2024-09-17T15:13:39.586 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T15:13:39.603 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:13:39.684 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} email: skanta@redhat.com first_in_suite: false job_id: '7908954' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMBWmuU3HKI6CDb3bSTo/lZhNv8pK3BRB/0FURB4+iA/9DWZUb53kN9dvfjLK8akrDGKkzocCqWX2MvMs5s8VF4= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWyTpwhRg7hYyYN9yp/NQF+jvVaVUL7QTWgzWVhIsFM/9TkdXxuwjsRhq3i8BtRT0DYFT9ydkHLDxkepw8wIKY= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T15:13:39.684 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T15:13:39.685 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T15:13:39.685 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T15:13:39.687 INFO:teuthology.task.internal:Checking packages... 2024-09-17T15:13:39.707 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T15:13:39.707 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T15:13:39.708 INFO:teuthology.packaging:ref: None 2024-09-17T15:13:39.708 INFO:teuthology.packaging:tag: None 2024-09-17T15:13:39.708 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:13:39.708 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:13:39.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T15:13:39.854 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975-g1889a165-1jammy 2024-09-17T15:13:39.856 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T15:13:39.863 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T15:13:39.863 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T15:13:39.895 INFO:teuthology.task.internal:Saving configuration 2024-09-17T15:13:39.909 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T15:13:39.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T15:13:39.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:03:43.266067', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPPZl1kxFhhCN3Qp/imzkjEppETjMhivFXSLR/f4D6mF'} 2024-09-17T15:13:39.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 15:03:43.264695', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFsWk/d9fY07dCrhEINLsyE78KnMsudPYRw46sg803e1'} 2024-09-17T15:13:39.978 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T15:13:39.984 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-17T15:13:39.984 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-17T15:13:39.984 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T15:13:40.062 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcb3429bbe0>, signals=[15]) 2024-09-17T15:13:40.062 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T15:13:40.068 INFO:teuthology.task.internal:Opening connections... 2024-09-17T15:13:40.068 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:13:40.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:13:40.149 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:13:40.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:13:40.230 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T15:13:40.237 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-09-17T15:13:40.242 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-09-17T15:13:40.243 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Ubuntu" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="22.04" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_CODENAME=jammy 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:ID=ubuntu 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE=debian 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-17T15:13:40.290 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-17T15:13:40.291 INFO:teuthology.orchestra.run.smithi086.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-17T15:13:40.291 INFO:teuthology.orchestra.run.smithi086.stdout:UBUNTU_CODENAME=jammy 2024-09-17T15:13:40.291 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-09-17T15:13:40.319 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-09-17T15:13:40.325 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-09-17T15:13:40.325 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-09-17T15:13:40.372 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-17T15:13:40.372 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Ubuntu" 2024-09-17T15:13:40.372 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="22.04" 2024-09-17T15:13:40.372 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_CODENAME=jammy 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:ID=ubuntu 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE=debian 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-17T15:13:40.373 INFO:teuthology.orchestra.run.smithi203.stdout:UBUNTU_CODENAME=jammy 2024-09-17T15:13:40.373 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-09-17T15:13:40.402 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T15:13:40.411 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T15:13:40.437 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T15:13:40.437 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:13:40.439 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-09-17T15:13:40.444 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T15:13:40.470 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T15:13:40.470 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:13:40.487 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-09-17T15:13:40.502 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T15:13:40.557 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T15:13:40.571 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T15:13:40.571 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T15:13:40.571 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:13:40.572 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T15:13:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:13:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:13:40.577 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-17T15:13:40.577 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-09-17T15:13:40.623 INFO:teuthology.orchestra.run.smithi086.stdout:5.15.0-56-generic 2024-09-17T15:13:40.623 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.15.0-56-generic 2024-09-17T15:13:40.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get clean 2024-09-17T15:13:40.625 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-56-generic 2024-09-17T15:13:40.625 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.15.0-56-generic 2024-09-17T15:13:40.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-09-17T15:13:40.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-17T15:13:40.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get update 2024-09-17T15:13:40.899 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:13:40.899 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-17T15:13:40.900 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-17T15:13:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-17T15:13:41.032 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:13:41.054 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-17T15:13:41.102 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-17T15:13:41.133 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-17T15:13:41.202 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-17T15:13:41.207 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-17T15:13:41.240 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-17T15:13:41.243 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-17T15:13:41.243 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-17T15:13:41.262 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-17T15:13:41.263 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-17T15:13:41.265 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-17T15:13:41.266 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [903 kB] 2024-09-17T15:13:41.269 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-17T15:13:41.271 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [628 kB] 2024-09-17T15:13:41.275 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-17T15:13:41.280 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-17T15:13:41.296 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-17T15:13:41.296 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-17T15:13:41.296 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-17T15:13:41.297 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-17T15:13:41.297 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-17T15:13:41.305 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-17T15:13:41.308 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-17T15:13:41.308 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-17T15:13:41.308 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-17T15:13:41.326 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-17T15:13:41.329 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-17T15:13:41.329 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-17T15:13:41.333 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-17T15:13:41.340 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-17T15:13:41.342 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-17T15:13:41.342 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-17T15:13:41.343 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-17T15:13:41.343 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-17T15:13:41.358 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-17T15:13:41.359 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-17T15:13:41.359 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-17T15:13:41.359 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-17T15:13:41.376 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-17T15:13:41.376 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-17T15:13:41.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-17T15:13:41.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-17T15:13:41.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-17T15:13:41.384 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-17T15:13:41.394 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-17T15:13:41.399 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-17T15:13:41.400 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-17T15:13:41.401 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-17T15:13:41.437 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-17T15:13:41.445 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-17T15:13:41.445 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [628 kB] 2024-09-17T15:13:41.448 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [903 kB] 2024-09-17T15:13:41.456 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-17T15:13:41.458 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-17T15:13:41.458 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-17T15:13:41.458 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-17T15:13:41.481 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-17T15:13:41.481 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-17T15:13:41.481 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-17T15:13:41.597 INFO:teuthology.orchestra.run.smithi086.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-17T15:13:41.793 INFO:teuthology.orchestra.run.smithi086.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-17T15:13:41.955 INFO:teuthology.orchestra.run.smithi086.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-17T15:13:41.973 INFO:teuthology.orchestra.run.smithi086.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-17T15:13:41.973 INFO:teuthology.orchestra.run.smithi086.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-17T15:13:42.045 INFO:teuthology.orchestra.run.smithi086.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-17T15:13:42.046 INFO:teuthology.orchestra.run.smithi086.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-17T15:13:42.068 INFO:teuthology.orchestra.run.smithi086.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-17T15:13:42.068 INFO:teuthology.orchestra.run.smithi086.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-17T15:13:42.089 INFO:teuthology.orchestra.run.smithi086.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-17T15:13:42.107 INFO:teuthology.orchestra.run.smithi086.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-17T15:13:42.112 INFO:teuthology.orchestra.run.smithi086.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-17T15:13:42.124 INFO:teuthology.orchestra.run.smithi086.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-17T15:13:42.125 INFO:teuthology.orchestra.run.smithi086.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-17T15:13:42.194 INFO:teuthology.orchestra.run.smithi086.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-17T15:13:42.195 INFO:teuthology.orchestra.run.smithi086.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-17T15:13:42.195 INFO:teuthology.orchestra.run.smithi086.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-17T15:13:42.196 INFO:teuthology.orchestra.run.smithi086.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-17T15:13:42.198 INFO:teuthology.orchestra.run.smithi086.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-17T15:13:42.206 INFO:teuthology.orchestra.run.smithi086.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-17T15:13:42.207 INFO:teuthology.orchestra.run.smithi086.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-17T15:13:42.208 INFO:teuthology.orchestra.run.smithi086.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-17T15:13:42.287 INFO:teuthology.orchestra.run.smithi086.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-17T15:13:42.288 INFO:teuthology.orchestra.run.smithi086.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-17T15:13:47.604 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 16.2 MB in 2s (7,691 kB/s) 2024-09-17T15:13:47.801 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 16.2 MB in 2s (6,986 kB/s) 2024-09-17T15:13:49.077 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:13:49.095 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T15:13:49.157 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:13:49.159 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:13:49.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T15:13:49.236 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:13:49.366 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-17T15:13:49.367 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-17T15:13:49.448 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2024-09-17T15:13:49.448 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2024-09-17T15:13:49.545 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:13:49.545 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:13:49.545 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:13:49.546 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:13:49.546 INFO:teuthology.orchestra.run.smithi203.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-17T15:13:49.546 INFO:teuthology.orchestra.run.smithi203.stdout: python2.7-minimal 2024-09-17T15:13:49.546 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:13:49.547 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-09-17T15:13:49.547 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T15:13:49.547 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-17T15:13:49.547 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-17T15:13:49.549 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-09-17T15:13:49.549 INFO:teuthology.orchestra.run.smithi203.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-17T15:13:49.549 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-09-17T15:13:49.549 INFO:teuthology.orchestra.run.smithi203.stdout: thermald 2024-09-17T15:13:49.598 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-09-17T15:13:49.598 INFO:teuthology.orchestra.run.smithi203.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T15:13:49.598 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-17T15:13:49.599 INFO:teuthology.orchestra.run.smithi203.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-17T15:13:49.600 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages will be upgraded: 2024-09-17T15:13:49.600 INFO:teuthology.orchestra.run.smithi203.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-17T15:13:49.619 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:13:49.619 INFO:teuthology.orchestra.run.smithi086.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:13:49.619 INFO:teuthology.orchestra.run.smithi086.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:13:49.619 INFO:teuthology.orchestra.run.smithi086.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:13:49.620 INFO:teuthology.orchestra.run.smithi086.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-17T15:13:49.620 INFO:teuthology.orchestra.run.smithi086.stdout: python2.7-minimal 2024-09-17T15:13:49.620 INFO:teuthology.orchestra.run.smithi086.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:13:49.621 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2024-09-17T15:13:49.621 INFO:teuthology.orchestra.run.smithi086.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T15:13:49.621 INFO:teuthology.orchestra.run.smithi086.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-17T15:13:49.621 INFO:teuthology.orchestra.run.smithi086.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-17T15:13:49.622 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2024-09-17T15:13:49.623 INFO:teuthology.orchestra.run.smithi086.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-17T15:13:49.623 INFO:teuthology.orchestra.run.smithi086.stdout:Recommended packages: 2024-09-17T15:13:49.623 INFO:teuthology.orchestra.run.smithi086.stdout: thermald 2024-09-17T15:13:49.670 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2024-09-17T15:13:49.670 INFO:teuthology.orchestra.run.smithi086.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T15:13:49.670 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-17T15:13:49.671 INFO:teuthology.orchestra.run.smithi086.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-17T15:13:49.672 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages will be upgraded: 2024-09-17T15:13:49.672 INFO:teuthology.orchestra.run.smithi086.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-17T15:13:49.685 INFO:teuthology.orchestra.run.smithi203.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T15:13:49.685 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 113 MB of archives. 2024-09-17T15:13:49.685 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-17T15:13:49.685 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-17T15:13:49.851 INFO:teuthology.orchestra.run.smithi086.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T15:13:49.851 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 113 MB of archives. 2024-09-17T15:13:49.851 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-17T15:13:49.851 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-17T15:13:50.101 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-17T15:13:50.188 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-17T15:13:50.998 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-17T15:13:50.998 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-17T15:13:50.998 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-17T15:13:51.120 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-17T15:13:51.141 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-17T15:13:51.342 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-17T15:13:51.601 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 113 MB in 2s (70.6 MB/s) 2024-09-17T15:13:51.691 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-17T15:13:51.803 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-17T15:13:53.873 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-17T15:13:53.878 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:13:53.911 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-17T15:13:53.911 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-17T15:13:53.911 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-17T15:13:53.967 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:13:54.292 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-17T15:13:54.338 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-17T15:13:54.748 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 113 MB in 5s (24.3 MB/s) 2024-09-17T15:13:54.923 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-17T15:13:56.160 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-17T15:13:56.181 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:13:56.277 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:13:56.729 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-17T15:13:56.750 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:13:56.788 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:13:56.880 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-17T15:13:56.884 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:13:56.962 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:13:59.630 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-17T15:13:59.651 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:13:59.742 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:00.218 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-17T15:14:00.238 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:14:00.276 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:04.051 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:04.178 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:04.536 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:04.699 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:05.019 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-17T15:14:05.030 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-17T15:14:05.069 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T15:14:06.351 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:06.484 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:06.766 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:06.896 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:07.199 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-17T15:14:07.221 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-17T15:14:07.265 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T15:14:10.818 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-17T15:14:10.855 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:14:10.892 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:12.535 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-17T15:14:12.571 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T15:14:12.609 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:13.493 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:13.593 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:13.926 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T15:14:14.048 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:14.166 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-17T15:14:14.295 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:15.604 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:15.604 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:15.604 INFO:teuthology.orchestra.run.smithi203.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:15.604 INFO:teuthology.orchestra.run.smithi203.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:15.756 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:15.844 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T15:14:15.961 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T15:14:16.286 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T15:14:17.112 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:17.459 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-17T15:14:17.588 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:17.817 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-17T15:14:18.492 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-17T15:14:18.619 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:18.917 INFO:teuthology.orchestra.run.smithi086.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:18.917 INFO:teuthology.orchestra.run.smithi086.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:18.917 INFO:teuthology.orchestra.run.smithi086.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:18.917 INFO:teuthology.orchestra.run.smithi086.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:19.072 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:20.102 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-17T15:14:20.102 INFO:teuthology.orchestra.run.smithi203.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:20.490 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-17T15:14:20.616 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-17T15:14:20.742 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:22.169 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T15:14:22.230 INFO:teuthology.orchestra.run.smithi086.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-17T15:14:22.230 INFO:teuthology.orchestra.run.smithi086.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:34.241 INFO:teuthology.orchestra.run.smithi203.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-17T15:14:34.241 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub' 2024-09-17T15:14:34.254 INFO:teuthology.orchestra.run.smithi203.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T15:14:34.273 INFO:teuthology.orchestra.run.smithi203.stdout:Generating grub configuration file ... 2024-09-17T15:14:34.546 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:34.561 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:34.889 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:34.896 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:34.972 INFO:teuthology.orchestra.run.smithi203.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T15:14:34.977 INFO:teuthology.orchestra.run.smithi203.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T15:14:35.136 INFO:teuthology.orchestra.run.smithi203.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T15:14:35.136 INFO:teuthology.orchestra.run.smithi203.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T15:14:35.136 INFO:teuthology.orchestra.run.smithi203.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T15:14:35.181 INFO:teuthology.orchestra.run.smithi203.stdout:done 2024-09-17T15:14:35.706 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-09-17T15:14:35.722 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-09-17T15:14:35.722 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-09-17T15:14:35.723 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.122.122 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-09-17T15:14:35.724 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-09-17T15:14:35.724 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-17T15:14:35.724 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:14:35.724 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:14:35.725 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T15:14:35.725 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:14:35.725 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get clean 2024-09-17T15:14:35.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-17T15:14:36.017 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T15:14:36.049 INFO:teuthology.orchestra.run.smithi086.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-17T15:14:36.050 INFO:teuthology.orchestra.run.smithi086.stdout:Sourcing file `/etc/default/grub' 2024-09-17T15:14:36.068 INFO:teuthology.orchestra.run.smithi086.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T15:14:36.074 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:14:36.085 INFO:teuthology.orchestra.run.smithi086.stdout:Generating grub configuration file ... 2024-09-17T15:14:36.091 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T15:14:36.108 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T15:14:36.345 INFO:teuthology.orchestra.run.smithi086.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:36.364 INFO:teuthology.orchestra.run.smithi086.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:36.651 INFO:teuthology.orchestra.run.smithi086.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:36.657 INFO:teuthology.orchestra.run.smithi086.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:36.733 INFO:teuthology.orchestra.run.smithi086.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T15:14:36.738 INFO:teuthology.orchestra.run.smithi086.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T15:14:36.898 INFO:teuthology.orchestra.run.smithi086.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T15:14:36.899 INFO:teuthology.orchestra.run.smithi086.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T15:14:36.899 INFO:teuthology.orchestra.run.smithi086.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T15:14:36.935 INFO:teuthology.orchestra.run.smithi086.stdout:done 2024-09-17T15:14:37.376 DEBUG:teuthology.orchestra.run.smithi086:> dpkg -s linux-image-generic 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Package: linux-image-generic 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Status: install ok installed 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Priority: optional 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Section: kernel 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Installed-Size: 21 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T15:14:37.399 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: amd64 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Source: linux-meta 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Version: 5.15.0.122.122 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Recommends: thermald 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout:Description: Generic Linux kernel image 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: This package will always depend on the latest generic kernel image 2024-09-17T15:14:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: available. 2024-09-17T15:14:37.400 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-17T15:14:37.400 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T15:14:37.400 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T15:14:37.401 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T15:14:37.401 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T15:14:37.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get clean 2024-09-17T15:14:37.474 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:14:37.493 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T15:14:37.556 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:14:37.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get update 2024-09-17T15:14:37.768 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-17T15:14:37.769 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-17T15:14:37.834 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:14:37.835 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T15:14:37.902 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T15:14:37.960 INFO:teuthology.orchestra.run.smithi203.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-17T15:14:37.960 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:14:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:14:37.960 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:14:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:14:37.961 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T15:14:37.962 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T15:14:37.962 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:14:37.983 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T15:14:38.004 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T15:14:38.005 DEBUG:teuthology.orchestra.run.smithi203:> dpkg -s linux-image-generic 2024-09-17T15:14:38.017 INFO:teuthology.orchestra.run.smithi203.stdout:Package: linux-image-generic 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Status: install ok installed 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Priority: optional 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Section: kernel 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Installed-Size: 21 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: amd64 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Source: linux-meta 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Version: 5.15.0.122.122 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Recommends: thermald 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout:Description: Generic Linux kernel image 2024-09-17T15:14:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: This package will always depend on the latest generic kernel image 2024-09-17T15:14:38.019 INFO:teuthology.orchestra.run.smithi203.stdout: available. 2024-09-17T15:14:38.019 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-17T15:14:38.063 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.mRYdXYh78O 2024-09-17T15:14:38.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub/grub.cfg /tmp/tmp.mRYdXYh78O 2024-09-17T15:14:38.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.mRYdXYh78O 2024-09-17T15:14:38.235 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.mRYdXYh78O is 10KB 2024-09-17T15:14:38.247 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.mRYdXYh78O 2024-09-17T15:14:38.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-17T15:14:38.307 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:14:38.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-17T15:14:38.307 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-17T15:14:38.374 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-17T15:14:38.374 DEBUG:teuthology.orchestra.run.smithi203:> sudo update-grub 2024-09-17T15:14:39.291 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:14:39.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T15:14:39.361 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:14:39.548 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2024-09-17T15:14:39.549 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2024-09-17T15:14:39.719 INFO:teuthology.orchestra.run.smithi086.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-17T15:14:39.719 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T15:14:39.720 INFO:teuthology.orchestra.run.smithi086.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:14:39.748 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub' 2024-09-17T15:14:39.749 INFO:teuthology.orchestra.run.smithi203.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T15:14:39.753 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-09-17T15:14:39.758 INFO:teuthology.orchestra.run.smithi086.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T15:14:39.761 DEBUG:teuthology.orchestra.run.smithi086:> dpkg -s linux-image-generic 2024-09-17T15:14:39.815 INFO:teuthology.orchestra.run.smithi086.stdout:Package: linux-image-generic 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Status: install ok installed 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Priority: optional 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Section: kernel 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Installed-Size: 21 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: amd64 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Source: linux-meta 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Version: 5.15.0.122.122 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T15:14:39.816 INFO:teuthology.orchestra.run.smithi086.stdout:Recommends: thermald 2024-09-17T15:14:39.817 INFO:teuthology.orchestra.run.smithi086.stdout:Description: Generic Linux kernel image 2024-09-17T15:14:39.817 INFO:teuthology.orchestra.run.smithi086.stdout: This package will always depend on the latest generic kernel image 2024-09-17T15:14:39.817 INFO:teuthology.orchestra.run.smithi086.stdout: available. 2024-09-17T15:14:39.817 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-09-17T15:14:39.861 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.2l7D7Q412s 2024-09-17T15:14:39.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub/grub.cfg /tmp/tmp.2l7D7Q412s 2024-09-17T15:14:39.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.2l7D7Q412s 2024-09-17T15:14:39.931 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:39.937 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:40.037 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.2l7D7Q412s is 10KB 2024-09-17T15:14:40.053 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.2l7D7Q412s 2024-09-17T15:14:40.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-17T15:14:40.113 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:14:40.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-17T15:14:40.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-17T15:14:40.181 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-17T15:14:40.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo update-grub 2024-09-17T15:14:40.216 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:40.221 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:40.259 INFO:teuthology.orchestra.run.smithi203.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T15:14:40.264 INFO:teuthology.orchestra.run.smithi203.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T15:14:40.396 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T15:14:40.396 INFO:teuthology.orchestra.run.smithi203.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T15:14:40.396 INFO:teuthology.orchestra.run.smithi203.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T15:14:40.413 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-09-17T15:14:40.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-09-17T15:14:41.350 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub' 2024-09-17T15:14:41.350 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T15:14:41.352 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-09-17T15:14:41.542 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T15:14:41.549 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T15:14:41.780 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T15:14:41.785 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T15:14:41.829 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T15:14:41.834 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T15:14:41.985 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T15:14:41.985 INFO:teuthology.orchestra.run.smithi086.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T15:14:41.985 INFO:teuthology.orchestra.run.smithi086.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T15:14:42.024 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-09-17T15:14:42.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-09-17T15:15:10.445 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T15:15:10.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:15:10.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:15:12.037 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T15:15:12.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:15:12.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:15:28.957 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-17T15:15:30.497 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-09-17T15:15:37.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:15:37.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:15:39.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:15:39.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:15:41.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-17T15:15:42.557 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-09-17T15:15:53.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:15:53.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:15:54.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:15:54.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:16:11.453 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-09-17T15:16:12.993 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-09-17T15:16:26.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:16:26.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:16:27.264 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-17T15:16:27.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:16:27.994 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:16:28.128 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:16:28.128 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-17T15:16:28.128 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-09-17T15:16:28.178 INFO:teuthology.orchestra.run.smithi203.stdout:5.15.0-122-generic 2024-09-17T15:16:28.178 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-17T15:16:28.178 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:16:28.178 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-17T15:16:28.363 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-09-17T15:16:29.179 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T15:16:29.179 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:16:29.355 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-09-17T15:16:29.377 DEBUG:teuthology.parallel:result is None 2024-09-17T15:16:29.515 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:16:29.515 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-17T15:16:29.516 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-09-17T15:16:29.520 INFO:teuthology.orchestra.run.smithi086.stdout:5.15.0-122-generic 2024-09-17T15:16:29.521 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-17T15:16:29.521 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T15:16:29.521 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-17T15:16:30.522 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T15:16:30.522 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T15:16:30.595 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-09-17T15:16:30.624 DEBUG:teuthology.parallel:result is None 2024-09-17T15:16:30.624 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T15:16:30.631 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T15:16:30.632 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:16:30.635 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T15:16:30.644 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T15:16:30.666 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T15:16:30.700 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T15:16:30.700 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:16:30.703 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T15:16:30.725 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T15:16:30.732 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T15:16:30.733 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:16:30.764 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T15:16:30.785 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:16:30.793 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:16:30.795 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:16:30.804 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T15:16:30.805 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T15:16:30.812 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T15:16:30.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:16:30.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T15:16:30.868 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T15:16:30.878 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T15:16:30.878 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:16:30.912 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T15:16:30.917 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:16:30.963 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:16:31.011 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:16:31.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:16:31.076 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T15:16:31.082 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T15:16:31.136 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:16:31.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T15:16:31.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-09-17T15:16:31.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-09-17T15:16:31.345 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T15:16:31.352 INFO:teuthology.task.internal:Starting timer... 2024-09-17T15:16:31.353 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T15:16:31.388 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T15:16:31.419 DEBUG:teuthology.task.selinux:Excluding smithi086: OS 'ubuntu' does not support SELinux 2024-09-17T15:16:31.419 DEBUG:teuthology.task.selinux:Excluding smithi203: OS 'ubuntu' does not support SELinux 2024-09-17T15:16:31.419 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T15:16:31.419 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-17T15:16:31.419 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T15:16:31.419 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T15:16:31.453 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T15:16:31.470 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T15:16:31.472 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T15:24:30.180 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-09-17T15:24:30.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:24:30.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:24:30.265 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-09-17T15:24:30.342 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-09-17T15:24:30.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:24:30.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T15:24:30.425 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-09-17T15:24:30.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-09-17T15:24:30.497 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T15:24:30.507 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T15:24:30.507 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:24:30.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:24:30.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T15:24:30.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:24:30.536 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-17T15:24:30.536 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Command line: ntpd -gq 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: ---------------------------------------------------- 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: ntp-4 is maintained by Network Time Foundation, 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: corporation. Support and training for ntp-4 are 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: available at https://www.nwtime.org/support 2024-09-17T15:24:30.537 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: ---------------------------------------------------- 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: proto: precision = 0.051 usec (-24) 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: basedate set to 2022-02-04 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: gps base set to 2022-02-06 (week 2196) 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T15:24:30.538 INFO:teuthology.orchestra.run.smithi086.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-17T15:24:30.539 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: restrict ::: KOD does nothing without LIMITED. 2024-09-17T15:24:30.539 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen and drop on 0 v6wildcard [::]:123 2024-09-17T15:24:30.539 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-17T15:24:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen normally on 2 lo 127.0.0.1:123 2024-09-17T15:24:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen normally on 3 enp3s0f1 172.21.15.86:123 2024-09-17T15:24:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen normally on 4 lo [::1]:123 2024-09-17T15:24:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d809%5]:123 2024-09-17T15:24:30.540 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:30 ntpd[18488]: Listening on routing socket on fd #22 for interface updates 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Command line: ntpd -gq 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: ---------------------------------------------------- 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: ntp-4 is maintained by Network Time Foundation, 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: corporation. Support and training for ntp-4 are 2024-09-17T15:24:30.566 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: available at https://www.nwtime.org/support 2024-09-17T15:24:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: ---------------------------------------------------- 2024-09-17T15:24:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: proto: precision = 0.063 usec (-24) 2024-09-17T15:24:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: basedate set to 2022-02-04 2024-09-17T15:24:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: gps base set to 2022-02-06 (week 2196) 2024-09-17T15:24:30.568 INFO:teuthology.orchestra.run.smithi203.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T15:24:30.568 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T15:24:30.568 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: restrict ::: KOD does nothing without LIMITED. 2024-09-17T15:24:30.568 INFO:teuthology.orchestra.run.smithi203.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-17T15:24:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen and drop on 0 v6wildcard [::]:123 2024-09-17T15:24:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-17T15:24:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen normally on 2 lo 127.0.0.1:123 2024-09-17T15:24:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen normally on 3 enp3s0f1 172.21.15.203:123 2024-09-17T15:24:30.569 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen normally on 4 lo [::1]:123 2024-09-17T15:24:30.570 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4a23%5]:123 2024-09-17T15:24:30.570 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:30 ntpd[18461]: Listening on routing socket on fd #22 for interface updates 2024-09-17T15:24:31.537 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:31.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:31 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:31.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:31 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:32.537 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:32 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:32.537 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:32 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.540 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:33 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:33.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:33 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.540 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.540 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.540 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:35 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:35 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:37.538 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:37.539 INFO:teuthology.orchestra.run.smithi086.stdout:17 Sep 15:24:37 ntpd[18488]: ntpd: time slew -0.000905 s 2024-09-17T15:24:37.539 INFO:teuthology.orchestra.run.smithi086.stdout:ntpd: time slew -0.000905s 2024-09-17T15:24:37.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-17T15:24:37.539 INFO:teuthology.orchestra.run.smithi086.stderr:17 Sep 15:24:37 ntpd[18488]: can't open /var/log/ntpstats/loopstats.20240917: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T15:24:37.567 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T15:24:37.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T15:24:37.568 INFO:teuthology.orchestra.run.smithi203.stdout:17 Sep 15:24:37 ntpd[18461]: ntpd: time slew -0.000965 s 2024-09-17T15:24:37.568 INFO:teuthology.orchestra.run.smithi203.stdout:ntpd: time slew -0.000965s 2024-09-17T15:24:37.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-17T15:24:37.568 INFO:teuthology.orchestra.run.smithi203.stderr:17 Sep 15:24:37 ntpd[18461]: can't open /var/log/ntpstats/loopstats.20240917: Permission denied 2024-09-17T15:24:37.591 INFO:teuthology.orchestra.run.smithi086.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T15:24:37.592 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================== 2024-09-17T15:24:37.592 INFO:teuthology.orchestra.run.smithi086.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.592 INFO:teuthology.orchestra.run.smithi086.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.592 INFO:teuthology.orchestra.run.smithi086.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.592 INFO:teuthology.orchestra.run.smithi086.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.618 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T15:24:37.618 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-09-17T15:24:37.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.619 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T15:24:37.620 INFO:teuthology.run_tasks:Running task install... 2024-09-17T15:24:37.628 DEBUG:teuthology.task.install:project ceph 2024-09-17T15:24:37.628 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:24:37.628 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T15:24:37.628 INFO:teuthology.task.install:Using flavor: default 2024-09-17T15:24:37.633 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T15:24:37.633 INFO:teuthology.task.install:extra packages: [] 2024-09-17T15:24:37.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-key list | grep Ceph 2024-09-17T15:24:37.634 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-key list | grep Ceph 2024-09-17T15:24:37.691 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-17T15:24:37.717 INFO:teuthology.orchestra.run.smithi203.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-17T15:24:37.720 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-17T15:24:37.720 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph.com (release key) 2024-09-17T15:24:37.721 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-17T15:24:37.722 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:24:37.742 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-17T15:24:37.742 INFO:teuthology.orchestra.run.smithi203.stdout:uid [ unknown] Ceph.com (release key) 2024-09-17T15:24:37.743 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-17T15:24:37.743 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:24:37.880 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default/ 2024-09-17T15:24:37.880 INFO:teuthology.task.install.deb:Package version is 19.3.0-4975-g1889a165-1jammy 2024-09-17T15:24:37.891 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default/ 2024-09-17T15:24:37.891 INFO:teuthology.task.install.deb:Package version is 19.3.0-4975-g1889a165-1jammy 2024-09-17T15:24:38.162 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:24:38.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-17T15:24:38.170 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:24:38.171 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-17T15:24:38.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get update 2024-09-17T15:24:38.184 DEBUG:teuthology.orchestra.run.smithi203:> sudo apt-get update 2024-09-17T15:24:38.341 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:24:38.348 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T15:24:38.352 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T15:24:38.359 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T15:24:38.370 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T15:24:38.376 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T15:24:38.464 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T15:24:38.471 INFO:teuthology.orchestra.run.smithi203.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T15:24:38.575 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy InRelease 2024-09-17T15:24:38.579 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy InRelease 2024-09-17T15:24:38.642 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-17T15:24:38.646 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-17T15:24:38.711 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-17T15:24:38.715 INFO:teuthology.orchestra.run.smithi203.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-17T15:24:38.779 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-09-17T15:24:38.783 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-17T15:24:38.855 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-17T15:24:38.862 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-09-17T15:24:38.946 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 28.6 kB in 1s (50.0 kB/s) 2024-09-17T15:24:38.952 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 28.6 kB in 1s (49.5 kB/s) 2024-09-17T15:24:39.952 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:24:39.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4975-g1889a165-1jammy cephadm=19.3.0-4975-g1889a165-1jammy ceph-mds=19.3.0-4975-g1889a165-1jammy ceph-mgr=19.3.0-4975-g1889a165-1jammy ceph-common=19.3.0-4975-g1889a165-1jammy ceph-fuse=19.3.0-4975-g1889a165-1jammy ceph-test=19.3.0-4975-g1889a165-1jammy ceph-volume=19.3.0-4975-g1889a165-1jammy radosgw=19.3.0-4975-g1889a165-1jammy python3-rados=19.3.0-4975-g1889a165-1jammy python3-rgw=19.3.0-4975-g1889a165-1jammy python3-cephfs=19.3.0-4975-g1889a165-1jammy python3-rbd=19.3.0-4975-g1889a165-1jammy libcephfs2=19.3.0-4975-g1889a165-1jammy libcephfs-dev=19.3.0-4975-g1889a165-1jammy librados2=19.3.0-4975-g1889a165-1jammy librbd1=19.3.0-4975-g1889a165-1jammy rbd-fuse=19.3.0-4975-g1889a165-1jammy 2024-09-17T15:24:40.002 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:24:40.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4975-g1889a165-1jammy cephadm=19.3.0-4975-g1889a165-1jammy ceph-mds=19.3.0-4975-g1889a165-1jammy ceph-mgr=19.3.0-4975-g1889a165-1jammy ceph-common=19.3.0-4975-g1889a165-1jammy ceph-fuse=19.3.0-4975-g1889a165-1jammy ceph-test=19.3.0-4975-g1889a165-1jammy ceph-volume=19.3.0-4975-g1889a165-1jammy radosgw=19.3.0-4975-g1889a165-1jammy python3-rados=19.3.0-4975-g1889a165-1jammy python3-rgw=19.3.0-4975-g1889a165-1jammy python3-cephfs=19.3.0-4975-g1889a165-1jammy python3-rbd=19.3.0-4975-g1889a165-1jammy libcephfs2=19.3.0-4975-g1889a165-1jammy libcephfs-dev=19.3.0-4975-g1889a165-1jammy librados2=19.3.0-4975-g1889a165-1jammy librbd1=19.3.0-4975-g1889a165-1jammy rbd-fuse=19.3.0-4975-g1889a165-1jammy 2024-09-17T15:24:40.031 INFO:teuthology.orchestra.run.smithi203.stdout:Reading package lists... 2024-09-17T15:24:40.074 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-09-17T15:24:40.246 INFO:teuthology.orchestra.run.smithi203.stdout:Building dependency tree... 2024-09-17T15:24:40.247 INFO:teuthology.orchestra.run.smithi203.stdout:Reading state information... 2024-09-17T15:24:40.260 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2024-09-17T15:24:40.261 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2024-09-17T15:24:40.441 INFO:teuthology.orchestra.run.smithi203.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:24:40.441 INFO:teuthology.orchestra.run.smithi203.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:24:40.441 INFO:teuthology.orchestra.run.smithi203.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:24:40.442 INFO:teuthology.orchestra.run.smithi203.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:24:40.442 INFO:teuthology.orchestra.run.smithi203.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T15:24:40.442 INFO:teuthology.orchestra.run.smithi203.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T15:24:40.442 INFO:teuthology.orchestra.run.smithi203.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout:The following additional packages will be installed: 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-17T15:24:40.443 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-17T15:24:40.444 INFO:teuthology.orchestra.run.smithi203.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-17T15:24:40.445 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-17T15:24:40.446 INFO:teuthology.orchestra.run.smithi203.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-17T15:24:40.446 INFO:teuthology.orchestra.run.smithi203.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-17T15:24:40.446 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-17T15:24:40.446 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-17T15:24:40.446 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet zip 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout:Suggested packages: 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-17T15:24:40.447 INFO:teuthology.orchestra.run.smithi203.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-17T15:24:40.448 INFO:teuthology.orchestra.run.smithi203.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-17T15:24:40.448 INFO:teuthology.orchestra.run.smithi203.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-17T15:24:40.448 INFO:teuthology.orchestra.run.smithi203.stdout:Recommended packages: 2024-09-17T15:24:40.448 INFO:teuthology.orchestra.run.smithi203.stdout: btrfs-tools 2024-09-17T15:24:40.449 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-17T15:24:40.450 INFO:teuthology.orchestra.run.smithi086.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-17T15:24:40.451 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet zip 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-17T15:24:40.452 INFO:teuthology.orchestra.run.smithi086.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-17T15:24:40.453 INFO:teuthology.orchestra.run.smithi086.stdout:Recommended packages: 2024-09-17T15:24:40.453 INFO:teuthology.orchestra.run.smithi086.stdout: btrfs-tools 2024-09-17T15:24:40.548 INFO:teuthology.orchestra.run.smithi203.stdout:The following NEW packages will be installed: 2024-09-17T15:24:40.548 INFO:teuthology.orchestra.run.smithi203.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-17T15:24:40.548 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-17T15:24:40.549 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-17T15:24:40.549 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-17T15:24:40.549 INFO:teuthology.orchestra.run.smithi203.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-17T15:24:40.549 INFO:teuthology.orchestra.run.smithi203.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-17T15:24:40.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-17T15:24:40.551 INFO:teuthology.orchestra.run.smithi203.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-17T15:24:40.552 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2024-09-17T15:24:40.552 INFO:teuthology.orchestra.run.smithi086.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-17T15:24:40.553 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-17T15:24:40.553 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-17T15:24:40.553 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-17T15:24:40.553 INFO:teuthology.orchestra.run.smithi086.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-17T15:24:40.553 INFO:teuthology.orchestra.run.smithi086.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-17T15:24:40.554 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-17T15:24:40.555 INFO:teuthology.orchestra.run.smithi086.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-17T15:24:40.610 INFO:teuthology.orchestra.run.smithi203.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-17T15:24:40.610 INFO:teuthology.orchestra.run.smithi203.stdout:Need to get 244 MB of archives. 2024-09-17T15:24:40.610 INFO:teuthology.orchestra.run.smithi203.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-17T15:24:40.610 INFO:teuthology.orchestra.run.smithi203.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-17T15:24:40.613 INFO:teuthology.orchestra.run.smithi086.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-17T15:24:40.613 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 244 MB of archives. 2024-09-17T15:24:40.613 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-17T15:24:40.613 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-17T15:24:40.699 INFO:teuthology.orchestra.run.smithi203.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-17T15:24:40.701 INFO:teuthology.orchestra.run.smithi203.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-17T15:24:40.702 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-17T15:24:40.704 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-17T15:24:40.718 INFO:teuthology.orchestra.run.smithi203.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-17T15:24:40.721 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-17T15:24:40.778 INFO:teuthology.orchestra.run.smithi203.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-17T15:24:40.780 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-17T15:24:40.783 INFO:teuthology.orchestra.run.smithi203.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-17T15:24:40.785 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-17T15:24:40.795 INFO:teuthology.orchestra.run.smithi203.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-17T15:24:40.797 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-17T15:24:40.799 INFO:teuthology.orchestra.run.smithi203.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-17T15:24:40.800 INFO:teuthology.orchestra.run.smithi203.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-17T15:24:40.801 INFO:teuthology.orchestra.run.smithi203.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-17T15:24:40.801 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-17T15:24:40.802 INFO:teuthology.orchestra.run.smithi203.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-17T15:24:40.803 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-17T15:24:40.803 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-17T15:24:40.804 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-17T15:24:40.810 INFO:teuthology.orchestra.run.smithi203.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-17T15:24:40.812 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-17T15:24:40.813 INFO:teuthology.orchestra.run.smithi203.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-17T15:24:40.814 INFO:teuthology.orchestra.run.smithi203.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-17T15:24:40.815 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-17T15:24:40.817 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-17T15:24:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-17T15:24:40.817 INFO:teuthology.orchestra.run.smithi203.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-17T15:24:40.818 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-17T15:24:40.819 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-17T15:24:40.834 INFO:teuthology.orchestra.run.smithi203.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-17T15:24:40.835 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-17T15:24:40.837 INFO:teuthology.orchestra.run.smithi203.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-17T15:24:40.838 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-17T15:24:40.838 INFO:teuthology.orchestra.run.smithi203.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-17T15:24:40.839 INFO:teuthology.orchestra.run.smithi203.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-17T15:24:40.839 INFO:teuthology.orchestra.run.smithi203.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-17T15:24:40.840 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-17T15:24:40.840 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-17T15:24:40.841 INFO:teuthology.orchestra.run.smithi086.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-17T15:24:40.841 INFO:teuthology.orchestra.run.smithi203.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-17T15:24:40.841 INFO:teuthology.orchestra.run.smithi203.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-17T15:24:40.841 INFO:teuthology.orchestra.run.smithi203.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-17T15:24:40.842 INFO:teuthology.orchestra.run.smithi086.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-17T15:24:40.842 INFO:teuthology.orchestra.run.smithi086.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-17T15:24:40.842 INFO:teuthology.orchestra.run.smithi086.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-17T15:24:40.851 INFO:teuthology.orchestra.run.smithi203.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-17T15:24:40.852 INFO:teuthology.orchestra.run.smithi203.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-17T15:24:40.853 INFO:teuthology.orchestra.run.smithi086.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-17T15:24:40.853 INFO:teuthology.orchestra.run.smithi086.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-17T15:24:40.869 INFO:teuthology.orchestra.run.smithi203.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-17T15:24:40.869 INFO:teuthology.orchestra.run.smithi203.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-17T15:24:40.870 INFO:teuthology.orchestra.run.smithi086.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-17T15:24:40.871 INFO:teuthology.orchestra.run.smithi086.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-17T15:24:40.872 INFO:teuthology.orchestra.run.smithi203.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-17T15:24:40.872 INFO:teuthology.orchestra.run.smithi203.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-17T15:24:40.872 INFO:teuthology.orchestra.run.smithi203.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-17T15:24:40.873 INFO:teuthology.orchestra.run.smithi203.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-17T15:24:40.873 INFO:teuthology.orchestra.run.smithi086.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-17T15:24:40.873 INFO:teuthology.orchestra.run.smithi086.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-17T15:24:40.874 INFO:teuthology.orchestra.run.smithi203.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-17T15:24:40.874 INFO:teuthology.orchestra.run.smithi203.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-17T15:24:40.874 INFO:teuthology.orchestra.run.smithi086.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-17T15:24:40.875 INFO:teuthology.orchestra.run.smithi086.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-17T15:24:40.875 INFO:teuthology.orchestra.run.smithi086.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-17T15:24:40.876 INFO:teuthology.orchestra.run.smithi086.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-17T15:24:40.886 INFO:teuthology.orchestra.run.smithi203.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-17T15:24:40.887 INFO:teuthology.orchestra.run.smithi086.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-17T15:24:40.887 INFO:teuthology.orchestra.run.smithi203.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-17T15:24:40.888 INFO:teuthology.orchestra.run.smithi086.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-17T15:24:40.903 INFO:teuthology.orchestra.run.smithi203.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-17T15:24:40.904 INFO:teuthology.orchestra.run.smithi203.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-17T15:24:40.904 INFO:teuthology.orchestra.run.smithi086.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-17T15:24:40.905 INFO:teuthology.orchestra.run.smithi086.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-17T15:24:40.909 INFO:teuthology.orchestra.run.smithi203.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-17T15:24:40.909 INFO:teuthology.orchestra.run.smithi203.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-17T15:24:40.909 INFO:teuthology.orchestra.run.smithi203.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-17T15:24:40.910 INFO:teuthology.orchestra.run.smithi086.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-17T15:24:40.910 INFO:teuthology.orchestra.run.smithi086.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-17T15:24:40.911 INFO:teuthology.orchestra.run.smithi203.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-17T15:24:40.911 INFO:teuthology.orchestra.run.smithi086.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-17T15:24:40.911 INFO:teuthology.orchestra.run.smithi086.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-17T15:24:40.911 INFO:teuthology.orchestra.run.smithi203.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-17T15:24:40.912 INFO:teuthology.orchestra.run.smithi086.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-17T15:24:40.914 INFO:teuthology.orchestra.run.smithi203.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-17T15:24:40.914 INFO:teuthology.orchestra.run.smithi086.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-17T15:24:40.920 INFO:teuthology.orchestra.run.smithi203.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-17T15:24:40.921 INFO:teuthology.orchestra.run.smithi086.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-17T15:24:40.922 INFO:teuthology.orchestra.run.smithi203.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-17T15:24:40.922 INFO:teuthology.orchestra.run.smithi086.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-17T15:24:40.931 INFO:teuthology.orchestra.run.smithi203.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-17T15:24:40.932 INFO:teuthology.orchestra.run.smithi086.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-17T15:24:40.934 INFO:teuthology.orchestra.run.smithi203.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-17T15:24:40.934 INFO:teuthology.orchestra.run.smithi203.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-17T15:24:40.935 INFO:teuthology.orchestra.run.smithi086.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-17T15:24:40.935 INFO:teuthology.orchestra.run.smithi086.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-17T15:24:40.935 INFO:teuthology.orchestra.run.smithi203.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-17T15:24:40.936 INFO:teuthology.orchestra.run.smithi086.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-17T15:24:40.937 INFO:teuthology.orchestra.run.smithi203.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-17T15:24:40.938 INFO:teuthology.orchestra.run.smithi086.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-17T15:24:40.962 INFO:teuthology.orchestra.run.smithi203.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4975-g1889a165-1jammy [3,624 kB] 2024-09-17T15:24:40.967 INFO:teuthology.orchestra.run.smithi086.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4975-g1889a165-1jammy [3,624 kB] 2024-09-17T15:24:40.988 INFO:teuthology.orchestra.run.smithi203.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-17T15:24:40.989 INFO:teuthology.orchestra.run.smithi203.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-17T15:24:40.990 INFO:teuthology.orchestra.run.smithi086.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-17T15:24:40.991 INFO:teuthology.orchestra.run.smithi203.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-17T15:24:40.991 INFO:teuthology.orchestra.run.smithi203.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-17T15:24:40.991 INFO:teuthology.orchestra.run.smithi086.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-17T15:24:40.992 INFO:teuthology.orchestra.run.smithi203.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-17T15:24:40.992 INFO:teuthology.orchestra.run.smithi086.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-17T15:24:40.992 INFO:teuthology.orchestra.run.smithi086.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-17T15:24:40.993 INFO:teuthology.orchestra.run.smithi086.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-17T15:24:41.008 INFO:teuthology.orchestra.run.smithi203.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-17T15:24:41.010 INFO:teuthology.orchestra.run.smithi203.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-17T15:24:41.010 INFO:teuthology.orchestra.run.smithi203.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-17T15:24:41.010 INFO:teuthology.orchestra.run.smithi086.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-17T15:24:41.012 INFO:teuthology.orchestra.run.smithi086.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-17T15:24:41.012 INFO:teuthology.orchestra.run.smithi086.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-17T15:24:41.025 INFO:teuthology.orchestra.run.smithi203.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-17T15:24:41.025 INFO:teuthology.orchestra.run.smithi203.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-17T15:24:41.025 INFO:teuthology.orchestra.run.smithi203.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-17T15:24:41.026 INFO:teuthology.orchestra.run.smithi203.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-17T15:24:41.026 INFO:teuthology.orchestra.run.smithi203.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-17T15:24:41.027 INFO:teuthology.orchestra.run.smithi086.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-17T15:24:41.027 INFO:teuthology.orchestra.run.smithi086.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-17T15:24:41.027 INFO:teuthology.orchestra.run.smithi203.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-17T15:24:41.028 INFO:teuthology.orchestra.run.smithi086.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-17T15:24:41.028 INFO:teuthology.orchestra.run.smithi086.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-17T15:24:41.028 INFO:teuthology.orchestra.run.smithi086.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-17T15:24:41.028 INFO:teuthology.orchestra.run.smithi086.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-17T15:24:41.042 INFO:teuthology.orchestra.run.smithi203.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-17T15:24:41.042 INFO:teuthology.orchestra.run.smithi203.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-17T15:24:41.043 INFO:teuthology.orchestra.run.smithi086.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-17T15:24:41.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-17T15:24:41.045 INFO:teuthology.orchestra.run.smithi203.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-17T15:24:41.047 INFO:teuthology.orchestra.run.smithi086.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-17T15:24:41.059 INFO:teuthology.orchestra.run.smithi203.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-17T15:24:41.060 INFO:teuthology.orchestra.run.smithi203.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-17T15:24:41.061 INFO:teuthology.orchestra.run.smithi203.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-17T15:24:41.061 INFO:teuthology.orchestra.run.smithi086.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-17T15:24:41.062 INFO:teuthology.orchestra.run.smithi086.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-17T15:24:41.063 INFO:teuthology.orchestra.run.smithi086.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-17T15:24:41.064 INFO:teuthology.orchestra.run.smithi203.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-17T15:24:41.066 INFO:teuthology.orchestra.run.smithi086.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-17T15:24:41.066 INFO:teuthology.orchestra.run.smithi203.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-17T15:24:41.067 INFO:teuthology.orchestra.run.smithi203.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-17T15:24:41.067 INFO:teuthology.orchestra.run.smithi203.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-17T15:24:41.068 INFO:teuthology.orchestra.run.smithi086.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-17T15:24:41.069 INFO:teuthology.orchestra.run.smithi086.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-17T15:24:41.069 INFO:teuthology.orchestra.run.smithi086.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-17T15:24:41.076 INFO:teuthology.orchestra.run.smithi203.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-17T15:24:41.077 INFO:teuthology.orchestra.run.smithi203.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-17T15:24:41.079 INFO:teuthology.orchestra.run.smithi086.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-17T15:24:41.079 INFO:teuthology.orchestra.run.smithi086.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-17T15:24:41.080 INFO:teuthology.orchestra.run.smithi203.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-17T15:24:41.082 INFO:teuthology.orchestra.run.smithi086.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-17T15:24:41.093 INFO:teuthology.orchestra.run.smithi203.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-17T15:24:41.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-17T15:24:41.094 INFO:teuthology.orchestra.run.smithi203.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-17T15:24:41.096 INFO:teuthology.orchestra.run.smithi086.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-17T15:24:41.096 INFO:teuthology.orchestra.run.smithi203.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-17T15:24:41.097 INFO:teuthology.orchestra.run.smithi203.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-17T15:24:41.097 INFO:teuthology.orchestra.run.smithi086.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-17T15:24:41.097 INFO:teuthology.orchestra.run.smithi086.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-17T15:24:41.098 INFO:teuthology.orchestra.run.smithi086.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-17T15:24:41.098 INFO:teuthology.orchestra.run.smithi086.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-17T15:24:41.102 INFO:teuthology.orchestra.run.smithi203.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-17T15:24:41.103 INFO:teuthology.orchestra.run.smithi203.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-17T15:24:41.105 INFO:teuthology.orchestra.run.smithi086.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-17T15:24:41.105 INFO:teuthology.orchestra.run.smithi086.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-17T15:24:41.110 INFO:teuthology.orchestra.run.smithi203.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-17T15:24:41.112 INFO:teuthology.orchestra.run.smithi203.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-17T15:24:41.113 INFO:teuthology.orchestra.run.smithi203.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-17T15:24:41.113 INFO:teuthology.orchestra.run.smithi086.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-17T15:24:41.115 INFO:teuthology.orchestra.run.smithi086.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-17T15:24:41.115 INFO:teuthology.orchestra.run.smithi086.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-17T15:24:41.859 INFO:teuthology.orchestra.run.smithi203.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4975-g1889a165-1jammy [3,332 kB] 2024-09-17T15:24:42.044 INFO:teuthology.orchestra.run.smithi086.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4975-g1889a165-1jammy [3,332 kB] 2024-09-17T15:24:42.327 INFO:teuthology.orchestra.run.smithi203.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4975-g1889a165-1jammy [722 kB] 2024-09-17T15:24:42.408 INFO:teuthology.orchestra.run.smithi203.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4975-g1889a165-1jammy [356 kB] 2024-09-17T15:24:42.470 INFO:teuthology.orchestra.run.smithi203.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4975-g1889a165-1jammy [32.8 kB] 2024-09-17T15:24:42.471 INFO:teuthology.orchestra.run.smithi203.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4975-g1889a165-1jammy [180 kB] 2024-09-17T15:24:42.524 INFO:teuthology.orchestra.run.smithi203.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4975-g1889a165-1jammy [71.3 kB] 2024-09-17T15:24:42.526 INFO:teuthology.orchestra.run.smithi203.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4975-g1889a165-1jammy [338 kB] 2024-09-17T15:24:42.595 INFO:teuthology.orchestra.run.smithi203.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4975-g1889a165-1jammy [7,966 kB] 2024-09-17T15:24:42.659 INFO:teuthology.orchestra.run.smithi086.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4975-g1889a165-1jammy [722 kB] 2024-09-17T15:24:42.804 INFO:teuthology.orchestra.run.smithi086.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4975-g1889a165-1jammy [356 kB] 2024-09-17T15:24:42.877 INFO:teuthology.orchestra.run.smithi086.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4975-g1889a165-1jammy [32.8 kB] 2024-09-17T15:24:42.877 INFO:teuthology.orchestra.run.smithi086.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4975-g1889a165-1jammy [180 kB] 2024-09-17T15:24:42.946 INFO:teuthology.orchestra.run.smithi086.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4975-g1889a165-1jammy [71.3 kB] 2024-09-17T15:24:42.948 INFO:teuthology.orchestra.run.smithi086.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4975-g1889a165-1jammy [338 kB] 2024-09-17T15:24:43.020 INFO:teuthology.orchestra.run.smithi086.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4975-g1889a165-1jammy [7,966 kB] 2024-09-17T15:24:43.731 INFO:teuthology.orchestra.run.smithi203.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4975-g1889a165-1jammy [112 kB] 2024-09-17T15:24:43.734 INFO:teuthology.orchestra.run.smithi203.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4975-g1889a165-1jammy [473 kB] 2024-09-17T15:24:43.809 INFO:teuthology.orchestra.run.smithi203.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4975-g1889a165-1jammy [28.2 MB] 2024-09-17T15:24:44.539 INFO:teuthology.orchestra.run.smithi086.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4975-g1889a165-1jammy [112 kB] 2024-09-17T15:24:44.541 INFO:teuthology.orchestra.run.smithi086.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4975-g1889a165-1jammy [473 kB] 2024-09-17T15:24:44.683 INFO:teuthology.orchestra.run.smithi086.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4975-g1889a165-1jammy [28.2 MB] 2024-09-17T15:24:47.720 INFO:teuthology.orchestra.run.smithi203.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4975-g1889a165-1jammy [5,209 kB] 2024-09-17T15:24:48.166 INFO:teuthology.orchestra.run.smithi203.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4975-g1889a165-1jammy [251 kB] 2024-09-17T15:24:48.197 INFO:teuthology.orchestra.run.smithi203.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4975-g1889a165-1jammy [124 kB] 2024-09-17T15:24:48.221 INFO:teuthology.orchestra.run.smithi203.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4975-g1889a165-1jammy [1,561 kB] 2024-09-17T15:24:48.377 INFO:teuthology.orchestra.run.smithi203.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4975-g1889a165-1jammy [6,310 kB] 2024-09-17T15:24:49.021 INFO:teuthology.orchestra.run.smithi203.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4975-g1889a165-1jammy [23.1 MB] 2024-09-17T15:24:49.269 INFO:teuthology.orchestra.run.smithi086.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4975-g1889a165-1jammy [5,209 kB] 2024-09-17T15:24:49.934 INFO:teuthology.orchestra.run.smithi086.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4975-g1889a165-1jammy [251 kB] 2024-09-17T15:24:49.962 INFO:teuthology.orchestra.run.smithi086.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4975-g1889a165-1jammy [124 kB] 2024-09-17T15:24:49.975 INFO:teuthology.orchestra.run.smithi086.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4975-g1889a165-1jammy [1,561 kB] 2024-09-17T15:24:50.137 INFO:teuthology.orchestra.run.smithi086.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4975-g1889a165-1jammy [6,310 kB] 2024-09-17T15:24:50.721 INFO:teuthology.orchestra.run.smithi086.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4975-g1889a165-1jammy [23.1 MB] 2024-09-17T15:24:51.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4975-g1889a165-1jammy [14.0 kB] 2024-09-17T15:24:51.017 INFO:teuthology.orchestra.run.smithi203.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4975-g1889a165-1jammy [880 kB] 2024-09-17T15:24:51.034 INFO:teuthology.orchestra.run.smithi203.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4975-g1889a165-1jammy [2,452 kB] 2024-09-17T15:24:51.235 INFO:teuthology.orchestra.run.smithi203.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4975-g1889a165-1jammy [813 kB] 2024-09-17T15:24:51.294 INFO:teuthology.orchestra.run.smithi203.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4975-g1889a165-1jammy [160 kB] 2024-09-17T15:24:51.297 INFO:teuthology.orchestra.run.smithi203.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4975-g1889a165-1jammy [61.5 MB] 2024-09-17T15:24:53.228 INFO:teuthology.orchestra.run.smithi086.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4975-g1889a165-1jammy [14.0 kB] 2024-09-17T15:24:53.229 INFO:teuthology.orchestra.run.smithi086.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4975-g1889a165-1jammy [880 kB] 2024-09-17T15:24:53.250 INFO:teuthology.orchestra.run.smithi086.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4975-g1889a165-1jammy [2,452 kB] 2024-09-17T15:24:53.529 INFO:teuthology.orchestra.run.smithi086.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4975-g1889a165-1jammy [813 kB] 2024-09-17T15:24:53.592 INFO:teuthology.orchestra.run.smithi086.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4975-g1889a165-1jammy [160 kB] 2024-09-17T15:24:53.626 INFO:teuthology.orchestra.run.smithi086.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4975-g1889a165-1jammy [61.5 MB] 2024-09-17T15:24:56.512 INFO:teuthology.orchestra.run.smithi203.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4975-g1889a165-1jammy [8,625 kB] 2024-09-17T15:24:56.799 INFO:teuthology.orchestra.run.smithi203.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4975-g1889a165-1jammy [14.2 kB] 2024-09-17T15:24:56.800 INFO:teuthology.orchestra.run.smithi203.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4975-g1889a165-1jammy [48.5 MB] 2024-09-17T15:24:58.024 INFO:teuthology.orchestra.run.smithi086.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4975-g1889a165-1jammy [8,625 kB] 2024-09-17T15:24:58.307 INFO:teuthology.orchestra.run.smithi086.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4975-g1889a165-1jammy [14.2 kB] 2024-09-17T15:24:58.307 INFO:teuthology.orchestra.run.smithi086.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4975-g1889a165-1jammy [48.5 MB] 2024-09-17T15:24:59.274 INFO:teuthology.orchestra.run.smithi203.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4975-g1889a165-1jammy [130 kB] 2024-09-17T15:24:59.277 INFO:teuthology.orchestra.run.smithi203.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4975-g1889a165-1jammy [39.0 kB] 2024-09-17T15:24:59.278 INFO:teuthology.orchestra.run.smithi203.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4975-g1889a165-1jammy [15.8 MB] 2024-09-17T15:24:59.848 INFO:teuthology.orchestra.run.smithi203.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4975-g1889a165-1jammy [91.3 kB] 2024-09-17T15:25:00.421 INFO:teuthology.orchestra.run.smithi203.stdout:Fetched 244 MB in 19s (12.6 MB/s) 2024-09-17T15:25:00.592 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-17T15:25:00.692 INFO:teuthology.orchestra.run.smithi203.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-17T15:25:00.698 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-17T15:25:00.782 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T15:25:00.921 INFO:teuthology.orchestra.run.smithi086.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4975-g1889a165-1jammy [130 kB] 2024-09-17T15:25:00.924 INFO:teuthology.orchestra.run.smithi086.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4975-g1889a165-1jammy [39.0 kB] 2024-09-17T15:25:00.925 INFO:teuthology.orchestra.run.smithi086.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4975-g1889a165-1jammy [15.8 MB] 2024-09-17T15:25:01.153 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-17T15:25:01.168 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-17T15:25:01.211 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T15:25:01.514 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-17T15:25:01.534 INFO:teuthology.orchestra.run.smithi086.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4975-g1889a165-1jammy [91.3 kB] 2024-09-17T15:25:01.541 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-17T15:25:01.573 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T15:25:02.027 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-17T15:25:02.053 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 244 MB in 21s (11.6 MB/s) 2024-09-17T15:25:02.102 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:02.205 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-17T15:25:02.303 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-17T15:25:02.308 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-17T15:25:02.379 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T15:25:02.513 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-17T15:25:02.540 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:02.580 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-17T15:25:02.671 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-17T15:25:02.707 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T15:25:02.933 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-17T15:25:02.960 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:03.000 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:03.043 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-17T15:25:03.069 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-17T15:25:03.109 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T15:25:03.362 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-17T15:25:03.388 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-17T15:25:03.428 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T15:25:03.555 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-17T15:25:03.581 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:03.629 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:03.756 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librados2. 2024-09-17T15:25:03.782 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../007-librados2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:03.823 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:03.957 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-17T15:25:03.984 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:04.015 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:04.259 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libnbd0. 2024-09-17T15:25:04.286 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-17T15:25:04.326 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-17T15:25:04.368 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-17T15:25:04.395 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T15:25:04.435 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:25:04.637 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librbd1. 2024-09-17T15:25:04.663 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../009-librbd1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:04.704 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:04.796 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-17T15:25:04.822 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-17T15:25:04.863 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T15:25:05.149 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librados2. 2024-09-17T15:25:05.157 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs2. 2024-09-17T15:25:05.175 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../007-librados2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:05.184 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:05.224 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:05.224 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:05.626 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libnbd0. 2024-09-17T15:25:05.652 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-17T15:25:05.660 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rados. 2024-09-17T15:25:05.675 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:05.684 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-17T15:25:05.718 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:06.004 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librbd1. 2024-09-17T15:25:06.031 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../009-librbd1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:06.038 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-17T15:25:06.064 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:06.070 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:06.199 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:06.466 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cephfs. 2024-09-17T15:25:06.481 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:06.523 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs2. 2024-09-17T15:25:06.524 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:06.550 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:06.590 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:06.810 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-17T15:25:06.826 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:06.842 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rados. 2024-09-17T15:25:06.857 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:06.860 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:07.042 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:07.163 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-17T15:25:07.189 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-17T15:25:07.222 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T15:25:07.320 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-17T15:25:07.346 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:07.411 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:07.533 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-prettytable. 2024-09-17T15:25:07.559 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-17T15:25:07.600 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-17T15:25:07.647 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cephfs. 2024-09-17T15:25:07.662 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:07.705 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:07.861 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rbd. 2024-09-17T15:25:07.876 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:07.919 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:08.016 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-17T15:25:08.042 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:08.082 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:08.264 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-17T15:25:08.290 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-17T15:25:08.331 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T15:25:08.419 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-17T15:25:08.445 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-17T15:25:08.485 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T15:25:08.617 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-17T15:25:08.633 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-17T15:25:08.675 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T15:25:08.805 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-prettytable. 2024-09-17T15:25:08.831 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-17T15:25:08.871 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-17T15:25:08.978 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-17T15:25:09.005 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-17T15:25:09.045 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T15:25:09.132 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rbd. 2024-09-17T15:25:09.154 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:09.199 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:09.390 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua5.1. 2024-09-17T15:25:09.417 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-17T15:25:09.457 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-17T15:25:09.593 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-17T15:25:09.619 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-17T15:25:09.659 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T15:25:09.760 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-any. 2024-09-17T15:25:09.787 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-17T15:25:09.827 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-17T15:25:09.895 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-17T15:25:09.910 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-17T15:25:09.954 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T15:25:10.079 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package zip. 2024-09-17T15:25:10.095 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-17T15:25:10.138 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking zip (3.0-12build2) ... 2024-09-17T15:25:10.248 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-17T15:25:10.263 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-17T15:25:10.306 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T15:25:10.486 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package unzip. 2024-09-17T15:25:10.513 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-17T15:25:10.616 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-17T15:25:10.634 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua5.1. 2024-09-17T15:25:10.660 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-17T15:25:10.701 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-17T15:25:10.987 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package luarocks. 2024-09-17T15:25:11.013 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-17T15:25:11.020 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-any. 2024-09-17T15:25:11.044 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-17T15:25:11.046 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-17T15:25:11.087 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-17T15:25:11.339 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package zip. 2024-09-17T15:25:11.354 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-17T15:25:11.398 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking zip (3.0-12build2) ... 2024-09-17T15:25:11.412 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package librgw2. 2024-09-17T15:25:11.427 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../026-librgw2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:11.470 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:11.726 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package unzip. 2024-09-17T15:25:11.752 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-17T15:25:11.792 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-17T15:25:12.081 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rgw. 2024-09-17T15:25:12.097 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:12.140 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:12.188 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package luarocks. 2024-09-17T15:25:12.214 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-17T15:25:12.254 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-17T15:25:12.493 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-17T15:25:12.519 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-17T15:25:12.559 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T15:25:12.636 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librgw2. 2024-09-17T15:25:12.663 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../026-librgw2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:12.703 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:12.846 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libradosstriper1. 2024-09-17T15:25:12.872 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:12.904 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:13.280 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rgw. 2024-09-17T15:25:13.296 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:13.333 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-common. 2024-09-17T15:25:13.347 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:13.359 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:13.408 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:14.417 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-17T15:25:14.444 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-17T15:25:14.483 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T15:25:14.794 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libradosstriper1. 2024-09-17T15:25:14.820 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:14.861 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:15.231 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-common. 2024-09-17T15:25:15.258 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:15.297 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:15.690 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-base. 2024-09-17T15:25:15.716 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:15.805 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:16.418 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-17T15:25:16.445 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-17T15:25:16.476 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-17T15:25:16.738 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cheroot. 2024-09-17T15:25:16.752 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-base. 2024-09-17T15:25:16.765 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-17T15:25:16.778 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:16.804 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T15:25:16.828 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:17.116 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-17T15:25:17.142 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-17T15:25:17.182 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-17T15:25:17.486 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-17T15:25:17.512 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-17T15:25:17.552 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-17T15:25:17.648 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-17T15:25:17.675 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-17T15:25:17.714 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-17T15:25:17.830 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-17T15:25:17.845 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-17T15:25:17.880 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-17T15:25:18.017 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cheroot. 2024-09-17T15:25:18.044 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-17T15:25:18.083 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T15:25:18.142 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempora. 2024-09-17T15:25:18.168 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-17T15:25:18.208 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-17T15:25:18.336 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-17T15:25:18.352 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-17T15:25:18.403 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-17T15:25:18.511 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-portend. 2024-09-17T15:25:18.538 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-17T15:25:18.578 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-17T15:25:18.706 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-17T15:25:18.733 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-17T15:25:18.772 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-17T15:25:18.881 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-17T15:25:18.907 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-17T15:25:18.947 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-17T15:25:19.075 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-17T15:25:19.102 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-17T15:25:19.141 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-17T15:25:19.218 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-17T15:25:19.244 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-17T15:25:19.284 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-17T15:25:19.419 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tempora. 2024-09-17T15:25:19.446 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-17T15:25:19.486 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-17T15:25:19.646 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-natsort. 2024-09-17T15:25:19.673 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-17T15:25:19.713 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-17T15:25:19.780 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-portend. 2024-09-17T15:25:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-17T15:25:19.847 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-17T15:25:20.016 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-logutils. 2024-09-17T15:25:20.042 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-17T15:25:20.074 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-17T15:25:20.150 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-17T15:25:20.176 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-17T15:25:20.216 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-17T15:25:20.369 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-17T15:25:20.395 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-17T15:25:20.436 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-17T15:25:20.519 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-17T15:25:20.545 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-17T15:25:20.585 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-17T15:25:20.722 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-mako. 2024-09-17T15:25:20.748 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-17T15:25:20.789 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T15:25:20.930 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-natsort. 2024-09-17T15:25:20.957 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-17T15:25:20.997 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-17T15:25:21.159 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-17T15:25:21.185 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-17T15:25:21.226 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-17T15:25:21.308 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-logutils. 2024-09-17T15:25:21.335 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-17T15:25:21.375 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-17T15:25:21.537 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-17T15:25:21.564 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-17T15:25:21.603 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-17T15:25:21.677 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-17T15:25:21.704 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-17T15:25:21.744 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-17T15:25:21.906 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webob. 2024-09-17T15:25:21.933 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-17T15:25:21.973 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T15:25:22.038 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-mako. 2024-09-17T15:25:22.065 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-17T15:25:22.096 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T15:25:22.301 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-waitress. 2024-09-17T15:25:22.328 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-17T15:25:22.363 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T15:25:22.399 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-17T15:25:22.426 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-17T15:25:22.466 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-17T15:25:22.646 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-tempita. 2024-09-17T15:25:22.673 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-17T15:25:22.713 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T15:25:22.785 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-17T15:25:22.812 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-17T15:25:22.852 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-17T15:25:23.016 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-paste. 2024-09-17T15:25:23.042 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-17T15:25:23.082 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T15:25:23.155 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webob. 2024-09-17T15:25:23.181 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-17T15:25:23.221 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T15:25:23.369 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-17T15:25:23.385 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-17T15:25:23.427 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T15:25:23.566 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-waitress. 2024-09-17T15:25:23.592 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-17T15:25:23.635 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T15:25:23.731 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-17T15:25:23.757 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-17T15:25:23.797 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-17T15:25:23.910 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tempita. 2024-09-17T15:25:23.937 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-17T15:25:23.968 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T15:25:24.109 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-webtest. 2024-09-17T15:25:24.136 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-17T15:25:24.175 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-17T15:25:24.263 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-paste. 2024-09-17T15:25:24.290 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-17T15:25:24.329 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T15:25:24.462 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pecan. 2024-09-17T15:25:24.488 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-17T15:25:24.528 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T15:25:24.624 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-17T15:25:24.647 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-17T15:25:24.682 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T15:25:24.884 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-certifi. 2024-09-17T15:25:24.910 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-17T15:25:24.942 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-17T15:25:24.959 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-17T15:25:24.986 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-17T15:25:25.026 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-17T15:25:25.253 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-idna. 2024-09-17T15:25:25.280 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-17T15:25:25.320 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T15:25:25.329 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webtest. 2024-09-17T15:25:25.356 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-17T15:25:25.396 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-17T15:25:25.648 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-urllib3. 2024-09-17T15:25:25.674 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-17T15:25:25.707 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pecan. 2024-09-17T15:25:25.714 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T15:25:25.734 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-17T15:25:25.773 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T15:25:26.043 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests. 2024-09-17T15:25:26.070 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-17T15:25:26.153 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-certifi. 2024-09-17T15:25:26.180 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-17T15:25:26.219 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-17T15:25:26.276 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T15:25:26.522 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-idna. 2024-09-17T15:25:26.549 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-17T15:25:26.588 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T15:25:26.621 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-17T15:25:26.648 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-17T15:25:26.688 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T15:25:26.883 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-urllib3. 2024-09-17T15:25:26.910 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-17T15:25:26.949 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T15:25:26.974 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-17T15:25:26.990 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:27.024 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:27.244 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-requests. 2024-09-17T15:25:27.271 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-17T15:25:27.310 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T15:25:27.413 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-17T15:25:27.441 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:27.479 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:27.630 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-17T15:25:27.657 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-17T15:25:27.696 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T15:25:27.774 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-17T15:25:27.790 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-17T15:25:27.824 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-17T15:25:27.958 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-17T15:25:27.973 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:28.016 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:28.127 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-openssl. 2024-09-17T15:25:28.154 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-17T15:25:28.194 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-17T15:25:28.404 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-17T15:25:28.431 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:28.470 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:28.539 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr. 2024-09-17T15:25:28.565 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:28.605 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:28.748 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-17T15:25:28.772 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-17T15:25:28.806 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-17T15:25:29.009 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mon. 2024-09-17T15:25:29.035 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:29.084 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:29.134 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-openssl. 2024-09-17T15:25:29.161 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-17T15:25:29.200 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-17T15:25:29.537 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr. 2024-09-17T15:25:29.564 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:29.603 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:29.672 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-osd. 2024-09-17T15:25:29.698 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:29.737 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:29.998 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mon. 2024-09-17T15:25:30.025 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:30.064 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:30.626 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-osd. 2024-09-17T15:25:30.653 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:30.692 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:30.824 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph. 2024-09-17T15:25:30.840 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../067-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:30.882 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:31.177 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-fuse. 2024-09-17T15:25:31.204 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:31.244 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mds. 2024-09-17T15:25:31.615 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:31.655 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:31.745 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph. 2024-09-17T15:25:31.761 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../067-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:31.803 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:32.075 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package cephadm. 2024-09-17T15:25:32.102 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../070-cephadm_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:32.141 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:32.165 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-fuse. 2024-09-17T15:25:32.192 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:32.231 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:32.422 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-17T15:25:32.448 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-17T15:25:32.480 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-17T15:25:32.567 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mds. 2024-09-17T15:25:32.594 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:32.625 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:32.808 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-17T15:25:32.823 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T15:25:32.857 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T15:25:33.170 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephadm. 2024-09-17T15:25:33.197 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../070-cephadm_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:33.236 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:33.556 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-17T15:25:33.583 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-17T15:25:33.622 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-17T15:25:33.863 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-babel. 2024-09-17T15:25:33.890 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T15:25:33.921 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T15:25:33.924 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-17T15:25:33.940 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T15:25:33.983 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T15:25:34.191 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-jinja2. 2024-09-17T15:25:34.218 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-17T15:25:34.258 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T15:25:34.544 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-17T15:25:34.571 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:34.619 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:34.923 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-17T15:25:34.938 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-17T15:25:34.973 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-17T15:25:34.979 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-babel. 2024-09-17T15:25:35.006 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T15:25:35.170 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T15:25:35.442 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-routes. 2024-09-17T15:25:35.448 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jinja2. 2024-09-17T15:25:35.469 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-17T15:25:35.476 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-17T15:25:35.506 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T15:25:35.508 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T15:25:35.827 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-17T15:25:35.838 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-17T15:25:35.853 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:35.864 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:35.893 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:35.904 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:36.221 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-17T15:25:36.248 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-17T15:25:36.287 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-17T15:25:36.575 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-routes. 2024-09-17T15:25:36.603 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-17T15:25:36.642 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T15:25:36.987 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-17T15:25:37.014 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:37.053 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:39.093 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-17T15:25:39.123 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-17T15:25:39.160 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T15:25:39.656 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-joblib. 2024-09-17T15:25:39.683 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-17T15:25:39.722 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T15:25:39.932 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-17T15:25:39.949 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-17T15:25:39.990 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T15:25:40.052 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-17T15:25:40.066 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-17T15:25:40.109 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-17T15:25:40.420 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-sklearn. 2024-09-17T15:25:40.448 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-17T15:25:40.494 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-joblib. 2024-09-17T15:25:40.521 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-17T15:25:40.560 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T15:25:40.572 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T15:25:40.922 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-17T15:25:40.949 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-17T15:25:40.988 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-17T15:25:41.299 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-17T15:25:41.316 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn. 2024-09-17T15:25:41.327 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:41.343 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-17T15:25:41.366 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:41.467 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T15:25:42.176 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-17T15:25:42.204 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:42.243 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:42.377 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-cachetools. 2024-09-17T15:25:42.405 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-17T15:25:42.444 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-17T15:25:42.747 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-17T15:25:42.775 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-17T15:25:42.814 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-17T15:25:43.151 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-17T15:25:43.178 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-17T15:25:43.196 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cachetools. 2024-09-17T15:25:43.217 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T15:25:43.223 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-17T15:25:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-17T15:25:43.537 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-rsa. 2024-09-17T15:25:43.565 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-17T15:25:43.565 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-17T15:25:43.592 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-17T15:25:43.604 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-17T15:25:43.631 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-17T15:25:43.874 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-google-auth. 2024-09-17T15:25:43.887 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-17T15:25:43.916 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-17T15:25:43.968 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-17T15:25:43.996 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-17T15:25:44.034 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T15:25:44.219 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-17T15:25:44.247 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-17T15:25:44.286 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T15:25:44.371 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rsa. 2024-09-17T15:25:44.399 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-17T15:25:44.437 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-17T15:25:44.573 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-websocket. 2024-09-17T15:25:44.600 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-17T15:25:44.631 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-17T15:25:44.757 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-google-auth. 2024-09-17T15:25:44.785 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-17T15:25:44.824 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-17T15:25:44.926 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-17T15:25:44.953 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-17T15:25:44.992 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T15:25:45.118 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-17T15:25:45.146 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-17T15:25:45.185 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T15:25:45.496 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-websocket. 2024-09-17T15:25:45.523 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-17T15:25:45.562 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-17T15:25:45.649 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-17T15:25:45.665 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:45.707 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:45.841 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-17T15:25:45.869 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-17T15:25:45.907 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T15:25:46.018 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-17T15:25:46.046 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-17T15:25:46.084 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T15:25:46.388 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-17T15:25:46.416 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T15:25:46.446 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T15:25:46.521 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-17T15:25:46.537 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:46.579 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:46.716 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package jq. 2024-09-17T15:25:46.743 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T15:25:46.782 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-17T15:25:46.906 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-17T15:25:46.935 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-17T15:25:46.973 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T15:25:47.071 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package socat. 2024-09-17T15:25:47.099 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-17T15:25:47.143 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-17T15:25:47.259 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-17T15:25:47.287 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T15:25:47.342 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T15:25:47.447 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package xmlstarlet. 2024-09-17T15:25:47.463 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-17T15:25:47.505 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-17T15:25:47.603 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package jq. 2024-09-17T15:25:47.631 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T15:25:47.669 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-17T15:25:47.860 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-test. 2024-09-17T15:25:47.884 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:47.923 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package socat. 2024-09-17T15:25:47.926 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:47.950 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-17T15:25:47.989 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-17T15:25:48.292 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package xmlstarlet. 2024-09-17T15:25:48.308 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-17T15:25:48.341 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-17T15:25:48.696 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-test. 2024-09-17T15:25:48.724 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:48.762 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:50.331 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package ceph-volume. 2024-09-17T15:25:50.359 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:50.389 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:50.677 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-17T15:25:50.693 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:50.735 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:51.025 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-volume. 2024-09-17T15:25:51.046 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-17T15:25:51.053 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T15:25:51.074 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-17T15:25:51.091 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:51.113 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T15:25:51.370 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-17T15:25:51.398 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:51.436 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:51.451 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-17T15:25:51.479 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-17T15:25:51.517 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-17T15:25:51.747 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-17T15:25:51.775 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-17T15:25:51.813 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T15:25:51.864 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package nvme-cli. 2024-09-17T15:25:51.892 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-17T15:25:51.922 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T15:25:52.135 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-17T15:25:52.163 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-17T15:25:52.201 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-17T15:25:52.336 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package pkg-config. 2024-09-17T15:25:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-17T15:25:52.402 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T15:25:52.497 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package nvme-cli. 2024-09-17T15:25:52.525 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-17T15:25:52.555 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T15:25:52.664 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-17T15:25:52.680 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-17T15:25:52.724 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-17T15:25:52.985 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package pkg-config. 2024-09-17T15:25:53.013 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-17T15:25:53.051 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T15:25:53.159 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-17T15:25:53.187 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-17T15:25:53.225 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-17T15:25:53.313 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-17T15:25:53.341 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-17T15:25:53.379 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-17T15:25:53.528 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pastescript. 2024-09-17T15:25:53.556 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-17T15:25:53.595 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-17T15:25:53.790 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-17T15:25:53.819 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-17T15:25:53.856 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-17T15:25:53.924 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pluggy. 2024-09-17T15:25:53.952 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-17T15:25:53.990 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-17T15:25:54.126 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastescript. 2024-09-17T15:25:54.142 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-17T15:25:54.176 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-17T15:25:54.277 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-psutil. 2024-09-17T15:25:54.305 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-17T15:25:54.343 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-17T15:25:54.471 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pluggy. 2024-09-17T15:25:54.499 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-17T15:25:54.537 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-17T15:25:54.697 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-py. 2024-09-17T15:25:54.724 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-17T15:25:54.763 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-17T15:25:54.848 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-psutil. 2024-09-17T15:25:54.877 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-17T15:25:54.914 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-17T15:25:55.059 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pygments. 2024-09-17T15:25:55.086 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-17T15:25:55.125 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T15:25:55.267 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-py. 2024-09-17T15:25:55.293 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-17T15:25:55.334 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-17T15:25:55.621 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-17T15:25:55.650 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-17T15:25:55.662 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pygments. 2024-09-17T15:25:55.688 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-17T15:25:55.691 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-17T15:25:55.728 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T15:25:55.974 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-toml. 2024-09-17T15:25:56.002 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-17T15:25:56.040 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-17T15:25:56.124 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-17T15:25:56.140 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-17T15:25:56.182 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-17T15:25:56.302 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-pytest. 2024-09-17T15:25:56.330 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-17T15:25:56.368 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T15:25:56.502 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-toml. 2024-09-17T15:25:56.530 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-17T15:25:56.576 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-17T15:25:56.722 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package python3-simplejson. 2024-09-17T15:25:56.750 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-17T15:25:56.789 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-17T15:25:56.862 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pytest. 2024-09-17T15:25:56.890 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-17T15:25:56.929 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T15:25:57.058 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-17T15:25:57.087 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-17T15:25:57.125 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-17T15:25:57.265 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplejson. 2024-09-17T15:25:57.294 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-17T15:25:57.331 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-17T15:25:57.609 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-17T15:25:57.626 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-17T15:25:57.668 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-17T15:25:57.879 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package radosgw. 2024-09-17T15:25:57.907 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../117-radosgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:57.954 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:58.430 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package radosgw. 2024-09-17T15:25:58.458 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../117-radosgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:58.496 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:58.866 INFO:teuthology.orchestra.run.smithi203.stdout:Selecting previously unselected package rbd-fuse. 2024-09-17T15:25:58.894 INFO:teuthology.orchestra.run.smithi203.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:58.932 INFO:teuthology.orchestra.run.smithi203.stdout:Unpacking rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:59.229 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-17T15:25:59.374 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package rbd-fuse. 2024-09-17T15:25:59.390 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T15:25:59.432 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:25:59.479 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T15:25:59.580 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T15:25:59.745 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-17T15:25:59.850 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-17T15:26:00.003 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T15:26:00.129 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T15:26:00.221 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-17T15:26:00.393 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-17T15:26:00.646 INFO:teuthology.orchestra.run.smithi203.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-17T15:26:00.717 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-17T15:26:00.740 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:01.169 INFO:teuthology.orchestra.run.smithi086.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-17T15:26:01.401 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user cephadm....done 2024-09-17T15:26:01.415 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:01.557 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T15:26:01.898 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user cephadm....done 2024-09-17T15:26:02.034 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-17T15:26:02.054 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T15:26:02.319 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-17T15:26:02.332 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-17T15:26:02.462 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-17T15:26:02.556 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-17T15:26:02.691 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-17T15:26:02.721 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-17T15:26:02.942 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-17T15:26:02.981 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T15:26:03.107 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-17T15:26:03.193 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T15:26:03.319 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-17T15:26:03.409 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T15:26:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T15:26:03.769 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-17T15:26:03.972 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-17T15:26:04.215 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-17T15:26:04.338 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-17T15:26:04.466 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-17T15:26:04.606 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-17T15:26:04.726 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-17T15:26:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-17T15:26:04.985 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-17T15:26:05.125 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-17T15:26:05.383 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-17T15:26:05.385 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-17T15:26:05.644 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:05.829 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:05.879 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T15:26:06.005 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-17T15:26:06.088 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T15:26:06.114 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T15:26:06.189 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-17T15:26:06.232 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T15:26:06.290 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T15:26:06.366 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-17T15:26:06.408 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T15:26:06.534 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-17T15:26:06.626 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-17T15:26:06.785 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-17T15:26:07.000 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-17T15:26:07.024 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-17T15:26:07.105 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T15:26:07.167 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-17T15:26:07.199 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-17T15:26:07.256 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-17T15:26:07.264 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T15:26:07.363 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-17T15:26:07.641 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T15:26:07.715 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T15:26:07.767 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-17T15:26:07.815 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-17T15:26:08.051 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T15:26:08.084 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T15:26:08.327 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-17T15:26:08.351 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-17T15:26:08.652 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up zip (3.0-12build2) ... 2024-09-17T15:26:08.671 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up zip (3.0-12build2) ... 2024-09-17T15:26:08.778 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T15:26:08.822 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T15:26:09.455 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T15:26:09.482 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T15:26:09.689 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T15:26:09.742 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T15:26:09.790 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-17T15:26:09.868 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-17T15:26:09.908 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T15:26:09.994 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T15:26:10.217 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-17T15:26:10.312 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-17T15:26:10.627 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-17T15:26:10.672 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-17T15:26:10.878 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T15:26:10.948 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T15:26:11.279 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T15:26:11.316 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T15:26:11.555 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T15:26:11.609 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T15:26:11.896 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-17T15:26:11.978 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-17T15:26:12.149 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-17T15:26:12.237 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-17T15:26:12.275 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:12.355 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:12.569 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T15:26:12.676 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T15:26:12.886 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T15:26:13.007 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T15:26:14.025 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T15:26:14.241 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T15:26:14.276 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:14.410 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-17T15:26:14.501 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:14.635 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-17T15:26:14.670 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T15:26:14.796 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-17T15:26:14.894 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T15:26:14.896 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T15:26:15.045 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-17T15:26:15.181 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-17T15:26:15.196 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T15:26:15.407 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-17T15:26:15.523 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-17T15:26:15.658 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:15.782 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-17T15:26:15.784 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-17T15:26:16.042 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:16.094 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-17T15:26:16.168 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-17T15:26:16.353 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-17T15:26:16.452 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-17T15:26:16.621 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-17T15:26:16.695 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-17T15:26:16.872 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-17T15:26:16.963 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-17T15:26:17.106 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-17T15:26:17.231 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-17T15:26:17.382 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T15:26:17.482 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-17T15:26:17.508 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-17T15:26:17.749 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T15:26:17.801 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T15:26:17.901 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-17T15:26:17.927 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T15:26:18.193 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T15:26:18.211 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T15:26:18.344 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T15:26:18.504 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-17T15:26:18.654 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T15:26:18.747 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T15:26:18.847 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-17T15:26:18.948 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T15:26:18.988 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-17T15:26:19.020 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-17T15:26:19.258 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T15:26:19.281 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T15:26:19.384 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T15:26:19.407 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-17T15:26:19.533 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T15:26:19.623 INFO:teuthology.orchestra.run.smithi203.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-17T15:26:19.769 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-17T15:26:20.028 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-17T15:26:20.110 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T15:26:20.154 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-17T15:26:20.261 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T15:26:20.397 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:20.506 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T15:26:20.679 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-17T15:26:20.816 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T15:26:20.947 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-17T15:26:21.098 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-17T15:26:21.349 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T15:26:21.367 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T15:26:21.475 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T15:26:21.651 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-17T15:26:21.777 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-17T15:26:21.784 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T15:26:22.045 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-17T15:26:22.077 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T15:26:22.345 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-17T15:26:22.390 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T15:26:22.496 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-17T15:26:22.632 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-17T15:26:22.789 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-17T15:26:23.003 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-17T15:26:23.167 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T15:26:23.287 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T15:26:23.435 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-17T15:26:23.640 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T15:26:23.774 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:23.845 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-17T15:26:23.900 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:24.052 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-17T15:26:24.146 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T15:26:24.405 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:24.491 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T15:26:24.522 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:24.617 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:24.624 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-17T15:26:24.743 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:24.852 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-17T15:26:24.891 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.017 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.138 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.144 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.298 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.354 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T15:26:25.354 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T15:26:25.424 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-17T15:26:25.709 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.835 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.845 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.944 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:25.979 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:26.080 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:26.140 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T15:26:26.140 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T15:26:26.181 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T15:26:26.612 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:26.721 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:26.856 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:26.982 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T15:26:27.401 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:27.528 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:27.654 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:27.780 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:27.880 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.227 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.353 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.365 INFO:teuthology.orchestra.run.smithi086.stdout:Adding group ceph....done 2024-09-17T15:26:28.479 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.605 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.739 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:28.758 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user ceph....done 2024-09-17T15:26:28.918 INFO:teuthology.orchestra.run.smithi086.stdout:Setting system user ceph properties....done 2024-09-17T15:26:28.932 INFO:teuthology.orchestra.run.smithi086.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-17T15:26:29.070 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-17T15:26:29.266 INFO:teuthology.orchestra.run.smithi203.stdout:Adding group ceph....done 2024-09-17T15:26:29.417 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-17T15:26:29.618 INFO:teuthology.orchestra.run.smithi203.stdout:Adding system user ceph....done 2024-09-17T15:26:29.794 INFO:teuthology.orchestra.run.smithi203.stdout:Setting system user ceph properties....done 2024-09-17T15:26:29.812 INFO:teuthology.orchestra.run.smithi203.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-17T15:26:29.915 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:29.945 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-17T15:26:30.041 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:30.242 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-17T15:26:30.476 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:26:30.476 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:26:30.699 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:30.825 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:31.275 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:26:31.275 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T15:26:31.305 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-17T15:26:31.735 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:31.786 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:32.011 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T15:26:32.012 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T15:26:32.040 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-17T15:26:32.463 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:32.486 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:32.677 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T15:26:32.678 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T15:26:32.712 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T15:26:32.712 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T15:26:33.181 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:33.188 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:33.425 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T15:26:33.425 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T15:26:33.432 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T15:26:33.432 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T15:26:33.915 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:33.915 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:34.042 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:34.180 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T15:26:34.180 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T15:26:34.193 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:34.411 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T15:26:34.412 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T15:26:34.642 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:34.870 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:34.877 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.029 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.131 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.146 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.281 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.389 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T15:26:35.389 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T15:26:35.625 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T15:26:35.734 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T15:26:35.880 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:35.887 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T15:26:36.023 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:36.149 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:36.300 INFO:teuthology.orchestra.run.smithi203.stdout:Setting up ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T15:26:36.644 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T15:26:36.804 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T15:26:37.013 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T15:26:37.064 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T15:26:37.675 INFO:teuthology.orchestra.run.smithi086.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T15:26:37.681 DEBUG:teuthology.parallel:result is None 2024-09-17T15:26:37.839 INFO:teuthology.orchestra.run.smithi203.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T15:26:38.546 INFO:teuthology.orchestra.run.smithi203.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T15:26:38.550 DEBUG:teuthology.parallel:result is None 2024-09-17T15:26:38.551 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:26:38.689 DEBUG:teuthology.orchestra.run.smithi086:> dpkg-query -W -f '${Version}' ceph 2024-09-17T15:26:38.708 INFO:teuthology.orchestra.run.smithi086.stdout:19.3.0-4975-g1889a165-1jammy 2024-09-17T15:26:38.709 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975-g1889a165-1jammy 2024-09-17T15:26:38.709 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975-g1889a165-1jammy is installed. 2024-09-17T15:26:38.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T15:26:38.849 DEBUG:teuthology.orchestra.run.smithi203:> dpkg-query -W -f '${Version}' ceph 2024-09-17T15:26:38.868 INFO:teuthology.orchestra.run.smithi203.stdout:19.3.0-4975-g1889a165-1jammy 2024-09-17T15:26:38.868 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975-g1889a165-1jammy 2024-09-17T15:26:38.868 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975-g1889a165-1jammy is installed. 2024-09-17T15:26:38.870 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T15:26:38.870 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:38.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:26:38.883 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:38.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T15:26:38.922 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T15:26:38.923 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:38.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:26:38.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:26:38.993 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:38.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T15:26:39.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T15:26:39.061 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T15:26:39.062 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:39.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:26:39.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:26:39.133 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:39.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T15:26:39.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T15:26:39.202 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T15:26:39.202 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:39.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:26:39.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:26:39.269 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:39.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T15:26:39.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T15:26:39.339 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T15:26:39.417 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T15:26:39.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-09-17T15:26:39.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-09-17T15:26:39.433 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T15:26:39.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:26:39.475 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T15:26:39.487 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T15:26:39.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:26:39.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T15:26:39.552 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T15:26:39.552 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T15:26:39.552 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954', 'branch': 'wip-bharath10-testing-2024-09-17-1348', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7908954', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 3421, 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath10-testing-2024-09-17-1348', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMBWmuU3HKI6CDb3bSTo/lZhNv8pK3BRB/0FURB4+iA/9DWZUb53kN9dvfjLK8akrDGKkzocCqWX2MvMs5s8VF4=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWyTpwhRg7hYyYN9yp/NQF+jvVaVUL7QTWgzWVhIsFM/9TkdXxuwjsRhq3i8BtRT0DYFT9ydkHLDxkepw8wIKY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_15:01:33', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T15:26:39.553 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:26:39.587 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T15:26:39.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:26:39.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T15:26:39.648 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:39.648 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:26:39.695 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:26:39.695 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-09-17 15:20:58.503042234 +0000 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-09-17 15:20:58.223044735 +0000 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-09-17 15:20:58.223044735 +0000 2024-09-17T15:26:39.743 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-09-17T15:26:39.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:26:39.797 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-09-17T15:26:39.797 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-09-17T15:26:39.797 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000371177 s, 1.4 MB/s 2024-09-17T15:26:39.798 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:26:39.848 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-09-17T15:26:39.894 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 894 Links: 1 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-09-17 15:20:58.731040198 +0000 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-09-17 15:20:58.727040234 +0000 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-09-17 15:20:58.727040234 +0000 2024-09-17T15:26:39.895 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-09-17T15:26:39.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:26:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-09-17T15:26:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-09-17T15:26:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000373284 s, 1.4 MB/s 2024-09-17T15:26:39.951 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:26:40.000 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-09-17T15:26:40.046 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:26:40.046 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.046 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-17T15:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-09-17 15:20:59.423034017 +0000 2024-09-17T15:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-09-17 15:20:59.187036125 +0000 2024-09-17T15:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-09-17 15:20:59.187036125 +0000 2024-09-17T15:26:40.047 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-09-17T15:26:40.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:26:40.101 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-09-17T15:26:40.101 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-09-17T15:26:40.101 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000419697 s, 1.2 MB/s 2024-09-17T15:26:40.102 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:26:40.151 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-09-17 15:20:59.871030016 +0000 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-09-17 15:20:59.639032088 +0000 2024-09-17T15:26:40.198 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-09-17 15:20:59.639032088 +0000 2024-09-17T15:26:40.199 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: - 2024-09-17T15:26:40.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:26:40.252 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-09-17T15:26:40.253 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-09-17T15:26:40.253 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000290527 s, 1.8 MB/s 2024-09-17T15:26:40.254 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:26:40.308 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:40.308 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:40.308 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T15:26:40.314 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T15:26:40.314 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-17 15:20:59.873078097 +0000 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-17 15:20:59.601080525 +0000 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-17 15:20:59.601080525 +0000 2024-09-17T15:26:40.363 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-17T15:26:40.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T15:26:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-17T15:26:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-17T15:26:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000448797 s, 1.1 MB/s 2024-09-17T15:26:40.419 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T15:26:40.468 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-09-17T15:26:40.514 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-17 15:21:00.353073809 +0000 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-17 15:21:00.089076167 +0000 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-17 15:21:00.089076167 +0000 2024-09-17T15:26:40.515 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-17T15:26:40.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T15:26:40.573 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-17T15:26:40.573 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-17T15:26:40.573 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000313829 s, 1.6 MB/s 2024-09-17T15:26:40.574 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T15:26:40.624 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-09-17T15:26:40.670 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-17 15:21:00.869069199 +0000 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-17 15:21:00.593071665 +0000 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-17 15:21:00.593071665 +0000 2024-09-17T15:26:40.671 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-17T15:26:40.671 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T15:26:40.724 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-17T15:26:40.724 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-17T15:26:40.724 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000303351 s, 1.7 MB/s 2024-09-17T15:26:40.725 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T15:26:40.771 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-09-17T15:26:40.818 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T15:26:40.818 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T15:26:40.818 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-17T15:26:40.819 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T15:26:40.819 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-09-17 15:21:01.405064412 +0000 2024-09-17T15:26:40.819 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-09-17 15:21:01.133066841 +0000 2024-09-17T15:26:40.819 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-09-17 15:21:01.133066841 +0000 2024-09-17T15:26:40.819 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2024-09-17T15:26:40.819 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T15:26:40.872 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-09-17T15:26:40.873 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-09-17T15:26:40.873 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.00033458 s, 1.5 MB/s 2024-09-17T15:26:40.874 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T15:26:40.923 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:40.924 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T15:26:40.924 INFO:tasks.ceph:Generating config... 2024-09-17T15:26:40.925 INFO:tasks.ceph:[client] debug monc = 20 2024-09-17T15:26:40.925 INFO:tasks.ceph:[client] debug ms = 1 2024-09-17T15:26:40.925 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T15:26:40.925 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-17T15:26:40.925 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-17T15:26:40.925 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-17T15:26:40.926 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-17T15:26:40.926 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-17T15:26:40.926 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-17T15:26:40.926 INFO:tasks.ceph:[global] ms type = async 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-17T15:26:40.926 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-17T15:26:40.927 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-17T15:26:40.927 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-17T15:26:40.927 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T15:26:40.927 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-17T15:26:40.928 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-17T15:26:40.928 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T15:26:40.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T15:26:40.962 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T15:26:41.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T15:26:41.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:26:41.130 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.86'), ('mon.c', '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]'), ('mon.b', '172.21.15.203')] 2024-09-17T15:26:41.131 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.203', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-17T15:26:41.131 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:41.132 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:26:41.178 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.86 --addv c '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]' --add b 172.21.15.203 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = quincy 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:26:41.265 INFO:teuthology.orchestra.run.smithi086.stdout:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 17 (quincy) 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:26:41.266 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T15:26:41.267 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T15:26:41.315 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b4e90329-4d4c-4253-907f-d2ba7ae38072... 2024-09-17T15:26:41.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:26:41.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-09-17T15:26:41.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-09-17T15:26:41.389 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.203" 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon election default strategy = 1 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject delay max = 1 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject delay probability = 0.005 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject delay type = mon 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject internal delays = 0.002 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 2500 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: ms type = async 2024-09-17T15:26:41.391 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-09-17T15:26:41.392 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-09-17T15:26:41.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression algorithm = snappy 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression mode = aggressive 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore write v2 = False 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore zero block detection = True 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-09-17T15:26:41.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify cached snaps = True 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify missing on start = True 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock override recovery settings = True 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue = debug_random 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue cut off = debug_random 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.395 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 10 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-09-17T15:26:41.396 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:26:41.397 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon min osdmap epochs = 25 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune interval = 2 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune min = 15 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune txsize = 2 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub inject missing keys = 0.05 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub interval = 300 2024-09-17T15:26:41.398 INFO:teuthology.orchestra.run.smithi086.stdout: paxos service trim min = 5 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-09-17T15:26:41.399 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-09-17T15:26:41.400 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-09-17T15:26:41.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-09-17T15:26:41.402 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-09-17T15:26:41.403 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790],172.21.15.203" 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 1 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay max = 1 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay probability = 0.005 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject delay type = mon 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject internal delays = 0.002 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 2500 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:26:41.404 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-09-17T15:26:41.405 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-09-17T15:26:41.406 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression algorithm = snappy 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression mode = aggressive 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore write v2 = False 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 20 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 20 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-09-17T15:26:41.407 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-17T15:26:41.408 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 10 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-09-17T15:26:41.409 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T15:26:41.410 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 25 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject crc mismatch = 0.01 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub inject missing keys = 0.05 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 5 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-09-17T15:26:41.411 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-09-17T15:26:41.412 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-09-17T15:26:41.413 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T15:26:41.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T15:26:41.500 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T15:26:41.500 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:41.500 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T15:26:41.546 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:41.546 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T15:26:41.594 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:26:41.594 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:41.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:26:41.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:26:41.658 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:26:41.658 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:41.706 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:26:41.707 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:41.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T15:26:41.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T15:26:41.731 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:26:41.731 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:41.779 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T15:26:41.779 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T15:26:41.779 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:26:41.864 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T15:26:41.881 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T15:26:41.881 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T15:26:41.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T15:26:41.959 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T15:26:41.970 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T15:26:41.970 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-17T15:26:41.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T15:26:41.983 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:41.984 INFO:tasks.ceph:role: osd.0 2024-09-17T15:26:41.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:26:41.984 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T15:26:42.050 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:42.050 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:42.050 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:42.050 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:42.050 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:42.051 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:42.051 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:42.051 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:42.051 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:42.051 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:42.530 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-09-17T15:26:42.533 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-09-17T15:26:42.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T15:26:42.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T15:26:42.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:42.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:42.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T15:26:42.734 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:42.734 INFO:tasks.ceph:role: osd.1 2024-09-17T15:26:42.734 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:26:42.734 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:42.798 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-09-17T15:26:43.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-09-17T15:26:43.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T15:26:43.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T15:26:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:43.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:43.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T15:26:43.406 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:43.406 INFO:tasks.ceph:role: osd.2 2024-09-17T15:26:43.406 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:26:43.406 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T15:26:43.469 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:43.469 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:43.469 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:43.469 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:43.470 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:43.935 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-09-17T15:26:43.938 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-09-17T15:26:43.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T15:26:44.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T15:26:44.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:44.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:44.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:44.140+0000 7f4fdddfe7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T15:26:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:44.140+0000 7f4fdddfe7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T15:26:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:44.140+0000 7f4fdddfe7c0 -1 bdev(0x556a777d3800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T15:26:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:44.140+0000 7f4fdddfe7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T15:26:47.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T15:26:47.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:47.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:47.400+0000 7fc6ae7e37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T15:26:47.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:47.400+0000 7fc6ae7e37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T15:26:47.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:47.400+0000 7fc6ae7e37c0 -1 bdev(0x556450769800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T15:26:47.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:47.400+0000 7fc6ae7e37c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T15:26:50.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T15:26:50.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:50.668+0000 7f274bbcd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T15:26:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:50.668+0000 7f274bbcd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T15:26:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:50.668+0000 7f274bbcd7c0 -1 bdev(0x5589fd0a7800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T15:26:50.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:26:50.668+0000 7f274bbcd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T15:26:53.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T15:26:53.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T15:26:53.902 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:53.902 INFO:tasks.ceph:role: osd.3 2024-09-17T15:26:53.902 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:26:53.902 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:53.966 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:54.424 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-09-17T15:26:54.426 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-17T15:26:54.426 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-17T15:26:54.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T15:26:54.574 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:54.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T15:26:54.630 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:54.630 INFO:tasks.ceph:role: osd.4 2024-09-17T15:26:54.630 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:26:54.630 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T15:26:54.696 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:54.696 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:54.696 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:54.696 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:54.697 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:55.117 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-09-17T15:26:55.119 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-17T15:26:55.119 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-17T15:26:55.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T15:26:55.234 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:55.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T15:26:55.287 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-17T15:26:55.288 INFO:tasks.ceph:role: osd.5 2024-09-17T15:26:55.288 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:26:55.288 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T15:26:55.349 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T15:26:55.350 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T15:26:55.813 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-09-17T15:26:55.815 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-09-17T15:26:55.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-17T15:26:55.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T15:26:55.920 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: /sbin/restorecon: command not found 2024-09-17T15:26:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:26:55.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:56.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:56.032+0000 7fe3fa1527c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T15:26:56.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:56.036+0000 7fe3fa1527c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T15:26:56.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:56.036+0000 7fe3fa1527c0 -1 bdev(0x561a8df19800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T15:26:56.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:56.036+0000 7fe3fa1527c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T15:26:59.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T15:26:59.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:26:59.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:59.300+0000 7f7d98e367c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T15:26:59.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:59.300+0000 7f7d98e367c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T15:26:59.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:59.300+0000 7f7d98e367c0 -1 bdev(0x5566538a3800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T15:26:59.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:26:59.300+0000 7f7d98e367c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T15:27:02.445 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T15:27:02.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:27:02.587 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:02.588+0000 7f5dd99067c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T15:27:02.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:02.588+0000 7f5dd99067c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T15:27:02.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:02.588+0000 7f5dd99067c0 -1 bdev(0x564c77b3b800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T15:27:02.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:02.588+0000 7f5dd99067c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T15:27:05.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T15:27:05.787 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T15:27:05.787 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:05.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T15:27:05.802 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:05.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T15:27:05.858 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:05.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T15:27:05.914 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:05.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T15:27:05.927 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:05.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T15:27:05.981 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:05.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T15:27:06.037 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:06.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T15:27:06.093 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:06.093 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T15:27:06.142 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T15:27:06.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T15:27:06.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC0n+lmPVKWCBAAe+iUIOf6tLWrDdMGN5hXjQ== 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC3n+lmWLAiGBAAqQEO1ePK84ppXtnfMH6LIg== 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-09-17T15:27:06.189 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC6n+lmigExKBAASpI0IobEuaYbvJ1R3ppnYg== 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCxn+lm5L6oMxAA3JFkXSMlnW5kGpt//0dfnA== 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDAn+lmGMPzARAAJDCUK2q9mjcBGahmA5Njuw== 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDDn+lmu1cMEhAAYDZ1KHj1hW4sPbjmy6zfiA== 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDGn+lmyr8uIxAAcd53Sk9Xe2Jblg7WBhEg8A== 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-09-17T15:27:06.190 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQCxn+lm0zJTORAAX1G+uKexOOzEVXPQYZVXsA== 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQC0n+lmPVKWCBAAe+iUIOf6tLWrDdMGN5hXjQ== 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQC3n+lmWLAiGBAAqQEO1ePK84ppXtnfMH6LIg== 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQC6n+lmigExKBAASpI0IobEuaYbvJ1R3ppnYg== 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCxn+lm5L6oMxAA3JFkXSMlnW5kGpt//0dfnA== 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-09-17T15:27:06.197 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDAn+lmGMPzARAAJDCUK2q9mjcBGahmA5Njuw== 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDDn+lmu1cMEhAAYDZ1KHj1hW4sPbjmy6zfiA== 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQDGn+lmyr8uIxAAcd53Sk9Xe2Jblg7WBhEg8A== 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-09-17T15:27:06.198 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQCxn+lm0zJTORAAX1G+uKexOOzEVXPQYZVXsA== 2024-09-17T15:27:06.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.235 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T15:27:06.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T15:27:06.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.510 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T15:27:06.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T15:27:06.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T15:27:06.844 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T15:27:06.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T15:27:06.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:27:06.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T15:27:06.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T15:27:07.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:27:07.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T15:27:07.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T15:27:07.126 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T15:27:07.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T15:27:07.232 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:27:07.234 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T15:27:07.277 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T15:27:07.278 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:27:07.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:27:07.280 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:27:07.281 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:27:07.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:27:07.283 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:27:07.283 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:27:07.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:27:07.323 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:27:07.323 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T15:27:07.323 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T15:27:07.323 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T15:27:07.326 INFO:tasks.ceph.mgr.x:Started 2024-09-17T15:27:07.326 DEBUG:tasks.ceph:set 0 configs 2024-09-17T15:27:07.326 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T15:27:07.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T15:27:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 10 monclient: get_monmap_and_config 2024-09-17T15:27:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 10 monclient: build_initial_monmap 2024-09-17T15:27:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 10 monclient: monmap: 2024-09-17T15:27:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:07.442599+0000 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:07.442599+0000 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:07.441 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 1 Processor -- start 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 1 -- start start 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.436+0000 7f2d7551a640 10 monclient: init 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d700a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-b con 0x7f2d700a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-a con 0x7f2d7013d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d700a68d0 con 0x7f2d700a4f70 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2d700a64a0 con 0x7f2d7013d180 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2d7009c230 con 0x7f2d700a5350 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d7551a640 10 monclient(hunting): authenticate will time out at 960.408221s 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38732/0 (socket says 172.21.15.86:38732) 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 learned_addr learned my addr 172.21.15.86:0/231276353 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40688/0 (socket says 172.21.15.86:40688) 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d7013d180 auth_method 0 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d700a5350 auth_method 0 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.440+0000 7f2d6effd640 1 -- 172.21.15.86:0/231276353 >> v1:172.21.15.203:6789/0 conn(0x7f2d700a4f70 legacy=0x7f2d7009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:07.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d7013d180 auth_method 0 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d700a5350 auth_method 0 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:07.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:07.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:07.644+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d700a5350 auth_method 0 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d7013d180 auth_method 0 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:08.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d7013d180 auth_method 0 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d700a5350 auth_method 0 2024-09-17T15:27:08.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:08.852+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): tick 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 msgr2=0x7f2d700a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d700a5350 0x7f2d700a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 msgr2=0x7f2d7013f570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.86:3300/0 conn(0x7f2d7013d180 0x7f2d7013f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 -- 172.21.15.86:0/231276353 >> v1:172.21.15.203:6789/0 conn(0x7f2d700a4f70 legacy=0x7f2d7009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:10.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient: _add_conns ranks=[0,4] 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d5c003770 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d5c007480 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 -- 172.21.15.86:0/231276353 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d7009c510 con 0x7f2d5c003770 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 1 -- 172.21.15.86:0/231276353 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2d700a64a0 con 0x7f2d5c007480 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6cff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d5c007480 auth_method 0 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:10.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:10.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:10.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:10.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:10.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:10.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.440+0000 7f2d6effd640 1 -- 172.21.15.86:0/231276353 >> v1:172.21.15.86:6789/0 conn(0x7f2d5c003770 legacy=0x7f2d5c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d5c007480 auth_method 0 2024-09-17T15:27:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:10.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:10.644+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d5c007480 auth_method 0 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.048+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:11.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d5c007480 auth_method 0 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:11.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:11.852+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.740+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3595296258 0 0) 0x7f2d7009c510 con 0x7f2d5c003770 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): my global_id is 4098 2024-09-17T15:27:12.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d58003530 con 0x7f2d5c003770 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2530592154 0 0) 0x7f2d58003530 con 0x7f2d5c003770 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 msgr2=0x7f2d5c009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 --2- 172.21.15.86:0/231276353 >> v2:172.21.15.203:3300/0 conn(0x7f2d5c007480 0x7f2d5c009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c00b0d0 con 0x7f2d5c003770 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:12.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f2d60001a40 con 0x7f2d5c003770 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: dump: 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:12.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 >> v1:172.21.15.86:6789/0 conn(0x7f2d5c003770 legacy=0x7f2d5c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 reap_dead start 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58003b20 0x7f2d58003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): picked mon.b con 0x7f2d58003b20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 0x7f2d58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): picked mon.c con 0x7f2d58004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): picked mon.a con 0x7f2d580099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2d700a64a0 con 0x7f2d580099c0 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5800c3a0 con 0x7f2d58003b20 2024-09-17T15:27:12.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5800c5a0 con 0x7f2d58004810 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6dffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58003b20 0x7f2d58003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d580099c0 auth_method 0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): get_auth_request con 0x7f2d58003b20 auth_method 0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 0x7f2d58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 msgr2=0x7f2d5800bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 msgr2=0x7f2d5800bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d58004810 auth_method 0 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:12.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58003b20 msgr2=0x7f2d58003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58003b20 0x7f2d58003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 msgr2=0x7f2d5800bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 monclient: found mon.c 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d58003530 con 0x7f2d58004810 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:12.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d7551a640 5 monclient: authenticate success, global_id 4098 2024-09-17T15:27:12.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.744+0000 7f2d7551a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6e7fc640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 msgr2=0x7f2d58004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 0x7f2d58004c90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d64019ce0 tx=0x7f2d6401e4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a64a47c894371138 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d640b4070 con 0x7f2d58004810 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: dump: 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:12.757 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d7551a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d640af410 con 0x7f2d58004810 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d7551a640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 1 -- 172.21.15.86:0/231276353 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d64019930 con 0x7f2d58004810 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d7551a640 4 set_mon_vals no callback set 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.752+0000 7f2d6dffb640 10 monclient: dump: 2024-09-17T15:27:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:12.759 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 10 monclient: shutdown 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 msgr2=0x7f2d58004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 0x7f2d58004c90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d64019ce0 tx=0x7f2d6401e4c0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- 172.21.15.86:0/231276353 shutdown_connections 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d580099c0 0x7f2d5800bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d58004810 0x7f2d58004c90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 --2- 172.21.15.86:0/231276353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58003b20 0x7f2d58003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- 172.21.15.86:0/231276353 >> 172.21.15.86:0/231276353 conn(0x7f2d700a7240 msgr2=0x7f2d700a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- 172.21.15.86:0/231276353 shutdown_connections 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- 172.21.15.86:0/231276353 wait complete. 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 10 monclient: build_initial_monmap 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 10 monclient: monmap: 2024-09-17T15:27:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 Processor -- start 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 1 -- start start 2024-09-17T15:27:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.756+0000 7f2d7551a640 10 monclient: init 2024-09-17T15:27:12.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 0x7f2d701342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-a con 0x7f2d58003b20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58004810 0x7f2d70134810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-b con 0x7f2d58004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 0x7f2d701cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): picked mon.noname-c con 0x7f2d580099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7009d070 con 0x7f2d58003b20 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7009cef0 con 0x7f2d58004810 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2d7009d1f0 con 0x7f2d580099c0 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d7551a640 10 monclient(hunting): authenticate will time out at 965.728833s 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 0x7f2d701342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 0x7f2d701cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 0x7f2d701342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44044/0 (socket says 172.21.15.86:44044) 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 1 -- 172.21.15.86:0/3679598547 learned_addr learned my addr 172.21.15.86:0/3679598547 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): get_auth_request con 0x7f2d58003b20 auth_method 0 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d580099c0 auth_method 0 2024-09-17T15:27:12.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58004810 0x7f2d70134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request con 0x7f2d58004810 auth_method 0 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:12.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient(hunting): handle_auth_done global_id 4103 payload 824 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/3679598547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58004810 msgr2=0x7f2d70134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58004810 0x7f2d70134810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 msgr2=0x7f2d701342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 0x7f2d701342d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/3679598547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d7009e450 con 0x7f2d580099c0 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 -- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 msgr2=0x7f2d701cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:12.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.760+0000 7f2d6f7fe640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 0x7f2d701cc8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d7006cc20 tx=0x7f2d68015d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68013ba0 con 0x7f2d580099c0 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: dump: 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:12.771 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d68013d40 con 0x7f2d580099c0 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68003be0 con 0x7f2d580099c0 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 10 monclient: dump: 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:12.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d6d7fa640 4 set_mon_vals no callback set 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 5 monclient: authenticate success, global_id 4103 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _renew_subs 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d7013bf90 con 0x7f2d580099c0 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _renew_subs 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d7013b740 con 0x7f2d580099c0 2024-09-17T15:27:12.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d680043c0 con 0x7f2d580099c0 2024-09-17T15:27:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2d680223f0 con 0x7f2d580099c0 2024-09-17T15:27:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.764+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d700a4600 con 0x7f2d580099c0 2024-09-17T15:27:12.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.776+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2d700a4600 con 0x7f2d580099c0 2024-09-17T15:27:12.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.776+0000 7f2d577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:12.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.776+0000 7f2d577fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.840+0000 7f2d7551a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:27:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.840+0000 7f2d7551a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:27:12.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.840+0000 7f2d7551a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:12.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.840+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f2d700a5750 con 0x7f2d580099c0 2024-09-17T15:27:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d577fe640 1 -- 172.21.15.86:0/3679598547 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f2d68027070 con 0x7f2d580099c0 2024-09-17T15:27:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-17T15:27:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d577fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-17T15:27:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 10 monclient: shutdown 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 msgr2=0x7f2d701cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 0x7f2d701cc8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2d7006cc20 tx=0x7f2d68015d10 comp rx=0 tx=0).stop 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 shutdown_connections 2024-09-17T15:27:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d580099c0 0x7f2d701cc8d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2d58004810 0x7f2d70134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 --2- 172.21.15.86:0/3679598547 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d58003b20 0x7f2d701342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:12.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 >> 172.21.15.86:0/3679598547 conn(0x7f2d700a7240 msgr2=0x7f2d70138240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:12.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 shutdown_connections 2024-09-17T15:27:12.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:12.848+0000 7f2d7551a640 1 -- 172.21.15.86:0/3679598547 wait complete. 2024-09-17T15:27:12.872 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T15:27:12.872 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T15:27:12.872 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:12.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T15:27:12.884 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:12.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T15:27:12.936 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-09-17T15:27:12.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T15:27:12.988 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:12.988 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T15:27:13.001 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:13.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T15:27:13.057 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-09-17T15:27:13.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T15:27:13.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 09486956-c4be-472a-a7b7-ae1b3e855a71 0 2024-09-17T15:27:13.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: get_monmap_and_config 2024-09-17T15:27:13.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: build_initial_monmap 2024-09-17T15:27:13.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: monmap: 2024-09-17T15:27:13.271 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:13.274234+0000 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:13.274234+0000 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:13.272 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:13.273 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:13.273 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 Processor -- start 2024-09-17T15:27:13.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 -- start start 2024-09-17T15:27:13.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: init 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93ac132950 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac13aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac132420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93ac13b1e0 0x7f93ac13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac13b1e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ac070940 con 0x7f93ac132950 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f93ac073390 con 0x7f93ac13b1e0 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f93ac073930 con 0x7f93ac132420 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93b0f9d640 10 monclient(hunting): authenticate will time out at 964.183321s 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93ac13b1e0 0x7f93ac13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aa59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f93ac132950 0x7f93ac132d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59784/0 (socket says 172.21.15.203:59784) 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aa59c640 1 -- 172.21.15.203:0/2322912392 learned_addr learned my addr 172.21.15.203:0/2322912392 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-17T15:27:13.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): get_auth_request con 0x7f93ac13b1e0 auth_method 0 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac13aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 -- 172.21.15.203:0/2322912392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 msgr2=0x7f93ac13aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac13aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.278 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 -- 172.21.15.203:0/2322912392 >> v1:172.21.15.203:6789/0 conn(0x7f93ac132950 legacy=0x7f93ac132d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:13.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:13.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 monclient: found mon.noname-c 2024-09-17T15:27:13.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:13.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.279 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.272+0000 7f93aad9d640 1 -- 172.21.15.203:0/2322912392 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac074610 con 0x7f93ac13b1e0 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.276+0000 7f93aad9d640 1 -- 172.21.15.203:0/2322912392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac13b1e0 msgr2=0x7f93ac13d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.276+0000 7f93aad9d640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac13b1e0 0x7f93ac13d5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009d90 tx=0x7f93a401c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6415a9fa04698be server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 1 -- 172.21.15.203:0/2322912392 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a40a67f0 con 0x7f93ac13b1e0 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: dump: 2024-09-17T15:27:13.283 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: _finish_auth 0 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:13.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 1 -- 172.21.15.203:0/2322912392 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93a40a6990 con 0x7f93ac13b1e0 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 1 -- 172.21.15.203:0/2322912392 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a40a6d60 con 0x7f93ac13b1e0 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 5 monclient: authenticate success, global_id 4105 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 4 set_mon_vals no callback set 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a959a640 10 monclient: dump: 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.285 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: shutdown 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- 172.21.15.203:0/2322912392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac13b1e0 msgr2=0x7f93ac13d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac13b1e0 0x7f93ac13d5d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009d90 tx=0x7f93a401c8c0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- 172.21.15.203:0/2322912392 shutdown_connections 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac13b1e0 0x7f93ac13d5d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/2322912392 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac13aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- 172.21.15.203:0/2322912392 >> 172.21.15.203:0/2322912392 conn(0x7f93ac0a7210 msgr2=0x7f93ac0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- 172.21.15.203:0/2322912392 shutdown_connections 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- 172.21.15.203:0/2322912392 wait complete. 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: build_initial_monmap 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: monmap: 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:13.287 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 Processor -- start 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- start start 2024-09-17T15:27:13.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: init 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac0a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac132420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-b con 0x7f93ac132950 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f93ac13a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac13b1e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac1406c0 con 0x7f93ac13b1e0 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.289 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f93ac140540 con 0x7f93ac132950 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f93ac140840 con 0x7f93ac132420 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93b0f9d640 10 monclient(hunting): authenticate will time out at 964.190351s 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51856/0 (socket says 172.21.15.203:51856) 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 learned_addr learned my addr 172.21.15.203:0/115782650 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): get_auth_request con 0x7f93ac132950 auth_method 0 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93aa59c640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac0a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 msgr2=0x7f93ac0a0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f93ac0a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 msgr2=0x7f93ac13a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f93ac13a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 monclient: found mon.noname-b 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:13.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ac13cc60 con 0x7f93ac132950 2024-09-17T15:27:13.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 msgr2=0x7f93ac139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:27:13.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 msgr2=0x7f93ac139b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:27:13.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c820 tx=0x7f939c00cce0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:13.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.280+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient: get_auth_request con 0x7f93ac132950 auth_method 0 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 0 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 msgr2=0x7f93ac139b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93ac132950 0x7f93ac139b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:13.484 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f938c0054e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): picked mon.noname-c con 0x7f93ac132420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 0x7f938c0083d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): picked mon.noname-b con 0x7f938c005f40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f938c00af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): picked mon.noname-a con 0x7f93ac13b1e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f938c00b9a0 con 0x7f93ac13b1e0 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f938c00bb20 con 0x7f938c005f40 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f938c00bd20 con 0x7f93ac132420 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f939affd640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:13.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 0x7f938c0083d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): get_auth_request con 0x7f938c005f40 auth_method 0 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f938c00af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 10 monclient(hunting): get_auth_request con 0x7f93ac13b1e0 auth_method 0 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:13.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aa59c640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f938c0054e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aa59c640 10 monclient(hunting): get_auth_request con 0x7f93ac132420 auth_method 0 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient(hunting): handle_auth_done global_id 4108 payload 293 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 msgr2=0x7f938c0054e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f938c0054e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 msgr2=0x7f938c00af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f938c00af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 monclient: found mon.noname-b 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938c00bfe0 con 0x7f938c005f40 2024-09-17T15:27:13.488 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.484+0000 7f93aad9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f938c00af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93a9d9b640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 msgr2=0x7f938c0083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93a9d9b640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 0x7f938c0083d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c820 tx=0x7f939c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c0077b0 con 0x7f938c005f40 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: dump: 2024-09-17T15:27:13.489 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: _finish_auth 0 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:13.490 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f939c020070 con 0x7f938c005f40 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 5 monclient: authenticate success, global_id 4108 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _renew_subs 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ac06ef00 con 0x7f938c005f40 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939c01b5f0 con 0x7f938c005f40 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 10 monclient: dump: 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.491 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93a8d99640 4 set_mon_vals no callback set 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _renew_subs 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ac0a0a30 con 0x7f938c005f40 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f939c01b790 con 0x7f938c005f40 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f939c0257d0 con 0x7f938c005f40 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.492 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.493 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.488+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ac132d30 con 0x7f938c005f40 2024-09-17T15:27:13.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.504+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f939c018070 con 0x7f938c005f40 2024-09-17T15:27:13.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.504+0000 7f939affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.504+0000 7f939affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:13.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.592+0000 7f93b0f9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "09486956-c4be-472a-a7b7-ae1b3e855a71", "id": 0}] 2024-09-17T15:27:13.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.592+0000 7f93b0f9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "09486956-c4be-472a-a7b7-ae1b3e855a71", "id": 0}] 2024-09-17T15:27:13.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.592+0000 7f93b0f9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.592+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "09486956-c4be-472a-a7b7-ae1b3e855a71", "id": 0} v 0) -- 0x7f93ac135e40 con 0x7f938c005f40 2024-09-17T15:27:13.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f939affd640 1 -- 172.21.15.203:0/115782650 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "09486956-c4be-472a-a7b7-ae1b3e855a71", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f939c01bad0 con 0x7f938c005f40 2024-09-17T15:27:13.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f939affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "09486956-c4be-472a-a7b7-ae1b3e855a71", "id": 0}] 2024-09-17T15:27:13.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f939affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:13.599 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-09-17T15:27:13.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 10 monclient: shutdown 2024-09-17T15:27:13.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:13.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 msgr2=0x7f938c0083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 0x7f938c0083d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c820 tx=0x7f939c008040 comp rx=0 tx=0).stop 2024-09-17T15:27:13.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 shutdown_connections 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93ac13b1e0 0x7f938c00af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f938c005f40 0x7f938c0083d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 --2- 172.21.15.203:0/115782650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93ac132420 0x7f938c0054e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 >> 172.21.15.203:0/115782650 conn(0x7f93ac0a7210 msgr2=0x7f93ac0a2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 shutdown_connections 2024-09-17T15:27:13.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.600+0000 7f93b0f9d640 1 -- 172.21.15.203:0/115782650 wait complete. 2024-09-17T15:27:13.615 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new c2fa0454-8b28-4acd-8c63-83331707cc17 1 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: get_monmap_and_config 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: build_initial_monmap 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: monmap: 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:13.728703+0000 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:13.728703+0000 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:13.726 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:13.727 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:13.727 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:13.727 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 Processor -- start 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 -- start start 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: init 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:27:13.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f04400a2d90 0x7f044009e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f04400a2d90 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04400a3170 0x7f04400a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-a con 0x7f04400a3170 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0440135b50 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f044009cfb0 con 0x7f0440135b50 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f04400a4960 con 0x7f04400a3170 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f044009d460 con 0x7f04400a2d90 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f0444b2a640 10 monclient(hunting): authenticate will time out at 964.637754s 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f04400a2d90 0x7f044009e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f04400a2d90 0x7f044009e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51890/0 (socket says 172.21.15.203:51890) 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 -- 172.21.15.203:0/2874630107 learned_addr learned my addr 172.21.15.203:0/2874630107 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): get_auth_request con 0x7f04400a2d90 auth_method 0 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043dd9b640 1 --2- 172.21.15.203:0/2874630107 >> v2:172.21.15.86:3300/0 conn(0x7f04400a3170 0x7f04400a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043dd9b640 10 monclient(hunting): get_auth_request con 0x7f04400a3170 auth_method 0 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 -- 172.21.15.203:0/2874630107 >> v2:172.21.15.86:3300/0 conn(0x7f04400a3170 msgr2=0x7f04400a3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 --2- 172.21.15.203:0/2874630107 >> v2:172.21.15.86:3300/0 conn(0x7f04400a3170 0x7f04400a3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 -- 172.21.15.203:0/2874630107 >> v1:172.21.15.86:6789/0 conn(0x7f0440135b50 legacy=0x7f0440135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 monclient: found mon.noname-c 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.728+0000 7f043e59c640 1 -- 172.21.15.203:0/2874630107 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044009d740 con 0x7f04400a2d90 2024-09-17T15:27:13.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043e59c640 1 -- 172.21.15.203:0/2874630107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04400a2d90 msgr2=0x7f044009e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:13.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043e59c640 1 --2- 172.21.15.203:0/2874630107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04400a2d90 0x7f044009e460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04400a2a50 tx=0x7f043001c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=967fc5ca632831f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:13.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 1 -- 172.21.15.203:0/2874630107 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300b0070 con 0x7f04400a2d90 2024-09-17T15:27:13.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: dump: 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: _finish_auth 0 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:13.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 1 -- 172.21.15.203:0/2874630107 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04300ab600 con 0x7f04400a2d90 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f0444b2a640 5 monclient: authenticate success, global_id 4111 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f0444b2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f0444b2a640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f0444b2a640 4 set_mon_vals no callback set 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 1 -- 172.21.15.203:0/2874630107 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300ab9d0 con 0x7f04400a2d90 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.732+0000 7f043d59a640 10 monclient: dump: 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.735 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.736 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: shutdown 2024-09-17T15:27:13.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- 172.21.15.203:0/2874630107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04400a2d90 msgr2=0x7f044009e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- 172.21.15.203:0/2874630107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04400a2d90 0x7f044009e460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04400a2a50 tx=0x7f043001c340 comp rx=0 tx=0).stop 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- 172.21.15.203:0/2874630107 shutdown_connections 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- 172.21.15.203:0/2874630107 >> v2:172.21.15.86:3300/0 conn(0x7f04400a3170 0x7f04400a3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- 172.21.15.203:0/2874630107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04400a2d90 0x7f044009e460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- 172.21.15.203:0/2874630107 >> 172.21.15.203:0/2874630107 conn(0x7f04400a8390 msgr2=0x7f04400a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- 172.21.15.203:0/2874630107 shutdown_connections 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- 172.21.15.203:0/2874630107 wait complete. 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: build_initial_monmap 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: monmap: 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:13.741 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 Processor -- start 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- start start 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: init 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:13.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04400a2d90 0x7f044013da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-c con 0x7f04400a2d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04400a3170 0x7f044013b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-a con 0x7f04400a3170 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): picked mon.noname-b con 0x7f0440135b50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0440140960 con 0x7f04400a3170 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04401407e0 con 0x7f0440135b50 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0440140ae0 con 0x7f04400a2d90 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f0444b2a640 10 monclient(hunting): authenticate will time out at 964.646555s 2024-09-17T15:27:13.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51898/0 (socket says 172.21.15.203:51898) 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 -- 172.21.15.203:0/1888968066 learned_addr learned my addr 172.21.15.203:0/1888968066 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): get_auth_request con 0x7f0440135b50 auth_method 0 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043dd9b640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04400a3170 0x7f044013b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043e59c640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04400a2d90 0x7f044013da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043dd9b640 10 monclient(hunting): get_auth_request con 0x7f04400a3170 auth_method 0 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043e59c640 10 monclient(hunting): get_auth_request con 0x7f04400a2d90 auth_method 0 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 -- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04400a2d90 msgr2=0x7f044013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04400a2d90 0x7f044013da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 -- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04400a3170 msgr2=0x7f044013b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04400a3170 0x7f044013b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 monclient: found mon.noname-b 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:13.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.736+0000 7f043ed9d640 1 -- 172.21.15.203:0/1888968066 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044011f950 con 0x7f0440135b50 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f043ed9d640 1 -- 172.21.15.203:0/1888968066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 msgr2=0x7f044013b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f043ed9d640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f043800c780 tx=0x7f043800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f043800ac60 con 0x7f0440135b50 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: dump: 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.746 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: _finish_auth 0 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 5 monclient: authenticate success, global_id 4114 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f043800ae00 con 0x7f0440135b50 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _renew_subs 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0440074bd0 con 0x7f0440135b50 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0438004bb0 con 0x7f0440135b50 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:13.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042ffff640 4 set_mon_vals no callback set 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 10 monclient: dump: 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _renew_subs 2024-09-17T15:27:13.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044011ee30 con 0x7f0440135b50 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0438005030 con 0x7f0440135b50 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f04380073e0 con 0x7f0440135b50 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.740+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044009e460 con 0x7f0440135b50 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.744+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f0438020070 con 0x7f0440135b50 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.744+0000 7f042effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:13.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.744+0000 7f042effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:13.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.812+0000 7f0444b2a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c2fa0454-8b28-4acd-8c63-83331707cc17", "id": 1}] 2024-09-17T15:27:13.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.812+0000 7f0444b2a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c2fa0454-8b28-4acd-8c63-83331707cc17", "id": 1}] 2024-09-17T15:27:13.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.812+0000 7f0444b2a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:13.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:13.812+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c2fa0454-8b28-4acd-8c63-83331707cc17", "id": 1} v 0) -- 0x7f0440073ea0 con 0x7f0440135b50 2024-09-17T15:27:14.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.528+0000 7f042effd640 1 -- 172.21.15.203:0/1888968066 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2fa0454-8b28-4acd-8c63-83331707cc17", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0438005550 con 0x7f0440135b50 2024-09-17T15:27:14.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.528+0000 7f042effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c2fa0454-8b28-4acd-8c63-83331707cc17", "id": 1}] 2024-09-17T15:27:14.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.528+0000 7f042effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:14.529 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-09-17T15:27:14.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.528+0000 7f0444b2a640 10 monclient: shutdown 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 msgr2=0x7f044013b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f043800c780 tx=0x7f043800cc40 comp rx=0 tx=0).stop 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 shutdown_connections 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0440135b50 0x7f044013b980 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04400a3170 0x7f044013b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 --2- 172.21.15.203:0/1888968066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04400a2d90 0x7f044013da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 >> 172.21.15.203:0/1888968066 conn(0x7f04400a8390 msgr2=0x7f0440073790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 shutdown_connections 2024-09-17T15:27:14.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.532+0000 7f0444b2a640 1 -- 172.21.15.203:0/1888968066 wait complete. 2024-09-17T15:27:14.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 7eafb358-7785-40f5-a8a4-dc97082623ca 2 2024-09-17T15:27:14.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: get_monmap_and_config 2024-09-17T15:27:14.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: build_initial_monmap 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: monmap: 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:14.669362+0000 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:14.669362+0000 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:14.667 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 Processor -- start 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 -- start start 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: init 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46cc0a0450 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46cc074860 0x7f46cc0a2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-c con 0x7f46cc074860 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46cc13ce70 0x7f46cc13f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-a con 0x7f46cc13ce70 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46cc0a2f20 con 0x7f46cc0a0450 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc0a0830 con 0x7f46cc13ce70 2024-09-17T15:27:14.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc070430 con 0x7f46cc074860 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46d129a640 10 monclient(hunting): authenticate will time out at 965.578555s 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46cc074860 0x7f46cc0a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46cc074860 0x7f46cc0a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51910/0 (socket says 172.21.15.203:51910) 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 -- 172.21.15.203:0/575516708 learned_addr learned my addr 172.21.15.203:0/575516708 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): get_auth_request con 0x7f46cc074860 auth_method 0 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:14.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46cb7fe640 1 --2- 172.21.15.203:0/575516708 >> v2:172.21.15.86:3300/0 conn(0x7f46cc13ce70 0x7f46cc13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 -- 172.21.15.203:0/575516708 >> v2:172.21.15.86:3300/0 conn(0x7f46cc13ce70 msgr2=0x7f46cc13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 --2- 172.21.15.203:0/575516708 >> v2:172.21.15.86:3300/0 conn(0x7f46cc13ce70 0x7f46cc13f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 -- 172.21.15.203:0/575516708 >> v1:172.21.15.86:6789/0 conn(0x7f46cc0a0450 legacy=0x7f46cc074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.668+0000 7f46c3fff640 1 -- 172.21.15.203:0/575516708 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc070710 con 0x7f46cc074860 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46c3fff640 1 -- 172.21.15.203:0/575516708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc074860 msgr2=0x7f46cc0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46c3fff640 1 --2- 172.21.15.203:0/575516708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc074860 0x7f46cc0a2860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009870 tx=0x7f46b401c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50cf67d9fe19ec81 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 1 -- 172.21.15.203:0/575516708 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40ab600 con 0x7f46cc074860 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: dump: 2024-09-17T15:27:14.676 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 1 -- 172.21.15.203:0/575516708 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46b40b0070 con 0x7f46cc074860 2024-09-17T15:27:14.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 5 monclient: authenticate success, global_id 4117 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 1 -- 172.21.15.203:0/575516708 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b40ab9c0 con 0x7f46cc074860 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 4 set_mon_vals no callback set 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46ca7fc640 10 monclient: dump: 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.678 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 10 monclient: shutdown 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 1 -- 172.21.15.203:0/575516708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc074860 msgr2=0x7f46cc0a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.672+0000 7f46d129a640 1 --2- 172.21.15.203:0/575516708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc074860 0x7f46cc0a2860 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009870 tx=0x7f46b401c440 comp rx=0 tx=0).stop 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- 172.21.15.203:0/575516708 shutdown_connections 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 --2- 172.21.15.203:0/575516708 >> v2:172.21.15.86:3300/0 conn(0x7f46cc13ce70 0x7f46cc13f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 --2- 172.21.15.203:0/575516708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc074860 0x7f46cc0a2860 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- 172.21.15.203:0/575516708 >> 172.21.15.203:0/575516708 conn(0x7f46cc0a3710 msgr2=0x7f46cc0a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- 172.21.15.203:0/575516708 shutdown_connections 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- 172.21.15.203:0/575516708 wait complete. 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient: build_initial_monmap 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient: monmap: 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:14.680 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 Processor -- start 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- start start 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient: init 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46cc074860 0x7f46cc134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-a con 0x7f46cc074860 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46cc0a0450 0x7f46cc1cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-c con 0x7f46cc0a0450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): picked mon.noname-b con 0x7f46cc13ce70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc1423d0 con 0x7f46cc074860 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc142250 con 0x7f46cc13ce70 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc142550 con 0x7f46cc0a0450 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46d129a640 10 monclient(hunting): authenticate will time out at 965.585574s 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51918/0 (socket says 172.21.15.203:51918) 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 -- 172.21.15.203:0/3863190468 learned_addr learned my addr 172.21.15.203:0/3863190468 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): get_auth_request con 0x7f46cc13ce70 auth_method 0 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46caffd640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46cc074860 0x7f46cc134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46c3fff640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46cc0a0450 0x7f46cc1cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46caffd640 10 monclient(hunting): get_auth_request con 0x7f46cc074860 auth_method 0 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46caffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46c3fff640 10 monclient(hunting): get_auth_request con 0x7f46cc0a0450 auth_method 0 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 -- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46cc0a0450 msgr2=0x7f46cc1cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46cc0a0450 0x7f46cc1cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 -- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46cc074860 msgr2=0x7f46cc134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46cc074860 0x7f46cc134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.676+0000 7f46cb7fe640 1 -- 172.21.15.203:0/3863190468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc1351c0 con 0x7f46cc13ce70 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46cb7fe640 1 -- 172.21.15.203:0/3863190468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 msgr2=0x7f46cc1d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46cb7fe640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46c400cd70 tx=0x7f46c400a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4003ba0 con 0x7f46cc13ce70 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: dump: 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.686 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c4003d40 con 0x7f46cc13ce70 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4004240 con 0x7f46cc13ce70 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 10 monclient: dump: 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c9ffb640 4 set_mon_vals no callback set 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 5 monclient: authenticate success, global_id 4120 2024-09-17T15:27:14.688 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _renew_subs 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc13cbf0 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _renew_subs 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1386b0 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f46c400e600 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f46c4016850 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.680+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc13c770 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.684+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f46c40047f0 con 0x7f46cc13ce70 2024-09-17T15:27:14.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.684+0000 7f46c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.684+0000 7f46c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:14.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.756+0000 7f46d129a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7eafb358-7785-40f5-a8a4-dc97082623ca", "id": 2}] 2024-09-17T15:27:14.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.756+0000 7f46d129a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7eafb358-7785-40f5-a8a4-dc97082623ca", "id": 2}] 2024-09-17T15:27:14.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.756+0000 7f46d129a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.756+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7eafb358-7785-40f5-a8a4-dc97082623ca", "id": 2} v 0) -- 0x7f46cc0a2860 con 0x7f46cc13ce70 2024-09-17T15:27:14.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46c37fe640 1 -- 172.21.15.203:0/3863190468 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7eafb358-7785-40f5-a8a4-dc97082623ca", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f46c4016430 con 0x7f46cc13ce70 2024-09-17T15:27:14.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7eafb358-7785-40f5-a8a4-dc97082623ca", "id": 2}] 2024-09-17T15:27:14.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:14.760 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-09-17T15:27:14.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 10 monclient: shutdown 2024-09-17T15:27:14.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:14.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 msgr2=0x7f46cc1d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f46c400cd70 tx=0x7f46c400a660 comp rx=0 tx=0).stop 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 shutdown_connections 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46cc13ce70 0x7f46cc1d04a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46cc0a0450 0x7f46cc1cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 --2- 172.21.15.203:0/3863190468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46cc074860 0x7f46cc134760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 >> 172.21.15.203:0/3863190468 conn(0x7f46cc0a3710 msgr2=0x7f46cc073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 shutdown_connections 2024-09-17T15:27:14.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.760+0000 7f46d129a640 1 -- 172.21.15.203:0/3863190468 wait complete. 2024-09-17T15:27:14.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 32ba0b4f-a5b8-4eee-8404-2823bf95ebc3 3 2024-09-17T15:27:14.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: get_monmap_and_config 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: build_initial_monmap 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: monmap: 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:14.890428+0000 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:14.890428+0000 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:14.888 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:14.889 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:14.889 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:14.889 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 Processor -- start 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 -- start start 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: init 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f881c0a50e0 0x7f881c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-c con 0x7f881c0a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:14.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f881c0a54c0 0x7f881c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-a con 0x7f881c0a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c137ea0 0x7f881c138350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-b con 0x7f881c137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.888+0000 7f88240d8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f881c0a6280 con 0x7f881c0a54c0 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f881c09c0a0 con 0x7f881c0a50e0 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f881c09c380 con 0x7f881c137ea0 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f88240d8640 10 monclient(hunting): authenticate will time out at 965.800026s 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f881c0a50e0 0x7f881c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f881c0a50e0 0x7f881c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51930/0 (socket says 172.21.15.203:51930) 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 -- 172.21.15.203:0/2544231025 learned_addr learned my addr 172.21.15.203:0/2544231025 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:14.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): get_auth_request con 0x7f881c0a50e0 auth_method 0 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821673640 1 --2- 172.21.15.203:0/2544231025 >> v2:172.21.15.86:3300/0 conn(0x7f881c0a54c0 0x7f881c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8822675640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c137ea0 0x7f881c138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821673640 10 monclient(hunting): get_auth_request con 0x7f881c0a54c0 auth_method 0 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821673640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821673640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821673640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8822675640 10 monclient(hunting): get_auth_request con 0x7f881c137ea0 auth_method 0 2024-09-17T15:27:14.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8822675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8822675640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8822675640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 -- 172.21.15.203:0/2544231025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c137ea0 msgr2=0x7f881c138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c137ea0 0x7f881c138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 -- 172.21.15.203:0/2544231025 >> v2:172.21.15.86:3300/0 conn(0x7f881c0a54c0 msgr2=0x7f881c0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 --2- 172.21.15.203:0/2544231025 >> v2:172.21.15.86:3300/0 conn(0x7f881c0a54c0 0x7f881c0a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient: _finish_hunting 0 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 monclient: found mon.noname-c 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.892+0000 7f8821e74640 1 -- 172.21.15.203:0/2544231025 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c09c660 con 0x7f881c0a50e0 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8821e74640 1 -- 172.21.15.203:0/2544231025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c0a50e0 msgr2=0x7f881c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8821e74640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c0a50e0 0x7f881c09cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f881c0aa620 tx=0x7f880c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21d2a9b693be183f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 1 -- 172.21.15.203:0/2544231025 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0b1070 con 0x7f881c0a50e0 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: dump: 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.902 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: _finish_auth 0 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: _check_auth_tickets 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 5 monclient: authenticate success, global_id 4123 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:14.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 1 -- 172.21.15.203:0/2544231025 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f880c0ac620 con 0x7f881c0a50e0 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 4 set_mon_vals no callback set 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 1 -- 172.21.15.203:0/2544231025 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c0ac9f0 con 0x7f881c0a50e0 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8820e72640 10 monclient: dump: 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.904 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: shutdown 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- 172.21.15.203:0/2544231025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c0a50e0 msgr2=0x7f881c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c0a50e0 0x7f881c09cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f881c0aa620 tx=0x7f880c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- 172.21.15.203:0/2544231025 shutdown_connections 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c137ea0 0x7f881c138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.905 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- 172.21.15.203:0/2544231025 >> v2:172.21.15.86:3300/0 conn(0x7f881c0a54c0 0x7f881c0a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- 172.21.15.203:0/2544231025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c0a50e0 0x7f881c09cd60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- 172.21.15.203:0/2544231025 >> 172.21.15.203:0/2544231025 conn(0x7f881c0aaae0 msgr2=0x7f881c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- 172.21.15.203:0/2544231025 shutdown_connections 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- 172.21.15.203:0/2544231025 wait complete. 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: build_initial_monmap 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: monmap: 2024-09-17T15:27:14.906 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:14.907 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:14.907 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:14.907 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 Processor -- start 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- start start 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: init 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:14.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f881c0a50e0 0x7f881c1cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-a con 0x7f881c0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c0a54c0 0x7f881c137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-c con 0x7f881c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): picked mon.noname-b con 0x7f881c137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0a6ae0 con 0x7f881c0a50e0 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f881c0a6960 con 0x7f881c137ea0 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f881c141b20 con 0x7f881c0a54c0 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:14.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f88240d8640 10 monclient(hunting): authenticate will time out at 965.807366s 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51936/0 (socket says 172.21.15.203:51936) 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 -- 172.21.15.203:0/3973043843 learned_addr learned my addr 172.21.15.203:0/3973043843 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): get_auth_request con 0x7f881c137ea0 auth_method 0 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-17T15:27:14.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 -- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c0a54c0 msgr2=0x7f881c137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c0a54c0 0x7f881c137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 -- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f881c0a50e0 msgr2=0x7f881c1cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8821e74640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f881c0a50e0 0x7f881c1cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f881c0a50e0 0x7f881c1cfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient: _finish_hunting 0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 monclient: found mon.noname-b 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.896+0000 7f8822675640 1 -- 172.21.15.203:0/3973043843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c13dd40 con 0x7f881c137ea0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f8822675640 1 -- 172.21.15.203:0/3973043843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 msgr2=0x7f881c13c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f8822675640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8814007e70 tx=0x7f881400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814013790 con 0x7f881c137ea0 2024-09-17T15:27:14.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: dump: 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:14.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8814019070 con 0x7f881c137ea0 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 5 monclient: authenticate success, global_id 4126 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _renew_subs 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881c0a7af0 con 0x7f881c137ea0 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8814013b90 con 0x7f881c137ea0 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 10 monclient: dump: 2024-09-17T15:27:14.913 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f8813fff640 4 set_mon_vals no callback set 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _renew_subs 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.914 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881c122ab0 con 0x7f881c137ea0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f881401e460 con 0x7f881c137ea0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f8814014ae0 con 0x7f881c137ea0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.900+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f881c0a58d0 con 0x7f881c137ea0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.904+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f8814014d80 con 0x7f881c137ea0 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.904+0000 7f88127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:14.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.904+0000 7f88127fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:14.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.972+0000 7f88240d8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "32ba0b4f-a5b8-4eee-8404-2823bf95ebc3", "id": 3}] 2024-09-17T15:27:14.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.972+0000 7f88240d8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "32ba0b4f-a5b8-4eee-8404-2823bf95ebc3", "id": 3}] 2024-09-17T15:27:14.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.972+0000 7f88240d8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:14.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.972+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "32ba0b4f-a5b8-4eee-8404-2823bf95ebc3", "id": 3} v 0) -- 0x7f881c1225f0 con 0x7f881c137ea0 2024-09-17T15:27:14.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88127fc640 1 -- 172.21.15.203:0/3973043843 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32ba0b4f-a5b8-4eee-8404-2823bf95ebc3", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f8814013d30 con 0x7f881c137ea0 2024-09-17T15:27:14.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "32ba0b4f-a5b8-4eee-8404-2823bf95ebc3", "id": 3}] 2024-09-17T15:27:14.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88127fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:14.979 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-09-17T15:27:14.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 10 monclient: shutdown 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 msgr2=0x7f881c13c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8814007e70 tx=0x7f881400e4b0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 shutdown_connections 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f881c137ea0 0x7f881c13c280 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f881c0a54c0 0x7f881c137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 --2- 172.21.15.203:0/3973043843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f881c0a50e0 0x7f881c1cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 >> 172.21.15.203:0/3973043843 conn(0x7f881c0aaae0 msgr2=0x7f881c0a3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 shutdown_connections 2024-09-17T15:27:14.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:14.980+0000 7f88240d8640 1 -- 172.21.15.203:0/3973043843 wait complete. 2024-09-17T15:27:14.996 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new b14d69b1-ee94-4f5a-b210-7893cb8790ea 4 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 10 monclient: get_monmap_and_config 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 10 monclient: monmap: 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:15.115042+0000 2024-09-17T15:27:15.112 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:15.115042+0000 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:15.113 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 1 Processor -- start 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 1 -- start start 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.112+0000 7ff66286b640 10 monclient: init 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff65c1381a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff65c06fb70 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff65c0a3da0 0x7ff65c140ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-c con 0x7ff65c0a3da0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:15.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65c06edd0 con 0x7ff65c1381a0 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65c06f0b0 con 0x7ff65c06fb70 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff65c0a4ab0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff66286b640 10 monclient(hunting): authenticate will time out at 966.024223s 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff65c0a3da0 0x7ff65c140ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff65b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff65c06fb70 0x7ff65c13d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59790/0 (socket says 172.21.15.203:59790) 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1162783332 learned_addr learned my addr 172.21.15.203:0/1162783332 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): get_auth_request con 0x7ff65c0a3da0 auth_method 0 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 1 -- 172.21.15.203:0/1162783332 >> v1:172.21.15.203:6789/0 conn(0x7ff65c06fb70 legacy=0x7ff65c13d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 1 -- 172.21.15.203:0/1162783332 >> v1:172.21.15.86:6789/0 conn(0x7ff65c1381a0 legacy=0x7ff65c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 1 monclient: found mon.noname-c 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.116+0000 7ff660e08640 1 -- 172.21.15.203:0/1162783332 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c138c90 con 0x7ff65c0a3da0 2024-09-17T15:27:15.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff660e08640 1 -- 172.21.15.203:0/1162783332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 msgr2=0x7ff65c140ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff660e08640 1 --2- 172.21.15.203:0/1162783332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c140ac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff65c138f20 tx=0x7ff65401c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7322392ecb2952ee server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 1 -- 172.21.15.203:0/1162783332 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540ac070 con 0x7ff65c0a3da0 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: dump: 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.123 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 5 monclient: authenticate success, global_id 4129 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 1 -- 172.21.15.203:0/1162783332 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6540a6740 con 0x7ff65c0a3da0 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 4 set_mon_vals no callback set 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 1 -- 172.21.15.203:0/1162783332 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6540a6b10 con 0x7ff65c0a3da0 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:15.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65affd640 10 monclient: dump: 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: shutdown 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- 172.21.15.203:0/1162783332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 msgr2=0x7ff65c140ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 --2- 172.21.15.203:0/1162783332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c140ac0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff65c138f20 tx=0x7ff65401c810 comp rx=0 tx=0).stop 2024-09-17T15:27:15.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- 172.21.15.203:0/1162783332 shutdown_connections 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 --2- 172.21.15.203:0/1162783332 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c140ac0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- 172.21.15.203:0/1162783332 >> 172.21.15.203:0/1162783332 conn(0x7ff65c0a8390 msgr2=0x7ff65c0a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- 172.21.15.203:0/1162783332 shutdown_connections 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- 172.21.15.203:0/1162783332 wait complete. 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: monmap: 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:15.126 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 Processor -- start 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- start start 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: init 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c06fb70 0x7ff65c137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-a con 0x7ff65c06fb70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-b con 0x7ff65c0a3da0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c1381a0 0x7ff65c13a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): picked mon.noname-c con 0x7ff65c1381a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c143a30 con 0x7ff65c06fb70 2024-09-17T15:27:15.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c143bb0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c143d30 con 0x7ff65c1381a0 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff66286b640 10 monclient(hunting): authenticate will time out at 966.031143s 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51956/0 (socket says 172.21.15.203:51956) 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1996380339 learned_addr learned my addr 172.21.15.203:0/1996380339 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): get_auth_request con 0x7ff65c0a3da0 auth_method 0 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65bfff640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c06fb70 0x7ff65c137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65bfff640 10 monclient(hunting): get_auth_request con 0x7ff65c06fb70 auth_method 0 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c1381a0 msgr2=0x7ff65c13a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c1381a0 0x7ff65c13a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c06fb70 msgr2=0x7ff65c137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c06fb70 0x7ff65c137190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.120+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1996380339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c13ba90 con 0x7ff65c0a3da0 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff65b7fe640 1 -- 172.21.15.203:0/1996380339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 msgr2=0x7ff65c1376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff65b7fe640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c470 tx=0x7ff64400c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644017070 con 0x7ff65c0a3da0 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: dump: 2024-09-17T15:27:15.131 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6440070d0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 5 monclient: authenticate success, global_id 4132 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _renew_subs 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c13d550 con 0x7ff65c0a3da0 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6440074c0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 10 monclient: dump: 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.133 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff65a7fc640 4 set_mon_vals no callback set 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _renew_subs 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c1402e0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff64401a070 con 0x7ff65c0a3da0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff644012910 con 0x7ff65c0a3da0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.124+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c138580 con 0x7ff65c0a3da0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.128+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7ff644021070 con 0x7ff65c0a3da0 2024-09-17T15:27:15.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.128+0000 7ff658ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.128+0000 7ff658ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:15.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.196+0000 7ff66286b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b14d69b1-ee94-4f5a-b210-7893cb8790ea", "id": 4}] 2024-09-17T15:27:15.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.196+0000 7ff66286b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b14d69b1-ee94-4f5a-b210-7893cb8790ea", "id": 4}] 2024-09-17T15:27:15.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.196+0000 7ff66286b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.196+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b14d69b1-ee94-4f5a-b210-7893cb8790ea", "id": 4} v 0) -- 0x7ff65c06d4b0 con 0x7ff65c0a3da0 2024-09-17T15:27:15.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff658ff9640 1 -- 172.21.15.203:0/1996380339 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b14d69b1-ee94-4f5a-b210-7893cb8790ea", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff644012b80 con 0x7ff65c0a3da0 2024-09-17T15:27:15.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff658ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b14d69b1-ee94-4f5a-b210-7893cb8790ea", "id": 4}] 2024-09-17T15:27:15.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff658ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:15.200 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-09-17T15:27:15.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff66286b640 10 monclient: shutdown 2024-09-17T15:27:15.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff66286b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 msgr2=0x7ff65c1376d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.200+0000 7ff66286b640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c470 tx=0x7ff64400c930 comp rx=0 tx=0).stop 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 shutdown_connections 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c1381a0 0x7ff65c13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff65c0a3da0 0x7ff65c1376d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 --2- 172.21.15.203:0/1996380339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c06fb70 0x7ff65c137190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 >> 172.21.15.203:0/1996380339 conn(0x7ff65c0a8390 msgr2=0x7ff65c1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 shutdown_connections 2024-09-17T15:27:15.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.204+0000 7ff66286b640 1 -- 172.21.15.203:0/1996380339 wait complete. 2024-09-17T15:27:15.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 37cddbcf-8bae-470a-97cf-2ef98e2e7fa1 5 2024-09-17T15:27:15.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.364+0000 7f2b12fcb640 10 monclient: get_monmap_and_config 2024-09-17T15:27:15.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.364+0000 7f2b12fcb640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.364+0000 7f2b12fcb640 10 monclient: monmap: 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:27:15.367765+0000 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:27:15.367765+0000 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.364+0000 7f2b12fcb640 1 Processor -- start 2024-09-17T15:27:15.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 1 -- start start 2024-09-17T15:27:15.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient: init 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2b04027000 0x7f2b040273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-c con 0x7f2b04027000 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b040bb9b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b04021f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b04023e40 con 0x7f2b04021f70 2024-09-17T15:27:15.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b04024120 con 0x7f2b040bb9b0 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2b04024400 con 0x7f2b04027000 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b12fcb640 10 monclient(hunting): authenticate will time out at 966.276552s 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2b04027000 0x7f2b040273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b117c8640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2b040bb9b0 0x7f2b04021860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:59792/0 (socket says 172.21.15.203:59792) 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b117c8640 1 -- 172.21.15.203:0/2703121849 learned_addr learned my addr 172.21.15.203:0/2703121849 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): get_auth_request con 0x7f2b04027000 auth_method 0 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 1 -- 172.21.15.203:0/2703121849 >> v1:172.21.15.203:6789/0 conn(0x7f2b040bb9b0 legacy=0x7f2b04021860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 1 -- 172.21.15.203:0/2703121849 >> v1:172.21.15.86:6789/0 conn(0x7f2b04021f70 legacy=0x7f2b040c3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 1 monclient: found mon.noname-c 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.368+0000 7f2b11fc9640 1 -- 172.21.15.203:0/2703121849 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b040246e0 con 0x7f2b04027000 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b11fc9640 1 -- 172.21.15.203:0/2703121849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b04027000 msgr2=0x7f2b040273e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b11fc9640 1 --2- 172.21.15.203:0/2703121849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b04027000 0x7f2b040273e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c066720 tx=0x7f2b0c0990d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7b2ef588a3738e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 1 -- 172.21.15.203:0/2703121849 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c121e50 con 0x7f2b04027000 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: dump: 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.372 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 1 -- 172.21.15.203:0/2703121849 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b0c125070 con 0x7f2b04027000 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 1 -- 172.21.15.203:0/2703121849 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c064260 con 0x7f2b04027000 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 5 monclient: authenticate success, global_id 4135 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 4 set_mon_vals no callback set 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b10fc7640 10 monclient: dump: 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.374 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: shutdown 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2703121849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b04027000 msgr2=0x7f2b040273e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2703121849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b04027000 0x7f2b040273e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c066720 tx=0x7f2b0c0990d0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2703121849 shutdown_connections 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2703121849 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b04027000 0x7f2b040273e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2703121849 >> 172.21.15.203:0/2703121849 conn(0x7f2b04006370 msgr2=0x7f2b04006780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2703121849 shutdown_connections 2024-09-17T15:27:15.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2703121849 wait complete. 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: monmap: 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 Processor -- start 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- start start 2024-09-17T15:27:15.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: init 2024-09-17T15:27:15.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b04021f70 0x7f2b04154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-a con 0x7f2b04021f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 0x7f2b04154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-c con 0x7f2b04027000 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): picked mon.noname-b con 0x7f2b040bb9b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2b040c79a0 con 0x7f2b04021f70 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2b04027ca0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2b040c7b20 con 0x7f2b04027000 2024-09-17T15:27:15.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b12fcb640 10 monclient(hunting): authenticate will time out at 966.283418s 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:51978/0 (socket says 172.21.15.203:51978) 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 -- 172.21.15.203:0/2865646018 learned_addr learned my addr 172.21.15.203:0/2865646018 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): get_auth_request con 0x7f2b040bb9b0 auth_method 0 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b117c8640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 0x7f2b04154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b11fc9640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b04021f70 0x7f2b04154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 -- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 msgr2=0x7f2b04154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 0x7f2b04154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 -- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b04021f70 msgr2=0x7f2b04154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b04021f70 0x7f2b04154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 monclient: found mon.noname-b 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b127ca640 1 -- 172.21.15.203:0/2865646018 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b040c3000 con 0x7f2b040bb9b0 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b117c8640 10 monclient: get_auth_request con 0x7f2b04027000 auth_method 0 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.372+0000 7f2b117c8640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 0x7f2b04154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:15.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b127ca640 1 -- 172.21.15.203:0/2865646018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 msgr2=0x7f2b040c05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b127ca640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800c410 tx=0x7f2b0800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08017070 con 0x7f2b040bb9b0 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: dump: 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.383 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b08003800 con 0x7f2b040bb9b0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b08003bf0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 5 monclient: authenticate success, global_id 4138 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _renew_subs 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b040c43c0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b037fe640 4 set_mon_vals no callback set 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 10 monclient: dump: 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _renew_subs 2024-09-17T15:27:15.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b040c40e0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2b0801a070 con 0x7f2b040bb9b0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f2b08006bf0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.376+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b040273e0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.380+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2b080067a0 con 0x7f2b040bb9b0 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.380+0000 7f2b027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.380+0000 7f2b027fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:15.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.452+0000 7f2b12fcb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "37cddbcf-8bae-470a-97cf-2ef98e2e7fa1", "id": 5}] 2024-09-17T15:27:15.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.452+0000 7f2b12fcb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "37cddbcf-8bae-470a-97cf-2ef98e2e7fa1", "id": 5}] 2024-09-17T15:27:15.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.452+0000 7f2b12fcb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:15.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.452+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "37cddbcf-8bae-470a-97cf-2ef98e2e7fa1", "id": 5} v 0) -- 0x7f2b04008300 con 0x7f2b040bb9b0 2024-09-17T15:27:15.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.456+0000 7f2b027fc640 1 -- 172.21.15.203:0/2865646018 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37cddbcf-8bae-470a-97cf-2ef98e2e7fa1", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f2b08007060 con 0x7f2b040bb9b0 2024-09-17T15:27:15.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.456+0000 7f2b027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "37cddbcf-8bae-470a-97cf-2ef98e2e7fa1", "id": 5}] 2024-09-17T15:27:15.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.456+0000 7f2b027fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:15.457 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-09-17T15:27:15.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 10 monclient: shutdown 2024-09-17T15:27:15.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 msgr2=0x7f2b040c05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2b0800c410 tx=0x7f2b0800c8d0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 shutdown_connections 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b040bb9b0 0x7f2b040c05a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b04027000 0x7f2b04154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 --2- 172.21.15.203:0/2865646018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b04021f70 0x7f2b04154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 >> 172.21.15.203:0/2865646018 conn(0x7f2b04006370 msgr2=0x7f2b0402bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 shutdown_connections 2024-09-17T15:27:15.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-09-17T15:27:15.460+0000 7f2b12fcb640 1 -- 172.21.15.203:0/2865646018 wait complete. 2024-09-17T15:27:15.473 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T15:27:15.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T15:27:15.476 INFO:tasks.ceph.osd.0:Started 2024-09-17T15:27:15.476 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T15:27:15.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T15:27:15.479 INFO:tasks.ceph.osd.1:Started 2024-09-17T15:27:15.479 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T15:27:15.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T15:27:15.481 INFO:tasks.ceph.osd.2:Started 2024-09-17T15:27:15.481 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T15:27:15.482 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T15:27:15.484 INFO:tasks.ceph.osd.3:Started 2024-09-17T15:27:15.484 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T15:27:15.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T15:27:15.486 INFO:tasks.ceph.osd.4:Started 2024-09-17T15:27:15.486 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T15:27:15.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T15:27:15.488 INFO:tasks.ceph.osd.5:Started 2024-09-17T15:27:15.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:15.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: get_monmap_and_config 2024-09-17T15:27:15.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: monmap: 2024-09-17T15:27:15.607 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:15.609893+0000 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:15.609893+0000 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:15.608 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 Processor -- start 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 -- start start 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: init 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-b con 0x7f2bcc0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 0x7f2bcc0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-a con 0x7f2bcc0a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2bcc137ea0 0x7f2bcc138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-c con 0x7f2bcc137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2bcc09bdc0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2bcc09d350 con 0x7f2bcc137ea0 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2bcc0a6530 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd3061640 10 monclient(hunting): authenticate will time out at 968.575226s 2024-09-17T15:27:15.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 0x7f2bcc0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49504/0 (socket says 172.21.15.86:49504) 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 0x7f2bcc0a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44066/0 (socket says 172.21.15.86:44066) 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/712200960 learned_addr learned my addr 172.21.15.86:0/712200960 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): get_auth_request con 0x7f2bcc0a50e0 auth_method 0 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): get_auth_request con 0x7f2bcc0a54c0 auth_method 0 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 1 --2- 172.21.15.86:0/712200960 >> v2:172.21.15.203:3300/0 conn(0x7f2bcc137ea0 0x7f2bcc138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): get_auth_request con 0x7f2bcc137ea0 auth_method 0 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd15fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/712200960 >> v2:172.21.15.203:3300/0 conn(0x7f2bcc137ea0 msgr2=0x7f2bcc138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 --2- 172.21.15.86:0/712200960 >> v2:172.21.15.203:3300/0 conn(0x7f2bcc137ea0 0x7f2bcc138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/712200960 >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 msgr2=0x7f2bcc0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 --2- 172.21.15.86:0/712200960 >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 0x7f2bcc0a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 monclient: found mon.noname-b 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.604+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/712200960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc09c150 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/712200960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 msgr2=0x7f2bcc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 --2- 172.21.15.86:0/712200960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc09c7c0 tx=0x7f2bbc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef67f81d6847af server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 1 -- 172.21.15.86:0/712200960 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0b0070 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: dump: 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.619 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 1 -- 172.21.15.86:0/712200960 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bbc0ab600 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 1 -- 172.21.15.86:0/712200960 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0ab9d0 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 5 monclient: authenticate success, global_id 4127 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 4 set_mon_vals no callback set 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcb7fe640 10 monclient: dump: 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.621 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: shutdown 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- 172.21.15.86:0/712200960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 msgr2=0x7f2bcc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- 172.21.15.86:0/712200960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc09c7c0 tx=0x7f2bbc01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- 172.21.15.86:0/712200960 shutdown_connections 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- 172.21.15.86:0/712200960 >> v2:172.21.15.203:3300/0 conn(0x7f2bcc137ea0 0x7f2bcc138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- 172.21.15.86:0/712200960 >> v2:172.21.15.86:3300/0 conn(0x7f2bcc0a54c0 0x7f2bcc0a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- 172.21.15.86:0/712200960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a50e0 0x7f2bcc09ce10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- 172.21.15.86:0/712200960 >> 172.21.15.86:0/712200960 conn(0x7f2bcc0aaae0 msgr2=0x7f2bcc0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- 172.21.15.86:0/712200960 shutdown_connections 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- 172.21.15.86:0/712200960 wait complete. 2024-09-17T15:27:15.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: build_initial_monmap 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: monmap: 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 Processor -- start 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- start start 2024-09-17T15:27:15.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: init 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-a con 0x7f2bcc0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 0x7f2bcc136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-c con 0x7f2bcc0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 0x7f2bcc13d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): picked mon.noname-b con 0x7f2bcc137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc06f660 con 0x7f2bcc0a50e0 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc06f4e0 con 0x7f2bcc137ea0 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:15.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2bcc141b20 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd3061640 10 monclient(hunting): authenticate will time out at 968.583028s 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 0x7f2bcc136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44068/0 (socket says 172.21.15.86:44068) 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 -- 172.21.15.86:0/2727169933 learned_addr learned my addr 172.21.15.86:0/2727169933 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 10 monclient(hunting): get_auth_request con 0x7f2bcc0a50e0 auth_method 0 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): get_auth_request con 0x7f2bcc0a54c0 auth_method 0 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:15.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd15fe640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 0x7f2bcc13d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient(hunting): handle_auth_done global_id 4130 payload 824 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 -- 172.21.15.86:0/2727169933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 msgr2=0x7f2bcc13d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 0x7f2bcc13d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 -- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 msgr2=0x7f2bcc1cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 monclient: found mon.noname-c 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bcbfff640 1 -- 172.21.15.86:0/2727169933 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc1d0da0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd15fe640 10 monclient: get_auth_request con 0x7f2bcc137ea0 auth_method 0 2024-09-17T15:27:15.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd15fe640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 0x7f2bcc13d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:15.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.612+0000 7f2bd0dfd640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:27:15.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bcbfff640 1 -- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 msgr2=0x7f2bcc136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:15.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bcbfff640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 0x7f2bcc136ff0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000e870 tx=0x7f2bc000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0017070 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: dump: 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.629 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bc000cb50 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0013470 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 10 monclient: dump: 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:15.630 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 5 monclient: authenticate success, global_id 4130 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _renew_subs 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc1226b0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bcaffd640 4 set_mon_vals no callback set 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _renew_subs 2024-09-17T15:27:15.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc0a0f60 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2bc001a070 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f2bc0006610 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.616+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc0a58d0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.620+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2bcc1d0da0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.620+0000 7f2bc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.620+0000 7f2bc97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:15.660 INFO:tasks.ceph.mgr.x.smithi203.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-17T15:27:15.660 INFO:tasks.ceph.mgr.x.smithi203.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-17T15:27:15.660 INFO:tasks.ceph.mgr.x.smithi203.stderr: from numpy import show_config as show_numpy_config 2024-09-17T15:27:15.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bcc074be0 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bc97fa640 1 -- 172.21.15.86:0/2727169933 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f2bc0013650 con 0x7f2bcc0a54c0 2024-09-17T15:27:15.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bc97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:15.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bc97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:15.684 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:15.684 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":8,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:15.454129+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 10 monclient: shutdown 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 msgr2=0x7f2bcc136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 0x7f2bcc136ff0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000e870 tx=0x7f2bc000ed30 comp rx=0 tx=0).stop 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 shutdown_connections 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bcc137ea0 0x7f2bcc13d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bcc0a54c0 0x7f2bcc136ff0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 --2- 172.21.15.86:0/2727169933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bcc0a50e0 0x7f2bcc1cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 >> 172.21.15.86:0/2727169933 conn(0x7f2bcc0aaae0 msgr2=0x7f2bcc09f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:15.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 shutdown_connections 2024-09-17T15:27:15.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:15.680+0000 7f2bd3061640 1 -- 172.21.15.86:0/2727169933 wait complete. 2024-09-17T15:27:15.700 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T15:27:15.701 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T15:27:16.305 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:16.430 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: get_monmap_and_config 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: build_initial_monmap 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: monmap: 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:27:16.433017+0000 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:27:16.433017+0000 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:16.431 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:16.432 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.434 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 Processor -- start 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 -- start start 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: init 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8548135df0 0x7f85481361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-c con 0x7f8548135df0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 0x7f854809f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-b con 0x7f85480a19a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8548136710 0x7f854809fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-a con 0x7f8548136710 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f85480a2490 con 0x7f8548136710 2024-09-17T15:27:16.435 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f85480a2a50 con 0x7f8548135df0 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f85480a21b0 con 0x7f85480a19a0 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854edf5640 10 monclient(hunting): authenticate will time out at 969.398529s 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8548136710 0x7f854809fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 0x7f854809f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 0x7f854809f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49518/0 (socket says 172.21.15.86:49518) 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8548136710 0x7f854809fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44078/0 (socket says 172.21.15.86:44078) 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 1 -- 172.21.15.86:0/3172522824 learned_addr learned my addr 172.21.15.86:0/3172522824 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): get_auth_request con 0x7f85480a19a0 auth_method 0 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:16.436 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): get_auth_request con 0x7f8548136710 auth_method 0 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854cb91640 1 --2- 172.21.15.86:0/3172522824 >> v2:172.21.15.203:3300/0 conn(0x7f8548135df0 0x7f85481361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.437 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854cb91640 10 monclient(hunting): get_auth_request con 0x7f8548135df0 auth_method 0 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854cb91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854cb91640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854cb91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient(hunting): handle_auth_done global_id 4137 payload 274 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 -- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 msgr2=0x7f854809f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 --2- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 0x7f854809f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 -- 172.21.15.86:0/3172522824 >> v2:172.21.15.203:3300/0 conn(0x7f8548135df0 msgr2=0x7f85481361d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 --2- 172.21.15.86:0/3172522824 >> v2:172.21.15.203:3300/0 conn(0x7f8548135df0 0x7f85481361d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient: _finish_hunting 0 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 monclient: found mon.noname-a 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:16.438 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.428+0000 7f854d392640 1 -- 172.21.15.86:0/3172522824 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85480a2770 con 0x7f8548136710 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854d392640 1 -- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8548136710 msgr2=0x7f854809fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854d392640 1 --2- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8548136710 0x7f854809fe50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f854809d0e0 tx=0x7f854001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d561220263c4fb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 1 -- 172.21.15.86:0/3172522824 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400b2070 con 0x7f8548136710 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:16.442 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: dump: 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:16.443 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 1 -- 172.21.15.86:0/3172522824 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85400a5780 con 0x7f8548136710 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 5 monclient: authenticate success, global_id 4137 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 4 set_mon_vals no callback set 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 1 -- 172.21.15.86:0/3172522824 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400a5b50 con 0x7f8548136710 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f85477fe640 10 monclient: dump: 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.444 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: shutdown 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8548136710 msgr2=0x7f854809fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8548136710 0x7f854809fe50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f854809d0e0 tx=0x7f854001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- 172.21.15.86:0/3172522824 shutdown_connections 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8548136710 0x7f854809fe50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.445 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- 172.21.15.86:0/3172522824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85480a19a0 0x7f854809f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- 172.21.15.86:0/3172522824 >> v2:172.21.15.203:3300/0 conn(0x7f8548135df0 0x7f85481361d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- 172.21.15.86:0/3172522824 >> 172.21.15.86:0/3172522824 conn(0x7f85480a2d70 msgr2=0x7f85480a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- 172.21.15.86:0/3172522824 shutdown_connections 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- 172.21.15.86:0/3172522824 wait complete. 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: build_initial_monmap 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: monmap: 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.446 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 Processor -- start 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- start start 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: init 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85480a19a0 0x7f85481d4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-a con 0x7f85480a19a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-c con 0x7f8548135df0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 0x7f8548137690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:16.447 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): picked mon.noname-b con 0x7f8548136710 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f854813f820 con 0x7f85480a19a0 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85480a07a0 con 0x7f8548136710 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f854813f9a0 con 0x7f8548135df0 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854edf5640 10 monclient(hunting): authenticate will time out at 969.406663s 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49526/0 (socket says 172.21.15.86:49526) 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 -- 172.21.15.86:0/2814018420 learned_addr learned my addr 172.21.15.86:0/2814018420 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): get_auth_request con 0x7f8548135df0 auth_method 0 2024-09-17T15:27:16.448 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85480a19a0 0x7f85481d4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): get_auth_request con 0x7f85480a19a0 auth_method 0 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:16.449 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854cb91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854d392640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 0x7f8548137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient(hunting): handle_auth_done global_id 4136 payload 824 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 -- 172.21.15.86:0/2814018420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 msgr2=0x7f8548137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 0x7f8548137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 -- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85480a19a0 msgr2=0x7f85481d4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85480a19a0 0x7f85481d4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f8547fff640 1 -- 172.21.15.86:0/2814018420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85481393e0 con 0x7f8548135df0 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854d392640 10 monclient: get_auth_request con 0x7f8548136710 auth_method 0 2024-09-17T15:27:16.450 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.436+0000 7f854d392640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 0x7f8548137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:16.452 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f8547fff640 1 -- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 msgr2=0x7f85481d52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:16.452 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f8547fff640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f853400e8e0 tx=0x7f853400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:16.452 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534017070 con 0x7f8548135df0 2024-09-17T15:27:16.452 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:16.452 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: dump: 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:16.453 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 5 monclient: authenticate success, global_id 4136 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _renew_subs 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853400cc10 con 0x7f8548135df0 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854813afc0 con 0x7f8548135df0 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534013470 con 0x7f8548135df0 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 10 monclient: dump: 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:16.454 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _renew_subs 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f8546ffd640 4 set_mon_vals no callback set 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f854809d3f0 con 0x7f8548135df0 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f853401a070 con 0x7f8548135df0 2024-09-17T15:27:16.455 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f8534013ce0 con 0x7f8548135df0 2024-09-17T15:27:16.456 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:16.456 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:16.456 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:16.456 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.440+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85481361d0 con 0x7f8548135df0 2024-09-17T15:27:16.456 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:16.444+0000 7f8547fff640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f853400e8e0 tx=0x7f853400eda0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f8534013610 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-17T15:27:16.587 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-17T15:27:16.588+0000 7fd8e5cbd7c0 -1 Falling back to public interface 2024-09-17T15:27:16.631 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-09-17T15:27:16.628+0000 7f84dfc537c0 -1 Falling back to public interface 2024-09-17T15:27:16.647 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-09-17T15:27:16.644+0000 7ff3c4f637c0 -1 Falling back to public interface 2024-09-17T15:27:16.754 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-17T15:27:16.756+0000 7fada2e0b7c0 -1 Falling back to public interface 2024-09-17T15:27:17.067 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-09-17T15:27:17.064+0000 7f49a57d87c0 -1 Falling back to public interface 2024-09-17T15:27:17.273 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.272+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f8534013610 con 0x7f8548135df0 2024-09-17T15:27:17.274 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.272+0000 7f85457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:17.274 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.272+0000 7f85457fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8548074000 con 0x7f8548135df0 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f85457fa640 1 -- 172.21.15.86:0/2814018420 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f85340059a0 con 0x7f8548135df0 2024-09-17T15:27:17.343 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f85457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:17.344 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f85457fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:17.344 INFO:teuthology.misc.health.smithi086.stdout: 2024-09-17T15:27:17.344 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":8,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:15.454129+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 10 monclient: shutdown 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 msgr2=0x7f85481d52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f853400e8e0 tx=0x7f853400eda0 comp rx=0 tx=0).stop 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 shutdown_connections 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8548136710 0x7f8548137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8548135df0 0x7f85481d52d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 --2- 172.21.15.86:0/2814018420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85480a19a0 0x7f85481d4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 >> 172.21.15.86:0/2814018420 conn(0x7f85480a2d70 msgr2=0x7f85481333c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 shutdown_connections 2024-09-17T15:27:17.345 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:17.340+0000 7f854edf5640 1 -- 172.21.15.86:0/2814018420 wait complete. 2024-09-17T15:27:17.358 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-17T15:27:17.360+0000 7fc5cfec67c0 -1 Falling back to public interface 2024-09-17T15:27:17.362 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-17T15:27:18.647 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-17T15:27:18.648+0000 7f9d05cb8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T15:27:18.650 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-09-17T15:27:18.652+0000 7f9d05cb8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T15:27:20.510 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-17T15:27:20.512+0000 7fd8e5cbd7c0 -1 osd.5 0 log_to_monitors true 2024-09-17T15:27:20.635 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-09-17T15:27:20.632+0000 7f84dfc537c0 -1 osd.0 0 log_to_monitors true 2024-09-17T15:27:20.681 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-17T15:27:20.684+0000 7fada2e0b7c0 -1 osd.3 0 log_to_monitors true 2024-09-17T15:27:20.717 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-09-17T15:27:20.712+0000 7ff3c4f637c0 -1 osd.1 0 log_to_monitors true 2024-09-17T15:27:21.167 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-09-17T15:27:21.164+0000 7f49a57d87c0 -1 osd.2 0 log_to_monitors true 2024-09-17T15:27:21.312 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-17T15:27:21.312+0000 7fc5cfec67c0 -1 osd.4 0 log_to_monitors true 2024-09-17T15:27:22.471 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-09-17T15:27:22.468+0000 7f84dbbfc640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T15:27:22.474 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-09-17T15:27:22.472+0000 7ff3c0f0c640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T15:27:22.502 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-09-17T15:27:22.500+0000 7f84d7a36640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:22.503 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-09-17T15:27:22.500+0000 7ff3bcd46640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:22.610 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-17T15:27:22.612+0000 7fd8e2478640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T15:27:22.613 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-09-17T15:27:22.616+0000 7fd8ddaa0640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:23.117 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-09-17T15:27:23.112+0000 7f49a1f93640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T15:27:23.130 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-09-17T15:27:23.128+0000 7f49995b3640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:23.301 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-17T15:27:23.300+0000 7fad9f5c6640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T15:27:23.322 INFO:tasks.ceph.osd.3.smithi203.stderr:2024-09-17T15:27:23.324+0000 7fad96be6640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:23.332 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-17T15:27:23.332+0000 7fc5cbe6f640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T15:27:23.350 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-09-17T15:27:23.352+0000 7fc5c3ca1640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T15:27:23.968 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.084+0000 7f528e7a0640 10 monclient: get_monmap_and_config 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.084+0000 7f528e7a0640 10 monclient: build_initial_monmap 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.084+0000 7f528e7a0640 10 monclient: monmap: 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:27:24.091236+0000 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:27:24.091236+0000 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.089 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:24.090 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:24.090 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:24.090 INFO:teuthology.misc.health.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:24.090 INFO:teuthology.misc.health.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:24.090 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.084+0000 7f528e7a0640 1 Processor -- start 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 1 -- start start 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient: init 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5288134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52880a1d80 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-b con 0x7f5288135530 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:24.092 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528809ed10 con 0x7f52880a1d80 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528809cce0 con 0x7f5288134a40 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f52880a2200 con 0x7f5288135530 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528e7a0640 10 monclient(hunting): authenticate will time out at 977.056736s 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f52877fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f52880a1d80 0x7f5288139f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:35984/0 (socket says 172.21.15.86:35984) 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42184/0 (socket says 172.21.15.86:42184) 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f52877fe640 1 -- 172.21.15.86:0/4040401905 learned_addr learned my addr 172.21.15.86:0/4040401905 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): get_auth_request con 0x7f5288135530 auth_method 0 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.093 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient(hunting): handle_auth_done global_id 4160 payload 274 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 -- 172.21.15.86:0/4040401905 >> v1:172.21.15.203:6789/0 conn(0x7f5288134a40 legacy=0x7f5288134e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 -- 172.21.15.86:0/4040401905 >> v1:172.21.15.86:6789/0 conn(0x7f52880a1d80 legacy=0x7f5288139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 monclient: found mon.noname-b 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.094 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.088+0000 7f528cd3d640 1 -- 172.21.15.86:0/4040401905 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528809e710 con 0x7f5288135530 2024-09-17T15:27:24.095 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528cd3d640 1 -- 172.21.15.86:0/4040401905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 msgr2=0x7f528813d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:24.095 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528cd3d640 1 --2- 172.21.15.86:0/4040401905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52780097f0 tx=0x7f527801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6743e2a196d34aa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.095 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 1 -- 172.21.15.86:0/4040401905 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52780b0070 con 0x7f5288135530 2024-09-17T15:27:24.095 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.095 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: dump: 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:24.096 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 1 -- 172.21.15.86:0/4040401905 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52780ab600 con 0x7f5288135530 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 1 -- 172.21.15.86:0/4040401905 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52780ab9d0 con 0x7f5288135530 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 5 monclient: authenticate success, global_id 4160 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 4 set_mon_vals no callback set 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f5286ffd640 10 monclient: dump: 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.097 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.098 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.098 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.098 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.098 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.098 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 10 monclient: shutdown 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 -- 172.21.15.86:0/4040401905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 msgr2=0x7f528813d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 --2- 172.21.15.86:0/4040401905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52780097f0 tx=0x7f527801c3a0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 -- 172.21.15.86:0/4040401905 shutdown_connections 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 --2- 172.21.15.86:0/4040401905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5288135530 0x7f528813d630 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.102 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 -- 172.21.15.86:0/4040401905 >> 172.21.15.86:0/4040401905 conn(0x7f52880a2d70 msgr2=0x7f52880a3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 -- 172.21.15.86:0/4040401905 shutdown_connections 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 1 -- 172.21.15.86:0/4040401905 wait complete. 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 10 monclient: build_initial_monmap 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.092+0000 7f528e7a0640 10 monclient: monmap: 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:24.103 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 Processor -- start 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- start start 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: init 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-c con 0x7f52880a1d80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5288134a40 0x7f5288139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-b con 0x7f5288134a40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5288135530 0x7f528813a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): picked mon.noname-a con 0x7f5288135530 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:24.104 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f528809eff0 con 0x7f5288135530 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52880a07e0 con 0x7f5288134a40 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f52881409b0 con 0x7f52880a1d80 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient(hunting): authenticate will time out at 977.064807s 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42198/0 (socket says 172.21.15.86:42198) 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 -- 172.21.15.86:0/2009430046 learned_addr learned my addr 172.21.15.86:0/2009430046 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): get_auth_request con 0x7f52880a1d80 auth_method 0 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.105 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5288135530 0x7f528813a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): get_auth_request con 0x7f5288135530 auth_method 0 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528cd3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52877fe640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5288134a40 0x7f5288139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52877fe640 10 monclient(hunting): get_auth_request con 0x7f5288134a40 auth_method 0 2024-09-17T15:27:24.106 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 -- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5288134a40 msgr2=0x7f5288139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5288134a40 0x7f5288139ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 -- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5288135530 msgr2=0x7f528813a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5288135530 0x7f528813a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 -- 172.21.15.86:0/2009430046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528811d290 con 0x7f52880a1d80 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 -- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 msgr2=0x7f5288133830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:24.107 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5287fff640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f528000c7c0 tx=0x7f528000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280016020 con 0x7f52880a1d80 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: dump: 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.108 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528000ac60 con 0x7f52880a1d80 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280012660 con 0x7f52880a1d80 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 5 monclient: authenticate success, global_id 4163 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _renew_subs 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f52867fc640 4 set_mon_vals no callback set 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528813b590 con 0x7f52880a1d80 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.109 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f5284ff9640 10 monclient: dump: 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr: 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _renew_subs 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.096+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f528813c530 con 0x7f52880a1d80 2024-09-17T15:27:24.110 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 4) ==== 48888+0+0 (secure 0 0 0) 0x7f5280019020 con 0x7f52880a1d80 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f528e7a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f528e7a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f528e7a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5288134e20 con 0x7f52880a1d80 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f5284ff9640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 0x7f526803df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.111 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(11..11 src has 1..11) ==== 3607+0+0 (secure 0 0 0) 0x7f528004fa40 con 0x7f52880a1d80 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f52877fe640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 0x7f526803df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.100+0000 7f52877fe640 10 monclient: get_auth_request con 0x7f526803bad0 auth_method 0 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.104+0000 7f52877fe640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 0x7f526803df90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5270007fd0 tx=0x7f5270009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.104+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5280020030 con 0x7f52880a1d80 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.104+0000 7f5284ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.112 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.104+0000 7f5284ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f528e7a0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f528e7a0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f528e7a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f528806d4a0 con 0x7f52880a1d80 2024-09-17T15:27:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f5284ff9640 1 -- 172.21.15.86:0/2009430046 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+8509 (secure 0 0 0) 0x7f5280011c60 con 0x7f52880a1d80 2024-09-17T15:27:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f5284ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.276+0000 7f5284ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:24.281 INFO:teuthology.misc.health.smithi086.stdout: 2024-09-17T15:27:24.282 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:23.470675+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 msgr2=0x7f526803df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 0x7f526803df90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5270007fd0 tx=0x7f5270009040 comp rx=0 tx=0).stop 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 10 monclient: shutdown 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 msgr2=0x7f5288133830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f528000c7c0 tx=0x7f528000cc80 comp rx=0 tx=0).stop 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 shutdown_connections 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f526803bad0 0x7f526803df90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5288135530 0x7f528813a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5288134a40 0x7f5288139ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 --2- 172.21.15.86:0/2009430046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52880a1d80 0x7f5288133830 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 >> 172.21.15.86:0/2009430046 conn(0x7f52880a2d70 msgr2=0x7f52880a4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 shutdown_connections 2024-09-17T15:27:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-09-17T15:27:24.280+0000 7f528e7a0640 1 -- 172.21.15.86:0/2009430046 wait complete. 2024-09-17T15:27:24.299 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-17T15:27:24.299 INFO:tasks.ceph:Creating RBD pool 2024-09-17T15:27:24.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: get_monmap_and_config 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: build_initial_monmap 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: monmap: 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:24.416365+0000 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:24.416365+0000 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:24.415 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:24.415 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:24.415 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:24.415 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:24.415 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 Processor -- start 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 -- start start 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: init 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:27:24.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9700a01a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 0x7fb9700a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9700a0580 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9700a11b0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb970074e00 con 0x7fb9700a11b0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb97009e350 con 0x7fb9700a0580 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb97009e630 con 0x7fb9700a01a0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb9770e5640 10 monclient(hunting): authenticate will time out at 977.381940s 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42208/0 (socket says 172.21.15.86:42208) 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 0x7fb9700a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 -- 172.21.15.86:0/4224597600 learned_addr learned my addr 172.21.15.86:0/4224597600 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): get_auth_request con 0x7fb9700a01a0 auth_method 0 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): get_auth_request con 0x7fb9700a0580 auth_method 0 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient(hunting): handle_auth_done global_id 4166 payload 274 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 -- 172.21.15.86:0/4224597600 >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 msgr2=0x7fb9700a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 --2- 172.21.15.86:0/4224597600 >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 0x7fb9700a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 -- 172.21.15.86:0/4224597600 >> v1:172.21.15.86:6789/0 conn(0x7fb9700a11b0 legacy=0x7fb97013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient: _finish_hunting 0 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 monclient: found mon.noname-b 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb974e81640 1 -- 172.21.15.86:0/4224597600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97009e910 con 0x7fb9700a01a0 2024-09-17T15:27:24.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.412+0000 7fb96ffff640 1 --2- 172.21.15.86:0/4224597600 >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 0x7fb9700a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:24.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb974e81640 1 -- 172.21.15.86:0/4224597600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 msgr2=0x7fb97009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:24.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb974e81640 1 --2- 172.21.15.86:0/4224597600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb95800bb10 tx=0x7fb95801eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79c6f09b00a2a2f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 1 -- 172.21.15.86:0/4224597600 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9580b4070 con 0x7fb9700a01a0 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: dump: 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 1 -- 172.21.15.86:0/4224597600 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9580a7aa0 con 0x7fb9700a01a0 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 1 -- 172.21.15.86:0/4224597600 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9580af440 con 0x7fb9700a01a0 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 5 monclient: authenticate success, global_id 4166 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 4 set_mon_vals no callback set 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb96f7fe640 10 monclient: dump: 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.425 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 10 monclient: shutdown 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 -- 172.21.15.86:0/4224597600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 msgr2=0x7fb97009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 --2- 172.21.15.86:0/4224597600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb95800bb10 tx=0x7fb95801eaf0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 -- 172.21.15.86:0/4224597600 shutdown_connections 2024-09-17T15:27:24.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 --2- 172.21.15.86:0/4224597600 >> v2:172.21.15.86:3300/0 conn(0x7fb9700a0580 0x7fb9700a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 --2- 172.21.15.86:0/4224597600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a01a0 0x7fb97009f010 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 -- 172.21.15.86:0/4224597600 >> 172.21.15.86:0/4224597600 conn(0x7fb9700a2f40 msgr2=0x7fb9700a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 -- 172.21.15.86:0/4224597600 shutdown_connections 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.416+0000 7fb9770e5640 1 -- 172.21.15.86:0/4224597600 wait complete. 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient: build_initial_monmap 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient: monmap: 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:24.427 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 Processor -- start 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 -- start start 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient: init 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9700a01a0 0x7fb970134d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-b con 0x7fb9700a01a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9700a0580 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a11b0 0x7fb97011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9700a11b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb97013e5e0 con 0x7fb9700a0580 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb97013e460 con 0x7fb9700a01a0 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb97013e760 con 0x7fb9700a11b0 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 10 monclient(hunting): authenticate will time out at 977.389069s 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53124/0 (socket says 172.21.15.86:53124) 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 -- 172.21.15.86:0/1438691249 learned_addr learned my addr 172.21.15.86:0/1438691249 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): get_auth_request con 0x7fb9700a0580 auth_method 0 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a11b0 0x7fb97011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): get_auth_request con 0x7fb9700a11b0 auth_method 0 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb975682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient(hunting): handle_auth_done global_id 4170 payload 824 2024-09-17T15:27:24.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 -- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a11b0 msgr2=0x7fb97011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a11b0 0x7fb97011c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 -- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9700a01a0 msgr2=0x7fb970134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9700a01a0 0x7fb970134d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 monclient: found mon.noname-a 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:24.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 -- 172.21.15.86:0/1438691249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97011f7a0 con 0x7fb9700a0580 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 -- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 msgr2=0x7fb970135280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96ffff640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb96000e9d0 tx=0x7fb96000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960017070 con 0x7fb9700a0580 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:24.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 10 monclient: dump: 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:24.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96000cb90 con 0x7fb9700a0580 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.420+0000 7fb9770e5640 5 monclient: authenticate success, global_id 4170 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb960013470 con 0x7fb9700a0580 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _renew_subs 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97011fa80 con 0x7fb9700a0580 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96effd640 4 set_mon_vals no callback set 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 10 monclient: dump: 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:24.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _renew_subs 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97006f6e0 con 0x7fb9700a0580 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:24.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97009d460 con 0x7fb9700a0580 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 4) ==== 48888+0+0 (secure 0 0 0) 0x7fb96001a070 con 0x7fb9700a0580 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 0x7fb95003dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 3607+0+0 (secure 0 0 0) 0x7fb96004f4e0 con 0x7fb9700a0580 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb974e81640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 0x7fb95003dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.424+0000 7fb974e81640 10 monclient: get_auth_request con 0x7fb95003bb20 auth_method 0 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.428+0000 7fb974e81640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 0x7fb95003dfe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9580ae000 tx=0x7fb958004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.428+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb960009380 con 0x7fb9700a0580 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.428+0000 7fb96d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:24.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.428+0000 7fb96d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:24.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.604+0000 7fb9770e5640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:27:24.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.604+0000 7fb9770e5640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:27:24.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.604+0000 7fb9770e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:24.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:24.604+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fb9701357c0 con 0x7fb9700a0580 2024-09-17T15:27:25.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.500+0000 7fb96d7fa640 1 -- 172.21.15.86:0/1438691249 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7fb960013b30 con 0x7fb9700a0580 2024-09-17T15:27:25.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.500+0000 7fb96d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-17T15:27:25.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.500+0000 7fb96d7fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-17T15:27:25.504 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'rbd' created 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 msgr2=0x7fb95003dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 0x7fb95003dfe0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb9580ae000 tx=0x7fb958004740 comp rx=0 tx=0).stop 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 10 monclient: shutdown 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 msgr2=0x7fb970135280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb96000e9d0 tx=0x7fb96000ee90 comp rx=0 tx=0).stop 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 shutdown_connections 2024-09-17T15:27:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb95003bb20 0x7fb95003dfe0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9700a11b0 0x7fb97011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9700a0580 0x7fb970135280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 --2- 172.21.15.86:0/1438691249 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9700a01a0 0x7fb970134d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 >> 172.21.15.86:0/1438691249 conn(0x7fb9700a2f40 msgr2=0x7fb9701365b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 shutdown_connections 2024-09-17T15:27:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.504+0000 7fb9770e5640 1 -- 172.21.15.86:0/1438691249 wait complete. 2024-09-17T15:27:25.531 DEBUG:teuthology.orchestra.run.smithi086:> rbd --cluster ceph pool init rbd 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: get_monmap_and_config 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: build_initial_monmap 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: monmap: 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:25.590627+0000 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:25.590627+0000 2024-09-17T15:27:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:25.589 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 Processor -- start 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 -- start start 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: init 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: _reopen_session rank -1 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-a-legacy con 0x55b332408be0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-b con 0x55b332408fc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-c-legacy con 0x55b3324a2ff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b332253c80 con 0x55b332408be0 2024-09-17T15:27:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55b332402800 con 0x55b3324a2ff0 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x55b3322d5160 con 0x55b332408fc0 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): _renew_subs 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.584+0000 7f6139b43100 10 monclient(hunting): authenticate will time out at 978.555985s 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42232/0 (socket says 172.21.15.86:42232) 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 -- 172.21.15.86:0/3805911053 learned_addr learned my addr 172.21.15.86:0/3805911053 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): get_auth_request con 0x55b332408fc0 auth_method 0 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient(hunting): handle_auth_done global_id 4178 payload 274 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 -- 172.21.15.86:0/3805911053 >> v1:172.21.15.203:6789/0 conn(0x55b3324a2ff0 legacy=0x55b3324a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 -- 172.21.15.86:0/3805911053 >> v1:172.21.15.86:6789/0 conn(0x55b332408be0 legacy=0x55b332402f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.588+0000 7f61337fe640 1 -- 172.21.15.86:0/3805911053 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b332401970 con 0x55b332408fc0 2024-09-17T15:27:25.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 -- 172.21.15.86:0/3805911053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 msgr2=0x55b3324093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:25.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 --2- 172.21.15.86:0/3805911053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f612800a5f0 tx=0x7f612801b370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba105ff229d79fa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:25.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 1 -- 172.21.15.86:0/3805911053 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61280af070 con 0x55b332408fc0 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: dump: 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 1 -- 172.21.15.86:0/3805911053 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61280a3e50 con 0x55b332408fc0 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 1 -- 172.21.15.86:0/3805911053 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128006890 con 0x55b332408fc0 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6132ffd640 10 monclient: dump: 2024-09-17T15:27:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:25.601 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 5 monclient: authenticate success, global_id 4178 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: get_monmap_and_config success 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 4 set_mon_vals no callback set 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: shutdown 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- 172.21.15.86:0/3805911053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 msgr2=0x55b3324093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 --2- 172.21.15.86:0/3805911053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f612800a5f0 tx=0x7f612801b370 comp rx=0 tx=0).stop 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- 172.21.15.86:0/3805911053 shutdown_connections 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 --2- 172.21.15.86:0/3805911053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b332408fc0 0x55b3324093a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- 172.21.15.86:0/3805911053 >> 172.21.15.86:0/3805911053 conn(0x55b3323eadf0 msgr2=0x55b3323eb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- 172.21.15.86:0/3805911053 shutdown_connections 2024-09-17T15:27:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- 172.21.15.86:0/3805911053 wait complete. 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: build_initial_monmap 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: monmap: 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 Processor -- start 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- start start 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: init 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: _reopen_session rank -1 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b332408be0 0x55b332485410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-b con 0x55b332408be0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 0x55b33249e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-a con 0x55b332408fc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 0x55b332483840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): picked mon.noname-c con 0x55b3324a2ff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x55b3322d8f70 con 0x55b332408fc0 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b3322e5d30 con 0x55b332408be0 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x55b33227aa70 con 0x55b3324a2ff0 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): _renew_subs 2024-09-17T15:27:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6139b43100 10 monclient(hunting): authenticate will time out at 978.563119s 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 0x55b33249e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 0x55b33249e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53136/0 (socket says 172.21.15.86:53136) 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 0x55b332483840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 learned_addr learned my addr 172.21.15.86:0/1962953059 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): get_auth_request con 0x55b332408fc0 auth_method 0 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b332408be0 0x55b332485410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): get_auth_request con 0x55b3324a2ff0 auth_method 0 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): get_auth_request con 0x55b332408be0 auth_method 0 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f6133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient(hunting): handle_auth_done global_id 4181 payload 824 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b332408be0 msgr2=0x55b332485410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b332408be0 0x55b332485410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 msgr2=0x55b33249e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 0x55b33249e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient: _finish_hunting 0 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 monclient: found mon.noname-c 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61398ae640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b332484920 con 0x55b3324a2ff0 2024-09-17T15:27:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.592+0000 7f61337fe640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b332408fc0 0x55b33249e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f61398ae640 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 msgr2=0x55b332483840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f61398ae640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 0x55b332483840 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f612c003150 tx=0x7f612c00b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c010790 con 0x55b3324a2ff0 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: dump: 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f612c016070 con 0x55b3324a2ff0 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 5 monclient: authenticate success, global_id 4181 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _renew_subs 2024-09-17T15:27:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b3324b0010 con 0x55b3324a2ff0 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c010b70 con 0x55b3324a2ff0 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 10 monclient: dump: 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f61327fc640 4 set_mon_vals no callback set 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _renew_subs 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55b332486650 con 0x55b3324a2ff0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 4) ==== 48888+0+0 (secure 0 0 0) 0x7f612c01b460 con 0x55b3324a2ff0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f611803a440 0x7f611803c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 4335+0+0 (secure 0 0 0) 0x7f612c04e680 con 0x55b3324a2ff0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55b3324afd90 con 0x55b3324a2ff0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6133fff640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f611803a440 0x7f611803c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.596+0000 7f6133fff640 10 monclient: get_auth_request con 0x7f611803a440 auth_method 0 2024-09-17T15:27:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:25.600+0000 7f6133fff640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f611803a440 0x7f611803c900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6124003f20 tx=0x7f61240027b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:26.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.656+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 141+0+0 (secure 0 0 0) 0x55b3324afd90 con 0x55b3324a2ff0 2024-09-17T15:27:26.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.656+0000 7f6123fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-17T15:27:26.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.656+0000 7f6123fff640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-17T15:27:26.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.656+0000 7f6139b43100 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:26.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.656+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61280a3e50 con 0x55b3324a2ff0 2024-09-17T15:27:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f612c01bd10 con 0x55b3324a2ff0 2024-09-17T15:27:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 8 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f612c04e330 con 0x55b3324a2ff0 2024-09-17T15:27:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6123fff640 10 monclient: handle_get_version_reply finishing 1 version 14 2024-09-17T15:27:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f61327fc640 10 monclient: _renew_subs 2024-09-17T15:27:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f61327fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f61327fc640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=14}) -- 0x7f6110002af0 con 0x55b3324a2ff0 2024-09-17T15:27:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 9 ==== osd_map(14..14 src has 1..14) ==== 1020+0+0 (secure 0 0 0) 0x7f612c018dd0 con 0x55b3324a2ff0 2024-09-17T15:27:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6139b43100 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x55b3324843a0 0x55b3324b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.664+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x55b3324b9850 con 0x55b3324843a0 2024-09-17T15:27:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.668+0000 7f61337fe640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x55b3324843a0 0x55b3324b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.668+0000 7f61337fe640 10 monclient: get_auth_request con 0x55b3324843a0 auth_method 0 2024-09-17T15:27:26.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:26.672+0000 7f61337fe640 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x55b3324843a0 0x55b3324b9310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:27.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.624+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 1 ==== osd_map(15..15 src has 1..15) ==== 296+0+0 (crc 0 0 0) 0x7f6128006cd0 con 0x55b3324843a0 2024-09-17T15:27:27.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.628+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 2 ==== osd_op_reply(1 rbd_trash [create] v15'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f61280b2040 con 0x55b3324843a0 2024-09-17T15:27:27.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) -- 0x55b33250c550 con 0x55b3324843a0 2024-09-17T15:27:27.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 3 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f61280b2040 con 0x55b3324843a0 2024-09-17T15:27:27.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e15) -- 0x55b33250b630 con 0x55b3324843a0 2024-09-17T15:27:27.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 4 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f61280b2040 con 0x55b3324843a0 2024-09-17T15:27:27.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f6131ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:27.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:27.636+0000 7f6131ffb640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f6114001f10 con 0x55b3324a2ff0 2024-09-17T15:27:28.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.596+0000 7f6122ffd640 10 monclient: tick 2024-09-17T15:27:28.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.596+0000 7f6122ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:28.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.596+0000 7f6122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:28.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.596+0000 7f6122ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 10 ==== pool_op_reply(tid 4 (0) Success v16) ==== 55+0+0 (secure 0 0 0) 0x7f612c014790 con 0x55b3324a2ff0 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f6123fff640 10 monclient: _renew_subs 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f6123fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=16}) -- 0x7f6118040bd0 con 0x55b3324a2ff0 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 11 ==== osd_map(16..16 src has 1..16) ==== 626+0+0 (secure 0 0 0) 0x7f612c006110 con 0x55b3324a2ff0 2024-09-17T15:27:28.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.624+0000 7f61327fc640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x7f61100057a0 con 0x55b3324843a0 2024-09-17T15:27:28.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.632+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 5 ==== osd_op_reply(5 rbd_info [create,write 0~8] v16'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f61280b2040 con 0x55b3324843a0 2024-09-17T15:27:28.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.632+0000 7f6131ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:28.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:28.632+0000 7f6131ffb640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v16) -- 0x7f6114003360 con 0x55b3324a2ff0 2024-09-17T15:27:29.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.596+0000 7f6122ffd640 10 monclient: tick 2024-09-17T15:27:29.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.596+0000 7f6122ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:29.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.596+0000 7f6122ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:29.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 12 ==== pool_op_reply(tid 6 (0) Success v17) ==== 43+0+0 (secure 0 0 0) 0x7f612c03b320 con 0x55b3324a2ff0 2024-09-17T15:27:29.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f6123fff640 10 monclient: _renew_subs 2024-09-17T15:27:29.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f6123fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7f6118040da0 con 0x55b3324a2ff0 2024-09-17T15:27:29.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f6123fff640 1 -- 172.21.15.86:0/1962953059 <== mon.2 v2:172.21.15.86:3301/0 13 ==== osd_map(17..17 src has 1..17) ==== 654+0+0 (secure 0 0 0) 0x7f612c046030 con 0x55b3324a2ff0 2024-09-17T15:27:29.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.628+0000 7f61327fc640 1 -- 172.21.15.86:0/1962953059 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7f6110005b40 con 0x55b3324843a0 2024-09-17T15:27:29.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f61337fe640 1 -- 172.21.15.86:0/1962953059 <== osd.5 v2:172.21.15.203:6800/3502598683 6 ==== osd_op_reply(7 rbd_info [write 0~19] v17'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f61280b2040 con 0x55b3324843a0 2024-09-17T15:27:29.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x55b3324843a0 msgr2=0x55b3324b9310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x55b3324843a0 0x55b3324b9310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f611803a440 msgr2=0x7f611803c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f611803a440 0x7f611803c900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6124003f20 tx=0x7f61240027b0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 10 monclient: shutdown 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 msgr2=0x55b332483840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 --2- 172.21.15.86:0/1962953059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b3324a2ff0 0x55b332483840 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f612c003150 tx=0x7f612c00b4b0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f61398ae640 1 -- 172.21.15.86:0/1962953059 reap_dead start 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 shutdown_connections 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 >> 172.21.15.86:0/1962953059 conn(0x55b3323eadf0 msgr2=0x55b332406bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 shutdown_connections 2024-09-17T15:27:29.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.636+0000 7f6139b43100 1 -- 172.21.15.86:0/1962953059 wait complete. 2024-09-17T15:27:29.646 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T15:27:29.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T15:27:29.647 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T15:27:29.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.772+0000 7f3a30ab1640 10 monclient: get_monmap_and_config 2024-09-17T15:27:29.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.772+0000 7f3a30ab1640 10 monclient: build_initial_monmap 2024-09-17T15:27:29.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.772+0000 7f3a30ab1640 10 monclient: monmap: 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:29.779866+0000 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:29.779866+0000 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:29.778 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 Processor -- start 2024-09-17T15:27:29.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 -- start start 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient: init 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a2c0a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 0x7f3a2c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a2c0a6280 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3a2c09d2a0 con 0x7f3a2c0a5bd0 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:29.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2c09c0a0 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a30ab1640 10 monclient(hunting): authenticate will time out at 982.745586s 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33088/0 (socket says 172.21.15.86:33088) 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 -- 172.21.15.86:0/4199946953 learned_addr learned my addr 172.21.15.86:0/4199946953 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): get_auth_request con 0x7f3a2c13d2f0 auth_method 0 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:29.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:29.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 1 --2- 172.21.15.86:0/4199946953 >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 0x7f3a2c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 10 monclient(hunting): get_auth_request con 0x7f3a2c0a5bd0 auth_method 0 2024-09-17T15:27:29.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:29.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:29.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2024-09-17T15:27:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 -- 172.21.15.86:0/4199946953 >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 msgr2=0x7f3a2c09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 --2- 172.21.15.86:0/4199946953 >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 0x7f3a2c09cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 -- 172.21.15.86:0/4199946953 >> v1:172.21.15.203:6789/0 conn(0x7f3a2c0a50e0 legacy=0x7f3a2c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 monclient: found mon.noname-b 2024-09-17T15:27:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a2ad9d640 1 -- 172.21.15.86:0/4199946953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c09c380 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.776+0000 7f3a29d9b640 1 --2- 172.21.15.86:0/4199946953 >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 0x7f3a2c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:27:29.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2ad9d640 1 -- 172.21.15.86:0/4199946953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 msgr2=0x7f3a2c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2ad9d640 1 --2- 172.21.15.86:0/4199946953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009e50 tx=0x7f3a2401c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=275864619aeb9457 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 1 -- 172.21.15.86:0/4199946953 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240a5600 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: dump: 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:29.792 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:29.793 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:29.793 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: _finish_auth 0 2024-09-17T15:27:29.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:29.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 1 -- 172.21.15.86:0/4199946953 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a240a57a0 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 1 -- 172.21.15.86:0/4199946953 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240a5b70 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a2959a640 10 monclient: dump: 2024-09-17T15:27:29.794 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 5 monclient: authenticate success, global_id 4187 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 4 set_mon_vals no callback set 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 10 monclient: shutdown 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:29.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 -- 172.21.15.86:0/4199946953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 msgr2=0x7f3a2c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/4199946953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009e50 tx=0x7f3a2401c860 comp rx=0 tx=0).stop 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 -- 172.21.15.86:0/4199946953 shutdown_connections 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/4199946953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c13d2f0 0x7f3a2c13f6e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/4199946953 >> v2:172.21.15.203:3300/0 conn(0x7f3a2c0a5bd0 0x7f3a2c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 -- 172.21.15.86:0/4199946953 >> 172.21.15.86:0/4199946953 conn(0x7f3a2c0aaae0 msgr2=0x7f3a2c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 -- 172.21.15.86:0/4199946953 shutdown_connections 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.780+0000 7f3a30ab1640 1 -- 172.21.15.86:0/4199946953 wait complete. 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: build_initial_monmap 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: monmap: 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:29.796 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 Processor -- start 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- start start 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: init 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:29.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 0x7f3a2c138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 0x7f3a2c09f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c142650 con 0x7f3a2c0a5bd0 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:29.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c0a6ab0 con 0x7f3a2c13d2f0 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2c1427d0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient(hunting): authenticate will time out at 982.752997s 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 0x7f3a2c138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 0x7f3a2c138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43272/0 (socket says 172.21.15.86:43272) 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 1 -- 172.21.15.86:0/1146601485 learned_addr learned my addr 172.21.15.86:0/1146601485 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33102/0 (socket says 172.21.15.86:33102) 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): get_auth_request con 0x7f3a2c0a5bd0 auth_method 0 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:29.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): get_auth_request con 0x7f3a2c0a50e0 auth_method 0 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2ad9d640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 0x7f3a2c09f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 -- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 msgr2=0x7f3a2c09f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 0x7f3a2c09f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 -- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 msgr2=0x7f3a2c138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 0x7f3a2c138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 monclient: found mon.noname-c 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 -- 172.21.15.86:0/1146601485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c1228d0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2ad9d640 10 monclient: get_auth_request con 0x7f3a2c13d2f0 auth_method 0 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2ad9d640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 0x7f3a2c09f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 -- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 msgr2=0x7f3a2c1cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:29.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a2a59c640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00c410 tx=0x7f3a1c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c017070 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: dump: 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:29.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: _finish_auth 0 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:29.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1c0069e0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 5 monclient: authenticate success, global_id 4190 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _renew_subs 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c122bb0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c006dd0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a1affd640 10 monclient: dump: 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a28d99640 4 set_mon_vals no callback set 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _renew_subs 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.784+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c0a79f0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a30ab1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:29.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a30ab1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a30ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2c09cd60 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3a1c01a070 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a1affd640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 0x7f3a0c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f3a1c04f130 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a29d9b640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 0x7f3a0c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.788+0000 7f3a29d9b640 10 monclient: get_auth_request con 0x7f3a0c03bb60 auth_method 0 2024-09-17T15:27:29.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.792+0000 7f3a29d9b640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 0x7f3a0c03e020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c13f040 tx=0x7f3a20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:29.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.792+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3a1c01bd70 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.792+0000 7f3a1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:29.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.792+0000 7f3a1affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a30ab1640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:27:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a30ab1640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:27:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a30ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f3a2c074be0 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a1affd640 1 -- 172.21.15.86:0/1146601485 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v4) ==== 86+0+159 (secure 0 0 0) 0x7f3a1c051170 con 0x7f3a2c0a50e0 2024-09-17T15:27:29.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-17T15:27:29.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.964+0000 7f3a1affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:29.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 msgr2=0x7f3a0c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 0x7f3a0c03e020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c13f040 tx=0x7f3a20008040 comp rx=0 tx=0).stop 2024-09-17T15:27:29.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 10 monclient: shutdown 2024-09-17T15:27:29.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:29.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 msgr2=0x7f3a2c1cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00c410 tx=0x7f3a1c00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 shutdown_connections 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0c03bb60 0x7f3a0c03e020 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a2c13d2f0 0x7f3a2c09f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a2c0a5bd0 0x7f3a2c138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 --2- 172.21.15.86:0/1146601485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a2c0a50e0 0x7f3a2c1cfb00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 >> 172.21.15.86:0/1146601485 conn(0x7f3a2c0aaae0 msgr2=0x7f3a2c0a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 shutdown_connections 2024-09-17T15:27:29.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:29.968+0000 7f3a30ab1640 1 -- 172.21.15.86:0/1146601485 wait complete. 2024-09-17T15:27:29.987 INFO:teuthology.orchestra.run.smithi086.stdout:[{"version":4,"timestamp":"2024-09-17T15:27:23.368783+0000","name":"","changes":[{"name":"osd.4/osd_mclock_max_capacity_iops_ssd","new_value":"5897.573932"}]}] 2024-09-17T15:27:29.987 INFO:tasks.ceph_manager:config epoch is 4 2024-09-17T15:27:29.988 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T15:27:29.988 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T15:27:29.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T15:27:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: get_monmap_and_config 2024-09-17T15:27:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: monmap: 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:30.102100+0000 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:30.102100+0000 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:30.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 Processor -- start 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 -- start start 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: init 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a000a5070 0x7f5a0009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-a con 0x7f5a000a5070 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a000a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a0013d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a000a6f40 con 0x7f5a000a5450 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a000a65e0 con 0x7f5a0013d280 2024-09-17T15:27:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5a0009c2a0 con 0x7f5a000a5070 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a07208640 10 monclient(hunting): authenticate will time out at 983.067495s 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a000a5070 0x7f5a0009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f59fffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5a000a5450 0x7f5a000a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37458/0 (socket says 172.21.15.86:37458) 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f59fffff640 1 -- 172.21.15.86:0/3327360977 learned_addr learned my addr 172.21.15.86:0/3327360977 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): get_auth_request con 0x7f5a000a5070 auth_method 0 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.096+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_done global_id 4188 payload 274 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 1 -- 172.21.15.86:0/3327360977 >> v1:172.21.15.203:6789/0 conn(0x7f5a0013d280 legacy=0x7f5a0013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 1 -- 172.21.15.86:0/3327360977 >> v1:172.21.15.86:6789/0 conn(0x7f5a000a5450 legacy=0x7f5a000a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 1 monclient: found mon.noname-a 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.100+0000 7f5a04fa4640 1 -- 172.21.15.86:0/3327360977 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0009c580 con 0x7f5a000a5070 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 -- 172.21.15.86:0/3327360977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a000a5070 msgr2=0x7f5a0009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 --2- 172.21.15.86:0/3327360977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a000a5070 0x7f5a0009cf60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097c0 tx=0x7f59f801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f562e32508d56589 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 1 -- 172.21.15.86:0/3327360977 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f80b0070 con 0x7f5a000a5070 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:30.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: dump: 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 5 monclient: authenticate success, global_id 4188 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 1 -- 172.21.15.86:0/3327360977 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f80ab600 con 0x7f5a000a5070 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 4 set_mon_vals no callback set 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 1 -- 172.21.15.86:0/3327360977 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f80ab9d0 con 0x7f5a000a5070 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59ff7fe640 10 monclient: dump: 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.111 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: shutdown 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- 172.21.15.86:0/3327360977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a000a5070 msgr2=0x7f5a0009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 --2- 172.21.15.86:0/3327360977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a000a5070 0x7f5a0009cf60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097c0 tx=0x7f59f801c360 comp rx=0 tx=0).stop 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- 172.21.15.86:0/3327360977 shutdown_connections 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 --2- 172.21.15.86:0/3327360977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a000a5070 0x7f5a0009cf60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- 172.21.15.86:0/3327360977 >> 172.21.15.86:0/3327360977 conn(0x7f5a000742a0 msgr2=0x7f5a000746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- 172.21.15.86:0/3327360977 shutdown_connections 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- 172.21.15.86:0/3327360977 wait complete. 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: monmap: 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 Processor -- start 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- start start 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: init 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-c con 0x7f5a000a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 0x7f5a00138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-b con 0x7f5a000a5450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 0x7f5a0013a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0013d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00143940 con 0x7f5a0013d280 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a001437c0 con 0x7f5a000a5450 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5a00143ac0 con 0x7f5a000a5070 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a07208640 10 monclient(hunting): authenticate will time out at 983.074526s 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33104/0 (socket says 172.21.15.86:33104) 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 0x7f5a0013a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 -- 172.21.15.86:0/4255977882 learned_addr learned my addr 172.21.15.86:0/4255977882 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:30.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): get_auth_request con 0x7f5a000a5070 auth_method 0 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): get_auth_request con 0x7f5a0013d280 auth_method 0 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59fffff640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 0x7f5a00138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient(hunting): handle_auth_done global_id 4191 payload 824 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 -- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 msgr2=0x7f5a001cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 -- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 msgr2=0x7f5a00138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 0x7f5a00138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 monclient: found mon.noname-a 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a057a5640 1 -- 172.21.15.86:0/4255977882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0013bcf0 con 0x7f5a0013d280 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59fffff640 10 monclient: get_auth_request con 0x7f5a000a5450 auth_method 0 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f5a04fa4640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:30.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.104+0000 7f59fffff640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 0x7f5a00138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a057a5640 1 -- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 msgr2=0x7f5a0013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a057a5640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 0x7f5a0013a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59f000c780 tx=0x7f59f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0016020 con 0x7f5a0013d280 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: dump: 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f000ac60 con 0x7f5a0013d280 2024-09-17T15:27:30.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 5 monclient: authenticate success, global_id 4191 2024-09-17T15:27:30.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _renew_subs 2024-09-17T15:27:30.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a000a1860 con 0x7f5a0013d280 2024-09-17T15:27:30.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0012660 con 0x7f5a0013d280 2024-09-17T15:27:30.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:30.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 10 monclient: dump: 2024-09-17T15:27:30.124 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.125 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.126 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.126 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.126 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59feffd640 4 set_mon_vals no callback set 2024-09-17T15:27:30.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _renew_subs 2024-09-17T15:27:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a001211f0 con 0x7f5a0013d280 2024-09-17T15:27:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0009cfe0 con 0x7f5a0013d280 2024-09-17T15:27:30.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f59f0019020 con 0x7f5a0013d280 2024-09-17T15:27:30.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 0x7f59e003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f59f004fca0 con 0x7f5a0013d280 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a04fa4640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 0x7f59e003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.108+0000 7f5a04fa4640 10 monclient: get_auth_request con 0x7f59e003bb60 auth_method 0 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.112+0000 7f5a04fa4640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 0x7f59e003e020 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59f801c190 tx=0x7f59f80a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.112+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f59f0020030 con 0x7f5a0013d280 2024-09-17T15:27:30.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.112+0000 7f59fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.112+0000 7f59fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:30.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.308+0000 7f5a07208640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:27:30.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.308+0000 7f5a07208640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:27:30.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.308+0000 7f5a07208640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.308+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f5a0013c850 con 0x7f5a0013d280 2024-09-17T15:27:30.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.312+0000 7f59fd7fa640 1 -- 172.21.15.86:0/4255977882 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93965 (secure 0 0 0) 0x7f59f000ae10 con 0x7f5a0013d280 2024-09-17T15:27:30.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.312+0000 7f59fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-17T15:27:30.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.312+0000 7f59fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:30.317 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 msgr2=0x7f59e003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 0x7f59e003e020 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f59f801c190 tx=0x7f59f80a6000 comp rx=0 tx=0).stop 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 10 monclient: shutdown 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 msgr2=0x7f5a0013a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 0x7f5a0013a230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f59f000c780 tx=0x7f59f000cc40 comp rx=0 tx=0).stop 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 shutdown_connections 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59e003bb60 0x7f59e003e020 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a0013d280 0x7f5a0013a230 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a000a5450 0x7f5a00138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 --2- 172.21.15.86:0/4255977882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a000a5070 0x7f5a001cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 >> 172.21.15.86:0/4255977882 conn(0x7f5a000742a0 msgr2=0x7f5a000a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 shutdown_connections 2024-09-17T15:27:30.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.316+0000 7f5a07208640 1 -- 172.21.15.86:0/4255977882 wait complete. 2024-09-17T15:27:30.333 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":2899236933},{"type":"v1","addr":"172.21.15.203:6825","nonce":2899236933}]},"active_addr":"172.21.15.203:6825/2899236933","active_change":"2024-09-17T15:27:17.611548+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3121703291}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":638852694}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3487148618}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3359708929}]}]} 2024-09-17T15:27:30.335 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T15:27:30.335 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T15:27:30.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: get_monmap_and_config 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: monmap: 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:30.446505+0000 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:30.446505+0000 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:30.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:30.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:30.445 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:30.445 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:30.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 Processor -- start 2024-09-17T15:27:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 -- start start 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: init 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d240689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7d240694c0 0x7f7d2406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-c con 0x7f7d240694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d2413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-b con 0x7f7d2413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d24074470 con 0x7f7d240689d0 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7d2409c900 con 0x7f7d240694c0 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2409cb90 con 0x7f7d2413aff0 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.440+0000 7f7d2a389640 10 monclient(hunting): authenticate will time out at 983.411928s 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d23fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7d240689d0 0x7f7d24068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37468/0 (socket says 172.21.15.86:37468) 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d23fff640 1 -- 172.21.15.86:0/1936921602 learned_addr learned my addr 172.21.15.86:0/1936921602 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 --2- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d2413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): get_auth_request con 0x7f7d2413aff0 auth_method 0 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d237fe640 1 --2- 172.21.15.86:0/1936921602 >> v2:172.21.15.203:3300/0 conn(0x7f7d240694c0 0x7f7d2406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d237fe640 10 monclient(hunting): get_auth_request con 0x7f7d240694c0 auth_method 0 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 -- 172.21.15.86:0/1936921602 >> v2:172.21.15.203:3300/0 conn(0x7f7d240694c0 msgr2=0x7f7d2406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 --2- 172.21.15.86:0/1936921602 >> v2:172.21.15.203:3300/0 conn(0x7f7d240694c0 0x7f7d2406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 -- 172.21.15.86:0/1936921602 >> v1:172.21.15.86:6789/0 conn(0x7f7d240689d0 legacy=0x7f7d24068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 monclient: found mon.noname-b 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:30.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.444+0000 7f7d28926640 1 -- 172.21.15.86:0/1936921602 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2409ce70 con 0x7f7d2413aff0 2024-09-17T15:27:30.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 1 -- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 msgr2=0x7f7d2413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 1 --2- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d2413d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009d90 tx=0x7f7d1c01c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=730a783c1c8f3397 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 1 -- 172.21.15.86:0/1936921602 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d1c0b2070 con 0x7f7d2413aff0 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: dump: 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 5 monclient: authenticate success, global_id 4196 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 1 -- 172.21.15.86:0/1936921602 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d1c0a5890 con 0x7f7d2413aff0 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 4 set_mon_vals no callback set 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 1 -- 172.21.15.86:0/1936921602 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2409c2d0 con 0x7f7d2413aff0 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:30.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d22ffd640 10 monclient: dump: 2024-09-17T15:27:30.456 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.456 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.456 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.456 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.457 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: shutdown 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 msgr2=0x7f7d2413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d2413d3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009d90 tx=0x7f7d1c01c830 comp rx=0 tx=0).stop 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- 172.21.15.86:0/1936921602 shutdown_connections 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1936921602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d2413d3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1936921602 >> v2:172.21.15.203:3300/0 conn(0x7f7d240694c0 0x7f7d2406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- 172.21.15.86:0/1936921602 >> 172.21.15.86:0/1936921602 conn(0x7f7d240a6440 msgr2=0x7f7d240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- 172.21.15.86:0/1936921602 shutdown_connections 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- 172.21.15.86:0/1936921602 wait complete. 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: monmap: 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.458 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 Processor -- start 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- start start 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: init 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-a con 0x7f7d240689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d240694c0 0x7f7d241d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-b con 0x7f7d240694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d24070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): picked mon.noname-c con 0x7f7d2413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7d24140670 con 0x7f7d240689d0 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7d241404f0 con 0x7f7d240694c0 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7d241407f0 con 0x7f7d2413aff0 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d2a389640 10 monclient(hunting): authenticate will time out at 983.418743s 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43298/0 (socket says 172.21.15.86:43298) 2024-09-17T15:27:30.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 -- 172.21.15.86:0/1362619764 learned_addr learned my addr 172.21.15.86:0/1362619764 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): get_auth_request con 0x7f7d240689d0 auth_method 0 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d24070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): get_auth_request con 0x7f7d2413aff0 auth_method 0 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d237fe640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d240694c0 0x7f7d241d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d237fe640 10 monclient(hunting): get_auth_request con 0x7f7d240694c0 auth_method 0 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient(hunting): handle_auth_done global_id 4194 payload 824 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 -- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 msgr2=0x7f7d24070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d24070e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 -- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d240694c0 msgr2=0x7f7d241d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d240694c0 0x7f7d241d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 monclient: found mon.noname-a 2024-09-17T15:27:30.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d23fff640 1 -- 172.21.15.86:0/1362619764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2409d460 con 0x7f7d240689d0 2024-09-17T15:27:30.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.448+0000 7f7d28926640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d24070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:30.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d23fff640 1 -- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 msgr2=0x7f7d241cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d23fff640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c470 tx=0x7f7d1800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18007070 con 0x7f7d240689d0 2024-09-17T15:27:30.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:30.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: dump: 2024-09-17T15:27:30.466 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.466 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.466 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d18007210 con 0x7f7d240689d0 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 5 monclient: authenticate success, global_id 4194 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _renew_subs 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2413d330 con 0x7f7d240689d0 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18007600 con 0x7f7d240689d0 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 10 monclient: dump: 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.469 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _renew_subs 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d227fc640 4 set_mon_vals no callback set 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d241356a0 con 0x7f7d240689d0 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d241214c0 con 0x7f7d240689d0 2024-09-17T15:27:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7d1801b400 con 0x7f7d240689d0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 0x7f7d1003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f7d1804fb80 con 0x7f7d240689d0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d237fe640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 0x7f7d1003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.452+0000 7f7d237fe640 10 monclient: get_auth_request con 0x7f7d1003bb60 auth_method 0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.456+0000 7f7d237fe640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 0x7f7d1003e020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d241210c0 tx=0x7f7d14008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.456+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d18010d10 con 0x7f7d240689d0 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.456+0000 7f7d20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.456+0000 7f7d20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:30.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.612+0000 7f7d2a389640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.612+0000 7f7d2a389640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.612+0000 7f7d2a389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.612+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d240729c0 con 0x7f7d240689d0 2024-09-17T15:27:30.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d20ff9640 1 -- 172.21.15.86:0/1362619764 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12413 (secure 0 0 0) 0x7f7d240729c0 con 0x7f7d240689d0 2024-09-17T15:27:30.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:30.620 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:30.620 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":17,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:29.623785+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:27:23.674476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:27:24.613917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.592045+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.760848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.159435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.733344+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.364717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.514816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 msgr2=0x7f7d1003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 0x7f7d1003e020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d241210c0 tx=0x7f7d14008040 comp rx=0 tx=0).stop 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 10 monclient: shutdown 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 msgr2=0x7f7d241cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.616+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c470 tx=0x7f7d1800c930 comp rx=0 tx=0).stop 2024-09-17T15:27:30.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 shutdown_connections 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d1003bb60 0x7f7d1003e020 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d2413aff0 0x7f7d24070e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d240694c0 0x7f7d241d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 --2- 172.21.15.86:0/1362619764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d240689d0 0x7f7d241cfc80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 >> 172.21.15.86:0/1362619764 conn(0x7f7d240a6440 msgr2=0x7f7d2409eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 shutdown_connections 2024-09-17T15:27:30.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.620+0000 7f7d2a389640 1 -- 172.21.15.86:0/1362619764 wait complete. 2024-09-17T15:27:30.637 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T15:27:30.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 10 monclient: get_monmap_and_config 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 10 monclient: monmap: 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:30.746639+0000 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:30.746639+0000 2024-09-17T15:27:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:30.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 1 Processor -- start 2024-09-17T15:27:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 1 -- start start 2024-09-17T15:27:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.740+0000 7fa9f8568640 10 monclient: init 2024-09-17T15:27:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9f00a5070 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 0x7fa9f009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-c con 0x7fa9f00a5b60 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-b con 0x7fa9f013d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f00a6f40 con 0x7fa9f00a5070 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f00a65e0 con 0x7fa9f00a5b60 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f009c2a0 con 0x7fa9f013d280 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f8568640 10 monclient(hunting): authenticate will time out at 983.712576s 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33138/0 (socket says 172.21.15.86:33138) 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2214245814 learned_addr learned my addr 172.21.15.86:0/2214245814 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): get_auth_request con 0x7fa9f013d280 auth_method 0 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2214245814 >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 msgr2=0x7fa9f009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f5b03640 1 --2- 172.21.15.86:0/2214245814 >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 0x7fa9f009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 --2- 172.21.15.86:0/2214245814 >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 0x7fa9f009cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2214245814 >> v1:172.21.15.203:6789/0 conn(0x7fa9f00a5070 legacy=0x7fa9f00a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 monclient: found mon.noname-b 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2214245814 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f009c580 con 0x7fa9f013d280 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f5b03640 10 monclient: get_auth_request con 0x7fa9f00a5b60 auth_method 0 2024-09-17T15:27:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.744+0000 7fa9f5b03640 1 --2- 172.21.15.86:0/2214245814 >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 0x7fa9f009cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2214245814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 msgr2=0x7fa9f013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 1 --2- 172.21.15.86:0/2214245814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009e50 tx=0x7fa9e801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfc80587caa90339 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 1 -- 172.21.15.86:0/2214245814 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a5530 con 0x7fa9f013d280 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: dump: 2024-09-17T15:27:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 1 -- 172.21.15.86:0/2214245814 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e80a56d0 con 0x7fa9f013d280 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 1 -- 172.21.15.86:0/2214245814 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80a5aa0 con 0x7fa9f013d280 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5302640 10 monclient: dump: 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 5 monclient: authenticate success, global_id 4202 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 4 set_mon_vals no callback set 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: shutdown 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- 172.21.15.86:0/2214245814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 msgr2=0x7fa9f013f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2214245814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8009e50 tx=0x7fa9e801c8f0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- 172.21.15.86:0/2214245814 shutdown_connections 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2214245814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013f670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2214245814 >> v2:172.21.15.203:3300/0 conn(0x7fa9f00a5b60 0x7fa9f009cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- 172.21.15.86:0/2214245814 >> 172.21.15.86:0/2214245814 conn(0x7fa9f00742a0 msgr2=0x7fa9f00746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- 172.21.15.86:0/2214245814 shutdown_connections 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- 172.21.15.86:0/2214245814 wait complete. 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: build_initial_monmap 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: monmap: 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 Processor -- start 2024-09-17T15:27:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- start start 2024-09-17T15:27:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: init 2024-09-17T15:27:30.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:30.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:30.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-a con 0x7fa9f00a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f00a5b60 0x7fa9f0138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-b con 0x7fa9f00a5b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): picked mon.noname-c con 0x7fa9f013d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f01425e0 con 0x7fa9f00a5070 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f00a6a70 con 0x7fa9f00a5b60 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f0142760 con 0x7fa9f013d280 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f8568640 10 monclient(hunting): authenticate will time out at 983.719554s 2024-09-17T15:27:30.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33144/0 (socket says 172.21.15.86:33144) 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43306/0 (socket says 172.21.15.86:43306) 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 1 -- 172.21.15.86:0/2947193216 learned_addr learned my addr 172.21.15.86:0/2947193216 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): get_auth_request con 0x7fa9f013d280 auth_method 0 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): get_auth_request con 0x7fa9f00a5070 auth_method 0 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6b05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5b03640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f00a5b60 0x7fa9f0138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5b03640 10 monclient(hunting): get_auth_request con 0x7fa9f00a5b60 auth_method 0 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5b03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5b03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f5b03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 1 -- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 msgr2=0x7fa9f013b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.748+0000 7fa9f6304640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 -- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f00a5b60 msgr2=0x7fa9f0138190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f00a5b60 0x7fa9f0138190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 10 monclient: _finish_hunting 0 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 monclient: found mon.noname-a 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 -- 172.21.15.86:0/2947193216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f0122b20 con 0x7fa9f00a5070 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6b05640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 -- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 msgr2=0x7fa9f01cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f6304640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00c3e0 tx=0x7fa9ec00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec0119a0 con 0x7fa9f00a5070 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: dump: 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:30.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ec017070 con 0x7fa9f00a5070 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 5 monclient: authenticate success, global_id 4197 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _renew_subs 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f006f300 con 0x7fa9f00a5070 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec011d80 con 0x7fa9f00a5070 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 10 monclient: dump: 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:30.766 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f4b01640 4 set_mon_vals no callback set 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _renew_subs 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f0122840 con 0x7fa9f00a5070 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f009cf60 con 0x7fa9f00a5070 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa9ec01c460 con 0x7fa9f00a5070 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 0x7fa9d403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.752+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7fa9ec04fc20 con 0x7fa9f00a5070 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.756+0000 7fa9f5b03640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 0x7fa9d403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.756+0000 7fa9f5b03640 10 monclient: get_auth_request con 0x7fa9d403bb60 auth_method 0 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.756+0000 7fa9f5b03640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 0x7fa9d403e020 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0139000 tx=0x7fa9e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:30.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.760+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa9ec010cf0 con 0x7fa9f00a5070 2024-09-17T15:27:30.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.760+0000 7fa9e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:30.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.760+0000 7fa9e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9f8568640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9f8568640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9f8568640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9f00008e0 con 0x7fa9f00a5070 2024-09-17T15:27:30.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9e6ffd640 1 -- 172.21.15.86:0/2947193216 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12413 (secure 0 0 0) 0x7fa9ec012420 con 0x7fa9f00a5070 2024-09-17T15:27:30.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:30.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.916+0000 7fa9e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:30.921 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":17,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:29.623785+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:27:23.674476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:27:24.613917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.592045+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.760848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.159435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.733344+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.364717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.514816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 msgr2=0x7fa9d403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 0x7fa9d403e020 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0139000 tx=0x7fa9e0008040 comp rx=0 tx=0).stop 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 10 monclient: shutdown 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 msgr2=0x7fa9f01cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00c3e0 tx=0x7fa9ec00c8a0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 shutdown_connections 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9d403bb60 0x7fa9d403e020 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9f013d280 0x7fa9f013b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f00a5b60 0x7fa9f0138190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 --2- 172.21.15.86:0/2947193216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9f00a5070 0x7fa9f01cfaf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:30.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 >> 172.21.15.86:0/2947193216 conn(0x7fa9f00742a0 msgr2=0x7fa9f00a0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:30.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 shutdown_connections 2024-09-17T15:27:30.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:30.920+0000 7fa9f8568640 1 -- 172.21.15.86:0/2947193216 wait complete. 2024-09-17T15:27:30.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T15:27:30.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T15:27:30.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T15:27:30.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T15:27:30.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T15:27:30.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T15:27:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: monmap: 2024-09-17T15:27:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.056756+0000 2024-09-17T15:27:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.056756+0000 2024-09-17T15:27:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.057 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: monmap: 2024-09-17T15:27:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.056757+0000 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.056757+0000 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.060 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.060 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.060 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.060 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.060 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 Processor -- start 2024-09-17T15:27:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 -- start start 2024-09-17T15:27:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: init 2024-09-17T15:27:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 Processor -- start 2024-09-17T15:27:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 -- start start 2024-09-17T15:27:31.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: init 2024-09-17T15:27:31.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-17T15:27:31.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:27:31.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf040a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf040a5bd0 0x7faf0409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-a con 0x7faf040a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-b con 0x7faf0413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf040a6280 con 0x7faf040a50e0 2024-09-17T15:27:31.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7faf0409d2a0 con 0x7faf040a5bd0 2024-09-17T15:27:31.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faf0409c0a0 con 0x7faf0413d2f0 2024-09-17T15:27:31.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf0b44d640 10 monclient(hunting): authenticate will time out at 984.022113s 2024-09-17T15:27:31.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bf40a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1bf40a5bd0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bf409cd60 0x7f1bf4140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409cd60 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40a6280 con 0x7f1bf40a5bd0 2024-09-17T15:27:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf409c0a0 con 0x7f1bf40a50e0 2024-09-17T15:27:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf409c380 con 0x7f1bf409cd60 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf95ad640 10 monclient(hunting): authenticate will time out at 984.022211s 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bf409cd60 0x7f1bf4140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf27fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1bf40a5bd0 0x7f1bf413d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37484/0 (socket says 172.21.15.86:37484) 2024-09-17T15:27:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf27fc640 1 -- 172.21.15.86:0/132791139 learned_addr learned my addr 172.21.15.86:0/132791139 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): get_auth_request con 0x7f1bf409cd60 auth_method 0 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf040a5bd0 0x7faf0409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33152/0 (socket says 172.21.15.86:33152) 2024-09-17T15:27:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 1 -- 172.21.15.86:0/3804094701 learned_addr learned my addr 172.21.15.86:0/3804094701 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): get_auth_request con 0x7faf0413d2f0 auth_method 0 2024-09-17T15:27:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): get_auth_request con 0x7faf040a5bd0 auth_method 0 2024-09-17T15:27:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf091e9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faf040a50e0 0x7faf040a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37500/0 (socket says 172.21.15.86:37500) 2024-09-17T15:27:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf099ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-09-17T15:27:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-09-17T15:27:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 1 -- 172.21.15.86:0/132791139 >> v1:172.21.15.203:6789/0 conn(0x7f1bf40a50e0 legacy=0x7f1bf40a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 1 -- 172.21.15.86:0/132791139 >> v1:172.21.15.86:6789/0 conn(0x7f1bf40a5bd0 legacy=0x7f1bf413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7f1bf37fe640 1 -- 172.21.15.86:0/132791139 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf409c660 con 0x7f1bf409cd60 2024-09-17T15:27:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 -- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 msgr2=0x7faf0413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 -- 172.21.15.86:0/3804094701 >> v1:172.21.15.86:6789/0 conn(0x7faf040a50e0 legacy=0x7faf040a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.052+0000 7faf089e8640 1 -- 172.21.15.86:0/3804094701 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0409c380 con 0x7faf040a5bd0 2024-09-17T15:27:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.056+0000 7faf099ea640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 -- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a5bd0 msgr2=0x7faf0409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a5bd0 0x7faf0409cd60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7182806a97715af6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 1 -- 172.21.15.86:0/3804094701 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80b0070 con 0x7faf040a5bd0 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: dump: 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.087 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 1 -- 172.21.15.86:0/3804094701 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef80ab600 con 0x7faf040a5bd0 2024-09-17T15:27:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 5 monclient: authenticate success, global_id 4203 2024-09-17T15:27:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 4 set_mon_vals no callback set 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 1 -- 172.21.15.86:0/3804094701 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80a4e60 con 0x7faf040a5bd0 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf03fff640 10 monclient: dump: 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: shutdown 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a5bd0 msgr2=0x7faf0409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a5bd0 0x7faf0409cd60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801c350 comp rx=0 tx=0).stop 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- 172.21.15.86:0/3804094701 shutdown_connections 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf0413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- 172.21.15.86:0/3804094701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a5bd0 0x7faf0409cd60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- 172.21.15.86:0/3804094701 >> 172.21.15.86:0/3804094701 conn(0x7faf040aaae0 msgr2=0x7faf040aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- 172.21.15.86:0/3804094701 shutdown_connections 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- 172.21.15.86:0/3804094701 wait complete. 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: monmap: 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf37fe640 1 -- 172.21.15.86:0/132791139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 msgr2=0x7f1bf4140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf37fe640 1 --2- 172.21.15.86:0/132791139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf4140a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009d90 tx=0x7f1bec01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60134ef9a729333 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 Processor -- start 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 1 -- 172.21.15.86:0/132791139 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0b2070 con 0x7f1bf409cd60 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: dump: 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 1 -- 172.21.15.86:0/132791139 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec0a5840 con 0x7f1bf409cd60 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 5 monclient: authenticate success, global_id 4200 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 1 -- 172.21.15.86:0/132791139 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0a5c10 con 0x7f1bf409cd60 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf1ffb640 10 monclient: dump: 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.095 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- start start 2024-09-17T15:27:31.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: init 2024-09-17T15:27:31.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf040a50e0 0x7faf0413a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-b con 0x7faf040a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf040a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-c con 0x7faf0413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a3610 0x7faf040a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): picked mon.noname-a con 0x7faf040a3610 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faf04142650 con 0x7faf040a3610 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faf040a6b00 con 0x7faf040a50e0 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faf041427d0 con 0x7faf0413d2f0 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf0b44d640 10 monclient(hunting): authenticate will time out at 984.031366s 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf040a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf040a30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33160/0 (socket says 172.21.15.86:33160) 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 -- 172.21.15.86:0/353649778 learned_addr learned my addr 172.21.15.86:0/353649778 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): get_auth_request con 0x7faf0413d2f0 auth_method 0 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 4 set_mon_vals no callback set 2024-09-17T15:27:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 10 monclient: shutdown 2024-09-17T15:27:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 1 -- 172.21.15.86:0/132791139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 msgr2=0x7f1bf4140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/132791139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf4140a20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009d90 tx=0x7f1bec01c800 comp rx=0 tx=0).stop 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf091e9640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf040a50e0 0x7faf0413a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf040a50e0 msgr2=0x7faf0413a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf040a50e0 0x7faf0413a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a3610 msgr2=0x7faf040a3a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf040a3610 0x7faf040a3a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf089e8640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0413b480 con 0x7faf0413d2f0 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 1 -- 172.21.15.86:0/132791139 shutdown_connections 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/132791139 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf4140a20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7f1bf95ad640 1 -- 172.21.15.86:0/132791139 >> 172.21.15.86:0/132791139 conn(0x7f1bf40aaae0 msgr2=0x7f1bf40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf091e9640 10 monclient: get_auth_request con 0x7faf040a50e0 auth_method 0 2024-09-17T15:27:31.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.060+0000 7faf091e9640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faf040a50e0 0x7faf0413a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- 172.21.15.86:0/132791139 shutdown_connections 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- 172.21.15.86:0/132791139 wait complete. 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient: monmap: 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 Processor -- start 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- start start 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient: init 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf409cd60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bf40a50e0 0x7f1bf4139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf40a5bd0 0x7f1bf4139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf40a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4143900 con 0x7f1bf409cd60 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4143780 con 0x7f1bf40a50e0 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf4143a80 con 0x7f1bf40a5bd0 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf95ad640 10 monclient(hunting): authenticate will time out at 984.033522s 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf41392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43338/0 (socket says 172.21.15.86:43338) 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 -- 172.21.15.86:0/1745232521 learned_addr learned my addr 172.21.15.86:0/1745232521 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): get_auth_request con 0x7f1bf409cd60 auth_method 0 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bf40a50e0 0x7f1bf4139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf27fc640 10 monclient(hunting): get_auth_request con 0x7f1bf40a50e0 auth_method 0 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient(hunting): handle_auth_done global_id 4206 payload 824 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf40a5bd0 msgr2=0x7f1bf4139d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf40a5bd0 0x7f1bf4139d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bf40a50e0 msgr2=0x7f1bf4139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bf40a50e0 0x7f1bf4139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7f1bf2ffd640 1 -- 172.21.15.86:0/1745232521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf413c460 con 0x7f1bf409cd60 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7faf089e8640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 msgr2=0x7faf040a30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7faf089e8640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf040a30d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.064+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf2ffd640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 msgr2=0x7f1bf41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf2ffd640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf41392d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1be400c450 tx=0x7f1be400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4012a40 con 0x7f1bf409cd60 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: dump: 2024-09-17T15:27:31.108 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be4017070 con 0x7f1bf409cd60 2024-09-17T15:27:31.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 5 monclient: authenticate success, global_id 4206 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 10 monclient: _renew_subs 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf413c740 con 0x7f1bf409cd60 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 10 monclient: _renew_subs 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf413a6c0 con 0x7f1bf409cd60 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4012e00 con 0x7f1bf409cd60 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 10 monclient: dump: 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf17fa640 4 set_mon_vals no callback set 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef80b0050 con 0x7faf0413d2f0 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: dump: 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef80b1040 con 0x7faf0413d2f0 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8004260 con 0x7faf0413d2f0 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: dump: 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 5 monclient: authenticate success, global_id 4211 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 10 monclient: _renew_subs 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0413d6d0 con 0x7faf0413d2f0 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf037fe640 4 set_mon_vals no callback set 2024-09-17T15:27:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 10 monclient: _renew_subs 2024-09-17T15:27:31.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf040a4b00 con 0x7faf0413d2f0 2024-09-17T15:27:31.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faf0413b8a0 con 0x7faf0413d2f0 2024-09-17T15:27:31.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1be401c460 con 0x7f1bf409cd60 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be003bbb0 0x7f1be003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f1be4010cf0 con 0x7f1bf409cd60 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7f1bd4001660 0x7f1bd4003b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 --> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1bd4006b10 con 0x7f1bd4001660 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faef80be060 con 0x7faf0413d2f0 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faef003bbb0 0x7faef003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7faef80e9b20 con 0x7faf0413d2f0 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7faef0041aa0 0x7faef0043ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faef800af30 con 0x7faef0041aa0 2024-09-17T15:27:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7faef80e9e00 con 0x7faf0413d2f0 2024-09-17T15:27:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf01ffb640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T15:27:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be003bbb0 0x7f1be003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf099ea640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7faef0041aa0 0x7faef0043ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf27fc640 10 monclient: get_auth_request con 0x7f1be003bbb0 auth_method 0 2024-09-17T15:27:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf099ea640 10 monclient: get_auth_request con 0x7faef0041aa0 auth_method 0 2024-09-17T15:27:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf091e9640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faef003bbb0 0x7faef003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7faf091e9640 10 monclient: get_auth_request con 0x7faef003bbb0 auth_method 0 2024-09-17T15:27:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf37fe640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7f1bd4001660 0x7f1bd4003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.068+0000 7f1bf37fe640 10 monclient: get_auth_request con 0x7f1bd4001660 auth_method 0 2024-09-17T15:27:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.072+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be003bbb0 0x7f1be003e070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007fd0 tx=0x7f1be8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.072+0000 7faf091e9640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faef003bbb0 0x7faef003e070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faef4005710 tx=0x7faef4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.072+0000 7faf099ea640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7faef0041aa0 0x7faef0043ee0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.076+0000 7f1bf37fe640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7f1bd4001660 0x7f1bd4003b20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.076+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== osd.0 v2:172.21.15.86:6800/3461875681 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faef800af30 con 0x7faef0041aa0 2024-09-17T15:27:31.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.076+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== osd.3 v2:172.21.15.203:6808/2535065757 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1bd4006b10 con 0x7f1bd4001660 2024-09-17T15:27:31.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 --> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faf0409cd60 con 0x7faef0041aa0 2024-09-17T15:27:31.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf01ffb640 1 -- 172.21.15.86:0/353649778 <== osd.0 v2:172.21.15.86:6800/3461875681 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faf0409cd60 con 0x7faef0041aa0 2024-09-17T15:27:31.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7faef0041aa0 msgr2=0x7faef0043ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7faef0041aa0 0x7faef0043ee0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faef003bbb0 msgr2=0x7faef003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faef003bbb0 0x7faef003e070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faef4005710 tx=0x7faef4009040 comp rx=0 tx=0).stop 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 10 monclient: shutdown 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 msgr2=0x7faf040a30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 --2- 172.21.15.86:0/353649778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf0413d2f0 0x7faf040a30d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801ab90 comp rx=0 tx=0).stop 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf099ea640 1 -- 172.21.15.86:0/353649778 reap_dead start 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 shutdown_connections 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 >> 172.21.15.86:0/353649778 conn(0x7faf040aaae0 msgr2=0x7faf0413c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 shutdown_connections 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.088+0000 7faf0b44d640 1 -- 172.21.15.86:0/353649778 wait complete. 2024-09-17T15:27:31.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.100+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 --> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1bd4005ba0 con 0x7f1bd4001660 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.100+0000 7f37b8cd1640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.100+0000 7f37b8cd1640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.100+0000 7f37b8cd1640 10 monclient: monmap: 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.107934+0000 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.107934+0000 2024-09-17T15:27:31.123 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bdb7fe640 1 -- 172.21.15.86:0/1745232521 <== osd.3 v2:172.21.15.203:6808/2535065757 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1bd4005ba0 con 0x7f1bd4001660 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7f1bd4001660 msgr2=0x7f1bd4003b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7f1bd4001660 0x7f1bd4003b20 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be003bbb0 msgr2=0x7f1be003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be003bbb0 0x7f1be003e070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007fd0 tx=0x7f1be8009040 comp rx=0 tx=0).stop 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 10 monclient: shutdown 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 msgr2=0x7f1bf41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 --2- 172.21.15.86:0/1745232521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf409cd60 0x7f1bf41392d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1be400c450 tx=0x7f1be400c910 comp rx=0 tx=0).stop 2024-09-17T15:27:31.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf37fe640 1 -- 172.21.15.86:0/1745232521 reap_dead start 2024-09-17T15:27:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 Processor -- start 2024-09-17T15:27:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 shutdown_connections 2024-09-17T15:27:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 >> 172.21.15.86:0/1745232521 conn(0x7f1bf40aaae0 msgr2=0x7f1bf40a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 -- start start 2024-09-17T15:27:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 shutdown_connections 2024-09-17T15:27:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f1bf95ad640 1 -- 172.21.15.86:0/1745232521 wait complete. 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient: init 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37b41381a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37b406fb70 0x7f37b40a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f37b41381a0 0x7f37b4138580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37514/0 (socket says 172.21.15.86:37514) 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-a con 0x7f37b406fb70 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37b406fb70 0x7f37b40a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: monmap: 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.109050+0000 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.109050+0000 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.129 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37b413d390 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37b406fb70 0x7f37b40a3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43340/0 (socket says 172.21.15.86:43340) 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 -- 172.21.15.86:0/313767715 learned_addr learned my addr 172.21.15.86:0/313767715 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b406edd0 con 0x7f37b41381a0 2024-09-17T15:27:31.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b406f0b0 con 0x7f37b413d390 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f37b40a4ab0 con 0x7f37b406fb70 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b8cd1640 10 monclient(hunting): authenticate will time out at 984.074137s 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): get_auth_request con 0x7f37b406fb70 auth_method 0 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 Processor -- start 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 -- start start 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: init 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 -- 172.21.15.86:0/313767715 >> v1:172.21.15.203:6789/0 conn(0x7f37b413d390 legacy=0x7f37b413f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 -- 172.21.15.86:0/313767715 >> v1:172.21.15.86:6789/0 conn(0x7f37b41381a0 legacy=0x7f37b4138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f37b1d9b640 1 -- 172.21.15.86:0/313767715 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b4138c90 con 0x7f37b406fb70 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 10 monclient: monmap: 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.110913+0000 2024-09-17T15:27:31.133 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.110913+0000 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d74134d90 0x7f4d74135170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-c con 0x7f4d74134d90 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 0x7f4d74138080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-a con 0x7f4d74137c50 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d741385c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d7409ffc0 con 0x7f4d741385c0 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4d7409ce10 con 0x7f4d74137c50 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4d740a1f60 con 0x7f4d74134d90 2024-09-17T15:27:31.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7f4d7a4d7640 10 monclient(hunting): authenticate will time out at 984.075375s 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 1 Processor -- start 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 1 -- start start 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.104+0000 7fce5163a640 10 monclient: init 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-b con 0x7fce4c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fce4c0a54c0 0x7fce4c0a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-c con 0x7fce4c0a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33170/0 (socket says 172.21.15.86:33170) 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce4c137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 -- 172.21.15.86:0/4276691236 learned_addr learned my addr 172.21.15.86:0/4276691236 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4c0a6280 con 0x7fce4c137ea0 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fce4c09c0a0 con 0x7fce4c0a54c0 2024-09-17T15:27:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d74134d90 0x7f4d74135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c09c380 con 0x7fce4c0a50e0 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d74134d90 0x7f4d74135170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:46376/0 (socket says 172.21.15.86:46376) 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 -- 172.21.15.86:0/3446217735 learned_addr learned my addr 172.21.15.86:0/3446217735 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): get_auth_request con 0x7f4d74134d90 auth_method 0 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce5163a640 10 monclient(hunting): authenticate will time out at 984.076978s 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): get_auth_request con 0x7fce4c0a50e0 auth_method 0 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 -- 172.21.15.86:0/4276691236 >> v2:172.21.15.203:3300/0 conn(0x7fce4c0a54c0 msgr2=0x7fce4c0a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 --2- 172.21.15.86:0/4276691236 >> v2:172.21.15.203:3300/0 conn(0x7fce4c0a54c0 0x7fce4c0a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 -- 172.21.15.86:0/4276691236 >> v1:172.21.15.86:6789/0 conn(0x7fce4c137ea0 legacy=0x7fce4c138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 monclient: found mon.noname-b 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7fce4affd640 1 -- 172.21.15.86:0/4276691236 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c09c660 con 0x7fce4c0a50e0 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 1 --2- 172.21.15.86:0/3446217735 >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 0x7f4d74138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): get_auth_request con 0x7f4d74137c50 auth_method 0 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: monmap: 2024-09-17T15:27:31.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.113913+0000 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.113913+0000 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient(hunting): handle_auth_done global_id 4228 payload 274 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 -- 172.21.15.86:0/3446217735 >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 msgr2=0x7f4d74138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 --2- 172.21.15.86:0/3446217735 >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 0x7f4d74138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 -- 172.21.15.86:0/3446217735 >> v1:172.21.15.86:6789/0 conn(0x7f4d741385c0 legacy=0x7f4d7413ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f4d73fff640 1 -- 172.21.15.86:0/3446217735 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d740a2240 con 0x7f4d74134d90 2024-09-17T15:27:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 Processor -- start 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 -- start start 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: init 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c500a2e70 0x7f7c50073d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-a con 0x7f7c500a2e70 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c500a3250 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c5013b080 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c500751c0 con 0x7f7c500a3250 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c500a49f0 con 0x7f7c5013b080 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7c500735b0 con 0x7f7c500a2e70 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.108+0000 7f7c54e51640 10 monclient(hunting): authenticate will time out at 984.079681s 2024-09-17T15:27:31.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b1d9b640 1 -- 172.21.15.86:0/313767715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 msgr2=0x7f37b40a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b1d9b640 1 --2- 172.21.15.86:0/313767715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b40a3da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ee42b33b0186d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4ed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7c5013b080 0x7f7c5013d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:58114/0 (socket says 172.21.15.86:58114) 2024-09-17T15:27:31.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4ed9d640 1 -- 172.21.15.86:0/470910352 learned_addr learned my addr 172.21.15.86:0/470910352 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 1 -- 172.21.15.86:0/313767715 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a80b0070 con 0x7f37b406fb70 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: dump: 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 1 -- 172.21.15.86:0/313767715 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a80ab600 con 0x7f37b406fb70 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 1 -- 172.21.15.86:0/313767715 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a80ab9d0 con 0x7f37b406fb70 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b159a640 10 monclient: dump: 2024-09-17T15:27:31.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 5 monclient: authenticate success, global_id 4209 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 4 set_mon_vals no callback set 2024-09-17T15:27:31.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 1 --2- 172.21.15.86:0/470910352 >> v2:172.21.15.86:3300/0 conn(0x7f7c500a2e70 0x7f7c50073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): get_auth_request con 0x7f7c500a2e70 auth_method 0 2024-09-17T15:27:31.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f4d737fe640 1 --2- 172.21.15.86:0/3446217735 >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 0x7f4d74138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: shutdown 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 msgr2=0x7f37b40a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 --2- 172.21.15.86:0/313767715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b40a3da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37a8009870 tx=0x7f37a801c420 comp rx=0 tx=0).stop 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 shutdown_connections 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 --2- 172.21.15.86:0/313767715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b40a3da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-17T15:27:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 >> 172.21.15.86:0/313767715 conn(0x7f37b40a8390 msgr2=0x7f37b40a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 1 -- 172.21.15.86:0/470910352 >> v1:172.21.15.203:6789/0 conn(0x7f7c5013b080 legacy=0x7f7c5013d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 1 -- 172.21.15.86:0/470910352 >> v1:172.21.15.86:6789/0 conn(0x7f7c500a3250 legacy=0x7f7c500a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 shutdown_connections 2024-09-17T15:27:31.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- 172.21.15.86:0/313767715 wait complete. 2024-09-17T15:27:31.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f7c4e59c640 1 -- 172.21.15.86:0/470910352 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c500723f0 con 0x7f7c500a2e70 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce4affd640 1 -- 172.21.15.86:0/4276691236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 msgr2=0x7fce4c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce4affd640 1 --2- 172.21.15.86:0/4276691236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009870 tx=0x7fce3c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c94bdb78813f794d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: monmap: 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.157 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 Processor -- start 2024-09-17T15:27:31.158 INFO:teuthology.orchestra.run.smithi086.stdout:47244640259 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 1 -- 172.21.15.86:0/4276691236 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0b0070 con 0x7fce4c0a50e0 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: dump: 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 1 -- 172.21.15.86:0/4276691236 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c0a4e50 con 0x7fce4c0a50e0 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 1 -- 172.21.15.86:0/4276691236 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0ab820 con 0x7fce4c0a50e0 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 5 monclient: authenticate success, global_id 4214 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 4 set_mon_vals no callback set 2024-09-17T15:27:31.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T15:27:31.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce49ffb640 10 monclient: dump: 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.162 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- start start 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: init 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b4139950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-a con 0x7f37b406fb70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37b41381a0 0x7f37b40a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-c con 0x7f37b41381a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37b413d390 0x7f37b40a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): picked mon.noname-b con 0x7f37b413d390 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f37b4143a50 con 0x7f37b406fb70 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37b41438d0 con 0x7f37b413d390 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f37b4143bd0 con 0x7f37b41381a0 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b8cd1640 10 monclient(hunting): authenticate will time out at 984.083577s 2024-09-17T15:27:31.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b4139950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b4139950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43380/0 (socket says 172.21.15.86:43380) 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 1 -- 172.21.15.86:0/3455188984 learned_addr learned my addr 172.21.15.86:0/3455188984 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): get_auth_request con 0x7f37b406fb70 auth_method 0 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b2d9d640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37b413d390 0x7f37b40a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 10 monclient: shutdown 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 msgr2=0x7fce4c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7fce5163a640 1 --2- 172.21.15.86:0/4276691236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009870 tx=0x7fce3c01c4b0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b2d9d640 10 monclient(hunting): get_auth_request con 0x7f37b413d390 auth_method 0 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.112+0000 7f37b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37b41381a0 msgr2=0x7f37b40a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37b41381a0 0x7f37b40a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37b413d390 msgr2=0x7f37b40a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37b413d390 0x7f37b40a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b40a1cb0 con 0x7f37b406fb70 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 shutdown_connections 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 --2- 172.21.15.86:0/4276691236 >> v2:172.21.15.203:3300/0 conn(0x7fce4c0a54c0 0x7fce4c0a58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 --2- 172.21.15.86:0/4276691236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a50e0 0x7fce4c09cd60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 >> 172.21.15.86:0/4276691236 conn(0x7fce4c0aaae0 msgr2=0x7fce4c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d73fff640 1 -- 172.21.15.86:0/3446217735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d74134d90 msgr2=0x7f4d74135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d73fff640 1 --2- 172.21.15.86:0/3446217735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d74134d90 0x7f4d74135170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d7406eb50 tx=0x7f4d6401c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e8c29fb8e028505 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 shutdown_connections 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- 172.21.15.86:0/4276691236 wait complete. 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient: monmap: 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 1 -- 172.21.15.86:0/3446217735 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d640b0070 con 0x7f4d74134d90 2024-09-17T15:27:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: dump: 2024-09-17T15:27:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 1 -- 172.21.15.86:0/3446217735 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d640ab600 con 0x7f4d74134d90 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 1 -- 172.21.15.86:0/3446217735 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d640ab9b0 con 0x7f4d74134d90 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 5 monclient: authenticate success, global_id 4228 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 4 set_mon_vals no callback set 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d72ffd640 10 monclient: dump: 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 Processor -- start 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- start start 2024-09-17T15:27:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient: init 2024-09-17T15:27:31.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: shutdown 2024-09-17T15:27:31.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/3446217735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d74134d90 msgr2=0x7f4d74135170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/3446217735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d74134d90 0x7f4d74135170 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4d7406eb50 tx=0x7f4d6401c490 comp rx=0 tx=0).stop 2024-09-17T15:27:31.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce4c0a50e0 0x7fce4c1d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-a con 0x7fce4c0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-c con 0x7fce4c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce4c137ea0 0x7fce4c13d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): picked mon.noname-b con 0x7fce4c137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c142cb0 con 0x7fce4c0a50e0 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c142b30 con 0x7fce4c137ea0 2024-09-17T15:27:31.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c142e30 con 0x7fce4c0a54c0 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce5163a640 10 monclient(hunting): authenticate will time out at 984.086042s 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/3446217735 shutdown_connections 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/3446217735 >> v2:172.21.15.86:3300/0 conn(0x7f4d74137c50 0x7f4d74138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/3446217735 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d74134d90 0x7f4d74135170 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/3446217735 >> 172.21.15.86:0/3446217735 conn(0x7f4d740a2d70 msgr2=0x7f4d740a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/3446217735 shutdown_connections 2024-09-17T15:27:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/3446217735 wait complete. 2024-09-17T15:27:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33200/0 (socket says 172.21.15.86:33200) 2024-09-17T15:27:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 -- 172.21.15.86:0/2661142464 learned_addr learned my addr 172.21.15.86:0/2661142464 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): get_auth_request con 0x7fce4c0a54c0 auth_method 0 2024-09-17T15:27:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce4c137ea0 0x7fce4c13d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): get_auth_request con 0x7fce4c137ea0 auth_method 0 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-17T15:27:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce4c137ea0 msgr2=0x7fce4c13d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce4c137ea0 0x7fce4c13d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce4c0a50e0 msgr2=0x7fce4c1d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce4c0a50e0 0x7fce4c1d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7fce4a7fc640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c0a18d0 con 0x7fce4c0a54c0 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: monmap: 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 Processor -- start 2024-09-17T15:27:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4e59c640 1 -- 172.21.15.86:0/470910352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c500a2e70 msgr2=0x7f7c50073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4e59c640 1 --2- 172.21.15.86:0/470910352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c500a2e70 0x7f7c50073d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097c0 tx=0x7f7c4001c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d33bd1d8abc7b49 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 msgr2=0x7f37b4139950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f37b259c640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b4139950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37a400c460 tx=0x7f37a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- start start 2024-09-17T15:27:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: init 2024-09-17T15:27:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 1 -- 172.21.15.86:0/470910352 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400b1070 con 0x7f7c500a2e70 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: dump: 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 1 -- 172.21.15.86:0/470910352 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c400ac640 con 0x7f7c500a2e70 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c54e51640 5 monclient: authenticate success, global_id 4215 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 1 -- 172.21.15.86:0/470910352 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400aca10 con 0x7f7c500a2e70 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c4d59a640 10 monclient: dump: 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c54e51640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f7c54e51640 4 set_mon_vals no callback set 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d74134d90 0x7f4d74133560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-c con 0x7f4d74134d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 0x7f4d74133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-a con 0x7f4d74137c50 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d741385c0 0x7f4d7413da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): picked mon.noname-b con 0x7f4d741385c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d74135830 con 0x7f4d74137c50 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d741356b0 con 0x7f4d741385c0 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4d7409d090 con 0x7f4d74134d90 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.116+0000 7f4d7a4d7640 10 monclient(hunting): authenticate will time out at 984.087981s 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4017070 con 0x7f37b406fb70 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: dump: 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a4005820 con 0x7f37b406fb70 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a4005c10 con 0x7f37b406fb70 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: dump: 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b0d99640 4 set_mon_vals no callback set 2024-09-17T15:27:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 0x7f4d74133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 0x7f4d74133aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43404/0 (socket says 172.21.15.86:43404) 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 -- 172.21.15.86:0/1497341772 learned_addr learned my addr 172.21.15.86:0/1497341772 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): get_auth_request con 0x7f4d74137c50 auth_method 0 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 5 monclient: authenticate success, global_id 4218 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 10 monclient: _renew_subs 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b409f580 con 0x7f37b406fb70 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 10 monclient: _renew_subs 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b413a5b0 con 0x7f37b406fb70 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: shutdown 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- 172.21.15.86:0/470910352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c500a2e70 msgr2=0x7f7c50073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 --2- 172.21.15.86:0/470910352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c500a2e70 0x7f7c50073d40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097c0 tx=0x7f7c4001c120 comp rx=0 tx=0).stop 2024-09-17T15:27:31.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2024-09-17T15:27:31.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d74134d90 msgr2=0x7f4d74133560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d74134d90 0x7f4d74133560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d741385c0 msgr2=0x7f4d7413da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d741385c0 0x7f4d7413da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d7411e140 con 0x7f4d74137c50 2024-09-17T15:27:31.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce4a7fc640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 msgr2=0x7fce4c136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce4a7fc640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce4000b580 tx=0x7fce4000ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- 172.21.15.86:0/470910352 shutdown_connections 2024-09-17T15:27:31.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 --2- 172.21.15.86:0/470910352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c500a2e70 0x7f7c50073d40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- 172.21.15.86:0/470910352 >> 172.21.15.86:0/470910352 conn(0x7f7c5009bc40 msgr2=0x7f7c500a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce4a7fc640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce4000b580 tx=0x7fce4000ba40 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fce40014070 mon_map magic: 0 2024-09-17T15:27:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- 172.21.15.86:0/470910352 shutdown_connections 2024-09-17T15:27:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40014070 con 0x7fce4c0a54c0 2024-09-17T15:27:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- 172.21.15.86:0/470910352 wait complete. 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: dump: 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce40010aa0 con 0x7fce4c0a54c0 2024-09-17T15:27:31.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 5 monclient: authenticate success, global_id 4217 2024-09-17T15:27:31.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce497fa640 4 set_mon_vals no callback set 2024-09-17T15:27:31.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37b4138580 con 0x7f37b406fb70 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f37a401a070 con 0x7f37b406fb70 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f379403bbb0 0x7f379403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f37a404f220 con 0x7f37b406fb70 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7f3794041a30 0x7f3794043e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37a4013e50 con 0x7f3794041a30 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f37a4056050 con 0x7f37b406fb70 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37a2ffd640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 10 monclient: _renew_subs 2024-09-17T15:27:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c121380 con 0x7fce4c0a54c0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b2d9d640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7f3794041a30 0x7f3794043e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b1d9b640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f379403bbb0 0x7f379403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b2d9d640 10 monclient: get_auth_request con 0x7f3794041a30 auth_method 0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 10 monclient: _renew_subs 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c13f720 con 0x7fce4c0a54c0 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f37b1d9b640 10 monclient: get_auth_request con 0x7f379403bbb0 auth_method 0 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: monmap: 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.202 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 Processor -- start 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fce4c0a4590 con 0x7fce4c0a54c0 2024-09-17T15:27:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 msgr2=0x7f4d74133aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d737fe640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 0x7f4d74133aa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800ebe0 tx=0x7f4d6800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68017070 con 0x7f4d74137c50 2024-09-17T15:27:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: dump: 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d6800ce40 con 0x7f4d74137c50 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d7a4d7640 5 monclient: authenticate success, global_id 4221 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d680136b0 con 0x7f4d74137c50 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d70ff9640 10 monclient: dump: 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f4d727fc640 4 set_mon_vals no callback set 2024-09-17T15:27:31.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- start start 2024-09-17T15:27:31.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: init 2024-09-17T15:27:31.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:31.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 0x7f7c501dcd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-b con 0x7f7c500a2e70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 0x7f7c501dd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-c con 0x7f7c500a3250 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c5013b080 0x7f7c501dd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): picked mon.noname-a con 0x7f7c5013b080 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7c501419d0 con 0x7f7c5013b080 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c500725c0 con 0x7f7c500a2e70 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7c50141b50 con 0x7f7c500a3250 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c54e51640 10 monclient(hunting): authenticate will time out at 984.091686s 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 0x7f7c501dd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.120+0000 7f7c4e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 0x7f7c501dcd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 0x7f7c501dd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33220/0 (socket says 172.21.15.86:33220) 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 0x7f7c501dcd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:46404/0 (socket says 172.21.15.86:46404) 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 1 -- 172.21.15.86:0/1009123828 learned_addr learned my addr 172.21.15.86:0/1009123828 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): get_auth_request con 0x7f7c500a3250 auth_method 0 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): get_auth_request con 0x7f7c500a2e70 auth_method 0 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 10 monclient: _renew_subs 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d7411e420 con 0x7f4d74137c50 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 10 monclient: _renew_subs 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d74140080 con 0x7f4d74137c50 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 msgr2=0x7f7c501dd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 0x7f7c501dd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c5013b080 msgr2=0x7f7c501dd7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4000f3c0 con 0x7fce4c0a54c0 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 10 monclient: dump: 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.213 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fce40017070 con 0x7fce4c0a54c0 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fce3803bb90 0x7fce3803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7fce40050210 con 0x7fce4c0a54c0 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7fce38041990 0x7fce38043dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce380444a0 con 0x7fce38041990 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c5013b080 0x7f7c501dd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 monclient: found mon.noname-b 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5006e110 con 0x7f7c500a2e70 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4dd9b640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c500a3250 0x7f7c501dd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce4affd640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fce3803bb90 0x7fce3803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce4affd640 10 monclient: get_auth_request con 0x7fce3803bb90 auth_method 0 2024-09-17T15:27:31.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f37b1d9b640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f379403bbb0 0x7f379403e070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f37a80ab000 tx=0x7f37a8004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d74135170 con 0x7f4d74137c50 2024-09-17T15:27:31.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce4b7fe640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7fce38041990 0x7fce38043dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce4b7fe640 10 monclient: get_auth_request con 0x7fce38041990 auth_method 0 2024-09-17T15:27:31.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4d6801a070 con 0x7f4d74137c50 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d6003bbe0 0x7f4d6003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f4d6804fe40 con 0x7f4d74137c50 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f4d60041ad0 0x7f4d60043f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d68009400 con 0x7f4d60041ad0 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f4d68052410 con 0x7f4d74137c50 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d70ff9640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d78a74640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f4d60041ad0 0x7f4d60043f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f4d78a74640 10 monclient: get_auth_request con 0x7f4d60041ad0 auth_method 0 2024-09-17T15:27:31.219 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T15:27:31.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f37b2d9d640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7f3794041a30 0x7f3794043e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 msgr2=0x7f7c501dcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c4e59c640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 0x7f7c501dcd20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c400ae000 tx=0x7f7c40018f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:31.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== osd.1 v2:172.21.15.86:6808/2659992775 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37a4013e50 con 0x7f3794041a30 2024-09-17T15:27:31.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.124+0000 7fce4affd640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fce3803bb90 0x7fce3803e050 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0ab000 tx=0x7fce3c004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f4d73fff640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d6003bbe0 0x7f4d6003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f4d73fff640 10 monclient: get_auth_request con 0x7f4d6003bbe0 auth_method 0 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400ba020 con 0x7f7c500a2e70 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: dump: 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.222 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c400b6ae0 con 0x7f7c500a2e70 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 5 monclient: authenticate success, global_id 4237 2024-09-17T15:27:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 10 monclient: _renew_subs 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c500a18c0 con 0x7f7c500a2e70 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400b1040 con 0x7f7c500a2e70 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: dump: 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.224 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c4cd99640 4 set_mon_vals no callback set 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 10 monclient: _renew_subs 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c500a00c0 con 0x7f7c500a2e70 2024-09-17T15:27:31.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7fce4b7fe640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7fce38041990 0x7fce38043dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c50073d40 con 0x7f7c500a2e70 2024-09-17T15:27:31.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== osd.2 v2:172.21.15.86:6816/3359134885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fce380444a0 con 0x7fce38041990 2024-09-17T15:27:31.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7c400bd020 con 0x7f7c500a2e70 2024-09-17T15:27:31.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f4d73fff640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d6003bbe0 0x7f4d6003e0a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d740a2ae0 tx=0x7f4d64004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c3003bbb0 0x7f7c3003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f7c400ea460 con 0x7f7c500a2e70 2024-09-17T15:27:31.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7f7c30041aa0 0x7f7c30043ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c400b3e50 con 0x7f7c30041aa0 2024-09-17T15:27:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f7c400ea740 con 0x7f7c500a2e70 2024-09-17T15:27:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c3effd640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-17T15:27:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f4d78a74640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f4d60041ad0 0x7f4d60043f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c4ed9d640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7f7c30041aa0 0x7f7c30043ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.128+0000 7f7c4ed9d640 10 monclient: get_auth_request con 0x7f7c30041aa0 auth_method 0 2024-09-17T15:27:31.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.132+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== osd.4 v2:172.21.15.203:6816/4139704649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d68009400 con 0x7f4d60041ad0 2024-09-17T15:27:31.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.136+0000 7f7c4ed9d640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7f7c30041aa0 0x7f7c30043ee0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.136+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== osd.5 v2:172.21.15.203:6800/3502598683 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c400b3e50 con 0x7f7c30041aa0 2024-09-17T15:27:31.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 --> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37b4139e90 con 0x7f3794041a30 2024-09-17T15:27:31.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37a2ffd640 1 -- 172.21.15.86:0/3455188984 <== osd.1 v2:172.21.15.86:6808/2659992775 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37b4139e90 con 0x7f3794041a30 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7f3794041a30 msgr2=0x7f3794043e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7f3794041a30 0x7f3794043e70 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f379403bbb0 msgr2=0x7f379403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f379403bbb0 0x7f379403e070 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f37a80ab000 tx=0x7f37a8004740 comp rx=0 tx=0).stop 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 10 monclient: shutdown 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 msgr2=0x7f37b4139950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 --2- 172.21.15.86:0/3455188984 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37b406fb70 0x7f37b4139950 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37a400c460 tx=0x7f37a400c920 comp rx=0 tx=0).stop 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b2d9d640 1 -- 172.21.15.86:0/3455188984 reap_dead start 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 shutdown_connections 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 >> 172.21.15.86:0/3455188984 conn(0x7f37b40a8390 msgr2=0x7f37b413f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 shutdown_connections 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.140+0000 7f37b8cd1640 1 -- 172.21.15.86:0/3455188984 wait complete. 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c5009f250 con 0x7f7c30041aa0 2024-09-17T15:27:31.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c3effd640 1 -- 172.21.15.86:0/1009123828 <== osd.5 v2:172.21.15.203:6800/3502598683 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7c5009f250 con 0x7f7c30041aa0 2024-09-17T15:27:31.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7f7c30041aa0 msgr2=0x7f7c30043ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7f7c30041aa0 0x7f7c30043ee0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c3003bbb0 msgr2=0x7f7c3003e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c3003bbb0 0x7f7c3003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 10 monclient: shutdown 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 msgr2=0x7f7c501dcd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 --2- 172.21.15.86:0/1009123828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c500a2e70 0x7f7c501dcd20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c400ae000 tx=0x7f7c40018f20 comp rx=0 tx=0).stop 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c4ed9d640 1 -- 172.21.15.86:0/1009123828 reap_dead start 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 shutdown_connections 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 >> 172.21.15.86:0/1009123828 conn(0x7f7c5009bc40 msgr2=0x7f7c5006e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 shutdown_connections 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.152+0000 7f7c54e51640 1 -- 172.21.15.86:0/1009123828 wait complete. 2024-09-17T15:27:31.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 --> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fce4c0a7940 con 0x7fce38041990 2024-09-17T15:27:31.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== osd.2 v2:172.21.15.86:6816/3359134885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fce4c0a7940 con 0x7fce38041990 2024-09-17T15:27:31.235 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7fce38041990 msgr2=0x7fce38043dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7fce38041990 0x7fce38043dd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fce3803bb90 msgr2=0x7fce3803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fce3803bb90 0x7fce3803e050 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0ab000 tx=0x7fce3c004740 comp rx=0 tx=0).stop 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 10 monclient: shutdown 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 20 monclient: shutdown canceling and discarding version request 1 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 msgr2=0x7fce4c136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 --2- 172.21.15.86:0/2661142464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce4c0a54c0 0x7fce4c136fb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fce4000b580 tx=0x7fce4000ba40 comp rx=0 tx=0).stop 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce4b7fe640 1 -- 172.21.15.86:0/2661142464 reap_dead start 2024-09-17T15:27:31.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce337fe640 1 -- 172.21.15.86:0/2661142464 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 0 0 0) 0x7fce400504f0 con 0x7fce4c0a54c0 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce337fe640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=17) 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 shutdown_connections 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 >> 172.21.15.86:0/2661142464 conn(0x7fce4c0aaae0 msgr2=0x7fce4c0a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 shutdown_connections 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.156+0000 7fce5163a640 1 -- 172.21.15.86:0/2661142464 wait complete. 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 --> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d74133fe0 con 0x7f4d60041ad0 2024-09-17T15:27:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d70ff9640 1 -- 172.21.15.86:0/1497341772 <== osd.4 v2:172.21.15.203:6816/4139704649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4d74133fe0 con 0x7f4d60041ad0 2024-09-17T15:27:31.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f4d60041ad0 msgr2=0x7f4d60043f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f4d60041ad0 0x7f4d60043f10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d6003bbe0 msgr2=0x7f4d6003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d6003bbe0 0x7f4d6003e0a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d740a2ae0 tx=0x7f4d64004740 comp rx=0 tx=0).stop 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 10 monclient: shutdown 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 msgr2=0x7f4d74133aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 --2- 172.21.15.86:0/1497341772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d74137c50 0x7f4d74133aa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4d6800ebe0 tx=0x7f4d6800c520 comp rx=0 tx=0).stop 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d78a74640 1 -- 172.21.15.86:0/1497341772 reap_dead start 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 shutdown_connections 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 >> 172.21.15.86:0/1497341772 conn(0x7f4d740a2d70 msgr2=0x7f4d7409e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 shutdown_connections 2024-09-17T15:27:31.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.160+0000 7f4d7a4d7640 1 -- 172.21.15.86:0/1497341772 wait complete. 2024-09-17T15:27:31.241 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T15:27:31.242 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T15:27:31.242 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T15:27:31.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7fd1db542640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7f77aef1b640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7f77aef1b640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7f77aef1b640 10 monclient: monmap: 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.375372+0000 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.375372+0000 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7fd1db542640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7fd1db542640 10 monclient: monmap: 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.375372+0000 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.375372+0000 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.368+0000 7f77aef1b640 1 Processor -- start 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 1 -- start start 2024-09-17T15:27:31.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient: init 2024-09-17T15:27:31.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 Processor -- start 2024-09-17T15:27:31.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 -- start start 2024-09-17T15:27:31.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient: init 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77a80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f77a80a5bd0 0x7f77a809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-c con 0x7f77a80a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77a813d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77a80a6280 con 0x7f77a80a50e0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77a809c0a0 con 0x7f77a813d2f0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f77a809d2a0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77aef1b640 10 monclient(hunting): authenticate will time out at 984.340749s 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1d40a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1d40a5bd0 0x7fd1d409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d40a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd1d413d2f0 0x7fd1d413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d413d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d40a6280 con 0x7fd1d40a50e0 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d409d2a0 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d409c0a0 con 0x7fd1d413d2f0 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1db542640 10 monclient(hunting): authenticate will time out at 984.340748s 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77accb7640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f77a80a50e0 0x7f77a80a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37542/0 (socket says 172.21.15.86:37542) 2024-09-17T15:27:31.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77accb7640 1 -- 172.21.15.86:0/617945485 learned_addr learned my addr 172.21.15.86:0/617945485 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1d40a5bd0 0x7fd1d409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1d40a5bd0 0x7fd1d409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43428/0 (socket says 172.21.15.86:43428) 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 -- 172.21.15.86:0/4046388154 learned_addr learned my addr 172.21.15.86:0/4046388154 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): get_auth_request con 0x7fd1d40a5bd0 auth_method 0 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 1 --2- 172.21.15.86:0/4046388154 >> v2:172.21.15.203:3300/0 conn(0x7fd1d413d2f0 0x7fd1d413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 1 --2- 172.21.15.86:0/617945485 >> v2:172.21.15.203:3300/0 conn(0x7f77a80a5bd0 0x7f77a809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): get_auth_request con 0x7fd1d413d2f0 auth_method 0 2024-09-17T15:27:31.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): get_auth_request con 0x7f77a80a5bd0 auth_method 0 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-09-17T15:27:31.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 -- 172.21.15.86:0/4046388154 >> v2:172.21.15.203:3300/0 conn(0x7fd1d413d2f0 msgr2=0x7fd1d413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 --2- 172.21.15.86:0/4046388154 >> v2:172.21.15.203:3300/0 conn(0x7fd1d413d2f0 0x7fd1d413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 -- 172.21.15.86:0/4046388154 >> v1:172.21.15.86:6789/0 conn(0x7fd1d40a50e0 legacy=0x7fd1d40a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7fd1d8add640 1 -- 172.21.15.86:0/4046388154 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d409c380 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 1 -- 172.21.15.86:0/617945485 >> v1:172.21.15.203:6789/0 conn(0x7f77a813d2f0 legacy=0x7f77a813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 1 -- 172.21.15.86:0/617945485 >> v1:172.21.15.86:6789/0 conn(0x7f77a80a50e0 legacy=0x7f77a80a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.372+0000 7f77a7fff640 1 -- 172.21.15.86:0/617945485 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a809c3d0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1d8add640 1 -- 172.21.15.86:0/4046388154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a5bd0 msgr2=0x7fd1d409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1d8add640 1 --2- 172.21.15.86:0/4046388154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d409cd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40a84e0 tx=0x7fd1d001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a308f7e4bbf0f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 1 -- 172.21.15.86:0/4046388154 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00b0070 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a7fff640 1 -- 172.21.15.86:0/617945485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a5bd0 msgr2=0x7f77a809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a7fff640 1 --2- 172.21.15.86:0/617945485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a5bd0 0x7f77a809cd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77940097c0 tx=0x7f779401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7327f824c036d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: dump: 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 1 -- 172.21.15.86:0/4046388154 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d00ab600 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 5 monclient: authenticate success, global_id 4224 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 4 set_mon_vals no callback set 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 1 -- 172.21.15.86:0/4046388154 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d00ab9d0 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1c7fff640 10 monclient: dump: 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.390 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 1 -- 172.21.15.86:0/617945485 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77940b0070 con 0x7f77a80a5bd0 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: dump: 2024-09-17T15:27:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 1 -- 172.21.15.86:0/617945485 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77940ab600 con 0x7f77a80a5bd0 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 5 monclient: authenticate success, global_id 4240 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 1 -- 172.21.15.86:0/617945485 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77940ab9d0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 4 set_mon_vals no callback set 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77a77fe640 10 monclient: dump: 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.393 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 10 monclient: shutdown 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 -- 172.21.15.86:0/4046388154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a5bd0 msgr2=0x7fd1d409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 --2- 172.21.15.86:0/4046388154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d409cd60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40a84e0 tx=0x7fd1d001c3b0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 -- 172.21.15.86:0/4046388154 shutdown_connections 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 --2- 172.21.15.86:0/4046388154 >> v2:172.21.15.203:3300/0 conn(0x7fd1d413d2f0 0x7fd1d413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 --2- 172.21.15.86:0/4046388154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d409cd60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 -- 172.21.15.86:0/4046388154 >> 172.21.15.86:0/4046388154 conn(0x7fd1d40aaae0 msgr2=0x7fd1d40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 -- 172.21.15.86:0/4046388154 shutdown_connections 2024-09-17T15:27:31.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 1 -- 172.21.15.86:0/4046388154 wait complete. 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 10 monclient: shutdown 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 -- 172.21.15.86:0/617945485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a5bd0 msgr2=0x7f77a809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 --2- 172.21.15.86:0/617945485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a5bd0 0x7f77a809cd60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f77940097c0 tx=0x7f779401c360 comp rx=0 tx=0).stop 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 -- 172.21.15.86:0/617945485 shutdown_connections 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 --2- 172.21.15.86:0/617945485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a5bd0 0x7f77a809cd60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 -- 172.21.15.86:0/617945485 >> 172.21.15.86:0/617945485 conn(0x7f77a80aaae0 msgr2=0x7f77a80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 -- 172.21.15.86:0/617945485 shutdown_connections 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7f77aef1b640 1 -- 172.21.15.86:0/617945485 wait complete. 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient: monmap: 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.376+0000 7fd1db542640 10 monclient: monmap: 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.398 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 Processor -- start 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 -- start start 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient: init 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 Processor -- start 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 -- start start 2024-09-17T15:27:31.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient: init 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a50e0 0x7fd1d4138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d40a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d4138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d40a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d4142650 con 0x7fd1d40a50e0 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d40a6ab0 con 0x7fd1d40a5bd0 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d41427d0 con 0x7fd1d413d2f0 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1db542640 10 monclient(hunting): authenticate will time out at 984.348624s 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33222/0 (socket says 172.21.15.86:33222) 2024-09-17T15:27:31.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 -- 172.21.15.86:0/3211037587 learned_addr learned my addr 172.21.15.86:0/3211037587 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): get_auth_request con 0x7fd1d413d2f0 auth_method 0 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d8add640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d4138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d8add640 10 monclient(hunting): get_auth_request con 0x7fd1d40a5bd0 auth_method 0 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d8add640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d8add640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d8add640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 -- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d40a5bd0 msgr2=0x7fd1d4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d4138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 -- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a50e0 msgr2=0x7fd1d4138100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a50e0 0x7fd1d4138100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 -- 172.21.15.86:0/3211037587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d40719f0 con 0x7fd1d413d2f0 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a50e0 0x7f77a81cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-b con 0x7f77a80a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 0x7f77a8138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-a con 0x7f77a80a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 0x7f77a809e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): picked mon.noname-c con 0x7f77a813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77a81439b0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77a8143830 con 0x7f77a80a50e0 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f77a8143b30 con 0x7f77a813d2f0 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77aef1b640 10 monclient(hunting): authenticate will time out at 984.349112s 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 0x7f77a809e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 0x7f77a809e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33230/0 (socket says 172.21.15.86:33230) 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 1 -- 172.21.15.86:0/4270913268 learned_addr learned my addr 172.21.15.86:0/4270913268 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): get_auth_request con 0x7f77a813d2f0 auth_method 0 2024-09-17T15:27:31.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 0x7f77a8138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): get_auth_request con 0x7f77a80a5bd0 auth_method 0 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a50e0 0x7f77a81cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77ad4b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): get_auth_request con 0x7f77a80a50e0 auth_method 0 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77accb7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 -- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 msgr2=0x7f77a809e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 0x7f77a809e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 -- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a50e0 msgr2=0x7f77a81cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a50e0 0x7f77a81cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7f77a7fff640 1 -- 172.21.15.86:0/4270913268 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a80a0e40 con 0x7f77a80a5bd0 2024-09-17T15:27:31.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 -- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 msgr2=0x7fd1d413abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1d9adf640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00efc0 tx=0x7fd1cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.380+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a7fff640 1 -- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 msgr2=0x7f77a8138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a7fff640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 0x7f77a8138200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77a806f8e0 tx=0x7f779401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77940b0050 con 0x7f77a80a5bd0 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: dump: 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.410 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 5 monclient: authenticate success, global_id 4227 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77940bc070 con 0x7f77a80a5bd0 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77940b5c60 con 0x7f77a80a5bd0 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _renew_subs 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a80a18d0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 10 monclient: dump: 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _renew_subs 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a80a0240 con 0x7f77a80a5bd0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a6ffd640 4 set_mon_vals no callback set 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7788005090 con 0x7f77a80a5bd0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f77940ab4c0 con 0x7f77a80a5bd0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 0x7f778c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f77940ea740 con 0x7f77a80a5bd0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77accb7640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 0x7f778c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7f77accb7640 10 monclient: get_auth_request con 0x7f778c03bb60 auth_method 0 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc018020 con 0x7fd1d413d2f0 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: dump: 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1cc014680 con 0x7fd1d413d2f0 2024-09-17T15:27:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc014a70 con 0x7fd1d413d2f0 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c5ffb640 10 monclient: dump: 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 5 monclient: authenticate success, global_id 4223 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _renew_subs 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d40a7cd0 con 0x7fd1d413d2f0 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _renew_subs 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d413c5e0 con 0x7fd1d413d2f0 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1c77fe640 4 set_mon_vals no callback set 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.384+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b4005090 con 0x7fd1d413d2f0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd1cc01b020 con 0x7fd1d413d2f0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7fd1c5ffb640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 0x7fd1b803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7fd1cc051350 con 0x7fd1d413d2f0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7fd1d92de640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 0x7fd1b803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7fd1d92de640 10 monclient: get_auth_request con 0x7fd1b803bbb0 auth_method 0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.388+0000 7f77accb7640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 0x7f778c03e020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f779c007ca0 tx=0x7f779c007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7fd1d92de640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 0x7fd1b803e070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007fd0 tx=0x7fd1c80079a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd1cc022030 con 0x7fd1d413d2f0 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7fd1c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7fd1c5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f77a80a0e40 con 0x7f77a80a5bd0 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.392+0000 7f77a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: monmap: 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.420 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.420148+0000 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.420148+0000 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 Processor -- start 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 -- start start 2024-09-17T15:27:31.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: init 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: monmap: 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.420950+0000 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.420950+0000 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T15:27:31.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 0x7fc7b009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b00a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7b00a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b00a6280 con 0x7fc7b00a54c0 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc7b009d2a0 con 0x7fc7b00a50e0 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b009c0a0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b832f640 10 monclient(hunting): authenticate will time out at 984.385622s 2024-09-17T15:27:31.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 Processor -- start 2024-09-17T15:27:31.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33236/0 (socket says 172.21.15.86:33236) 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 -- 172.21.15.86:0/2353428630 learned_addr learned my addr 172.21.15.86:0/2353428630 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): get_auth_request con 0x7fc7b013d2f0 auth_method 0 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 1 --2- 172.21.15.86:0/2353428630 >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 0x7fc7b009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): get_auth_request con 0x7fc7b00a50e0 auth_method 0 2024-09-17T15:27:31.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-17T15:27:31.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 -- 172.21.15.86:0/2353428630 >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 msgr2=0x7fc7b009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/2353428630 >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 0x7fc7b009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 -- 172.21.15.86:0/2353428630 >> v1:172.21.15.203:6789/0 conn(0x7fc7b00a54c0 legacy=0x7fc7b00a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 monclient: found mon.noname-b 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b68cc640 1 -- 172.21.15.86:0/2353428630 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b009c380 con 0x7fc7b013d2f0 2024-09-17T15:27:31.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fc7b60cb640 1 --2- 172.21.15.86:0/2353428630 >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 0x7fc7b009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 -- start start 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: init 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbe00a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcbe00a5bd0 0x7fcbe009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe00a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbe013d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe00a6280 con 0x7fcbe00a50e0 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe009c0a0 con 0x7fcbe013d2f0 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe009d2a0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbe64e7640 10 monclient(hunting): authenticate will time out at 984.386753s 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcbe00a50e0 0x7fcbe00a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37560/0 (socket says 172.21.15.86:37560) 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdffff640 1 -- 172.21.15.86:0/3794379721 learned_addr learned my addr 172.21.15.86:0/3794379721 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/3794379721 >> v2:172.21.15.86:3300/0 conn(0x7fcbe00a5bd0 0x7fcbe009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbe00a5bd0 auth_method 0 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-17T15:27:31.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3794379721 >> v1:172.21.15.203:6789/0 conn(0x7fcbe013d2f0 legacy=0x7fcbe013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3794379721 >> v1:172.21.15.86:6789/0 conn(0x7fcbe00a50e0 legacy=0x7fcbe00a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.416+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3794379721 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe009c3d0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b68cc640 1 -- 172.21.15.86:0/2353428630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 msgr2=0x7fc7b013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/2353428630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009e50 tx=0x7fc7a801c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5df5cf76f2394460 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 1 -- 172.21.15.86:0/2353428630 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80a5600 con 0x7fc7b013d2f0 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: dump: 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 1 -- 172.21.15.86:0/2353428630 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a80a57a0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 1 -- 172.21.15.86:0/2353428630 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80a5b70 con 0x7fc7b013d2f0 2024-09-17T15:27:31.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b50c9640 10 monclient: dump: 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.440 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b832f640 5 monclient: authenticate success, global_id 4229 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b832f640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.420+0000 7fc7b832f640 4 set_mon_vals no callback set 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: shutdown 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- 172.21.15.86:0/2353428630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 msgr2=0x7fc7b013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- 172.21.15.86:0/2353428630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009e50 tx=0x7fc7a801c860 comp rx=0 tx=0).stop 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- 172.21.15.86:0/2353428630 shutdown_connections 2024-09-17T15:27:31.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- 172.21.15.86:0/2353428630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b013f6e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- 172.21.15.86:0/2353428630 >> v2:172.21.15.203:3300/0 conn(0x7fc7b00a50e0 0x7fc7b009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- 172.21.15.86:0/2353428630 >> 172.21.15.86:0/2353428630 conn(0x7fc7b00aaae0 msgr2=0x7fc7b00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- 172.21.15.86:0/2353428630 shutdown_connections 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- 172.21.15.86:0/2353428630 wait complete. 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3794379721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 msgr2=0x7fcbe009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/3794379721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe009cd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009a20 tx=0x7fcbd401c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a4bb8fa5995c33 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3794379721 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40ab650 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: dump: 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3794379721 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd40b0070 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3794379721 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40aba10 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbdeffd640 10 monclient: dump: 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 5 monclient: authenticate success, global_id 4230 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 4 set_mon_vals no callback set 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: monmap: 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.446 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 Processor -- start 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- start start 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: init 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 10 monclient: shutdown 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 -- 172.21.15.86:0/3794379721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 msgr2=0x7fcbe009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 --2- 172.21.15.86:0/3794379721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe009cd60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009a20 tx=0x7fcbd401c5c0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: monmap: 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.429146+0000 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.429146+0000 2024-09-17T15:27:31.448 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 Processor -- start 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 -- start start 2024-09-17T15:27:31.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: init 2024-09-17T15:27:31.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 -- 172.21.15.86:0/3794379721 shutdown_connections 2024-09-17T15:27:31.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 --2- 172.21.15.86:0/3794379721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe009cd60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 -- 172.21.15.86:0/3794379721 >> 172.21.15.86:0/3794379721 conn(0x7fcbe00aaae0 msgr2=0x7fcbe00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 -- 172.21.15.86:0/3794379721 shutdown_connections 2024-09-17T15:27:31.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:31.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-a con 0x7fc7b00a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b00a54c0 0x7fc7b0138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7b00a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b009f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7b013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0142650 con 0x7fc7b00a50e0 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b00a6ab0 con 0x7fc7b00a54c0 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b01427d0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b832f640 10 monclient(hunting): authenticate will time out at 984.394649s 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43452/0 (socket says 172.21.15.86:43452) 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 1 -- 172.21.15.86:0/3700017687 learned_addr learned my addr 172.21.15.86:0/3700017687 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): get_auth_request con 0x7fc7b00a50e0 auth_method 0 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b009f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): get_auth_request con 0x7fc7b013d2f0 auth_method 0 2024-09-17T15:27:31.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b5409e080 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8b540a3860 0x7f8b540a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-c con 0x7f8b540a3860 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b540a11b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b540a2950 con 0x7f8b5409e080 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b540a1d10 con 0x7f8b540a11b0 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8b540a3c70 con 0x7f8b540a3860 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b58af4640 10 monclient(hunting): authenticate will time out at 984.394786s 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8b540a3860 0x7f8b540a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8b540a3860 0x7f8b540a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:46470/0 (socket says 172.21.15.86:46470) 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 -- 172.21.15.86:0/3674025068 learned_addr learned my addr 172.21.15.86:0/3674025068 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): get_auth_request con 0x7f8b540a3860 auth_method 0 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b58ca640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b00a54c0 0x7fc7b0138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b58ca640 10 monclient(hunting): get_auth_request con 0x7fc7b00a54c0 auth_method 0 2024-09-17T15:27:31.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b58ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b58ca640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b58ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient(hunting): handle_auth_done global_id 4232 payload 824 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 -- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b00a54c0 msgr2=0x7fc7b0138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b00a54c0 0x7fc7b0138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 -- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 msgr2=0x7fc7b01cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b68cc640 1 -- 172.21.15.86:0/3700017687 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b013f7f0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fc7b60cb640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 -- 172.21.15.86:0/3674025068 >> v1:172.21.15.203:6789/0 conn(0x7f8b540a11b0 legacy=0x7f8b540a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 -- 172.21.15.86:0/3674025068 >> v1:172.21.15.86:6789/0 conn(0x7f8b5409e080 legacy=0x7f8b5408c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 monclient: found mon.noname-c 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:31.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7f8b51d9b640 1 -- 172.21.15.86:0/3674025068 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b54148a50 con 0x7f8b540a3860 2024-09-17T15:27:31.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 -- 172.21.15.86:0/3794379721 wait complete. 2024-09-17T15:27:31.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 10 monclient: monmap: 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.424+0000 7fcbe64e7640 1 Processor -- start 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 -- start start 2024-09-17T15:27:31.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient: init 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 0x7fcbe01cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe00a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe00a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013d2f0 0x7fcbe009e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe01439b0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe0143830 con 0x7fcbe00a50e0 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe0143b30 con 0x7fcbe013d2f0 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbe64e7640 10 monclient(hunting): authenticate will time out at 984.396885s 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43466/0 (socket says 172.21.15.86:43466) 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/750952098 learned_addr learned my addr 172.21.15.86:0/750952098 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbe00a5bd0 auth_method 0 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 0x7fcbe01cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 10 monclient(hunting): get_auth_request con 0x7fcbe00a50e0 auth_method 0 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_done global_id 4236 payload 824 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013d2f0 msgr2=0x7fcbe009e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013d2f0 0x7fcbe009e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 msgr2=0x7fcbe01cfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 0x7fcbe01cfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/750952098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe0121c10 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdffff640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 0x7fcbe01cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:27:31.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b68cc640 1 -- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 msgr2=0x7fc7b009f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b68cc640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b009f870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00a6490 tx=0x7fc7a8019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: get_monmap_and_config 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: monmap: 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:31.434172+0000 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:31.434172+0000 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:31.468 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:31.469 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:31.469 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:31.469 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:31.469 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80bc070 con 0x7fc7b013d2f0 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: dump: 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.470 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a80b8850 con 0x7fc7b013d2f0 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80a5600 con 0x7fc7b013d2f0 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 10 monclient: dump: 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.471 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 5 monclient: authenticate success, global_id 4232 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 10 monclient: _renew_subs 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b01228d0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 10 monclient: _renew_subs 2024-09-17T15:27:31.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01d0a10 con 0x7fc7b013d2f0 2024-09-17T15:27:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7b48c8640 4 set_mon_vals no callback set 2024-09-17T15:27:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc7a80b8c60 con 0x7fc7b013d2f0 2024-09-17T15:27:31.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 Processor -- start 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 -- start start 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: init 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faad40a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faad40a5bd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 0x7faad4140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-b con 0x7faad409cd60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad409d210 con 0x7faad40a50e0 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad40a6280 con 0x7faad40a5bd0 2024-09-17T15:27:31.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 msgr2=0x7fcbe0138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40ad040 tx=0x7fcbd401ae00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 0x7fc79403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7fc7a80ea560 con 0x7fc7b013d2f0 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faad409c0a0 con 0x7faad409cd60 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faadb2d7640 10 monclient(hunting): authenticate will time out at 984.399852s 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faad9073640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faad40a50e0 0x7faad40a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37568/0 (socket says 172.21.15.86:37568) 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.428+0000 7faad9073640 1 -- 172.21.15.86:0/2374626306 learned_addr learned my addr 172.21.15.86:0/2374626306 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 1 --2- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 0x7faad4140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): get_auth_request con 0x7faad409cd60 auth_method 0 2024-09-17T15:27:31.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40aa690 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: dump: 2024-09-17T15:27:31.477 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd40aa850 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40aac40 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 10 monclient: dump: 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.479 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 5 monclient: authenticate success, global_id 4236 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 10 monclient: _renew_subs 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 1 -- 172.21.15.86:0/750952098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe00a0af0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbddffb640 4 set_mon_vals no callback set 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 10 monclient: _renew_subs 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbe64e7640 1 -- 172.21.15.86:0/750952098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe00a0db0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b51d9b640 1 -- 172.21.15.86:0/3674025068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a3860 msgr2=0x7f8b540a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b51d9b640 1 --2- 172.21.15.86:0/3674025068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a3860 0x7f8b540a0c70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009870 tx=0x7f8b4401c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67e4a8908a5c5e15 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b832f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b832f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b832f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b013d810 con 0x7fc7b013d2f0 2024-09-17T15:27:31.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbc6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbc6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbc6ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b60cb640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 0x7fc79403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fc7b60cb640 10 monclient: get_auth_request con 0x7fc79403bbb0 auth_method 0 2024-09-17T15:27:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb8005090 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcbd40bd020 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 0x7fcbcc03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7fcbd40ea500 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdffff640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 0x7fcbcc03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 1 -- 172.21.15.86:0/3674025068 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b440b0070 con 0x7f8b540a3860 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: dump: 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 1 -- 172.21.15.86:0/3674025068 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b440ab600 con 0x7f8b540a3860 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 1 -- 172.21.15.86:0/3674025068 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b440ab9d0 con 0x7f8b540a3860 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 1 -- 172.21.15.86:0/2374626306 >> v1:172.21.15.203:6789/0 conn(0x7faad40a5bd0 legacy=0x7faad413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 1 -- 172.21.15.86:0/2374626306 >> v1:172.21.15.86:6789/0 conn(0x7faad40a50e0 legacy=0x7faad40a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 1 monclient: found mon.noname-b 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b5159a640 10 monclient: dump: 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7faad9874640 1 -- 172.21.15.86:0/2374626306 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad409c380 con 0x7faad409cd60 2024-09-17T15:27:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7fcbdffff640 10 monclient: get_auth_request con 0x7fcbcc03bbb0 auth_method 0 2024-09-17T15:27:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 5 monclient: authenticate success, global_id 4255 2024-09-17T15:27:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 4 set_mon_vals no callback set 2024-09-17T15:27:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 10 monclient: shutdown 2024-09-17T15:27:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 -- 172.21.15.86:0/3674025068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a3860 msgr2=0x7f8b540a0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 --2- 172.21.15.86:0/3674025068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a3860 0x7f8b540a0c70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009870 tx=0x7f8b4401c420 comp rx=0 tx=0).stop 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 -- 172.21.15.86:0/3674025068 shutdown_connections 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 --2- 172.21.15.86:0/3674025068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a3860 0x7f8b540a0c70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 -- 172.21.15.86:0/3674025068 >> 172.21.15.86:0/3674025068 conn(0x7f8b540a2d70 msgr2=0x7f8b540a3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 -- 172.21.15.86:0/3674025068 shutdown_connections 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.432+0000 7f8b58af4640 1 -- 172.21.15.86:0/3674025068 wait complete. 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7fc7b60cb640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 0x7fc79403e070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009580 tx=0x7fc7ac009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7faad9874640 1 -- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 msgr2=0x7faad4140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7faad9874640 1 --2- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 0x7faad4140a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faacc009dc0 tx=0x7faacc01c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a27465c285baa58d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7fcbdffff640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 0x7fcbcc03e070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0007fd0 tx=0x7fcbd0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcbd40c4060 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7fcbdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7fcbdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient: monmap: 2024-09-17T15:27:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 Processor -- start 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 -- start start 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient: init 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:31.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-c con 0x7f8b5409e080 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a11b0 0x7f8b541ea680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-b con 0x7f8b540a11b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 0x7f8b54142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): picked mon.noname-a con 0x7f8b540a3860 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8b54154ad0 con 0x7f8b540a3860 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8b5409ce90 con 0x7f8b540a11b0 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8b54154c50 con 0x7f8b5409e080 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b58af4640 10 monclient(hunting): authenticate will time out at 984.405462s 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33278/0 (socket says 172.21.15.86:33278) 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 1 -- 172.21.15.86:0/2510588149 learned_addr learned my addr 172.21.15.86:0/2510588149 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): get_auth_request con 0x7f8b5409e080 auth_method 0 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 0x7f8b54142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): get_auth_request con 0x7f8b540a3860 auth_method 0 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b51d9b640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a11b0 0x7f8b541ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b51d9b640 10 monclient(hunting): get_auth_request con 0x7f8b540a11b0 auth_method 0 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 -- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 msgr2=0x7f8b541ea140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 -- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a11b0 msgr2=0x7f8b541ea680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a11b0 0x7f8b541ea680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b52d9d640 1 -- 172.21.15.86:0/2510588149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b541538b0 con 0x7f8b540a3860 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.436+0000 7f8b5259c640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b52d9d640 1 -- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 msgr2=0x7f8b54142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b52d9d640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 0x7f8b54142720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c7b0 tx=0x7f8b4c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c016020 con 0x7f8b540a3860 2024-09-17T15:27:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: dump: 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 5 monclient: authenticate success, global_id 4239 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _renew_subs 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b54146540 con 0x7f8b540a3860 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _renew_subs 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b541441e0 con 0x7f8b540a3860 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4c00abe0 con 0x7f8b540a3860 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c00ce60 con 0x7f8b540a3860 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b42ffd640 10 monclient: dump: 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b50d99640 4 set_mon_vals no callback set 2024-09-17T15:27:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.440+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b38005090 con 0x7f8b540a3860 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8b4c019020 con 0x7f8b540a3860 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7f8b42ffd640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 0x7f8b34082030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7f8b4c0521d0 con 0x7f8b540a3860 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc7a80c5020 con 0x7fc7b013d2f0 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7fc7a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7fc7a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7f8b5259c640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 0x7f8b34082030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.444+0000 7f8b5259c640 10 monclient: get_auth_request con 0x7f8b3407fb70 auth_method 0 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 1 -- 172.21.15.86:0/2374626306 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc0b2070 con 0x7faad409cd60 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: dump: 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 1 -- 172.21.15.86:0/2374626306 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faacc0a5800 con 0x7faad409cd60 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 1 -- 172.21.15.86:0/2374626306 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc0a5bd0 con 0x7faad409cd60 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 5 monclient: authenticate success, global_id 4235 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 4 set_mon_vals no callback set 2024-09-17T15:27:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faacbfff640 10 monclient: dump: 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: shutdown 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 msgr2=0x7faad4140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 --2- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 0x7faad4140a20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faacc009dc0 tx=0x7faacc01c820 comp rx=0 tx=0).stop 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- 172.21.15.86:0/2374626306 shutdown_connections 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 --2- 172.21.15.86:0/2374626306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad409cd60 0x7faad4140a20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- 172.21.15.86:0/2374626306 >> 172.21.15.86:0/2374626306 conn(0x7faad40aaae0 msgr2=0x7faad40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- 172.21.15.86:0/2374626306 shutdown_connections 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- 172.21.15.86:0/2374626306 wait complete. 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8b4c020020 con 0x7f8b540a3860 2024-09-17T15:27:31.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7f8b42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7f8b42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: build_initial_monmap 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: monmap: 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:31.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 Processor -- start 2024-09-17T15:27:31.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7f8b5259c640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 0x7f8b34082030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b48004470 tx=0x7f8b48003700 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- start start 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: init 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad409cd60 0x7faad41dce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-b con 0x7faad409cd60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40a50e0 0x7faad41dd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-c con 0x7faad40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): picked mon.noname-a con 0x7faad40a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faad4143900 con 0x7faad40a5bd0 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faad4143780 con 0x7faad409cd60 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faad4143a80 con 0x7faad40a50e0 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faadb2d7640 10 monclient(hunting): authenticate will time out at 984.419429s 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43482/0 (socket says 172.21.15.86:43482) 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 1 -- 172.21.15.86:0/1511510093 learned_addr learned my addr 172.21.15.86:0/1511510093 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): get_auth_request con 0x7faad40a5bd0 auth_method 0 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:31.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.448+0000 7faad9874640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 10 monclient(hunting): handle_auth_done global_id 4242 payload 824 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 -- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40a50e0 msgr2=0x7faad41dd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40a50e0 0x7faad41dd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 -- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad409cd60 msgr2=0x7faad41dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad409cd60 0x7faad41dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 10 monclient: _finish_hunting 0 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 monclient: found mon.noname-a 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 -- 172.21.15.86:0/1511510093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad413cb80 con 0x7faad40a5bd0 2024-09-17T15:27:31.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 -- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 msgr2=0x7faad41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faad9874640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faacc01d0c0 tx=0x7faacc0195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc0bd020 con 0x7faad40a5bd0 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 10 monclient: dump: 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faac9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 5 monclient: authenticate success, global_id 4242 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 10 monclient: _renew_subs 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 1 -- 172.21.15.86:0/1511510093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad413bfe0 con 0x7faad40a5bd0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 10 monclient: _renew_subs 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.452+0000 7faadb2d7640 1 -- 172.21.15.86:0/1511510093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad40a18d0 con 0x7faad40a5bd0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faacc0b3040 con 0x7faad40a5bd0 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faabf7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faacb7fe640 4 set_mon_vals no callback set 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faacc0b2070 con 0x7faad40a5bd0 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faabf7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faabf7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab4005090 con 0x7faad40a5bd0 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 10 monclient: dump: 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faacc0c0020 con 0x7faad40a5bd0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 0x7faab803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.456+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4402+0+0 (secure 0 0 0) 0x7faacc0eaf60 con 0x7faad40a5bd0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faad9073640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 0x7faab803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faad9073640 10 monclient: get_auth_request con 0x7faab803bb60 auth_method 0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faacc0c7030 con 0x7faad40a5bd0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faac9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faac9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.460+0000 7faad9073640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 0x7faab803e020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faad0004fa0 tx=0x7faad0007ab0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7b832f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7b832f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7b832f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc7b009cd60 con 0x7fc7b013d2f0 2024-09-17T15:27:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7a6ffd640 1 -- 172.21.15.86:0/3700017687 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc7b013f7f0 con 0x7fc7b013d2f0 2024-09-17T15:27:31.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:31.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.612+0000 7fc7a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.617 INFO:teuthology.orchestra.run.smithi086.stdout:47244640259 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 msgr2=0x7fc79403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 0x7fc79403e070 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009580 tx=0x7fc7ac009190 comp rx=0 tx=0).stop 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 10 monclient: shutdown 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 msgr2=0x7fc7b009f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b009f870 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc7b00a6490 tx=0x7fc7a8019660 comp rx=0 tx=0).stop 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 shutdown_connections 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc79403bbb0 0x7fc79403e070 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7b013d2f0 0x7fc7b009f870 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7b00a54c0 0x7fc7b0138220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 --2- 172.21.15.86:0/3700017687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7b00a50e0 0x7fc7b01cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 >> 172.21.15.86:0/3700017687 conn(0x7fc7b00aaae0 msgr2=0x7fc7b00a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 shutdown_connections 2024-09-17T15:27:31.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.616+0000 7fc7b832f640 1 -- 172.21.15.86:0/3700017687 wait complete. 2024-09-17T15:27:31.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faabf7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:27:31.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faabf7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:27:31.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faabf7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7faab4005380 con 0x7faad40a5bd0 2024-09-17T15:27:31.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faac9ffb640 1 -- 172.21.15.86:0/1511510093 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faacc016560 con 0x7faad40a5bd0 2024-09-17T15:27:31.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faac9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-17T15:27:31.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.628+0000 7faac9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.633 INFO:teuthology.orchestra.run.smithi086.stdout:47244640259 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 msgr2=0x7faab803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 0x7faab803e020 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7faad0004fa0 tx=0x7faad0007ab0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 10 monclient: shutdown 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 msgr2=0x7faad41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faacc01d0c0 tx=0x7faacc0195c0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 shutdown_connections 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faab803bb60 0x7faab803e020 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad40a5bd0 0x7faad41392d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad40a50e0 0x7faad41dd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 --2- 172.21.15.86:0/1511510093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad409cd60 0x7faad41dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 >> 172.21.15.86:0/1511510093 conn(0x7faad40aaae0 msgr2=0x7faad40a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 shutdown_connections 2024-09-17T15:27:31.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.632+0000 7faabf7fe640 1 -- 172.21.15.86:0/1511510093 wait complete. 2024-09-17T15:27:31.639 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-09-17T15:27:31.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2024-09-17T15:27:31.652 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:31.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.688+0000 7fd1db542640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:27:31.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.688+0000 7fd1db542640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.688+0000 7fd1db542640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.688+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd1b4005650 con 0x7fd1d413d2f0 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1c5ffb640 1 -- 172.21.15.86:0/3211037587 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd1cc010040 con 0x7fd1d413d2f0 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1c5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.694 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 msgr2=0x7fd1b803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 0x7fd1b803e070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8007fd0 tx=0x7fd1c80079a0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 10 monclient: shutdown 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 msgr2=0x7fd1d413abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00efc0 tx=0x7fd1cc00c490 comp rx=0 tx=0).stop 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 shutdown_connections 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1b803bbb0 0x7fd1b803e070 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1d413d2f0 0x7fd1d413abe0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d40a5bd0 0x7fd1d4138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 --2- 172.21.15.86:0/3211037587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1d40a50e0 0x7fd1d4138100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 >> 172.21.15.86:0/3211037587 conn(0x7fd1d40aaae0 msgr2=0x7fd1d40a9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 shutdown_connections 2024-09-17T15:27:31.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.692+0000 7fd1db542640 1 -- 172.21.15.86:0/3211037587 wait complete. 2024-09-17T15:27:31.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.5 2024-09-17T15:27:31.717 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.724+0000 7f77aef1b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:27:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.724+0000 7f77aef1b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.724+0000 7f77aef1b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.724+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7788005380 con 0x7f77a80a5bd0 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77a57fa640 1 -- 172.21.15.86:0/4270913268 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7794016070 con 0x7f77a80a5bd0 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.730 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 msgr2=0x7f778c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 0x7f778c03e020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f779c007ca0 tx=0x7f779c007bf0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 10 monclient: shutdown 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 msgr2=0x7f77a8138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 0x7f77a8138200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77a806f8e0 tx=0x7f779401aba0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 shutdown_connections 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f778c03bb60 0x7f778c03e020 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77a813d2f0 0x7f77a809e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77a80a5bd0 0x7f77a8138200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 --2- 172.21.15.86:0/4270913268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77a80a50e0 0x7f77a81cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 >> 172.21.15.86:0/4270913268 conn(0x7f77a80aaae0 msgr2=0x7f77a80a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 shutdown_connections 2024-09-17T15:27:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7f77aef1b640 1 -- 172.21.15.86:0/4270913268 wait complete. 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7fcbc6ffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7fcbc6ffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7fcbc6ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.728+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fcbb8005380 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbdcff9640 1 -- 172.21.15.86:0/750952098 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcbd40162d0 con 0x7fcbe00a5bd0 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.734 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 msgr2=0x7fcbcc03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 0x7fcbcc03e070 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0007fd0 tx=0x7fcbd0009040 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 10 monclient: shutdown 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 msgr2=0x7fcbe0138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40ad040 tx=0x7fcbd401ae00 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 shutdown_connections 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bbb0 0x7fcbcc03e070 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013d2f0 0x7fcbe009e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00a5bd0 0x7fcbe0138290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 --2- 172.21.15.86:0/750952098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00a50e0 0x7fcbe01cfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 >> 172.21.15.86:0/750952098 conn(0x7fcbe00aaae0 msgr2=0x7fcbe00a3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 shutdown_connections 2024-09-17T15:27:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.732+0000 7fcbc6ffd640 1 -- 172.21.15.86:0/750952098 wait complete. 2024-09-17T15:27:31.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.4 2024-09-17T15:27:31.751 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:31.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-17T15:27:31.752 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b58af4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b58af4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b58af4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8b38005380 con 0x7f8b540a3860 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b42ffd640 1 -- 172.21.15.86:0/2510588149 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8b4c022d10 con 0x7f8b540a3860 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.752+0000 7f8b42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:31.757 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:31.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 msgr2=0x7f8b34082030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 0x7f8b34082030 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b48004470 tx=0x7f8b48003700 comp rx=0 tx=0).stop 2024-09-17T15:27:31.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 10 monclient: shutdown 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 msgr2=0x7f8b54142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 0x7f8b54142720 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c00c7b0 tx=0x7f8b4c00cc70 comp rx=0 tx=0).stop 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 shutdown_connections 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8b3407fb70 0x7f8b34082030 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b540a3860 0x7f8b54142720 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b540a11b0 0x7f8b541ea680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 --2- 172.21.15.86:0/2510588149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b5409e080 0x7f8b541ea140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 >> 172.21.15.86:0/2510588149 conn(0x7f8b540a2d70 msgr2=0x7f8b5406ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 shutdown_connections 2024-09-17T15:27:31.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:31.756+0000 7f8b58af4640 1 -- 172.21.15.86:0/2510588149 wait complete. 2024-09-17T15:27:31.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.1 2024-09-17T15:27:31.773 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:32.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T15:27:32.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: get_monmap_and_config 2024-09-17T15:27:32.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: build_initial_monmap 2024-09-17T15:27:32.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: monmap: 2024-09-17T15:27:32.766 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:32.768960+0000 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:32.768960+0000 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:32.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 Processor -- start 2024-09-17T15:27:32.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 -- start start 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: init 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c700a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1c700a5bd0 0x7f1c7009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c700a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c7013d2f0 0x7f1c7013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c7013d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c700a6280 con 0x7f1c700a50e0 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1c7009d2a0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:32.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1c7009c0a0 con 0x7f1c700a5bd0 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c74b5d640 10 monclient(hunting): authenticate will time out at 985.735180s 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c700a50e0 0x7f1c700a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37580/0 (socket says 172.21.15.86:37580) 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6e59c640 1 -- 172.21.15.86:0/4082050543 learned_addr learned my addr 172.21.15.86:0/4082050543 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/4082050543 >> v2:172.21.15.86:3300/0 conn(0x7f1c7013d2f0 0x7f1c7013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): get_auth_request con 0x7f1c7013d2f0 auth_method 0 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c65d9b640 1 --2- 172.21.15.86:0/4082050543 >> v2:172.21.15.203:3300/0 conn(0x7f1c700a5bd0 0x7f1c7009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c65d9b640 10 monclient(hunting): get_auth_request con 0x7f1c700a5bd0 auth_method 0 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/4082050543 >> v2:172.21.15.203:3300/0 conn(0x7f1c700a5bd0 msgr2=0x7f1c7009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/4082050543 >> v2:172.21.15.203:3300/0 conn(0x7f1c700a5bd0 0x7f1c7009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/4082050543 >> v1:172.21.15.86:6789/0 conn(0x7f1c700a50e0 legacy=0x7f1c700a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.764+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/4082050543 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c7009c380 con 0x7f1c7013d2f0 2024-09-17T15:27:32.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/4082050543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 msgr2=0x7f1c7013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/4082050543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c7013f6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c7006cc20 tx=0x7f1c6801c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ef3bf382d8bd4a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 1 -- 172.21.15.86:0/4082050543 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c680b2070 con 0x7f1c7013d2f0 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: dump: 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:32.774 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 1 -- 172.21.15.86:0/4082050543 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c680a5760 con 0x7f1c7013d2f0 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 1 -- 172.21.15.86:0/4082050543 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c680a5b30 con 0x7f1c7013d2f0 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 5 monclient: authenticate success, global_id 4245 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 4 set_mon_vals no callback set 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:32.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6dd9b640 10 monclient: dump: 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:32.776 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: shutdown 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- 172.21.15.86:0/4082050543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 msgr2=0x7f1c7013f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/4082050543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c7013f6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1c7006cc20 tx=0x7f1c6801c7c0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- 172.21.15.86:0/4082050543 shutdown_connections 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/4082050543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c7013f6e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/4082050543 >> v2:172.21.15.203:3300/0 conn(0x7f1c700a5bd0 0x7f1c7009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- 172.21.15.86:0/4082050543 >> 172.21.15.86:0/4082050543 conn(0x7f1c700aaae0 msgr2=0x7f1c700aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- 172.21.15.86:0/4082050543 shutdown_connections 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- 172.21.15.86:0/4082050543 wait complete. 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: build_initial_monmap 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: monmap: 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 Processor -- start 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- start start 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: init 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:32.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c700a50e0 0x7f1c701213b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c700a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c700a5bd0 0x7f1c70138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c700a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c7013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1c70142650 con 0x7f1c7013d2f0 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c700a6ad0 con 0x7f1c700a50e0 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c701427d0 con 0x7f1c700a5bd0 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c74b5d640 10 monclient(hunting): authenticate will time out at 985.742429s 2024-09-17T15:27:32.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c700a5bd0 0x7f1c70138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43498/0 (socket says 172.21.15.86:43498) 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/1439329835 learned_addr learned my addr 172.21.15.86:0/1439329835 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): get_auth_request con 0x7f1c7013d2f0 auth_method 0 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): get_auth_request con 0x7f1c700a5bd0 auth_method 0 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:32.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6e59c640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c700a50e0 0x7f1c701213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6e59c640 10 monclient(hunting): get_auth_request con 0x7f1c700a50e0 auth_method 0 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c700a5bd0 msgr2=0x7f1c70138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:32.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c700a5bd0 0x7f1c70138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c700a50e0 msgr2=0x7f1c701213b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c700a50e0 0x7f1c701213b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.772+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/1439329835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c700a7af0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c6ed9d640 1 -- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 msgr2=0x7f1c701cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c6ed9d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801ae70 tx=0x7f1c68019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c680bb020 con 0x7f1c7013d2f0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: dump: 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:32.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c680ad650 con 0x7f1c7013d2f0 2024-09-17T15:27:32.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 5 monclient: authenticate success, global_id 4248 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _renew_subs 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c701228d0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c680b2040 con 0x7f1c7013d2f0 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 10 monclient: dump: 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:32.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _renew_subs 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c700a18d0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c6d59a640 4 set_mon_vals no callback set 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1c680be020 con 0x7f1c7013d2f0 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c7009cd60 con 0x7f1c7013d2f0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 0x7f1c5003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f1c680ea4f0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c6e59c640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 0x7f1c5003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.776+0000 7f1c6e59c640 10 monclient: get_auth_request con 0x7f1c5003bb60 auth_method 0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.780+0000 7f1c6e59c640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 0x7f1c5003e020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c70121290 tx=0x7f1c60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.780+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1c680c5030 con 0x7f1c7013d2f0 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.780+0000 7f1c677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:32.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.780+0000 7f1c677fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:32.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c74b5d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:32.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c74b5d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:32.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c74b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:32.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1c70074be0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c677fe640 1 -- 172.21.15.86:0/1439329835 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1c700a7af0 con 0x7f1c7013d2f0 2024-09-17T15:27:32.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-17T15:27:32.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.940+0000 7f1c677fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:32.944 INFO:teuthology.orchestra.run.smithi086.stdout:47244640260 2024-09-17T15:27:32.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 msgr2=0x7f1c5003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:32.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 0x7f1c5003e020 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1c70121290 tx=0x7f1c60009040 comp rx=0 tx=0).stop 2024-09-17T15:27:32.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 10 monclient: shutdown 2024-09-17T15:27:32.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 msgr2=0x7f1c701cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1c6801ae70 tx=0x7f1c68019560 comp rx=0 tx=0).stop 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 shutdown_connections 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c5003bb60 0x7f1c5003e020 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c7013d2f0 0x7f1c701cfab0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c700a5bd0 0x7f1c70138220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 --2- 172.21.15.86:0/1439329835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c700a50e0 0x7f1c701213b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 >> 172.21.15.86:0/1439329835 conn(0x7f1c700aaae0 msgr2=0x7f1c700a3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 shutdown_connections 2024-09-17T15:27:32.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:32.944+0000 7f1c74b5d640 1 -- 172.21.15.86:0/1439329835 wait complete. 2024-09-17T15:27:32.962 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-17T15:27:32.962 DEBUG:teuthology.parallel:result is None 2024-09-17T15:27:32.962 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T15:27:32.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: get_monmap_and_config 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: monmap: 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.076 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:33.078573+0000 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:33.078573+0000 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:33.077 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 1 Processor -- start 2024-09-17T15:27:33.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 1 -- start start 2024-09-17T15:27:33.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: init 2024-09-17T15:27:33.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.072+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64b40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f64b4068db0 0x7f64b40691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-c con 0x7f64b4068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f64b4139f00 0x7f64b413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-a con 0x7f64b4139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b409d770 con 0x7f64b40689d0 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f64b409ed00 con 0x7f64b4139f00 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f64b409efe0 con 0x7f64b4068db0 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b8c68640 10 monclient(hunting): authenticate will time out at 986.044164s 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f64b40689d0 0x7f64b409ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37582/0 (socket says 172.21.15.86:37582) 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b259c640 1 -- 172.21.15.86:0/3234101478 learned_addr learned my addr 172.21.15.86:0/3234101478 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/3234101478 >> v2:172.21.15.86:3300/0 conn(0x7f64b4139f00 0x7f64b413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): get_auth_request con 0x7f64b4139f00 auth_method 0 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b1d9b640 1 --2- 172.21.15.86:0/3234101478 >> v2:172.21.15.203:3300/0 conn(0x7f64b4068db0 0x7f64b40691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b1d9b640 10 monclient(hunting): get_auth_request con 0x7f64b4068db0 auth_method 0 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 -- 172.21.15.86:0/3234101478 >> v2:172.21.15.203:3300/0 conn(0x7f64b4068db0 msgr2=0x7f64b40691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/3234101478 >> v2:172.21.15.203:3300/0 conn(0x7f64b4068db0 0x7f64b40691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 -- 172.21.15.86:0/3234101478 >> v1:172.21.15.86:6789/0 conn(0x7f64b40689d0 legacy=0x7f64b409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.076+0000 7f64b2d9d640 1 -- 172.21.15.86:0/3234101478 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b4069700 con 0x7f64b4139f00 2024-09-17T15:27:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 -- 172.21.15.86:0/3234101478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 msgr2=0x7f64b413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/3234101478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413c2f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64b40a06b0 tx=0x7f64ac01c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a67bb7756ff482 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 1 -- 172.21.15.86:0/3234101478 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0b0070 con 0x7f64b4139f00 2024-09-17T15:27:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: dump: 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 1 -- 172.21.15.86:0/3234101478 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64ac0ab600 con 0x7f64b4139f00 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 5 monclient: authenticate success, global_id 4251 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 1 -- 172.21.15.86:0/3234101478 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0a4e60 con 0x7f64b4139f00 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 4 set_mon_vals no callback set 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b159a640 10 monclient: dump: 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: shutdown 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- 172.21.15.86:0/3234101478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 msgr2=0x7f64b413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- 172.21.15.86:0/3234101478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413c2f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64b40a06b0 tx=0x7f64ac01c300 comp rx=0 tx=0).stop 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- 172.21.15.86:0/3234101478 shutdown_connections 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- 172.21.15.86:0/3234101478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413c2f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- 172.21.15.86:0/3234101478 >> v2:172.21.15.203:3300/0 conn(0x7f64b4068db0 0x7f64b40691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- 172.21.15.86:0/3234101478 >> 172.21.15.86:0/3234101478 conn(0x7f64b40a6440 msgr2=0x7f64b40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- 172.21.15.86:0/3234101478 shutdown_connections 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- 172.21.15.86:0/3234101478 wait complete. 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: monmap: 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:33.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 Processor -- start 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- start start 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: init 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 0x7f64b4132640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-b con 0x7f64b40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64b4068db0 0x7f64b413a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-c con 0x7f64b4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): picked mon.noname-a con 0x7f64b4139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f64b413f3e0 con 0x7f64b4139f00 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b413f260 con 0x7f64b40689d0 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f64b413f560 con 0x7f64b4068db0 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b8c68640 10 monclient(hunting): authenticate will time out at 986.050866s 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64b4068db0 0x7f64b413a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43514/0 (socket says 172.21.15.86:43514) 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 -- 172.21.15.86:0/2161197289 learned_addr learned my addr 172.21.15.86:0/2161197289 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): get_auth_request con 0x7f64b4139f00 auth_method 0 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): get_auth_request con 0x7f64b4068db0 auth_method 0 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b259c640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 0x7f64b4132640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 -- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64b4068db0 msgr2=0x7f64b413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64b4068db0 0x7f64b413a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 -- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 msgr2=0x7f64b4132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 0x7f64b4132640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b2d9d640 1 -- 172.21.15.86:0/2161197289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b4073970 con 0x7f64b4139f00 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b259c640 10 monclient: get_auth_request con 0x7f64b40689d0 auth_method 0 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.080+0000 7f64b259c640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 0x7f64b4132640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b2d9d640 1 -- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 msgr2=0x7f64b413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b2d9d640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f64ac01a9b0 tx=0x7f64ac0190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0b9020 con 0x7f64b4139f00 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:33.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: dump: 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64ac0a4e60 con 0x7f64b4139f00 2024-09-17T15:27:33.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 5 monclient: authenticate success, global_id 4254 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _renew_subs 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b40a1a90 con 0x7f64b4139f00 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0b0040 con 0x7f64b4139f00 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 10 monclient: dump: 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _renew_subs 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b4072270 con 0x7f64b4139f00 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b0d99640 4 set_mon_vals no callback set 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b40691c0 con 0x7f64b4139f00 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f64ac0bc020 con 0x7f64b4139f00 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 0x7f649403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f64ac0e9e80 con 0x7f64b4139f00 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b259c640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 0x7f649403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.084+0000 7f64b259c640 10 monclient: get_auth_request con 0x7f649403bb60 auth_method 0 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.088+0000 7f64b259c640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 0x7f649403e020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f649c00a850 tx=0x7f649c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.088+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f64ac0c3030 con 0x7f64b4139f00 2024-09-17T15:27:33.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.088+0000 7f64aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.088+0000 7f64aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:33.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.244+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f64b40a15d0 con 0x7f649403bb60 2024-09-17T15:27:33.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.244+0000 7f64aaffd640 1 -- 172.21.15.86:0/2161197289 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34227 (secure 0 0 0) 0x7f64b40a15d0 con 0x7f649403bb60 2024-09-17T15:27:33.248 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:33.249 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 msgr2=0x7f649403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 0x7f649403e020 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f649c00a850 tx=0x7f649c008040 comp rx=0 tx=0).stop 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 10 monclient: shutdown 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 msgr2=0x7f64b413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f64ac01a9b0 tx=0x7f64ac0190a0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 shutdown_connections 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f649403bb60 0x7f649403e020 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64b4139f00 0x7f64b413a850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64b4068db0 0x7f64b413a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 --2- 172.21.15.86:0/2161197289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b40689d0 0x7f64b4132640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 >> 172.21.15.86:0/2161197289 conn(0x7f64b40a6440 msgr2=0x7f64b409ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 shutdown_connections 2024-09-17T15:27:33.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.248+0000 7f64b8c68640 1 -- 172.21.15.86:0/2161197289 wait complete. 2024-09-17T15:27:33.265 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-17T15:27:32.373660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163304,"kb_used_data":1112,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067736,"statfs":{"total":579820584960,"available":579653361664,"internally_reserved":0,"allocated":1138688,"data_stored":1375588,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.747771"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738413+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:27:29.738413+0000","last_peered":"2024-09-17T15:27:29.738413+0000","last_clean":"2024-09-17T15:27:29.738413+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:27:29.738413+0000","last_undegraded":"2024-09-17T15:27:29.738413+0000","last_fullsized":"2024-09-17T15:27:29.738413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.642137+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:27:29.642137+0000","last_peered":"2024-09-17T15:27:29.642137+0000","last_clean":"2024-09-17T15:27:29.642137+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:27:29.642137+0000","last_undegraded":"2024-09-17T15:27:29.642137+0000","last_fullsized":"2024-09-17T15:27:29.642137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.739086+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:27:29.739086+0000","last_peered":"2024-09-17T15:27:29.739086+0000","last_clean":"2024-09-17T15:27:29.739086+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:27:29.739086+0000","last_undegraded":"2024-09-17T15:27:29.739086+0000","last_fullsized":"2024-09-17T15:27:29.739086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.641995+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:27:29.641995+0000","last_peered":"2024-09-17T15:27:29.641995+0000","last_clean":"2024-09-17T15:27:29.641995+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:27:29.641995+0000","last_undegraded":"2024-09-17T15:27:29.641995+0000","last_fullsized":"2024-09-17T15:27:29.641995+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":23,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.639322+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:27:29.639322+0000","last_peered":"2024-09-17T15:27:29.639322+0000","last_clean":"2024-09-17T15:27:29.639322+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:27:29.639322+0000","last_undegraded":"2024-09-17T15:27:29.639322+0000","last_fullsized":"2024-09-17T15:27:29.639322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.631572+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:27:29.631572+0000","last_peered":"2024-09-17T15:27:29.631572+0000","last_clean":"2024-09-17T15:27:29.631572+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:27:29.631572+0000","last_undegraded":"2024-09-17T15:27:29.631572+0000","last_fullsized":"2024-09-17T15:27:29.631572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738599+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:27:29.738599+0000","last_peered":"2024-09-17T15:27:29.738599+0000","last_clean":"2024-09-17T15:27:29.738599+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:27:29.738599+0000","last_undegraded":"2024-09-17T15:27:29.738599+0000","last_fullsized":"2024-09-17T15:27:29.738599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.638123+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:27:29.638123+0000","last_peered":"2024-09-17T15:27:29.638123+0000","last_clean":"2024-09-17T15:27:29.638123+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:27:29.638123+0000","last_undegraded":"2024-09-17T15:27:29.638123+0000","last_fullsized":"2024-09-17T15:27:29.638123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":109,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.737605+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:27:29.737605+0000","last_peered":"2024-09-17T15:27:29.737605+0000","last_clean":"2024-09-17T15:27:29.737605+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:27:29.737605+0000","last_undegraded":"2024-09-17T15:27:29.737605+0000","last_fullsized":"2024-09-17T15:27:29.737605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":52968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":52968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:27:33.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:27:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: get_monmap_and_config 2024-09-17T15:27:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: monmap: 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:33.382016+0000 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:33.382016+0000 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:33.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 Processor -- start 2024-09-17T15:27:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 -- start start 2024-09-17T15:27:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: init 2024-09-17T15:27:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 0x7fdca4069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca4068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdca4134a40 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca409d770 con 0x7fdca4134a40 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdca409ed00 con 0x7fdca4068db0 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdca409efe0 con 0x7fdca40689d0 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdcaa90e640 10 monclient(hunting): authenticate will time out at 986.347486s 2024-09-17T15:27:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca8eab640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdca4134a40 0x7fdca4134ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37588/0 (socket says 172.21.15.86:37588) 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca8eab640 1 -- 172.21.15.86:0/2059558967 learned_addr learned my addr 172.21.15.86:0/2059558967 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): get_auth_request con 0x7fdca40689d0 auth_method 0 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca37fe640 1 --2- 172.21.15.86:0/2059558967 >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 0x7fdca4069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 1 -- 172.21.15.86:0/2059558967 >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 msgr2=0x7fdca4069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.376+0000 7fdca3fff640 1 --2- 172.21.15.86:0/2059558967 >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 0x7fdca4069190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 1 -- 172.21.15.86:0/2059558967 >> v1:172.21.15.86:6789/0 conn(0x7fdca4134a40 legacy=0x7fdca4134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 1 monclient: found mon.noname-b 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca3fff640 1 -- 172.21.15.86:0/2059558967 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca40a04e0 con 0x7fdca40689d0 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca37fe640 10 monclient: get_auth_request con 0x7fdca4068db0 auth_method 0 2024-09-17T15:27:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.380+0000 7fdca37fe640 1 --2- 172.21.15.86:0/2059558967 >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 0x7fdca4069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:33.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca3fff640 1 -- 172.21.15.86:0/2059558967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 msgr2=0x7fdca409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca3fff640 1 --2- 172.21.15.86:0/2059558967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca409ffa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc980097c0 tx=0x7fdc9801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cee1635f231d600 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 1 -- 172.21.15.86:0/2059558967 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc980b2070 con 0x7fdca40689d0 2024-09-17T15:27:33.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: dump: 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 5 monclient: authenticate success, global_id 4247 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 1 -- 172.21.15.86:0/2059558967 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc980ad640 con 0x7fdca40689d0 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 1 -- 172.21.15.86:0/2059558967 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc980ada10 con 0x7fdca40689d0 2024-09-17T15:27:33.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdca2ffd640 10 monclient: dump: 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:33.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 4 set_mon_vals no callback set 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 10 monclient: shutdown 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 -- 172.21.15.86:0/2059558967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 msgr2=0x7fdca409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/2059558967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca409ffa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdc980097c0 tx=0x7fdc9801c3f0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 -- 172.21.15.86:0/2059558967 shutdown_connections 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/2059558967 >> v2:172.21.15.203:3300/0 conn(0x7fdca4068db0 0x7fdca4069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/2059558967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca409ffa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 -- 172.21.15.86:0/2059558967 >> 172.21.15.86:0/2059558967 conn(0x7fdca40a6440 msgr2=0x7fdca40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 -- 172.21.15.86:0/2059558967 shutdown_connections 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.384+0000 7fdcaa90e640 1 -- 172.21.15.86:0/2059558967 wait complete. 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient: monmap: 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:33.396 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:33.396 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 Processor -- start 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 -- start start 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient: init 2024-09-17T15:27:33.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca413bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-c con 0x7fdca40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-a con 0x7fdca4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:33.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 0x7fdca41cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): picked mon.noname-b con 0x7fdca4134a40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:33.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdca413dac0 con 0x7fdca4068db0 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdca413d940 con 0x7fdca4134a40 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdca413dc40 con 0x7fdca40689d0 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdcaa90e640 10 monclient(hunting): authenticate will time out at 986.357261s 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca413bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43520/0 (socket says 172.21.15.86:43520) 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 1 -- 172.21.15.86:0/1133970725 learned_addr learned my addr 172.21.15.86:0/1133970725 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:33.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): get_auth_request con 0x7fdca4068db0 auth_method 0 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): get_auth_request con 0x7fdca40689d0 auth_method 0 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca8eab640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 0x7fdca41cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 -- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 msgr2=0x7fdca41cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 0x7fdca41cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 -- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 msgr2=0x7fdca4133aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 monclient: found mon.noname-c 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca3fff640 1 -- 172.21.15.86:0/1133970725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca413c750 con 0x7fdca40689d0 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca8eab640 10 monclient: get_auth_request con 0x7fdca4134a40 auth_method 0 2024-09-17T15:27:33.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca8eab640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 0x7fdca41cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:33.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.388+0000 7fdca37fe640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca3fff640 1 -- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 msgr2=0x7fdca413bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca3fff640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca413bca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc980ac000 tx=0x7fdc98019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc980ba030 con 0x7fdca40689d0 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: dump: 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc980b6620 con 0x7fdca40689d0 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc980b2040 con 0x7fdca40689d0 2024-09-17T15:27:33.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 10 monclient: dump: 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca1ffb640 4 set_mon_vals no callback set 2024-09-17T15:27:33.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 5 monclient: authenticate success, global_id 4250 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _renew_subs 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca40a28b0 con 0x7fdca40689d0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _renew_subs 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca411cd80 con 0x7fdca40689d0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca409ffa0 con 0x7fdca40689d0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdc980ad500 con 0x7fdca40689d0 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 0x7fdc9003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7fdc980e98b0 con 0x7fdca40689d0 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca37fe640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 0x7fdc9003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.392+0000 7fdca37fe640 10 monclient: get_auth_request con 0x7fdc9003bb90 auth_method 0 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.396+0000 7fdca37fe640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 0x7fdc9003e050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdca413cdc0 tx=0x7fdc94008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.396+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdc98016070 con 0x7fdca40689d0 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.396+0000 7fdca0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.396+0000 7fdca0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:33.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.552+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdca411f4c0 con 0x7fdc9003bb90 2024-09-17T15:27:33.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.552+0000 7fdca0ff9640 1 -- 172.21.15.86:0/1133970725 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34227 (secure 0 0 0) 0x7fdca411f4c0 con 0x7fdc9003bb90 2024-09-17T15:27:33.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:33.558 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 msgr2=0x7fdc9003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 0x7fdc9003e050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdca413cdc0 tx=0x7fdc94008040 comp rx=0 tx=0).stop 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 10 monclient: shutdown 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 msgr2=0x7fdca413bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca413bca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdc980ac000 tx=0x7fdc98019190 comp rx=0 tx=0).stop 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 shutdown_connections 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc9003bb90 0x7fdc9003e050 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdca4134a40 0x7fdca41cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdca4068db0 0x7fdca4133aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 --2- 172.21.15.86:0/1133970725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdca40689d0 0x7fdca413bca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 >> 172.21.15.86:0/1133970725 conn(0x7fdca40a6440 msgr2=0x7fdca4135ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 shutdown_connections 2024-09-17T15:27:33.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.556+0000 7fdcaa90e640 1 -- 172.21.15.86:0/1133970725 wait complete. 2024-09-17T15:27:33.574 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-17T15:27:32.373660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163304,"kb_used_data":1112,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067736,"statfs":{"total":579820584960,"available":579653361664,"internally_reserved":0,"allocated":1138688,"data_stored":1375588,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.747771"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738413+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:27:29.738413+0000","last_peered":"2024-09-17T15:27:29.738413+0000","last_clean":"2024-09-17T15:27:29.738413+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:27:29.738413+0000","last_undegraded":"2024-09-17T15:27:29.738413+0000","last_fullsized":"2024-09-17T15:27:29.738413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.642137+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:27:29.642137+0000","last_peered":"2024-09-17T15:27:29.642137+0000","last_clean":"2024-09-17T15:27:29.642137+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:27:29.642137+0000","last_undegraded":"2024-09-17T15:27:29.642137+0000","last_fullsized":"2024-09-17T15:27:29.642137+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.739086+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:27:29.739086+0000","last_peered":"2024-09-17T15:27:29.739086+0000","last_clean":"2024-09-17T15:27:29.739086+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:27:29.739086+0000","last_undegraded":"2024-09-17T15:27:29.739086+0000","last_fullsized":"2024-09-17T15:27:29.739086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.641995+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:27:29.641995+0000","last_peered":"2024-09-17T15:27:29.641995+0000","last_clean":"2024-09-17T15:27:29.641995+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:27:29.641995+0000","last_undegraded":"2024-09-17T15:27:29.641995+0000","last_fullsized":"2024-09-17T15:27:29.641995+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":23,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.639322+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:27:29.639322+0000","last_peered":"2024-09-17T15:27:29.639322+0000","last_clean":"2024-09-17T15:27:29.639322+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:27:29.639322+0000","last_undegraded":"2024-09-17T15:27:29.639322+0000","last_fullsized":"2024-09-17T15:27:29.639322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.631572+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:27:29.631572+0000","last_peered":"2024-09-17T15:27:29.631572+0000","last_clean":"2024-09-17T15:27:29.631572+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:27:29.631572+0000","last_undegraded":"2024-09-17T15:27:29.631572+0000","last_fullsized":"2024-09-17T15:27:29.631572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738599+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:27:29.738599+0000","last_peered":"2024-09-17T15:27:29.738599+0000","last_clean":"2024-09-17T15:27:29.738599+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:27:29.738599+0000","last_undegraded":"2024-09-17T15:27:29.738599+0000","last_fullsized":"2024-09-17T15:27:29.738599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.638123+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:27:29.638123+0000","last_peered":"2024-09-17T15:27:29.638123+0000","last_clean":"2024-09-17T15:27:29.638123+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:27:29.638123+0000","last_undegraded":"2024-09-17T15:27:29.638123+0000","last_fullsized":"2024-09-17T15:27:29.638123+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'72","reported_seq":109,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.737605+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:27:29.737605+0000","last_peered":"2024-09-17T15:27:29.737605+0000","last_clean":"2024-09-17T15:27:29.737605+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:27:29.737605+0000","last_undegraded":"2024-09-17T15:27:29.737605+0000","last_fullsized":"2024-09-17T15:27:29.737605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":52968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":172032,"data_stored":52968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:27:33.574 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T15:27:33.575 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T15:27:33.575 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T15:27:33.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: get_monmap_and_config 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: monmap: 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:33.681486+0000 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:33.681486+0000 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:33.679 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:33.680 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:33.680 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:33.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 Processor -- start 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 -- start start 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: init 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb30c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb30c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb30c13aff0 0x7fb30c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb30c13aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb30c074470 con 0x7fb30c068db0 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb30c09c8b0 con 0x7fb30c13aff0 2024-09-17T15:27:33.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c09cb90 con 0x7fb30c0689d0 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb3108b7640 10 monclient(hunting): authenticate will time out at 986.646794s 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33348/0 (socket says 172.21.15.86:33348) 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 -- 172.21.15.86:0/3834714349 learned_addr learned my addr 172.21.15.86:0/3834714349 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): get_auth_request con 0x7fb30c0689d0 auth_method 0 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 1 --2- 172.21.15.86:0/3834714349 >> v2:172.21.15.86:3300/0 conn(0x7fb30c13aff0 0x7fb30c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): get_auth_request con 0x7fb30c13aff0 auth_method 0 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 -- 172.21.15.86:0/3834714349 >> v2:172.21.15.86:3300/0 conn(0x7fb30c13aff0 msgr2=0x7fb30c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 --2- 172.21.15.86:0/3834714349 >> v2:172.21.15.86:3300/0 conn(0x7fb30c13aff0 0x7fb30c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 -- 172.21.15.86:0/3834714349 >> v1:172.21.15.203:6789/0 conn(0x7fb30c068db0 legacy=0x7fb30c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 monclient: found mon.noname-b 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:33.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.676+0000 7fb30a59c640 1 -- 172.21.15.86:0/3834714349 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c09ce70 con 0x7fb30c0689d0 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30a59c640 1 -- 172.21.15.86:0/3834714349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 msgr2=0x7fb30c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30a59c640 1 --2- 172.21.15.86:0/3834714349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009870 tx=0x7fb2fc01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc00ae87d1856d0f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 1 -- 172.21.15.86:0/3834714349 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc0b1070 con 0x7fb30c0689d0 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: dump: 2024-09-17T15:27:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 1 -- 172.21.15.86:0/3834714349 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2fc0ac620 con 0x7fb30c0689d0 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 1 -- 172.21.15.86:0/3834714349 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30c09c2d0 con 0x7fb30c0689d0 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb3108b7640 5 monclient: authenticate success, global_id 4253 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb3108b7640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb3108b7640 4 set_mon_vals no callback set 2024-09-17T15:27:33.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.680+0000 7fb30959a640 10 monclient: dump: 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: shutdown 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- 172.21.15.86:0/3834714349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 msgr2=0x7fb30c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- 172.21.15.86:0/3834714349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc009870 tx=0x7fb2fc01c420 comp rx=0 tx=0).stop 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- 172.21.15.86:0/3834714349 shutdown_connections 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- 172.21.15.86:0/3834714349 >> v2:172.21.15.86:3300/0 conn(0x7fb30c13aff0 0x7fb30c13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- 172.21.15.86:0/3834714349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c0689d0 0x7fb30c06f840 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- 172.21.15.86:0/3834714349 >> 172.21.15.86:0/3834714349 conn(0x7fb30c0a6440 msgr2=0x7fb30c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- 172.21.15.86:0/3834714349 shutdown_connections 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- 172.21.15.86:0/3834714349 wait complete. 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: build_initial_monmap 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: monmap: 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:33.695 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 Processor -- start 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- start start 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: init 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c0689d0 0x7fb30c11ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb30c0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:33.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c068db0 0x7fb30c135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-c con 0x7fb30c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb30c13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c140610 con 0x7fb30c13aff0 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c140490 con 0x7fb30c0689d0 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c140790 con 0x7fb30c068db0 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb3108b7640 10 monclient(hunting): authenticate will time out at 986.653653s 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43536/0 (socket says 172.21.15.86:43536) 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 -- 172.21.15.86:0/1960276425 learned_addr learned my addr 172.21.15.86:0/1960276425 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): get_auth_request con 0x7fb30c13aff0 auth_method 0 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c068db0 0x7fb30c135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): get_auth_request con 0x7fb30c068db0 auth_method 0 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb309d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30a59c640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c0689d0 0x7fb30c11ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30a59c640 10 monclient(hunting): get_auth_request con 0x7fb30c0689d0 auth_method 0 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 -- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c068db0 msgr2=0x7fb30c135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c068db0 0x7fb30c135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 -- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c0689d0 msgr2=0x7fb30c11ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c0689d0 0x7fb30c11ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 monclient: found mon.noname-a 2024-09-17T15:27:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.684+0000 7fb30ad9d640 1 -- 172.21.15.86:0/1960276425 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c120470 con 0x7fb30c13aff0 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb30ad9d640 1 -- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 msgr2=0x7fb30c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb30ad9d640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb304009dd0 tx=0x7fb30400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304018020 con 0x7fb30c13aff0 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: dump: 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.702 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040146a0 con 0x7fb30c13aff0 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 5 monclient: authenticate success, global_id 4263 2024-09-17T15:27:33.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _renew_subs 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30c136780 con 0x7fb30c13aff0 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304014a90 con 0x7fb30c13aff0 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 10 monclient: dump: 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:33.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb308d99640 4 set_mon_vals no callback set 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _renew_subs 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30c120190 con 0x7fb30c13aff0 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30c06f840 con 0x7fb30c13aff0 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb30401b020 con 0x7fb30c13aff0 2024-09-17T15:27:33.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 0x7fb2ec03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7fb304050980 con 0x7fb30c13aff0 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb30a59c640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 0x7fb2ec03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.688+0000 7fb30a59c640 10 monclient: get_auth_request con 0x7fb2ec03bb60 auth_method 0 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.692+0000 7fb30a59c640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 0x7fb2ec03e020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb30c069ad0 tx=0x7fb2fc0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.692+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb304022070 con 0x7fb30c13aff0 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.692+0000 7fb2faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:33.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.692+0000 7fb2faffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:33.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.884+0000 7fb3108b7640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.884+0000 7fb3108b7640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb30c11fc30 con 0x7fb30c13aff0 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb2faffd640 1 -- 172.21.15.86:0/1960276425 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb304010040 con 0x7fb30c13aff0 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb2faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb2faffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:33.890 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 msgr2=0x7fb2ec03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 0x7fb2ec03e020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb30c069ad0 tx=0x7fb2fc0a7000 comp rx=0 tx=0).stop 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 10 monclient: shutdown 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 msgr2=0x7fb30c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb304009dd0 tx=0x7fb30400c490 comp rx=0 tx=0).stop 2024-09-17T15:27:33.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 shutdown_connections 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2ec03bb60 0x7fb2ec03e020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb30c13aff0 0x7fb30c11c8c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb30c068db0 0x7fb30c135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 --2- 172.21.15.86:0/1960276425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb30c0689d0 0x7fb30c11ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 >> 172.21.15.86:0/1960276425 conn(0x7fb30c0a6440 msgr2=0x7fb30c0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 shutdown_connections 2024-09-17T15:27:33.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:33.888+0000 7fb3108b7640 1 -- 172.21.15.86:0/1960276425 wait complete. 2024-09-17T15:27:33.908 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T15:27:33.908 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-17T15:27:33.922 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-17T15:27:33.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:27:34.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.036+0000 7fa4c6580640 10 monclient: get_monmap_and_config 2024-09-17T15:27:34.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.036+0000 7fa4c6580640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.036+0000 7fa4c6580640 10 monclient: monmap: 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:34.043888+0000 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:34.043888+0000 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:34.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 Processor -- start 2024-09-17T15:27:34.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 -- start start 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient: init 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4c00a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa4c00a54c0 0x7fa4c00a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c00a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4c013d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c00a6530 con 0x7fa4c00a50e0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c009c150 con 0x7fa4c013d2f0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c009c430 con 0x7fa4c00a54c0 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4c6580640 10 monclient(hunting): authenticate will time out at 987.009693s 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa4c00a50e0 0x7fa4c009ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37594/0 (socket says 172.21.15.86:37594) 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bffff640 1 -- 172.21.15.86:0/377985198 learned_addr learned my addr 172.21.15.86:0/377985198 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 1 --2- 172.21.15.86:0/377985198 >> v2:172.21.15.203:3300/0 conn(0x7fa4c00a54c0 0x7fa4c00a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): get_auth_request con 0x7fa4c00a54c0 auth_method 0 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 1 -- 172.21.15.86:0/377985198 >> v1:172.21.15.203:6789/0 conn(0x7fa4c013d2f0 legacy=0x7fa4c013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 1 -- 172.21.15.86:0/377985198 >> v1:172.21.15.86:6789/0 conn(0x7fa4c00a50e0 legacy=0x7fa4c009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 1 monclient: found mon.noname-c 2024-09-17T15:27:34.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:27:34.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.040+0000 7fa4bf7fe640 1 -- 172.21.15.86:0/377985198 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c00a6940 con 0x7fa4c00a54c0 2024-09-17T15:27:34.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4bf7fe640 1 -- 172.21.15.86:0/377985198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 msgr2=0x7fa4c00a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4bf7fe640 1 --2- 172.21.15.86:0/377985198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c00a58d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40097c0 tx=0x7fa4b401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6634629a599aff98 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4beffd640 1 -- 172.21.15.86:0/377985198 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40b0070 con 0x7fa4c00a54c0 2024-09-17T15:27:34.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:34.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.044+0000 7fa4beffd640 10 monclient: dump: 2024-09-17T15:27:34.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.051 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 5 monclient: authenticate success, global_id 4279 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 1 -- 172.21.15.86:0/377985198 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b40ab600 con 0x7fa4c00a54c0 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 1 -- 172.21.15.86:0/377985198 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b40ab9d0 con 0x7fa4c00a54c0 2024-09-17T15:27:34.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 4 set_mon_vals no callback set 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4beffd640 10 monclient: dump: 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.053 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: shutdown 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- 172.21.15.86:0/377985198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 msgr2=0x7fa4c00a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 --2- 172.21.15.86:0/377985198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c00a58d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40097c0 tx=0x7fa4b401c360 comp rx=0 tx=0).stop 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- 172.21.15.86:0/377985198 shutdown_connections 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 --2- 172.21.15.86:0/377985198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c00a58d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- 172.21.15.86:0/377985198 >> 172.21.15.86:0/377985198 conn(0x7fa4c00aaae0 msgr2=0x7fa4c00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- 172.21.15.86:0/377985198 shutdown_connections 2024-09-17T15:27:34.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- 172.21.15.86:0/377985198 wait complete. 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: monmap: 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.060 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 Processor -- start 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- start start 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: init 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:34.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 0x7fa4c0139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-a con 0x7fa4c00a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c01385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-b con 0x7fa4c00a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): picked mon.noname-c con 0x7fa4c013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c01439b0 con 0x7fa4c00a50e0 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c0143830 con 0x7fa4c00a54c0 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c0143b30 con 0x7fa4c013d2f0 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c6580640 10 monclient(hunting): authenticate will time out at 987.018792s 2024-09-17T15:27:34.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33372/0 (socket says 172.21.15.86:33372) 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 -- 172.21.15.86:0/3310811658 learned_addr learned my addr 172.21.15.86:0/3310811658 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): get_auth_request con 0x7fa4c013d2f0 auth_method 0 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 0x7fa4c0139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): get_auth_request con 0x7fa4c00a50e0 auth_method 0 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c01385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): get_auth_request con 0x7fa4c00a54c0 auth_method 0 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 -- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 msgr2=0x7fa4c01385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c01385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 -- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 msgr2=0x7fa4c0139ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 0x7fa4c0139ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 monclient: found mon.noname-c 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4c4b1d640 1 -- 172.21.15.86:0/3310811658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c00a18d0 con 0x7fa4c013d2f0 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.048+0000 7fa4bffff640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 0x7fa4c0139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:34.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c4b1d640 1 -- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 msgr2=0x7fa4c0138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c4b1d640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c7b0 tx=0x7fa4b800cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8016020 con 0x7fa4c013d2f0 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: dump: 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.066 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 5 monclient: authenticate success, global_id 4259 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _renew_subs 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4b800ac00 con 0x7fa4c013d2f0 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c013b290 con 0x7fa4c013d2f0 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4b8012420 con 0x7fa4c013d2f0 2024-09-17T15:27:34.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 10 monclient: dump: 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.068 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bdffb640 4 set_mon_vals no callback set 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _renew_subs 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c013a220 con 0x7fa4c013d2f0 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c00a58d0 con 0x7fa4c013d2f0 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa4b8019020 con 0x7fa4c013d2f0 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 0x7fa4ac03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7fa4b804fa50 con 0x7fa4c013d2f0 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bffff640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 0x7fa4ac03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.052+0000 7fa4bffff640 10 monclient: get_auth_request con 0x7fa4ac03bb60 auth_method 0 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.056+0000 7fa4bffff640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 0x7fa4ac03e020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0007ca0 tx=0x7fa4b0007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.056+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa4b8020070 con 0x7fa4c013d2f0 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.056+0000 7fa4bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.056+0000 7fa4bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:34.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4c6580640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:34.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4c6580640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:34.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4c6580640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa4c013ad10 con 0x7fa4c013d2f0 2024-09-17T15:27:34.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4bcff9640 1 -- 172.21.15.86:0/3310811658 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12370 (secure 0 0 0) 0x7fa4b8012840 con 0x7fa4c013d2f0 2024-09-17T15:27:34.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:27:34.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.216+0000 7fa4bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:34.220 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:34.220 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":18,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:27:32.368573+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:27:23.674476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:27:24.613917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.592045+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.760848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.159435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.733344+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.364717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.514816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 msgr2=0x7fa4ac03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 0x7fa4ac03e020 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0007ca0 tx=0x7fa4b0007bf0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 10 monclient: shutdown 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 msgr2=0x7fa4c0138af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800c7b0 tx=0x7fa4b800cc70 comp rx=0 tx=0).stop 2024-09-17T15:27:34.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 shutdown_connections 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa4ac03bb60 0x7fa4ac03e020 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4c013d2f0 0x7fa4c0138af0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c00a54c0 0x7fa4c01385b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 --2- 172.21.15.86:0/3310811658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4c00a50e0 0x7fa4c0139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 >> 172.21.15.86:0/3310811658 conn(0x7fa4c00aaae0 msgr2=0x7fa4c00a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 shutdown_connections 2024-09-17T15:27:34.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.220+0000 7fa4c6580640 1 -- 172.21.15.86:0/3310811658 wait complete. 2024-09-17T15:27:34.237 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T15:27:23.674476+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-17T15:27:24.613917+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 17, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T15:27:34.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.356+0000 7f8fb664e640 10 monclient: get_monmap_and_config 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.356+0000 7f8fb664e640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.356+0000 7f8fb664e640 10 monclient: monmap: 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:34.363326+0000 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:34.363326+0000 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:34.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:34.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 Processor -- start 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 -- start start 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient: init 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8fb00a0c70 0x7f8fb00a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb00a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb00a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fb0135f90 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb00a2130 con 0x7f8fb0135f90 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb00a2410 con 0x7f8fb00a0c70 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb00a26f0 con 0x7f8fb00a1590 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fb664e640 10 monclient(hunting): authenticate will time out at 987.328858s 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33384/0 (socket says 172.21.15.86:33384) 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 -- 172.21.15.86:0/895416862 learned_addr learned my addr 172.21.15.86:0/895416862 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): get_auth_request con 0x7f8fb00a1590 auth_method 0 2024-09-17T15:27:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8faffff640 1 --2- 172.21.15.86:0/895416862 >> v2:172.21.15.203:3300/0 conn(0x7f8fb00a0c70 0x7f8fb00a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8faffff640 10 monclient(hunting): get_auth_request con 0x7f8fb00a0c70 auth_method 0 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8faffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8faffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8faffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 -- 172.21.15.86:0/895416862 >> v2:172.21.15.203:3300/0 conn(0x7f8fb00a0c70 msgr2=0x7f8fb00a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 --2- 172.21.15.86:0/895416862 >> v2:172.21.15.203:3300/0 conn(0x7f8fb00a0c70 0x7f8fb00a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 -- 172.21.15.86:0/895416862 >> v1:172.21.15.203:6789/0 conn(0x7f8fb0135f90 legacy=0x7f8fb013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:34.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.360+0000 7f8fa77fe640 1 -- 172.21.15.86:0/895416862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb009f9f0 con 0x7f8fb00a1590 2024-09-17T15:27:34.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fa77fe640 1 -- 172.21.15.86:0/895416862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 msgr2=0x7f8fb0135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fa77fe640 1 --2- 172.21.15.86:0/895416862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009870 tx=0x7f8f9801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f007e9c3eeeca701 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 1 -- 172.21.15.86:0/895416862 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f980b1070 con 0x7f8fb00a1590 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: dump: 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fb664e640 5 monclient: authenticate success, global_id 4262 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fb664e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 1 -- 172.21.15.86:0/895416862 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f980ac600 con 0x7f8fb00a1590 2024-09-17T15:27:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fb664e640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8fb664e640 4 set_mon_vals no callback set 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 1 -- 172.21.15.86:0/895416862 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f980ac9d0 con 0x7f8fb00a1590 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.364+0000 7f8faf7fe640 10 monclient: dump: 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: shutdown 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- 172.21.15.86:0/895416862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 msgr2=0x7f8fb0135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- 172.21.15.86:0/895416862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009870 tx=0x7f8f9801c420 comp rx=0 tx=0).stop 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- 172.21.15.86:0/895416862 shutdown_connections 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- 172.21.15.86:0/895416862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb0135a50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- 172.21.15.86:0/895416862 >> v2:172.21.15.203:3300/0 conn(0x7f8fb00a0c70 0x7f8fb00a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- 172.21.15.86:0/895416862 >> 172.21.15.86:0/895416862 conn(0x7f8fb00a2f40 msgr2=0x7f8fb00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- 172.21.15.86:0/895416862 shutdown_connections 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- 172.21.15.86:0/895416862 wait complete. 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: monmap: 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:34.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 Processor -- start 2024-09-17T15:27:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- start start 2024-09-17T15:27:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: init 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fb00a0c70 0x7f8fb01ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb00a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb013d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb00a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb0135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb009ce60 con 0x7f8fb0135f90 2024-09-17T15:27:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb009cce0 con 0x7f8fb00a0c70 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb009ff50 con 0x7f8fb00a1590 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb664e640 10 monclient(hunting): authenticate will time out at 987.338207s 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43544/0 (socket says 172.21.15.86:43544) 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 -- 172.21.15.86:0/2670523988 learned_addr learned my addr 172.21.15.86:0/2670523988 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): get_auth_request con 0x7f8fb0135f90 auth_method 0 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb013d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): get_auth_request con 0x7f8fb00a1590 auth_method 0 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 -- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 msgr2=0x7f8fb013d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb013d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 -- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fb00a0c70 msgr2=0x7f8fb01ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fb00a0c70 0x7f8fb01ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 monclient: found mon.noname-a 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.368+0000 7f8fb4beb640 1 -- 172.21.15.86:0/2670523988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb006f9b0 con 0x7f8fb0135f90 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb4beb640 1 -- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 msgr2=0x7f8fb013d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb4beb640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c700 tx=0x7f8fa800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8004b20 con 0x7f8fb0135f90 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: dump: 2024-09-17T15:27:34.383 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 5 monclient: authenticate success, global_id 4269 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa8004cc0 con 0x7f8fb0135f90 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa800aa90 con 0x7f8fb0135f90 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _renew_subs 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb009d460 con 0x7f8fb0135f90 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fae7fc640 4 set_mon_vals no callback set 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 10 monclient: dump: 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.385 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _renew_subs 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb01388e0 con 0x7f8fb0135f90 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb00a1050 con 0x7f8fb0135f90 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8fa800ac70 con 0x7f8fb0135f90 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 0x7f8f9003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f8fa804fe90 con 0x7f8fb0135f90 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8faffff640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 0x7f8f9003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.372+0000 7f8faffff640 10 monclient: get_auth_request con 0x7f8f9003bb60 auth_method 0 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.376+0000 7f8faffff640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 0x7f8f9003e020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00098f0 tx=0x7f8fa0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.376+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8fa8020070 con 0x7f8fb0135f90 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.376+0000 7f8fad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.376+0000 7f8fad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:34.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fb664e640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fb664e640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fb664e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8fb0139440 con 0x7f8fb0135f90 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fad7fa640 1 -- 172.21.15.86:0/2670523988 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) ==== 93+0+10 (secure 0 0 0) 0x7f8fa8005210 con 0x7f8fb0135f90 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.548+0000 7f8fad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:34.552 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 1 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 msgr2=0x7f8f9003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 0x7f8f9003e020 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00098f0 tx=0x7f8fa0008040 comp rx=0 tx=0).stop 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 10 monclient: shutdown 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 msgr2=0x7f8fb013d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c700 tx=0x7f8fa800cbc0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 shutdown_connections 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8f9003bb60 0x7f8f9003e020 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb0135f90 0x7f8fb013d6c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb00a1590 0x7f8fb013d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 --2- 172.21.15.86:0/2670523988 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fb00a0c70 0x7f8fb01ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 >> 172.21.15.86:0/2670523988 conn(0x7f8fb00a2f40 msgr2=0x7f8fb00729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 shutdown_connections 2024-09-17T15:27:34.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.552+0000 7f8fb664e640 1 -- 172.21.15.86:0/2670523988 wait complete. 2024-09-17T15:27:34.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: get_monmap_and_config 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: monmap: 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:34.694501+0000 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:34.694501+0000 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:34.693 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 1 Processor -- start 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 1 -- start start 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: init 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:27:34.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.688+0000 7ff5599d4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff5540a50e0 0x7ff55409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-a con 0x7ff5540a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5540a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff55413d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5540a6280 con 0x7ff55413d2f0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff55409c0a0 con 0x7ff5540a54c0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff55409d2a0 con 0x7ff5540a50e0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5599d4640 10 monclient(hunting): authenticate will time out at 987.660096s 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff5540a50e0 0x7ff55409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5537fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff55413d2f0 0x7ff55413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37604/0 (socket says 172.21.15.86:37604) 2024-09-17T15:27:34.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff5537fe640 1 -- 172.21.15.86:0/2073055779 learned_addr learned my addr 172.21.15.86:0/2073055779 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): get_auth_request con 0x7ff5540a50e0 auth_method 0 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 1 -- 172.21.15.86:0/2073055779 >> v1:172.21.15.203:6789/0 conn(0x7ff5540a54c0 legacy=0x7ff5540a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 1 -- 172.21.15.86:0/2073055779 >> v1:172.21.15.86:6789/0 conn(0x7ff55413d2f0 legacy=0x7ff55413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 1 monclient: found mon.noname-a 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.692+0000 7ff552ffd640 1 -- 172.21.15.86:0/2073055779 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55409c3d0 con 0x7ff5540a50e0 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 -- 172.21.15.86:0/2073055779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 msgr2=0x7ff55409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 --2- 172.21.15.86:0/2073055779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff55409cd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1ee71067300ffe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 1 -- 172.21.15.86:0/2073055779 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b0070 con 0x7ff5540a50e0 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: dump: 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 1 -- 172.21.15.86:0/2073055779 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480ab600 con 0x7ff5540a50e0 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 1 -- 172.21.15.86:0/2073055779 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480ab9d0 con 0x7ff5540a50e0 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 5 monclient: authenticate success, global_id 4272 2024-09-17T15:27:34.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 4 set_mon_vals no callback set 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff551ffb640 10 monclient: dump: 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.701 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: shutdown 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- 172.21.15.86:0/2073055779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 msgr2=0x7ff55409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 --2- 172.21.15.86:0/2073055779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff55409cd60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff5480097c0 tx=0x7ff54801c360 comp rx=0 tx=0).stop 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- 172.21.15.86:0/2073055779 shutdown_connections 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 --2- 172.21.15.86:0/2073055779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff55409cd60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- 172.21.15.86:0/2073055779 >> 172.21.15.86:0/2073055779 conn(0x7ff5540aaae0 msgr2=0x7ff5540aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- 172.21.15.86:0/2073055779 shutdown_connections 2024-09-17T15:27:34.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- 172.21.15.86:0/2073055779 wait complete. 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: build_initial_monmap 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: monmap: 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:34.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 Processor -- start 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- start start 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: init 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-a con 0x7ff5540a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 0x7ff5541381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-b con 0x7ff5540a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff55413d2f0 0x7ff55413a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): picked mon.noname-c con 0x7ff55413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5541439b0 con 0x7ff5540a50e0 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff554143830 con 0x7ff5540a54c0 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff554143b30 con 0x7ff55413d2f0 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5599d4640 10 monclient(hunting): authenticate will time out at 987.667240s 2024-09-17T15:27:34.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43562/0 (socket says 172.21.15.86:43562) 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 -- 172.21.15.86:0/3542900798 learned_addr learned my addr 172.21.15.86:0/3542900798 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): get_auth_request con 0x7ff5540a50e0 auth_method 0 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff55413d2f0 0x7ff55413a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): get_auth_request con 0x7ff55413d2f0 auth_method 0 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 -- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff55413d2f0 msgr2=0x7ff55413a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff55413d2f0 0x7ff55413a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 -- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 msgr2=0x7ff5541381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5527fc640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 0x7ff5541381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 0x7ff5541381c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 monclient: found mon.noname-a 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff552ffd640 1 -- 172.21.15.86:0/3542900798 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5541d0d90 con 0x7ff5540a50e0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5527fc640 10 monclient: get_auth_request con 0x7ff5540a54c0 auth_method 0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.696+0000 7ff5527fc640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 0x7ff5541381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff552ffd640 1 -- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 msgr2=0x7ff5541cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff552ffd640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff54801c190 tx=0x7ff548019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b9070 con 0x7ff5540a50e0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: dump: 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: _finish_auth 0 2024-09-17T15:27:34.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5480b5690 con 0x7ff5540a50e0 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480b0040 con 0x7ff5540a50e0 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 10 monclient: dump: 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:34.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 5 monclient: authenticate success, global_id 4275 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _renew_subs 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5540a18d0 con 0x7ff5540a50e0 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5517fa640 4 set_mon_vals no callback set 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _renew_subs 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5540a7e30 con 0x7ff5540a50e0 2024-09-17T15:27:34.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff55409cd60 con 0x7ff5540a50e0 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff5480bc070 con 0x7ff5540a50e0 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 0x7ff54003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7ff5480e9df0 con 0x7ff5540a50e0 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.700+0000 7ff5527fc640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 0x7ff54003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:34.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.704+0000 7ff5527fc640 10 monclient: get_auth_request con 0x7ff54003bb90 auth_method 0 2024-09-17T15:27:34.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.704+0000 7ff5527fc640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 0x7ff54003e050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff554140b20 tx=0x7ff544009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:34.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.704+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff5480c3070 con 0x7ff5540a50e0 2024-09-17T15:27:34.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.704+0000 7ff53bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:34.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.704+0000 7ff53bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff5599d4640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff5599d4640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff5599d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7ff55413c900 con 0x7ff5540a50e0 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff53bfff640 1 -- 172.21.15.86:0/3542900798 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) ==== 92+0+10 (secure 0 0 0) 0x7ff5540a18d0 con 0x7ff5540a50e0 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff53bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-17T15:27:34.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.872+0000 7ff53bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:34.877 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 8 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 msgr2=0x7ff54003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 0x7ff54003e050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff554140b20 tx=0x7ff544009040 comp rx=0 tx=0).stop 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 10 monclient: shutdown 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 msgr2=0x7ff5541cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff54801c190 tx=0x7ff548019100 comp rx=0 tx=0).stop 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 shutdown_connections 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff54003bb90 0x7ff54003e050 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff55413d2f0 0x7ff55413a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5540a54c0 0x7ff5541381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 --2- 172.21.15.86:0/3542900798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5540a50e0 0x7ff5541cfe80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:34.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 >> 172.21.15.86:0/3542900798 conn(0x7ff5540aaae0 msgr2=0x7ff5540a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:34.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 shutdown_connections 2024-09-17T15:27:34.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:34.876+0000 7ff5599d4640 1 -- 172.21.15.86:0/3542900798 wait complete. 2024-09-17T15:27:34.894 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-17T15:27:34.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: get_monmap_and_config 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: monmap: 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:35.017568+0000 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:35.017568+0000 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:35.016 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 Processor -- start 2024-09-17T15:27:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 -- start start 2024-09-17T15:27:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: init 2024-09-17T15:27:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71b00689d0 0x7f71b006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-c con 0x7f71b00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71b0068db0 0x7f71b0069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-a con 0x7f71b0068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-b con 0x7f71b0135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f71b009c8b0 con 0x7f71b0068db0 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f71b009cb90 con 0x7f71b00689d0 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f71b009ce70 con 0x7f71b0135ae0 2024-09-17T15:27:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71b5546640 10 monclient(hunting): authenticate will time out at 987.983076s 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71b0068db0 0x7f71b0069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33418/0 (socket says 172.21.15.86:33418) 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 -- 172.21.15.86:0/2100832764 learned_addr learned my addr 172.21.15.86:0/2100832764 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): get_auth_request con 0x7f71b0135ae0 auth_method 0 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): get_auth_request con 0x7f71b0068db0 auth_method 0 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71aeffd640 1 --2- 172.21.15.86:0/2100832764 >> v2:172.21.15.203:3300/0 conn(0x7f71b00689d0 0x7f71b006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71aeffd640 10 monclient(hunting): get_auth_request con 0x7f71b00689d0 auth_method 0 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 -- 172.21.15.86:0/2100832764 >> v2:172.21.15.203:3300/0 conn(0x7f71b00689d0 msgr2=0x7f71b006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 --2- 172.21.15.86:0/2100832764 >> v2:172.21.15.203:3300/0 conn(0x7f71b00689d0 0x7f71b006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 -- 172.21.15.86:0/2100832764 >> v2:172.21.15.86:3300/0 conn(0x7f71b0068db0 msgr2=0x7f71b0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 --2- 172.21.15.86:0/2100832764 >> v2:172.21.15.86:3300/0 conn(0x7f71b0068db0 0x7f71b0069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.012+0000 7f71af7fe640 1 -- 172.21.15.86:0/2100832764 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b0069740 con 0x7f71b0135ae0 2024-09-17T15:27:35.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71af7fe640 1 -- 172.21.15.86:0/2100832764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 msgr2=0x7f71b0135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71af7fe640 1 --2- 172.21.15.86:0/2100832764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b0069a70 tx=0x7f71a001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fea08dc9c42d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71adffb640 1 -- 172.21.15.86:0/2100832764 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a00b0070 con 0x7f71b0135ae0 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.016+0000 7f71adffb640 10 monclient: dump: 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 5 monclient: authenticate success, global_id 4271 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 1 -- 172.21.15.86:0/2100832764 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a00ab600 con 0x7f71b0135ae0 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 4 set_mon_vals no callback set 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 1 -- 172.21.15.86:0/2100832764 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a00ab9d0 con 0x7f71b0135ae0 2024-09-17T15:27:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71adffb640 10 monclient: dump: 2024-09-17T15:27:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: shutdown 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- 172.21.15.86:0/2100832764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 msgr2=0x7f71b0135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- 172.21.15.86:0/2100832764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f71b0069a70 tx=0x7f71a001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- 172.21.15.86:0/2100832764 shutdown_connections 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- 172.21.15.86:0/2100832764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0135ae0 0x7f71b0135f70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- 172.21.15.86:0/2100832764 >> v2:172.21.15.86:3300/0 conn(0x7f71b0068db0 0x7f71b0069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- 172.21.15.86:0/2100832764 >> v2:172.21.15.203:3300/0 conn(0x7f71b00689d0 0x7f71b006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- 172.21.15.86:0/2100832764 >> 172.21.15.86:0/2100832764 conn(0x7f71b00a6440 msgr2=0x7f71b00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- 172.21.15.86:0/2100832764 shutdown_connections 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- 172.21.15.86:0/2100832764 wait complete. 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: monmap: 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 Processor -- start 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- start start 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: init 2024-09-17T15:27:35.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-a con 0x7f71b00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 0x7f71b011cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-c con 0x7f71b0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71b0135ae0 0x7f71b0134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): picked mon.noname-b con 0x7f71b0135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f71b013f820 con 0x7f71b00689d0 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f71b009c2d0 con 0x7f71b0135ae0 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f71b013f9a0 con 0x7f71b0068db0 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71b5546640 10 monclient(hunting): authenticate will time out at 987.990096s 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43582/0 (socket says 172.21.15.86:43582) 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 0x7f71b011cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 -- 172.21.15.86:0/2546963657 learned_addr learned my addr 172.21.15.86:0/2546963657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): get_auth_request con 0x7f71b00689d0 auth_method 0 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): get_auth_request con 0x7f71b0068db0 auth_method 0 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71af7fe640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71b0135ae0 0x7f71b0134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71af7fe640 10 monclient(hunting): get_auth_request con 0x7f71b0135ae0 auth_method 0 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-17T15:27:35.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 -- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 msgr2=0x7f71b011cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 0x7f71b011cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 -- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71b0135ae0 msgr2=0x7f71b0134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71b0135ae0 0x7f71b0134c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 monclient: found mon.noname-a 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71aeffd640 1 -- 172.21.15.86:0/2546963657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71b011e2d0 con 0x7f71b00689d0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.020+0000 7f71ae7fc640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 0x7f71b011cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71aeffd640 1 -- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 msgr2=0x7f71b011ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71aeffd640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71a8007ef0 tx=0x7f71a800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8013790 con 0x7f71b00689d0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: dump: 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a8019070 con 0x7f71b00689d0 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a8013b70 con 0x7f71b00689d0 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 5 monclient: authenticate success, global_id 4281 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 10 monclient: _renew_subs 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 1 -- 172.21.15.86:0/2546963657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71b00a39e0 con 0x7f71b00689d0 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71ad7fa640 4 set_mon_vals no callback set 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 10 monclient: dump: 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.038 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 10 monclient: _renew_subs 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71b5546640 1 -- 172.21.15.86:0/2546963657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71b01351d0 con 0x7f71b00689d0 2024-09-17T15:27:35.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a57fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7188005090 con 0x7f71b00689d0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f71a8013d10 con 0x7f71b00689d0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 0x7f719003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f71a80506e0 con 0x7f71b00689d0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71ae7fc640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 0x7f719003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.024+0000 7f71ae7fc640 10 monclient: get_auth_request con 0x7f719003bb60 auth_method 0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.028+0000 7f71ae7fc640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 0x7f719003e020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f71b01cee40 tx=0x7f7198009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.028+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f71a804f270 con 0x7f71b00689d0 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.028+0000 7f71a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.028+0000 7f71a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:35.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.196+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7188000bd0 con 0x7f719003bb60 2024-09-17T15:27:35.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a77fe640 1 -- 172.21.15.86:0/2546963657 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34123 (secure 0 0 0) 0x7f7188000bd0 con 0x7f719003bb60 2024-09-17T15:27:35.202 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:35.202 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 msgr2=0x7f719003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 0x7f719003e020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f71b01cee40 tx=0x7f7198009190 comp rx=0 tx=0).stop 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 10 monclient: shutdown 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 msgr2=0x7f71b011ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71a8007ef0 tx=0x7f71a800e4b0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 shutdown_connections 2024-09-17T15:27:35.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f719003bb60 0x7f719003e020 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71b0135ae0 0x7f71b0134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71b0068db0 0x7f71b011cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 --2- 172.21.15.86:0/2546963657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71b00689d0 0x7f71b011ee30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 >> 172.21.15.86:0/2546963657 conn(0x7f71b00a6440 msgr2=0x7f71b013c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 shutdown_connections 2024-09-17T15:27:35.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.200+0000 7f71a57fa640 1 -- 172.21.15.86:0/2546963657 wait complete. 2024-09-17T15:27:35.220 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-17T15:27:33.625174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163320,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067720,"statfs":{"total":579820584960,"available":579653345280,"internally_reserved":0,"allocated":1163264,"data_stored":1384792,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.999286"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738413+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:27:29.738413+0000","last_peered":"2024-09-17T15:27:29.738413+0000","last_clean":"2024-09-17T15:27:29.738413+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:27:29.738413+0000","last_undegraded":"2024-09-17T15:27:29.738413+0000","last_fullsized":"2024-09-17T15:27:29.738413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.646282+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:27:32.646282+0000","last_peered":"2024-09-17T15:27:32.646282+0000","last_clean":"2024-09-17T15:27:32.646282+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:27:32.646282+0000","last_undegraded":"2024-09-17T15:27:32.646282+0000","last_fullsized":"2024-09-17T15:27:32.646282+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.739086+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:27:29.739086+0000","last_peered":"2024-09-17T15:27:29.739086+0000","last_clean":"2024-09-17T15:27:29.739086+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:27:29.739086+0000","last_undegraded":"2024-09-17T15:27:29.739086+0000","last_fullsized":"2024-09-17T15:27:29.739086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.646063+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:27:32.646063+0000","last_peered":"2024-09-17T15:27:32.646063+0000","last_clean":"2024-09-17T15:27:32.646063+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:27:32.646063+0000","last_undegraded":"2024-09-17T15:27:32.646063+0000","last_fullsized":"2024-09-17T15:27:32.646063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.620455+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:27:32.620455+0000","last_peered":"2024-09-17T15:27:32.620455+0000","last_clean":"2024-09-17T15:27:32.620455+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:27:32.620455+0000","last_undegraded":"2024-09-17T15:27:32.620455+0000","last_fullsized":"2024-09-17T15:27:32.620455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.631572+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:27:29.631572+0000","last_peered":"2024-09-17T15:27:29.631572+0000","last_clean":"2024-09-17T15:27:29.631572+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:27:29.631572+0000","last_undegraded":"2024-09-17T15:27:29.631572+0000","last_fullsized":"2024-09-17T15:27:29.631572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738599+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:27:29.738599+0000","last_peered":"2024-09-17T15:27:29.738599+0000","last_clean":"2024-09-17T15:27:29.738599+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:27:29.738599+0000","last_undegraded":"2024-09-17T15:27:29.738599+0000","last_fullsized":"2024-09-17T15:27:29.738599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.620479+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:27:32.620479+0000","last_peered":"2024-09-17T15:27:32.620479+0000","last_clean":"2024-09-17T15:27:32.620479+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:27:32.620479+0000","last_undegraded":"2024-09-17T15:27:32.620479+0000","last_fullsized":"2024-09-17T15:27:32.620479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":109,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.737605+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:27:29.737605+0000","last_peered":"2024-09-17T15:27:29.737605+0000","last_clean":"2024-09-17T15:27:29.737605+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:27:29.737605+0000","last_undegraded":"2024-09-17T15:27:29.737605+0000","last_fullsized":"2024-09-17T15:27:29.737605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":57570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":57570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:27:35.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: get_monmap_and_config 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: monmap: 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:35.338558+0000 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:35.338558+0000 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:35.337 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 1 Processor -- start 2024-09-17T15:27:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 1 -- start start 2024-09-17T15:27:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: init 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0ecc0a5070 0x7f0ecc09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc0a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ecc0a5450 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-b con 0x7f0ecc13d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.332+0000 7f0ed0fe7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ecc0a6f40 con 0x7f0ecc0a5450 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0ecc0a65e0 con 0x7f0ecc0a5070 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ecc09c2a0 con 0x7f0ecc13d280 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ed0fe7640 10 monclient(hunting): authenticate will time out at 988.304047s 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33432/0 (socket says 172.21.15.86:33432) 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 -- 172.21.15.86:0/323376345 learned_addr learned my addr 172.21.15.86:0/323376345 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): get_auth_request con 0x7f0ecc13d280 auth_method 0 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0eca59c640 1 --2- 172.21.15.86:0/323376345 >> v2:172.21.15.203:3300/0 conn(0x7f0ecc0a5070 0x7f0ecc09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0eca59c640 10 monclient(hunting): get_auth_request con 0x7f0ecc0a5070 auth_method 0 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0eca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0eca59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0eca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 -- 172.21.15.86:0/323376345 >> v2:172.21.15.203:3300/0 conn(0x7f0ecc0a5070 msgr2=0x7f0ecc09cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 --2- 172.21.15.86:0/323376345 >> v2:172.21.15.203:3300/0 conn(0x7f0ecc0a5070 0x7f0ecc09cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 -- 172.21.15.86:0/323376345 >> v1:172.21.15.203:6789/0 conn(0x7f0ecc0a5450 legacy=0x7f0ecc0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 monclient: found mon.noname-b 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.336+0000 7f0ecad9d640 1 -- 172.21.15.86:0/323376345 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc09c580 con 0x7f0ecc13d280 2024-09-17T15:27:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 1 -- 172.21.15.86:0/323376345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 msgr2=0x7f0ecc13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 1 --2- 172.21.15.86:0/323376345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009e50 tx=0x7f0ec401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4359e372f760e3ed server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 1 -- 172.21.15.86:0/323376345 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec40a5530 con 0x7f0ecc13d280 2024-09-17T15:27:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: dump: 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 1 -- 172.21.15.86:0/323376345 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec40a56d0 con 0x7f0ecc13d280 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 5 monclient: authenticate success, global_id 4277 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 1 -- 172.21.15.86:0/323376345 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec40a5aa0 con 0x7f0ecc13d280 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 4 set_mon_vals no callback set 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec9d9b640 10 monclient: dump: 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: shutdown 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/323376345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 msgr2=0x7f0ecc13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/323376345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009e50 tx=0x7f0ec401c8f0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/323376345 shutdown_connections 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/323376345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13f670 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/323376345 >> v2:172.21.15.203:3300/0 conn(0x7f0ecc0a5070 0x7f0ecc09cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/323376345 >> 172.21.15.86:0/323376345 conn(0x7f0ecc0742a0 msgr2=0x7f0ecc0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/323376345 shutdown_connections 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/323376345 wait complete. 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: monmap: 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 Processor -- start 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- start start 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: init 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ecc0a5070 0x7f0ecc137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-b con 0x7f0ecc0a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-a con 0x7f0ecc0a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc13d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc142760 con 0x7f0ecc0a5450 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ecc1425e0 con 0x7f0ecc0a5070 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ecc1428e0 con 0x7f0ecc13d280 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ed0fe7640 10 monclient(hunting): authenticate will time out at 988.310931s 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43588/0 (socket says 172.21.15.86:43588) 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 -- 172.21.15.86:0/1284993877 learned_addr learned my addr 172.21.15.86:0/1284993877 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): get_auth_request con 0x7f0ecc0a5450 auth_method 0 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): get_auth_request con 0x7f0ecc13d280 auth_method 0 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0eca59c640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ecc0a5070 0x7f0ecc137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0eca59c640 10 monclient(hunting): get_auth_request con 0x7f0ecc0a5070 auth_method 0 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0eca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0eca59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0eca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 -- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 msgr2=0x7f0ecc13b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 -- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ecc0a5070 msgr2=0x7f0ecc137ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ecc0a5070 0x7f0ecc137ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 monclient: found mon.noname-a 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ec3fff640 1 -- 172.21.15.86:0/1284993877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc122710 con 0x7f0ecc0a5450 2024-09-17T15:27:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.340+0000 7f0ecad9d640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec3fff640 1 -- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 msgr2=0x7f0ecc13b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec3fff640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400c450 tx=0x7f0eb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4012a40 con 0x7f0ecc0a5450 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: dump: 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb4017070 con 0x7f0ecc0a5450 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 5 monclient: authenticate success, global_id 4284 2024-09-17T15:27:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _renew_subs 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ecc1229f0 con 0x7f0ecc0a5450 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4012e40 con 0x7f0ecc0a5450 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 10 monclient: dump: 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _renew_subs 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ecc1214c0 con 0x7f0ecc0a5450 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec8d99640 4 set_mon_vals no callback set 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ecc09cf60 con 0x7f0ecc0a5450 2024-09-17T15:27:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0eb401c460 con 0x7f0ecc0a5450 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 0x7f0eb003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f0eb404fb50 con 0x7f0ecc0a5450 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0eca59c640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 0x7f0eb003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.344+0000 7f0eca59c640 10 monclient: get_auth_request con 0x7f0eb003bb60 auth_method 0 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.348+0000 7f0eca59c640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 0x7f0eb003e020 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00b9d0 tx=0x7f0ebc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.348+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0eb401ac60 con 0x7f0ecc0a5450 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.348+0000 7f0ec2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.348+0000 7f0ec2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.500+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ecc122490 con 0x7f0eb003bb60 2024-09-17T15:27:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.504+0000 7f0ec2ffd640 1 -- 172.21.15.86:0/1284993877 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34123 (secure 0 0 0) 0x7f0ecc122490 con 0x7f0eb003bb60 2024-09-17T15:27:35.507 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:27:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 msgr2=0x7f0eb003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 0x7f0eb003e020 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc00b9d0 tx=0x7f0ebc008040 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 10 monclient: shutdown 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 msgr2=0x7f0ecc13b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0eb400c450 tx=0x7f0eb400c910 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 shutdown_connections 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0eb003bb60 0x7f0eb003e020 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ecc13d280 0x7f0ecc13b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ecc0a5450 0x7f0ecc13b220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 --2- 172.21.15.86:0/1284993877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ecc0a5070 0x7f0ecc137ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 >> 172.21.15.86:0/1284993877 conn(0x7f0ecc0742a0 msgr2=0x7f0ecc0a3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 shutdown_connections 2024-09-17T15:27:35.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.508+0000 7f0ed0fe7640 1 -- 172.21.15.86:0/1284993877 wait complete. 2024-09-17T15:27:35.524 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-17T15:27:33.625174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163320,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067720,"statfs":{"total":579820584960,"available":579653345280,"internally_reserved":0,"allocated":1163264,"data_stored":1384792,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.999286"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738413+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:27:29.738413+0000","last_peered":"2024-09-17T15:27:29.738413+0000","last_clean":"2024-09-17T15:27:29.738413+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:27:29.738413+0000","last_undegraded":"2024-09-17T15:27:29.738413+0000","last_fullsized":"2024-09-17T15:27:29.738413+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.646282+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:27:32.646282+0000","last_peered":"2024-09-17T15:27:32.646282+0000","last_clean":"2024-09-17T15:27:32.646282+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:27:32.646282+0000","last_undegraded":"2024-09-17T15:27:32.646282+0000","last_fullsized":"2024-09-17T15:27:32.646282+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.739086+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:27:29.739086+0000","last_peered":"2024-09-17T15:27:29.739086+0000","last_clean":"2024-09-17T15:27:29.739086+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:27:29.739086+0000","last_undegraded":"2024-09-17T15:27:29.739086+0000","last_fullsized":"2024-09-17T15:27:29.739086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.646063+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:27:32.646063+0000","last_peered":"2024-09-17T15:27:32.646063+0000","last_clean":"2024-09-17T15:27:32.646063+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:27:32.646063+0000","last_undegraded":"2024-09-17T15:27:32.646063+0000","last_fullsized":"2024-09-17T15:27:32.646063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":25,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.620455+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:27:32.620455+0000","last_peered":"2024-09-17T15:27:32.620455+0000","last_clean":"2024-09-17T15:27:32.620455+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:27:32.620455+0000","last_undegraded":"2024-09-17T15:27:32.620455+0000","last_fullsized":"2024-09-17T15:27:32.620455+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.631572+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:27:29.631572+0000","last_peered":"2024-09-17T15:27:29.631572+0000","last_clean":"2024-09-17T15:27:29.631572+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:27:29.631572+0000","last_undegraded":"2024-09-17T15:27:29.631572+0000","last_fullsized":"2024-09-17T15:27:29.631572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.738599+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:27:29.738599+0000","last_peered":"2024-09-17T15:27:29.738599+0000","last_clean":"2024-09-17T15:27:29.738599+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:27:29.738599+0000","last_undegraded":"2024-09-17T15:27:29.738599+0000","last_fullsized":"2024-09-17T15:27:29.738599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-17T15:27:32.620479+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:27:32.620479+0000","last_peered":"2024-09-17T15:27:32.620479+0000","last_clean":"2024-09-17T15:27:32.620479+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:27:32.620479+0000","last_undegraded":"2024-09-17T15:27:32.620479+0000","last_fullsized":"2024-09-17T15:27:32.620479+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'72","reported_seq":109,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-17T15:27:29.737605+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:27:29.737605+0000","last_peered":"2024-09-17T15:27:29.737605+0000","last_clean":"2024-09-17T15:27:29.737605+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:27:29.737605+0000","last_undegraded":"2024-09-17T15:27:29.737605+0000","last_fullsized":"2024-09-17T15:27:29.737605+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":57570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":52949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":57570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":229376,"data_stored":581877,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:27:35.529 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-17T15:27:35.529 DEBUG:tasks.mon_thrash:Yielding 2024-09-17T15:27:35.529 INFO:teuthology.run_tasks:Running task rados... 2024-09-17T15:27:35.542 INFO:tasks.rados:Beginning rados... 2024-09-17T15:27:35.542 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-17T15:27:35.542 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath10-testing-2024-09-17-1348'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath10-testing-2024-09-17-1348', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T15:27:35.542 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-17T15:27:35.542 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-17T15:27:35.585 INFO:tasks.rados:joining rados 2024-09-17T15:27:35.585 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-17T15:27:35.586 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726586855, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-17T15:27:35.586 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:27:35.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:27:35.586 INFO:tasks.rados:clients are ['client.0'] 2024-09-17T15:27:35.587 INFO:tasks.rados:starting run 0 out of 1 2024-09-17T15:27:35.587 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-17T15:27:35.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: get_monmap_and_config 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: monmap: 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:35.701608+0000 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:35.701608+0000 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:35.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 Processor -- start 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 -- start start 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: init 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-b con 0x7f89380a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89380a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f893813d2f0 0x7f893813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-a con 0x7f893813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893809d350 con 0x7f89380a54c0 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f893809bdc0 con 0x7f893813d2f0 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f89380a6580 con 0x7f89380a50e0 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893fc06640 10 monclient(hunting): authenticate will time out at 988.667005s 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33448/0 (socket says 172.21.15.86:33448) 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d1a1640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f89380a54c0 0x7f89380a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37608/0 (socket says 172.21.15.86:37608) 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 -- 172.21.15.86:0/3742715501 learned_addr learned my addr 172.21.15.86:0/3742715501 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): get_auth_request con 0x7f89380a50e0 auth_method 0 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 1 --2- 172.21.15.86:0/3742715501 >> v2:172.21.15.86:3300/0 conn(0x7f893813d2f0 0x7f893813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): get_auth_request con 0x7f893813d2f0 auth_method 0 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 -- 172.21.15.86:0/3742715501 >> v2:172.21.15.86:3300/0 conn(0x7f893813d2f0 msgr2=0x7f893813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 --2- 172.21.15.86:0/3742715501 >> v2:172.21.15.86:3300/0 conn(0x7f893813d2f0 0x7f893813f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 -- 172.21.15.86:0/3742715501 >> v1:172.21.15.86:6789/0 conn(0x7f89380a54c0 legacy=0x7f89380a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 monclient: found mon.noname-b 2024-09-17T15:27:35.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.696+0000 7f893d9a2640 1 -- 172.21.15.86:0/3742715501 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893809c150 con 0x7f89380a50e0 2024-09-17T15:27:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.700+0000 7f893d9a2640 1 -- 172.21.15.86:0/3742715501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 msgr2=0x7f893809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.700+0000 7f893d9a2640 1 --2- 172.21.15.86:0/3742715501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f893809c770 tx=0x7f893401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ef232fd42486c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.700+0000 7f893c9a0640 1 -- 172.21.15.86:0/3742715501 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340b0070 con 0x7f89380a50e0 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: dump: 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 1 -- 172.21.15.86:0/3742715501 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89340ab600 con 0x7f89380a50e0 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 1 -- 172.21.15.86:0/3742715501 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89340ab9d0 con 0x7f89380a50e0 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893c9a0640 10 monclient: dump: 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.709 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 5 monclient: authenticate success, global_id 4283 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 4 set_mon_vals no callback set 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: shutdown 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- 172.21.15.86:0/3742715501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 msgr2=0x7f893809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- 172.21.15.86:0/3742715501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f893809c770 tx=0x7f893401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- 172.21.15.86:0/3742715501 shutdown_connections 2024-09-17T15:27:35.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- 172.21.15.86:0/3742715501 >> v2:172.21.15.86:3300/0 conn(0x7f893813d2f0 0x7f893813f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- 172.21.15.86:0/3742715501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89380a50e0 0x7f893809ce10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- 172.21.15.86:0/3742715501 >> 172.21.15.86:0/3742715501 conn(0x7f89380aaae0 msgr2=0x7f89380aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- 172.21.15.86:0/3742715501 shutdown_connections 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- 172.21.15.86:0/3742715501 wait complete. 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: monmap: 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:35.711 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:35.712 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:35.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 Processor -- start 2024-09-17T15:27:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- start start 2024-09-17T15:27:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: init 2024-09-17T15:27:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 0x7f8938121190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-b con 0x7f89380a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-a con 0x7f89380a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 0x7f89380a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): picked mon.noname-c con 0x7f893813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8938142850 con 0x7f89380a54c0 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f89381426d0 con 0x7f89380a50e0 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f89381429d0 con 0x7f893813d2f0 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893fc06640 10 monclient(hunting): authenticate will time out at 988.675907s 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.704+0000 7f893d1a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43608/0 (socket says 172.21.15.86:43608) 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 0x7f89380a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 1 -- 172.21.15.86:0/2927443425 learned_addr learned my addr 172.21.15.86:0/2927443425 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): get_auth_request con 0x7f89380a54c0 auth_method 0 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): get_auth_request con 0x7f893813d2f0 auth_method 0 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 0x7f8938121190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 10 monclient(hunting): get_auth_request con 0x7f89380a50e0 auth_method 0 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 -- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 msgr2=0x7f8938121190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 0x7f8938121190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 -- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 msgr2=0x7f8938138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 monclient: found mon.noname-c 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 -- 172.21.15.86:0/2927443425 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89381226b0 con 0x7f893813d2f0 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d9a2640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 0x7f8938121190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893d1a1640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 -- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 msgr2=0x7f89380a2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893e1a3640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 0x7f89380a2a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f893000efc0 tx=0x7f893000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930018020 con 0x7f893813d2f0 2024-09-17T15:27:35.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: dump: 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 5 monclient: authenticate success, global_id 4286 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89300146a0 con 0x7f893813d2f0 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8930014a90 con 0x7f893813d2f0 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _renew_subs 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89380a7b10 con 0x7f893813d2f0 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892ffff640 4 set_mon_vals no callback set 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f892e7fc640 10 monclient: dump: 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _renew_subs 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89381223d0 con 0x7f893813d2f0 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.708+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f893809ce10 con 0x7f893813d2f0 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f893001b020 con 0x7f893813d2f0 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f892e7fc640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 0x7f891c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f8930050d30 con 0x7f893813d2f0 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f893d9a2640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 0x7f891c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f893d9a2640 10 monclient: get_auth_request con 0x7f891c03bb90 auth_method 0 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: get_monmap_and_config 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: monmap: 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:35.717705+0000 2024-09-17T15:27:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:35.717705+0000 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 Processor -- start 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 -- start start 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: init 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-b con 0x7f51b00a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f51b00a54c0 0x7f51b00a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-c con 0x7f51b00a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51b0137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51b00a6280 con 0x7f51b0137ea0 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f51b009c0a0 con 0x7f51b00a54c0 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51b009c380 con 0x7f51b00a50e0 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b6f61640 10 monclient(hunting): authenticate will time out at 988.682852s 2024-09-17T15:27:35.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33470/0 (socket says 172.21.15.86:33470) 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 -- 172.21.15.86:0/1640849023 learned_addr learned my addr 172.21.15.86:0/1640849023 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): get_auth_request con 0x7f51b00a50e0 auth_method 0 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 -- 172.21.15.86:0/1640849023 >> v2:172.21.15.203:3300/0 conn(0x7f51b00a54c0 msgr2=0x7f51b00a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 --2- 172.21.15.86:0/1640849023 >> v2:172.21.15.203:3300/0 conn(0x7f51b00a54c0 0x7f51b00a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 -- 172.21.15.86:0/1640849023 >> v1:172.21.15.203:6789/0 conn(0x7f51b0137ea0 legacy=0x7f51b0138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 monclient: found mon.noname-b 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f51b4cfd640 1 -- 172.21.15.86:0/1640849023 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b009c660 con 0x7f51b00a50e0 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.712+0000 7f893d9a2640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 0x7f891c03e050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f893809c770 tx=0x7f8934008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8930022020 con 0x7f893813d2f0 2024-09-17T15:27:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f892e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f892e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b4cfd640 1 -- 172.21.15.86:0/1640849023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 msgr2=0x7f51b009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b4cfd640 1 --2- 172.21.15.86:0/1640849023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009870 tx=0x7f51a401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=854a16ac710727a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 1 -- 172.21.15.86:0/1640849023 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a40b0070 con 0x7f51b00a50e0 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: dump: 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b6f61640 5 monclient: authenticate success, global_id 4289 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 1 -- 172.21.15.86:0/1640849023 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51a40a4e50 con 0x7f51b00a50e0 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b6f61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b6f61640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51b6f61640 4 set_mon_vals no callback set 2024-09-17T15:27:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 1 -- 172.21.15.86:0/1640849023 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a40ab820 con 0x7f51b00a50e0 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.716+0000 7f51af7fe640 10 monclient: dump: 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.738 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 10 monclient: shutdown 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- 172.21.15.86:0/1640849023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 msgr2=0x7f51b009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 --2- 172.21.15.86:0/1640849023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f51a4009870 tx=0x7f51a401c4b0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- 172.21.15.86:0/1640849023 shutdown_connections 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 --2- 172.21.15.86:0/1640849023 >> v2:172.21.15.203:3300/0 conn(0x7f51b00a54c0 0x7f51b00a58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 --2- 172.21.15.86:0/1640849023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b009cd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- 172.21.15.86:0/1640849023 >> 172.21.15.86:0/1640849023 conn(0x7f51b00aaae0 msgr2=0x7f51b00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- 172.21.15.86:0/1640849023 shutdown_connections 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- 172.21.15.86:0/1640849023 wait complete. 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 10 monclient: build_initial_monmap 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 10 monclient: monmap: 2024-09-17T15:27:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 Processor -- start 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 1 -- start start 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.720+0000 7f51b6f61640 10 monclient: init 2024-09-17T15:27:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b01d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-c con 0x7f51b00a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-a con 0x7f51b00a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51b0137ea0 0x7f51b013d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): picked mon.noname-b con 0x7f51b0137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f51b0142cb0 con 0x7f51b00a54c0 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51b0142b30 con 0x7f51b0137ea0 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51b0142e30 con 0x7f51b00a50e0 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient(hunting): authenticate will time out at 988.692332s 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b01d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43618/0 (socket says 172.21.15.86:43618) 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 -- 172.21.15.86:0/399667852 learned_addr learned my addr 172.21.15.86:0/399667852 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): get_auth_request con 0x7f51b00a54c0 auth_method 0 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): get_auth_request con 0x7f51b00a50e0 auth_method 0 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b54fe640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51b0137ea0 0x7f51b013d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b54fe640 10 monclient(hunting): get_auth_request con 0x7f51b0137ea0 auth_method 0 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b54fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b54fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b54fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2024-09-17T15:27:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 -- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 msgr2=0x7f51b01d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b01d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 -- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51b0137ea0 msgr2=0x7f51b013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51b0137ea0 0x7f51b013d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 monclient: found mon.noname-a 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 -- 172.21.15.86:0/399667852 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b00a18d0 con 0x7f51b00a54c0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b4cfd640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b01d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 -- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 msgr2=0x7f51b0136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51affff640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f519c00b550 tx=0x7f519c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c014070 con 0x7f51b00a54c0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: dump: 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: _finish_auth 0 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:27:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 5 monclient: authenticate success, global_id 4293 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _renew_subs 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f519c010aa0 con 0x7f51b00a54c0 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b0121380 con 0x7f51b00a54c0 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519c00f3c0 con 0x7f51b00a54c0 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51ad7fa640 10 monclient: dump: 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:35.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51aeffd640 4 set_mon_vals no callback set 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _renew_subs 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.724+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b013f720 con 0x7f51b00a54c0 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f519c017070 con 0x7f51b00a54c0 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b6f61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b6f61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b6f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51b00a7940 con 0x7f51b00a54c0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51ad7fa640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 0x7f519403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f519c04fbf0 con 0x7f51b00a54c0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b4cfd640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 0x7f519403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b4cfd640 10 monclient: get_auth_request con 0x7f519403bb90 auth_method 0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.728+0000 7f51b4cfd640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 0x7f519403e050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51b006f890 tx=0x7f51a40a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.732+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f51b00a18d0 con 0x7f51b00a54c0 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.732+0000 7f51ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.732+0000 7f51ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.900+0000 7f51b6f61640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-17T15:27:35.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.900+0000 7f51b6f61640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-17T15:27:35.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.900+0000 7f51b6f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:35.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.900+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} v 0) -- 0x7f51b009cd60 con 0x7f51b00a54c0 2024-09-17T15:27:35.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:27:35.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:35.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:35.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f89381d0a40 con 0x7f893813d2f0 2024-09-17T15:27:35.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f892e7fc640 1 -- 172.21.15.86:0/2927443425 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f8930010040 con 0x7f893813d2f0 2024-09-17T15:27:35.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f892e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:35.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f892e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:35.943 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:27:35.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 msgr2=0x7f891c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 0x7f891c03e050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f893809c770 tx=0x7f8934008040 comp rx=0 tx=0).stop 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 10 monclient: shutdown 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 msgr2=0x7f89380a2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 0x7f89380a2a00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f893000efc0 tx=0x7f893000c490 comp rx=0 tx=0).stop 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 shutdown_connections 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f891c03bb90 0x7f891c03e050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f893813d2f0 0x7f89380a2a00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89380a54c0 0x7f8938138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.940+0000 7f893fc06640 1 --2- 172.21.15.86:0/2927443425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89380a50e0 0x7f8938121190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:35.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.944+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 >> 172.21.15.86:0/2927443425 conn(0x7f89380aaae0 msgr2=0x7f893813f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:35.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.944+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 shutdown_connections 2024-09-17T15:27:35.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:35.944+0000 7f893fc06640 1 -- 172.21.15.86:0/2927443425 wait complete. 2024-09-17T15:27:35.961 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:27:35.961 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:27:35.961 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:27:35.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:27:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.060+0000 7f73fb469640 10 monclient: get_monmap_and_config 2024-09-17T15:27:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.060+0000 7f73fb469640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.060+0000 7f73fb469640 10 monclient: monmap: 2024-09-17T15:27:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.065 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:36.067735+0000 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:36.067735+0000 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:36.066 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 Processor -- start 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 -- start start 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient: init 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T15:27:36.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73f40a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73f40a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73f413e190 0x7f73f4140580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-a con 0x7f73f413e190 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f40a68d0 con 0x7f73f40a4f70 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f40a64a0 con 0x7f73f40a5350 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f73f409c7f0 con 0x7f73f413e190 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73fb469640 10 monclient(hunting): authenticate will time out at 989.033171s 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73f413e190 0x7f73f4140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9205640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f73f40a4f70 0x7f73f409cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37618/0 (socket says 172.21.15.86:37618) 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73f413e190 0x7f73f4140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43632/0 (socket says 172.21.15.86:43632) 2024-09-17T15:27:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9205640 1 -- 172.21.15.86:0/1568034159 learned_addr learned my addr 172.21.15.86:0/1568034159 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): get_auth_request con 0x7f73f413e190 auth_method 0 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 -- 172.21.15.86:0/1568034159 >> v1:172.21.15.203:6789/0 conn(0x7f73f40a5350 legacy=0x7f73f40a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 -- 172.21.15.86:0/1568034159 >> v1:172.21.15.86:6789/0 conn(0x7f73f40a4f70 legacy=0x7f73f409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 monclient: found mon.noname-a 2024-09-17T15:27:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.064+0000 7f73f9a06640 1 -- 172.21.15.86:0/1568034159 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f409c230 con 0x7f73f413e190 2024-09-17T15:27:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73f9a06640 1 -- 172.21.15.86:0/1568034159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f413e190 msgr2=0x7f73f4140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73f9a06640 1 --2- 172.21.15.86:0/1568034159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f413e190 0x7f73f4140580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009d90 tx=0x7f73ec01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2165377472a30915 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 1 -- 172.21.15.86:0/1568034159 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0b2070 con 0x7f73f413e190 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: dump: 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 1 -- 172.21.15.86:0/1568034159 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ec0a5940 con 0x7f73f413e190 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 1 -- 172.21.15.86:0/1568034159 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0a5d10 con 0x7f73f413e190 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73ebfff640 10 monclient: dump: 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 5 monclient: authenticate success, global_id 4296 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 4 set_mon_vals no callback set 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 10 monclient: shutdown 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- 172.21.15.86:0/1568034159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f413e190 msgr2=0x7f73f4140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 --2- 172.21.15.86:0/1568034159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f413e190 0x7f73f4140580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f73ec009d90 tx=0x7f73ec01c800 comp rx=0 tx=0).stop 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- 172.21.15.86:0/1568034159 shutdown_connections 2024-09-17T15:27:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 --2- 172.21.15.86:0/1568034159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f413e190 0x7f73f4140580 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- 172.21.15.86:0/1568034159 >> 172.21.15.86:0/1568034159 conn(0x7f73f40a7240 msgr2=0x7f73f40a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- 172.21.15.86:0/1568034159 shutdown_connections 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- 172.21.15.86:0/1568034159 wait complete. 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 10 monclient: monmap: 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 Processor -- start 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 1 -- start start 2024-09-17T15:27:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.068+0000 7f73fb469640 10 monclient: init 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f40a4f70 0x7f73f4135390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-b con 0x7f73f40a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 0x7f73f41358d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-a con 0x7f73f40a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 0x7f73f41cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): picked mon.noname-c con 0x7f73f413e190 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4143bd0 con 0x7f73f40a5350 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4143a50 con 0x7f73f40a4f70 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73f4143d50 con 0x7f73f413e190 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): authenticate will time out at 989.040214s 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 0x7f73f41358d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 0x7f73f41cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 0x7f73f41358d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43636/0 (socket says 172.21.15.86:43636) 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 1 -- 172.21.15.86:0/3414355398 learned_addr learned my addr 172.21.15.86:0/3414355398 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): get_auth_request con 0x7f73f40a5350 auth_method 0 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): get_auth_request con 0x7f73f413e190 auth_method 0 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9205640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f40a4f70 0x7f73f4135390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9205640 10 monclient(hunting): get_auth_request con 0x7f73f40a4f70 auth_method 0 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9205640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9205640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f40a4f70 msgr2=0x7f73f4135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f40a4f70 0x7f73f4135390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 msgr2=0x7f73f41358d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 0x7f73f41358d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 monclient: found mon.noname-c 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f41387e0 con 0x7f73f413e190 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f8a04640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a5350 0x7f73f41358d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 msgr2=0x7f73f41cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73f9a06640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 0x7f73f41cffe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0ac000 tx=0x7f73ec0195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0bd070 con 0x7f73f413e190 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: dump: 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ec0b8880 con 0x7f73f413e190 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec0b2040 con 0x7f73f413e190 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 10 monclient: dump: 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 5 monclient: authenticate success, global_id 4295 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _renew_subs 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73eb7fe640 4 set_mon_vals no callback set 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f413ea30 con 0x7f73f413e190 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _renew_subs 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f41375a0 con 0x7f73f413e190 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 0x7f73f41d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73f4138c00 con 0x7f73f41d0520 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.072+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f73ec0b8c70 con 0x7f73f413e190 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 0x7f73f41d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient: get_auth_request con 0x7f73f41d0520 auth_method 0 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d803bbb0 0x7f73d803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-17T15:27:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73f413e570 con 0x7f73f41d0520 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f73ec0eac80 con 0x7f73f413e190 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f8a04640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d803bbb0 0x7f73d803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f8a04640 10 monclient: get_auth_request con 0x7f73d803bbb0 auth_method 0 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 msgr2=0x7f73f41d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f9205640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 0x7f73f41d0900 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e500 tx=0x7f73f000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f0004e40 con 0x7f73f41d0520 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 10 monclient: dump: 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f73f0018070 con 0x7f73f41d0520 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 msgr2=0x7f73f41d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 0x7f73f41d0900 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e500 tx=0x7f73f000e9c0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 msgr2=0x7f73f41d0900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73e9ffb640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f41d0520 0x7f73f41d0900 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.076+0000 7f73f8a04640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d803bbb0 0x7f73d803e070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f73f4140220 tx=0x7f73e4007ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:27:36.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 0x7f73f40a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73f413cf60 con 0x7f73f40a3980 2024-09-17T15:27:36.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 0x7f73f40a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient: get_auth_request con 0x7f73f40a3980 auth_method 0 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 1 -- 172.21.15.86:0/3414355398 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f73f409cef0 con 0x7f73f40a3980 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 msgr2=0x7f73f40a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73f9205640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 0x7f73f40a3d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e4d0 tx=0x7f73f0004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f00066f0 con 0x7f73f40a3980 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 10 monclient: dump: 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f73f00066f0 con 0x7f73f40a3980 2024-09-17T15:27:36.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 msgr2=0x7f73f40a3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 0x7f73f40a3d90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e4d0 tx=0x7f73f0004950 comp rx=0 tx=0).stop 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 msgr2=0x7f73f40a3d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73e9ffb640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f40a3980 0x7f73f40a3d90 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f73f000e4d0 tx=0x7f73f0004950 comp rx=0 tx=0).stop 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 8, 2024-09-17T15:27:36.098 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28727, 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 23, 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.099 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:27:36.100 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.101 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:27:36.102 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.103 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.104 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.105 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:27:36.106 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:27:36.108 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:27:36.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:27:36.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:27:36.111 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d803bbb0 msgr2=0x7f73d803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.084+0000 7f73fb469640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d803bbb0 0x7f73d803e070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f73f4140220 tx=0x7f73e4007ed0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 10 monclient: shutdown 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 msgr2=0x7f73f41cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 --2- 172.21.15.86:0/3414355398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73f413e190 0x7f73f41cffe0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0ac000 tx=0x7f73ec0195a0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 reap_dead start 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73f9a06640 1 -- 172.21.15.86:0/3414355398 reap_dead start 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 shutdown_connections 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 >> 172.21.15.86:0/3414355398 conn(0x7f73f40a7240 msgr2=0x7f73f413d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 shutdown_connections 2024-09-17T15:27:36.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.088+0000 7f73fb469640 1 -- 172.21.15.86:0/3414355398 wait complete. 2024-09-17T15:27:36.114 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:27:36.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:27:36.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:27:36.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: get_monmap_and_config 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: monmap: 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:36.218252+0000 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:36.218252+0000 2024-09-17T15:27:36.216 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:36.217 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 Processor -- start 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 -- start start 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: init 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff32409d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-a con 0x7ff32409e260 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-b con 0x7ff32413a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff32409ff00 con 0x7ff32409d770 2024-09-17T15:27:36.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff32409e7f0 con 0x7ff32409e260 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff3240a2110 con 0x7ff32413a440 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff32a36c640 10 monclient(hunting): authenticate will time out at 989.183589s 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43654/0 (socket says 172.21.15.86:43654) 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33506/0 (socket says 172.21.15.86:33506) 2024-09-17T15:27:36.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 1 -- 172.21.15.86:0/2864897841 learned_addr learned my addr 172.21.15.86:0/2864897841 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): get_auth_request con 0x7ff32409e260 auth_method 0 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): get_auth_request con 0x7ff32413a440 auth_method 0 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.212+0000 7ff328909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 10 monclient(hunting): handle_auth_done global_id 4298 payload 274 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 1 -- 172.21.15.86:0/2864897841 >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 msgr2=0x7ff324139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 1 --2- 172.21.15.86:0/2864897841 >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 1 -- 172.21.15.86:0/2864897841 >> v1:172.21.15.203:6789/0 conn(0x7ff32409d770 legacy=0x7ff32409db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 1 monclient: found mon.noname-b 2024-09-17T15:27:36.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff328909640 1 -- 172.21.15.86:0/2864897841 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32409ec30 con 0x7ff32413a440 2024-09-17T15:27:36.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.216+0000 7ff3237fe640 1 --2- 172.21.15.86:0/2864897841 >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff328909640 1 -- 172.21.15.86:0/2864897841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 msgr2=0x7ff32413c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff328909640 1 --2- 172.21.15.86:0/2864897841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0097c0 tx=0x7ff31c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89d196c316bdb647 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 1 -- 172.21.15.86:0/2864897841 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0b0070 con 0x7ff32413a440 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: dump: 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 5 monclient: authenticate success, global_id 4298 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 1 -- 172.21.15.86:0/2864897841 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff31c0ab600 con 0x7ff32413a440 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 1 -- 172.21.15.86:0/2864897841 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31c0ab9d0 con 0x7ff32413a440 2024-09-17T15:27:36.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 4 set_mon_vals no callback set 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff322ffd640 10 monclient: dump: 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.232 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: shutdown 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- 172.21.15.86:0/2864897841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 msgr2=0x7ff32413c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- 172.21.15.86:0/2864897841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0097c0 tx=0x7ff31c01c360 comp rx=0 tx=0).stop 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- 172.21.15.86:0/2864897841 shutdown_connections 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- 172.21.15.86:0/2864897841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32413a440 0x7ff32413c830 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- 172.21.15.86:0/2864897841 >> v2:172.21.15.86:3300/0 conn(0x7ff32409e260 0x7ff324139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- 172.21.15.86:0/2864897841 >> 172.21.15.86:0/2864897841 conn(0x7ff3240a2f40 msgr2=0x7ff3240a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- 172.21.15.86:0/2864897841 shutdown_connections 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- 172.21.15.86:0/2864897841 wait complete. 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: monmap: 2024-09-17T15:27:36.233 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 Processor -- start 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- start start 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: init 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 0x7ff324131000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-c con 0x7ff32409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 0x7ff324131540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-a con 0x7ff32409e260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32413a440 0x7ff3241d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): picked mon.noname-b con 0x7ff32413a440 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff3240a1850 con 0x7ff32409e260 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3240a16d0 con 0x7ff32413a440 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff3240a19d0 con 0x7ff32409d770 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff32a36c640 10 monclient(hunting): authenticate will time out at 989.190688s 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 0x7ff324131540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 0x7ff324131000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 0x7ff324131540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43664/0 (socket says 172.21.15.86:43664) 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 -- 172.21.15.86:0/1330057568 learned_addr learned my addr 172.21.15.86:0/1330057568 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): get_auth_request con 0x7ff32409e260 auth_method 0 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 0x7ff324131000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33516/0 (socket says 172.21.15.86:33516) 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): get_auth_request con 0x7ff32409d770 auth_method 0 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 msgr2=0x7ff324131000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 0x7ff324131000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32413a440 msgr2=0x7ff3241d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32413a440 0x7ff3241d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff3237fe640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32411d9f0 con 0x7ff32409e260 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.220+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff32409d770 0x7ff324131000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff3237fe640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 msgr2=0x7ff324131540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff3237fe640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 0x7ff324131540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff31800eb70 tx=0x7ff31800d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff318010850 con 0x7ff32409e260 2024-09-17T15:27:36.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: dump: 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3180109f0 con 0x7ff32409e260 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 5 monclient: authenticate success, global_id 4311 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 monclient: _renew_subs 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32411fad0 con 0x7ff32409e260 2024-09-17T15:27:36.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff318010da0 con 0x7ff32409e260 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 10 monclient: dump: 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.241 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff3227fc640 4 set_mon_vals no callback set 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 monclient: _renew_subs 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3240a2780 con 0x7ff32409e260 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 0x7ff3241d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff32409db50 con 0x7ff3241d07f0 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 0x7ff3241d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient: get_auth_request con 0x7ff3241d07f0 auth_method 0 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff323fff640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3241d1110 con 0x7ff3241d07f0 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff31800f3d0 con 0x7ff32409e260 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff31003bb60 0x7ff31003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7ff318050680 con 0x7ff32409e260 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff328909640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff31003bb60 0x7ff31003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.224+0000 7ff328909640 10 monclient: get_auth_request con 0x7ff31003bb60 auth_method 0 2024-09-17T15:27:36.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff323fff640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 msgr2=0x7ff3241d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 0x7ff3241d0bd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff31400ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff314018030 con 0x7ff3241d07f0 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 10 monclient: dump: 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.246 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff31400ae40 con 0x7ff3241d07f0 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 msgr2=0x7ff3241d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 0x7ff3241d0bd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff31400ab10 comp rx=0 tx=0).stop 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 msgr2=0x7ff3241d0bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff320ff9640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241d07f0 0x7ff3241d0bd0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff31400ab10 comp rx=0 tx=0).stop 2024-09-17T15:27:36.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.228+0000 7ff328909640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff31003bb60 0x7ff31003e020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0097c0 tx=0x7ff31c004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff32a36c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff32a36c640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 0x7ff324136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff32a36c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff32411e610 con 0x7ff3241367a0 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 0x7ff324136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient: get_auth_request con 0x7ff3241367a0 auth_method 0 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.232+0000 7ff323fff640 1 -- 172.21.15.86:0/1330057568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff32411f850 con 0x7ff3241367a0 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff323fff640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 msgr2=0x7ff324136bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff323fff640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 0x7ff324136bb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff314004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff31401ec20 con 0x7ff3241367a0 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 10 monclient: dump: 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.249 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7ff31401ec20 con 0x7ff3241367a0 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 msgr2=0x7ff324136bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 0x7ff324136bb0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff314004a80 comp rx=0 tx=0).stop 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 msgr2=0x7ff324136bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff320ff9640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3241367a0 0x7ff324136bb0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff314009fd0 tx=0x7ff314004a80 comp rx=0 tx=0).stop 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 8, 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28879, 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:27:36.251 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 23, 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.252 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.253 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:27:36.254 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:27:36.255 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.256 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.258 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.260 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.263 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:27:36.264 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff31003bb60 msgr2=0x7ff31003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff31003bb60 0x7ff31003e020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff31c0097c0 tx=0x7ff31c004740 comp rx=0 tx=0).stop 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 10 monclient: shutdown 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 msgr2=0x7ff324131540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 --2- 172.21.15.86:0/1330057568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff32409e260 0x7ff324131540 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff31800eb70 tx=0x7ff31800d720 comp rx=0 tx=0).stop 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff328909640 1 -- 172.21.15.86:0/1330057568 reap_dead start 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff328909640 1 -- 172.21.15.86:0/1330057568 reap_dead start 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 shutdown_connections 2024-09-17T15:27:36.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 >> 172.21.15.86:0/1330057568 conn(0x7ff3240a2f40 msgr2=0x7ff324134e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 shutdown_connections 2024-09-17T15:27:36.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.236+0000 7ff32a36c640 1 -- 172.21.15.86:0/1330057568 wait complete. 2024-09-17T15:27:36.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:27:36.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:27:36.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:27:36.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: get_monmap_and_config 2024-09-17T15:27:36.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: monmap: 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:36.376125+0000 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:36.376125+0000 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:36.375 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:36.375 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:36.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 Processor -- start 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 -- start start 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: init 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:27:36.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92700a50e0 0x7f927009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-a con 0x7f92700a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f92700a54c0 0x7f92700a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-c con 0x7f92700a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 0x7f9270138350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-b con 0x7f9270137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f92700a6280 con 0x7f92700a50e0 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f927009c0a0 con 0x7f92700a54c0 2024-09-17T15:27:36.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f927009c380 con 0x7f9270137ea0 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f9274d15640 10 monclient(hunting): authenticate will time out at 989.341579s 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92700a50e0 0x7f927009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 0x7f9270138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92700a50e0 0x7f927009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43680/0 (socket says 172.21.15.86:43680) 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 -- 172.21.15.86:0/1188255495 learned_addr learned my addr 172.21.15.86:0/1188255495 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): get_auth_request con 0x7f92700a50e0 auth_method 0 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): get_auth_request con 0x7f9270137ea0 auth_method 0 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1188255495 >> v2:172.21.15.203:3300/0 conn(0x7f92700a54c0 0x7f92700a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926dd9b640 10 monclient(hunting): get_auth_request con 0x7f92700a54c0 auth_method 0 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 -- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 msgr2=0x7f9270138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 0x7f9270138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 -- 172.21.15.86:0/1188255495 >> v2:172.21.15.203:3300/0 conn(0x7f92700a54c0 msgr2=0x7f92700a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 --2- 172.21.15.86:0/1188255495 >> v2:172.21.15.203:3300/0 conn(0x7f92700a54c0 0x7f92700a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 monclient: found mon.noname-a 2024-09-17T15:27:36.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926e59c640 1 -- 172.21.15.86:0/1188255495 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927009c660 con 0x7f92700a50e0 2024-09-17T15:27:36.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.372+0000 7f926ed9d640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 0x7f9270138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926e59c640 1 -- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a50e0 msgr2=0x7f927009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926e59c640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a50e0 0x7f927009cd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f927006cc20 tx=0x7f926801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ec75a1168c1e26 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 1 -- 172.21.15.86:0/1188255495 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680a5750 con 0x7f92700a50e0 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:36.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: dump: 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 1 -- 172.21.15.86:0/1188255495 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92680a58f0 con 0x7f92700a50e0 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 1 -- 172.21.15.86:0/1188255495 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680a5cc0 con 0x7f92700a50e0 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 5 monclient: authenticate success, global_id 4314 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 4 set_mon_vals no callback set 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f926d59a640 10 monclient: dump: 2024-09-17T15:27:36.390 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 10 monclient: shutdown 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 -- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a50e0 msgr2=0x7f927009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a50e0 0x7f927009cd60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f927006cc20 tx=0x7f926801c8b0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 -- 172.21.15.86:0/1188255495 shutdown_connections 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9270137ea0 0x7f9270138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 --2- 172.21.15.86:0/1188255495 >> v2:172.21.15.203:3300/0 conn(0x7f92700a54c0 0x7f92700a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 --2- 172.21.15.86:0/1188255495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a50e0 0x7f927009cd60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 -- 172.21.15.86:0/1188255495 >> 172.21.15.86:0/1188255495 conn(0x7f92700aaae0 msgr2=0x7f92700aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 -- 172.21.15.86:0/1188255495 shutdown_connections 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.376+0000 7f9274d15640 1 -- 172.21.15.86:0/1188255495 wait complete. 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient: monmap: 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 Processor -- start 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 -- start start 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient: init 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:36.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 0x7f92701cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-c con 0x7f92700a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a54c0 0x7f9270137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-a con 0x7f92700a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9270137ea0 0x7f9270137650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): picked mon.noname-b con 0x7f9270137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f92700a6ae0 con 0x7f92700a54c0 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92700a6960 con 0x7f9270137ea0 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9270141b20 con 0x7f92700a50e0 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f9274d15640 10 monclient(hunting): authenticate will time out at 989.349355s 2024-09-17T15:27:36.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 0x7f92701cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 0x7f92701cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33546/0 (socket says 172.21.15.86:33546) 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 -- 172.21.15.86:0/1322553073 learned_addr learned my addr 172.21.15.86:0/1322553073 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): get_auth_request con 0x7f92700a50e0 auth_method 0 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a54c0 0x7f9270137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): get_auth_request con 0x7f92700a54c0 auth_method 0 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9270137ea0 msgr2=0x7f9270137650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9270137ea0 0x7f9270137650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a54c0 msgr2=0x7f9270137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a54c0 0x7f9270137110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 monclient: found mon.noname-c 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926e59c640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927013dd10 con 0x7f92700a50e0 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.380+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92700a54c0 0x7f9270137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926e59c640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 msgr2=0x7f92701cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926e59c640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 0x7f92701cfde0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92680af040 tx=0x7f9268019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680b6890 con 0x7f92700a50e0 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: dump: 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92680bc070 con 0x7f92700a50e0 2024-09-17T15:27:36.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680b6c70 con 0x7f92700a50e0 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 10 monclient: dump: 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926cd99640 4 set_mon_vals no callback set 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 5 monclient: authenticate success, global_id 4313 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 monclient: _renew_subs 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92700a4560 con 0x7f92700a50e0 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 monclient: _renew_subs 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92700a18d0 con 0x7f92700a50e0 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 0x7f927013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f927006a0e0 con 0x7f927013c2a0 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f92680c1460 con 0x7f92700a50e0 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f925003bb90 0x7f925003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 0x7f927013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f92680eaa60 con 0x7f92700a50e0 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient: get_auth_request con 0x7f927013c2a0 auth_method 0 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926ed9d640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f925003bb90 0x7f925003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926ed9d640 10 monclient: get_auth_request con 0x7f925003bb90 auth_method 0 2024-09-17T15:27:36.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 10 monclient(hunting): handle_auth_done global_id 4312 payload 824 2024-09-17T15:27:36.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.384+0000 7f926dd9b640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92700a7ae0 con 0x7f927013c2a0 2024-09-17T15:27:36.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f926dd9b640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 msgr2=0x7f927013c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 0x7f927013c680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f925800e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f926ed9d640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f925003bb90 0x7f925003e050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f92701d03c0 tx=0x7f9260009460 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258018020 con 0x7f927013c2a0 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 10 monclient: dump: 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.405 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9258010e90 con 0x7f927013c2a0 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 msgr2=0x7f927013c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 0x7f927013c680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f925800e5a0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 msgr2=0x7f927013c680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.388+0000 7f9266ffd640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013c2a0 0x7f927013c680 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f925800e5a0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.392+0000 7f9274d15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.392+0000 7f9274d15640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 0x7f92700a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.392+0000 7f9274d15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.392+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92700a58d0 con 0x7f927013cbc0 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 0x7f92700a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient: get_auth_request con 0x7f927013cbc0 auth_method 0 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 1 -- 172.21.15.86:0/1322553073 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9270074be0 con 0x7f927013cbc0 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 msgr2=0x7f92700a2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.396+0000 7f926dd9b640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 0x7f92700a2c50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f92580046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258018030 con 0x7f927013cbc0 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:27:36.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 10 monclient: dump: 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f925801f270 con 0x7f927013cbc0 2024-09-17T15:27:36.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 msgr2=0x7f92700a2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 0x7f92700a2c50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f92580046d0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 msgr2=0x7f92700a2c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9266ffd640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f927013cbc0 0x7f92700a2c50 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9258010b90 tx=0x7f92580046d0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:36.409 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 8, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 29003, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 23, 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:27:36.410 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:27:36.411 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.412 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:27:36.413 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:36.414 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:36.415 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.417 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:36.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.420 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.421 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.422 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:27:36.423 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f925003bb90 msgr2=0x7f925003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f925003bb90 0x7f925003e050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f92701d03c0 tx=0x7f9260009460 comp rx=0 tx=0).stop 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 10 monclient: shutdown 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 msgr2=0x7f92701cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 --2- 172.21.15.86:0/1322553073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92700a50e0 0x7f92701cfde0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f92680af040 tx=0x7f9268019650 comp rx=0 tx=0).stop 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f926ed9d640 1 -- 172.21.15.86:0/1322553073 reap_dead start 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f926ed9d640 1 -- 172.21.15.86:0/1322553073 reap_dead start 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 shutdown_connections 2024-09-17T15:27:36.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 >> 172.21.15.86:0/1322553073 conn(0x7f92700aaae0 msgr2=0x7f92700a1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 shutdown_connections 2024-09-17T15:27:36.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.400+0000 7f9274d15640 1 -- 172.21.15.86:0/1322553073 wait complete. 2024-09-17T15:27:36.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:27:36.426 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:27:36.426 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:27:36.426 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:27:36.426 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:27:36.427 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:27:36.427 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:27:36.447 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:27:36.448 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:27:36.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:27:36.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: get_monmap_and_config 2024-09-17T15:27:36.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: monmap: 2024-09-17T15:27:36.551 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.551 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:36.553894+0000 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:36.553894+0000 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:36.552 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 Processor -- start 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 -- start start 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: init 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-b con 0x7f78c00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78c0068db0 0x7f78c00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-a con 0x7f78c0068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78c0135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c009c8b0 con 0x7f78c0135ae0 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f78c0074500 con 0x7f78c0068db0 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f78c009cbe0 con 0x7f78c00689d0 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78c5043640 10 monclient(hunting): authenticate will time out at 989.519248s 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78c0068db0 0x7f78c00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33560/0 (socket says 172.21.15.86:33560) 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78c0068db0 0x7f78c00691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43698/0 (socket says 172.21.15.86:43698) 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 1 -- 172.21.15.86:0/531003913 learned_addr learned my addr 172.21.15.86:0/531003913 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): get_auth_request con 0x7f78c00689d0 auth_method 0 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bf59e640 1 -- 172.21.15.86:0/531003913 >> v1:172.21.15.203:6789/0 conn(0x7f78c0135ae0 legacy=0x7f78c0135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): get_auth_request con 0x7f78c0068db0 auth_method 0 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient(hunting): handle_auth_done global_id 4320 payload 274 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 -- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 msgr2=0x7f78c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 -- 172.21.15.86:0/531003913 >> v1:172.21.15.203:6789/0 conn(0x7f78c0135ae0 legacy=0x7f78c0135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 monclient: found mon.noname-a 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78be59c640 1 -- 172.21.15.86:0/531003913 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c009ce70 con 0x7f78c0068db0 2024-09-17T15:27:36.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.548+0000 7f78bed9d640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.552+0000 7f78be59c640 1 -- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c0068db0 msgr2=0x7f78c00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.552+0000 7f78be59c640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c0068db0 0x7f78c00691d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78b40097c0 tx=0x7f78b401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9033830d8af629dc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.552+0000 7f78bdd9b640 1 -- 172.21.15.86:0/531003913 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40b0070 con 0x7f78c0068db0 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: dump: 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.563 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 1 -- 172.21.15.86:0/531003913 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b40ab600 con 0x7f78c0068db0 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 5 monclient: authenticate success, global_id 4320 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:36.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 1 -- 172.21.15.86:0/531003913 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40ab9d0 con 0x7f78c0068db0 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 4 set_mon_vals no callback set 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bdd9b640 10 monclient: dump: 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.565 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: shutdown 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c0068db0 msgr2=0x7f78c00691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c0068db0 0x7f78c00691d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f78b40097c0 tx=0x7f78b401c300 comp rx=0 tx=0).stop 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- 172.21.15.86:0/531003913 shutdown_connections 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c0068db0 0x7f78c00691d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- 172.21.15.86:0/531003913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c00689d0 0x7f78c006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- 172.21.15.86:0/531003913 >> 172.21.15.86:0/531003913 conn(0x7f78c00a6440 msgr2=0x7f78c00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- 172.21.15.86:0/531003913 shutdown_connections 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- 172.21.15.86:0/531003913 wait complete. 2024-09-17T15:27:36.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: build_initial_monmap 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: monmap: 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 Processor -- start 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- start start 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: init 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 0x7f78c0135060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-a con 0x7f78c00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-c con 0x7f78c0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 0x7f78c01d0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): picked mon.noname-b con 0x7f78c0135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01409b0 con 0x7f78c00689d0 2024-09-17T15:27:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78c0140830 con 0x7f78c0135ae0 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f78c0140b30 con 0x7f78c0068db0 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78c5043640 10 monclient(hunting): authenticate will time out at 989.525955s 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 0x7f78c0135060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33562/0 (socket says 172.21.15.86:33562) 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): get_auth_request con 0x7f78c00689d0 auth_method 0 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 -- 172.21.15.86:0/3990213094 learned_addr learned my addr 172.21.15.86:0/3990213094 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): get_auth_request con 0x7f78c0068db0 auth_method 0 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bf59e640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 msgr2=0x7f78c01d0790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bf59e640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 0x7f78c01d0790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 msgr2=0x7f78c01d0790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 0x7f78c01d0790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 msgr2=0x7f78c0135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 0x7f78c0135060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient: _finish_hunting 0 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 monclient: found mon.noname-c 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78be59c640 1 -- 172.21.15.86:0/3990213094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c013db60 con 0x7f78c0068db0 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.556+0000 7f78bed9d640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 0x7f78c0135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78be59c640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 msgr2=0x7f78c01d0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:36.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78be59c640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78b401cba0 tx=0x7f78b4019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40b0040 con 0x7f78c0068db0 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: dump: 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: _finish_auth 0 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b40b9070 con 0x7f78c0068db0 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 5 monclient: authenticate success, global_id 4319 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _renew_subs 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c0120cd0 con 0x7f78c0068db0 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b40aaa10 con 0x7f78c0068db0 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 10 monclient: dump: 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:36.574 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _renew_subs 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78bd59a640 4 set_mon_vals no callback set 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c009ee80 con 0x7f78c0068db0 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c006f840 con 0x7f78c0068db0 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f78b40ab4c0 con 0x7f78c0068db0 2024-09-17T15:27:36.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 0x7f78ac03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:36.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f78b40e9a10 con 0x7f78c0068db0 2024-09-17T15:27:36.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78bed9d640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 0x7f78ac03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:36.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.560+0000 7f78bed9d640 10 monclient: get_auth_request con 0x7f78ac03bb10 auth_method 0 2024-09-17T15:27:36.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.564+0000 7f78bed9d640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 0x7f78ac03dfd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009630 tx=0x7f78b0009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:36.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.564+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f78b4016070 con 0x7f78c0068db0 2024-09-17T15:27:36.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.564+0000 7f78a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:36.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.564+0000 7f78a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:36.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78c5043640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:27:36.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78c5043640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78c5043640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78c01396f0 con 0x7f78c0068db0 2024-09-17T15:27:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78a77fe640 1 -- 172.21.15.86:0/3990213094 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f78b40b3390 con 0x7f78c0068db0 2024-09-17T15:27:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.780+0000 7f78a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:36.785 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 msgr2=0x7f78ac03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 0x7f78ac03dfd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b0009630 tx=0x7f78b0009240 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 10 monclient: shutdown 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 msgr2=0x7f78c01d0250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f78b401cba0 tx=0x7f78b4019100 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 shutdown_connections 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f78ac03bb10 0x7f78ac03dfd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78c0135ae0 0x7f78c01d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78c0068db0 0x7f78c01d0250 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 --2- 172.21.15.86:0/3990213094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78c00689d0 0x7f78c0135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 >> 172.21.15.86:0/3990213094 conn(0x7f78c00a6440 msgr2=0x7f78c00a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 shutdown_connections 2024-09-17T15:27:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:36.784+0000 7f78c5043640 1 -- 172.21.15.86:0/3990213094 wait complete. 2024-09-17T15:27:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:38.724+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:38.724+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:38.724+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:38.724+0000 7f519bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:39.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.724+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:39.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.724+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:39.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.724+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:39.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: get_monmap_and_config 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: build_initial_monmap 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: monmap: 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:39.933451+0000 2024-09-17T15:27:39.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:39.933451+0000 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:39.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 Processor -- start 2024-09-17T15:27:39.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 -- start start 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: init 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f3e240689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 0x7f3e240691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3e24068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e24135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e2409c8b0 con 0x7f3e24135ae0 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3e24074500 con 0x7f3e24068db0 2024-09-17T15:27:39.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e2409cbe0 con 0x7f3e240689d0 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e29b2c640 10 monclient(hunting): authenticate will time out at 992.898962s 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40068/0 (socket says 172.21.15.86:40068) 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 -- 172.21.15.86:0/185034010 learned_addr learned my addr 172.21.15.86:0/185034010 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): get_auth_request con 0x7f3e240689d0 auth_method 0 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:39.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e22ffd640 1 -- 172.21.15.86:0/185034010 >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 msgr2=0x7f3e240691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e23fff640 1 -- 172.21.15.86:0/185034010 >> v1:172.21.15.203:6789/0 conn(0x7f3e24135ae0 legacy=0x7f3e24135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e22ffd640 1 --2- 172.21.15.86:0/185034010 >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 0x7f3e240691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e227fc640 1 -- 172.21.15.86:0/185034010 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 -- 172.21.15.86:0/185034010 >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 msgr2=0x7f3e240691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 --2- 172.21.15.86:0/185034010 >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 0x7f3e240691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 -- 172.21.15.86:0/185034010 >> v1:172.21.15.203:6789/0 conn(0x7f3e24135ae0 legacy=0x7f3e24135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:39.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:39.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 monclient: found mon.noname-b 2024-09-17T15:27:39.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:39.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:27:39.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.928+0000 7f3e237fe640 1 -- 172.21.15.86:0/185034010 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2409ce70 con 0x7f3e240689d0 2024-09-17T15:27:39.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 1 -- 172.21.15.86:0/185034010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 msgr2=0x7f3e2406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 1 --2- 172.21.15.86:0/185034010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe543f3f20bb9ddf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 1 -- 172.21.15.86:0/185034010 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e140b2070 con 0x7f3e240689d0 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: dump: 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.940 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 1 -- 172.21.15.86:0/185034010 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e140ad680 con 0x7f3e240689d0 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 5 monclient: authenticate success, global_id 4322 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:39.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 4 set_mon_vals no callback set 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 1 -- 172.21.15.86:0/185034010 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e140ada50 con 0x7f3e240689d0 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e227fc640 10 monclient: dump: 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: shutdown 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- 172.21.15.86:0/185034010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 msgr2=0x7f3e2406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/185034010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009870 tx=0x7f3e1401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- 172.21.15.86:0/185034010 shutdown_connections 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/185034010 >> v2:172.21.15.203:3300/0 conn(0x7f3e24068db0 0x7f3e240691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/185034010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e2406f840 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- 172.21.15.86:0/185034010 >> 172.21.15.86:0/185034010 conn(0x7f3e240a6440 msgr2=0x7f3e240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- 172.21.15.86:0/185034010 shutdown_connections 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- 172.21.15.86:0/185034010 wait complete. 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: build_initial_monmap 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: monmap: 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:39.944 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 Processor -- start 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- start start 2024-09-17T15:27:39.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: init 2024-09-17T15:27:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e241ce070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-c con 0x7f3e240689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:39.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-a con 0x7f3e24068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 0x7f3e241ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): picked mon.noname-b con 0x7f3e24135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e241409b0 con 0x7f3e24068db0 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24140830 con 0x7f3e24135ae0 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e24140b30 con 0x7f3e240689d0 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e29b2c640 10 monclient(hunting): authenticate will time out at 992.907161s 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57856/0 (socket says 172.21.15.86:57856) 2024-09-17T15:27:39.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 -- 172.21.15.86:0/926489855 learned_addr learned my addr 172.21.15.86:0/926489855 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): get_auth_request con 0x7f3e24068db0 auth_method 0 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e23fff640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 msgr2=0x7f3e241ceaf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e23fff640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 0x7f3e241ceaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e241ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): get_auth_request con 0x7f3e240689d0 auth_method 0 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:39.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient(hunting): handle_auth_done global_id 4326 payload 824 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 msgr2=0x7f3e241ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e241ce070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 msgr2=0x7f3e241ceaf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 0x7f3e241ceaf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 monclient: found mon.noname-a 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:39.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.936+0000 7f3e22ffd640 1 -- 172.21.15.86:0/926489855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2413cb40 con 0x7f3e24068db0 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e22ffd640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 msgr2=0x7f3e241ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e22ffd640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00b530 tx=0x7f3e0c00b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c013020 con 0x7f3e24068db0 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: dump: 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:39.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: _finish_auth 0 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e0c005070 con 0x7f3e24068db0 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 5 monclient: authenticate success, global_id 4326 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _renew_subs 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2411f1b0 con 0x7f3e24068db0 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c005460 con 0x7f3e24068db0 2024-09-17T15:27:39.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e217fa640 4 set_mon_vals no callback set 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 10 monclient: dump: 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:39.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _renew_subs 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e2411df70 con 0x7f3e24068db0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3e0c016020 con 0x7f3e24068db0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e2406f840 con 0x7f3e24068db0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 0x7f3e0403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f3e0c04f9e0 con 0x7f3e24068db0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e237fe640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 0x7f3e0403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.940+0000 7f3e237fe640 10 monclient: get_auth_request con 0x7f3e0403bb10 auth_method 0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.944+0000 7f3e237fe640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 0x7f3e0403dfd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e2406fdb0 tx=0x7f3e140a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:39.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.948+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3e0c01d020 con 0x7f3e24068db0 2024-09-17T15:27:39.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.948+0000 7f3e1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:39.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:39.948+0000 7f3e1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:40.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e29b2c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:27:40.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e29b2c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:40.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e29b2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:40.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e2406d4f0 con 0x7f3e24068db0 2024-09-17T15:27:40.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e1bfff640 1 -- 172.21.15.86:0/926489855 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3e0c04e570 con 0x7f3e24068db0 2024-09-17T15:27:40.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:40.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.180+0000 7f3e1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:40.184 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 msgr2=0x7f3e0403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 0x7f3e0403dfd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3e2406fdb0 tx=0x7f3e140a7000 comp rx=0 tx=0).stop 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 10 monclient: shutdown 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 msgr2=0x7f3e241ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:40.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c00b530 tx=0x7f3e0c00b9f0 comp rx=0 tx=0).stop 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 shutdown_connections 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e0403bb10 0x7f3e0403dfd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e24135ae0 0x7f3e241ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e24068db0 0x7f3e241ce5b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 --2- 172.21.15.86:0/926489855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e240689d0 0x7f3e241ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 >> 172.21.15.86:0/926489855 conn(0x7f3e240a6440 msgr2=0x7f3e240a2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 shutdown_connections 2024-09-17T15:27:40.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.184+0000 7f3e29b2c640 1 -- 172.21.15.86:0/926489855 wait complete. 2024-09-17T15:27:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.728+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.728+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:40.728+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:41.728+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:41.728+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:41.728+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:42.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:42.728+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:42.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:42.728+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:42.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:42.728+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:43.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: get_monmap_and_config 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: build_initial_monmap 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: monmap: 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:43.329890+0000 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:43.329890+0000 2024-09-17T15:27:43.327 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:43.328 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:43.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 Processor -- start 2024-09-17T15:27:43.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 -- start start 2024-09-17T15:27:43.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: init 2024-09-17T15:27:43.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:43.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb300689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb300694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-c con 0x7feb3006f840 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb30074470 con 0x7feb300694c0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3009c8b0 con 0x7feb300689d0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7feb3009cb90 con 0x7feb3006f840 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb35cfa640 10 monclient(hunting): authenticate will time out at 996.295485s 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feb300694c0 0x7feb3013aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:60064/0 (socket says 172.21.15.86:60064) 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2f7fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7feb300689d0 legacy=0x7feb30068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 learned_addr learned my addr 172.21.15.86:0/1284179337 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:43.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2ffff640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 msgr2=0x7feb3013e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:43.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:43.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.328+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.86:6789/0 conn(0x7feb300694c0 legacy=0x7feb3013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.536+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 msgr2=0x7feb3013e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.536+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.203:6789/0 conn(0x7feb300689d0 legacy=0x7feb30068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:43.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.536+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:43.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.723+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:43.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.723+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:43.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.723+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:43.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.939+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.203:6789/0 conn(0x7feb300689d0 legacy=0x7feb30068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:43.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.939+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 msgr2=0x7feb3013e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:43.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:43.939+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:44.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:44.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:44.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:44.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.747+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.203:6789/0 conn(0x7feb300689d0 legacy=0x7feb30068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:44.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.747+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 msgr2=0x7feb3013e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:44.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:44.747+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:45.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:45.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:45.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:45.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:45.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:45.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): tick 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 msgr2=0x7feb3013e720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb3006f840 0x7feb3013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:46.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.203:6789/0 conn(0x7feb300689d0 legacy=0x7feb30068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.86:6789/0 conn(0x7feb300694c0 legacy=0x7feb3013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient: _add_conns ranks=[3,2] 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7feb10003750 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7feb10006260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7feb3009cb90 con 0x7feb10003750 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feb10008e00 con 0x7feb10006260 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.327+0000 7feb2d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:46.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 10 monclient(hunting): get_auth_request con 0x7feb10006260 auth_method 0 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb10003750 auth_method 0 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:46.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:27:46.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.331+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:46.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 10 monclient(hunting): get_auth_request con 0x7feb10006260 auth_method 0 2024-09-17T15:27:46.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb10003750 auth_method 0 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.535+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 10 monclient(hunting): get_auth_request con 0x7feb10006260 auth_method 0 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb10003750 auth_method 0 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:46.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:46.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:46.939+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:47.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:47.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:47.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:47.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:47.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:47.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 10 monclient(hunting): get_auth_request con 0x7feb10006260 auth_method 0 2024-09-17T15:27:47.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb10003750 auth_method 0 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:47.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:47.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:47.743+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:27:48.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:48.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:48.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:48.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:48.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:48.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:48.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:48.727+0000 7f519bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 10 monclient(hunting): tick 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 msgr2=0x7feb10008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb10006260 0x7feb10008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 msgr2=0x7feb10005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10003750 0x7feb10005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2d7fa640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.327+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 reap_dead start 2024-09-17T15:27:49.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb300694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 0x7feb10006710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7feb10003bc0 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7feb10006c50 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb3009ce70 con 0x7feb300694c0 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7feb3009cb90 con 0x7feb10006c50 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:49.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 1 -- 172.21.15.86:0/1284179337 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7feb10008e00 con 0x7feb10003bc0 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 msgr2=0x7feb10006710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 0x7feb10006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 10 monclient(hunting): get_auth_request con 0x7feb10006c50 auth_method 0 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:49.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:49.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.331+0000 7feb2f7fe640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.86:6789/0 conn(0x7feb300694c0 legacy=0x7feb1000b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 msgr2=0x7feb10006710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 0x7feb10006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 10 monclient(hunting): get_auth_request con 0x7feb10006c50 auth_method 0 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.535+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:27:49.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:49.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:49.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 msgr2=0x7feb10006710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 0x7feb10006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 10 monclient(hunting): get_auth_request con 0x7feb10006c50 auth_method 0 2024-09-17T15:27:49.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:27:49.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:49.939+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883612939 0 0) 0x7feb3009ce70 con 0x7feb300694c0 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): my global_id is 4329 2024-09-17T15:27:50.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb1c003530 con 0x7feb300694c0 2024-09-17T15:27:50.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7f519bfff640 10 monclient: tick 2024-09-17T15:27:50.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7f519bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:50.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7f519bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:50.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4233012359 0 0) 0x7feb1c003530 con 0x7feb300694c0 2024-09-17T15:27:50.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 msgr2=0x7feb10006710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:50.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.203:3300/0 conn(0x7feb10003bc0 0x7feb10006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 msgr2=0x7feb100070e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 --2- 172.21.15.86:0/1284179337 >> v2:172.21.15.86:3300/0 conn(0x7feb10006c50 0x7feb100070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3009c8b0 con 0x7feb300694c0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb35cfa640 5 monclient: authenticate success, global_id 4329 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7feb20009ca0 con 0x7feb300694c0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb35cfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:50.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: dump: 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:50.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 >> v1:172.21.15.86:6789/0 conn(0x7feb300694c0 legacy=0x7feb1000b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c003a30 0x7feb1c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): picked mon.c con 0x7feb1c003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb1c004810 0x7feb1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): picked mon.b con 0x7feb1c004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 0x7feb1c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): picked mon.a con 0x7feb1c0052f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feb3009cb90 con 0x7feb1c0052f0 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb10008e00 con 0x7feb1c004810 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c00c2c0 con 0x7feb1c003a30 2024-09-17T15:27:50.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb35cfa640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2effd640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb1c004810 msgr2=0x7feb1c004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2effd640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb1c004810 0x7feb1c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 0x7feb1c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c003a30 0x7feb1c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): get_auth_request con 0x7feb1c0052f0 auth_method 0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb1c003a30 auth_method 0 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:50.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 10 monclient(hunting): handle_auth_done global_id 4329 payload 293 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c003a30 msgr2=0x7feb1c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c003a30 0x7feb1c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.727+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb1c004810 msgr2=0x7feb1c004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb1c004810 0x7feb1c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 1 monclient: found mon.a 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:50.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c003530 con 0x7feb1c0052f0 2024-09-17T15:27:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 10 monclient: _finish_auth 0 2024-09-17T15:27:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:50.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.731+0000 7feb2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.735+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 reap_dead start 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.735+0000 7feb2ffff640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 msgr2=0x7feb1c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.735+0000 7feb2ffff640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 0x7feb1c00b9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feb3006fc20 tx=0x7feb2801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3571ac369e392c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2800e680 con 0x7feb1c0052f0 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: dump: 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.739 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb280039f0 con 0x7feb1c0052f0 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:50.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 1 -- 172.21.15.86:0/1284179337 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28003de0 con 0x7feb1c0052f0 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 4 set_mon_vals no callback set 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb2e7fc640 10 monclient: dump: 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 10 monclient: shutdown 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 msgr2=0x7feb1c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 0x7feb1c00b9d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feb3006fc20 tx=0x7feb2801c8b0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- 172.21.15.86:0/1284179337 shutdown_connections 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 --2- 172.21.15.86:0/1284179337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb1c0052f0 0x7feb1c00b9d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- 172.21.15.86:0/1284179337 >> 172.21.15.86:0/1284179337 conn(0x7feb300a6440 msgr2=0x7feb300a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- 172.21.15.86:0/1284179337 shutdown_connections 2024-09-17T15:27:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- 172.21.15.86:0/1284179337 wait complete. 2024-09-17T15:27:50.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 10 monclient: build_initial_monmap 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 10 monclient: monmap: 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 Processor -- start 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 1 -- start start 2024-09-17T15:27:50.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.739+0000 7feb35cfa640 10 monclient: init 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-c con 0x7feb1c0052f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 0x7feb30134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-b con 0x7feb301ce0c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 0x7feb30123690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): picked mon.noname-a con 0x7feb30125cb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feb301362f0 con 0x7feb30125cb0 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feb3009c2d0 con 0x7feb301ce0c0 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feb30136470 con 0x7feb1c0052f0 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb35cfa640 10 monclient(hunting): authenticate will time out at 1003.708743s 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 0x7feb30123690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2effd640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 msgr2=0x7feb30134fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36956/0 (socket says 172.21.15.86:36956) 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/812673553 learned_addr learned my addr 172.21.15.86:0/812673553 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request con 0x7feb1c0052f0 auth_method 0 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2effd640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 0x7feb30134fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): get_auth_request con 0x7feb30125cb0 auth_method 0 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 msgr2=0x7feb30134fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 0x7feb30134fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 msgr2=0x7feb30123690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 0x7feb30123690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/812673553 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb3009d460 con 0x7feb1c0052f0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2ffff640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 0x7feb30123690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 -- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 msgr2=0x7feb30134aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.743+0000 7feb2f7fe640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb2000bee0 tx=0x7feb20005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2001d020 con 0x7feb1c0052f0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: dump: 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb20019890 con 0x7feb1c0052f0 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 5 monclient: authenticate success, global_id 4328 2024-09-17T15:27:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _renew_subs 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb30135520 con 0x7feb1c0052f0 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb20019c80 con 0x7feb1c0052f0 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 10 monclient: dump: 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:50.752 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _renew_subs 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb2dffb640 4 set_mon_vals no callback set 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb30126c10 con 0x7feb1c0052f0 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb30136800 con 0x7feb1c0052f0 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7feb20020020 con 0x7feb1c0052f0 2024-09-17T15:27:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 0x7feb1003ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7feb200518e0 con 0x7feb1c0052f0 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb2effd640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 0x7feb1003ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.747+0000 7feb2effd640 10 monclient: get_auth_request con 0x7feb1003c800 auth_method 0 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.751+0000 7feb2effd640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 0x7feb1003ecc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb30070760 tx=0x7feb18009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.751+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7feb20027020 con 0x7feb1c0052f0 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.751+0000 7feb27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.751+0000 7feb27fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:50.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb35cfa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:27:50.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb35cfa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:50.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb35cfa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:50.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb30127510 con 0x7feb1c0052f0 2024-09-17T15:27:50.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb27fff640 1 -- 172.21.15.86:0/812673553 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7feb20022df0 con 0x7feb1c0052f0 2024-09-17T15:27:50.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:27:50.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.987+0000 7feb27fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:50.988 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:27:50.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 msgr2=0x7feb1003ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:50.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 0x7feb1003ecc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feb30070760 tx=0x7feb18009040 comp rx=0 tx=0).stop 2024-09-17T15:27:50.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 10 monclient: shutdown 2024-09-17T15:27:50.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 msgr2=0x7feb30134aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb2000bee0 tx=0x7feb20005480 comp rx=0 tx=0).stop 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 shutdown_connections 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb1003c800 0x7feb1003ecc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb30125cb0 0x7feb30123690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feb301ce0c0 0x7feb30134fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 --2- 172.21.15.86:0/812673553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb1c0052f0 0x7feb30134aa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 >> 172.21.15.86:0/812673553 conn(0x7feb300a6440 msgr2=0x7feb30070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 shutdown_connections 2024-09-17T15:27:50.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:50.991+0000 7feb35cfa640 1 -- 172.21.15.86:0/812673553 wait complete. 2024-09-17T15:27:51.006 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:27:51.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:27:51.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:27:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.111+0000 7f16a190c640 10 monclient: get_monmap_and_config 2024-09-17T15:27:51.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.111+0000 7f16a190c640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.111+0000 7f16a190c640 10 monclient: monmap: 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:51.115472+0000 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:51.115472+0000 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:51.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 Processor -- start 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 -- start start 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient: init 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f169c0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f169c0a5bd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f169c09cd60 0x7f169c140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-c con 0x7f169c09cd60 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c0a6280 con 0x7f169c0a50e0 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c09c0a0 con 0x7f169c0a5bd0 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f169c09c380 con 0x7f169c09cd60 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 10 monclient(hunting): authenticate will time out at 1004.080883s 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f1693fff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f169c0a5bd0 legacy=0x7f169c13d2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f169c0a50e0 0x7f169c0a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48206/0 (socket says 172.21.15.86:48206) 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169b7fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f169c09cd60 msgr2=0x7f169c140a20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169affd640 1 -- 172.21.15.86:0/469447940 learned_addr learned my addr 172.21.15.86:0/469447940 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:51.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169b7fe640 1 --2- 172.21.15.86:0/469447940 >> v2:172.21.15.203:3300/0 conn(0x7f169c09cd60 0x7f169c140a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169affd640 1 -- 172.21.15.86:0/469447940 >> v1:172.21.15.86:6789/0 conn(0x7f169c0a50e0 legacy=0x7f169c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3167150100 0 0) 0x7f169c0a6280 con 0x7f169c0a50e0 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): my global_id is 4335 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1688003530 con 0x7f169c0a50e0 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2390596118 0 0) 0x7f1688003530 con 0x7f169c0a50e0 2024-09-17T15:27:51.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 >> v2:172.21.15.203:3300/0 conn(0x7f169c09cd60 msgr2=0x7f169c140a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 --2- 172.21.15.86:0/469447940 >> v2:172.21.15.203:3300/0 conn(0x7f169c09cd60 0x7f169c140a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 >> v1:172.21.15.203:6789/0 conn(0x7f169c0a5bd0 legacy=0x7f169c13d2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c09c660 con 0x7f169c0a50e0 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f168c0019b0 con 0x7f169c0a50e0 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 5 monclient: authenticate success, global_id 4335 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f16a190c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: dump: 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 >> v1:172.21.15.86:6789/0 conn(0x7f169c0a50e0 legacy=0x7f169c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1688003a30 0x7f1688003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): picked mon.a con 0x7f1688003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f1688004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): picked mon.c con 0x7f1688004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16880099c0 0x7f168800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): picked mon.b con 0x7f16880099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f169c09c380 con 0x7f1688003a30 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f168800c3a0 con 0x7f16880099c0 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f168800c5a0 con 0x7f1688004810 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.115+0000 7f169a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f16a190c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f1688004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1688003a30 0x7f1688003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): get_auth_request con 0x7f1688004810 auth_method 0 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169b7fe640 1 -- 172.21.15.86:0/469447940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16880099c0 msgr2=0x7f168800bdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): get_auth_request con 0x7f1688003a30 auth_method 0 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169b7fe640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16880099c0 0x7f168800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 293 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 -- 172.21.15.86:0/469447940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16880099c0 msgr2=0x7f168800bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f16880099c0 0x7f168800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 -- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1688003a30 msgr2=0x7f1688003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1688003a30 0x7f1688003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 monclient: found mon.c 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f169b7fe640 1 -- 172.21.15.86:0/469447940 reap_dead start 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 1 -- 172.21.15.86:0/469447940 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1688003530 con 0x7f1688004810 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.119+0000 7f1693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f1693fff640 1 -- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 msgr2=0x7f1688004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f1693fff640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f1688004c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1684003940 tx=0x7f168401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2236fe500bf3c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684009520 con 0x7f1688004810 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: dump: 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.132 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1684003bc0 con 0x7f1688004810 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 1 -- 172.21.15.86:0/469447940 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1684005070 con 0x7f1688004810 2024-09-17T15:27:51.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 4 set_mon_vals no callback set 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169a7fc640 10 monclient: dump: 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: shutdown 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 msgr2=0x7f1688004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f1688004c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1684003940 tx=0x7f168401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- 172.21.15.86:0/469447940 shutdown_connections 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 --2- 172.21.15.86:0/469447940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f1688004c90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- 172.21.15.86:0/469447940 >> 172.21.15.86:0/469447940 conn(0x7f169c0aaae0 msgr2=0x7f169c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- 172.21.15.86:0/469447940 shutdown_connections 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- 172.21.15.86:0/469447940 wait complete. 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: monmap: 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:51.135 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 Processor -- start 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- start start 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: init 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:27:51.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f169c1d0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-c con 0x7f1688004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f169c1d0970 0x7f169c125750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-b con 0x7f169c1d0970 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): picked mon.noname-a con 0x7f169c1d0db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0a6ae0 con 0x7f169c1d0db0 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0a6960 con 0x7f169c1d0970 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f169c1386b0 con 0x7f1688004810 2024-09-17T15:27:51.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f16a190c640 10 monclient(hunting): authenticate will time out at 1004.090862s 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f169c1d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54602/0 (socket says 172.21.15.86:54602) 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f169c1d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36972/0 (socket says 172.21.15.86:36972) 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 learned_addr learned my addr 172.21.15.86:0/1483221137 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): get_auth_request con 0x7f169c1d0db0 auth_method 0 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f1693fff640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f169c1d0970 msgr2=0x7f169c125750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): get_auth_request con 0x7f1688004810 auth_method 0 2024-09-17T15:27:51.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f1693fff640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f169c1d0970 0x7f169c125750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 msgr2=0x7f169c1d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f169c1d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f169c1d0970 msgr2=0x7f169c125750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f169c1d0970 0x7f169c125750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c128f80 con 0x7f169c1d0db0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.123+0000 7f169affd640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1688004810 0x7f169c1d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 msgr2=0x7f169c125c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169b7fe640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f169c06fc80 tx=0x7f1694009e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694015670 con 0x7f169c1d0db0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: dump: 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1694010070 con 0x7f169c1d0db0 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 5 monclient: authenticate success, global_id 4338 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 monclient: _renew_subs 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c0a18d0 con 0x7f169c1d0db0 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1694015a70 con 0x7f169c1d0db0 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 10 monclient: dump: 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1699ffb640 4 set_mon_vals no callback set 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 monclient: _renew_subs 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c1271c0 con 0x7f169c1d0db0 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 0x7f169c1265b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f169c138aa0 con 0x7f169c1261d0 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 0x7f169c1265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient: get_auth_request con 0x7f169c1261d0 auth_method 0 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f169401e3f0 con 0x7f169c1d0db0 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f169affd640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f169c126af0 con 0x7f169c1261d0 2024-09-17T15:27:51.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f167c03bb40 0x7f167c03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f1694050540 con 0x7f169c1d0db0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1693fff640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f167c03bb40 0x7f167c03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.127+0000 7f1693fff640 10 monclient: get_auth_request con 0x7f167c03bb40 auth_method 0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f169affd640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 msgr2=0x7f169c1265b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f169affd640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 0x7f169c1265b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f168c001ec0 tx=0x7f168c012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c015440 con 0x7f169c1261d0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 10 monclient: dump: 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.145 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f168c01a070 con 0x7f169c1261d0 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 msgr2=0x7f169c1265b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 0x7f169c1265b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f168c001ec0 tx=0x7f168c012690 comp rx=0 tx=0).stop 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 msgr2=0x7f169c1265b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1698ff9640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1261d0 0x7f169c1265b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.131+0000 7f1693fff640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f167c03bb40 0x7f167c03e000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1684006fd0 tx=0x7f1684005a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f16a190c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:27:51.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f16a190c640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 0x7f169c121140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f16a190c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f169c1295e0 con 0x7f169c120d30 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 0x7f169c121140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient: get_auth_request con 0x7f169c120d30 auth_method 0 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.135+0000 7f169affd640 1 -- 172.21.15.86:0/1483221137 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f169c127d20 con 0x7f169c120d30 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f169affd640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 msgr2=0x7f169c121140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f169affd640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 0x7f169c121140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f168c016000 tx=0x7f168c007b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c015d90 con 0x7f169c120d30 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 10 monclient: dump: 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f168c015d90 con 0x7f169c120d30 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 msgr2=0x7f169c121140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 0x7f169c121140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f168c016000 tx=0x7f168c007b50 comp rx=0 tx=0).stop 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 msgr2=0x7f169c121140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f1698ff9640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c120d30 0x7f169c121140 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f168c016000 tx=0x7f168c007b50 comp rx=0 tx=0).stop 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:51.151 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 12, 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 43783, 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.152 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:27:51.153 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.154 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:27:51.155 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.156 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:27:51.157 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.158 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:27:51.161 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:27:51.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.163 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.165 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:27:51.165 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f167c03bb40 msgr2=0x7f167c03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f167c03bb40 0x7f167c03e000 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1684006fd0 tx=0x7f1684005a90 comp rx=0 tx=0).stop 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 10 monclient: shutdown 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 msgr2=0x7f169c125c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f169b7fe640 1 -- 172.21.15.86:0/1483221137 reap_dead start 2024-09-17T15:27:51.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f169c06fc80 tx=0x7f1694009e70 comp rx=0 tx=0).stop 2024-09-17T15:27:51.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 shutdown_connections 2024-09-17T15:27:51.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 --2- 172.21.15.86:0/1483221137 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f169c1d0db0 0x7f169c125c90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 >> 172.21.15.86:0/1483221137 conn(0x7f169c0aaae0 msgr2=0x7f169c0a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 shutdown_connections 2024-09-17T15:27:51.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.139+0000 7f16a190c640 1 -- 172.21.15.86:0/1483221137 wait complete. 2024-09-17T15:27:51.167 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:27:51.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:27:51.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:27:51.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 10 monclient: get_monmap_and_config 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 10 monclient: monmap: 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:51.271087+0000 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:51.271087+0000 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:51.269 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:51.270 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 1 Processor -- start 2024-09-17T15:27:51.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 1 -- start start 2024-09-17T15:27:51.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.267+0000 7f07423ef640 10 monclient: init 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f073c0a0c70 0x7f073c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-a con 0x7f073c0a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f073c0a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:27:51.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 0x7f073c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-c con 0x7f073c136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f073c0a2130 con 0x7f073c0a1590 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f073c0a2410 con 0x7f073c0a0c70 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f073c0a26f0 con 0x7f073c136160 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f07423ef640 10 monclient(hunting): authenticate will time out at 1004.236283s 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f073c0a0c70 0x7f073c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f073c0a0c70 0x7f073c0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54622/0 (socket says 172.21.15.86:54622) 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073b7fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f073c0a1590 legacy=0x7f073c135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:27:51.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 -- 172.21.15.86:0/1011175497 learned_addr learned my addr 172.21.15.86:0/1011175497 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): get_auth_request con 0x7f073c0a0c70 auth_method 0 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f074098c640 1 -- 172.21.15.86:0/1011175497 >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 msgr2=0x7f073c13df30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f074098c640 1 --2- 172.21.15.86:0/1011175497 >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 0x7f073c13df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 -- 172.21.15.86:0/1011175497 >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 msgr2=0x7f073c13df30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 --2- 172.21.15.86:0/1011175497 >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 0x7f073c13df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 -- 172.21.15.86:0/1011175497 >> v1:172.21.15.203:6789/0 conn(0x7f073c0a1590 legacy=0x7f073c135a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 monclient: found mon.noname-a 2024-09-17T15:27:51.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.271+0000 7f073bfff640 1 -- 172.21.15.86:0/1011175497 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c09f9f0 con 0x7f073c0a0c70 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 -- 172.21.15.86:0/1011175497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a0c70 msgr2=0x7f073c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 --2- 172.21.15.86:0/1011175497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a0c70 0x7f073c0a1050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f073c06f840 tx=0x7f073401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce95a1f0c922870f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 1 -- 172.21.15.86:0/1011175497 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340a5770 con 0x7f073c0a0c70 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: dump: 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 1 -- 172.21.15.86:0/1011175497 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07340a5910 con 0x7f073c0a0c70 2024-09-17T15:27:51.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 5 monclient: authenticate success, global_id 4347 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 1 -- 172.21.15.86:0/1011175497 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340a5ce0 con 0x7f073c0a0c70 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 4 set_mon_vals no callback set 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073affd640 10 monclient: dump: 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: shutdown 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- 172.21.15.86:0/1011175497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a0c70 msgr2=0x7f073c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- 172.21.15.86:0/1011175497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a0c70 0x7f073c0a1050 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f073c06f840 tx=0x7f073401c8d0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- 172.21.15.86:0/1011175497 shutdown_connections 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- 172.21.15.86:0/1011175497 >> v2:172.21.15.203:3300/0 conn(0x7f073c136160 0x7f073c13df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- 172.21.15.86:0/1011175497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a0c70 0x7f073c0a1050 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- 172.21.15.86:0/1011175497 >> 172.21.15.86:0/1011175497 conn(0x7f073c0a2f40 msgr2=0x7f073c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- 172.21.15.86:0/1011175497 shutdown_connections 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- 172.21.15.86:0/1011175497 wait complete. 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: monmap: 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:51.285 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 Processor -- start 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- start start 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: init 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 0x7f073c1cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-c con 0x7f073c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a1590 0x7f073c11e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-a con 0x7f073c0a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f073c136160 0x7f073c135310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): picked mon.noname-b con 0x7f073c136160 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f073c0a00d0 con 0x7f073c0a1590 2024-09-17T15:27:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f073c09ff50 con 0x7f073c136160 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f073c140c90 con 0x7f073c0a0c70 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f07423ef640 10 monclient(hunting): authenticate will time out at 1004.243225s 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 0x7f073c1cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a1590 0x7f073c11e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 0x7f073c1cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36980/0 (socket says 172.21.15.86:36980) 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 -- 172.21.15.86:0/1348476597 learned_addr learned my addr 172.21.15.86:0/1348476597 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): get_auth_request con 0x7f073c0a0c70 auth_method 0 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): get_auth_request con 0x7f073c0a1590 auth_method 0 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f074098c640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f073c136160 msgr2=0x7f073c135310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f074098c640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f073c136160 0x7f073c135310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f073c136160 msgr2=0x7f073c135310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:51.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f073c136160 0x7f073c135310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a1590 msgr2=0x7f073c11e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a1590 0x7f073c11e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 monclient: found mon.noname-c 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073bfff640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f073c1ceda0 con 0x7f073c0a0c70 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.275+0000 7f073b7fe640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f073c0a1590 0x7f073c11e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073bfff640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 msgr2=0x7f073c1cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073bfff640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 0x7f073c1cdea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f073401d170 tx=0x7f0734019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340bc020 con 0x7f073c0a0c70 2024-09-17T15:27:51.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: dump: 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07340b8880 con 0x7f073c0a0c70 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 5 monclient: authenticate success, global_id 4334 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 monclient: _renew_subs 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073c0700e0 con 0x7f073c0a0c70 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340a5770 con 0x7f073c0a0c70 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 10 monclient: dump: 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 monclient: _renew_subs 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f073c11f400 con 0x7f073c0a0c70 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0739ffb640 4 set_mon_vals no callback set 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 0x7f073c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f073c0a1050 con 0x7f073c135850 2024-09-17T15:27:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 0x7f073c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient: get_auth_request con 0x7f073c135850 auth_method 0 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f07340bf020 con 0x7f073c0a0c70 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f073b7fe640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f073c09d3e0 con 0x7f073c135850 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f072003bb10 0x7f072003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 4374+0+0 (secure 0 0 0) 0x7f07340eab30 con 0x7f073c0a0c70 2024-09-17T15:27:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f074098c640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f072003bb10 0x7f072003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.279+0000 7f074098c640 10 monclient: get_auth_request con 0x7f072003bb10 auth_method 0 2024-09-17T15:27:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f073b7fe640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 msgr2=0x7f073c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f073b7fe640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 0x7f073c11c8c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f0724005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724016070 con 0x7f073c135850 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 10 monclient: dump: 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f072400ed00 con 0x7f073c135850 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 msgr2=0x7f073c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 0x7f073c11c8c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f0724005850 comp rx=0 tx=0).stop 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 msgr2=0x7f073c11c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f0738ff9640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c135850 0x7f073c11c8c0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f0724005850 comp rx=0 tx=0).stop 2024-09-17T15:27:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.283+0000 7f074098c640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f072003bb10 0x7f072003dfd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f073c071410 tx=0x7f072c006cc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f07423ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f07423ef640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 0x7f073c11d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f07423ef640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f073c11d750 con 0x7f073c11ce00 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 0x7f073c11d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient: get_auth_request con 0x7f073c11ce00 auth_method 0 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 824 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.287+0000 7f073b7fe640 1 -- 172.21.15.86:0/1348476597 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f073c06d4f0 con 0x7f073c11ce00 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f073b7fe640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 msgr2=0x7f073c11d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f073b7fe640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 0x7f073c11d210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f07240059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0724016040 con 0x7f073c11ce00 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 10 monclient: dump: 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.299 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f0724005c60 con 0x7f073c11ce00 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 msgr2=0x7f073c11d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 0x7f073c11d210 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f07240059e0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 msgr2=0x7f073c11d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:27:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f0738ff9640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c11ce00 0x7f073c11d210 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0724003940 tx=0x7f07240059e0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 12, 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 43935, 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:27:51.301 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.303 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:27:51.304 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:27:51.305 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:27:51.306 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:27:51.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:27:51.311 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:27:51.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:27:51.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:27:51.314 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:27:51.314 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f072003bb10 msgr2=0x7f072003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f072003bb10 0x7f072003dfd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f073c071410 tx=0x7f072c006cc0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 10 monclient: shutdown 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 msgr2=0x7f073c1cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 --2- 172.21.15.86:0/1348476597 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f073c0a0c70 0x7f073c1cdea0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f073401d170 tx=0x7f0734019670 comp rx=0 tx=0).stop 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f074098c640 1 -- 172.21.15.86:0/1348476597 reap_dead start 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f074098c640 1 -- 172.21.15.86:0/1348476597 reap_dead start 2024-09-17T15:27:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 shutdown_connections 2024-09-17T15:27:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 >> 172.21.15.86:0/1348476597 conn(0x7f073c0a2f40 msgr2=0x7f073c0714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 shutdown_connections 2024-09-17T15:27:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.291+0000 7f07423ef640 1 -- 172.21.15.86:0/1348476597 wait complete. 2024-09-17T15:27:51.316 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:27:51.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:27:51.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.695+0000 7f51ad7fa640 1 -- 172.21.15.86:0/399667852 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]=0 pool 'unique_pool_0' created v19) ==== 130+0+0 (secure 0 0 0) 0x7f519c04e780 con 0x7f51b00a54c0 2024-09-17T15:27:51.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-17T15:27:51.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51ad7fa640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' created 2024-09-17T15:27:51.697 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_0' created 2024-09-17T15:27:51.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 msgr2=0x7f519403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 0x7f519403e050 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f51b006f890 tx=0x7f51a40a6000 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 10 monclient: shutdown 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 msgr2=0x7f51b0136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f519c00b550 tx=0x7f519c00ba10 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 shutdown_connections 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f519403bb90 0x7f519403e050 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51b0137ea0 0x7f51b013d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51b00a54c0 0x7f51b0136fb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 --2- 172.21.15.86:0/399667852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51b00a50e0 0x7f51b01d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 >> 172.21.15.86:0/399667852 conn(0x7f51b00aaae0 msgr2=0x7f51b00a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 shutdown_connections 2024-09-17T15:27:51.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.699+0000 7f51b6f61640 1 -- 172.21.15.86:0/399667852 wait complete. 2024-09-17T15:27:51.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: get_monmap_and_config 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: monmap: 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:27:51.858265+0000 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:27:51.858265+0000 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:27:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 Processor -- start 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 -- start start 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: init 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0bc0a0c00 addr v1:172.21.15.86:6789/0 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 0x7fb0bc13aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-c con 0x7fb0bc06f920 addr v2:172.21.15.203:3300/0 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0bc0a16f0 0x7fb0bc13d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb0bc0a16f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:27:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0bc0a2130 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb0bc0a2410 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb0bc0a26f0 con 0x7fb0bc06f920 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0c18d9640 10 monclient(hunting): authenticate will time out at 1004.823689s 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0baffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb0bc0a0c00 0x7fb0bc0a0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48214/0 (socket says 172.21.15.86:48214) 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0baffd640 1 -- 172.21.15.86:0/2415408483 learned_addr learned my addr 172.21.15.86:0/2415408483 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0bb7fe640 1 --2- 172.21.15.86:0/2415408483 >> v2:172.21.15.86:3300/0 conn(0x7fb0bc0a16f0 0x7fb0bc13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/2415408483 >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 msgr2=0x7fb0bc13aec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0bb7fe640 10 monclient(hunting): get_auth_request con 0x7fb0bc0a16f0 auth_method 0 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0ba7fc640 1 --2- 172.21.15.86:0/2415408483 >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 0x7fb0bc13aec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.855+0000 7fb0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 1 -- 172.21.15.86:0/2415408483 >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 msgr2=0x7fb0bc13aec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 1 --2- 172.21.15.86:0/2415408483 >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 0x7fb0bc13aec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 1 -- 172.21.15.86:0/2415408483 >> v1:172.21.15.86:6789/0 conn(0x7fb0bc0a0c00 legacy=0x7fb0bc0a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 1 monclient: found mon.noname-a 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.859+0000 7fb0bb7fe640 1 -- 172.21.15.86:0/2415408483 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc0a29d0 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 1 -- 172.21.15.86:0/2415408483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a16f0 msgr2=0x7fb0bc13d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 1 --2- 172.21.15.86:0/2415408483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a16f0 0x7fb0bc13d410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc06eb30 tx=0x7fb0b401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684a460f2753a08e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 1 -- 172.21.15.86:0/2415408483 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40b2070 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: dump: 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.863 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 1 -- 172.21.15.86:0/2415408483 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b40a5990 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 5 monclient: authenticate success, global_id 4353 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: get_monmap_and_config success 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 1 -- 172.21.15.86:0/2415408483 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40a5d60 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 4 set_mon_vals no callback set 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0b9ffb640 10 monclient: dump: 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: shutdown 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- 172.21.15.86:0/2415408483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a16f0 msgr2=0x7fb0bc13d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/2415408483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a16f0 0x7fb0bc13d410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc06eb30 tx=0x7fb0b401c810 comp rx=0 tx=0).stop 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- 172.21.15.86:0/2415408483 shutdown_connections 2024-09-17T15:27:51.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/2415408483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a16f0 0x7fb0bc13d410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/2415408483 >> v2:172.21.15.203:3300/0 conn(0x7fb0bc06f920 0x7fb0bc13aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- 172.21.15.86:0/2415408483 >> 172.21.15.86:0/2415408483 conn(0x7fb0bc0a2d00 msgr2=0x7fb0bc0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:51.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- 172.21.15.86:0/2415408483 shutdown_connections 2024-09-17T15:27:51.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- 172.21.15.86:0/2415408483 wait complete. 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: build_initial_monmap 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: monmap: 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:27:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 Processor -- start 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- start start 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: init 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: _reopen_session rank -1 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 0x7fb0bc1cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-b con 0x7fb0bc06f920 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-a con 0x7fb0bc0a0c00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0bc0a16f0 0x7fb0bc138f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): picked mon.noname-c con 0x7fb0bc0a16f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc140500 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb0bc140380 con 0x7fb0bc06f920 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb0bc140680 con 0x7fb0bc0a16f0 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): _renew_subs 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0c18d9640 10 monclient(hunting): authenticate will time out at 1004.831277s 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54636/0 (socket says 172.21.15.86:54636) 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/1113729674 learned_addr learned my addr 172.21.15.86:0/1113729674 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): get_auth_request con 0x7fb0bc0a0c00 auth_method 0 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0baffd640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 msgr2=0x7fb0bc1cda90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:27:51.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0bc0a16f0 0x7fb0bc138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0baffd640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 0x7fb0bc1cda90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): get_auth_request con 0x7fb0bc0a16f0 auth_method 0 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-17T15:27:51.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0bc0a16f0 msgr2=0x7fb0bc138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0bc0a16f0 0x7fb0bc138f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 msgr2=0x7fb0bc1cda90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 0x7fb0bc1cda90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 monclient: found mon.noname-a 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.863+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/1113729674 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0bc1ce9a0 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0ba7fc640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 msgr2=0x7fb0bc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0ba7fc640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b530 tx=0x7fb0ac00b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac013020 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:27:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: dump: 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: _finish_auth 0 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:27:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ac005070 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 5 monclient: authenticate success, global_id 4356 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _renew_subs 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0bc13d610 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac005460 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 10 monclient: dump: 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.876 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _renew_subs 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0b97fa640 4 set_mon_vals no callback set 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0bc13c7e0 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:51.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0bc13ce00 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb0ac016020 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 0x7fb0a803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.867+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4698+0+0 (secure 0 0 0) 0x7fb0ac04fa90 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.871+0000 7fb0baffd640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 0x7fb0a803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.871+0000 7fb0baffd640 10 monclient: get_auth_request con 0x7fb0a803bb60 auth_method 0 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.871+0000 7fb0baffd640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 0x7fb0a803e020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0004820 tx=0x7fb0b0007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:27:51.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.871+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:27:51.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.875+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb0bc13ce00 con 0x7fb0bc0a0c00 2024-09-17T15:27:51.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.875+0000 7fb0a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:27:51.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:51.875+0000 7fb0a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:27:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.047+0000 7fb0c18d9640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-17T15:27:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.047+0000 7fb0c18d9640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-17T15:27:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.047+0000 7fb0c18d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:27:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.047+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} v 0) -- 0x7fb0bc13b5a0 con 0x7fb0bc0a0c00 2024-09-17T15:27:52.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.699+0000 7fb0a3fff640 1 -- 172.21.15.86:0/1113729674 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]=0 enabled application 'rados' on pool 'unique_pool_0' v20) ==== 197+0+0 (secure 0 0 0) 0x7fb0ac01d020 con 0x7fb0bc0a0c00 2024-09-17T15:27:52.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.699+0000 7fb0a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-17T15:27:52.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.699+0000 7fb0a3fff640 10 monclient: _finish_command 2 = system:0 enabled application 'rados' on pool 'unique_pool_0' 2024-09-17T15:27:52.700 INFO:teuthology.orchestra.run.smithi086.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-17T15:27:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 msgr2=0x7fb0a803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 0x7fb0a803e020 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0004820 tx=0x7fb0b0007b30 comp rx=0 tx=0).stop 2024-09-17T15:27:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 10 monclient: shutdown 2024-09-17T15:27:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 msgr2=0x7fb0bc131c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac00b530 tx=0x7fb0ac00b9f0 comp rx=0 tx=0).stop 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 shutdown_connections 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb0a803bb60 0x7fb0a803e020 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0bc0a16f0 0x7fb0bc138f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0bc0a0c00 0x7fb0bc131c60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 --2- 172.21.15.86:0/1113729674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0bc06f920 0x7fb0bc1cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 >> 172.21.15.86:0/1113729674 conn(0x7fb0bc0a2d00 msgr2=0x7fb0bc135e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 shutdown_connections 2024-09-17T15:27:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:27:52.703+0000 7fb0c18d9640 1 -- 172.21.15.86:0/1113729674 wait complete. 2024-09-17T15:27:53.732 DEBUG:teuthology.orchestra.run.smithi203:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight read -> 100 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight write -> 50 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight delete -> 50 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight snap_create -> 50 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight snap_remove -> 50 2024-09-17T15:27:53.755 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight rollback -> 50 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight copy_from -> 50 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout:adding op weight write_excl -> 50 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout:ceph version 19.3.0-4975-g1889a165 (1889a165b5128787ec693450a3d1c5df5d6dc203) squid (dev) 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout:Configuration: 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Number of operations: 4000 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Number of objects: 50 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Max in flight operations: 16 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Object size (in bytes): 4000000 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Write stride min: 400000 2024-09-17T15:27:53.756 INFO:tasks.rados.rados.0.smithi203.stdout: Write stride max: 800000 2024-09-17T15:27:54.724 INFO:tasks.rados.rados.0.smithi203.stdout:1: write initial oid 1 2024-09-17T15:27:54.724 INFO:tasks.rados.rados.0.smithi203.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-17T15:27:54.731 INFO:tasks.rados.rados.0.smithi203.stdout:1: writing smithi20328086-1 from 416807 to 892056 tid 1 2024-09-17T15:27:54.740 INFO:tasks.rados.rados.0.smithi203.stdout:1: writing smithi20328086-1 from 1542129 to 2016807 tid 2 2024-09-17T15:27:54.740 INFO:tasks.rados.rados.0.smithi203.stdout:2: write initial oid 2 2024-09-17T15:27:54.740 INFO:tasks.rados.rados.0.smithi203.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-17T15:27:54.748 INFO:tasks.rados.rados.0.smithi203.stdout:2: writing smithi20328086-2 from 416807 to 892056 tid 1 2024-09-17T15:27:54.757 INFO:tasks.rados.rados.0.smithi203.stdout:2: writing smithi20328086-2 from 1542129 to 2016807 tid 2 2024-09-17T15:27:54.757 INFO:tasks.rados.rados.0.smithi203.stdout:3: write initial oid 3 2024-09-17T15:27:54.757 INFO:tasks.rados.rados.0.smithi203.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-17T15:27:54.765 INFO:tasks.rados.rados.0.smithi203.stdout:3: writing smithi20328086-3 from 433614 to 984112 tid 1 2024-09-17T15:27:54.773 INFO:tasks.rados.rados.0.smithi203.stdout:3: writing smithi20328086-3 from 1600611 to 2033614 tid 2 2024-09-17T15:27:54.773 INFO:tasks.rados.rados.0.smithi203.stdout:4: write initial oid 4 2024-09-17T15:27:54.773 INFO:tasks.rados.rados.0.smithi203.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-17T15:27:54.781 INFO:tasks.rados.rados.0.smithi203.stdout:4: writing smithi20328086-4 from 450421 to 1076168 tid 1 2024-09-17T15:27:54.788 INFO:tasks.rados.rados.0.smithi203.stdout:4: writing smithi20328086-4 from 1659093 to 2050421 tid 2 2024-09-17T15:27:54.788 INFO:tasks.rados.rados.0.smithi203.stdout:5: write initial oid 5 2024-09-17T15:27:54.788 INFO:tasks.rados.rados.0.smithi203.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-17T15:27:54.796 INFO:tasks.rados.rados.0.smithi203.stdout:5: writing smithi20328086-5 from 467228 to 1168224 tid 1 2024-09-17T15:27:54.802 INFO:tasks.rados.rados.0.smithi203.stdout:5: writing smithi20328086-5 from 1717575 to 2067228 tid 2 2024-09-17T15:27:54.802 INFO:tasks.rados.rados.0.smithi203.stdout:6: write initial oid 6 2024-09-17T15:27:54.802 INFO:tasks.rados.rados.0.smithi203.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-17T15:27:54.811 INFO:tasks.rados.rados.0.smithi203.stdout:6: writing smithi20328086-6 from 484035 to 1260280 tid 1 2024-09-17T15:27:54.816 INFO:tasks.rados.rados.0.smithi203.stdout:6: writing smithi20328086-6 from 2059704 to 2084035 tid 2 2024-09-17T15:27:54.817 INFO:tasks.rados.rados.0.smithi203.stdout:7: write initial oid 7 2024-09-17T15:27:54.817 INFO:tasks.rados.rados.0.smithi203.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-17T15:27:54.823 INFO:tasks.rados.rados.0.smithi203.stdout:7: writing smithi20328086-7 from 500842 to 952336 tid 1 2024-09-17T15:27:54.830 INFO:tasks.rados.rados.0.smithi203.stdout:7: writing smithi20328086-7 from 1718186 to 2100842 tid 2 2024-09-17T15:27:54.830 INFO:tasks.rados.rados.0.smithi203.stdout:8: write initial oid 8 2024-09-17T15:27:54.830 INFO:tasks.rados.rados.0.smithi203.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-17T15:27:54.837 INFO:tasks.rados.rados.0.smithi203.stdout:8: writing smithi20328086-8 from 517649 to 1044392 tid 1 2024-09-17T15:27:54.843 INFO:tasks.rados.rados.0.smithi203.stdout:8: writing smithi20328086-8 from 1776668 to 2117649 tid 2 2024-09-17T15:27:54.843 INFO:tasks.rados.rados.0.smithi203.stdout:9: write initial oid 9 2024-09-17T15:27:54.843 INFO:tasks.rados.rados.0.smithi203.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-17T15:27:54.850 INFO:tasks.rados.rados.0.smithi203.stdout:9: writing smithi20328086-9 from 534456 to 1252801 tid 1 2024-09-17T15:27:54.855 INFO:tasks.rados.rados.0.smithi203.stdout:9: writing smithi20328086-9 from 1951503 to 2134456 tid 2 2024-09-17T15:27:54.855 INFO:tasks.rados.rados.0.smithi203.stdout:10: write initial oid 10 2024-09-17T15:27:54.855 INFO:tasks.rados.rados.0.smithi203.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-17T15:27:54.862 INFO:tasks.rados.rados.0.smithi203.stdout:10: writing smithi20328086-10 from 551263 to 1344857 tid 1 2024-09-17T15:27:54.867 INFO:tasks.rados.rados.0.smithi203.stdout:10: writing smithi20328086-10 from 1893632 to 2151263 tid 2 2024-09-17T15:27:54.867 INFO:tasks.rados.rados.0.smithi203.stdout:11: write initial oid 11 2024-09-17T15:27:54.867 INFO:tasks.rados.rados.0.smithi203.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-17T15:27:54.873 INFO:tasks.rados.rados.0.smithi203.stdout:11: writing smithi20328086-11 from 568070 to 1036913 tid 1 2024-09-17T15:27:54.879 INFO:tasks.rados.rados.0.smithi203.stdout:11: writing smithi20328086-11 from 1552114 to 2168070 tid 2 2024-09-17T15:27:54.879 INFO:tasks.rados.rados.0.smithi203.stdout:12: write initial oid 12 2024-09-17T15:27:54.879 INFO:tasks.rados.rados.0.smithi203.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-17T15:27:54.886 INFO:tasks.rados.rados.0.smithi203.stdout:12: writing smithi20328086-12 from 584877 to 1128969 tid 1 2024-09-17T15:27:54.891 INFO:tasks.rados.rados.0.smithi203.stdout:12: writing smithi20328086-12 from 1610596 to 2172599 tid 2 2024-09-17T15:27:54.892 INFO:tasks.rados.rados.0.smithi203.stdout:13: write initial oid 13 2024-09-17T15:27:54.892 INFO:tasks.rados.rados.0.smithi203.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-17T15:27:54.898 INFO:tasks.rados.rados.0.smithi203.stdout:13: writing smithi20328086-13 from 601684 to 1221025 tid 1 2024-09-17T15:27:54.904 INFO:tasks.rados.rados.0.smithi203.stdout:13: writing smithi20328086-13 from 1669078 to 2201684 tid 2 2024-09-17T15:27:54.904 INFO:tasks.rados.rados.0.smithi203.stdout:14: write initial oid 14 2024-09-17T15:27:54.904 INFO:tasks.rados.rados.0.smithi203.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-17T15:27:54.911 INFO:tasks.rados.rados.0.smithi203.stdout:14: writing smithi20328086-14 from 618491 to 1313081 tid 1 2024-09-17T15:27:54.916 INFO:tasks.rados.rados.0.smithi203.stdout:14: writing smithi20328086-14 from 2011207 to 2218491 tid 2 2024-09-17T15:27:54.917 INFO:tasks.rados.rados.0.smithi203.stdout:15: write initial oid 15 2024-09-17T15:27:54.917 INFO:tasks.rados.rados.0.smithi203.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-17T15:27:54.924 INFO:tasks.rados.rados.0.smithi203.stdout:15: writing smithi20328086-15 from 635298 to 1405137 tid 1 2024-09-17T15:27:54.929 INFO:tasks.rados.rados.0.smithi203.stdout:15: writing smithi20328086-15 from 2069689 to 2235298 tid 2 2024-09-17T15:27:54.929 INFO:tasks.rados.rados.0.smithi203.stdout:16: write initial oid 16 2024-09-17T15:27:54.929 INFO:tasks.rados.rados.0.smithi203.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-17T15:27:54.935 INFO:tasks.rados.rados.0.smithi203.stdout:16: writing smithi20328086-16 from 652105 to 1097193 tid 1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:16: writing smithi20328086-16 from 1728171 to 2181159 tid 2 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: finishing write tid 1 to smithi20328086-1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: finishing write tid 2 to smithi20328086-1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: finishing write tid 3 to smithi20328086-1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: finishing write tid 4 to smithi20328086-1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: oid 1 updating version 0 to 1 2024-09-17T15:27:54.942 INFO:tasks.rados.rados.0.smithi203.stdout:1: oid 1 updating version 1 to 2 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:1: oid 1 updating version 2 to 3 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:2: finishing write tid 1 to smithi20328086-2 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:1: done (15 left) 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:17: write initial oid 17 2024-09-17T15:27:54.943 INFO:tasks.rados.rados.0.smithi203.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-17T15:27:54.949 INFO:tasks.rados.rados.0.smithi203.stdout:17: writing smithi20328086-17 from 668912 to 1305602 tid 1 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout:17: writing smithi20328086-17 from 1903006 to 2268912 tid 2 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout:2: finishing write tid 2 to smithi20328086-2 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout:2: finishing write tid 3 to smithi20328086-2 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout:2: finishing write tid 4 to smithi20328086-2 2024-09-17T15:27:54.955 INFO:tasks.rados.rados.0.smithi203.stdout:2: oid 2 updating version 0 to 3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:2: oid 2 version 3 is already newer than 2 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:2: oid 2 version 3 is already newer than 1 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: finishing write tid 1 to smithi20328086-3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: finishing write tid 2 to smithi20328086-3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: finishing write tid 3 to smithi20328086-3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: finishing write tid 4 to smithi20328086-3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: oid 3 updating version 0 to 2 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: oid 3 version 2 is already newer than 1 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: oid 3 updating version 2 to 3 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-17T15:27:54.956 INFO:tasks.rados.rados.0.smithi203.stdout:4: finishing write tid 1 to smithi20328086-4 2024-09-17T15:27:54.957 INFO:tasks.rados.rados.0.smithi203.stdout:2: done (15 left) 2024-09-17T15:27:54.957 INFO:tasks.rados.rados.0.smithi203.stdout:3: done (14 left) 2024-09-17T15:27:54.957 INFO:tasks.rados.rados.0.smithi203.stdout:18: write initial oid 18 2024-09-17T15:27:54.957 INFO:tasks.rados.rados.0.smithi203.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-17T15:27:54.963 INFO:tasks.rados.rados.0.smithi203.stdout:18: writing smithi20328086-18 from 685719 to 1397658 tid 1 2024-09-17T15:27:54.968 INFO:tasks.rados.rados.0.smithi203.stdout:18: writing smithi20328086-18 from 1845135 to 2285719 tid 2 2024-09-17T15:27:54.968 INFO:tasks.rados.rados.0.smithi203.stdout:19: write initial oid 19 2024-09-17T15:27:54.968 INFO:tasks.rados.rados.0.smithi203.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-17T15:27:54.976 INFO:tasks.rados.rados.0.smithi203.stdout:19: writing smithi20328086-19 from 702526 to 1489714 tid 1 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:19: writing smithi20328086-19 from 1903617 to 2302526 tid 2 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: finishing write tid 2 to smithi20328086-4 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: finishing write tid 3 to smithi20328086-4 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: finishing write tid 4 to smithi20328086-4 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: oid 4 updating version 0 to 3 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: oid 4 version 3 is already newer than 2 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: oid 4 version 3 is already newer than 1 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-17T15:27:54.981 INFO:tasks.rados.rados.0.smithi203.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: finishing write tid 1 to smithi20328086-5 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: finishing write tid 2 to smithi20328086-5 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: finishing write tid 3 to smithi20328086-5 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: finishing write tid 4 to smithi20328086-5 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: oid 5 updating version 0 to 1 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: oid 5 updating version 1 to 2 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: oid 5 updating version 2 to 3 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:6: finishing write tid 1 to smithi20328086-6 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:4: done (15 left) 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:5: done (14 left) 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:20: write initial oid 20 2024-09-17T15:27:54.982 INFO:tasks.rados.rados.0.smithi203.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-17T15:27:54.988 INFO:tasks.rados.rados.0.smithi203.stdout:20: writing smithi20328086-20 from 719333 to 1181770 tid 1 2024-09-17T15:27:54.994 INFO:tasks.rados.rados.0.smithi203.stdout:20: writing smithi20328086-20 from 1962099 to 2319333 tid 2 2024-09-17T15:27:54.994 INFO:tasks.rados.rados.0.smithi203.stdout:21: write initial oid 21 2024-09-17T15:27:54.994 INFO:tasks.rados.rados.0.smithi203.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-17T15:27:55.001 INFO:tasks.rados.rados.0.smithi203.stdout:21: writing smithi20328086-21 from 736140 to 1273826 tid 1 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:21: writing smithi20328086-21 from 2020581 to 2336140 tid 2 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: finishing write tid 2 to smithi20328086-6 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: finishing write tid 3 to smithi20328086-6 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: finishing write tid 4 to smithi20328086-6 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: oid 6 updating version 0 to 5 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: oid 6 updating version 5 to 6 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: oid 6 version 6 is already newer than 4 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-17T15:27:55.008 INFO:tasks.rados.rados.0.smithi203.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:7: finishing write tid 1 to smithi20328086-7 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:7: finishing write tid 2 to smithi20328086-7 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:7: finishing write tid 3 to smithi20328086-7 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:6: done (15 left) 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:22: write initial oid 22 2024-09-17T15:27:55.009 INFO:tasks.rados.rados.0.smithi203.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-17T15:27:55.015 INFO:tasks.rados.rados.0.smithi203.stdout:22: writing smithi20328086-22 from 752947 to 1365882 tid 1 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:22: writing smithi20328086-22 from 1962710 to 2352947 tid 2 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:7: finishing write tid 4 to smithi20328086-7 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:7: oid 7 updating version 0 to 6 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:7: oid 7 version 6 is already newer than 5 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:7: oid 7 version 6 is already newer than 4 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-17T15:27:55.021 INFO:tasks.rados.rados.0.smithi203.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-17T15:27:55.022 INFO:tasks.rados.rados.0.smithi203.stdout:8: finishing write tid 1 to smithi20328086-8 2024-09-17T15:27:55.022 INFO:tasks.rados.rados.0.smithi203.stdout:7: done (15 left) 2024-09-17T15:27:55.022 INFO:tasks.rados.rados.0.smithi203.stdout:23: write initial oid 23 2024-09-17T15:27:55.022 INFO:tasks.rados.rados.0.smithi203.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-17T15:27:55.029 INFO:tasks.rados.rados.0.smithi203.stdout:23: writing smithi20328086-23 from 769754 to 1457938 tid 1 2024-09-17T15:27:55.034 INFO:tasks.rados.rados.0.smithi203.stdout:23: writing smithi20328086-23 from 2021192 to 2369754 tid 2 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: finishing write tid 2 to smithi20328086-8 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: finishing write tid 3 to smithi20328086-8 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: finishing write tid 4 to smithi20328086-8 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: oid 8 updating version 0 to 2 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: oid 8 version 2 is already newer than 1 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: oid 8 updating version 2 to 3 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:9: finishing write tid 1 to smithi20328086-9 2024-09-17T15:27:55.035 INFO:tasks.rados.rados.0.smithi203.stdout:9: finishing write tid 2 to smithi20328086-9 2024-09-17T15:27:55.036 INFO:tasks.rados.rados.0.smithi203.stdout:9: finishing write tid 3 to smithi20328086-9 2024-09-17T15:27:55.036 INFO:tasks.rados.rados.0.smithi203.stdout:8: done (15 left) 2024-09-17T15:27:55.036 INFO:tasks.rados.rados.0.smithi203.stdout:24: write initial oid 24 2024-09-17T15:27:55.036 INFO:tasks.rados.rados.0.smithi203.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-17T15:27:55.041 INFO:tasks.rados.rados.0.smithi203.stdout:24: writing smithi20328086-24 from 786561 to 1266347 tid 1 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:24: writing smithi20328086-24 from 1796027 to 2263691 tid 2 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:9: finishing write tid 4 to smithi20328086-9 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:9: oid 9 updating version 0 to 1 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:9: oid 9 updating version 1 to 3 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:9: oid 9 version 3 is already newer than 2 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:10: finishing write tid 1 to smithi20328086-10 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:10: finishing write tid 2 to smithi20328086-10 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:10: finishing write tid 3 to smithi20328086-10 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:10: finishing write tid 4 to smithi20328086-10 2024-09-17T15:27:55.048 INFO:tasks.rados.rados.0.smithi203.stdout:10: oid 10 updating version 0 to 2 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:10: oid 10 updating version 2 to 3 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:10: oid 10 version 3 is already newer than 1 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:11: finishing write tid 1 to smithi20328086-11 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:9: done (15 left) 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:10: done (14 left) 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:25: write initial oid 25 2024-09-17T15:27:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-17T15:27:55.053 INFO:tasks.rados.rados.0.smithi203.stdout:25: writing smithi20328086-25 from 403368 to 958403 tid 1 2024-09-17T15:27:55.060 INFO:tasks.rados.rados.0.smithi203.stdout:25: writing smithi20328086-25 from 1454509 to 2182184 tid 2 2024-09-17T15:27:55.060 INFO:tasks.rados.rados.0.smithi203.stdout:26: write initial oid 26 2024-09-17T15:27:55.060 INFO:tasks.rados.rados.0.smithi203.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-17T15:27:55.066 INFO:tasks.rados.rados.0.smithi203.stdout:26: writing smithi20328086-26 from 420175 to 1050459 tid 1 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:26: writing smithi20328086-26 from 1796638 to 2267971 tid 2 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: finishing write tid 2 to smithi20328086-11 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: finishing write tid 3 to smithi20328086-11 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: finishing write tid 4 to smithi20328086-11 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: oid 11 updating version 0 to 3 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: oid 11 version 3 is already newer than 2 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: oid 11 version 3 is already newer than 1 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:12: finishing write tid 1 to smithi20328086-12 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:12: finishing write tid 2 to smithi20328086-12 2024-09-17T15:27:55.074 INFO:tasks.rados.rados.0.smithi203.stdout:12: finishing write tid 3 to smithi20328086-12 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:12: finishing write tid 4 to smithi20328086-12 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:12: oid 12 updating version 0 to 2 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:12: oid 12 version 2 is already newer than 1 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:12: oid 12 updating version 2 to 3 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: finishing write tid 1 to smithi20328086-13 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: finishing write tid 2 to smithi20328086-13 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: finishing write tid 3 to smithi20328086-13 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: finishing write tid 4 to smithi20328086-13 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: oid 13 updating version 0 to 5 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: oid 13 version 5 is already newer than 4 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: oid 13 updating version 5 to 6 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-17T15:27:55.075 INFO:tasks.rados.rados.0.smithi203.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: finishing write tid 1 to smithi20328086-14 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: finishing write tid 2 to smithi20328086-14 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: finishing write tid 3 to smithi20328086-14 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: finishing write tid 4 to smithi20328086-14 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: oid 14 updating version 0 to 9 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: oid 14 version 9 is already newer than 8 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: oid 14 version 9 is already newer than 7 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: finishing write tid 1 to smithi20328086-15 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: finishing write tid 2 to smithi20328086-15 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: finishing write tid 3 to smithi20328086-15 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: finishing write tid 4 to smithi20328086-15 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: oid 15 updating version 0 to 4 2024-09-17T15:27:55.076 INFO:tasks.rados.rados.0.smithi203.stdout:15: oid 15 updating version 4 to 5 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:15: oid 15 updating version 5 to 6 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:16: finishing write tid 1 to smithi20328086-16 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:11: done (15 left) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:12: done (14 left) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:13: done (13 left) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:14: done (12 left) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:15: done (11 left) 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:27: write initial oid 27 2024-09-17T15:27:55.077 INFO:tasks.rados.rados.0.smithi203.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-17T15:27:55.079 INFO:tasks.rados.rados.0.smithi203.stdout:27: writing smithi20328086-27 from 436982 to 1142515 tid 1 2024-09-17T15:27:55.087 INFO:tasks.rados.rados.0.smithi203.stdout:27: writing smithi20328086-27 from 1855120 to 2436982 tid 2 2024-09-17T15:27:55.087 INFO:tasks.rados.rados.0.smithi203.stdout:28: write initial oid 28 2024-09-17T15:27:55.087 INFO:tasks.rados.rados.0.smithi203.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-17T15:27:55.094 INFO:tasks.rados.rados.0.smithi203.stdout:28: writing smithi20328086-28 from 453789 to 1234571 tid 1 2024-09-17T15:27:55.100 INFO:tasks.rados.rados.0.smithi203.stdout:28: writing smithi20328086-28 from 1913602 to 2388604 tid 2 2024-09-17T15:27:55.100 INFO:tasks.rados.rados.0.smithi203.stdout:29: write initial oid 29 2024-09-17T15:27:55.101 INFO:tasks.rados.rados.0.smithi203.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-17T15:27:55.105 INFO:tasks.rados.rados.0.smithi203.stdout:29: writing smithi20328086-29 from 470596 to 926627 tid 1 2024-09-17T15:27:55.113 INFO:tasks.rados.rados.0.smithi203.stdout:29: writing smithi20328086-29 from 1572084 to 2190744 tid 2 2024-09-17T15:27:55.113 INFO:tasks.rados.rados.0.smithi203.stdout:30: write initial oid 30 2024-09-17T15:27:55.113 INFO:tasks.rados.rados.0.smithi203.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-17T15:27:55.120 INFO:tasks.rados.rados.0.smithi203.stdout:30: writing smithi20328086-30 from 487403 to 1018683 tid 1 2024-09-17T15:27:55.124 INFO:tasks.rados.rados.0.smithi203.stdout:30: writing smithi20328086-30 from 1514213 to 1992884 tid 2 2024-09-17T15:27:55.124 INFO:tasks.rados.rados.0.smithi203.stdout:31: write initial oid 31 2024-09-17T15:27:55.124 INFO:tasks.rados.rados.0.smithi203.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-17T15:27:55.130 INFO:tasks.rados.rados.0.smithi203.stdout:31: writing smithi20328086-31 from 504210 to 1110739 tid 1 2024-09-17T15:27:55.137 INFO:tasks.rados.rados.0.smithi203.stdout:31: writing smithi20328086-31 from 1572695 to 2195024 tid 2 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: finishing write tid 2 to smithi20328086-16 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: finishing write tid 3 to smithi20328086-16 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: finishing write tid 4 to smithi20328086-16 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: oid 16 updating version 0 to 1 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: oid 16 updating version 1 to 3 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: oid 16 version 3 is already newer than 2 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:17: finishing write tid 1 to smithi20328086-17 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:17: finishing write tid 2 to smithi20328086-17 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:17: finishing write tid 3 to smithi20328086-17 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:17: finishing write tid 4 to smithi20328086-17 2024-09-17T15:27:55.138 INFO:tasks.rados.rados.0.smithi203.stdout:17: oid 17 updating version 0 to 6 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:17: oid 17 version 6 is already newer than 5 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:17: oid 17 version 6 is already newer than 4 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: finishing write tid 1 to smithi20328086-18 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: finishing write tid 2 to smithi20328086-18 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: finishing write tid 3 to smithi20328086-18 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: finishing write tid 4 to smithi20328086-18 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: oid 18 updating version 0 to 5 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: oid 18 updating version 5 to 6 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: oid 18 version 6 is already newer than 4 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:19: finishing write tid 1 to smithi20328086-19 2024-09-17T15:27:55.139 INFO:tasks.rados.rados.0.smithi203.stdout:19: finishing write tid 2 to smithi20328086-19 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: finishing write tid 3 to smithi20328086-19 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: finishing write tid 4 to smithi20328086-19 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: oid 19 updating version 0 to 4 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: oid 19 updating version 4 to 5 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: oid 19 updating version 5 to 6 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: finishing write tid 1 to smithi20328086-20 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: finishing write tid 2 to smithi20328086-20 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: finishing write tid 3 to smithi20328086-20 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: finishing write tid 4 to smithi20328086-20 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: oid 20 updating version 0 to 12 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: oid 20 version 12 is already newer than 11 2024-09-17T15:27:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:20: oid 20 version 12 is already newer than 10 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 12 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: finishing write tid 1 to smithi20328086-21 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: finishing write tid 2 to smithi20328086-21 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: finishing write tid 3 to smithi20328086-21 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: finishing write tid 4 to smithi20328086-21 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: oid 21 updating version 0 to 2 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: oid 21 version 2 is already newer than 1 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: oid 21 updating version 2 to 3 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:23: finishing write tid 1 to smithi20328086-23 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:23: finishing write tid 2 to smithi20328086-23 2024-09-17T15:27:55.141 INFO:tasks.rados.rados.0.smithi203.stdout:23: finishing write tid 3 to smithi20328086-23 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: finishing write tid 4 to smithi20328086-23 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: oid 23 updating version 0 to 14 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: oid 23 version 14 is already newer than 13 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: oid 23 updating version 14 to 15 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:16: done (15 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:17: done (14 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:18: done (13 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:19: done (12 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:20: done (11 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:21: done (10 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:23: done (9 left) 2024-09-17T15:27:55.142 INFO:tasks.rados.rados.0.smithi203.stdout:32: write initial oid 32 2024-09-17T15:27:55.143 INFO:tasks.rados.rados.0.smithi203.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-17T15:27:55.144 INFO:tasks.rados.rados.0.smithi203.stdout:32: writing smithi20328086-32 from 521017 to 1319148 tid 1 2024-09-17T15:27:55.149 INFO:tasks.rados.rados.0.smithi203.stdout:32: writing smithi20328086-32 from 1747530 to 2229870 tid 2 2024-09-17T15:27:55.149 INFO:tasks.rados.rados.0.smithi203.stdout:33: write initial oid 33 2024-09-17T15:27:55.149 INFO:tasks.rados.rados.0.smithi203.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-17T15:27:55.155 INFO:tasks.rados.rados.0.smithi203.stdout:33: writing smithi20328086-33 from 537824 to 1011204 tid 1 2024-09-17T15:27:55.163 INFO:tasks.rados.rados.0.smithi203.stdout:33: writing smithi20328086-33 from 1806012 to 2432010 tid 2 2024-09-17T15:27:55.163 INFO:tasks.rados.rados.0.smithi203.stdout:34: write initial oid 34 2024-09-17T15:27:55.163 INFO:tasks.rados.rados.0.smithi203.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-17T15:27:55.170 INFO:tasks.rados.rados.0.smithi203.stdout:34: writing smithi20328086-34 from 554631 to 1103260 tid 1 2024-09-17T15:27:55.176 INFO:tasks.rados.rados.0.smithi203.stdout:34: writing smithi20328086-34 from 1748141 to 2234150 tid 2 2024-09-17T15:27:55.176 INFO:tasks.rados.rados.0.smithi203.stdout:35: write initial oid 35 2024-09-17T15:27:55.176 INFO:tasks.rados.rados.0.smithi203.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-17T15:27:55.183 INFO:tasks.rados.rados.0.smithi203.stdout:35: writing smithi20328086-35 from 571438 to 1195316 tid 1 2024-09-17T15:27:55.190 INFO:tasks.rados.rados.0.smithi203.stdout:35: writing smithi20328086-35 from 1806623 to 2436290 tid 2 2024-09-17T15:27:55.190 INFO:tasks.rados.rados.0.smithi203.stdout:36: write initial oid 36 2024-09-17T15:27:55.190 INFO:tasks.rados.rados.0.smithi203.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-17T15:27:55.197 INFO:tasks.rados.rados.0.smithi203.stdout:36: writing smithi20328086-36 from 588245 to 1287372 tid 1 2024-09-17T15:27:55.204 INFO:tasks.rados.rados.0.smithi203.stdout:36: writing smithi20328086-36 from 1865105 to 2354783 tid 2 2024-09-17T15:27:55.204 INFO:tasks.rados.rados.0.smithi203.stdout:37: write initial oid 37 2024-09-17T15:27:55.204 INFO:tasks.rados.rados.0.smithi203.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-17T15:27:55.212 INFO:tasks.rados.rados.0.smithi203.stdout:37: writing smithi20328086-37 from 605052 to 1379428 tid 1 2024-09-17T15:27:55.218 INFO:tasks.rados.rados.0.smithi203.stdout:37: writing smithi20328086-37 from 1923587 to 2556923 tid 2 2024-09-17T15:27:55.219 INFO:tasks.rados.rados.0.smithi203.stdout:38: write initial oid 38 2024-09-17T15:27:55.219 INFO:tasks.rados.rados.0.smithi203.stdout:24: finishing write tid 1 to smithi20328086-24 2024-09-17T15:27:55.219 INFO:tasks.rados.rados.0.smithi203.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-17T15:27:55.225 INFO:tasks.rados.rados.0.smithi203.stdout:38: writing smithi20328086-38 from 621859 to 1071484 tid 1 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout:38: writing smithi20328086-38 from 1865716 to 2621859 tid 2 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout:24: finishing write tid 2 to smithi20328086-24 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout:24: finishing write tid 3 to smithi20328086-24 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout:24: finishing write tid 4 to smithi20328086-24 2024-09-17T15:27:55.234 INFO:tasks.rados.rados.0.smithi203.stdout:24: oid 24 updating version 0 to 6 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:24: oid 24 version 6 is already newer than 5 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:24: oid 24 version 6 is already newer than 4 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:24: done (15 left) 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:39: write initial oid 39 2024-09-17T15:27:55.235 INFO:tasks.rados.rados.0.smithi203.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-17T15:27:55.240 INFO:tasks.rados.rados.0.smithi203.stdout:39: writing smithi20328086-39 from 638666 to 1163540 tid 1 2024-09-17T15:27:55.248 INFO:tasks.rados.rados.0.smithi203.stdout:39: writing smithi20328086-39 from 1924198 to 2561203 tid 2 2024-09-17T15:27:55.248 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.248 INFO:tasks.rados.rados.0.smithi203.stdout:25: finishing write tid 1 to smithi20328086-25 2024-09-17T15:27:55.248 INFO:tasks.rados.rados.0.smithi203.stdout:25: finishing write tid 2 to smithi20328086-25 2024-09-17T15:27:55.248 INFO:tasks.rados.rados.0.smithi203.stdout:25: finishing write tid 3 to smithi20328086-25 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: finishing write tid 4 to smithi20328086-25 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: oid 25 updating version 0 to 7 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: oid 25 updating version 7 to 8 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: oid 25 updating version 8 to 9 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:26: finishing write tid 1 to smithi20328086-26 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:25: done (15 left) 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:40: write initial oid 40 2024-09-17T15:27:55.249 INFO:tasks.rados.rados.0.smithi203.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-17T15:27:55.256 INFO:tasks.rados.rados.0.smithi203.stdout:40: writing smithi20328086-40 from 655473 to 1371949 tid 1 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:40: writing smithi20328086-40 from 2099033 to 2655473 tid 2 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:26: finishing write tid 2 to smithi20328086-26 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:26: finishing write tid 3 to smithi20328086-26 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:26: finishing write tid 4 to smithi20328086-26 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:26: oid 26 updating version 0 to 6 2024-09-17T15:27:55.264 INFO:tasks.rados.rados.0.smithi203.stdout:26: oid 26 version 6 is already newer than 4 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:26: oid 26 version 6 is already newer than 5 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:27: finishing write tid 1 to smithi20328086-27 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:26: done (15 left) 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:41: write initial oid 41 2024-09-17T15:27:55.265 INFO:tasks.rados.rados.0.smithi203.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-17T15:27:55.272 INFO:tasks.rados.rados.0.smithi203.stdout:41: writing smithi20328086-41 from 672280 to 1464005 tid 1 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:41: writing smithi20328086-41 from 2157515 to 2672280 tid 2 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: finishing write tid 2 to smithi20328086-27 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: finishing write tid 3 to smithi20328086-27 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: finishing write tid 4 to smithi20328086-27 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: oid 27 updating version 0 to 5 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: oid 27 updating version 5 to 6 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: oid 27 version 6 is already newer than 4 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-17T15:27:55.279 INFO:tasks.rados.rados.0.smithi203.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: finishing write tid 1 to smithi20328086-28 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: finishing write tid 2 to smithi20328086-28 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: finishing write tid 3 to smithi20328086-28 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: finishing write tid 4 to smithi20328086-28 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: oid 28 updating version 0 to 8 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: oid 28 updating version 8 to 9 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: oid 28 version 9 is already newer than 7 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:29: finishing write tid 1 to smithi20328086-29 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:27: done (15 left) 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:28: done (14 left) 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:42: write initial oid 42 2024-09-17T15:27:55.280 INFO:tasks.rados.rados.0.smithi203.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-17T15:27:55.285 INFO:tasks.rados.rados.0.smithi203.stdout:42: writing smithi20328086-42 from 689087 to 1156061 tid 1 2024-09-17T15:27:55.293 INFO:tasks.rados.rados.0.smithi203.stdout:42: writing smithi20328086-42 from 1699644 to 2483976 tid 2 2024-09-17T15:27:55.293 INFO:tasks.rados.rados.0.smithi203.stdout:43: write initial oid 43 2024-09-17T15:27:55.293 INFO:tasks.rados.rados.0.smithi203.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-17T15:27:55.300 INFO:tasks.rados.rados.0.smithi203.stdout:43: writing smithi20328086-43 from 705894 to 1248117 tid 1 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:43: writing smithi20328086-43 from 1758126 to 2402469 tid 2 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: finishing write tid 2 to smithi20328086-29 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: finishing write tid 3 to smithi20328086-29 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: finishing write tid 4 to smithi20328086-29 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: oid 29 updating version 0 to 10 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: oid 29 updating version 10 to 11 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:29: oid 29 updating version 11 to 12 2024-09-17T15:27:55.307 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 12 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: finishing write tid 1 to smithi20328086-30 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: finishing write tid 2 to smithi20328086-30 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: finishing write tid 3 to smithi20328086-30 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: finishing write tid 4 to smithi20328086-30 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: oid 30 updating version 0 to 7 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: oid 30 updating version 7 to 9 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: oid 30 version 9 is already newer than 8 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 9 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:29: done (15 left) 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:30: done (14 left) 2024-09-17T15:27:55.308 INFO:tasks.rados.rados.0.smithi203.stdout:44: write initial oid 44 2024-09-17T15:27:55.309 INFO:tasks.rados.rados.0.smithi203.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-17T15:27:55.314 INFO:tasks.rados.rados.0.smithi203.stdout:44: writing smithi20328086-44 from 722701 to 1340173 tid 1 2024-09-17T15:27:55.321 INFO:tasks.rados.rados.0.smithi203.stdout:44: writing smithi20328086-44 from 1816608 to 2604609 tid 2 2024-09-17T15:27:55.321 INFO:tasks.rados.rados.0.smithi203.stdout:45: write initial oid 45 2024-09-17T15:27:55.321 INFO:tasks.rados.rados.0.smithi203.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-17T15:27:55.330 INFO:tasks.rados.rados.0.smithi203.stdout:45: writing smithi20328086-45 from 739508 to 1432229 tid 1 2024-09-17T15:27:55.336 INFO:tasks.rados.rados.0.smithi203.stdout:45: writing smithi20328086-45 from 1875090 to 2523102 tid 2 2024-09-17T15:27:55.336 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.336 INFO:tasks.rados.rados.0.smithi203.stdout:33: finishing write tid 1 to smithi20328086-33 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: finishing write tid 2 to smithi20328086-33 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: finishing write tid 3 to smithi20328086-33 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: finishing write tid 4 to smithi20328086-33 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: oid 33 updating version 0 to 5 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: oid 33 version 5 is already newer than 4 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: oid 33 updating version 5 to 6 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: finishing write tid 1 to smithi20328086-34 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: finishing write tid 2 to smithi20328086-34 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: finishing write tid 3 to smithi20328086-34 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: finishing write tid 4 to smithi20328086-34 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: oid 34 updating version 0 to 2 2024-09-17T15:27:55.337 INFO:tasks.rados.rados.0.smithi203.stdout:34: oid 34 version 2 is already newer than 1 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:34: oid 34 updating version 2 to 3 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:33: done (15 left) 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:34: done (14 left) 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:46: write initial oid 46 2024-09-17T15:27:55.338 INFO:tasks.rados.rados.0.smithi203.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-17T15:27:55.345 INFO:tasks.rados.rados.0.smithi203.stdout:46: writing smithi20328086-46 from 756315 to 1524285 tid 1 2024-09-17T15:27:55.352 INFO:tasks.rados.rados.0.smithi203.stdout:46: writing smithi20328086-46 from 1933572 to 2725242 tid 2 2024-09-17T15:27:55.352 INFO:tasks.rados.rados.0.smithi203.stdout:47: write initial oid 47 2024-09-17T15:27:55.352 INFO:tasks.rados.rados.0.smithi203.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-17T15:27:55.359 INFO:tasks.rados.rados.0.smithi203.stdout:47: writing smithi20328086-47 from 773122 to 1332694 tid 1 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:47: writing smithi20328086-47 from 1992054 to 2643735 tid 2 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: finishing write tid 1 to smithi20328086-35 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: finishing write tid 2 to smithi20328086-35 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: finishing write tid 3 to smithi20328086-35 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: finishing write tid 4 to smithi20328086-35 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: oid 35 updating version 0 to 4 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: oid 35 updating version 4 to 5 2024-09-17T15:27:55.367 INFO:tasks.rados.rados.0.smithi203.stdout:35: oid 35 updating version 5 to 6 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: finishing write tid 1 to smithi20328086-36 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: finishing write tid 2 to smithi20328086-36 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: finishing write tid 3 to smithi20328086-36 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: finishing write tid 4 to smithi20328086-36 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: oid 36 updating version 0 to 12 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: oid 36 version 12 is already newer than 11 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: oid 36 version 12 is already newer than 10 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:37: finishing write tid 1 to smithi20328086-37 2024-09-17T15:27:55.368 INFO:tasks.rados.rados.0.smithi203.stdout:37: finishing write tid 2 to smithi20328086-37 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: finishing write tid 3 to smithi20328086-37 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: finishing write tid 4 to smithi20328086-37 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: oid 37 updating version 0 to 8 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: oid 37 version 8 is already newer than 7 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: oid 37 updating version 8 to 9 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:38: finishing write tid 1 to smithi20328086-38 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:39: finishing write tid 1 to smithi20328086-39 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:35: done (15 left) 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:36: done (14 left) 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:37: done (13 left) 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:48: write initial oid 48 2024-09-17T15:27:55.369 INFO:tasks.rados.rados.0.smithi203.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-17T15:27:55.374 INFO:tasks.rados.rados.0.smithi203.stdout:48: writing smithi20328086-48 from 789929 to 1424750 tid 1 2024-09-17T15:27:55.382 INFO:tasks.rados.rados.0.smithi203.stdout:48: writing smithi20328086-48 from 2050536 to 2789929 tid 2 2024-09-17T15:27:55.382 INFO:tasks.rados.rados.0.smithi203.stdout:49: write initial oid 49 2024-09-17T15:27:55.383 INFO:tasks.rados.rados.0.smithi203.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-17T15:27:55.389 INFO:tasks.rados.rados.0.smithi203.stdout:49: writing smithi20328086-49 from 406736 to 1116806 tid 1 2024-09-17T15:27:55.396 INFO:tasks.rados.rados.0.smithi203.stdout:49: writing smithi20328086-49 from 1709018 to 2364368 tid 2 2024-09-17T15:27:55.396 INFO:tasks.rados.rados.0.smithi203.stdout:50: write initial oid 50 2024-09-17T15:27:55.396 INFO:tasks.rados.rados.0.smithi203.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-17T15:27:55.403 INFO:tasks.rados.rados.0.smithi203.stdout:50: writing smithi20328086-50 from 423543 to 1208862 tid 1 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:50: writing smithi20328086-50 from 1767500 to 2566508 tid 2 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:39: finishing write tid 2 to smithi20328086-39 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:32: finishing write tid 1 to smithi20328086-32 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:22: finishing write tid 1 to smithi20328086-22 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:32: finishing write tid 2 to smithi20328086-32 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:32: finishing write tid 3 to smithi20328086-32 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:38: finishing write tid 2 to smithi20328086-38 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:22: finishing write tid 2 to smithi20328086-22 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:22: finishing write tid 3 to smithi20328086-22 2024-09-17T15:27:55.412 INFO:tasks.rados.rados.0.smithi203.stdout:32: finishing write tid 4 to smithi20328086-32 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:32: oid 32 updating version 0 to 2 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:32: oid 32 version 2 is already newer than 1 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:32: oid 32 updating version 2 to 3 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:31: finishing write tid 1 to smithi20328086-31 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: finishing write tid 3 to smithi20328086-39 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:31: finishing write tid 2 to smithi20328086-31 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: finishing write tid 4 to smithi20328086-39 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: oid 39 updating version 0 to 11 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: oid 39 updating version 11 to 12 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: oid 39 version 12 is already newer than 10 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-17T15:27:55.413 INFO:tasks.rados.rados.0.smithi203.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:31: finishing write tid 3 to smithi20328086-31 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:32: done (15 left) 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:39: done (14 left) 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:51: read oid 14 snap -1 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:51: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:52: read oid 24 snap -1 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:52: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 16 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:31: finishing write tid 4 to smithi20328086-31 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:31: oid 31 updating version 0 to 4 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:31: oid 31 updating version 4 to 5 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:31: oid 31 updating version 5 to 6 2024-09-17T15:27:55.414 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:22: finishing write tid 4 to smithi20328086-22 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:22: oid 22 updating version 0 to 3 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:22: oid 22 version 3 is already newer than 2 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:22: oid 22 version 3 is already newer than 1 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: finishing write tid 3 to smithi20328086-38 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: finishing write tid 4 to smithi20328086-38 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: oid 38 updating version 0 to 9 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: oid 38 version 9 is already newer than 7 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: oid 38 version 9 is already newer than 8 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-17T15:27:55.415 INFO:tasks.rados.rados.0.smithi203.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:22: done (15 left) 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:31: done (14 left) 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:38: done (13 left) 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:53: copy_from oid 34 from oid 25 current snap is 0 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: finishing write tid 1 to smithi20328086-40 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: finishing write tid 2 to smithi20328086-40 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: finishing write tid 3 to smithi20328086-40 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: finishing write tid 4 to smithi20328086-40 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: oid 40 updating version 0 to 9 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: oid 40 version 9 is already newer than 8 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: oid 40 version 9 is already newer than 7 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:41: finishing write tid 1 to smithi20328086-41 2024-09-17T15:27:55.416 INFO:tasks.rados.rados.0.smithi203.stdout:41: finishing write tid 2 to smithi20328086-41 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: finishing write tid 3 to smithi20328086-41 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: finishing write tid 4 to smithi20328086-41 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: oid 41 updating version 0 to 1 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: oid 41 updating version 1 to 2 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: oid 41 updating version 2 to 3 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 3 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: finishing write tid 1 to smithi20328086-42 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: finishing write tid 2 to smithi20328086-42 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: finishing write tid 3 to smithi20328086-42 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: finishing write tid 4 to smithi20328086-42 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: oid 42 updating version 0 to 7 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: oid 42 updating version 7 to 9 2024-09-17T15:27:55.417 INFO:tasks.rados.rados.0.smithi203.stdout:42: oid 42 version 9 is already newer than 8 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:43: finishing write tid 1 to smithi20328086-43 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:40: done (13 left) 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:41: done (12 left) 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:42: done (11 left) 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:54: write (excl) oid 10 current snap is 0 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:43: finishing write tid 2 to smithi20328086-43 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:43: finishing write tid 3 to smithi20328086-43 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:54: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-17T15:27:55.418 INFO:tasks.rados.rados.0.smithi203.stdout:54: writing smithi20328086-10 from 440350 to 900918 tid 1 2024-09-17T15:27:55.422 INFO:tasks.rados.rados.0.smithi203.stdout:54: writing smithi20328086-10 from 1309629 to 1852295 tid 2 2024-09-17T15:27:55.428 INFO:tasks.rados.rados.0.smithi203.stdout:54: writing smithi20328086-10 from 2323973 to 2840350 tid 3 2024-09-17T15:27:55.428 INFO:tasks.rados.rados.0.smithi203.stdout:55: copy_from oid 5 from oid 8 current snap is 0 2024-09-17T15:27:55.428 INFO:tasks.rados.rados.0.smithi203.stdout:43: finishing write tid 4 to smithi20328086-43 2024-09-17T15:27:55.428 INFO:tasks.rados.rados.0.smithi203.stdout:43: oid 43 updating version 0 to 6 2024-09-17T15:27:55.428 INFO:tasks.rados.rados.0.smithi203.stdout:43: oid 43 version 6 is already newer than 5 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:43: oid 43 version 6 is already newer than 4 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: finishing write tid 1 to smithi20328086-44 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: finishing write tid 2 to smithi20328086-44 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:43: done (12 left) 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:56: rollback oid 3 current snap is 0 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:56: done (12 left) 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:57: write oid 43 current snap is 0 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: finishing write tid 3 to smithi20328086-44 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: finishing write tid 4 to smithi20328086-44 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: oid 44 updating version 0 to 11 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: oid 44 updating version 11 to 12 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:44: oid 44 version 12 is already newer than 10 2024-09-17T15:27:55.429 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: finishing write tid 1 to smithi20328086-45 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: finishing write tid 2 to smithi20328086-45 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: finishing write tid 3 to smithi20328086-45 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: finishing write tid 4 to smithi20328086-45 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: oid 45 updating version 0 to 6 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: oid 45 version 6 is already newer than 5 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: oid 45 version 6 is already newer than 4 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-17T15:27:55.430 INFO:tasks.rados.rados.0.smithi203.stdout:57: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-17T15:27:55.433 INFO:tasks.rados.rados.0.smithi203.stdout:57: writing smithi20328086-43 from 457157 to 992974 tid 1 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:57: writing smithi20328086-43 from 1768111 to 2170788 tid 2 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:44: done (12 left) 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:45: done (11 left) 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:58: read oid 3 snap -1 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:58: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:59: delete oid 4 current snap is 0 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: finishing write tid 1 to smithi20328086-46 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: finishing write tid 2 to smithi20328086-46 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: finishing write tid 3 to smithi20328086-46 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: finishing write tid 4 to smithi20328086-46 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: oid 46 updating version 0 to 11 2024-09-17T15:27:55.441 INFO:tasks.rados.rados.0.smithi203.stdout:46: oid 46 updating version 11 to 12 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:46: oid 46 version 12 is already newer than 10 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: finishing write tid 1 to smithi20328086-47 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: finishing write tid 2 to smithi20328086-47 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: finishing write tid 3 to smithi20328086-47 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: finishing write tid 4 to smithi20328086-47 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: oid 47 updating version 0 to 8 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: oid 47 version 8 is already newer than 7 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: oid 47 updating version 8 to 9 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:48: finishing write tid 1 to smithi20328086-48 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:48: finishing write tid 2 to smithi20328086-48 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:48: finishing write tid 3 to smithi20328086-48 2024-09-17T15:27:55.442 INFO:tasks.rados.rados.0.smithi203.stdout:48: finishing write tid 4 to smithi20328086-48 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:48: oid 48 updating version 0 to 15 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:48: oid 48 version 15 is already newer than 13 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:48: oid 48 version 15 is already newer than 14 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: finishing write tid 1 to smithi20328086-49 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: finishing write tid 2 to smithi20328086-49 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: finishing write tid 3 to smithi20328086-49 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: finishing write tid 4 to smithi20328086-49 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: oid 49 updating version 0 to 15 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: oid 49 version 15 is already newer than 13 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: oid 49 version 15 is already newer than 14 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T15:27:55.443 INFO:tasks.rados.rados.0.smithi203.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-17T15:27:55.444 INFO:tasks.rados.rados.0.smithi203.stdout:50: finishing write tid 1 to smithi20328086-50 2024-09-17T15:27:55.444 INFO:tasks.rados.rados.0.smithi203.stdout:51: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-17T15:27:55.455 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 887109 2024-09-17T15:27:55.558 INFO:tasks.rados.rados.0.smithi203.stdout:50: finishing write tid 2 to smithi20328086-50 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: finishing write tid 3 to smithi20328086-50 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: finishing write tid 4 to smithi20328086-50 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: oid 50 updating version 0 to 5 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: oid 50 updating version 5 to 6 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: oid 50 version 6 is already newer than 4 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 6 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-17T15:27:55.559 INFO:tasks.rados.rados.0.smithi203.stdout:52: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:27:55.573 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:27:55.599 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: finishing write tid 1 to smithi20328086-10 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:53: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: finishing write tid 2 to smithi20328086-10 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: finishing write tid 3 to smithi20328086-10 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: finishing write tid 4 to smithi20328086-10 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:53: finishing copy_from to smithi20328086-34 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: finishing write tid 5 to smithi20328086-10 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: oid 10 updating version 0 to 13 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: oid 10 version 13 is already newer than 11 2024-09-17T15:27:55.626 INFO:tasks.rados.rados.0.smithi203.stdout:54: oid 10 version 13 is already newer than 10 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:54: oid 10 version 13 is already newer than 12 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 13 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:54: left oid 10 (ObjNum 50 snap 0 seq_num 50) 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:55: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:55: finishing copy_from to smithi20328086-5 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: finishing write tid 1 to smithi20328086-43 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: finishing write tid 2 to smithi20328086-43 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: finishing write tid 3 to smithi20328086-43 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: finishing write tid 4 to smithi20328086-43 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: oid 43 updating version 0 to 9 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: oid 43 version 9 is already newer than 7 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: oid 43 version 9 is already newer than 8 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 9 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-17T15:27:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:57: left oid 43 (ObjNum 51 snap 0 seq_num 51) 2024-09-17T15:27:55.628 INFO:tasks.rados.rados.0.smithi203.stdout:58: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-17T15:27:55.629 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 549546 2024-09-17T15:27:55.644 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:46: done (12 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:47: done (11 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:48: done (10 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:49: done (9 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:50: done (8 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:51: done (7 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:52: done (6 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:53: done (5 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:54: done (4 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:55: done (3 left) 2024-09-17T15:27:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:57: done (2 left) 2024-09-17T15:27:55.688 INFO:tasks.rados.rados.0.smithi203.stdout:58: done (1 left) 2024-09-17T15:27:55.688 INFO:tasks.rados.rados.0.smithi203.stdout:59: done (0 left) 2024-09-17T15:27:55.688 INFO:tasks.rados.rados.0.smithi203.stdout:60: rollback oid 42 current snap is 0 2024-09-17T15:27:55.688 INFO:tasks.rados.rados.0.smithi203.stdout:60: done (0 left) 2024-09-17T15:27:55.688 INFO:tasks.rados.rados.0.smithi203.stdout:61: delete oid 4 current snap is 0 2024-09-17T15:27:55.689 INFO:tasks.rados.rados.0.smithi203.stdout:61: done (0 left) 2024-09-17T15:27:55.689 INFO:tasks.rados.rados.0.smithi203.stdout:62: delete oid 47 current snap is 0 2024-09-17T15:27:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:62: done (0 left) 2024-09-17T15:27:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:63: rollback oid 46 current snap is 0 2024-09-17T15:27:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:63: done (0 left) 2024-09-17T15:27:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:64: delete oid 39 current snap is 0 2024-09-17T15:27:55.697 INFO:tasks.rados.rados.0.smithi203.stdout:64: done (0 left) 2024-09-17T15:27:55.697 INFO:tasks.rados.rados.0.smithi203.stdout:65: delete oid 2 current snap is 0 2024-09-17T15:27:55.700 INFO:tasks.rados.rados.0.smithi203.stdout:65: done (0 left) 2024-09-17T15:27:55.700 INFO:tasks.rados.rados.0.smithi203.stdout:66: write oid 26 current snap is 0 2024-09-17T15:27:55.700 INFO:tasks.rados.rados.0.smithi203.stdout:66: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-17T15:27:55.721 INFO:tasks.rados.rados.0.smithi203.stdout:66: writing smithi20328086-26 from 557999 to 1261663 tid 1 2024-09-17T15:27:55.736 INFO:tasks.rados.rados.0.smithi203.stdout:66: writing smithi20328086-26 from 1719003 to 2132687 tid 2 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:66: writing smithi20328086-26 from 2632287 to 2957999 tid 3 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:67: read oid 43 snap -1 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:67: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:68: write oid 30 current snap is 0 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:66: finishing write tid 1 to smithi20328086-26 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:66: finishing write tid 2 to smithi20328086-26 2024-09-17T15:27:55.749 INFO:tasks.rados.rados.0.smithi203.stdout:68: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-17T15:27:55.768 INFO:tasks.rados.rados.0.smithi203.stdout:68: writing smithi20328086-30 from 574806 to 1353719 tid 1 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:68: writing smithi20328086-30 from 2061132 to 2618474 tid 2 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:69: read oid 49 snap -1 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:69: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:70: delete oid 46 current snap is 0 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:66: finishing write tid 3 to smithi20328086-26 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:66: finishing write tid 4 to smithi20328086-26 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:66: finishing write tid 5 to smithi20328086-26 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:66: oid 26 updating version 0 to 18 2024-09-17T15:27:55.785 INFO:tasks.rados.rados.0.smithi203.stdout:66: oid 26 version 18 is already newer than 16 2024-09-17T15:27:55.786 INFO:tasks.rados.rados.0.smithi203.stdout:66: oid 26 version 18 is already newer than 17 2024-09-17T15:27:55.786 INFO:tasks.rados.rados.0.smithi203.stdout:66: oid 26 updating version 18 to 19 2024-09-17T15:27:55.786 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 19 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-17T15:27:55.786 INFO:tasks.rados.rados.0.smithi203.stdout:66: left oid 26 (ObjNum 57 snap 0 seq_num 57) 2024-09-17T15:27:55.786 INFO:tasks.rados.rados.0.smithi203.stdout:67: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-17T15:27:55.831 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 281040 2024-09-17T15:27:55.919 INFO:tasks.rados.rados.0.smithi203.stdout:68: finishing write tid 1 to smithi20328086-30 2024-09-17T15:27:55.919 INFO:tasks.rados.rados.0.smithi203.stdout:69: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-17T15:27:55.990 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2257539 2024-09-17T15:27:56.054 INFO:tasks.rados.rados.0.smithi203.stdout:68: finishing write tid 2 to smithi20328086-30 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: finishing write tid 3 to smithi20328086-30 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: finishing write tid 4 to smithi20328086-30 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: oid 30 updating version 0 to 10 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: oid 30 updating version 10 to 12 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: oid 30 version 12 is already newer than 11 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 12 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: left oid 30 (ObjNum 58 snap 0 seq_num 58) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:66: done (4 left) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:67: done (3 left) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:68: done (2 left) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:69: done (1 left) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:70: done (0 left) 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:71: write oid 46 current snap is 0 2024-09-17T15:27:56.055 INFO:tasks.rados.rados.0.smithi203.stdout:71: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:71: writing smithi20328086-46 from 608420 to 1137831 tid 1 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:71: writing smithi20328086-46 from 1778096 to 2339107 tid 2 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:71: writing smithi20328086-46 from 2881850 to 3008420 tid 3 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:72: rollback oid 6 current snap is 0 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:72: done (1 left) 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:73: read oid 42 snap -1 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:73: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:74: snap_create 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:71: finishing write tid 1 to smithi20328086-46 2024-09-17T15:27:56.056 INFO:tasks.rados.rados.0.smithi203.stdout:71: finishing write tid 2 to smithi20328086-46 2024-09-17T15:27:56.057 INFO:tasks.rados.rados.0.smithi203.stdout:71: finishing write tid 3 to smithi20328086-46 2024-09-17T15:27:56.057 INFO:tasks.rados.rados.0.smithi203.stdout:71: finishing write tid 4 to smithi20328086-46 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: finishing write tid 5 to smithi20328086-46 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: oid 46 updating version 0 to 16 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: oid 46 updating version 16 to 17 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: oid 46 version 17 is already newer than 15 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: oid 46 version 17 is already newer than 14 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 17 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:71: left oid 46 (ObjNum 60 snap 0 seq_num 60) 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:73: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-17T15:27:56.058 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 526340 2024-09-17T15:27:56.750 INFO:tasks.rados.rados.0.smithi203.stdout:71: done (2 left) 2024-09-17T15:27:56.750 INFO:tasks.rados.rados.0.smithi203.stdout:73: done (1 left) 2024-09-17T15:27:56.751 INFO:tasks.rados.rados.0.smithi203.stdout:74: done (0 left) 2024-09-17T15:27:56.751 INFO:tasks.rados.rados.0.smithi203.stdout:75: write (excl) oid 5 current snap is 1 2024-09-17T15:27:56.751 INFO:tasks.rados.rados.0.smithi203.stdout:75: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-17T15:27:56.769 INFO:tasks.rados.rados.0.smithi203.stdout:75: writing smithi20328086-5 from 625227 to 1346240 tid 1 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:75: writing smithi20328086-5 from 1952931 to 2657600 tid 2 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:76: copy_from oid 9 from oid 6 current snap is 1 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:75: finishing write tid 1 to smithi20328086-5 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:77: read oid 15 snap 0 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:77: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:78: copy_from oid 32 from oid 34 current snap is 1 2024-09-17T15:27:56.789 INFO:tasks.rados.rados.0.smithi203.stdout:79: snap_create 2024-09-17T15:27:56.795 INFO:tasks.rados.rados.0.smithi203.stdout:75: finishing write tid 2 to smithi20328086-5 2024-09-17T15:27:56.796 INFO:tasks.rados.rados.0.smithi203.stdout:75: finishing write tid 3 to smithi20328086-5 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:75: finishing write tid 4 to smithi20328086-5 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:75: oid 5 updating version 0 to 9 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:75: oid 5 updating version 9 to 11 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:75: oid 5 version 11 is already newer than 10 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 11 (ObjNum 61 snap 1 seq_num 61) dirty exists 2024-09-17T15:27:56.798 INFO:tasks.rados.rados.0.smithi203.stdout:75: left oid 5 (ObjNum 61 snap 1 seq_num 61) 2024-09-17T15:27:56.800 INFO:tasks.rados.rados.0.smithi203.stdout:77: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-17T15:27:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:76: finishing copy_from racing read to smithi20328086-9 2024-09-17T15:27:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:76: finishing copy_from to smithi20328086-9 2024-09-17T15:27:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-17T15:27:56.887 INFO:tasks.rados.rados.0.smithi203.stdout:78: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:27:56.887 INFO:tasks.rados.rados.0.smithi203.stdout:78: finishing copy_from to smithi20328086-32 2024-09-17T15:27:56.887 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T15:27:58.168 INFO:tasks.rados.rados.0.smithi203.stdout:75: done (4 left) 2024-09-17T15:27:58.168 INFO:tasks.rados.rados.0.smithi203.stdout:76: done (3 left) 2024-09-17T15:27:58.168 INFO:tasks.rados.rados.0.smithi203.stdout:77: done (2 left) 2024-09-17T15:27:58.169 INFO:tasks.rados.rados.0.smithi203.stdout:78: done (1 left) 2024-09-17T15:27:58.169 INFO:tasks.rados.rados.0.smithi203.stdout:79: done (0 left) 2024-09-17T15:27:58.169 INFO:tasks.rados.rados.0.smithi203.stdout:80: write (excl) oid 13 current snap is 2 2024-09-17T15:27:58.169 INFO:tasks.rados.rados.0.smithi203.stdout:80: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-17T15:27:58.188 INFO:tasks.rados.rados.0.smithi203.stdout:80: writing smithi20328086-13 from 642034 to 1438296 tid 1 2024-09-17T15:27:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:80: writing smithi20328086-13 from 1895060 to 2459740 tid 2 2024-09-17T15:27:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:81: copy_from oid 2 from oid 24 current snap is 2 2024-09-17T15:27:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:82: write (excl) oid 5 current snap is 2 2024-09-17T15:27:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:82: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-17T15:27:58.218 INFO:tasks.rados.rados.0.smithi203.stdout:82: writing smithi20328086-5 from 658841 to 1130352 tid 1 2024-09-17T15:27:58.231 INFO:tasks.rados.rados.0.smithi203.stdout:82: writing smithi20328086-5 from 1553542 to 2261880 tid 2 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:82: writing smithi20328086-5 from 2964119 to 3058841 tid 3 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:83: copy_from oid 41 from oid 33 current snap is 2 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: finishing write tid 1 to smithi20328086-13 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: finishing write tid 2 to smithi20328086-13 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: finishing write tid 3 to smithi20328086-13 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: finishing write tid 4 to smithi20328086-13 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: oid 13 updating version 0 to 16 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: oid 13 updating version 16 to 17 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: oid 13 version 17 is already newer than 15 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 17 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-09-17T15:27:58.240 INFO:tasks.rados.rados.0.smithi203.stdout:80: left oid 13 (ObjNum 62 snap 2 seq_num 62) 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:82: finishing write tid 1 to smithi20328086-5 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:81: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:81: finishing copy_from to smithi20328086-2 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:80: done (3 left) 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:81: done (2 left) 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:84: rollback oid 39 current snap is 2 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 39 to 1 2024-09-17T15:27:58.241 INFO:tasks.rados.rados.0.smithi203.stdout:85: snap_remove snap 0 2024-09-17T15:27:59.169 INFO:tasks.rados.rados.0.smithi203.stdout:85: done (3 left) 2024-09-17T15:27:59.169 INFO:tasks.rados.rados.0.smithi203.stdout:86: read oid 50 snap 1 2024-09-17T15:27:59.170 INFO:tasks.rados.rados.0.smithi203.stdout:86: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T15:27:59.170 INFO:tasks.rados.rados.0.smithi203.stdout:87: write oid 10 current snap is 2 2024-09-17T15:27:59.170 INFO:tasks.rados.rados.0.smithi203.stdout:87: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-17T15:27:59.181 INFO:tasks.rados.rados.0.smithi203.stdout:87: writing smithi20328086-10 from 675648 to 1222408 tid 1 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:87: writing smithi20328086-10 from 2012024 to 2580373 tid 2 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:88: read oid 37 snap -1 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:88: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:89: read oid 31 snap -1 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:89: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-17T15:27:59.195 INFO:tasks.rados.rados.0.smithi203.stdout:90: write (excl) oid 35 current snap is 2 2024-09-17T15:27:59.196 INFO:tasks.rados.rados.0.smithi203.stdout:90: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-17T15:27:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:90: writing smithi20328086-35 from 692455 to 1314464 tid 1 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:90: writing smithi20328086-35 from 2070506 to 2782513 tid 2 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:91: copy_from oid 36 from oid 12 current snap is 2 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: finishing write tid 2 to smithi20328086-5 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:84: finishing rollback tid 1 to smithi20328086-39 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 0 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: finishing write tid 3 to smithi20328086-5 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: finishing write tid 4 to smithi20328086-5 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: finishing write tid 5 to smithi20328086-5 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: oid 5 updating version 0 to 14 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: oid 5 version 14 is already newer than 13 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: oid 5 updating version 14 to 15 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:82: oid 5 updating version 15 to 16 2024-09-17T15:27:59.226 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 16 (ObjNum 63 snap 2 seq_num 63) dirty exists 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:82: left oid 5 (ObjNum 63 snap 2 seq_num 63) 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:83: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:83: finishing copy_from to smithi20328086-41 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:82: done (8 left) 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:83: done (7 left) 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:84: done (6 left) 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:92: write oid 18 current snap is 2 2024-09-17T15:27:59.227 INFO:tasks.rados.rados.0.smithi203.stdout:92: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-17T15:27:59.238 INFO:tasks.rados.rados.0.smithi203.stdout:92: writing smithi20328086-18 from 709262 to 1406520 tid 1 2024-09-17T15:27:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:92: writing smithi20328086-18 from 2012635 to 2584653 tid 2 2024-09-17T15:27:59.254 INFO:tasks.rados.rados.0.smithi203.stdout:92: writing smithi20328086-18 from 3046388 to 3109262 tid 3 2024-09-17T15:27:59.254 INFO:tasks.rados.rados.0.smithi203.stdout:93: write (excl) oid 33 current snap is 2 2024-09-17T15:27:59.254 INFO:tasks.rados.rados.0.smithi203.stdout:93: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-17T15:27:59.266 INFO:tasks.rados.rados.0.smithi203.stdout:93: writing smithi20328086-33 from 726069 to 1498576 tid 1 2024-09-17T15:27:59.275 INFO:tasks.rados.rados.0.smithi203.stdout:93: writing smithi20328086-33 from 2071117 to 2786793 tid 2 2024-09-17T15:27:59.276 INFO:tasks.rados.rados.0.smithi203.stdout:94: read oid 47 snap -1 2024-09-17T15:27:59.276 INFO:tasks.rados.rados.0.smithi203.stdout:94: expect deleted 2024-09-17T15:27:59.276 INFO:tasks.rados.rados.0.smithi203.stdout:95: delete oid 3 current snap is 2 2024-09-17T15:27:59.276 INFO:tasks.rados.rados.0.smithi203.stdout:86: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-17T15:27:59.417 INFO:tasks.rados.rados.0.smithi203.stdout:87: finishing write tid 1 to smithi20328086-10 2024-09-17T15:27:59.417 INFO:tasks.rados.rados.0.smithi203.stdout:89: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-17T15:27:59.422 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 966660 2024-09-17T15:27:59.438 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:27:59.482 INFO:tasks.rados.rados.0.smithi203.stdout:87: finishing write tid 2 to smithi20328086-10 2024-09-17T15:27:59.482 INFO:tasks.rados.rados.0.smithi203.stdout:87: finishing write tid 3 to smithi20328086-10 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:87: finishing write tid 4 to smithi20328086-10 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:87: oid 10 updating version 0 to 22 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:87: oid 10 updating version 22 to 23 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:87: oid 10 version 23 is already newer than 21 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 23 (ObjNum 64 snap 2 seq_num 64) dirty exists 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:87: left oid 10 (ObjNum 64 snap 2 seq_num 64) 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:88: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-17T15:27:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:27:59.521 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1993652 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: finishing write tid 1 to smithi20328086-35 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: finishing write tid 2 to smithi20328086-35 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: finishing write tid 3 to smithi20328086-35 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: finishing write tid 4 to smithi20328086-35 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: oid 35 updating version 0 to 10 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: oid 35 updating version 10 to 11 2024-09-17T15:27:59.553 INFO:tasks.rados.rados.0.smithi203.stdout:90: oid 35 updating version 11 to 12 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 12 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:90: left oid 35 (ObjNum 65 snap 2 seq_num 65) 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: finishing write tid 1 to smithi20328086-18 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: finishing write tid 2 to smithi20328086-18 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: finishing write tid 3 to smithi20328086-18 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: finishing write tid 4 to smithi20328086-18 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: finishing write tid 5 to smithi20328086-18 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: oid 18 updating version 0 to 15 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: oid 18 version 15 is already newer than 14 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: oid 18 version 15 is already newer than 12 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: oid 18 version 15 is already newer than 13 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 15 (ObjNum 66 snap 2 seq_num 66) dirty exists 2024-09-17T15:27:59.554 INFO:tasks.rados.rados.0.smithi203.stdout:92: left oid 18 (ObjNum 66 snap 2 seq_num 66) 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: finishing write tid 1 to smithi20328086-33 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: finishing write tid 2 to smithi20328086-33 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: finishing write tid 3 to smithi20328086-33 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: finishing write tid 4 to smithi20328086-33 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: oid 33 updating version 0 to 21 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: oid 33 updating version 21 to 22 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: oid 33 version 22 is already newer than 20 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 22 (ObjNum 67 snap 2 seq_num 67) dirty exists 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:93: left oid 33 (ObjNum 67 snap 2 seq_num 67) 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:91: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:91: finishing copy_from to smithi20328086-36 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:86: done (9 left) 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:87: done (8 left) 2024-09-17T15:27:59.555 INFO:tasks.rados.rados.0.smithi203.stdout:88: done (7 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:89: done (6 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:90: done (5 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:91: done (4 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:92: done (3 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:93: done (2 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:94: done (1 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:95: done (0 left) 2024-09-17T15:27:59.556 INFO:tasks.rados.rados.0.smithi203.stdout:96: snap_create 2024-09-17T15:28:00.178 INFO:tasks.rados.rados.0.smithi203.stdout:96: done (0 left) 2024-09-17T15:28:00.179 INFO:tasks.rados.rados.0.smithi203.stdout:97: copy_from oid 33 from oid 44 current snap is 3 2024-09-17T15:28:00.179 INFO:tasks.rados.rados.0.smithi203.stdout:98: snap_create 2024-09-17T15:28:00.212 INFO:tasks.rados.rados.0.smithi203.stdout:97: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:28:00.214 INFO:tasks.rados.rados.0.smithi203.stdout:97: finishing copy_from to smithi20328086-33 2024-09-17T15:28:00.214 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-17T15:28:01.181 INFO:tasks.rados.rados.0.smithi203.stdout:97: done (1 left) 2024-09-17T15:28:01.182 INFO:tasks.rados.rados.0.smithi203.stdout:98: done (0 left) 2024-09-17T15:28:01.182 INFO:tasks.rados.rados.0.smithi203.stdout:99: snap_remove snap 3 2024-09-17T15:28:02.195 INFO:tasks.rados.rados.0.smithi203.stdout:99: done (0 left) 2024-09-17T15:28:02.195 INFO:tasks.rados.rados.0.smithi203.stdout:100: read oid 12 snap -1 2024-09-17T15:28:02.195 INFO:tasks.rados.rados.0.smithi203.stdout:100: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:28:02.195 INFO:tasks.rados.rados.0.smithi203.stdout:101: write (excl) oid 40 current snap is 4 2024-09-17T15:28:02.195 INFO:tasks.rados.rados.0.smithi203.stdout:101: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-17T15:28:02.212 INFO:tasks.rados.rados.0.smithi203.stdout:101: writing smithi20328086-40 from 759683 to 1399041 tid 1 2024-09-17T15:28:02.233 INFO:tasks.rados.rados.0.smithi203.stdout:101: writing smithi20328086-40 from 1904434 to 2623779 tid 2 2024-09-17T15:28:02.241 INFO:tasks.rados.rados.0.smithi203.stdout:101: writing smithi20328086-40 from 3128657 to 3159683 tid 3 2024-09-17T15:28:02.241 INFO:tasks.rados.rados.0.smithi203.stdout:102: read oid 48 snap -1 2024-09-17T15:28:02.242 INFO:tasks.rados.rados.0.smithi203.stdout:102: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-17T15:28:02.242 INFO:tasks.rados.rados.0.smithi203.stdout:103: write oid 32 current snap is 4 2024-09-17T15:28:02.242 INFO:tasks.rados.rados.0.smithi203.stdout:103: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-17T15:28:02.259 INFO:tasks.rados.rados.0.smithi203.stdout:103: writing smithi20328086-32 from 776490 to 1491097 tid 1 2024-09-17T15:28:02.271 INFO:tasks.rados.rados.0.smithi203.stdout:103: writing smithi20328086-32 from 2246563 to 2825919 tid 2 2024-09-17T15:28:02.272 INFO:tasks.rados.rados.0.smithi203.stdout:104: snap_create 2024-09-17T15:28:02.272 INFO:tasks.rados.rados.0.smithi203.stdout:100: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:28:02.272 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 190035 2024-09-17T15:28:02.310 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:02.359 INFO:tasks.rados.rados.0.smithi203.stdout:101: finishing write tid 1 to smithi20328086-40 2024-09-17T15:28:02.360 INFO:tasks.rados.rados.0.smithi203.stdout:101: finishing write tid 2 to smithi20328086-40 2024-09-17T15:28:02.360 INFO:tasks.rados.rados.0.smithi203.stdout:101: finishing write tid 3 to smithi20328086-40 2024-09-17T15:28:02.360 INFO:tasks.rados.rados.0.smithi203.stdout:102: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-17T15:28:02.395 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1059998 2024-09-17T15:28:02.441 INFO:tasks.rados.rados.0.smithi203.stdout:101: finishing write tid 4 to smithi20328086-40 2024-09-17T15:28:02.441 INFO:tasks.rados.rados.0.smithi203.stdout:101: finishing write tid 5 to smithi20328086-40 2024-09-17T15:28:02.441 INFO:tasks.rados.rados.0.smithi203.stdout:101: oid 40 updating version 0 to 17 2024-09-17T15:28:02.441 INFO:tasks.rados.rados.0.smithi203.stdout:101: oid 40 version 17 is already newer than 14 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:101: oid 40 version 17 is already newer than 15 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:101: oid 40 version 17 is already newer than 16 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 17 (ObjNum 69 snap 4 seq_num 69) dirty exists 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:101: left oid 40 (ObjNum 69 snap 4 seq_num 69) 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: finishing write tid 1 to smithi20328086-32 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: finishing write tid 2 to smithi20328086-32 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: finishing write tid 3 to smithi20328086-32 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: finishing write tid 4 to smithi20328086-32 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: oid 32 updating version 0 to 14 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: oid 32 version 14 is already newer than 12 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: oid 32 version 14 is already newer than 13 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 14 (ObjNum 70 snap 4 seq_num 70) dirty exists 2024-09-17T15:28:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:103: left oid 32 (ObjNum 70 snap 4 seq_num 70) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:100: done (4 left) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:101: done (3 left) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:102: done (2 left) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:103: done (1 left) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:104: done (0 left) 2024-09-17T15:28:03.202 INFO:tasks.rados.rados.0.smithi203.stdout:105: delete oid 16 current snap is 5 2024-09-17T15:28:03.212 INFO:tasks.rados.rados.0.smithi203.stdout:105: done (0 left) 2024-09-17T15:28:03.212 INFO:tasks.rados.rados.0.smithi203.stdout:106: delete oid 7 current snap is 5 2024-09-17T15:28:03.218 INFO:tasks.rados.rados.0.smithi203.stdout:106: done (0 left) 2024-09-17T15:28:03.218 INFO:tasks.rados.rados.0.smithi203.stdout:107: snap_remove snap 4 2024-09-17T15:28:04.213 INFO:tasks.rados.rados.0.smithi203.stdout:107: done (0 left) 2024-09-17T15:28:04.213 INFO:tasks.rados.rados.0.smithi203.stdout:108: write oid 31 current snap is 5 2024-09-17T15:28:04.213 INFO:tasks.rados.rados.0.smithi203.stdout:108: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-17T15:28:04.228 INFO:tasks.rados.rados.0.smithi203.stdout:108: writing smithi20328086-31 from 426911 to 967265 tid 1 2024-09-17T15:28:04.251 INFO:tasks.rados.rados.0.smithi203.stdout:108: writing smithi20328086-31 from 1622009 to 2348692 tid 2 2024-09-17T15:28:04.264 INFO:tasks.rados.rados.0.smithi203.stdout:108: writing smithi20328086-31 from 3121996 to 3226911 tid 3 2024-09-17T15:28:04.265 INFO:tasks.rados.rados.0.smithi203.stdout:109: delete oid 33 current snap is 5 2024-09-17T15:28:04.265 INFO:tasks.rados.rados.0.smithi203.stdout:108: finishing write tid 1 to smithi20328086-31 2024-09-17T15:28:04.265 INFO:tasks.rados.rados.0.smithi203.stdout:108: finishing write tid 2 to smithi20328086-31 2024-09-17T15:28:04.269 INFO:tasks.rados.rados.0.smithi203.stdout:108: finishing write tid 3 to smithi20328086-31 2024-09-17T15:28:04.269 INFO:tasks.rados.rados.0.smithi203.stdout:108: finishing write tid 4 to smithi20328086-31 2024-09-17T15:28:04.270 INFO:tasks.rados.rados.0.smithi203.stdout:108: finishing write tid 5 to smithi20328086-31 2024-09-17T15:28:04.270 INFO:tasks.rados.rados.0.smithi203.stdout:108: oid 31 updating version 0 to 10 2024-09-17T15:28:04.270 INFO:tasks.rados.rados.0.smithi203.stdout:108: oid 31 updating version 10 to 11 2024-09-17T15:28:04.270 INFO:tasks.rados.rados.0.smithi203.stdout:108: oid 31 version 11 is already newer than 8 2024-09-17T15:28:04.271 INFO:tasks.rados.rados.0.smithi203.stdout:108: oid 31 version 11 is already newer than 9 2024-09-17T15:28:04.271 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 11 (ObjNum 73 snap 5 seq_num 73) dirty exists 2024-09-17T15:28:04.271 INFO:tasks.rados.rados.0.smithi203.stdout:108: left oid 31 (ObjNum 73 snap 5 seq_num 73) 2024-09-17T15:28:04.272 INFO:tasks.rados.rados.0.smithi203.stdout:108: done (1 left) 2024-09-17T15:28:04.272 INFO:tasks.rados.rados.0.smithi203.stdout:109: done (0 left) 2024-09-17T15:28:04.273 INFO:tasks.rados.rados.0.smithi203.stdout:110: snap_remove snap 1 2024-09-17T15:28:05.210 INFO:tasks.rados.rados.0.smithi203.stdout:110: done (0 left) 2024-09-17T15:28:05.210 INFO:tasks.rados.rados.0.smithi203.stdout:111: delete oid 7 current snap is 5 2024-09-17T15:28:05.215 INFO:tasks.rados.rados.0.smithi203.stdout:111: done (0 left) 2024-09-17T15:28:05.215 INFO:tasks.rados.rados.0.smithi203.stdout:112: write (excl) oid 5 current snap is 5 2024-09-17T15:28:05.215 INFO:tasks.rados.rados.0.smithi203.stdout:112: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-17T15:28:05.237 INFO:tasks.rados.rados.0.smithi203.stdout:112: writing smithi20328086-5 from 477332 to 1243433 tid 1 2024-09-17T15:28:05.253 INFO:tasks.rados.rados.0.smithi203.stdout:112: writing smithi20328086-5 from 1681102 to 2155112 tid 2 2024-09-17T15:28:05.269 INFO:tasks.rados.rados.0.smithi203.stdout:112: writing smithi20328086-5 from 2687912 to 3277332 tid 3 2024-09-17T15:28:05.269 INFO:tasks.rados.rados.0.smithi203.stdout:113: write (excl) oid 15 current snap is 5 2024-09-17T15:28:05.269 INFO:tasks.rados.rados.0.smithi203.stdout:112: finishing write tid 1 to smithi20328086-5 2024-09-17T15:28:05.270 INFO:tasks.rados.rados.0.smithi203.stdout:112: finishing write tid 2 to smithi20328086-5 2024-09-17T15:28:05.270 INFO:tasks.rados.rados.0.smithi203.stdout:113: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-17T15:28:05.282 INFO:tasks.rados.rados.0.smithi203.stdout:113: writing smithi20328086-15 from 494139 to 1051842 tid 1 2024-09-17T15:28:05.296 INFO:tasks.rados.rados.0.smithi203.stdout:113: writing smithi20328086-15 from 1455937 to 2189958 tid 2 2024-09-17T15:28:05.309 INFO:tasks.rados.rados.0.smithi203.stdout:113: writing smithi20328086-15 from 2948041 to 3294139 tid 3 2024-09-17T15:28:05.309 INFO:tasks.rados.rados.0.smithi203.stdout:114: delete oid 30 current snap is 5 2024-09-17T15:28:05.309 INFO:tasks.rados.rados.0.smithi203.stdout:112: finishing write tid 3 to smithi20328086-5 2024-09-17T15:28:05.309 INFO:tasks.rados.rados.0.smithi203.stdout:112: finishing write tid 4 to smithi20328086-5 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: finishing write tid 5 to smithi20328086-5 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: oid 5 updating version 0 to 32 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: oid 5 version 32 is already newer than 31 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: oid 5 updating version 32 to 34 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: oid 5 version 34 is already newer than 33 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 34 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:112: left oid 5 (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:113: finishing write tid 1 to smithi20328086-15 2024-09-17T15:28:05.310 INFO:tasks.rados.rados.0.smithi203.stdout:113: finishing write tid 2 to smithi20328086-15 2024-09-17T15:28:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:113: finishing write tid 3 to smithi20328086-15 2024-09-17T15:28:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:113: finishing write tid 4 to smithi20328086-15 2024-09-17T15:28:05.318 INFO:tasks.rados.rados.0.smithi203.stdout:113: finishing write tid 5 to smithi20328086-15 2024-09-17T15:28:05.318 INFO:tasks.rados.rados.0.smithi203.stdout:113: oid 15 updating version 0 to 16 2024-09-17T15:28:05.318 INFO:tasks.rados.rados.0.smithi203.stdout:113: oid 15 version 16 is already newer than 14 2024-09-17T15:28:05.318 INFO:tasks.rados.rados.0.smithi203.stdout:113: oid 15 version 16 is already newer than 15 2024-09-17T15:28:05.318 INFO:tasks.rados.rados.0.smithi203.stdout:113: oid 15 updating version 16 to 17 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 17 (ObjNum 77 snap 5 seq_num 77) dirty exists 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:113: left oid 15 (ObjNum 77 snap 5 seq_num 77) 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:112: done (2 left) 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:113: done (1 left) 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:114: done (0 left) 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:115: write oid 1 current snap is 5 2024-09-17T15:28:05.319 INFO:tasks.rados.rados.0.smithi203.stdout:115: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-17T15:28:05.332 INFO:tasks.rados.rados.0.smithi203.stdout:115: writing smithi20328086-1 from 527753 to 1235954 tid 1 2024-09-17T15:28:05.344 INFO:tasks.rados.rados.0.smithi203.stdout:115: writing smithi20328086-1 from 1856548 to 2594238 tid 2 2024-09-17T15:28:05.351 INFO:tasks.rados.rados.0.smithi203.stdout:115: writing smithi20328086-1 from 3286534 to 3327753 tid 3 2024-09-17T15:28:05.351 INFO:tasks.rados.rados.0.smithi203.stdout:116: copy_from oid 26 from oid 46 current snap is 5 2024-09-17T15:28:05.351 INFO:tasks.rados.rados.0.smithi203.stdout:115: finishing write tid 1 to smithi20328086-1 2024-09-17T15:28:05.351 INFO:tasks.rados.rados.0.smithi203.stdout:117: snap_create 2024-09-17T15:28:05.354 INFO:tasks.rados.rados.0.smithi203.stdout:115: finishing write tid 2 to smithi20328086-1 2024-09-17T15:28:05.354 INFO:tasks.rados.rados.0.smithi203.stdout:115: finishing write tid 3 to smithi20328086-1 2024-09-17T15:28:05.354 INFO:tasks.rados.rados.0.smithi203.stdout:115: finishing write tid 4 to smithi20328086-1 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: finishing write tid 5 to smithi20328086-1 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: oid 1 updating version 0 to 19 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: oid 1 version 19 is already newer than 18 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: oid 1 version 19 is already newer than 17 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: oid 1 updating version 19 to 20 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 20 (ObjNum 79 snap 5 seq_num 79) dirty exists 2024-09-17T15:28:05.355 INFO:tasks.rados.rados.0.smithi203.stdout:115: left oid 1 (ObjNum 79 snap 5 seq_num 79) 2024-09-17T15:28:05.378 INFO:tasks.rados.rados.0.smithi203.stdout:116: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:28:05.387 INFO:tasks.rados.rados.0.smithi203.stdout:116: finishing copy_from to smithi20328086-26 2024-09-17T15:28:05.387 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 17 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-17T15:28:06.227 INFO:tasks.rados.rados.0.smithi203.stdout:115: done (2 left) 2024-09-17T15:28:06.227 INFO:tasks.rados.rados.0.smithi203.stdout:116: done (1 left) 2024-09-17T15:28:06.227 INFO:tasks.rados.rados.0.smithi203.stdout:117: done (0 left) 2024-09-17T15:28:06.227 INFO:tasks.rados.rados.0.smithi203.stdout:118: read oid 26 snap -1 2024-09-17T15:28:06.227 INFO:tasks.rados.rados.0.smithi203.stdout:118: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-17T15:28:06.228 INFO:tasks.rados.rados.0.smithi203.stdout:119: snap_remove snap 5 2024-09-17T15:28:07.235 INFO:tasks.rados.rados.0.smithi203.stdout:119: done (1 left) 2024-09-17T15:28:07.235 INFO:tasks.rados.rados.0.smithi203.stdout:120: write oid 46 current snap is 6 2024-09-17T15:28:07.235 INFO:tasks.rados.rados.0.smithi203.stdout:120: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-17T15:28:07.246 INFO:tasks.rados.rados.0.smithi203.stdout:120: writing smithi20328086-46 from 544560 to 1328010 tid 1 2024-09-17T15:28:07.255 INFO:tasks.rados.rados.0.smithi203.stdout:120: writing smithi20328086-46 from 1915030 to 2396378 tid 2 2024-09-17T15:28:07.263 INFO:tasks.rados.rados.0.smithi203.stdout:120: writing smithi20328086-46 from 2797604 to 3277365 tid 3 2024-09-17T15:28:07.263 INFO:tasks.rados.rados.0.smithi203.stdout:121: delete oid 6 current snap is 6 2024-09-17T15:28:07.263 INFO:tasks.rados.rados.0.smithi203.stdout:118: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-17T15:28:07.397 INFO:tasks.rados.rados.0.smithi203.stdout:120: finishing write tid 1 to smithi20328086-46 2024-09-17T15:28:07.397 INFO:tasks.rados.rados.0.smithi203.stdout:120: finishing write tid 2 to smithi20328086-46 2024-09-17T15:28:07.397 INFO:tasks.rados.rados.0.smithi203.stdout:120: finishing write tid 3 to smithi20328086-46 2024-09-17T15:28:07.397 INFO:tasks.rados.rados.0.smithi203.stdout:120: finishing write tid 4 to smithi20328086-46 2024-09-17T15:28:07.397 INFO:tasks.rados.rados.0.smithi203.stdout:120: finishing write tid 5 to smithi20328086-46 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:120: oid 46 updating version 0 to 23 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:120: oid 46 updating version 23 to 24 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:120: oid 46 updating version 24 to 26 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:120: oid 46 version 26 is already newer than 25 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 26 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-09-17T15:28:07.398 INFO:tasks.rados.rados.0.smithi203.stdout:120: left oid 46 (ObjNum 80 snap 6 seq_num 80) 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:118: done (2 left) 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:120: done (1 left) 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:121: done (0 left) 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:122: read oid 28 snap -1 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:122: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:123: write (excl) oid 37 current snap is 6 2024-09-17T15:28:07.913 INFO:tasks.rados.rados.0.smithi203.stdout:123: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-17T15:28:07.923 INFO:tasks.rados.rados.0.smithi203.stdout:123: writing smithi20328086-37 from 578174 to 1112122 tid 1 2024-09-17T15:28:07.932 INFO:tasks.rados.rados.0.smithi203.stdout:123: writing smithi20328086-37 from 1515641 to 2000658 tid 2 2024-09-17T15:28:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:123: writing smithi20328086-37 from 2736097 to 3238402 tid 3 2024-09-17T15:28:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:124: read oid 23 snap -1 2024-09-17T15:28:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:124: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-17T15:28:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:125: read oid 43 snap -1 2024-09-17T15:28:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:125: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-17T15:28:07.945 INFO:tasks.rados.rados.0.smithi203.stdout:126: copy_from oid 26 from oid 13 current snap is 6 2024-09-17T15:28:07.945 INFO:tasks.rados.rados.0.smithi203.stdout:122: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-17T15:28:08.014 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: finishing write tid 1 to smithi20328086-37 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: finishing write tid 2 to smithi20328086-37 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: finishing write tid 3 to smithi20328086-37 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: finishing write tid 4 to smithi20328086-37 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: finishing write tid 5 to smithi20328086-37 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: oid 37 updating version 0 to 25 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: oid 37 updating version 25 to 28 2024-09-17T15:28:08.072 INFO:tasks.rados.rados.0.smithi203.stdout:123: oid 37 version 28 is already newer than 27 2024-09-17T15:28:08.073 INFO:tasks.rados.rados.0.smithi203.stdout:123: oid 37 version 28 is already newer than 26 2024-09-17T15:28:08.073 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 28 (ObjNum 82 snap 6 seq_num 82) dirty exists 2024-09-17T15:28:08.073 INFO:tasks.rados.rados.0.smithi203.stdout:123: left oid 37 (ObjNum 82 snap 6 seq_num 82) 2024-09-17T15:28:08.073 INFO:tasks.rados.rados.0.smithi203.stdout:124: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-17T15:28:08.094 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:08.132 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2347019 2024-09-17T15:28:08.132 INFO:tasks.rados.rados.0.smithi203.stdout:125: expect (ObjNum 51 snap 0 seq_num 51) 2024-09-17T15:28:08.215 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:126: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:126: finishing copy_from to smithi20328086-26 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 17 (ObjNum 62 snap 2 seq_num 62) dirty exists 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:122: done (4 left) 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:123: done (3 left) 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:124: done (2 left) 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:125: done (1 left) 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:126: done (0 left) 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:127: copy_from oid 7 from oid 48 current snap is 6 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:128: write oid 45 current snap is 6 2024-09-17T15:28:08.261 INFO:tasks.rados.rados.0.smithi203.stdout:128: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-17T15:28:08.283 INFO:tasks.rados.rados.0.smithi203.stdout:128: writing smithi20328086-45 from 594981 to 1204178 tid 1 2024-09-17T15:28:08.307 INFO:tasks.rados.rados.0.smithi203.stdout:128: writing smithi20328086-45 from 1974123 to 2719151 tid 2 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:128: writing smithi20328086-45 from 3279873 to 3394981 tid 3 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:129: snap_create 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:127: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:127: finishing copy_from to smithi20328086-7 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:128: finishing write tid 1 to smithi20328086-45 2024-09-17T15:28:08.317 INFO:tasks.rados.rados.0.smithi203.stdout:128: finishing write tid 2 to smithi20328086-45 2024-09-17T15:28:08.320 INFO:tasks.rados.rados.0.smithi203.stdout:128: finishing write tid 3 to smithi20328086-45 2024-09-17T15:28:08.321 INFO:tasks.rados.rados.0.smithi203.stdout:128: finishing write tid 4 to smithi20328086-45 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: finishing write tid 5 to smithi20328086-45 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: oid 45 updating version 0 to 10 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: oid 45 updating version 10 to 13 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: oid 45 version 13 is already newer than 12 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: oid 45 version 13 is already newer than 11 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 13 (ObjNum 83 snap 6 seq_num 83) dirty exists 2024-09-17T15:28:08.322 INFO:tasks.rados.rados.0.smithi203.stdout:128: left oid 45 (ObjNum 83 snap 6 seq_num 83) 2024-09-17T15:28:09.238 INFO:tasks.rados.rados.0.smithi203.stdout:127: done (2 left) 2024-09-17T15:28:09.239 INFO:tasks.rados.rados.0.smithi203.stdout:128: done (1 left) 2024-09-17T15:28:09.239 INFO:tasks.rados.rados.0.smithi203.stdout:129: done (0 left) 2024-09-17T15:28:09.239 INFO:tasks.rados.rados.0.smithi203.stdout:130: snap_create 2024-09-17T15:28:10.637 INFO:tasks.rados.rados.0.smithi203.stdout:130: done (0 left) 2024-09-17T15:28:10.637 INFO:tasks.rados.rados.0.smithi203.stdout:131: read oid 28 snap 6 2024-09-17T15:28:10.637 INFO:tasks.rados.rados.0.smithi203.stdout:131: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-17T15:28:10.637 INFO:tasks.rados.rados.0.smithi203.stdout:132: write (excl) oid 12 current snap is 8 2024-09-17T15:28:10.637 INFO:tasks.rados.rados.0.smithi203.stdout:132: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-17T15:28:10.654 INFO:tasks.rados.rados.0.smithi203.stdout:132: writing smithi20328086-12 from 611788 to 1012587 tid 1 2024-09-17T15:28:10.674 INFO:tasks.rados.rados.0.smithi203.stdout:132: writing smithi20328086-12 from 1748958 to 2237644 tid 2 2024-09-17T15:28:10.692 INFO:tasks.rados.rados.0.smithi203.stdout:132: writing smithi20328086-12 from 2907296 to 3411788 tid 3 2024-09-17T15:28:10.692 INFO:tasks.rados.rados.0.smithi203.stdout:133: copy_from oid 40 from oid 24 current snap is 8 2024-09-17T15:28:10.692 INFO:tasks.rados.rados.0.smithi203.stdout:131: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: finishing write tid 1 to smithi20328086-12 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: finishing write tid 2 to smithi20328086-12 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: finishing write tid 3 to smithi20328086-12 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: finishing write tid 4 to smithi20328086-12 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: finishing write tid 5 to smithi20328086-12 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: oid 12 updating version 0 to 21 2024-09-17T15:28:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:132: oid 12 updating version 21 to 23 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:132: oid 12 updating version 23 to 24 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:132: oid 12 version 24 is already newer than 22 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 24 (ObjNum 84 snap 8 seq_num 84) dirty exists 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:132: left oid 12 (ObjNum 84 snap 8 seq_num 84) 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:133: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:133: finishing copy_from to smithi20328086-40 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:131: done (2 left) 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:132: done (1 left) 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:133: done (0 left) 2024-09-17T15:28:10.812 INFO:tasks.rados.rados.0.smithi203.stdout:134: delete oid 32 current snap is 8 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:134: done (0 left) 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:135: read oid 46 snap -1 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:135: expect (ObjNum 80 snap 6 seq_num 80) 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:136: read oid 8 snap -1 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:136: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:28:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:137: delete oid 29 current snap is 8 2024-09-17T15:28:10.827 INFO:tasks.rados.rados.0.smithi203.stdout:137: done (2 left) 2024-09-17T15:28:10.827 INFO:tasks.rados.rados.0.smithi203.stdout:138: delete oid 31 current snap is 8 2024-09-17T15:28:10.829 INFO:tasks.rados.rados.0.smithi203.stdout:136: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:28:10.830 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:10.871 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1575481 2024-09-17T15:28:10.889 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:10.917 INFO:tasks.rados.rados.0.smithi203.stdout:135: expect (ObjNum 80 snap 6 seq_num 80) 2024-09-17T15:28:10.983 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3083350 2024-09-17T15:28:10.994 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:11.046 INFO:tasks.rados.rados.0.smithi203.stdout:135: done (2 left) 2024-09-17T15:28:11.047 INFO:tasks.rados.rados.0.smithi203.stdout:136: done (1 left) 2024-09-17T15:28:11.047 INFO:tasks.rados.rados.0.smithi203.stdout:138: done (0 left) 2024-09-17T15:28:11.047 INFO:tasks.rados.rados.0.smithi203.stdout:139: snap_create 2024-09-17T15:28:11.316 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:28:11.317 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:28:11.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:28:11.320 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:28:11.320 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:28:11.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:11.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 10 monclient: get_monmap_and_config 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 10 monclient: build_initial_monmap 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 10 monclient: monmap: 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:11.449965+0000 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:11.449965+0000 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:11.448 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:11.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 1 Processor -- start 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 1 -- start start 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.447+0000 7f0764f7a640 10 monclient: init 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-c con 0x7f07600689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-a con 0x7f0760068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0760135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f076009c8b0 con 0x7f0760135ae0 2024-09-17T15:28:11.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f076009cb90 con 0x7f0760068db0 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f076009ce70 con 0x7f07600689d0 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f0764f7a640 10 monclient(hunting): authenticate will time out at 1024.416289s 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49984/0 (socket says 172.21.15.86:49984) 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 learned_addr learned my addr 172.21.15.86:0/2811447311 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0760068db0 auth_method 0 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:11.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f07600689d0 auth_method 0 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:11.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:11.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.451+0000 7f075ed9d640 1 -- 172.21.15.86:0/2811447311 >> v1:172.21.15.203:6789/0 conn(0x7f0760135ae0 legacy=0x7f0760135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0760068db0 auth_method 0 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f07600689d0 auth_method 0 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:11.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:11.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:11.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:11.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:11.655+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:12.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0760068db0 auth_method 0 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f07600689d0 auth_method 0 2024-09-17T15:28:12.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:12.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0760068db0 auth_method 0 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f07600689d0 auth_method 0 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:12.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): tick 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 msgr2=0x7f076006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.203:3300/0 conn(0x7f07600689d0 0x7f076006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 -- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 msgr2=0x7f0760069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 --2- 172.21.15.86:0/2811447311 >> v2:172.21.15.86:3300/0 conn(0x7f0760068db0 0x7f0760069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 -- 172.21.15.86:0/2811447311 >> v1:172.21.15.203:6789/0 conn(0x7f0760135ae0 legacy=0x7f0760135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient: _add_conns ranks=[2,0] 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f0740003740 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07400062a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 -- 172.21.15.86:0/2811447311 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0760069740 con 0x7f07400062a0 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 1 -- 172.21.15.86:0/2811447311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f076009cb90 con 0x7f0740003740 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f074ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0740003740 auth_method 0 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:14.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:14.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:14.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:14.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:14.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.450+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> v1:172.21.15.86:6789/0 conn(0x7f07400062a0 legacy=0x7f0740008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:14.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:14.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0740003740 auth_method 0 2024-09-17T15:28:14.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:14.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:14.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:14.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:14.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:14.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:14.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:14.654+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:15.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:15.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0740003740 auth_method 0 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:15.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.058+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:15.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0740003740 auth_method 0 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:15.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:15.862+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1433896643 0 0) 0x7f0760069740 con 0x7f07400062a0 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): _negotiate 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): my global_id is 4365 2024-09-17T15:28:16.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0748003530 con 0x7f07400062a0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 187395592 0 0) 0x7f0748003530 con 0x7f07400062a0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 msgr2=0x7f0740005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0740003740 0x7f0740005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _finish_hunting 0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074000b0f0 con 0x7f07400062a0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _finish_auth 0 2024-09-17T15:28:16.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f075400f040 con 0x7f07400062a0 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: dump: 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:16.463 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 >> v1:172.21.15.86:6789/0 conn(0x7f07400062a0 legacy=0x7f0740008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 1 -- 172.21.15.86:0/2811447311 reap_dead start 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f0748003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): picked mon.b con 0x7f0748003ad0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 0x7f0748004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): picked mon.c con 0x7f0748004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 0x7f074800ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): picked mon.a con 0x7f0748005400 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:16.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f076009cb90 con 0x7f0748005400 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f074800c350 con 0x7f0748003ad0 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f074800c550 con 0x7f0748004810 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075d59a640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 0x7f0748004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 0x7f074800ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0748004810 auth_method 0 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): get_auth_request con 0x7f0748005400 auth_method 0 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f0748003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0748003ad0 auth_method 0 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:16.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient(hunting): handle_auth_done global_id 4365 payload 293 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f0748003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f0748003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 msgr2=0x7f074800ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 0x7f074800ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.458+0000 7f075dd9b640 1 monclient: found mon.c 2024-09-17T15:28:16.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0748003530 con 0x7f0748004810 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f0764f7a640 5 monclient: authenticate success, global_id 4365 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f0764f7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:16.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.462+0000 7f075ed9d640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 0x7f074800ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:16.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.466+0000 7f075dd9b640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 msgr2=0x7f0748004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:16.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.466+0000 7f075dd9b640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 0x7f0748004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f075400c900 tx=0x7f07540201e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4197acf39359e2c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540af070 con 0x7f0748004810 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: dump: 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:16.480 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07540b3400 con 0x7f0748004810 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 1 -- 172.21.15.86:0/2811447311 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07540142f0 con 0x7f0748004810 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 4 set_mon_vals no callback set 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f075d59a640 10 monclient: dump: 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:16.481 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:16.482 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 10 monclient: shutdown 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 -- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 msgr2=0x7f0748004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 0x7f0748004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f075400c900 tx=0x7f07540201e0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 -- 172.21.15.86:0/2811447311 shutdown_connections 2024-09-17T15:28:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748005400 0x7f074800ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748004810 0x7f0748004c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 --2- 172.21.15.86:0/2811447311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f0748003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 -- 172.21.15.86:0/2811447311 >> 172.21.15.86:0/2811447311 conn(0x7f07600a6440 msgr2=0x7f07600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 -- 172.21.15.86:0/2811447311 shutdown_connections 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.478+0000 7f0764f7a640 1 -- 172.21.15.86:0/2811447311 wait complete. 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient: build_initial_monmap 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient: monmap: 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:16.485 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:16.485 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:16.485 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 Processor -- start 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 -- start start 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient: init 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-b con 0x7f0748003ad0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-a con 0x7f0748004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): picked mon.noname-c con 0x7f0748005400 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f076009c240 con 0x7f0748004810 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f076009c0c0 con 0x7f0748003ad0 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f076006f840 con 0x7f0748005400 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f0764f7a640 10 monclient(hunting): authenticate will time out at 1029.451295s 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42306/0 (socket says 172.21.15.86:42306) 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 learned_addr learned my addr 172.21.15.86:0/3616448963 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 10 monclient(hunting): get_auth_request con 0x7f0748005400 auth_method 0 2024-09-17T15:28:16.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0748004810 auth_method 0 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 msgr2=0x7f07601236d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 msgr2=0x7f07601236d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 msgr2=0x7f07601cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 msgr2=0x7f07601cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:16.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075dd9b640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0748003ad0 auth_method 0 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f07601cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f07601cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:16.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.482+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): get_auth_request con 0x7f0748005400 auth_method 0 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): get_auth_request con 0x7f0748004810 auth_method 0 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 10 monclient(hunting): get_auth_request con 0x7f0748003ad0 auth_method 0 2024-09-17T15:28:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f07601cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f07601cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient(hunting): handle_auth_done global_id 4349 payload 824 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 msgr2=0x7f07601cd0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 msgr2=0x7f07601cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:28:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 monclient: found mon.noname-c 2024-09-17T15:28:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0760134a90 con 0x7f0748005400 2024-09-17T15:28:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075dd9b640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 msgr2=0x7f07601236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.686+0000 7f075ed9d640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0760125c30 tx=0x7f0758005ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758011a10 con 0x7f0748005400 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: dump: 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: _finish_auth 0 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0758007e50 con 0x7f0748005400 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758011dd0 con 0x7f0748005400 2024-09-17T15:28:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 5 monclient: authenticate success, global_id 4349 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _renew_subs 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f076009d460 con 0x7f0748005400 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 10 monclient: dump: 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:16.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:16.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074ffff640 4 set_mon_vals no callback set 2024-09-17T15:28:16.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _renew_subs 2024-09-17T15:28:16.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0760127210 con 0x7f0748005400 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f076006fb70 con 0x7f0748005400 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0758013040 con 0x7f0748005400 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 0x7f074003eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 4755+0+0 (secure 0 0 0) 0x7f075804fff0 con 0x7f0748005400 2024-09-17T15:28:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 0x7f074003eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.690+0000 7f075e59c640 10 monclient: get_auth_request con 0x7f074003c660 auth_method 0 2024-09-17T15:28:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.694+0000 7f075e59c640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 0x7f074003eb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0750009fd0 tx=0x7f075000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.698+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f07580015a0 con 0x7f0748005400 2024-09-17T15:28:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.698+0000 7f074effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.698+0000 7f074effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:16.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.950+0000 7f0764f7a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:16.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.950+0000 7f0764f7a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:16.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.950+0000 7f0764f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:16.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.950+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0760125cf0 con 0x7f0748005400 2024-09-17T15:28:16.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f074effd640 1 -- 172.21.15.86:0/3616448963 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0758015c20 con 0x7f0748005400 2024-09-17T15:28:16.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f074effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:16.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f074effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:16.954 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:16.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 msgr2=0x7f074003eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 0x7f074003eb20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0750009fd0 tx=0x7f075000b040 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 10 monclient: shutdown 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 msgr2=0x7f07601236d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0760125c30 tx=0x7f0758005ba0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 shutdown_connections 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f074003c660 0x7f074003eb20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0748005400 0x7f07601236d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0748004810 0x7f07601cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 --2- 172.21.15.86:0/3616448963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0748003ad0 0x7f07601cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 >> 172.21.15.86:0/3616448963 conn(0x7f07600a6440 msgr2=0x7f07600a27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:16.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 shutdown_connections 2024-09-17T15:28:16.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:16.954+0000 7f0764f7a640 1 -- 172.21.15.86:0/3616448963 wait complete. 2024-09-17T15:28:16.973 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:28:16.973 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:28:16.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: get_monmap_and_config 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: monmap: 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:17.092845+0000 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:17.092845+0000 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:17.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:17.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:17.092 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:17.092 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:17.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 Processor -- start 2024-09-17T15:28:17.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 -- start start 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: init 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fcd340a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcd340a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcd3413d2f0 0x7fcd3413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fcd3413d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd340a6280 con 0x7fcd340a54c0 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcd3409d2a0 con 0x7fcd3413d2f0 2024-09-17T15:28:17.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcd3409c0a0 con 0x7fcd340a50e0 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd39a3e640 10 monclient(hunting): authenticate will time out at 1030.059242s 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42324/0 (socket says 172.21.15.86:42324) 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 -- 172.21.15.86:0/2082645981 learned_addr learned my addr 172.21.15.86:0/2082645981 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): get_auth_request con 0x7fcd340a50e0 auth_method 0 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd337fe640 1 --2- 172.21.15.86:0/2082645981 >> v2:172.21.15.203:3300/0 conn(0x7fcd3413d2f0 0x7fcd3413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd337fe640 10 monclient(hunting): get_auth_request con 0x7fcd3413d2f0 auth_method 0 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 -- 172.21.15.86:0/2082645981 >> v2:172.21.15.203:3300/0 conn(0x7fcd3413d2f0 msgr2=0x7fcd3413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 --2- 172.21.15.86:0/2082645981 >> v2:172.21.15.203:3300/0 conn(0x7fcd3413d2f0 0x7fcd3413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 -- 172.21.15.86:0/2082645981 >> v1:172.21.15.203:6789/0 conn(0x7fcd340a54c0 legacy=0x7fcd340a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 monclient: found mon.noname-b 2024-09-17T15:28:17.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.090+0000 7fcd32ffd640 1 -- 172.21.15.86:0/2082645981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd3409c380 con 0x7fcd340a50e0 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.094+0000 7fcd32ffd640 1 -- 172.21.15.86:0/2082645981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 msgr2=0x7fcd3409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.094+0000 7fcd32ffd640 1 --2- 172.21.15.86:0/2082645981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd2401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=975b78740f169454 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 1 -- 172.21.15.86:0/2082645981 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b0070 con 0x7fcd340a50e0 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: dump: 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.102 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 1 -- 172.21.15.86:0/2082645981 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd240ab600 con 0x7fcd340a50e0 2024-09-17T15:28:17.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 5 monclient: authenticate success, global_id 4352 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 1 -- 172.21.15.86:0/2082645981 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd240ab9d0 con 0x7fcd340a50e0 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 4 set_mon_vals no callback set 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd31ffb640 10 monclient: dump: 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: shutdown 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- 172.21.15.86:0/2082645981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 msgr2=0x7fcd3409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/2082645981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009870 tx=0x7fcd2401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- 172.21.15.86:0/2082645981 shutdown_connections 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/2082645981 >> v2:172.21.15.203:3300/0 conn(0x7fcd3413d2f0 0x7fcd3413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/2082645981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd3409cd60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- 172.21.15.86:0/2082645981 >> 172.21.15.86:0/2082645981 conn(0x7fcd340aaae0 msgr2=0x7fcd340aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- 172.21.15.86:0/2082645981 shutdown_connections 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- 172.21.15.86:0/2082645981 wait complete. 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: monmap: 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:17.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 Processor -- start 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- start start 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: init 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd341340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fcd340a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd340a54c0 0x7fcd34134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fcd340a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 0x7fcd340a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fcd3413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcd34142650 con 0x7fcd3413d2f0 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcd340a6b00 con 0x7fcd340a54c0 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcd341427d0 con 0x7fcd340a50e0 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd39a3e640 10 monclient(hunting): authenticate will time out at 1030.066281s 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd341340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 0x7fcd340a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd341340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42328/0 (socket says 172.21.15.86:42328) 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 0x7fcd340a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50026/0 (socket says 172.21.15.86:50026) 2024-09-17T15:28:17.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 -- 172.21.15.86:0/434776953 learned_addr learned my addr 172.21.15.86:0/434776953 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): get_auth_request con 0x7fcd340a50e0 auth_method 0 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): get_auth_request con 0x7fcd3413d2f0 auth_method 0 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd340a54c0 0x7fcd34134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd340a54c0 msgr2=0x7fcd34134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd340a54c0 0x7fcd34134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 msgr2=0x7fcd340a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 0x7fcd340a1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 monclient: found mon.noname-c 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd32ffd640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd34139b50 con 0x7fcd340a50e0 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd327fc640 10 monclient: get_auth_request con 0x7fcd340a54c0 auth_method 0 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd340a54c0 0x7fcd34134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:17.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.098+0000 7fcd337fe640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3413d2f0 0x7fcd340a1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd32ffd640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 msgr2=0x7fcd341340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd32ffd640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd341340c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd240ae040 tx=0x7fcd2401ac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd240ba020 con 0x7fcd340a50e0 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: dump: 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.114 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 5 monclient: authenticate success, global_id 4355 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd240b6630 con 0x7fcd340a50e0 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd240b0040 con 0x7fcd340a50e0 2024-09-17T15:28:17.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 10 monclient: dump: 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd317fa640 4 set_mon_vals no callback set 2024-09-17T15:28:17.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 monclient: _renew_subs 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd34139e30 con 0x7fcd340a50e0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 monclient: _renew_subs 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd340a18d0 con 0x7fcd340a50e0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 0x7fcd340a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3406a0e0 con 0x7fcd340a2460 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 0x7fcd340a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient: get_auth_request con 0x7fcd340a2460 auth_method 0 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcd240bd020 con 0x7fcd340a50e0 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd327fc640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd3413da90 con 0x7fcd340a2460 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcd1803bb60 0x7fcd1803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(36..36 src has 1..36) ==== 4755+0+0 (secure 0 0 0) 0x7fcd240ea300 con 0x7fcd340a50e0 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd337fe640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcd1803bb60 0x7fcd1803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.102+0000 7fcd337fe640 10 monclient: get_auth_request con 0x7fcd1803bb60 auth_method 0 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd327fc640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 msgr2=0x7fcd340a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 0x7fcd340a2840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00d830 tx=0x7fcd1c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c006d60 con 0x7fcd340a2460 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 10 monclient: dump: 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd1c006d60 con 0x7fcd340a2460 2024-09-17T15:28:17.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 msgr2=0x7fcd340a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 0x7fcd340a2840 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00d830 tx=0x7fcd1c00dcf0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 msgr2=0x7fcd340a2840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd2bfff640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2460 0x7fcd340a2840 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c00d830 tx=0x7fcd1c00dcf0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.106+0000 7fcd337fe640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcd1803bb60 0x7fcd1803e020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd3413a850 tx=0x7fcd2c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd39a3e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 0x7fcd3409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd39a3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3413d6d0 con 0x7fcd340a2d80 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 0x7fcd3409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient: get_auth_request con 0x7fcd340a2d80 auth_method 0 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.110+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd327fc640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-09-17T15:28:17.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd327fc640 1 -- 172.21.15.86:0/434776953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd3409cd60 con 0x7fcd340a2d80 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd327fc640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 msgr2=0x7fcd3409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd327fc640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 0x7fcd3409e5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c017000 tx=0x7fcd1c003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c01a3f0 con 0x7fcd340a2d80 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 10 monclient: dump: 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fcd1c0243f0 con 0x7fcd340a2d80 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:17.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 msgr2=0x7fcd3409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 0x7fcd3409e5e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c017000 tx=0x7fcd1c003af0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 msgr2=0x7fcd3409e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd2bfff640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd340a2d80 0x7fcd3409e5e0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c017000 tx=0x7fcd1c003af0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:17.124 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 69758, 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.125 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.126 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:17.127 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:17.128 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:17.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:17.130 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.131 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.132 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.134 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.135 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.136 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:28:17.137 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.137 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.137 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:17.137 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcd1803bb60 msgr2=0x7fcd1803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcd1803bb60 0x7fcd1803e020 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcd3413a850 tx=0x7fcd2c008040 comp rx=0 tx=0).stop 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 10 monclient: shutdown 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 msgr2=0x7fcd341340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 --2- 172.21.15.86:0/434776953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd340a50e0 0x7fcd341340c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd240ae040 tx=0x7fcd2401ac30 comp rx=0 tx=0).stop 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd337fe640 1 -- 172.21.15.86:0/434776953 reap_dead start 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd337fe640 1 -- 172.21.15.86:0/434776953 reap_dead start 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 shutdown_connections 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 >> 172.21.15.86:0/434776953 conn(0x7fcd340aaae0 msgr2=0x7fcd3413c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 shutdown_connections 2024-09-17T15:28:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.114+0000 7fcd39a3e640 1 -- 172.21.15.86:0/434776953 wait complete. 2024-09-17T15:28:17.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:17.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:28:17.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:28:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: get_monmap_and_config 2024-09-17T15:28:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: monmap: 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:17.252616+0000 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:17.252616+0000 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:17.252 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 Processor -- start 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 -- start start 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: init 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 0x7f0ddc09db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc09d770 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ddc09e090 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ddc135150 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ddc09ff00 con 0x7f0ddc09e090 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ddc09ec30 con 0x7f0ddc135150 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0ddc09e7f0 con 0x7f0ddc09d770 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de45ba640 10 monclient(hunting): authenticate will time out at 1030.218105s 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de1b55640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0ddc09e090 0x7f0ddc134a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42464/0 (socket says 172.21.15.86:42464) 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 learned_addr learned my addr 172.21.15.86:0/2880684220 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 1 --2- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 0x7f0ddc09db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 10 monclient(hunting): get_auth_request con 0x7f0ddc09d770 auth_method 0 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 1 -- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 msgr2=0x7f0ddc09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 1 -- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 msgr2=0x7f0ddc09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 1 --2- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 0x7f0ddc09db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.250+0000 7f0de2356640 1 --2- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 0x7f0ddc09db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 >> v1:172.21.15.86:6789/0 conn(0x7f0ddc09e090 legacy=0x7f0ddc134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 251946581 0 0) 0x7f0ddc09ff00 con 0x7f0ddc09e090 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): _negotiate 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 1 -- 172.21.15.86:0/2880684220 >> v1:172.21.15.203:6789/0 conn(0x7f0ddc135150 legacy=0x7f0ddc13cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): my global_id is 4380 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dc8003530 con 0x7f0ddc09e090 2024-09-17T15:28:17.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 703952458 0 0) 0x7f0dc8003530 con 0x7f0ddc09e090 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 msgr2=0x7f0ddc09db50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 --2- 172.21.15.86:0/2880684220 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc09d770 0x7f0ddc09db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 >> v1:172.21.15.203:6789/0 conn(0x7f0ddc135150 legacy=0x7f0ddc13cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc09ef10 con 0x7f0ddc09e090 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de45ba640 5 monclient: authenticate success, global_id 4380 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de45ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f0dd00019b0 con 0x7f0ddc09e090 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: dump: 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 >> v1:172.21.15.86:6789/0 conn(0x7f0ddc09e090 legacy=0x7f0ddc134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dc8003b20 0x7f0dc8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): picked mon.b con 0x7f0dc8003b20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:17.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 0x7f0dc8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): picked mon.a con 0x7f0dc8004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc800bbd0 0x7f0dc800dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): picked mon.c con 0x7f0dc800bbd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc800e5b0 con 0x7f0dc8004810 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc800e760 con 0x7f0dc8003b20 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0dc800e960 con 0x7f0dc800bbd0 2024-09-17T15:28:17.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1354640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de45ba640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 0x7f0dc8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): get_auth_request con 0x7f0dc8004810 auth_method 0 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc800bbd0 0x7f0dc800dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dc8003b20 0x7f0dc8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): get_auth_request con 0x7f0dc800bbd0 auth_method 0 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 10 monclient(hunting): get_auth_request con 0x7f0dc8003b20 auth_method 0 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient(hunting): handle_auth_done global_id 4380 payload 293 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc800bbd0 msgr2=0x7f0dc800dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc800bbd0 0x7f0dc800dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dc8003b20 msgr2=0x7f0dc8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dc8003b20 0x7f0dc8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 monclient: found mon.a 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc8003530 con 0x7f0dc8004810 2024-09-17T15:28:17.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de1b55640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2356640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dc8003b20 0x7f0dc8003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:28:17.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.254+0000 7f0de2b57640 1 -- 172.21.15.86:0/2880684220 reap_dead start 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1b55640 1 -- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 msgr2=0x7f0dc8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1b55640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 0x7f0dc8004c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00014e0 tx=0x7f0dd001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33de807eda12da37 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd000c0e0 con 0x7f0dc8004810 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: dump: 2024-09-17T15:28:17.275 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd00a9020 con 0x7f0dc8004810 2024-09-17T15:28:17.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 4 set_mon_vals no callback set 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 1 -- 172.21.15.86:0/2880684220 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0005b00 con 0x7f0dc8004810 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de1354640 10 monclient: dump: 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 10 monclient: shutdown 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 -- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 msgr2=0x7f0dc8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 0x7f0dc8004c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0dd00014e0 tx=0x7f0dd001cd00 comp rx=0 tx=0).stop 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 -- 172.21.15.86:0/2880684220 shutdown_connections 2024-09-17T15:28:17.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 --2- 172.21.15.86:0/2880684220 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc8004810 0x7f0dc8004c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 -- 172.21.15.86:0/2880684220 >> 172.21.15.86:0/2880684220 conn(0x7f0ddc0a2f40 msgr2=0x7f0ddc0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 -- 172.21.15.86:0/2880684220 shutdown_connections 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.258+0000 7f0de45ba640 1 -- 172.21.15.86:0/2880684220 wait complete. 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient: monmap: 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 Processor -- start 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 -- start start 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient: init 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc8004810 0x7f0ddc1294e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-c con 0x7f0dc8004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ddc129a20 0x7f0ddc129ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc129a20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 0x7f0ddc1db480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc1d9090 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc134a40 con 0x7f0ddc1d9090 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc0a2110 con 0x7f0ddc129a20 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc134bc0 con 0x7f0dc8004810 2024-09-17T15:28:17.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de45ba640 10 monclient(hunting): authenticate will time out at 1030.229378s 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc8004810 0x7f0ddc1294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 0x7f0ddc1db480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc8004810 0x7f0ddc1294e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42352/0 (socket says 172.21.15.86:42352) 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 1 -- 172.21.15.86:0/3609265107 learned_addr learned my addr 172.21.15.86:0/3609265107 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): get_auth_request con 0x7f0ddc1d9090 auth_method 0 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): get_auth_request con 0x7f0dc8004810 auth_method 0 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de1b55640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ddc129a20 0x7f0ddc129ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de1b55640 10 monclient(hunting): get_auth_request con 0x7f0ddc129a20 auth_method 0 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de1b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de1b55640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de1b55640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc8004810 msgr2=0x7f0ddc1294e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc8004810 0x7f0ddc1294e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ddc129a20 msgr2=0x7f0ddc129ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ddc129a20 0x7f0ddc129ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 monclient: found mon.noname-a 2024-09-17T15:28:17.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.262+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc12cb30 con 0x7f0ddc1d9090 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 msgr2=0x7f0ddc1db480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2b57640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 0x7f0ddc1db480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0a2c60 tx=0x7f0dd80019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8012770 con 0x7f0ddc1d9090 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: dump: 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 5 monclient: authenticate success, global_id 4383 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd8018070 con 0x7f0ddc1d9090 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8012b90 con 0x7f0ddc1d9090 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 10 monclient: dump: 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.287 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de0b53640 4 set_mon_vals no callback set 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 monclient: _renew_subs 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ddc074600 con 0x7f0ddc1d9090 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 monclient: _renew_subs 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ddc12a7d0 con 0x7f0ddc1d9090 2024-09-17T15:28:17.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 0x7f0ddc11cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc09d460 con 0x7f0ddc11c8c0 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0dd801d460 con 0x7f0ddc1d9090 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 0x7f0ddc11cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient: get_auth_request con 0x7f0ddc11c8c0 auth_method 0 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0dc003bbe0 0x7f0dc003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4755+0+0 (secure 0 0 0) 0x7f0dd804fb40 con 0x7f0ddc1d9090 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de2356640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ddc09ff00 con 0x7f0ddc11c8c0 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de1b55640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0dc003bbe0 0x7f0dc003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.266+0000 7f0de1b55640 10 monclient: get_auth_request con 0x7f0dc003bbe0 auth_method 0 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0de2356640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 msgr2=0x7f0ddc11cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0de2356640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 0x7f0ddc11cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd401a020 con 0x7f0ddc11c8c0 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 10 monclient: dump: 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.291 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0dd4016660 con 0x7f0ddc11c8c0 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 msgr2=0x7f0ddc11cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 0x7f0ddc11cca0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4007700 comp rx=0 tx=0).stop 2024-09-17T15:28:17.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 msgr2=0x7f0ddc11cca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0dceffd640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11c8c0 0x7f0ddc11cca0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4007700 comp rx=0 tx=0).stop 2024-09-17T15:28:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.270+0000 7f0de1b55640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0dc003bbe0 0x7f0dc003e0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009000 tx=0x7f0ddc06d7c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de45ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de45ba640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 0x7f0ddc1df9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de45ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc06d4f0 con 0x7f0ddc11d1e0 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 0x7f0ddc1df9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient: get_auth_request con 0x7f0ddc11d1e0 auth_method 0 2024-09-17T15:28:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.274+0000 7f0de2356640 1 -- 172.21.15.86:0/3609265107 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ddc1204c0 con 0x7f0ddc11d1e0 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de2356640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 msgr2=0x7f0ddc1df9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de2356640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 0x7f0ddc1df9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd401a070 con 0x7f0ddc11d1e0 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 10 monclient: dump: 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.297 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f0dd401f410 con 0x7f0ddc11d1e0 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 msgr2=0x7f0ddc1df9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 0x7f0ddc1df9e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4004ac0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 msgr2=0x7f0ddc1df9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0dceffd640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ddc11d1e0 0x7f0ddc1df9e0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4003fd0 tx=0x7f0dd4004ac0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 69922, 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:17.299 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.300 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:17.302 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:17.303 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.304 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.305 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.311 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:17.314 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0dc003bbe0 msgr2=0x7f0dc003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0dc003bbe0 0x7f0dc003e0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009000 tx=0x7f0ddc06d7c0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 10 monclient: shutdown 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 msgr2=0x7f0ddc1db480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 --2- 172.21.15.86:0/3609265107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc1d9090 0x7f0ddc1db480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0a2c60 tx=0x7f0dd80019d0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 reap_dead start 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de2b57640 1 -- 172.21.15.86:0/3609265107 reap_dead start 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 shutdown_connections 2024-09-17T15:28:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 >> 172.21.15.86:0/3609265107 conn(0x7f0ddc0a2f40 msgr2=0x7f0ddc073bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 shutdown_connections 2024-09-17T15:28:17.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.278+0000 7f0de45ba640 1 -- 172.21.15.86:0/3609265107 wait complete. 2024-09-17T15:28:17.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:17.317 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:28:17.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:28:17.381 INFO:tasks.rados.rados.0.smithi203.stdout:139: done (0 left) 2024-09-17T15:28:17.381 INFO:tasks.rados.rados.0.smithi203.stdout:140: read oid 15 snap 8 2024-09-17T15:28:17.381 INFO:tasks.rados.rados.0.smithi203.stdout:140: expect (ObjNum 77 snap 5 seq_num 77) 2024-09-17T15:28:17.381 INFO:tasks.rados.rados.0.smithi203.stdout:141: snap_create 2024-09-17T15:28:17.395 INFO:tasks.rados.rados.0.smithi203.stdout:140: expect (ObjNum 77 snap 5 seq_num 77) 2024-09-17T15:28:17.395 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: get_monmap_and_config 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: monmap: 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:17.436180+0000 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:17.436180+0000 2024-09-17T15:28:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:17.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 Processor -- start 2024-09-17T15:28:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 -- start start 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: init 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50680a50e0 0x7f506809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-a con 0x7f50680a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a54c0 0x7f50680a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-b con 0x7f50680a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 0x7f50681382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-c con 0x7f5068137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f50680a6280 con 0x7f50680a50e0 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f506809c0a0 con 0x7f5068137ea0 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f506809c380 con 0x7f50680a54c0 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506f1e2640 10 monclient(hunting): authenticate will time out at 1030.401655s 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50680a50e0 0x7f506809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a54c0 0x7f50680a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50680a50e0 0x7f506809cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50080/0 (socket says 172.21.15.86:50080) 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 -- 172.21.15.86:0/2537490233 learned_addr learned my addr 172.21.15.86:0/2537490233 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): get_auth_request con 0x7f50680a50e0 auth_method 0 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): get_auth_request con 0x7f50680a54c0 auth_method 0 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 -- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a54c0 msgr2=0x7f50680a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 --2- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a54c0 0x7f50680a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 -- 172.21.15.86:0/2537490233 >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 msgr2=0x7f50681382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506d77f640 1 --2- 172.21.15.86:0/2537490233 >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 0x7f50681382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 --2- 172.21.15.86:0/2537490233 >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 0x7f50681382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 monclient: found mon.noname-a 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506cf7e640 1 -- 172.21.15.86:0/2537490233 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506809c660 con 0x7f50680a50e0 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506d77f640 10 monclient: get_auth_request con 0x7f5068137ea0 auth_method 0 2024-09-17T15:28:17.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.434+0000 7f506d77f640 1 --2- 172.21.15.86:0/2537490233 >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 0x7f50681382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506cf7e640 1 -- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50680a50e0 msgr2=0x7f506809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506cf7e640 1 --2- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50680a50e0 0x7f506809cd60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f506806f8c0 tx=0x7f506001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c0148fca0e05c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 1 -- 172.21.15.86:0/2537490233 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600a56f0 con 0x7f50680a50e0 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:17.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: dump: 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 5 monclient: authenticate success, global_id 4386 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 1 -- 172.21.15.86:0/2537490233 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50600a5890 con 0x7f50680a50e0 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 4 set_mon_vals no callback set 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 1 -- 172.21.15.86:0/2537490233 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600a5c60 con 0x7f50680a50e0 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f50677fe640 10 monclient: dump: 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 10 monclient: shutdown 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 -- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50680a50e0 msgr2=0x7f506809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 --2- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50680a50e0 0x7f506809cd60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f506806f8c0 tx=0x7f506001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 -- 172.21.15.86:0/2537490233 shutdown_connections 2024-09-17T15:28:17.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 --2- 172.21.15.86:0/2537490233 >> v2:172.21.15.203:3300/0 conn(0x7f5068137ea0 0x7f50681382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 --2- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a54c0 0x7f50680a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 --2- 172.21.15.86:0/2537490233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50680a50e0 0x7f506809cd60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 -- 172.21.15.86:0/2537490233 >> 172.21.15.86:0/2537490233 conn(0x7f50680aaae0 msgr2=0x7f50680aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 -- 172.21.15.86:0/2537490233 shutdown_connections 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 1 -- 172.21.15.86:0/2537490233 wait complete. 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.438+0000 7f506f1e2640 10 monclient: monmap: 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.451 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 Processor -- start 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- start start 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: init 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 0x7f50681211f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-c con 0x7f50680a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50680a54c0 0x7f506813c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-b con 0x7f50680a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5068137ea0 0x7f5068137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): picked mon.noname-a con 0x7f5068137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f50680a6ae0 con 0x7f5068137ea0 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50680a6960 con 0x7f50680a54c0 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5068141b20 con 0x7f50680a50e0 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient(hunting): authenticate will time out at 1030.408632s 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 0x7f50681211f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5068137ea0 0x7f5068137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 0x7f50681211f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42382/0 (socket says 172.21.15.86:42382) 2024-09-17T15:28:17.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 -- 172.21.15.86:0/1634356157 learned_addr learned my addr 172.21.15.86:0/1634356157 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): get_auth_request con 0x7f50680a50e0 auth_method 0 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): get_auth_request con 0x7f5068137ea0 auth_method 0 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5067fff640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50680a54c0 0x7f506813c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5067fff640 10 monclient(hunting): get_auth_request con 0x7f50680a54c0 auth_method 0 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5067fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50680a54c0 msgr2=0x7f506813c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50680a54c0 0x7f506813c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5068137ea0 msgr2=0x7f5068137050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5068137ea0 0x7f5068137050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 monclient: found mon.noname-c 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50680a7ab0 con 0x7f50680a50e0 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506d77f640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5068137ea0 0x7f5068137050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 msgr2=0x7f50681211f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506cf7e640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 0x7f50681211f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f506001af60 tx=0x7f5060019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600bc020 con 0x7f50680a50e0 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: dump: 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.456 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50600b8880 con 0x7f50680a50e0 2024-09-17T15:28:17.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 5 monclient: authenticate success, global_id 4370 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _renew_subs 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f506813dc30 con 0x7f50680a50e0 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600a56f0 con 0x7f50680a50e0 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f50657fa640 10 monclient: dump: 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.458 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _renew_subs 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f5066ffd640 4 set_mon_vals no callback set 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.442+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5068122430 con 0x7f50680a50e0 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506f1e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506f1e2640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 0x7f50681cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5068122950 con 0x7f5068137590 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f50600bf020 con 0x7f50680a50e0 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 0x7f50681cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient: get_auth_request con 0x7f5068137590 auth_method 0 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f50657fa640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f504803bb60 0x7f504803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 4755+0+0 (secure 0 0 0) 0x7f50600eae00 con 0x7f50680a50e0 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506d77f640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f504803bb60 0x7f504803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f506d77f640 10 monclient: get_auth_request con 0x7f504803bb60 auth_method 0 2024-09-17T15:28:17.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 10 monclient(hunting): handle_auth_done global_id 14104 payload 824 2024-09-17T15:28:17.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5068122740 con 0x7f5068137590 2024-09-17T15:28:17.461 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 582974 2024-09-17T15:28:17.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 msgr2=0x7f50681cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.446+0000 7f5067fff640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 0x7f50681cfd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f5050011a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050004230 con 0x7f5068137590 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 10 monclient: dump: 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f506d77f640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f504803bb60 0x7f504803e020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f505800ae30 tx=0x7f5058008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5050004230 con 0x7f5068137590 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 msgr2=0x7f50681cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 0x7f50681cfd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f5050011a60 comp rx=0 tx=0).stop 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 msgr2=0x7f50681cfd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.450+0000 7f50657fa640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5068137590 0x7f50681cfd90 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f5050011a60 comp rx=0 tx=0).stop 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f506f1e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f506f1e2640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 0x7f50681d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f506f1e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f506806a0e0 con 0x7f50681d02d0 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 0x7f50681d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient: get_auth_request con 0x7f50681d02d0 auth_method 0 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 10 monclient(hunting): handle_auth_done global_id 14107 payload 824 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.454+0000 7f5067fff640 1 -- 172.21.15.86:0/1634356157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50681d0bf0 con 0x7f50681d02d0 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f5067fff640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 msgr2=0x7f50681d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f5067fff640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 0x7f50681d06b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f50500046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505002b020 con 0x7f50681d02d0 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 10 monclient: dump: 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.467 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f5050015460 con 0x7f50681d02d0 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 msgr2=0x7f50681d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 0x7f50681d06b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f50500046d0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 msgr2=0x7f50681d06b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f50657fa640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50681d02d0 0x7f50681d06b0 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5068121800 tx=0x7f50500046d0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:17.468 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 6059, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:17.469 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.470 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.471 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:17.472 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:17.473 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.474 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.476 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.477 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:17.480 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:17.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:17.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:17.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:17.481 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:17.481 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f504803bb60 msgr2=0x7f504803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f504803bb60 0x7f504803e020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f505800ae30 tx=0x7f5058008040 comp rx=0 tx=0).stop 2024-09-17T15:28:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 10 monclient: shutdown 2024-09-17T15:28:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 msgr2=0x7f50681211f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 --2- 172.21.15.86:0/1634356157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50680a50e0 0x7f50681211f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f506001af60 tx=0x7f5060019650 comp rx=0 tx=0).stop 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506d77f640 1 -- 172.21.15.86:0/1634356157 reap_dead start 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506d77f640 1 -- 172.21.15.86:0/1634356157 reap_dead start 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 shutdown_connections 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 >> 172.21.15.86:0/1634356157 conn(0x7f50680aaae0 msgr2=0x7f506813e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 shutdown_connections 2024-09-17T15:28:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.458+0000 7f506f1e2640 1 -- 172.21.15.86:0/1634356157 wait complete. 2024-09-17T15:28:17.483 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:17.483 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:28:17.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 10 monclient: get_monmap_and_config 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 10 monclient: monmap: 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:17.594886+0000 2024-09-17T15:28:17.593 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:17.594886+0000 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:17.594 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 1 Processor -- start 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 1 -- start start 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.590+0000 7ffbea299640 10 monclient: init 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffbe40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe40694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffbe413aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbe4074470 con 0x7ffbe40689d0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbe409c8b0 con 0x7ffbe413aff0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe409cbe0 con 0x7ffbe40694c0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbea299640 10 monclient(hunting): authenticate will time out at 1030.560566s 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42384/0 (socket says 172.21.15.86:42384) 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 -- 172.21.15.86:0/2855846583 learned_addr learned my addr 172.21.15.86:0/2855846583 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): get_auth_request con 0x7ffbe40694c0 auth_method 0 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 -- 172.21.15.86:0/2855846583 >> v1:172.21.15.203:6789/0 conn(0x7ffbe413aff0 legacy=0x7ffbe413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 -- 172.21.15.86:0/2855846583 >> v1:172.21.15.86:6789/0 conn(0x7ffbe40689d0 legacy=0x7ffbe4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 monclient: found mon.noname-b 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.594+0000 7ffbe37fe640 1 -- 172.21.15.86:0/2855846583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe409ce70 con 0x7ffbe40694c0 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe37fe640 1 -- 172.21.15.86:0/2855846583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 msgr2=0x7ffbe406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe37fe640 1 --2- 172.21.15.86:0/2855846583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40098a0 tx=0x7ffbd401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7999ac37c98e352a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 1 -- 172.21.15.86:0/2855846583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40b2070 con 0x7ffbe40694c0 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: dump: 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.600 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 5 monclient: authenticate success, global_id 4373 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:17.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 1 -- 172.21.15.86:0/2855846583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbd40ad680 con 0x7ffbe40694c0 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 4 set_mon_vals no callback set 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 1 -- 172.21.15.86:0/2855846583 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe409c2d0 con 0x7ffbe40694c0 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe2ffd640 10 monclient: dump: 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.602 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: shutdown 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- 172.21.15.86:0/2855846583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 msgr2=0x7ffbe406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 --2- 172.21.15.86:0/2855846583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40098a0 tx=0x7ffbd401c440 comp rx=0 tx=0).stop 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- 172.21.15.86:0/2855846583 shutdown_connections 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 --2- 172.21.15.86:0/2855846583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe40694c0 0x7ffbe406f840 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- 172.21.15.86:0/2855846583 >> 172.21.15.86:0/2855846583 conn(0x7ffbe40a6440 msgr2=0x7ffbe40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- 172.21.15.86:0/2855846583 shutdown_connections 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- 172.21.15.86:0/2855846583 wait complete. 2024-09-17T15:28:17.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: build_initial_monmap 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: monmap: 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:17.604 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 Processor -- start 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- start start 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: init 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 0x7ffbe411c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe40689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 0x7ffbe411ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe40694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4141ca0 con 0x7ffbe40689d0 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe4141b20 con 0x7ffbe40694c0 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe4141e20 con 0x7ffbe413aff0 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbea299640 10 monclient(hunting): authenticate will time out at 1030.567403s 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 0x7ffbe411c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42400/0 (socket says 172.21.15.86:42400) 2024-09-17T15:28:17.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 0x7ffbe411c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50096/0 (socket says 172.21.15.86:50096) 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 -- 172.21.15.86:0/928483366 learned_addr learned my addr 172.21.15.86:0/928483366 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): get_auth_request con 0x7ffbe413aff0 auth_method 0 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): get_auth_request con 0x7ffbe40689d0 auth_method 0 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe37fe640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 0x7ffbe411ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient(hunting): handle_auth_done global_id 4376 payload 824 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 -- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 msgr2=0x7ffbe411ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 0x7ffbe411ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 -- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 msgr2=0x7ffbe411c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 0x7ffbe411c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient: _finish_hunting 0 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 monclient: found mon.noname-c 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe8836640 1 -- 172.21.15.86:0/928483366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe411f0e0 con 0x7ffbe413aff0 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe37fe640 10 monclient: get_auth_request con 0x7ffbe40694c0 auth_method 0 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.598+0000 7ffbe37fe640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 0x7ffbe411ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe8836640 1 -- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 msgr2=0x7ffbe411d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe8836640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c780 tx=0x7ffbdc00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc00ac60 con 0x7ffbe413aff0 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: dump: 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbdc00ae00 con 0x7ffbe413aff0 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc004bb0 con 0x7ffbe413aff0 2024-09-17T15:28:17.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 5 monclient: authenticate success, global_id 4376 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _renew_subs 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe411f3c0 con 0x7ffbe413aff0 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe27fc640 4 set_mon_vals no callback set 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 10 monclient: dump: 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:17.614 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _renew_subs 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe411ee00 con 0x7ffbe413aff0 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe406f840 con 0x7ffbe413aff0 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ffbdc005030 con 0x7ffbe413aff0 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 0x7ffbd003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(37..37 src has 1..37) ==== 4755+0+0 (secure 0 0 0) 0x7ffbdc050170 con 0x7ffbe413aff0 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe3fff640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 0x7ffbd003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.602+0000 7ffbe3fff640 10 monclient: get_auth_request con 0x7ffbd003bbe0 auth_method 0 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.606+0000 7ffbe3fff640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 0x7ffbd003e0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8007fd0 tx=0x7ffbd8007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.606+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffbdc020070 con 0x7ffbe413aff0 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.610+0000 7ffbe0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:17.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.610+0000 7ffbe0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:17.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.834+0000 7ffbea299640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:28:17.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.834+0000 7ffbea299640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:28:17.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.834+0000 7ffbea299640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:17.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.834+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ffbe406d4f0 con 0x7ffbe413aff0 2024-09-17T15:28:17.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbe0ff9640 1 -- 172.21.15.86:0/928483366 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ffbdc004e10 con 0x7ffbe413aff0 2024-09-17T15:28:17.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbe0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:28:17.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbe0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 msgr2=0x7ffbd003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 0x7ffbd003e0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8007fd0 tx=0x7ffbd8007e40 comp rx=0 tx=0).stop 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 10 monclient: shutdown 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 msgr2=0x7ffbe411d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c780 tx=0x7ffbdc00cc40 comp rx=0 tx=0).stop 2024-09-17T15:28:17.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 shutdown_connections 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffbd003bbe0 0x7ffbd003e0a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbe413aff0 0x7ffbe411d340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40694c0 0x7ffbe411ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 --2- 172.21.15.86:0/928483366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbe40689d0 0x7ffbe411c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 >> 172.21.15.86:0/928483366 conn(0x7ffbe40a6440 msgr2=0x7ffbe413ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 shutdown_connections 2024-09-17T15:28:17.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:17.838+0000 7ffbea299640 1 -- 172.21.15.86:0/928483366 wait complete. 2024-09-17T15:28:17.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=755326562,config=3541143443,health=861809967,kv=3591778543,logm=1997889477}) 2024-09-17T15:28:17.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=1981080768,config=3541143443,health=861809967,kv=2794928421,logm=3118833233}) 2024-09-17T15:28:17.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=755326562,config=3541143443,health=861809967,kv=3591778543,logm=1997889477}) 2024-09-17T15:28:17.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.842+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=11,kv=11,logm=68} crc {auth=1793889709,config=3541143443,health=1648279203,kv=3591778543,logm=2465178598}) 2024-09-17T15:28:17.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230765801}) 2024-09-17T15:28:17.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055637149}) 2024-09-17T15:28:17.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230765801}) 2024-09-17T15:28:17.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790536481}) 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198875759}) 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.846+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224754288}) 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198875759}) 2024-09-17T15:28:17.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944526610}) 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797857903}) 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3243295126,mdsmap=3164546943}) 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797857903}) 2024-09-17T15:28:17.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.850+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=1019475358,mdsmap=2815901553}) 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=5,osdmap=49} crc {mdsmap=2815901553,mgr=4050415671,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2936955070,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3311363537,osd_pg_creating=4154890195,osd_snap=1813110910,osdmap=2068718978}) 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6,osdmap=43} crc {mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=937865332,mon_config_key=4289266023,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4154890196,osd_snap=855962214,osdmap=4163505738}) 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=5,osdmap=49} crc {mdsmap=2815901553,mgr=4050415671,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2936955070,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3311363537,osd_pg_creating=4154890195,osd_snap=1813110910,osdmap=2068718978}) 2024-09-17T15:28:17.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.854+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6,osdmap=46} crc {mdsmap=2815901553,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2109985499,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3616902286,osd_pg_creating=4154890195,osd_snap=855962214,osdmap=3230029739}) 2024-09-17T15:28:17.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=1418600816}) 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=26} crc {osdmap=449931400}) 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=1418600816}) 2024-09-17T15:28:17.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:28:17.858+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=27} crc {osdmap=2897227029}) 2024-09-17T15:28:18.384 INFO:tasks.rados.rados.0.smithi203.stdout:140: done (1 left) 2024-09-17T15:28:18.384 INFO:tasks.rados.rados.0.smithi203.stdout:141: done (0 left) 2024-09-17T15:28:18.384 INFO:tasks.rados.rados.0.smithi203.stdout:142: rollback oid 35 current snap is 10 2024-09-17T15:28:18.385 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 9 2024-09-17T15:28:18.385 INFO:tasks.rados.rados.0.smithi203.stdout:143: write oid 43 current snap is 10 2024-09-17T15:28:18.385 INFO:tasks.rados.rados.0.smithi203.stdout:143: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-17T15:28:18.406 INFO:tasks.rados.rados.0.smithi203.stdout:143: writing smithi20328086-43 from 679016 to 1380811 tid 1 2024-09-17T15:28:18.423 INFO:tasks.rados.rados.0.smithi203.stdout:143: writing smithi20328086-43 from 1866533 to 2362557 tid 2 2024-09-17T15:28:18.437 INFO:tasks.rados.rados.0.smithi203.stdout:143: writing smithi20328086-43 from 2900635 to 3303278 tid 3 2024-09-17T15:28:18.438 INFO:tasks.rados.rados.0.smithi203.stdout:144: snap_remove snap 7 2024-09-17T15:28:18.857 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:28:18.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: get_monmap_and_config 2024-09-17T15:28:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: build_initial_monmap 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: monmap: 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:18.981697+0000 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:18.981697+0000 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:18.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 Processor -- start 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 -- start start 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: init 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70b80689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70b80694c0 0x7f70b806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-c con 0x7f70b80694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 0x7f70b813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-a con 0x7f70b813aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b809c8b0 con 0x7f70b80689d0 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f70b8074500 con 0x7f70b813aff0 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f70b809cb90 con 0x7f70b80694c0 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70c0541640 10 monclient(hunting): authenticate will time out at 1031.947196s 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 0x7f70b813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 0x7f70b813d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56094/0 (socket says 172.21.15.86:56094) 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 1 -- 172.21.15.86:0/4074328271 learned_addr learned my addr 172.21.15.86:0/4074328271 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): get_auth_request con 0x7f70b813aff0 auth_method 0 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 1 --2- 172.21.15.86:0/4074328271 >> v2:172.21.15.203:3300/0 conn(0x7f70b80694c0 0x7f70b806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): get_auth_request con 0x7f70b80694c0 auth_method 0 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.978+0000 7f70bdadc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 10 monclient(hunting): handle_auth_done global_id 14110 payload 274 2024-09-17T15:28:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 1 -- 172.21.15.86:0/4074328271 >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 msgr2=0x7f70b813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 1 --2- 172.21.15.86:0/4074328271 >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 0x7f70b813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 1 -- 172.21.15.86:0/4074328271 >> v1:172.21.15.86:6789/0 conn(0x7f70b80689d0 legacy=0x7f70b8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 10 monclient: _finish_hunting 0 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 1 monclient: found mon.noname-c 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.982+0000 7f70bdadc640 1 -- 172.21.15.86:0/4074328271 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b809ce70 con 0x7f70b80694c0 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bdadc640 1 -- 172.21.15.86:0/4074328271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 msgr2=0x7f70b806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bdadc640 1 --2- 172.21.15.86:0/4074328271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b806f840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f70b8069940 tx=0x7f70a801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1f4dfd2df54c1c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 1 -- 172.21.15.86:0/4074328271 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a80b0070 con 0x7f70b80694c0 2024-09-17T15:28:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: dump: 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: _finish_auth 0 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: _check_auth_tickets 2024-09-17T15:28:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 1 -- 172.21.15.86:0/4074328271 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a80ab600 con 0x7f70b80694c0 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 1 -- 172.21.15.86:0/4074328271 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b809c2d0 con 0x7f70b80694c0 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70bd2db640 10 monclient: dump: 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:18.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:18.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:18.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 5 monclient: authenticate success, global_id 14110 2024-09-17T15:28:18.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:18.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 4 set_mon_vals no callback set 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 10 monclient: shutdown 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- 172.21.15.86:0/4074328271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 msgr2=0x7f70b806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 --2- 172.21.15.86:0/4074328271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b806f840 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f70b8069940 tx=0x7f70a801c300 comp rx=0 tx=0).stop 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- 172.21.15.86:0/4074328271 shutdown_connections 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 --2- 172.21.15.86:0/4074328271 >> v2:172.21.15.86:3300/0 conn(0x7f70b813aff0 0x7f70b813d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 --2- 172.21.15.86:0/4074328271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b806f840 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- 172.21.15.86:0/4074328271 >> 172.21.15.86:0/4074328271 conn(0x7f70b80a6440 msgr2=0x7f70b80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- 172.21.15.86:0/4074328271 shutdown_connections 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- 172.21.15.86:0/4074328271 wait complete. 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 10 monclient: build_initial_monmap 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 10 monclient: monmap: 2024-09-17T15:28:18.995 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 Processor -- start 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 1 -- start start 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.986+0000 7f70c0541640 10 monclient: init 2024-09-17T15:28:18.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-a con 0x7f70b80689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-b con 0x7f70b80694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 0x7f70b809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): picked mon.noname-c con 0x7f70b813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f70b81405c0 con 0x7f70b80689d0 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f70b8140440 con 0x7f70b80694c0 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f70b8140740 con 0x7f70b813aff0 2024-09-17T15:28:18.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient(hunting): authenticate will time out at 1031.956456s 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 0x7f70b809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56100/0 (socket says 172.21.15.86:56100) 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 -- 172.21.15.86:0/1343528294 learned_addr learned my addr 172.21.15.86:0/1343528294 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): get_auth_request con 0x7f70b80689d0 auth_method 0 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:18.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): get_auth_request con 0x7f70b813aff0 auth_method 0 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 10 monclient(hunting): get_auth_request con 0x7f70b80694c0 auth_method 0 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 -- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 msgr2=0x7f70b809dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 0x7f70b809dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 -- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 msgr2=0x7f70b809d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b809d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 monclient: found mon.noname-a 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 -- 172.21.15.86:0/1343528294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b80728b0 con 0x7f70b80689d0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70bdadc640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b809d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70beade640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 0x7f70b809dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 -- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 msgr2=0x7f70b80732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70be2dd640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c450 tx=0x7f70b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4012a40 con 0x7f70b80689d0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:19.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: dump: 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 5 monclient: authenticate success, global_id 4398 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b4017070 con 0x7f70b80689d0 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _renew_subs 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b813b3d0 con 0x7f70b80689d0 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4012e40 con 0x7f70b80689d0 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70aeffd640 10 monclient: dump: 2024-09-17T15:28:19.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70affff640 4 set_mon_vals no callback set 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _renew_subs 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.990+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b80725d0 con 0x7f70b80689d0 2024-09-17T15:28:19.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f70b401c460 con 0x7f70b80689d0 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70c0541640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70c0541640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70c0541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b806f840 con 0x7f70b80689d0 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70aeffd640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 0x7f709c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4711+0+0 (secure 0 0 0) 0x7f70b404fc80 con 0x7f70b80689d0 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70bdadc640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 0x7f709c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70bdadc640 10 monclient: get_auth_request con 0x7f709c03bb60 auth_method 0 2024-09-17T15:28:19.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.994+0000 7f70bdadc640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 0x7f709c03e020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a80ab000 tx=0x7f70a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.998+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f70b401ac60 con 0x7f70b80689d0 2024-09-17T15:28:19.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.998+0000 7f70aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:18.998+0000 7f70aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:19.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f70b806d4f0 con 0x7f70b80689d0 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70aeffd640 1 -- 172.21.15.86:0/1343528294 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f70b404e6c0 con 0x7f70b80689d0 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:19.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:19.229 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:19.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 msgr2=0x7f709c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 0x7f709c03e020 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a80ab000 tx=0x7f70a8008040 comp rx=0 tx=0).stop 2024-09-17T15:28:19.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 10 monclient: shutdown 2024-09-17T15:28:19.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 msgr2=0x7f70b80732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f70b400c450 tx=0x7f70b400c910 comp rx=0 tx=0).stop 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 shutdown_connections 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709c03bb60 0x7f709c03e020 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b813aff0 0x7f70b809dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70b80694c0 0x7f70b809d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 --2- 172.21.15.86:0/1343528294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b80689d0 0x7f70b80732b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 >> 172.21.15.86:0/1343528294 conn(0x7f70b80a6440 msgr2=0x7f70b813b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 shutdown_connections 2024-09-17T15:28:19.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.226+0000 7f70c0541640 1 -- 172.21.15.86:0/1343528294 wait complete. 2024-09-17T15:28:19.244 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:28:19.244 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:28:19.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:28:19.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:28:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: get_monmap_and_config 2024-09-17T15:28:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: monmap: 2024-09-17T15:28:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:19.356575+0000 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:19.356575+0000 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:19.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 Processor -- start 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 -- start start 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: init 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff40c134b10 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff40c134650 0x7ff40c13ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c134650 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff40c13d410 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff40c0a1010 con 0x7ff40c134b10 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff40c06ffd0 con 0x7ff40c13d410 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c0a0010 con 0x7ff40c134650 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff413783640 10 monclient(hunting): authenticate will time out at 1032.322397s 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff41151f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff40c134b10 0x7ff40c134ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42956/0 (socket says 172.21.15.86:42956) 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff41151f640 1 -- 172.21.15.86:0/2988499363 learned_addr learned my addr 172.21.15.86:0/2988499363 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 1 --2- 172.21.15.86:0/2988499363 >> v2:172.21.15.203:3300/0 conn(0x7ff40c134650 0x7ff40c13ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): get_auth_request con 0x7ff40c134650 auth_method 0 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.354+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff41151f640 1 -- 172.21.15.86:0/2988499363 >> v1:172.21.15.86:6789/0 conn(0x7ff40c134b10 legacy=0x7ff40c134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2982723727 0 0) 0x7ff40c0a1010 con 0x7ff40c134b10 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 10 monclient(hunting): _negotiate 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 10 monclient(hunting): my global_id is 4401 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3f8003530 con 0x7ff40c134b10 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff411d20640 1 -- 172.21.15.86:0/2988499363 >> v1:172.21.15.203:6789/0 conn(0x7ff40c13d410 legacy=0x7ff40c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 10 monclient(hunting): handle_auth_done global_id 14116 payload 274 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 1 -- 172.21.15.86:0/2988499363 >> v1:172.21.15.203:6789/0 conn(0x7ff40c13d410 legacy=0x7ff40c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 1 -- 172.21.15.86:0/2988499363 >> v1:172.21.15.86:6789/0 conn(0x7ff40c134b10 legacy=0x7ff40c134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 989337493 0 0) 0x7ff3f8003530 con 0x7ff40c134b10 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 1 monclient: found mon.noname-c 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff410d1e640 1 -- 172.21.15.86:0/2988499363 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c0702b0 con 0x7ff40c134650 2024-09-17T15:28:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.358+0000 7ff403fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-17T15:28:19.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff410d1e640 1 -- 172.21.15.86:0/2988499363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c134650 msgr2=0x7ff40c13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff410d1e640 1 --2- 172.21.15.86:0/2988499363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c134650 0x7ff40c13ced0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0098c0 tx=0x7ff3fc01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8b65722734f4641 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0b0070 con 0x7ff40c134650 2024-09-17T15:28:19.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:19.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: dump: 2024-09-17T15:28:19.364 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3fc0ab600 con 0x7ff40c134650 2024-09-17T15:28:19.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 1 -- 172.21.15.86:0/2988499363 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0ab9d0 con 0x7ff40c134650 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff403fff640 10 monclient: dump: 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 5 monclient: authenticate success, global_id 14116 2024-09-17T15:28:19.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:19.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 4 set_mon_vals no callback set 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 10 monclient: shutdown 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 -- 172.21.15.86:0/2988499363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c134650 msgr2=0x7ff40c13ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 --2- 172.21.15.86:0/2988499363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c134650 0x7ff40c13ced0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0098c0 tx=0x7ff3fc01c460 comp rx=0 tx=0).stop 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 -- 172.21.15.86:0/2988499363 shutdown_connections 2024-09-17T15:28:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 --2- 172.21.15.86:0/2988499363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c134650 0x7ff40c13ced0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 -- 172.21.15.86:0/2988499363 >> 172.21.15.86:0/2988499363 conn(0x7ff40c0a7210 msgr2=0x7ff40c0a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 -- 172.21.15.86:0/2988499363 shutdown_connections 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 1 -- 172.21.15.86:0/2988499363 wait complete. 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.362+0000 7ff413783640 10 monclient: monmap: 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 Processor -- start 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- start start 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: init 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 0x7ff40c137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c134650 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c134b10 0x7ff40c137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): picked mon.noname-a con 0x7ff40c134b10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c13d410 0x7ff40c1384f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c13d410 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c143ad0 con 0x7ff40c134b10 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c143950 con 0x7ff40c13d410 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c143c50 con 0x7ff40c134650 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient(hunting): authenticate will time out at 1032.332795s 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 0x7ff40c137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 0x7ff40c137a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43822/0 (socket says 172.21.15.86:43822) 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 -- 172.21.15.86:0/79124625 learned_addr learned my addr 172.21.15.86:0/79124625 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): get_auth_request con 0x7ff40c134650 auth_method 0 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c134b10 0x7ff40c137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): get_auth_request con 0x7ff40c134b10 auth_method 0 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff411d20640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c13d410 0x7ff40c1384f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff411d20640 10 monclient(hunting): get_auth_request con 0x7ff40c13d410 auth_method 0 2024-09-17T15:28:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff411d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff411d20640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff411d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c13d410 msgr2=0x7ff40c1384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c13d410 0x7ff40c1384f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c134b10 msgr2=0x7ff40c137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c134b10 0x7ff40c137fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 monclient: found mon.noname-c 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c13a570 con 0x7ff40c134650 2024-09-17T15:28:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 msgr2=0x7ff40c137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff41151f640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 0x7ff40c137a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff40800c530 tx=0x7ff40800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408017070 con 0x7ff40c134650 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: dump: 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff408007070 con 0x7ff40c134650 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 5 monclient: authenticate success, global_id 4382 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _renew_subs 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c073b30 con 0x7ff40c134650 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff408007460 con 0x7ff40c134650 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff401ffb640 10 monclient: dump: 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff402ffd640 4 set_mon_vals no callback set 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _renew_subs 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.366+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c13c4a0 con 0x7ff40c134650 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff413783640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff413783640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 0x7ff40c131880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c06eee0 con 0x7ff40c1314a0 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 0x7ff40c131880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient: get_auth_request con 0x7ff40c1314a0 auth_method 0 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff40801a070 con 0x7ff40c134650 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff40c134ef0 con 0x7ff40c1314a0 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f003bb60 0x7ff3f003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(38..38 src has 1..38) ==== 4711+0+0 (secure 0 0 0) 0x7ff40804f280 con 0x7ff40c134650 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff411d20640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f003bb60 0x7ff3f003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff411d20640 10 monclient: get_auth_request con 0x7ff3f003bb60 auth_method 0 2024-09-17T15:28:19.382 INFO:tasks.rados.rados.0.smithi203.stdout:144: done (2 left) 2024-09-17T15:28:19.383 INFO:tasks.rados.rados.0.smithi203.stdout:145: snap_remove snap 6 2024-09-17T15:28:19.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 msgr2=0x7ff40c131880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff410d1e640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 0x7ff40c131880 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0a0dd0 tx=0x7ff3fc01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0bb030 con 0x7ff40c1314a0 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 10 monclient: dump: 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff3fc0abdb0 con 0x7ff40c1314a0 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 msgr2=0x7ff40c131880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 0x7ff40c131880 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0a0dd0 tx=0x7ff3fc01aca0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 msgr2=0x7ff40c131880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff401ffb640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c1314a0 0x7ff40c131880 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0a0dd0 tx=0x7ff3fc01aca0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.370+0000 7ff411d20640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f003bb60 0x7ff3f003e020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff40c138ff0 tx=0x7ff404009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff413783640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff413783640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 0x7ff40c13d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff413783640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c1407e0 con 0x7ff40c131dc0 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 0x7ff40c13d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient: get_auth_request con 0x7ff40c131dc0 auth_method 0 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.378+0000 7ff410d1e640 1 -- 172.21.15.86:0/79124625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff40c06ddc0 con 0x7ff40c131dc0 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff410d1e640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 msgr2=0x7ff40c13d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff410d1e640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 0x7ff40c13d7f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc019070 tx=0x7ff3fc019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc0bb050 con 0x7ff40c131dc0 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 10 monclient: dump: 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7ff3fc0b7460 con 0x7ff40c131dc0 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 msgr2=0x7ff40c13d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 0x7ff40c13d7f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc019070 tx=0x7ff3fc019200 comp rx=0 tx=0).stop 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 msgr2=0x7ff40c13d7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff401ffb640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c131dc0 0x7ff40c13d7f0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc019070 tx=0x7ff3fc019200 comp rx=0 tx=0).stop 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72026, 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.397 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:19.398 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:19.400 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:19.401 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.402 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:19.403 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:19.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f003bb60 msgr2=0x7ff3f003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f003bb60 0x7ff3f003e020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff40c138ff0 tx=0x7ff404009040 comp rx=0 tx=0).stop 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 10 monclient: shutdown 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 msgr2=0x7ff40c137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 --2- 172.21.15.86:0/79124625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c134650 0x7ff40c137a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff40800c530 tx=0x7ff40800c9f0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff411d20640 1 -- 172.21.15.86:0/79124625 reap_dead start 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff411d20640 1 -- 172.21.15.86:0/79124625 reap_dead start 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 shutdown_connections 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 >> 172.21.15.86:0/79124625 conn(0x7ff40c0a7210 msgr2=0x7ff40c0a4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 shutdown_connections 2024-09-17T15:28:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.382+0000 7ff413783640 1 -- 172.21.15.86:0/79124625 wait complete. 2024-09-17T15:28:19.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:19.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:28:19.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: get_monmap_and_config 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: monmap: 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:19.520115+0000 2024-09-17T15:28:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:19.520115+0000 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:19.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 Processor -- start 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 -- start start 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: init 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec540a50e0 0x7fec5409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-a con 0x7fec540a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec540a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec5413d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec540a6280 con 0x7fec5413d2f0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5409c0a0 con 0x7fec540a54c0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fec5409d2a0 con 0x7fec540a50e0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5b180640 10 monclient(hunting): authenticate will time out at 1032.485601s 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec540a50e0 0x7fec5409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec540a50e0 0x7fec5409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56128/0 (socket says 172.21.15.86:56128) 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 1 -- 172.21.15.86:0/2685338362 learned_addr learned my addr 172.21.15.86:0/2685338362 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): get_auth_request con 0x7fec540a50e0 auth_method 0 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5971d640 1 --1- 172.21.15.86:0/2685338362 >> v1:172.21.15.86:6789/0 conn(0x7fec5413d2f0 0x7fec5413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42970/0 (socket says 172.21.15.86:42970) 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec5971d640 1 -- 172.21.15.86:0/2685338362 >> v1:172.21.15.86:6789/0 conn(0x7fec5413d2f0 legacy=0x7fec5413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.518+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 10 monclient(hunting): handle_auth_done global_id 4413 payload 274 2024-09-17T15:28:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 1 -- 172.21.15.86:0/2685338362 >> v1:172.21.15.203:6789/0 conn(0x7fec540a54c0 legacy=0x7fec540a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 1 -- 172.21.15.86:0/2685338362 >> v1:172.21.15.86:6789/0 conn(0x7fec5413d2f0 legacy=0x7fec5413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 1 monclient: found mon.noname-a 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.522+0000 7fec58f1c640 1 -- 172.21.15.86:0/2685338362 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5409c3d0 con 0x7fec540a50e0 2024-09-17T15:28:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 1 -- 172.21.15.86:0/2685338362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a50e0 msgr2=0x7fec5409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 1 --2- 172.21.15.86:0/2685338362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a50e0 0x7fec5409cd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fec44009ed0 tx=0x7fec4401c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78cbb30127b7bef server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 1 -- 172.21.15.86:0/2685338362 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440a4ad0 con 0x7fec540a50e0 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: dump: 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 5 monclient: authenticate success, global_id 4413 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 1 -- 172.21.15.86:0/2685338362 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec440a4c70 con 0x7fec540a50e0 2024-09-17T15:28:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 1 -- 172.21.15.86:0/2685338362 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440ac610 con 0x7fec540a50e0 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec537fe640 10 monclient: dump: 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 4 set_mon_vals no callback set 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: shutdown 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- 172.21.15.86:0/2685338362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a50e0 msgr2=0x7fec5409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 --2- 172.21.15.86:0/2685338362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a50e0 0x7fec5409cd60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fec44009ed0 tx=0x7fec4401c570 comp rx=0 tx=0).stop 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- 172.21.15.86:0/2685338362 shutdown_connections 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 --2- 172.21.15.86:0/2685338362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a50e0 0x7fec5409cd60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- 172.21.15.86:0/2685338362 >> 172.21.15.86:0/2685338362 conn(0x7fec540aaae0 msgr2=0x7fec540aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- 172.21.15.86:0/2685338362 shutdown_connections 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- 172.21.15.86:0/2685338362 wait complete. 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: monmap: 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 Processor -- start 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- start start 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: init 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 0x7fec54137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-c con 0x7fec540a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 0x7fec541392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-a con 0x7fec540a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec5413d2f0 0x7fec54139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): picked mon.noname-b con 0x7fec5413d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fec541441d0 con 0x7fec540a54c0 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec54144050 con 0x7fec5413d2f0 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fec54144350 con 0x7fec540a50e0 2024-09-17T15:28:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5b180640 10 monclient(hunting): authenticate will time out at 1032.494888s 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 0x7fec54137ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 0x7fec541392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 0x7fec541392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56134/0 (socket says 172.21.15.86:56134) 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 0x7fec54137ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43832/0 (socket says 172.21.15.86:43832) 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 -- 172.21.15.86:0/4149916622 learned_addr learned my addr 172.21.15.86:0/4149916622 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): get_auth_request con 0x7fec540a54c0 auth_method 0 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): get_auth_request con 0x7fec540a50e0 auth_method 0 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5971d640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec5413d2f0 0x7fec54139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5971d640 10 monclient(hunting): get_auth_request con 0x7fec5413d2f0 auth_method 0 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5971d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5971d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec5971d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 msgr2=0x7fec54137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 0x7fec54137ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec5413d2f0 msgr2=0x7fec54139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec5413d2f0 0x7fec54139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 monclient: found mon.noname-a 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec53fff640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec54120d30 con 0x7fec540a54c0 2024-09-17T15:28:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.526+0000 7fec58f1c640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec540a50e0 0x7fec54137ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec53fff640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 msgr2=0x7fec541392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec53fff640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 0x7fec541392d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c450 tx=0x7fec4c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c0119a0 con 0x7fec540a54c0 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: dump: 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec4c017070 con 0x7fec540a54c0 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 5 monclient: authenticate success, global_id 4419 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 monclient: _renew_subs 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec541219b0 con 0x7fec540a54c0 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec4c011da0 con 0x7fec540a54c0 2024-09-17T15:28:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 10 monclient: dump: 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec527fc640 4 set_mon_vals no callback set 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 monclient: _renew_subs 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec541216d0 con 0x7fec540a54c0 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 0x7fec54139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fec5409cd60 con 0x7fec54138420 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 0x7fec54139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient: get_auth_request con 0x7fec54138420 auth_method 0 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fec4c01c460 con 0x7fec540a54c0 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec58f1c640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec540008e0 con 0x7fec54138420 2024-09-17T15:28:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec3403bbb0 0x7fec3403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4767+0+0 (secure 0 0 0) 0x7fec4c04fd20 con 0x7fec540a54c0 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5971d640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec3403bbb0 0x7fec3403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.530+0000 7fec5971d640 10 monclient: get_auth_request con 0x7fec3403bbb0 auth_method 0 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec58f1c640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 msgr2=0x7fec54139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec58f1c640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 0x7fec54139d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad000 tx=0x7fec44019630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440b5800 con 0x7fec54138420 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 10 monclient: dump: 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec440b5800 con 0x7fec54138420 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 msgr2=0x7fec54139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 0x7fec54139d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad000 tx=0x7fec44019630 comp rx=0 tx=0).stop 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 msgr2=0x7fec54139d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec517fa640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec54138420 0x7fec54139d50 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad000 tx=0x7fec44019630 comp rx=0 tx=0).stop 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.534+0000 7fec5971d640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec3403bbb0 0x7fec3403e070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec5413b5a0 tx=0x7fec3c0019d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec5b180640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec5b180640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 0x7fec5413d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec5b180640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fec5413a670 con 0x7fec5413a290 2024-09-17T15:28:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 0x7fec5413d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient: get_auth_request con 0x7fec5413a290 auth_method 0 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.538+0000 7fec58f1c640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-09-17T15:28:19.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec58f1c640 1 -- 172.21.15.86:0/4149916622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec5413b7c0 con 0x7fec5413a290 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec58f1c640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 msgr2=0x7fec5413d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec58f1c640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 0x7fec5413d6d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec44009ed0 tx=0x7fec440a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec440039a0 con 0x7fec5413a290 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 10 monclient: dump: 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.556 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fec440c4570 con 0x7fec5413a290 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 msgr2=0x7fec5413d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 0x7fec5413d6d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec44009ed0 tx=0x7fec440a6000 comp rx=0 tx=0).stop 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 msgr2=0x7fec5413d6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec517fa640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5413a290 0x7fec5413d6d0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fec44009ed0 tx=0x7fec440a6000 comp rx=0 tx=0).stop 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72186, 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:19.570 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.571 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.572 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:19.573 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec3403bbb0 msgr2=0x7fec3403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec3403bbb0 0x7fec3403e070 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fec5413b5a0 tx=0x7fec3c0019d0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 10 monclient: shutdown 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 msgr2=0x7fec541392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 --2- 172.21.15.86:0/4149916622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec540a54c0 0x7fec541392d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c450 tx=0x7fec4c00c910 comp rx=0 tx=0).stop 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5971d640 1 -- 172.21.15.86:0/4149916622 reap_dead start 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5971d640 1 -- 172.21.15.86:0/4149916622 reap_dead start 2024-09-17T15:28:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 shutdown_connections 2024-09-17T15:28:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 >> 172.21.15.86:0/4149916622 conn(0x7fec540aaae0 msgr2=0x7fec540a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 shutdown_connections 2024-09-17T15:28:19.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.542+0000 7fec5b180640 1 -- 172.21.15.86:0/4149916622 wait complete. 2024-09-17T15:28:19.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:19.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:28:19.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: get_monmap_and_config 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: monmap: 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:19.692952+0000 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:19.692952+0000 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:19.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 Processor -- start 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 -- start start 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: init 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f734c135f80 0x7f734c072310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-c con 0x7f734c135f80 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f734c073cf0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f734c13ae70 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f734c136a70 con 0x7f734c073cf0 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f734c074100 con 0x7f734c13ae70 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f734c0a4c00 con 0x7f734c135f80 2024-09-17T15:28:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f73518d4640 10 monclient(hunting): authenticate will time out at 1032.658448s 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734a7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f734c073cf0 0x7f734c136360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42972/0 (socket says 172.21.15.86:42972) 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734a7fc640 1 -- 172.21.15.86:0/1109032154 learned_addr learned my addr 172.21.15.86:0/1109032154 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 1 --2- 172.21.15.86:0/1109032154 >> v2:172.21.15.203:3300/0 conn(0x7f734c135f80 0x7f734c072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): get_auth_request con 0x7f734c135f80 auth_method 0 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.690+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 1 -- 172.21.15.86:0/1109032154 >> v1:172.21.15.86:6789/0 conn(0x7f734c073cf0 legacy=0x7f734c136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2372070428 0 0) 0x7f734c136a70 con 0x7f734c073cf0 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): my global_id is 4422 2024-09-17T15:28:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7334003530 con 0x7f734c073cf0 2024-09-17T15:28:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 >> v1:172.21.15.203:6789/0 conn(0x7f734c13ae70 legacy=0x7f734c13d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2530741717 0 0) 0x7f7334003530 con 0x7f734c073cf0 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 >> v2:172.21.15.203:3300/0 conn(0x7f734c135f80 msgr2=0x7f734c072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 --2- 172.21.15.86:0/1109032154 >> v2:172.21.15.203:3300/0 conn(0x7f734c135f80 0x7f734c072310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 >> v1:172.21.15.203:6789/0 conn(0x7f734c13ae70 legacy=0x7f734c13d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c0a4950 con 0x7f734c073cf0 2024-09-17T15:28:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f73400019b0 con 0x7f734c073cf0 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:28:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: dump: 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 >> v1:172.21.15.86:6789/0 conn(0x7f734c073cf0 legacy=0x7f734c136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7334003b20 0x7f7334003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): picked mon.b con 0x7f7334003b20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7334004810 0x7f7334004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): picked mon.a con 0x7f7334004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 0x7f733400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): picked mon.c con 0x7f73340053d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f733400c2e0 con 0x7f7334004810 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f733400c490 con 0x7f7334003b20 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f733400c690 con 0x7f73340053d0 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f7349ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 0x7f733400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): get_auth_request con 0x7f73340053d0 auth_method 0 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7334004810 0x7f7334004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 10 monclient(hunting): get_auth_request con 0x7f7334004810 auth_method 0 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734affd640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7334003b20 0x7f7334003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient(hunting): handle_auth_done global_id 4422 payload 293 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7334003b20 msgr2=0x7f7334003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7334003b20 0x7f7334003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7334004810 msgr2=0x7f7334004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7334004810 0x7f7334004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 monclient: found mon.c 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7334003530 con 0x7f73340053d0 2024-09-17T15:28:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734affd640 10 monclient: get_auth_request con 0x7f7334003b20 auth_method 0 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734affd640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7334003b20 0x7f7334003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f73518d4640 5 monclient: authenticate success, global_id 4422 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f73518d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734a7fc640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7334004810 0x7f7334004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:28:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.694+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 reap_dead start 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f734b7fe640 1 -- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 msgr2=0x7f733400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f734b7fe640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 0x7f733400ba30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7344002090 tx=0x7f734401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c6041f9f597659e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344001dc0 con 0x7f73340053d0 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: dump: 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73440a9070 con 0x7f73340053d0 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 1 -- 172.21.15.86:0/1109032154 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73440a5640 con 0x7f73340053d0 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 4 set_mon_vals no callback set 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f7349ffb640 10 monclient: dump: 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 10 monclient: shutdown 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 msgr2=0x7f733400ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 0x7f733400ba30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7344002090 tx=0x7f734401c8d0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- 172.21.15.86:0/1109032154 shutdown_connections 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 --2- 172.21.15.86:0/1109032154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73340053d0 0x7f733400ba30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- 172.21.15.86:0/1109032154 >> 172.21.15.86:0/1109032154 conn(0x7f734c09bc70 msgr2=0x7f734c0a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- 172.21.15.86:0/1109032154 shutdown_connections 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- 172.21.15.86:0/1109032154 wait complete. 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 10 monclient: monmap: 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:19.724 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 Processor -- start 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 1 -- start start 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.698+0000 7f73518d4640 10 monclient: init 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73340053d0 0x7f734c139630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-b con 0x7f73340053d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 0x7f734c1374d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-c con 0x7f734c137050 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f734c137a10 0x7f734c1d4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): picked mon.noname-a con 0x7f734c137a10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f734c0726a0 con 0x7f734c137a10 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f734c072520 con 0x7f73340053d0 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f734c136360 con 0x7f734c137050 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient(hunting): authenticate will time out at 1032.668334s 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 0x7f734c1374d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f734c137a10 0x7f734c1d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 0x7f734c1374d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43868/0 (socket says 172.21.15.86:43868) 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 -- 172.21.15.86:0/579380560 learned_addr learned my addr 172.21.15.86:0/579380560 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): get_auth_request con 0x7f734c137050 auth_method 0 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): get_auth_request con 0x7f734c137a10 auth_method 0 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734affd640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73340053d0 0x7f734c139630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734affd640 10 monclient(hunting): get_auth_request con 0x7f73340053d0 auth_method 0 2024-09-17T15:28:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73340053d0 msgr2=0x7f734c139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73340053d0 0x7f734c139630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f734c137a10 msgr2=0x7f734c1d4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f734c137a10 0x7f734c1d4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 monclient: found mon.noname-c 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c09f660 con 0x7f734c137050 2024-09-17T15:28:19.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734b7fe640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f734c137a10 0x7f734c1d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 msgr2=0x7f734c1374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f734a7fc640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 0x7f734c1374d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7340005fc0 tx=0x7f73400074e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734000e950 con 0x7f734c137050 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: dump: 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.730 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7340018070 con 0x7f734c137050 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734000ecf0 con 0x7f734c137050 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 5 monclient: authenticate success, global_id 4394 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _renew_subs 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c139fc0 con 0x7f734c137050 2024-09-17T15:28:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f7333fff640 10 monclient: dump: 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73497fa640 4 set_mon_vals no callback set 2024-09-17T15:28:19.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _renew_subs 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.702+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c13a870 con 0x7f734c137050 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f73518d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f73518d4640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 0x7f734c13b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f734c074cb0 con 0x7f734c137e20 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f734001d460 con 0x7f734c137050 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f7333fff640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f733803bbb0 0x7f733803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(39..39 src has 1..39) ==== 4767+0+0 (secure 0 0 0) 0x7f734004fd80 con 0x7f734c137050 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 0x7f734c13b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734b7fe640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f733803bbb0 0x7f733803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient: get_auth_request con 0x7f734c137e20 auth_method 0 2024-09-17T15:28:19.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734b7fe640 10 monclient: get_auth_request con 0x7f733803bbb0 auth_method 0 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 10 monclient(hunting): handle_auth_done global_id 14131 payload 824 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f734c1366f0 con 0x7f734c137e20 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734b7fe640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f733803bbb0 0x7f733803e070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f734c0a6270 tx=0x7f73440018b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 msgr2=0x7f734c13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.706+0000 7f734affd640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 0x7f734c13b250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c012c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c01b020 con 0x7f734c137e20 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 10 monclient: dump: 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f733c010840 con 0x7f734c137e20 2024-09-17T15:28:19.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 msgr2=0x7f734c13b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 0x7f734c13b250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c012c10 comp rx=0 tx=0).stop 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 msgr2=0x7f734c13b250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.710+0000 7f7333fff640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c137e20 0x7f734c13b250 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c012c10 comp rx=0 tx=0).stop 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.714+0000 7f73518d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.714+0000 7f73518d4640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 0x7f734c13bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.714+0000 7f73518d4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.714+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f734c06a0e0 con 0x7f734c13b790 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 0x7f734c13bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient: get_auth_request con 0x7f734c13b790 auth_method 0 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 10 monclient(hunting): handle_auth_done global_id 14134 payload 824 2024-09-17T15:28:19.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 1 -- 172.21.15.86:0/579380560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f734c0a8330 con 0x7f734c13b790 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 msgr2=0x7f734c13bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f734affd640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 0x7f734c13bba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c01b050 con 0x7f734c13b790 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f7333fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f7333fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:19.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.718+0000 7f7333fff640 10 monclient: dump: 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f733c003620 con 0x7f734c13b790 2024-09-17T15:28:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 msgr2=0x7f734c13bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 0x7f734c13bba0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c004740 comp rx=0 tx=0).stop 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 msgr2=0x7f734c13bba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f7333fff640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f734c13b790 0x7f734c13bba0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f733c012810 tx=0x7f733c004740 comp rx=0 tx=0).stop 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:19.740 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 20, 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 8319, 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:19.742 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.743 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:19.744 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.745 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.746 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:19.747 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:19.748 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:19.749 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.751 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:19.752 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.753 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.754 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:19.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:19.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:19.757 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:19.757 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f733803bbb0 msgr2=0x7f733803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f733803bbb0 0x7f733803e070 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f734c0a6270 tx=0x7f73440018b0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 10 monclient: shutdown 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 msgr2=0x7f734c1374d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 --2- 172.21.15.86:0/579380560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f734c137050 0x7f734c1374d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7340005fc0 tx=0x7f73400074e0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f734b7fe640 1 -- 172.21.15.86:0/579380560 reap_dead start 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f734b7fe640 1 -- 172.21.15.86:0/579380560 reap_dead start 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 shutdown_connections 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 >> 172.21.15.86:0/579380560 conn(0x7f734c09bc70 msgr2=0x7f734c09c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 shutdown_connections 2024-09-17T15:28:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.722+0000 7f73518d4640 1 -- 172.21.15.86:0/579380560 wait complete. 2024-09-17T15:28:19.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:28:19.760 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:28:19.760 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:28:19.760 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:28:19.760 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:28:19.761 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:28:19.761 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:28:19.771 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:28:19.772 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:28:19.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: get_monmap_and_config 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: monmap: 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:19.928661+0000 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:19.928661+0000 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:19.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 Processor -- start 2024-09-17T15:28:19.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 -- start start 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: init 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbadc069260 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbadc074c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbadc09c890 0x7fbadc13ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc09c890 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc09e410 con 0x7fbadc069260 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc09e6f0 con 0x7fbadc074c30 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc09d150 con 0x7fbadc09c890 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae24d8640 10 monclient(hunting): authenticate will time out at 1032.894931s 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbadc09c890 0x7fbadc13ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbadc09c890 0x7fbadc13ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56178/0 (socket says 172.21.15.86:56178) 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 -- 172.21.15.86:0/3035960345 learned_addr learned my addr 172.21.15.86:0/3035960345 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): get_auth_request con 0x7fbadc09c890 auth_method 0 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbadbfff640 1 --1- 172.21.15.86:0/3035960345 >> v1:172.21.15.86:6789/0 conn(0x7fbadc069260 0x7fbadc09ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42984/0 (socket says 172.21.15.86:42984) 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 -- 172.21.15.86:0/3035960345 >> v1:172.21.15.203:6789/0 conn(0x7fbadc074c30 legacy=0x7fbadc09c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 -- 172.21.15.86:0/3035960345 >> v1:172.21.15.86:6789/0 conn(0x7fbadc069260 legacy=0x7fbadc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 monclient: found mon.noname-a 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.926+0000 7fbae0a75640 1 -- 172.21.15.86:0/3035960345 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc09e9d0 con 0x7fbadc09c890 2024-09-17T15:28:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae0a75640 1 -- 172.21.15.86:0/3035960345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc09c890 msgr2=0x7fbadc13ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae0a75640 1 --2- 172.21.15.86:0/3035960345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc09c890 0x7fbadc13ca90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbad400d180 tx=0x7fbad401ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b5cff403fab1c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 1 -- 172.21.15.86:0/3035960345 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad40ad070 con 0x7fbadc09c890 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: dump: 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 1 -- 172.21.15.86:0/3035960345 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad40a7b90 con 0x7fbadc09c890 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae24d8640 5 monclient: authenticate success, global_id 4428 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae24d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 1 -- 172.21.15.86:0/3035960345 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad40b1400 con 0x7fbadc09c890 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae24d8640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbae24d8640 4 set_mon_vals no callback set 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.930+0000 7fbadb7fe640 10 monclient: dump: 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.937 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: shutdown 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- 172.21.15.86:0/3035960345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc09c890 msgr2=0x7fbadc13ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3035960345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc09c890 0x7fbadc13ca90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbad400d180 tx=0x7fbad401ead0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- 172.21.15.86:0/3035960345 shutdown_connections 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3035960345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc09c890 0x7fbadc13ca90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- 172.21.15.86:0/3035960345 >> 172.21.15.86:0/3035960345 conn(0x7fbadc0a6440 msgr2=0x7fbadc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- 172.21.15.86:0/3035960345 shutdown_connections 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- 172.21.15.86:0/3035960345 wait complete. 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: build_initial_monmap 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: monmap: 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:19.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 Processor -- start 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- start start 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: init 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 0x7fbadc1d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc074c30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbadc09c890 0x7fbadc133330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc09c890 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc13fb80 con 0x7fbadc074c30 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc13fa00 con 0x7fbadc09c890 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc13fd00 con 0x7fbadc069260 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae24d8640 10 monclient(hunting): authenticate will time out at 1032.901999s 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbadbfff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 msgr2=0x7fbadc1d0170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbadbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 0x7fbadc1d0170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:19.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56180/0 (socket says 172.21.15.86:56180) 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 -- 172.21.15.86:0/3514160017 learned_addr learned my addr 172.21.15.86:0/3514160017 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): get_auth_request con 0x7fbadc074c30 auth_method 0 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient(hunting): handle_auth_done global_id 4431 payload 824 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbae0a75640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbadc09c890 0x7fbadc133330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 -- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 msgr2=0x7fbadc1d0170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 0x7fbadc1d0170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 -- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbadc09c890 msgr2=0x7fbadc133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbadc09c890 0x7fbadc133330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 monclient: found mon.noname-a 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.934+0000 7fbad37fe640 1 -- 172.21.15.86:0/3514160017 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc13c8b0 con 0x7fbadc074c30 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad37fe640 1 -- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 msgr2=0x7fbadc132df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad37fe640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c450 tx=0x7fbac400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4012a40 con 0x7fbadc074c30 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: dump: 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.947 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: _finish_auth 0 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbac4017070 con 0x7fbadc074c30 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 5 monclient: authenticate success, global_id 4431 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _renew_subs 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc09d460 con 0x7fbadc074c30 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac4012e40 con 0x7fbadc074c30 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 10 monclient: dump: 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:19.949 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _renew_subs 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc13c5d0 con 0x7fbadc074c30 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbadaffd640 4 set_mon_vals no callback set 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc09ba70 con 0x7fbadc074c30 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbac401c460 con 0x7fbadc074c30 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 0x7fbabc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4767+0+0 (secure 0 0 0) 0x7fbac404fdf0 con 0x7fbadc074c30 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbadbfff640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 0x7fbabc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:19.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.938+0000 7fbadbfff640 10 monclient: get_auth_request con 0x7fbabc03bb60 auth_method 0 2024-09-17T15:28:19.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.942+0000 7fbadbfff640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 0x7fbabc03e020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbacc004920 tx=0x7fbacc007a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:19.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.942+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbac401c930 con 0x7fbadc074c30 2024-09-17T15:28:19.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.942+0000 7fbad97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:19.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:19.942+0000 7fbad97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:20.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbae24d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbae24d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbae24d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbadc06d4f0 con 0x7fbadc074c30 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbad97fa640 1 -- 172.21.15.86:0/3514160017 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fbac401b4b0 con 0x7fbadc074c30 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbad97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.170+0000 7fbad97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:20.174 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 msgr2=0x7fbabc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 0x7fbabc03e020 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbacc004920 tx=0x7fbacc007a70 comp rx=0 tx=0).stop 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 10 monclient: shutdown 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 msgr2=0x7fbadc132df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbac400c450 tx=0x7fbac400c910 comp rx=0 tx=0).stop 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 shutdown_connections 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbabc03bb60 0x7fbabc03e020 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbadc09c890 0x7fbadc133330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbadc074c30 0x7fbadc132df0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:20.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 --2- 172.21.15.86:0/3514160017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbadc069260 0x7fbadc1d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 >> 172.21.15.86:0/3514160017 conn(0x7fbadc0a6440 msgr2=0x7fbadc073140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 shutdown_connections 2024-09-17T15:28:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:20.174+0000 7fbae24d8640 1 -- 172.21.15.86:0/3514160017 wait complete. 2024-09-17T15:28:23.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: get_monmap_and_config 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: build_initial_monmap 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: monmap: 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:23.325062+0000 2024-09-17T15:28:23.323 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:23.325062+0000 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:23.324 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 Processor -- start 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 -- start start 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: init 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4ea4068db0 0x7f4ea40691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea4068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ea4135ae0 0x7f4ea4135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea4135ae0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4ea4074500 con 0x7f4ea4135ae0 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4ea409c8b0 con 0x7f4ea4068db0 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea409cb90 con 0x7f4ea40689d0 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4eaa959640 10 monclient(hunting): authenticate will time out at 1036.290545s 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea3fff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 msgr2=0x7f4ea406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:23.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ea4135ae0 0x7f4ea4135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ea4135ae0 0x7f4ea4135f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56196/0 (socket says 172.21.15.86:56196) 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 -- 172.21.15.86:0/3080953311 learned_addr learned my addr 172.21.15.86:0/3080953311 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): get_auth_request con 0x7f4ea4135ae0 auth_method 0 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:23.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea37fe640 1 --2- 172.21.15.86:0/3080953311 >> v2:172.21.15.203:3300/0 conn(0x7f4ea4068db0 0x7f4ea40691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea37fe640 10 monclient(hunting): get_auth_request con 0x7f4ea4068db0 auth_method 0 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient(hunting): handle_auth_done global_id 4434 payload 274 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 -- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 msgr2=0x7f4ea406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 --2- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 -- 172.21.15.86:0/3080953311 >> v2:172.21.15.203:3300/0 conn(0x7f4ea4068db0 msgr2=0x7f4ea40691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 --2- 172.21.15.86:0/3080953311 >> v2:172.21.15.203:3300/0 conn(0x7f4ea4068db0 0x7f4ea40691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient: _finish_hunting 0 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 monclient: found mon.noname-a 2024-09-17T15:28:23.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:23.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:23.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.322+0000 7f4ea8ef6640 1 -- 172.21.15.86:0/3080953311 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea409ce70 con 0x7f4ea4135ae0 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.326+0000 7f4ea8ef6640 1 -- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4135ae0 msgr2=0x7f4ea4135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 1 --2- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4135ae0 0x7f4ea4135f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409cac0 tx=0x7f4e9c01cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88c1a2b09e65beb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 1 -- 172.21.15.86:0/3080953311 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c0b0070 con 0x7f4ea4135ae0 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: dump: 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.334 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 1 -- 172.21.15.86:0/3080953311 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e9c0ab600 con 0x7f4ea4135ae0 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 5 monclient: authenticate success, global_id 4434 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:23.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 1 -- 172.21.15.86:0/3080953311 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c0ab9d0 con 0x7f4ea4135ae0 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 4 set_mon_vals no callback set 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea2ffd640 10 monclient: dump: 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:23.336 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: shutdown 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4135ae0 msgr2=0x7f4ea4135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4135ae0 0x7f4ea4135f70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea409cac0 tx=0x7f4e9c01cda0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- 172.21.15.86:0/3080953311 shutdown_connections 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4135ae0 0x7f4ea4135f70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- 172.21.15.86:0/3080953311 >> v2:172.21.15.203:3300/0 conn(0x7f4ea4068db0 0x7f4ea40691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- 172.21.15.86:0/3080953311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- 172.21.15.86:0/3080953311 >> 172.21.15.86:0/3080953311 conn(0x7f4ea40a6440 msgr2=0x7f4ea40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- 172.21.15.86:0/3080953311 shutdown_connections 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- 172.21.15.86:0/3080953311 wait complete. 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: build_initial_monmap 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: monmap: 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:23.338 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 Processor -- start 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- start start 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: init 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:23.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea41340a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea4135ae0 0x7f4ea411dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea4135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea409c2d0 con 0x7f4ea4068db0 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea409be70 con 0x7f4ea4135ae0 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea413f820 con 0x7f4ea40689d0 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4eaa959640 10 monclient(hunting): authenticate will time out at 1036.298540s 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea3fff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 msgr2=0x7f4ea41340a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea41340a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56210/0 (socket says 172.21.15.86:56210) 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 -- 172.21.15.86:0/2536536467 learned_addr learned my addr 172.21.15.86:0/2536536467 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): get_auth_request con 0x7f4ea4068db0 auth_method 0 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:23.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea4135ae0 0x7f4ea411dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 10 monclient(hunting): get_auth_request con 0x7f4ea4135ae0 auth_method 0 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea8ef6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient(hunting): handle_auth_done global_id 4437 payload 824 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 -- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 msgr2=0x7f4ea41340a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea41340a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 -- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea4135ae0 msgr2=0x7f4ea411dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea4135ae0 0x7f4ea411dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 monclient: found mon.noname-a 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:23.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.330+0000 7f4ea37fe640 1 -- 172.21.15.86:0/2536536467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea41217a0 con 0x7f4ea4068db0 2024-09-17T15:28:23.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea37fe640 1 -- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 msgr2=0x7f4ea41345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea37fe640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007e90 tx=0x7f4e8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea37fe640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007e90 tx=0x7f4e8c00e4b0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f4e8c013790 mon_map magic: 0 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c013790 con 0x7f4ea4068db0 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: dump: 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c019070 con 0x7f4ea4068db0 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 5 monclient: authenticate success, global_id 4437 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _renew_subs 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea1ffb640 4 set_mon_vals no callback set 2024-09-17T15:28:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea4121a80 con 0x7f4ea4068db0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _renew_subs 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea4120840 con 0x7f4ea4068db0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c013b70 con 0x7f4ea4068db0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea40691d0 con 0x7f4ea4068db0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 10 monclient: dump: 2024-09-17T15:28:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4e8c013d10 con 0x7f4ea4068db0 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea0ff9640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 0x7f4e8403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea3fff640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 0x7f4e8403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.334+0000 7f4ea3fff640 10 monclient: get_auth_request con 0x7f4e8403bb10 auth_method 0 2024-09-17T15:28:23.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.338+0000 7f4ea3fff640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 0x7f4e8403dfd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4e94006fd0 tx=0x7f4e94006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:23.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.338+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:28:23.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.342+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4767+0+0 (secure 0 0 0) 0x7f4e8c050890 con 0x7f4ea4068db0 2024-09-17T15:28:23.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.958+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4e8c04f2a0 con 0x7f4ea4068db0 2024-09-17T15:28:23.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.958+0000 7f4ea0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:23.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:23.958+0000 7f4ea0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4eaa959640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4eaa959640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4eaa959640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4ea406d4f0 con 0x7f4ea4068db0 2024-09-17T15:28:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4ea0ff9640 1 -- 172.21.15.86:0/2536536467 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f4e8c01ec60 con 0x7f4ea4068db0 2024-09-17T15:28:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4ea0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.206+0000 7f4ea0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:24.209 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 msgr2=0x7f4e8403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 0x7f4e8403dfd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4e94006fd0 tx=0x7f4e94006d40 comp rx=0 tx=0).stop 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 10 monclient: shutdown 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 msgr2=0x7f4ea41345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c007e90 tx=0x7f4e8c00e4b0 comp rx=0 tx=0).stop 2024-09-17T15:28:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 shutdown_connections 2024-09-17T15:28:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e8403bb10 0x7f4e8403dfd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea4135ae0 0x7f4ea411dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ea4068db0 0x7f4ea41345e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 --2- 172.21.15.86:0/2536536467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ea40689d0 0x7f4ea41340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 >> 172.21.15.86:0/2536536467 conn(0x7f4ea40a6440 msgr2=0x7f4ea4134f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:24.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 shutdown_connections 2024-09-17T15:28:24.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:24.210+0000 7f4eaa959640 1 -- 172.21.15.86:0/2536536467 wait complete. 2024-09-17T15:28:27.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: get_monmap_and_config 2024-09-17T15:28:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: build_initial_monmap 2024-09-17T15:28:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: monmap: 2024-09-17T15:28:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:27.356 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:27.358289+0000 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:27.358289+0000 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:27.357 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 Processor -- start 2024-09-17T15:28:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 -- start start 2024-09-17T15:28:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: init 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb54c0689d0 0x7fb54c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 0x7fb54c069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb54c135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb54c09c8b0 con 0x7fb54c135ae0 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb54c09cb90 con 0x7fb54c0689d0 2024-09-17T15:28:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb54c09ce70 con 0x7fb54c068db0 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb5533ac640 10 monclient(hunting): authenticate will time out at 1040.323782s 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb551148640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb54c0689d0 0x7fb54c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.354+0000 7fb551148640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb54c0689d0 0x7fb54c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56222/0 (socket says 172.21.15.86:56222) 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 -- 172.21.15.86:0/814615717 learned_addr learned my addr 172.21.15.86:0/814615717 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): get_auth_request con 0x7fb54c0689d0 auth_method 0 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb550947640 1 --2- 172.21.15.86:0/814615717 >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 0x7fb54c069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient(hunting): handle_auth_done global_id 4440 payload 274 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 -- 172.21.15.86:0/814615717 >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 msgr2=0x7fb54c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 --2- 172.21.15.86:0/814615717 >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 0x7fb54c069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 -- 172.21.15.86:0/814615717 >> v1:172.21.15.203:6789/0 conn(0x7fb54c135ae0 legacy=0x7fb54c135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient: _finish_hunting 0 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 monclient: found mon.noname-a 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb551148640 1 -- 172.21.15.86:0/814615717 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c069740 con 0x7fb54c0689d0 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb550947640 10 monclient: get_auth_request con 0x7fb54c068db0 auth_method 0 2024-09-17T15:28:27.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.358+0000 7fb550947640 1 --2- 172.21.15.86:0/814615717 >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 0x7fb54c069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb551148640 1 -- 172.21.15.86:0/814615717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c0689d0 msgr2=0x7fb54c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb551148640 1 --2- 172.21.15.86:0/814615717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c0689d0 0x7fb54c06f840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a05c0 tx=0x7fb54401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963fe960a2e83d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 1 -- 172.21.15.86:0/814615717 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5440b0070 con 0x7fb54c0689d0 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: dump: 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.364 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: _finish_auth 0 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 1 -- 172.21.15.86:0/814615717 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5440ab600 con 0x7fb54c0689d0 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 5 monclient: authenticate success, global_id 4440 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:27.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 1 -- 172.21.15.86:0/814615717 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5440ab9d0 con 0x7fb54c0689d0 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 4 set_mon_vals no callback set 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb54bfff640 10 monclient: dump: 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:27.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: shutdown 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- 172.21.15.86:0/814615717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c0689d0 msgr2=0x7fb54c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- 172.21.15.86:0/814615717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c0689d0 0x7fb54c06f840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a05c0 tx=0x7fb54401c470 comp rx=0 tx=0).stop 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- 172.21.15.86:0/814615717 shutdown_connections 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- 172.21.15.86:0/814615717 >> v2:172.21.15.203:3300/0 conn(0x7fb54c068db0 0x7fb54c069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- 172.21.15.86:0/814615717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c0689d0 0x7fb54c06f840 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- 172.21.15.86:0/814615717 >> 172.21.15.86:0/814615717 conn(0x7fb54c0a6440 msgr2=0x7fb54c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- 172.21.15.86:0/814615717 shutdown_connections 2024-09-17T15:28:27.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- 172.21.15.86:0/814615717 wait complete. 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: build_initial_monmap 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: monmap: 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:27.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 Processor -- start 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- start start 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: init 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb54c0689d0 0x7fb54c11dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-b con 0x7fb54c0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:27.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 0x7fb54c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-c con 0x7fb54c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): picked mon.noname-a con 0x7fb54c135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c140a40 con 0x7fb54c135ae0 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c09c2d0 con 0x7fb54c0689d0 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb54c140bc0 con 0x7fb54c068db0 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.362+0000 7fb5533ac640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient(hunting): authenticate will time out at 1040.332013s 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb550947640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 msgr2=0x7fb54c11c8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb550947640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 0x7fb54c11c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:27.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56226/0 (socket says 172.21.15.86:56226) 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 -- 172.21.15.86:0/3239425634 learned_addr learned my addr 172.21.15.86:0/3239425634 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): get_auth_request con 0x7fb54c135ae0 auth_method 0 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551148640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb54c0689d0 0x7fb54c11dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551148640 10 monclient(hunting): get_auth_request con 0x7fb54c0689d0 auth_method 0 2024-09-17T15:28:27.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551148640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551148640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551148640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 -- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 msgr2=0x7fb54c11c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 0x7fb54c11c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 -- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb54c0689d0 msgr2=0x7fb54c11dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb54c0689d0 0x7fb54c11dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient: _finish_hunting 0 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 monclient: found mon.noname-a 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 -- 172.21.15.86:0/3239425634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c0a2840 con 0x7fb54c135ae0 2024-09-17T15:28:27.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 -- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 msgr2=0x7fb54c134b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb551949640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c3c0 tx=0x7fb53c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0119a0 con 0x7fb54c135ae0 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: dump: 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:27.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb53c017070 con 0x7fb54c135ae0 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c011d80 con 0x7fb54c135ae0 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 5 monclient: authenticate success, global_id 4443 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _renew_subs 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c09d460 con 0x7fb54c135ae0 2024-09-17T15:28:27.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb54b7fe640 4 set_mon_vals no callback set 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 10 monclient: dump: 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:27.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _renew_subs 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c11ce00 con 0x7fb54c135ae0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb53c01c460 con 0x7fb54c135ae0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.366+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c06f840 con 0x7fb54c135ae0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.370+0000 7fb549ffb640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 0x7fb53803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.370+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4767+0+0 (secure 0 0 0) 0x7fb53c04fce0 con 0x7fb54c135ae0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.370+0000 7fb551148640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 0x7fb53803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:27.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.370+0000 7fb551148640 10 monclient: get_auth_request con 0x7fb53803bb60 auth_method 0 2024-09-17T15:28:27.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.370+0000 7fb551148640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 0x7fb53803e020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a0620 tx=0x7fb544008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:27.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.374+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb53c01ac60 con 0x7fb54c135ae0 2024-09-17T15:28:27.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.374+0000 7fb549ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:27.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.374+0000 7fb549ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb5533ac640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb5533ac640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb5533ac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb54c1350c0 con 0x7fb54c135ae0 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb549ffb640 1 -- 172.21.15.86:0/3239425634 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb53c04e6f0 con 0x7fb54c135ae0 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb549ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.598+0000 7fb549ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:27.601 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:27.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 msgr2=0x7fb53803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 0x7fb53803e020 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a0620 tx=0x7fb544008040 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 10 monclient: shutdown 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 msgr2=0x7fb54c134b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c3c0 tx=0x7fb53c00c880 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 shutdown_connections 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb53803bb60 0x7fb53803e020 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb54c135ae0 0x7fb54c134b80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb54c068db0 0x7fb54c11c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 --2- 172.21.15.86:0/3239425634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb54c0689d0 0x7fb54c11dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 >> 172.21.15.86:0/3239425634 conn(0x7fb54c0a6440 msgr2=0x7fb54c0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:27.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 shutdown_connections 2024-09-17T15:28:27.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:27.602+0000 7fb5533ac640 1 -- 172.21.15.86:0/3239425634 wait complete. 2024-09-17T15:28:30.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: get_monmap_and_config 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: build_initial_monmap 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: monmap: 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:30.752997+0000 2024-09-17T15:28:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:30.752997+0000 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:30.752 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 Processor -- start 2024-09-17T15:28:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 -- start start 2024-09-17T15:28:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: init 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-b con 0x7fe7000689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-c con 0x7fe700068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-a con 0x7fe700135ae0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe700074500 con 0x7fe700135ae0 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe70009c8b0 con 0x7fe700068db0 2024-09-17T15:28:30.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe70009cb90 con 0x7fe7000689d0 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe70562b640 10 monclient(hunting): authenticate will time out at 1043.718520s 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6feffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 msgr2=0x7fe70006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6feffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49514/0 (socket says 172.21.15.86:49514) 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 learned_addr learned my addr 172.21.15.86:0/2274046047 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe700135ae0 auth_method 0 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:30.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe700068db0 auth_method 0 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:30.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.750+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6feffd640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 msgr2=0x7fe70006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6feffd640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe700135ae0 auth_method 0 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe700068db0 auth_method 0 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:30.954+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:28:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe700135ae0 auth_method 0 2024-09-17T15:28:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6feffd640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 msgr2=0x7fe70006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6feffd640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe700068db0 auth_method 0 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:31.358+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6feffd640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 msgr2=0x7fe70006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6feffd640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe700135ae0 auth_method 0 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:32.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:32.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:32.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe700068db0 auth_method 0 2024-09-17T15:28:32.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:32.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:32.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:32.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:28:32.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:28:32.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:32.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:32.162+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): tick 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 msgr2=0x7fe70006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7000689d0 0x7fe70006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 msgr2=0x7fe7000691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:33.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.203:3300/0 conn(0x7fe700068db0 0x7fe7000691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 -- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 msgr2=0x7fe700135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 --2- 172.21.15.86:0/2274046047 >> v2:172.21.15.86:3300/0 conn(0x7fe700135ae0 0x7fe700135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient: _add_conns ranks=[1,0] 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6ec0036c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6ec0073f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 -- 172.21.15.86:0/2274046047 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe70009ce70 con 0x7fe6ec0073f0 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 1 -- 172.21.15.86:0/2274046047 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec00c130 con 0x7fe6ec0036c0 2024-09-17T15:28:33.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fcff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:33.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> v1:172.21.15.86:6789/0 conn(0x7fe6ec0073f0 legacy=0x7fe6ec009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:33.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:33.754+0000 7fe6feffd640 1 -- 172.21.15.86:0/2274046047 >> v1:172.21.15.203:6789/0 conn(0x7fe6ec0036c0 legacy=0x7fe6ec005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:34.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007782175 0 0) 0x7fe70009ce70 con 0x7fe6ec0073f0 2024-09-17T15:28:34.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 10 monclient(hunting): _negotiate 2024-09-17T15:28:34.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:34.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:34.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 10 monclient(hunting): my global_id is 4446 2024-09-17T15:28:34.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.598+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6e8003530 con 0x7fe6ec0073f0 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:145: done (2 left) 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:146: snap_create 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:142: finishing rollback tid 0 to smithi20328086-35 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:142: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:142: finishing rollback tid 2 to smithi20328086-35 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 17 (ObjNum 65 snap 2 seq_num 65) dirty exists 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: finishing write tid 1 to smithi20328086-43 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: finishing write tid 2 to smithi20328086-43 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: finishing write tid 3 to smithi20328086-43 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: finishing write tid 4 to smithi20328086-43 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: finishing write tid 5 to smithi20328086-43 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: oid 43 updating version 0 to 18 2024-09-17T15:28:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:143: oid 43 updating version 18 to 19 2024-09-17T15:28:34.604 INFO:tasks.rados.rados.0.smithi203.stdout:143: oid 43 version 19 is already newer than 17 2024-09-17T15:28:34.604 INFO:tasks.rados.rados.0.smithi203.stdout:143: oid 43 updating version 19 to 20 2024-09-17T15:28:34.604 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 20 (ObjNum 88 snap 10 seq_num 88) dirty exists 2024-09-17T15:28:34.604 INFO:tasks.rados.rados.0.smithi203.stdout:143: left oid 43 (ObjNum 88 snap 10 seq_num 88) 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 643214784 0 0) 0x7fe6e8003530 con 0x7fe6ec0073f0 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 >> v1:172.21.15.203:6789/0 conn(0x7fe6ec0036c0 legacy=0x7fe6ec005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _finish_hunting 0 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec00d390 con 0x7fe6ec0073f0 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:34.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fe6f400f040 con 0x7fe6ec0073f0 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: dump: 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:34.608 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 >> v1:172.21.15.86:6789/0 conn(0x7fe6ec0073f0 legacy=0x7fe6ec009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 reap_dead start 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 0x7fe6e8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): picked mon.b con 0x7fe6e8003a60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe6e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): picked mon.a con 0x7fe6e8004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 0x7fe6e800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): picked mon.c con 0x7fe6e80099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:34.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e800c3a0 con 0x7fe6e8004810 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6e800c520 con 0x7fe6e8003a60 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6e800c720 con 0x7fe6e80099c0 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 msgr2=0x7fe6e800bdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fdffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 0x7fe6e800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe6e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request con 0x7fe6e8004810 auth_method 0 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:34.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 0x7fe6e8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 10 monclient(hunting): get_auth_request con 0x7fe6e8003a60 auth_method 0 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient(hunting): handle_auth_done global_id 4446 payload 293 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 msgr2=0x7fe6e800bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 0x7fe6e800bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 msgr2=0x7fe6e8003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 0x7fe6e8003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 monclient: found mon.a 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6e8003530 con 0x7fe6e8004810 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient: _finish_auth 0 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe70562b640 5 monclient: authenticate success, global_id 4446 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe70562b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.602+0000 7fe6feffd640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 0x7fe6e8003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fe7fc640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 msgr2=0x7fe6e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fe7fc640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe6e8004c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4000ee0 tx=0x7fe6f401d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acc94a1af329c71 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f40af020 con 0x7fe6e8004810 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: dump: 2024-09-17T15:28:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f40a7680 con 0x7fe6e8004810 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe70562b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe70562b640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe70562b640 4 set_mon_vals no callback set 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 1 -- 172.21.15.86:0/2274046047 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f400eea0 con 0x7fe6e8004810 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.610+0000 7fe6fdffb640 10 monclient: dump: 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:34.616 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: shutdown 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 msgr2=0x7fe6e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe6e8004c90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4000ee0 tx=0x7fe6f401d870 comp rx=0 tx=0).stop 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- 172.21.15.86:0/2274046047 shutdown_connections 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e80099c0 0x7fe6e800bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe6e8004c90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- 172.21.15.86:0/2274046047 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8003a60 0x7fe6e8003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- 172.21.15.86:0/2274046047 >> 172.21.15.86:0/2274046047 conn(0x7fe7000a6440 msgr2=0x7fe7000a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:34.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- 172.21.15.86:0/2274046047 shutdown_connections 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- 172.21.15.86:0/2274046047 wait complete. 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: build_initial_monmap 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: monmap: 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:34.619 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 Processor -- start 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- start start 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: init 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 0x7fe7001cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-c con 0x7fe6e8003a60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe700134060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-a con 0x7fe6e8004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): picked mon.noname-b con 0x7fe6e80099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:34.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe700136280 con 0x7fe6e8004810 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe700136100 con 0x7fe6e80099c0 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe700136400 con 0x7fe6e8003a60 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe70562b640 10 monclient(hunting): authenticate will time out at 1047.582825s 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6feffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 msgr2=0x7fe7001cf5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6feffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 0x7fe7001cf5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:59646/0 (socket says 172.21.15.86:59646) 2024-09-17T15:28:34.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/3116647669 learned_addr learned my addr 172.21.15.86:0/3116647669 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe6e80099c0 auth_method 0 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient(hunting): handle_auth_done global_id 14146 payload 824 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 msgr2=0x7fe7001cf5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 0x7fe7001cf5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 msgr2=0x7fe700134060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe700134060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:34.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 monclient: found mon.noname-b 2024-09-17T15:28:34.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:34.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.203:3300/0 2024-09-17T15:28:34.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.614+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/3116647669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe700121560 con 0x7fe6e80099c0 2024-09-17T15:28:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.618+0000 7fe6ff7fe640 1 -- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 msgr2=0x7fe700070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.618+0000 7fe6ff7fe640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007360 tx=0x7fe6f0012a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f001b030 con 0x7fe6e80099c0 2024-09-17T15:28:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:34.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: dump: 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f0016990 con 0x7fe6e80099c0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 5 monclient: authenticate success, global_id 14146 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _renew_subs 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7001d0470 con 0x7fe6e80099c0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000f040 con 0x7fe6e80099c0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 10 monclient: dump: 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:34.626 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6fd7fa640 4 set_mon_vals no callback set 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _renew_subs 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe70009d750 con 0x7fe6e80099c0 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:28:34.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe700136790 con 0x7fe6e80099c0 2024-09-17T15:28:34.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe6f001e030 con 0x7fe6e80099c0 2024-09-17T15:28:34.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 0x7fe6ec03eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:34.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4767+0+0 (secure 0 0 0) 0x7fe6f00505a0 con 0x7fe6e80099c0 2024-09-17T15:28:34.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6feffd640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 0x7fe6ec03eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:34.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.622+0000 7fe6feffd640 10 monclient: get_auth_request con 0x7fe6ec03c620 auth_method 0 2024-09-17T15:28:34.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.626+0000 7fe6feffd640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 0x7fe6ec03eae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8001230 tx=0x7fe6f8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:34.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.634+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe6f0025030 con 0x7fe6e80099c0 2024-09-17T15:28:34.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.634+0000 7fe6e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:34.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.634+0000 7fe6e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:34.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:34.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:34.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:28:34.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe70006d4f0 con 0x7fe6e80099c0 2024-09-17T15:28:34.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe6e77fe640 1 -- 172.21.15.86:0/3116647669 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fe6f0016dc0 con 0x7fe6e80099c0 2024-09-17T15:28:34.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe6e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:34.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe6e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:34.872 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 msgr2=0x7fe6ec03eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 0x7fe6ec03eae0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8001230 tx=0x7fe6f8009040 comp rx=0 tx=0).stop 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 10 monclient: shutdown 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 msgr2=0x7fe700070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:34.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.870+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec007360 tx=0x7fe6f0012a30 comp rx=0 tx=0).stop 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 shutdown_connections 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6ec03c620 0x7fe6ec03eae0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80099c0 0x7fe700070f30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6e8004810 0x7fe700134060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 --2- 172.21.15.86:0/3116647669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6e8003a60 0x7fe7001cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 >> 172.21.15.86:0/3116647669 conn(0x7fe7000a6440 msgr2=0x7fe700135220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 shutdown_connections 2024-09-17T15:28:34.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.874+0000 7fe70562b640 1 -- 172.21.15.86:0/3116647669 wait complete. 2024-09-17T15:28:34.890 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:28:34.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:28:34.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:28:34.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 10 monclient: get_monmap_and_config 2024-09-17T15:28:34.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 10 monclient: build_initial_monmap 2024-09-17T15:28:34.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 10 monclient: monmap: 2024-09-17T15:28:34.997 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:34.997 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:34.999007+0000 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:34.999007+0000 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:34.998 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 1 Processor -- start 2024-09-17T15:28:35.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 1 -- start start 2024-09-17T15:28:35.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.994+0000 7f4fb57bf640 10 monclient: init 2024-09-17T15:28:35.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb00a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-b con 0x7f4fb00a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fb00a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4fb0136160 0x7f4fb013df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-c con 0x7f4fb0136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb00a2130 con 0x7f4fb00a1590 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb00a2410 con 0x7f4fb0136160 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb00a26f0 con 0x7f4fb00a0c70 2024-09-17T15:28:35.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fb57bf640 10 monclient(hunting): authenticate will time out at 1047.964381s 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faeffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 msgr2=0x7f4fb00a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faeffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb00a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fae7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4fb00a1590 0x7f4fb0135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54606/0 (socket says 172.21.15.86:54606) 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4fae7fc640 1 -- 172.21.15.86:0/1073027743 learned_addr learned my addr 172.21.15.86:0/1073027743 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/1073027743 >> v2:172.21.15.203:3300/0 conn(0x7f4fb0136160 0x7f4fb013df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): get_auth_request con 0x7f4fb0136160 auth_method 0 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_done global_id 14149 payload 274 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 -- 172.21.15.86:0/1073027743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 msgr2=0x7f4fb00a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/1073027743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb00a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 -- 172.21.15.86:0/1073027743 >> v1:172.21.15.86:6789/0 conn(0x7f4fb00a1590 legacy=0x7f4fb0135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 monclient: found mon.noname-c 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:34.998+0000 7f4faf7fe640 1 -- 172.21.15.86:0/1073027743 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb009f9f0 con 0x7f4fb0136160 2024-09-17T15:28:35.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4faf7fe640 1 -- 172.21.15.86:0/1073027743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb0136160 msgr2=0x7f4fb013df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/1073027743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb0136160 0x7f4fb013df30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80097c0 tx=0x7f4fa801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82e9090096fa2d78 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 1 -- 172.21.15.86:0/1073027743 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80b0070 con 0x7f4fb0136160 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: dump: 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: _finish_auth 0 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: _check_auth_tickets 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 1 -- 172.21.15.86:0/1073027743 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa80ab600 con 0x7f4fb0136160 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 5 monclient: authenticate success, global_id 14149 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:35.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 1 -- 172.21.15.86:0/1073027743 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80ab9d0 con 0x7f4fb0136160 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 4 set_mon_vals no callback set 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fadffb640 10 monclient: dump: 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.007 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 10 monclient: shutdown 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- 172.21.15.86:0/1073027743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb0136160 msgr2=0x7f4fb013df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/1073027743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb0136160 0x7f4fb013df30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4fa80097c0 tx=0x7f4fa801c380 comp rx=0 tx=0).stop 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- 172.21.15.86:0/1073027743 shutdown_connections 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/1073027743 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb0136160 0x7f4fb013df30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/1073027743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb00a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- 172.21.15.86:0/1073027743 >> 172.21.15.86:0/1073027743 conn(0x7f4fb00a2f40 msgr2=0x7f4fb00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- 172.21.15.86:0/1073027743 shutdown_connections 2024-09-17T15:28:35.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- 172.21.15.86:0/1073027743 wait complete. 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 10 monclient: build_initial_monmap 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 10 monclient: monmap: 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:35.014 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 Processor -- start 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 1 -- start start 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.002+0000 7f4fb57bf640 10 monclient: init 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb01cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-c con 0x7f4fb00a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb00a1590 0x7f4fb011e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-b con 0x7f4fb00a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 0x7f4fb0135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): picked mon.noname-a con 0x7f4fb0136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:35.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb00a00d0 con 0x7f4fb0136160 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb009ff50 con 0x7f4fb00a1590 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb0140c90 con 0x7f4fb00a0c70 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): authenticate will time out at 1047.972382s 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faeffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 msgr2=0x7f4fb01cdd80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faeffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb01cdd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 0x7f4fb0135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 0x7f4fb0135160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49578/0 (socket says 172.21.15.86:49578) 2024-09-17T15:28:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 learned_addr learned my addr 172.21.15.86:0/2591815936 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): get_auth_request con 0x7f4fb0136160 auth_method 0 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fae7fc640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb00a1590 0x7f4fb011e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient(hunting): handle_auth_done global_id 4449 payload 824 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 msgr2=0x7f4fb01cdd80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4fb00a0c70 0x7f4fb01cdd80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb00a1590 msgr2=0x7f4fb011e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb00a1590 0x7f4fb011e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 monclient: found mon.noname-a 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb0135e00 con 0x7f4fb0136160 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 msgr2=0x7f4fb0135160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4faf7fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 0x7f4fb0135160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4fa801c1b0 tx=0x7f4fa8019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80b9070 con 0x7f4fb0136160 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: dump: 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.018 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa80b5680 con 0x7f4fb0136160 2024-09-17T15:28:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 5 monclient: authenticate success, global_id 4449 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _renew_subs 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb006f9b0 con 0x7f4fb0136160 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80b0040 con 0x7f4fb0136160 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fa77fe640 10 monclient: dump: 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _renew_subs 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fad7fa640 4 set_mon_vals no callback set 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb011faf0 con 0x7f4fb0136160 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 0x7f4fb011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.006+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb00a1050 con 0x7f4fb01356a0 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 0x7f4fb011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient: get_auth_request con 0x7f4fb01356a0 auth_method 0 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4fa80bc070 con 0x7f4fb0136160 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f9003bb10 0x7f4f9003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4767+0+0 (secure 0 0 0) 0x7f4fa80e9ef0 con 0x7f4fb0136160 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fb00008e0 con 0x7f4fb01356a0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fae7fc640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f9003bb10 0x7f4f9003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fae7fc640 10 monclient: get_auth_request con 0x7f4f9003bb10 auth_method 0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 msgr2=0x7f4fb011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4faeffd640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 0x7f4fb011c8c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0003d10 tx=0x7f4fa0009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0005560 con 0x7f4fb01356a0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 10 monclient: dump: 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.023 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4fa0005560 con 0x7f4fb01356a0 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 msgr2=0x7f4fb011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 0x7f4fb011c8c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0003d10 tx=0x7f4fa0009f30 comp rx=0 tx=0).stop 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 msgr2=0x7f4fb011c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:35.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fa77fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb01356a0 0x7f4fb011c8c0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0003d10 tx=0x7f4fa0009f30 comp rx=0 tx=0).stop 2024-09-17T15:28:35.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.010+0000 7f4fae7fc640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f9003bb10 0x7f4f9003dfd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0135ff0 tx=0x7f4f98009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fb57bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:35.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 0x7f4fb011d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fb57bf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb011d750 con 0x7f4fb011ce00 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 0x7f4fb011d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient: get_auth_request con 0x7f4fb011ce00 auth_method 0 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 1 -- 172.21.15.86:0/2591815936 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4fb011f250 con 0x7f4fb011ce00 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 msgr2=0x7f4fb011d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4faeffd640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 0x7f4fb011d210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0014000 tx=0x7f4fa0005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa001aa30 con 0x7f4fb011ce00 2024-09-17T15:28:35.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 10 monclient: dump: 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f4fa001aa30 con 0x7f4fb011ce00 2024-09-17T15:28:35.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 msgr2=0x7f4fb011d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 0x7f4fb011d210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0014000 tx=0x7f4fa0005d60 comp rx=0 tx=0).stop 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 msgr2=0x7f4fb011d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.018+0000 7f4fa77fe640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb011ce00 0x7f4fb011d210 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:28:35.029 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 24, 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 87661, 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.031 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:35.032 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.033 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.034 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.035 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.036 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:35.038 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.040 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.041 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.042 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:35.042 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f9003bb10 msgr2=0x7f4f9003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f9003bb10 0x7f4f9003dfd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0135ff0 tx=0x7f4f98009040 comp rx=0 tx=0).stop 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 10 monclient: shutdown 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 msgr2=0x7f4fb0135160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 --2- 172.21.15.86:0/2591815936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4fb0136160 0x7f4fb0135160 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4fa801c1b0 tx=0x7f4fa8019120 comp rx=0 tx=0).stop 2024-09-17T15:28:35.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 reap_dead start 2024-09-17T15:28:35.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4faf7fe640 1 -- 172.21.15.86:0/2591815936 reap_dead start 2024-09-17T15:28:35.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 shutdown_connections 2024-09-17T15:28:35.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 >> 172.21.15.86:0/2591815936 conn(0x7f4fb00a2f40 msgr2=0x7f4fb0071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:35.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 shutdown_connections 2024-09-17T15:28:35.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.022+0000 7f4fb57bf640 1 -- 172.21.15.86:0/2591815936 wait complete. 2024-09-17T15:28:35.044 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:28:35.044 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:28:35.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: get_monmap_and_config 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: build_initial_monmap 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: monmap: 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:35.160186+0000 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:35.160186+0000 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.158 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:35.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:35.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:35.159 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:35.159 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:35.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 Processor -- start 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 -- start start 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: init 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0e600a50e0 0x7f0e6009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-a con 0x7f0e600a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e600a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 0x7f0e6013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-b con 0x7f0e6013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e600a6280 con 0x7f0e600a54c0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6009d2a0 con 0x7f0e600a50e0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6009c0a0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e687ad640 10 monclient(hunting): authenticate will time out at 1048.125685s 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66d4a640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 msgr2=0x7f0e6013f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66d4a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 0x7f0e6013f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0e600a50e0 0x7f0e6009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e65d48640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0e600a54c0 0x7f0e600a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54612/0 (socket says 172.21.15.86:54612) 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0e600a50e0 0x7f0e6009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49606/0 (socket says 172.21.15.86:49606) 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e65d48640 1 -- 172.21.15.86:0/1383844117 learned_addr learned my addr 172.21.15.86:0/1383844117 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): get_auth_request con 0x7f0e600a50e0 auth_method 0 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 -- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 msgr2=0x7f0e6013f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 --2- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 0x7f0e6013f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 -- 172.21.15.86:0/1383844117 >> v1:172.21.15.86:6789/0 conn(0x7f0e600a54c0 legacy=0x7f0e600a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient: _finish_hunting 0 2024-09-17T15:28:35.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 monclient: found mon.noname-a 2024-09-17T15:28:35.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:35.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.158+0000 7f0e66549640 1 -- 172.21.15.86:0/1383844117 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6009c380 con 0x7f0e600a50e0 2024-09-17T15:28:35.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e66549640 1 -- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e600a50e0 msgr2=0x7f0e6009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e66549640 1 --2- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e600a50e0 0x7f0e6009cd60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e540097c0 tx=0x7f0e5401c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63d5e31f4f48ec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 1 -- 172.21.15.86:0/1383844117 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e540b0070 con 0x7f0e600a50e0 2024-09-17T15:28:35.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: dump: 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: _finish_auth 0 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: _check_auth_tickets 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 5 monclient: authenticate success, global_id 4458 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 1 -- 172.21.15.86:0/1383844117 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e540ab600 con 0x7f0e600a50e0 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 4 set_mon_vals no callback set 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 1 -- 172.21.15.86:0/1383844117 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e540ab9d0 con 0x7f0e600a50e0 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e65547640 10 monclient: dump: 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 10 monclient: shutdown 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 -- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e600a50e0 msgr2=0x7f0e6009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 --2- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e600a50e0 0x7f0e6009cd60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e540097c0 tx=0x7f0e5401c350 comp rx=0 tx=0).stop 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 -- 172.21.15.86:0/1383844117 shutdown_connections 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 --2- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e6013d2f0 0x7f0e6013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 --2- 172.21.15.86:0/1383844117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e600a50e0 0x7f0e6009cd60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 -- 172.21.15.86:0/1383844117 >> 172.21.15.86:0/1383844117 conn(0x7f0e600aaae0 msgr2=0x7f0e600aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:35.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 -- 172.21.15.86:0/1383844117 shutdown_connections 2024-09-17T15:28:35.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 1 -- 172.21.15.86:0/1383844117 wait complete. 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.162+0000 7f0e687ad640 10 monclient: build_initial_monmap 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: monmap: 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 Processor -- start 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- start start 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: init 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600a50e0 0x7f0e601cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-b con 0x7f0e600a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e600a54c0 0x7f0e601d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-c con 0x7f0e600a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): picked mon.noname-a con 0x7f0e6013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0e60142650 con 0x7f0e6013d2f0 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e600a6b00 con 0x7f0e600a50e0 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0e601427d0 con 0x7f0e600a54c0 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient(hunting): authenticate will time out at 1048.132889s 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e65d48640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e600a54c0 msgr2=0x7f0e601d02a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e65d48640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e600a54c0 0x7f0e601d02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49612/0 (socket says 172.21.15.86:49612) 2024-09-17T15:28:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 learned_addr learned my addr 172.21.15.86:0/772781293 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): get_auth_request con 0x7f0e6013d2f0 auth_method 0 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e600a54c0 msgr2=0x7f0e601d02a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600a50e0 0x7f0e601cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e600a54c0 0x7f0e601d02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600a50e0 msgr2=0x7f0e601cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e600a50e0 0x7f0e601cfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient: _finish_hunting 0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 monclient: found mon.noname-a 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6013d6d0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 msgr2=0x7f0e601d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800b490 tx=0x7f0e5800b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e66d4a640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800b490 tx=0x7f0e5800b950 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f0e580060d0 config(0 keys) 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e58017070 con 0x7f0e6013d2f0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: dump: 2024-09-17T15:28:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e580060d0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e52ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 5 monclient: authenticate success, global_id 4461 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _renew_subs 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e64d46640 4 set_mon_vals no callback set 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6013b4b0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _renew_subs 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.166+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6013bc30 con 0x7f0e6013d2f0 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e687ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e687ad640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e600a18d0 con 0x7f0e60134e70 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e580064a0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 10 monclient: dump: 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0e5801a070 con 0x7f0e6013d2f0 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient: get_auth_request con 0x7f0e60134e70 auth_method 0 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e52ffd640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e4403bb40 0x7f0e4403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 msgr2=0x7f0e60135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 msgr2=0x7f0e60135250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e65d48640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e4403bb40 0x7f0e4403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.170+0000 7f0e65d48640 10 monclient: get_auth_request con 0x7f0e4403bb40 auth_method 0 2024-09-17T15:28:35.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.174+0000 7f0e65d48640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e4403bb40 0x7f0e4403e000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e601d0e20 tx=0x7f0e5c005110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient: get_auth_request con 0x7f0e60134e70 auth_method 0 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 10 monclient(hunting): handle_auth_done global_id 14155 payload 824 2024-09-17T15:28:35.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.374+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e6009cd60 con 0x7f0e60134e70 2024-09-17T15:28:35.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 msgr2=0x7f0e60135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e540ad000 tx=0x7f0e540190f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e540b0040 con 0x7f0e60134e70 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 10 monclient: dump: 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0e540093e0 con 0x7f0e60134e70 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 msgr2=0x7f0e60135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e540ad000 tx=0x7f0e540190f0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 msgr2=0x7f0e60135250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.378+0000 7f0e52ffd640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60134e70 0x7f0e60135250 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e540ad000 tx=0x7f0e540190f0 comp rx=0 tx=0).stop 2024-09-17T15:28:35.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e687ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:28:35.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e687ad640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 0x7f0e6009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:35.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e687ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:35.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e60074be0 con 0x7f0e60135790 2024-09-17T15:28:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 0x7f0e6009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient: get_auth_request con 0x7f0e60135790 auth_method 0 2024-09-17T15:28:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:35.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:35.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:35.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:35.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:35.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 10 monclient(hunting): handle_auth_done global_id 14158 payload 824 2024-09-17T15:28:35.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.386+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e60135ba0 con 0x7f0e60135790 2024-09-17T15:28:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.390+0000 7f0e66549640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 msgr2=0x7f0e6009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.390+0000 7f0e66549640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 0x7f0e6009e5e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e5401c180 tx=0x7f0e540a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:35.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e540b0040 con 0x7f0e60135790 2024-09-17T15:28:35.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 10 monclient: dump: 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:35.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f0e54003d30 con 0x7f0e60135790 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 msgr2=0x7f0e6009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 0x7f0e6009e5e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e5401c180 tx=0x7f0e540a8000 comp rx=0 tx=0).stop 2024-09-17T15:28:35.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 msgr2=0x7f0e6009e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e60135790 0x7f0e6009e5e0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0e5401c180 tx=0x7f0e540a8000 comp rx=0 tx=0).stop 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 24, 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 23994, 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:28:35.397 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.398 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.399 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:28:35.400 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:28:35.401 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:28:35.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:28:35.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:28:35.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:28:35.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:28:35.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:35.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:28:35.410 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e4403bb40 msgr2=0x7f0e4403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e4403bb40 0x7f0e4403e000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e601d0e20 tx=0x7f0e5c005110 comp rx=0 tx=0).stop 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 10 monclient: shutdown 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 msgr2=0x7f0e601d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 --2- 172.21.15.86:0/772781293 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e6013d2f0 0x7f0e601d07e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e5800b490 tx=0x7f0e5800b950 comp rx=0 tx=0).stop 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 reap_dead start 2024-09-17T15:28:35.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e66d4a640 1 -- 172.21.15.86:0/772781293 reap_dead start 2024-09-17T15:28:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e52ffd640 1 -- 172.21.15.86:0/772781293 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4767+0+0 (unknown 0 0 0) 0x7f0e5804f3a0 con 0x7f0e6013d2f0 2024-09-17T15:28:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 shutdown_connections 2024-09-17T15:28:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 >> 172.21.15.86:0/772781293 conn(0x7f0e600aaae0 msgr2=0x7f0e6013f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 shutdown_connections 2024-09-17T15:28:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:35.394+0000 7f0e687ad640 1 -- 172.21.15.86:0/772781293 wait complete. 2024-09-17T15:28:35.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:28:35.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:28:35.592 INFO:tasks.rados.rados.0.smithi203.stdout:142: done (2 left) 2024-09-17T15:28:35.592 INFO:tasks.rados.rados.0.smithi203.stdout:143: done (1 left) 2024-09-17T15:28:35.592 INFO:tasks.rados.rados.0.smithi203.stdout:146: done (0 left) 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:147: read oid 24 snap -1 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:147: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:148: read oid 21 snap -1 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:148: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:149: write oid 23 current snap is 11 2024-09-17T15:28:35.593 INFO:tasks.rados.rados.0.smithi203.stdout:149: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-17T15:28:35.617 INFO:tasks.rados.rados.0.smithi203.stdout:149: writing smithi20328086-23 from 695823 to 1472867 tid 1 2024-09-17T15:28:35.635 INFO:tasks.rados.rados.0.smithi203.stdout:149: writing smithi20328086-23 from 1925015 to 2564697 tid 2 2024-09-17T15:28:35.648 INFO:tasks.rados.rados.0.smithi203.stdout:149: writing smithi20328086-23 from 3328058 to 3495823 tid 3 2024-09-17T15:28:35.648 INFO:tasks.rados.rados.0.smithi203.stdout:150: snap_remove snap 8 2024-09-17T15:28:36.795 INFO:tasks.rados.rados.0.smithi203.stdout:150: done (3 left) 2024-09-17T15:28:36.796 INFO:tasks.rados.rados.0.smithi203.stdout:151: read oid 25 snap -1 2024-09-17T15:28:36.796 INFO:tasks.rados.rados.0.smithi203.stdout:151: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:28:36.796 INFO:tasks.rados.rados.0.smithi203.stdout:152: copy_from oid 11 from oid 18 current snap is 11 2024-09-17T15:28:36.796 INFO:tasks.rados.rados.0.smithi203.stdout:148: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-17T15:28:36.864 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:36.915 INFO:tasks.rados.rados.0.smithi203.stdout:147: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:28:36.915 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:36.960 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 373735 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: finishing write tid 1 to smithi20328086-23 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: finishing write tid 2 to smithi20328086-23 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: finishing write tid 3 to smithi20328086-23 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: finishing write tid 4 to smithi20328086-23 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: finishing write tid 5 to smithi20328086-23 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: oid 23 updating version 0 to 26 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: oid 23 version 26 is already newer than 24 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: oid 23 version 26 is already newer than 25 2024-09-17T15:28:36.984 INFO:tasks.rados.rados.0.smithi203.stdout:149: oid 23 updating version 26 to 27 2024-09-17T15:28:36.985 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 27 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-17T15:28:36.985 INFO:tasks.rados.rados.0.smithi203.stdout:149: left oid 23 (ObjNum 89 snap 11 seq_num 89) 2024-09-17T15:28:36.985 INFO:tasks.rados.rados.0.smithi203.stdout:151: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:28:37.000 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:37.047 INFO:tasks.rados.rados.0.smithi203.stdout:152: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:152: finishing copy_from to smithi20328086-11 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 15 (ObjNum 66 snap 2 seq_num 66) dirty exists 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:147: done (4 left) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:148: done (3 left) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:149: done (2 left) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:151: done (1 left) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:152: done (0 left) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:153: read oid 45 snap -1 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:153: expect (ObjNum 83 snap 6 seq_num 83) 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:154: copy_from oid 19 from oid 30 current snap is 11 2024-09-17T15:28:37.048 INFO:tasks.rados.rados.0.smithi203.stdout:155: delete oid 37 current snap is 11 2024-09-17T15:28:37.054 INFO:tasks.rados.rados.0.smithi203.stdout:155: done (2 left) 2024-09-17T15:28:37.054 INFO:tasks.rados.rados.0.smithi203.stdout:156: rollback oid 40 current snap is 11 2024-09-17T15:28:37.054 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 10 2024-09-17T15:28:37.054 INFO:tasks.rados.rados.0.smithi203.stdout:154: finishing copy_from to smithi20328086-19 2024-09-17T15:28:37.054 INFO:tasks.rados.rados.0.smithi203.stdout:154: got expected ENOENT (src dne) 2024-09-17T15:28:37.055 INFO:tasks.rados.rados.0.smithi203.stdout:157: snap_remove snap 9 2024-09-17T15:28:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:157: done (3 left) 2024-09-17T15:28:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:158: snap_create 2024-09-17T15:28:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:154: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:28:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:153: expect (ObjNum 83 snap 6 seq_num 83) 2024-09-17T15:28:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:37.541 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1836172 2024-09-17T15:28:37.572 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:37.623 INFO:tasks.rados.rados.0.smithi203.stdout:156: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:28:37.623 INFO:tasks.rados.rados.0.smithi203.stdout:156: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:28:37.623 INFO:tasks.rados.rados.0.smithi203.stdout:156: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:28:37.623 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 31 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:153: done (3 left) 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:154: done (2 left) 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:156: done (1 left) 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:158: done (0 left) 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:159: read oid 18 snap -1 2024-09-17T15:28:38.480 INFO:tasks.rados.rados.0.smithi203.stdout:159: expect (ObjNum 66 snap 2 seq_num 66) 2024-09-17T15:28:38.481 INFO:tasks.rados.rados.0.smithi203.stdout:160: snap_create 2024-09-17T15:28:38.495 INFO:tasks.rados.rados.0.smithi203.stdout:159: expect (ObjNum 66 snap 2 seq_num 66) 2024-09-17T15:28:38.541 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:39.481 INFO:tasks.rados.rados.0.smithi203.stdout:159: done (1 left) 2024-09-17T15:28:39.481 INFO:tasks.rados.rados.0.smithi203.stdout:160: done (0 left) 2024-09-17T15:28:39.481 INFO:tasks.rados.rados.0.smithi203.stdout:161: write (excl) oid 3 current snap is 13 2024-09-17T15:28:39.481 INFO:tasks.rados.rados.0.smithi203.stdout:161: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-17T15:28:39.501 INFO:tasks.rados.rados.0.smithi203.stdout:161: writing smithi20328086-3 from 729437 to 1256979 tid 1 2024-09-17T15:28:39.522 INFO:tasks.rados.rados.0.smithi203.stdout:161: writing smithi20328086-3 from 1925626 to 2568977 tid 2 2024-09-17T15:28:39.537 INFO:tasks.rados.rados.0.smithi203.stdout:161: writing smithi20328086-3 from 3266551 to 3529437 tid 3 2024-09-17T15:28:39.537 INFO:tasks.rados.rados.0.smithi203.stdout:162: delete oid 43 current snap is 13 2024-09-17T15:28:39.537 INFO:tasks.rados.rados.0.smithi203.stdout:161: finishing write tid 1 to smithi20328086-3 2024-09-17T15:28:39.537 INFO:tasks.rados.rados.0.smithi203.stdout:161: finishing write tid 2 to smithi20328086-3 2024-09-17T15:28:39.542 INFO:tasks.rados.rados.0.smithi203.stdout:161: finishing write tid 3 to smithi20328086-3 2024-09-17T15:28:39.542 INFO:tasks.rados.rados.0.smithi203.stdout:161: finishing write tid 4 to smithi20328086-3 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: finishing write tid 5 to smithi20328086-3 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: oid 3 updating version 0 to 29 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: oid 3 version 29 is already newer than 27 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: oid 3 updating version 29 to 30 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: oid 3 version 30 is already newer than 28 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 30 (ObjNum 91 snap 13 seq_num 91) dirty exists 2024-09-17T15:28:39.543 INFO:tasks.rados.rados.0.smithi203.stdout:161: left oid 3 (ObjNum 91 snap 13 seq_num 91) 2024-09-17T15:28:39.544 INFO:tasks.rados.rados.0.smithi203.stdout:161: done (1 left) 2024-09-17T15:28:39.544 INFO:tasks.rados.rados.0.smithi203.stdout:162: done (0 left) 2024-09-17T15:28:39.544 INFO:tasks.rados.rados.0.smithi203.stdout:163: snap_remove snap 10 2024-09-17T15:28:40.490 INFO:tasks.rados.rados.0.smithi203.stdout:163: done (0 left) 2024-09-17T15:28:40.490 INFO:tasks.rados.rados.0.smithi203.stdout:164: write oid 42 current snap is 13 2024-09-17T15:28:40.490 INFO:tasks.rados.rados.0.smithi203.stdout:164: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-17T15:28:40.512 INFO:tasks.rados.rados.0.smithi203.stdout:164: writing smithi20328086-42 from 763051 to 1557444 tid 1 2024-09-17T15:28:40.529 INFO:tasks.rados.rados.0.smithi203.stdout:164: writing smithi20328086-42 from 2158943 to 2805963 tid 2 2024-09-17T15:28:40.540 INFO:tasks.rados.rados.0.smithi203.stdout:164: writing smithi20328086-42 from 3437750 to 3563051 tid 3 2024-09-17T15:28:40.540 INFO:tasks.rados.rados.0.smithi203.stdout:165: delete oid 9 current snap is 13 2024-09-17T15:28:40.540 INFO:tasks.rados.rados.0.smithi203.stdout:164: finishing write tid 1 to smithi20328086-42 2024-09-17T15:28:40.540 INFO:tasks.rados.rados.0.smithi203.stdout:164: finishing write tid 2 to smithi20328086-42 2024-09-17T15:28:40.546 INFO:tasks.rados.rados.0.smithi203.stdout:165: done (1 left) 2024-09-17T15:28:40.546 INFO:tasks.rados.rados.0.smithi203.stdout:166: copy_from oid 15 from oid 30 current snap is 13 2024-09-17T15:28:40.546 INFO:tasks.rados.rados.0.smithi203.stdout:167: write (excl) oid 2 current snap is 13 2024-09-17T15:28:40.546 INFO:tasks.rados.rados.0.smithi203.stdout:167: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-17T15:28:40.562 INFO:tasks.rados.rados.0.smithi203.stdout:167: writing smithi20328086-2 from 796665 to 1341556 tid 1 2024-09-17T15:28:40.575 INFO:tasks.rados.rados.0.smithi203.stdout:167: writing smithi20328086-2 from 1759554 to 2410243 tid 2 2024-09-17T15:28:40.587 INFO:tasks.rados.rados.0.smithi203.stdout:167: writing smithi20328086-2 from 2976243 to 3596665 tid 3 2024-09-17T15:28:40.587 INFO:tasks.rados.rados.0.smithi203.stdout:168: write (excl) oid 47 current snap is 13 2024-09-17T15:28:40.587 INFO:tasks.rados.rados.0.smithi203.stdout:168: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-17T15:28:40.596 INFO:tasks.rados.rados.0.smithi203.stdout:168: writing smithi20328086-47 from 413472 to 1033612 tid 1 2024-09-17T15:28:40.606 INFO:tasks.rados.rados.0.smithi203.stdout:168: writing smithi20328086-47 from 1818036 to 2328736 tid 2 2024-09-17T15:28:40.615 INFO:tasks.rados.rados.0.smithi203.stdout:168: writing smithi20328086-47 from 3120019 to 3613472 tid 3 2024-09-17T15:28:40.616 INFO:tasks.rados.rados.0.smithi203.stdout:169: snap_create 2024-09-17T15:28:40.616 INFO:tasks.rados.rados.0.smithi203.stdout:164: finishing write tid 3 to smithi20328086-42 2024-09-17T15:28:40.616 INFO:tasks.rados.rados.0.smithi203.stdout:166: finishing copy_from to smithi20328086-15 2024-09-17T15:28:40.616 INFO:tasks.rados.rados.0.smithi203.stdout:166: got expected ENOENT (src dne) 2024-09-17T15:28:40.617 INFO:tasks.rados.rados.0.smithi203.stdout:166: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:28:40.617 INFO:tasks.rados.rados.0.smithi203.stdout:164: finishing write tid 4 to smithi20328086-42 2024-09-17T15:28:40.617 INFO:tasks.rados.rados.0.smithi203.stdout:164: finishing write tid 5 to smithi20328086-42 2024-09-17T15:28:40.617 INFO:tasks.rados.rados.0.smithi203.stdout:164: oid 42 updating version 0 to 35 2024-09-17T15:28:40.617 INFO:tasks.rados.rados.0.smithi203.stdout:164: oid 42 version 35 is already newer than 34 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:164: oid 42 version 35 is already newer than 32 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:164: oid 42 version 35 is already newer than 33 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 35 (ObjNum 93 snap 13 seq_num 93) dirty exists 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:164: left oid 42 (ObjNum 93 snap 13 seq_num 93) 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:167: finishing write tid 1 to smithi20328086-2 2024-09-17T15:28:40.618 INFO:tasks.rados.rados.0.smithi203.stdout:167: finishing write tid 2 to smithi20328086-2 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: finishing write tid 3 to smithi20328086-2 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: finishing write tid 4 to smithi20328086-2 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: finishing write tid 5 to smithi20328086-2 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: oid 2 updating version 0 to 19 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: oid 2 updating version 19 to 20 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: oid 2 updating version 20 to 21 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: oid 2 version 21 is already newer than 18 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 21 (ObjNum 95 snap 13 seq_num 95) dirty exists 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:167: left oid 2 (ObjNum 95 snap 13 seq_num 95) 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:168: finishing write tid 1 to smithi20328086-47 2024-09-17T15:28:40.619 INFO:tasks.rados.rados.0.smithi203.stdout:168: finishing write tid 2 to smithi20328086-47 2024-09-17T15:28:40.620 INFO:tasks.rados.rados.0.smithi203.stdout:168: finishing write tid 3 to smithi20328086-47 2024-09-17T15:28:40.621 INFO:tasks.rados.rados.0.smithi203.stdout:168: finishing write tid 4 to smithi20328086-47 2024-09-17T15:28:40.621 INFO:tasks.rados.rados.0.smithi203.stdout:168: finishing write tid 5 to smithi20328086-47 2024-09-17T15:28:40.621 INFO:tasks.rados.rados.0.smithi203.stdout:168: oid 47 updating version 0 to 21 2024-09-17T15:28:40.621 INFO:tasks.rados.rados.0.smithi203.stdout:168: oid 47 version 21 is already newer than 19 2024-09-17T15:28:40.621 INFO:tasks.rados.rados.0.smithi203.stdout:168: oid 47 version 21 is already newer than 18 2024-09-17T15:28:40.622 INFO:tasks.rados.rados.0.smithi203.stdout:168: oid 47 version 21 is already newer than 20 2024-09-17T15:28:40.622 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 21 (ObjNum 96 snap 13 seq_num 96) dirty exists 2024-09-17T15:28:40.622 INFO:tasks.rados.rados.0.smithi203.stdout:168: left oid 47 (ObjNum 96 snap 13 seq_num 96) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:164: done (4 left) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:166: done (3 left) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:167: done (2 left) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:168: done (1 left) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:169: done (0 left) 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:170: copy_from oid 44 from oid 36 current snap is 14 2024-09-17T15:28:41.491 INFO:tasks.rados.rados.0.smithi203.stdout:171: snap_create 2024-09-17T15:28:41.517 INFO:tasks.rados.rados.0.smithi203.stdout:170: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:28:41.520 INFO:tasks.rados.rados.0.smithi203.stdout:170: finishing copy_from to smithi20328086-44 2024-09-17T15:28:41.520 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:28:42.390 INFO:tasks.rados.rados.0.smithi203.stdout:170: done (1 left) 2024-09-17T15:28:42.391 INFO:tasks.rados.rados.0.smithi203.stdout:171: done (0 left) 2024-09-17T15:28:42.391 INFO:tasks.rados.rados.0.smithi203.stdout:172: copy_from oid 26 from oid 23 current snap is 15 2024-09-17T15:28:42.391 INFO:tasks.rados.rados.0.smithi203.stdout:173: copy_from oid 50 from oid 1 current snap is 15 2024-09-17T15:28:42.391 INFO:tasks.rados.rados.0.smithi203.stdout:174: snap_remove snap 2 2024-09-17T15:28:43.392 INFO:tasks.rados.rados.0.smithi203.stdout:174: done (2 left) 2024-09-17T15:28:43.392 INFO:tasks.rados.rados.0.smithi203.stdout:175: snap_create 2024-09-17T15:28:43.392 INFO:tasks.rados.rados.0.smithi203.stdout:172: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:28:43.392 INFO:tasks.rados.rados.0.smithi203.stdout:172: finishing copy_from to smithi20328086-26 2024-09-17T15:28:43.393 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 27 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-17T15:28:43.393 INFO:tasks.rados.rados.0.smithi203.stdout:173: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:28:43.393 INFO:tasks.rados.rados.0.smithi203.stdout:173: finishing copy_from to smithi20328086-50 2024-09-17T15:28:43.393 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 20 (ObjNum 79 snap 5 seq_num 79) dirty exists 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:172: done (2 left) 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:173: done (1 left) 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:175: done (0 left) 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:176: copy_from oid 2 from oid 14 current snap is 16 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:177: write (excl) oid 26 current snap is 16 2024-09-17T15:28:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:177: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-17T15:28:44.412 INFO:tasks.rados.rados.0.smithi203.stdout:177: writing smithi20328086-26 from 430279 to 1125668 tid 1 2024-09-17T15:28:44.425 INFO:tasks.rados.rados.0.smithi203.stdout:177: writing smithi20328086-26 from 1876518 to 2530876 tid 2 2024-09-17T15:28:44.434 INFO:tasks.rados.rados.0.smithi203.stdout:177: writing smithi20328086-26 from 3031089 to 3567886 tid 3 2024-09-17T15:28:44.434 INFO:tasks.rados.rados.0.smithi203.stdout:178: snap_remove snap 13 2024-09-17T15:28:45.407 INFO:tasks.rados.rados.0.smithi203.stdout:178: done (2 left) 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:179: read oid 35 snap 15 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:179: expect (ObjNum 65 snap 2 seq_num 65) 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:180: rollback oid 25 current snap is 16 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 15 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:176: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:176: finishing copy_from to smithi20328086-2 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:177: finishing write tid 1 to smithi20328086-26 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:177: finishing write tid 2 to smithi20328086-26 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:177: finishing write tid 3 to smithi20328086-26 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:177: finishing write tid 4 to smithi20328086-26 2024-09-17T15:28:45.408 INFO:tasks.rados.rados.0.smithi203.stdout:177: finishing write tid 5 to smithi20328086-26 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: oid 26 updating version 0 to 32 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: oid 26 updating version 32 to 33 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: oid 26 updating version 33 to 35 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: oid 26 version 35 is already newer than 34 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 35 (ObjNum 97 snap 16 seq_num 97) dirty exists 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: left oid 26 (ObjNum 97 snap 16 seq_num 97) 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:176: done (3 left) 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:177: done (2 left) 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:181: copy_from oid 14 from oid 21 current snap is 16 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:182: read oid 16 snap -1 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:182: expect deleted 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:183: write (excl) oid 11 current snap is 16 2024-09-17T15:28:45.409 INFO:tasks.rados.rados.0.smithi203.stdout:183: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-17T15:28:45.421 INFO:tasks.rados.rados.0.smithi203.stdout:183: writing smithi20328086-11 from 447086 to 1217724 tid 1 2024-09-17T15:28:45.437 INFO:tasks.rados.rados.0.smithi203.stdout:183: writing smithi20328086-11 from 1935000 to 2733016 tid 2 2024-09-17T15:28:45.446 INFO:tasks.rados.rados.0.smithi203.stdout:183: writing smithi20328086-11 from 3458512 to 3647086 tid 3 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:184: copy_from oid 18 from oid 33 current snap is 16 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:180: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:180: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:180: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 32 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-17T15:28:45.447 INFO:tasks.rados.rados.0.smithi203.stdout:179: expect (ObjNum 65 snap 2 seq_num 65) 2024-09-17T15:28:45.590 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:45.637 INFO:tasks.rados.rados.0.smithi203.stdout:183: finishing write tid 1 to smithi20328086-11 2024-09-17T15:28:45.637 INFO:tasks.rados.rados.0.smithi203.stdout:181: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:28:45.637 INFO:tasks.rados.rados.0.smithi203.stdout:181: finishing copy_from to smithi20328086-14 2024-09-17T15:28:45.637 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-17T15:28:45.637 INFO:tasks.rados.rados.0.smithi203.stdout:183: finishing write tid 2 to smithi20328086-11 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: finishing write tid 3 to smithi20328086-11 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: finishing write tid 4 to smithi20328086-11 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: finishing write tid 5 to smithi20328086-11 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: oid 11 updating version 0 to 53 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: oid 11 version 53 is already newer than 52 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: oid 11 version 53 is already newer than 51 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: oid 11 updating version 53 to 54 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 54 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:183: left oid 11 (ObjNum 98 snap 16 seq_num 98) 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:184: finishing copy_from to smithi20328086-18 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:184: got expected ENOENT (src dne) 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:184: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:28:45.638 INFO:tasks.rados.rados.0.smithi203.stdout:179: done (5 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:180: done (4 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:181: done (3 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:182: done (2 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:183: done (1 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:184: done (0 left) 2024-09-17T15:28:45.639 INFO:tasks.rados.rados.0.smithi203.stdout:185: snap_create 2024-09-17T15:28:46.420 INFO:tasks.rados.rados.0.smithi203.stdout:185: done (0 left) 2024-09-17T15:28:46.421 INFO:tasks.rados.rados.0.smithi203.stdout:186: copy_from oid 37 from oid 39 current snap is 17 2024-09-17T15:28:46.421 INFO:tasks.rados.rados.0.smithi203.stdout:187: write oid 3 current snap is 17 2024-09-17T15:28:46.421 INFO:tasks.rados.rados.0.smithi203.stdout:187: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-17T15:28:46.437 INFO:tasks.rados.rados.0.smithi203.stdout:187: writing smithi20328086-3 from 463893 to 1026133 tid 1 2024-09-17T15:28:46.457 INFO:tasks.rados.rados.0.smithi203.stdout:187: writing smithi20328086-3 from 1593482 to 2251509 tid 2 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:187: writing smithi20328086-3 from 2685935 to 3245276 tid 3 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:188: delete oid 50 current snap is 17 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:186: finishing copy_from to smithi20328086-37 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:186: got expected ENOENT (src dne) 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:186: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:187: finishing write tid 1 to smithi20328086-3 2024-09-17T15:28:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:187: finishing write tid 2 to smithi20328086-3 2024-09-17T15:28:46.479 INFO:tasks.rados.rados.0.smithi203.stdout:187: finishing write tid 3 to smithi20328086-3 2024-09-17T15:28:46.480 INFO:tasks.rados.rados.0.smithi203.stdout:187: finishing write tid 4 to smithi20328086-3 2024-09-17T15:28:46.480 INFO:tasks.rados.rados.0.smithi203.stdout:187: finishing write tid 5 to smithi20328086-3 2024-09-17T15:28:46.480 INFO:tasks.rados.rados.0.smithi203.stdout:187: oid 3 updating version 0 to 44 2024-09-17T15:28:46.481 INFO:tasks.rados.rados.0.smithi203.stdout:187: oid 3 version 44 is already newer than 43 2024-09-17T15:28:46.481 INFO:tasks.rados.rados.0.smithi203.stdout:187: oid 3 version 44 is already newer than 42 2024-09-17T15:28:46.481 INFO:tasks.rados.rados.0.smithi203.stdout:187: oid 3 version 44 is already newer than 41 2024-09-17T15:28:46.481 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 44 (ObjNum 99 snap 17 seq_num 99) dirty exists 2024-09-17T15:28:46.481 INFO:tasks.rados.rados.0.smithi203.stdout:187: left oid 3 (ObjNum 99 snap 17 seq_num 99) 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:186: done (2 left) 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:187: done (1 left) 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:188: done (0 left) 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:189: read oid 14 snap 15 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:189: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-17T15:28:46.486 INFO:tasks.rados.rados.0.smithi203.stdout:190: snap_remove snap 14 2024-09-17T15:28:47.391 INFO:tasks.rados.rados.0.smithi203.stdout:190: done (1 left) 2024-09-17T15:28:47.391 INFO:tasks.rados.rados.0.smithi203.stdout:191: snap_create 2024-09-17T15:28:47.391 INFO:tasks.rados.rados.0.smithi203.stdout:189: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-17T15:28:47.457 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:48.392 INFO:tasks.rados.rados.0.smithi203.stdout:189: done (1 left) 2024-09-17T15:28:48.392 INFO:tasks.rados.rados.0.smithi203.stdout:191: done (0 left) 2024-09-17T15:28:48.392 INFO:tasks.rados.rados.0.smithi203.stdout:192: copy_from oid 6 from oid 49 current snap is 18 2024-09-17T15:28:48.392 INFO:tasks.rados.rados.0.smithi203.stdout:193: copy_from oid 33 from oid 24 current snap is 18 2024-09-17T15:28:48.392 INFO:tasks.rados.rados.0.smithi203.stdout:194: delete oid 3 current snap is 18 2024-09-17T15:28:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:194: done (2 left) 2024-09-17T15:28:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:195: delete oid 30 current snap is 18 2024-09-17T15:28:48.408 INFO:tasks.rados.rados.0.smithi203.stdout:195: done (2 left) 2024-09-17T15:28:48.408 INFO:tasks.rados.rados.0.smithi203.stdout:196: snap_remove snap 11 2024-09-17T15:28:49.395 INFO:tasks.rados.rados.0.smithi203.stdout:196: done (2 left) 2024-09-17T15:28:49.395 INFO:tasks.rados.rados.0.smithi203.stdout:197: copy_from oid 21 from oid 7 current snap is 18 2024-09-17T15:28:49.395 INFO:tasks.rados.rados.0.smithi203.stdout:193: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:192: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:192: finishing copy_from to smithi20328086-6 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:193: finishing copy_from to smithi20328086-33 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:192: done (2 left) 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:193: done (1 left) 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:198: copy_from oid 44 from oid 50 current snap is 18 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:199: write (excl) oid 29 current snap is 18 2024-09-17T15:28:49.396 INFO:tasks.rados.rados.0.smithi203.stdout:199: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-17T15:28:49.401 INFO:tasks.rados.rados.0.smithi203.stdout:199: writing smithi20328086-29 from 531121 to 994357 tid 1 2024-09-17T15:28:49.409 INFO:tasks.rados.rados.0.smithi203.stdout:199: writing smithi20328086-29 from 1711057 to 2376422 tid 2 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:199: writing smithi20328086-29 from 3079274 to 3731121 tid 3 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:200: rollback oid 32 current snap is 18 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 16 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:198: finishing copy_from to smithi20328086-44 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:198: got expected ENOENT (src dne) 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:198: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:199: finishing write tid 1 to smithi20328086-29 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:198: done (3 left) 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:201: read oid 14 snap 16 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:201: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-17T15:28:49.418 INFO:tasks.rados.rados.0.smithi203.stdout:202: read oid 40 snap -1 2024-09-17T15:28:49.419 INFO:tasks.rados.rados.0.smithi203.stdout:202: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:28:49.419 INFO:tasks.rados.rados.0.smithi203.stdout:203: delete oid 16 current snap is 18 2024-09-17T15:28:49.419 INFO:tasks.rados.rados.0.smithi203.stdout:200: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:28:49.419 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 3606057640) dirty dne 2024-09-17T15:28:49.419 INFO:tasks.rados.rados.0.smithi203.stdout:199: finishing write tid 2 to smithi20328086-29 2024-09-17T15:28:49.423 INFO:tasks.rados.rados.0.smithi203.stdout:200: done (5 left) 2024-09-17T15:28:49.423 INFO:tasks.rados.rados.0.smithi203.stdout:203: done (4 left) 2024-09-17T15:28:49.423 INFO:tasks.rados.rados.0.smithi203.stdout:204: write oid 19 current snap is 18 2024-09-17T15:28:49.423 INFO:tasks.rados.rados.0.smithi203.stdout:204: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-17T15:28:49.430 INFO:tasks.rados.rados.0.smithi203.stdout:204: writing smithi20328086-19 from 564735 to 1178469 tid 1 2024-09-17T15:28:49.438 INFO:tasks.rados.rados.0.smithi203.stdout:204: writing smithi20328086-19 from 1828021 to 2497055 tid 2 2024-09-17T15:28:49.445 INFO:tasks.rados.rados.0.smithi203.stdout:204: writing smithi20328086-19 from 3134120 to 3764735 tid 3 2024-09-17T15:28:49.445 INFO:tasks.rados.rados.0.smithi203.stdout:205: write oid 45 current snap is 18 2024-09-17T15:28:49.445 INFO:tasks.rados.rados.0.smithi203.stdout:201: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-17T15:28:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1348463 2024-09-17T15:28:49.518 INFO:tasks.rados.rados.0.smithi203.stdout:199: finishing write tid 3 to smithi20328086-29 2024-09-17T15:28:49.518 INFO:tasks.rados.rados.0.smithi203.stdout:199: finishing write tid 4 to smithi20328086-29 2024-09-17T15:28:49.518 INFO:tasks.rados.rados.0.smithi203.stdout:199: finishing write tid 5 to smithi20328086-29 2024-09-17T15:28:49.518 INFO:tasks.rados.rados.0.smithi203.stdout:199: oid 29 updating version 0 to 35 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:199: oid 29 updating version 35 to 41 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:199: oid 29 version 41 is already newer than 40 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:199: oid 29 updating version 41 to 42 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 42 (ObjNum 103 snap 18 seq_num 103) dirty exists 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:199: left oid 29 (ObjNum 103 snap 18 seq_num 103) 2024-09-17T15:28:49.519 INFO:tasks.rados.rados.0.smithi203.stdout:202: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:28:49.536 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1323524 2024-09-17T15:28:49.576 INFO:tasks.rados.rados.0.smithi203.stdout:197: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:197: finishing copy_from to smithi20328086-21 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: finishing write tid 1 to smithi20328086-19 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: finishing write tid 2 to smithi20328086-19 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: finishing write tid 3 to smithi20328086-19 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: finishing write tid 4 to smithi20328086-19 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: finishing write tid 5 to smithi20328086-19 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: oid 19 updating version 0 to 41 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: oid 19 updating version 41 to 44 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: oid 19 version 44 is already newer than 42 2024-09-17T15:28:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:204: oid 19 version 44 is already newer than 43 2024-09-17T15:28:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 44 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-17T15:28:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:204: left oid 19 (ObjNum 105 snap 18 seq_num 105) 2024-09-17T15:28:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:205: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-17T15:28:49.588 INFO:tasks.rados.rados.0.smithi203.stdout:205: writing smithi20328086-45 from 581542 to 1270525 tid 1 2024-09-17T15:28:49.597 INFO:tasks.rados.rados.0.smithi203.stdout:205: writing smithi20328086-45 from 1886503 to 2299195 tid 2 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:205: writing smithi20328086-45 from 2761543 to 3432494 tid 3 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:197: done (5 left) 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:199: done (4 left) 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:201: done (3 left) 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:202: done (2 left) 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:204: done (1 left) 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:206: write (excl) oid 35 current snap is 18 2024-09-17T15:28:49.609 INFO:tasks.rados.rados.0.smithi203.stdout:206: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-17T15:28:49.618 INFO:tasks.rados.rados.0.smithi203.stdout:206: writing smithi20328086-35 from 598349 to 1078934 tid 1 2024-09-17T15:28:49.629 INFO:tasks.rados.rados.0.smithi203.stdout:206: writing smithi20328086-35 from 1544985 to 2217688 tid 2 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:206: writing smithi20328086-35 from 2905319 to 3387542 tid 3 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:207: rollback oid 32 current snap is 18 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 15 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:205: finishing write tid 1 to smithi20328086-45 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:205: finishing write tid 2 to smithi20328086-45 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:205: finishing write tid 3 to smithi20328086-45 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:205: finishing write tid 4 to smithi20328086-45 2024-09-17T15:28:49.639 INFO:tasks.rados.rados.0.smithi203.stdout:205: finishing write tid 5 to smithi20328086-45 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: oid 45 updating version 0 to 18 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: oid 45 updating version 18 to 19 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: oid 45 version 19 is already newer than 17 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: oid 45 updating version 19 to 20 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 20 (ObjNum 106 snap 18 seq_num 106) dirty exists 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: left oid 45 (ObjNum 106 snap 18 seq_num 106) 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:206: finishing write tid 1 to smithi20328086-35 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:206: finishing write tid 2 to smithi20328086-35 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:205: done (2 left) 2024-09-17T15:28:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:208: snap_remove snap 12 2024-09-17T15:28:50.409 INFO:tasks.rados.rados.0.smithi203.stdout:208: done (2 left) 2024-09-17T15:28:50.409 INFO:tasks.rados.rados.0.smithi203.stdout:209: write (excl) oid 45 current snap is 18 2024-09-17T15:28:50.409 INFO:tasks.rados.rados.0.smithi203.stdout:209: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-17T15:28:50.419 INFO:tasks.rados.rados.0.smithi203.stdout:209: writing smithi20328086-45 from 615156 to 1170990 tid 1 2024-09-17T15:28:50.427 INFO:tasks.rados.rados.0.smithi203.stdout:209: writing smithi20328086-45 from 1603467 to 2019828 tid 2 2024-09-17T15:28:50.439 INFO:tasks.rados.rados.0.smithi203.stdout:209: writing smithi20328086-45 from 2816389 to 3509884 tid 3 2024-09-17T15:28:50.439 INFO:tasks.rados.rados.0.smithi203.stdout:210: rollback oid 5 current snap is 18 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 15 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:207: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 3606057640) dirty dne 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: finishing write tid 3 to smithi20328086-35 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: finishing write tid 4 to smithi20328086-35 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: finishing write tid 5 to smithi20328086-35 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: oid 35 updating version 0 to 29 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: oid 35 updating version 29 to 31 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: oid 35 version 31 is already newer than 30 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: oid 35 updating version 31 to 32 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 32 (ObjNum 107 snap 18 seq_num 107) dirty exists 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:206: left oid 35 (ObjNum 107 snap 18 seq_num 107) 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:209: finishing write tid 1 to smithi20328086-45 2024-09-17T15:28:50.440 INFO:tasks.rados.rados.0.smithi203.stdout:209: finishing write tid 2 to smithi20328086-45 2024-09-17T15:28:50.441 INFO:tasks.rados.rados.0.smithi203.stdout:206: done (3 left) 2024-09-17T15:28:50.441 INFO:tasks.rados.rados.0.smithi203.stdout:207: done (2 left) 2024-09-17T15:28:50.441 INFO:tasks.rados.rados.0.smithi203.stdout:211: read oid 11 snap -1 2024-09-17T15:28:50.441 INFO:tasks.rados.rados.0.smithi203.stdout:211: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-17T15:28:50.441 INFO:tasks.rados.rados.0.smithi203.stdout:212: snap_create 2024-09-17T15:28:50.445 INFO:tasks.rados.rados.0.smithi203.stdout:210: finishing rollback tid 0 to smithi20328086-5 2024-09-17T15:28:50.448 INFO:tasks.rados.rados.0.smithi203.stdout:210: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:28:50.448 INFO:tasks.rados.rados.0.smithi203.stdout:209: finishing write tid 3 to smithi20328086-45 2024-09-17T15:28:50.448 INFO:tasks.rados.rados.0.smithi203.stdout:210: finishing rollback tid 2 to smithi20328086-5 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 43 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: finishing write tid 4 to smithi20328086-45 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: finishing write tid 5 to smithi20328086-45 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: oid 45 updating version 0 to 26 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: oid 45 version 26 is already newer than 24 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: oid 45 version 26 is already newer than 23 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:209: oid 45 version 26 is already newer than 25 2024-09-17T15:28:50.449 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 26 (ObjNum 108 snap 18 seq_num 108) dirty exists 2024-09-17T15:28:50.450 INFO:tasks.rados.rados.0.smithi203.stdout:209: left oid 45 (ObjNum 108 snap 18 seq_num 108) 2024-09-17T15:28:50.457 INFO:tasks.rados.rados.0.smithi203.stdout:211: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-17T15:28:50.457 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:50.597 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:51.409 INFO:tasks.rados.rados.0.smithi203.stdout:209: done (3 left) 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:210: done (2 left) 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:211: done (1 left) 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:212: done (0 left) 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:213: rollback oid 33 current snap is 19 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 18 2024-09-17T15:28:51.410 INFO:tasks.rados.rados.0.smithi203.stdout:214: delete oid 48 current snap is 19 2024-09-17T15:28:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:214: done (1 left) 2024-09-17T15:28:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:215: read oid 44 snap -1 2024-09-17T15:28:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:215: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:28:51.421 INFO:tasks.rados.rados.0.smithi203.stdout:216: copy_from oid 43 from oid 5 current snap is 19 2024-09-17T15:28:51.421 INFO:tasks.rados.rados.0.smithi203.stdout:217: write (excl) oid 50 current snap is 19 2024-09-17T15:28:51.421 INFO:tasks.rados.rados.0.smithi203.stdout:217: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-17T15:28:51.434 INFO:tasks.rados.rados.0.smithi203.stdout:217: writing smithi20328086-50 from 648770 to 1355102 tid 1 2024-09-17T15:28:51.445 INFO:tasks.rados.rados.0.smithi203.stdout:217: writing smithi20328086-50 from 2120431 to 2540461 tid 2 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:217: writing smithi20328086-50 from 3271235 to 3703627 tid 3 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:218: snap_create 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:213: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:213: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:213: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:28:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:215: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:28:51.457 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:28:51.483 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 760765 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:217: finishing write tid 1 to smithi20328086-50 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:216: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:217: finishing write tid 2 to smithi20328086-50 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:216: finishing copy_from to smithi20328086-43 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 43 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:217: finishing write tid 3 to smithi20328086-50 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:217: finishing write tid 4 to smithi20328086-50 2024-09-17T15:28:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:217: finishing write tid 5 to smithi20328086-50 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:217: oid 50 updating version 0 to 40 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:217: oid 50 updating version 40 to 43 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:217: oid 50 version 43 is already newer than 41 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:217: oid 50 version 43 is already newer than 42 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 43 (ObjNum 110 snap 19 seq_num 110) dirty exists 2024-09-17T15:28:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:217: left oid 50 (ObjNum 110 snap 19 seq_num 110) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:213: done (4 left) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:215: done (3 left) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:216: done (2 left) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:217: done (1 left) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:218: done (0 left) 2024-09-17T15:28:52.394 INFO:tasks.rados.rados.0.smithi203.stdout:219: delete oid 40 current snap is 20 2024-09-17T15:28:52.400 INFO:tasks.rados.rados.0.smithi203.stdout:219: done (0 left) 2024-09-17T15:28:52.400 INFO:tasks.rados.rados.0.smithi203.stdout:220: read oid 32 snap 19 2024-09-17T15:28:52.400 INFO:tasks.rados.rados.0.smithi203.stdout:220: expect deleted 2024-09-17T15:28:52.400 INFO:tasks.rados.rados.0.smithi203.stdout:221: write (excl) oid 17 current snap is 20 2024-09-17T15:28:52.400 INFO:tasks.rados.rados.0.smithi203.stdout:221: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-17T15:28:52.416 INFO:tasks.rados.rados.0.smithi203.stdout:221: writing smithi20328086-17 from 682384 to 1139214 tid 1 2024-09-17T15:28:52.430 INFO:tasks.rados.rados.0.smithi203.stdout:221: writing smithi20328086-17 from 1721042 to 2144741 tid 2 2024-09-17T15:28:52.446 INFO:tasks.rados.rados.0.smithi203.stdout:221: writing smithi20328086-17 from 2809728 to 3264664 tid 3 2024-09-17T15:28:52.454 INFO:tasks.rados.rados.0.smithi203.stdout:221: writing smithi20328086-17 from 3731357 to 3882384 tid 4 2024-09-17T15:28:52.454 INFO:tasks.rados.rados.0.smithi203.stdout:222: write (excl) oid 49 current snap is 20 2024-09-17T15:28:52.454 INFO:tasks.rados.rados.0.smithi203.stdout:222: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-17T15:28:52.467 INFO:tasks.rados.rados.0.smithi203.stdout:222: writing smithi20328086-49 from 699191 to 1231270 tid 1 2024-09-17T15:28:52.477 INFO:tasks.rados.rados.0.smithi203.stdout:222: writing smithi20328086-49 from 1779524 to 2346881 tid 2 2024-09-17T15:28:52.487 INFO:tasks.rados.rados.0.smithi203.stdout:222: writing smithi20328086-49 from 2837151 to 3503359 tid 3 2024-09-17T15:28:52.487 INFO:tasks.rados.rados.0.smithi203.stdout:223: snap_remove snap 16 2024-09-17T15:28:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:223: done (3 left) 2024-09-17T15:28:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:224: snap_create 2024-09-17T15:28:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 1 to smithi20328086-17 2024-09-17T15:28:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 2 to smithi20328086-17 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 3 to smithi20328086-17 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 4 to smithi20328086-17 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 5 to smithi20328086-17 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: finishing write tid 6 to smithi20328086-17 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: oid 17 updating version 0 to 69 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: oid 17 updating version 69 to 70 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: oid 17 updating version 70 to 71 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: oid 17 version 71 is already newer than 68 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: oid 17 updating version 71 to 72 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 72 (ObjNum 112 snap 20 seq_num 112) dirty exists 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:221: left oid 17 (ObjNum 112 snap 20 seq_num 112) 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:222: finishing write tid 1 to smithi20328086-49 2024-09-17T15:28:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:222: finishing write tid 2 to smithi20328086-49 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: finishing write tid 3 to smithi20328086-49 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: finishing write tid 4 to smithi20328086-49 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: finishing write tid 5 to smithi20328086-49 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: oid 49 updating version 0 to 50 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: oid 49 version 50 is already newer than 49 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: oid 49 updating version 50 to 51 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: oid 49 updating version 51 to 52 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 52 (ObjNum 113 snap 20 seq_num 113) dirty exists 2024-09-17T15:28:53.430 INFO:tasks.rados.rados.0.smithi203.stdout:222: left oid 49 (ObjNum 113 snap 20 seq_num 113) 2024-09-17T15:28:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:220: done (3 left) 2024-09-17T15:28:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:221: done (2 left) 2024-09-17T15:28:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:222: done (1 left) 2024-09-17T15:28:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:224: done (0 left) 2024-09-17T15:28:54.436 INFO:tasks.rados.rados.0.smithi203.stdout:225: snap_create 2024-09-17T15:28:55.419 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:28:55.419 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:28:55.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:28:55.422 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:28:55.423 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:28:55.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:225: done (0 left) 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:226: rollback oid 1 current snap is 22 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 20 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:227: read oid 39 snap -1 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:227: expect deleted 2024-09-17T15:28:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:228: snap_remove snap 19 2024-09-17T15:28:55.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 10 monclient: get_monmap_and_config 2024-09-17T15:28:55.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 10 monclient: build_initial_monmap 2024-09-17T15:28:55.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 10 monclient: monmap: 2024-09-17T15:28:55.541 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:55.541 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:55.543235+0000 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:55.543235+0000 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:55.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 1 Processor -- start 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 1 -- start start 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.537+0000 7f448ddf8640 10 monclient: init 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44880a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44880a54c0 0x7f44880a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-a con 0x7f44880a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f448813d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44880a6530 con 0x7f448813d2f0 2024-09-17T15:28:55.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448809c150 con 0x7f44880a50e0 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f448809c430 con 0x7f44880a54c0 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f448ddf8640 10 monclient(hunting): authenticate will time out at 1068.508477s 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44880a54c0 0x7f44880a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f44880a54c0 0x7f44880a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47316/0 (socket says 172.21.15.86:47316) 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f4487fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f448813d2f0 0x7f448813f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42490/0 (socket says 172.21.15.86:42490) 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 -- 172.21.15.86:0/1677563131 learned_addr learned my addr 172.21.15.86:0/1677563131 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): get_auth_request con 0x7f44880a54c0 auth_method 0 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:55.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient(hunting): handle_auth_done global_id 4464 payload 274 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 -- 172.21.15.86:0/1677563131 >> v1:172.21.15.203:6789/0 conn(0x7f44880a50e0 legacy=0x7f448809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 -- 172.21.15.86:0/1677563131 >> v1:172.21.15.86:6789/0 conn(0x7f448813d2f0 legacy=0x7f448813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 monclient: found mon.noname-a 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.541+0000 7f447ffff640 1 -- 172.21.15.86:0/1677563131 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44880a6940 con 0x7f44880a54c0 2024-09-17T15:28:55.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 -- 172.21.15.86:0/1677563131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44880a54c0 msgr2=0x7f44880a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:55.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 --2- 172.21.15.86:0/1677563131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44880a54c0 0x7f44880a58d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4470009870 tx=0x7f447001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1910a1f1c1ae0151 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:55.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 1 -- 172.21.15.86:0/1677563131 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44700b0070 con 0x7f44880a54c0 2024-09-17T15:28:55.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:55.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: dump: 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:55.548 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: _finish_auth 0 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 1 -- 172.21.15.86:0/1677563131 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44700ab600 con 0x7f44880a54c0 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 1 -- 172.21.15.86:0/1677563131 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44700ab9d0 con 0x7f44880a54c0 2024-09-17T15:28:55.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4486ffd640 10 monclient: dump: 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 5 monclient: authenticate success, global_id 4464 2024-09-17T15:28:55.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:55.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 4 set_mon_vals no callback set 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: shutdown 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- 172.21.15.86:0/1677563131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44880a54c0 msgr2=0x7f44880a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1677563131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44880a54c0 0x7f44880a58d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4470009870 tx=0x7f447001c4b0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- 172.21.15.86:0/1677563131 shutdown_connections 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1677563131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f44880a54c0 0x7f44880a58d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- 172.21.15.86:0/1677563131 >> 172.21.15.86:0/1677563131 conn(0x7f44880aaae0 msgr2=0x7f44880aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- 172.21.15.86:0/1677563131 shutdown_connections 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- 172.21.15.86:0/1677563131 wait complete. 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: build_initial_monmap 2024-09-17T15:28:55.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: monmap: 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 Processor -- start 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- start start 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: init 2024-09-17T15:28:55.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 0x7f44881cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-b con 0x7f44880a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-c con 0x7f44880a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): picked mon.noname-a con 0x7f448813d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4488143a50 con 0x7f448813d2f0 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f44881438d0 con 0x7f44880a50e0 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4488143bd0 con 0x7f44880a54c0 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:55.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f448ddf8640 10 monclient(hunting): authenticate will time out at 1068.515461s 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47330/0 (socket says 172.21.15.86:47330) 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 -- 172.21.15.86:0/1648588625 learned_addr learned my addr 172.21.15.86:0/1648588625 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): get_auth_request con 0x7f448813d2f0 auth_method 0 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 10 monclient(hunting): get_auth_request con 0x7f44880a54c0 auth_method 0 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:55.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 msgr2=0x7f4488138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 msgr2=0x7f4488138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f447ffff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient(hunting): handle_auth_done global_id 4467 payload 824 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f44877fe640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 0x7f44881cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 msgr2=0x7f4488138110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 msgr2=0x7f44881cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 0x7f44881cfe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient: _finish_hunting 0 2024-09-17T15:28:55.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 monclient: found mon.noname-a 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f4487fff640 1 -- 172.21.15.86:0/1648588625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44880a3630 con 0x7f448813d2f0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f44877fe640 10 monclient: get_auth_request con 0x7f44880a50e0 auth_method 0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.545+0000 7f44877fe640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 0x7f44881cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4487fff640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 msgr2=0x7f44880a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4487fff640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f448000c6b0 tx=0x7f448000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480017070 con 0x7f448813d2f0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: dump: 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:55.560 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: _finish_auth 0 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f448000a640 con 0x7f448813d2f0 2024-09-17T15:28:55.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448000aa30 con 0x7f448813d2f0 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 10 monclient: dump: 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:55.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f44867fc640 4 set_mon_vals no callback set 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 5 monclient: authenticate success, global_id 4467 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _renew_subs 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44881211f0 con 0x7f448813d2f0 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _renew_subs 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44880a18d0 con 0x7f448813d2f0 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44881213c0 con 0x7f448813d2f0 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.549+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f448001a070 con 0x7f448813d2f0 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.553+0000 7f4484ff9640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 0x7f446803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.553+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 4755+0+0 (secure 0 0 0) 0x7f448004f5a0 con 0x7f448813d2f0 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.553+0000 7f44877fe640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 0x7f446803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.553+0000 7f44877fe640 10 monclient: get_auth_request con 0x7f446803bbb0 auth_method 0 2024-09-17T15:28:55.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.553+0000 7f44877fe640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 0x7f446803e070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4478003e40 tx=0x7f4478009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:55.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.557+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f448001a360 con 0x7f448813d2f0 2024-09-17T15:28:55.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.557+0000 7f4484ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:55.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.557+0000 7f4484ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f448ddf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f448ddf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f448ddf8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44880a58d0 con 0x7f448813d2f0 2024-09-17T15:28:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f4484ff9640 1 -- 172.21.15.86:0/1648588625 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f44800079d0 con 0x7f448813d2f0 2024-09-17T15:28:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f4484ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.781+0000 7f4484ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:55.785 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 msgr2=0x7f446803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 0x7f446803e070 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4478003e40 tx=0x7f4478009040 comp rx=0 tx=0).stop 2024-09-17T15:28:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 10 monclient: shutdown 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 msgr2=0x7f44880a1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f448000c6b0 tx=0x7f448000cb70 comp rx=0 tx=0).stop 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 shutdown_connections 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f446803bbb0 0x7f446803e070 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f448813d2f0 0x7f44880a1b70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44880a54c0 0x7f4488138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 --2- 172.21.15.86:0/1648588625 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44880a50e0 0x7f44881cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 >> 172.21.15.86:0/1648588625 conn(0x7f44880aaae0 msgr2=0x7f44880a0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 shutdown_connections 2024-09-17T15:28:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:55.785+0000 7f448ddf8640 1 -- 172.21.15.86:0/1648588625 wait complete. 2024-09-17T15:28:56.453 INFO:tasks.rados.rados.0.smithi203.stdout:228: done (2 left) 2024-09-17T15:28:56.454 INFO:tasks.rados.rados.0.smithi203.stdout:229: snap_remove snap 18 2024-09-17T15:28:57.395 INFO:tasks.rados.rados.0.smithi203.stdout:229: done (2 left) 2024-09-17T15:28:57.395 INFO:tasks.rados.rados.0.smithi203.stdout:230: snap_remove snap 15 2024-09-17T15:28:58.396 INFO:tasks.rados.rados.0.smithi203.stdout:230: done (2 left) 2024-09-17T15:28:58.396 INFO:tasks.rados.rados.0.smithi203.stdout:231: snap_create 2024-09-17T15:28:58.397 INFO:tasks.rados.rados.0.smithi203.stdout:226: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:28:58.397 INFO:tasks.rados.rados.0.smithi203.stdout:226: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:28:58.397 INFO:tasks.rados.rados.0.smithi203.stdout:226: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:28:58.397 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 52 (ObjNum 79 snap 5 seq_num 79) dirty exists 2024-09-17T15:28:58.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:28:58.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: get_monmap_and_config 2024-09-17T15:28:58.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: build_initial_monmap 2024-09-17T15:28:58.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: monmap: 2024-09-17T15:28:58.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:28:58.933286+0000 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:28:58.933286+0000 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:28:58.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 Processor -- start 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 -- start start 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: init 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f01d81319b0 0x7f01d8131d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-c con 0x7f01d81319b0 addr v2:172.21.15.203:3300/0 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01d809d840 addr v1:172.21.15.203:6789/0 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f01d80a2d50 0x7f01d813ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-a con 0x7f01d80a2d50 addr v2:172.21.15.86:3300/0 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d809c5b0 con 0x7f01d809d840 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f01d80a40f0 con 0x7f01d80a2d50 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f01d80a42c0 con 0x7f01d81319b0 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01df13a640 10 monclient(hunting): authenticate will time out at 1071.899150s 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f01d80a2d50 0x7f01d813ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f01d80a2d50 0x7f01d813ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39668/0 (socket says 172.21.15.86:39668) 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 -- 172.21.15.86:0/1063838904 learned_addr learned my addr 172.21.15.86:0/1063838904 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): get_auth_request con 0x7f01d80a2d50 auth_method 0 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:58.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dced6640 1 --2- 172.21.15.86:0/1063838904 >> v2:172.21.15.203:3300/0 conn(0x7f01d81319b0 0x7f01d8131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dced6640 10 monclient(hunting): get_auth_request con 0x7f01d81319b0 auth_method 0 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dced6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dced6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dced6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_done global_id 4470 payload 274 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 -- 172.21.15.86:0/1063838904 >> v2:172.21.15.203:3300/0 conn(0x7f01d81319b0 msgr2=0x7f01d8131d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:58.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 --2- 172.21.15.86:0/1063838904 >> v2:172.21.15.203:3300/0 conn(0x7f01d81319b0 0x7f01d8131d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 -- 172.21.15.86:0/1063838904 >> v1:172.21.15.203:6789/0 conn(0x7f01d809d840 legacy=0x7f01d80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient: _finish_hunting 0 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 monclient: found mon.noname-a 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:58.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.929+0000 7f01dd6d7640 1 -- 172.21.15.86:0/1063838904 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d80a4500 con 0x7f01d80a2d50 2024-09-17T15:28:58.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01dd6d7640 1 -- 172.21.15.86:0/1063838904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d80a2d50 msgr2=0x7f01d813ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01dd6d7640 1 --2- 172.21.15.86:0/1063838904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d80a2d50 0x7f01d813ff10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f01d806dad0 tx=0x7f01d001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2477a3a44352e21c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 1 -- 172.21.15.86:0/1063838904 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00a5640 con 0x7f01d80a2d50 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: dump: 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: _finish_auth 0 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 1 -- 172.21.15.86:0/1063838904 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d00a57e0 con 0x7f01d80a2d50 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 1 -- 172.21.15.86:0/1063838904 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00a5bb0 con 0x7f01d80a2d50 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01d77fe640 10 monclient: dump: 2024-09-17T15:28:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 5 monclient: authenticate success, global_id 4470 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: get_monmap_and_config success 2024-09-17T15:28:58.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 4 set_mon_vals no callback set 2024-09-17T15:28:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: shutdown 2024-09-17T15:28:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- 172.21.15.86:0/1063838904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d80a2d50 msgr2=0x7f01d813ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 --2- 172.21.15.86:0/1063838904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d80a2d50 0x7f01d813ff10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f01d806dad0 tx=0x7f01d001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- 172.21.15.86:0/1063838904 shutdown_connections 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 --2- 172.21.15.86:0/1063838904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d80a2d50 0x7f01d813ff10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 --2- 172.21.15.86:0/1063838904 >> v2:172.21.15.203:3300/0 conn(0x7f01d81319b0 0x7f01d8131d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- 172.21.15.86:0/1063838904 >> 172.21.15.86:0/1063838904 conn(0x7f01d806d170 msgr2=0x7f01d80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- 172.21.15.86:0/1063838904 shutdown_connections 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- 172.21.15.86:0/1063838904 wait complete. 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: build_initial_monmap 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: monmap: 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:28:58.948 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 Processor -- start 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 -- start start 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: init 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: _reopen_session rank -1 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-c con 0x7f01d809d840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:28:58.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d80a2d50 0x7f01d813e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.937+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-b con 0x7f01d80a2d50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): picked mon.noname-a con 0x7f01d81319b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f01d8142e80 con 0x7f01d81319b0 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f01d809c890 con 0x7f01d80a2d50 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f01d8143000 con 0x7f01d809d840 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): _renew_subs 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient(hunting): authenticate will time out at 1071.908046s 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39670/0 (socket says 172.21.15.86:39670) 2024-09-17T15:28:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 -- 172.21.15.86:0/664260460 learned_addr learned my addr 172.21.15.86:0/664260460 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): get_auth_request con 0x7f01d81319b0 auth_method 0 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 10 monclient(hunting): get_auth_request con 0x7f01d809d840 auth_method 0 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 msgr2=0x7f01d813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 msgr2=0x7f01d813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:28:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d7fff640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d80a2d50 0x7f01d813e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dced6640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d7fff640 10 monclient(hunting): get_auth_request con 0x7f01d80a2d50 auth_method 0 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient(hunting): handle_auth_done global_id 4473 payload 824 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 msgr2=0x7f01d813f240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d80a2d50 msgr2=0x7f01d813e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d80a2d50 0x7f01d813e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient: _finish_hunting 0 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 monclient: found mon.noname-a 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:28:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 -- 172.21.15.86:0/664260460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d813d120 con 0x7f01d81319b0 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 msgr2=0x7f01d81392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01dd6d7640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01d00af040 tx=0x7f01d0019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00bc020 con 0x7f01d81319b0 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: dump: 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: _finish_auth 0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 5 monclient: authenticate success, global_id 4473 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d00b8a90 con 0x7f01d81319b0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _renew_subs 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d00a5640 con 0x7f01d81319b0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d813bb30 con 0x7f01d81319b0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:28:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d57fa640 10 monclient: dump: 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01d6ffd640 4 set_mon_vals no callback set 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _renew_subs 2024-09-17T15:28:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d813c860 con 0x7f01d81319b0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.941+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d8131d90 con 0x7f01d81319b0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f01d00bf020 con 0x7f01d81319b0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01d57fa640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 0x7f01c403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4783+0+0 (secure 0 0 0) 0x7f01d00eb080 con 0x7f01d81319b0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01dced6640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 0x7f01c403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:28:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01dced6640 10 monclient: get_auth_request con 0x7f01c403bb60 auth_method 0 2024-09-17T15:28:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.945+0000 7f01dced6640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 0x7f01c403e020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b990 tx=0x7f01cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:28:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.949+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f01d00c6030 con 0x7f01d81319b0 2024-09-17T15:28:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.949+0000 7f01d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:28:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:58.949+0000 7f01d57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:28:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01df13a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:28:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01df13a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01df13a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:28:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f01d809c0e0 con 0x7f01d81319b0 2024-09-17T15:28:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01d57fa640 1 -- 172.21.15.86:0/664260460 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f01d00165c0 con 0x7f01d81319b0 2024-09-17T15:28:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:28:59.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.185+0000 7f01d57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:28:59.189 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":24,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 msgr2=0x7f01c403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 0x7f01c403e020 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00b990 tx=0x7f01cc008040 comp rx=0 tx=0).stop 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 10 monclient: shutdown 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 msgr2=0x7f01d81392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f01d00af040 tx=0x7f01d0019650 comp rx=0 tx=0).stop 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 shutdown_connections 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01c403bb60 0x7f01c403e020 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01d81319b0 0x7f01d81392e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:59.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d80a2d50 0x7f01d813e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 --2- 172.21.15.86:0/664260460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01d809d840 0x7f01d813f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:28:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 >> 172.21.15.86:0/664260460 conn(0x7f01d806d170 msgr2=0x7f01d80a06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:28:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 shutdown_connections 2024-09-17T15:28:59.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:28:59.189+0000 7f01df13a640 1 -- 172.21.15.86:0/664260460 wait complete. 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:226: done (2 left) 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:227: done (1 left) 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:231: done (0 left) 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:232: copy_from oid 5 from oid 17 current snap is 23 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:233: rollback oid 6 current snap is 23 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 21 2024-09-17T15:28:59.399 INFO:tasks.rados.rados.0.smithi203.stdout:234: delete oid 24 current snap is 23 2024-09-17T15:28:59.408 INFO:tasks.rados.rados.0.smithi203.stdout:233: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:28:59.409 INFO:tasks.rados.rados.0.smithi203.stdout:234: done (2 left) 2024-09-17T15:28:59.409 INFO:tasks.rados.rados.0.smithi203.stdout:235: write oid 28 current snap is 23 2024-09-17T15:28:59.409 INFO:tasks.rados.rados.0.smithi203.stdout:235: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-17T15:28:59.437 INFO:tasks.rados.rados.0.smithi203.stdout:235: writing smithi20328086-28 from 732805 to 1531735 tid 1 2024-09-17T15:28:59.458 INFO:tasks.rados.rados.0.smithi203.stdout:235: writing smithi20328086-28 from 2296488 to 2867514 tid 2 2024-09-17T15:28:59.469 INFO:tasks.rados.rados.0.smithi203.stdout:235: writing smithi20328086-28 from 3291997 to 3697102 tid 3 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:236: write (excl) oid 39 current snap is 23 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:233: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:233: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 64 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:232: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:232: finishing copy_from to smithi20328086-5 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 72 (ObjNum 112 snap 20 seq_num 112) dirty exists 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:235: finishing write tid 1 to smithi20328086-28 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:235: finishing write tid 2 to smithi20328086-28 2024-09-17T15:28:59.470 INFO:tasks.rados.rados.0.smithi203.stdout:236: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-17T15:28:59.483 INFO:tasks.rados.rados.0.smithi203.stdout:236: writing smithi20328086-39 from 749612 to 1223791 tid 1 2024-09-17T15:28:59.495 INFO:tasks.rados.rados.0.smithi203.stdout:236: writing smithi20328086-39 from 1954970 to 2386007 tid 2 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:236: writing smithi20328086-39 from 2919420 to 3535797 tid 3 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:232: done (3 left) 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:233: done (2 left) 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:237: delete oid 37 current snap is 23 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:235: finishing write tid 3 to smithi20328086-28 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:235: finishing write tid 4 to smithi20328086-28 2024-09-17T15:28:59.508 INFO:tasks.rados.rados.0.smithi203.stdout:235: finishing write tid 5 to smithi20328086-28 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:235: oid 28 updating version 0 to 70 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:235: oid 28 version 70 is already newer than 69 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:235: oid 28 version 70 is already newer than 68 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:235: oid 28 updating version 70 to 71 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 71 (ObjNum 115 snap 23 seq_num 115) dirty exists 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:235: left oid 28 (ObjNum 115 snap 23 seq_num 115) 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:236: finishing write tid 1 to smithi20328086-39 2024-09-17T15:28:59.509 INFO:tasks.rados.rados.0.smithi203.stdout:236: finishing write tid 2 to smithi20328086-39 2024-09-17T15:28:59.514 INFO:tasks.rados.rados.0.smithi203.stdout:236: finishing write tid 3 to smithi20328086-39 2024-09-17T15:28:59.515 INFO:tasks.rados.rados.0.smithi203.stdout:236: finishing write tid 4 to smithi20328086-39 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:236: finishing write tid 5 to smithi20328086-39 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:236: oid 39 updating version 0 to 61 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:236: oid 39 version 61 is already newer than 60 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:236: oid 39 updating version 61 to 62 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:236: oid 39 version 62 is already newer than 59 2024-09-17T15:28:59.516 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 62 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-17T15:28:59.517 INFO:tasks.rados.rados.0.smithi203.stdout:236: left oid 39 (ObjNum 116 snap 23 seq_num 116) 2024-09-17T15:28:59.517 INFO:tasks.rados.rados.0.smithi203.stdout:235: done (2 left) 2024-09-17T15:28:59.517 INFO:tasks.rados.rados.0.smithi203.stdout:236: done (1 left) 2024-09-17T15:28:59.517 INFO:tasks.rados.rados.0.smithi203.stdout:237: done (0 left) 2024-09-17T15:28:59.517 INFO:tasks.rados.rados.0.smithi203.stdout:238: rollback oid 26 current snap is 23 2024-09-17T15:28:59.518 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 20 2024-09-17T15:28:59.518 INFO:tasks.rados.rados.0.smithi203.stdout:239: delete oid 6 current snap is 23 2024-09-17T15:28:59.521 INFO:tasks.rados.rados.0.smithi203.stdout:239: done (1 left) 2024-09-17T15:28:59.521 INFO:tasks.rados.rados.0.smithi203.stdout:240: copy_from oid 45 from oid 44 current snap is 23 2024-09-17T15:28:59.521 INFO:tasks.rados.rados.0.smithi203.stdout:241: rollback oid 32 current snap is 23 2024-09-17T15:28:59.522 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 21 2024-09-17T15:28:59.522 INFO:tasks.rados.rados.0.smithi203.stdout:242: copy_from oid 50 from oid 8 current snap is 23 2024-09-17T15:28:59.522 INFO:tasks.rados.rados.0.smithi203.stdout:243: rollback oid 42 current snap is 23 2024-09-17T15:28:59.522 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 22 2024-09-17T15:28:59.522 INFO:tasks.rados.rados.0.smithi203.stdout:244: snap_remove snap 17 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:244: done (5 left) 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:245: snap_create 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:241: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 3056417917) dirty dne 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:238: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:238: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:238: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 75 (ObjNum 97 snap 16 seq_num 97) dirty exists 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:243: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:29:00.474 INFO:tasks.rados.rados.0.smithi203.stdout:243: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:243: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 60 (ObjNum 93 snap 13 seq_num 93) dirty exists 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:240: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:242: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:240: finishing copy_from to smithi20328086-45 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:242: finishing copy_from to smithi20328086-50 2024-09-17T15:29:00.475 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:29:02.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:02.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: get_monmap_and_config 2024-09-17T15:29:02.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: build_initial_monmap 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: monmap: 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:02.341377+0000 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:02.341377+0000 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:02.340 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:02.341 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 Processor -- start 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 -- start start 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: init 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-a con 0x7f5a340a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:02.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-b con 0x7f5a340a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-c con 0x7f5a34137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5a3409bdc0 con 0x7f5a340a50e0 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5a3409d350 con 0x7f5a34137ea0 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5a340a6530 con 0x7f5a340a54c0 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3acd2640 10 monclient(hunting): authenticate will time out at 1075.306977s 2024-09-17T15:29:02.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39686/0 (socket says 172.21.15.86:39686) 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 learned_addr learned my addr 172.21.15.86:0/3939887468 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a340a50e0 auth_method 0 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a340a54c0 auth_method 0 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a34137ea0 auth_method 0 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.337+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a340a50e0 auth_method 0 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a340a54c0 auth_method 0 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a34137ea0 auth_method 0 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.541+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a340a54c0 auth_method 0 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a340a50e0 auth_method 0 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a34137ea0 auth_method 0 2024-09-17T15:29:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:02.945+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:03.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:03.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:03.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a340a50e0 auth_method 0 2024-09-17T15:29:03.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a340a54c0 auth_method 0 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:03.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a34137ea0 auth_method 0 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:03.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:03.749+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:05.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): tick 2024-09-17T15:29:05.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:29:05.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 monclient(hunting): continuing hunt 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 msgr2=0x7f5a340a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a340a54c0 0x7f5a340a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 msgr2=0x7f5a34138330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.203:3300/0 conn(0x7f5a34137ea0 0x7f5a34138330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 -- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 msgr2=0x7f5a3409ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 --2- 172.21.15.86:0/3939887468 >> v2:172.21.15.86:3300/0 conn(0x7f5a340a50e0 0x7f5a3409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient: _add_conns ranks=[0,1] 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a1c003680 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a1c0073b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 -- 172.21.15.86:0/3939887468 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a3409c150 con 0x7f5a1c003680 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 1 -- 172.21.15.86:0/3939887468 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1c00c0f0 con 0x7f5a1c0073b0 2024-09-17T15:29:05.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a327fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:05.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3939887468 >> v1:172.21.15.86:6789/0 conn(0x7f5a1c003680 legacy=0x7f5a1c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:05.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> v1:172.21.15.203:6789/0 conn(0x7f5a1c0073b0 legacy=0x7f5a1c009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:05.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.341+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:29:05.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1748329499 0 0) 0x7f5a1c00c0f0 con 0x7f5a1c0073b0 2024-09-17T15:29:05.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient(hunting): _negotiate 2024-09-17T15:29:05.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:05.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient(hunting): my global_id is 14167 2024-09-17T15:29:05.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a20003530 con 0x7f5a1c0073b0 2024-09-17T15:29:05.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1077813953 0 0) 0x7f5a20003530 con 0x7f5a1c0073b0 2024-09-17T15:29:05.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 >> v1:172.21.15.86:6789/0 conn(0x7f5a1c003680 legacy=0x7f5a1c005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:05.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:29:05.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1c00d350 con 0x7f5a1c0073b0 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f5a2800f040 con 0x7f5a1c0073b0 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.909+0000 7f5a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient: dump: 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:05.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 >> v1:172.21.15.203:6789/0 conn(0x7f5a1c0073b0 legacy=0x7f5a1c009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 1 -- 172.21.15.86:0/3939887468 reap_dead start 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:29:05.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20003bc0 0x7f5a20003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): picked mon.a con 0x7f5a20003bc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 0x7f5a20004db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): picked mon.c con 0x7f5a20004930 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a2000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): picked mon.b con 0x7f5a200099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2000c500 con 0x7f5a20003bc0 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2000c6b0 con 0x7f5a200099c0 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5a2000c8b0 con 0x7f5a20004930 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:05.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a337fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 0x7f5a20004db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a20004930 auth_method 0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a2000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a200099c0 auth_method 0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20003bc0 0x7f5a20003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a20003bc0 auth_method 0 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a38a6e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient(hunting): handle_auth_done global_id 14167 payload 293 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 msgr2=0x7f5a2000bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:05.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a2000bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20003bc0 msgr2=0x7f5a20003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20003bc0 0x7f5a20003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 monclient: found mon.c 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a20003530 con 0x7f5a20004930 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient: _finish_auth 0 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3acd2640 5 monclient: authenticate success, global_id 14167 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3acd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:05.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.913+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a2000bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a33fff640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 msgr2=0x7f5a20004db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 0x7f5a20004db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a28006e90 tx=0x7f5a2801d870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644709855669c04 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2800ee10 con 0x7f5a20004930 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: dump: 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:05.924 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a3acd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a280ab640 con 0x7f5a20004930 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a3acd2640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:05.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a3acd2640 4 set_mon_vals no callback set 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 1 -- 172.21.15.86:0/3939887468 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a280ab9f0 con 0x7f5a20004930 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.917+0000 7f5a337fe640 10 monclient: dump: 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:05.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 10 monclient: shutdown 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 msgr2=0x7f5a20004db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 0x7f5a20004db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5a28006e90 tx=0x7f5a2801d870 comp rx=0 tx=0).stop 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3939887468 shutdown_connections 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a2000bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20004930 0x7f5a20004db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3939887468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20003bc0 0x7f5a20003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3939887468 >> 172.21.15.86:0/3939887468 conn(0x7f5a340aaae0 msgr2=0x7f5a340aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3939887468 shutdown_connections 2024-09-17T15:29:05.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3939887468 wait complete. 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 10 monclient: build_initial_monmap 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 10 monclient: monmap: 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:05.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 Processor -- start 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 1 -- start start 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.921+0000 7f5a3acd2640 10 monclient: init 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-c con 0x7f5a20003bc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 0x7f5a341373c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-a con 0x7f5a20004930 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a34120d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): picked mon.noname-b con 0x7f5a200099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5a34138640 con 0x7f5a20004930 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5a341384c0 con 0x7f5a200099c0 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5a341387c0 con 0x7f5a20003bc0 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3acd2640 10 monclient(hunting): authenticate will time out at 1078.892729s 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40438/0 (socket says 172.21.15.86:40438) 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3538821934 learned_addr learned my addr 172.21.15.86:0/3538821934 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request con 0x7f5a20003bc0 auth_method 0 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 0x7f5a341373c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): get_auth_request con 0x7f5a20004930 auth_method 0 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 msgr2=0x7f5a34136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 msgr2=0x7f5a34136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3926f640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a34120d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3926f640 10 monclient(hunting): get_auth_request con 0x7f5a200099c0 auth_method 0 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3926f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3926f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:05.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a3926f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 msgr2=0x7f5a34136e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 msgr2=0x7f5a34120d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a34120d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 monclient: found mon.noname-a 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 -- 172.21.15.86:0/3538821934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a340a18d0 con 0x7f5a20004930 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 msgr2=0x7f5a341373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a33fff640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 0x7f5a341373c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5a280aa000 tx=0x7f5a2801a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.925+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2800edf0 con 0x7f5a20004930 2024-09-17T15:29:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: dump: 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: _finish_auth 0 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: _check_auth_tickets 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a280abe50 con 0x7f5a20004930 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 5 monclient: authenticate success, global_id 4476 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _renew_subs 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a341bfff0 con 0x7f5a20004930 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004020 con 0x7f5a20004930 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 10 monclient: dump: 2024-09-17T15:29:05.937 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _renew_subs 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a32ffd640 4 set_mon_vals no callback set 2024-09-17T15:29:05.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a34124860 con 0x7f5a20004930 2024-09-17T15:29:05.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5a28004200 con 0x7f5a20004930 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a3409bdc0 con 0x7f5a20004930 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 0x7f5a1c03e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 4783+0+0 (secure 0 0 0) 0x7f5a280ead50 con 0x7f5a20004930 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 0x7f5a1c03e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.929+0000 7f5a38a6e640 10 monclient: get_auth_request con 0x7f5a1c03c480 auth_method 0 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.933+0000 7f5a38a6e640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 0x7f5a1c03e940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000c10 tx=0x7f5a24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.933+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5a280af020 con 0x7f5a20004930 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.933+0000 7f5a317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:05.933+0000 7f5a317fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:238: done (5 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:240: done (4 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:241: done (3 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:242: done (2 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:243: done (1 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:245: done (0 left) 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:246: rollback oid 16 current snap is 24 2024-09-17T15:29:05.975 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 20 2024-09-17T15:29:05.976 INFO:tasks.rados.rados.0.smithi203.stdout:247: snap_create 2024-09-17T15:29:05.976 INFO:tasks.rados.rados.0.smithi203.stdout:246: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:29:05.976 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 0 (ObjNum 129 snap 0 seq_num 3084040578) dirty dne 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5a34074be0 con 0x7f5a20004930 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a317fa640 1 -- 172.21.15.86:0/3538821934 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f5a28017680 con 0x7f5a20004930 2024-09-17T15:29:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:06.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a317fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:06.168 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:06.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 msgr2=0x7f5a1c03e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:06.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 0x7f5a1c03e940 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5a24000c10 tx=0x7f5a24008040 comp rx=0 tx=0).stop 2024-09-17T15:29:06.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 10 monclient: shutdown 2024-09-17T15:29:06.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 msgr2=0x7f5a341373c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 0x7f5a341373c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5a280aa000 tx=0x7f5a2801a710 comp rx=0 tx=0).stop 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 shutdown_connections 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5a1c03c480 0x7f5a1c03e940 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5a200099c0 0x7f5a34120d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a20004930 0x7f5a341373c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 --2- 172.21.15.86:0/3538821934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a20003bc0 0x7f5a34136e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.165+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 >> 172.21.15.86:0/3538821934 conn(0x7f5a340aaae0 msgr2=0x7f5a341c2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.169+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 shutdown_connections 2024-09-17T15:29:06.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:06.169+0000 7f5a3acd2640 1 -- 172.21.15.86:0/3538821934 wait complete. 2024-09-17T15:29:09.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: get_monmap_and_config 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: build_initial_monmap 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: monmap: 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:09.313525+0000 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:09.313525+0000 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:09.312 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:09.313 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:09.313 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 Processor -- start 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 -- start start 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: init 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-a con 0x7f73c00689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:09.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73c0068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-c con 0x7f73c013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73c009c8b0 con 0x7f73c0068db0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f73c0074500 con 0x7f73c00689d0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f73c009cb90 con 0x7f73c013aff0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73c5d25640 10 monclient(hunting): authenticate will time out at 1082.279064s 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33554/0 (socket says 172.21.15.86:33554) 2024-09-17T15:29:09.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 learned_addr learned my addr 172.21.15.86:0/3833814019 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request con 0x7f73c00689d0 auth_method 0 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 10 monclient(hunting): get_auth_request con 0x7f73c013aff0 auth_method 0 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:09.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.309+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:09.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.313+0000 7f73beffd640 1 -- 172.21.15.86:0/3833814019 >> v1:172.21.15.203:6789/0 conn(0x7f73c0068db0 legacy=0x7f73c0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request con 0x7f73c00689d0 auth_method 0 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:09.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 10 monclient(hunting): get_auth_request con 0x7f73c013aff0 auth_method 0 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.513+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request con 0x7f73c00689d0 auth_method 0 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:09.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 10 monclient(hunting): get_auth_request con 0x7f73c013aff0 auth_method 0 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:09.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:09.917+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:10.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:10.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:10.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:10.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:10.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 10 monclient(hunting): get_auth_request con 0x7f73c013aff0 auth_method 0 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:10.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:10.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:10.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:10.721+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 325261633 0 0) 0x7f73c009c8b0 con 0x7f73c0068db0 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): my global_id is 14173 2024-09-17T15:29:11.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a8003530 con 0x7f73c0068db0 2024-09-17T15:29:11.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1589723583 0 0) 0x7f73a8003530 con 0x7f73c0068db0 2024-09-17T15:29:11.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 msgr2=0x7f73c013d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.203:3300/0 conn(0x7f73c013aff0 0x7f73c013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 msgr2=0x7f73c006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 --2- 172.21.15.86:0/3833814019 >> v2:172.21.15.86:3300/0 conn(0x7f73c00689d0 0x7f73c006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c009ce70 con 0x7f73c0068db0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _finish_auth 0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f73b00019f0 con 0x7f73c0068db0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:11.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: dump: 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 >> v1:172.21.15.203:6789/0 conn(0x7f73c0068db0 legacy=0x7f73c0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:11.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 0x7f73a8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): picked mon.b con 0x7f73a8003b20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8004810 0x7f73a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): picked mon.c con 0x7f73a8004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73a8005390 0x7f73a800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): picked mon.a con 0x7f73a8005390 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73c009cb90 con 0x7f73a8005390 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73a800c2d0 con 0x7f73a8003b20 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73a800c4d0 con 0x7f73a8004810 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:11.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.932+0000 7f73be7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8004810 0x7f73a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 0x7f73a8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request con 0x7f73a8003b20 auth_method 0 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): get_auth_request con 0x7f73a8004810 auth_method 0 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:11.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient(hunting): handle_auth_done global_id 14173 payload 293 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8004810 msgr2=0x7f73a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8004810 0x7f73a8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73a8005390 0x7f73a800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73a8005390 msgr2=0x7f73a800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73a8005390 0x7f73a800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 monclient: found mon.b 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a8003530 con 0x7f73a8003b20 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:11.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73c5d25640 5 monclient: authenticate success, global_id 14173 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73c5d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bffff640 10 monclient: get_auth_request con 0x7f73a8005390 auth_method 0 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bffff640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73a8005390 0x7f73a800b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73beffd640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8004810 0x7f73a8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:11.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.936+0000 7f73bffff640 1 -- 172.21.15.86:0/3833814019 reap_dead start 2024-09-17T15:29:11.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73bf7fe640 1 -- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 msgr2=0x7f73a8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:11.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 0x7f73a8003f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73b4011980 tx=0x7f73b401ce70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f7d48bf16721c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b40b2070 con 0x7f73a8003b20 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: dump: 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:11.948 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b40a5cd0 con 0x7f73a8003b20 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 1 -- 172.21.15.86:0/3833814019 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b400f040 con 0x7f73a8003b20 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 4 set_mon_vals no callback set 2024-09-17T15:29:11.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73be7fc640 10 monclient: dump: 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:11.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 10 monclient: shutdown 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 -- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 msgr2=0x7f73a8003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 0x7f73a8003f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f73b4011980 tx=0x7f73b401ce70 comp rx=0 tx=0).stop 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 -- 172.21.15.86:0/3833814019 shutdown_connections 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 --2- 172.21.15.86:0/3833814019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73a8003b20 0x7f73a8003f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 -- 172.21.15.86:0/3833814019 >> 172.21.15.86:0/3833814019 conn(0x7f73c00a6440 msgr2=0x7f73c00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 -- 172.21.15.86:0/3833814019 shutdown_connections 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.944+0000 7f73c5d25640 1 -- 172.21.15.86:0/3833814019 wait complete. 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient: build_initial_monmap 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient: monmap: 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 Processor -- start 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 -- start start 2024-09-17T15:29:11.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient: init 2024-09-17T15:29:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-c con 0x7f73a8003b20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73c0134fb0 0x7f73c0135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-b con 0x7f73c0134fb0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 0x7f73c01cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): picked mon.noname-a con 0x7f73c01cd2b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73c009c2d0 con 0x7f73c01cd2b0 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73c01362f0 con 0x7f73c0134fb0 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73c0136470 con 0x7f73a8003b20 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73c5d25640 10 monclient(hunting): authenticate will time out at 1084.918788s 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33102/0 (socket says 172.21.15.86:33102) 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 -- 172.21.15.86:0/43889794 learned_addr learned my addr 172.21.15.86:0/43889794 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request con 0x7f73a8003b20 auth_method 0 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 msgr2=0x7f73c01d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 msgr2=0x7f73c01d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 0x7f73c01cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): get_auth_request con 0x7f73c01cd2b0 auth_method 0 2024-09-17T15:29:11.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73c0134fb0 0x7f73c0135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): get_auth_request con 0x7f73c0134fb0 auth_method 0 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 msgr2=0x7f73c01d0830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73c0134fb0 msgr2=0x7f73c0135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73c0134fb0 0x7f73c0135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 monclient: found mon.noname-a 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.948+0000 7f73bffff640 1 -- 172.21.15.86:0/43889794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73c009f220 con 0x7f73c01cd2b0 2024-09-17T15:29:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73bffff640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 msgr2=0x7f73c01cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73bffff640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 0x7f73c01cd6d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f73c0069970 tx=0x7f73b8004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b801b020 con 0x7f73c01cd2b0 2024-09-17T15:29:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: dump: 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: _finish_auth 0 2024-09-17T15:29:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b8007ba0 con 0x7f73c01cd2b0 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8017470 con 0x7f73c01cd2b0 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 10 monclient: dump: 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73bdffb640 4 set_mon_vals no callback set 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 5 monclient: authenticate success, global_id 4479 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _renew_subs 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73c01d1290 con 0x7f73c01cd2b0 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _renew_subs 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73c0127220 con 0x7f73c01cd2b0 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f73b801e020 con 0x7f73c01cd2b0 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c0136800 con 0x7f73c01cd2b0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 0x7f73ac03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 4755+0+0 (secure 0 0 0) 0x7f73b8050a40 con 0x7f73c01cd2b0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 0x7f73ac03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.952+0000 7f73bf7fe640 10 monclient: get_auth_request con 0x7f73ac03bb60 auth_method 0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.956+0000 7f73bf7fe640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 0x7f73ac03e020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f73b4000c10 tx=0x7f73b4004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.960+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f73b8025030 con 0x7f73c01cd2b0 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.960+0000 7f73a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:11.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:11.960+0000 7f73a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:12.004 INFO:tasks.rados.rados.0.smithi203.stdout:246: done (1 left) 2024-09-17T15:29:12.004 INFO:tasks.rados.rados.0.smithi203.stdout:247: done (0 left) 2024-09-17T15:29:12.004 INFO:tasks.rados.rados.0.smithi203.stdout:248: delete oid 22 current snap is 25 2024-09-17T15:29:12.014 INFO:tasks.rados.rados.0.smithi203.stdout:248: done (0 left) 2024-09-17T15:29:12.014 INFO:tasks.rados.rados.0.smithi203.stdout:249: rollback oid 37 current snap is 25 2024-09-17T15:29:12.014 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 23 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:250: rollback oid 46 current snap is 25 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 23 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:251: copy_from oid 4 from oid 3 current snap is 25 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:252: delete oid 38 current snap is 25 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:249: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:29:12.015 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 0 (ObjNum 129 snap 0 seq_num 3607097256) dirty dne 2024-09-17T15:29:12.024 INFO:tasks.rados.rados.0.smithi203.stdout:250: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:250: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:250: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 68 (ObjNum 80 snap 6 seq_num 80) dirty exists 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:249: done (3 left) 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:250: done (2 left) 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:252: done (1 left) 2024-09-17T15:29:12.026 INFO:tasks.rados.rados.0.smithi203.stdout:253: snap_remove snap 23 2024-09-17T15:29:12.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:12.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:12.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:12.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f73c0127970 con 0x7f73c01cd2b0 2024-09-17T15:29:12.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73a7fff640 1 -- 172.21.15.86:0/43889794 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f73c01d1290 con 0x7f73c01cd2b0 2024-09-17T15:29:12.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:12.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:12.216 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":32,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 msgr2=0x7f73ac03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 0x7f73ac03e020 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f73b4000c10 tx=0x7f73b4004740 comp rx=0 tx=0).stop 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 10 monclient: shutdown 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 msgr2=0x7f73c01cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 0x7f73c01cd6d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f73c0069970 tx=0x7f73b8004700 comp rx=0 tx=0).stop 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 shutdown_connections 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73ac03bb60 0x7f73ac03e020 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73c01cd2b0 0x7f73c01cd6d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73c0134fb0 0x7f73c0135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 --2- 172.21.15.86:0/43889794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73a8003b20 0x7f73c01d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.212+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 >> 172.21.15.86:0/43889794 conn(0x7f73c00a6440 msgr2=0x7f73c0072560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.216+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 shutdown_connections 2024-09-17T15:29:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:12.216+0000 7f73c5d25640 1 -- 172.21.15.86:0/43889794 wait complete. 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:253: done (1 left) 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:254: rollback oid 19 current snap is 25 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 22 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:251: finishing copy_from to smithi20328086-4 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:251: got expected ENOENT (src dne) 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:251: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:29:12.401 INFO:tasks.rados.rados.0.smithi203.stdout:251: done (1 left) 2024-09-17T15:29:12.402 INFO:tasks.rados.rados.0.smithi203.stdout:255: read oid 42 snap -1 2024-09-17T15:29:12.402 INFO:tasks.rados.rados.0.smithi203.stdout:255: expect (ObjNum 93 snap 13 seq_num 93) 2024-09-17T15:29:12.402 INFO:tasks.rados.rados.0.smithi203.stdout:256: snap_create 2024-09-17T15:29:12.416 INFO:tasks.rados.rados.0.smithi203.stdout:254: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:29:12.416 INFO:tasks.rados.rados.0.smithi203.stdout:254: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:29:12.417 INFO:tasks.rados.rados.0.smithi203.stdout:254: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:29:12.417 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 71 (ObjNum 105 snap 18 seq_num 105) dirty exists 2024-09-17T15:29:12.418 INFO:tasks.rados.rados.0.smithi203.stdout:255: expect (ObjNum 93 snap 13 seq_num 93) 2024-09-17T15:29:12.523 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:12.599 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2436728 2024-09-17T15:29:15.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: get_monmap_and_config 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: build_initial_monmap 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: monmap: 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:15.369249+0000 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:15.369249+0000 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:15.369 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 Processor -- start 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 -- start start 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: init 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd79c0689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:15.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd79c139f00 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd79c09d770 con 0x7fd79c139f00 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd79c09ed00 con 0x7fd79c0689d0 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd79c09efe0 con 0x7fd79c068db0 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a2275640 10 monclient(hunting): authenticate will time out at 1088.335251s 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a0812640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd79c139f00 0x7fd79c13c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37282/0 (socket says 172.21.15.86:37282) 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd7a0812640 1 -- 172.21.15.86:0/2342777759 learned_addr learned my addr 172.21.15.86:0/2342777759 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c068db0 auth_method 0 2024-09-17T15:29:15.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.364+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:15.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.368+0000 7fd7a0812640 1 -- 172.21.15.86:0/2342777759 >> v1:172.21.15.86:6789/0 conn(0x7fd79c139f00 legacy=0x7fd79c13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:15.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.368+0000 7fd79bfff640 1 -- 172.21.15.86:0/2342777759 >> v1:172.21.15.203:6789/0 conn(0x7fd79c0689d0 legacy=0x7fd79c09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:15.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:15.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c068db0 auth_method 0 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:15.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.568+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c068db0 auth_method 0 2024-09-17T15:29:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:15.972+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:16.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c068db0 auth_method 0 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:16.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:16.776+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:17.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4261187885 0 0) 0x7fd79c09ed00 con 0x7fd79c0689d0 2024-09-17T15:29:17.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): _negotiate 2024-09-17T15:29:17.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:17.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:17.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): my global_id is 14179 2024-09-17T15:29:17.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd784003530 con 0x7fd79c0689d0 2024-09-17T15:29:17.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 104523493 0 0) 0x7fd784003530 con 0x7fd79c0689d0 2024-09-17T15:29:17.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 msgr2=0x7fd79c0691c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:17.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 --2- 172.21.15.86:0/2342777759 >> v2:172.21.15.203:3300/0 conn(0x7fd79c068db0 0x7fd79c0691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 >> v1:172.21.15.86:6789/0 conn(0x7fd79c139f00 legacy=0x7fd79c13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _finish_hunting 0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c069700 con 0x7fd79c0689d0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _finish_auth 0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _check_auth_tickets 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fd78c001d80 con 0x7fd79c0689d0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: dump: 2024-09-17T15:29:17.997 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 >> v1:172.21.15.203:6789/0 conn(0x7fd79c0689d0 legacy=0x7fd79c09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:17.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd784003b20 0x7fd784003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): picked mon.a con 0x7fd784003b20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd784004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): picked mon.b con 0x7fd784004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7840099c0 0x7fd78400bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.988+0000 7fd79affd640 10 monclient(hunting): picked mon.c con 0x7fd7840099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c09efe0 con 0x7fd784003b20 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd78400c3d0 con 0x7fd784004810 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd78400c5d0 con 0x7fd7840099c0 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79affd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd784003b20 0x7fd784003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd784004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7840099c0 0x7fd78400bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd784004810 auth_method 0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): get_auth_request con 0x7fd7840099c0 auth_method 0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): get_auth_request con 0x7fd784003b20 auth_method 0 2024-09-17T15:29:18.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient(hunting): handle_auth_done global_id 14179 payload 293 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7840099c0 msgr2=0x7fd78400bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:18.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7840099c0 0x7fd78400bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd784003b20 msgr2=0x7fd784003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd784003b20 0x7fd784003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 monclient: found mon.b 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd784003530 con 0x7fd784004810 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd79b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a0812640 1 -- 172.21.15.86:0/2342777759 reap_dead start 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a2275640 5 monclient: authenticate success, global_id 14179 2024-09-17T15:29:18.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:17.992+0000 7fd7a2275640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:18.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79b7fe640 1 -- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 msgr2=0x7fd784004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:18.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd784004c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd790013c10 tx=0x7fd79001f390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1db21480b634108 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:18.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7900b4070 con 0x7fd784004810 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: dump: 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:18.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7900ab050 con 0x7fd784004810 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 4 set_mon_vals no callback set 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 1 -- 172.21.15.86:0/2342777759 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd790013890 con 0x7fd784004810 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd79affd640 10 monclient: dump: 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.006 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:18.007 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:18.007 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:18.007 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:18.007 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:18.007 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 10 monclient: shutdown 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 -- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 msgr2=0x7fd784004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd784004c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd790013c10 tx=0x7fd79001f390 comp rx=0 tx=0).stop 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 -- 172.21.15.86:0/2342777759 shutdown_connections 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 --2- 172.21.15.86:0/2342777759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd784004c90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 -- 172.21.15.86:0/2342777759 >> 172.21.15.86:0/2342777759 conn(0x7fd79c0a6440 msgr2=0x7fd79c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 -- 172.21.15.86:0/2342777759 shutdown_connections 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.000+0000 7fd7a2275640 1 -- 172.21.15.86:0/2342777759 wait complete. 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient: build_initial_monmap 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient: monmap: 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:18.009 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:18.010 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:18.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 Processor -- start 2024-09-17T15:29:18.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 -- start start 2024-09-17T15:29:18.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient: init 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd79c133a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-b con 0x7fd784004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c121be0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): picked mon.noname-a con 0x7fd79c11f5c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c1355e0 con 0x7fd79c11f5c0 2024-09-17T15:29:18.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c135460 con 0x7fd784004810 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c135760 con 0x7fd79c121be0 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a2275640 10 monclient(hunting): authenticate will time out at 1090.974554s 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33624/0 (socket says 172.21.15.86:33624) 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 -- 172.21.15.86:0/185699510 learned_addr learned my addr 172.21.15.86:0/185699510 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request con 0x7fd79c121be0 auth_method 0 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): get_auth_request con 0x7fd79c11f5c0 auth_method 0 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd79c133a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 msgr2=0x7fd79c133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 msgr2=0x7fd79c133f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): get_auth_request con 0x7fd784004810 auth_method 0 2024-09-17T15:29:18.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79b7fe640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd79bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 msgr2=0x7fd79c133f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 msgr2=0x7fd79c133a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd79c133a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient: _finish_hunting 0 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 monclient: found mon.noname-a 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.004+0000 7fd7a0812640 1 -- 172.21.15.86:0/185699510 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c0a1680 con 0x7fd79c11f5c0 2024-09-17T15:29:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a0812640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 msgr2=0x7fd79c11fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a0812640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c10 tx=0x7fd794007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794011790 con 0x7fd79c11f5c0 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: dump: 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:18.018 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: _finish_auth 0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794018070 con 0x7fd79c11f5c0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 5 monclient: authenticate success, global_id 4482 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _renew_subs 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c134480 con 0x7fd79c11f5c0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794011bb0 con 0x7fd79c11f5c0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 10 monclient: dump: 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd79a7fc640 4 set_mon_vals no callback set 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _renew_subs 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c1233e0 con 0x7fd79c11f5c0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd79401d460 con 0x7fd79c11f5c0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c09ed00 con 0x7fd79c11f5c0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 0x7fd78803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 4811+0+0 (secure 0 0 0) 0x7fd79404fa40 con 0x7fd79c11f5c0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd79bfff640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 0x7fd78803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.008+0000 7fd79bfff640 10 monclient: get_auth_request con 0x7fd78803bbb0 auth_method 0 2024-09-17T15:29:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.012+0000 7fd79bfff640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 0x7fd78803e070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0014a0 tx=0x7fd78c007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:18.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.016+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd794022020 con 0x7fd79c11f5c0 2024-09-17T15:29:18.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.016+0000 7fd798ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:18.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.016+0000 7fd798ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:18.055 INFO:tasks.rados.rados.0.smithi203.stdout:254: done (2 left) 2024-09-17T15:29:18.055 INFO:tasks.rados.rados.0.smithi203.stdout:255: done (1 left) 2024-09-17T15:29:18.055 INFO:tasks.rados.rados.0.smithi203.stdout:256: done (0 left) 2024-09-17T15:29:18.055 INFO:tasks.rados.rados.0.smithi203.stdout:257: snap_remove snap 22 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.252+0000 7fd7a2275640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.252+0000 7fd7a2275640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd79c06d4f0 con 0x7fd79c11f5c0 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd798ff9640 1 -- 172.21.15.86:0/185699510 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fd79404e420 con 0x7fd79c11f5c0 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd798ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:18.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd798ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:18.260 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 msgr2=0x7fd78803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 0x7fd78803e070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0014a0 tx=0x7fd78c007e40 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 10 monclient: shutdown 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 msgr2=0x7fd79c11fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd794000c10 tx=0x7fd794007430 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 shutdown_connections 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd78803bbb0 0x7fd78803e070 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd79c11f5c0 0x7fd79c11fa00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd79c121be0 0x7fd79c133f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 --2- 172.21.15.86:0/185699510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd784004810 0x7fd79c133a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:18.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 >> 172.21.15.86:0/185699510 conn(0x7fd79c0a6440 msgr2=0x7fd79c138d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:18.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 shutdown_connections 2024-09-17T15:29:18.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:18.256+0000 7fd7a2275640 1 -- 172.21.15.86:0/185699510 wait complete. 2024-09-17T15:29:19.859 INFO:tasks.rados.rados.0.smithi203.stdout:257: done (0 left) 2024-09-17T15:29:19.860 INFO:tasks.rados.rados.0.smithi203.stdout:258: delete oid 28 current snap is 26 2024-09-17T15:29:19.891 INFO:tasks.rados.rados.0.smithi203.stdout:258: done (0 left) 2024-09-17T15:29:19.891 INFO:tasks.rados.rados.0.smithi203.stdout:259: read oid 20 snap -1 2024-09-17T15:29:19.891 INFO:tasks.rados.rados.0.smithi203.stdout:259: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:29:19.891 INFO:tasks.rados.rados.0.smithi203.stdout:260: write (excl) oid 4 current snap is 26 2024-09-17T15:29:19.892 INFO:tasks.rados.rados.0.smithi203.stdout:260: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-17T15:29:19.909 INFO:tasks.rados.rados.0.smithi203.stdout:260: writing smithi20328086-4 from 450454 to 1092480 tid 1 2024-09-17T15:29:19.925 INFO:tasks.rados.rados.0.smithi203.stdout:260: writing smithi20328086-4 from 1505862 to 2050454 tid 2 2024-09-17T15:29:19.926 INFO:tasks.rados.rados.0.smithi203.stdout:261: read oid 25 snap -1 2024-09-17T15:29:19.926 INFO:tasks.rados.rados.0.smithi203.stdout:261: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:29:19.926 INFO:tasks.rados.rados.0.smithi203.stdout:262: write oid 24 current snap is 26 2024-09-17T15:29:19.926 INFO:tasks.rados.rados.0.smithi203.stdout:262: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-17T15:29:19.943 INFO:tasks.rados.rados.0.smithi203.stdout:262: writing smithi20328086-24 from 467261 to 1184536 tid 1 2024-09-17T15:29:19.954 INFO:tasks.rados.rados.0.smithi203.stdout:262: writing smithi20328086-24 from 1847991 to 2067261 tid 2 2024-09-17T15:29:19.954 INFO:tasks.rados.rados.0.smithi203.stdout:263: read oid 13 snap -1 2024-09-17T15:29:19.954 INFO:tasks.rados.rados.0.smithi203.stdout:263: expect (ObjNum 62 snap 2 seq_num 62) 2024-09-17T15:29:19.954 INFO:tasks.rados.rados.0.smithi203.stdout:264: write oid 10 current snap is 26 2024-09-17T15:29:19.954 INFO:tasks.rados.rados.0.smithi203.stdout:264: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-17T15:29:19.967 INFO:tasks.rados.rados.0.smithi203.stdout:264: writing smithi20328086-10 from 484068 to 1276592 tid 1 2024-09-17T15:29:19.975 INFO:tasks.rados.rados.0.smithi203.stdout:264: writing smithi20328086-10 from 1906473 to 2084068 tid 2 2024-09-17T15:29:19.975 INFO:tasks.rados.rados.0.smithi203.stdout:265: delete oid 37 current snap is 26 2024-09-17T15:29:19.975 INFO:tasks.rados.rados.0.smithi203.stdout:259: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:29:20.065 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:20.092 INFO:tasks.rados.rados.0.smithi203.stdout:260: finishing write tid 1 to smithi20328086-4 2024-09-17T15:29:20.092 INFO:tasks.rados.rados.0.smithi203.stdout:260: finishing write tid 2 to smithi20328086-4 2024-09-17T15:29:20.093 INFO:tasks.rados.rados.0.smithi203.stdout:261: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:29:20.117 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:20.159 INFO:tasks.rados.rados.0.smithi203.stdout:260: finishing write tid 3 to smithi20328086-4 2024-09-17T15:29:20.159 INFO:tasks.rados.rados.0.smithi203.stdout:260: finishing write tid 4 to smithi20328086-4 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:260: oid 4 updating version 0 to 37 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:260: oid 4 updating version 37 to 39 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:260: oid 4 version 39 is already newer than 38 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 39 (ObjNum 122 snap 26 seq_num 122) dirty exists 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:260: left oid 4 (ObjNum 122 snap 26 seq_num 122) 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: finishing write tid 1 to smithi20328086-24 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: finishing write tid 2 to smithi20328086-24 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: finishing write tid 3 to smithi20328086-24 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: finishing write tid 4 to smithi20328086-24 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: oid 24 updating version 0 to 56 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: oid 24 version 56 is already newer than 55 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: oid 24 updating version 56 to 57 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 57 (ObjNum 123 snap 26 seq_num 123) dirty exists 2024-09-17T15:29:20.160 INFO:tasks.rados.rados.0.smithi203.stdout:262: left oid 24 (ObjNum 123 snap 26 seq_num 123) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:259: done (6 left) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:260: done (5 left) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:261: done (4 left) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:262: done (3 left) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:265: done (2 left) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:266: read oid 34 snap 20 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:266: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:29:20.161 INFO:tasks.rados.rados.0.smithi203.stdout:267: snap_remove snap 25 2024-09-17T15:29:20.877 INFO:tasks.rados.rados.0.smithi203.stdout:267: done (3 left) 2024-09-17T15:29:20.877 INFO:tasks.rados.rados.0.smithi203.stdout:268: snap_remove snap 21 2024-09-17T15:29:20.877 INFO:tasks.rados.rados.0.smithi203.stdout:263: expect (ObjNum 62 snap 2 seq_num 62) 2024-09-17T15:29:20.877 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:20.923 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 426510 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: finishing write tid 1 to smithi20328086-10 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: finishing write tid 2 to smithi20328086-10 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: finishing write tid 3 to smithi20328086-10 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: finishing write tid 4 to smithi20328086-10 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: oid 10 updating version 0 to 36 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: oid 10 version 36 is already newer than 35 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: oid 10 updating version 36 to 37 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 37 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:264: left oid 10 (ObjNum 124 snap 26 seq_num 124) 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:266: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-17T15:29:21.010 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:21.034 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:21.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: get_monmap_and_config 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: monmap: 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:21.408968+0000 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:21.408968+0000 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:21.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:21.409 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 Processor -- start 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 -- start start 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: init 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87e80689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87e80694c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f87e806f840 0x7f87e813e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-c con 0x7f87e806f840 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e8074470 con 0x7f87e80689d0 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e809c8b0 con 0x7f87e80694c0 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f87e809cb90 con 0x7f87e806f840 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87ecaa2640 10 monclient(hunting): authenticate will time out at 1094.374610s 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e659c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f87e80689d0 0x7f87e8068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46182/0 (socket says 172.21.15.86:46182) 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e659c640 1 -- 172.21.15.86:0/2977972168 learned_addr learned my addr 172.21.15.86:0/2977972168 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 1 --2- 172.21.15.86:0/2977972168 >> v2:172.21.15.203:3300/0 conn(0x7f87e806f840 0x7f87e813e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): get_auth_request con 0x7f87e806f840 auth_method 0 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_done global_id 14185 payload 274 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 1 -- 172.21.15.86:0/2977972168 >> v1:172.21.15.203:6789/0 conn(0x7f87e80694c0 legacy=0x7f87e813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 1 -- 172.21.15.86:0/2977972168 >> v1:172.21.15.86:6789/0 conn(0x7f87e80689d0 legacy=0x7f87e8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 1 monclient: found mon.noname-c 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:29:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.404+0000 7f87e6d9d640 1 -- 172.21.15.86:0/2977972168 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e809ce70 con 0x7f87e806f840 2024-09-17T15:29:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 -- 172.21.15.86:0/2977972168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e806f840 msgr2=0x7f87e813e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 --2- 172.21.15.86:0/2977972168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e806f840 0x7f87e813e720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009d90 tx=0x7f87e001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfd316adf7a26a40 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 1 -- 172.21.15.86:0/2977972168 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00b2070 con 0x7f87e806f840 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: dump: 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 1 -- 172.21.15.86:0/2977972168 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e00a5970 con 0x7f87e806f840 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 1 -- 172.21.15.86:0/2977972168 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e809c2d0 con 0x7f87e806f840 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e559a640 10 monclient: dump: 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.417 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 5 monclient: authenticate success, global_id 14185 2024-09-17T15:29:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 4 set_mon_vals no callback set 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: shutdown 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- 172.21.15.86:0/2977972168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e806f840 msgr2=0x7f87e813e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/2977972168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e806f840 0x7f87e813e720 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87e0009d90 tx=0x7f87e001c830 comp rx=0 tx=0).stop 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- 172.21.15.86:0/2977972168 shutdown_connections 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/2977972168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e806f840 0x7f87e813e720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- 172.21.15.86:0/2977972168 >> 172.21.15.86:0/2977972168 conn(0x7f87e80a6440 msgr2=0x7f87e80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- 172.21.15.86:0/2977972168 shutdown_connections 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- 172.21.15.86:0/2977972168 wait complete. 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: monmap: 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 Processor -- start 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- start start 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: init 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 0x7f87e81c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-a con 0x7f87e80689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e80694c0 0x7f87e81c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-b con 0x7f87e80694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): picked mon.noname-c con 0x7f87e806f840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87e809dbc0 con 0x7f87e80689d0 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87e809da40 con 0x7f87e80694c0 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f87e809dd40 con 0x7f87e806f840 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87ecaa2640 10 monclient(hunting): authenticate will time out at 1094.382975s 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 0x7f87e81c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46280/0 (socket says 172.21.15.86:46280) 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 -- 172.21.15.86:0/3667936237 learned_addr learned my addr 172.21.15.86:0/3667936237 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): get_auth_request con 0x7f87e806f840 auth_method 0 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): get_auth_request con 0x7f87e80689d0 auth_method 0 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient(hunting): handle_auth_done global_id 24101 payload 824 2024-09-17T15:29:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e5d9b640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e80694c0 0x7f87e81c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 -- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e80694c0 msgr2=0x7f87e81c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e80694c0 0x7f87e81c8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 -- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 msgr2=0x7f87e81c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 0x7f87e81c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 monclient: found mon.noname-c 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e6d9d640 1 -- 172.21.15.86:0/3667936237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e8073a50 con 0x7f87e806f840 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.412+0000 7f87e659c640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 0x7f87e81c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87e6d9d640 1 -- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 msgr2=0x7f87e81c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87e6d9d640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b0040 tx=0x7f87e00195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00bd020 con 0x7f87e806f840 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: dump: 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87e00b8880 con 0x7f87e806f840 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 5 monclient: authenticate success, global_id 24101 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _renew_subs 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e81365e0 con 0x7f87e806f840 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87e00b2040 con 0x7f87e806f840 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 10 monclient: dump: 2024-09-17T15:29:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87e4d99640 4 set_mon_vals no callback set 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _renew_subs 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e809d460 con 0x7f87e806f840 2024-09-17T15:29:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f87e00c0020 con 0x7f87e806f840 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e8068db0 con 0x7f87e806f840 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 0x7f87c803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 4783+0+0 (secure 0 0 0) 0x7f87e00eb000 con 0x7f87e806f840 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87e659c640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 0x7f87c803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.416+0000 7f87e659c640 10 monclient: get_auth_request con 0x7f87c803bb60 auth_method 0 2024-09-17T15:29:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.420+0000 7f87e659c640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 0x7f87c803e020 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87dc007ca0 tx=0x7f87dc007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.424+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f87e00c7030 con 0x7f87e806f840 2024-09-17T15:29:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.424+0000 7f87d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.424+0000 7f87d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87ecaa2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87ecaa2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87ecaa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87e806d4f0 con 0x7f87e806f840 2024-09-17T15:29:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87d6ffd640 1 -- 172.21.15.86:0/3667936237 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f87e8073a50 con 0x7f87e806f840 2024-09-17T15:29:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.652+0000 7f87d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:21.659 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 msgr2=0x7f87c803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 0x7f87c803e020 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87dc007ca0 tx=0x7f87dc007bf0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 10 monclient: shutdown 2024-09-17T15:29:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 msgr2=0x7f87e81c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b0040 tx=0x7f87e00195d0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 shutdown_connections 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f87c803bb60 0x7f87c803e020 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87e806f840 0x7f87e81c8da0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87e80694c0 0x7f87e81c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 --2- 172.21.15.86:0/3667936237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87e80689d0 0x7f87e81c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 >> 172.21.15.86:0/3667936237 conn(0x7f87e80a6440 msgr2=0x7f87e812fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 shutdown_connections 2024-09-17T15:29:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.656+0000 7f87ecaa2640 1 -- 172.21.15.86:0/3667936237 wait complete. 2024-09-17T15:29:21.676 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:29:21.676 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:29:21.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:29:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: get_monmap_and_config 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: monmap: 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:21.784964+0000 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:21.784964+0000 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:21.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:21.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:21.785 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:21.785 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:21.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 Processor -- start 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 -- start start 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: init 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e80a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 0x7fa8e80a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e80a5350 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8e8137d30 0x7fa8e81381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e8137d30 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e809d430 con 0x7fa8e8137d30 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa8e80a68d0 con 0x7fa8e80a5350 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e80a64a0 con 0x7fa8e80a4f70 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ef54a640 10 monclient(hunting): authenticate will time out at 1094.750748s 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8e8137d30 0x7fa8e81381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8e8137d30 0x7fa8e81381a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56478/0 (socket says 172.21.15.86:56478) 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46286/0 (socket says 172.21.15.86:46286) 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 -- 172.21.15.86:0/4235369033 learned_addr learned my addr 172.21.15.86:0/4235369033 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): get_auth_request con 0x7fa8e8137d30 auth_method 0 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): get_auth_request con 0x7fa8e80a4f70 auth_method 0 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/4235369033 >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 0x7fa8e80a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 -- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 msgr2=0x7fa8e809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 -- 172.21.15.86:0/4235369033 >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 msgr2=0x7fa8e80a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 --2- 172.21.15.86:0/4235369033 >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 0x7fa8e80a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 monclient: found mon.noname-a 2024-09-17T15:29:21.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8edae7640 1 -- 172.21.15.86:0/4235369033 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e809c230 con 0x7fa8e8137d30 2024-09-17T15:29:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ecae5640 10 monclient: get_auth_request con 0x7fa8e80a5350 auth_method 0 2024-09-17T15:29:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/4235369033 >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 0x7fa8e80a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:29:21.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.780+0000 7fa8ed2e6640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8edae7640 1 -- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8137d30 msgr2=0x7fa8e81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8edae7640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8137d30 0x7fa8e81381a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8069c50 tx=0x7fa8e001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21185ac1077ba409 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 1 -- 172.21.15.86:0/4235369033 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00a5690 con 0x7fa8e8137d30 2024-09-17T15:29:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: dump: 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8ef54a640 5 monclient: authenticate success, global_id 4488 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 1 -- 172.21.15.86:0/4235369033 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e00a5830 con 0x7fa8e8137d30 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8ef54a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8ef54a640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8ef54a640 4 set_mon_vals no callback set 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 1 -- 172.21.15.86:0/4235369033 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e00a5c00 con 0x7fa8e8137d30 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.784+0000 7fa8dffff640 10 monclient: dump: 2024-09-17T15:29:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: shutdown 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8137d30 msgr2=0x7fa8e81381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8137d30 0x7fa8e81381a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8069c50 tx=0x7fa8e001c940 comp rx=0 tx=0).stop 2024-09-17T15:29:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- 172.21.15.86:0/4235369033 shutdown_connections 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e8137d30 0x7fa8e81381a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/4235369033 >> v2:172.21.15.203:3300/0 conn(0x7fa8e80a5350 0x7fa8e80a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/4235369033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e80a4f70 0x7fa8e809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- 172.21.15.86:0/4235369033 >> 172.21.15.86:0/4235369033 conn(0x7fa8e80a7240 msgr2=0x7fa8e80a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- 172.21.15.86:0/4235369033 shutdown_connections 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- 172.21.15.86:0/4235369033 wait complete. 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: monmap: 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 Processor -- start 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- start start 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: init 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 0x7fa8e81cfe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-a con 0x7fa8e80a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8e80a5350 0x7fa8e8137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-b con 0x7fa8e80a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8137d30 0x7fa8e8137550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): picked mon.noname-c con 0x7fa8e8137d30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e806f9c0 con 0x7fa8e80a4f70 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e806f840 con 0x7fa8e80a5350 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa8e81419b0 con 0x7fa8e8137d30 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ef54a640 10 monclient(hunting): authenticate will time out at 1094.757718s 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 0x7fa8e81cfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8137d30 0x7fa8e8137550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 0x7fa8e81cfe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56488/0 (socket says 172.21.15.86:56488) 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 -- 172.21.15.86:0/1037605638 learned_addr learned my addr 172.21.15.86:0/1037605638 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): get_auth_request con 0x7fa8e80a4f70 auth_method 0 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): get_auth_request con 0x7fa8e8137d30 auth_method 0 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8edae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient(hunting): handle_auth_done global_id 4491 payload 824 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8137d30 msgr2=0x7fa8e8137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8e8137d30 0x7fa8e8137550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8e80a5350 msgr2=0x7fa8e8137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8e80a5350 0x7fa8e8137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 monclient: found mon.noname-a 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.788+0000 7fa8ed2e6640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e81229d0 con 0x7fa8e80a4f70 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ed2e6640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 msgr2=0x7fa8e81cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ed2e6640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 0x7fa8e81cfe50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800ebe0 tx=0x7fa8d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8017070 con 0x7fa8e80a4f70 2024-09-17T15:29:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: dump: 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d800ce40 con 0x7fa8e80a4f70 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 5 monclient: authenticate success, global_id 4491 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 monclient: _renew_subs 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e806ef50 con 0x7fa8e80a4f70 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8013690 con 0x7fa8e80a4f70 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 10 monclient: dump: 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8df7fe640 4 set_mon_vals no callback set 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 monclient: _renew_subs 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e81226f0 con 0x7fa8e80a4f70 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 0x7fa8e813d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e80a5750 con 0x7fa8e813d130 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 0x7fa8e813d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient: get_auth_request con 0x7fa8e813d130 auth_method 0 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa8d801a070 con 0x7fa8e80a4f70 2024-09-17T15:29:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_done global_id 4494 payload 824 2024-09-17T15:29:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8e80a5ff0 con 0x7fa8e813d130 2024-09-17T15:29:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8cc03bb60 0x7fa8cc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4783+0+0 (secure 0 0 0) 0x7fa8d804ffa0 con 0x7fa8e80a4f70 2024-09-17T15:29:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8edae7640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8cc03bb60 0x7fa8cc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8edae7640 10 monclient: get_auth_request con 0x7fa8cc03bb60 auth_method 0 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 msgr2=0x7fa8e813d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.792+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 0x7fa8e813d510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8e809dcf0 tx=0x7fa8e400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4015070 con 0x7fa8e813d130 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 10 monclient: dump: 2024-09-17T15:29:21.812 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa8e4007a70 con 0x7fa8e813d130 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:21.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 msgr2=0x7fa8e813d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 0x7fa8e813d510 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8e809dcf0 tx=0x7fa8e400cb80 comp rx=0 tx=0).stop 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 msgr2=0x7fa8e813d510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8ddffb640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813d130 0x7fa8e813d510 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa8e809dcf0 tx=0x7fa8e400cb80 comp rx=0 tx=0).stop 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.796+0000 7fa8edae7640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8cc03bb60 0x7fa8cc03e020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8e809e420 tx=0x7fa8e00a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ef54a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 0x7fa8e8139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ef54a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e813de80 con 0x7fa8e813da50 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 0x7fa8e8139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient: get_auth_request con 0x7fa8e813da50 auth_method 0 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.800+0000 7fa8ecae5640 1 -- 172.21.15.86:0/1037605638 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8e81211d0 con 0x7fa8e813da50 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ecae5640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 msgr2=0x7fa8e8139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ecae5640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 0x7fa8e8139160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4006b20 tx=0x7fa8e40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4028020 con 0x7fa8e813da50 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 10 monclient: dump: 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.815 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fa8e40113f0 con 0x7fa8e813da50 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 msgr2=0x7fa8e8139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 0x7fa8e8139160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4006b20 tx=0x7fa8e40046d0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 msgr2=0x7fa8e8139160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ddffb640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e813da50 0x7fa8e8139160 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4006b20 tx=0x7fa8e40046d0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 134448, 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:21.817 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.818 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.819 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:21.820 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.821 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.822 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.823 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.825 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:21.826 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:21.829 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8cc03bb60 msgr2=0x7fa8cc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8cc03bb60 0x7fa8cc03e020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8e809e420 tx=0x7fa8e00a7070 comp rx=0 tx=0).stop 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 10 monclient: shutdown 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 msgr2=0x7fa8e81cfe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 --2- 172.21.15.86:0/1037605638 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8e80a4f70 0x7fa8e81cfe50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800ebe0 tx=0x7fa8d800c520 comp rx=0 tx=0).stop 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8edae7640 1 -- 172.21.15.86:0/1037605638 reap_dead start 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8edae7640 1 -- 172.21.15.86:0/1037605638 reap_dead start 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 shutdown_connections 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 >> 172.21.15.86:0/1037605638 conn(0x7fa8e80a7240 msgr2=0x7fa8e80a9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 shutdown_connections 2024-09-17T15:29:21.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.804+0000 7fa8ef54a640 1 -- 172.21.15.86:0/1037605638 wait complete. 2024-09-17T15:29:21.832 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:21.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:29:21.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:263: done (3 left) 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:264: done (2 left) 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:266: done (1 left) 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:268: done (0 left) 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:269: write oid 42 current snap is 26 2024-09-17T15:29:21.887 INFO:tasks.rados.rados.0.smithi203.stdout:269: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-17T15:29:21.905 INFO:tasks.rados.rados.0.smithi203.stdout:269: writing smithi20328086-42 from 517682 to 1060704 tid 1 2024-09-17T15:29:21.923 INFO:tasks.rados.rados.0.smithi203.stdout:269: writing smithi20328086-42 from 1623437 to 2117682 tid 2 2024-09-17T15:29:21.923 INFO:tasks.rados.rados.0.smithi203.stdout:270: snap_create 2024-09-17T15:29:21.923 INFO:tasks.rados.rados.0.smithi203.stdout:269: finishing write tid 1 to smithi20328086-42 2024-09-17T15:29:21.929 INFO:tasks.rados.rados.0.smithi203.stdout:269: finishing write tid 2 to smithi20328086-42 2024-09-17T15:29:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:269: finishing write tid 3 to smithi20328086-42 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:269: finishing write tid 4 to smithi20328086-42 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:269: oid 42 updating version 0 to 82 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:269: oid 42 version 82 is already newer than 81 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:269: oid 42 version 82 is already newer than 80 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 82 (ObjNum 126 snap 26 seq_num 126) dirty exists 2024-09-17T15:29:21.931 INFO:tasks.rados.rados.0.smithi203.stdout:269: left oid 42 (ObjNum 126 snap 26 seq_num 126) 2024-09-17T15:29:21.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 10 monclient: get_monmap_and_config 2024-09-17T15:29:21.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 10 monclient: monmap: 2024-09-17T15:29:21.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:21.950835+0000 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:21.950835+0000 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:21.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 1 Processor -- start 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 1 -- start start 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.944+0000 7f316262a640 10 monclient: init 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f315c0a4f70 0x7f315c09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): picked mon.noname-a con 0x7f315c0a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f315c0a5350 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 0x7f315c13f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): picked mon.noname-b con 0x7f315c13d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315c09d430 con 0x7f315c0a5350 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f315c09c7f0 con 0x7f315c0a4f70 2024-09-17T15:29:21.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f315c0a68d0 con 0x7f315c13d180 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f316262a640 10 monclient(hunting): authenticate will time out at 1094.916206s 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f315c0a4f70 0x7f315c09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f315c0a4f70 0x7f315c09cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56508/0 (socket says 172.21.15.86:56508) 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 0x7f315c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 -- 172.21.15.86:0/3383853060 learned_addr learned my addr 172.21.15.86:0/3383853060 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): get_auth_request con 0x7f315c0a4f70 auth_method 0 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315b7fe640 1 --1- 172.21.15.86:0/3383853060 >> v1:172.21.15.86:6789/0 conn(0x7f315c0a5350 0x7f315c0a5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46192/0 (socket says 172.21.15.86:46192) 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): get_auth_request con 0x7f315c13d180 auth_method 0 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient(hunting): handle_auth_done global_id 4500 payload 274 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 -- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 msgr2=0x7f315c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 0x7f315c13f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 -- 172.21.15.86:0/3383853060 >> v1:172.21.15.86:6789/0 conn(0x7f315c0a5350 legacy=0x7f315c0a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 monclient: found mon.noname-a 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f315bfff640 1 -- 172.21.15.86:0/3383853060 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c0a64a0 con 0x7f315c0a4f70 2024-09-17T15:29:21.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.948+0000 7f3160bc7640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 0x7f315c13f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 -- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a4f70 msgr2=0x7f315c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a4f70 0x7f315c09cef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f315c06e190 tx=0x7f314c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ade16a749e05be server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 1 -- 172.21.15.86:0/3383853060 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0b0070 con 0x7f315c0a4f70 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: dump: 2024-09-17T15:29:21.959 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 1 -- 172.21.15.86:0/3383853060 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c0ab600 con 0x7f315c0a4f70 2024-09-17T15:29:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 1 -- 172.21.15.86:0/3383853060 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0ab9d0 con 0x7f315c0a4f70 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 5 monclient: authenticate success, global_id 4500 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 4 set_mon_vals no callback set 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315affd640 10 monclient: dump: 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: shutdown 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a4f70 msgr2=0x7f315c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a4f70 0x7f315c09cef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f315c06e190 tx=0x7f314c01c3b0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- 172.21.15.86:0/3383853060 shutdown_connections 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c13d180 0x7f315c13f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- 172.21.15.86:0/3383853060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a4f70 0x7f315c09cef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- 172.21.15.86:0/3383853060 >> 172.21.15.86:0/3383853060 conn(0x7f315c0a7240 msgr2=0x7f315c0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- 172.21.15.86:0/3383853060 shutdown_connections 2024-09-17T15:29:21.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- 172.21.15.86:0/3383853060 wait complete. 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: build_initial_monmap 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: monmap: 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 Processor -- start 2024-09-17T15:29:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- start start 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: init 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 0x7f315c1cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): picked mon.noname-c con 0x7f315c0a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a5350 0x7f315c0a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): picked mon.noname-a con 0x7f315c0a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f315c13d180 0x7f315c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): picked mon.noname-b con 0x7f315c13d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f315c1424e0 con 0x7f315c0a5350 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f315c06f9a0 con 0x7f315c13d180 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f315c142660 con 0x7f315c0a4f70 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f316262a640 10 monclient(hunting): authenticate will time out at 1094.923015s 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 0x7f315c1cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a5350 0x7f315c0a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 0x7f315c1cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46304/0 (socket says 172.21.15.86:46304) 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 -- 172.21.15.86:0/3376411123 learned_addr learned my addr 172.21.15.86:0/3376411123 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): get_auth_request con 0x7f315c0a4f70 auth_method 0 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): get_auth_request con 0x7f315c0a5350 auth_method 0 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient(hunting): handle_auth_done global_id 24113 payload 824 2024-09-17T15:29:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f315c13d180 msgr2=0x7f315c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f315c13d180 0x7f315c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a5350 msgr2=0x7f315c0a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f315c0a5350 0x7f315c0a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 monclient: found mon.noname-c 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.952+0000 7f315bfff640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c0a3840 con 0x7f315c0a4f70 2024-09-17T15:29:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315bfff640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 msgr2=0x7f315c1cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315bfff640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 0x7f315c1cfea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f314c01cc50 tx=0x7f314c019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0b0040 con 0x7f315c0a4f70 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: dump: 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.972 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: _finish_auth 0 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c0aa5b0 con 0x7f315c0a4f70 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 5 monclient: authenticate success, global_id 24113 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 monclient: _renew_subs 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c09f5e0 con 0x7f315c0a4f70 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c0aa980 con 0x7f315c0a4f70 2024-09-17T15:29:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 10 monclient: dump: 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315a7fc640 4 set_mon_vals no callback set 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 monclient: _renew_subs 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c0a43a0 con 0x7f315c0a4f70 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 0x7f315c0a21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f315c13d7b0 con 0x7f315c0a1de0 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 0x7f315c0a21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient: get_auth_request con 0x7f315c0a1de0 auth_method 0 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f314c0aab20 con 0x7f315c0a4f70 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 10 monclient(hunting): handle_auth_done global_id 24116 payload 824 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f315b7fe640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f315c09cef0 con 0x7f315c0a1de0 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(73..73 src has 1..73) ==== 4783+0+0 (secure 0 0 0) 0x7f314c0e9c70 con 0x7f315c0a4f70 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3160bc7640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.956+0000 7f3160bc7640 10 monclient: get_auth_request con 0x7f314803bb60 auth_method 0 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f315b7fe640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 msgr2=0x7f315c0a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f315b7fe640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 0x7f315c0a21c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f315000ebc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150004d60 con 0x7f315c0a1de0 2024-09-17T15:29:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 10 monclient: dump: 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3150019070 con 0x7f315c0a1de0 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 msgr2=0x7f315c0a21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 0x7f315c0a21c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f315000ebc0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 msgr2=0x7f315c0a21c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3158ff9640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a1de0 0x7f315c0a21c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f315000ebc0 comp rx=0 tx=0).stop 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.960+0000 7f3160bc7640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3154009bd0 tx=0x7f315400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f316262a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f316262a640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 0x7f315c139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f316262a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f315c0a1760 con 0x7f315c0a2700 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 0x7f315c139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient: get_auth_request con 0x7f315c0a2700 auth_method 0 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 10 monclient(hunting): handle_auth_done global_id 24119 payload 824 2024-09-17T15:29:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.964+0000 7f315b7fe640 1 -- 172.21.15.86:0/3376411123 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f315c0a5ff0 con 0x7f315c0a2700 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f315b7fe640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 msgr2=0x7f315c139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f315b7fe640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 0x7f315c139160 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f3150004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315001e230 con 0x7f315c0a2700 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 10 monclient: dump: 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f315001e230 con 0x7f315c0a2700 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 msgr2=0x7f315c139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 0x7f315c139160 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f3150004940 comp rx=0 tx=0).stop 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 msgr2=0x7f315c139160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3158ff9640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a2700 0x7f315c139160 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f315000e700 tx=0x7f3150004940 comp rx=0 tx=0).stop 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:21.982 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 26466, 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.983 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.984 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.985 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:21.986 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:21.987 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.988 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.989 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:21.992 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:21.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:21.994 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:21.995 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 msgr2=0x7f314803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3154009bd0 tx=0x7f315400b040 comp rx=0 tx=0).stop 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 10 monclient: shutdown 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 msgr2=0x7f315c1cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 --2- 172.21.15.86:0/3376411123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f315c0a4f70 0x7f315c1cfea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f314c01cc50 tx=0x7f314c019150 comp rx=0 tx=0).stop 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3160bc7640 1 -- 172.21.15.86:0/3376411123 reap_dead start 2024-09-17T15:29:21.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f3160bc7640 1 -- 172.21.15.86:0/3376411123 reap_dead start 2024-09-17T15:29:21.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 shutdown_connections 2024-09-17T15:29:21.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 >> 172.21.15.86:0/3376411123 conn(0x7f315c0a7240 msgr2=0x7f315c13c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:21.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 shutdown_connections 2024-09-17T15:29:21.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:21.968+0000 7f316262a640 1 -- 172.21.15.86:0/3376411123 wait complete. 2024-09-17T15:29:21.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:21.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:29:21.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:29:22.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: get_monmap_and_config 2024-09-17T15:29:22.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: build_initial_monmap 2024-09-17T15:29:22.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: monmap: 2024-09-17T15:29:22.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:22.101542+0000 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:22.101542+0000 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:22.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 Processor -- start 2024-09-17T15:29:22.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 -- start start 2024-09-17T15:29:22.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: init 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9880a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-b con 0x7fb9880a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb98813d180 0x7fb98813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-a con 0x7fb98813d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9880a68d0 con 0x7fb9880a4f70 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb9880a64a0 con 0x7fb98813d180 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb98809c230 con 0x7fb9880a5350 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb98d88f640 10 monclient(hunting): authenticate will time out at 1095.067040s 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb98813d180 0x7fb98813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46346/0 (socket says 172.21.15.86:46346) 2024-09-17T15:29:22.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb98813d180 0x7fb98813f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56522/0 (socket says 172.21.15.86:56522) 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 1 -- 172.21.15.86:0/580393396 learned_addr learned my addr 172.21.15.86:0/580393396 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): get_auth_request con 0x7fb9880a5350 auth_method 0 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): get_auth_request con 0x7fb98813d180 auth_method 0 2024-09-17T15:29:22.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb986ffd640 1 --1- 172.21.15.86:0/580393396 >> v1:172.21.15.86:6789/0 conn(0x7fb9880a4f70 0x7fb98809cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46198/0 (socket says 172.21.15.86:46198) 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient(hunting): handle_auth_done global_id 4506 payload 274 2024-09-17T15:29:22.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 -- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 msgr2=0x7fb9880a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 -- 172.21.15.86:0/580393396 >> v1:172.21.15.86:6789/0 conn(0x7fb9880a4f70 legacy=0x7fb98809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient: _finish_hunting 0 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 monclient: found mon.noname-a 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:22.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:22.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9877fe640 1 -- 172.21.15.86:0/580393396 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98809c510 con 0x7fb98813d180 2024-09-17T15:29:22.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.096+0000 7fb9867fc640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:22.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb9877fe640 1 -- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 msgr2=0x7fb98813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb9877fe640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813f570 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d90 tx=0x7fb98001c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83dd382c51eab94 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 1 -- 172.21.15.86:0/580393396 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9800b2070 con 0x7fb98813d180 2024-09-17T15:29:22.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:22.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: dump: 2024-09-17T15:29:22.125 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.125 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.126 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.127 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.127 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: _finish_auth 0 2024-09-17T15:29:22.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:29:22.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 1 -- 172.21.15.86:0/580393396 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9800a5870 con 0x7fb98813d180 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 1 -- 172.21.15.86:0/580393396 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9800a5c40 con 0x7fb98813d180 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb98d88f640 5 monclient: authenticate success, global_id 4506 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb98d88f640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:22.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb98d88f640 4 set_mon_vals no callback set 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.100+0000 7fb985ffb640 10 monclient: dump: 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: shutdown 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 msgr2=0x7fb98813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813f570 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d90 tx=0x7fb98001c7d0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- 172.21.15.86:0/580393396 shutdown_connections 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813f570 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- 172.21.15.86:0/580393396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a5350 0x7fb9880a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- 172.21.15.86:0/580393396 >> 172.21.15.86:0/580393396 conn(0x7fb9880a7240 msgr2=0x7fb9880a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- 172.21.15.86:0/580393396 shutdown_connections 2024-09-17T15:29:22.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- 172.21.15.86:0/580393396 wait complete. 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: build_initial_monmap 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: monmap: 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 Processor -- start 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- start start 2024-09-17T15:29:22.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: init 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 0x7fb9881396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-c con 0x7fb9880a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880a5350 0x7fb988139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-b con 0x7fb9880a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): picked mon.noname-a con 0x7fb98813d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb988142710 con 0x7fb98813d180 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb988142590 con 0x7fb9880a5350 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb988142890 con 0x7fb9880a4f70 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb98d88f640 10 monclient(hunting): authenticate will time out at 1095.073805s 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 0x7fb9881396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56534/0 (socket says 172.21.15.86:56534) 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 1 -- 172.21.15.86:0/3015672666 learned_addr learned my addr 172.21.15.86:0/3015672666 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): get_auth_request con 0x7fb9880a4f70 auth_method 0 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): get_auth_request con 0x7fb98813d180 auth_method 0 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient(hunting): handle_auth_done global_id 24125 payload 824 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880a5350 msgr2=0x7fb988139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9867fc640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880a5350 0x7fb988139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880a5350 0x7fb988139c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 msgr2=0x7fb98813a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 monclient: found mon.noname-c 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb986ffd640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb98809f200 con 0x7fb9880a4f70 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.104+0000 7fb9877fe640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb98813d180 0x7fb98813a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb986ffd640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 msgr2=0x7fb9881396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb986ffd640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 0x7fb9881396e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c410 tx=0x7fb97c005790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c017070 con 0x7fb9880a4f70 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: dump: 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: _finish_auth 0 2024-09-17T15:29:22.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb97c005820 con 0x7fb9880a4f70 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97c005c10 con 0x7fb9880a4f70 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 5 monclient: authenticate success, global_id 24125 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 monclient: _renew_subs 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98813d560 con 0x7fb9880a4f70 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb984ff9640 4 set_mon_vals no callback set 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 10 monclient: dump: 2024-09-17T15:29:22.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 monclient: _renew_subs 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb98809ef20 con 0x7fb9880a4f70 2024-09-17T15:29:22.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 0x7fb9881d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb98813bc30 con 0x7fb9881cfe00 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb97c01a070 con 0x7fb9880a4f70 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 0x7fb9881d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb97403bb90 0x7fb97403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient: get_auth_request con 0x7fb9881cfe00 auth_method 0 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(73..73 src has 1..73) ==== 4783+0+0 (secure 0 0 0) 0x7fb97c04f320 con 0x7fb9880a4f70 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9877fe640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb97403bb90 0x7fb97403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9877fe640 10 monclient: get_auth_request con 0x7fb97403bb90 auth_method 0 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 10 monclient(hunting): handle_auth_done global_id 14188 payload 824 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.108+0000 7fb9867fc640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9880a5750 con 0x7fb9881cfe00 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb9867fc640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 msgr2=0x7fb9881d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb9867fc640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 0x7fb9881d01e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb98813b740 tx=0x7fb978010c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97800f040 con 0x7fb9881cfe00 2024-09-17T15:29:22.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 10 monclient: dump: 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.141 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb9877fe640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb97403bb90 0x7fb97403e050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb98001d070 tx=0x7fb9800a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb97801c400 con 0x7fb9881cfe00 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 msgr2=0x7fb9881d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 0x7fb9881d01e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb98813b740 tx=0x7fb978010c20 comp rx=0 tx=0).stop 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 msgr2=0x7fb9881d01e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.112+0000 7fb96ffff640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881cfe00 0x7fb9881d01e0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb98813b740 tx=0x7fb978010c20 comp rx=0 tx=0).stop 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.116+0000 7fb98d88f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.116+0000 7fb98d88f640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 0x7fb9880a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.116+0000 7fb98d88f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.116+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880a5ff0 con 0x7fb9881d0720 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 0x7fb9880a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient: get_auth_request con 0x7fb9881d0720 auth_method 0 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 10 monclient(hunting): handle_auth_done global_id 14191 payload 824 2024-09-17T15:29:22.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 1 -- 172.21.15.86:0/3015672666 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb988069920 con 0x7fb9881d0720 2024-09-17T15:29:22.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 msgr2=0x7fb9880a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb9867fc640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 0x7fb9880a1930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb97801b000 tx=0x7fb9780046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978021770 con 0x7fb9881d0720 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb96ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb96ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.120+0000 7fb96ffff640 10 monclient: dump: 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.145 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fb9780212c0 con 0x7fb9881d0720 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 msgr2=0x7fb9880a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 0x7fb9880a1930 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb97801b000 tx=0x7fb9780046d0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 msgr2=0x7fb9880a1930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb96ffff640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9881d0720 0x7fb9880a1930 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:22.146 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 70724, 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:22.147 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:22.148 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.149 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:22.150 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:22.151 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:22.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:22.153 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:22.154 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:22.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.156 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:22.158 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:22.159 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:22.160 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:22.160 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:22.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb97403bb90 msgr2=0x7fb97403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb97403bb90 0x7fb97403e050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb98001d070 tx=0x7fb9800a7070 comp rx=0 tx=0).stop 2024-09-17T15:29:22.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 10 monclient: shutdown 2024-09-17T15:29:22.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:22.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 msgr2=0x7fb9881396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 --2- 172.21.15.86:0/3015672666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9880a4f70 0x7fb9881396e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00c410 tx=0x7fb97c005790 comp rx=0 tx=0).stop 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb9877fe640 1 -- 172.21.15.86:0/3015672666 reap_dead start 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb9877fe640 1 -- 172.21.15.86:0/3015672666 reap_dead start 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 shutdown_connections 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 >> 172.21.15.86:0/3015672666 conn(0x7fb9880a7240 msgr2=0x7fb98813c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 shutdown_connections 2024-09-17T15:29:22.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.124+0000 7fb98d88f640 1 -- 172.21.15.86:0/3015672666 wait complete. 2024-09-17T15:29:22.162 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:22.162 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:29:22.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:29:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 10 monclient: get_monmap_and_config 2024-09-17T15:29:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 10 monclient: build_initial_monmap 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 10 monclient: monmap: 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:22.266896+0000 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:22.266896+0000 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:22.266 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:22.267 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:22.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 1 Processor -- start 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 1 -- start start 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.260+0000 7f745ece1640 10 monclient: init 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f74580689d0 0x7f745806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-c con 0x7f74580689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:22.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7458068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-b con 0x7f745813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7458074470 con 0x7f7458068db0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f745809c900 con 0x7f74580689d0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f745809cb90 con 0x7f745813aff0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ece1640 10 monclient(hunting): authenticate will time out at 1095.232342s 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f7457fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7458068db0 0x7f7458069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46210/0 (socket says 172.21.15.86:46210) 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f7457fff640 1 -- 172.21.15.86:0/2459768807 learned_addr learned my addr 172.21.15.86:0/2459768807 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 --2- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): get_auth_request con 0x7f745813aff0 auth_method 0 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient(hunting): handle_auth_done global_id 24128 payload 274 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 -- 172.21.15.86:0/2459768807 >> v2:172.21.15.203:3300/0 conn(0x7f74580689d0 msgr2=0x7f745806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745ca7d640 1 --2- 172.21.15.86:0/2459768807 >> v2:172.21.15.203:3300/0 conn(0x7f74580689d0 0x7f745806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 --2- 172.21.15.86:0/2459768807 >> v2:172.21.15.203:3300/0 conn(0x7f74580689d0 0x7f745806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 -- 172.21.15.86:0/2459768807 >> v1:172.21.15.86:6789/0 conn(0x7f7458068db0 legacy=0x7f7458069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient: _finish_hunting 0 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 monclient: found mon.noname-b 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.264+0000 7f745d27e640 1 -- 172.21.15.86:0/2459768807 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745809ce70 con 0x7f745813aff0 2024-09-17T15:29:22.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 1 -- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 msgr2=0x7f745813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 1 --2- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745813d3e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50700d6d1744019d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 1 -- 172.21.15.86:0/2459768807 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74480b0070 con 0x7f745813aff0 2024-09-17T15:29:22.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: dump: 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:22.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 5 monclient: authenticate success, global_id 24128 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 1 -- 172.21.15.86:0/2459768807 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74480ab600 con 0x7f745813aff0 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 4 set_mon_vals no callback set 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 1 -- 172.21.15.86:0/2459768807 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745809c2d0 con 0x7f745813aff0 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f74577fe640 10 monclient: dump: 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: shutdown 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 msgr2=0x7f745813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745813d3e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c360 comp rx=0 tx=0).stop 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- 172.21.15.86:0/2459768807 shutdown_connections 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- 172.21.15.86:0/2459768807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745813d3e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- 172.21.15.86:0/2459768807 >> v2:172.21.15.203:3300/0 conn(0x7f74580689d0 0x7f745806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- 172.21.15.86:0/2459768807 >> 172.21.15.86:0/2459768807 conn(0x7f74580a6440 msgr2=0x7f74580a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- 172.21.15.86:0/2459768807 shutdown_connections 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- 172.21.15.86:0/2459768807 wait complete. 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: build_initial_monmap 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: monmap: 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:22.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 Processor -- start 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- start start 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: init 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74580689d0 0x7f7458073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-a con 0x7f74580689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458068db0 0x7f745809d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-b con 0x7f7458068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): picked mon.noname-c con 0x7f745813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7458140670 con 0x7f74580689d0 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74581404f0 con 0x7f7458068db0 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f74581407f0 con 0x7f745813aff0 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ece1640 10 monclient(hunting): authenticate will time out at 1095.239721s 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46378/0 (socket says 172.21.15.86:46378) 2024-09-17T15:29:22.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 1 -- 172.21.15.86:0/2920691569 learned_addr learned my addr 172.21.15.86:0/2920691569 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): get_auth_request con 0x7f745813aff0 auth_method 0 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ca7d640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74580689d0 0x7f7458073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ca7d640 10 monclient(hunting): get_auth_request con 0x7f74580689d0 auth_method 0 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ca7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ca7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745ca7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.268+0000 7f745d27e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ca7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ca7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ca7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 10 monclient(hunting): handle_auth_done global_id 24131 payload 824 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 -- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458068db0 msgr2=0x7f745809d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f7457fff640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458068db0 0x7f745809d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458068db0 0x7f745809d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 -- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74580689d0 msgr2=0x7f7458073270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74580689d0 0x7f7458073270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 10 monclient: _finish_hunting 0 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 monclient: found mon.noname-c 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 -- 172.21.15.86:0/2920691569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7458072900 con 0x7f745813aff0 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 -- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 msgr2=0x7f745809dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745d27e640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f7448019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74480b0040 con 0x7f745813aff0 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: dump: 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: _finish_auth 0 2024-09-17T15:29:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: _check_auth_tickets 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74480b9070 con 0x7f745813aff0 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74480aa9b0 con 0x7f745813aff0 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:22.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 10 monclient: dump: 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:22.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 5 monclient: authenticate success, global_id 24131 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _renew_subs 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74580a3650 con 0x7f745813aff0 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f7456ffd640 4 set_mon_vals no callback set 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _renew_subs 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f745809f110 con 0x7f745813aff0 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745806f840 con 0x7f745813aff0 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f74480ab4c0 con 0x7f745813aff0 2024-09-17T15:29:22.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.272+0000 7f74557fa640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 0x7f743803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.276+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(73..73 src has 1..73) ==== 4783+0+0 (secure 0 0 0) 0x7f74480e9ec0 con 0x7f745813aff0 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.276+0000 7f745ca7d640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 0x7f743803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.276+0000 7f745ca7d640 10 monclient: get_auth_request con 0x7f743803bbb0 auth_method 0 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.276+0000 7f745ca7d640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 0x7f743803e070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f744000aae0 tx=0x7f7440008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.280+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7448016070 con 0x7f745813aff0 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.280+0000 7f74557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:22.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.280+0000 7f74557fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:22.401 INFO:tasks.rados.rados.0.smithi203.stdout:269: done (1 left) 2024-09-17T15:29:22.401 INFO:tasks.rados.rados.0.smithi203.stdout:270: done (0 left) 2024-09-17T15:29:22.401 INFO:tasks.rados.rados.0.smithi203.stdout:271: read oid 48 snap -1 2024-09-17T15:29:22.402 INFO:tasks.rados.rados.0.smithi203.stdout:271: expect deleted 2024-09-17T15:29:22.402 INFO:tasks.rados.rados.0.smithi203.stdout:272: read oid 27 snap -1 2024-09-17T15:29:22.402 INFO:tasks.rados.rados.0.smithi203.stdout:272: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T15:29:22.402 INFO:tasks.rados.rados.0.smithi203.stdout:273: snap_create 2024-09-17T15:29:22.414 INFO:tasks.rados.rados.0.smithi203.stdout:272: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-17T15:29:22.422 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1354409 2024-09-17T15:29:22.508 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 196416 2024-09-17T15:29:22.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.504+0000 7f745ece1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:29:22.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.504+0000 7f745ece1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:29:22.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.504+0000 7f745ece1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:22.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.504+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f745806d4f0 con 0x7f745813aff0 2024-09-17T15:29:22.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=11,logm=70} crc {auth=1346015519,config=3541143443,health=1704768867,kv=3591778543,logm=2328860333}) 2024-09-17T15:29:22.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=11,logm=66} crc {auth=786559226,config=3541143443,health=3897446751,kv=3591778543,logm=2712569145}) 2024-09-17T15:29:22.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=10,kv=11,logm=70} crc {auth=1346015519,config=3541143443,health=1704768867,kv=3591778543,logm=2328860333}) 2024-09-17T15:29:22.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=11,logm=66} crc {auth=2557019349,config=3541143443,health=2609900051,kv=2552736648,logm=3535531353}) 2024-09-17T15:29:22.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f74557fa640 1 -- 172.21.15.86:0/2920691569 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f74480a9e50 con 0x7f745813aff0 2024-09-17T15:29:22.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f74557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:29:22.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f74557fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:22.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248318690}) 2024-09-17T15:29:22.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3538461926}) 2024-09-17T15:29:22.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4248318690}) 2024-09-17T15:29:22.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.508+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907634214}) 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 msgr2=0x7f743803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 0x7f743803e070 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f744000aae0 tx=0x7f7440008040 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 10 monclient: shutdown 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 msgr2=0x7f745809dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f7448019100 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 shutdown_connections 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f743803bbb0 0x7f743803e070 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.508+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f745813aff0 0x7f745809dce0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.512+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7458068db0 0x7f745809d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.512+0000 7f745ece1640 1 --2- 172.21.15.86:0/2920691569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74580689d0 0x7f7458073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.512+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 >> 172.21.15.86:0/2920691569 conn(0x7f74580a6440 msgr2=0x7f74580a2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:22.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.512+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 shutdown_connections 2024-09-17T15:29:22.516 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.516 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876272083}) 2024-09-17T15:29:22.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:22.512+0000 7f745ece1640 1 -- 172.21.15.86:0/2920691569 wait complete. 2024-09-17T15:29:22.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174355910}) 2024-09-17T15:29:22.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876272083}) 2024-09-17T15:29:22.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2194328490}) 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105216347}) 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=452149977}) 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105216347}) 2024-09-17T15:29:22.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306660181}) 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207896786}) 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657747509}) 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207896786}) 2024-09-17T15:29:22.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.512+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849991024}) 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815936983}) 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1950080864}) 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815936983}) 2024-09-17T15:29:22.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3180958778}) 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=26} crc {logm=2828004346,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1908846706,mon_config_key=2933492657,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=3965751285,osd_snap=1516640082,osdmap=4010370216}) 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=29} crc {logm=2828004346,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1908846706,mon_config_key=1176994871,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=3965751285,osd_snap=3581122374,osdmap=4200129105}) 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=26} crc {logm=2828004346,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1908846706,mon_config_key=2933492657,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=3965751285,osd_snap=1516640082,osdmap=4010370216}) 2024-09-17T15:29:22.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.516+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=12,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=27} crc {logm=2035962316,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1908846706,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=3965751285,osd_snap=3581122374,osdmap=97157785}) 2024-09-17T15:29:22.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=4272973227}) 2024-09-17T15:29:22.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=94} crc {osdmap=3523577055}) 2024-09-17T15:29:22.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:29:22.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=95} crc {osdmap=4272973227}) 2024-09-17T15:29:22.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:29:22.520+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=3334944536}) 2024-09-17T15:29:22.533 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:29:23.401 INFO:tasks.rados.rados.0.smithi203.stdout:271: done (2 left) 2024-09-17T15:29:23.401 INFO:tasks.rados.rados.0.smithi203.stdout:272: done (1 left) 2024-09-17T15:29:23.402 INFO:tasks.rados.rados.0.smithi203.stdout:273: done (0 left) 2024-09-17T15:29:23.402 INFO:tasks.rados.rados.0.smithi203.stdout:274: snap_remove snap 20 2024-09-17T15:29:23.534 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:29:23.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:23.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: get_monmap_and_config 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: build_initial_monmap 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: monmap: 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:23.661955+0000 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:23.661955+0000 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:23.661 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:23.662 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:23.662 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 Processor -- start 2024-09-17T15:29:23.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 -- start start 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: init 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6a80a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a80a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6a80a6530 con 0x7fb6a80a50e0 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb6a809bdf0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a809c1a0 con 0x7fb6a813d2f0 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6ad04a640 10 monclient(hunting): authenticate will time out at 1096.627429s 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56554/0 (socket says 172.21.15.86:56554) 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46384/0 (socket says 172.21.15.86:46384) 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 1 -- 172.21.15.86:0/2945922852 learned_addr learned my addr 172.21.15.86:0/2945922852 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): get_auth_request con 0x7fb6a80a54c0 auth_method 0 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): get_auth_request con 0x7fb6a813d2f0 auth_method 0 2024-09-17T15:29:23.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.656+0000 7fb6a759e640 10 monclient(hunting): handle_auth_done global_id 24134 payload 274 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 1 -- 172.21.15.86:0/2945922852 >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 msgr2=0x7fb6a80a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 1 --2- 172.21.15.86:0/2945922852 >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 1 -- 172.21.15.86:0/2945922852 >> v1:172.21.15.203:6789/0 conn(0x7fb6a80a50e0 legacy=0x7fb6a809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:23.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 10 monclient: _finish_hunting 0 2024-09-17T15:29:23.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 1 monclient: found mon.noname-b 2024-09-17T15:29:23.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:23.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a759e640 1 -- 172.21.15.86:0/2945922852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a809c460 con 0x7fb6a813d2f0 2024-09-17T15:29:23.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.660+0000 7fb6a659c640 1 --2- 172.21.15.86:0/2945922852 >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a759e640 1 -- 172.21.15.86:0/2945922852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 msgr2=0x7fb6a813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a759e640 1 --2- 172.21.15.86:0/2945922852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097c0 tx=0x7fb69801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abf75bb5c971208c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 1 -- 172.21.15.86:0/2945922852 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980b0070 con 0x7fb6a813d2f0 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:29:23.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: dump: 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:29:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 5 monclient: authenticate success, global_id 24134 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 1 -- 172.21.15.86:0/2945922852 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6980ab600 con 0x7fb6a813d2f0 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 1 -- 172.21.15.86:0/2945922852 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6980ab9d0 con 0x7fb6a813d2f0 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 4 set_mon_vals no callback set 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a5d9b640 10 monclient: dump: 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.672 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: shutdown 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- 172.21.15.86:0/2945922852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 msgr2=0x7fb6a813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/2945922852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb6980097c0 tx=0x7fb69801c360 comp rx=0 tx=0).stop 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- 172.21.15.86:0/2945922852 shutdown_connections 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/2945922852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a813d2f0 0x7fb6a813f6e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/2945922852 >> v2:172.21.15.86:3300/0 conn(0x7fb6a80a54c0 0x7fb6a80a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- 172.21.15.86:0/2945922852 >> 172.21.15.86:0/2945922852 conn(0x7fb6a80aaae0 msgr2=0x7fb6a80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- 172.21.15.86:0/2945922852 shutdown_connections 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- 172.21.15.86:0/2945922852 wait complete. 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: build_initial_monmap 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: monmap: 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:23.674 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 Processor -- start 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- start start 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: init 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 0x7fb6a8121480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a813d2f0 0x7fb6a81382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:23.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a8142850 con 0x7fb6a80a50e0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a81426d0 con 0x7fb6a813d2f0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a81429d0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6ad04a640 10 monclient(hunting): authenticate will time out at 1096.634636s 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 0x7fb6a8121480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46398/0 (socket says 172.21.15.86:46398) 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 0x7fb6a8121480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56568/0 (socket says 172.21.15.86:56568) 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 -- 172.21.15.86:0/506721405 learned_addr learned my addr 172.21.15.86:0/506721405 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): get_auth_request con 0x7fb6a80a54c0 auth_method 0 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:23.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): get_auth_request con 0x7fb6a80a50e0 auth_method 0 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient(hunting): handle_auth_done global_id 24137 payload 824 2024-09-17T15:29:23.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 -- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a813d2f0 msgr2=0x7fb6a81382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a813d2f0 0x7fb6a81382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 -- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 msgr2=0x7fb6a8121480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 0x7fb6a8121480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient: _finish_hunting 0 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 monclient: found mon.noname-c 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.664+0000 7fb6a659c640 1 -- 172.21.15.86:0/506721405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a80a9f90 con 0x7fb6a80a54c0 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6a659c640 1 -- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 msgr2=0x7fb6a80a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6a659c640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000efc0 tx=0x7fb6a000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0014680 con 0x7fb6a80a54c0 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: dump: 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a0019070 con 0x7fb6a80a54c0 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:23.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0014a20 con 0x7fb6a80a54c0 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 10 monclient: dump: 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 5 monclient: authenticate success, global_id 24137 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6a4d99640 4 set_mon_vals no callback set 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _renew_subs 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a80a18d0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _renew_subs 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a8122c80 con 0x7fb6a80a54c0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb6a0012050 con 0x7fb6a80a54c0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a80a58d0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 0x7fb68803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 4755+0+0 (secure 0 0 0) 0x7fb6a00504d0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6a6d9d640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 0x7fb68803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.668+0000 7fb6a6d9d640 10 monclient: get_auth_request con 0x7fb68803bbe0 auth_method 0 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.672+0000 7fb6a6d9d640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 0x7fb68803e0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80a1350 tx=0x7fb690007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.672+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb6a0010040 con 0x7fb6a80a54c0 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.672+0000 7fb69f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:23.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.672+0000 7fb69f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:23.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb6ad04a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:23.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb6ad04a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:23.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb6ad04a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:23.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6a8074be0 con 0x7fb6a80a54c0 2024-09-17T15:29:23.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb69f7fe640 1 -- 172.21.15.86:0/506721405 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fb6a0007c70 con 0x7fb6a80a54c0 2024-09-17T15:29:23.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb69f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:23.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.892+0000 7fb69f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:23.899 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 msgr2=0x7fb68803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 0x7fb68803e0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80a1350 tx=0x7fb690007e40 comp rx=0 tx=0).stop 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 10 monclient: shutdown 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 msgr2=0x7fb6a80a3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000efc0 tx=0x7fb6a000c490 comp rx=0 tx=0).stop 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 shutdown_connections 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb68803bbe0 0x7fb68803e0a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6a813d2f0 0x7fb6a81382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a80a54c0 0x7fb6a80a3300 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 --2- 172.21.15.86:0/506721405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80a50e0 0x7fb6a8121480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:23.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 >> 172.21.15.86:0/506721405 conn(0x7fb6a80aaae0 msgr2=0x7fb6a80a06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:23.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 shutdown_connections 2024-09-17T15:29:23.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:23.896+0000 7fb6ad04a640 1 -- 172.21.15.86:0/506721405 wait complete. 2024-09-17T15:29:23.916 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:29:23.916 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:29:23.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:29:23.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:29:24.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: get_monmap_and_config 2024-09-17T15:29:24.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: monmap: 2024-09-17T15:29:24.023 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:24.024648+0000 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:24.024648+0000 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:24.024 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 Processor -- start 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 -- start start 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: init 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 0x7f0e7009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-b con 0x7f0e700a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e700a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e7013d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e7009d430 con 0x7f0e7013d180 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e700a68d0 con 0x7f0e700a5350 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7009c7f0 con 0x7f0e700a4f70 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e7786a640 10 monclient(hunting): authenticate will time out at 1096.990140s 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 0x7f0e7009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75e07640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0e7013d180 0x7f0e7013f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46222/0 (socket says 172.21.15.86:46222) 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75e07640 1 -- 172.21.15.86:0/1616979774 learned_addr learned my addr 172.21.15.86:0/1616979774 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): get_auth_request con 0x7f0e700a4f70 auth_method 0 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient(hunting): handle_auth_done global_id 24140 payload 274 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 1 -- 172.21.15.86:0/1616979774 >> v1:172.21.15.203:6789/0 conn(0x7f0e700a5350 legacy=0x7f0e700a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 1 -- 172.21.15.86:0/1616979774 >> v1:172.21.15.86:6789/0 conn(0x7f0e7013d180 legacy=0x7f0e7013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 1 monclient: found mon.noname-b 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:29:24.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.020+0000 7f0e75606640 1 -- 172.21.15.86:0/1616979774 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e700a64f0 con 0x7f0e700a4f70 2024-09-17T15:29:24.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e75606640 1 -- 172.21.15.86:0/1616979774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 msgr2=0x7f0e7009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e75606640 1 --2- 172.21.15.86:0/1616979774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 0x7f0e7009cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e70069c50 tx=0x7f0e6c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=704a4431765ec63c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 1 -- 172.21.15.86:0/1616979774 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c0b0070 con 0x7f0e700a4f70 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: dump: 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 1 -- 172.21.15.86:0/1616979774 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6c0ab600 con 0x7f0e700a4f70 2024-09-17T15:29:24.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 5 monclient: authenticate success, global_id 24140 2024-09-17T15:29:24.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 1 -- 172.21.15.86:0/1616979774 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c0ab9d0 con 0x7f0e700a4f70 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 4 set_mon_vals no callback set 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e67fff640 10 monclient: dump: 2024-09-17T15:29:24.034 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 10 monclient: shutdown 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 -- 172.21.15.86:0/1616979774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 msgr2=0x7f0e7009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 --2- 172.21.15.86:0/1616979774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 0x7f0e7009cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0e70069c50 tx=0x7f0e6c01c300 comp rx=0 tx=0).stop 2024-09-17T15:29:24.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 -- 172.21.15.86:0/1616979774 shutdown_connections 2024-09-17T15:29:24.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 --2- 172.21.15.86:0/1616979774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a4f70 0x7f0e7009cef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 -- 172.21.15.86:0/1616979774 >> 172.21.15.86:0/1616979774 conn(0x7f0e700a7240 msgr2=0x7f0e700a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 -- 172.21.15.86:0/1616979774 shutdown_connections 2024-09-17T15:29:24.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.024+0000 7f0e7786a640 1 -- 172.21.15.86:0/1616979774 wait complete. 2024-09-17T15:29:24.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient: monmap: 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 Processor -- start 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 -- start start 2024-09-17T15:29:24.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient: init 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e700a4f70 0x7f0e700a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-b con 0x7f0e700a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 0x7f0e700a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-c con 0x7f0e700a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e7013d180 0x7f0e70138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): picked mon.noname-a con 0x7f0e7013d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70143990 con 0x7f0e7013d180 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e70143810 con 0x7f0e700a4f70 2024-09-17T15:29:24.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0e70143b10 con 0x7f0e700a5350 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e7786a640 10 monclient(hunting): authenticate will time out at 1096.997227s 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e7013d180 0x7f0e70138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 0x7f0e700a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 0x7f0e700a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46416/0 (socket says 172.21.15.86:46416) 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 -- 172.21.15.86:0/611805476 learned_addr learned my addr 172.21.15.86:0/611805476 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): get_auth_request con 0x7f0e700a5350 auth_method 0 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): get_auth_request con 0x7f0e7013d180 auth_method 0 2024-09-17T15:29:24.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient(hunting): handle_auth_done global_id 24143 payload 824 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75606640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e700a4f70 0x7f0e700a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e700a4f70 msgr2=0x7f0e700a4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e700a4f70 0x7f0e700a4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e7013d180 msgr2=0x7f0e70138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e7013d180 0x7f0e70138240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 monclient: found mon.noname-c 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:29:24.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e70140a80 con 0x7f0e700a5350 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e75e07640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e7013d180 0x7f0e70138240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 msgr2=0x7f0e700a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.028+0000 7f0e74e05640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 0x7f0e700a2280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c410 tx=0x7f0e6000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60017070 con 0x7f0e700a5350 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: dump: 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.047 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 5 monclient: authenticate success, global_id 24143 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e60003820 con 0x7f0e700a5350 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 monclient: _renew_subs 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:24.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e700a1760 con 0x7f0e700a5350 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e60003c10 con 0x7f0e700a5350 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: dump: 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.049 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e677fe640 4 set_mon_vals no callback set 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 monclient: _renew_subs 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e70121320 con 0x7f0e700a5350 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 0x7f0e701d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0e700a32b0 con 0x7f0e701d0000 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0e6001a070 con 0x7f0e700a5350 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 0x7f0e701d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient: get_auth_request con 0x7f0e701d0000 auth_method 0 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e5403bb90 0x7f0e5403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 10 monclient(hunting): handle_auth_done global_id 4524 payload 824 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e7009cef0 con 0x7f0e701d0000 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(75..75 src has 1..75) ==== 4755+0+0 (secure 0 0 0) 0x7f0e6004f410 con 0x7f0e700a5350 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75e07640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e5403bb90 0x7f0e5403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75e07640 10 monclient: get_auth_request con 0x7f0e5403bb90 auth_method 0 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 msgr2=0x7f0e701d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e75606640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 0x7f0e701d03e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0ad000 tx=0x7f0e6c01aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c0bb020 con 0x7f0e701d0000 2024-09-17T15:29:24.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: dump: 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0e6c0b6470 con 0x7f0e701d0000 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 msgr2=0x7f0e701d03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 0x7f0e701d03e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0ad000 tx=0x7f0e6c01aba0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 msgr2=0x7f0e701d03e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.032+0000 7f0e65ffb640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0000 0x7f0e701d03e0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0ad000 tx=0x7f0e6c01aba0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.036+0000 7f0e75e07640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e5403bb90 0x7f0e5403e050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e68005c10 tx=0x7f0e68009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e7786a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e7786a640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 0x7f0e7013d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e7786a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0e700a5ff0 con 0x7f0e701d0920 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 0x7f0e7013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient: get_auth_request con 0x7f0e701d0920 auth_method 0 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 10 monclient(hunting): handle_auth_done global_id 4527 payload 824 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.040+0000 7f0e75606640 1 -- 172.21.15.86:0/611805476 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0e70069920 con 0x7f0e701d0920 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e75606640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 msgr2=0x7f0e7013d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e75606640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 0x7f0e7013d560 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009740 tx=0x7f0e6c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6c0bb030 con 0x7f0e701d0920 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 10 monclient: dump: 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f0e6c0c2020 con 0x7f0e701d0920 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 msgr2=0x7f0e7013d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 0x7f0e7013d560 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009740 tx=0x7f0e6c019100 comp rx=0 tx=0).stop 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 msgr2=0x7f0e7013d560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e65ffb640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e701d0920 0x7f0e7013d560 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c009740 tx=0x7f0e6c019100 comp rx=0 tx=0).stop 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 136688, 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:24.059 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.060 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.062 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.063 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.064 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.066 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.067 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.069 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.070 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.071 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:24.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:24.073 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e5403bb90 msgr2=0x7f0e5403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0e5403bb90 0x7f0e5403e050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e68005c10 tx=0x7f0e68009040 comp rx=0 tx=0).stop 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 10 monclient: shutdown 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 msgr2=0x7f0e700a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 --2- 172.21.15.86:0/611805476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e700a5350 0x7f0e700a2280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000c410 tx=0x7f0e6000c8d0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e75e07640 1 -- 172.21.15.86:0/611805476 reap_dead start 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e75e07640 1 -- 172.21.15.86:0/611805476 reap_dead start 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 shutdown_connections 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 >> 172.21.15.86:0/611805476 conn(0x7f0e700a7240 msgr2=0x7f0e700a07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 shutdown_connections 2024-09-17T15:29:24.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.044+0000 7f0e7786a640 1 -- 172.21.15.86:0/611805476 wait complete. 2024-09-17T15:29:24.076 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:24.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:29:24.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: get_monmap_and_config 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: monmap: 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:24.188990+0000 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:24.188990+0000 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:24.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:24.189 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:24.189 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:24.189 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 Processor -- start 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 -- start start 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: init 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32340a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32340a5bd0 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f323409cd60 0x7f3234140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-a con 0x7f323409cd60 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32340a6280 con 0x7f32340a5bd0 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f323409c0a0 con 0x7f32340a50e0 2024-09-17T15:29:24.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f323409c380 con 0x7f323409cd60 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f323a46d640 10 monclient(hunting): authenticate will time out at 1097.154742s 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f323409cd60 0x7f3234140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f323409cd60 0x7f3234140a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56596/0 (socket says 172.21.15.86:56596) 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 -- 172.21.15.86:0/2698783175 learned_addr learned my addr 172.21.15.86:0/2698783175 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): get_auth_request con 0x7f323409cd60 auth_method 0 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient(hunting): handle_auth_done global_id 4530 payload 274 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 -- 172.21.15.86:0/2698783175 >> v1:172.21.15.203:6789/0 conn(0x7f32340a50e0 legacy=0x7f32340a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 -- 172.21.15.86:0/2698783175 >> v1:172.21.15.86:6789/0 conn(0x7f32340a5bd0 legacy=0x7f323413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 monclient: found mon.noname-a 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.184+0000 7f3238a0a640 1 -- 172.21.15.86:0/2698783175 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323409c660 con 0x7f323409cd60 2024-09-17T15:29:24.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3238a0a640 1 -- 172.21.15.86:0/2698783175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f323409cd60 msgr2=0x7f3234140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3238a0a640 1 --2- 172.21.15.86:0/2698783175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f323409cd60 0x7f3234140a20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f322c009e50 tx=0x7f322c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6034ecc1e4751f5c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 1 -- 172.21.15.86:0/2698783175 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0a5620 con 0x7f323409cd60 2024-09-17T15:29:24.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: dump: 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f323a46d640 5 monclient: authenticate success, global_id 4530 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f323a46d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 1 -- 172.21.15.86:0/2698783175 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c0a57c0 con 0x7f323409cd60 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 1 -- 172.21.15.86:0/2698783175 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0a5b90 con 0x7f323409cd60 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f323a46d640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f323a46d640 4 set_mon_vals no callback set 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.188+0000 7f3232ffd640 10 monclient: dump: 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: shutdown 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- 172.21.15.86:0/2698783175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f323409cd60 msgr2=0x7f3234140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 --2- 172.21.15.86:0/2698783175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f323409cd60 0x7f3234140a20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f322c009e50 tx=0x7f322c01c8c0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- 172.21.15.86:0/2698783175 shutdown_connections 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 --2- 172.21.15.86:0/2698783175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f323409cd60 0x7f3234140a20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- 172.21.15.86:0/2698783175 >> 172.21.15.86:0/2698783175 conn(0x7f32340aaae0 msgr2=0x7f32340aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- 172.21.15.86:0/2698783175 shutdown_connections 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- 172.21.15.86:0/2698783175 wait complete. 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: monmap: 2024-09-17T15:29:24.203 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 Processor -- start 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- start start 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: init 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:29:24.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f323409cd60 0x7f32341340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-b con 0x7f323409cd60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32340a50e0 0x7f3234134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-c con 0x7f32340a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 0x7f32341392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): picked mon.noname-a con 0x7f32340a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3234143900 con 0x7f32340a5bd0 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3234143780 con 0x7f323409cd60 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3234143a80 con 0x7f32340a50e0 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f323a46d640 10 monclient(hunting): authenticate will time out at 1097.161766s 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 0x7f32341392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32340a50e0 0x7f3234134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 0x7f32341392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56598/0 (socket says 172.21.15.86:56598) 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 learned_addr learned my addr 172.21.15.86:0/411287068 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): get_auth_request con 0x7f32340a5bd0 auth_method 0 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): get_auth_request con 0x7f32340a50e0 auth_method 0 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f32337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f323409cd60 0x7f32341340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient(hunting): handle_auth_done global_id 4533 payload 824 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32340a50e0 msgr2=0x7f3234134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f32340a50e0 0x7f3234134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f323409cd60 msgr2=0x7f32341340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f323409cd60 0x7f32341340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 monclient: found mon.noname-a 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323413c450 con 0x7f32340a5bd0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3233fff640 10 monclient: get_auth_request con 0x7f323409cd60 auth_method 0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.192+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f323409cd60 0x7f32341340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 msgr2=0x7f32341392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3238a0a640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 0x7f32341392d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f322c009e50 tx=0x7f322c019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c0bc020 con 0x7f32340a5bd0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: dump: 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.208 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f322c0b8880 con 0x7f32340a5bd0 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 5 monclient: authenticate success, global_id 4533 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 monclient: _renew_subs 2024-09-17T15:29:24.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323413c730 con 0x7f32340a5bd0 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f322c009510 con 0x7f32340a5bd0 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 10 monclient: dump: 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.210 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 monclient: _renew_subs 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3231ffb640 4 set_mon_vals no callback set 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323413b4f0 con 0x7f32340a5bd0 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 0x7f3234139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f32340a54c0 con 0x7f3234139810 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 0x7f3234139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient: get_auth_request con 0x7f3234139810 auth_method 0 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f322c0bf020 con 0x7f32340a5bd0 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 10 monclient(hunting): handle_auth_done global_id 24149 payload 824 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3233fff640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f323413caa0 con 0x7f3234139810 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f322003bbb0 0x7f322003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 4755+0+0 (secure 0 0 0) 0x7f322c0eac10 con 0x7f32340a5bd0 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f32337fe640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f322003bbb0 0x7f322003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.196+0000 7f32337fe640 10 monclient: get_auth_request con 0x7f322003bbb0 auth_method 0 2024-09-17T15:29:24.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3233fff640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 msgr2=0x7f3234139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 0x7f3234139bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f322400d480 tx=0x7f322400d940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224012a10 con 0x7f3234139810 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 10 monclient: dump: 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.215 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3224012a10 con 0x7f3234139810 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 msgr2=0x7f3234139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 0x7f3234139bf0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f322400d480 tx=0x7f322400d940 comp rx=0 tx=0).stop 2024-09-17T15:29:24.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 msgr2=0x7f3234139bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f3230ff9640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3234139810 0x7f3234139bf0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f322400d480 tx=0x7f322400d940 comp rx=0 tx=0).stop 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.200+0000 7f32337fe640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f322003bbb0 0x7f322003e070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f323413fe70 tx=0x7f3228007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f323a46d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f323a46d640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 0x7f32340a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f323a46d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3234074be0 con 0x7f323413a130 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 0x7f32340a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient: get_auth_request con 0x7f323413a130 auth_method 0 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 10 monclient(hunting): handle_auth_done global_id 24152 payload 824 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.204+0000 7f3233fff640 1 -- 172.21.15.86:0/411287068 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f323413a540 con 0x7f323413a130 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3233fff640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 msgr2=0x7f32340a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3233fff640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 0x7f32340a05f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3224017000 tx=0x7f3224003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3224007650 con 0x7f323413a130 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 10 monclient: dump: 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f3224000e50 con 0x7f323413a130 2024-09-17T15:29:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 msgr2=0x7f32340a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 0x7f32340a05f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3224017000 tx=0x7f3224003770 comp rx=0 tx=0).stop 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 msgr2=0x7f32340a05f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3230ff9640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f323413a130 0x7f32340a05f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:24.220 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28706, 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:24.221 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.222 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.223 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:24.224 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.225 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.226 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.227 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.228 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.229 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.231 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.232 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.233 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.233 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.233 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:24.233 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f322003bbb0 msgr2=0x7f322003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f322003bbb0 0x7f322003e070 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f323413fe70 tx=0x7f3228007e40 comp rx=0 tx=0).stop 2024-09-17T15:29:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 10 monclient: shutdown 2024-09-17T15:29:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 msgr2=0x7f32341392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 --2- 172.21.15.86:0/411287068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32340a5bd0 0x7f32341392d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f322c009e50 tx=0x7f322c019660 comp rx=0 tx=0).stop 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 reap_dead start 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f3238a0a640 1 -- 172.21.15.86:0/411287068 reap_dead start 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 shutdown_connections 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 >> 172.21.15.86:0/411287068 conn(0x7f32340aaae0 msgr2=0x7f32340a29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 shutdown_connections 2024-09-17T15:29:24.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.208+0000 7f323a46d640 1 -- 172.21.15.86:0/411287068 wait complete. 2024-09-17T15:29:24.235 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:24.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:29:24.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:29:24.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.340+0000 7f2e12feb640 10 monclient: get_monmap_and_config 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.340+0000 7f2e12feb640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.340+0000 7f2e12feb640 10 monclient: monmap: 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:24.347854+0000 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:24.347854+0000 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:24.347 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:24.348 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 Processor -- start 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 -- start start 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient: init 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e0c134d90 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e0c0a1130 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e0c13e680 0x7f2e0c140a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0c13e680 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0c135880 con 0x7f2e0c134d90 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0c0a1540 con 0x7f2e0c0a1130 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2e0c0a21b0 con 0x7f2e0c13e680 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e12feb640 10 monclient(hunting): authenticate will time out at 1097.313015s 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e10d87640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e0c134d90 0x7f2e0c135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46236/0 (socket says 172.21.15.86:46236) 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e10d87640 1 -- 172.21.15.86:0/2918104742 learned_addr learned my addr 172.21.15.86:0/2918104742 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 1 --2- 172.21.15.86:0/2918104742 >> v2:172.21.15.86:3300/0 conn(0x7f2e0c13e680 0x7f2e0c140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): get_auth_request con 0x7f2e0c13e680 auth_method 0 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient(hunting): handle_auth_done global_id 4536 payload 274 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 1 -- 172.21.15.86:0/2918104742 >> v1:172.21.15.203:6789/0 conn(0x7f2e0c0a1130 legacy=0x7f2e0c13cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 1 -- 172.21.15.86:0/2918104742 >> v1:172.21.15.86:6789/0 conn(0x7f2e0c134d90 legacy=0x7f2e0c135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:24.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 1 monclient: found mon.noname-a 2024-09-17T15:29:24.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.344+0000 7f2e11588640 1 -- 172.21.15.86:0/2918104742 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0c073b10 con 0x7f2e0c13e680 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e11588640 1 -- 172.21.15.86:0/2918104742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c13e680 msgr2=0x7f2e0c140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e11588640 1 --2- 172.21.15.86:0/2918104742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c13e680 0x7f2e0c140a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0097f0 tx=0x7f2dfc01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0a2ddec46d81a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 1 -- 172.21.15.86:0/2918104742 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0b0070 con 0x7f2e0c13e680 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: dump: 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.354 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 1 -- 172.21.15.86:0/2918104742 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dfc0ab600 con 0x7f2e0c13e680 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 5 monclient: authenticate success, global_id 4536 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 1 -- 172.21.15.86:0/2918104742 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0ab9d0 con 0x7f2e0c13e680 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 4 set_mon_vals no callback set 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0b7fe640 10 monclient: dump: 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.356 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: shutdown 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- 172.21.15.86:0/2918104742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c13e680 msgr2=0x7f2e0c140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 --2- 172.21.15.86:0/2918104742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c13e680 0x7f2e0c140a70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0097f0 tx=0x7f2dfc01c330 comp rx=0 tx=0).stop 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- 172.21.15.86:0/2918104742 shutdown_connections 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 --2- 172.21.15.86:0/2918104742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c13e680 0x7f2e0c140a70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- 172.21.15.86:0/2918104742 >> 172.21.15.86:0/2918104742 conn(0x7f2e0c0a2f40 msgr2=0x7f2e0c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- 172.21.15.86:0/2918104742 shutdown_connections 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- 172.21.15.86:0/2918104742 wait complete. 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: build_initial_monmap 2024-09-17T15:29:24.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: monmap: 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 Processor -- start 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- start start 2024-09-17T15:29:24.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: init 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 0x7f2e0c133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-c con 0x7f2e0c0a1130 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 0x7f2e0c13acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0c134d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13e680 0x7f2e0c13b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): picked mon.noname-b con 0x7f2e0c13e680 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c143950 con 0x7f2e0c134d90 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c1437d0 con 0x7f2e0c13e680 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e0c143ad0 con 0x7f2e0c0a1130 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e12feb640 10 monclient(hunting): authenticate will time out at 1097.319852s 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e10d87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 0x7f2e0c133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.348+0000 7f2e0bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 0x7f2e0c13acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 0x7f2e0c13acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56622/0 (socket says 172.21.15.86:56622) 2024-09-17T15:29:24.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 0x7f2e0c133ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:46448/0 (socket says 172.21.15.86:46448) 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 -- 172.21.15.86:0/1097253216 learned_addr learned my addr 172.21.15.86:0/1097253216 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request con 0x7f2e0c134d90 auth_method 0 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): get_auth_request con 0x7f2e0c0a1130 auth_method 0 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e11588640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13e680 0x7f2e0c13b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient(hunting): handle_auth_done global_id 4539 payload 824 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 msgr2=0x7f2e0c133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 0x7f2e0c133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13e680 msgr2=0x7f2e0c13b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13e680 0x7f2e0c13b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 monclient: found mon.noname-a 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0c09d460 con 0x7f2e0c134d90 2024-09-17T15:29:24.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e11588640 10 monclient: get_auth_request con 0x7f2e0c13e680 auth_method 0 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e11588640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13e680 0x7f2e0c13b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e0c0a1130 0x7f2e0c133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 msgr2=0x7f2e0c13acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0bfff640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 0x7f2e0c13acb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2df400c7c0 tx=0x7f2df400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4016020 con 0x7f2e0c134d90 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: dump: 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: _finish_auth 0 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: _check_auth_tickets 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 5 monclient: authenticate success, global_id 4539 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df400ac60 con 0x7f2e0c134d90 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 monclient: _renew_subs 2024-09-17T15:29:24.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e0c11d180 con 0x7f2e0c134d90 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df4012660 con 0x7f2e0c134d90 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 10 monclient: dump: 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 monclient: _renew_subs 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e0affd640 4 set_mon_vals no callback set 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0c13d4a0 con 0x7f2e0c134d90 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 0x7f2e0c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c13c720 con 0x7f2e0c13bef0 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2df4019020 con 0x7f2e0c134d90 2024-09-17T15:29:24.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e10d87640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 0x7f2e0c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.352+0000 7f2e097fa640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2dec03bb90 0x7f2dec03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient: get_auth_request con 0x7f2e0c13bef0 auth_method 0 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 4755+0+0 (secure 0 0 0) 0x7f2df4050270 con 0x7f2e0c134d90 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e11588640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2dec03bb90 0x7f2dec03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e11588640 10 monclient: get_auth_request con 0x7f2dec03bb90 auth_method 0 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 10 monclient(hunting): handle_auth_done global_id 14194 payload 824 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e0c0a5bf0 con 0x7f2e0c13bef0 2024-09-17T15:29:24.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 msgr2=0x7f2e0c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e10d87640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 0x7f2e0c02d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e040072a0 tx=0x7f2e04007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e11588640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2dec03bb90 0x7f2dec03e050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c06d160 tx=0x7f2dfc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0400fce0 con 0x7f2e0c13bef0 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.356+0000 7f2e097fa640 10 monclient: dump: 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e0400fce0 con 0x7f2e0c13bef0 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 msgr2=0x7f2e0c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 0x7f2e0c02d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2e040072a0 tx=0x7f2e04007760 comp rx=0 tx=0).stop 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 msgr2=0x7f2e0c02d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.360+0000 7f2e097fa640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c13bef0 0x7f2e0c02d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e12feb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e12feb640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 0x7f2e0c02dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e12feb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0c11cde0 con 0x7f2e0c02d990 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 0x7f2e0c02dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient: get_auth_request con 0x7f2e0c02d990 auth_method 0 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:24.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 10 monclient(hunting): handle_auth_done global_id 14197 payload 824 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.364+0000 7f2e10d87640 1 -- 172.21.15.86:0/1097253216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e0c0a5630 con 0x7f2e0c02d990 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e10d87640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 msgr2=0x7f2e0c02dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e10d87640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 0x7f2e0c02dd70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e04017000 tx=0x7f2e04004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e04006450 con 0x7f2e0c02d990 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 10 monclient: dump: 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f2e04006450 con 0x7f2e0c02d990 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 msgr2=0x7f2e0c02dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 0x7f2e0c02dd70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e04017000 tx=0x7f2e04004950 comp rx=0 tx=0).stop 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 msgr2=0x7f2e0c02dd70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:24.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e097fa640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e0c02d990 0x7f2e0c02dd70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 38, 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72972, 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:29:24.378 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.379 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.380 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:24.381 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:24.382 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:24.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.384 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:24.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:24.387 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:24.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.390 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:24.391 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:24.392 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2dec03bb90 msgr2=0x7f2dec03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2dec03bb90 0x7f2dec03e050 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c06d160 tx=0x7f2dfc008040 comp rx=0 tx=0).stop 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 10 monclient: shutdown 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 msgr2=0x7f2e0c13acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 --2- 172.21.15.86:0/1097253216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e0c134d90 0x7f2e0c13acb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2df400c7c0 tx=0x7f2df400cc80 comp rx=0 tx=0).stop 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e11588640 1 -- 172.21.15.86:0/1097253216 reap_dead start 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e11588640 1 -- 172.21.15.86:0/1097253216 reap_dead start 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 shutdown_connections 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 >> 172.21.15.86:0/1097253216 conn(0x7f2e0c0a2f40 msgr2=0x7f2e0c09ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:24.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 shutdown_connections 2024-09-17T15:29:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:24.368+0000 7f2e12feb640 1 -- 172.21.15.86:0/1097253216 wait complete. 2024-09-17T15:29:24.394 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:29:24.394 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:29:24.394 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-17T15:29:24.394 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:29:24.394 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:29:24.395 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:29:24.395 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:29:24.404 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:29:24.404 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:29:24.405 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:29:24.405 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:29:39.405 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:29:39.406 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:29:39.406 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:29:39.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: get_monmap_and_config 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: build_initial_monmap 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: monmap: 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:39.533917+0000 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:39.533917+0000 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:39.533 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:39.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:39.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:39.534 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:39.534 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:39.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 Processor -- start 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 -- start start 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: init 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73680a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f73680a5b60 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f736813d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a6f40 con 0x7f73680a5070 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73680a65e0 con 0x7f736813d280 2024-09-17T15:29:39.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f736809c2a0 con 0x7f73680a5b60 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f736d99d640 10 monclient(hunting): authenticate will time out at 1112.499320s 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f7366ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f73680a5070 0x7f73680a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41366/0 (socket says 172.21.15.86:41366) 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 learned_addr learned my addr 172.21.15.86:0/1648469135 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.531+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:39.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:39.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:39.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 >> v1:172.21.15.86:6789/0 conn(0x7f73680a5070 legacy=0x7f73680a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:39.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.535+0000 7f73677fe640 1 -- 172.21.15.86:0/1648469135 >> v1:172.21.15.203:6789/0 conn(0x7f736813d280 legacy=0x7f736813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:39.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:40.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:40.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): tick 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f736809cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f736809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 >> v1:172.21.15.203:6789/0 conn(0x7f736813d280 legacy=0x7f736813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 >> v1:172.21.15.86:6789/0 conn(0x7f73680a5070 legacy=0x7f73680a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient: _add_conns ranks=[2,3] 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f7354003840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f73540063a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f736809c2a0 con 0x7f73540063a0 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7354008f20 con 0x7f7354003840 2024-09-17T15:29:42.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7364ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f7354005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f7366ffd640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.531+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:42.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:42.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.735+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f7354005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:42.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.735+0000 7f7366ffd640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:42.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:42.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:42.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:42.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:42.739+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:29:43.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.139+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f7354005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:43.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.139+0000 7f7366ffd640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:43.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.143+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:29:43.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.943+0000 7f7366ffd640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f7354005c70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:43.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.943+0000 7f7366ffd640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:43.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:43.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:29:43.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:43.947+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:29:45.467 INFO:tasks.rados.rados.0.smithi203.stdout:274: done (0 left) 2024-09-17T15:29:45.467 INFO:tasks.rados.rados.0.smithi203.stdout:275: delete oid 26 current snap is 28 2024-09-17T15:29:45.488 INFO:tasks.rados.rados.0.smithi203.stdout:275: done (0 left) 2024-09-17T15:29:45.488 INFO:tasks.rados.rados.0.smithi203.stdout:276: read oid 8 snap -1 2024-09-17T15:29:45.488 INFO:tasks.rados.rados.0.smithi203.stdout:276: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:45.488 INFO:tasks.rados.rados.0.smithi203.stdout:277: write oid 27 current snap is 28 2024-09-17T15:29:45.488 INFO:tasks.rados.rados.0.smithi203.stdout:277: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-17T15:29:45.508 INFO:tasks.rados.rados.0.smithi203.stdout:277: writing smithi20328086-27 from 551296 to 1244816 tid 1 2024-09-17T15:29:45.523 INFO:tasks.rados.rados.0.smithi203.stdout:277: writing smithi20328086-27 from 2024048 to 2151296 tid 2 2024-09-17T15:29:45.523 INFO:tasks.rados.rados.0.smithi203.stdout:278: snap_remove snap 24 2024-09-17T15:29:45.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): tick 2024-09-17T15:29:45.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:29:45.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:29:45.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f7354005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f7354005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 msgr2=0x7f7354008860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f73677fe640 1 -- 172.21.15.86:0/1648469135 reap_dead start 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f735400b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f73680a5b60 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f73540063a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f735400b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f7354003840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:45.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f736809c2a0 con 0x7f73540063a0 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7354008f20 con 0x7f73680a5b60 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 1 -- 172.21.15.86:0/1648469135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7354005cd0 con 0x7f7354003840 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.531+0000 7f7364ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73677fe640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f735400b970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73677fe640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f735400b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.86:3300/0 conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f735400b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f7366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient(hunting): handle_auth_done global_id 4542 payload 274 2024-09-17T15:29:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f735400b970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f735400b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 msgr2=0x7f735400b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f735400b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient: _finish_hunting 0 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 monclient: found mon.noname-a 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.535+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736809c580 con 0x7f73540063a0 2024-09-17T15:29:45.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f73667fc640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73540063a0 msgr2=0x7f7354008860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f73667fc640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73540063a0 0x7f7354008860 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f73680a6000 tx=0x7f735c01dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bfb3346c1da964a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 1 -- 172.21.15.86:0/1648469135 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0b3070 con 0x7f73540063a0 2024-09-17T15:29:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: dump: 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: _finish_auth 0 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:29:45.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 1 -- 172.21.15.86:0/1648469135 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f735c0a6af0 con 0x7f73540063a0 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 1 -- 172.21.15.86:0/1648469135 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c00f040 con 0x7f73540063a0 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 5 monclient: authenticate success, global_id 4542 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 4 set_mon_vals no callback set 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f7365ffb640 10 monclient: dump: 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.546 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.547 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 10 monclient: shutdown 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 -- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73540063a0 msgr2=0x7f7354008860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73540063a0 0x7f7354008860 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f73680a6000 tx=0x7f735c01dd30 comp rx=0 tx=0).stop 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 -- 172.21.15.86:0/1648469135 shutdown_connections 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f735400b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 --2- 172.21.15.86:0/1648469135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73540063a0 0x7f7354008860 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 --2- 172.21.15.86:0/1648469135 >> v2:172.21.15.203:3300/0 conn(0x7f73680a5b60 0x7f735400b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 -- 172.21.15.86:0/1648469135 >> 172.21.15.86:0/1648469135 conn(0x7f73680742a0 msgr2=0x7f73680746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 -- 172.21.15.86:0/1648469135 shutdown_connections 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.543+0000 7f736d99d640 1 -- 172.21.15.86:0/1648469135 wait complete. 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient: build_initial_monmap 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient: monmap: 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 Processor -- start 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 -- start start 2024-09-17T15:29:45.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient: init 2024-09-17T15:29:45.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:45.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:29:45.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f73681d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-c con 0x7f7354003840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73540063a0 0x7f73681d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-b con 0x7f73540063a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): picked mon.noname-a con 0x7f73680a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7368138640 con 0x7f73680a5b60 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73680a6a70 con 0x7f73540063a0 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73681387c0 con 0x7f7354003840 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:45.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f7366ffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f73681d06b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f736d99d640 10 monclient(hunting): authenticate will time out at 1118.514405s 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f7366ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f73681d06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49792/0 (socket says 172.21.15.86:49792) 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 -- 172.21.15.86:0/2891824657 learned_addr learned my addr 172.21.15.86:0/2891824657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): get_auth_request con 0x7f73680a5b60 auth_method 0 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73540063a0 0x7f73681d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): get_auth_request con 0x7f73540063a0 auth_method 0 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient(hunting): handle_auth_done global_id 4545 payload 824 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 -- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 msgr2=0x7f73681d06b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f73681d06b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 -- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73540063a0 msgr2=0x7f73681d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73540063a0 0x7f73681d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:29:45.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 monclient: found mon.noname-a 2024-09-17T15:29:45.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:45.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.547+0000 7f73677fe640 1 -- 172.21.15.86:0/2891824657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73681379e0 con 0x7f73680a5b60 2024-09-17T15:29:45.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f73677fe640 1 -- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 msgr2=0x7f73681235f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f73677fe640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73680a62d0 tx=0x7f736000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360017070 con 0x7f73680a5b60 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: dump: 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: _finish_auth 0 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736000a610 con 0x7f73680a5b60 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736000aa00 con 0x7f73680a5b60 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 10 monclient: dump: 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 5 monclient: authenticate success, global_id 4545 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f73657fa640 4 set_mon_vals no callback set 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _renew_subs 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73681250b0 con 0x7f73680a5b60 2024-09-17T15:29:45.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _renew_subs 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7368127410 con 0x7f73680a5b60 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7368138bf0 con 0x7f73680a5b60 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f736001a070 con 0x7f73680a5b60 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 0x7f735403f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.551+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4783+0+0 (secure 0 0 0) 0x7f736004f360 con 0x7f73680a5b60 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.555+0000 7f7366ffd640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 0x7f735403f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.555+0000 7f7366ffd640 10 monclient: get_auth_request con 0x7f735403cc30 auth_method 0 2024-09-17T15:29:45.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.555+0000 7f7366ffd640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 0x7f735403f0f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f73580014b0 tx=0x7f7358001d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.559+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f736001a360 con 0x7f73680a5b60 2024-09-17T15:29:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.559+0000 7f734ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.559+0000 7f734ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.791+0000 7f736d99d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.791+0000 7f736d99d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.791+0000 7f736d99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.791+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7368125c10 con 0x7f73680a5b60 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.791+0000 7f734ffff640 1 -- 172.21.15.86:0/2891824657 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f736004f0d0 con 0x7f73680a5b60 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f734ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:29:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f734ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:45.796 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":40,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:29:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 msgr2=0x7f735403f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 0x7f735403f0f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f73580014b0 tx=0x7f7358001d80 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 10 monclient: shutdown 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 msgr2=0x7f73681235f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73680a62d0 tx=0x7f736000cb90 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 shutdown_connections 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f735403cc30 0x7f735403f0f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73680a5b60 0x7f73681235f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73540063a0 0x7f73681d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 --2- 172.21.15.86:0/2891824657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7354003840 0x7f73681d06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 >> 172.21.15.86:0/2891824657 conn(0x7f73680742a0 msgr2=0x7f73680a1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 shutdown_connections 2024-09-17T15:29:45.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.795+0000 7f736d99d640 1 -- 172.21.15.86:0/2891824657 wait complete. 2024-09-17T15:29:45.814 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:29:45.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:29:45.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:29:45.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: get_monmap_and_config 2024-09-17T15:29:45.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: build_initial_monmap 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: monmap: 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:45.936544+0000 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:45.936544+0000 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:45.936 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 Processor -- start 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 -- start start 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: init 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1b80a4f70 0x7fd1b809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b80a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1b80a5350 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 0x7fd1b813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b813d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:29:45.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b809d430 con 0x7fd1b80a5350 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b809c7f0 con 0x7fd1b80a4f70 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b80a68d0 con 0x7fd1b813d180 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1bcb72640 10 monclient(hunting): authenticate will time out at 1118.902099s 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1b80a4f70 0x7fd1b809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b5d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd1b80a5350 0x7fd1b80a5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41368/0 (socket says 172.21.15.86:41368) 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b5d9b640 1 -- 172.21.15.86:0/1739961890 learned_addr learned my addr 172.21.15.86:0/1739961890 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 --2- 172.21.15.86:0/1739961890 >> v2:172.21.15.86:3300/0 conn(0x7fd1b80a4f70 0x7fd1b809cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49804/0 (socket says 172.21.15.86:49804) 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): get_auth_request con 0x7fd1b80a4f70 auth_method 0 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 -- 172.21.15.86:0/1739961890 >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 msgr2=0x7fd1b813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b6d9d640 1 --2- 172.21.15.86:0/1739961890 >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 0x7fd1b813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 --2- 172.21.15.86:0/1739961890 >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 0x7fd1b813f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 -- 172.21.15.86:0/1739961890 >> v1:172.21.15.86:6789/0 conn(0x7fd1b80a5350 legacy=0x7fd1b80a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient: _finish_hunting 0 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 monclient: found mon.noname-a 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b659c640 1 -- 172.21.15.86:0/1739961890 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b80a64a0 con 0x7fd1b80a4f70 2024-09-17T15:29:45.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b6d9d640 10 monclient: get_auth_request con 0x7fd1b813d180 auth_method 0 2024-09-17T15:29:45.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.935+0000 7fd1b6d9d640 1 --2- 172.21.15.86:0/1739961890 >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 0x7fd1b813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:29:45.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b659c640 1 -- 172.21.15.86:0/1739961890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 msgr2=0x7fd1b809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b659c640 1 --2- 172.21.15.86:0/1739961890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b809cef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1b809c970 tx=0x7fd1b001c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6c05327be61fd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 1 -- 172.21.15.86:0/1739961890 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00b0070 con 0x7fd1b80a4f70 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: dump: 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.943 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: _finish_auth 0 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: _check_auth_tickets 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 1 -- 172.21.15.86:0/1739961890 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00ab600 con 0x7fd1b80a4f70 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 1 -- 172.21.15.86:0/1739961890 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00ab9d0 con 0x7fd1b80a4f70 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1bcb72640 5 monclient: authenticate success, global_id 4548 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1bcb72640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1bcb72640 4 set_mon_vals no callback set 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.939+0000 7fd1b559a640 10 monclient: dump: 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.944 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.945 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: shutdown 2024-09-17T15:29:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- 172.21.15.86:0/1739961890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 msgr2=0x7fd1b809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/1739961890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b809cef0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd1b809c970 tx=0x7fd1b001c2f0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- 172.21.15.86:0/1739961890 shutdown_connections 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/1739961890 >> v2:172.21.15.203:3300/0 conn(0x7fd1b813d180 0x7fd1b813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/1739961890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b809cef0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- 172.21.15.86:0/1739961890 >> 172.21.15.86:0/1739961890 conn(0x7fd1b80a7240 msgr2=0x7fd1b80a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- 172.21.15.86:0/1739961890 shutdown_connections 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- 172.21.15.86:0/1739961890 wait complete. 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: build_initial_monmap 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: monmap: 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:45.950 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 Processor -- start 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- start start 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: init 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b80a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:45.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1b80a5350 0x7fd1b80a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b80a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b813d180 0x7fd1b80a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): picked mon.noname-b con 0x7fd1b813d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8142450 con 0x7fd1b80a4f70 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81422d0 con 0x7fd1b813d180 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b81425d0 con 0x7fd1b80a5350 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1bcb72640 10 monclient(hunting): authenticate will time out at 1118.909841s 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b5d9b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1b80a5350 msgr2=0x7fd1b80a22d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:45.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49814/0 (socket says 172.21.15.86:49814) 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 -- 172.21.15.86:0/2124544912 learned_addr learned my addr 172.21.15.86:0/2124544912 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): get_auth_request con 0x7fd1b80a4f70 auth_method 0 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1b80a5350 0x7fd1b80a22d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient(hunting): handle_auth_done global_id 4551 payload 824 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1b80a5350 msgr2=0x7fd1b80a22d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1b80a5350 0x7fd1b80a22d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b813d180 msgr2=0x7fd1b80a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1b813d180 0x7fd1b80a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient: _finish_hunting 0 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 monclient: found mon.noname-a 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.943+0000 7fd1b659c640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b813de60 con 0x7fd1b80a4f70 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b659c640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 msgr2=0x7fd1b80a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b659c640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0019f20 tx=0x7fd1b0019090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00b9020 con 0x7fd1b80a4f70 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: dump: 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.954 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b00b5890 con 0x7fd1b80a4f70 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 5 monclient: authenticate success, global_id 4551 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 monclient: _renew_subs 2024-09-17T15:29:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b813d910 con 0x7fd1b80a4f70 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b00b0040 con 0x7fd1b80a4f70 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 10 monclient: dump: 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b4d99640 4 set_mon_vals no callback set 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 monclient: _renew_subs 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b80a42c0 con 0x7fd1b80a4f70 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 0x7fd1b81cff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8139b70 con 0x7fd1b81cfba0 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 0x7fd1b81cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient: get_auth_request con 0x7fd1b81cfba0 auth_method 0 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd1b00bc020 con 0x7fd1b80a4f70 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_done global_id 4554 payload 824 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b5d9b640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1b809cef0 con 0x7fd1b81cfba0 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd19803bb10 0x7fd19803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4783+0+0 (secure 0 0 0) 0x7fd1b00ea270 con 0x7fd1b80a4f70 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b6d9d640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd19803bb10 0x7fd19803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.947+0000 7fd1b6d9d640 10 monclient: get_auth_request con 0x7fd19803bb10 auth_method 0 2024-09-17T15:29:45.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1b5d9b640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 msgr2=0x7fd1b81cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1b5d9b640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 0x7fd1b81cff80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1b813eb10 tx=0x7fd1ac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac017070 con 0x7fd1b81cfba0 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 10 monclient: dump: 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1ac0070d0 con 0x7fd1b81cfba0 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 msgr2=0x7fd1b81cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 0x7fd1b81cff80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1b813eb10 tx=0x7fd1ac00c8e0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 msgr2=0x7fd1b81cff80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1a6ffd640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81cfba0 0x7fd1b81cff80 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd1b813eb10 tx=0x7fd1ac00c8e0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.951+0000 7fd1b6d9d640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd19803bb10 0x7fd19803dfd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd1b813c0b0 tx=0x7fd1a80083b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1bcb72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 0x7fd1b809e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:45.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1bcb72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8069920 con 0x7fd1b81d04c0 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 0x7fd1b809e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient: get_auth_request con 0x7fd1b81d04c0 auth_method 0 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 10 monclient(hunting): handle_auth_done global_id 4557 payload 824 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.955+0000 7fd1b5d9b640 1 -- 172.21.15.86:0/2124544912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1b81395b0 con 0x7fd1b81d04c0 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1b5d9b640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 msgr2=0x7fd1b809e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:45.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1b5d9b640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 0x7fd1b809e470 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac011000 tx=0x7fd1ac004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac017050 con 0x7fd1b81d04c0 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 10 monclient: dump: 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:45.964 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7fd1ac018040 con 0x7fd1b81d04c0 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 msgr2=0x7fd1b809e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 0x7fd1b809e470 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac011000 tx=0x7fd1ac004a20 comp rx=0 tx=0).stop 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 msgr2=0x7fd1b809e470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1a6ffd640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b81d04c0 0x7fd1b809e470 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac011000 tx=0x7fd1ac004a20 comp rx=0 tx=0).stop 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:45.965 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 40, 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 158603, 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:45.967 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:45.968 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:45.969 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:45.970 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:45.971 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:45.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:45.973 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:45.974 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:45.975 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:29:45.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:45.978 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:45.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:45.978 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:45.978 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd19803bb10 msgr2=0x7fd19803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd19803bb10 0x7fd19803dfd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd1b813c0b0 tx=0x7fd1a80083b0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 10 monclient: shutdown 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 msgr2=0x7fd1b80a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1b6d9d640 1 -- 172.21.15.86:0/2124544912 reap_dead start 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0019f20 tx=0x7fd1b0019090 comp rx=0 tx=0).stop 2024-09-17T15:29:45.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 shutdown_connections 2024-09-17T15:29:45.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 --2- 172.21.15.86:0/2124544912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1b80a4f70 0x7fd1b80a1d90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:45.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 >> 172.21.15.86:0/2124544912 conn(0x7fd1b80a7240 msgr2=0x7fd1b813c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:45.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 shutdown_connections 2024-09-17T15:29:45.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:45.959+0000 7fd1bcb72640 1 -- 172.21.15.86:0/2124544912 wait complete. 2024-09-17T15:29:45.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:29:45.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:29:45.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:29:46.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: get_monmap_and_config 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: build_initial_monmap 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: monmap: 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:29:46.092646+0000 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:29:46.092646+0000 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:29:46.092 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:29:46.093 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:29:46.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 Processor -- start 2024-09-17T15:29:46.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 -- start start 2024-09-17T15:29:46.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: init 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe52c09d770 addr v1:172.21.15.86:6789/0 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe52c09e260 0x7fe52c139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): picked mon.noname-a con 0x7fe52c09e260 addr v2:172.21.15.86:3300/0 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe52c13a440 addr v1:172.21.15.203:6789/0 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52c09ff00 con 0x7fe52c09d770 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52c09ec30 con 0x7fe52c13a440 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe52c09e7f0 con 0x7fe52c09e260 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe533520640 10 monclient(hunting): authenticate will time out at 1119.058082s 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe52c09e260 0x7fe52c139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe5312bc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe52c09d770 0x7fe52c09db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41370/0 (socket says 172.21.15.86:41370) 2024-09-17T15:29:46.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): get_auth_request con 0x7fe52c09e260 auth_method 0 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe5312bc640 1 -- 172.21.15.86:0/3556507597 learned_addr learned my addr 172.21.15.86:0/3556507597 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient(hunting): handle_auth_done global_id 4560 payload 274 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 1 -- 172.21.15.86:0/3556507597 >> v1:172.21.15.203:6789/0 conn(0x7fe52c13a440 legacy=0x7fe52c13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 1 -- 172.21.15.86:0/3556507597 >> v1:172.21.15.86:6789/0 conn(0x7fe52c09d770 legacy=0x7fe52c09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient: _finish_hunting 0 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 1 monclient: found mon.noname-a 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:46.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:46.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.091+0000 7fe530abb640 1 -- 172.21.15.86:0/3556507597 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52c09ef10 con 0x7fe52c09e260 2024-09-17T15:29:46.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe530abb640 1 -- 172.21.15.86:0/3556507597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 msgr2=0x7fe52c139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe530abb640 1 --2- 172.21.15.86:0/3556507597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c139f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00ba60 tx=0x7fe51c01ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1026045a1f6cb78 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 1 -- 172.21.15.86:0/3556507597 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b4070 con 0x7fe52c09e260 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: dump: 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.099 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: _finish_auth 0 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: _check_auth_tickets 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 5 monclient: authenticate success, global_id 4560 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 1 -- 172.21.15.86:0/3556507597 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c0a7850 con 0x7fe52c09e260 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 10 monclient: get_monmap_and_config success 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 1 -- 172.21.15.86:0/3556507597 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0a7c20 con 0x7fe52c09e260 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 4 set_mon_vals no callback set 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe523fff640 10 monclient: dump: 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 10 monclient: shutdown 2024-09-17T15:29:46.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 -- 172.21.15.86:0/3556507597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 msgr2=0x7fe52c139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 --2- 172.21.15.86:0/3556507597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c139f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00ba60 tx=0x7fe51c01ea30 comp rx=0 tx=0).stop 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 -- 172.21.15.86:0/3556507597 shutdown_connections 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 --2- 172.21.15.86:0/3556507597 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c139f00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 -- 172.21.15.86:0/3556507597 >> 172.21.15.86:0/3556507597 conn(0x7fe52c0a2f40 msgr2=0x7fe52c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 -- 172.21.15.86:0/3556507597 shutdown_connections 2024-09-17T15:29:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.095+0000 7fe533520640 1 -- 172.21.15.86:0/3556507597 wait complete. 2024-09-17T15:29:46.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient: build_initial_monmap 2024-09-17T15:29:46.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient: monmap: 2024-09-17T15:29:46.106 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 Processor -- start 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 -- start start 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient: init 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient: _reopen_session rank -1 2024-09-17T15:29:46.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52c09d770 0x7fe52c1d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): picked mon.noname-c con 0x7fe52c09d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): picked mon.noname-a con 0x7fe52c09e260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c13a440 0x7fe52c0729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): picked mon.noname-b con 0x7fe52c13a440 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe5312bc640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52c09d770 msgr2=0x7fe52c1d0130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c0a1850 con 0x7fe52c09e260 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c0a16d0 con 0x7fe52c13a440 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe52c0a19d0 con 0x7fe52c09d770 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe5312bc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52c09d770 0x7fe52c1d0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): _renew_subs 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe533520640 10 monclient(hunting): authenticate will time out at 1119.065419s 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49862/0 (socket says 172.21.15.86:49862) 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 -- 172.21.15.86:0/400798736 learned_addr learned my addr 172.21.15.86:0/400798736 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): get_auth_request con 0x7fe52c09e260 auth_method 0 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:46.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52c09d770 msgr2=0x7fe52c1d0130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52c09d770 0x7fe52c1d0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c13a440 msgr2=0x7fe52c0729a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe531abd640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c13a440 0x7fe52c0729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c13a440 0x7fe52c0729a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient: _finish_hunting 0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 monclient: found mon.noname-a 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.099+0000 7fe530abb640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52c122720 con 0x7fe52c09e260 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe530abb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 msgr2=0x7fe52c135040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe530abb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0b2040 tx=0x7fe51c01b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe530abb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0b2040 tx=0x7fe51c01b7d0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fe51c0b4040 mon_map magic: 0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b4040 con 0x7fe52c09e260 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: dump: 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: _finish_auth 0 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:29:46.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c0bd070 con 0x7fe52c09e260 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 5 monclient: authenticate success, global_id 4563 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 monclient: _renew_subs 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52c122a00 con 0x7fe52c09e260 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5237fe640 4 set_mon_vals no callback set 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 monclient: _renew_subs 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52c09c6b0 con 0x7fe52c09e260 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 0x7fe52c02d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c121200 con 0x7fe52c02d450 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0afc50 con 0x7fe52c09e260 2024-09-17T15:29:46.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 10 monclient: dump: 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe51c0c2460 con 0x7fe52c09e260 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 0x7fe52c02d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe521ffb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe51003bb40 0x7fe51003e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient: get_auth_request con 0x7fe52c02d450 auth_method 0 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe531abd640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe51003bb40 0x7fe51003e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe531abd640 10 monclient: get_auth_request con 0x7fe51003bb40 auth_method 0 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 10 monclient(hunting): handle_auth_done global_id 14206 payload 824 2024-09-17T15:29:46.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.103+0000 7fe5312bc640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe52c1d10b0 con 0x7fe52c02d450 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe5312bc640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 msgr2=0x7fe52c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe5312bc640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 0x7fe52c02d830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe528004880 tx=0x7fe52800c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe531abd640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe51003bb40 0x7fe51003e000 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe52c074640 tx=0x7fe524009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528017070 con 0x7fe52c02d450 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 10 monclient: dump: 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.116 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5280070d0 con 0x7fe52c02d450 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 msgr2=0x7fe52c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 0x7fe52c02d830 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe528004880 tx=0x7fe52800c8e0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 msgr2=0x7fe52c02d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.107+0000 7fe521ffb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02d450 0x7fe52c02d830 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe528004880 tx=0x7fe52800c8e0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.111+0000 7fe533520640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.111+0000 7fe533520640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 0x7fe52c02e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.111+0000 7fe533520640 10 monclient(hunting): start opening mon connection 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.111+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c09d460 con 0x7fe52c02dd70 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 0x7fe52c02e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient: get_auth_request con 0x7fe52c02dd70 auth_method 0 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:29:46.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:29:46.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 10 monclient(hunting): handle_auth_done global_id 14209 payload 824 2024-09-17T15:29:46.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 1 -- 172.21.15.86:0/400798736 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe52c09db50 con 0x7fe52c02dd70 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 msgr2=0x7fe52c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe5312bc640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 0x7fe52c02e180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe528014000 tx=0x7fe528004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528017030 con 0x7fe52c02dd70 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe521ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe521ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.115+0000 7fe521ffb640 10 monclient: dump: 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:29:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7fe528022410 con 0x7fe52c02dd70 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 msgr2=0x7fe52c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 0x7fe52c02e180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe528014000 tx=0x7fe528004a20 comp rx=0 tx=0).stop 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 msgr2=0x7fe52c02e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:29:46.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe52c02dd70 0x7fe52c02e180 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe528014000 tx=0x7fe528004a20 comp rx=0 tx=0).stop 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 40, 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 94715, 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.122 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:46.123 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:46.124 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:29:46.125 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:29:46.126 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:29:46.127 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.128 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:46.129 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.131 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:29:46.132 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.133 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:29:46.134 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:29:46.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:29:46.135 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:29:46.135 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:29:46.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe51003bb40 msgr2=0x7fe51003e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe51003bb40 0x7fe51003e000 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe52c074640 tx=0x7fe524009040 comp rx=0 tx=0).stop 2024-09-17T15:29:46.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 10 monclient: shutdown 2024-09-17T15:29:46.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:29:46.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 msgr2=0x7fe52c135040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 --2- 172.21.15.86:0/400798736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52c09e260 0x7fe52c135040 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0b2040 tx=0x7fe51c01b7d0 comp rx=0 tx=0).stop 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe531abd640 1 -- 172.21.15.86:0/400798736 reap_dead start 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe521ffb640 1 -- 172.21.15.86:0/400798736 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 4783+0+0 (unknown 0 0 0) 0x7fe51c0ea4f0 con 0x7fe52c09e260 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe531abd640 1 -- 172.21.15.86:0/400798736 reap_dead start 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 shutdown_connections 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 >> 172.21.15.86:0/400798736 conn(0x7fe52c0a2f40 msgr2=0x7fe52c13cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 shutdown_connections 2024-09-17T15:29:46.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:29:46.119+0000 7fe533520640 1 -- 172.21.15.86:0/400798736 wait complete. 2024-09-17T15:29:46.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:29:46.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:29:46.475 INFO:tasks.rados.rados.0.smithi203.stdout:278: done (2 left) 2024-09-17T15:29:46.475 INFO:tasks.rados.rados.0.smithi203.stdout:279: write (excl) oid 26 current snap is 28 2024-09-17T15:29:46.475 INFO:tasks.rados.rados.0.smithi203.stdout:279: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-17T15:29:46.496 INFO:tasks.rados.rados.0.smithi203.stdout:279: writing smithi20328086-26 from 568103 to 1336872 tid 1 2024-09-17T15:29:46.509 INFO:tasks.rados.rados.0.smithi203.stdout:279: writing smithi20328086-26 from 2082530 to 2168103 tid 2 2024-09-17T15:29:46.509 INFO:tasks.rados.rados.0.smithi203.stdout:280: copy_from oid 49 from oid 17 current snap is 28 2024-09-17T15:29:46.509 INFO:tasks.rados.rados.0.smithi203.stdout:276: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:46.526 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:46.584 INFO:tasks.rados.rados.0.smithi203.stdout:277: finishing write tid 1 to smithi20328086-27 2024-09-17T15:29:46.584 INFO:tasks.rados.rados.0.smithi203.stdout:277: finishing write tid 2 to smithi20328086-27 2024-09-17T15:29:46.584 INFO:tasks.rados.rados.0.smithi203.stdout:277: finishing write tid 3 to smithi20328086-27 2024-09-17T15:29:46.584 INFO:tasks.rados.rados.0.smithi203.stdout:277: finishing write tid 4 to smithi20328086-27 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:277: oid 27 updating version 0 to 88 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:277: oid 27 updating version 88 to 90 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:277: oid 27 version 90 is already newer than 89 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 90 (ObjNum 128 snap 28 seq_num 128) dirty exists 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:277: left oid 27 (ObjNum 128 snap 28 seq_num 128) 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: finishing write tid 1 to smithi20328086-26 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: finishing write tid 2 to smithi20328086-26 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: finishing write tid 3 to smithi20328086-26 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: finishing write tid 4 to smithi20328086-26 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: oid 26 updating version 0 to 108 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: oid 26 version 108 is already newer than 107 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: oid 26 version 108 is already newer than 102 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 108 (ObjNum 129 snap 28 seq_num 129) dirty exists 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:279: left oid 26 (ObjNum 129 snap 28 seq_num 129) 2024-09-17T15:29:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:280: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:280: finishing copy_from to smithi20328086-49 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 72 (ObjNum 112 snap 20 seq_num 112) dirty exists 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:276: done (3 left) 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:277: done (2 left) 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:279: done (1 left) 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:280: done (0 left) 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:281: write (excl) oid 27 current snap is 28 2024-09-17T15:29:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:281: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-17T15:29:46.602 INFO:tasks.rados.rados.0.smithi203.stdout:281: writing smithi20328086-27 from 584910 to 1145281 tid 1 2024-09-17T15:29:46.619 INFO:tasks.rados.rados.0.smithi203.stdout:281: writing smithi20328086-27 from 1857365 to 2184910 tid 2 2024-09-17T15:29:46.620 INFO:tasks.rados.rados.0.smithi203.stdout:282: delete oid 18 current snap is 28 2024-09-17T15:29:46.620 INFO:tasks.rados.rados.0.smithi203.stdout:281: finishing write tid 1 to smithi20328086-27 2024-09-17T15:29:46.627 INFO:tasks.rados.rados.0.smithi203.stdout:281: finishing write tid 2 to smithi20328086-27 2024-09-17T15:29:46.628 INFO:tasks.rados.rados.0.smithi203.stdout:281: finishing write tid 3 to smithi20328086-27 2024-09-17T15:29:46.628 INFO:tasks.rados.rados.0.smithi203.stdout:282: done (1 left) 2024-09-17T15:29:46.628 INFO:tasks.rados.rados.0.smithi203.stdout:283: copy_from oid 46 from oid 31 current snap is 28 2024-09-17T15:29:46.629 INFO:tasks.rados.rados.0.smithi203.stdout:284: rollback oid 8 current snap is 28 2024-09-17T15:29:46.629 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 27 2024-09-17T15:29:46.629 INFO:tasks.rados.rados.0.smithi203.stdout:285: snap_remove snap 26 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:285: done (3 left) 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:286: snap_create 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:281: finishing write tid 4 to smithi20328086-27 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:281: oid 27 updating version 0 to 97 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:281: oid 27 updating version 97 to 99 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:281: oid 27 version 99 is already newer than 98 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 99 (ObjNum 130 snap 28 seq_num 130) dirty exists 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:281: left oid 27 (ObjNum 130 snap 28 seq_num 130) 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:284: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:283: finishing copy_from to smithi20328086-46 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:283: got expected ENOENT (src dne) 2024-09-17T15:29:47.405 INFO:tasks.rados.rados.0.smithi203.stdout:284: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:29:47.406 INFO:tasks.rados.rados.0.smithi203.stdout:283: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:29:47.406 INFO:tasks.rados.rados.0.smithi203.stdout:284: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:29:47.406 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 27 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:29:48.404 INFO:tasks.rados.rados.0.smithi203.stdout:281: done (3 left) 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:283: done (2 left) 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:284: done (1 left) 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:286: done (0 left) 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:287: rollback oid 31 current snap is 29 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 28 2024-09-17T15:29:48.405 INFO:tasks.rados.rados.0.smithi203.stdout:288: delete oid 18 current snap is 29 2024-09-17T15:29:48.410 INFO:tasks.rados.rados.0.smithi203.stdout:287: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:29:48.410 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 0 (ObjNum 129 snap 0 seq_num 3084744526) dirty dne 2024-09-17T15:29:48.413 INFO:tasks.rados.rados.0.smithi203.stdout:287: done (1 left) 2024-09-17T15:29:48.413 INFO:tasks.rados.rados.0.smithi203.stdout:288: done (0 left) 2024-09-17T15:29:48.413 INFO:tasks.rados.rados.0.smithi203.stdout:289: read oid 17 snap -1 2024-09-17T15:29:48.414 INFO:tasks.rados.rados.0.smithi203.stdout:289: expect (ObjNum 112 snap 20 seq_num 112) 2024-09-17T15:29:48.414 INFO:tasks.rados.rados.0.smithi203.stdout:290: write (excl) oid 26 current snap is 29 2024-09-17T15:29:48.414 INFO:tasks.rados.rados.0.smithi203.stdout:290: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-17T15:29:48.436 INFO:tasks.rados.rados.0.smithi203.stdout:290: writing smithi20328086-26 from 635331 to 1421449 tid 1 2024-09-17T15:29:48.450 INFO:tasks.rados.rados.0.smithi203.stdout:290: writing smithi20328086-26 from 1916458 to 2235331 tid 2 2024-09-17T15:29:48.450 INFO:tasks.rados.rados.0.smithi203.stdout:291: delete oid 44 current snap is 29 2024-09-17T15:29:48.450 INFO:tasks.rados.rados.0.smithi203.stdout:289: expect (ObjNum 112 snap 20 seq_num 112) 2024-09-17T15:29:48.623 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1551686 2024-09-17T15:29:48.664 INFO:tasks.rados.rados.0.smithi203.stdout:290: finishing write tid 1 to smithi20328086-26 2024-09-17T15:29:48.664 INFO:tasks.rados.rados.0.smithi203.stdout:290: finishing write tid 2 to smithi20328086-26 2024-09-17T15:29:48.664 INFO:tasks.rados.rados.0.smithi203.stdout:290: finishing write tid 3 to smithi20328086-26 2024-09-17T15:29:48.664 INFO:tasks.rados.rados.0.smithi203.stdout:290: finishing write tid 4 to smithi20328086-26 2024-09-17T15:29:48.664 INFO:tasks.rados.rados.0.smithi203.stdout:290: oid 26 updating version 0 to 116 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:290: oid 26 updating version 116 to 117 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:290: oid 26 updating version 117 to 118 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 118 (ObjNum 133 snap 29 seq_num 133) dirty exists 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:290: left oid 26 (ObjNum 133 snap 29 seq_num 133) 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:289: done (2 left) 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:290: done (1 left) 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:291: done (0 left) 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:292: copy_from oid 2 from oid 44 current snap is 29 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:293: read oid 30 snap -1 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:293: expect deleted 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:294: write oid 9 current snap is 29 2024-09-17T15:29:48.665 INFO:tasks.rados.rados.0.smithi203.stdout:294: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-17T15:29:48.683 INFO:tasks.rados.rados.0.smithi203.stdout:294: writing smithi20328086-9 from 668945 to 1205561 tid 1 2024-09-17T15:29:48.699 INFO:tasks.rados.rados.0.smithi203.stdout:294: writing smithi20328086-9 from 1633422 to 2124785 tid 2 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:295: read oid 33 snap -1 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:295: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:296: delete oid 49 current snap is 29 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:292: finishing copy_from to smithi20328086-2 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:292: got expected ENOENT (src dne) 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:292: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:29:48.700 INFO:tasks.rados.rados.0.smithi203.stdout:294: finishing write tid 1 to smithi20328086-9 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: finishing write tid 2 to smithi20328086-9 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: finishing write tid 3 to smithi20328086-9 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: finishing write tid 4 to smithi20328086-9 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: oid 9 updating version 0 to 64 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: oid 9 version 64 is already newer than 63 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: oid 9 version 64 is already newer than 62 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 64 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-17T15:29:48.706 INFO:tasks.rados.rados.0.smithi203.stdout:294: left oid 9 (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:292: done (4 left) 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:293: done (3 left) 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:294: done (2 left) 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:296: done (1 left) 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:297: write oid 7 current snap is 29 2024-09-17T15:29:48.707 INFO:tasks.rados.rados.0.smithi203.stdout:297: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-17T15:29:48.725 INFO:tasks.rados.rados.0.smithi203.stdout:297: writing smithi20328086-7 from 702559 to 1106026 tid 1 2024-09-17T15:29:48.742 INFO:tasks.rados.rados.0.smithi203.stdout:297: writing smithi20328086-7 from 1750386 to 2245418 tid 2 2024-09-17T15:29:48.742 INFO:tasks.rados.rados.0.smithi203.stdout:298: rollback oid 16 current snap is 29 2024-09-17T15:29:48.743 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 27 2024-09-17T15:29:48.743 INFO:tasks.rados.rados.0.smithi203.stdout:295: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:29:48.769 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:48.795 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: finishing write tid 1 to smithi20328086-7 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:298: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 0 (ObjNum 129 snap 0 seq_num 3607094432) dirty dne 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: finishing write tid 2 to smithi20328086-7 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: finishing write tid 3 to smithi20328086-7 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: finishing write tid 4 to smithi20328086-7 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: oid 7 updating version 0 to 40 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: oid 7 version 40 is already newer than 39 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: oid 7 updating version 40 to 41 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 41 (ObjNum 137 snap 29 seq_num 137) dirty exists 2024-09-17T15:29:48.822 INFO:tasks.rados.rados.0.smithi203.stdout:297: left oid 7 (ObjNum 137 snap 29 seq_num 137) 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:295: done (2 left) 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:297: done (1 left) 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:298: done (0 left) 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:299: rollback oid 44 current snap is 29 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 28 2024-09-17T15:29:48.823 INFO:tasks.rados.rados.0.smithi203.stdout:300: snap_remove snap 27 2024-09-17T15:29:49.407 INFO:tasks.rados.rados.0.smithi203.stdout:300: done (1 left) 2024-09-17T15:29:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:301: delete oid 40 current snap is 29 2024-09-17T15:29:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:299: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:29:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:299: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:29:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 98 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:29:49.416 INFO:tasks.rados.rados.0.smithi203.stdout:299: done (1 left) 2024-09-17T15:29:49.416 INFO:tasks.rados.rados.0.smithi203.stdout:301: done (0 left) 2024-09-17T15:29:49.416 INFO:tasks.rados.rados.0.smithi203.stdout:302: snap_create 2024-09-17T15:29:50.415 INFO:tasks.rados.rados.0.smithi203.stdout:302: done (0 left) 2024-09-17T15:29:50.416 INFO:tasks.rados.rados.0.smithi203.stdout:303: copy_from oid 45 from oid 23 current snap is 30 2024-09-17T15:29:50.416 INFO:tasks.rados.rados.0.smithi203.stdout:304: snap_remove snap 29 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:304: done (1 left) 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:305: write oid 13 current snap is 30 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:303: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:303: finishing copy_from to smithi20328086-45 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 27 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-17T15:29:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:305: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-17T15:29:51.439 INFO:tasks.rados.rados.0.smithi203.stdout:305: writing smithi20328086-13 from 736173 to 1290138 tid 1 2024-09-17T15:29:51.455 INFO:tasks.rados.rados.0.smithi203.stdout:305: writing smithi20328086-13 from 1867350 to 2336173 tid 2 2024-09-17T15:29:51.455 INFO:tasks.rados.rados.0.smithi203.stdout:303: done (1 left) 2024-09-17T15:29:51.455 INFO:tasks.rados.rados.0.smithi203.stdout:306: copy_from oid 44 from oid 43 current snap is 30 2024-09-17T15:29:51.455 INFO:tasks.rados.rados.0.smithi203.stdout:305: finishing write tid 1 to smithi20328086-13 2024-09-17T15:29:51.455 INFO:tasks.rados.rados.0.smithi203.stdout:307: write (excl) oid 47 current snap is 30 2024-09-17T15:29:51.456 INFO:tasks.rados.rados.0.smithi203.stdout:307: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-17T15:29:51.476 INFO:tasks.rados.rados.0.smithi203.stdout:307: writing smithi20328086-47 from 752980 to 1382194 tid 1 2024-09-17T15:29:51.487 INFO:tasks.rados.rados.0.smithi203.stdout:307: writing smithi20328086-47 from 1809479 to 2352980 tid 2 2024-09-17T15:29:51.487 INFO:tasks.rados.rados.0.smithi203.stdout:308: snap_remove snap 28 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:308: done (3 left) 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:309: delete oid 39 current snap is 30 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: finishing write tid 2 to smithi20328086-13 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: finishing write tid 3 to smithi20328086-13 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: finishing write tid 4 to smithi20328086-13 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: oid 13 updating version 0 to 53 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: oid 13 updating version 53 to 54 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: oid 13 updating version 54 to 55 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 55 (ObjNum 139 snap 30 seq_num 139) dirty exists 2024-09-17T15:29:52.406 INFO:tasks.rados.rados.0.smithi203.stdout:305: left oid 13 (ObjNum 139 snap 30 seq_num 139) 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: finishing write tid 1 to smithi20328086-47 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:306: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: finishing write tid 2 to smithi20328086-47 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:306: finishing copy_from to smithi20328086-44 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 43 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: finishing write tid 3 to smithi20328086-47 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: finishing write tid 4 to smithi20328086-47 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: oid 47 updating version 0 to 39 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: oid 47 version 39 is already newer than 38 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: oid 47 updating version 39 to 40 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 40 (ObjNum 140 snap 30 seq_num 140) dirty exists 2024-09-17T15:29:52.407 INFO:tasks.rados.rados.0.smithi203.stdout:307: left oid 47 (ObjNum 140 snap 30 seq_num 140) 2024-09-17T15:29:52.414 INFO:tasks.rados.rados.0.smithi203.stdout:305: done (3 left) 2024-09-17T15:29:52.414 INFO:tasks.rados.rados.0.smithi203.stdout:306: done (2 left) 2024-09-17T15:29:52.414 INFO:tasks.rados.rados.0.smithi203.stdout:307: done (1 left) 2024-09-17T15:29:52.414 INFO:tasks.rados.rados.0.smithi203.stdout:309: done (0 left) 2024-09-17T15:29:52.414 INFO:tasks.rados.rados.0.smithi203.stdout:310: snap_create 2024-09-17T15:29:53.419 INFO:tasks.rados.rados.0.smithi203.stdout:310: done (0 left) 2024-09-17T15:29:53.420 INFO:tasks.rados.rados.0.smithi203.stdout:311: snap_create 2024-09-17T15:29:54.431 INFO:tasks.rados.rados.0.smithi203.stdout:311: done (0 left) 2024-09-17T15:29:54.431 INFO:tasks.rados.rados.0.smithi203.stdout:312: snap_remove snap 30 2024-09-17T15:29:55.434 INFO:tasks.rados.rados.0.smithi203.stdout:312: done (0 left) 2024-09-17T15:29:55.434 INFO:tasks.rados.rados.0.smithi203.stdout:313: read oid 3 snap -1 2024-09-17T15:29:55.434 INFO:tasks.rados.rados.0.smithi203.stdout:313: expect deleted 2024-09-17T15:29:55.434 INFO:tasks.rados.rados.0.smithi203.stdout:314: write oid 35 current snap is 32 2024-09-17T15:29:55.434 INFO:tasks.rados.rados.0.smithi203.stdout:314: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-17T15:29:55.459 INFO:tasks.rados.rados.0.smithi203.stdout:314: writing smithi20328086-35 from 786594 to 1566306 tid 1 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:314: writing smithi20328086-35 from 2326443 to 2386594 tid 2 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:315: rollback oid 8 current snap is 32 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 31 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:313: done (2 left) 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:316: rollback oid 38 current snap is 32 2024-09-17T15:29:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 31 2024-09-17T15:29:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:317: delete oid 15 current snap is 32 2024-09-17T15:29:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:316: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:29:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 0 (ObjNum 81 snap 0 seq_num 3082984779) dirty dne 2024-09-17T15:29:55.481 INFO:tasks.rados.rados.0.smithi203.stdout:314: finishing write tid 1 to smithi20328086-35 2024-09-17T15:29:55.481 INFO:tasks.rados.rados.0.smithi203.stdout:314: finishing write tid 2 to smithi20328086-35 2024-09-17T15:29:55.483 INFO:tasks.rados.rados.0.smithi203.stdout:315: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:29:55.483 INFO:tasks.rados.rados.0.smithi203.stdout:314: finishing write tid 3 to smithi20328086-35 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:314: finishing write tid 4 to smithi20328086-35 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:314: oid 35 updating version 0 to 44 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:314: oid 35 version 44 is already newer than 43 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:314: oid 35 version 44 is already newer than 42 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 44 (ObjNum 142 snap 32 seq_num 142) dirty exists 2024-09-17T15:29:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:314: left oid 35 (ObjNum 142 snap 32 seq_num 142) 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:315: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:315: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 46 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:314: done (3 left) 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:315: done (2 left) 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:316: done (1 left) 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:317: done (0 left) 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:318: rollback oid 13 current snap is 32 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 31 2024-09-17T15:29:55.491 INFO:tasks.rados.rados.0.smithi203.stdout:319: rollback oid 11 current snap is 32 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 31 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:320: read oid 43 snap 31 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:320: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:321: read oid 50 snap -1 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:321: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:322: write (excl) oid 46 current snap is 32 2024-09-17T15:29:55.492 INFO:tasks.rados.rados.0.smithi203.stdout:322: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-17T15:29:55.508 INFO:tasks.rados.rados.0.smithi203.stdout:322: writing smithi20328086-46 from 420208 to 950418 tid 1 2024-09-17T15:29:55.528 INFO:tasks.rados.rados.0.smithi203.stdout:322: writing smithi20328086-46 from 1527054 to 2293104 tid 2 2024-09-17T15:29:55.528 INFO:tasks.rados.rados.0.smithi203.stdout:323: delete oid 41 current snap is 32 2024-09-17T15:29:55.528 INFO:tasks.rados.rados.0.smithi203.stdout:320: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:29:55.654 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:55.727 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:55.799 INFO:tasks.rados.rados.0.smithi203.stdout:319: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:29:55.800 INFO:tasks.rados.rados.0.smithi203.stdout:319: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:29:55.800 INFO:tasks.rados.rados.0.smithi203.stdout:321: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:55.800 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:55.825 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1477516 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:318: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:318: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:318: finishing rollback tid 2 to smithi20328086-13 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 61 (ObjNum 139 snap 30 seq_num 139) dirty exists 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:319: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 108 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: finishing write tid 1 to smithi20328086-46 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: finishing write tid 2 to smithi20328086-46 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: finishing write tid 3 to smithi20328086-46 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: finishing write tid 4 to smithi20328086-46 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: oid 46 updating version 0 to 106 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: oid 46 updating version 106 to 107 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:322: oid 46 version 107 is already newer than 105 2024-09-17T15:29:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 107 (ObjNum 144 snap 32 seq_num 144) dirty exists 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:322: left oid 46 (ObjNum 144 snap 32 seq_num 144) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:318: done (5 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:319: done (4 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:320: done (3 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:321: done (2 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:322: done (1 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:323: done (0 left) 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:324: rollback oid 2 current snap is 32 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 31 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:325: rollback oid 48 current snap is 32 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 31 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:326: copy_from oid 47 from oid 36 current snap is 32 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:327: rollback oid 23 current snap is 32 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 31 2024-09-17T15:29:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:328: write oid 34 current snap is 32 2024-09-17T15:29:55.864 INFO:tasks.rados.rados.0.smithi203.stdout:328: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-17T15:29:55.883 INFO:tasks.rados.rados.0.smithi203.stdout:328: writing smithi20328086-34 from 453822 to 1250883 tid 1 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:328: writing smithi20328086-34 from 1760371 to 2413737 tid 2 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:329: read oid 8 snap -1 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:329: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:330: delete oid 10 current snap is 32 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:325: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 0 (ObjNum 113 snap 0 seq_num 3606124432) dirty dne 2024-09-17T15:29:55.902 INFO:tasks.rados.rados.0.smithi203.stdout:324: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:324: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:327: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:324: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 49 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:327: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:327: finishing rollback tid 2 to smithi20328086-23 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 81 (ObjNum 89 snap 11 seq_num 89) dirty exists 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:326: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:326: finishing copy_from to smithi20328086-47 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:29:55.903 INFO:tasks.rados.rados.0.smithi203.stdout:328: finishing write tid 1 to smithi20328086-34 2024-09-17T15:29:55.909 INFO:tasks.rados.rados.0.smithi203.stdout:329: expect (ObjNum 7 snap 0 seq_num 7) 2024-09-17T15:29:55.913 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 767489 2024-09-17T15:29:55.958 INFO:tasks.rados.rados.0.smithi203.stdout:328: finishing write tid 2 to smithi20328086-34 2024-09-17T15:29:55.958 INFO:tasks.rados.rados.0.smithi203.stdout:328: finishing write tid 3 to smithi20328086-34 2024-09-17T15:29:55.958 INFO:tasks.rados.rados.0.smithi203.stdout:324: done (6 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:325: done (5 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:326: done (4 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:327: done (3 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:329: done (2 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:330: done (1 left) 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:331: delete oid 23 current snap is 32 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:328: finishing write tid 4 to smithi20328086-34 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:328: oid 34 updating version 0 to 46 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:328: oid 34 updating version 46 to 48 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:328: oid 34 version 48 is already newer than 47 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 48 (ObjNum 146 snap 32 seq_num 146) dirty exists 2024-09-17T15:29:55.959 INFO:tasks.rados.rados.0.smithi203.stdout:328: left oid 34 (ObjNum 146 snap 32 seq_num 146) 2024-09-17T15:29:55.962 INFO:tasks.rados.rados.0.smithi203.stdout:328: done (1 left) 2024-09-17T15:29:55.963 INFO:tasks.rados.rados.0.smithi203.stdout:331: done (0 left) 2024-09-17T15:29:55.963 INFO:tasks.rados.rados.0.smithi203.stdout:332: read oid 43 snap -1 2024-09-17T15:29:55.963 INFO:tasks.rados.rados.0.smithi203.stdout:332: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:29:55.963 INFO:tasks.rados.rados.0.smithi203.stdout:333: write (excl) oid 45 current snap is 32 2024-09-17T15:29:55.963 INFO:tasks.rados.rados.0.smithi203.stdout:333: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-17T15:29:55.970 INFO:tasks.rados.rados.0.smithi203.stdout:333: writing smithi20328086-45 from 504243 to 1127051 tid 1 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:333: writing smithi20328086-45 from 1819464 to 2336510 tid 2 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:334: read oid 7 snap -1 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:334: expect (ObjNum 137 snap 29 seq_num 137) 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:335: rollback oid 37 current snap is 32 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 31 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:332: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:29:55.981 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.176 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.248 INFO:tasks.rados.rados.0.smithi203.stdout:335: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:29:56.249 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 0 (ObjNum 33 snap 0 seq_num 3586005048) dirty dne 2024-09-17T15:29:56.249 INFO:tasks.rados.rados.0.smithi203.stdout:333: finishing write tid 1 to smithi20328086-45 2024-09-17T15:29:56.249 INFO:tasks.rados.rados.0.smithi203.stdout:334: expect (ObjNum 137 snap 29 seq_num 137) 2024-09-17T15:29:56.249 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.338 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2210261 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: finishing write tid 2 to smithi20328086-45 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: finishing write tid 3 to smithi20328086-45 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: finishing write tid 4 to smithi20328086-45 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: oid 45 updating version 0 to 53 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: oid 45 version 53 is already newer than 51 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: oid 45 version 53 is already newer than 52 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 53 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: left oid 45 (ObjNum 149 snap 32 seq_num 149) 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:332: done (3 left) 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:333: done (2 left) 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:334: done (1 left) 2024-09-17T15:29:56.346 INFO:tasks.rados.rados.0.smithi203.stdout:335: done (0 left) 2024-09-17T15:29:56.347 INFO:tasks.rados.rados.0.smithi203.stdout:336: rollback oid 8 current snap is 32 2024-09-17T15:29:56.347 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 31 2024-09-17T15:29:56.347 INFO:tasks.rados.rados.0.smithi203.stdout:337: read oid 36 snap -1 2024-09-17T15:29:56.347 INFO:tasks.rados.rados.0.smithi203.stdout:337: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:29:56.347 INFO:tasks.rados.rados.0.smithi203.stdout:338: delete oid 17 current snap is 32 2024-09-17T15:29:56.354 INFO:tasks.rados.rados.0.smithi203.stdout:338: done (2 left) 2024-09-17T15:29:56.355 INFO:tasks.rados.rados.0.smithi203.stdout:339: write (excl) oid 40 current snap is 32 2024-09-17T15:29:56.355 INFO:tasks.rados.rados.0.smithi203.stdout:339: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-17T15:29:56.374 INFO:tasks.rados.rados.0.smithi203.stdout:339: writing smithi20328086-40 from 537857 to 1311163 tid 1 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:339: writing smithi20328086-40 from 1936428 to 2457143 tid 2 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:340: read oid 49 snap -1 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:340: expect deleted 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:341: read oid 33 snap -1 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:341: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:29:56.393 INFO:tasks.rados.rados.0.smithi203.stdout:342: rollback oid 41 current snap is 32 2024-09-17T15:29:56.394 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 31 2024-09-17T15:29:56.394 INFO:tasks.rados.rados.0.smithi203.stdout:336: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:29:56.394 INFO:tasks.rados.rados.0.smithi203.stdout:337: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:29:56.394 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.462 INFO:tasks.rados.rados.0.smithi203.stdout:336: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:336: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 51 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: finishing write tid 1 to smithi20328086-40 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:342: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: finishing write tid 2 to smithi20328086-40 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:342: finishing rollback tid 2 to smithi20328086-41 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 46 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: finishing write tid 3 to smithi20328086-40 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: finishing write tid 4 to smithi20328086-40 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: oid 40 updating version 0 to 111 2024-09-17T15:29:56.463 INFO:tasks.rados.rados.0.smithi203.stdout:339: oid 40 updating version 111 to 113 2024-09-17T15:29:56.464 INFO:tasks.rados.rados.0.smithi203.stdout:339: oid 40 version 113 is already newer than 112 2024-09-17T15:29:56.464 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 113 (ObjNum 151 snap 32 seq_num 151) dirty exists 2024-09-17T15:29:56.464 INFO:tasks.rados.rados.0.smithi203.stdout:339: left oid 40 (ObjNum 151 snap 32 seq_num 151) 2024-09-17T15:29:56.464 INFO:tasks.rados.rados.0.smithi203.stdout:341: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:29:56.464 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.488 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.514 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:336: done (5 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:337: done (4 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:339: done (3 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:340: done (2 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:341: done (1 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:342: done (0 left) 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:343: rollback oid 26 current snap is 32 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 31 2024-09-17T15:29:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:344: snap_create 2024-09-17T15:29:56.550 INFO:tasks.rados.rados.0.smithi203.stdout:343: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:29:56.551 INFO:tasks.rados.rados.0.smithi203.stdout:343: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:29:56.551 INFO:tasks.rados.rados.0.smithi203.stdout:343: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:29:56.551 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 132 (ObjNum 133 snap 29 seq_num 133) dirty exists 2024-09-17T15:29:57.411 INFO:tasks.rados.rados.0.smithi203.stdout:343: done (1 left) 2024-09-17T15:29:57.412 INFO:tasks.rados.rados.0.smithi203.stdout:344: done (0 left) 2024-09-17T15:29:57.412 INFO:tasks.rados.rados.0.smithi203.stdout:345: copy_from oid 25 from oid 5 current snap is 33 2024-09-17T15:29:57.412 INFO:tasks.rados.rados.0.smithi203.stdout:346: snap_create 2024-09-17T15:29:57.437 INFO:tasks.rados.rados.0.smithi203.stdout:345: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:29:57.447 INFO:tasks.rados.rados.0.smithi203.stdout:345: finishing copy_from to smithi20328086-25 2024-09-17T15:29:57.447 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 72 (ObjNum 112 snap 20 seq_num 112) dirty exists 2024-09-17T15:29:58.419 INFO:tasks.rados.rados.0.smithi203.stdout:345: done (1 left) 2024-09-17T15:29:58.420 INFO:tasks.rados.rados.0.smithi203.stdout:346: done (0 left) 2024-09-17T15:29:58.420 INFO:tasks.rados.rados.0.smithi203.stdout:347: copy_from oid 48 from oid 45 current snap is 34 2024-09-17T15:29:58.420 INFO:tasks.rados.rados.0.smithi203.stdout:348: read oid 21 snap -1 2024-09-17T15:29:58.420 INFO:tasks.rados.rados.0.smithi203.stdout:348: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-17T15:29:58.420 INFO:tasks.rados.rados.0.smithi203.stdout:349: snap_create 2024-09-17T15:29:58.434 INFO:tasks.rados.rados.0.smithi203.stdout:348: expect (ObjNum 47 snap 0 seq_num 47) 2024-09-17T15:29:58.513 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:29:58.543 INFO:tasks.rados.rados.0.smithi203.stdout:347: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:29:58.543 INFO:tasks.rados.rados.0.smithi203.stdout:347: finishing copy_from to smithi20328086-48 2024-09-17T15:29:58.543 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 53 (ObjNum 149 snap 32 seq_num 149) dirty exists 2024-09-17T15:29:59.421 INFO:tasks.rados.rados.0.smithi203.stdout:347: done (2 left) 2024-09-17T15:29:59.421 INFO:tasks.rados.rados.0.smithi203.stdout:348: done (1 left) 2024-09-17T15:29:59.421 INFO:tasks.rados.rados.0.smithi203.stdout:349: done (0 left) 2024-09-17T15:29:59.421 INFO:tasks.rados.rados.0.smithi203.stdout:350: snap_remove snap 34 2024-09-17T15:30:00.436 INFO:tasks.rados.rados.0.smithi203.stdout:350: done (0 left) 2024-09-17T15:30:00.436 INFO:tasks.rados.rados.0.smithi203.stdout:351: delete oid 28 current snap is 35 2024-09-17T15:30:00.442 INFO:tasks.rados.rados.0.smithi203.stdout:351: done (0 left) 2024-09-17T15:30:00.442 INFO:tasks.rados.rados.0.smithi203.stdout:352: write oid 2 current snap is 35 2024-09-17T15:30:00.442 INFO:tasks.rados.rados.0.smithi203.stdout:352: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-17T15:30:00.449 INFO:tasks.rados.rados.0.smithi203.stdout:352: writing smithi20328086-2 from 571471 to 1211628 tid 1 2024-09-17T15:30:00.465 INFO:tasks.rados.rados.0.smithi203.stdout:352: writing smithi20328086-2 from 1653392 to 2177776 tid 2 2024-09-17T15:30:00.465 INFO:tasks.rados.rados.0.smithi203.stdout:353: snap_remove snap 32 2024-09-17T15:30:01.440 INFO:tasks.rados.rados.0.smithi203.stdout:353: done (1 left) 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:354: read oid 23 snap -1 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:354: expect deleted 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: finishing write tid 1 to smithi20328086-2 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: finishing write tid 2 to smithi20328086-2 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: finishing write tid 3 to smithi20328086-2 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: finishing write tid 4 to smithi20328086-2 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: oid 2 updating version 0 to 53 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: oid 2 version 53 is already newer than 52 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: oid 2 version 53 is already newer than 51 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 53 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: left oid 2 (ObjNum 153 snap 35 seq_num 153) 2024-09-17T15:30:01.441 INFO:tasks.rados.rados.0.smithi203.stdout:352: done (1 left) 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:355: read oid 1 snap -1 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:355: expect (ObjNum 79 snap 5 seq_num 79) 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:356: read oid 18 snap -1 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:356: expect deleted 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:357: copy_from oid 30 from oid 31 current snap is 35 2024-09-17T15:30:01.442 INFO:tasks.rados.rados.0.smithi203.stdout:358: snap_create 2024-09-17T15:30:01.447 INFO:tasks.rados.rados.0.smithi203.stdout:357: finishing copy_from to smithi20328086-30 2024-09-17T15:30:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:357: got expected ENOENT (src dne) 2024-09-17T15:30:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:357: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:30:01.451 INFO:tasks.rados.rados.0.smithi203.stdout:355: expect (ObjNum 79 snap 5 seq_num 79) 2024-09-17T15:30:01.548 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 416820 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:354: done (4 left) 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:355: done (3 left) 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:356: done (2 left) 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:357: done (1 left) 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:358: done (0 left) 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:359: rollback oid 29 current snap is 36 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 29 to 31 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:360: read oid 37 snap -1 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:360: expect deleted 2024-09-17T15:30:02.410 INFO:tasks.rados.rados.0.smithi203.stdout:361: read oid 20 snap -1 2024-09-17T15:30:02.411 INFO:tasks.rados.rados.0.smithi203.stdout:361: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:30:02.411 INFO:tasks.rados.rados.0.smithi203.stdout:362: copy_from oid 38 from oid 47 current snap is 36 2024-09-17T15:30:02.411 INFO:tasks.rados.rados.0.smithi203.stdout:363: write oid 36 current snap is 36 2024-09-17T15:30:02.411 INFO:tasks.rados.rados.0.smithi203.stdout:363: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-17T15:30:02.432 INFO:tasks.rados.rados.0.smithi203.stdout:363: writing smithi20328086-36 from 588278 to 1303684 tid 1 2024-09-17T15:30:02.451 INFO:tasks.rados.rados.0.smithi203.stdout:363: writing smithi20328086-36 from 1711874 to 2379916 tid 2 2024-09-17T15:30:02.451 INFO:tasks.rados.rados.0.smithi203.stdout:364: write (excl) oid 28 current snap is 36 2024-09-17T15:30:02.451 INFO:tasks.rados.rados.0.smithi203.stdout:364: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-17T15:30:02.471 INFO:tasks.rados.rados.0.smithi203.stdout:364: writing smithi20328086-28 from 605085 to 1395740 tid 1 2024-09-17T15:30:02.484 INFO:tasks.rados.rados.0.smithi203.stdout:364: writing smithi20328086-28 from 2170356 to 2605085 tid 2 2024-09-17T15:30:02.484 INFO:tasks.rados.rados.0.smithi203.stdout:365: write oid 44 current snap is 36 2024-09-17T15:30:02.484 INFO:tasks.rados.rados.0.smithi203.stdout:365: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-17T15:30:02.494 INFO:tasks.rados.rados.0.smithi203.stdout:365: writing smithi20328086-44 from 621892 to 1087796 tid 1 2024-09-17T15:30:02.504 INFO:tasks.rados.rados.0.smithi203.stdout:365: writing smithi20328086-44 from 1712485 to 2384196 tid 2 2024-09-17T15:30:02.504 INFO:tasks.rados.rados.0.smithi203.stdout:366: rollback oid 10 current snap is 36 2024-09-17T15:30:02.504 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 31 2024-09-17T15:30:02.504 INFO:tasks.rados.rados.0.smithi203.stdout:361: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:30:02.578 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:359: finishing rollback tid 0 to smithi20328086-29 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:359: finishing rollback tid 1 to smithi20328086-29 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:359: finishing rollback tid 2 to smithi20328086-29 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 75 (ObjNum 103 snap 18 seq_num 103) dirty exists 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:362: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:362: finishing copy_from to smithi20328086-38 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:363: finishing write tid 1 to smithi20328086-36 2024-09-17T15:30:02.606 INFO:tasks.rados.rados.0.smithi203.stdout:363: finishing write tid 2 to smithi20328086-36 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: finishing write tid 3 to smithi20328086-36 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: finishing write tid 4 to smithi20328086-36 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: oid 36 updating version 0 to 135 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: oid 36 updating version 135 to 136 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: oid 36 updating version 136 to 137 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 137 (ObjNum 154 snap 36 seq_num 154) dirty exists 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:363: left oid 36 (ObjNum 154 snap 36 seq_num 154) 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: finishing write tid 1 to smithi20328086-28 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: finishing write tid 2 to smithi20328086-28 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: finishing write tid 3 to smithi20328086-28 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: finishing write tid 4 to smithi20328086-28 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: oid 28 updating version 0 to 140 2024-09-17T15:30:02.607 INFO:tasks.rados.rados.0.smithi203.stdout:364: oid 28 version 140 is already newer than 138 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:364: oid 28 version 140 is already newer than 139 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 140 (ObjNum 155 snap 36 seq_num 155) dirty exists 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:364: left oid 28 (ObjNum 155 snap 36 seq_num 155) 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: finishing write tid 1 to smithi20328086-44 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:366: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:366: finishing rollback tid 2 to smithi20328086-10 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 66 (ObjNum 124 snap 26 seq_num 124) dirty exists 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: finishing write tid 2 to smithi20328086-44 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: finishing write tid 3 to smithi20328086-44 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: finishing write tid 4 to smithi20328086-44 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: oid 44 updating version 0 to 115 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: oid 44 updating version 115 to 117 2024-09-17T15:30:02.608 INFO:tasks.rados.rados.0.smithi203.stdout:365: oid 44 version 117 is already newer than 116 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 117 (ObjNum 156 snap 36 seq_num 156) dirty exists 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:365: left oid 44 (ObjNum 156 snap 36 seq_num 156) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:359: done (7 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:360: done (6 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:361: done (5 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:362: done (4 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:363: done (3 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:364: done (2 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:365: done (1 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:366: done (0 left) 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:367: write (excl) oid 15 current snap is 36 2024-09-17T15:30:02.609 INFO:tasks.rados.rados.0.smithi203.stdout:367: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-17T15:30:02.616 INFO:tasks.rados.rados.0.smithi203.stdout:367: writing smithi20328086-15 from 638699 to 1179852 tid 1 2024-09-17T15:30:02.627 INFO:tasks.rados.rados.0.smithi203.stdout:367: writing smithi20328086-15 from 1770967 to 2302689 tid 2 2024-09-17T15:30:02.627 INFO:tasks.rados.rados.0.smithi203.stdout:368: write oid 34 current snap is 36 2024-09-17T15:30:02.627 INFO:tasks.rados.rados.0.smithi203.stdout:368: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-17T15:30:02.639 INFO:tasks.rados.rados.0.smithi203.stdout:368: writing smithi20328086-34 from 655506 to 1271908 tid 1 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:368: writing smithi20328086-34 from 1829449 to 2504829 tid 2 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:369: copy_from oid 4 from oid 16 current snap is 36 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: finishing write tid 1 to smithi20328086-15 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: finishing write tid 2 to smithi20328086-15 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: finishing write tid 3 to smithi20328086-15 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: finishing write tid 4 to smithi20328086-15 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: oid 15 updating version 0 to 39 2024-09-17T15:30:02.651 INFO:tasks.rados.rados.0.smithi203.stdout:367: oid 15 version 39 is already newer than 38 2024-09-17T15:30:02.652 INFO:tasks.rados.rados.0.smithi203.stdout:367: oid 15 updating version 39 to 40 2024-09-17T15:30:02.652 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 40 (ObjNum 157 snap 36 seq_num 157) dirty exists 2024-09-17T15:30:02.652 INFO:tasks.rados.rados.0.smithi203.stdout:367: left oid 15 (ObjNum 157 snap 36 seq_num 157) 2024-09-17T15:30:02.652 INFO:tasks.rados.rados.0.smithi203.stdout:367: done (2 left) 2024-09-17T15:30:02.652 INFO:tasks.rados.rados.0.smithi203.stdout:370: snap_remove snap 31 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:370: done (2 left) 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:371: write (excl) oid 14 current snap is 36 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: finishing write tid 1 to smithi20328086-34 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:369: finishing copy_from to smithi20328086-4 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:369: got expected ENOENT (src dne) 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:369: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: finishing write tid 2 to smithi20328086-34 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: finishing write tid 3 to smithi20328086-34 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: finishing write tid 4 to smithi20328086-34 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: oid 34 updating version 0 to 51 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: oid 34 version 51 is already newer than 50 2024-09-17T15:30:03.461 INFO:tasks.rados.rados.0.smithi203.stdout:368: oid 34 updating version 51 to 52 2024-09-17T15:30:03.462 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 52 (ObjNum 158 snap 36 seq_num 158) dirty exists 2024-09-17T15:30:03.462 INFO:tasks.rados.rados.0.smithi203.stdout:368: left oid 34 (ObjNum 158 snap 36 seq_num 158) 2024-09-17T15:30:03.462 INFO:tasks.rados.rados.0.smithi203.stdout:371: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-17T15:30:03.479 INFO:tasks.rados.rados.0.smithi203.stdout:371: writing smithi20328086-14 from 672313 to 1363964 tid 1 2024-09-17T15:30:03.492 INFO:tasks.rados.rados.0.smithi203.stdout:371: writing smithi20328086-14 from 1887931 to 2306969 tid 2 2024-09-17T15:30:03.492 INFO:tasks.rados.rados.0.smithi203.stdout:368: done (2 left) 2024-09-17T15:30:03.492 INFO:tasks.rados.rados.0.smithi203.stdout:369: done (1 left) 2024-09-17T15:30:03.492 INFO:tasks.rados.rados.0.smithi203.stdout:372: snap_create 2024-09-17T15:30:03.492 INFO:tasks.rados.rados.0.smithi203.stdout:371: finishing write tid 1 to smithi20328086-14 2024-09-17T15:30:03.499 INFO:tasks.rados.rados.0.smithi203.stdout:371: finishing write tid 2 to smithi20328086-14 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: finishing write tid 3 to smithi20328086-14 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: finishing write tid 4 to smithi20328086-14 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: oid 14 updating version 0 to 88 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: oid 14 version 88 is already newer than 87 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: oid 14 version 88 is already newer than 86 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 88 (ObjNum 159 snap 36 seq_num 159) dirty exists 2024-09-17T15:30:03.500 INFO:tasks.rados.rados.0.smithi203.stdout:371: left oid 14 (ObjNum 159 snap 36 seq_num 159) 2024-09-17T15:30:04.467 INFO:tasks.rados.rados.0.smithi203.stdout:371: done (1 left) 2024-09-17T15:30:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:372: done (0 left) 2024-09-17T15:30:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:373: snap_create 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:373: done (0 left) 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:374: read oid 34 snap -1 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:374: expect (ObjNum 158 snap 36 seq_num 158) 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:375: read oid 43 snap -1 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:375: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:376: copy_from oid 41 from oid 2 current snap is 38 2024-09-17T15:30:05.470 INFO:tasks.rados.rados.0.smithi203.stdout:377: snap_remove snap 36 2024-09-17T15:30:06.138 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:30:06.139 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:30:06.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:30:06.142 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:30:06.142 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:30:06.142 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:30:06.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:30:21.143 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:30:21.144 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:30:21.144 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:30:21.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: get_monmap_and_config 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: build_initial_monmap 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: monmap: 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:21.273694+0000 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:21.273694+0000 2024-09-17T15:30:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:21.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 Processor -- start 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 -- start start 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: init 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-a con 0x7fbde80689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbde8068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbde8139f00 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde809d770 con 0x7fbde8068db0 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde809ed00 con 0x7fbde8139f00 2024-09-17T15:30:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbde809efe0 con 0x7fbde80689d0 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdf0349640 10 monclient(hunting): authenticate will time out at 1154.239291s 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48976/0 (socket says 172.21.15.86:48976) 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 learned_addr learned my addr 172.21.15.86:0/2024390671 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbde80689d0 auth_method 0 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:21.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:21.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:21.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.270+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:21.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.274+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> v1:172.21.15.86:6789/0 conn(0x7fbde8068db0 legacy=0x7fbde80691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:21.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.274+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 >> v1:172.21.15.203:6789/0 conn(0x7fbde8139f00 legacy=0x7fbde813c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbde80689d0 auth_method 0 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:21.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:21.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:21.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.474+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:21.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:21.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbde80689d0 auth_method 0 2024-09-17T15:30:21.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:21.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:21.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:21.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:21.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:21.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:21.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:21.878+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbde80689d0 auth_method 0 2024-09-17T15:30:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:22.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:22.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:22.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:22.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:22.682+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): tick 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 msgr2=0x7fbde809ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.86:3300/0 conn(0x7fbde80689d0 0x7fbde809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 >> v1:172.21.15.203:6789/0 conn(0x7fbde8139f00 legacy=0x7fbde813c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 >> v1:172.21.15.86:6789/0 conn(0x7fbde8068db0 legacy=0x7fbde80691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient: _add_conns ranks=[4,2] 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:24.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): picked mon.noname-c con 0x7fbdcc0037f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): picked mon.noname-b con 0x7fbdcc008540 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbde809efe0 con 0x7fbdcc0037f0 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbdcc00b130 con 0x7fbdcc008540 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbddffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbded8e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.270+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.274+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:24.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:24.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.478+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:24.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:24.882+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:25.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:25.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:25.686+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:26.555 INFO:tasks.rados.rados.0.smithi203.stdout:377: done (3 left) 2024-09-17T15:30:26.555 INFO:tasks.rados.rados.0.smithi203.stdout:378: read oid 9 snap -1 2024-09-17T15:30:26.555 INFO:tasks.rados.rados.0.smithi203.stdout:378: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:30:26.555 INFO:tasks.rados.rados.0.smithi203.stdout:379: snap_remove snap 35 2024-09-17T15:30:27.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): tick 2024-09-17T15:30:27.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 reap_dead start 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbde8139f00 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): picked mon.noname-c con 0x7fbdcc0037f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:27.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): picked mon.noname-b con 0x7fbdcc008540 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde80a04e0 con 0x7fbde8139f00 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbde809efe0 con 0x7fbdcc0037f0 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 1 -- 172.21.15.86:0/2024390671 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbdcc00b130 con 0x7fbdcc008540 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.270+0000 7fbddffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbded8e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_done global_id 34100 payload 274 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 msgr2=0x7fbdcc005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 >> v1:172.21.15.203:6789/0 conn(0x7fbde8139f00 legacy=0x7fbdcc00d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 monclient: found mon.noname-b 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.274+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdcc00fd90 con 0x7fbdcc008540 2024-09-17T15:30:27.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdee8e6640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbde809d3b0 tx=0x7fbde001c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4329f153609e867b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 1 -- 172.21.15.86:0/2024390671 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0005d50 con 0x7fbdcc008540 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: dump: 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 1 -- 172.21.15.86:0/2024390671 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde00ad070 con 0x7fbdcc008540 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 5 monclient: authenticate success, global_id 34100 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 1 -- 172.21.15.86:0/2024390671 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007290 con 0x7fbdcc008540 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbded0e3640 10 monclient: dump: 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:27.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 4 set_mon_vals no callback set 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 10 monclient: shutdown 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 -- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 msgr2=0x7fbdcc00aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbde809d3b0 tx=0x7fbde001c740 comp rx=0 tx=0).stop 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 -- 172.21.15.86:0/2024390671 shutdown_connections 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 --2- 172.21.15.86:0/2024390671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdcc008540 0x7fbdcc00aa00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 --2- 172.21.15.86:0/2024390671 >> v2:172.21.15.203:3300/0 conn(0x7fbdcc0037f0 0x7fbdcc005c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 -- 172.21.15.86:0/2024390671 >> 172.21.15.86:0/2024390671 conn(0x7fbde80a6440 msgr2=0x7fbde80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 -- 172.21.15.86:0/2024390671 shutdown_connections 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.282+0000 7fbdf0349640 1 -- 172.21.15.86:0/2024390671 wait complete. 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient: monmap: 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 Processor -- start 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 -- start start 2024-09-17T15:30:27.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient: init 2024-09-17T15:30:27.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-a con 0x7fbdcc0037f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdcc008540 0x7fbde8133b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-b con 0x7fbdcc008540 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): picked mon.noname-c con 0x7fbde8139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbde81355e0 con 0x7fbdcc0037f0 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8135460 con 0x7fbdcc008540 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbde8135760 con 0x7fbde8139f00 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdf0349640 10 monclient(hunting): authenticate will time out at 1160.254673s 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49014/0 (socket says 172.21.15.86:49014) 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48798/0 (socket says 172.21.15.86:48798) 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 1 -- 172.21.15.86:0/1941250440 learned_addr learned my addr 172.21.15.86:0/1941250440 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request con 0x7fbdcc0037f0 auth_method 0 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): get_auth_request con 0x7fbde8139f00 auth_method 0 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbded8e4640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdcc008540 0x7fbde8133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbded8e4640 10 monclient(hunting): get_auth_request con 0x7fbdcc008540 auth_method 0 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbded8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbded8e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbded8e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient(hunting): handle_auth_done global_id 34103 payload 824 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 -- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdcc008540 msgr2=0x7fbde8133b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdcc008540 0x7fbde8133b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 -- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 msgr2=0x7fbde81cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 monclient: found mon.noname-c 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee8e6640 1 -- 172.21.15.86:0/1941250440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde8121180 con 0x7fbde8139f00 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.286+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdee8e6640 1 -- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 msgr2=0x7fbde8134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdee8e6640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbde00a9000 tx=0x7fbde0019530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00b3de0 con 0x7fbde8139f00 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: dump: 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.299 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde0007070 con 0x7fbde8139f00 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0007460 con 0x7fbde8139f00 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 5 monclient: authenticate success, global_id 34103 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _renew_subs 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddffff640 4 set_mon_vals no callback set 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde809d460 con 0x7fbde8139f00 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 10 monclient: dump: 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _renew_subs 2024-09-17T15:30:27.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde81234e0 con 0x7fbde8139f00 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbde0007600 con 0x7fbde8139f00 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde809ba70 con 0x7fbde8139f00 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 0x7fbdcc03f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(96..96 src has 1..96) ==== 4799+0+0 (secure 0 0 0) 0x7fbde00ee0f0 con 0x7fbde8139f00 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.290+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 0x7fbdcc03f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.294+0000 7fbdee0e5640 10 monclient: get_auth_request con 0x7fbdcc03cd00 auth_method 0 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.294+0000 7fbdee0e5640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 0x7fbdcc03f1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbde4019f40 tx=0x7fbde401b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.298+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbde00164d0 con 0x7fbde8139f00 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.298+0000 7fbddeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.298+0000 7fbddeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:379: done (4 left) 2024-09-17T15:30:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:380: snap_create 2024-09-17T15:30:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:375: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:30:27.536 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:27.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.538+0000 7fbdf0349640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:30:27.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.542+0000 7fbdf0349640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:27.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.542+0000 7fbdf0349640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.542+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbde8121ce0 con 0x7fbde8139f00 2024-09-17T15:30:27.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbddeffd640 1 -- 172.21.15.86:0/1941250440 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fbde00ecae0 con 0x7fbde8139f00 2024-09-17T15:30:27.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbddeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:27.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbddeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:27.548 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 msgr2=0x7fbdcc03f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 0x7fbdcc03f1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbde4019f40 tx=0x7fbde401b040 comp rx=0 tx=0).stop 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 10 monclient: shutdown 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 msgr2=0x7fbde8134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbde00a9000 tx=0x7fbde0019530 comp rx=0 tx=0).stop 2024-09-17T15:30:27.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 shutdown_connections 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbdcc03cd00 0x7fbdcc03f1c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde8139f00 0x7fbde8134040 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbdcc008540 0x7fbde8133b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 --2- 172.21.15.86:0/1941250440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdcc0037f0 0x7fbde81cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 >> 172.21.15.86:0/1941250440 conn(0x7fbde80a6440 msgr2=0x7fbde80a29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 shutdown_connections 2024-09-17T15:30:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.546+0000 7fbdf0349640 1 -- 172.21.15.86:0/1941250440 wait complete. 2024-09-17T15:30:27.567 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:30:27.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:30:27.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:30:27.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.670+0000 7f9dc6708640 10 monclient: get_monmap_and_config 2024-09-17T15:30:27.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.670+0000 7f9dc6708640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.670+0000 7f9dc6708640 10 monclient: monmap: 2024-09-17T15:30:27.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:27.675754+0000 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:27.675754+0000 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:27.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 Processor -- start 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 -- start start 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient: init 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9dc00a4f70 0x7f9dc009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc00a4f70 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dc00a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc013d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc009d430 con 0x7f9dc00a5350 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc009c7f0 con 0x7f9dc00a4f70 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc00a68d0 con 0x7f9dc013d180 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc6708640 10 monclient(hunting): authenticate will time out at 1160.641537s 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48816/0 (socket says 172.21.15.86:48816) 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/3538866407 learned_addr learned my addr 172.21.15.86:0/3538866407 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): get_auth_request con 0x7f9dc013d180 auth_method 0 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_done global_id 34106 payload 274 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/3538866407 >> v2:172.21.15.203:3300/0 conn(0x7f9dc00a4f70 msgr2=0x7f9dc009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dbffff640 1 --2- 172.21.15.86:0/3538866407 >> v2:172.21.15.203:3300/0 conn(0x7f9dc00a4f70 0x7f9dc009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 --2- 172.21.15.86:0/3538866407 >> v2:172.21.15.203:3300/0 conn(0x7f9dc00a4f70 0x7f9dc009cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/3538866407 >> v1:172.21.15.203:6789/0 conn(0x7f9dc00a5350 legacy=0x7f9dc00a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 monclient: found mon.noname-b 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.674+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/3538866407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc00a64a0 con 0x7f9dc013d180 2024-09-17T15:30:27.682 INFO:tasks.rados.rados.0.smithi203.stdout:374: expect (ObjNum 158 snap 36 seq_num 158) 2024-09-17T15:30:27.682 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/3538866407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 msgr2=0x7f9dc013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc4ca5640 1 --2- 172.21.15.86:0/3538866407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9dc006e190 tx=0x7f9db801c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f51a24593a37e7c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 1 -- 172.21.15.86:0/3538866407 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db80a56c0 con 0x7f9dc013d180 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: dump: 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.685 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 1 -- 172.21.15.86:0/3538866407 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db80a5860 con 0x7f9dc013d180 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 5 monclient: authenticate success, global_id 34106 2024-09-17T15:30:27.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 1 -- 172.21.15.86:0/3538866407 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db80a5c30 con 0x7f9dc013d180 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 4 set_mon_vals no callback set 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dbf7fe640 10 monclient: dump: 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.687 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 10 monclient: shutdown 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 -- 172.21.15.86:0/3538866407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 msgr2=0x7f9dc013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 --2- 172.21.15.86:0/3538866407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9dc006e190 tx=0x7f9db801c880 comp rx=0 tx=0).stop 2024-09-17T15:30:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 -- 172.21.15.86:0/3538866407 shutdown_connections 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 --2- 172.21.15.86:0/3538866407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc013d180 0x7f9dc013f570 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 --2- 172.21.15.86:0/3538866407 >> v2:172.21.15.203:3300/0 conn(0x7f9dc00a4f70 0x7f9dc009cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 -- 172.21.15.86:0/3538866407 >> 172.21.15.86:0/3538866407 conn(0x7f9dc00a7240 msgr2=0x7f9dc00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 -- 172.21.15.86:0/3538866407 shutdown_connections 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 1 -- 172.21.15.86:0/3538866407 wait complete. 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.678+0000 7f9dc6708640 10 monclient: monmap: 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:27.689 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 Processor -- start 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 -- start start 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient: init 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 0x7f9dc01cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc00a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00a5350 0x7f9dc01d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-b con 0x7f9dc00a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc013d180 0x7f9dc01d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc013d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc01424e0 con 0x7f9dc013d180 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc006f9a0 con 0x7f9dc00a5350 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc0142660 con 0x7f9dc00a4f70 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc6708640 10 monclient(hunting): authenticate will time out at 1160.648973s 2024-09-17T15:30:27.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 0x7f9dc01cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 0x7f9dc01cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48822/0 (socket says 172.21.15.86:48822) 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 -- 172.21.15.86:0/1382965092 learned_addr learned my addr 172.21.15.86:0/1382965092 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): get_auth_request con 0x7f9dc00a4f70 auth_method 0 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc013d180 0x7f9dc01d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): get_auth_request con 0x7f9dc013d180 auth_method 0 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dc4ca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient(hunting): handle_auth_done global_id 34109 payload 824 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00a5350 msgr2=0x7f9dc01d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9dc00a5350 0x7f9dc01d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc013d180 msgr2=0x7f9dc01d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc013d180 0x7f9dc01d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 monclient: found mon.noname-c 2024-09-17T15:30:27.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc013de80 con 0x7f9dc00a4f70 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 msgr2=0x7f9dc01cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbffff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 0x7f9dc01cfaf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c7f0 tx=0x7f9db000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.682+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db0015070 con 0x7f9dc00a4f70 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: dump: 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 5 monclient: authenticate success, global_id 34109 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db0004240 con 0x7f9dc00a4f70 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db0007830 con 0x7f9dc00a4f70 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbeffd640 4 set_mon_vals no callback set 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 10 monclient: dump: 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 monclient: _renew_subs 2024-09-17T15:30:27.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc013b2d0 con 0x7f9dc00a4f70 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 monclient: _renew_subs 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc0139160 con 0x7f9dc00a4f70 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 0x7f9dc01350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc009cef0 con 0x7f9dc0134d00 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 0x7f9dc01350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient: get_auth_request con 0x7f9dc0134d00 auth_method 0 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9db7fff640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dc00a5ff0 con 0x7f9dc0134d00 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9db0007a10 con 0x7f9dc00a4f70 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9da403bbb0 0x7f9da403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(97..97 src has 1..97) ==== 4811+0+0 (secure 0 0 0) 0x7f9db004ff40 con 0x7f9dc00a4f70 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc4ca5640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9da403bbb0 0x7f9da403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.686+0000 7f9dc4ca5640 10 monclient: get_auth_request con 0x7f9da403bbb0 auth_method 0 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9db7fff640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 msgr2=0x7f9dc01350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9db7fff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 0x7f9dc01350e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9da800c340 tx=0x7f9da800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8017070 con 0x7f9dc0134d00 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 10 monclient: dump: 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9da80047a0 con 0x7f9dc0134d00 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 msgr2=0x7f9dc01350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 0x7f9dc01350e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9da800c340 tx=0x7f9da800c800 comp rx=0 tx=0).stop 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 msgr2=0x7f9dc01350e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dbd7fa640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0134d00 0x7f9dc01350e0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9da800c340 tx=0x7f9da800c800 comp rx=0 tx=0).stop 2024-09-17T15:30:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.690+0000 7f9dc4ca5640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9da403bbb0 0x7f9da403e070 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9dc009d9c0 tx=0x7f9db80a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9dc6708640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9dc6708640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 0x7f9dc009df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9dc6708640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc0069920 con 0x7f9dc0135620 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 0x7f9dc009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient: get_auth_request con 0x7f9dc0135620 auth_method 0 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 10 monclient(hunting): handle_auth_done global_id 4575 payload 824 2024-09-17T15:30:27.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.694+0000 7f9db7fff640 1 -- 172.21.15.86:0/1382965092 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9dc013aae0 con 0x7f9dc0135620 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9db7fff640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 msgr2=0x7f9dc009df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9db7fff640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 0x7f9dc009df40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9da8014000 tx=0x7f9da8003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da8017030 con 0x7f9dc0135620 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 10 monclient: dump: 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f9da80192d0 con 0x7f9dc0135620 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 msgr2=0x7f9dc009df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 0x7f9dc009df40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9da8014000 tx=0x7f9da8003770 comp rx=0 tx=0).stop 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 msgr2=0x7f9dc009df40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dbd7fa640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc0135620 0x7f9dc009df40 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9da8014000 tx=0x7f9da8003770 comp rx=0 tx=0).stop 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 200342, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:30:27.708 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.709 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:27.711 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.712 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:27.714 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:27.715 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:27.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.718 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:27.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:27.720 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:27.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9da403bbb0 msgr2=0x7f9da403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9da403bbb0 0x7f9da403e070 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9dc009d9c0 tx=0x7f9db80a7070 comp rx=0 tx=0).stop 2024-09-17T15:30:27.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 10 monclient: shutdown 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 msgr2=0x7f9dc01cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 --2- 172.21.15.86:0/1382965092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc00a4f70 0x7f9dc01cfaf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c7f0 tx=0x7f9db000ccb0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/1382965092 reap_dead start 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc4ca5640 1 -- 172.21.15.86:0/1382965092 reap_dead start 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 shutdown_connections 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 >> 172.21.15.86:0/1382965092 conn(0x7f9dc00a7240 msgr2=0x7f9dc013c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 shutdown_connections 2024-09-17T15:30:27.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.698+0000 7f9dc6708640 1 -- 172.21.15.86:0/1382965092 wait complete. 2024-09-17T15:30:27.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:27.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:30:27.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:30:27.828 INFO:tasks.rados.rados.0.smithi203.stdout:376: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:30:27.828 INFO:tasks.rados.rados.0.smithi203.stdout:376: finishing copy_from to smithi20328086-41 2024-09-17T15:30:27.828 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 53 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-17T15:30:27.828 INFO:tasks.rados.rados.0.smithi203.stdout:378: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:30:27.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: get_monmap_and_config 2024-09-17T15:30:27.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: monmap: 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:27.836241+0000 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:27.836241+0000 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:27.836 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 Processor -- start 2024-09-17T15:30:27.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 -- start start 2024-09-17T15:30:27.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: init 2024-09-17T15:30:27.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d40a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-b con 0x7ff5d40a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5d40a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff5d4136160 0x7ff5d413df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-c con 0x7ff5d4136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5d40a2130 con 0x7ff5d40a1590 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d40a2410 con 0x7ff5d4136160 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d40a26f0 con 0x7ff5d40a0c70 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5dbe33640 10 monclient(hunting): authenticate will time out at 1160.801738s 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d93ce640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff5d40a1590 0x7ff5d4135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40050/0 (socket says 172.21.15.86:40050) 2024-09-17T15:30:27.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d93ce640 1 -- 172.21.15.86:0/2745466320 learned_addr learned my addr 172.21.15.86:0/2745466320 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d40a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): get_auth_request con 0x7ff5d40a0c70 auth_method 0 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5da3d0640 1 --2- 172.21.15.86:0/2745466320 >> v2:172.21.15.203:3300/0 conn(0x7ff5d4136160 0x7ff5d413df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5da3d0640 10 monclient(hunting): get_auth_request con 0x7ff5d4136160 auth_method 0 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5da3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5da3d0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5da3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_done global_id 34112 payload 274 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/2745466320 >> v2:172.21.15.203:3300/0 conn(0x7ff5d4136160 msgr2=0x7ff5d413df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/2745466320 >> v2:172.21.15.203:3300/0 conn(0x7ff5d4136160 0x7ff5d413df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/2745466320 >> v1:172.21.15.86:6789/0 conn(0x7ff5d40a1590 legacy=0x7ff5d4135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 monclient: found mon.noname-b 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.834+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/2745466320 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d409f9f0 con 0x7ff5d40a0c70 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 msgr2=0x7ff5d40a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d40a1050 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009d90 tx=0x7ff5c401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b41b6ef219ce459 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 1 -- 172.21.15.86:0/2745466320 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c40b2070 con 0x7ff5d40a0c70 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: dump: 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.842 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 1 -- 172.21.15.86:0/2745466320 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c40a5890 con 0x7ff5d40a0c70 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 5 monclient: authenticate success, global_id 34112 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 1 -- 172.21.15.86:0/2745466320 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c40a5c60 con 0x7ff5d40a0c70 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 4 set_mon_vals no callback set 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5d8bcd640 10 monclient: dump: 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.845 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 10 monclient: shutdown 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 -- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 msgr2=0x7ff5d40a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d40a1050 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009d90 tx=0x7ff5c401c830 comp rx=0 tx=0).stop 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 -- 172.21.15.86:0/2745466320 shutdown_connections 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/2745466320 >> v2:172.21.15.203:3300/0 conn(0x7ff5d4136160 0x7ff5d413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/2745466320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d40a1050 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 -- 172.21.15.86:0/2745466320 >> 172.21.15.86:0/2745466320 conn(0x7ff5d40a2f40 msgr2=0x7ff5d40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 -- 172.21.15.86:0/2745466320 shutdown_connections 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 1 -- 172.21.15.86:0/2745466320 wait complete. 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.838+0000 7ff5dbe33640 10 monclient: monmap: 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.850 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 Processor -- start 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- start start 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: init 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d41cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-c con 0x7ff5d40a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5d40a1590 0x7ff5d412fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-b con 0x7ff5d40a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5d4136160 0x7ff5d41301a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): picked mon.noname-a con 0x7ff5d4136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d40a00d0 con 0x7ff5d4136160 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d409ff50 con 0x7ff5d40a1590 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d4140c90 con 0x7ff5d40a0c70 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient(hunting): authenticate will time out at 1160.808835s 2024-09-17T15:30:27.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d41cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d41cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48838/0 (socket says 172.21.15.86:48838) 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/1112291590 learned_addr learned my addr 172.21.15.86:0/1112291590 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5d4136160 0x7ff5d41301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): get_auth_request con 0x7ff5d40a0c70 auth_method 0 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): get_auth_request con 0x7ff5d4136160 auth_method 0 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d93ce640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5d40a1590 0x7ff5d412fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d93ce640 10 monclient(hunting): get_auth_request con 0x7ff5d40a1590 auth_method 0 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d93ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d93ce640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d93ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient(hunting): handle_auth_done global_id 34115 payload 824 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5d40a1590 msgr2=0x7ff5d412fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5d40a1590 0x7ff5d412fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5d4136160 msgr2=0x7ff5d41301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5d4136160 0x7ff5d41301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 monclient: found mon.noname-c 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d4071d50 con 0x7ff5d40a0c70 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5da3d0640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5d4136160 0x7ff5d41301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 msgr2=0x7ff5d41cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5d9bcf640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d41cff60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5c401c660 tx=0x7ff5c40195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c40bd020 con 0x7ff5d40a0c70 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: dump: 2024-09-17T15:30:27.855 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: _finish_auth 0 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:27.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c40b3040 con 0x7ff5d40a0c70 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 5 monclient: authenticate success, global_id 34115 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _renew_subs 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d40730c0 con 0x7ff5d40a0c70 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c40b2070 con 0x7ff5d40a0c70 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5ca7fc640 10 monclient: dump: 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5cbfff640 4 set_mon_vals no callback set 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _renew_subs 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.842+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d40733a0 con 0x7ff5d40a0c70 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5dbe33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 0x7ff5d402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d409dd40 con 0x7ff5d4070480 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 0x7ff5d402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient: get_auth_request con 0x7ff5d4070480 auth_method 0 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff5c40c0020 con 0x7ff5d40a0c70 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_done global_id 34118 payload 824 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5d4072160 con 0x7ff5d4070480 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5c003bbb0 0x7ff5c003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(97..97 src has 1..97) ==== 4811+0+0 (secure 0 0 0) 0x7ff5c40ead30 con 0x7ff5d40a0c70 2024-09-17T15:30:27.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5da3d0640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5c003bbb0 0x7ff5c003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5da3d0640 10 monclient: get_auth_request con 0x7ff5c003bbb0 auth_method 0 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 msgr2=0x7ff5d402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5d93ce640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 0x7ff5d402d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e500 tx=0x7ff5d000e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0004e40 con 0x7ff5d4070480 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 10 monclient: dump: 2024-09-17T15:30:27.862 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff5d0018070 con 0x7ff5d4070480 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 msgr2=0x7ff5d402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 0x7ff5d402d450 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e500 tx=0x7ff5d000e9c0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 msgr2=0x7ff5d402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.846+0000 7ff5ca7fc640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070480 0x7ff5d402d450 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e500 tx=0x7ff5d000e9c0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.850+0000 7ff5da3d0640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5c003bbb0 0x7ff5c003e070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4132eb0 tx=0x7ff5cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5dbe33640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 0x7ff5d402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5dbe33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d4135600 con 0x7ff5d4070860 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 0x7ff5d402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient: get_auth_request con 0x7ff5d4070860 auth_method 0 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 10 monclient(hunting): handle_auth_done global_id 34121 payload 824 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.854+0000 7ff5d93ce640 1 -- 172.21.15.86:0/1112291590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5d4133680 con 0x7ff5d4070860 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5d93ce640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 msgr2=0x7ff5d402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5d93ce640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 0x7ff5d402d990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e4d0 tx=0x7ff5d0004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00064e0 con 0x7ff5d4070860 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 10 monclient: dump: 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.865 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7ff5d00064e0 con 0x7ff5d4070860 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 msgr2=0x7ff5d402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 0x7ff5d402d990 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e4d0 tx=0x7ff5d0004950 comp rx=0 tx=0).stop 2024-09-17T15:30:27.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 msgr2=0x7ff5d402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5ca7fc640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d4070860 0x7ff5d402d990 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000e4d0 tx=0x7ff5d0004950 comp rx=0 tx=0).stop 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 21639, 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:27.867 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.868 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.869 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:27.870 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:27.871 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:27.871 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:27.871 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:27.872 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:27.873 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.876 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:27.877 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:27.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:27.880 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:27.881 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5c003bbb0 msgr2=0x7ff5c003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5c003bbb0 0x7ff5c003e070 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4132eb0 tx=0x7ff5cc008040 comp rx=0 tx=0).stop 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 10 monclient: shutdown 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 msgr2=0x7ff5d41cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 --2- 172.21.15.86:0/1112291590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5d40a0c70 0x7ff5d41cff60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff5c401c660 tx=0x7ff5c40195d0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5da3d0640 1 -- 172.21.15.86:0/1112291590 reap_dead start 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5da3d0640 1 -- 172.21.15.86:0/1112291590 reap_dead start 2024-09-17T15:30:27.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 shutdown_connections 2024-09-17T15:30:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 >> 172.21.15.86:0/1112291590 conn(0x7ff5d40a2f40 msgr2=0x7ff5d4132f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 shutdown_connections 2024-09-17T15:30:27.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.858+0000 7ff5dbe33640 1 -- 172.21.15.86:0/1112291590 wait complete. 2024-09-17T15:30:27.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:27.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:30:27.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:30:27.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: get_monmap_and_config 2024-09-17T15:30:27.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: build_initial_monmap 2024-09-17T15:30:27.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: monmap: 2024-09-17T15:30:27.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:27.988831+0000 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:27.988831+0000 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:27.989 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:27.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 Processor -- start 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 -- start start 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: init 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 0x7f875809db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-b con 0x7f875809d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f875809e090 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8758135150 0x7f875813cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-a con 0x7f8758135150 addr v2:172.21.15.86:3300/0 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875809ff00 con 0x7f875809e090 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f875809e7f0 con 0x7f8758135150 2024-09-17T15:30:27.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f87580a2110 con 0x7f875809d770 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875fcba640 10 monclient(hunting): authenticate will time out at 1160.954534s 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8758135150 0x7f875813cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 0x7f875809db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875d255640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f875809e090 0x7f8758134a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40066/0 (socket says 172.21.15.86:40066) 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8758135150 0x7f875813cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49066/0 (socket says 172.21.15.86:49066) 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875d255640 1 -- 172.21.15.86:0/626348933 learned_addr learned my addr 172.21.15.86:0/626348933 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): get_auth_request con 0x7f8758135150 auth_method 0 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): get_auth_request con 0x7f875809d770 auth_method 0 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient(hunting): handle_auth_done global_id 4581 payload 274 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 -- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 msgr2=0x7f875809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 0x7f875809db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:27.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 -- 172.21.15.86:0/626348933 >> v1:172.21.15.86:6789/0 conn(0x7f875809e090 legacy=0x7f8758134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient: _finish_hunting 0 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 monclient: found mon.noname-a 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875e257640 1 -- 172.21.15.86:0/626348933 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875809ec30 con 0x7f8758135150 2024-09-17T15:30:27.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.986+0000 7f875da56640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 0x7f875809db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:27.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875e257640 1 -- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8758135150 msgr2=0x7f875813cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875e257640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8758135150 0x7f875813cf20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8750009d90 tx=0x7f875001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748cefdf4f7e74e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 1 -- 172.21.15.86:0/626348933 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500ac070 con 0x7f8758135150 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: dump: 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:27.999 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: _finish_auth 0 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: _check_auth_tickets 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 1 -- 172.21.15.86:0/626348933 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87500a6680 con 0x7f8758135150 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 5 monclient: authenticate success, global_id 4581 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 1 -- 172.21.15.86:0/626348933 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500a6a50 con 0x7f8758135150 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 4 set_mon_vals no callback set 2024-09-17T15:30:28.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875ca54640 10 monclient: dump: 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 10 monclient: shutdown 2024-09-17T15:30:28.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 -- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8758135150 msgr2=0x7f875813cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8758135150 0x7f875813cf20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8750009d90 tx=0x7f875001c830 comp rx=0 tx=0).stop 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 -- 172.21.15.86:0/626348933 shutdown_connections 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8758135150 0x7f875813cf20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 --2- 172.21.15.86:0/626348933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f875809d770 0x7f875809db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.990+0000 7f875fcba640 1 -- 172.21.15.86:0/626348933 >> 172.21.15.86:0/626348933 conn(0x7f87580a2f40 msgr2=0x7f87580a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- 172.21.15.86:0/626348933 shutdown_connections 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- 172.21.15.86:0/626348933 wait complete. 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient: build_initial_monmap 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient: monmap: 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:28.002 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 Processor -- start 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- start start 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient: init 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:30:28.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f875809d770 0x7f87581cdf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-a con 0x7f875809d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875809e090 0x7f875811e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-b con 0x7f875809e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): picked mon.noname-c con 0x7f8758135150 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f875809cce0 con 0x7f875809d770 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f875809ce60 con 0x7f875809e090 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f875813fc80 con 0x7f8758135150 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875fcba640 10 monclient(hunting): authenticate will time out at 1160.961533s 2024-09-17T15:30:28.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f875809d770 0x7f87581cdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48888/0 (socket says 172.21.15.86:48888) 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 learned_addr learned my addr 172.21.15.86:0/4070694276 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): get_auth_request con 0x7f8758135150 auth_method 0 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f875809d770 0x7f87581cdf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49078/0 (socket says 172.21.15.86:49078) 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): get_auth_request con 0x7f875809d770 auth_method 0 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient(hunting): handle_auth_done global_id 34127 payload 824 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875809e090 msgr2=0x7f875811e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875d255640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875809e090 0x7f875811e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875809e090 0x7f875811e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f875809d770 msgr2=0x7f87581cdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f875809d770 0x7f87581cdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient: _finish_hunting 0 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 monclient: found mon.noname-c 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f875811f420 con 0x7f8758135150 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875d255640 10 monclient: get_auth_request con 0x7f875809e090 auth_method 0 2024-09-17T15:30:28.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.994+0000 7f875d255640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875809e090 0x7f875811e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 msgr2=0x7f8758134220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875e257640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8750009d90 tx=0x7f87500195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500c0070 con 0x7f8758135150 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: dump: 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 5 monclient: authenticate success, global_id 34127 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 monclient: _renew_subs 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87500a6480 con 0x7f8758135150 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:28.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f875809d460 con 0x7f8758135150 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87500ac040 con 0x7f8758135150 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 10 monclient: dump: 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.014 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874ffff640 4 set_mon_vals no callback set 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 monclient: _renew_subs 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f875811fcc0 con 0x7f8758135150 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f87500c3070 con 0x7f8758135150 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 0x7f875811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87580a07a0 con 0x7f87581347c0 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f873c03bbb0 0x7f873c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(97..97 src has 1..97) ==== 4811+0+0 (secure 0 0 0) 0x7f87500ea4f0 con 0x7f8758135150 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 0x7f875811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient: get_auth_request con 0x7f87581347c0 auth_method 0 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875d255640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f873c03bbb0 0x7f873c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875d255640 10 monclient: get_auth_request con 0x7f873c03bbb0 auth_method 0 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 10 monclient(hunting): handle_auth_done global_id 14224 payload 824 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:27.998+0000 7f875da56640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8758134ba0 con 0x7f87581347c0 2024-09-17T15:30:28.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f875d255640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f873c03bbb0 0x7f873c03e070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8748007fd0 tx=0x7f8748009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f875da56640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 msgr2=0x7f875811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f875da56640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 0x7f875811c8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f8754011e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754014440 con 0x7f87581347c0 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 10 monclient: dump: 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.017 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8754014440 con 0x7f87581347c0 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 msgr2=0x7f875811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 0x7f875811c8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f8754011e90 comp rx=0 tx=0).stop 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 msgr2=0x7f875811c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.002+0000 7f874e7fc640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87581347c0 0x7f875811c8c0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f8754011e90 comp rx=0 tx=0).stop 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875fcba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875fcba640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 0x7f875811d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875fcba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f875811d750 con 0x7f875811ce00 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875da56640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 0x7f875811d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875da56640 10 monclient: get_auth_request con 0x7f875811ce00 auth_method 0 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875da56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875da56640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.006+0000 7f875da56640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 10 monclient(hunting): handle_auth_done global_id 14227 payload 824 2024-09-17T15:30:28.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 1 -- 172.21.15.86:0/4070694276 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f875809db50 con 0x7f875811ce00 2024-09-17T15:30:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 msgr2=0x7f875811d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:28.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f875da56640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 0x7f875811d210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f87540046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8754022980 con 0x7f875811ce00 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f874e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f874e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.010+0000 7f874e7fc640 10 monclient: dump: 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.021 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f875402b470 con 0x7f875811ce00 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 msgr2=0x7f875811d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 0x7f875811d210 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f87540046d0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 msgr2=0x7f875811d210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:28.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f874e7fc640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f875811ce00 0x7f875811d210 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87540119d0 tx=0x7f87540046d0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 136610, 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:28.023 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:28.024 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:28.025 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:28.026 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:28.027 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:28.028 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:28.029 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:28.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:28.031 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.033 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.034 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.035 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.036 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:28.037 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f873c03bbb0 msgr2=0x7f873c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f873c03bbb0 0x7f873c03e070 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8748007fd0 tx=0x7f8748009040 comp rx=0 tx=0).stop 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 10 monclient: shutdown 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 msgr2=0x7f8758134220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8750009d90 tx=0x7f87500195d0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875e257640 1 -- 172.21.15.86:0/4070694276 reap_dead start 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 shutdown_connections 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 --2- 172.21.15.86:0/4070694276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8758135150 0x7f8758134220 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 >> 172.21.15.86:0/4070694276 conn(0x7f87580a2f40 msgr2=0x7f8758072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:28.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 shutdown_connections 2024-09-17T15:30:28.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.014+0000 7f875fcba640 1 -- 172.21.15.86:0/4070694276 wait complete. 2024-09-17T15:30:28.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:28.040 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:30:28.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: get_monmap_and_config 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: build_initial_monmap 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: monmap: 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:28.149443+0000 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:28.149443+0000 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:28.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:28.150 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:28.150 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:28.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 Processor -- start 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 -- start start 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: init 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 0x7ff40c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c0a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff40c0a5450 0x7ff40c0a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c0a5450 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff40c137e30 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:28.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff40c0a6f40 con 0x7ff40c137e30 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff40c0a65e0 con 0x7ff40c0a5450 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c09c2a0 con 0x7ff40c0a5070 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff41169a640 10 monclient(hunting): authenticate will time out at 1161.114831s 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 0x7ff40c09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff40c137e30 0x7ff40c138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37136/0 (socket says 172.21.15.86:37136) 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40b7fe640 1 -- 172.21.15.86:0/1830983571 learned_addr learned my addr 172.21.15.86:0/1830983571 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): get_auth_request con 0x7ff40c0a5070 auth_method 0 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 1 --2- 172.21.15.86:0/1830983571 >> v2:172.21.15.203:3300/0 conn(0x7ff40c0a5450 0x7ff40c0a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): get_auth_request con 0x7ff40c0a5450 auth_method 0 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient(hunting): handle_auth_done global_id 34130 payload 274 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 -- 172.21.15.86:0/1830983571 >> v2:172.21.15.203:3300/0 conn(0x7ff40c0a5450 msgr2=0x7ff40c0a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 --2- 172.21.15.86:0/1830983571 >> v2:172.21.15.203:3300/0 conn(0x7ff40c0a5450 0x7ff40c0a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 -- 172.21.15.86:0/1830983571 >> v1:172.21.15.86:6789/0 conn(0x7ff40c137e30 legacy=0x7ff40c138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient: _finish_hunting 0 2024-09-17T15:30:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 monclient: found mon.noname-b 2024-09-17T15:30:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:28.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.146+0000 7ff40affd640 1 -- 172.21.15.86:0/1830983571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c09c580 con 0x7ff40c0a5070 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.150+0000 7ff40affd640 1 -- 172.21.15.86:0/1830983571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 msgr2=0x7ff40c09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.150+0000 7ff40affd640 1 --2- 172.21.15.86:0/1830983571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 0x7ff40c09cf60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4000097c0 tx=0x7ff40001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63ed119d32b87a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 1 -- 172.21.15.86:0/1830983571 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000b0070 con 0x7ff40c0a5070 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: dump: 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: _finish_auth 0 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 1 -- 172.21.15.86:0/1830983571 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4000ab600 con 0x7ff40c0a5070 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:28.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 1 -- 172.21.15.86:0/1830983571 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4000ab9d0 con 0x7ff40c0a5070 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff409ffb640 10 monclient: dump: 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.158 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 5 monclient: authenticate success, global_id 34130 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 4 set_mon_vals no callback set 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: shutdown 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- 172.21.15.86:0/1830983571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 msgr2=0x7ff40c09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- 172.21.15.86:0/1830983571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 0x7ff40c09cf60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff4000097c0 tx=0x7ff40001c360 comp rx=0 tx=0).stop 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- 172.21.15.86:0/1830983571 shutdown_connections 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- 172.21.15.86:0/1830983571 >> v2:172.21.15.203:3300/0 conn(0x7ff40c0a5450 0x7ff40c0a5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- 172.21.15.86:0/1830983571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5070 0x7ff40c09cf60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- 172.21.15.86:0/1830983571 >> 172.21.15.86:0/1830983571 conn(0x7ff40c0742a0 msgr2=0x7ff40c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- 172.21.15.86:0/1830983571 shutdown_connections 2024-09-17T15:30:28.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- 172.21.15.86:0/1830983571 wait complete. 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: build_initial_monmap 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: monmap: 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 Processor -- start 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- start start 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: init 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-c con 0x7ff40c0a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 0x7ff40c13d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-a con 0x7ff40c137e30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c137330 0x7ff40c1d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): picked mon.noname-b con 0x7ff40c137330 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c142ac0 con 0x7ff40c137e30 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff40c0a6a70 con 0x7ff40c137330 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff40c142c40 con 0x7ff40c0a5450 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff41169a640 10 monclient(hunting): authenticate will time out at 1161.122444s 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 0x7ff40c13d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39636/0 (socket says 172.21.15.86:39636) 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): get_auth_request con 0x7ff40c137e30 auth_method 0 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 1 -- 172.21.15.86:0/299289949 learned_addr learned my addr 172.21.15.86:0/299289949 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): get_auth_request con 0x7ff40c0a5450 auth_method 0 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient(hunting): handle_auth_done global_id 4587 payload 824 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 -- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 msgr2=0x7ff40c13fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40b7fe640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c137330 0x7ff40c1d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 -- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c137330 msgr2=0x7ff40c1d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c137330 0x7ff40c1d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 monclient: found mon.noname-a 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40a7fc640 1 -- 172.21.15.86:0/299289949 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff40c0a1860 con 0x7ff40c137e30 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.154+0000 7ff40affd640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff40a7fc640 1 -- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 msgr2=0x7ff40c13d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff40a7fc640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 0x7ff40c13d8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00efc0 tx=0x7ff3fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc018020 con 0x7ff40c137e30 2024-09-17T15:30:28.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: dump: 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: _finish_auth 0 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3fc014680 con 0x7ff40c137e30 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc014a50 con 0x7ff40c137e30 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 10 monclient: dump: 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:28.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff408ff9640 4 set_mon_vals no callback set 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 5 monclient: authenticate success, global_id 4587 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _renew_subs 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff40c0a1580 con 0x7ff40c137e30 2024-09-17T15:30:28.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _renew_subs 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff40c121420 con 0x7ff40c137e30 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff40c09cf60 con 0x7ff40c137e30 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff3fc01b020 con 0x7ff40c137e30 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 0x7ff3f803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 4811+0+0 (secure 0 0 0) 0x7ff3fc050ed0 con 0x7ff40c137e30 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff40affd640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 0x7ff3f803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:28.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.158+0000 7ff40affd640 10 monclient: get_auth_request con 0x7ff3f803bb60 auth_method 0 2024-09-17T15:30:28.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.162+0000 7ff40affd640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 0x7ff3f803e020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4000097c0 tx=0x7ff400008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:28.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.162+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff3fc022030 con 0x7ff40c137e30 2024-09-17T15:30:28.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.162+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:28.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.162+0000 7ff3f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff41169a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:30:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff41169a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:30:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff41169a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff40c1215f0 con 0x7ff40c137e30 2024-09-17T15:30:28.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff3f37fe640 1 -- 172.21.15.86:0/299289949 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff3fc010040 con 0x7ff40c137e30 2024-09-17T15:30:28.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff3f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:30:28.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.382+0000 7ff3f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:28.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2092053002,config=3541143443,health=906682742,kv=3591778543,logm=1875187178}) 2024-09-17T15:30:28.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=2092053002,config=3541143443,health=2223430764,kv=2348081265,logm=2288996785}) 2024-09-17T15:30:28.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2092053002,config=3541143443,health=906682742,kv=3591778543,logm=1875187178}) 2024-09-17T15:30:28.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.382+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=9,logm=67} crc {auth=2092053002,config=3541143443,health=1645065264,kv=3867796606,logm=3166274576}) 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 msgr2=0x7ff3f803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 0x7ff3f803e020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4000097c0 tx=0x7ff400008040 comp rx=0 tx=0).stop 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 10 monclient: shutdown 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 msgr2=0x7ff40c13d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 0x7ff40c13d8f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00efc0 tx=0x7ff3fc00c490 comp rx=0 tx=0).stop 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 shutdown_connections 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3f803bb60 0x7ff3f803e020 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff40c137330 0x7ff40c1d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff40c137e30 0x7ff40c13d8f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 --2- 172.21.15.86:0/299289949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff40c0a5450 0x7ff40c13fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 >> 172.21.15.86:0/299289949 conn(0x7ff40c0742a0 msgr2=0x7ff40c0a3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 shutdown_connections 2024-09-17T15:30:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:28.386+0000 7ff41169a640 1 -- 172.21.15.86:0/299289949 wait complete. 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644541885}) 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910149965}) 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644541885}) 2024-09-17T15:30:28.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990623821}) 2024-09-17T15:30:28.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462072584}) 2024-09-17T15:30:28.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929915424}) 2024-09-17T15:30:28.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462072584}) 2024-09-17T15:30:28.392 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.386+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173653143}) 2024-09-17T15:30:28.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.394 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696270061}) 2024-09-17T15:30:28.394 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3473448619}) 2024-09-17T15:30:28.394 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.394 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696270061}) 2024-09-17T15:30:28.394 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383353257}) 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349945052}) 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303383940}) 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349945052}) 2024-09-17T15:30:28.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.390+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047893695}) 2024-09-17T15:30:28.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110840247}) 2024-09-17T15:30:28.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052446862}) 2024-09-17T15:30:28.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2110840247}) 2024-09-17T15:30:28.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.394+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1374147565}) 2024-09-17T15:30:28.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273993877}) 2024-09-17T15:30:28.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3377541681}) 2024-09-17T15:30:28.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273993877}) 2024-09-17T15:30:28.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043050196}) 2024-09-17T15:30:28.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=2082081361,mdsmap=3140514009,mgr=333540546,mgr_command_descs=1770323759,mgrstat=1048213252}) 2024-09-17T15:30:28.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=2806302204,mdsmap=2815901553,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=288080862,mon_config_key=1147139052}) 2024-09-17T15:30:28.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=9} crc {logm=2082081361,mdsmap=3140514009,mgr=333540546,mgr_command_descs=1770323759,mgrstat=1048213252}) 2024-09-17T15:30:28.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.398+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=211243361,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1153853277}) 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=19,osdmap=55} crc {mgrstat=507179242,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=598337686,osd_metadata=2356047796,osd_pg_creating=73130827,osd_snap=1864780857,osdmap=1214013670}) 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=56} crc {mgrstat=507179242,mon_config_key=2265748168,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=73130827,osd_snap=2867486463,osdmap=2714818775}) 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=2,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=19,osdmap=55} crc {mgrstat=507179242,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=598337686,osd_metadata=2356047796,osd_pg_creating=73130827,osd_snap=1864780857,osdmap=1214013670}) 2024-09-17T15:30:28.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.402+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16,osdmap=56} crc {mgrstat=507179242,mon_config_key=2888077779,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=73130827,osd_snap=3619668204,osdmap=764370194}) 2024-09-17T15:30:28.405 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:30:28.409 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.409 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3281960577}) 2024-09-17T15:30:28.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=97} crc {osdmap=4287635925}) 2024-09-17T15:30:28.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3281960577}) 2024-09-17T15:30:28.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.406+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=703235838}) 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=1} crc {osdmap=1581044621}) 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=2} crc {osdmap=2304751980}) 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=1} crc {osdmap=1581044621}) 2024-09-17T15:30:28.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:30:28.410+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=2} crc {osdmap=2304751980}) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:374: done (4 left) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:375: done (3 left) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:376: done (2 left) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:378: done (1 left) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:380: done (0 left) 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:381: copy_from oid 49 from oid 35 current snap is 39 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:382: write (excl) oid 17 current snap is 39 2024-09-17T15:30:28.420 INFO:tasks.rados.rados.0.smithi203.stdout:382: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-17T15:30:28.440 INFO:tasks.rados.rados.0.smithi203.stdout:382: writing smithi20328086-17 from 689120 to 1172373 tid 1 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:382: writing smithi20328086-17 from 1946413 to 2625462 tid 2 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:383: rollback oid 21 current snap is 39 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 37 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:381: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:382: finishing write tid 1 to smithi20328086-17 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:381: finishing copy_from to smithi20328086-49 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 44 (ObjNum 142 snap 32 seq_num 142) dirty exists 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:381: done (2 left) 2024-09-17T15:30:28.461 INFO:tasks.rados.rados.0.smithi203.stdout:384: write oid 2 current snap is 39 2024-09-17T15:30:28.462 INFO:tasks.rados.rados.0.smithi203.stdout:384: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-17T15:30:28.469 INFO:tasks.rados.rados.0.smithi203.stdout:384: writing smithi20328086-2 from 705927 to 1264429 tid 1 2024-09-17T15:30:28.480 INFO:tasks.rados.rados.0.smithi203.stdout:384: writing smithi20328086-2 from 2004895 to 2427602 tid 2 2024-09-17T15:30:28.480 INFO:tasks.rados.rados.0.smithi203.stdout:385: write oid 48 current snap is 39 2024-09-17T15:30:28.480 INFO:tasks.rados.rados.0.smithi203.stdout:385: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-17T15:30:28.490 INFO:tasks.rados.rados.0.smithi203.stdout:385: writing smithi20328086-48 from 722734 to 1356485 tid 1 2024-09-17T15:30:28.501 INFO:tasks.rados.rados.0.smithi203.stdout:385: writing smithi20328086-48 from 2063377 to 2722734 tid 2 2024-09-17T15:30:28.501 INFO:tasks.rados.rados.0.smithi203.stdout:386: rollback oid 43 current snap is 39 2024-09-17T15:30:28.501 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 33 2024-09-17T15:30:28.501 INFO:tasks.rados.rados.0.smithi203.stdout:383: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:30:28.501 INFO:tasks.rados.rados.0.smithi203.stdout:383: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:383: finishing rollback tid 2 to smithi20328086-21 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 151 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: finishing write tid 2 to smithi20328086-17 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: finishing write tid 3 to smithi20328086-17 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: finishing write tid 4 to smithi20328086-17 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: oid 17 updating version 0 to 126 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: oid 17 updating version 126 to 128 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: oid 17 version 128 is already newer than 127 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 128 (ObjNum 160 snap 39 seq_num 160) dirty exists 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: left oid 17 (ObjNum 160 snap 39 seq_num 160) 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:382: done (4 left) 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:383: done (3 left) 2024-09-17T15:30:28.502 INFO:tasks.rados.rados.0.smithi203.stdout:387: write oid 47 current snap is 39 2024-09-17T15:30:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:387: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-17T15:30:28.511 INFO:tasks.rados.rados.0.smithi203.stdout:387: writing smithi20328086-47 from 739541 to 1448541 tid 1 2024-09-17T15:30:28.518 INFO:tasks.rados.rados.0.smithi203.stdout:387: writing smithi20328086-47 from 2121859 to 2548235 tid 2 2024-09-17T15:30:28.518 INFO:tasks.rados.rados.0.smithi203.stdout:388: snap_remove snap 38 2024-09-17T15:30:29.406 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:30:29.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: finishing write tid 1 to smithi20328086-2 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: finishing write tid 2 to smithi20328086-2 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: finishing write tid 3 to smithi20328086-2 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: finishing write tid 4 to smithi20328086-2 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: oid 2 updating version 0 to 61 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: oid 2 version 61 is already newer than 60 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: oid 2 version 61 is already newer than 59 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 61 (ObjNum 161 snap 39 seq_num 161) dirty exists 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:384: left oid 2 (ObjNum 161 snap 39 seq_num 161) 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:385: finishing write tid 1 to smithi20328086-48 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:385: finishing write tid 2 to smithi20328086-48 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:385: finishing write tid 3 to smithi20328086-48 2024-09-17T15:30:29.425 INFO:tasks.rados.rados.0.smithi203.stdout:385: finishing write tid 4 to smithi20328086-48 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:385: oid 48 updating version 0 to 86 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:385: oid 48 updating version 86 to 87 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:385: oid 48 updating version 87 to 88 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 88 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:385: left oid 48 (ObjNum 162 snap 39 seq_num 162) 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:386: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:386: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:386: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 56 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:387: finishing write tid 1 to smithi20328086-47 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:387: finishing write tid 2 to smithi20328086-47 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:387: finishing write tid 3 to smithi20328086-47 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:387: finishing write tid 4 to smithi20328086-47 2024-09-17T15:30:29.426 INFO:tasks.rados.rados.0.smithi203.stdout:387: oid 47 updating version 0 to 59 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:387: oid 47 version 59 is already newer than 57 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:387: oid 47 version 59 is already newer than 58 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 59 (ObjNum 163 snap 39 seq_num 163) dirty exists 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:387: left oid 47 (ObjNum 163 snap 39 seq_num 163) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:384: done (4 left) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:385: done (3 left) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:386: done (2 left) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:387: done (1 left) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:388: done (0 left) 2024-09-17T15:30:29.427 INFO:tasks.rados.rados.0.smithi203.stdout:389: snap_create 2024-09-17T15:30:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: get_monmap_and_config 2024-09-17T15:30:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: build_initial_monmap 2024-09-17T15:30:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: monmap: 2024-09-17T15:30:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:29.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:29.533760+0000 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:29.533760+0000 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:29.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 Processor -- start 2024-09-17T15:30:29.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 -- start start 2024-09-17T15:30:29.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: init 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f672c0a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 0x7f672c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): picked mon.noname-b con 0x7f672c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f672c13d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672c0a6530 con 0x7f672c13d2f0 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672c09c150 con 0x7f672c0a50e0 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f672c09c430 con 0x7f672c0a54c0 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673328f640 10 monclient(hunting): authenticate will time out at 1162.499384s 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673182c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f672c13d2f0 0x7f672c13f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37142/0 (socket says 172.21.15.86:37142) 2024-09-17T15:30:29.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673182c640 1 -- 172.21.15.86:0/3607895605 learned_addr learned my addr 172.21.15.86:0/3607895605 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 1 --2- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 0x7f672c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): get_auth_request con 0x7f672c0a54c0 auth_method 0 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient(hunting): handle_auth_done global_id 34136 payload 274 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 1 -- 172.21.15.86:0/3607895605 >> v1:172.21.15.203:6789/0 conn(0x7f672c0a50e0 legacy=0x7f672c09ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 1 -- 172.21.15.86:0/3607895605 >> v1:172.21.15.86:6789/0 conn(0x7f672c13d2f0 legacy=0x7f672c13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient: _finish_hunting 0 2024-09-17T15:30:29.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 1 monclient: found mon.noname-b 2024-09-17T15:30:29.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:29.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.530+0000 7f673082a640 1 -- 172.21.15.86:0/3607895605 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c0a6940 con 0x7f672c0a54c0 2024-09-17T15:30:29.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 1 -- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 msgr2=0x7f672c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 1 --2- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 0x7f672c0a58d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f671c0097f0 tx=0x7f671c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19b25a18b1a9323d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 1 -- 172.21.15.86:0/3607895605 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c0b0070 con 0x7f672c0a54c0 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: dump: 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.540 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: _finish_auth 0 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: _check_auth_tickets 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 5 monclient: authenticate success, global_id 34136 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 1 -- 172.21.15.86:0/3607895605 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f671c0ab600 con 0x7f672c0a54c0 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 4 set_mon_vals no callback set 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 1 -- 172.21.15.86:0/3607895605 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c0ab9d0 con 0x7f672c0a54c0 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f6723fff640 10 monclient: dump: 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: shutdown 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 msgr2=0x7f672c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 --2- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 0x7f672c0a58d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f671c0097f0 tx=0x7f671c01c3a0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- 172.21.15.86:0/3607895605 shutdown_connections 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 --2- 172.21.15.86:0/3607895605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a54c0 0x7f672c0a58d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- 172.21.15.86:0/3607895605 >> 172.21.15.86:0/3607895605 conn(0x7f672c0aaae0 msgr2=0x7f672c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- 172.21.15.86:0/3607895605 shutdown_connections 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- 172.21.15.86:0/3607895605 wait complete. 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: build_initial_monmap 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: monmap: 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:29.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 Processor -- start 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- start start 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: init 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): picked mon.noname-c con 0x7f672c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f672c0a54c0 0x7f672c138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): picked mon.noname-b con 0x7f672c0a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 0x7f672c0a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): picked mon.noname-a con 0x7f672c13d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f672c143a10 con 0x7f672c13d2f0 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f672c143890 con 0x7f672c0a54c0 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f672c143b90 con 0x7f672c0a50e0 2024-09-17T15:30:29.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673328f640 10 monclient(hunting): authenticate will time out at 1162.506588s 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 0x7f672c0a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39652/0 (socket says 172.21.15.86:39652) 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 -- 172.21.15.86:0/2949540857 learned_addr learned my addr 172.21.15.86:0/2949540857 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): get_auth_request con 0x7f672c0a50e0 auth_method 0 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): get_auth_request con 0x7f672c13d2f0 auth_method 0 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f672c0a54c0 0x7f672c138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 10 monclient(hunting): get_auth_request con 0x7f672c0a54c0 auth_method 0 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673082a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient(hunting): handle_auth_done global_id 34139 payload 824 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 -- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f672c0a54c0 msgr2=0x7f672c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f672c0a54c0 0x7f672c138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 -- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 msgr2=0x7f672c0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 0x7f672c0a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient: _finish_hunting 0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 monclient: found mon.noname-c 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673102b640 1 -- 172.21.15.86:0/2949540857 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c121a90 con 0x7f672c0a50e0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.538+0000 7f673182c640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 0x7f672c0a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673102b640 1 -- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 msgr2=0x7f672c1cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673102b640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f672800c480 tx=0x7f672800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728012a40 con 0x7f672c0a50e0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:29.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: dump: 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: _finish_auth 0 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:30:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6728017070 con 0x7f672c0a50e0 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 5 monclient: authenticate success, global_id 34139 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _renew_subs 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c0a18d0 con 0x7f672c0a50e0 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6728012e40 con 0x7f672c0a50e0 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 10 monclient: dump: 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.554 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f67237fe640 4 set_mon_vals no callback set 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _renew_subs 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c0a4240 con 0x7f672c0a50e0 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:29.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f672801c460 con 0x7f672c0a50e0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672c0a58d0 con 0x7f672c0a50e0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 0x7f671003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7f672804fce0 con 0x7f672c0a50e0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673082a640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 0x7f671003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.542+0000 7f673082a640 10 monclient: get_auth_request con 0x7f671003bbb0 auth_method 0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.546+0000 7f673082a640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 0x7f671003e070 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f671c0ab000 tx=0x7f671c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.546+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f672804e6e0 con 0x7f672c0a50e0 2024-09-17T15:30:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.546+0000 7f6721ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:29.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.546+0000 7f6721ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:30:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:29.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f672c0a36e0 con 0x7f672c0a50e0 2024-09-17T15:30:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f6721ffb640 1 -- 172.21.15.86:0/2949540857 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f672801c930 con 0x7f672c0a50e0 2024-09-17T15:30:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f6721ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:29.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f6721ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:29.785 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":46,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:30:29.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 msgr2=0x7f671003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 0x7f671003e070 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f671c0ab000 tx=0x7f671c008040 comp rx=0 tx=0).stop 2024-09-17T15:30:29.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 10 monclient: shutdown 2024-09-17T15:30:29.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 msgr2=0x7f672c1cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f672800c480 tx=0x7f672800c940 comp rx=0 tx=0).stop 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 shutdown_connections 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f671003bbb0 0x7f671003e070 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f672c13d2f0 0x7f672c0a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f672c0a54c0 0x7f672c138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 --2- 172.21.15.86:0/2949540857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f672c0a50e0 0x7f672c1cfe20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.782+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 >> 172.21.15.86:0/2949540857 conn(0x7f672c0aaae0 msgr2=0x7f672c0a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.786+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 shutdown_connections 2024-09-17T15:30:29.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.786+0000 7f673328f640 1 -- 172.21.15.86:0/2949540857 wait complete. 2024-09-17T15:30:29.803 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:30:29.803 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:30:29.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:30:29.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.910+0000 7f200c3c9640 10 monclient: get_monmap_and_config 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.910+0000 7f200c3c9640 10 monclient: build_initial_monmap 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.910+0000 7f200c3c9640 10 monclient: monmap: 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.915 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:29.915813+0000 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:29.915813+0000 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:29.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 Processor -- start 2024-09-17T15:30:29.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 -- start start 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient: init 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f20040a50e0 0x7f200409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-c con 0x7f20040a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20040a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f200413d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20040a6280 con 0x7f200413d2f0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f200409c0a0 con 0x7f20040a54c0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f200409d2a0 con 0x7f20040a50e0 2024-09-17T15:30:29.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200c3c9640 10 monclient(hunting): authenticate will time out at 1162.881452s 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a966640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f200413d2f0 0x7f200413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37148/0 (socket says 172.21.15.86:37148) 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a966640 1 -- 172.21.15.86:0/3152364254 learned_addr learned my addr 172.21.15.86:0/3152364254 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 1 --2- 172.21.15.86:0/3152364254 >> v2:172.21.15.203:3300/0 conn(0x7f20040a50e0 0x7f200409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): get_auth_request con 0x7f20040a50e0 auth_method 0 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient(hunting): handle_auth_done global_id 14236 payload 274 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 1 -- 172.21.15.86:0/3152364254 >> v1:172.21.15.203:6789/0 conn(0x7f20040a54c0 legacy=0x7f20040a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 1 -- 172.21.15.86:0/3152364254 >> v1:172.21.15.86:6789/0 conn(0x7f200413d2f0 legacy=0x7f200413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient: _finish_hunting 0 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 1 monclient: found mon.noname-c 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:30:29.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.914+0000 7f200a165640 1 -- 172.21.15.86:0/3152364254 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f200409c3d0 con 0x7f20040a50e0 2024-09-17T15:30:29.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200a165640 1 -- 172.21.15.86:0/3152364254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a50e0 msgr2=0x7f200409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200a165640 1 --2- 172.21.15.86:0/3152364254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a50e0 0x7f200409cd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097c0 tx=0x7f1ff401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dff38212d7f6c8b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 1 -- 172.21.15.86:0/3152364254 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40b0070 con 0x7f20040a50e0 2024-09-17T15:30:29.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: dump: 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: _finish_auth 0 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: _check_auth_tickets 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 5 monclient: authenticate success, global_id 14236 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 1 -- 172.21.15.86:0/3152364254 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff40ab600 con 0x7f20040a50e0 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 1 -- 172.21.15.86:0/3152364254 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40ab9d0 con 0x7f20040a50e0 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f2009163640 10 monclient: dump: 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 4 set_mon_vals no callback set 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 10 monclient: shutdown 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 -- 172.21.15.86:0/3152364254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a50e0 msgr2=0x7f200409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 --2- 172.21.15.86:0/3152364254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a50e0 0x7f200409cd60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40097c0 tx=0x7f1ff401c360 comp rx=0 tx=0).stop 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 -- 172.21.15.86:0/3152364254 shutdown_connections 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 --2- 172.21.15.86:0/3152364254 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a50e0 0x7f200409cd60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 -- 172.21.15.86:0/3152364254 >> 172.21.15.86:0/3152364254 conn(0x7f20040aaae0 msgr2=0x7f20040aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.918+0000 7f200c3c9640 1 -- 172.21.15.86:0/3152364254 shutdown_connections 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 -- 172.21.15.86:0/3152364254 wait complete. 2024-09-17T15:30:29.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient: build_initial_monmap 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient: monmap: 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:29.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 Processor -- start 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 -- start start 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient: init 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 0x7f20041cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-a con 0x7f20040a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a54c0 0x7f20041381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-b con 0x7f20040a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200413d2f0 0x7f200413a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): picked mon.noname-c con 0x7f200413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f20041439b0 con 0x7f20040a50e0 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2004143830 con 0x7f20040a54c0 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2004143b30 con 0x7f200413d2f0 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200c3c9640 10 monclient(hunting): authenticate will time out at 1162.889608s 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 0x7f20041cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 0x7f20041cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:45134/0 (socket says 172.21.15.86:45134) 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 -- 172.21.15.86:0/566339421 learned_addr learned my addr 172.21.15.86:0/566339421 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): get_auth_request con 0x7f20040a50e0 auth_method 0 2024-09-17T15:30:29.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200413d2f0 0x7f200413a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): get_auth_request con 0x7f200413d2f0 auth_method 0 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient(hunting): handle_auth_done global_id 4593 payload 824 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200413d2f0 msgr2=0x7f200413a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200413d2f0 0x7f200413a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a54c0 0x7f20041381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a54c0 msgr2=0x7f20041381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a54c0 0x7f20041381c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient: _finish_hunting 0 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 monclient: found mon.noname-a 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a165640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20041d0d90 con 0x7f20040a50e0 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f2009964640 10 monclient: get_auth_request con 0x7f20040a54c0 auth_method 0 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f200a966640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200413d2f0 0x7f200413a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.922+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20040a54c0 0x7f20041381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:30:29.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200a165640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 msgr2=0x7f20041cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200a165640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 0x7f20041cfe80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ff401cc00 tx=0x7f1ff401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40b0050 con 0x7f20040a50e0 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: dump: 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.939 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: _finish_auth 0 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff40bc070 con 0x7f20040a50e0 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 5 monclient: authenticate success, global_id 4593 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 monclient: _renew_subs 2024-09-17T15:30:29.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20040a18d0 con 0x7f20040a50e0 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40b5c80 con 0x7f20040a50e0 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2008962640 4 set_mon_vals no callback set 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 10 monclient: dump: 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 monclient: _renew_subs 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20040a7e30 con 0x7f20040a50e0 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 0x7f200413ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f20040a1d90 con 0x7f200413aab0 2024-09-17T15:30:29.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 0x7f200413ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient: get_auth_request con 0x7f200413aab0 auth_method 0 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 10 monclient(hunting): handle_auth_done global_id 4596 payload 824 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f200413b3d0 con 0x7f200413aab0 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1ff40ab4c0 con 0x7f20040a50e0 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fe803bb90 0x7f1fe803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7f1ff40ea320 con 0x7f20040a50e0 2024-09-17T15:30:29.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200a966640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fe803bb90 0x7f1fe803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f200a966640 10 monclient: get_auth_request con 0x7f1fe803bb90 auth_method 0 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 msgr2=0x7f200413ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.926+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 0x7f200413ae90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20040a7bc0 tx=0x7f200000d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2000012a10 con 0x7f200413aab0 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 10 monclient: dump: 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.944 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2000012a10 con 0x7f200413aab0 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 msgr2=0x7f200413ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 0x7f200413ae90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20040a7bc0 tx=0x7f200000d8e0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 msgr2=0x7f200413ae90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f1ffaffd640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413aab0 0x7f200413ae90 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f20040a7bc0 tx=0x7f200000d8e0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.930+0000 7f200a966640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fe803bb90 0x7f1fe803e050 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005c10 tx=0x7f1ffc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f200c3c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:29.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f200c3c9640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 0x7f200413dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f200c3c9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f200409cd60 con 0x7f200413d6d0 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 0x7f200413dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient: get_auth_request con 0x7f200413d6d0 auth_method 0 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:29.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.934+0000 7f2009964640 1 -- 172.21.15.86:0/566339421 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f200413c900 con 0x7f200413d6d0 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f2009964640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 msgr2=0x7f200413dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f2009964640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 0x7f200413dae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2000017000 tx=0x7f2000003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20000243f0 con 0x7f200413d6d0 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 10 monclient: dump: 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:29.949 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f20000243f0 con 0x7f200413d6d0 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 msgr2=0x7f200413dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 0x7f200413dae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2000017000 tx=0x7f2000003770 comp rx=0 tx=0).stop 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 msgr2=0x7f200413dae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f1ffaffd640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f200413d6d0 0x7f200413dae0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.950 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 202582, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:30:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:29.963 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fe803bb90 msgr2=0x7f1fe803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fe803bb90 0x7f1fe803e050 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc005c10 tx=0x7f1ffc009040 comp rx=0 tx=0).stop 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 10 monclient: shutdown 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 msgr2=0x7f20041cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 --2- 172.21.15.86:0/566339421 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040a50e0 0x7f20041cfe80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1ff401cc00 tx=0x7f1ff401aba0 comp rx=0 tx=0).stop 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200a966640 1 -- 172.21.15.86:0/566339421 reap_dead start 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200a966640 1 -- 172.21.15.86:0/566339421 reap_dead start 2024-09-17T15:30:29.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 shutdown_connections 2024-09-17T15:30:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 >> 172.21.15.86:0/566339421 conn(0x7f20040aaae0 msgr2=0x7f20040a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 shutdown_connections 2024-09-17T15:30:29.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:29.938+0000 7f200c3c9640 1 -- 172.21.15.86:0/566339421 wait complete. 2024-09-17T15:30:29.966 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:29.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:30:29.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:30:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: get_monmap_and_config 2024-09-17T15:30:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: build_initial_monmap 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: monmap: 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:30.072229+0000 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:30.072229+0000 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:30.073 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:30.073 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 Processor -- start 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 -- start start 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: init 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T15:30:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf000a4900 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdf000a13d0 0x7fdf0013adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-c con 0x7fdf000a13d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 0x7fdf0013d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-b con 0x7fdf0013b300 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf00069ae0 con 0x7fdf000a4900 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdf0006ed80 con 0x7fdf000a13d0 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdf000a4ce0 con 0x7fdf0013b300 2024-09-17T15:30:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdf04fd1640 10 monclient(hunting): authenticate will time out at 1163.037737s 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefe59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdf000a4900 0x7fdf000a0cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37152/0 (socket says 172.21.15.86:37152) 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefe59c640 1 -- 172.21.15.86:0/2995196803 learned_addr learned my addr 172.21.15.86:0/2995196803 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 --2- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 0x7fdf0013d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): get_auth_request con 0x7fdf0013b300 auth_method 0 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/2995196803 >> v2:172.21.15.203:3300/0 conn(0x7fdf000a13d0 0x7fdf0013adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefdd9b640 10 monclient(hunting): get_auth_request con 0x7fdf000a13d0 auth_method 0 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient(hunting): handle_auth_done global_id 34145 payload 274 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 -- 172.21.15.86:0/2995196803 >> v2:172.21.15.203:3300/0 conn(0x7fdf000a13d0 msgr2=0x7fdf0013adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 --2- 172.21.15.86:0/2995196803 >> v2:172.21.15.203:3300/0 conn(0x7fdf000a13d0 0x7fdf0013adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 -- 172.21.15.86:0/2995196803 >> v1:172.21.15.86:6789/0 conn(0x7fdf000a4900 legacy=0x7fdf000a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:30:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 monclient: found mon.noname-b 2024-09-17T15:30:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.070+0000 7fdefed9d640 1 -- 172.21.15.86:0/2995196803 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf00132c30 con 0x7fdf0013b300 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefed9d640 1 -- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 msgr2=0x7fdf0013d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefed9d640 1 --2- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 0x7fdf0013d6f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d90 tx=0x7fdef801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da0d75e5eb50374 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 1 -- 172.21.15.86:0/2995196803 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef80ac070 con 0x7fdf0013b300 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: dump: 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.086 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: _finish_auth 0 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: _check_auth_tickets 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 5 monclient: authenticate success, global_id 34145 2024-09-17T15:30:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 1 -- 172.21.15.86:0/2995196803 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef80a6680 con 0x7fdf0013b300 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 4 set_mon_vals no callback set 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 1 -- 172.21.15.86:0/2995196803 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef80a6a50 con 0x7fdf0013b300 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdefd59a640 10 monclient: dump: 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 10 monclient: shutdown 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 -- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 msgr2=0x7fdf0013d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 0x7fdf0013d6f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdef8009d90 tx=0x7fdef801c830 comp rx=0 tx=0).stop 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 -- 172.21.15.86:0/2995196803 shutdown_connections 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/2995196803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0013b300 0x7fdf0013d6f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/2995196803 >> v2:172.21.15.203:3300/0 conn(0x7fdf000a13d0 0x7fdf0013adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 -- 172.21.15.86:0/2995196803 >> 172.21.15.86:0/2995196803 conn(0x7fdf000a3e10 msgr2=0x7fdf000a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 -- 172.21.15.86:0/2995196803 shutdown_connections 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 1 -- 172.21.15.86:0/2995196803 wait complete. 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 10 monclient: build_initial_monmap 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.074+0000 7fdf04fd1640 10 monclient: monmap: 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 Processor -- start 2024-09-17T15:30:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 -- start start 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient: init 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-c con 0x7fdf000a13d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf000a4900 0x7fdf0009b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-b con 0x7fdf000a4900 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf0013b300 0x7fdf0009b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): picked mon.noname-a con 0x7fdf0013b300 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdf001405d0 con 0x7fdf0013b300 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdf00140450 con 0x7fdf000a4900 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdf00140750 con 0x7fdf000a13d0 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient(hunting): authenticate will time out at 1163.045056s 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39674/0 (socket says 172.21.15.86:39674) 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf0013b300 0x7fdf0009b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 -- 172.21.15.86:0/3891874405 learned_addr learned my addr 172.21.15.86:0/3891874405 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): get_auth_request con 0x7fdf000a13d0 auth_method 0 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): get_auth_request con 0x7fdf0013b300 auth_method 0 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient(hunting): handle_auth_done global_id 34148 payload 824 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf000a4900 msgr2=0x7fdf0009b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf000a4900 0x7fdf0009b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf000a4900 0x7fdf0009b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf0013b300 msgr2=0x7fdf0009b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf0013b300 0x7fdf0009b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient: _finish_hunting 0 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 monclient: found mon.noname-c 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf00069710 con 0x7fdf000a13d0 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefed9d640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf0013b300 0x7fdf0009b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 msgr2=0x7fdf0009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefe59c640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c470 tx=0x7fdef000c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0017070 con 0x7fdf000a13d0 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: dump: 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: _finish_auth 0 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef00070d0 con 0x7fdf000a13d0 2024-09-17T15:30:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 5 monclient: authenticate success, global_id 34148 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient: _renew_subs 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0012dc80 con 0x7fdf000a13d0 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00074c0 con 0x7fdf000a13d0 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdeeeffd640 10 monclient: dump: 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.078+0000 7fdefcd99640 4 set_mon_vals no callback set 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 10 monclient: _renew_subs 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf0009d3b0 con 0x7fdf000a13d0 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 0x7fdf00131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdf0009c2f0 con 0x7fdf0009bf10 2024-09-17T15:30:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 0x7fdf00131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient: get_auth_request con 0x7fdf0009bf10 auth_method 0 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefe59c640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c470 tx=0x7fdef000c930 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fdef004f4f0 osd_map(99..99 src has 1..99) 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdef001a070 con 0x7fdf000a13d0 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_done global_id 34151 payload 824 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf000a0cc0 con 0x7fdf0009bf10 2024-09-17T15:30:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdee003bb60 0x7fdee003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefed9d640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdee003bb60 0x7fdee003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefed9d640 10 monclient: get_auth_request con 0x7fdee003bb60 auth_method 0 2024-09-17T15:30:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 msgr2=0x7fdf00131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 0x7fdf00131520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf00136a40 tx=0x7fdef400e900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4017070 con 0x7fdf0009bf10 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 10 monclient: dump: 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdef4013900 con 0x7fdf0009bf10 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 msgr2=0x7fdf00131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 0x7fdf00131520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdf00136a40 tx=0x7fdef400e900 comp rx=0 tx=0).stop 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 msgr2=0x7fdf00131520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.082+0000 7fdeeeffd640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf0009bf10 0x7fdf00131520 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.086+0000 7fdefed9d640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdee003bb60 0x7fdee003e020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf0006fa40 tx=0x7fdef80a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdf04fd1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 0x7fdf00131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdf04fd1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdf0006e5b0 con 0x7fdf00131a60 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 0x7fdf00131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient: get_auth_request con 0x7fdf00131a60 auth_method 0 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 10 monclient(hunting): handle_auth_done global_id 34154 payload 824 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 1 -- 172.21.15.86:0/3891874405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf0012dec0 con 0x7fdf00131a60 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 msgr2=0x7fdf00131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.090+0000 7fdefdd9b640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 0x7fdf00131e40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf00136a40 tx=0x7fdef4004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef400f040 con 0x7fdf00131a60 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 10 monclient: dump: 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fdef4012780 con 0x7fdf00131a60 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 msgr2=0x7fdf00131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 0x7fdf00131e40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf00136a40 tx=0x7fdef4004740 comp rx=0 tx=0).stop 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 msgr2=0x7fdf00131e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf00131a60 0x7fdf00131e40 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdf00136a40 tx=0x7fdef4004740 comp rx=0 tx=0).stop 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:30.107 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 23875, 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.108 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.109 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:30.110 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:30.111 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.112 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:30.113 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:30.115 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.116 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:30.117 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:30.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:30.119 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdee003bb60 msgr2=0x7fdee003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdee003bb60 0x7fdee003e020 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf0006fa40 tx=0x7fdef80a7000 comp rx=0 tx=0).stop 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 10 monclient: shutdown 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 msgr2=0x7fdf0009aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 --2- 172.21.15.86:0/3891874405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf000a13d0 0x7fdf0009aa70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdef000c470 tx=0x7fdef000c930 comp rx=0 tx=0).stop 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdefed9d640 1 -- 172.21.15.86:0/3891874405 reap_dead start 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdeeeffd640 1 -- 172.21.15.86:0/3891874405 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (unknown 0 0 0) 0x7fdef004f4f0 con 0x7fdf000a13d0 2024-09-17T15:30:30.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdefed9d640 1 -- 172.21.15.86:0/3891874405 reap_dead start 2024-09-17T15:30:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 shutdown_connections 2024-09-17T15:30:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 >> 172.21.15.86:0/3891874405 conn(0x7fdf000a3e10 msgr2=0x7fdf000a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 shutdown_connections 2024-09-17T15:30:30.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.094+0000 7fdf04fd1640 1 -- 172.21.15.86:0/3891874405 wait complete. 2024-09-17T15:30:30.122 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:30.122 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:30:30.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:30:30.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: get_monmap_and_config 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: build_initial_monmap 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: monmap: 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:30.228457+0000 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:30.228457+0000 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:30.229 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 Processor -- start 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 -- start start 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: init 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca200a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca2013af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): picked mon.noname-b con 0x7fca200a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca2013b450 0x7fca2013d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): picked mon.noname-c con 0x7fca2013b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca200a2130 con 0x7fca200a0c70 2024-09-17T15:30:30.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fca200a2410 con 0x7fca2013b450 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fca200a26f0 con 0x7fca200a1760 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca27788640 10 monclient(hunting): authenticate will time out at 1163.193838s 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca2013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca25524640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fca200a0c70 0x7fca200a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37160/0 (socket says 172.21.15.86:37160) 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca25524640 1 -- 172.21.15.86:0/2698451474 learned_addr learned my addr 172.21.15.86:0/2698451474 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): get_auth_request con 0x7fca200a1760 auth_method 0 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient(hunting): handle_auth_done global_id 34157 payload 274 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 -- 172.21.15.86:0/2698451474 >> v2:172.21.15.203:3300/0 conn(0x7fca2013b450 msgr2=0x7fca2013d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 --2- 172.21.15.86:0/2698451474 >> v2:172.21.15.203:3300/0 conn(0x7fca2013b450 0x7fca2013d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 -- 172.21.15.86:0/2698451474 >> v1:172.21.15.86:6789/0 conn(0x7fca200a0c70 legacy=0x7fca200a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient: _finish_hunting 0 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 monclient: found mon.noname-b 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.226+0000 7fca24d23640 1 -- 172.21.15.86:0/2698451474 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2009f9f0 con 0x7fca200a1760 2024-09-17T15:30:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca24d23640 1 -- 172.21.15.86:0/2698451474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 msgr2=0x7fca2013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca24d23640 1 --2- 172.21.15.86:0/2698451474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca2013af10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0097c0 tx=0x7fca1c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=437d3ed4995cf7c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 1 -- 172.21.15.86:0/2698451474 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0b0070 con 0x7fca200a1760 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: dump: 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: _finish_auth 0 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: _check_auth_tickets 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 5 monclient: authenticate success, global_id 34157 2024-09-17T15:30:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 1 -- 172.21.15.86:0/2698451474 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca1c0ab600 con 0x7fca200a1760 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 4 set_mon_vals no callback set 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 1 -- 172.21.15.86:0/2698451474 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0ab9d0 con 0x7fca200a1760 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca17fff640 10 monclient: dump: 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 10 monclient: shutdown 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 -- 172.21.15.86:0/2698451474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 msgr2=0x7fca2013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 --2- 172.21.15.86:0/2698451474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca2013af10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fca1c0097c0 tx=0x7fca1c01c360 comp rx=0 tx=0).stop 2024-09-17T15:30:30.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 -- 172.21.15.86:0/2698451474 shutdown_connections 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 --2- 172.21.15.86:0/2698451474 >> v2:172.21.15.203:3300/0 conn(0x7fca2013b450 0x7fca2013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 --2- 172.21.15.86:0/2698451474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca2013af10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 -- 172.21.15.86:0/2698451474 >> 172.21.15.86:0/2698451474 conn(0x7fca200a2f40 msgr2=0x7fca200a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 -- 172.21.15.86:0/2698451474 shutdown_connections 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 -- 172.21.15.86:0/2698451474 wait complete. 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 10 monclient: build_initial_monmap 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 10 monclient: monmap: 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:30.242 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.230+0000 7fca27788640 1 Processor -- start 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- start start 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: init 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca200a0c70 0x7fca2011efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): picked mon.noname-b con 0x7fca200a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca20135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): picked mon.noname-c con 0x7fca200a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2013b450 0x7fca201cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): picked mon.noname-a con 0x7fca2013b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fca20140720 con 0x7fca2013b450 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca201405a0 con 0x7fca200a0c70 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fca201408a0 con 0x7fca200a1760 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient(hunting): authenticate will time out at 1163.200505s 2024-09-17T15:30:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca20135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca20135e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39732/0 (socket says 172.21.15.86:39732) 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 -- 172.21.15.86:0/4074147051 learned_addr learned my addr 172.21.15.86:0/4074147051 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): get_auth_request con 0x7fca200a1760 auth_method 0 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2013b450 0x7fca201cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25524640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca200a0c70 0x7fca2011efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 10 monclient(hunting): get_auth_request con 0x7fca2013b450 auth_method 0 2024-09-17T15:30:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25524640 10 monclient(hunting): get_auth_request con 0x7fca200a0c70 auth_method 0 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25524640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25524640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient(hunting): handle_auth_done global_id 34160 payload 824 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca200a0c70 msgr2=0x7fca2011efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca200a0c70 0x7fca2011efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2013b450 msgr2=0x7fca201cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2013b450 0x7fca201cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient: _finish_hunting 0 2024-09-17T15:30:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 monclient: found mon.noname-c 2024-09-17T15:30:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca2009d460 con 0x7fca200a1760 2024-09-17T15:30:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca25d25640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fca2013b450 0x7fca201cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 msgr2=0x7fca20135e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca24d23640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca20135e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fca1c01cc00 tx=0x7fca1c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0b9070 con 0x7fca200a1760 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: dump: 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: _finish_auth 0 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca1c0b5690 con 0x7fca200a1760 2024-09-17T15:30:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 5 monclient: authenticate success, global_id 34160 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1c0b0040 con 0x7fca200a1760 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _renew_subs 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca16ffd640 4 set_mon_vals no callback set 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca2013bfa0 con 0x7fca200a1760 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca15ffb640 10 monclient: dump: 2024-09-17T15:30:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _renew_subs 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:30:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.234+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca201204f0 con 0x7fca200a1760 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca27788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fca1c0bc070 con 0x7fca200a1760 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca27788640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 0x7fca2002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca200a1050 con 0x7fca201206c0 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fca0403bb90 0x7fca0403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7fca1c0e9e70 con 0x7fca200a1760 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 0x7fca2002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient: get_auth_request con 0x7fca201206c0 auth_method 0 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25d25640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fca0403bb90 0x7fca0403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25d25640 10 monclient: get_auth_request con 0x7fca0403bb90 auth_method 0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 10 monclient(hunting): handle_auth_done global_id 14245 payload 824 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca2013ba50 con 0x7fca201206c0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25d25640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fca0403bb90 0x7fca0403e050 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca20120cc0 tx=0x7fca18009590 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 msgr2=0x7fca2002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca25524640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 0x7fca2002d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e460 tx=0x7fca1000e920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10005d00 con 0x7fca201206c0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.238+0000 7fca15ffb640 10 monclient: dump: 2024-09-17T15:30:30.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca10018070 con 0x7fca201206c0 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:30.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 msgr2=0x7fca2002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 0x7fca2002d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e460 tx=0x7fca1000e920 comp rx=0 tx=0).stop 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 msgr2=0x7fca2002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.242+0000 7fca15ffb640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca201206c0 0x7fca2002d450 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca1000e460 tx=0x7fca1000e920 comp rx=0 tx=0).stop 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca27788640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca27788640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 0x7fca2002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca27788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca200730f0 con 0x7fca20136380 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 0x7fca2002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient: get_auth_request con 0x7fca20136380 auth_method 0 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:30.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 10 monclient(hunting): handle_auth_done global_id 14248 payload 824 2024-09-17T15:30:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.246+0000 7fca25524640 1 -- 172.21.15.86:0/4074147051 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca2006d4f0 con 0x7fca20136380 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca25524640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 msgr2=0x7fca2002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca25524640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 0x7fca2002d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca10017000 tx=0x7fca10004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100074e0 con 0x7fca20136380 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 10 monclient: dump: 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:30.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fca100074e0 con 0x7fca20136380 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 msgr2=0x7fca2002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 0x7fca2002d990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca10017000 tx=0x7fca10004950 comp rx=0 tx=0).stop 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 msgr2=0x7fca2002d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca15ffb640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca20136380 0x7fca2002d990 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fca10017000 tx=0x7fca10004950 comp rx=0 tx=0).stop 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 46, 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 138850, 2024-09-17T15:30:30.262 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.263 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.264 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:30.265 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:30.266 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:30.267 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.268 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.269 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.270 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:30.272 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:30.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:30:30.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:30.275 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:30.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fca0403bb90 msgr2=0x7fca0403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fca0403bb90 0x7fca0403e050 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca20120cc0 tx=0x7fca18009590 comp rx=0 tx=0).stop 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 10 monclient: shutdown 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 msgr2=0x7fca20135e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 --2- 172.21.15.86:0/4074147051 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fca200a1760 0x7fca20135e40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fca1c01cc00 tx=0x7fca1c019100 comp rx=0 tx=0).stop 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca25d25640 1 -- 172.21.15.86:0/4074147051 reap_dead start 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca25d25640 1 -- 172.21.15.86:0/4074147051 reap_dead start 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 shutdown_connections 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 >> 172.21.15.86:0/4074147051 conn(0x7fca200a2f40 msgr2=0x7fca2013d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 shutdown_connections 2024-09-17T15:30:30.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:30.250+0000 7fca27788640 1 -- 172.21.15.86:0/4074147051 wait complete. 2024-09-17T15:30:30.278 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:30:30.278 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:30:30.278 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-17T15:30:30.278 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:30:30.278 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:30:30.278 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:30:30.278 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:30:30.287 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:30:30.288 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:30:30.288 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:30:30.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:30:45.288 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:30:45.289 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:30:45.289 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:30:45.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: get_monmap_and_config 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: build_initial_monmap 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: monmap: 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:45.418314+0000 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:45.418314+0000 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:45.418 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:45.419 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:45.419 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:45.419 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:45.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 Processor -- start 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 -- start start 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: init 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-b con 0x7f83880a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83880a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-a con 0x7f838813d280 addr v2:172.21.15.86:3300/0 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83880a6f40 con 0x7f83880a5450 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f83880a65e0 con 0x7f838813d280 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:45.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f838809c2a0 con 0x7f83880a5070 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838fc7e640 10 monclient(hunting): authenticate will time out at 1178.383985s 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.414+0000 7f838da1a640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 msgr2=0x7f838809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838da1a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838d219640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f83880a5450 0x7f83880a5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:60820/0 (socket says 172.21.15.86:60820) 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838d219640 1 -- 172.21.15.86:0/1081033224 learned_addr learned my addr 172.21.15.86:0/1081033224 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 10 monclient(hunting): get_auth_request con 0x7f838813d280 auth_method 0 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:45.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:45.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:30:45.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:45.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:45.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:45.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.418+0000 7f838d219640 1 -- 172.21.15.86:0/1081033224 >> v1:172.21.15.86:6789/0 conn(0x7f83880a5450 legacy=0x7f83880a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 msgr2=0x7f838809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 10 monclient(hunting): get_auth_request con 0x7f838813d280 auth_method 0 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:30:45.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:45.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:45.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:45.622+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 10 monclient(hunting): get_auth_request con 0x7f838813d280 auth_method 0 2024-09-17T15:30:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 msgr2=0x7f838809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:46.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.026+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:46.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 msgr2=0x7f838809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:46.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:46.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:46.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 10 monclient(hunting): get_auth_request con 0x7f838813d280 auth_method 0 2024-09-17T15:30:46.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:46.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:46.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:46.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:30:46.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:46.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:46.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:46.830+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): tick 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 msgr2=0x7f838809cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:48.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83880a5070 0x7f838809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 msgr2=0x7f838813f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.86:3300/0 conn(0x7f838813d280 0x7f838813f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 -- 172.21.15.86:0/1081033224 >> v1:172.21.15.86:6789/0 conn(0x7f83880a5450 legacy=0x7f83880a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient: _add_conns ranks=[4,1] 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): picked mon.noname-c con 0x7f8374003750 addr v2:172.21.15.203:3300/0 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8374006220 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 -- 172.21.15.86:0/1081033224 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838809c580 con 0x7f8374006220 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 1 -- 172.21.15.86:0/1081033224 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f83880a65e0 con 0x7f8374003750 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f83877fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8374003750 auth_method 0 2024-09-17T15:30:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.418+0000 7f838d219640 1 -- 172.21.15.86:0/1081033224 >> v1:172.21.15.203:6789/0 conn(0x7f8374006220 legacy=0x7f83740086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:48.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:48.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8374003750 auth_method 0 2024-09-17T15:30:48.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:48.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:48.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:48.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:48.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:48.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:48.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:48.622+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:30:49.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:49.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8374003750 auth_method 0 2024-09-17T15:30:49.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:49.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.026+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:30:49.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:49.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8374003750 auth_method 0 2024-09-17T15:30:49.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:49.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:49.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:49.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:30:49.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:30:49.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:30:49.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:49.830+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708819750 0 0) 0x7f838809c580 con 0x7f8374006220 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): _negotiate 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): my global_id is 14251 2024-09-17T15:30:50.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8370003530 con 0x7f8374006220 2024-09-17T15:30:50.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2113006565 0 0) 0x7f8370003530 con 0x7f8374006220 2024-09-17T15:30:50.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 msgr2=0x7f8374005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 --2- 172.21.15.86:0/1081033224 >> v2:172.21.15.203:3300/0 conn(0x7f8374003750 0x7f8374005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837400b050 con 0x7f8374006220 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f837c001970 con 0x7f8374006220 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838fc7e640 5 monclient: authenticate success, global_id 14251 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838fc7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:30:50.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: dump: 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 >> v1:172.21.15.203:6789/0 conn(0x7f8374006220 legacy=0x7f83740086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 reap_dead start 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 0x7f8370003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): picked mon.b con 0x7f8370003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 0x7f8370004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): picked mon.c con 0x7f8370004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 0x7f837000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): picked mon.a con 0x7f83700053d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83880a65e0 con 0x7f83700053d0 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f837000c260 con 0x7f8370003a30 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f837000c460 con 0x7f8370004810 2024-09-17T15:30:50.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838d219640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 msgr2=0x7f8370004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838ca18640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838d219640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 0x7f8370004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838fc7e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 0x7f837000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): get_auth_request con 0x7f83700053d0 auth_method 0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 0x7f8370003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8370003a30 auth_method 0 2024-09-17T15:30:50.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient(hunting): handle_auth_done global_id 14251 payload 293 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 msgr2=0x7f8370004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 0x7f8370004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 msgr2=0x7f8370003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 0x7f8370003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 monclient: found mon.a 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8370003530 con 0x7f83700053d0 2024-09-17T15:30:50.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838e21b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.326+0000 7f838da1a640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 0x7f8370003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:30:50.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.334+0000 7f838e21b640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 msgr2=0x7f837000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.334+0000 7f838e21b640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 0x7f837000b9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f83880a6280 tx=0x7f837801cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33e5a9418629f6b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83780ab070 con 0x7f83700053d0 2024-09-17T15:30:50.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: dump: 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83780a5cf0 con 0x7f83700053d0 2024-09-17T15:30:50.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838fc7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838fc7e640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 1 -- 172.21.15.86:0/1081033224 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837800f040 con 0x7f83700053d0 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838fc7e640 4 set_mon_vals no callback set 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.350+0000 7f838ca18640 10 monclient: dump: 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.356 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.357 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.357 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.357 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.357 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: shutdown 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 msgr2=0x7f837000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 0x7f837000b9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f83880a6280 tx=0x7f837801cec0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- 172.21.15.86:0/1081033224 shutdown_connections 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83700053d0 0x7f837000b9d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8370004810 0x7f8370004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- 172.21.15.86:0/1081033224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370003a30 0x7f8370003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- 172.21.15.86:0/1081033224 >> 172.21.15.86:0/1081033224 conn(0x7f83880742a0 msgr2=0x7f83880746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- 172.21.15.86:0/1081033224 shutdown_connections 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- 172.21.15.86:0/1081033224 wait complete. 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: build_initial_monmap 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: monmap: 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 Processor -- start 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- start start 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: init 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-a con 0x7f8370003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370004810 0x7f83881381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-b con 0x7f8370004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:50.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 0x7f83881386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): picked mon.noname-c con 0x7f83700053d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83880a69e0 con 0x7f8370003a30 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83880a6860 con 0x7f8370004810 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f838809cf60 con 0x7f83700053d0 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838fc7e640 10 monclient(hunting): authenticate will time out at 1183.323561s 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838e21b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 msgr2=0x7f83881386e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838e21b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 0x7f83881386e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51168/0 (socket says 172.21.15.86:51168) 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 1 -- 172.21.15.86:0/3956758779 learned_addr learned my addr 172.21.15.86:0/3956758779 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): get_auth_request con 0x7f8370003a30 auth_method 0 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838d219640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370004810 0x7f83881381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838da1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838d219640 10 monclient(hunting): get_auth_request con 0x7f8370004810 auth_method 0 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838d219640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838d219640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.354+0000 7f838d219640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838d219640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838d219640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838d219640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 10 monclient(hunting): handle_auth_done global_id 4608 payload 824 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 -- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 msgr2=0x7f83881386e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 0x7f83881386e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 -- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370004810 msgr2=0x7f83881381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370004810 0x7f83881381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 monclient: found mon.noname-a 2024-09-17T15:30:50.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 -- 172.21.15.86:0/3956758779 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8388124870 con 0x7f8370003a30 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 -- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 msgr2=0x7f83881cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838da1a640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8380010e40 tx=0x7f8380010f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838001d600 con 0x7f8370003a30 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: dump: 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f838001d7a0 con 0x7f8370003a30 2024-09-17T15:30:50.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838001db50 con 0x7f8370003a30 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 5 monclient: authenticate success, global_id 4608 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 10 monclient: _renew_subs 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83881242b0 con 0x7f8370003a30 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f8387fff640 4 set_mon_vals no callback set 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f83867fc640 10 monclient: dump: 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 10 monclient: _renew_subs 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.358+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8388122830 con 0x7f8370003a30 2024-09-17T15:30:50.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f838001c3a0 con 0x7f8370003a30 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838fc7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838fc7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838fc7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83880a5830 con 0x7f8370003a30 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f83867fc640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 0x7f837403e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7f83800518f0 con 0x7f8370003a30 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838d219640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 0x7f837403e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838d219640 10 monclient: get_auth_request con 0x7f837403c3f0 auth_method 0 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.362+0000 7f838d219640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 0x7f837403e8b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8388124070 tx=0x7f837c001a30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.366+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8380019070 con 0x7f8370003a30 2024-09-17T15:30:50.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.366+0000 7f83867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.366+0000 7f83867fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:50.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f838fc7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:30:50.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f838fc7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f838fc7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8388069920 con 0x7f8370003a30 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f83867fc640 1 -- 172.21.15.86:0/3956758779 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f8380007020 con 0x7f8370003a30 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f83867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.602+0000 7f83867fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:50.606 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":48,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 msgr2=0x7f837403e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 0x7f837403e8b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8388124070 tx=0x7f837c001a30 comp rx=0 tx=0).stop 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 10 monclient: shutdown 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 msgr2=0x7f83881cc790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8380010e40 tx=0x7f8380010f30 comp rx=0 tx=0).stop 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 shutdown_connections 2024-09-17T15:30:50.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f837403c3f0 0x7f837403e8b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83700053d0 0x7f83881386e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8370004810 0x7f83881381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 --2- 172.21.15.86:0/3956758779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8370003a30 0x7f83881cc790 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 >> 172.21.15.86:0/3956758779 conn(0x7f83880742a0 msgr2=0x7f83880a2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 shutdown_connections 2024-09-17T15:30:50.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.606+0000 7f838fc7e640 1 -- 172.21.15.86:0/3956758779 wait complete. 2024-09-17T15:30:50.626 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:30:50.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:30:50.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:30:50.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: get_monmap_and_config 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: build_initial_monmap 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: monmap: 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:50.748450+0000 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:50.748450+0000 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:50.748 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:50.749 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:50.749 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 Processor -- start 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 -- start start 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: init 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c2009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-b con 0x7f3c200a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c200a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c2013d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:50.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c2009d430 con 0x7f3c2013d180 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c200a68d0 con 0x7f3c200a5350 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3c2009c7f0 con 0x7f3c200a4f70 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c27294640 10 monclient(hunting): authenticate will time out at 1183.714124s 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c25030640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 msgr2=0x7f3c2009cef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c25030640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c2009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c25831640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3c2013d180 0x7f3c2013f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45614/0 (socket says 172.21.15.86:45614) 2024-09-17T15:30:50.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.746+0000 7f3c25831640 1 -- 172.21.15.86:0/733667502 learned_addr learned my addr 172.21.15.86:0/733667502 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25831640 1 -- 172.21.15.86:0/733667502 >> v1:172.21.15.86:6789/0 conn(0x7f3c2013d180 legacy=0x7f3c2013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c2482f640 1 -- 172.21.15.86:0/733667502 >> v1:172.21.15.203:6789/0 conn(0x7f3c200a5350 legacy=0x7f3c200a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 928381607 0 0) 0x7f3c2009d430 con 0x7f3c2013d180 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _negotiate 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): my global_id is 4611 2024-09-17T15:30:50.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c0c003530 con 0x7f3c2013d180 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2340146177 0 0) 0x7f3c200a68d0 con 0x7f3c200a5350 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _negotiate 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): my global_id is 14257 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c2009d430 con 0x7f3c200a5350 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 463141472 0 0) 0x7f3c0c003530 con 0x7f3c2013d180 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 msgr2=0x7f3c2009cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c2009cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 >> v1:172.21.15.203:6789/0 conn(0x7f3c200a5350 legacy=0x7f3c200a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:30:50.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c200a64f0 con 0x7f3c2013d180 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f3c18001d20 con 0x7f3c2013d180 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c27294640 5 monclient: authenticate success, global_id 4611 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c27294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: dump: 2024-09-17T15:30:50.756 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 >> v1:172.21.15.86:6789/0 conn(0x7f3c2013d180 legacy=0x7f3c2013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:30:50.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 0x7f3c0c004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): picked mon.a con 0x7f3c0c003c90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c0c004810 0x7f3c0c004c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): picked mon.b con 0x7f3c0c004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c0c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): picked mon.c con 0x7f3c200a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2009c7f0 con 0x7f3c0c003c90 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c00bf10 con 0x7f3c0c004810 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0c00c090 con 0x7f3c200a4f70 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c17fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:30:50.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c27294640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 0x7f3c0c004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25831640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 msgr2=0x7f3c0c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25831640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c0c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): get_auth_request con 0x7f3c0c003c90 auth_method 0 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c2482f640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c0c004810 0x7f3c0c004c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient(hunting): handle_auth_done global_id 4611 payload 293 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 msgr2=0x7f3c0c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:50.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c200a4f70 0x7f3c0c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c0c004810 msgr2=0x7f3c0c004c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c0c004810 0x7f3c0c004c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 monclient: found mon.a 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 1 -- 172.21.15.86:0/733667502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c003530 con 0x7f3c0c003c90 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c25030640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c2482f640 10 monclient: get_auth_request con 0x7f3c0c004810 auth_method 0 2024-09-17T15:30:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.750+0000 7f3c2482f640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c0c004810 0x7f3c0c004c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c25030640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 msgr2=0x7f3c0c004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c25030640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 0x7f3c0c004090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009920 tx=0x7f3c1c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a579dd9b355e95 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c009520 con 0x7f3c0c003c90 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: dump: 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c1c003bc0 con 0x7f3c0c003c90 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:50.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 4 set_mon_vals no callback set 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 1 -- 172.21.15.86:0/733667502 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c005070 con 0x7f3c0c003c90 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c17fff640 10 monclient: dump: 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.768 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 10 monclient: shutdown 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 msgr2=0x7f3c0c004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 --2- 172.21.15.86:0/733667502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0c003c90 0x7f3c0c004090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009920 tx=0x7f3c1c01c400 comp rx=0 tx=0).stop 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c25831640 1 -- 172.21.15.86:0/733667502 reap_dead start 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- 172.21.15.86:0/733667502 shutdown_connections 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- 172.21.15.86:0/733667502 >> 172.21.15.86:0/733667502 conn(0x7f3c200a7240 msgr2=0x7f3c200a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- 172.21.15.86:0/733667502 shutdown_connections 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- 172.21.15.86:0/733667502 wait complete. 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 10 monclient: build_initial_monmap 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 10 monclient: monmap: 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:50.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 Processor -- start 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 1 -- start start 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.754+0000 7f3c27294640 10 monclient: init 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:30:50.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0c003c90 0x7f3c201cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0c003c90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 0x7f3c20120bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-a con 0x7f3c201385b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c20138a30 0x7f3c20121100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): picked mon.noname-b con 0x7f3c20138a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c20141830 con 0x7f3c201385b0 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201416b0 con 0x7f3c20138a30 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3c201419b0 con 0x7f3c0c003c90 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): authenticate will time out at 1183.724195s 2024-09-17T15:30:50.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0c003c90 msgr2=0x7f3c201cccc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0c003c90 0x7f3c201cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 0x7f3c20120bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 0x7f3c20120bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51182/0 (socket says 172.21.15.86:51182) 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 -- 172.21.15.86:0/1024628584 learned_addr learned my addr 172.21.15.86:0/1024628584 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): get_auth_request con 0x7f3c201385b0 auth_method 0 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient(hunting): handle_auth_done global_id 4614 payload 824 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25831640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c20138a30 0x7f3c20121100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0c003c90 msgr2=0x7f3c201cccc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:30:50.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0c003c90 0x7f3c201cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c20138a30 msgr2=0x7f3c20121100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c20138a30 0x7f3c20121100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 monclient: found mon.noname-a 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c200a1760 con 0x7f3c201385b0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 msgr2=0x7f3c20120bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c2482f640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 0x7f3c20120bc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000d7d0 tx=0x7f3c1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000de80 con 0x7f3c201385b0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:50.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: dump: 2024-09-17T15:30:50.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c10017070 con 0x7f3c201385b0 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 5 monclient: authenticate success, global_id 4614 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _renew_subs 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c20123070 con 0x7f3c201385b0 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100059d0 con 0x7f3c201385b0 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c15ffb640 10 monclient: dump: 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.778 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c16ffd640 4 set_mon_vals no callback set 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _renew_subs 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c201cd200 con 0x7f3c201385b0 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 0x7f3c200a1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201240b0 con 0x7f3c20121640 2024-09-17T15:30:50.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 0x7f3c200a1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient: get_auth_request con 0x7f3c20121640 auth_method 0 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.758+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3c10005bb0 con 0x7f3c201385b0 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25030640 10 monclient(hunting): handle_auth_done global_id 4617 payload 824 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25030640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c20121a20 con 0x7f3c20121640 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0403bb10 0x7f3c0403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7f3c1004ffa0 con 0x7f3c201385b0 2024-09-17T15:30:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25831640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0403bb10 0x7f3c0403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25831640 10 monclient: get_auth_request con 0x7f3c0403bb10 auth_method 0 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25030640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 msgr2=0x7f3c200a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25030640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 0x7f3c200a1e60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0aa000 tx=0x7f3c0c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c009520 con 0x7f3c20121640 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 10 monclient: dump: 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.781 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3c1c009520 con 0x7f3c20121640 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 msgr2=0x7f3c200a1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 0x7f3c200a1e60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0aa000 tx=0x7f3c0c005840 comp rx=0 tx=0).stop 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 msgr2=0x7f3c200a1e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:50.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c15ffb640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c20121640 0x7f3c200a1e60 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c0aa000 tx=0x7f3c0c005840 comp rx=0 tx=0).stop 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.762+0000 7f3c25831640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0403bb10 0x7f3c0403dfd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c2006e1c0 tx=0x7f3c18009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c27294640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c27294640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 0x7f3c200a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c27294640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201238f0 con 0x7f3c200a23a0 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 0x7f3c200a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient: get_auth_request con 0x7f3c200a23a0 auth_method 0 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 10 monclient(hunting): handle_auth_done global_id 4620 payload 824 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 1 -- 172.21.15.86:0/1024628584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c200a5730 con 0x7f3c200a23a0 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 msgr2=0x7f3c200a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c25030640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 0x7f3c200a27b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c006fd0 tx=0x7f3c1c0090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c0bc980 con 0x7f3c200a23a0 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.770+0000 7f3c15ffb640 10 monclient: dump: 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f3c1c0bc2a0 con 0x7f3c200a23a0 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:50.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 msgr2=0x7f3c200a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 0x7f3c200a27b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c006fd0 tx=0x7f3c1c0090e0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 msgr2=0x7f3c200a27b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c15ffb640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c200a23a0 0x7f3c200a27b0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c006fd0 tx=0x7f3c1c0090e0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 48, 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 223413, 2024-09-17T15:30:50.787 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.788 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.789 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.791 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:50.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:50.793 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:50.794 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.796 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.798 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:30:50.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:50.800 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:50.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0403bb10 msgr2=0x7f3c0403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0403bb10 0x7f3c0403dfd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3c2006e1c0 tx=0x7f3c18009040 comp rx=0 tx=0).stop 2024-09-17T15:30:50.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 10 monclient: shutdown 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 msgr2=0x7f3c20120bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 --2- 172.21.15.86:0/1024628584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c201385b0 0x7f3c20120bc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000d7d0 tx=0x7f3c1000dc90 comp rx=0 tx=0).stop 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c25831640 1 -- 172.21.15.86:0/1024628584 reap_dead start 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c25831640 1 -- 172.21.15.86:0/1024628584 reap_dead start 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 shutdown_connections 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 >> 172.21.15.86:0/1024628584 conn(0x7f3c200a7240 msgr2=0x7f3c200a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 shutdown_connections 2024-09-17T15:30:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.774+0000 7f3c27294640 1 -- 172.21.15.86:0/1024628584 wait complete. 2024-09-17T15:30:50.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:30:50.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:30:50.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:30:50.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: get_monmap_and_config 2024-09-17T15:30:50.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: build_initial_monmap 2024-09-17T15:30:50.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: monmap: 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:30:50.916148+0000 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:30:50.916148+0000 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:30:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:30:50.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 Processor -- start 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 -- start start 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: init 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f900c0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 0x7f900c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-b con 0x7f900c0a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f900c13d2f0 0x7f900c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-a con 0x7f900c13d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:30:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f900c0a6280 con 0x7f900c0a50e0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f900c09d2a0 con 0x7f900c13d2f0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f900c09c0a0 con 0x7f900c0a5bd0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f9013b17640 10 monclient(hunting): authenticate will time out at 1183.881575s 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90110b2640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 msgr2=0x7f900c09cd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90110b2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 0x7f900c09cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f900c13d2f0 0x7f900c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90118b3640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f900c0a50e0 0x7f900c0a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45628/0 (socket says 172.21.15.86:45628) 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90118b3640 1 -- 172.21.15.86:0/2133294108 learned_addr learned my addr 172.21.15.86:0/2133294108 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): get_auth_request con 0x7f900c13d2f0 auth_method 0 2024-09-17T15:30:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient(hunting): handle_auth_done global_id 4623 payload 274 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 -- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 msgr2=0x7f900c09cd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 --2- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 0x7f900c09cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 -- 172.21.15.86:0/2133294108 >> v1:172.21.15.86:6789/0 conn(0x7f900c0a50e0 legacy=0x7f900c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 monclient: found mon.noname-a 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.914+0000 7f90120b4640 1 -- 172.21.15.86:0/2133294108 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c09c380 con 0x7f900c13d2f0 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90120b4640 1 -- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c13d2f0 msgr2=0x7f900c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90120b4640 1 --2- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c13d2f0 0x7f900c13f6e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9008009d90 tx=0x7f900801c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c15b2a658e7626 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 1 -- 172.21.15.86:0/2133294108 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90080b2070 con 0x7f900c13d2f0 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: dump: 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 1 -- 172.21.15.86:0/2133294108 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90080a5860 con 0x7f900c13d2f0 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 1 -- 172.21.15.86:0/2133294108 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90080a5c30 con 0x7f900c13d2f0 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 5 monclient: authenticate success, global_id 4623 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 10 monclient: get_monmap_and_config success 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 4 set_mon_vals no callback set 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f90108b1640 10 monclient: dump: 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 10 monclient: shutdown 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 -- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c13d2f0 msgr2=0x7f900c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 --2- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c13d2f0 0x7f900c13f6e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9008009d90 tx=0x7f900801c820 comp rx=0 tx=0).stop 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 -- 172.21.15.86:0/2133294108 shutdown_connections 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 --2- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c13d2f0 0x7f900c13f6e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 --2- 172.21.15.86:0/2133294108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a5bd0 0x7f900c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 -- 172.21.15.86:0/2133294108 >> 172.21.15.86:0/2133294108 conn(0x7f900c0aaae0 msgr2=0x7f900c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 -- 172.21.15.86:0/2133294108 shutdown_connections 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 1 -- 172.21.15.86:0/2133294108 wait complete. 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 10 monclient: build_initial_monmap 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.918+0000 7f9013b17640 10 monclient: monmap: 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 Processor -- start 2024-09-17T15:30:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 -- start start 2024-09-17T15:30:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient: init 2024-09-17T15:30:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient: _reopen_session rank -1 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a50e0 0x7f900c1cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-c con 0x7f900c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 0x7f900c1d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-a con 0x7f900c0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c13d2f0 0x7f900c1d0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): picked mon.noname-b con 0x7f900c13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f900c142650 con 0x7f900c0a5bd0 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f900c0a6b00 con 0x7f900c13d2f0 2024-09-17T15:30:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f900c1427d0 con 0x7f900c0a50e0 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): _renew_subs 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient(hunting): authenticate will time out at 1183.888816s 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90118b3640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a50e0 msgr2=0x7f900c1cfae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90118b3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a50e0 0x7f900c1cfae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 0x7f900c1d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 0x7f900c1d0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51222/0 (socket says 172.21.15.86:51222) 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 -- 172.21.15.86:0/646167884 learned_addr learned my addr 172.21.15.86:0/646167884 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): get_auth_request con 0x7f900c0a5bd0 auth_method 0 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c13d2f0 0x7f900c1d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): get_auth_request con 0x7f900c13d2f0 auth_method 0 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90120b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient(hunting): handle_auth_done global_id 4626 payload 824 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a50e0 msgr2=0x7f900c1cfae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f900c0a50e0 0x7f900c1cfae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c13d2f0 msgr2=0x7f900c1d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c13d2f0 0x7f900c1d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient: _finish_hunting 0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 monclient: found mon.noname-a 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c13d960 con 0x7f900c0a5bd0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 msgr2=0x7f900c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90110b2640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 0x7f900c1d0020 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b550 tx=0x7f8ffc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc003de0 con 0x7f900c0a5bd0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: dump: 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: _finish_auth 0 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: _check_auth_tickets 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ffc015070 con 0x7f900c0a5bd0 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:30:50.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc005330 con 0x7f900c0a5bd0 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f90027fc640 10 monclient: dump: 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 5 monclient: authenticate success, global_id 4626 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient: _renew_subs 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9003fff640 4 set_mon_vals no callback set 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.922+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f900c13ab20 con 0x7f900c0a5bd0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 10 monclient: _renew_subs 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f900c139460 con 0x7f900c0a5bd0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 0x7f900c135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f900c13f910 con 0x7f900c134e70 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8ffc005510 con 0x7f900c0a5bd0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 0x7f900c135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90027fc640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ff003bb60 0x7f8ff003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient: get_auth_request con 0x7f900c134e70 auth_method 0 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 4783+0+0 (secure 0 0 0) 0x7f8ffc04f990 con 0x7f900c0a5bd0 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90120b4640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ff003bb60 0x7f8ff003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90120b4640 10 monclient: get_auth_request con 0x7f8ff003bb60 auth_method 0 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 10 monclient(hunting): handle_auth_done global_id 14263 payload 824 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.926+0000 7f90118b3640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f900c13a840 con 0x7f900c134e70 2024-09-17T15:30:50.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90118b3640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 msgr2=0x7f900c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90118b3640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 0x7f900c135250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9004004880 tx=0x7f900400c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90120b4640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ff003bb60 0x7f8ff003e020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f90080ac000 tx=0x7f90080a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004017070 con 0x7f900c134e70 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 10 monclient: dump: 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90040070d0 con 0x7f900c134e70 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 msgr2=0x7f900c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 0x7f900c135250 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9004004880 tx=0x7f900400c8e0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 msgr2=0x7f900c135250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:50.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.930+0000 7f90027fc640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c134e70 0x7f900c135250 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9004004880 tx=0x7f900400c8e0 comp rx=0 tx=0).stop 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f9013b17640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f9013b17640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 0x7f900c09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f9013b17640 10 monclient(hunting): start opening mon connection 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f900c13aec0 con 0x7f900c135790 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 0x7f900c09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient: get_auth_request con 0x7f900c135790 auth_method 0 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:30:50.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.934+0000 7f90118b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90118b3640 10 monclient(hunting): handle_auth_done global_id 14266 payload 824 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90118b3640 1 -- 172.21.15.86:0/646167884 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f900c074be0 con 0x7f900c135790 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90118b3640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 msgr2=0x7f900c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90118b3640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 0x7f900c09e5e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9004014000 tx=0x7f9004004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004017030 con 0x7f900c135790 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 10 monclient: dump: 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:30:50.945 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f9004022410 con 0x7f900c135790 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 msgr2=0x7f900c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 0x7f900c09e5e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9004014000 tx=0x7f9004004a20 comp rx=0 tx=0).stop 2024-09-17T15:30:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 msgr2=0x7f900c09e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.938+0000 7f90027fc640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f900c135790 0x7f900c09e5e0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9004014000 tx=0x7f9004004a20 comp rx=0 tx=0).stop 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 48, 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 159538, 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:30:50.947 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.948 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.949 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:30:50.950 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:30:50.951 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:30:50.952 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.955 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:30:50.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:30:50.959 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ff003bb60 msgr2=0x7f8ff003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ff003bb60 0x7f8ff003e020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f90080ac000 tx=0x7f90080a7070 comp rx=0 tx=0).stop 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 10 monclient: shutdown 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 msgr2=0x7f900c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 --2- 172.21.15.86:0/646167884 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f900c0a5bd0 0x7f900c1d0020 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00b550 tx=0x7f8ffc00ba10 comp rx=0 tx=0).stop 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f90120b4640 1 -- 172.21.15.86:0/646167884 reap_dead start 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f90120b4640 1 -- 172.21.15.86:0/646167884 reap_dead start 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 shutdown_connections 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 >> 172.21.15.86:0/646167884 conn(0x7f900c0aaae0 msgr2=0x7f900c13deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 shutdown_connections 2024-09-17T15:30:50.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:30:50.942+0000 7f9013b17640 1 -- 172.21.15.86:0/646167884 wait complete. 2024-09-17T15:30:50.962 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:30:50.962 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:30:51.342 INFO:tasks.rados.rados.0.smithi203.stdout:389: done (0 left) 2024-09-17T15:30:51.342 INFO:tasks.rados.rados.0.smithi203.stdout:390: copy_from oid 49 from oid 48 current snap is 40 2024-09-17T15:30:51.342 INFO:tasks.rados.rados.0.smithi203.stdout:391: snap_remove snap 37 2024-09-17T15:30:52.351 INFO:tasks.rados.rados.0.smithi203.stdout:391: done (1 left) 2024-09-17T15:30:52.351 INFO:tasks.rados.rados.0.smithi203.stdout:392: snap_create 2024-09-17T15:30:52.351 INFO:tasks.rados.rados.0.smithi203.stdout:390: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:30:52.351 INFO:tasks.rados.rados.0.smithi203.stdout:390: finishing copy_from to smithi20328086-49 2024-09-17T15:30:52.351 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 88 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:30:52.419 INFO:tasks.rados.rados.0.smithi203.stdout:390: done (1 left) 2024-09-17T15:30:52.419 INFO:tasks.rados.rados.0.smithi203.stdout:392: done (0 left) 2024-09-17T15:30:52.419 INFO:tasks.rados.rados.0.smithi203.stdout:393: read oid 25 snap -1 2024-09-17T15:30:52.419 INFO:tasks.rados.rados.0.smithi203.stdout:393: expect (ObjNum 112 snap 20 seq_num 112) 2024-09-17T15:30:52.419 INFO:tasks.rados.rados.0.smithi203.stdout:394: snap_remove snap 39 2024-09-17T15:30:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:394: done (1 left) 2024-09-17T15:30:53.428 INFO:tasks.rados.rados.0.smithi203.stdout:395: write oid 2 current snap is 41 2024-09-17T15:30:53.429 INFO:tasks.rados.rados.0.smithi203.stdout:395: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-17T15:30:53.453 INFO:tasks.rados.rados.0.smithi203.stdout:395: writing smithi20328086-2 from 756348 to 1540597 tid 1 2024-09-17T15:30:53.474 INFO:tasks.rados.rados.0.smithi203.stdout:395: writing smithi20328086-2 from 2063988 to 2750375 tid 2 2024-09-17T15:30:53.474 INFO:tasks.rados.rados.0.smithi203.stdout:396: write oid 15 current snap is 41 2024-09-17T15:30:53.474 INFO:tasks.rados.rados.0.smithi203.stdout:396: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-17T15:30:53.490 INFO:tasks.rados.rados.0.smithi203.stdout:396: writing smithi20328086-15 from 773155 to 1232653 tid 1 2024-09-17T15:30:53.499 INFO:tasks.rados.rados.0.smithi203.stdout:396: writing smithi20328086-15 from 1722470 to 2152515 tid 2 2024-09-17T15:30:53.505 INFO:tasks.rados.rados.0.smithi203.stdout:396: writing smithi20328086-15 from 2648676 to 2773155 tid 3 2024-09-17T15:30:53.506 INFO:tasks.rados.rados.0.smithi203.stdout:397: copy_from oid 47 from oid 33 current snap is 41 2024-09-17T15:30:53.506 INFO:tasks.rados.rados.0.smithi203.stdout:393: expect (ObjNum 112 snap 20 seq_num 112) 2024-09-17T15:30:53.595 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3832781 2024-09-17T15:30:53.634 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3595487 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: finishing write tid 1 to smithi20328086-2 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: finishing write tid 2 to smithi20328086-2 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: finishing write tid 3 to smithi20328086-2 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: finishing write tid 4 to smithi20328086-2 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: oid 2 updating version 0 to 68 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: oid 2 updating version 68 to 69 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: oid 2 version 69 is already newer than 67 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 69 (ObjNum 164 snap 41 seq_num 164) dirty exists 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:395: left oid 2 (ObjNum 164 snap 41 seq_num 164) 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:396: finishing write tid 1 to smithi20328086-15 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:396: finishing write tid 2 to smithi20328086-15 2024-09-17T15:30:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:396: finishing write tid 3 to smithi20328086-15 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: finishing write tid 4 to smithi20328086-15 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: finishing write tid 5 to smithi20328086-15 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: oid 15 updating version 0 to 45 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: oid 15 updating version 45 to 46 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: oid 15 version 46 is already newer than 44 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: oid 15 updating version 46 to 47 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 47 (ObjNum 165 snap 41 seq_num 165) dirty exists 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: left oid 15 (ObjNum 165 snap 41 seq_num 165) 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:397: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:397: finishing copy_from to smithi20328086-47 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:393: done (3 left) 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:395: done (2 left) 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:396: done (1 left) 2024-09-17T15:30:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:397: done (0 left) 2024-09-17T15:30:53.706 INFO:tasks.rados.rados.0.smithi203.stdout:398: snap_remove snap 40 2024-09-17T15:30:54.453 INFO:tasks.rados.rados.0.smithi203.stdout:398: done (0 left) 2024-09-17T15:30:54.453 INFO:tasks.rados.rados.0.smithi203.stdout:399: write (excl) oid 47 current snap is 41 2024-09-17T15:30:54.453 INFO:tasks.rados.rados.0.smithi203.stdout:399: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-17T15:30:54.469 INFO:tasks.rados.rados.0.smithi203.stdout:399: writing smithi20328086-47 from 789962 to 1324709 tid 1 2024-09-17T15:30:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:399: writing smithi20328086-47 from 1780952 to 2471008 tid 2 2024-09-17T15:30:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:400: snap_create 2024-09-17T15:30:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:399: finishing write tid 1 to smithi20328086-47 2024-09-17T15:30:54.493 INFO:tasks.rados.rados.0.smithi203.stdout:399: finishing write tid 2 to smithi20328086-47 2024-09-17T15:30:54.493 INFO:tasks.rados.rados.0.smithi203.stdout:399: finishing write tid 3 to smithi20328086-47 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:399: finishing write tid 4 to smithi20328086-47 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:399: oid 47 updating version 0 to 70 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:399: oid 47 version 70 is already newer than 69 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:399: oid 47 version 70 is already newer than 68 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 70 (ObjNum 166 snap 41 seq_num 166) dirty exists 2024-09-17T15:30:54.494 INFO:tasks.rados.rados.0.smithi203.stdout:399: left oid 47 (ObjNum 166 snap 41 seq_num 166) 2024-09-17T15:30:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:399: done (1 left) 2024-09-17T15:30:55.438 INFO:tasks.rados.rados.0.smithi203.stdout:400: done (0 left) 2024-09-17T15:30:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:401: write (excl) oid 50 current snap is 42 2024-09-17T15:30:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:401: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-17T15:30:55.450 INFO:tasks.rados.rados.0.smithi203.stdout:401: writing smithi20328086-50 from 406769 to 1016765 tid 1 2024-09-17T15:30:55.461 INFO:tasks.rados.rados.0.smithi203.stdout:401: writing smithi20328086-50 from 1439434 to 1873148 tid 2 2024-09-17T15:30:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:401: writing smithi20328086-50 from 2303522 to 2806769 tid 3 2024-09-17T15:30:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:402: read oid 12 snap 41 2024-09-17T15:30:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:402: expect (ObjNum 84 snap 8 seq_num 84) 2024-09-17T15:30:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:403: read oid 20 snap 33 2024-09-17T15:30:55.473 INFO:tasks.rados.rados.0.smithi203.stdout:403: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:30:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:404: read oid 47 snap 41 2024-09-17T15:30:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:404: expect (ObjNum 166 snap 41 seq_num 166) 2024-09-17T15:30:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:405: write (excl) oid 21 current snap is 42 2024-09-17T15:30:55.474 INFO:tasks.rados.rados.0.smithi203.stdout:405: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-17T15:30:55.484 INFO:tasks.rados.rados.0.smithi203.stdout:405: writing smithi20328086-21 from 423576 to 825174 tid 1 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:405: writing smithi20328086-21 from 1497916 to 2191641 tid 2 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:406: delete oid 11 current snap is 42 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: finishing write tid 1 to smithi20328086-50 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: finishing write tid 2 to smithi20328086-50 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: finishing write tid 3 to smithi20328086-50 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: finishing write tid 4 to smithi20328086-50 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: finishing write tid 5 to smithi20328086-50 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: oid 50 updating version 0 to 58 2024-09-17T15:30:55.500 INFO:tasks.rados.rados.0.smithi203.stdout:401: oid 50 updating version 58 to 59 2024-09-17T15:30:55.501 INFO:tasks.rados.rados.0.smithi203.stdout:401: oid 50 updating version 59 to 60 2024-09-17T15:30:55.501 INFO:tasks.rados.rados.0.smithi203.stdout:401: oid 50 updating version 60 to 61 2024-09-17T15:30:55.501 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 61 (ObjNum 167 snap 42 seq_num 167) dirty exists 2024-09-17T15:30:55.501 INFO:tasks.rados.rados.0.smithi203.stdout:401: left oid 50 (ObjNum 167 snap 42 seq_num 167) 2024-09-17T15:30:55.501 INFO:tasks.rados.rados.0.smithi203.stdout:402: expect (ObjNum 84 snap 8 seq_num 84) 2024-09-17T15:30:55.598 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:55.689 INFO:tasks.rados.rados.0.smithi203.stdout:403: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-17T15:30:55.695 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1072258 2024-09-17T15:30:55.709 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:55.735 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 500495 2024-09-17T15:30:55.757 INFO:tasks.rados.rados.0.smithi203.stdout:404: expect (ObjNum 166 snap 41 seq_num 166) 2024-09-17T15:30:55.844 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:30:55.887 INFO:tasks.rados.rados.0.smithi203.stdout:405: finishing write tid 1 to smithi20328086-21 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: finishing write tid 2 to smithi20328086-21 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: finishing write tid 3 to smithi20328086-21 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: finishing write tid 4 to smithi20328086-21 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: oid 21 updating version 0 to 163 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: oid 21 version 163 is already newer than 162 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: oid 21 version 163 is already newer than 161 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 163 (ObjNum 168 snap 42 seq_num 168) dirty exists 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: left oid 21 (ObjNum 168 snap 42 seq_num 168) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:401: done (5 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:402: done (4 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:403: done (3 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:404: done (2 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:405: done (1 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:406: done (0 left) 2024-09-17T15:30:55.888 INFO:tasks.rados.rados.0.smithi203.stdout:407: snap_remove snap 41 2024-09-17T15:30:56.453 INFO:tasks.rados.rados.0.smithi203.stdout:407: done (0 left) 2024-09-17T15:30:56.453 INFO:tasks.rados.rados.0.smithi203.stdout:408: write oid 22 current snap is 42 2024-09-17T15:30:56.454 INFO:tasks.rados.rados.0.smithi203.stdout:408: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-17T15:30:56.470 INFO:tasks.rados.rados.0.smithi203.stdout:408: writing smithi20328086-22 from 457190 to 1009286 tid 1 2024-09-17T15:30:56.489 INFO:tasks.rados.rados.0.smithi203.stdout:408: writing smithi20328086-22 from 1614880 to 2195921 tid 2 2024-09-17T15:30:56.498 INFO:tasks.rados.rados.0.smithi203.stdout:408: writing smithi20328086-22 from 2785791 to 2857190 tid 3 2024-09-17T15:30:56.499 INFO:tasks.rados.rados.0.smithi203.stdout:409: delete oid 21 current snap is 42 2024-09-17T15:30:56.499 INFO:tasks.rados.rados.0.smithi203.stdout:408: finishing write tid 1 to smithi20328086-22 2024-09-17T15:30:56.499 INFO:tasks.rados.rados.0.smithi203.stdout:408: finishing write tid 2 to smithi20328086-22 2024-09-17T15:30:56.502 INFO:tasks.rados.rados.0.smithi203.stdout:408: finishing write tid 3 to smithi20328086-22 2024-09-17T15:30:56.503 INFO:tasks.rados.rados.0.smithi203.stdout:409: done (1 left) 2024-09-17T15:30:56.503 INFO:tasks.rados.rados.0.smithi203.stdout:410: snap_remove snap 33 2024-09-17T15:30:57.420 INFO:tasks.rados.rados.0.smithi203.stdout:410: done (1 left) 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:411: read oid 39 snap -1 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:411: expect deleted 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:412: write (excl) oid 13 current snap is 42 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: finishing write tid 4 to smithi20328086-22 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: finishing write tid 5 to smithi20328086-22 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: oid 22 updating version 0 to 35 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: oid 22 updating version 35 to 36 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: oid 22 version 36 is already newer than 34 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: oid 22 updating version 36 to 37 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 37 (ObjNum 170 snap 42 seq_num 170) dirty exists 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:408: left oid 22 (ObjNum 170 snap 42 seq_num 170) 2024-09-17T15:30:57.421 INFO:tasks.rados.rados.0.smithi203.stdout:412: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-17T15:30:57.441 INFO:tasks.rados.rados.0.smithi203.stdout:412: writing smithi20328086-13 from 490804 to 1193398 tid 1 2024-09-17T15:30:57.458 INFO:tasks.rados.rados.0.smithi203.stdout:412: writing smithi20328086-13 from 1615491 to 2200201 tid 2 2024-09-17T15:30:57.469 INFO:tasks.rados.rados.0.smithi203.stdout:412: writing smithi20328086-13 from 2724284 to 2890804 tid 3 2024-09-17T15:30:57.469 INFO:tasks.rados.rados.0.smithi203.stdout:408: done (2 left) 2024-09-17T15:30:57.469 INFO:tasks.rados.rados.0.smithi203.stdout:413: snap_create 2024-09-17T15:30:57.469 INFO:tasks.rados.rados.0.smithi203.stdout:412: finishing write tid 1 to smithi20328086-13 2024-09-17T15:30:57.469 INFO:tasks.rados.rados.0.smithi203.stdout:412: finishing write tid 2 to smithi20328086-13 2024-09-17T15:30:57.475 INFO:tasks.rados.rados.0.smithi203.stdout:412: finishing write tid 3 to smithi20328086-13 2024-09-17T15:30:57.481 INFO:tasks.rados.rados.0.smithi203.stdout:412: finishing write tid 4 to smithi20328086-13 2024-09-17T15:30:57.481 INFO:tasks.rados.rados.0.smithi203.stdout:412: finishing write tid 5 to smithi20328086-13 2024-09-17T15:30:57.481 INFO:tasks.rados.rados.0.smithi203.stdout:412: oid 13 updating version 0 to 73 2024-09-17T15:30:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:412: oid 13 updating version 73 to 74 2024-09-17T15:30:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:412: oid 13 version 74 is already newer than 71 2024-09-17T15:30:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:412: oid 13 version 74 is already newer than 72 2024-09-17T15:30:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 74 (ObjNum 172 snap 42 seq_num 172) dirty exists 2024-09-17T15:30:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:412: left oid 13 (ObjNum 172 snap 42 seq_num 172) 2024-09-17T15:30:58.773 INFO:tasks.rados.rados.0.smithi203.stdout:411: done (2 left) 2024-09-17T15:30:58.773 INFO:tasks.rados.rados.0.smithi203.stdout:412: done (1 left) 2024-09-17T15:30:58.773 INFO:tasks.rados.rados.0.smithi203.stdout:413: done (0 left) 2024-09-17T15:30:58.773 INFO:tasks.rados.rados.0.smithi203.stdout:414: snap_remove snap 42 2024-09-17T15:30:59.782 INFO:tasks.rados.rados.0.smithi203.stdout:414: done (0 left) 2024-09-17T15:30:59.782 INFO:tasks.rados.rados.0.smithi203.stdout:415: write oid 39 current snap is 43 2024-09-17T15:30:59.782 INFO:tasks.rados.rados.0.smithi203.stdout:415: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-17T15:30:59.802 INFO:tasks.rados.rados.0.smithi203.stdout:415: writing smithi20328086-39 from 507611 to 1285454 tid 1 2024-09-17T15:30:59.822 INFO:tasks.rados.rados.0.smithi203.stdout:415: writing smithi20328086-39 from 2073973 to 2518694 tid 2 2024-09-17T15:30:59.822 INFO:tasks.rados.rados.0.smithi203.stdout:416: rollback oid 2 current snap is 43 2024-09-17T15:30:59.822 INFO:tasks.rados.rados.0.smithi203.stdout:416: done (1 left) 2024-09-17T15:30:59.822 INFO:tasks.rados.rados.0.smithi203.stdout:417: snap_create 2024-09-17T15:30:59.823 INFO:tasks.rados.rados.0.smithi203.stdout:415: finishing write tid 1 to smithi20328086-39 2024-09-17T15:30:59.828 INFO:tasks.rados.rados.0.smithi203.stdout:415: finishing write tid 2 to smithi20328086-39 2024-09-17T15:30:59.829 INFO:tasks.rados.rados.0.smithi203.stdout:415: finishing write tid 3 to smithi20328086-39 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:415: finishing write tid 4 to smithi20328086-39 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:415: oid 39 updating version 0 to 95 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:415: oid 39 version 95 is already newer than 94 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:415: oid 39 version 95 is already newer than 93 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 95 (ObjNum 173 snap 43 seq_num 173) dirty exists 2024-09-17T15:30:59.830 INFO:tasks.rados.rados.0.smithi203.stdout:415: left oid 39 (ObjNum 173 snap 43 seq_num 173) 2024-09-17T15:31:00.782 INFO:tasks.rados.rados.0.smithi203.stdout:415: done (1 left) 2024-09-17T15:31:00.782 INFO:tasks.rados.rados.0.smithi203.stdout:417: done (0 left) 2024-09-17T15:31:00.782 INFO:tasks.rados.rados.0.smithi203.stdout:418: write (excl) oid 2 current snap is 44 2024-09-17T15:31:00.782 INFO:tasks.rados.rados.0.smithi203.stdout:418: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-17T15:31:00.797 INFO:tasks.rados.rados.0.smithi203.stdout:418: writing smithi20328086-2 from 524418 to 977510 tid 1 2024-09-17T15:31:00.820 INFO:tasks.rados.rados.0.smithi203.stdout:418: writing smithi20328086-2 from 1732455 to 2320834 tid 2 2024-09-17T15:31:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:418: writing smithi20328086-2 from 2779130 to 2924418 tid 3 2024-09-17T15:31:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:419: read oid 30 snap 43 2024-09-17T15:31:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:419: expect deleted 2024-09-17T15:31:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:420: write (excl) oid 12 current snap is 44 2024-09-17T15:31:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:420: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-17T15:31:00.845 INFO:tasks.rados.rados.0.smithi203.stdout:420: writing smithi20328086-12 from 541225 to 1185919 tid 1 2024-09-17T15:31:00.857 INFO:tasks.rados.rados.0.smithi203.stdout:420: writing smithi20328086-12 from 1907290 to 2355680 tid 2 2024-09-17T15:31:00.858 INFO:tasks.rados.rados.0.smithi203.stdout:421: write oid 35 current snap is 44 2024-09-17T15:31:00.858 INFO:tasks.rados.rados.0.smithi203.stdout:421: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-17T15:31:00.868 INFO:tasks.rados.rados.0.smithi203.stdout:421: writing smithi20328086-35 from 558032 to 1277975 tid 1 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:421: writing smithi20328086-35 from 1849419 to 2441467 tid 2 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:422: copy_from oid 46 from oid 7 current snap is 44 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:418: finishing write tid 1 to smithi20328086-2 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:418: finishing write tid 2 to smithi20328086-2 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:423: write oid 8 current snap is 44 2024-09-17T15:31:00.877 INFO:tasks.rados.rados.0.smithi203.stdout:423: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-17T15:31:00.886 INFO:tasks.rados.rados.0.smithi203.stdout:423: writing smithi20328086-8 from 574839 to 1370031 tid 1 2024-09-17T15:31:00.893 INFO:tasks.rados.rados.0.smithi203.stdout:423: writing smithi20328086-8 from 1907901 to 2359960 tid 2 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:424: rollback oid 43 current snap is 44 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 43 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: finishing write tid 3 to smithi20328086-2 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: finishing write tid 4 to smithi20328086-2 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: finishing write tid 5 to smithi20328086-2 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: oid 2 updating version 0 to 75 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: oid 2 updating version 75 to 76 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: oid 2 updating version 76 to 77 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: oid 2 updating version 77 to 78 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 78 (ObjNum 174 snap 44 seq_num 174) dirty exists 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:418: left oid 2 (ObjNum 174 snap 44 seq_num 174) 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:420: finishing write tid 1 to smithi20328086-12 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:420: finishing write tid 2 to smithi20328086-12 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:420: finishing write tid 3 to smithi20328086-12 2024-09-17T15:31:00.894 INFO:tasks.rados.rados.0.smithi203.stdout:420: finishing write tid 4 to smithi20328086-12 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:420: oid 12 updating version 0 to 53 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:420: oid 12 version 53 is already newer than 52 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:420: oid 12 version 53 is already newer than 51 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 53 (ObjNum 175 snap 44 seq_num 175) dirty exists 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:420: left oid 12 (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:418: done (6 left) 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:419: done (5 left) 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:420: done (4 left) 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:425: rollback oid 41 current snap is 44 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 43 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:421: finishing write tid 1 to smithi20328086-35 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:421: finishing write tid 2 to smithi20328086-35 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:421: finishing write tid 3 to smithi20328086-35 2024-09-17T15:31:00.895 INFO:tasks.rados.rados.0.smithi203.stdout:421: finishing write tid 4 to smithi20328086-35 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:421: oid 35 updating version 0 to 62 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:421: oid 35 updating version 62 to 64 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:421: oid 35 version 64 is already newer than 63 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 64 (ObjNum 176 snap 44 seq_num 176) dirty exists 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:421: left oid 35 (ObjNum 176 snap 44 seq_num 176) 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:421: done (4 left) 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:426: write oid 5 current snap is 44 2024-09-17T15:31:00.896 INFO:tasks.rados.rados.0.smithi203.stdout:426: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-17T15:31:00.899 INFO:tasks.rados.rados.0.smithi203.stdout:426: writing smithi20328086-5 from 591646 to 1062087 tid 1 2024-09-17T15:31:00.906 INFO:tasks.rados.rados.0.smithi203.stdout:426: writing smithi20328086-5 from 1566383 to 2162100 tid 2 2024-09-17T15:31:00.912 INFO:tasks.rados.rados.0.smithi203.stdout:426: writing smithi20328086-5 from 2888822 to 2991646 tid 3 2024-09-17T15:31:00.912 INFO:tasks.rados.rados.0.smithi203.stdout:427: write (excl) oid 6 current snap is 44 2024-09-17T15:31:00.912 INFO:tasks.rados.rados.0.smithi203.stdout:427: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-17T15:31:00.919 INFO:tasks.rados.rados.0.smithi203.stdout:427: writing smithi20328086-6 from 608453 to 1154143 tid 1 2024-09-17T15:31:00.925 INFO:tasks.rados.rados.0.smithi203.stdout:427: writing smithi20328086-6 from 1624865 to 2080593 tid 2 2024-09-17T15:31:00.932 INFO:tasks.rados.rados.0.smithi203.stdout:427: writing smithi20328086-6 from 2632598 to 3008453 tid 3 2024-09-17T15:31:00.932 INFO:tasks.rados.rados.0.smithi203.stdout:428: delete oid 37 current snap is 44 2024-09-17T15:31:00.932 INFO:tasks.rados.rados.0.smithi203.stdout:422: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: finishing write tid 1 to smithi20328086-8 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:422: finishing copy_from to smithi20328086-46 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 41 (ObjNum 137 snap 29 seq_num 137) dirty exists 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: finishing write tid 2 to smithi20328086-8 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: finishing write tid 3 to smithi20328086-8 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: finishing write tid 4 to smithi20328086-8 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: oid 8 updating version 0 to 75 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: oid 8 version 75 is already newer than 74 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: oid 8 updating version 75 to 76 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 76 (ObjNum 177 snap 44 seq_num 177) dirty exists 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:423: left oid 8 (ObjNum 177 snap 44 seq_num 177) 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:424: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:424: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:31:00.933 INFO:tasks.rados.rados.0.smithi203.stdout:424: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 69 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:425: finishing rollback tid 0 to smithi20328086-41 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:425: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:425: finishing rollback tid 2 to smithi20328086-41 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 73 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: finishing write tid 1 to smithi20328086-5 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: finishing write tid 2 to smithi20328086-5 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: finishing write tid 3 to smithi20328086-5 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: finishing write tid 4 to smithi20328086-5 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: finishing write tid 5 to smithi20328086-5 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: oid 5 updating version 0 to 75 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: oid 5 version 75 is already newer than 74 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: oid 5 updating version 75 to 76 2024-09-17T15:31:00.934 INFO:tasks.rados.rados.0.smithi203.stdout:426: oid 5 version 76 is already newer than 73 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 76 (ObjNum 178 snap 44 seq_num 178) dirty exists 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:426: left oid 5 (ObjNum 178 snap 44 seq_num 178) 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:427: finishing write tid 1 to smithi20328086-6 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:427: finishing write tid 2 to smithi20328086-6 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:422: done (6 left) 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:423: done (5 left) 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:424: done (4 left) 2024-09-17T15:31:00.935 INFO:tasks.rados.rados.0.smithi203.stdout:425: done (3 left) 2024-09-17T15:31:00.936 INFO:tasks.rados.rados.0.smithi203.stdout:426: done (2 left) 2024-09-17T15:31:00.936 INFO:tasks.rados.rados.0.smithi203.stdout:428: done (1 left) 2024-09-17T15:31:00.936 INFO:tasks.rados.rados.0.smithi203.stdout:429: snap_remove snap 43 2024-09-17T15:31:01.796 INFO:tasks.rados.rados.0.smithi203.stdout:429: done (1 left) 2024-09-17T15:31:01.796 INFO:tasks.rados.rados.0.smithi203.stdout:430: copy_from oid 17 from oid 46 current snap is 44 2024-09-17T15:31:01.796 INFO:tasks.rados.rados.0.smithi203.stdout:427: finishing write tid 3 to smithi20328086-6 2024-09-17T15:31:01.796 INFO:tasks.rados.rados.0.smithi203.stdout:427: finishing write tid 4 to smithi20328086-6 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: finishing write tid 5 to smithi20328086-6 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: oid 6 updating version 0 to 96 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: oid 6 version 96 is already newer than 93 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: oid 6 version 96 is already newer than 95 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: oid 6 version 96 is already newer than 94 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 96 (ObjNum 179 snap 44 seq_num 179) dirty exists 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: left oid 6 (ObjNum 179 snap 44 seq_num 179) 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:427: done (1 left) 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:431: copy_from oid 22 from oid 33 current snap is 44 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:432: read oid 47 snap -1 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:432: expect (ObjNum 166 snap 41 seq_num 166) 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:433: rollback oid 27 current snap is 44 2024-09-17T15:31:01.797 INFO:tasks.rados.rados.0.smithi203.stdout:433: done (3 left) 2024-09-17T15:31:01.798 INFO:tasks.rados.rados.0.smithi203.stdout:434: write oid 1 current snap is 44 2024-09-17T15:31:01.798 INFO:tasks.rados.rados.0.smithi203.stdout:434: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-17T15:31:01.805 INFO:tasks.rados.rados.0.smithi203.stdout:434: writing smithi20328086-1 from 642067 to 1338255 tid 1 2024-09-17T15:31:01.813 INFO:tasks.rados.rados.0.smithi203.stdout:434: writing smithi20328086-1 from 2025476 to 2484873 tid 2 2024-09-17T15:31:01.818 INFO:tasks.rados.rados.0.smithi203.stdout:434: writing smithi20328086-1 from 2971091 to 3042067 tid 3 2024-09-17T15:31:01.818 INFO:tasks.rados.rados.0.smithi203.stdout:435: read oid 26 snap -1 2024-09-17T15:31:01.818 INFO:tasks.rados.rados.0.smithi203.stdout:435: expect (ObjNum 133 snap 29 seq_num 133) 2024-09-17T15:31:01.818 INFO:tasks.rados.rados.0.smithi203.stdout:436: read oid 21 snap -1 2024-09-17T15:31:01.819 INFO:tasks.rados.rados.0.smithi203.stdout:436: expect deleted 2024-09-17T15:31:01.819 INFO:tasks.rados.rados.0.smithi203.stdout:437: read oid 38 snap -1 2024-09-17T15:31:01.819 INFO:tasks.rados.rados.0.smithi203.stdout:437: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:31:01.819 INFO:tasks.rados.rados.0.smithi203.stdout:438: write (excl) oid 7 current snap is 44 2024-09-17T15:31:01.819 INFO:tasks.rados.rados.0.smithi203.stdout:438: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-17T15:31:01.829 INFO:tasks.rados.rados.0.smithi203.stdout:438: writing smithi20328086-7 from 658874 to 1430311 tid 1 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:438: writing smithi20328086-7 from 2083958 to 2687013 tid 2 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:439: rollback oid 39 current snap is 44 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:439: done (8 left) 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:440: read oid 27 snap -1 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:440: expect (ObjNum 130 snap 28 seq_num 130) 2024-09-17T15:31:01.838 INFO:tasks.rados.rados.0.smithi203.stdout:441: read oid 9 snap -1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:441: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:442: snap_create 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:430: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: finishing write tid 1 to smithi20328086-1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: finishing write tid 2 to smithi20328086-1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: finishing write tid 3 to smithi20328086-1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: finishing write tid 4 to smithi20328086-1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: finishing write tid 5 to smithi20328086-1 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: oid 1 updating version 0 to 99 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: oid 1 version 99 is already newer than 98 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: oid 1 version 99 is already newer than 97 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: oid 1 updating version 99 to 100 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 100 (ObjNum 181 snap 44 seq_num 181) dirty exists 2024-09-17T15:31:01.839 INFO:tasks.rados.rados.0.smithi203.stdout:434: left oid 1 (ObjNum 181 snap 44 seq_num 181) 2024-09-17T15:31:01.840 INFO:tasks.rados.rados.0.smithi203.stdout:430: finishing copy_from to smithi20328086-17 2024-09-17T15:31:01.840 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 41 (ObjNum 137 snap 29 seq_num 137) dirty exists 2024-09-17T15:31:01.840 INFO:tasks.rados.rados.0.smithi203.stdout:432: expect (ObjNum 166 snap 41 seq_num 166) 2024-09-17T15:31:01.840 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 359151 2024-09-17T15:31:01.934 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:01.977 INFO:tasks.rados.rados.0.smithi203.stdout:437: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:31:01.977 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:02.011 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1979640 2024-09-17T15:31:02.022 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1623017 2024-09-17T15:31:02.026 INFO:tasks.rados.rados.0.smithi203.stdout:435: expect (ObjNum 133 snap 29 seq_num 133) 2024-09-17T15:31:02.026 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:02.063 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1212632 2024-09-17T15:31:02.100 INFO:tasks.rados.rados.0.smithi203.stdout:431: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:31:02.100 INFO:tasks.rados.rados.0.smithi203.stdout:431: finishing copy_from to smithi20328086-22 2024-09-17T15:31:02.100 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:31:02.100 INFO:tasks.rados.rados.0.smithi203.stdout:440: expect (ObjNum 130 snap 28 seq_num 130) 2024-09-17T15:31:02.200 INFO:tasks.rados.rados.0.smithi203.stdout:438: finishing write tid 1 to smithi20328086-7 2024-09-17T15:31:02.200 INFO:tasks.rados.rados.0.smithi203.stdout:438: finishing write tid 2 to smithi20328086-7 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: finishing write tid 3 to smithi20328086-7 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: finishing write tid 4 to smithi20328086-7 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: oid 7 updating version 0 to 81 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: oid 7 updating version 81 to 82 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: oid 7 updating version 82 to 83 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 83 (ObjNum 182 snap 44 seq_num 182) dirty exists 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:438: left oid 7 (ObjNum 182 snap 44 seq_num 182) 2024-09-17T15:31:02.201 INFO:tasks.rados.rados.0.smithi203.stdout:441: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:31:02.229 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 885133 2024-09-17T15:31:02.243 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:02.420 INFO:tasks.rados.rados.0.smithi203.stdout:430: done (10 left) 2024-09-17T15:31:02.420 INFO:tasks.rados.rados.0.smithi203.stdout:431: done (9 left) 2024-09-17T15:31:02.420 INFO:tasks.rados.rados.0.smithi203.stdout:432: done (8 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:434: done (7 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:435: done (6 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:436: done (5 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:437: done (4 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:438: done (3 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:440: done (2 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:441: done (1 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:442: done (0 left) 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:443: write oid 21 current snap is 45 2024-09-17T15:31:02.421 INFO:tasks.rados.rados.0.smithi203.stdout:443: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-17T15:31:02.431 INFO:tasks.rados.rados.0.smithi203.stdout:443: writing smithi20328086-21 from 675681 to 1238720 tid 1 2024-09-17T15:31:02.442 INFO:tasks.rados.rados.0.smithi203.stdout:443: writing smithi20328086-21 from 1858793 to 2605506 tid 2 2024-09-17T15:31:02.446 INFO:tasks.rados.rados.0.smithi203.stdout:443: writing smithi20328086-21 from 3025937 to 3075681 tid 3 2024-09-17T15:31:02.446 INFO:tasks.rados.rados.0.smithi203.stdout:444: write oid 20 current snap is 45 2024-09-17T15:31:02.446 INFO:tasks.rados.rados.0.smithi203.stdout:444: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-17T15:31:02.456 INFO:tasks.rados.rados.0.smithi203.stdout:444: writing smithi20328086-20 from 692488 to 1330776 tid 1 2024-09-17T15:31:02.466 INFO:tasks.rados.rados.0.smithi203.stdout:444: writing smithi20328086-20 from 1917275 to 2523999 tid 2 2024-09-17T15:31:02.466 INFO:tasks.rados.rados.0.smithi203.stdout:445: read oid 4 snap -1 2024-09-17T15:31:02.466 INFO:tasks.rados.rados.0.smithi203.stdout:445: expect (ObjNum 122 snap 26 seq_num 122) 2024-09-17T15:31:02.466 INFO:tasks.rados.rados.0.smithi203.stdout:446: read oid 12 snap 44 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:446: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:447: copy_from oid 25 from oid 41 current snap is 45 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:448: read oid 36 snap -1 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:448: expect (ObjNum 154 snap 36 seq_num 154) 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:449: snap_create 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: finishing write tid 1 to smithi20328086-21 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: finishing write tid 2 to smithi20328086-21 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: finishing write tid 3 to smithi20328086-21 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: finishing write tid 4 to smithi20328086-21 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: finishing write tid 5 to smithi20328086-21 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: oid 21 updating version 0 to 173 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: oid 21 version 173 is already newer than 172 2024-09-17T15:31:02.467 INFO:tasks.rados.rados.0.smithi203.stdout:443: oid 21 updating version 173 to 174 2024-09-17T15:31:02.468 INFO:tasks.rados.rados.0.smithi203.stdout:443: oid 21 version 174 is already newer than 171 2024-09-17T15:31:02.468 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 174 (ObjNum 183 snap 45 seq_num 183) dirty exists 2024-09-17T15:31:02.468 INFO:tasks.rados.rados.0.smithi203.stdout:443: left oid 21 (ObjNum 183 snap 45 seq_num 183) 2024-09-17T15:31:02.468 INFO:tasks.rados.rados.0.smithi203.stdout:444: finishing write tid 1 to smithi20328086-20 2024-09-17T15:31:02.481 INFO:tasks.rados.rados.0.smithi203.stdout:445: expect (ObjNum 122 snap 26 seq_num 122) 2024-09-17T15:31:02.481 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 260753 2024-09-17T15:31:02.510 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1218122 2024-09-17T15:31:02.525 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:02.546 INFO:tasks.rados.rados.0.smithi203.stdout:446: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:31:02.546 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 22814 2024-09-17T15:31:02.717 INFO:tasks.rados.rados.0.smithi203.stdout:444: finishing write tid 2 to smithi20328086-20 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: finishing write tid 3 to smithi20328086-20 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: finishing write tid 4 to smithi20328086-20 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: oid 20 updating version 0 to 103 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: oid 20 version 103 is already newer than 102 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: oid 20 updating version 103 to 104 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 104 (ObjNum 184 snap 45 seq_num 184) dirty exists 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:444: left oid 20 (ObjNum 184 snap 45 seq_num 184) 2024-09-17T15:31:02.718 INFO:tasks.rados.rados.0.smithi203.stdout:448: expect (ObjNum 154 snap 36 seq_num 154) 2024-09-17T15:31:02.789 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:02.829 INFO:tasks.rados.rados.0.smithi203.stdout:447: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:31:02.829 INFO:tasks.rados.rados.0.smithi203.stdout:447: finishing copy_from to smithi20328086-25 2024-09-17T15:31:02.829 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 73 (ObjNum 153 snap 35 seq_num 153) dirty exists 2024-09-17T15:31:03.432 INFO:tasks.rados.rados.0.smithi203.stdout:443: done (6 left) 2024-09-17T15:31:03.432 INFO:tasks.rados.rados.0.smithi203.stdout:444: done (5 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:445: done (4 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:446: done (3 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:447: done (2 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:448: done (1 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:449: done (0 left) 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:450: write (excl) oid 19 current snap is 46 2024-09-17T15:31:03.433 INFO:tasks.rados.rados.0.smithi203.stdout:450: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-17T15:31:03.443 INFO:tasks.rados.rados.0.smithi203.stdout:450: writing smithi20328086-19 from 709295 to 1422832 tid 1 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:450: writing smithi20328086-19 from 1859404 to 2609786 tid 2 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:451: rollback oid 18 current snap is 46 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 44 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:452: read oid 29 snap -1 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:452: expect (ObjNum 103 snap 18 seq_num 103) 2024-09-17T15:31:03.454 INFO:tasks.rados.rados.0.smithi203.stdout:453: delete oid 3 current snap is 46 2024-09-17T15:31:03.455 INFO:tasks.rados.rados.0.smithi203.stdout:450: finishing write tid 1 to smithi20328086-19 2024-09-17T15:31:03.456 INFO:tasks.rados.rados.0.smithi203.stdout:451: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:31:03.456 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 0 (ObjNum 129 snap 0 seq_num 3607075048) dirty dne 2024-09-17T15:31:03.458 INFO:tasks.rados.rados.0.smithi203.stdout:451: done (3 left) 2024-09-17T15:31:03.458 INFO:tasks.rados.rados.0.smithi203.stdout:453: done (2 left) 2024-09-17T15:31:03.458 INFO:tasks.rados.rados.0.smithi203.stdout:454: snap_remove snap 45 2024-09-17T15:31:04.442 INFO:tasks.rados.rados.0.smithi203.stdout:454: done (2 left) 2024-09-17T15:31:04.442 INFO:tasks.rados.rados.0.smithi203.stdout:455: write oid 37 current snap is 46 2024-09-17T15:31:04.442 INFO:tasks.rados.rados.0.smithi203.stdout:455: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-17T15:31:04.453 INFO:tasks.rados.rados.0.smithi203.stdout:455: writing smithi20328086-37 from 742909 to 1206944 tid 1 2024-09-17T15:31:04.467 INFO:tasks.rados.rados.0.smithi203.stdout:455: writing smithi20328086-37 from 1976368 to 2730419 tid 2 2024-09-17T15:31:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:456: read oid 28 snap -1 2024-09-17T15:31:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:456: expect (ObjNum 155 snap 36 seq_num 155) 2024-09-17T15:31:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:457: write (excl) oid 30 current snap is 46 2024-09-17T15:31:04.468 INFO:tasks.rados.rados.0.smithi203.stdout:457: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-17T15:31:04.480 INFO:tasks.rados.rados.0.smithi203.stdout:457: writing smithi20328086-30 from 759716 to 1299000 tid 1 2024-09-17T15:31:04.492 INFO:tasks.rados.rados.0.smithi203.stdout:457: writing smithi20328086-30 from 2034850 to 2648912 tid 2 2024-09-17T15:31:04.492 INFO:tasks.rados.rados.0.smithi203.stdout:458: read oid 15 snap -1 2024-09-17T15:31:04.492 INFO:tasks.rados.rados.0.smithi203.stdout:458: expect (ObjNum 165 snap 41 seq_num 165) 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:459: delete oid 45 current snap is 46 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: finishing write tid 2 to smithi20328086-19 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: finishing write tid 3 to smithi20328086-19 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: finishing write tid 4 to smithi20328086-19 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: oid 19 updating version 0 to 98 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: oid 19 updating version 98 to 99 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: oid 19 updating version 99 to 100 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 100 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:450: left oid 19 (ObjNum 185 snap 46 seq_num 185) 2024-09-17T15:31:04.493 INFO:tasks.rados.rados.0.smithi203.stdout:452: expect (ObjNum 103 snap 18 seq_num 103) 2024-09-17T15:31:04.505 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2722013 2024-09-17T15:31:04.525 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:04.572 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:04.611 INFO:tasks.rados.rados.0.smithi203.stdout:455: finishing write tid 1 to smithi20328086-37 2024-09-17T15:31:04.611 INFO:tasks.rados.rados.0.smithi203.stdout:455: finishing write tid 2 to smithi20328086-37 2024-09-17T15:31:04.611 INFO:tasks.rados.rados.0.smithi203.stdout:455: finishing write tid 3 to smithi20328086-37 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:455: finishing write tid 4 to smithi20328086-37 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:455: oid 37 updating version 0 to 105 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:455: oid 37 version 105 is already newer than 103 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:455: oid 37 version 105 is already newer than 104 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 105 (ObjNum 187 snap 46 seq_num 187) dirty exists 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:455: left oid 37 (ObjNum 187 snap 46 seq_num 187) 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:456: expect (ObjNum 155 snap 36 seq_num 155) 2024-09-17T15:31:04.612 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:04.642 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1927498 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: finishing write tid 1 to smithi20328086-30 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: finishing write tid 2 to smithi20328086-30 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: finishing write tid 3 to smithi20328086-30 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: finishing write tid 4 to smithi20328086-30 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: oid 30 updating version 0 to 58 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: oid 30 version 58 is already newer than 56 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: oid 30 version 58 is already newer than 57 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 58 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:457: left oid 30 (ObjNum 188 snap 46 seq_num 188) 2024-09-17T15:31:04.684 INFO:tasks.rados.rados.0.smithi203.stdout:458: expect (ObjNum 165 snap 41 seq_num 165) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:450: done (6 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:452: done (5 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:455: done (4 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:456: done (3 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:457: done (2 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:458: done (1 left) 2024-09-17T15:31:04.796 INFO:tasks.rados.rados.0.smithi203.stdout:459: done (0 left) 2024-09-17T15:31:04.797 INFO:tasks.rados.rados.0.smithi203.stdout:460: snap_create 2024-09-17T15:31:05.443 INFO:tasks.rados.rados.0.smithi203.stdout:460: done (0 left) 2024-09-17T15:31:05.444 INFO:tasks.rados.rados.0.smithi203.stdout:461: rollback oid 6 current snap is 47 2024-09-17T15:31:05.444 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 44 2024-09-17T15:31:05.444 INFO:tasks.rados.rados.0.smithi203.stdout:462: write oid 20 current snap is 47 2024-09-17T15:31:05.444 INFO:tasks.rados.rados.0.smithi203.stdout:462: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-17T15:31:05.468 INFO:tasks.rados.rados.0.smithi203.stdout:462: writing smithi20328086-20 from 793330 to 1483112 tid 1 2024-09-17T15:31:05.488 INFO:tasks.rados.rados.0.smithi203.stdout:462: writing smithi20328086-20 from 2035461 to 2653192 tid 2 2024-09-17T15:31:05.496 INFO:tasks.rados.rados.0.smithi203.stdout:462: writing smithi20328086-20 from 3101545 to 3193330 tid 3 2024-09-17T15:31:05.496 INFO:tasks.rados.rados.0.smithi203.stdout:463: snap_remove snap 46 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:463: done (2 left) 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:464: delete oid 41 current snap is 47 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:461: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:461: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:461: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 108 (ObjNum 179 snap 44 seq_num 179) dirty exists 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:462: finishing write tid 1 to smithi20328086-20 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:462: finishing write tid 2 to smithi20328086-20 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:462: finishing write tid 3 to smithi20328086-20 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:462: finishing write tid 4 to smithi20328086-20 2024-09-17T15:31:06.471 INFO:tasks.rados.rados.0.smithi203.stdout:462: finishing write tid 5 to smithi20328086-20 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:462: oid 20 updating version 0 to 112 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:462: oid 20 updating version 112 to 113 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:462: oid 20 version 113 is already newer than 111 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:462: oid 20 version 113 is already newer than 110 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 113 (ObjNum 190 snap 47 seq_num 190) dirty exists 2024-09-17T15:31:06.472 INFO:tasks.rados.rados.0.smithi203.stdout:462: left oid 20 (ObjNum 190 snap 47 seq_num 190) 2024-09-17T15:31:06.477 INFO:tasks.rados.rados.0.smithi203.stdout:461: done (2 left) 2024-09-17T15:31:06.477 INFO:tasks.rados.rados.0.smithi203.stdout:462: done (1 left) 2024-09-17T15:31:06.477 INFO:tasks.rados.rados.0.smithi203.stdout:464: done (0 left) 2024-09-17T15:31:06.477 INFO:tasks.rados.rados.0.smithi203.stdout:465: write oid 18 current snap is 47 2024-09-17T15:31:06.477 INFO:tasks.rados.rados.0.smithi203.stdout:465: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-17T15:31:06.490 INFO:tasks.rados.rados.0.smithi203.stdout:465: writing smithi20328086-18 from 426944 to 983577 tid 1 2024-09-17T15:31:06.504 INFO:tasks.rados.rados.0.smithi203.stdout:465: writing smithi20328086-18 from 1468778 to 2090178 tid 2 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:465: writing smithi20328086-18 from 2872744 to 3226944 tid 3 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:466: copy_from oid 35 from oid 3 current snap is 47 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:465: finishing write tid 1 to smithi20328086-18 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:465: finishing write tid 2 to smithi20328086-18 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:467: copy_from oid 24 from oid 50 current snap is 47 2024-09-17T15:31:06.520 INFO:tasks.rados.rados.0.smithi203.stdout:468: snap_create 2024-09-17T15:31:06.521 INFO:tasks.rados.rados.0.smithi203.stdout:466: finishing copy_from to smithi20328086-35 2024-09-17T15:31:06.522 INFO:tasks.rados.rados.0.smithi203.stdout:466: got expected ENOENT (src dne) 2024-09-17T15:31:06.522 INFO:tasks.rados.rados.0.smithi203.stdout:466: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:31:06.526 INFO:tasks.rados.rados.0.smithi203.stdout:465: finishing write tid 3 to smithi20328086-18 2024-09-17T15:31:06.528 INFO:tasks.rados.rados.0.smithi203.stdout:465: finishing write tid 4 to smithi20328086-18 2024-09-17T15:31:06.528 INFO:tasks.rados.rados.0.smithi203.stdout:465: finishing write tid 5 to smithi20328086-18 2024-09-17T15:31:06.528 INFO:tasks.rados.rados.0.smithi203.stdout:465: oid 18 updating version 0 to 79 2024-09-17T15:31:06.528 INFO:tasks.rados.rados.0.smithi203.stdout:465: oid 18 updating version 79 to 80 2024-09-17T15:31:06.529 INFO:tasks.rados.rados.0.smithi203.stdout:465: oid 18 updating version 80 to 82 2024-09-17T15:31:06.529 INFO:tasks.rados.rados.0.smithi203.stdout:465: oid 18 version 82 is already newer than 81 2024-09-17T15:31:06.529 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 82 (ObjNum 192 snap 47 seq_num 192) dirty exists 2024-09-17T15:31:06.529 INFO:tasks.rados.rados.0.smithi203.stdout:465: left oid 18 (ObjNum 192 snap 47 seq_num 192) 2024-09-17T15:31:06.552 INFO:tasks.rados.rados.0.smithi203.stdout:467: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:31:06.556 INFO:tasks.rados.rados.0.smithi203.stdout:467: finishing copy_from to smithi20328086-24 2024-09-17T15:31:06.557 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 61 (ObjNum 167 snap 42 seq_num 167) dirty exists 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:465: done (3 left) 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:466: done (2 left) 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:467: done (1 left) 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:468: done (0 left) 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:469: write (excl) oid 15 current snap is 48 2024-09-17T15:31:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:469: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-17T15:31:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:469: writing smithi20328086-15 from 443751 to 1075633 tid 1 2024-09-17T15:31:07.465 INFO:tasks.rados.rados.0.smithi203.stdout:469: writing smithi20328086-15 from 1810907 to 2575965 tid 2 2024-09-17T15:31:07.474 INFO:tasks.rados.rados.0.smithi203.stdout:469: writing smithi20328086-15 from 3067461 to 3243751 tid 3 2024-09-17T15:31:07.474 INFO:tasks.rados.rados.0.smithi203.stdout:470: copy_from oid 20 from oid 6 current snap is 48 2024-09-17T15:31:07.474 INFO:tasks.rados.rados.0.smithi203.stdout:469: finishing write tid 1 to smithi20328086-15 2024-09-17T15:31:07.474 INFO:tasks.rados.rados.0.smithi203.stdout:469: finishing write tid 2 to smithi20328086-15 2024-09-17T15:31:07.475 INFO:tasks.rados.rados.0.smithi203.stdout:471: copy_from oid 10 from oid 31 current snap is 48 2024-09-17T15:31:07.475 INFO:tasks.rados.rados.0.smithi203.stdout:472: snap_create 2024-09-17T15:31:07.478 INFO:tasks.rados.rados.0.smithi203.stdout:469: finishing write tid 3 to smithi20328086-15 2024-09-17T15:31:07.480 INFO:tasks.rados.rados.0.smithi203.stdout:469: finishing write tid 4 to smithi20328086-15 2024-09-17T15:31:07.480 INFO:tasks.rados.rados.0.smithi203.stdout:469: finishing write tid 5 to smithi20328086-15 2024-09-17T15:31:07.480 INFO:tasks.rados.rados.0.smithi203.stdout:469: oid 15 updating version 0 to 63 2024-09-17T15:31:07.480 INFO:tasks.rados.rados.0.smithi203.stdout:469: oid 15 version 63 is already newer than 62 2024-09-17T15:31:07.480 INFO:tasks.rados.rados.0.smithi203.stdout:469: oid 15 version 63 is already newer than 61 2024-09-17T15:31:07.481 INFO:tasks.rados.rados.0.smithi203.stdout:469: oid 15 version 63 is already newer than 60 2024-09-17T15:31:07.481 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 63 (ObjNum 193 snap 48 seq_num 193) dirty exists 2024-09-17T15:31:07.481 INFO:tasks.rados.rados.0.smithi203.stdout:469: left oid 15 (ObjNum 193 snap 48 seq_num 193) 2024-09-17T15:31:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:471: finishing copy_from to smithi20328086-10 2024-09-17T15:31:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:471: got expected ENOENT (src dne) 2024-09-17T15:31:07.487 INFO:tasks.rados.rados.0.smithi203.stdout:471: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:31:07.500 INFO:tasks.rados.rados.0.smithi203.stdout:470: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:31:07.503 INFO:tasks.rados.rados.0.smithi203.stdout:470: finishing copy_from to smithi20328086-20 2024-09-17T15:31:07.503 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 108 (ObjNum 179 snap 44 seq_num 179) dirty exists 2024-09-17T15:31:08.829 INFO:tasks.rados.rados.0.smithi203.stdout:469: done (3 left) 2024-09-17T15:31:08.829 INFO:tasks.rados.rados.0.smithi203.stdout:470: done (2 left) 2024-09-17T15:31:08.829 INFO:tasks.rados.rados.0.smithi203.stdout:471: done (1 left) 2024-09-17T15:31:08.829 INFO:tasks.rados.rados.0.smithi203.stdout:472: done (0 left) 2024-09-17T15:31:08.829 INFO:tasks.rados.rados.0.smithi203.stdout:473: delete oid 27 current snap is 49 2024-09-17T15:31:08.834 INFO:tasks.rados.rados.0.smithi203.stdout:473: done (0 left) 2024-09-17T15:31:08.834 INFO:tasks.rados.rados.0.smithi203.stdout:474: read oid 50 snap -1 2024-09-17T15:31:08.834 INFO:tasks.rados.rados.0.smithi203.stdout:474: expect (ObjNum 167 snap 42 seq_num 167) 2024-09-17T15:31:08.834 INFO:tasks.rados.rados.0.smithi203.stdout:475: delete oid 45 current snap is 49 2024-09-17T15:31:08.841 INFO:tasks.rados.rados.0.smithi203.stdout:475: done (1 left) 2024-09-17T15:31:08.841 INFO:tasks.rados.rados.0.smithi203.stdout:476: rollback oid 30 current snap is 49 2024-09-17T15:31:08.841 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 30 to 47 2024-09-17T15:31:08.841 INFO:tasks.rados.rados.0.smithi203.stdout:477: read oid 45 snap -1 2024-09-17T15:31:08.841 INFO:tasks.rados.rados.0.smithi203.stdout:477: expect deleted 2024-09-17T15:31:08.842 INFO:tasks.rados.rados.0.smithi203.stdout:478: snap_create 2024-09-17T15:31:08.851 INFO:tasks.rados.rados.0.smithi203.stdout:476: finishing rollback tid 0 to smithi20328086-30 2024-09-17T15:31:08.853 INFO:tasks.rados.rados.0.smithi203.stdout:474: expect (ObjNum 167 snap 42 seq_num 167) 2024-09-17T15:31:08.853 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:08.941 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:08.985 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:476: finishing rollback tid 1 to smithi20328086-30 2024-09-17T15:31:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:476: finishing rollback tid 2 to smithi20328086-30 2024-09-17T15:31:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 67 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:31:09.843 INFO:tasks.rados.rados.0.smithi203.stdout:474: done (3 left) 2024-09-17T15:31:09.843 INFO:tasks.rados.rados.0.smithi203.stdout:476: done (2 left) 2024-09-17T15:31:09.843 INFO:tasks.rados.rados.0.smithi203.stdout:477: done (1 left) 2024-09-17T15:31:09.843 INFO:tasks.rados.rados.0.smithi203.stdout:478: done (0 left) 2024-09-17T15:31:09.843 INFO:tasks.rados.rados.0.smithi203.stdout:479: snap_remove snap 49 2024-09-17T15:31:10.847 INFO:tasks.rados.rados.0.smithi203.stdout:479: done (0 left) 2024-09-17T15:31:10.847 INFO:tasks.rados.rados.0.smithi203.stdout:480: snap_remove snap 48 2024-09-17T15:31:10.963 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:31:10.963 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:31:10.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:31:10.966 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:31:10.966 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:31:10.966 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:31:10.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:31:25.967 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:31:25.967 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:31:25.967 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:31:25.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:26.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: get_monmap_and_config 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: monmap: 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:26.097183+0000 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:26.097183+0000 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:26.097 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:26.098 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:26.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 Processor -- start 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 -- start start 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: init 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea280a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 0x7fea280a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-b con 0x7fea280a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea2813d2f0 0x7fea2813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-a con 0x7fea2813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea280a6530 con 0x7fea280a50e0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fea2809c150 con 0x7fea2813d2f0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fea2809c430 con 0x7fea280a54c0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2e63c640 10 monclient(hunting): authenticate will time out at 1219.062526s 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea2813d2f0 0x7fea2813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 0x7fea280a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea2813d2f0 0x7fea2813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48258/0 (socket says 172.21.15.86:48258) 2024-09-17T15:31:26.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 -- 172.21.15.86:0/3580516140 learned_addr learned my addr 172.21.15.86:0/3580516140 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): get_auth_request con 0x7fea2813d2f0 auth_method 0 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): get_auth_request con 0x7fea280a54c0 auth_method 0 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient(hunting): handle_auth_done global_id 4629 payload 274 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 -- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 msgr2=0x7fea280a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 0x7fea280a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 -- 172.21.15.86:0/3580516140 >> v1:172.21.15.203:6789/0 conn(0x7fea280a50e0 legacy=0x7fea2809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 monclient: found mon.noname-a 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea2cbd9640 1 -- 172.21.15.86:0/3580516140 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea280a6940 con 0x7fea2813d2f0 2024-09-17T15:31:26.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.093+0000 7fea277fe640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 0x7fea280a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2cbd9640 1 -- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea2813d2f0 msgr2=0x7fea2813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2cbd9640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea2813d2f0 0x7fea2813f6e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fea20009e50 tx=0x7fea2001c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4745f6d96957c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 1 -- 172.21.15.86:0/3580516140 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea200a5750 con 0x7fea2813d2f0 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: dump: 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 1 -- 172.21.15.86:0/3580516140 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea200a58f0 con 0x7fea2813d2f0 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 5 monclient: authenticate success, global_id 4629 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:26.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 1 -- 172.21.15.86:0/3580516140 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea200a5cc0 con 0x7fea2813d2f0 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 4 set_mon_vals no callback set 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea26ffd640 10 monclient: dump: 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.110 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 10 monclient: shutdown 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 1 -- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea2813d2f0 msgr2=0x7fea2813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.097+0000 7fea2e63c640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea2813d2f0 0x7fea2813f6e0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fea20009e50 tx=0x7fea2001c8f0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- 172.21.15.86:0/3580516140 shutdown_connections 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea2813d2f0 0x7fea2813f6e0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 --2- 172.21.15.86:0/3580516140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a54c0 0x7fea280a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- 172.21.15.86:0/3580516140 >> 172.21.15.86:0/3580516140 conn(0x7fea280aaae0 msgr2=0x7fea280aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- 172.21.15.86:0/3580516140 shutdown_connections 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- 172.21.15.86:0/3580516140 wait complete. 2024-09-17T15:31:26.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient: monmap: 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 Processor -- start 2024-09-17T15:31:26.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- start start 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient: init 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 0x7fea281cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-c con 0x7fea280a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-a con 0x7fea280a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 0x7fea280a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): picked mon.noname-b con 0x7fea2813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea28142800 con 0x7fea280a54c0 2024-09-17T15:31:26.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea28142680 con 0x7fea2813d2f0 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fea28142980 con 0x7fea280a50e0 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2e63c640 10 monclient(hunting): authenticate will time out at 1219.069544s 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48260/0 (socket says 172.21.15.86:48260) 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 -- 172.21.15.86:0/1072735833 learned_addr learned my addr 172.21.15.86:0/1072735833 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): get_auth_request con 0x7fea280a54c0 auth_method 0 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 0x7fea281cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): get_auth_request con 0x7fea280a50e0 auth_method 0 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2cbd9640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 0x7fea280a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient(hunting): handle_auth_done global_id 4632 payload 824 2024-09-17T15:31:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 -- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 msgr2=0x7fea281cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 0x7fea281cfc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 -- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 msgr2=0x7fea280a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 0x7fea280a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 monclient: found mon.noname-a 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea277fe640 1 -- 172.21.15.86:0/1072735833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea280a4740 con 0x7fea280a54c0 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2cbd9640 10 monclient: get_auth_request con 0x7fea2813d2f0 auth_method 0 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea27fff640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 0x7fea281cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.101+0000 7fea2cbd9640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 0x7fea280a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea277fe640 1 -- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 msgr2=0x7fea281381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea277fe640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea1800ebe0 tx=0x7fea1800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18017070 con 0x7fea280a54c0 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: dump: 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.117 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 5 monclient: authenticate success, global_id 4632 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea1800cde0 con 0x7fea280a54c0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _renew_subs 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea280a18d0 con 0x7fea280a54c0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea18013690 con 0x7fea280a54c0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:26.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 10 monclient: dump: 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea267fc640 4 set_mon_vals no callback set 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _renew_subs 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea281222a0 con 0x7fea280a54c0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea280a58d0 con 0x7fea280a54c0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fea1801a070 con 0x7fea280a54c0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 0x7fea0803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 4783+0+0 (secure 0 0 0) 0x7fea1804ff60 con 0x7fea280a54c0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea27fff640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 0x7fea0803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.105+0000 7fea27fff640 10 monclient: get_auth_request con 0x7fea0803bb60 auth_method 0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.109+0000 7fea27fff640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 0x7fea0803e020 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea280a4c70 tx=0x7fea10007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.109+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fea18013d50 con 0x7fea280a54c0 2024-09-17T15:31:26.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.109+0000 7fea24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.109+0000 7fea24ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:26.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:26.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:26.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea2813f6e0 con 0x7fea280a54c0 2024-09-17T15:31:26.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea24ff9640 1 -- 172.21.15.86:0/1072735833 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fea18023d00 con 0x7fea280a54c0 2024-09-17T15:31:26.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:26.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea24ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:26.349 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:26.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 msgr2=0x7fea0803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 0x7fea0803e020 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fea280a4c70 tx=0x7fea10007e40 comp rx=0 tx=0).stop 2024-09-17T15:31:26.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 10 monclient: shutdown 2024-09-17T15:31:26.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 msgr2=0x7fea281381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fea1800ebe0 tx=0x7fea1800c520 comp rx=0 tx=0).stop 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 shutdown_connections 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea0803bb60 0x7fea0803e020 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea2813d2f0 0x7fea280a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea280a54c0 0x7fea281381b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 --2- 172.21.15.86:0/1072735833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea280a50e0 0x7fea281cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 >> 172.21.15.86:0/1072735833 conn(0x7fea280aaae0 msgr2=0x7fea280a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 shutdown_connections 2024-09-17T15:31:26.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.345+0000 7fea2e63c640 1 -- 172.21.15.86:0/1072735833 wait complete. 2024-09-17T15:31:26.367 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:31:26.367 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:31:26.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:31:26.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: get_monmap_and_config 2024-09-17T15:31:26.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: monmap: 2024-09-17T15:31:26.477 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:26.478250+0000 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:26.478250+0000 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:26.478 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:26.479 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:26.479 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:26.479 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:26.479 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 Processor -- start 2024-09-17T15:31:26.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 -- start start 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: init 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-b con 0x7f17380a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17380a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 0x7f173813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-a con 0x7f173813d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f173809d430 con 0x7f17380a5350 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f173809c7f0 con 0x7f173813d180 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f17380a68d0 con 0x7f17380a4f70 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173e63a640 10 monclient(hunting): authenticate will time out at 1219.443706s 2024-09-17T15:31:26.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173cbd7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 0x7f173813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49568/0 (socket says 172.21.15.86:49568) 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 1 -- 172.21.15.86:0/3718000737 learned_addr learned my addr 172.21.15.86:0/3718000737 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173cbd7640 10 monclient(hunting): get_auth_request con 0x7f173813d180 auth_method 0 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173cbd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173cbd7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f173cbd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 10 monclient(hunting): get_auth_request con 0x7f17380a4f70 auth_method 0 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.473+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f173cbd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f173cbd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f173cbd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 10 monclient(hunting): handle_auth_done global_id 44105 payload 274 2024-09-17T15:31:26.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 1 -- 172.21.15.86:0/3718000737 >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 msgr2=0x7f173813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 1 --2- 172.21.15.86:0/3718000737 >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 0x7f173813f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 1 -- 172.21.15.86:0/3718000737 >> v1:172.21.15.203:6789/0 conn(0x7f17380a5350 legacy=0x7f17380a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 1 monclient: found mon.noname-b 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f1737fff640 1 -- 172.21.15.86:0/3718000737 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17380a64a0 con 0x7f17380a4f70 2024-09-17T15:31:26.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.477+0000 7f173cbd7640 1 --2- 172.21.15.86:0/3718000737 >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 0x7f173813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 -- 172.21.15.86:0/3718000737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 msgr2=0x7f173809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 --2- 172.21.15.86:0/3718000737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f173809e420 tx=0x7f173001c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84a4332c5da8a251 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 1 -- 172.21.15.86:0/3718000737 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300a56e0 con 0x7f17380a4f70 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: dump: 2024-09-17T15:31:26.495 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 5 monclient: authenticate success, global_id 44105 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 1 -- 172.21.15.86:0/3718000737 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17300a5880 con 0x7f17380a4f70 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 1 -- 172.21.15.86:0/3718000737 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300a5c50 con 0x7f17380a4f70 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1736ffd640 10 monclient: dump: 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 4 set_mon_vals no callback set 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: shutdown 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- 172.21.15.86:0/3718000737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 msgr2=0x7f173809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- 172.21.15.86:0/3718000737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f173809e420 tx=0x7f173001c8a0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- 172.21.15.86:0/3718000737 shutdown_connections 2024-09-17T15:31:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- 172.21.15.86:0/3718000737 >> v2:172.21.15.86:3300/0 conn(0x7f173813d180 0x7f173813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- 172.21.15.86:0/3718000737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f173809cef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- 172.21.15.86:0/3718000737 >> 172.21.15.86:0/3718000737 conn(0x7f17380a7240 msgr2=0x7f17380a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- 172.21.15.86:0/3718000737 shutdown_connections 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- 172.21.15.86:0/3718000737 wait complete. 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: monmap: 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:26.499 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 Processor -- start 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- start start 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: init 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f17380a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-c con 0x7f17380a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17380a5350 0x7f17380a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-a con 0x7f17380a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:26.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f173813d180 0x7f17380a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): picked mon.noname-b con 0x7f173813d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17381424e0 con 0x7f17380a5350 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f173806f950 con 0x7f173813d180 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1738142660 con 0x7f17380a4f70 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f173e63a640 10 monclient(hunting): authenticate will time out at 1219.450570s 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f17380a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17380a5350 0x7f17380a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f17380a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49582/0 (socket says 172.21.15.86:49582) 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 -- 172.21.15.86:0/465639279 learned_addr learned my addr 172.21.15.86:0/465639279 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): get_auth_request con 0x7f17380a4f70 auth_method 0 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): get_auth_request con 0x7f17380a5350 auth_method 0 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient(hunting): handle_auth_done global_id 44108 payload 824 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f173813d180 msgr2=0x7f17380a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f173813d180 0x7f17380a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17380a5350 msgr2=0x7f17380a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17380a5350 0x7f17380a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 monclient: found mon.noname-c 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f1737fff640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f173809f590 con 0x7f17380a4f70 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.481+0000 7f17377fe640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17380a5350 0x7f17380a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1737fff640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 msgr2=0x7f17380a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1737fff640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f17380a1e70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17300af040 tx=0x7f1730019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300bc020 con 0x7f17380a4f70 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: dump: 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.504 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17300b8880 con 0x7f17380a4f70 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 5 monclient: authenticate success, global_id 44108 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17300a56e0 con 0x7f17380a4f70 2024-09-17T15:31:26.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 monclient: _renew_subs 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17380a1760 con 0x7f17380a4f70 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1735ffb640 4 set_mon_vals no callback set 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 10 monclient: dump: 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.506 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 monclient: _renew_subs 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17380a3930 con 0x7f17380a4f70 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 0x7f17381cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f173809eea0 con 0x7f17381cfc10 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 0x7f17381cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient: get_auth_request con 0x7f17381cfc10 auth_method 0 2024-09-17T15:31:26.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f17300bf020 con 0x7f17380a4f70 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 10 monclient(hunting): handle_auth_done global_id 4641 payload 824 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f17377fe640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f173809cef0 con 0x7f17381cfc10 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f172403bb90 0x7f172403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(120..120 src has 1..120) ==== 4783+0+0 (secure 0 0 0) 0x7f17300eac00 con 0x7f17380a4f70 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173cbd7640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f172403bb90 0x7f172403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.485+0000 7f173cbd7640 10 monclient: get_auth_request con 0x7f172403bb90 auth_method 0 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f17377fe640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 msgr2=0x7f17381cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f17377fe640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 0x7f17381cfff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c00d860 con 0x7f17381cfc10 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 10 monclient: dump: 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.509 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f172c018070 con 0x7f17381cfc10 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 msgr2=0x7f17381cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 0x7f17381cfff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c00ff70 comp rx=0 tx=0).stop 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 msgr2=0x7f17381cfff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f1734ff9640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381cfc10 0x7f17381cfff0 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c00ff70 comp rx=0 tx=0).stop 2024-09-17T15:31:26.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.489+0000 7f173cbd7640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f172403bb90 0x7f172403e050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1728006e90 tx=0x7f1728006de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f173e63a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f173e63a640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 0x7f1738139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f173e63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f17380a5ff0 con 0x7f17381d0530 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 0x7f1738139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient: get_auth_request con 0x7f17381d0530 auth_method 0 2024-09-17T15:31:26.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 10 monclient(hunting): handle_auth_done global_id 4644 payload 824 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.493+0000 7f17377fe640 1 -- 172.21.15.86:0/465639279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1738069920 con 0x7f17381d0530 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f17377fe640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 msgr2=0x7f1738139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f17377fe640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 0x7f1738139160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c001030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f172c003600 con 0x7f17381d0530 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f1734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f1734ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.497+0000 7f1734ff9640 10 monclient: dump: 2024-09-17T15:31:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f172c003600 con 0x7f17381d0530 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:26.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 msgr2=0x7f1738139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 0x7f1738139160 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c001030 comp rx=0 tx=0).stop 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 msgr2=0x7f1738139160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f1734ff9640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17381d0530 0x7f1738139160 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f173809d9c0 tx=0x7f172c001030 comp rx=0 tx=0).stop 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:26.515 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 259141, 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.516 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.517 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:26.518 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:26.519 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.520 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:26.523 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:26.524 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.525 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.526 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.527 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.529 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:26.529 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f172403bb90 msgr2=0x7f172403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f172403bb90 0x7f172403e050 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1728006e90 tx=0x7f1728006de0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 10 monclient: shutdown 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 msgr2=0x7f17380a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 --2- 172.21.15.86:0/465639279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17380a4f70 0x7f17380a1e70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17300af040 tx=0x7f1730019640 comp rx=0 tx=0).stop 2024-09-17T15:31:26.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173cbd7640 1 -- 172.21.15.86:0/465639279 reap_dead start 2024-09-17T15:31:26.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173cbd7640 1 -- 172.21.15.86:0/465639279 reap_dead start 2024-09-17T15:31:26.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 shutdown_connections 2024-09-17T15:31:26.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 >> 172.21.15.86:0/465639279 conn(0x7f17380a7240 msgr2=0x7f173813c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 shutdown_connections 2024-09-17T15:31:26.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.501+0000 7f173e63a640 1 -- 172.21.15.86:0/465639279 wait complete. 2024-09-17T15:31:26.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:26.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:31:26.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:31:26.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.645+0000 7f458ff79640 10 monclient: get_monmap_and_config 2024-09-17T15:31:26.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.645+0000 7f458ff79640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.645+0000 7f458ff79640 10 monclient: monmap: 2024-09-17T15:31:26.651 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.651 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:26.651969+0000 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:26.651969+0000 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:26.653 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:26.653 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 Processor -- start 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 -- start start 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient: init 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 0x7f45880a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-c con 0x7f45880a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f4588135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-b con 0x7f45880a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4588135f90 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45880a2130 con 0x7f4588135f90 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f45880a2410 con 0x7f45880a0c70 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f45880a26f0 con 0x7f45880a1590 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458ff79640 10 monclient(hunting): authenticate will time out at 1219.617502s 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458e516640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4588135f90 0x7f458813cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43824/0 (socket says 172.21.15.86:43824) 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458e516640 1 -- 172.21.15.86:0/189148730 learned_addr learned my addr 172.21.15.86:0/189148730 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 --2- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f4588135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): get_auth_request con 0x7f45880a1590 auth_method 0 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 1 --2- 172.21.15.86:0/189148730 >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 0x7f45880a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 10 monclient(hunting): get_auth_request con 0x7f45880a0c70 auth_method 0 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient(hunting): handle_auth_done global_id 44111 payload 274 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 -- 172.21.15.86:0/189148730 >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 msgr2=0x7f45880a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 --2- 172.21.15.86:0/189148730 >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 0x7f45880a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 -- 172.21.15.86:0/189148730 >> v1:172.21.15.86:6789/0 conn(0x7f4588135f90 legacy=0x7f458813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 monclient: found mon.noname-b 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458d514640 1 -- 172.21.15.86:0/189148730 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458809f9f0 con 0x7f45880a1590 2024-09-17T15:31:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.649+0000 7f458dd15640 1 --2- 172.21.15.86:0/189148730 >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 0x7f45880a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:31:26.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458d514640 1 -- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 msgr2=0x7f4588135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458d514640 1 --2- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f4588135a50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45840097c0 tx=0x7f458401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f095f4ffc7bb9aeb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 1 -- 172.21.15.86:0/189148730 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45840b0070 con 0x7f45880a1590 2024-09-17T15:31:26.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: dump: 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.663 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 5 monclient: authenticate success, global_id 44111 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 1 -- 172.21.15.86:0/189148730 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45840ab600 con 0x7f45880a1590 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 1 -- 172.21.15.86:0/189148730 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45840ab9d0 con 0x7f45880a1590 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:26.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 4 set_mon_vals no callback set 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458cd13640 10 monclient: dump: 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.665 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 10 monclient: shutdown 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 -- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 msgr2=0x7f4588135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 --2- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f4588135a50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45840097c0 tx=0x7f458401c360 comp rx=0 tx=0).stop 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 -- 172.21.15.86:0/189148730 shutdown_connections 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 --2- 172.21.15.86:0/189148730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f4588135a50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 --2- 172.21.15.86:0/189148730 >> v2:172.21.15.203:3300/0 conn(0x7f45880a0c70 0x7f45880a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 -- 172.21.15.86:0/189148730 >> 172.21.15.86:0/189148730 conn(0x7f45880a2f40 msgr2=0x7f45880a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 -- 172.21.15.86:0/189148730 shutdown_connections 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.653+0000 7f458ff79640 1 -- 172.21.15.86:0/189148730 wait complete. 2024-09-17T15:31:26.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: monmap: 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 Processor -- start 2024-09-17T15:31:26.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 -- start start 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: init 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45880a0c70 0x7f458813d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-b con 0x7f45880a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f45881cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-c con 0x7f45880a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): picked mon.noname-a con 0x7f4588135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f458809ce60 con 0x7f4588135f90 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f458809cce0 con 0x7f45880a0c70 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f458809ff50 con 0x7f45880a1590 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient(hunting): authenticate will time out at 1219.625035s 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f45881cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48314/0 (socket says 172.21.15.86:48314) 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 learned_addr learned my addr 172.21.15.86:0/2459978756 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): get_auth_request con 0x7f4588135f90 auth_method 0 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): get_auth_request con 0x7f45880a1590 auth_method 0 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458d514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient(hunting): handle_auth_done global_id 4647 payload 824 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458dd15640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45880a0c70 0x7f458813d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 msgr2=0x7f45881cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45880a1590 0x7f45881cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45880a0c70 msgr2=0x7f458813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45880a0c70 0x7f458813d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 monclient: found mon.noname-a 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458809d460 con 0x7f4588135f90 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 msgr2=0x7f45881ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458e516640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f458000ca40 tx=0x7f458000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458000ade0 con 0x7f4588135f90 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: dump: 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4580012400 con 0x7f4588135f90 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 5 monclient: authenticate success, global_id 4647 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: _renew_subs 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458813c330 con 0x7f4588135f90 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800127f0 con 0x7f4588135f90 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457a7fc640 10 monclient: dump: 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.677 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.657+0000 7f457bfff640 4 set_mon_vals no callback set 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 10 monclient: _renew_subs 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458813d890 con 0x7f4588135f90 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 0x7f4588130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f458813c850 con 0x7f45881ce7c0 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4580012a10 con 0x7f4588135f90 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 0x7f4588130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient: get_auth_request con 0x7f45881ce7c0 auth_method 0 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f456c03bb60 0x7f456c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(120..120 src has 1..120) ==== 4783+0+0 (secure 0 0 0) 0x7f4580050550 con 0x7f4588135f90 2024-09-17T15:31:26.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 10 monclient(hunting): handle_auth_done global_id 44117 payload 824 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45881ceba0 con 0x7f45881ce7c0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458d514640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f456c03bb60 0x7f456c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458d514640 10 monclient: get_auth_request con 0x7f456c03bb60 auth_method 0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 msgr2=0x7f4588130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f458dd15640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 0x7f4588130a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c00f900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c018020 con 0x7f45881ce7c0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 10 monclient: dump: 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f457c014b80 con 0x7f45881ce7c0 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 msgr2=0x7f4588130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 0x7f4588130a00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c00f900 comp rx=0 tx=0).stop 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 msgr2=0x7f4588130a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.661+0000 7f457a7fc640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45881ce7c0 0x7f4588130a00 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c00f900 comp rx=0 tx=0).stop 2024-09-17T15:31:26.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.665+0000 7f458d514640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f456c03bb60 0x7f456c03e020 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f45880a40b0 tx=0x7f4584008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458ff79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458ff79640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 0x7f4588131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458ff79640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4588131890 con 0x7f4588130f40 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 0x7f4588131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient: get_auth_request con 0x7f4588130f40 auth_method 0 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 10 monclient(hunting): handle_auth_done global_id 44120 payload 824 2024-09-17T15:31:26.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.669+0000 7f458dd15640 1 -- 172.21.15.86:0/2459978756 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45880a1050 con 0x7f4588130f40 2024-09-17T15:31:26.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458dd15640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 msgr2=0x7f4588131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458dd15640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 0x7f4588131350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c018070 con 0x7f4588130f40 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 10 monclient: dump: 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f457c004220 con 0x7f4588130f40 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 msgr2=0x7f4588131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 0x7f4588131350 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 msgr2=0x7f4588131350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f457a7fc640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4588130f40 0x7f4588131350 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f457c00f440 tx=0x7f457c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.685 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 15627, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:31:26.686 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.687 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.688 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.689 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.690 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.691 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:26.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.696 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.697 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:26.698 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f456c03bb60 msgr2=0x7f456c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f456c03bb60 0x7f456c03e020 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f45880a40b0 tx=0x7f4584008040 comp rx=0 tx=0).stop 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 10 monclient: shutdown 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 msgr2=0x7f45881ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f458000ca40 tx=0x7f458000cf00 comp rx=0 tx=0).stop 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458e516640 1 -- 172.21.15.86:0/2459978756 reap_dead start 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 shutdown_connections 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 --2- 172.21.15.86:0/2459978756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4588135f90 0x7f45881ce280 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 >> 172.21.15.86:0/2459978756 conn(0x7f45880a2f40 msgr2=0x7f4588072930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 shutdown_connections 2024-09-17T15:31:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.673+0000 7f458ff79640 1 -- 172.21.15.86:0/2459978756 wait complete. 2024-09-17T15:31:26.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:26.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:31:26.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:31:26.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: get_monmap_and_config 2024-09-17T15:31:26.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: monmap: 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:26.816153+0000 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:26.816153+0000 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.816 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:26.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:26.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:26.817 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:26.817 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:26.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 Processor -- start 2024-09-17T15:31:26.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 -- start start 2024-09-17T15:31:26.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: init 2024-09-17T15:31:26.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-b con 0x7fedd80a4900 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fedd80a1200 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedd81361b0 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd8069ae0 con 0x7fedd81361b0 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedd8132c30 con 0x7fedd80a1200 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fedd80a1640 con 0x7fedd80a4900 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedde7f2640 10 monclient(hunting): authenticate will time out at 1219.781856s 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49614/0 (socket says 172.21.15.86:49614) 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 -- 172.21.15.86:0/867125600 learned_addr learned my addr 172.21.15.86:0/867125600 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): get_auth_request con 0x7fedd80a4900 auth_method 0 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient(hunting): handle_auth_done global_id 44123 payload 274 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 -- 172.21.15.86:0/867125600 >> v1:172.21.15.203:6789/0 conn(0x7fedd80a1200 legacy=0x7fedd8135aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 -- 172.21.15.86:0/867125600 >> v1:172.21.15.86:6789/0 conn(0x7fedd81361b0 legacy=0x7fedd813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 monclient: found mon.noname-b 2024-09-17T15:31:26.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.813+0000 7fedd7fff640 1 -- 172.21.15.86:0/867125600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd809dbe0 con 0x7fedd80a4900 2024-09-17T15:31:26.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd7fff640 1 -- 172.21.15.86:0/867125600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 msgr2=0x7fedd80a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd7fff640 1 --2- 172.21.15.86:0/867125600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fedd8070a50 tx=0x7fedd001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f612081b5b6038fe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 1 -- 172.21.15.86:0/867125600 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00a64e0 con 0x7fedd80a4900 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: dump: 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.823 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 5 monclient: authenticate success, global_id 44123 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 1 -- 172.21.15.86:0/867125600 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd00a6680 con 0x7fedd80a4900 2024-09-17T15:31:26.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 1 -- 172.21.15.86:0/867125600 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00a6a50 con 0x7fedd80a4900 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedd6ffd640 10 monclient: dump: 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.825 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 4 set_mon_vals no callback set 2024-09-17T15:31:26.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 10 monclient: shutdown 2024-09-17T15:31:26.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 1 -- 172.21.15.86:0/867125600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 msgr2=0x7fedd80a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.817+0000 7fedde7f2640 1 --2- 172.21.15.86:0/867125600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fedd8070a50 tx=0x7fedd001c8f0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- 172.21.15.86:0/867125600 shutdown_connections 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 --2- 172.21.15.86:0/867125600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd80a0cc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- 172.21.15.86:0/867125600 >> 172.21.15.86:0/867125600 conn(0x7fedd80a3e10 msgr2=0x7fedd80a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- 172.21.15.86:0/867125600 shutdown_connections 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- 172.21.15.86:0/867125600 wait complete. 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient: monmap: 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:26.832 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 Processor -- start 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- start start 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient: init 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 0x7fedd809a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-a con 0x7fedd80a1200 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd809aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-c con 0x7fedd80a4900 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd81361b0 0x7fedd8070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): picked mon.noname-b con 0x7fedd81361b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80a8240 con 0x7fedd80a1200 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fedd80a4ce0 con 0x7fedd81361b0 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fedd8141f30 con 0x7fedd80a4900 2024-09-17T15:31:26.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedde7f2640 10 monclient(hunting): authenticate will time out at 1219.790006s 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 0x7fedd809a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd809aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 0x7fedd809a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48326/0 (socket says 172.21.15.86:48326) 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd809aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49624/0 (socket says 172.21.15.86:49624) 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 1 -- 172.21.15.86:0/128640641 learned_addr learned my addr 172.21.15.86:0/128640641 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): get_auth_request con 0x7fedd80a1200 auth_method 0 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): get_auth_request con 0x7fedd80a4900 auth_method 0 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient(hunting): handle_auth_done global_id 44126 payload 824 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd81361b0 msgr2=0x7fedd8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd81361b0 0x7fedd8070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 msgr2=0x7fedd809a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 0x7fedd809a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 monclient: found mon.noname-c 2024-09-17T15:31:26.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd77fe640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd812fd00 con 0x7fedd80a4900 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.821+0000 7fedd7fff640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedd80a1200 0x7fedd809a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd77fe640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 msgr2=0x7fedd809aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd77fe640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd809aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00b3e0 tx=0x7fedcc00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc015020 con 0x7fedd80a4900 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: dump: 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.837 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 5 monclient: authenticate success, global_id 44126 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedcc011a40 con 0x7fedd80a4900 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 monclient: _renew_subs 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd8073c50 con 0x7fedd80a4900 2024-09-17T15:31:26.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedcc011e30 con 0x7fedd80a4900 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 10 monclient: dump: 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 monclient: _renew_subs 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd813f090 con 0x7fedd80a4900 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd5ffb640 4 set_mon_vals no callback set 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 0x7fedd8071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fedd809e560 con 0x7fedd80735f0 2024-09-17T15:31:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 0x7fedd8071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fedcc018020 con 0x7fedd80a4900 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient: get_auth_request con 0x7fedd80735f0 auth_method 0 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fedc403bb60 0x7fedc403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(120..120 src has 1..120) ==== 4783+0+0 (secure 0 0 0) 0x7fedcc04fce0 con 0x7fedd80a4900 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 10 monclient(hunting): handle_auth_done global_id 14272 payload 824 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7fedd7fff640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedd80a0cc0 con 0x7fedd80735f0 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7feddcd8f640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fedc403bb60 0x7fedc403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.825+0000 7feddcd8f640 10 monclient: get_auth_request con 0x7fedc403bb60 auth_method 0 2024-09-17T15:31:26.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd7fff640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 msgr2=0x7fedd8071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd7fff640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 0x7fedd8071500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fedd0000f90 tx=0x7fedd001b130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7feddcd8f640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fedc403bb60 0x7fedc403e020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fedc8007fd0 tx=0x7fedc80079c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00a6de0 con 0x7fedd80735f0 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 10 monclient: dump: 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.844 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fedd00a6de0 con 0x7fedd80735f0 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 msgr2=0x7fedd8071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 0x7fedd8071500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fedd0000f90 tx=0x7fedd001b130 comp rx=0 tx=0).stop 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 msgr2=0x7fedd8071500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.829+0000 7fedd4ff9640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd80735f0 0x7fedd8071500 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fedd0000f90 tx=0x7fedd001b130 comp rx=0 tx=0).stop 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedde7f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedde7f2640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 0x7fedd8131520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedde7f2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fedd8071e50 con 0x7fedd8071a40 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 0x7fedd8131520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient: get_auth_request con 0x7fedd8071a40 auth_method 0 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 10 monclient(hunting): handle_auth_done global_id 14275 payload 824 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.837+0000 7fedd7fff640 1 -- 172.21.15.86:0/128640641 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fedd806e5b0 con 0x7fedd8071a40 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd7fff640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 msgr2=0x7fedd8131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd7fff640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 0x7fedd8131520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fedd809e6e0 tx=0x7fedd0019690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00c8070 con 0x7fedd8071a40 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 10 monclient: dump: 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.848 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fedd00c4970 con 0x7fedd8071a40 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 msgr2=0x7fedd8131520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 0x7fedd8131520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fedd809e6e0 tx=0x7fedd0019690 comp rx=0 tx=0).stop 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 msgr2=0x7fedd8131520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:26.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedd4ff9640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fedd8071a40 0x7fedd8131520 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fedd809e6e0 tx=0x7fedd0019690 comp rx=0 tx=0).stop 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 195445, 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:26.850 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.851 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.852 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.853 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:26.854 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:26.855 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.856 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.857 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:26.858 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.859 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.860 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:26.861 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.862 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:26.863 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fedc403bb60 msgr2=0x7fedc403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fedc403bb60 0x7fedc403e020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fedc8007fd0 tx=0x7fedc80079c0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 10 monclient: shutdown 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 msgr2=0x7fedd809aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 --2- 172.21.15.86:0/128640641 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedd80a4900 0x7fedd809aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fedcc00b3e0 tx=0x7fedcc00b8a0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7feddcd8f640 1 -- 172.21.15.86:0/128640641 reap_dead start 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7feddcd8f640 1 -- 172.21.15.86:0/128640641 reap_dead start 2024-09-17T15:31:26.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 shutdown_connections 2024-09-17T15:31:26.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 >> 172.21.15.86:0/128640641 conn(0x7fedd80a3e10 msgr2=0x7fedd80a1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 shutdown_connections 2024-09-17T15:31:26.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.841+0000 7fedde7f2640 1 -- 172.21.15.86:0/128640641 wait complete. 2024-09-17T15:31:26.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:26.866 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:31:26.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:31:26.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.973+0000 7f26d9bb7640 10 monclient: get_monmap_and_config 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.973+0000 7f26d9bb7640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.973+0000 7f26d9bb7640 10 monclient: monmap: 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:26.979441+0000 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:26.979441+0000 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:26.979 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:26.980 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 Processor -- start 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 -- start start 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient: init 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 0x7f26d406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-c con 0x7f26d40689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26d4068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:26.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-b con 0x7f26d413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d4074470 con 0x7f26d4068db0 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f26d409c900 con 0x7f26d40689d0 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f26d409cb90 con 0x7f26d413aff0 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d9bb7640 10 monclient(hunting): authenticate will time out at 1219.944873s 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49638/0 (socket says 172.21.15.86:49638) 2024-09-17T15:31:26.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 -- 172.21.15.86:0/3930549229 learned_addr learned my addr 172.21.15.86:0/3930549229 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): get_auth_request con 0x7f26d413aff0 auth_method 0 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d37fe640 1 --2- 172.21.15.86:0/3930549229 >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 0x7f26d406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient(hunting): handle_auth_done global_id 44129 payload 274 2024-09-17T15:31:26.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 -- 172.21.15.86:0/3930549229 >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 msgr2=0x7f26d406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 --2- 172.21.15.86:0/3930549229 >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 0x7f26d406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 -- 172.21.15.86:0/3930549229 >> v1:172.21.15.203:6789/0 conn(0x7f26d4068db0 legacy=0x7f26d4069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 monclient: found mon.noname-b 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d3fff640 1 -- 172.21.15.86:0/3930549229 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d409ce70 con 0x7f26d413aff0 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d37fe640 10 monclient: get_auth_request con 0x7f26d40689d0 auth_method 0 2024-09-17T15:31:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.977+0000 7f26d37fe640 1 --2- 172.21.15.86:0/3930549229 >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 0x7f26d406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d3fff640 1 -- 172.21.15.86:0/3930549229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d3fff640 1 --2- 172.21.15.86:0/3930549229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009870 tx=0x7f26cc01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c47decad3c895ee server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 1 -- 172.21.15.86:0/3930549229 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc0b1070 con 0x7f26d413aff0 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: dump: 2024-09-17T15:31:26.989 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: _finish_auth 0 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:26.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 1 -- 172.21.15.86:0/3930549229 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26cc0ac620 con 0x7f26d413aff0 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 5 monclient: authenticate success, global_id 44129 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 4 set_mon_vals no callback set 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 1 -- 172.21.15.86:0/3930549229 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d409c2d0 con 0x7f26d413aff0 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d27fc640 10 monclient: dump: 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.991 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: shutdown 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- 172.21.15.86:0/3930549229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/3930549229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f26cc009870 tx=0x7f26cc01c420 comp rx=0 tx=0).stop 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- 172.21.15.86:0/3930549229 shutdown_connections 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/3930549229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d413d3e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/3930549229 >> v2:172.21.15.203:3300/0 conn(0x7f26d40689d0 0x7f26d406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- 172.21.15.86:0/3930549229 >> 172.21.15.86:0/3930549229 conn(0x7f26d40a6440 msgr2=0x7f26d40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- 172.21.15.86:0/3930549229 shutdown_connections 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- 172.21.15.86:0/3930549229 wait complete. 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: build_initial_monmap 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: monmap: 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:26.993 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 Processor -- start 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- start start 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: init 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26d4131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-b con 0x7f26d40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26d4131c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-a con 0x7f26d4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient(hunting): picked mon.noname-c con 0x7f26d413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f26d4140670 con 0x7f26d4068db0 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.981+0000 7f26d9bb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f26d41404f0 con 0x7f26d40689d0 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f26d41407f0 con 0x7f26d413aff0 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient(hunting): authenticate will time out at 1219.952015s 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26d4131c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49652/0 (socket says 172.21.15.86:49652) 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 learned_addr learned my addr 172.21.15.86:0/1419780962 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:26.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): get_auth_request con 0x7f26d413aff0 auth_method 0 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26d4131c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48338/0 (socket says 172.21.15.86:48338) 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): get_auth_request con 0x7f26d4068db0 auth_method 0 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:26.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26d4131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 10 monclient(hunting): get_auth_request con 0x7f26d40689d0 auth_method 0 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_done global_id 44132 payload 824 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 msgr2=0x7f26d4131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26d4131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 msgr2=0x7f26d4131c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26d4131c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:31:26.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 monclient: found mon.noname-c 2024-09-17T15:31:26.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:26.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:26.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d4072fe0 con 0x7f26d413aff0 2024-09-17T15:31:26.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26d4131c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:27.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d406fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:27.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f26cc01c250 tx=0x7f26cc01ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:27.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc0b1040 con 0x7f26d413aff0 2024-09-17T15:31:27.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:27.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: dump: 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: _finish_auth 0 2024-09-17T15:31:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26cc0bc070 con 0x7f26d413aff0 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 5 monclient: authenticate success, global_id 44132 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient: _renew_subs 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d40732c0 con 0x7f26d413aff0 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc0b7690 con 0x7f26d413aff0 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: dump: 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:27.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient: _renew_subs 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d1ffb640 4 set_mon_vals no callback set 2024-09-17T15:31:27.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d40a3850 con 0x7f26d413aff0 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d406fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d406fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f26cc01c250 tx=0x7f26cc01ac60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f26cc01c250 tx=0x7f26cc01ac60 comp rx=0 tx=0).stop 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 msgr2=0x7f26d406fde0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26d413aff0 0x7f26d406fde0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26b4005880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): picked mon.b con 0x7f26d40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26b4008280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): picked mon.a con 0x7f26d4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26b4008c90 0x7f26b400b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): picked mon.c con 0x7f26b4008c90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f26b400bbd0 con 0x7f26d4068db0 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f26b400bd50 con 0x7f26d40689d0 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f26b400bed0 con 0x7f26b4008c90 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26cbfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26b4008280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): get_auth_request con 0x7f26d4068db0 auth_method 0 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26b4008c90 0x7f26b400b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): get_auth_request con 0x7f26b4008c90 auth_method 0 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d9bb7640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d37fe640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26b4005880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient(hunting): handle_auth_done global_id 44132 payload 293 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26b4008c90 msgr2=0x7f26b400b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26b4008c90 0x7f26b400b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 msgr2=0x7f26b4005880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26b4005880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 monclient: found mon.a 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f26b400c1f0 con 0x7f26d4068db0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.985+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26d413d140 con 0x7f26d4068db0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26bc00f3f0 con 0x7f26d4068db0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d37fe640 10 monclient: get_auth_request con 0x7f26d40689d0 auth_method 0 2024-09-17T15:31:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d37fe640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26b4005880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 msgr2=0x7f26b4008280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d2ffd640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26b4008280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00f620 tx=0x7f26cc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26bc003fd0 con 0x7f26d4068db0 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 10 monclient: dump: 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26bc007020 con 0x7f26d4068db0 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d17fa640 4 set_mon_vals no callback set 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f26bc0071d0 con 0x7f26d4068db0 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 0x7f26b4046370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 4 ==== osd_map(120..120 src has 1..120) ==== 4783+0+0 (secure 0 0 0) 0x7f26bc010040 con 0x7f26d4068db0 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d37fe640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 0x7f26b4046370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.989+0000 7f26d37fe640 10 monclient: get_auth_request con 0x7f26d413aff0 auth_method 0 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.993+0000 7f26d37fe640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 0x7f26b4046370 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f26c400f560 tx=0x7f26c4007680 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.993+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f26bc04c860 con 0x7f26d4068db0 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.993+0000 7f26cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.993+0000 7f26cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.997+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f26bc04cbe0 con 0x7f26d4068db0 2024-09-17T15:31:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:26.997+0000 7f26cbfff640 10 monclient: handle_mon_command_ack 1 not found 2024-09-17T15:31:27.028 INFO:tasks.rados.rados.0.smithi203.stdout:480: done (0 left) 2024-09-17T15:31:27.028 INFO:tasks.rados.rados.0.smithi203.stdout:481: read oid 21 snap -1 2024-09-17T15:31:27.028 INFO:tasks.rados.rados.0.smithi203.stdout:481: expect (ObjNum 183 snap 45 seq_num 183) 2024-09-17T15:31:27.028 INFO:tasks.rados.rados.0.smithi203.stdout:482: snap_create 2024-09-17T15:31:27.038 INFO:tasks.rados.rados.0.smithi203.stdout:481: expect (ObjNum 183 snap 45 seq_num 183) 2024-09-17T15:31:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.217+0000 7f26d9bb7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:31:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.217+0000 7f26d9bb7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:31:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.217+0000 7f26d9bb7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.217+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f26d406f840 con 0x7f26d4068db0 2024-09-17T15:31:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.221+0000 7f26cbfff640 1 -- 172.21.15.86:0/1419780962 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f26bc014680 con 0x7f26d4068db0 2024-09-17T15:31:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.221+0000 7f26cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:31:27.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.221+0000 7f26cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:27.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=3025376367,config=3541143443,health=1017346500,kv=3885551602,logm=776009319}) 2024-09-17T15:31:27.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=11,kv=10,logm=66} crc {auth=202132855,config=3541143443,health=2385462494,kv=2732825231,logm=2185974390}) 2024-09-17T15:31:27.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=10,logm=66} crc {auth=3025376367,config=3541143443,health=1017346500,kv=3885551602,logm=776009319}) 2024-09-17T15:31:27.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.221+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=10,kv=10,logm=67} crc {auth=202132855,config=3541143443,health=1281764271,kv=2898202873,logm=1351217624}) 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 msgr2=0x7f26b4046370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 0x7f26b4046370 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f26c400f560 tx=0x7f26c4007680 comp rx=0 tx=0).stop 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 10 monclient: shutdown 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 msgr2=0x7f26b4008280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26b4008280 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f26bc00f620 tx=0x7f26cc008040 comp rx=0 tx=0).stop 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 shutdown_connections 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f26d413aff0 0x7f26b4046370 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26b4008c90 0x7f26b400b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26d4068db0 0x7f26b4008280 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 --2- 172.21.15.86:0/1419780962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f26d40689d0 0x7f26b4005880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:27.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 >> 172.21.15.86:0/1419780962 conn(0x7f26d40a6440 msgr2=0x7f26d40a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:27.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 shutdown_connections 2024-09-17T15:31:27.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:27.225+0000 7f26d9bb7640 1 -- 172.21.15.86:0/1419780962 wait complete. 2024-09-17T15:31:27.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540971663}) 2024-09-17T15:31:27.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087170850}) 2024-09-17T15:31:27.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540971663}) 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=25844266}) 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231923642}) 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491156582}) 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231923642}) 2024-09-17T15:31:27.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.225+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984314241}) 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950311006}) 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148652346}) 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=950311006}) 2024-09-17T15:31:27.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516897790}) 2024-09-17T15:31:27.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882373284}) 2024-09-17T15:31:27.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582539483}) 2024-09-17T15:31:27.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882373284}) 2024-09-17T15:31:27.235 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.229+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4293822174}) 2024-09-17T15:31:27.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129476995}) 2024-09-17T15:31:27.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=522496497}) 2024-09-17T15:31:27.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129476995}) 2024-09-17T15:31:27.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849796677}) 2024-09-17T15:31:27.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775997266}) 2024-09-17T15:31:27.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135591214}) 2024-09-17T15:31:27.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3775997266}) 2024-09-17T15:31:27.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.233+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882176037}) 2024-09-17T15:31:27.240 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.240 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624530031}) 2024-09-17T15:31:27.240 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2836338341}) 2024-09-17T15:31:27.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624530031}) 2024-09-17T15:31:27.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=755144148}) 2024-09-17T15:31:27.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179737778}) 2024-09-17T15:31:27.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=237776661}) 2024-09-17T15:31:27.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179737778}) 2024-09-17T15:31:27.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.237+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105080740}) 2024-09-17T15:31:27.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=1} crc {logm=194985150,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1836730088,mon_config_key=2965230982,monmap=4191199434,nvmeofgw=773687688}) 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=12,monmap=3,nvmeofgw=2} crc {logm=3489065817,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=371823223,mon_config_key=399731099,monmap=4191199434,nvmeofgw=1596038039}) 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=1} crc {logm=194985150,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1836730088,mon_config_key=2965230982,monmap=4191199434,nvmeofgw=773687688}) 2024-09-17T15:31:27.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=13} crc {logm=2596466217,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=81733170,mon_config_key=2445621393}) 2024-09-17T15:31:27.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.248 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=70} crc {nvmeofgw=1228700967,osd_metadata=4043941930,osd_pg_creating=386741304,osd_snap=1541085543,osdmap=1230916905}) 2024-09-17T15:31:27.248 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=69} crc {nvmeofgw=3140514009,osd_metadata=4043941930,osd_pg_creating=386741304,osd_snap=56734184,osdmap=1278226221}) 2024-09-17T15:31:27.248 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.248 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=70} crc {nvmeofgw=1228700967,osd_metadata=4043941930,osd_pg_creating=386741304,osd_snap=1541085543,osdmap=1230916905}) 2024-09-17T15:31:27.248 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.241+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=70} crc {nvmeofgw=3140514009,osd_metadata=4043941930,osd_pg_creating=386741304,osd_snap=1300645193,osdmap=2521168872}) 2024-09-17T15:31:27.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2827439195}) 2024-09-17T15:31:27.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2569951223}) 2024-09-17T15:31:27.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2827439195}) 2024-09-17T15:31:27.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.245+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2555703913}) 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=364632271}) 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=22} crc {osdmap=21186903}) 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=364632271}) 2024-09-17T15:31:27.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:31:27.249+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=2562527642}) 2024-09-17T15:31:27.428 INFO:tasks.rados.rados.0.smithi203.stdout:481: done (1 left) 2024-09-17T15:31:27.428 INFO:tasks.rados.rados.0.smithi203.stdout:482: done (0 left) 2024-09-17T15:31:27.428 INFO:tasks.rados.rados.0.smithi203.stdout:483: copy_from oid 29 from oid 30 current snap is 51 2024-09-17T15:31:27.428 INFO:tasks.rados.rados.0.smithi203.stdout:484: delete oid 4 current snap is 51 2024-09-17T15:31:27.438 INFO:tasks.rados.rados.0.smithi203.stdout:484: done (1 left) 2024-09-17T15:31:27.439 INFO:tasks.rados.rados.0.smithi203.stdout:485: copy_from oid 46 from oid 49 current snap is 51 2024-09-17T15:31:27.439 INFO:tasks.rados.rados.0.smithi203.stdout:486: read oid 32 snap 47 2024-09-17T15:31:27.439 INFO:tasks.rados.rados.0.smithi203.stdout:486: expect deleted 2024-09-17T15:31:27.439 INFO:tasks.rados.rados.0.smithi203.stdout:487: snap_remove snap 44 2024-09-17T15:31:28.245 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:31:28.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: get_monmap_and_config 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: monmap: 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:28.373718+0000 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:28.373718+0000 2024-09-17T15:31:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:28.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 Processor -- start 2024-09-17T15:31:28.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 -- start start 2024-09-17T15:31:28.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: init 2024-09-17T15:31:28.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3f600689d0 0x7f3f6009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f600689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f60068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 0x7f3f6013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f60139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f6009d770 con 0x7f3f60068db0 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3f6009ed00 con 0x7f3f600689d0 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6009efe0 con 0x7f3f60139f00 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f6634d640 10 monclient(hunting): authenticate will time out at 1221.339251s 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f5f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3f60068db0 0x7f3f600691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44766/0 (socket says 172.21.15.86:44766) 2024-09-17T15:31:28.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/3071844341 learned_addr learned my addr 172.21.15.86:0/3071844341 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 --2- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 0x7f3f6013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): get_auth_request con 0x7f3f60139f00 auth_method 0 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient(hunting): handle_auth_done global_id 44135 payload 274 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 -- 172.21.15.86:0/3071844341 >> v2:172.21.15.203:3300/0 conn(0x7f3f600689d0 msgr2=0x7f3f6009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 --2- 172.21.15.86:0/3071844341 >> v2:172.21.15.203:3300/0 conn(0x7f3f600689d0 0x7f3f6009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 -- 172.21.15.86:0/3071844341 >> v1:172.21.15.86:6789/0 conn(0x7f3f60068db0 legacy=0x7f3f600691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 monclient: found mon.noname-b 2024-09-17T15:31:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.369+0000 7f3f648ea640 1 -- 172.21.15.86:0/3071844341 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f600a04e0 con 0x7f3f60139f00 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 1 -- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 msgr2=0x7f3f6013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 1 --2- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 0x7f3f6013c2f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3f600a0770 tx=0x7f3f5801c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4307038976d7ad99 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 1 -- 172.21.15.86:0/3071844341 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f580b2070 con 0x7f3f60139f00 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: dump: 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 1 -- 172.21.15.86:0/3071844341 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f580a58d0 con 0x7f3f60139f00 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 5 monclient: authenticate success, global_id 44135 2024-09-17T15:31:28.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 1 -- 172.21.15.86:0/3071844341 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f580a5ca0 con 0x7f3f60139f00 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 4 set_mon_vals no callback set 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5effd640 10 monclient: dump: 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.382 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.383 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.383 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: shutdown 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 msgr2=0x7f3f6013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 0x7f3f6013c2f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3f600a0770 tx=0x7f3f5801c8a0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- 172.21.15.86:0/3071844341 shutdown_connections 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- 172.21.15.86:0/3071844341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60139f00 0x7f3f6013c2f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- 172.21.15.86:0/3071844341 >> v2:172.21.15.203:3300/0 conn(0x7f3f600689d0 0x7f3f6009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- 172.21.15.86:0/3071844341 >> 172.21.15.86:0/3071844341 conn(0x7f3f600a6440 msgr2=0x7f3f600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- 172.21.15.86:0/3071844341 shutdown_connections 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- 172.21.15.86:0/3071844341 wait complete. 2024-09-17T15:31:28.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: monmap: 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:28.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 Processor -- start 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- start start 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: init 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f600689d0 0x7f3f601cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-b con 0x7f3f600689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-c con 0x7f3f60068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 0x7f3f601331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): picked mon.noname-a con 0x7f3f60139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6013f3e0 con 0x7f3f60139f00 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6013f260 con 0x7f3f600689d0 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6013f560 con 0x7f3f60068db0 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f6634d640 10 monclient(hunting): authenticate will time out at 1221.347067s 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 0x7f3f601331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37496/0 (socket says 172.21.15.86:37496) 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/1557048623 learned_addr learned my addr 172.21.15.86:0/1557048623 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:28.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): get_auth_request con 0x7f3f60068db0 auth_method 0 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): get_auth_request con 0x7f3f60139f00 auth_method 0 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5ffff640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f600689d0 0x7f3f601cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5ffff640 10 monclient(hunting): get_auth_request con 0x7f3f600689d0 auth_method 0 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f600689d0 msgr2=0x7f3f601cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f600689d0 0x7f3f601cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 msgr2=0x7f3f601331e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 0x7f3f601331e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/1557048623 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f60138b60 con 0x7f3f60068db0 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.377+0000 7f3f648ea640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 0x7f3f601331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5f7fe640 1 -- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 msgr2=0x7f3f60132ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5f7fe640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c470 tx=0x7f3f5400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54007070 con 0x7f3f60068db0 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:28.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: dump: 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f54007210 con 0x7f3f60068db0 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 5 monclient: authenticate success, global_id 44138 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _renew_subs 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6009d460 con 0x7f3f60068db0 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f54007600 con 0x7f3f60068db0 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 10 monclient: dump: 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.396 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _renew_subs 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f6011ddc0 con 0x7f3f60068db0 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5dffb640 4 set_mon_vals no callback set 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f6009ffa0 con 0x7f3f60068db0 2024-09-17T15:31:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3f5401b400 con 0x7f3f60068db0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 0x7f3f4c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(122..122 src has 1..122) ==== 4783+0+0 (secure 0 0 0) 0x7f3f5404fc60 con 0x7f3f60068db0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5ffff640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 0x7f3f4c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.381+0000 7f3f5ffff640 10 monclient: get_auth_request con 0x7f3f4c03bbb0 auth_method 0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.385+0000 7f3f5ffff640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 0x7f3f4c03e070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f50006fd0 tx=0x7f3f50008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.385+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3f54010d10 con 0x7f3f60068db0 2024-09-17T15:31:28.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.385+0000 7f3f5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.385+0000 7f3f5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:28.442 INFO:tasks.rados.rados.0.smithi203.stdout:487: done (3 left) 2024-09-17T15:31:28.442 INFO:tasks.rados.rados.0.smithi203.stdout:488: snap_remove snap 50 2024-09-17T15:31:28.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f6634d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:28.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f6634d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:28.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f6634d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3f6006d4f0 con 0x7f3f60068db0 2024-09-17T15:31:28.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f5cff9640 1 -- 172.21.15.86:0/1557048623 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3f54012400 con 0x7f3f60068db0 2024-09-17T15:31:28.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:28.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.633+0000 7f3f5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:28.638 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 msgr2=0x7f3f4c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 0x7f3f4c03e070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f50006fd0 tx=0x7f3f50008040 comp rx=0 tx=0).stop 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 10 monclient: shutdown 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 msgr2=0x7f3f60132ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f5400c470 tx=0x7f3f5400c930 comp rx=0 tx=0).stop 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 shutdown_connections 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3f4c03bbb0 0x7f3f4c03e070 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f60139f00 0x7f3f601331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f60068db0 0x7f3f60132ca0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 --2- 172.21.15.86:0/1557048623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f600689d0 0x7f3f601cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 >> 172.21.15.86:0/1557048623 conn(0x7f3f600a6440 msgr2=0x7f3f6013b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 shutdown_connections 2024-09-17T15:31:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.637+0000 7f3f6634d640 1 -- 172.21.15.86:0/1557048623 wait complete. 2024-09-17T15:31:28.657 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:31:28.657 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:31:28.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:31:28.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:31:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: get_monmap_and_config 2024-09-17T15:31:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: monmap: 2024-09-17T15:31:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:28.780096+0000 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:28.780096+0000 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:28.780 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 Processor -- start 2024-09-17T15:31:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 -- start start 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: init 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4700a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4700a5bd0 0x7fe47009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe4700a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe47013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4700a6280 con 0x7fe4700a50e0 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe47009d2a0 con 0x7fe4700a5bd0 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe47009c0a0 con 0x7fe47013d2f0 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe474c0e640 10 monclient(hunting): authenticate will time out at 1221.745543s 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4700a5bd0 0x7fe47009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe4700a5bd0 0x7fe47009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35238/0 (socket says 172.21.15.86:35238) 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 -- 172.21.15.86:0/3112245437 learned_addr learned my addr 172.21.15.86:0/3112245437 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): get_auth_request con 0x7fe4700a5bd0 auth_method 0 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37504/0 (socket says 172.21.15.86:37504) 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): get_auth_request con 0x7fe47013d2f0 auth_method 0 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_done global_id 4659 payload 274 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 -- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 msgr2=0x7fe47013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 -- 172.21.15.86:0/3112245437 >> v1:172.21.15.203:6789/0 conn(0x7fe4700a50e0 legacy=0x7fe4700a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:31:28.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:28.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46dd9b640 1 -- 172.21.15.86:0/3112245437 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47009c380 con 0x7fe4700a5bd0 2024-09-17T15:31:28.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.777+0000 7fe46ed9d640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:28.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46dd9b640 1 -- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4700a5bd0 msgr2=0x7fe47009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4700a5bd0 0x7fe47009cd60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87447a542466ae93 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 1 -- 172.21.15.86:0/3112245437 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4640b0070 con 0x7fe4700a5bd0 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: dump: 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 5 monclient: authenticate success, global_id 4659 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 1 -- 172.21.15.86:0/3112245437 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4640ab600 con 0x7fe4700a5bd0 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 4 set_mon_vals no callback set 2024-09-17T15:31:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 1 -- 172.21.15.86:0/3112245437 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4640ab9d0 con 0x7fe4700a5bd0 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe46d59a640 10 monclient: dump: 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 10 monclient: shutdown 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 -- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4700a5bd0 msgr2=0x7fe47009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4700a5bd0 0x7fe47009cd60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe464009870 tx=0x7fe46401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 -- 172.21.15.86:0/3112245437 shutdown_connections 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe47013d2f0 0x7fe47013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 --2- 172.21.15.86:0/3112245437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4700a5bd0 0x7fe47009cd60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 -- 172.21.15.86:0/3112245437 >> 172.21.15.86:0/3112245437 conn(0x7fe4700aaae0 msgr2=0x7fe4700aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 -- 172.21.15.86:0/3112245437 shutdown_connections 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 1 -- 172.21.15.86:0/3112245437 wait complete. 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.781+0000 7fe474c0e640 10 monclient: monmap: 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:28.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 Processor -- start 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- start start 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: init 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4700a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:28.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4700a5bd0 0x7fe4701d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4700a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe47013d2f0 0x7fe4701d0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): picked mon.noname-a con 0x7fe47013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe470142650 con 0x7fe47013d2f0 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe4700a6b00 con 0x7fe4700a5bd0 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe4701427d0 con 0x7fe4700a50e0 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient(hunting): authenticate will time out at 1221.752744s 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37508/0 (socket says 172.21.15.86:37508) 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 -- 172.21.15.86:0/4000221822 learned_addr learned my addr 172.21.15.86:0/4000221822 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): get_auth_request con 0x7fe4700a50e0 auth_method 0 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46ed9d640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe47013d2f0 0x7fe4701d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46ed9d640 10 monclient(hunting): get_auth_request con 0x7fe47013d2f0 auth_method 0 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4700a5bd0 0x7fe4701d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4700a5bd0 msgr2=0x7fe4701d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4700a5bd0 0x7fe4701d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe47013d2f0 msgr2=0x7fe4701d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe47013d2f0 0x7fe4701d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 monclient: found mon.noname-c 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe47013b190 con 0x7fe4700a50e0 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46dd9b640 10 monclient: get_auth_request con 0x7fe4700a5bd0 auth_method 0 2024-09-17T15:31:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4700a5bd0 0x7fe4701d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:31:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 msgr2=0x7fe4701cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46e59c640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c450 tx=0x7fe46000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460012a40 con 0x7fe4700a50e0 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: dump: 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe460017070 con 0x7fe4700a50e0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 5 monclient: authenticate success, global_id 44144 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _renew_subs 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe47013d980 con 0x7fe4700a50e0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe460012e40 con 0x7fe4700a50e0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe45effd640 10 monclient: dump: 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe46cd99640 4 set_mon_vals no callback set 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _renew_subs 2024-09-17T15:31:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.785+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4700a4840 con 0x7fe4700a50e0 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe474c0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe474c0e640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 0x7fe470135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe47006a0e0 con 0x7fe470134e70 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 0x7fe470135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient: get_auth_request con 0x7fe470134e70 auth_method 0 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46e59c640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c450 tx=0x7fe46000c910 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fe46004fd00 osd_map(123..123 src has 1..123) 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe46001c460 con 0x7fe4700a50e0 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_done global_id 4665 payload 824 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe47009cd60 con 0x7fe470134e70 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe45003bb90 0x7fe45003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46ed9d640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe45003bb90 0x7fe45003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46ed9d640 10 monclient: get_auth_request con 0x7fe45003bb90 auth_method 0 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 msgr2=0x7fe470135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 0x7fe470135250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ad000 tx=0x7fe46401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4640ba070 con 0x7fe470134e70 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 10 monclient: dump: 2024-09-17T15:31:28.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe464009490 con 0x7fe470134e70 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 msgr2=0x7fe470135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 0x7fe470135250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ad000 tx=0x7fe46401ac30 comp rx=0 tx=0).stop 2024-09-17T15:31:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 msgr2=0x7fe470135250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.789+0000 7fe45effd640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470134e70 0x7fe470135250 secure :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe4640ad000 tx=0x7fe46401ac30 comp rx=0 tx=0).stop 2024-09-17T15:31:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.793+0000 7fe46ed9d640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe45003bb90 0x7fe45003e050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4700a8520 tx=0x7fe46800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe474c0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe474c0e640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 0x7fe47009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe474c0e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe47013af70 con 0x7fe470135790 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 0x7fe47009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient: get_auth_request con 0x7fe470135790 auth_method 0 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.797+0000 7fe46dd9b640 1 -- 172.21.15.86:0/4000221822 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe470139990 con 0x7fe470135790 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe46dd9b640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 msgr2=0x7fe47009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe46dd9b640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 0x7fe47009e5e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe46401c220 tx=0x7fe464019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4640ba040 con 0x7fe470135790 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 10 monclient: dump: 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fe464003cc0 con 0x7fe470135790 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 msgr2=0x7fe47009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 0x7fe47009e5e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe46401c220 tx=0x7fe464019190 comp rx=0 tx=0).stop 2024-09-17T15:31:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 msgr2=0x7fe47009e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe470135790 0x7fe47009e5e0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe46401c220 tx=0x7fe464019190 comp rx=0 tx=0).stop 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 261444, 2024-09-17T15:31:28.814 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.815 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.816 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:28.817 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.818 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:28.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.820 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.821 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:28.822 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.825 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.826 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:28.827 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe45003bb90 msgr2=0x7fe45003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe45003bb90 0x7fe45003e050 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe4700a8520 tx=0x7fe46800b040 comp rx=0 tx=0).stop 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 10 monclient: shutdown 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 msgr2=0x7fe4701cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 --2- 172.21.15.86:0/4000221822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4700a50e0 0x7fe4701cfba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe46000c450 tx=0x7fe46000c910 comp rx=0 tx=0).stop 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe46ed9d640 1 -- 172.21.15.86:0/4000221822 reap_dead start 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe45effd640 1 -- 172.21.15.86:0/4000221822 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (unknown 0 0 0) 0x7fe46004fd00 con 0x7fe4700a50e0 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe46ed9d640 1 -- 172.21.15.86:0/4000221822 reap_dead start 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 shutdown_connections 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 >> 172.21.15.86:0/4000221822 conn(0x7fe4700aaae0 msgr2=0x7fe47013c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 shutdown_connections 2024-09-17T15:31:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.801+0000 7fe474c0e640 1 -- 172.21.15.86:0/4000221822 wait complete. 2024-09-17T15:31:28.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:28.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:31:28.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: get_monmap_and_config 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: monmap: 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:28.944742+0000 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:28.944742+0000 2024-09-17T15:31:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:28.945 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 Processor -- start 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 -- start start 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: init 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f780409d770 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f780409e260 0x7f7804139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): picked mon.noname-a con 0x7f780409e260 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:28.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f780413a440 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780409ff00 con 0x7f780409d770 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780409ec30 con 0x7f780413a440 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f780409e7f0 con 0x7f780409e260 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780c383640 10 monclient(hunting): authenticate will time out at 1221.910222s 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f780409e260 0x7f7804139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f780409e260 0x7f7804139f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35270/0 (socket says 172.21.15.86:35270) 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 -- 172.21.15.86:0/3762345470 learned_addr learned my addr 172.21.15.86:0/3762345470 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): get_auth_request con 0x7f780409e260 auth_method 0 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780a11f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f780409d770 0x7f780409db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44782/0 (socket says 172.21.15.86:44782) 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient(hunting): handle_auth_done global_id 4671 payload 274 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 -- 172.21.15.86:0/3762345470 >> v1:172.21.15.203:6789/0 conn(0x7f780413a440 legacy=0x7f780413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 -- 172.21.15.86:0/3762345470 >> v1:172.21.15.86:6789/0 conn(0x7f780409d770 legacy=0x7f780409db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 monclient: found mon.noname-a 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:28.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.941+0000 7f780991e640 1 -- 172.21.15.86:0/3762345470 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780409ef10 con 0x7f780409e260 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780991e640 1 -- 172.21.15.86:0/3762345470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780409e260 msgr2=0x7f7804139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780991e640 1 --2- 172.21.15.86:0/3762345470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780409e260 0x7f7804139f00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77f400bb10 tx=0x7f77f401eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f0a8e92a03dab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 1 -- 172.21.15.86:0/3762345470 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40b4070 con 0x7f780409e260 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: dump: 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 1 -- 172.21.15.86:0/3762345470 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f40a7910 con 0x7f780409e260 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 1 -- 172.21.15.86:0/3762345470 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40a7ce0 con 0x7f780409e260 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 5 monclient: authenticate success, global_id 4671 2024-09-17T15:31:28.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 4 set_mon_vals no callback set 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780911d640 10 monclient: dump: 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 10 monclient: shutdown 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 -- 172.21.15.86:0/3762345470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780409e260 msgr2=0x7f7804139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 --2- 172.21.15.86:0/3762345470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780409e260 0x7f7804139f00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77f400bb10 tx=0x7f77f401eaf0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 -- 172.21.15.86:0/3762345470 shutdown_connections 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 --2- 172.21.15.86:0/3762345470 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780409e260 0x7f7804139f00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 -- 172.21.15.86:0/3762345470 >> 172.21.15.86:0/3762345470 conn(0x7f78040a2f40 msgr2=0x7f78040a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 -- 172.21.15.86:0/3762345470 shutdown_connections 2024-09-17T15:31:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.945+0000 7f780c383640 1 -- 172.21.15.86:0/3762345470 wait complete. 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient: build_initial_monmap 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient: monmap: 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:28.959 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 Processor -- start 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 -- start start 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient: init 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): picked mon.noname-c con 0x7f780409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f780409e260 0x7f7804134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): picked mon.noname-b con 0x7f780409e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 0x7f7804135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): picked mon.noname-a con 0x7f780413a440 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f78040a1850 con 0x7f780413a440 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78040a16d0 con 0x7f780409e260 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f78040a19d0 con 0x7f780409d770 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780c383640 10 monclient(hunting): authenticate will time out at 1221.917710s 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 0x7f7804135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37522/0 (socket says 172.21.15.86:37522) 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 0x7f7804135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35274/0 (socket says 172.21.15.86:35274) 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 -- 172.21.15.86:0/1576884117 learned_addr learned my addr 172.21.15.86:0/1576884117 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:28.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): get_auth_request con 0x7f780409d770 auth_method 0 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): get_auth_request con 0x7f780413a440 auth_method 0 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f780409e260 0x7f7804134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 10 monclient(hunting): get_auth_request con 0x7f780409e260 auth_method 0 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient(hunting): handle_auth_done global_id 44147 payload 824 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f780409e260 msgr2=0x7f7804134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f780409e260 0x7f7804134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 msgr2=0x7f7804135340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 0x7f7804135340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient: _finish_hunting 0 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 monclient: found mon.noname-c 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a11f640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780413a900 con 0x7f780409d770 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f780409e260 0x7f7804134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.949+0000 7f780a920640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f780413a440 0x7f7804135340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780a11f640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 msgr2=0x7f78041d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780a11f640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f780000c460 tx=0x7f780000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800017070 con 0x7f780409d770 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: dump: 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.964 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7800005820 con 0x7f780409d770 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 5 monclient: authenticate success, global_id 44147 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 monclient: _renew_subs 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7804120460 con 0x7f780409d770 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800005c10 con 0x7f780409d770 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 10 monclient: dump: 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780891c640 4 set_mon_vals no callback set 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 monclient: _renew_subs 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041227c0 con 0x7f780409d770 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 0x7f780402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f780409db50 con 0x7f780411f1b0 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 0x7f780402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient: get_auth_request con 0x7f780411f1b0 auth_method 0 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780a11f640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f780000c460 tx=0x7f780000c920 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f780001a070 mgrmap(e 5) 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f780001a070 con 0x7f780409d770 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 10 monclient(hunting): handle_auth_done global_id 44150 payload 824 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780991e640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7804072e00 con 0x7f780411f1b0 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f77faffd640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77e803bb90 0x7f77e803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780a920640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77e803bb90 0x7f77e803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.953+0000 7f780a920640 10 monclient: get_auth_request con 0x7f77e803bb90 auth_method 0 2024-09-17T15:31:28.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f780991e640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 msgr2=0x7f780402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 0x7f780402d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f77f40b1000 tx=0x7f77f401d330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40be020 con 0x7f780411f1b0 2024-09-17T15:31:28.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 10 monclient: dump: 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f77f400b730 con 0x7f780411f1b0 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:28.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 msgr2=0x7f780402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 0x7f780402d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f77f40b1000 tx=0x7f77f401d330 comp rx=0 tx=0).stop 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 msgr2=0x7f780402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f77faffd640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f1b0 0x7f780402d450 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f77f40b1000 tx=0x7f77f401d330 comp rx=0 tx=0).stop 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.957+0000 7f780a920640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77e803bb90 0x7f77e803e050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f78040a2c80 tx=0x7f77fc003ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780c383640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780c383640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 0x7f780402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780c383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f780406d4f0 con 0x7f780411f590 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 0x7f780402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient: get_auth_request con 0x7f780411f590 auth_method 0 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:28.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.961+0000 7f780991e640 1 -- 172.21.15.86:0/1576884117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7804120fc0 con 0x7f780411f590 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780991e640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 msgr2=0x7f780402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780991e640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 0x7f780402d990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f77f400bb10 tx=0x7f77f401b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40be070 con 0x7f780411f590 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 10 monclient: dump: 2024-09-17T15:31:28.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f77f40ba420 con 0x7f780411f590 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 msgr2=0x7f780402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 0x7f780402d990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f77f400bb10 tx=0x7f77f401b890 comp rx=0 tx=0).stop 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 msgr2=0x7f780402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780411f590 0x7f780402d990 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f77f400bb10 tx=0x7f77f401b890 comp rx=0 tx=0).stop 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 17919, 2024-09-17T15:31:28.975 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.976 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.977 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:28.978 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:28.979 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.980 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.981 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:28.983 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:28.984 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.986 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:28.987 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77e803bb90 msgr2=0x7f77e803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77e803bb90 0x7f77e803e050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f78040a2c80 tx=0x7f77fc003ca0 comp rx=0 tx=0).stop 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 10 monclient: shutdown 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 msgr2=0x7f78041d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 --2- 172.21.15.86:0/1576884117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f780409d770 0x7f78041d0020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f780000c460 tx=0x7f780000c920 comp rx=0 tx=0).stop 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780a920640 1 -- 172.21.15.86:0/1576884117 reap_dead start 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f77faffd640 1 -- 172.21.15.86:0/1576884117 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (unknown 0 0 0) 0x7f780004f3b0 con 0x7f780409d770 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780a920640 1 -- 172.21.15.86:0/1576884117 reap_dead start 2024-09-17T15:31:28.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 shutdown_connections 2024-09-17T15:31:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 >> 172.21.15.86:0/1576884117 conn(0x7f78040a2f40 msgr2=0x7f78040a5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 shutdown_connections 2024-09-17T15:31:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:28.965+0000 7f780c383640 1 -- 172.21.15.86:0/1576884117 wait complete. 2024-09-17T15:31:28.990 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:28.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:31:28.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:31:29.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: get_monmap_and_config 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: build_initial_monmap 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: monmap: 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:29.104451+0000 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:29.104451+0000 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:29.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:29.105 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:29.105 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:29.105 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:29.105 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 Processor -- start 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 -- start start 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: init 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 0x7f4bb80a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bb80a1590 0x7f4bb8135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb80a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bb8135f90 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bb80a2130 con 0x7f4bb8135f90 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4bb80a2410 con 0x7f4bb80a1590 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4bb80a26f0 con 0x7f4bb80a0c70 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bbd3a1640 10 monclient(hunting): authenticate will time out at 1222.069972s 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bb80a1590 0x7f4bb8135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4bb8135f90 0x7f4bb813cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44796/0 (socket says 172.21.15.86:44796) 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb77fe640 1 -- 172.21.15.86:0/1390552687 learned_addr learned my addr 172.21.15.86:0/1390552687 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): get_auth_request con 0x7f4bb80a1590 auth_method 0 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb6ffd640 1 --2- 172.21.15.86:0/1390552687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 0x7f4bb80a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_done global_id 4677 payload 274 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 -- 172.21.15.86:0/1390552687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 msgr2=0x7f4bb80a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/1390552687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 0x7f4bb80a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 -- 172.21.15.86:0/1390552687 >> v1:172.21.15.86:6789/0 conn(0x7f4bb8135f90 legacy=0x7f4bb813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient: _finish_hunting 0 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 monclient: found mon.noname-a 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb67fc640 1 -- 172.21.15.86:0/1390552687 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb809f9f0 con 0x7f4bb80a1590 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb6ffd640 10 monclient: get_auth_request con 0x7f4bb80a0c70 auth_method 0 2024-09-17T15:31:29.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.101+0000 7f4bb6ffd640 1 --2- 172.21.15.86:0/1390552687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 0x7f4bb80a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb67fc640 1 -- 172.21.15.86:0/1390552687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a1590 msgr2=0x7f4bb8135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/1390552687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a1590 0x7f4bb8135a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4bb806d160 tx=0x7f4bac01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954521880626adf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 1 -- 172.21.15.86:0/1390552687 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0b0070 con 0x7f4bb80a1590 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: dump: 2024-09-17T15:31:29.111 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 5 monclient: authenticate success, global_id 4677 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 1 -- 172.21.15.86:0/1390552687 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bac0ab600 con 0x7f4bb80a1590 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 1 -- 172.21.15.86:0/1390552687 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0ab9d0 con 0x7f4bb80a1590 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 4 set_mon_vals no callback set 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bb5ffb640 10 monclient: dump: 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 10 monclient: shutdown 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/1390552687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a1590 msgr2=0x7f4bb8135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/1390552687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a1590 0x7f4bb8135a50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4bb806d160 tx=0x7f4bac01c340 comp rx=0 tx=0).stop 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/1390552687 shutdown_connections 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/1390552687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a1590 0x7f4bb8135a50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/1390552687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a0c70 0x7f4bb80a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/1390552687 >> 172.21.15.86:0/1390552687 conn(0x7f4bb80a2f40 msgr2=0x7f4bb80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/1390552687 shutdown_connections 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.105+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/1390552687 wait complete. 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient: build_initial_monmap 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient: monmap: 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:29.115 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:29.116 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:29.116 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 Processor -- start 2024-09-17T15:31:29.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 -- start start 2024-09-17T15:31:29.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient: init 2024-09-17T15:31:29.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 0x7f4bb813d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb80a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a1590 0x7f4bb809d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb80a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb8135f90 0x7f4bb809dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb8135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb809ce60 con 0x7f4bb80a0c70 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb809cce0 con 0x7f4bb80a1590 2024-09-17T15:31:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb809ff50 con 0x7f4bb8135f90 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bbd3a1640 10 monclient(hunting): authenticate will time out at 1222.077210s 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 0x7f4bb813d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 0x7f4bb813d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35280/0 (socket says 172.21.15.86:35280) 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3293705135 learned_addr learned my addr 172.21.15.86:0/3293705135 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): get_auth_request con 0x7f4bb80a0c70 auth_method 0 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb8135f90 0x7f4bb809dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request con 0x7f4bb8135f90 auth_method 0 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb8135f90 msgr2=0x7f4bb809dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a1590 0x7f4bb809d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb8135f90 0x7f4bb809dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a1590 msgr2=0x7f4bb809d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a1590 0x7f4bb809d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 monclient: found mon.noname-a 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb813b440 con 0x7f4bb80a0c70 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 msgr2=0x7f4bb813d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.109+0000 7f4bb6ffd640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 0x7f4bb813d160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c7c0 tx=0x7f4ba800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8007c40 con 0x7f4bb80a0c70 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: dump: 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8007de0 con 0x7f4bb80a0c70 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8011630 con 0x7f4bb80a0c70 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 5 monclient: authenticate success, global_id 4680 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 10 monclient: dump: 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 monclient: _renew_subs 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb813d6a0 con 0x7f4bb80a0c70 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb57fa640 4 set_mon_vals no callback set 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 monclient: _renew_subs 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb806fec0 con 0x7f4bb80a0c70 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 0x7f4bb8071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb806f990 con 0x7f4bb809e220 2024-09-17T15:31:29.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4ba80117d0 con 0x7f4bb80a0c70 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 0x7f4bb8071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient: get_auth_request con 0x7f4bb809e220 auth_method 0 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bbb0 0x7f4ba403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (secure 0 0 0) 0x7f4ba8050260 con 0x7f4bb80a0c70 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bbb0 0x7f4ba403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb77fe640 10 monclient: get_auth_request con 0x7f4ba403bbb0 auth_method 0 2024-09-17T15:31:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_done global_id 14287 payload 824 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.113+0000 7f4bb67fc640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bb813aec0 con 0x7f4bb809e220 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4bb67fc640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 msgr2=0x7f4bb8071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 0x7f4bb8071240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0ad000 tx=0x7f4bac01ab80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bbb0 0x7f4ba403e070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0005500 tx=0x7f4bb0005490 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0b0050 con 0x7f4bb809e220 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 10 monclient: dump: 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4bac0b1040 con 0x7f4bb809e220 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 msgr2=0x7f4bb8071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 0x7f4bb8071240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0ad000 tx=0x7f4bac01ab80 comp rx=0 tx=0).stop 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 msgr2=0x7f4bb8071240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.117+0000 7f4b9f7fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb809e220 0x7f4bb8071240 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0ad000 tx=0x7f4bac01ab80 comp rx=0 tx=0).stop 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.121+0000 7f4bbd3a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.121+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 0x7f4bb8071b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.121+0000 7f4bbd3a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.121+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb809e600 con 0x7f4bb8071780 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.121+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 0x7f4bb8071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient: get_auth_request con 0x7f4bb8071780 auth_method 0 2024-09-17T15:31:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 10 monclient(hunting): handle_auth_done global_id 14290 payload 824 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 1 -- 172.21.15.86:0/3293705135 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4bb80a1050 con 0x7f4bb8071780 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 msgr2=0x7f4bb8071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4bb67fc640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 0x7f4bb8071b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4bac000c10 tx=0x7f4bac0190e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0bf720 con 0x7f4bb8071780 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4b9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.125+0000 7f4b9f7fe640 10 monclient: dump: 2024-09-17T15:31:29.133 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f4bac003e70 con 0x7f4bb8071780 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 msgr2=0x7f4bb8071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 0x7f4bb8071b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4bac000c10 tx=0x7f4bac0190e0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 msgr2=0x7f4bb8071b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4b9f7fe640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb8071780 0x7f4bb8071b60 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4bac000c10 tx=0x7f4bac0190e0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 50, 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 197729, 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:29.135 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:29.136 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:29.137 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:29.138 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:29.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.140 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.141 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:29.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:29.143 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:29.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:29.146 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:29.147 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bbb0 msgr2=0x7f4ba403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bbb0 0x7f4ba403e070 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0005500 tx=0x7f4bb0005490 comp rx=0 tx=0).stop 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 10 monclient: shutdown 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 msgr2=0x7f4bb813d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 --2- 172.21.15.86:0/3293705135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a0c70 0x7f4bb813d160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c7c0 tx=0x7f4ba800cc80 comp rx=0 tx=0).stop 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3293705135 reap_dead start 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3293705135 reap_dead start 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 shutdown_connections 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 >> 172.21.15.86:0/3293705135 conn(0x7f4bb80a2f40 msgr2=0x7f4bb8072b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 shutdown_connections 2024-09-17T15:31:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.129+0000 7f4bbd3a1640 1 -- 172.21.15.86:0/3293705135 wait complete. 2024-09-17T15:31:29.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:31:29.150 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:31:29.150 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:31:29.151 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:31:29.151 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:31:29.151 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:31:29.151 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:31:29.175 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:31:29.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:31:29.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: get_monmap_and_config 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: build_initial_monmap 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: monmap: 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.284 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:29.284986+0000 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:29.284986+0000 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:29.285 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 Processor -- start 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 -- start start 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: init 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f762c0a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f762c0a5b60 0x7f762c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-a con 0x7f762c0a5b60 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f762c13d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c0a6f40 con 0x7f762c0a5070 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f762c0a65e0 con 0x7f762c13d280 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f762c09c2a0 con 0x7f762c0a5b60 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f76310a1640 10 monclient(hunting): authenticate will time out at 1222.250157s 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f762c0a5b60 0x7f762c09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762b59e640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f762c13d280 legacy=0x7f762c13f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762ad9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f762c0a5070 0x7f762c0a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44812/0 (socket says 172.21.15.86:44812) 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762ad9d640 1 -- 172.21.15.86:0/1098713431 learned_addr learned my addr 172.21.15.86:0/1098713431 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): get_auth_request con 0x7f762c0a5b60 auth_method 0 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient(hunting): handle_auth_done global_id 4683 payload 274 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 1 -- 172.21.15.86:0/1098713431 >> v1:172.21.15.203:6789/0 conn(0x7f762c13d280 legacy=0x7f762c13f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 1 -- 172.21.15.86:0/1098713431 >> v1:172.21.15.86:6789/0 conn(0x7f762c0a5070 legacy=0x7f762c0a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 1 monclient: found mon.noname-a 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.281+0000 7f762a59c640 1 -- 172.21.15.86:0/1098713431 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c09c580 con 0x7f762c0a5b60 2024-09-17T15:31:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 -- 172.21.15.86:0/1098713431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 msgr2=0x7f762c09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 --2- 172.21.15.86:0/1098713431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c09cf60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7620009d90 tx=0x7f762001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fd19c06959bc09 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 1 -- 172.21.15.86:0/1098713431 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200ac070 con 0x7f762c0a5b60 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: dump: 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: _finish_auth 0 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 1 -- 172.21.15.86:0/1098713431 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76200a6760 con 0x7f762c0a5b60 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 1 -- 172.21.15.86:0/1098713431 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200a6b30 con 0x7f762c0a5b60 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 5 monclient: authenticate success, global_id 4683 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 4 set_mon_vals no callback set 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f7629d9b640 10 monclient: dump: 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.294 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: shutdown 2024-09-17T15:31:29.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- 172.21.15.86:0/1098713431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 msgr2=0x7f762c09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 --2- 172.21.15.86:0/1098713431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c09cf60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7620009d90 tx=0x7f762001c830 comp rx=0 tx=0).stop 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- 172.21.15.86:0/1098713431 shutdown_connections 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 --2- 172.21.15.86:0/1098713431 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c09cf60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- 172.21.15.86:0/1098713431 >> 172.21.15.86:0/1098713431 conn(0x7f762c0742a0 msgr2=0x7f762c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- 172.21.15.86:0/1098713431 shutdown_connections 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- 172.21.15.86:0/1098713431 wait complete. 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: build_initial_monmap 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: monmap: 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:29.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 Processor -- start 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- start start 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: init 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 0x7f762c137d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-b con 0x7f762c0a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-a con 0x7f762c0a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f762c13d280 0x7f762c13a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): picked mon.noname-c con 0x7f762c13d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f762c143940 con 0x7f762c0a5b60 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f762c1437c0 con 0x7f762c0a5070 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f762c143ac0 con 0x7f762c13d280 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f76310a1640 10 monclient(hunting): authenticate will time out at 1222.258883s 2024-09-17T15:31:29.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35308/0 (socket says 172.21.15.86:35308) 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 -- 172.21.15.86:0/1592419592 learned_addr learned my addr 172.21.15.86:0/1592419592 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f762c13d280 0x7f762c13a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): get_auth_request con 0x7f762c0a5b60 auth_method 0 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762ad9d640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 msgr2=0x7f762c137d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): get_auth_request con 0x7f762c13d280 auth_method 0 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762ad9d640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 0x7f762c137d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:29.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient(hunting): handle_auth_done global_id 4686 payload 824 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f762c13d280 msgr2=0x7f762c13a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f762c13d280 0x7f762c13a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 msgr2=0x7f762c137d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 0x7f762c137d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 monclient: found mon.noname-a 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:29.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.289+0000 7f762a59c640 1 -- 172.21.15.86:0/1592419592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f762c121860 con 0x7f762c0a5b60 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f762a59c640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 msgr2=0x7f762c139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f762a59c640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f762001c660 tx=0x7f76200195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200c0070 con 0x7f762c0a5b60 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: dump: 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: _finish_auth 0 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: _check_auth_tickets 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76200a7880 con 0x7f762c0a5b60 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76200ac050 con 0x7f762c0a5b60 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 10 monclient: dump: 2024-09-17T15:31:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 5 monclient: authenticate success, global_id 4686 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _renew_subs 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f7628d99640 4 set_mon_vals no callback set 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762c120cc0 con 0x7f762c0a5b60 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _renew_subs 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762c0a1860 con 0x7f762c0a5b60 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f762c09cfe0 con 0x7f762c0a5b60 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f76200af070 con 0x7f762c0a5b60 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 0x7f761803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (secure 0 0 0) 0x7f76200ea4b0 con 0x7f762c0a5b60 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f762ad9d640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 0x7f761803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:29.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.293+0000 7f762ad9d640 10 monclient: get_auth_request con 0x7f761803bb10 auth_method 0 2024-09-17T15:31:29.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.297+0000 7f762ad9d640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 0x7f761803dfd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f761c004820 tx=0x7f761c007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:29.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.297+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:31:29.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.301+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7620016540 con 0x7f762c0a5b60 2024-09-17T15:31:29.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.301+0000 7f76137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:29.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.301+0000 7f76137fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.525+0000 7f76310a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.525+0000 7f76310a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.525+0000 7f76310a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.525+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f762c069920 con 0x7f762c0a5b60 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76137fe640 1 -- 172.21.15.86:0/1592419592 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f76200b1d00 con 0x7f762c0a5b60 2024-09-17T15:31:29.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:29.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76137fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:29.532 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 msgr2=0x7f761803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 0x7f761803dfd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f761c004820 tx=0x7f761c007b30 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 10 monclient: shutdown 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 msgr2=0x7f762c139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f762001c660 tx=0x7f76200195d0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 shutdown_connections 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f761803bb10 0x7f761803dfd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f762c13d280 0x7f762c13a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f762c0a5b60 0x7f762c139fd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 --2- 172.21.15.86:0/1592419592 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f762c0a5070 0x7f762c137d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:29.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 >> 172.21.15.86:0/1592419592 conn(0x7f762c0742a0 msgr2=0x7f762c0a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:29.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 shutdown_connections 2024-09-17T15:31:29.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:29.529+0000 7f76310a1640 1 -- 172.21.15.86:0/1592419592 wait complete. 2024-09-17T15:31:32.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: get_monmap_and_config 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: build_initial_monmap 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: monmap: 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:32.677254+0000 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:32.677254+0000 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:32.678 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 Processor -- start 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 -- start start 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: init 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa3000689d0 0x7fa30006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): picked mon.noname-c con 0x7fa3000689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa300068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa30013aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30009c8b0 con 0x7fa30013aff0 2024-09-17T15:31:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30009cb90 con 0x7fa300068db0 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa300074500 con 0x7fa3000689d0 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa308384640 10 monclient(hunting): authenticate will time out at 1225.642930s 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa306120640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fa3000689d0 msgr2=0x7fa30006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa30591f640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fa300068db0 legacy=0x7fa300069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa306921640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa30013aff0 0x7fa30013d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44824/0 (socket says 172.21.15.86:44824) 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 learned_addr learned my addr 172.21.15.86:0/784250407 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.673+0000 7fa306120640 1 --2- 172.21.15.86:0/784250407 >> v2:172.21.15.203:3300/0 conn(0x7fa3000689d0 0x7fa30006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 >> v1:172.21.15.86:6789/0 conn(0x7fa30013aff0 legacy=0x7fa30013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753823162 0 0) 0x7fa30009c8b0 con 0x7fa30013aff0 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): _negotiate 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): my global_id is 4689 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2ec003530 con 0x7fa30013aff0 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2915750603 0 0) 0x7fa2ec003530 con 0x7fa30013aff0 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 >> v2:172.21.15.203:3300/0 conn(0x7fa3000689d0 msgr2=0x7fa30006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 --2- 172.21.15.86:0/784250407 >> v2:172.21.15.203:3300/0 conn(0x7fa3000689d0 0x7fa30006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 >> v1:172.21.15.203:6789/0 conn(0x7fa300068db0 legacy=0x7fa300069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _finish_hunting 0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30009ce70 con 0x7fa30013aff0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _finish_auth 0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _check_auth_tickets 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fa2fc001d20 con 0x7fa30013aff0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:32.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: dump: 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:32.686 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 >> v1:172.21.15.86:6789/0 conn(0x7fa30013aff0 legacy=0x7fa30013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2ec003a80 0x7fa2ec003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): picked mon.b con 0x7fa2ec003a80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec004810 0x7fa2ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): picked mon.c con 0x7fa2ec004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): picked mon.a con 0x7fa2ec0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec00c3a0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ec00c550 con 0x7fa2ec003a80 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa2ec00c750 con 0x7fa2ec004810 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30511e640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306120640 1 -- 172.21.15.86:0/784250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2ec003a80 msgr2=0x7fa2ec003e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306120640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2ec003a80 0x7fa2ec003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec004810 0x7fa2ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): get_auth_request con 0x7fa2ec0099c0 auth_method 0 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): get_auth_request con 0x7fa2ec004810 auth_method 0 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:32.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa30591f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient(hunting): handle_auth_done global_id 4689 payload 293 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec004810 msgr2=0x7fa2ec004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec004810 0x7fa2ec004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2ec003a80 msgr2=0x7fa2ec003e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2ec003a80 0x7fa2ec003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient: _finish_hunting 0 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 monclient: found mon.a 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:32.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ec003530 con 0x7fa2ec0099c0 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient: _finish_auth 0 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 10 monclient: _check_auth_tickets 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa306921640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa308384640 5 monclient: authenticate success, global_id 4689 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.677+0000 7fa308384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:32.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 reap_dead start 2024-09-17T15:31:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa306921640 1 -- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 msgr2=0x7fa2ec00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa306921640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc001f40 tx=0x7fa2fc01d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab0d02fd1a631e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc0b0070 con 0x7fa2ec0099c0 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 10 monclient: dump: 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2fc0ab680 con 0x7fa2ec0099c0 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa306921640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc001f40 tx=0x7fa2fc01d400 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fa2fc00a6d0 mon_map magic: 0 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 4 set_mon_vals no callback set 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 10 monclient: shutdown 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 -- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 msgr2=0x7fa2ec00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc001f40 tx=0x7fa2fc01d400 comp rx=0 tx=0).stop 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 1 -- 172.21.15.86:0/784250407 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fa2fc00a6d0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa30511e640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 -- 172.21.15.86:0/784250407 shutdown_connections 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 --2- 172.21.15.86:0/784250407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2ec0099c0 0x7fa2ec00bdb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 -- 172.21.15.86:0/784250407 >> 172.21.15.86:0/784250407 conn(0x7fa3000a6440 msgr2=0x7fa3000a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 -- 172.21.15.86:0/784250407 shutdown_connections 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.681+0000 7fa308384640 1 -- 172.21.15.86:0/784250407 wait complete. 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient: build_initial_monmap 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient: monmap: 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:32.701 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 Processor -- start 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 -- start start 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient: init 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): picked mon.noname-c con 0x7fa2ec0099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 0x7fa3001caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): picked mon.noname-b con 0x7fa3001caaa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 0x7fa30012df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): picked mon.noname-a con 0x7fa30012dad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa3000691e0 con 0x7fa30012dad0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa30006fb10 con 0x7fa3001caaa0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa300069360 con 0x7fa2ec0099c0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa308384640 10 monclient(hunting): authenticate will time out at 1225.653282s 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 0x7fa30012df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37568/0 (socket says 172.21.15.86:37568) 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 -- 172.21.15.86:0/1899119513 learned_addr learned my addr 172.21.15.86:0/1899119513 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): get_auth_request con 0x7fa2ec0099c0 auth_method 0 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): get_auth_request con 0x7fa30012dad0 auth_method 0 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa30591f640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 msgr2=0x7fa3001caf20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa30591f640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 0x7fa3001caf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient(hunting): handle_auth_done global_id 44162 payload 824 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 msgr2=0x7fa3001caf20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 0x7fa3001caf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 msgr2=0x7fa30012df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 0x7fa30012df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient: _finish_hunting 0 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 monclient: found mon.noname-c 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306120640 1 -- 172.21.15.86:0/1899119513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3000a3180 con 0x7fa2ec0099c0 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.685+0000 7fa306921640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 0x7fa30012df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa306120640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 msgr2=0x7fa3001ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:32.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa306120640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800b560 tx=0x7fa2f800ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8013020 con 0x7fa2ec0099c0 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: dump: 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 5 monclient: authenticate success, global_id 44162 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f8005070 con 0x7fa2ec0099c0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _renew_subs 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30009d460 con 0x7fa2ec0099c0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f8005460 con 0x7fa2ec0099c0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 10 monclient: dump: 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa30491d640 4 set_mon_vals no callback set 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _renew_subs 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa300134d40 con 0x7fa2ec0099c0 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa2f8016020 con 0x7fa2ec0099c0 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa300135ae0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 0x7fa2e403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (secure 0 0 0) 0x7fa2f804fbf0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa30591f640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 0x7fa2e403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.689+0000 7fa30591f640 10 monclient: get_auth_request con 0x7fa2e403bb10 auth_method 0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.693+0000 7fa30591f640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 0x7fa2e403dfd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa300133050 tx=0x7fa2f0006f60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.693+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa2f801d020 con 0x7fa2ec0099c0 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.693+0000 7fa2f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.693+0000 7fa2f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:32.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa308384640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:32.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa308384640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:32.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa308384640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:32.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa30006d4f0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa2f6ffd640 1 -- 172.21.15.86:0/1899119513 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa2f800e3c0 con 0x7fa2ec0099c0 2024-09-17T15:31:32.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa2f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:32.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.921+0000 7fa2f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:32.926 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:32.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 msgr2=0x7fa2e403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 0x7fa2e403dfd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa300133050 tx=0x7fa2f0006f60 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 10 monclient: shutdown 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 msgr2=0x7fa3001ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa2f800b560 tx=0x7fa2f800ba20 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 shutdown_connections 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa2e403bb10 0x7fa2e403dfd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa30012dad0 0x7fa30012df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3001caaa0 0x7fa3001caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 --2- 172.21.15.86:0/1899119513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2ec0099c0 0x7fa3001ca560 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 >> 172.21.15.86:0/1899119513 conn(0x7fa3000a6440 msgr2=0x7fa3000a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:32.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 shutdown_connections 2024-09-17T15:31:32.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:32.925+0000 7fa308384640 1 -- 172.21.15.86:0/1899119513 wait complete. 2024-09-17T15:31:35.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: get_monmap_and_config 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: build_initial_monmap 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: monmap: 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:36.069608+0000 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:36.069608+0000 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.069 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:36.070 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:36.070 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 Processor -- start 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 -- start start 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: init 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 0x7f70a406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-c con 0x7f70a40689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70a4068db0 0x7f70a4069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-a con 0x7f70a4068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70a4135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a409c8b0 con 0x7f70a4135ae0 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:36.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f70a409cb90 con 0x7f70a4068db0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f70a409ce70 con 0x7f70a40689d0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70aa201640 10 monclient(hunting): authenticate will time out at 1229.035147s 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a3fff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f70a4135ae0 legacy=0x7f70a4135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70a4068db0 0x7f70a4069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70a4068db0 0x7f70a4069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35326/0 (socket says 172.21.15.86:35326) 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 -- 172.21.15.86:0/3822157614 learned_addr learned my addr 172.21.15.86:0/3822157614 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): get_auth_request con 0x7f70a4068db0 auth_method 0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a37fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 msgr2=0x7f70a406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a37fe640 1 --2- 172.21.15.86:0/3822157614 >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 0x7f70a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:36.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_done global_id 4695 payload 274 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 -- 172.21.15.86:0/3822157614 >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 msgr2=0x7f70a406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 --2- 172.21.15.86:0/3822157614 >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 0x7f70a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 -- 172.21.15.86:0/3822157614 >> v1:172.21.15.203:6789/0 conn(0x7f70a4135ae0 legacy=0x7f70a4135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:36.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.065+0000 7f70a2ffd640 1 -- 172.21.15.86:0/3822157614 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a4069740 con 0x7f70a4068db0 2024-09-17T15:31:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.069+0000 7f70a2ffd640 1 -- 172.21.15.86:0/3822157614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a4068db0 msgr2=0x7f70a4069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.069+0000 7f70a2ffd640 1 --2- 172.21.15.86:0/3822157614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a4068db0 0x7f70a4069200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f70a4069a70 tx=0x7f709401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae73f039c1ab430 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 1 -- 172.21.15.86:0/3822157614 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70940b0070 con 0x7f70a4068db0 2024-09-17T15:31:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:36.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: dump: 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: _finish_auth 0 2024-09-17T15:31:36.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 1 -- 172.21.15.86:0/3822157614 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70940ab600 con 0x7f70a4068db0 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 1 -- 172.21.15.86:0/3822157614 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70940ab9d0 con 0x7f70a4068db0 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 5 monclient: authenticate success, global_id 4695 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 4 set_mon_vals no callback set 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a27fc640 10 monclient: dump: 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:36.077 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:36.078 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: shutdown 2024-09-17T15:31:36.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- 172.21.15.86:0/3822157614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a4068db0 msgr2=0x7f70a4069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- 172.21.15.86:0/3822157614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a4068db0 0x7f70a4069200 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f70a4069a70 tx=0x7f709401c490 comp rx=0 tx=0).stop 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- 172.21.15.86:0/3822157614 shutdown_connections 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- 172.21.15.86:0/3822157614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a4068db0 0x7f70a4069200 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- 172.21.15.86:0/3822157614 >> v2:172.21.15.203:3300/0 conn(0x7f70a40689d0 0x7f70a406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- 172.21.15.86:0/3822157614 >> 172.21.15.86:0/3822157614 conn(0x7f70a40a6440 msgr2=0x7f70a40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- 172.21.15.86:0/3822157614 shutdown_connections 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- 172.21.15.86:0/3822157614 wait complete. 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: build_initial_monmap 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: monmap: 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:36.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 Processor -- start 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- start start 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: init 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:31:36.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a40689d0 0x7f70a4134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-a con 0x7f70a40689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-c con 0x7f70a4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 0x7f70a41cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): picked mon.noname-b con 0x7f70a4135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4140a40 con 0x7f70a40689d0 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f70a409c2d0 con 0x7f70a4135ae0 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f70a4140bc0 con 0x7f70a4068db0 2024-09-17T15:31:36.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70aa201640 10 monclient(hunting): authenticate will time out at 1229.042438s 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37584/0 (socket says 172.21.15.86:37584) 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a3fff640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 msgr2=0x7f70a41cdfa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 -- 172.21.15.86:0/2447816706 learned_addr learned my addr 172.21.15.86:0/2447816706 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): get_auth_request con 0x7f70a4068db0 auth_method 0 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a3fff640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 0x7f70a41cdfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a40689d0 0x7f70a4134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:36.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): get_auth_request con 0x7f70a40689d0 auth_method 0 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient(hunting): handle_auth_done global_id 44165 payload 824 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 -- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 msgr2=0x7f70a41cdfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 0x7f70a41cdfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 -- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a40689d0 msgr2=0x7f70a4134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:36.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a40689d0 0x7f70a4134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 monclient: found mon.noname-c 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.073+0000 7f70a2ffd640 1 -- 172.21.15.86:0/2447816706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a4073ae0 con 0x7f70a4068db0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70a2ffd640 1 -- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 msgr2=0x7f70a4134fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70a2ffd640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f709401cd30 tx=0x7f709401acd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70940ba020 con 0x7f70a4068db0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: dump: 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:36.088 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: _finish_auth 0 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70940b6630 con 0x7f70a4068db0 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 5 monclient: authenticate success, global_id 44165 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _renew_subs 2024-09-17T15:31:36.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a409d460 con 0x7f70a4068db0 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70940b0040 con 0x7f70a4068db0 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 10 monclient: dump: 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:36.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70a1ffb640 4 set_mon_vals no callback set 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _renew_subs 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a411ee40 con 0x7f70a4068db0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a406f840 con 0x7f70a4068db0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f70940bd020 con 0x7f70a4068db0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 0x7f709003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 4783+0+0 (secure 0 0 0) 0x7f70940e9e90 con 0x7f70a4068db0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70a37fe640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 0x7f709003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.077+0000 7f70a37fe640 10 monclient: get_auth_request con 0x7f709003bb60 auth_method 0 2024-09-17T15:31:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.081+0000 7f70a37fe640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 0x7f709003e020 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f70980093d0 tx=0x7f7098009360 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.081+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f70940c4070 con 0x7f70a4068db0 2024-09-17T15:31:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.081+0000 7f708bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:36.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.081+0000 7f708bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:36.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f70aa201640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:36.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f70aa201640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:36.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f70aa201640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:36.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f70a406d4f0 con 0x7f70a4068db0 2024-09-17T15:31:36.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f708bfff640 1 -- 172.21.15.86:0/2447816706 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f7094016110 con 0x7f70a4068db0 2024-09-17T15:31:36.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f708bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:36.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.297+0000 7f708bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:36.302 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 msgr2=0x7f709003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 0x7f709003e020 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f70980093d0 tx=0x7f7098009360 comp rx=0 tx=0).stop 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 10 monclient: shutdown 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 msgr2=0x7f70a4134fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f709401cd30 tx=0x7f709401acd0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 shutdown_connections 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f709003bb60 0x7f709003e020 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f70a4135ae0 0x7f70a41cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70a4068db0 0x7f70a4134fe0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 --2- 172.21.15.86:0/2447816706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70a40689d0 0x7f70a4134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 >> 172.21.15.86:0/2447816706 conn(0x7f70a40a6440 msgr2=0x7f70a413d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 shutdown_connections 2024-09-17T15:31:36.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:36.301+0000 7f70aa201640 1 -- 172.21.15.86:0/2447816706 wait complete. 2024-09-17T15:31:39.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:31:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: get_monmap_and_config 2024-09-17T15:31:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: build_initial_monmap 2024-09-17T15:31:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: monmap: 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:39.445145+0000 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:39.445145+0000 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:39.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 Processor -- start 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 -- start start 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: init 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-c con 0x7fa83c0a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa83c0a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-a con 0x7fa83c13d280 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83c0a6f40 con 0x7fa83c0a5450 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa83c0a65e0 con 0x7fa83c13d280 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa83c09c2a0 con 0x7fa83c0a5070 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa84158d640 10 monclient(hunting): authenticate will time out at 1232.410498s 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83affd640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 msgr2=0x7fa83c09cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83a7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa83c0a5450 0x7fa83c0a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53122/0 (socket says 172.21.15.86:53122) 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 learned_addr learned my addr 172.21.15.86:0/3785901127 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39342/0 (socket says 172.21.15.86:39342) 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request con 0x7fa83c13d280 auth_method 0 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83affd640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.441+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.445+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.86:6789/0 conn(0x7fa83c0a5450 legacy=0x7fa83c0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 msgr2=0x7fa83c09cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83affd640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request con 0x7fa83c13d280 auth_method 0 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:31:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:39.645+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:31:40.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 msgr2=0x7fa83c09cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:40.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83affd640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:31:40.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:40.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request con 0x7fa83c13d280 auth_method 0 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:40.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.049+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:31:40.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 msgr2=0x7fa83c09cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:40.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83affd640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request con 0x7fa83c13d280 auth_method 0 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:40.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:40.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:40.853+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): tick 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 msgr2=0x7fa83c09cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.203:3300/0 conn(0x7fa83c0a5070 0x7fa83c09cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 -- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 msgr2=0x7fa83c13f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:42.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 --2- 172.21.15.86:0/3785901127 >> v2:172.21.15.86:3300/0 conn(0x7fa83c13d280 0x7fa83c13f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.86:6789/0 conn(0x7fa83c0a5450 legacy=0x7fa83c0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient: _add_conns ranks=[1,2] 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa828003740 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fa828007410 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 -- 172.21.15.86:0/3785901127 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83c09c580 con 0x7fa828003740 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 1 -- 172.21.15.86:0/3785901127 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c0a65e0 con 0x7fa828007410 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa838ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.203:6789/0 conn(0x7fa828003740 legacy=0x7fa828005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request con 0x7fa828007410 auth_method 0 2024-09-17T15:31:42.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:42.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.441+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:42.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.203:6789/0 conn(0x7fa828003740 legacy=0x7fa828005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request con 0x7fa828007410 auth_method 0 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:42.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:42.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:42.645+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:31:43.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:43.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.203:6789/0 conn(0x7fa828003740 legacy=0x7fa828005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:43.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request con 0x7fa828007410 auth_method 0 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:31:43.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.049+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:31:43.502 INFO:tasks.rados.rados.0.smithi203.stdout:488: done (3 left) 2024-09-17T15:31:43.502 INFO:tasks.rados.rados.0.smithi203.stdout:489: write oid 36 current snap is 51 2024-09-17T15:31:43.502 INFO:tasks.rados.rados.0.smithi203.stdout:489: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-17T15:31:43.519 INFO:tasks.rados.rados.0.smithi203.stdout:489: writing smithi20328086-36 from 510979 to 1043857 tid 1 2024-09-17T15:31:43.541 INFO:tasks.rados.rados.0.smithi203.stdout:489: writing smithi20328086-36 from 1528482 to 2300878 tid 2 2024-09-17T15:31:43.556 INFO:tasks.rados.rados.0.smithi203.stdout:489: writing smithi20328086-36 from 3060800 to 3310979 tid 3 2024-09-17T15:31:43.557 INFO:tasks.rados.rados.0.smithi203.stdout:490: write oid 33 current snap is 51 2024-09-17T15:31:43.557 INFO:tasks.rados.rados.0.smithi203.stdout:490: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-17T15:31:43.571 INFO:tasks.rados.rados.0.smithi203.stdout:490: writing smithi20328086-33 from 527786 to 1252266 tid 1 2024-09-17T15:31:43.580 INFO:tasks.rados.rados.0.smithi203.stdout:490: writing smithi20328086-33 from 1703317 to 2219371 tid 2 2024-09-17T15:31:43.589 INFO:tasks.rados.rados.0.smithi203.stdout:490: writing smithi20328086-33 from 2804576 to 3239611 tid 3 2024-09-17T15:31:43.589 INFO:tasks.rados.rados.0.smithi203.stdout:491: read oid 10 snap -1 2024-09-17T15:31:43.590 INFO:tasks.rados.rados.0.smithi203.stdout:491: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-17T15:31:43.590 INFO:tasks.rados.rados.0.smithi203.stdout:492: write (excl) oid 47 current snap is 51 2024-09-17T15:31:43.590 INFO:tasks.rados.rados.0.smithi203.stdout:492: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-17T15:31:43.600 INFO:tasks.rados.rados.0.smithi203.stdout:492: writing smithi20328086-47 from 544593 to 1344322 tid 1 2024-09-17T15:31:43.607 INFO:tasks.rados.rados.0.smithi203.stdout:492: writing smithi20328086-47 from 1761799 to 2537864 tid 2 2024-09-17T15:31:43.613 INFO:tasks.rados.rados.0.smithi203.stdout:492: writing smithi20328086-47 from 2948352 to 3344593 tid 3 2024-09-17T15:31:43.613 INFO:tasks.rados.rados.0.smithi203.stdout:493: read oid 41 snap -1 2024-09-17T15:31:43.613 INFO:tasks.rados.rados.0.smithi203.stdout:493: expect deleted 2024-09-17T15:31:43.613 INFO:tasks.rados.rados.0.smithi203.stdout:494: rollback oid 48 current snap is 51 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 47 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:483: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:483: finishing copy_from to smithi20328086-29 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 67 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:485: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:485: finishing copy_from to smithi20328086-46 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 88 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: finishing write tid 1 to smithi20328086-36 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: finishing write tid 2 to smithi20328086-36 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: finishing write tid 3 to smithi20328086-36 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: finishing write tid 4 to smithi20328086-36 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: finishing write tid 5 to smithi20328086-36 2024-09-17T15:31:43.614 INFO:tasks.rados.rados.0.smithi203.stdout:489: oid 36 updating version 0 to 176 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:489: oid 36 updating version 176 to 177 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:489: oid 36 updating version 177 to 178 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:489: oid 36 updating version 178 to 179 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 179 (ObjNum 197 snap 51 seq_num 197) dirty exists 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:489: left oid 36 (ObjNum 197 snap 51 seq_num 197) 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:490: finishing write tid 1 to smithi20328086-33 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:490: finishing write tid 2 to smithi20328086-33 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:490: finishing write tid 3 to smithi20328086-33 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:490: finishing write tid 4 to smithi20328086-33 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:483: done (8 left) 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:485: done (7 left) 2024-09-17T15:31:43.615 INFO:tasks.rados.rados.0.smithi203.stdout:486: done (6 left) 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:489: done (5 left) 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:495: write oid 17 current snap is 51 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: finishing write tid 5 to smithi20328086-33 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: oid 33 updating version 0 to 79 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: oid 33 version 79 is already newer than 78 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: oid 33 updating version 79 to 80 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: oid 33 version 80 is already newer than 77 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 80 (ObjNum 198 snap 51 seq_num 198) dirty exists 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:490: left oid 33 (ObjNum 198 snap 51 seq_num 198) 2024-09-17T15:31:43.616 INFO:tasks.rados.rados.0.smithi203.stdout:491: expect (ObjNum 124 snap 26 seq_num 124) 2024-09-17T15:31:43.677 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2034674 2024-09-17T15:31:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:492: finishing write tid 1 to smithi20328086-47 2024-09-17T15:31:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:492: finishing write tid 2 to smithi20328086-47 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: finishing write tid 3 to smithi20328086-47 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: finishing write tid 4 to smithi20328086-47 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: finishing write tid 5 to smithi20328086-47 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: oid 47 updating version 0 to 86 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: oid 47 version 86 is already newer than 85 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: oid 47 updating version 86 to 87 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: oid 47 version 87 is already newer than 84 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 87 (ObjNum 199 snap 51 seq_num 199) dirty exists 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:492: left oid 47 (ObjNum 199 snap 51 seq_num 199) 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:494: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:494: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:31:43.753 INFO:tasks.rados.rados.0.smithi203.stdout:495: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-17T15:31:43.758 INFO:tasks.rados.rados.0.smithi203.stdout:495: writing smithi20328086-17 from 561400 to 1036378 tid 1 2024-09-17T15:31:43.766 INFO:tasks.rados.rados.0.smithi203.stdout:495: writing smithi20328086-17 from 1820281 to 2340004 tid 2 2024-09-17T15:31:43.775 INFO:tasks.rados.rados.0.smithi203.stdout:495: writing smithi20328086-17 from 2859422 to 3317001 tid 3 2024-09-17T15:31:43.775 INFO:tasks.rados.rados.0.smithi203.stdout:490: done (5 left) 2024-09-17T15:31:43.775 INFO:tasks.rados.rados.0.smithi203.stdout:491: done (4 left) 2024-09-17T15:31:43.776 INFO:tasks.rados.rados.0.smithi203.stdout:492: done (3 left) 2024-09-17T15:31:43.776 INFO:tasks.rados.rados.0.smithi203.stdout:493: done (2 left) 2024-09-17T15:31:43.776 INFO:tasks.rados.rados.0.smithi203.stdout:496: write (excl) oid 10 current snap is 51 2024-09-17T15:31:43.776 INFO:tasks.rados.rados.0.smithi203.stdout:496: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-17T15:31:43.786 INFO:tasks.rados.rados.0.smithi203.stdout:496: writing smithi20328086-10 from 578207 to 1128434 tid 1 2024-09-17T15:31:43.798 INFO:tasks.rados.rados.0.smithi203.stdout:496: writing smithi20328086-10 from 1762410 to 2542144 tid 2 2024-09-17T15:31:43.805 INFO:tasks.rados.rados.0.smithi203.stdout:496: writing smithi20328086-10 from 3286845 to 3378207 tid 3 2024-09-17T15:31:43.806 INFO:tasks.rados.rados.0.smithi203.stdout:497: write (excl) oid 31 current snap is 51 2024-09-17T15:31:43.806 INFO:tasks.rados.rados.0.smithi203.stdout:497: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-17T15:31:43.814 INFO:tasks.rados.rados.0.smithi203.stdout:497: writing smithi20328086-31 from 595014 to 1220490 tid 1 2024-09-17T15:31:43.822 INFO:tasks.rados.rados.0.smithi203.stdout:497: writing smithi20328086-31 from 1820892 to 2344284 tid 2 2024-09-17T15:31:43.828 INFO:tasks.rados.rados.0.smithi203.stdout:497: writing smithi20328086-31 from 2797915 to 3394391 tid 3 2024-09-17T15:31:43.828 INFO:tasks.rados.rados.0.smithi203.stdout:498: write oid 2 current snap is 51 2024-09-17T15:31:43.828 INFO:tasks.rados.rados.0.smithi203.stdout:498: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-17T15:31:43.836 INFO:tasks.rados.rados.0.smithi203.stdout:498: writing smithi20328086-2 from 611821 to 1312546 tid 1 2024-09-17T15:31:43.844 INFO:tasks.rados.rados.0.smithi203.stdout:498: writing smithi20328086-2 from 1879374 to 2662777 tid 2 2024-09-17T15:31:43.850 INFO:tasks.rados.rados.0.smithi203.stdout:498: writing smithi20328086-2 from 3341691 to 3411821 tid 3 2024-09-17T15:31:43.850 INFO:tasks.rados.rados.0.smithi203.stdout:499: delete oid 8 current snap is 51 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:494: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 111 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: finishing write tid 1 to smithi20328086-17 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: finishing write tid 2 to smithi20328086-17 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: finishing write tid 3 to smithi20328086-17 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: finishing write tid 4 to smithi20328086-17 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: finishing write tid 5 to smithi20328086-17 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: oid 17 updating version 0 to 142 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: oid 17 version 142 is already newer than 141 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: oid 17 version 142 is already newer than 139 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: oid 17 version 142 is already newer than 140 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 142 (ObjNum 200 snap 51 seq_num 200) dirty exists 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:495: left oid 17 (ObjNum 200 snap 51 seq_num 200) 2024-09-17T15:31:43.851 INFO:tasks.rados.rados.0.smithi203.stdout:496: finishing write tid 1 to smithi20328086-10 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: finishing write tid 2 to smithi20328086-10 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: finishing write tid 3 to smithi20328086-10 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: finishing write tid 4 to smithi20328086-10 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: finishing write tid 5 to smithi20328086-10 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: oid 10 updating version 0 to 79 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: oid 10 version 79 is already newer than 78 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: oid 10 version 79 is already newer than 76 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: oid 10 version 79 is already newer than 77 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 79 (ObjNum 201 snap 51 seq_num 201) dirty exists 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:496: left oid 10 (ObjNum 201 snap 51 seq_num 201) 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:497: finishing write tid 1 to smithi20328086-31 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:497: finishing write tid 2 to smithi20328086-31 2024-09-17T15:31:43.852 INFO:tasks.rados.rados.0.smithi203.stdout:497: finishing write tid 3 to smithi20328086-31 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: finishing write tid 4 to smithi20328086-31 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: finishing write tid 5 to smithi20328086-31 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: oid 31 updating version 0 to 48 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: oid 31 updating version 48 to 51 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: oid 31 version 51 is already newer than 50 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: oid 31 version 51 is already newer than 49 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 51 (ObjNum 202 snap 51 seq_num 202) dirty exists 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:497: left oid 31 (ObjNum 202 snap 51 seq_num 202) 2024-09-17T15:31:43.853 INFO:tasks.rados.rados.0.smithi203.stdout:498: finishing write tid 1 to smithi20328086-2 2024-09-17T15:31:43.854 INFO:tasks.rados.rados.0.smithi203.stdout:498: finishing write tid 2 to smithi20328086-2 2024-09-17T15:31:43.854 INFO:tasks.rados.rados.0.smithi203.stdout:498: finishing write tid 3 to smithi20328086-2 2024-09-17T15:31:43.854 INFO:tasks.rados.rados.0.smithi203.stdout:498: finishing write tid 4 to smithi20328086-2 2024-09-17T15:31:43.855 INFO:tasks.rados.rados.0.smithi203.stdout:498: finishing write tid 5 to smithi20328086-2 2024-09-17T15:31:43.855 INFO:tasks.rados.rados.0.smithi203.stdout:498: oid 2 updating version 0 to 87 2024-09-17T15:31:43.855 INFO:tasks.rados.rados.0.smithi203.stdout:498: oid 2 updating version 87 to 88 2024-09-17T15:31:43.855 INFO:tasks.rados.rados.0.smithi203.stdout:498: oid 2 version 88 is already newer than 85 2024-09-17T15:31:43.856 INFO:tasks.rados.rados.0.smithi203.stdout:498: oid 2 version 88 is already newer than 86 2024-09-17T15:31:43.856 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 88 (ObjNum 203 snap 51 seq_num 203) dirty exists 2024-09-17T15:31:43.856 INFO:tasks.rados.rados.0.smithi203.stdout:498: left oid 2 (ObjNum 203 snap 51 seq_num 203) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:494: done (5 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:495: done (4 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:496: done (3 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:497: done (2 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:498: done (1 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:499: done (0 left) 2024-09-17T15:31:43.857 INFO:tasks.rados.rados.0.smithi203.stdout:500: write (excl) oid 8 current snap is 51 2024-09-17T15:31:43.858 INFO:tasks.rados.rados.0.smithi203.stdout:500: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request con 0x7fa828007410 auth_method 0 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:43.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83affd640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.203:6789/0 conn(0x7fa828003740 legacy=0x7fa828005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_done global_id 44168 payload 274 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> v1:172.21.15.203:6789/0 conn(0x7fa828003740 legacy=0x7fa828005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 1 monclient: found mon.noname-b 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:31:43.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.852+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82800b110 con 0x7fa828007410 2024-09-17T15:31:43.862 INFO:tasks.rados.rados.0.smithi203.stdout:500: writing smithi20328086-8 from 645435 to 1096658 tid 1 2024-09-17T15:31:43.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa83a7fc640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:43.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa830005b60 tx=0x7fa83001cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1394608696708e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:43.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 1 -- 172.21.15.86:0/3785901127 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8300ab070 con 0x7fa828007410 2024-09-17T15:31:43.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:43.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:31:43.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: dump: 2024-09-17T15:31:43.865 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: _finish_auth 0 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 1 -- 172.21.15.86:0/3785901127 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8300a59d0 con 0x7fa828007410 2024-09-17T15:31:43.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 5 monclient: authenticate success, global_id 44168 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 4 set_mon_vals no callback set 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 1 -- 172.21.15.86:0/3785901127 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8300a5dc0 con 0x7fa828007410 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa839ffb640 10 monclient: dump: 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:43.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 10 monclient: shutdown 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 1 -- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 msgr2=0x7fa8280098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 1 --2- 172.21.15.86:0/3785901127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa828007410 0x7fa8280098d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa830005b60 tx=0x7fa83001cd10 comp rx=0 tx=0).stop 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa83b7fe640 1 -- 172.21.15.86:0/3785901127 reap_dead start 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 1 -- 172.21.15.86:0/3785901127 shutdown_connections 2024-09-17T15:31:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.860+0000 7fa84158d640 1 -- 172.21.15.86:0/3785901127 >> 172.21.15.86:0/3785901127 conn(0x7fa83c0742a0 msgr2=0x7fa83c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- 172.21.15.86:0/3785901127 shutdown_connections 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- 172.21.15.86:0/3785901127 wait complete. 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient: build_initial_monmap 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient: monmap: 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 Processor -- start 2024-09-17T15:31:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- start start 2024-09-17T15:31:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient: init 2024-09-17T15:31:43.871 INFO:tasks.rados.rados.0.smithi203.stdout:500: writing smithi20328086-8 from 1879985 to 2667057 tid 2 2024-09-17T15:31:43.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:43.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:31:43.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:43.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-c con 0x7fa83c1213a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:43.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83c138210 0x7fa83c1cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-a con 0x7fa83c138210 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 0x7fa83c1cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): picked mon.noname-b con 0x7fa83c138690 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c1425e0 con 0x7fa83c138210 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c0a6a70 con 0x7fa83c138690 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c142760 con 0x7fa83c1213a0 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa84158d640 10 monclient(hunting): authenticate will time out at 1236.835224s 2024-09-17T15:31:43.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83b7fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 msgr2=0x7fa83c1cffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53214/0 (socket says 172.21.15.86:53214) 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 -- 172.21.15.86:0/828907632 learned_addr learned my addr 172.21.15.86:0/828907632 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83c138210 0x7fa83c1cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): get_auth_request con 0x7fa83c1213a0 auth_method 0 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83b7fe640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 0x7fa83c1cffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request con 0x7fa83c138210 auth_method 0 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:43.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient(hunting): handle_auth_done global_id 44171 payload 824 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 -- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 msgr2=0x7fa83c1cffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 0x7fa83c1cffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 -- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83c138210 msgr2=0x7fa83c1cfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83c138210 0x7fa83c1cfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient: _finish_hunting 0 2024-09-17T15:31:43.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 monclient: found mon.noname-c 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.864+0000 7fa83affd640 1 -- 172.21.15.86:0/828907632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c06f0c0 con 0x7fa83c1213a0 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa83affd640 1 -- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 msgr2=0x7fa83c09f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa83affd640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00b950 tx=0x7fa82c00bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00cd40 con 0x7fa83c1213a0 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: dump: 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.878 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: _finish_auth 0 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: _check_auth_tickets 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c010030 con 0x7fa83c1213a0 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 5 monclient: authenticate success, global_id 44171 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 10 monclient: _renew_subs 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:43.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c0a1860 con 0x7fa83c1213a0 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c015660 con 0x7fa83c1213a0 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8237fe640 10 monclient: dump: 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 10 monclient: _renew_subs 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c122ba0 con 0x7fa83c1213a0 2024-09-17T15:31:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.868+0000 7fa8397fa640 4 set_mon_vals no callback set 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa84158d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa84158d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa84158d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c005090 con 0x7fa83c1213a0 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa82c0143d0 con 0x7fa83c1213a0 2024-09-17T15:31:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa8237fe640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 0x7fa82803f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(124..124 src has 1..124) ==== 4799+0+0 (secure 0 0 0) 0x7fa82c04f920 con 0x7fa83c1213a0 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 0x7fa82803f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.872+0000 7fa83a7fc640 10 monclient: get_auth_request con 0x7fa82803cb90 auth_method 0 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.876+0000 7fa83a7fc640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 0x7fa82803f050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8300ad000 tx=0x7fa830010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.876+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa82c01a030 con 0x7fa83c1213a0 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.876+0000 7fa8237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:43.876+0000 7fa8237fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:500: writing smithi20328086-8 from 3280184 to 3445435 tid 3 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:501: rollback oid 9 current snap is 51 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 47 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:500: finishing write tid 1 to smithi20328086-8 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:502: copy_from oid 24 from oid 26 current snap is 51 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:503: copy_from oid 27 from oid 19 current snap is 51 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:504: delete oid 41 current snap is 51 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:504: done (4 left) 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:505: read oid 43 snap -1 2024-09-17T15:31:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:505: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:506: delete oid 45 current snap is 51 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:500: finishing write tid 2 to smithi20328086-8 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:500: finishing write tid 3 to smithi20328086-8 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:501: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:506: done (5 left) 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:507: read oid 25 snap -1 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:507: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:508: rollback oid 34 current snap is 51 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 47 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:509: write oid 18 current snap is 51 2024-09-17T15:31:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:509: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-17T15:31:43.893 INFO:tasks.rados.rados.0.smithi203.stdout:509: writing smithi20328086-18 from 695856 to 1489179 tid 1 2024-09-17T15:31:43.901 INFO:tasks.rados.rados.0.smithi203.stdout:509: writing smithi20328086-18 from 2171784 to 2706183 tid 2 2024-09-17T15:31:43.906 INFO:tasks.rados.rados.0.smithi203.stdout:509: writing smithi20328086-18 from 3362453 to 3495856 tid 3 2024-09-17T15:31:43.906 INFO:tasks.rados.rados.0.smithi203.stdout:510: write oid 36 current snap is 51 2024-09-17T15:31:43.906 INFO:tasks.rados.rados.0.smithi203.stdout:510: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-17T15:31:43.913 INFO:tasks.rados.rados.0.smithi203.stdout:510: writing smithi20328086-36 from 712663 to 1181235 tid 1 2024-09-17T15:31:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:510: writing smithi20328086-36 from 1713913 to 2391970 tid 2 2024-09-17T15:31:43.926 INFO:tasks.rados.rados.0.smithi203.stdout:510: writing smithi20328086-36 from 2873523 to 3465256 tid 3 2024-09-17T15:31:43.926 INFO:tasks.rados.rados.0.smithi203.stdout:511: delete oid 35 current snap is 51 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:501: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:501: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 98 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: finishing write tid 4 to smithi20328086-8 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: finishing write tid 5 to smithi20328086-8 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: oid 8 updating version 0 to 93 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: oid 8 version 93 is already newer than 92 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: oid 8 version 93 is already newer than 91 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: oid 8 version 93 is already newer than 90 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 93 (ObjNum 205 snap 51 seq_num 205) dirty exists 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:500: left oid 8 (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:508: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:505: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:31:43.927 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa84158d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:31:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa84158d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa84158d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:31:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa81c005380 con 0x7fa83c1213a0 2024-09-17T15:31:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa8237fe640 1 -- 172.21.15.86:0/828907632 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fa82c0148a0 con 0x7fa83c1213a0 2024-09-17T15:31:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa8237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:31:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.124+0000 7fa8237fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:44.130 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:31:44.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 msgr2=0x7fa82803f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 0x7fa82803f050 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8300ad000 tx=0x7fa830010040 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 10 monclient: shutdown 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 msgr2=0x7fa83c09f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00b950 tx=0x7fa82c00bd50 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 shutdown_connections 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82803cb90 0x7fa82803f050 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa83c138690 0x7fa83c1cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83c138210 0x7fa83c1cfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 --2- 172.21.15.86:0/828907632 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83c1213a0 0x7fa83c09f920 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 >> 172.21.15.86:0/828907632 conn(0x7fa83c0742a0 msgr2=0x7fa83c0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:44.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 shutdown_connections 2024-09-17T15:31:44.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.128+0000 7fa84158d640 1 -- 172.21.15.86:0/828907632 wait complete. 2024-09-17T15:31:44.148 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:31:44.148 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:31:44.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:31:44.183 INFO:tasks.rados.rados.0.smithi203.stdout:508: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:31:44.183 INFO:tasks.rados.rados.0.smithi203.stdout:507: expect (ObjNum 153 snap 35 seq_num 153) 2024-09-17T15:31:44.183 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:44.263 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1515078 2024-09-17T15:31:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: get_monmap_and_config 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: build_initial_monmap 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: monmap: 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:44.268523+0000 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:44.268523+0000 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:44.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 Processor -- start 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 -- start start 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: init 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d200a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d200a1760 addr v1:172.21.15.86:6789/0 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d2013c630 0x7f4d2013ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4d2013c630 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d200a2130 con 0x7f4d200a1760 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d200a26f0 con 0x7f4d200a0c70 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4d200a2410 con 0x7f4d2013c630 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d257e2640 10 monclient(hunting): authenticate will time out at 1237.234093s 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d1effd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f4d200a0c70 legacy=0x7f4d200a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d1f7fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f4d2013c630 msgr2=0x7f4d2013ea20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d1f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4d2013c630 0x7f4d2013ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d1e7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4d200a1760 0x7f4d2013af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53138/0 (socket says 172.21.15.86:53138) 2024-09-17T15:31:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.264+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3269637331 learned_addr learned my addr 172.21.15.86:0/3269637331 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3269637331 >> v1:172.21.15.86:6789/0 conn(0x7f4d200a1760 legacy=0x7f4d2013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592983951 0 0) 0x7f4d200a2130 con 0x7f4d200a1760 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): my global_id is 4704 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d0c003530 con 0x7f4d200a1760 2024-09-17T15:31:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2171009768 0 0) 0x7f4d0c003530 con 0x7f4d200a1760 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 >> v2:172.21.15.203:3300/0 conn(0x7f4d2013c630 msgr2=0x7f4d2013ea20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 --2- 172.21.15.86:0/3269637331 >> v2:172.21.15.203:3300/0 conn(0x7f4d2013c630 0x7f4d2013ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 >> v1:172.21.15.203:6789/0 conn(0x7f4d200a0c70 legacy=0x7f4d200a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2009f9f0 con 0x7f4d200a1760 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _finish_auth 0 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f4d080019b0 con 0x7f4d200a1760 2024-09-17T15:31:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d257e2640 5 monclient: authenticate success, global_id 4704 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d257e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: dump: 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 >> v1:172.21.15.86:6789/0 conn(0x7f4d200a1760 legacy=0x7f4d2013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d0c003a30 0x7f4d0c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): picked mon.b con 0x7f4d0c003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c004810 0x7f4d0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): picked mon.c con 0x7f4d0c004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): picked mon.a con 0x7f4d0c0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200a2410 con 0x7f4d0c0099c0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0c00c3a0 con 0x7f4d0c003a30 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4d0c00c5a0 con 0x7f4d0c004810 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1dffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d257e2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1effd640 1 -- 172.21.15.86:0/3269637331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d0c003a30 msgr2=0x7f4d0c003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1effd640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d0c003a30 0x7f4d0c003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c004810 0x7f4d0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): get_auth_request con 0x7f4d0c0099c0 auth_method 0 2024-09-17T15:31:44.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): get_auth_request con 0x7f4d0c004810 auth_method 0 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient(hunting): handle_auth_done global_id 4704 payload 293 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c004810 msgr2=0x7f4d0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c004810 0x7f4d0c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3269637331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d0c003a30 msgr2=0x7f4d0c003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d0c003a30 0x7f4d0c003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 monclient: found mon.a 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3269637331 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d0c003530 con 0x7f4d0c0099c0 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:31:44.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.268+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3269637331 reap_dead start 2024-09-17T15:31:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 msgr2=0x7f4d0c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d18003940 tx=0x7f4d1801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce40309bf7714525 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18009520 con 0x7f4d0c0099c0 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 10 monclient: dump: 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d18003940 tx=0x7f4d1801c4b0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f4d18005070 mon_map magic: 0 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d18003bc0 con 0x7f4d0c0099c0 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 4 set_mon_vals no callback set 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 10 monclient: shutdown 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 -- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 msgr2=0x7f4d0c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4d18003940 tx=0x7f4d1801c4b0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 1 -- 172.21.15.86:0/3269637331 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f4d18005070 con 0x7f4d0c0099c0 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d1dffb640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 -- 172.21.15.86:0/3269637331 shutdown_connections 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3269637331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d0c0099c0 0x7f4d0c00bdb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 -- 172.21.15.86:0/3269637331 >> 172.21.15.86:0/3269637331 conn(0x7f4d200a2f40 msgr2=0x7f4d200a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 -- 172.21.15.86:0/3269637331 shutdown_connections 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 -- 172.21.15.86:0/3269637331 wait complete. 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 10 monclient: build_initial_monmap 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 10 monclient: monmap: 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.272+0000 7f4d257e2640 1 Processor -- start 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- start start 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: init 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c0099c0 0x7f4d2013af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-c con 0x7f4d0c0099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 0x7f4d201363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4d20135f50 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d201cd6e0 0x7f4d201cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4d201cd6e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200a11d0 con 0x7f4d20135f50 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200a1050 con 0x7f4d201cd6e0 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4d200a1350 con 0x7f4d0c0099c0 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c0099c0 0x7f4d2013af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1f7fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d201cd6e0 msgr2=0x7f4d201cdb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c0099c0 0x7f4d2013af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53236/0 (socket says 172.21.15.86:53236) 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 0x7f4d201363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 1 -- 172.21.15.86:0/3472660740 learned_addr learned my addr 172.21.15.86:0/3472660740 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d201cd6e0 0x7f4d201cdb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient(hunting): authenticate will time out at 1237.244850s 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): get_auth_request con 0x7f4d0c0099c0 auth_method 0 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): get_auth_request con 0x7f4d20135f50 auth_method 0 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient(hunting): handle_auth_done global_id 4707 payload 824 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c0099c0 msgr2=0x7f4d2013af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d0c0099c0 0x7f4d2013af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d201cd6e0 msgr2=0x7f4d201cdb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d201cd6e0 0x7f4d201cdb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 monclient: found mon.noname-a 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2012e130 con 0x7f4d20135f50 2024-09-17T15:31:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 msgr2=0x7f4d201363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 0x7f4d201363d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d08001bd0 tx=0x7f4d08007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1e7fc640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 0x7f4d201363d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d08001bd0 tx=0x7f4d08007720 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f4d08007e50 mon_map magic: 0 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08007e50 con 0x7f4d20135f50 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: dump: 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: _finish_auth 0 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: _check_auth_tickets 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d08018070 con 0x7f4d20135f50 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d177fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 5 monclient: authenticate success, global_id 4707 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _renew_subs 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d1d7fa640 4 set_mon_vals no callback set 2024-09-17T15:31:44.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d201309e0 con 0x7f4d20135f50 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _renew_subs 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.276+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d20130140 con 0x7f4d20135f50 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d257e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 0x7f4d201ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d200a2130 con 0x7f4d20130310 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 0x7f4d201ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient: get_auth_request con 0x7f4d20130310 auth_method 0 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0800fe00 con 0x7f4d20135f50 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: dump: 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.297 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4d0801d460 con 0x7f4d20135f50 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 10 monclient(hunting): handle_auth_done global_id 4710 payload 824 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d2012f1e0 con 0x7f4d20130310 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d0003bce0 0x7f4d0003e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d0003bce0 0x7f4d0003e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1f7fe640 10 monclient: get_auth_request con 0x7f4d0003bce0 auth_method 0 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 msgr2=0x7f4d201ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d1effd640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 0x7f4d201ce0d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4d10007f60 tx=0x7f4d10007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d100054e0 con 0x7f4d20130310 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: dump: 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4d10018070 con 0x7f4d20130310 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 msgr2=0x7f4d201ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 0x7f4d201ce0d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4d10007f60 tx=0x7f4d10007f90 comp rx=0 tx=0).stop 2024-09-17T15:31:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 msgr2=0x7f4d201ce0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.280+0000 7f4d177fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20130310 0x7f4d201ce0d0 secure :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4d10007f60 tx=0x7f4d10007f90 comp rx=0 tx=0).stop 2024-09-17T15:31:44.301 INFO:tasks.rados.rados.0.smithi203.stdout:508: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:31:44.301 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 70 (ObjNum 158 snap 36 seq_num 158) dirty exists 2024-09-17T15:31:44.301 INFO:tasks.rados.rados.0.smithi203.stdout:503: finishing copy_from racing read to smithi20328086-27 2024-09-17T15:31:44.301 INFO:tasks.rados.rados.0.smithi203.stdout:503: finishing copy_from to smithi20328086-27 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 100 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: finishing write tid 1 to smithi20328086-18 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:502: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:502: finishing copy_from to smithi20328086-24 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 132 (ObjNum 133 snap 29 seq_num 133) dirty exists 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: finishing write tid 2 to smithi20328086-18 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: finishing write tid 3 to smithi20328086-18 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: finishing write tid 4 to smithi20328086-18 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: finishing write tid 5 to smithi20328086-18 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: oid 18 updating version 0 to 97 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: oid 18 version 97 is already newer than 95 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: oid 18 updating version 97 to 98 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:509: oid 18 version 98 is already newer than 96 2024-09-17T15:31:44.302 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 98 (ObjNum 208 snap 51 seq_num 208) dirty exists 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:509: left oid 18 (ObjNum 208 snap 51 seq_num 208) 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: finishing write tid 1 to smithi20328086-36 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: finishing write tid 2 to smithi20328086-36 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: finishing write tid 3 to smithi20328086-36 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: finishing write tid 4 to smithi20328086-36 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: finishing write tid 5 to smithi20328086-36 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: oid 36 updating version 0 to 182 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: oid 36 updating version 182 to 183 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: oid 36 version 183 is already newer than 180 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: oid 36 version 183 is already newer than 181 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 183 (ObjNum 209 snap 51 seq_num 209) dirty exists 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:510: left oid 36 (ObjNum 209 snap 51 seq_num 209) 2024-09-17T15:31:44.303 INFO:tasks.rados.rados.0.smithi203.stdout:500: done (9 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:501: done (8 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:502: done (7 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:503: done (6 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:505: done (5 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:507: done (4 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:508: done (3 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:509: done (2 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:510: done (1 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:511: done (0 left) 2024-09-17T15:31:44.304 INFO:tasks.rados.rados.0.smithi203.stdout:512: snap_create 2024-09-17T15:31:44.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.284+0000 7f4d1f7fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d0003bce0 0x7f4d0003e1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4d200a1ba0 tx=0x7f4d18005a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d257e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 0x7f4d20137300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d257e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4d20136910 con 0x7f4d20136ef0 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 0x7f4d20137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient: get_auth_request con 0x7f4d20136ef0 auth_method 0 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 10 monclient(hunting): handle_auth_done global_id 4713 payload 824 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.288+0000 7f4d1effd640 1 -- 172.21.15.86:0/3472660740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4d201d8a90 con 0x7f4d20136ef0 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d1effd640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 msgr2=0x7f4d20137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d1effd640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 0x7f4d20137300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4d10015040 tx=0x7f4d10005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10013600 con 0x7f4d20136ef0 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 10 monclient: dump: 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.307 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f4d10013600 con 0x7f4d20136ef0 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 msgr2=0x7f4d20137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 0x7f4d20137300 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4d10015040 tx=0x7f4d10005d60 comp rx=0 tx=0).stop 2024-09-17T15:31:44.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 msgr2=0x7f4d20137300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 54, 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 276936, 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:44.315 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.316 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:44.317 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:44.318 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.320 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.321 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:44.322 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:44.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20136ef0 0x7f4d20137300 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4d10015040 tx=0x7f4d10005d60 comp rx=0 tx=0).stop 2024-09-17T15:31:44.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d0003bce0 msgr2=0x7f4d0003e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4d0003bce0 0x7f4d0003e1a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4d200a1ba0 tx=0x7f4d18005a90 comp rx=0 tx=0).stop 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 10 monclient: shutdown 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 msgr2=0x7f4d201363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 --2- 172.21.15.86:0/3472660740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d20135f50 0x7f4d201363d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4d08001bd0 tx=0x7f4d08007720 comp rx=0 tx=0).stop 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3472660740 reap_dead start 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d177fe640 1 -- 172.21.15.86:0/3472660740 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4799+0+0 (unknown 0 0 0) 0x7f4d0804fe00 con 0x7f4d20135f50 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d1f7fe640 1 -- 172.21.15.86:0/3472660740 reap_dead start 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 shutdown_connections 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 >> 172.21.15.86:0/3472660740 conn(0x7f4d200a2f40 msgr2=0x7f4d200a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 shutdown_connections 2024-09-17T15:31:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.292+0000 7f4d257e2640 1 -- 172.21.15.86:0/3472660740 wait complete. 2024-09-17T15:31:44.325 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:31:44.325 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:31:44.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.432+0000 7febb3fee640 10 monclient: get_monmap_and_config 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.432+0000 7febb3fee640 10 monclient: build_initial_monmap 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.432+0000 7febb3fee640 10 monclient: monmap: 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:31:44.439720+0000 2024-09-17T15:31:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:31:44.439720+0000 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:31:44.440 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 Processor -- start 2024-09-17T15:31:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 -- start start 2024-09-17T15:31:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient: init 2024-09-17T15:31:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febac0a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febac0a1760 0x7febac13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-a con 0x7febac0a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 0x7febac13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-c con 0x7febac13b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febac0a2130 con 0x7febac0a0c70 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7febac0a2410 con 0x7febac0a1760 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7febac0a26f0 con 0x7febac13b450 2024-09-17T15:31:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb3fee640 10 monclient(hunting): authenticate will time out at 1237.405358s 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1d8a640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7febac0a0c70 legacy=0x7febac0a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb258b640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 msgr2=0x7febac13d840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febac0a1760 0x7febac13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb258b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 0x7febac13d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febac0a1760 0x7febac13af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39428/0 (socket says 172.21.15.86:39428) 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 -- 172.21.15.86:0/2225821850 learned_addr learned my addr 172.21.15.86:0/2225821850 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): get_auth_request con 0x7febac0a1760 auth_method 0 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient(hunting): handle_auth_done global_id 4716 payload 274 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 -- 172.21.15.86:0/2225821850 >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 msgr2=0x7febac13d840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 --2- 172.21.15.86:0/2225821850 >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 0x7febac13d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 -- 172.21.15.86:0/2225821850 >> v1:172.21.15.203:6789/0 conn(0x7febac0a0c70 legacy=0x7febac0a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient: _finish_hunting 0 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 monclient: found mon.noname-a 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.436+0000 7febb1589640 1 -- 172.21.15.86:0/2225821850 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febac09f9f0 con 0x7febac0a1760 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb1589640 1 -- 172.21.15.86:0/2225821850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac0a1760 msgr2=0x7febac13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb1589640 1 --2- 172.21.15.86:0/2225821850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac0a1760 0x7febac13af10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7febac06f840 tx=0x7feba801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ef7adce31b0039 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 1 -- 172.21.15.86:0/2225821850 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba80b1070 con 0x7febac0a1760 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: dump: 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.449 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: _finish_auth 0 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: _check_auth_tickets 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 1 -- 172.21.15.86:0/2225821850 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba80a8050 con 0x7febac0a1760 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 5 monclient: authenticate success, global_id 4716 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:44.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 10 monclient: get_monmap_and_config success 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 1 -- 172.21.15.86:0/2225821850 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba80ac860 con 0x7febac0a1760 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 4 set_mon_vals no callback set 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb0d88640 10 monclient: dump: 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 10 monclient: shutdown 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 -- 172.21.15.86:0/2225821850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac0a1760 msgr2=0x7febac13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 --2- 172.21.15.86:0/2225821850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac0a1760 0x7febac13af10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7febac06f840 tx=0x7feba801c400 comp rx=0 tx=0).stop 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 -- 172.21.15.86:0/2225821850 shutdown_connections 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 --2- 172.21.15.86:0/2225821850 >> v2:172.21.15.203:3300/0 conn(0x7febac13b450 0x7febac13d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 --2- 172.21.15.86:0/2225821850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac0a1760 0x7febac13af10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 -- 172.21.15.86:0/2225821850 >> 172.21.15.86:0/2225821850 conn(0x7febac0a2f40 msgr2=0x7febac0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 -- 172.21.15.86:0/2225821850 shutdown_connections 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 1 -- 172.21.15.86:0/2225821850 wait complete. 2024-09-17T15:31:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 10 monclient: build_initial_monmap 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.440+0000 7febb3fee640 10 monclient: monmap: 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 Processor -- start 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- start start 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: init 2024-09-17T15:31:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _reopen_session rank -1 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febac0a0c70 0x7febac072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-b con 0x7febac0a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac0a1760 0x7febac0732d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-c con 0x7febac0a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 0x7febac1380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): picked mon.noname-a con 0x7febac13b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7febac140720 con 0x7febac13b450 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febac1405a0 con 0x7febac0a0c70 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7febac1408a0 con 0x7febac0a1760 2024-09-17T15:31:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): _renew_subs 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient(hunting): authenticate will time out at 1237.412755s 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1d8a640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febac0a0c70 msgr2=0x7febac072d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1d8a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febac0a0c70 0x7febac072d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac0a1760 0x7febac0732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 0x7febac1380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 0x7febac1380c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39440/0 (socket says 172.21.15.86:39440) 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): get_auth_request con 0x7febac0a1760 auth_method 0 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 learned_addr learned my addr 172.21.15.86:0/3001842837 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): get_auth_request con 0x7febac13b450 auth_method 0 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb1589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac0a1760 msgr2=0x7febac0732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac0a1760 0x7febac0732d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febac0a0c70 msgr2=0x7febac072d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febac0a0c70 0x7febac072d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient: _finish_hunting 0 2024-09-17T15:31:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 monclient: found mon.noname-a 2024-09-17T15:31:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:31:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febac070850 con 0x7febac13b450 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 msgr2=0x7febac1380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb258b640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 0x7febac1380c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feba400be50 tx=0x7feba400bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4016630 con 0x7febac13b450 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: dump: 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: _finish_auth 0 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba4010030 con 0x7febac13b450 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 5 monclient: authenticate success, global_id 4719 2024-09-17T15:31:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _renew_subs 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febac09d460 con 0x7febac13b450 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4016a50 con 0x7febac13b450 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba27fc640 10 monclient: dump: 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.462 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7feba3fff640 4 set_mon_vals no callback set 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _renew_subs 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.444+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febac070570 con 0x7febac13b450 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb3fee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb3fee640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 0x7febac02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7febac0701f0 con 0x7febac1cddd0 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 0x7febac02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient: get_auth_request con 0x7febac1cddd0 auth_method 0 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7feba40153e0 con 0x7febac13b450 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 10 monclient(hunting): handle_auth_done global_id 44180 payload 824 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febac0a1050 con 0x7febac1cddd0 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb9003bb40 0x7feb9003e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 4799+0+0 (secure 0 0 0) 0x7feba40509e0 con 0x7febac13b450 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1589640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb9003bb40 0x7feb9003e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1589640 10 monclient: get_auth_request con 0x7feb9003bb40 auth_method 0 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 msgr2=0x7febac02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7febb1d8a640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 0x7febac02d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c005270 con 0x7febac1cddd0 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 10 monclient: dump: 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb9c015070 con 0x7febac1cddd0 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 msgr2=0x7febac02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 0x7febac02d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c00b910 comp rx=0 tx=0).stop 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 msgr2=0x7febac02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.448+0000 7feba27fc640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac1cddd0 0x7febac02d450 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c00b910 comp rx=0 tx=0).stop 2024-09-17T15:31:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.452+0000 7febb1589640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb9003bb40 0x7feb9003e000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febac06d160 tx=0x7feba80a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb3fee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb3fee640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 0x7febac02dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb3fee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7febac1341c0 con 0x7febac02d990 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 0x7febac02dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient: get_auth_request con 0x7febac02d990 auth_method 0 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:31:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 10 monclient(hunting): handle_auth_done global_id 44183 payload 824 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.456+0000 7febb1d8a640 1 -- 172.21.15.86:0/3001842837 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febac06faf0 con 0x7febac02d990 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb1d8a640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 msgr2=0x7febac02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb1d8a640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 0x7febac02dda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c01ae60 con 0x7febac02d990 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 10 monclient: dump: 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.469 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4533 (secure 0 0 0) 0x7feb9c010480 con 0x7febac02d990 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 msgr2=0x7febac02dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 0x7febac02dda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c005a30 comp rx=0 tx=0).stop 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 msgr2=0x7febac02dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:31:44.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7feba27fc640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febac02d990 0x7febac02dda0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00b450 tx=0x7feb9c005a30 comp rx=0 tx=0).stop 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 54, 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 33415, 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:31:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:31:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:31:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:31:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:31:44.482 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:31:44.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:31:44.483 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:31:44.483 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb9003bb40 msgr2=0x7feb9003e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feb9003bb40 0x7feb9003e000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febac06d160 tx=0x7feba80a7000 comp rx=0 tx=0).stop 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 10 monclient: shutdown 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 msgr2=0x7febac1380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 --2- 172.21.15.86:0/3001842837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febac13b450 0x7febac1380c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feba400be50 tx=0x7feba400bf40 comp rx=0 tx=0).stop 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 reap_dead start 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb258b640 1 -- 172.21.15.86:0/3001842837 reap_dead start 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 shutdown_connections 2024-09-17T15:31:44.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 >> 172.21.15.86:0/3001842837 conn(0x7febac0a2f40 msgr2=0x7febac131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:31:44.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 shutdown_connections 2024-09-17T15:31:44.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:31:44.460+0000 7febb3fee640 1 -- 172.21.15.86:0/3001842837 wait complete. 2024-09-17T15:31:44.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:31:44.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:512: done (0 left) 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:513: copy_from oid 31 from oid 22 current snap is 52 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:514: rollback oid 13 current snap is 52 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 51 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:515: rollback oid 49 current snap is 52 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 47 2024-09-17T15:31:44.513 INFO:tasks.rados.rados.0.smithi203.stdout:516: delete oid 25 current snap is 52 2024-09-17T15:31:44.519 INFO:tasks.rados.rados.0.smithi203.stdout:516: done (3 left) 2024-09-17T15:31:44.519 INFO:tasks.rados.rados.0.smithi203.stdout:517: rollback oid 38 current snap is 52 2024-09-17T15:31:44.519 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 47 2024-09-17T15:31:44.519 INFO:tasks.rados.rados.0.smithi203.stdout:518: snap_create 2024-09-17T15:31:44.521 INFO:tasks.rados.rados.0.smithi203.stdout:515: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:31:44.521 INFO:tasks.rados.rados.0.smithi203.stdout:515: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:31:44.521 INFO:tasks.rados.rados.0.smithi203.stdout:515: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:31:44.521 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 187 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:31:44.522 INFO:tasks.rados.rados.0.smithi203.stdout:514: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:31:44.526 INFO:tasks.rados.rados.0.smithi203.stdout:514: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:31:44.526 INFO:tasks.rados.rados.0.smithi203.stdout:514: finishing rollback tid 2 to smithi20328086-13 2024-09-17T15:31:44.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 83 (ObjNum 172 snap 42 seq_num 172) dirty exists 2024-09-17T15:31:44.526 INFO:tasks.rados.rados.0.smithi203.stdout:517: finishing rollback tid 0 to smithi20328086-38 2024-09-17T15:31:44.527 INFO:tasks.rados.rados.0.smithi203.stdout:517: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:31:44.527 INFO:tasks.rados.rados.0.smithi203.stdout:517: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:31:44.527 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 87 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:31:44.539 INFO:tasks.rados.rados.0.smithi203.stdout:513: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:31:44.542 INFO:tasks.rados.rados.0.smithi203.stdout:513: finishing copy_from to smithi20328086-31 2024-09-17T15:31:44.542 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:513: done (4 left) 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:514: done (3 left) 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:515: done (2 left) 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:517: done (1 left) 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:518: done (0 left) 2024-09-17T15:31:45.517 INFO:tasks.rados.rados.0.smithi203.stdout:519: snap_remove snap 52 2024-09-17T15:31:46.530 INFO:tasks.rados.rados.0.smithi203.stdout:519: done (0 left) 2024-09-17T15:31:46.530 INFO:tasks.rados.rados.0.smithi203.stdout:520: write oid 31 current snap is 53 2024-09-17T15:31:46.530 INFO:tasks.rados.rados.0.smithi203.stdout:520: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-17T15:31:46.553 INFO:tasks.rados.rados.0.smithi203.stdout:520: writing smithi20328086-31 from 763084 to 1457403 tid 1 2024-09-17T15:31:46.569 INFO:tasks.rados.rados.0.smithi203.stdout:520: writing smithi20328086-31 from 1889359 to 2431096 tid 2 2024-09-17T15:31:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:520: writing smithi20328086-31 from 2955792 to 3497694 tid 3 2024-09-17T15:31:46.585 INFO:tasks.rados.rados.0.smithi203.stdout:521: snap_create 2024-09-17T15:31:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:520: finishing write tid 1 to smithi20328086-31 2024-09-17T15:31:46.586 INFO:tasks.rados.rados.0.smithi203.stdout:520: finishing write tid 2 to smithi20328086-31 2024-09-17T15:31:46.592 INFO:tasks.rados.rados.0.smithi203.stdout:520: finishing write tid 3 to smithi20328086-31 2024-09-17T15:31:46.593 INFO:tasks.rados.rados.0.smithi203.stdout:520: finishing write tid 4 to smithi20328086-31 2024-09-17T15:31:46.593 INFO:tasks.rados.rados.0.smithi203.stdout:520: finishing write tid 5 to smithi20328086-31 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:520: oid 31 updating version 0 to 55 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:520: oid 31 version 55 is already newer than 53 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:520: oid 31 version 55 is already newer than 54 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:520: oid 31 version 55 is already newer than 52 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 55 (ObjNum 212 snap 53 seq_num 212) dirty exists 2024-09-17T15:31:46.594 INFO:tasks.rados.rados.0.smithi203.stdout:520: left oid 31 (ObjNum 212 snap 53 seq_num 212) 2024-09-17T15:31:47.428 INFO:tasks.rados.rados.0.smithi203.stdout:520: done (1 left) 2024-09-17T15:31:47.429 INFO:tasks.rados.rados.0.smithi203.stdout:521: done (0 left) 2024-09-17T15:31:47.429 INFO:tasks.rados.rados.0.smithi203.stdout:522: snap_create 2024-09-17T15:31:48.431 INFO:tasks.rados.rados.0.smithi203.stdout:522: done (0 left) 2024-09-17T15:31:48.431 INFO:tasks.rados.rados.0.smithi203.stdout:523: delete oid 5 current snap is 55 2024-09-17T15:31:48.443 INFO:tasks.rados.rados.0.smithi203.stdout:523: done (0 left) 2024-09-17T15:31:48.443 INFO:tasks.rados.rados.0.smithi203.stdout:524: write oid 50 current snap is 55 2024-09-17T15:31:48.443 INFO:tasks.rados.rados.0.smithi203.stdout:524: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-17T15:31:48.465 INFO:tasks.rados.rados.0.smithi203.stdout:524: writing smithi20328086-50 from 796698 to 1357868 tid 1 2024-09-17T15:31:48.485 INFO:tasks.rados.rados.0.smithi203.stdout:524: writing smithi20328086-50 from 2006323 to 2551729 tid 2 2024-09-17T15:31:48.500 INFO:tasks.rados.rados.0.smithi203.stdout:524: writing smithi20328086-50 from 3126991 to 3596698 tid 3 2024-09-17T15:31:48.500 INFO:tasks.rados.rados.0.smithi203.stdout:525: read oid 45 snap -1 2024-09-17T15:31:48.500 INFO:tasks.rados.rados.0.smithi203.stdout:525: expect deleted 2024-09-17T15:31:48.500 INFO:tasks.rados.rados.0.smithi203.stdout:526: write oid 49 current snap is 55 2024-09-17T15:31:48.500 INFO:tasks.rados.rados.0.smithi203.stdout:524: finishing write tid 1 to smithi20328086-50 2024-09-17T15:31:48.501 INFO:tasks.rados.rados.0.smithi203.stdout:524: finishing write tid 2 to smithi20328086-50 2024-09-17T15:31:48.501 INFO:tasks.rados.rados.0.smithi203.stdout:526: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-17T15:31:48.512 INFO:tasks.rados.rados.0.smithi203.stdout:526: writing smithi20328086-49 from 413505 to 1049924 tid 1 2024-09-17T15:31:48.528 INFO:tasks.rados.rados.0.smithi203.stdout:526: writing smithi20328086-49 from 1664805 to 2353869 tid 2 2024-09-17T15:31:48.541 INFO:tasks.rados.rados.0.smithi203.stdout:526: writing smithi20328086-49 from 3038061 to 3530132 tid 3 2024-09-17T15:31:48.541 INFO:tasks.rados.rados.0.smithi203.stdout:527: write oid 18 current snap is 55 2024-09-17T15:31:48.541 INFO:tasks.rados.rados.0.smithi203.stdout:527: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-17T15:31:48.551 INFO:tasks.rados.rados.0.smithi203.stdout:527: writing smithi20328086-18 from 430312 to 1141980 tid 1 2024-09-17T15:31:48.560 INFO:tasks.rados.rados.0.smithi203.stdout:527: writing smithi20328086-18 from 1723287 to 2272362 tid 2 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:527: writing smithi20328086-18 from 2781837 to 3485180 tid 3 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:528: copy_from oid 28 from oid 12 current snap is 55 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: finishing write tid 3 to smithi20328086-50 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: finishing write tid 4 to smithi20328086-50 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: finishing write tid 5 to smithi20328086-50 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: oid 50 updating version 0 to 69 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: oid 50 version 69 is already newer than 67 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: oid 50 updating version 69 to 70 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: oid 50 version 70 is already newer than 68 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 70 (ObjNum 214 snap 55 seq_num 214) dirty exists 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: left oid 50 (ObjNum 214 snap 55 seq_num 214) 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:524: done (4 left) 2024-09-17T15:31:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:525: done (3 left) 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:529: delete oid 10 current snap is 55 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: finishing write tid 1 to smithi20328086-49 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: finishing write tid 2 to smithi20328086-49 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: finishing write tid 3 to smithi20328086-49 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: finishing write tid 4 to smithi20328086-49 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: finishing write tid 5 to smithi20328086-49 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: oid 49 updating version 0 to 191 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: oid 49 version 191 is already newer than 190 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: oid 49 version 191 is already newer than 189 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: oid 49 updating version 191 to 192 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 192 (ObjNum 215 snap 55 seq_num 215) dirty exists 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:526: left oid 49 (ObjNum 215 snap 55 seq_num 215) 2024-09-17T15:31:48.571 INFO:tasks.rados.rados.0.smithi203.stdout:527: finishing write tid 1 to smithi20328086-18 2024-09-17T15:31:48.572 INFO:tasks.rados.rados.0.smithi203.stdout:527: finishing write tid 2 to smithi20328086-18 2024-09-17T15:31:48.580 INFO:tasks.rados.rados.0.smithi203.stdout:527: finishing write tid 3 to smithi20328086-18 2024-09-17T15:31:48.583 INFO:tasks.rados.rados.0.smithi203.stdout:526: done (3 left) 2024-09-17T15:31:48.583 INFO:tasks.rados.rados.0.smithi203.stdout:529: done (2 left) 2024-09-17T15:31:48.583 INFO:tasks.rados.rados.0.smithi203.stdout:530: copy_from oid 7 from oid 29 current snap is 55 2024-09-17T15:31:48.583 INFO:tasks.rados.rados.0.smithi203.stdout:531: snap_remove snap 51 2024-09-17T15:31:49.547 INFO:tasks.rados.rados.0.smithi203.stdout:531: done (3 left) 2024-09-17T15:31:49.547 INFO:tasks.rados.rados.0.smithi203.stdout:532: write oid 30 current snap is 55 2024-09-17T15:31:49.547 INFO:tasks.rados.rados.0.smithi203.stdout:532: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-17T15:31:49.555 INFO:tasks.rados.rados.0.smithi203.stdout:532: writing smithi20328086-30 from 463926 to 926092 tid 1 2024-09-17T15:31:49.565 INFO:tasks.rados.rados.0.smithi203.stdout:532: writing smithi20328086-30 from 1723898 to 2160289 tid 2 2024-09-17T15:31:49.576 INFO:tasks.rados.rados.0.smithi203.stdout:532: writing smithi20328086-30 from 2603977 to 3329864 tid 3 2024-09-17T15:31:49.576 INFO:tasks.rados.rados.0.smithi203.stdout:533: read oid 43 snap -1 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:533: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:534: snap_create 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: finishing write tid 4 to smithi20328086-18 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: finishing write tid 5 to smithi20328086-18 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: oid 18 updating version 0 to 100 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: oid 18 updating version 100 to 103 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: oid 18 version 103 is already newer than 101 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: oid 18 version 103 is already newer than 102 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 103 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:527: left oid 18 (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:528: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:528: finishing copy_from to smithi20328086-28 2024-09-17T15:31:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 53 (ObjNum 175 snap 44 seq_num 175) dirty exists 2024-09-17T15:31:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:530: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:31:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:530: finishing copy_from to smithi20328086-7 2024-09-17T15:31:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 67 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:31:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:532: finishing write tid 1 to smithi20328086-30 2024-09-17T15:31:49.578 INFO:tasks.rados.rados.0.smithi203.stdout:532: finishing write tid 2 to smithi20328086-30 2024-09-17T15:31:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:533: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:31:49.765 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1414445 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: finishing write tid 3 to smithi20328086-30 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: finishing write tid 4 to smithi20328086-30 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: finishing write tid 5 to smithi20328086-30 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: oid 30 updating version 0 to 76 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: oid 30 version 76 is already newer than 75 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: oid 30 version 76 is already newer than 73 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:532: oid 30 version 76 is already newer than 74 2024-09-17T15:31:49.822 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 76 (ObjNum 218 snap 55 seq_num 218) dirty exists 2024-09-17T15:31:49.823 INFO:tasks.rados.rados.0.smithi203.stdout:532: left oid 30 (ObjNum 218 snap 55 seq_num 218) 2024-09-17T15:31:50.736 INFO:tasks.rados.rados.0.smithi203.stdout:527: done (5 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:528: done (4 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:530: done (3 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:532: done (2 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:533: done (1 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:534: done (0 left) 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:535: copy_from oid 12 from oid 22 current snap is 56 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:536: write oid 35 current snap is 56 2024-09-17T15:31:50.737 INFO:tasks.rados.rados.0.smithi203.stdout:536: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-17T15:31:50.747 INFO:tasks.rados.rados.0.smithi203.stdout:536: writing smithi20328086-35 from 480733 to 1018148 tid 1 2024-09-17T15:31:50.762 INFO:tasks.rados.rados.0.smithi203.stdout:536: writing smithi20328086-35 from 1782380 to 2478782 tid 2 2024-09-17T15:31:50.772 INFO:tasks.rados.rados.0.smithi203.stdout:536: writing smithi20328086-35 from 3031400 to 3568559 tid 3 2024-09-17T15:31:50.772 INFO:tasks.rados.rados.0.smithi203.stdout:537: snap_remove snap 47 2024-09-17T15:31:51.743 INFO:tasks.rados.rados.0.smithi203.stdout:537: done (2 left) 2024-09-17T15:31:51.743 INFO:tasks.rados.rados.0.smithi203.stdout:538: delete oid 32 current snap is 56 2024-09-17T15:31:51.743 INFO:tasks.rados.rados.0.smithi203.stdout:535: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: finishing write tid 1 to smithi20328086-35 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:535: finishing copy_from to smithi20328086-12 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: finishing write tid 2 to smithi20328086-35 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: finishing write tid 3 to smithi20328086-35 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: finishing write tid 4 to smithi20328086-35 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: finishing write tid 5 to smithi20328086-35 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: oid 35 updating version 0 to 75 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: oid 35 updating version 75 to 76 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: oid 35 version 76 is already newer than 73 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: oid 35 version 76 is already newer than 74 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 76 (ObjNum 219 snap 56 seq_num 219) dirty exists 2024-09-17T15:31:51.744 INFO:tasks.rados.rados.0.smithi203.stdout:536: left oid 35 (ObjNum 219 snap 56 seq_num 219) 2024-09-17T15:31:51.757 INFO:tasks.rados.rados.0.smithi203.stdout:535: done (2 left) 2024-09-17T15:31:51.757 INFO:tasks.rados.rados.0.smithi203.stdout:536: done (1 left) 2024-09-17T15:31:51.757 INFO:tasks.rados.rados.0.smithi203.stdout:538: done (0 left) 2024-09-17T15:31:51.757 INFO:tasks.rados.rados.0.smithi203.stdout:539: read oid 39 snap -1 2024-09-17T15:31:51.757 INFO:tasks.rados.rados.0.smithi203.stdout:539: expect (ObjNum 173 snap 43 seq_num 173) 2024-09-17T15:31:51.758 INFO:tasks.rados.rados.0.smithi203.stdout:540: write oid 41 current snap is 56 2024-09-17T15:31:51.758 INFO:tasks.rados.rados.0.smithi203.stdout:540: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-17T15:31:51.771 INFO:tasks.rados.rados.0.smithi203.stdout:540: writing smithi20328086-41 from 514347 to 918613 tid 1 2024-09-17T15:31:51.787 INFO:tasks.rados.rados.0.smithi203.stdout:540: writing smithi20328086-41 from 1499344 to 2199415 tid 2 2024-09-17T15:31:51.801 INFO:tasks.rados.rados.0.smithi203.stdout:540: writing smithi20328086-41 from 2686246 to 3362302 tid 3 2024-09-17T15:31:51.801 INFO:tasks.rados.rados.0.smithi203.stdout:541: read oid 22 snap -1 2024-09-17T15:31:51.802 INFO:tasks.rados.rados.0.smithi203.stdout:541: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:31:51.802 INFO:tasks.rados.rados.0.smithi203.stdout:542: write (excl) oid 1 current snap is 56 2024-09-17T15:31:51.802 INFO:tasks.rados.rados.0.smithi203.stdout:542: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-17T15:31:51.812 INFO:tasks.rados.rados.0.smithi203.stdout:542: writing smithi20328086-1 from 531154 to 1010669 tid 1 2024-09-17T15:31:51.822 INFO:tasks.rados.rados.0.smithi203.stdout:542: writing smithi20328086-1 from 1557826 to 2001555 tid 2 2024-09-17T15:31:51.833 INFO:tasks.rados.rados.0.smithi203.stdout:542: writing smithi20328086-1 from 2713669 to 3200997 tid 3 2024-09-17T15:31:51.837 INFO:tasks.rados.rados.0.smithi203.stdout:542: writing smithi20328086-1 from 3679295 to 3731154 tid 4 2024-09-17T15:31:51.837 INFO:tasks.rados.rados.0.smithi203.stdout:543: snap_remove snap 54 2024-09-17T15:31:52.431 INFO:tasks.rados.rados.0.smithi203.stdout:543: done (4 left) 2024-09-17T15:31:52.431 INFO:tasks.rados.rados.0.smithi203.stdout:544: snap_create 2024-09-17T15:31:52.431 INFO:tasks.rados.rados.0.smithi203.stdout:539: expect (ObjNum 173 snap 43 seq_num 173) 2024-09-17T15:31:52.467 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2229479 2024-09-17T15:31:52.486 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 503847 2024-09-17T15:31:52.564 INFO:tasks.rados.rados.0.smithi203.stdout:540: finishing write tid 1 to smithi20328086-41 2024-09-17T15:31:52.564 INFO:tasks.rados.rados.0.smithi203.stdout:540: finishing write tid 2 to smithi20328086-41 2024-09-17T15:31:52.564 INFO:tasks.rados.rados.0.smithi203.stdout:541: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:31:52.619 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2280537 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: finishing write tid 3 to smithi20328086-41 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: finishing write tid 4 to smithi20328086-41 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: finishing write tid 5 to smithi20328086-41 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: oid 41 updating version 0 to 84 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: oid 41 updating version 84 to 85 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: oid 41 version 85 is already newer than 83 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:540: oid 41 updating version 85 to 86 2024-09-17T15:31:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 86 (ObjNum 221 snap 56 seq_num 221) dirty exists 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:540: left oid 41 (ObjNum 221 snap 56 seq_num 221) 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 1 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 2 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 3 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 4 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 5 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: finishing write tid 6 to smithi20328086-1 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: oid 1 updating version 0 to 128 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: oid 1 updating version 128 to 130 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: oid 1 version 130 is already newer than 127 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: oid 1 version 130 is already newer than 129 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:542: oid 1 updating version 130 to 131 2024-09-17T15:31:52.624 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 131 (ObjNum 222 snap 56 seq_num 222) dirty exists 2024-09-17T15:31:52.625 INFO:tasks.rados.rados.0.smithi203.stdout:542: left oid 1 (ObjNum 222 snap 56 seq_num 222) 2024-09-17T15:31:53.431 INFO:tasks.rados.rados.0.smithi203.stdout:539: done (4 left) 2024-09-17T15:31:53.431 INFO:tasks.rados.rados.0.smithi203.stdout:540: done (3 left) 2024-09-17T15:31:53.431 INFO:tasks.rados.rados.0.smithi203.stdout:541: done (2 left) 2024-09-17T15:31:53.432 INFO:tasks.rados.rados.0.smithi203.stdout:542: done (1 left) 2024-09-17T15:31:53.432 INFO:tasks.rados.rados.0.smithi203.stdout:544: done (0 left) 2024-09-17T15:31:53.432 INFO:tasks.rados.rados.0.smithi203.stdout:545: write oid 45 current snap is 57 2024-09-17T15:31:53.432 INFO:tasks.rados.rados.0.smithi203.stdout:545: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-17T15:31:53.441 INFO:tasks.rados.rados.0.smithi203.stdout:545: writing smithi20328086-45 from 547961 to 1102725 tid 1 2024-09-17T15:31:53.450 INFO:tasks.rados.rados.0.smithi203.stdout:545: writing smithi20328086-45 from 1616308 to 2320048 tid 2 2024-09-17T15:31:53.459 INFO:tasks.rados.rados.0.smithi203.stdout:545: writing smithi20328086-45 from 2741092 to 3439692 tid 3 2024-09-17T15:31:53.459 INFO:tasks.rados.rados.0.smithi203.stdout:546: write (excl) oid 4 current snap is 57 2024-09-17T15:31:53.459 INFO:tasks.rados.rados.0.smithi203.stdout:545: finishing write tid 1 to smithi20328086-45 2024-09-17T15:31:53.459 INFO:tasks.rados.rados.0.smithi203.stdout:546: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-17T15:31:53.467 INFO:tasks.rados.rados.0.smithi203.stdout:546: writing smithi20328086-4 from 564768 to 1194781 tid 1 2024-09-17T15:31:53.474 INFO:tasks.rados.rados.0.smithi203.stdout:546: writing smithi20328086-4 from 1674790 to 2122188 tid 2 2024-09-17T15:31:53.483 INFO:tasks.rados.rados.0.smithi203.stdout:546: writing smithi20328086-4 from 2768515 to 3394740 tid 3 2024-09-17T15:31:53.483 INFO:tasks.rados.rados.0.smithi203.stdout:547: write (excl) oid 36 current snap is 57 2024-09-17T15:31:53.483 INFO:tasks.rados.rados.0.smithi203.stdout:547: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-17T15:31:53.493 INFO:tasks.rados.rados.0.smithi203.stdout:547: writing smithi20328086-36 from 581575 to 1286837 tid 1 2024-09-17T15:31:53.501 INFO:tasks.rados.rados.0.smithi203.stdout:547: writing smithi20328086-36 from 1733272 to 2440681 tid 2 2024-09-17T15:31:53.510 INFO:tasks.rados.rados.0.smithi203.stdout:547: writing smithi20328086-36 from 3195938 to 3633435 tid 3 2024-09-17T15:31:53.510 INFO:tasks.rados.rados.0.smithi203.stdout:548: write oid 11 current snap is 57 2024-09-17T15:31:53.510 INFO:tasks.rados.rados.0.smithi203.stdout:548: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-17T15:31:53.518 INFO:tasks.rados.rados.0.smithi203.stdout:548: writing smithi20328086-11 from 598382 to 1378893 tid 1 2024-09-17T15:31:53.525 INFO:tasks.rados.rados.0.smithi203.stdout:548: writing smithi20328086-11 from 2075401 to 2526468 tid 2 2024-09-17T15:31:53.532 INFO:tasks.rados.rados.0.smithi203.stdout:548: writing smithi20328086-11 from 3107008 to 3755777 tid 3 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:549: delete oid 33 current snap is 57 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: finishing write tid 2 to smithi20328086-45 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: finishing write tid 3 to smithi20328086-45 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: finishing write tid 4 to smithi20328086-45 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: finishing write tid 5 to smithi20328086-45 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: oid 45 updating version 0 to 73 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: oid 45 version 73 is already newer than 70 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: oid 45 version 73 is already newer than 72 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: oid 45 version 73 is already newer than 71 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 73 (ObjNum 223 snap 57 seq_num 223) dirty exists 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:545: left oid 45 (ObjNum 223 snap 57 seq_num 223) 2024-09-17T15:31:53.533 INFO:tasks.rados.rados.0.smithi203.stdout:546: finishing write tid 1 to smithi20328086-4 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: finishing write tid 2 to smithi20328086-4 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: finishing write tid 3 to smithi20328086-4 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: finishing write tid 4 to smithi20328086-4 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: finishing write tid 5 to smithi20328086-4 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: oid 4 updating version 0 to 75 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: oid 4 version 75 is already newer than 74 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: oid 4 updating version 75 to 77 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: oid 4 version 77 is already newer than 76 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 77 (ObjNum 224 snap 57 seq_num 224) dirty exists 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:546: left oid 4 (ObjNum 224 snap 57 seq_num 224) 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:547: finishing write tid 1 to smithi20328086-36 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:547: finishing write tid 2 to smithi20328086-36 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:547: finishing write tid 3 to smithi20328086-36 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:547: finishing write tid 4 to smithi20328086-36 2024-09-17T15:31:53.534 INFO:tasks.rados.rados.0.smithi203.stdout:547: finishing write tid 5 to smithi20328086-36 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:547: oid 36 updating version 0 to 210 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:547: oid 36 updating version 210 to 212 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:547: oid 36 version 212 is already newer than 211 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:547: oid 36 updating version 212 to 213 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 213 (ObjNum 225 snap 57 seq_num 225) dirty exists 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:547: left oid 36 (ObjNum 225 snap 57 seq_num 225) 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:548: finishing write tid 1 to smithi20328086-11 2024-09-17T15:31:53.535 INFO:tasks.rados.rados.0.smithi203.stdout:548: finishing write tid 2 to smithi20328086-11 2024-09-17T15:31:53.539 INFO:tasks.rados.rados.0.smithi203.stdout:548: finishing write tid 3 to smithi20328086-11 2024-09-17T15:31:53.539 INFO:tasks.rados.rados.0.smithi203.stdout:548: finishing write tid 4 to smithi20328086-11 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: finishing write tid 5 to smithi20328086-11 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: oid 11 updating version 0 to 147 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: oid 11 updating version 147 to 148 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: oid 11 version 148 is already newer than 146 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: oid 11 version 148 is already newer than 145 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 148 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:548: left oid 11 (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:545: done (4 left) 2024-09-17T15:31:53.540 INFO:tasks.rados.rados.0.smithi203.stdout:546: done (3 left) 2024-09-17T15:31:53.541 INFO:tasks.rados.rados.0.smithi203.stdout:547: done (2 left) 2024-09-17T15:31:53.541 INFO:tasks.rados.rados.0.smithi203.stdout:548: done (1 left) 2024-09-17T15:31:53.541 INFO:tasks.rados.rados.0.smithi203.stdout:549: done (0 left) 2024-09-17T15:31:53.541 INFO:tasks.rados.rados.0.smithi203.stdout:550: delete oid 30 current snap is 57 2024-09-17T15:31:53.547 INFO:tasks.rados.rados.0.smithi203.stdout:550: done (0 left) 2024-09-17T15:31:53.547 INFO:tasks.rados.rados.0.smithi203.stdout:551: copy_from oid 10 from oid 46 current snap is 57 2024-09-17T15:31:53.547 INFO:tasks.rados.rados.0.smithi203.stdout:552: copy_from oid 39 from oid 3 current snap is 57 2024-09-17T15:31:53.547 INFO:tasks.rados.rados.0.smithi203.stdout:553: write oid 40 current snap is 57 2024-09-17T15:31:53.547 INFO:tasks.rados.rados.0.smithi203.stdout:553: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-17T15:31:53.555 INFO:tasks.rados.rados.0.smithi203.stdout:553: writing smithi20328086-40 from 648803 to 1371414 tid 1 2024-09-17T15:31:53.563 INFO:tasks.rados.rados.0.smithi203.stdout:553: writing smithi20328086-40 from 1967200 to 2681947 tid 2 2024-09-17T15:31:53.569 INFO:tasks.rados.rados.0.smithi203.stdout:553: writing smithi20328086-40 from 3421983 to 3848803 tid 3 2024-09-17T15:31:53.570 INFO:tasks.rados.rados.0.smithi203.stdout:554: snap_remove snap 56 2024-09-17T15:31:54.440 INFO:tasks.rados.rados.0.smithi203.stdout:554: done (3 left) 2024-09-17T15:31:54.440 INFO:tasks.rados.rados.0.smithi203.stdout:555: write oid 33 current snap is 57 2024-09-17T15:31:54.440 INFO:tasks.rados.rados.0.smithi203.stdout:555: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-17T15:31:54.449 INFO:tasks.rados.rados.0.smithi203.stdout:555: writing smithi20328086-33 from 665610 to 1463470 tid 1 2024-09-17T15:31:54.454 INFO:tasks.rados.rados.0.smithi203.stdout:555: writing smithi20328086-33 from 1909329 to 2367734 tid 2 2024-09-17T15:31:54.460 INFO:tasks.rados.rados.0.smithi203.stdout:555: writing smithi20328086-33 from 2816700 to 3226910 tid 3 2024-09-17T15:31:54.460 INFO:tasks.rados.rados.0.smithi203.stdout:556: write oid 31 current snap is 57 2024-09-17T15:31:54.461 INFO:tasks.rados.rados.0.smithi203.stdout:556: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-17T15:31:54.468 INFO:tasks.rados.rados.0.smithi203.stdout:556: writing smithi20328086-31 from 682417 to 1155526 tid 1 2024-09-17T15:31:54.474 INFO:tasks.rados.rados.0.smithi203.stdout:556: writing smithi20328086-31 from 1567811 to 2169874 tid 2 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:556: writing smithi20328086-31 from 2844123 to 3465605 tid 3 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:557: copy_from oid 27 from oid 30 current snap is 57 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:552: finishing copy_from to smithi20328086-39 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:552: got expected ENOENT (src dne) 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:558: rollback oid 38 current snap is 57 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 55 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:552: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:553: finishing write tid 1 to smithi20328086-40 2024-09-17T15:31:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:553: finishing write tid 2 to smithi20328086-40 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:552: done (6 left) 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:559: rollback oid 3 current snap is 57 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 55 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:551: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:553: finishing write tid 3 to smithi20328086-40 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:553: finishing write tid 4 to smithi20328086-40 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:560: write (excl) oid 37 current snap is 57 2024-09-17T15:31:54.483 INFO:tasks.rados.rados.0.smithi203.stdout:560: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-17T15:31:54.489 INFO:tasks.rados.rados.0.smithi203.stdout:560: writing smithi20328086-37 from 699224 to 1247582 tid 1 2024-09-17T15:31:54.496 INFO:tasks.rados.rados.0.smithi203.stdout:560: writing smithi20328086-37 from 2026293 to 2488367 tid 2 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:560: writing smithi20328086-37 from 3271546 to 3704300 tid 3 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:561: delete oid 50 current snap is 57 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:551: finishing copy_from to smithi20328086-10 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 88 (ObjNum 162 snap 39 seq_num 162) dirty exists 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:553: finishing write tid 5 to smithi20328086-40 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:553: oid 40 updating version 0 to 150 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:553: oid 40 updating version 150 to 151 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:553: oid 40 updating version 151 to 152 2024-09-17T15:31:54.505 INFO:tasks.rados.rados.0.smithi203.stdout:553: oid 40 updating version 152 to 153 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 153 (ObjNum 229 snap 57 seq_num 229) dirty exists 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:553: left oid 40 (ObjNum 229 snap 57 seq_num 229) 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: finishing write tid 1 to smithi20328086-33 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: finishing write tid 2 to smithi20328086-33 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: finishing write tid 3 to smithi20328086-33 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: finishing write tid 4 to smithi20328086-33 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: finishing write tid 5 to smithi20328086-33 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: oid 33 updating version 0 to 91 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: oid 33 updating version 91 to 93 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: oid 33 version 93 is already newer than 92 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: oid 33 version 93 is already newer than 88 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 93 (ObjNum 230 snap 57 seq_num 230) dirty exists 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:555: left oid 33 (ObjNum 230 snap 57 seq_num 230) 2024-09-17T15:31:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:556: finishing write tid 1 to smithi20328086-31 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:556: finishing write tid 2 to smithi20328086-31 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:559: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:557: finishing copy_from to smithi20328086-27 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:557: got expected ENOENT (src dne) 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:557: finishing copy_from racing read to smithi20328086-27 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:558: finishing rollback tid 0 to smithi20328086-38 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:558: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:556: finishing write tid 3 to smithi20328086-31 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:558: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 108 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-17T15:31:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:556: finishing write tid 4 to smithi20328086-31 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: finishing write tid 5 to smithi20328086-31 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: oid 31 updating version 0 to 56 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: oid 31 updating version 56 to 59 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: oid 31 version 59 is already newer than 58 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: oid 31 version 59 is already newer than 57 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 59 (ObjNum 231 snap 57 seq_num 231) dirty exists 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:556: left oid 31 (ObjNum 231 snap 57 seq_num 231) 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:560: finishing write tid 1 to smithi20328086-37 2024-09-17T15:31:54.508 INFO:tasks.rados.rados.0.smithi203.stdout:560: finishing write tid 2 to smithi20328086-37 2024-09-17T15:31:54.509 INFO:tasks.rados.rados.0.smithi203.stdout:560: finishing write tid 3 to smithi20328086-37 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: finishing write tid 4 to smithi20328086-37 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: finishing write tid 5 to smithi20328086-37 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: oid 37 updating version 0 to 118 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: oid 37 version 118 is already newer than 116 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: oid 37 version 118 is already newer than 115 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:560: oid 37 version 118 is already newer than 117 2024-09-17T15:31:54.510 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 118 (ObjNum 232 snap 57 seq_num 232) dirty exists 2024-09-17T15:31:54.511 INFO:tasks.rados.rados.0.smithi203.stdout:560: left oid 37 (ObjNum 232 snap 57 seq_num 232) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:551: done (8 left) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:553: done (7 left) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:555: done (6 left) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:556: done (5 left) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:557: done (4 left) 2024-09-17T15:31:54.515 INFO:tasks.rados.rados.0.smithi203.stdout:558: done (3 left) 2024-09-17T15:31:54.516 INFO:tasks.rados.rados.0.smithi203.stdout:559: done (2 left) 2024-09-17T15:31:54.516 INFO:tasks.rados.rados.0.smithi203.stdout:560: done (1 left) 2024-09-17T15:31:54.516 INFO:tasks.rados.rados.0.smithi203.stdout:561: done (0 left) 2024-09-17T15:31:54.516 INFO:tasks.rados.rados.0.smithi203.stdout:562: snap_remove snap 55 2024-09-17T15:31:55.452 INFO:tasks.rados.rados.0.smithi203.stdout:562: done (0 left) 2024-09-17T15:31:55.452 INFO:tasks.rados.rados.0.smithi203.stdout:563: snap_create 2024-09-17T15:31:56.457 INFO:tasks.rados.rados.0.smithi203.stdout:563: done (0 left) 2024-09-17T15:31:56.457 INFO:tasks.rados.rados.0.smithi203.stdout:564: write (excl) oid 25 current snap is 58 2024-09-17T15:31:56.457 INFO:tasks.rados.rados.0.smithi203.stdout:564: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-17T15:31:56.469 INFO:tasks.rados.rados.0.smithi203.stdout:564: writing smithi20328086-25 from 732838 to 1431694 tid 1 2024-09-17T15:31:56.478 INFO:tasks.rados.rados.0.smithi203.stdout:564: writing smithi20328086-25 from 2026904 to 2492647 tid 2 2024-09-17T15:31:56.489 INFO:tasks.rados.rados.0.smithi203.stdout:564: writing smithi20328086-25 from 3210039 to 3781690 tid 3 2024-09-17T15:31:56.490 INFO:tasks.rados.rados.0.smithi203.stdout:565: write (excl) oid 31 current snap is 58 2024-09-17T15:31:56.490 INFO:tasks.rados.rados.0.smithi203.stdout:565: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-17T15:31:56.502 INFO:tasks.rados.rados.0.smithi203.stdout:565: writing smithi20328086-31 from 749645 to 1523750 tid 1 2024-09-17T15:31:56.513 INFO:tasks.rados.rados.0.smithi203.stdout:565: writing smithi20328086-31 from 2085386 to 2694787 tid 2 2024-09-17T15:31:56.522 INFO:tasks.rados.rados.0.smithi203.stdout:565: writing smithi20328086-31 from 3237462 to 3949645 tid 3 2024-09-17T15:31:56.522 INFO:tasks.rados.rados.0.smithi203.stdout:566: write (excl) oid 13 current snap is 58 2024-09-17T15:31:56.522 INFO:tasks.rados.rados.0.smithi203.stdout:566: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-17T15:31:56.531 INFO:tasks.rados.rados.0.smithi203.stdout:566: writing smithi20328086-13 from 766452 to 1332159 tid 1 2024-09-17T15:31:56.537 INFO:tasks.rados.rados.0.smithi203.stdout:566: writing smithi20328086-13 from 1860221 to 2329633 tid 2 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:566: writing smithi20328086-13 from 2981238 to 3575433 tid 3 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:567: read oid 45 snap -1 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:567: expect (ObjNum 223 snap 57 seq_num 223) 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:568: read oid 7 snap -1 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:568: expect (ObjNum 188 snap 46 seq_num 188) 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:569: write oid 27 current snap is 58 2024-09-17T15:31:56.545 INFO:tasks.rados.rados.0.smithi203.stdout:569: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-17T15:31:56.553 INFO:tasks.rados.rados.0.smithi203.stdout:569: writing smithi20328086-27 from 783259 to 1424215 tid 1 2024-09-17T15:31:56.560 INFO:tasks.rados.rados.0.smithi203.stdout:569: writing smithi20328086-27 from 1918703 to 2531773 tid 2 2024-09-17T15:31:56.565 INFO:tasks.rados.rados.0.smithi203.stdout:569: writing smithi20328086-27 from 3008661 to 3414128 tid 3 2024-09-17T15:31:56.568 INFO:tasks.rados.rados.0.smithi203.stdout:569: writing smithi20328086-27 from 3821939 to 3983259 tid 4 2024-09-17T15:31:56.568 INFO:tasks.rados.rados.0.smithi203.stdout:570: rollback oid 34 current snap is 58 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 57 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:564: finishing write tid 1 to smithi20328086-25 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:564: finishing write tid 2 to smithi20328086-25 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:571: read oid 32 snap -1 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:571: expect deleted 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:572: write oid 48 current snap is 58 2024-09-17T15:31:56.569 INFO:tasks.rados.rados.0.smithi203.stdout:572: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-17T15:31:56.574 INFO:tasks.rados.rados.0.smithi203.stdout:572: writing smithi20328086-48 from 400066 to 1116271 tid 1 2024-09-17T15:31:56.580 INFO:tasks.rados.rados.0.smithi203.stdout:572: writing smithi20328086-48 from 1860832 to 2000066 tid 2 2024-09-17T15:31:56.580 INFO:tasks.rados.rados.0.smithi203.stdout:573: rollback oid 19 current snap is 58 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 53 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: finishing write tid 3 to smithi20328086-25 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: finishing write tid 4 to smithi20328086-25 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: finishing write tid 5 to smithi20328086-25 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: oid 25 updating version 0 to 134 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: oid 25 updating version 134 to 136 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: oid 25 version 136 is already newer than 135 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: oid 25 updating version 136 to 137 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 137 (ObjNum 234 snap 58 seq_num 234) dirty exists 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:564: left oid 25 (ObjNum 234 snap 58 seq_num 234) 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:565: finishing write tid 1 to smithi20328086-31 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:565: finishing write tid 2 to smithi20328086-31 2024-09-17T15:31:56.581 INFO:tasks.rados.rados.0.smithi203.stdout:565: finishing write tid 3 to smithi20328086-31 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: finishing write tid 4 to smithi20328086-31 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: finishing write tid 5 to smithi20328086-31 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: oid 31 updating version 0 to 61 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: oid 31 updating version 61 to 62 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: oid 31 updating version 62 to 63 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: oid 31 version 63 is already newer than 60 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 63 (ObjNum 235 snap 58 seq_num 235) dirty exists 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:565: left oid 31 (ObjNum 235 snap 58 seq_num 235) 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: finishing write tid 1 to smithi20328086-13 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: finishing write tid 2 to smithi20328086-13 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: finishing write tid 3 to smithi20328086-13 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: finishing write tid 4 to smithi20328086-13 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: finishing write tid 5 to smithi20328086-13 2024-09-17T15:31:56.582 INFO:tasks.rados.rados.0.smithi203.stdout:566: oid 13 updating version 0 to 115 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:566: oid 13 version 115 is already newer than 112 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:566: oid 13 version 115 is already newer than 114 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:566: oid 13 version 115 is already newer than 113 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 115 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:566: left oid 13 (ObjNum 236 snap 58 seq_num 236) 2024-09-17T15:31:56.583 INFO:tasks.rados.rados.0.smithi203.stdout:567: expect (ObjNum 223 snap 57 seq_num 223) 2024-09-17T15:31:56.699 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:568: expect (ObjNum 188 snap 46 seq_num 188) 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 964225 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 1 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 2 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 3 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 4 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:570: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 5 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: finishing write tid 6 to smithi20328086-27 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: oid 27 updating version 0 to 133 2024-09-17T15:31:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:569: oid 27 updating version 133 to 135 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:569: oid 27 version 135 is already newer than 134 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:569: oid 27 updating version 135 to 136 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:569: oid 27 updating version 136 to 137 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 137 (ObjNum 237 snap 58 seq_num 237) dirty exists 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:569: left oid 27 (ObjNum 237 snap 58 seq_num 237) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:564: done (9 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:565: done (8 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:566: done (7 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:567: done (6 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:568: done (5 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:569: done (4 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:571: done (3 left) 2024-09-17T15:31:56.954 INFO:tasks.rados.rados.0.smithi203.stdout:574: read oid 35 snap -1 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:574: expect (ObjNum 219 snap 56 seq_num 219) 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:575: snap_create 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:570: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:570: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 84 (ObjNum 158 snap 36 seq_num 158) dirty exists 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: finishing write tid 1 to smithi20328086-48 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: finishing write tid 2 to smithi20328086-48 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: finishing write tid 3 to smithi20328086-48 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: finishing write tid 4 to smithi20328086-48 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: oid 48 updating version 0 to 123 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: oid 48 updating version 123 to 124 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:572: oid 48 version 124 is already newer than 122 2024-09-17T15:31:56.955 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 124 (ObjNum 238 snap 58 seq_num 238) dirty exists 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:572: left oid 48 (ObjNum 238 snap 58 seq_num 238) 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:573: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:573: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:573: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 128 (ObjNum 185 snap 46 seq_num 185) dirty exists 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:574: expect (ObjNum 219 snap 56 seq_num 219) 2024-09-17T15:31:56.956 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2230798 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:570: done (4 left) 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:572: done (3 left) 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:573: done (2 left) 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:574: done (1 left) 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:575: done (0 left) 2024-09-17T15:31:57.432 INFO:tasks.rados.rados.0.smithi203.stdout:576: snap_create 2024-09-17T15:31:58.435 INFO:tasks.rados.rados.0.smithi203.stdout:576: done (0 left) 2024-09-17T15:31:58.435 INFO:tasks.rados.rados.0.smithi203.stdout:577: read oid 38 snap -1 2024-09-17T15:31:58.435 INFO:tasks.rados.rados.0.smithi203.stdout:577: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:31:58.435 INFO:tasks.rados.rados.0.smithi203.stdout:578: snap_remove snap 53 2024-09-17T15:31:59.438 INFO:tasks.rados.rados.0.smithi203.stdout:578: done (1 left) 2024-09-17T15:31:59.438 INFO:tasks.rados.rados.0.smithi203.stdout:579: snap_create 2024-09-17T15:31:59.439 INFO:tasks.rados.rados.0.smithi203.stdout:577: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-17T15:31:59.439 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:31:59.471 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 195506 2024-09-17T15:31:59.500 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 393777 2024-09-17T15:32:00.450 INFO:tasks.rados.rados.0.smithi203.stdout:577: done (1 left) 2024-09-17T15:32:00.450 INFO:tasks.rados.rados.0.smithi203.stdout:579: done (0 left) 2024-09-17T15:32:00.450 INFO:tasks.rados.rados.0.smithi203.stdout:580: rollback oid 8 current snap is 61 2024-09-17T15:32:00.450 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 60 2024-09-17T15:32:00.450 INFO:tasks.rados.rados.0.smithi203.stdout:581: snap_remove snap 58 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:581: done (1 left) 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:582: rollback oid 43 current snap is 61 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 57 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:580: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:580: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:580: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 119 (ObjNum 205 snap 51 seq_num 205) dirty exists 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:580: done (1 left) 2024-09-17T15:32:01.448 INFO:tasks.rados.rados.0.smithi203.stdout:583: delete oid 2 current snap is 61 2024-09-17T15:32:01.457 INFO:tasks.rados.rados.0.smithi203.stdout:582: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:32:01.457 INFO:tasks.rados.rados.0.smithi203.stdout:582: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:32:01.458 INFO:tasks.rados.rados.0.smithi203.stdout:582: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:32:01.458 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 90 (ObjNum 76 snap 5 seq_num 76) dirty exists 2024-09-17T15:32:01.465 INFO:tasks.rados.rados.0.smithi203.stdout:582: done (1 left) 2024-09-17T15:32:01.465 INFO:tasks.rados.rados.0.smithi203.stdout:583: done (0 left) 2024-09-17T15:32:01.465 INFO:tasks.rados.rados.0.smithi203.stdout:584: snap_create 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:584: done (0 left) 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:585: read oid 25 snap -1 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:585: expect (ObjNum 234 snap 58 seq_num 234) 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:586: read oid 4 snap -1 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:586: expect (ObjNum 224 snap 57 seq_num 224) 2024-09-17T15:32:02.434 INFO:tasks.rados.rados.0.smithi203.stdout:587: delete oid 40 current snap is 62 2024-09-17T15:32:02.447 INFO:tasks.rados.rados.0.smithi203.stdout:587: done (2 left) 2024-09-17T15:32:02.447 INFO:tasks.rados.rados.0.smithi203.stdout:588: delete oid 33 current snap is 62 2024-09-17T15:32:02.454 INFO:tasks.rados.rados.0.smithi203.stdout:585: expect (ObjNum 234 snap 58 seq_num 234) 2024-09-17T15:32:02.454 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:02.540 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:02.582 INFO:tasks.rados.rados.0.smithi203.stdout:586: expect (ObjNum 224 snap 57 seq_num 224) 2024-09-17T15:32:02.637 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3537843 2024-09-17T15:32:02.644 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:585: done (2 left) 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:586: done (1 left) 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:588: done (0 left) 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:589: read oid 36 snap 61 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:589: expect (ObjNum 225 snap 57 seq_num 225) 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:590: copy_from oid 41 from oid 33 current snap is 62 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:591: read oid 28 snap -1 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:591: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:592: write (excl) oid 47 current snap is 62 2024-09-17T15:32:02.686 INFO:tasks.rados.rados.0.smithi203.stdout:592: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-17T15:32:02.693 INFO:tasks.rados.rados.0.smithi203.stdout:592: writing smithi20328086-47 from 467294 to 1084495 tid 1 2024-09-17T15:32:02.698 INFO:tasks.rados.rados.0.smithi203.stdout:592: writing smithi20328086-47 from 1578407 to 2067294 tid 2 2024-09-17T15:32:02.698 INFO:tasks.rados.rados.0.smithi203.stdout:593: snap_remove snap 57 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:593: done (4 left) 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:594: snap_create 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:590: finishing copy_from to smithi20328086-41 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:590: got expected ENOENT (src dne) 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:590: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:32:03.465 INFO:tasks.rados.rados.0.smithi203.stdout:589: expect (ObjNum 225 snap 57 seq_num 225) 2024-09-17T15:32:03.482 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 746853 2024-09-17T15:32:03.609 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:03.703 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:03.798 INFO:tasks.rados.rados.0.smithi203.stdout:592: finishing write tid 1 to smithi20328086-47 2024-09-17T15:32:03.798 INFO:tasks.rados.rados.0.smithi203.stdout:592: finishing write tid 2 to smithi20328086-47 2024-09-17T15:32:03.798 INFO:tasks.rados.rados.0.smithi203.stdout:592: finishing write tid 3 to smithi20328086-47 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:592: finishing write tid 4 to smithi20328086-47 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:592: oid 47 updating version 0 to 124 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:592: oid 47 version 124 is already newer than 123 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:592: oid 47 updating version 124 to 125 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 125 (ObjNum 242 snap 62 seq_num 242) dirty exists 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:592: left oid 47 (ObjNum 242 snap 62 seq_num 242) 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:591: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:32:03.799 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:03.858 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:03.941 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1798131 2024-09-17T15:32:04.485 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:32:04.486 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:32:04.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:32:04.489 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:32:04.489 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:32:04.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: get_monmap_and_config 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: build_initial_monmap 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: monmap: 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:04.621299+0000 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:04.621299+0000 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:04.621 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:04.622 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:04.622 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 Processor -- start 2024-09-17T15:32:04.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 -- start start 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: init 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7dec0689d0 0x7f7dec06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7dec068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7dec13aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec09c8b0 con 0x7f7dec068db0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dec09cb90 con 0x7f7dec13aff0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7dec074500 con 0x7f7dec0689d0 2024-09-17T15:32:04.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df3167640 10 monclient(hunting): authenticate will time out at 1257.586797s 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7dec0689d0 0x7f7dec06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7debfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7dec068db0 0x7f7dec069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40366/0 (socket says 172.21.15.86:40366) 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7debfff640 1 -- 172.21.15.86:0/2139336165 learned_addr learned my addr 172.21.15.86:0/2139336165 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): get_auth_request con 0x7f7dec0689d0 auth_method 0 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient(hunting): handle_auth_done global_id 4722 payload 274 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 1 -- 172.21.15.86:0/2139336165 >> v1:172.21.15.203:6789/0 conn(0x7f7dec13aff0 legacy=0x7f7dec13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 1 -- 172.21.15.86:0/2139336165 >> v1:172.21.15.86:6789/0 conn(0x7f7dec068db0 legacy=0x7f7dec069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient: _finish_hunting 0 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 1 monclient: found mon.noname-a 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.616+0000 7f7df0f03640 1 -- 172.21.15.86:0/2139336165 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec09ce70 con 0x7f7dec0689d0 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df0f03640 1 -- 172.21.15.86:0/2139336165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 msgr2=0x7f7dec06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df0f03640 1 --2- 172.21.15.86:0/2139336165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec06f840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7dec069940 tx=0x7f7ddc01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3cf19c1943bb04 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 1 -- 172.21.15.86:0/2139336165 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0b0070 con 0x7f7dec0689d0 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: dump: 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.628 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 1 -- 172.21.15.86:0/2139336165 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc0ab600 con 0x7f7dec0689d0 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df3167640 5 monclient: authenticate success, global_id 4722 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df3167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df3167640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:04.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7df3167640 4 set_mon_vals no callback set 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 1 -- 172.21.15.86:0/2139336165 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec09c2d0 con 0x7f7dec0689d0 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.620+0000 7f7deb7fe640 10 monclient: dump: 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:04.630 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: shutdown 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- 172.21.15.86:0/2139336165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 msgr2=0x7f7dec06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 --2- 172.21.15.86:0/2139336165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec06f840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7dec069940 tx=0x7f7ddc01c340 comp rx=0 tx=0).stop 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- 172.21.15.86:0/2139336165 shutdown_connections 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 --2- 172.21.15.86:0/2139336165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec06f840 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- 172.21.15.86:0/2139336165 >> 172.21.15.86:0/2139336165 conn(0x7f7dec0a6440 msgr2=0x7f7dec0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- 172.21.15.86:0/2139336165 shutdown_connections 2024-09-17T15:32:04.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- 172.21.15.86:0/2139336165 wait complete. 2024-09-17T15:32:04.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: build_initial_monmap 2024-09-17T15:32:04.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: monmap: 2024-09-17T15:32:04.635 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:04.635 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 Processor -- start 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- start start 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: init 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:04.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-a con 0x7f7dec0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 0x7f7dec11ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-b con 0x7f7dec068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dec13aff0 0x7f7dec11d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): picked mon.noname-c con 0x7f7dec13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec141c30 con 0x7f7dec0689d0 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec141ab0 con 0x7f7dec068db0 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec141db0 con 0x7f7dec13aff0 2024-09-17T15:32:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df3167640 10 monclient(hunting): authenticate will time out at 1257.594112s 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47790/0 (socket says 172.21.15.86:47790) 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 -- 172.21.15.86:0/4185605643 learned_addr learned my addr 172.21.15.86:0/4185605643 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): get_auth_request con 0x7f7dec0689d0 auth_method 0 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dec13aff0 0x7f7dec11d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): get_auth_request con 0x7f7dec13aff0 auth_method 0 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df1704640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient(hunting): handle_auth_done global_id 4725 payload 824 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7debfff640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 0x7f7dec11ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 -- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dec13aff0 msgr2=0x7f7dec11d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dec13aff0 0x7f7dec11d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 -- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 msgr2=0x7f7dec11ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 0x7f7dec11ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient: _finish_hunting 0 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 monclient: found mon.noname-a 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7df0f03640 1 -- 172.21.15.86:0/4185605643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec11eff0 con 0x7f7dec0689d0 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7debfff640 10 monclient: get_auth_request con 0x7f7dec068db0 auth_method 0 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.624+0000 7f7debfff640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 0x7f7dec11ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df0f03640 1 -- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 msgr2=0x7f7dec11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df0f03640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0ae040 tx=0x7f7ddc01ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0b0050 con 0x7f7dec0689d0 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: dump: 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: _finish_auth 0 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc0bc070 con 0x7f7dec0689d0 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0b5a80 con 0x7f7dec0689d0 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 5 monclient: authenticate success, global_id 4725 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _renew_subs 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7dea7fc640 4 set_mon_vals no callback set 2024-09-17T15:32:04.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dec11f2d0 con 0x7f7dec0689d0 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 10 monclient: dump: 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:04.643 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _renew_subs 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dec11ed10 con 0x7f7dec0689d0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dec06f840 con 0x7f7dec0689d0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7ddc0b5c20 con 0x7f7dec0689d0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 0x7f7dcc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 4755+0+0 (secure 0 0 0) 0x7f7ddc0ea920 con 0x7f7dec0689d0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7debfff640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 0x7f7dcc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:04.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.628+0000 7f7debfff640 10 monclient: get_auth_request con 0x7f7dcc03bb60 auth_method 0 2024-09-17T15:32:04.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.632+0000 7f7debfff640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 0x7f7dcc03e020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4007fd0 tx=0x7f7dd4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:04.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.632+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7ddc016050 con 0x7f7dec0689d0 2024-09-17T15:32:04.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.632+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:04.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.632+0000 7f7de97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:04.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:04.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:04.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:04.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7dec11de30 con 0x7f7dec0689d0 2024-09-17T15:32:04.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7de97fa640 1 -- 172.21.15.86:0/4185605643 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f7ddc0e9330 con 0x7f7dec0689d0 2024-09-17T15:32:04.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7de97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:04.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7de97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:04.873 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 msgr2=0x7f7dcc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 0x7f7dcc03e020 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4007fd0 tx=0x7f7dd4009040 comp rx=0 tx=0).stop 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 10 monclient: shutdown 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 msgr2=0x7f7dec11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:04.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0ae040 tx=0x7f7ddc01ab80 comp rx=0 tx=0).stop 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 shutdown_connections 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.868+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7dcc03bb60 0x7f7dcc03e020 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7dec13aff0 0x7f7dec11d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dec068db0 0x7f7dec11ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 --2- 172.21.15.86:0/4185605643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7dec0689d0 0x7f7dec11c8c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 >> 172.21.15.86:0/4185605643 conn(0x7f7dec0a6440 msgr2=0x7f7dec0a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 shutdown_connections 2024-09-17T15:32:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:04.872+0000 7f7df3167640 1 -- 172.21.15.86:0/4185605643 wait complete. 2024-09-17T15:32:05.227 INFO:tasks.rados.rados.0.smithi203.stdout:589: done (4 left) 2024-09-17T15:32:05.227 INFO:tasks.rados.rados.0.smithi203.stdout:590: done (3 left) 2024-09-17T15:32:05.227 INFO:tasks.rados.rados.0.smithi203.stdout:591: done (2 left) 2024-09-17T15:32:05.227 INFO:tasks.rados.rados.0.smithi203.stdout:592: done (1 left) 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:594: done (0 left) 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:595: read oid 2 snap -1 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:595: expect deleted 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:596: rollback oid 21 current snap is 63 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 60 2024-09-17T15:32:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:597: delete oid 41 current snap is 63 2024-09-17T15:32:05.240 INFO:tasks.rados.rados.0.smithi203.stdout:595: done (2 left) 2024-09-17T15:32:05.240 INFO:tasks.rados.rados.0.smithi203.stdout:597: done (1 left) 2024-09-17T15:32:05.240 INFO:tasks.rados.rados.0.smithi203.stdout:598: write oid 2 current snap is 63 2024-09-17T15:32:05.240 INFO:tasks.rados.rados.0.smithi203.stdout:598: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-17T15:32:05.256 INFO:tasks.rados.rados.0.smithi203.stdout:598: writing smithi20328086-2 from 500908 to 984960 tid 1 2024-09-17T15:32:05.275 INFO:tasks.rados.rados.0.smithi203.stdout:598: writing smithi20328086-2 from 1411724 to 2100908 tid 2 2024-09-17T15:32:05.275 INFO:tasks.rados.rados.0.smithi203.stdout:599: write (excl) oid 3 current snap is 63 2024-09-17T15:32:05.275 INFO:tasks.rados.rados.0.smithi203.stdout:599: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-17T15:32:05.291 INFO:tasks.rados.rados.0.smithi203.stdout:599: writing smithi20328086-3 from 517715 to 1077016 tid 1 2024-09-17T15:32:05.303 INFO:tasks.rados.rados.0.smithi203.stdout:599: writing smithi20328086-3 from 1870206 to 2117715 tid 2 2024-09-17T15:32:05.303 INFO:tasks.rados.rados.0.smithi203.stdout:600: read oid 44 snap -1 2024-09-17T15:32:05.303 INFO:tasks.rados.rados.0.smithi203.stdout:600: expect (ObjNum 156 snap 36 seq_num 156) 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:601: rollback oid 27 current snap is 63 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 59 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:596: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:596: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:596: finishing rollback tid 2 to smithi20328086-21 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 227 (ObjNum 183 snap 45 seq_num 183) dirty exists 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:598: finishing write tid 1 to smithi20328086-2 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:599: finishing write tid 1 to smithi20328086-3 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:596: done (4 left) 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:602: copy_from oid 6 from oid 7 current snap is 63 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:603: read oid 50 snap -1 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:603: expect deleted 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:604: read oid 16 snap -1 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:604: expect deleted 2024-09-17T15:32:05.304 INFO:tasks.rados.rados.0.smithi203.stdout:605: snap_remove snap 61 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:605: done (7 left) 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:606: delete oid 5 current snap is 63 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: finishing write tid 2 to smithi20328086-2 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: finishing write tid 3 to smithi20328086-2 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: finishing write tid 4 to smithi20328086-2 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: oid 2 updating version 0 to 107 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: oid 2 version 107 is already newer than 105 2024-09-17T15:32:05.479 INFO:tasks.rados.rados.0.smithi203.stdout:598: oid 2 version 107 is already newer than 106 2024-09-17T15:32:05.480 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 107 (ObjNum 244 snap 63 seq_num 244) dirty exists 2024-09-17T15:32:05.480 INFO:tasks.rados.rados.0.smithi203.stdout:598: left oid 2 (ObjNum 244 snap 63 seq_num 244) 2024-09-17T15:32:05.480 INFO:tasks.rados.rados.0.smithi203.stdout:599: finishing write tid 2 to smithi20328086-3 2024-09-17T15:32:05.480 INFO:tasks.rados.rados.0.smithi203.stdout:600: expect (ObjNum 156 snap 36 seq_num 156) 2024-09-17T15:32:05.545 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 822283 2024-09-17T15:32:05.608 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1551241 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:602: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: finishing write tid 3 to smithi20328086-3 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: finishing write tid 4 to smithi20328086-3 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: oid 3 updating version 0 to 143 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: oid 3 version 143 is already newer than 142 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: oid 3 updating version 143 to 144 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 144 (ObjNum 245 snap 63 seq_num 245) dirty exists 2024-09-17T15:32:05.642 INFO:tasks.rados.rados.0.smithi203.stdout:599: left oid 3 (ObjNum 245 snap 63 seq_num 245) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:601: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:602: finishing copy_from to smithi20328086-6 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 67 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:601: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:601: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 148 (ObjNum 237 snap 58 seq_num 237) dirty exists 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:598: done (7 left) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:599: done (6 left) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:600: done (5 left) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:601: done (4 left) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:602: done (3 left) 2024-09-17T15:32:05.643 INFO:tasks.rados.rados.0.smithi203.stdout:603: done (2 left) 2024-09-17T15:32:05.644 INFO:tasks.rados.rados.0.smithi203.stdout:604: done (1 left) 2024-09-17T15:32:05.644 INFO:tasks.rados.rados.0.smithi203.stdout:606: done (0 left) 2024-09-17T15:32:05.644 INFO:tasks.rados.rados.0.smithi203.stdout:607: write (excl) oid 21 current snap is 63 2024-09-17T15:32:05.644 INFO:tasks.rados.rados.0.smithi203.stdout:607: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-17T15:32:05.662 INFO:tasks.rados.rados.0.smithi203.stdout:607: writing smithi20328086-21 from 551329 to 1261128 tid 1 2024-09-17T15:32:05.677 INFO:tasks.rados.rados.0.smithi203.stdout:607: writing smithi20328086-21 from 1870817 to 2151329 tid 2 2024-09-17T15:32:05.678 INFO:tasks.rados.rados.0.smithi203.stdout:607: finishing write tid 1 to smithi20328086-21 2024-09-17T15:32:05.678 INFO:tasks.rados.rados.0.smithi203.stdout:608: rollback oid 7 current snap is 63 2024-09-17T15:32:05.678 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 59 2024-09-17T15:32:05.678 INFO:tasks.rados.rados.0.smithi203.stdout:609: delete oid 19 current snap is 63 2024-09-17T15:32:05.686 INFO:tasks.rados.rados.0.smithi203.stdout:608: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:32:05.686 INFO:tasks.rados.rados.0.smithi203.stdout:608: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:32:05.688 INFO:tasks.rados.rados.0.smithi203.stdout:608: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:32:05.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 111 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:32:05.688 INFO:tasks.rados.rados.0.smithi203.stdout:608: done (2 left) 2024-09-17T15:32:05.688 INFO:tasks.rados.rados.0.smithi203.stdout:609: done (1 left) 2024-09-17T15:32:05.688 INFO:tasks.rados.rados.0.smithi203.stdout:610: snap_create 2024-09-17T15:32:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:607: finishing write tid 2 to smithi20328086-21 2024-09-17T15:32:05.693 INFO:tasks.rados.rados.0.smithi203.stdout:607: finishing write tid 3 to smithi20328086-21 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:607: finishing write tid 4 to smithi20328086-21 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:607: oid 21 updating version 0 to 232 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:607: oid 21 version 232 is already newer than 228 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:607: oid 21 version 232 is already newer than 231 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 232 (ObjNum 247 snap 63 seq_num 247) dirty exists 2024-09-17T15:32:05.694 INFO:tasks.rados.rados.0.smithi203.stdout:607: left oid 21 (ObjNum 247 snap 63 seq_num 247) 2024-09-17T15:32:06.485 INFO:tasks.rados.rados.0.smithi203.stdout:607: done (1 left) 2024-09-17T15:32:06.486 INFO:tasks.rados.rados.0.smithi203.stdout:610: done (0 left) 2024-09-17T15:32:06.486 INFO:tasks.rados.rados.0.smithi203.stdout:611: read oid 25 snap -1 2024-09-17T15:32:06.486 INFO:tasks.rados.rados.0.smithi203.stdout:611: expect (ObjNum 234 snap 58 seq_num 234) 2024-09-17T15:32:06.486 INFO:tasks.rados.rados.0.smithi203.stdout:612: write oid 47 current snap is 64 2024-09-17T15:32:06.486 INFO:tasks.rados.rados.0.smithi203.stdout:612: seq_num 249 ranges {584943=460297,1587781=597162} 2024-09-17T15:32:06.504 INFO:tasks.rados.rados.0.smithi203.stdout:612: writing smithi20328086-47 from 584943 to 1045240 tid 1 2024-09-17T15:32:06.522 INFO:tasks.rados.rados.0.smithi203.stdout:612: writing smithi20328086-47 from 1587781 to 2184943 tid 2 2024-09-17T15:32:06.523 INFO:tasks.rados.rados.0.smithi203.stdout:613: snap_create 2024-09-17T15:32:06.523 INFO:tasks.rados.rados.0.smithi203.stdout:611: expect (ObjNum 234 snap 58 seq_num 234) 2024-09-17T15:32:06.612 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:06.700 INFO:tasks.rados.rados.0.smithi203.stdout:612: finishing write tid 1 to smithi20328086-47 2024-09-17T15:32:06.700 INFO:tasks.rados.rados.0.smithi203.stdout:612: finishing write tid 2 to smithi20328086-47 2024-09-17T15:32:06.700 INFO:tasks.rados.rados.0.smithi203.stdout:612: finishing write tid 3 to smithi20328086-47 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:612: finishing write tid 4 to smithi20328086-47 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:612: oid 47 updating version 0 to 135 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:612: oid 47 version 135 is already newer than 133 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:612: oid 47 version 135 is already newer than 134 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 135 (ObjNum 249 snap 64 seq_num 249) dirty exists 2024-09-17T15:32:06.701 INFO:tasks.rados.rados.0.smithi203.stdout:612: left oid 47 (ObjNum 249 snap 64 seq_num 249) 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:611: done (2 left) 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:612: done (1 left) 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:613: done (0 left) 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:614: rollback oid 11 current snap is 65 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 63 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:615: copy_from oid 37 from oid 1 current snap is 65 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:616: write oid 7 current snap is 65 2024-09-17T15:32:07.445 INFO:tasks.rados.rados.0.smithi203.stdout:616: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-17T15:32:07.465 INFO:tasks.rados.rados.0.smithi203.stdout:616: writing smithi20328086-7 from 601750 to 1137296 tid 1 2024-09-17T15:32:07.485 INFO:tasks.rados.rados.0.smithi203.stdout:616: writing smithi20328086-7 from 1929910 to 2201750 tid 2 2024-09-17T15:32:07.485 INFO:tasks.rados.rados.0.smithi203.stdout:617: read oid 41 snap -1 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:617: expect deleted 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:618: snap_create 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:614: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:614: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:614: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 169 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:32:07.486 INFO:tasks.rados.rados.0.smithi203.stdout:616: finishing write tid 1 to smithi20328086-7 2024-09-17T15:32:07.489 INFO:tasks.rados.rados.0.smithi203.stdout:616: finishing write tid 2 to smithi20328086-7 2024-09-17T15:32:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:615: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: finishing write tid 3 to smithi20328086-7 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: finishing write tid 4 to smithi20328086-7 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: oid 7 updating version 0 to 113 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: oid 7 updating version 113 to 114 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: oid 7 updating version 114 to 115 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 115 (ObjNum 250 snap 65 seq_num 250) dirty exists 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:616: left oid 7 (ObjNum 250 snap 65 seq_num 250) 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:615: finishing copy_from to smithi20328086-37 2024-09-17T15:32:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 131 (ObjNum 222 snap 56 seq_num 222) dirty exists 2024-09-17T15:32:07.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: get_monmap_and_config 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: build_initial_monmap 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: monmap: 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:08.021782+0000 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:08.021782+0000 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:08.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:08.022 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 Processor -- start 2024-09-17T15:32:08.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 -- start start 2024-09-17T15:32:08.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: init 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 0x7f941406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): picked mon.noname-c con 0x7f94140689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9414068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f941413aff0 0x7f941413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): picked mon.noname-a con 0x7f941413aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f941409c8b0 con 0x7f9414068db0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9414074500 con 0x7f941413aff0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f941409cb90 con 0x7f94140689d0 2024-09-17T15:32:08.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9419726640 10 monclient(hunting): authenticate will time out at 1260.987334s 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f941413aff0 0x7f941413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f941413aff0 0x7f941413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47820/0 (socket says 172.21.15.86:47820) 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 -- 172.21.15.86:0/2623865548 learned_addr learned my addr 172.21.15.86:0/2623865548 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): get_auth_request con 0x7f941413aff0 auth_method 0 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9412ffd640 1 --2- 172.21.15.86:0/2623865548 >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 0x7f941406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient(hunting): handle_auth_done global_id 4728 payload 274 2024-09-17T15:32:08.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 -- 172.21.15.86:0/2623865548 >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 msgr2=0x7f941406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 --2- 172.21.15.86:0/2623865548 >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 0x7f941406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 -- 172.21.15.86:0/2623865548 >> v1:172.21.15.203:6789/0 conn(0x7f9414068db0 legacy=0x7f9414069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f94137fe640 1 -- 172.21.15.86:0/2623865548 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941409ce70 con 0x7f941413aff0 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9412ffd640 10 monclient: get_auth_request con 0x7f94140689d0 auth_method 0 2024-09-17T15:32:08.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.016+0000 7f9412ffd640 1 --2- 172.21.15.86:0/2623865548 >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 0x7f941406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94137fe640 1 -- 172.21.15.86:0/2623865548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f941413aff0 msgr2=0x7f941413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94137fe640 1 --2- 172.21.15.86:0/2623865548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f941413aff0 0x7f941413d3e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9414069940 tx=0x7f940c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56da73ab645791c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 1 -- 172.21.15.86:0/2623865548 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941409c2d0 con 0x7f941413aff0 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: dump: 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 1 -- 172.21.15.86:0/2623865548 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940c0a5730 con 0x7f941413aff0 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 5 monclient: authenticate success, global_id 4728 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:08.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 1 -- 172.21.15.86:0/2623865548 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c0a5b00 con 0x7f941413aff0 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 4 set_mon_vals no callback set 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9411ffb640 10 monclient: dump: 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: shutdown 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- 172.21.15.86:0/2623865548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f941413aff0 msgr2=0x7f941413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- 172.21.15.86:0/2623865548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f941413aff0 0x7f941413d3e0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9414069940 tx=0x7f940c01c890 comp rx=0 tx=0).stop 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- 172.21.15.86:0/2623865548 shutdown_connections 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- 172.21.15.86:0/2623865548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f941413aff0 0x7f941413d3e0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- 172.21.15.86:0/2623865548 >> v2:172.21.15.203:3300/0 conn(0x7f94140689d0 0x7f941406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- 172.21.15.86:0/2623865548 >> 172.21.15.86:0/2623865548 conn(0x7f94140a6440 msgr2=0x7f94140a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- 172.21.15.86:0/2623865548 shutdown_connections 2024-09-17T15:32:08.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- 172.21.15.86:0/2623865548 wait complete. 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: build_initial_monmap 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: monmap: 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 Processor -- start 2024-09-17T15:32:08.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- start start 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: init 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): picked mon.noname-c con 0x7f94140689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): picked mon.noname-a con 0x7f9414068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f941413aff0 0x7f9414070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): picked mon.noname-b con 0x7f941413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94141405f0 con 0x7f9414068db0 2024-09-17T15:32:08.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9414140470 con 0x7f941413aff0 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9414140770 con 0x7f94140689d0 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9419726640 10 monclient(hunting): authenticate will time out at 1260.994710s 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56172/0 (socket says 172.21.15.86:56172) 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47828/0 (socket says 172.21.15.86:47828) 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 -- 172.21.15.86:0/985496555 learned_addr learned my addr 172.21.15.86:0/985496555 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): get_auth_request con 0x7f94140689d0 auth_method 0 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:08.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): get_auth_request con 0x7f9414068db0 auth_method 0 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94137fe640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f941413aff0 0x7f9414070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient(hunting): handle_auth_done global_id 44189 payload 824 2024-09-17T15:32:08.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 -- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f941413aff0 msgr2=0x7f9414070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f941413aff0 0x7f9414070d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 -- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 msgr2=0x7f9414132a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 monclient: found mon.noname-c 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f9412ffd640 1 -- 172.21.15.86:0/985496555 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941411d0d0 con 0x7f94140689d0 2024-09-17T15:32:08.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.024+0000 7f94127fc640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:32:08.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9412ffd640 1 -- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 msgr2=0x7f9414132540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:08.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9412ffd640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f940800c7c0 tx=0x7f940800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:08.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408007c40 con 0x7f94140689d0 2024-09-17T15:32:08.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: dump: 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:08.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9408007de0 con 0x7f94140689d0 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408011630 con 0x7f94140689d0 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 10 monclient: dump: 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:08.043 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 5 monclient: authenticate success, global_id 44189 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _renew_subs 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9414073130 con 0x7f94140689d0 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f94117fa640 4 set_mon_vals no callback set 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _renew_subs 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f941413b5d0 con 0x7f94140689d0 2024-09-17T15:32:08.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f941413d5a0 con 0x7f94140689d0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f94080117d0 con 0x7f94140689d0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 0x7f940003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(148..148 src has 1..148) ==== 4783+0+0 (secure 0 0 0) 0x7f94080507c0 con 0x7f94140689d0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f94127fc640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 0x7f940003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.028+0000 7f94127fc640 10 monclient: get_auth_request con 0x7f940003bb60 auth_method 0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.032+0000 7f94127fc640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 0x7f940003e020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f94140749b0 tx=0x7f9404007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.032+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9408015020 con 0x7f94140689d0 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.032+0000 7f93fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:08.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.032+0000 7f93fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:08.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f9419726640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f9419726640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f9419726640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f941406d4f0 con 0x7f94140689d0 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f93fb7fe640 1 -- 172.21.15.86:0/985496555 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9408040630 con 0x7f94140689d0 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f93fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:08.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.260+0000 7f93fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:08.267 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 msgr2=0x7f940003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 0x7f940003e020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f94140749b0 tx=0x7f9404007e40 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 10 monclient: shutdown 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 msgr2=0x7f9414132540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f940800c7c0 tx=0x7f940800cc80 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 shutdown_connections 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940003bb60 0x7f940003e020 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f941413aff0 0x7f9414070d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9414068db0 0x7f9414132a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 --2- 172.21.15.86:0/985496555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94140689d0 0x7f9414132540 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 >> 172.21.15.86:0/985496555 conn(0x7f94140a6440 msgr2=0x7f94140a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:08.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 shutdown_connections 2024-09-17T15:32:08.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:08.264+0000 7f9419726640 1 -- 172.21.15.86:0/985496555 wait complete. 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:614: done (4 left) 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:615: done (3 left) 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:616: done (2 left) 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:617: done (1 left) 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:618: done (0 left) 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:619: copy_from oid 48 from oid 13 current snap is 66 2024-09-17T15:32:08.494 INFO:tasks.rados.rados.0.smithi203.stdout:620: snap_create 2024-09-17T15:32:08.532 INFO:tasks.rados.rados.0.smithi203.stdout:619: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:32:08.538 INFO:tasks.rados.rados.0.smithi203.stdout:619: finishing copy_from to smithi20328086-48 2024-09-17T15:32:08.538 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 115 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:32:09.493 INFO:tasks.rados.rados.0.smithi203.stdout:619: done (1 left) 2024-09-17T15:32:09.493 INFO:tasks.rados.rados.0.smithi203.stdout:620: done (0 left) 2024-09-17T15:32:09.493 INFO:tasks.rados.rados.0.smithi203.stdout:621: read oid 29 snap 59 2024-09-17T15:32:09.493 INFO:tasks.rados.rados.0.smithi203.stdout:621: expect (ObjNum 188 snap 46 seq_num 188) 2024-09-17T15:32:09.493 INFO:tasks.rados.rados.0.smithi203.stdout:622: read oid 12 snap -1 2024-09-17T15:32:09.494 INFO:tasks.rados.rados.0.smithi203.stdout:622: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:32:09.494 INFO:tasks.rados.rados.0.smithi203.stdout:623: snap_remove snap 66 2024-09-17T15:32:10.687 INFO:tasks.rados.rados.0.smithi203.stdout:623: done (2 left) 2024-09-17T15:32:10.687 INFO:tasks.rados.rados.0.smithi203.stdout:624: read oid 14 snap 64 2024-09-17T15:32:10.688 INFO:tasks.rados.rados.0.smithi203.stdout:624: expect (ObjNum 159 snap 36 seq_num 159) 2024-09-17T15:32:10.688 INFO:tasks.rados.rados.0.smithi203.stdout:625: delete oid 1 current snap is 67 2024-09-17T15:32:10.688 INFO:tasks.rados.rados.0.smithi203.stdout:621: expect (ObjNum 188 snap 46 seq_num 188) 2024-09-17T15:32:10.698 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1779950 2024-09-17T15:32:10.762 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1458205 2024-09-17T15:32:10.792 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:10.828 INFO:tasks.rados.rados.0.smithi203.stdout:622: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:32:10.828 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:10.880 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 110760 2024-09-17T15:32:10.905 INFO:tasks.rados.rados.0.smithi203.stdout:624: expect (ObjNum 159 snap 36 seq_num 159) 2024-09-17T15:32:10.930 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2599248 2024-09-17T15:32:10.935 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:11.014 INFO:tasks.rados.rados.0.smithi203.stdout:621: done (3 left) 2024-09-17T15:32:11.014 INFO:tasks.rados.rados.0.smithi203.stdout:622: done (2 left) 2024-09-17T15:32:11.014 INFO:tasks.rados.rados.0.smithi203.stdout:624: done (1 left) 2024-09-17T15:32:11.014 INFO:tasks.rados.rados.0.smithi203.stdout:625: done (0 left) 2024-09-17T15:32:11.014 INFO:tasks.rados.rados.0.smithi203.stdout:626: snap_create 2024-09-17T15:32:11.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: get_monmap_and_config 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: monmap: 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:11.413833+0000 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:11.413833+0000 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.413 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:11.414 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:11.414 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:11.414 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:11.414 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:11.414 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 Processor -- start 2024-09-17T15:32:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 -- start start 2024-09-17T15:32:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: init 2024-09-17T15:32:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-c con 0x7f67100689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6710068db0 0x7f6710069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-a con 0x7f6710068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6710135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f671009c8b0 con 0x7f6710135ae0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f671009cb90 con 0x7f6710068db0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f671009ce70 con 0x7f67100689d0 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f67164d5640 10 monclient(hunting): authenticate will time out at 1264.379412s 2024-09-17T15:32:11.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6710068db0 0x7f6710069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6710068db0 0x7f6710069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38978/0 (socket says 172.21.15.86:38978) 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 1 -- 172.21.15.86:0/465970302 learned_addr learned my addr 172.21.15.86:0/465970302 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): get_auth_request con 0x7f6710068db0 auth_method 0 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670ffff640 1 --2- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670ffff640 10 monclient(hunting): get_auth_request con 0x7f67100689d0 auth_method 0 2024-09-17T15:32:11.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.408+0000 7f670ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670ffff640 1 -- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 msgr2=0x7f671006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670ffff640 1 -- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 msgr2=0x7f671006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670ffff640 1 --2- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:11.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670ffff640 1 --2- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f6714a72640 1 -- 172.21.15.86:0/465970302 >> v1:172.21.15.86:6789/0 conn(0x7f6710135ae0 legacy=0x7f6710135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 1 -- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 msgr2=0x7f671006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 1 --2- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 1 -- 172.21.15.86:0/465970302 >> v1:172.21.15.86:6789/0 conn(0x7f6710135ae0 legacy=0x7f6710135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:11.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.412+0000 7f670f7fe640 1 -- 172.21.15.86:0/465970302 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6710069740 con 0x7f6710068db0 2024-09-17T15:32:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670f7fe640 1 -- 172.21.15.86:0/465970302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 msgr2=0x7f6710069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670f7fe640 1 --2- 172.21.15.86:0/465970302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710069200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6710069a70 tx=0x7f670401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f590e8d6a30cdf87 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 1 -- 172.21.15.86:0/465970302 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67040a5680 con 0x7f6710068db0 2024-09-17T15:32:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: dump: 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 1 -- 172.21.15.86:0/465970302 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67040a5820 con 0x7f6710068db0 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f67164d5640 5 monclient: authenticate success, global_id 4734 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f67164d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f67164d5640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f67164d5640 4 set_mon_vals no callback set 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 1 -- 172.21.15.86:0/465970302 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67040a5bf0 con 0x7f6710068db0 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.416+0000 7f670effd640 10 monclient: dump: 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.425 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: shutdown 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- 172.21.15.86:0/465970302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 msgr2=0x7f6710069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- 172.21.15.86:0/465970302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710069200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6710069a70 tx=0x7f670401c9c0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- 172.21.15.86:0/465970302 shutdown_connections 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- 172.21.15.86:0/465970302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710069200 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- 172.21.15.86:0/465970302 >> v2:172.21.15.203:3300/0 conn(0x7f67100689d0 0x7f671006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- 172.21.15.86:0/465970302 >> 172.21.15.86:0/465970302 conn(0x7f67100a6440 msgr2=0x7f67100a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:11.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- 172.21.15.86:0/465970302 shutdown_connections 2024-09-17T15:32:11.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- 172.21.15.86:0/465970302 wait complete. 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: monmap: 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:11.431 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 Processor -- start 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- start start 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: init 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-b con 0x7f67100689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-a con 0x7f6710068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6710135ae0 0x7f671011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): picked mon.noname-c con 0x7f6710135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6710140a40 con 0x7f6710068db0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f671009c2d0 con 0x7f67100689d0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6710140bc0 con 0x7f6710135ae0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f67164d5640 10 monclient(hunting): authenticate will time out at 1264.389836s 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6710135ae0 0x7f671011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38984/0 (socket says 172.21.15.86:38984) 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 -- 172.21.15.86:0/1427682125 learned_addr learned my addr 172.21.15.86:0/1427682125 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): get_auth_request con 0x7f6710068db0 auth_method 0 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): get_auth_request con 0x7f6710135ae0 auth_method 0 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f6714a72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 10 monclient(hunting): get_auth_request con 0x7f67100689d0 auth_method 0 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient(hunting): handle_auth_done global_id 4737 payload 824 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 msgr2=0x7f671011dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 msgr2=0x7f671011dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6710135ae0 msgr2=0x7f671011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6710135ae0 0x7f671011c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 msgr2=0x7f671011dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670ffff640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.420+0000 7f670f7fe640 1 -- 172.21.15.86:0/1427682125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671011f300 con 0x7f6710068db0 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670f7fe640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 msgr2=0x7f6710134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670f7fe640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f670401a5f0 tx=0x7f6704019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67040bc020 con 0x7f6710068db0 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: dump: 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 5 monclient: authenticate success, global_id 4737 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67040b8880 con 0x7f6710068db0 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _renew_subs 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67100a3860 con 0x7f6710068db0 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67040a5680 con 0x7f6710068db0 2024-09-17T15:32:11.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 10 monclient: dump: 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _renew_subs 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f671011ed40 con 0x7f6710068db0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670e7fc640 4 set_mon_vals no callback set 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671006f840 con 0x7f6710068db0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f67040bf020 con 0x7f6710068db0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 0x7f66fc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 4783+0+0 (secure 0 0 0) 0x7f67040eae80 con 0x7f6710068db0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670ffff640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 0x7f66fc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.424+0000 7f670ffff640 10 monclient: get_auth_request con 0x7f66fc03bb60 auth_method 0 2024-09-17T15:32:11.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.428+0000 7f670ffff640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 0x7f66fc03e020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6700006aa0 tx=0x7f6700006a30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.428+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f67040c6030 con 0x7f6710068db0 2024-09-17T15:32:11.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.428+0000 7f670cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.428+0000 7f670cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:11.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f67164d5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:11.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f67164d5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:11.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f67164d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f671011f540 con 0x7f6710068db0 2024-09-17T15:32:11.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f670cff9640 1 -- 172.21.15.86:0/1427682125 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6704016640 con 0x7f6710068db0 2024-09-17T15:32:11.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f670cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:11.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.656+0000 7f670cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:11.662 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:11.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 msgr2=0x7f66fc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 0x7f66fc03e020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6700006aa0 tx=0x7f6700006a30 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 10 monclient: shutdown 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 msgr2=0x7f6710134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f670401a5f0 tx=0x7f6704019760 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 shutdown_connections 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f66fc03bb60 0x7f66fc03e020 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6710135ae0 0x7f671011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6710068db0 0x7f6710134c50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 --2- 172.21.15.86:0/1427682125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67100689d0 0x7f671011dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 >> 172.21.15.86:0/1427682125 conn(0x7f67100a6440 msgr2=0x7f671013d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 shutdown_connections 2024-09-17T15:32:11.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.660+0000 7f67164d5640 1 -- 172.21.15.86:0/1427682125 wait complete. 2024-09-17T15:32:11.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:32:11.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:32:11.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:32:11.691 INFO:tasks.rados.rados.0.smithi203.stdout:626: done (0 left) 2024-09-17T15:32:11.691 INFO:tasks.rados.rados.0.smithi203.stdout:627: rollback oid 26 current snap is 68 2024-09-17T15:32:11.691 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 62 2024-09-17T15:32:11.691 INFO:tasks.rados.rados.0.smithi203.stdout:628: read oid 43 snap 59 2024-09-17T15:32:11.691 INFO:tasks.rados.rados.0.smithi203.stdout:628: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:32:11.692 INFO:tasks.rados.rados.0.smithi203.stdout:629: read oid 35 snap 59 2024-09-17T15:32:11.692 INFO:tasks.rados.rados.0.smithi203.stdout:629: expect (ObjNum 219 snap 56 seq_num 219) 2024-09-17T15:32:11.692 INFO:tasks.rados.rados.0.smithi203.stdout:630: write (excl) oid 49 current snap is 68 2024-09-17T15:32:11.692 INFO:tasks.rados.rados.0.smithi203.stdout:630: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-17T15:32:11.710 INFO:tasks.rados.rados.0.smithi203.stdout:630: writing smithi20328086-49 from 635364 to 1037761 tid 1 2024-09-17T15:32:11.730 INFO:tasks.rados.rados.0.smithi203.stdout:630: writing smithi20328086-49 from 1763227 to 2235364 tid 2 2024-09-17T15:32:11.730 INFO:tasks.rados.rados.0.smithi203.stdout:631: write oid 36 current snap is 68 2024-09-17T15:32:11.730 INFO:tasks.rados.rados.0.smithi203.stdout:631: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-17T15:32:11.747 INFO:tasks.rados.rados.0.smithi203.stdout:631: writing smithi20328086-36 from 652171 to 1129817 tid 1 2024-09-17T15:32:11.761 INFO:tasks.rados.rados.0.smithi203.stdout:631: writing smithi20328086-36 from 1821709 to 2252171 tid 2 2024-09-17T15:32:11.761 INFO:tasks.rados.rados.0.smithi203.stdout:632: snap_remove snap 64 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: get_monmap_and_config 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: monmap: 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:11.792088+0000 2024-09-17T15:32:11.791 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:11.792088+0000 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:11.792 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 Processor -- start 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 -- start start 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: init 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:32:11.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfd40a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 0x7fdfd409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-b con 0x7fdfd40a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdfd413d2f0 0x7fdfd413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-a con 0x7fdfd413d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd40a6280 con 0x7fdfd40a50e0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd409d2a0 con 0x7fdfd413d2f0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd409c0a0 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd98c4640 10 monclient(hunting): authenticate will time out at 1264.757467s 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 0x7fdfd409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdfd40a50e0 0x7fdfd40a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43394/0 (socket says 172.21.15.86:43394) 2024-09-17T15:32:11.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd2ffd640 1 -- 172.21.15.86:0/3527137727 learned_addr learned my addr 172.21.15.86:0/3527137727 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): get_auth_request con 0x7fdfd40a5bd0 auth_method 0 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd37fe640 1 --2- 172.21.15.86:0/3527137727 >> v2:172.21.15.86:3300/0 conn(0x7fdfd413d2f0 0x7fdfd413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd37fe640 10 monclient(hunting): get_auth_request con 0x7fdfd413d2f0 auth_method 0 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient(hunting): handle_auth_done global_id 44195 payload 274 2024-09-17T15:32:11.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 -- 172.21.15.86:0/3527137727 >> v2:172.21.15.86:3300/0 conn(0x7fdfd413d2f0 msgr2=0x7fdfd413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 --2- 172.21.15.86:0/3527137727 >> v2:172.21.15.86:3300/0 conn(0x7fdfd413d2f0 0x7fdfd413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 -- 172.21.15.86:0/3527137727 >> v1:172.21.15.86:6789/0 conn(0x7fdfd40a50e0 legacy=0x7fdfd40a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 monclient: found mon.noname-b 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:32:11.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.788+0000 7fdfd27fc640 1 -- 172.21.15.86:0/3527137727 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd409c380 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd27fc640 1 -- 172.21.15.86:0/3527137727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 msgr2=0x7fdfd409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd27fc640 1 --2- 172.21.15.86:0/3527137727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 0x7fdfd409cd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80097c0 tx=0x7fdfc801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9077b67004f86c1b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 1 -- 172.21.15.86:0/3527137727 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80b0070 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: dump: 2024-09-17T15:32:11.800 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.801 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 1 -- 172.21.15.86:0/3527137727 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc80ab600 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 5 monclient: authenticate success, global_id 44195 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 1 -- 172.21.15.86:0/3527137727 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80a4e60 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 4 set_mon_vals no callback set 2024-09-17T15:32:11.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd1ffb640 10 monclient: dump: 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 10 monclient: shutdown 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:11.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- 172.21.15.86:0/3527137727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 msgr2=0x7fdfd409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/3527137727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 0x7fdfd409cd60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdfc80097c0 tx=0x7fdfc801c330 comp rx=0 tx=0).stop 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- 172.21.15.86:0/3527137727 shutdown_connections 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/3527137727 >> v2:172.21.15.86:3300/0 conn(0x7fdfd413d2f0 0x7fdfd413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/3527137727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd40a5bd0 0x7fdfd409cd60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- 172.21.15.86:0/3527137727 >> 172.21.15.86:0/3527137727 conn(0x7fdfd40aaae0 msgr2=0x7fdfd40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- 172.21.15.86:0/3527137727 shutdown_connections 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- 172.21.15.86:0/3527137727 wait complete. 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 10 monclient: monmap: 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 Processor -- start 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 1 -- start start 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.792+0000 7fdfd98c4640 10 monclient: init 2024-09-17T15:32:11.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd40a50e0 0x7fdfd41213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-a con 0x7fdfd40a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfd40a5bd0 0x7fdfd409fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-b con 0x7fdfd40a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 0x7fdfd4138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): picked mon.noname-c con 0x7fdfd413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd4142650 con 0x7fdfd40a50e0 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd40a6b00 con 0x7fdfd40a5bd0 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd41427d0 con 0x7fdfd413d2f0 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): authenticate will time out at 1264.764152s 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd40a50e0 0x7fdfd41213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 0x7fdfd4138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 0x7fdfd4138260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59636/0 (socket says 172.21.15.86:59636) 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 learned_addr learned my addr 172.21.15.86:0/171882289 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): get_auth_request con 0x7fdfd413d2f0 auth_method 0 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd40a50e0 0x7fdfd41213f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38994/0 (socket says 172.21.15.86:38994) 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): get_auth_request con 0x7fdfd40a50e0 auth_method 0 2024-09-17T15:32:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient(hunting): handle_auth_done global_id 44198 payload 824 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd27fc640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfd40a5bd0 0x7fdfd409fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfd40a5bd0 msgr2=0x7fdfd409fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfd40a5bd0 0x7fdfd409fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd40a50e0 msgr2=0x7fdfd41213f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd40a50e0 0x7fdfd41213f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 monclient: found mon.noname-c 2024-09-17T15:32:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:32:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd40a12b0 con 0x7fdfd413d2f0 2024-09-17T15:32:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 msgr2=0x7fdfd4138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd37fe640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 0x7fdfd4138260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d600 tx=0x7fdfc400dac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4012800 con 0x7fdfd413d2f0 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: dump: 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfc4019070 con 0x7fdfd413d2f0 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4012c00 con 0x7fdfd413d2f0 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfbbfff640 10 monclient: dump: 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.814 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 5 monclient: authenticate success, global_id 44198 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _renew_subs 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd17fa640 4 set_mon_vals no callback set 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd40a0d00 con 0x7fdfd413d2f0 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _renew_subs 2024-09-17T15:32:11.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd413d940 con 0x7fdfd413d2f0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 0x7fdfd41cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd406a0e0 con 0x7fdfd41387a0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 0x7fdfd41cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient: get_auth_request con 0x7fdfd41387a0 auth_method 0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.796+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdfc401e460 con 0x7fdfd413d2f0 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_done global_id 4746 payload 824 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfd409cd60 con 0x7fdfd41387a0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdfc003bbb0 0x7fdfc003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 4755+0+0 (secure 0 0 0) 0x7fdfc40502e0 con 0x7fdfd413d2f0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd27fc640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdfc003bbb0 0x7fdfc003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd27fc640 10 monclient: get_auth_request con 0x7fdfc003bbb0 auth_method 0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 msgr2=0x7fdfd41cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd2ffd640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 0x7fdfd41cfba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc007980 tx=0x7fdfcc00f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc017030 con 0x7fdfd41387a0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 10 monclient: dump: 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdfcc006020 con 0x7fdfd41387a0 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 msgr2=0x7fdfd41cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 0x7fdfd41cfba0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc007980 tx=0x7fdfcc00f5a0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 msgr2=0x7fdfd41cfba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfbbfff640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41387a0 0x7fdfd41cfba0 secure :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc007980 tx=0x7fdfcc00f5a0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.800+0000 7fdfd27fc640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdfc003bbb0 0x7fdfc003e070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40aa620 tx=0x7fdfc8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 0x7fdfd41d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd41d0a00 con 0x7fdfd41d00e0 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 0x7fdfd41d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient: get_auth_request con 0x7fdfd41d00e0 auth_method 0 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2024-09-17T15:32:11.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 1 -- 172.21.15.86:0/171882289 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdfd4074be0 con 0x7fdfd41d00e0 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 msgr2=0x7fdfd41d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd2ffd640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 0x7fdfd41d04c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc017000 tx=0x7fdfcc004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfcc01d240 con 0x7fdfd41d00e0 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 10 monclient: dump: 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.821 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fdfcc01d240 con 0x7fdfd41d00e0 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 msgr2=0x7fdfd41d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 0x7fdfd41d04c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc017000 tx=0x7fdfcc004cd0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 msgr2=0x7fdfd41d04c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:11.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfbbfff640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfd41d00e0 0x7fdfd41d04c0 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc017000 tx=0x7fdfcc004cd0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 304452, 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:11.823 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.824 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.825 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:11.826 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.827 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.828 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.830 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:11.831 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.832 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:11.833 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.834 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.835 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:11.836 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:11.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdfc003bbb0 msgr2=0x7fdfc003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdfc003bbb0 0x7fdfc003e070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdfd40aa620 tx=0x7fdfc8008040 comp rx=0 tx=0).stop 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 10 monclient: shutdown 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 msgr2=0x7fdfd4138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 --2- 172.21.15.86:0/171882289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfd413d2f0 0x7fdfd4138260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d600 tx=0x7fdfc400dac0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 reap_dead start 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd37fe640 1 -- 172.21.15.86:0/171882289 reap_dead start 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 shutdown_connections 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 >> 172.21.15.86:0/171882289 conn(0x7fdfd40aaae0 msgr2=0x7fdfd40a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 shutdown_connections 2024-09-17T15:32:11.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.808+0000 7fdfd98c4640 1 -- 172.21.15.86:0/171882289 wait complete. 2024-09-17T15:32:11.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:11.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:32:11.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.944+0000 7fe6bf5df640 10 monclient: get_monmap_and_config 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.944+0000 7fe6bf5df640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.944+0000 7fe6bf5df640 10 monclient: monmap: 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:11.951954+0000 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:11.951954+0000 2024-09-17T15:32:11.951 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:11.952 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 Processor -- start 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 -- start start 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient: init 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 0x7fe6b809f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b80a01a0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6b80a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6b80a1380 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b8074e00 con 0x7fe6b80a1380 2024-09-17T15:32:11.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b809e350 con 0x7fe6b80a0580 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b809e630 con 0x7fe6b80a01a0 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bf5df640 10 monclient(hunting): authenticate will time out at 1264.917397s 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bdb7c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe6b80a1380 0x7fe6b813ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43404/0 (socket says 172.21.15.86:43404) 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bdb7c640 1 -- 172.21.15.86:0/3725483991 learned_addr learned my addr 172.21.15.86:0/3725483991 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 0x7fe6b809f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request con 0x7fe6b80a01a0 auth_method 0 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 msgr2=0x7fe6b809f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:11.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 msgr2=0x7fe6b809f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 0x7fe6b809f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:11.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 0x7fe6b809f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6bdb7c640 1 -- 172.21.15.86:0/3725483991 >> v1:172.21.15.86:6789/0 conn(0x7fe6b80a1380 legacy=0x7fe6b813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694188895 0 0) 0x7fe6b8074e00 con 0x7fe6b80a1380 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 10 monclient(hunting): _negotiate 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 10 monclient(hunting): my global_id is 4752 2024-09-17T15:32:11.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.948+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6a4003530 con 0x7fe6b80a1380 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3234041224 0 0) 0x7fe6a4003530 con 0x7fe6b80a1380 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 msgr2=0x7fe6b809f010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 --2- 172.21.15.86:0/3725483991 >> v2:172.21.15.203:3300/0 conn(0x7fe6b80a01a0 0x7fe6b809f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 >> v1:172.21.15.203:6789/0 conn(0x7fe6b80a0580 legacy=0x7fe6b80a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b809e910 con 0x7fe6b80a1380 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fe6b0001d30 con 0x7fe6b80a1380 2024-09-17T15:32:11.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: dump: 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 >> v1:172.21.15.86:6789/0 conn(0x7fe6b80a1380 legacy=0x7fe6b813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a4003ad0 0x7fe6a4003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): picked mon.b con 0x7fe6a4003ad0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): picked mon.a con 0x7fe6a4004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6a4005400 0x7fe6a400ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): picked mon.c con 0x7fe6a4005400 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b809e630 con 0x7fe6a4004810 2024-09-17T15:32:11.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a400c350 con 0x7fe6a4003ad0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a400c550 con 0x7fe6a4005400 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6abfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request con 0x7fe6a4004810 auth_method 0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6a4005400 0x7fe6a400ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): get_auth_request con 0x7fe6a4005400 auth_method 0 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a4003ad0 0x7fe6a4003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request con 0x7fe6a4003ad0 auth_method 0 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bd37b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bd37b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_done global_id 4752 payload 293 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6a4005400 msgr2=0x7fe6a400ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6a4005400 0x7fe6a400ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3725483991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a4003ad0 msgr2=0x7fe6a4003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6a4003ad0 0x7fe6a4003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 monclient: found mon.a 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3725483991 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a4003530 con 0x7fe6a4004810 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bcb7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bdb7c640 1 -- 172.21.15.86:0/3725483991 reap_dead start 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bf5df640 5 monclient: authenticate success, global_id 4752 2024-09-17T15:32:11.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.952+0000 7fe6bf5df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6a4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6b809fa50 tx=0x7fe6b401c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5578890963328a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40b0070 con 0x7fe6a4004810 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: dump: 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.971 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b40ab620 con 0x7fe6a4004810 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 4 set_mon_vals no callback set 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 1 -- 172.21.15.86:0/3725483991 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40039b0 con 0x7fe6a4004810 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6abfff640 10 monclient: dump: 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: shutdown 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6a4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6b809fa50 tx=0x7fe6b401c2c0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3725483991 shutdown_connections 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3725483991 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6a4004c90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3725483991 >> 172.21.15.86:0/3725483991 conn(0x7fe6b80a2f40 msgr2=0x7fe6b80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3725483991 shutdown_connections 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3725483991 wait complete. 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: build_initial_monmap 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: monmap: 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 Processor -- start 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- start start 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: init 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6b811fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-a con 0x7fe6a4004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b811dbd0 0x7fe6b8133b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6b811dbd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b8134090 0x7fe6b81d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): picked mon.noname-b con 0x7fe6b8134090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8136bd0 con 0x7fe6a4004810 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b8136a50 con 0x7fe6b8134090 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b8136d50 con 0x7fe6b811dbd0 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bf5df640 10 monclient(hunting): authenticate will time out at 1264.927289s 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6b811fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b811dbd0 0x7fe6b8133b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6b811fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39048/0 (socket says 172.21.15.86:39048) 2024-09-17T15:32:11.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3816644855 learned_addr learned my addr 172.21.15.86:0/3816644855 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request con 0x7fe6a4004810 auth_method 0 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request con 0x7fe6b811dbd0 auth_method 0 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient(hunting): handle_auth_done global_id 4755 payload 824 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bdb7c640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b8134090 0x7fe6b81d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b811dbd0 msgr2=0x7fe6b8133b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b811dbd0 0x7fe6b8133b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b8134090 msgr2=0x7fe6b81d0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6b8134090 0x7fe6b81d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient: _finish_hunting 0 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 monclient: found mon.noname-a 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.956+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b811f1b0 con 0x7fe6a4004810 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bd37b640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6b811fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bd37b640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6b811fd10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac010870 tx=0x7fe6ac010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac019020 con 0x7fe6a4004810 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: dump: 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.979 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6ac00eb60 con 0x7fe6a4004810 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 5 monclient: authenticate success, global_id 4755 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 monclient: _renew_subs 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b8120c70 con 0x7fe6a4004810 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac00f040 con 0x7fe6a4004810 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 10 monclient: dump: 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 monclient: _renew_subs 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b811e200 con 0x7fe6a4004810 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6ab7fe640 4 set_mon_vals no callback set 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 0x7fe6b81d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b81d1060 con 0x7fe6b81d0740 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 0x7fe6b81d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient: get_auth_request con 0x7fe6b81d0740 auth_method 0 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe6ac01c020 con 0x7fe6a4004810 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_done global_id 44204 payload 824 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6b809e350 con 0x7fe6b81d0740 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe69c03bb60 0x7fe69c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4755+0+0 (secure 0 0 0) 0x7fe6ac0503f0 con 0x7fe6a4004810 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bdb7c640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe69c03bb60 0x7fe69c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.960+0000 7fe6bdb7c640 10 monclient: get_auth_request con 0x7fe69c03bb60 auth_method 0 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 msgr2=0x7fe6b81d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 0x7fe6b81d0b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40ad000 tx=0x7fe6a4005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40b0040 con 0x7fe6b81d0740 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:11.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 10 monclient: dump: 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6b40ab600 con 0x7fe6b81d0740 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 msgr2=0x7fe6b81d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 0x7fe6b81d0b20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40ad000 tx=0x7fe6a4005840 comp rx=0 tx=0).stop 2024-09-17T15:32:11.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 msgr2=0x7fe6b81d0b20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:11.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6a9ffb640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b81d0740 0x7fe6b81d0b20 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40ad000 tx=0x7fe6a4005840 comp rx=0 tx=0).stop 2024-09-17T15:32:11.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.964+0000 7fe6bdb7c640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe69c03bb60 0x7fe69c03e020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80a2c60 tx=0x7fe6b0006040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:11.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 0x7fe6b813d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:11.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b81344a0 con 0x7fe6b813d500 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 0x7fe6b813d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient: get_auth_request con 0x7fe6b813d500 auth_method 0 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 10 monclient(hunting): handle_auth_done global_id 44207 payload 824 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3816644855 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6b806d4f0 con 0x7fe6b813d500 2024-09-17T15:32:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 msgr2=0x7fe6b813d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bcb7a640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 0x7fe6b813d910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6b401c300 tx=0x7fe6b40a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40b0040 con 0x7fe6b813d500 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 10 monclient: dump: 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:11.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fe6b40c7410 con 0x7fe6b813d500 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 msgr2=0x7fe6b813d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 0x7fe6b813d910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6b401c300 tx=0x7fe6b40a6000 comp rx=0 tx=0).stop 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 msgr2=0x7fe6b813d910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6a9ffb640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6b813d500 0x7fe6b813d910 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe6b401c300 tx=0x7fe6b40a6000 comp rx=0 tx=0).stop 2024-09-17T15:32:11.989 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 60926, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:32:11.990 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.991 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.992 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:11.993 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:11.994 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.995 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:11.996 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:11.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:12.000 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:12.001 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:12.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe69c03bb60 msgr2=0x7fe69c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe69c03bb60 0x7fe69c03e020 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80a2c60 tx=0x7fe6b0006040 comp rx=0 tx=0).stop 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 10 monclient: shutdown 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 msgr2=0x7fe6b811fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 --2- 172.21.15.86:0/3816644855 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6a4004810 0x7fe6b811fd10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac010870 tx=0x7fe6ac010d30 comp rx=0 tx=0).stop 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bdb7c640 1 -- 172.21.15.86:0/3816644855 reap_dead start 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bdb7c640 1 -- 172.21.15.86:0/3816644855 reap_dead start 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 shutdown_connections 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 >> 172.21.15.86:0/3816644855 conn(0x7fe6b80a2f40 msgr2=0x7fe6b80a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 shutdown_connections 2024-09-17T15:32:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:11.972+0000 7fe6bf5df640 1 -- 172.21.15.86:0/3816644855 wait complete. 2024-09-17T15:32:12.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:12.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:32:12.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: get_monmap_and_config 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: build_initial_monmap 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: monmap: 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:12.120097+0000 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:12.120097+0000 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:12.120 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 Processor -- start 2024-09-17T15:32:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 -- start start 2024-09-17T15:32:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: init 2024-09-17T15:32:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-c con 0x7f42e40a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f42e40a1590 0x7f42e4135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-a con 0x7f42e40a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42e4135f90 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e40a2130 con 0x7f42e4135f90 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f42e40a2410 con 0x7f42e40a1590 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f42e40a26f0 con 0x7f42e40a0c70 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42eb260640 10 monclient(hunting): authenticate will time out at 1265.085505s 2024-09-17T15:32:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f42e40a1590 0x7f42e4135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f42e40a1590 0x7f42e4135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39052/0 (socket says 172.21.15.86:39052) 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e97fd640 1 --1- 172.21.15.86:0/1026584945 >> v1:172.21.15.86:6789/0 conn(0x7f42e4135f90 0x7f42e413cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43416/0 (socket says 172.21.15.86:43416) 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 1 -- 172.21.15.86:0/1026584945 learned_addr learned my addr 172.21.15.86:0/1026584945 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): get_auth_request con 0x7f42e40a1590 auth_method 0 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request con 0x7f42e40a0c70 auth_method 0 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 msgr2=0x7f42e40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 msgr2=0x7f42e40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42e97fd640 1 -- 172.21.15.86:0/1026584945 >> v1:172.21.15.86:6789/0 conn(0x7f42e4135f90 legacy=0x7f42e413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.116+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 10 monclient(hunting): handle_auth_done global_id 4758 payload 274 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 1 -- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 msgr2=0x7f42e40a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 1 --2- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 1 -- 172.21.15.86:0/1026584945 >> v1:172.21.15.86:6789/0 conn(0x7f42e4135f90 legacy=0x7f42e413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 1 monclient: found mon.noname-a 2024-09-17T15:32:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:12.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:12.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.120+0000 7f42dbfff640 1 -- 172.21.15.86:0/1026584945 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e409f9f0 con 0x7f42e40a1590 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42dbfff640 1 -- 172.21.15.86:0/1026584945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e40a1590 msgr2=0x7f42e4135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42dbfff640 1 --2- 172.21.15.86:0/1026584945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e40a1590 0x7f42e4135a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42e406d160 tx=0x7f42d401c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a97d539420019fd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 1 -- 172.21.15.86:0/1026584945 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d40b1070 con 0x7f42e40a1590 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: dump: 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 5 monclient: authenticate success, global_id 4758 2024-09-17T15:32:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 1 -- 172.21.15.86:0/1026584945 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d40a47a0 con 0x7f42e40a1590 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 1 -- 172.21.15.86:0/1026584945 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d40a4b70 con 0x7f42e40a1590 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 4 set_mon_vals no callback set 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42db7fe640 10 monclient: dump: 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: shutdown 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- 172.21.15.86:0/1026584945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e40a1590 msgr2=0x7f42e4135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- 172.21.15.86:0/1026584945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e40a1590 0x7f42e4135a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f42e406d160 tx=0x7f42d401c060 comp rx=0 tx=0).stop 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- 172.21.15.86:0/1026584945 shutdown_connections 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- 172.21.15.86:0/1026584945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e40a1590 0x7f42e4135a50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- 172.21.15.86:0/1026584945 >> v2:172.21.15.203:3300/0 conn(0x7f42e40a0c70 0x7f42e40a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- 172.21.15.86:0/1026584945 >> 172.21.15.86:0/1026584945 conn(0x7f42e40a2f40 msgr2=0x7f42e40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- 172.21.15.86:0/1026584945 shutdown_connections 2024-09-17T15:32:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- 172.21.15.86:0/1026584945 wait complete. 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: build_initial_monmap 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: monmap: 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:12.137 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 Processor -- start 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- start start 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: init 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e40a0c70 0x7f42e41d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-b con 0x7f42e40a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42e40a1590 0x7f42e41d1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-c con 0x7f42e40a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 0x7f42e41d1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): picked mon.noname-a con 0x7f42e4135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f42e4141800 con 0x7f42e4135f90 2024-09-17T15:32:12.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42e4141680 con 0x7f42e40a0c70 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f42e4141980 con 0x7f42e40a1590 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.124+0000 7f42eb260640 10 monclient(hunting): authenticate will time out at 1265.095967s 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 0x7f42e41d1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42e40a1590 0x7f42e41d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 0x7f42e41d1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39054/0 (socket says 172.21.15.86:39054) 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 learned_addr learned my addr 172.21.15.86:0/1323032083 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): get_auth_request con 0x7f42e4135f90 auth_method 0 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42e40a1590 0x7f42e41d1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59680/0 (socket says 172.21.15.86:59680) 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): get_auth_request con 0x7f42e40a1590 auth_method 0 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e40a0c70 0x7f42e41d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:12.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request con 0x7f42e40a0c70 auth_method 0 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e8ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42e40a1590 msgr2=0x7f42e41d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42e40a1590 0x7f42e41d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e40a0c70 msgr2=0x7f42e41d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e40a0c70 0x7f42e41d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient: _finish_hunting 0 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 monclient: found mon.noname-a 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:12.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e409f610 con 0x7f42e4135f90 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 msgr2=0x7f42e41d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42e97fd640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 0x7f42e41d1680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00c900 tx=0x7f42dc00cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc018070 con 0x7f42e4135f90 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: dump: 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: _finish_auth 0 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42dc00a8a0 con 0x7f42e4135f90 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc00ac90 con 0x7f42e4135f90 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42d97fa640 10 monclient: dump: 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:12.143 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 5 monclient: authenticate success, global_id 4764 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 monclient: _renew_subs 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42daffd640 4 set_mon_vals no callback set 2024-09-17T15:32:12.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e409eec0 con 0x7f42e4135f90 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 monclient: _renew_subs 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e413c8d0 con 0x7f42e4135f90 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.128+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42e40a1050 con 0x7f42e413d310 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f42dc0070d0 con 0x7f42e4135f90 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e413d310 auth_method 0 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42d97fa640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f42d003bb60 0x7f42d003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 4755+0+0 (secure 0 0 0) 0x7f42dc04f6c0 con 0x7f42e4135f90 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42dbfff640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f42d003bb60 0x7f42d003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42dbfff640 10 monclient: get_auth_request con 0x7f42d003bb60 auth_method 0 2024-09-17T15:32:12.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.132+0000 7f42dbfff640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f42d003bb60 0x7f42d003e020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f42e406f840 tx=0x7f42d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:12.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e413d310 auth_method 0 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:12.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.336+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:32:12.442 INFO:tasks.rados.rados.0.smithi203.stdout:632: done (5 left) 2024-09-17T15:32:12.442 INFO:tasks.rados.rados.0.smithi203.stdout:633: copy_from oid 30 from oid 34 current snap is 68 2024-09-17T15:32:12.443 INFO:tasks.rados.rados.0.smithi203.stdout:627: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:32:12.443 INFO:tasks.rados.rados.0.smithi203.stdout:627: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:32:12.443 INFO:tasks.rados.rados.0.smithi203.stdout:627: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:32:12.443 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 236 (ObjNum 133 snap 29 seq_num 133) dirty exists 2024-09-17T15:32:12.443 INFO:tasks.rados.rados.0.smithi203.stdout:629: expect (ObjNum 219 snap 56 seq_num 219) 2024-09-17T15:32:12.450 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 784123 2024-09-17T15:32:12.504 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:12.548 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:12.587 INFO:tasks.rados.rados.0.smithi203.stdout:628: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:32:12.659 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:12.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:12.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e413d310 auth_method 0 2024-09-17T15:32:12.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:12.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:12.740+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:32:12.759 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2056731 2024-09-17T15:32:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:630: finishing write tid 1 to smithi20328086-49 2024-09-17T15:32:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:630: finishing write tid 2 to smithi20328086-49 2024-09-17T15:32:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:630: finishing write tid 3 to smithi20328086-49 2024-09-17T15:32:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:630: finishing write tid 4 to smithi20328086-49 2024-09-17T15:32:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:630: oid 49 updating version 0 to 240 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:630: oid 49 version 240 is already newer than 238 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:630: oid 49 version 240 is already newer than 239 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 240 (ObjNum 252 snap 68 seq_num 252) dirty exists 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:630: left oid 49 (ObjNum 252 snap 68 seq_num 252) 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:631: finishing write tid 1 to smithi20328086-36 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:631: finishing write tid 2 to smithi20328086-36 2024-09-17T15:32:12.799 INFO:tasks.rados.rados.0.smithi203.stdout:631: finishing write tid 3 to smithi20328086-36 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:631: finishing write tid 4 to smithi20328086-36 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:631: oid 36 updating version 0 to 243 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:631: oid 36 updating version 243 to 244 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:631: oid 36 version 244 is already newer than 242 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 244 (ObjNum 253 snap 68 seq_num 253) dirty exists 2024-09-17T15:32:12.800 INFO:tasks.rados.rados.0.smithi203.stdout:631: left oid 36 (ObjNum 253 snap 68 seq_num 253) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:633: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:633: finishing copy_from to smithi20328086-30 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 84 (ObjNum 158 snap 36 seq_num 158) dirty exists 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:627: done (5 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:628: done (4 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:629: done (3 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:630: done (2 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:631: done (1 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:633: done (0 left) 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:634: rollback oid 31 current snap is 68 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 59 2024-09-17T15:32:12.801 INFO:tasks.rados.rados.0.smithi203.stdout:635: snap_remove snap 67 2024-09-17T15:32:13.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:13.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e413d310 auth_method 0 2024-09-17T15:32:13.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:13.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:13.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:13.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:13.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:13.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:13.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:13.544+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:32:13.710 INFO:tasks.rados.rados.0.smithi203.stdout:635: done (1 left) 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:636: read oid 24 snap -1 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:636: expect (ObjNum 133 snap 29 seq_num 133) 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:634: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:634: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:634: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 70 (ObjNum 235 snap 58 seq_num 235) dirty exists 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:634: done (1 left) 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:637: write (excl) oid 31 current snap is 68 2024-09-17T15:32:13.711 INFO:tasks.rados.rados.0.smithi203.stdout:637: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-17T15:32:13.728 INFO:tasks.rados.rados.0.smithi203.stdout:637: writing smithi20328086-31 from 668978 to 1221873 tid 1 2024-09-17T15:32:13.743 INFO:tasks.rados.rados.0.smithi203.stdout:637: writing smithi20328086-31 from 1763838 to 2268978 tid 2 2024-09-17T15:32:13.743 INFO:tasks.rados.rados.0.smithi203.stdout:638: copy_from oid 26 from oid 29 current snap is 68 2024-09-17T15:32:13.743 INFO:tasks.rados.rados.0.smithi203.stdout:636: expect (ObjNum 133 snap 29 seq_num 133) 2024-09-17T15:32:13.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1749464 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: finishing write tid 1 to smithi20328086-31 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: finishing write tid 2 to smithi20328086-31 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: finishing write tid 3 to smithi20328086-31 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: finishing write tid 4 to smithi20328086-31 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: oid 31 updating version 0 to 71 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: oid 31 updating version 71 to 74 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: oid 31 updating version 74 to 75 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 75 (ObjNum 254 snap 68 seq_num 254) dirty exists 2024-09-17T15:32:13.800 INFO:tasks.rados.rados.0.smithi203.stdout:637: left oid 31 (ObjNum 254 snap 68 seq_num 254) 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:638: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:638: finishing copy_from to smithi20328086-26 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 67 (ObjNum 188 snap 46 seq_num 188) dirty exists 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:636: done (2 left) 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:637: done (1 left) 2024-09-17T15:32:13.801 INFO:tasks.rados.rados.0.smithi203.stdout:638: done (0 left) 2024-09-17T15:32:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:639: copy_from oid 6 from oid 15 current snap is 68 2024-09-17T15:32:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:640: write (excl) oid 50 current snap is 68 2024-09-17T15:32:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:640: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-17T15:32:13.823 INFO:tasks.rados.rados.0.smithi203.stdout:640: writing smithi20328086-50 from 685785 to 1313929 tid 1 2024-09-17T15:32:13.839 INFO:tasks.rados.rados.0.smithi203.stdout:640: writing smithi20328086-50 from 1822320 to 2285785 tid 2 2024-09-17T15:32:13.839 INFO:tasks.rados.rados.0.smithi203.stdout:641: delete oid 23 current snap is 68 2024-09-17T15:32:13.839 INFO:tasks.rados.rados.0.smithi203.stdout:639: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:32:13.839 INFO:tasks.rados.rados.0.smithi203.stdout:639: finishing copy_from to smithi20328086-6 2024-09-17T15:32:13.840 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 63 (ObjNum 193 snap 48 seq_num 193) dirty exists 2024-09-17T15:32:13.840 INFO:tasks.rados.rados.0.smithi203.stdout:640: finishing write tid 1 to smithi20328086-50 2024-09-17T15:32:13.842 INFO:tasks.rados.rados.0.smithi203.stdout:639: done (2 left) 2024-09-17T15:32:13.842 INFO:tasks.rados.rados.0.smithi203.stdout:641: done (1 left) 2024-09-17T15:32:13.842 INFO:tasks.rados.rados.0.smithi203.stdout:642: write (excl) oid 21 current snap is 68 2024-09-17T15:32:13.842 INFO:tasks.rados.rados.0.smithi203.stdout:642: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-17T15:32:13.864 INFO:tasks.rados.rados.0.smithi203.stdout:642: writing smithi20328086-21 from 719399 to 1498041 tid 1 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:642: writing smithi20328086-21 from 1939284 to 2319399 tid 2 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:643: write (excl) oid 37 current snap is 68 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: finishing write tid 2 to smithi20328086-50 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: finishing write tid 3 to smithi20328086-50 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: finishing write tid 4 to smithi20328086-50 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: oid 50 updating version 0 to 85 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: oid 50 updating version 85 to 86 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: oid 50 version 86 is already newer than 84 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 86 (ObjNum 255 snap 68 seq_num 255) dirty exists 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:640: left oid 50 (ObjNum 255 snap 68 seq_num 255) 2024-09-17T15:32:13.876 INFO:tasks.rados.rados.0.smithi203.stdout:643: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-17T15:32:13.888 INFO:tasks.rados.rados.0.smithi203.stdout:643: writing smithi20328086-37 from 736206 to 1190097 tid 1 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:643: writing smithi20328086-37 from 1881413 to 2336206 tid 2 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:640: done (2 left) 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:644: write oid 32 current snap is 68 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:642: finishing write tid 1 to smithi20328086-21 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:642: finishing write tid 2 to smithi20328086-21 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:642: finishing write tid 3 to smithi20328086-21 2024-09-17T15:32:13.902 INFO:tasks.rados.rados.0.smithi203.stdout:644: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-17T15:32:13.916 INFO:tasks.rados.rados.0.smithi203.stdout:644: writing smithi20328086-32 from 753013 to 1398506 tid 1 2024-09-17T15:32:13.925 INFO:tasks.rados.rados.0.smithi203.stdout:644: writing smithi20328086-32 from 2056248 to 2353013 tid 2 2024-09-17T15:32:13.925 INFO:tasks.rados.rados.0.smithi203.stdout:645: write oid 23 current snap is 68 2024-09-17T15:32:13.925 INFO:tasks.rados.rados.0.smithi203.stdout:645: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-17T15:32:13.937 INFO:tasks.rados.rados.0.smithi203.stdout:645: writing smithi20328086-23 from 769820 to 1490562 tid 1 2024-09-17T15:32:13.944 INFO:tasks.rados.rados.0.smithi203.stdout:645: writing smithi20328086-23 from 2114730 to 2369820 tid 2 2024-09-17T15:32:13.944 INFO:tasks.rados.rados.0.smithi203.stdout:646: read oid 17 snap 59 2024-09-17T15:32:13.944 INFO:tasks.rados.rados.0.smithi203.stdout:646: expect (ObjNum 200 snap 51 seq_num 200) 2024-09-17T15:32:13.944 INFO:tasks.rados.rados.0.smithi203.stdout:647: delete oid 1 current snap is 68 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:642: finishing write tid 4 to smithi20328086-21 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:642: oid 21 updating version 0 to 261 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:642: oid 21 version 261 is already newer than 259 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:642: oid 21 version 261 is already newer than 260 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 261 (ObjNum 257 snap 68 seq_num 257) dirty exists 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:642: left oid 21 (ObjNum 257 snap 68 seq_num 257) 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: finishing write tid 1 to smithi20328086-37 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: finishing write tid 2 to smithi20328086-37 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: finishing write tid 3 to smithi20328086-37 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: finishing write tid 4 to smithi20328086-37 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: oid 37 updating version 0 to 150 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: oid 37 updating version 150 to 152 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:643: oid 37 version 152 is already newer than 151 2024-09-17T15:32:13.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 152 (ObjNum 258 snap 68 seq_num 258) dirty exists 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:643: left oid 37 (ObjNum 258 snap 68 seq_num 258) 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: finishing write tid 1 to smithi20328086-32 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: finishing write tid 2 to smithi20328086-32 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: finishing write tid 3 to smithi20328086-32 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: finishing write tid 4 to smithi20328086-32 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: oid 32 updating version 0 to 89 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: oid 32 version 89 is already newer than 87 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: oid 32 version 89 is already newer than 88 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 89 (ObjNum 259 snap 68 seq_num 259) dirty exists 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:644: left oid 32 (ObjNum 259 snap 68 seq_num 259) 2024-09-17T15:32:13.946 INFO:tasks.rados.rados.0.smithi203.stdout:645: finishing write tid 1 to smithi20328086-23 2024-09-17T15:32:13.950 INFO:tasks.rados.rados.0.smithi203.stdout:645: finishing write tid 2 to smithi20328086-23 2024-09-17T15:32:13.950 INFO:tasks.rados.rados.0.smithi203.stdout:645: finishing write tid 3 to smithi20328086-23 2024-09-17T15:32:13.950 INFO:tasks.rados.rados.0.smithi203.stdout:645: finishing write tid 4 to smithi20328086-23 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:645: oid 23 updating version 0 to 151 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:645: oid 23 version 151 is already newer than 150 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:645: oid 23 version 151 is already newer than 149 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 151 (ObjNum 260 snap 68 seq_num 260) dirty exists 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:645: left oid 23 (ObjNum 260 snap 68 seq_num 260) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:642: done (5 left) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:643: done (4 left) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:644: done (3 left) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:645: done (2 left) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:647: done (1 left) 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:648: rollback oid 13 current snap is 68 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 63 2024-09-17T15:32:13.951 INFO:tasks.rados.rados.0.smithi203.stdout:649: snap_create 2024-09-17T15:32:13.963 INFO:tasks.rados.rados.0.smithi203.stdout:648: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:32:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:648: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:32:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:648: finishing rollback tid 2 to smithi20328086-13 2024-09-17T15:32:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 127 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:32:13.972 INFO:tasks.rados.rados.0.smithi203.stdout:646: expect (ObjNum 200 snap 51 seq_num 200) 2024-09-17T15:32:13.972 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:14.076 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:14.146 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1087966 2024-09-17T15:32:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:646: done (2 left) 2024-09-17T15:32:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:648: done (1 left) 2024-09-17T15:32:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:649: done (0 left) 2024-09-17T15:32:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:650: copy_from oid 31 from oid 30 current snap is 69 2024-09-17T15:32:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:651: write oid 48 current snap is 69 2024-09-17T15:32:14.769 INFO:tasks.rados.rados.0.smithi203.stdout:651: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-17T15:32:14.775 INFO:tasks.rados.rados.0.smithi203.stdout:651: writing smithi20328086-48 from 403434 to 874674 tid 1 2024-09-17T15:32:14.782 INFO:tasks.rados.rados.0.smithi203.stdout:651: writing smithi20328086-48 from 1315341 to 1716097 tid 2 2024-09-17T15:32:14.782 INFO:tasks.rados.rados.0.smithi203.stdout:652: copy_from oid 24 from oid 11 current snap is 69 2024-09-17T15:32:14.782 INFO:tasks.rados.rados.0.smithi203.stdout:653: delete oid 46 current snap is 69 2024-09-17T15:32:14.790 INFO:tasks.rados.rados.0.smithi203.stdout:651: finishing write tid 1 to smithi20328086-48 2024-09-17T15:32:14.791 INFO:tasks.rados.rados.0.smithi203.stdout:651: finishing write tid 2 to smithi20328086-48 2024-09-17T15:32:14.792 INFO:tasks.rados.rados.0.smithi203.stdout:651: finishing write tid 3 to smithi20328086-48 2024-09-17T15:32:14.792 INFO:tasks.rados.rados.0.smithi203.stdout:651: finishing write tid 4 to smithi20328086-48 2024-09-17T15:32:14.792 INFO:tasks.rados.rados.0.smithi203.stdout:651: oid 48 updating version 0 to 154 2024-09-17T15:32:14.792 INFO:tasks.rados.rados.0.smithi203.stdout:651: oid 48 updating version 154 to 156 2024-09-17T15:32:14.792 INFO:tasks.rados.rados.0.smithi203.stdout:651: oid 48 version 156 is already newer than 155 2024-09-17T15:32:14.793 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 156 (ObjNum 262 snap 69 seq_num 262) dirty exists 2024-09-17T15:32:14.793 INFO:tasks.rados.rados.0.smithi203.stdout:651: left oid 48 (ObjNum 262 snap 69 seq_num 262) 2024-09-17T15:32:14.797 INFO:tasks.rados.rados.0.smithi203.stdout:651: done (3 left) 2024-09-17T15:32:14.797 INFO:tasks.rados.rados.0.smithi203.stdout:653: done (2 left) 2024-09-17T15:32:14.797 INFO:tasks.rados.rados.0.smithi203.stdout:654: delete oid 25 current snap is 69 2024-09-17T15:32:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:650: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:32:14.810 INFO:tasks.rados.rados.0.smithi203.stdout:650: finishing copy_from to smithi20328086-31 2024-09-17T15:32:14.811 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 84 (ObjNum 158 snap 36 seq_num 158) dirty exists 2024-09-17T15:32:14.816 INFO:tasks.rados.rados.0.smithi203.stdout:652: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:32:14.816 INFO:tasks.rados.rados.0.smithi203.stdout:650: done (2 left) 2024-09-17T15:32:14.816 INFO:tasks.rados.rados.0.smithi203.stdout:654: done (1 left) 2024-09-17T15:32:14.817 INFO:tasks.rados.rados.0.smithi203.stdout:655: snap_create 2024-09-17T15:32:14.819 INFO:tasks.rados.rados.0.smithi203.stdout:652: finishing copy_from to smithi20328086-24 2024-09-17T15:32:14.819 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 169 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:32:15.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.128+0000 7f42cbfff640 10 monclient: tick 2024-09-17T15:32:15.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.128+0000 7f42cbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:15.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.128+0000 7f42cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:15.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.128+0000 7f42cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e413d310 auth_method 0 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:15.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_done global_id 54100 payload 824 2024-09-17T15:32:15.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.148+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42e413b3b0 con 0x7f42e413d310 2024-09-17T15:32:15.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:15.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42e0018990 tx=0x7f42e0018e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e000f040 con 0x7f42e413d310 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 10 monclient: dump: 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.156 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42e0022400 con 0x7f42e413d310 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42e0018990 tx=0x7f42e0018e50 comp rx=0 tx=0).stop 2024-09-17T15:32:15.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 msgr2=0x7f42e412e9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:15.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.152+0000 7f42d97fa640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e413d310 0x7f42e412e9e0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f42e0018990 tx=0x7f42e0018e50 comp rx=0 tx=0).stop 2024-09-17T15:32:15.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42eb260640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:15.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42eb260640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 0x7f42e412f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42eb260640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f42e412f870 con 0x7f42e412ef20 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 0x7f42e412f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient: get_auth_request con 0x7f42e412ef20 auth_method 0 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 10 monclient(hunting): handle_auth_done global_id 54103 payload 824 2024-09-17T15:32:15.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.160+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42e406d4f0 con 0x7f42e412ef20 2024-09-17T15:32:15.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42e8ffc640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 msgr2=0x7f42e412f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:15.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42e8ffc640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 0x7f42e412f330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f42e0022000 tx=0x7f42e00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e0027040 con 0x7f42e412ef20 2024-09-17T15:32:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:15.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 10 monclient: dump: 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f42e00225b0 con 0x7f42e412ef20 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 msgr2=0x7f42e412f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 0x7f42e412f330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f42e0022000 tx=0x7f42e00046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 msgr2=0x7f42e412f330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:15.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42d97fa640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f42e412ef20 0x7f42e412f330 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f42e0022000 tx=0x7f42e00046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10599, 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:15.171 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:15.172 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:15.173 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:15.174 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:15.175 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:15.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:15.177 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:15.178 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.179 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:15.180 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.182 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:15.183 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:15.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f42d003bb60 msgr2=0x7f42d003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f42d003bb60 0x7f42d003e020 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f42e406f840 tx=0x7f42d4008040 comp rx=0 tx=0).stop 2024-09-17T15:32:15.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 10 monclient: shutdown 2024-09-17T15:32:15.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 msgr2=0x7f42e41d1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 --2- 172.21.15.86:0/1323032083 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42e4135f90 0x7f42e41d1680 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00c900 tx=0x7f42dc00cdc0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 reap_dead start 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42e97fd640 1 -- 172.21.15.86:0/1323032083 reap_dead start 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 shutdown_connections 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 >> 172.21.15.86:0/1323032083 conn(0x7f42e40a2f40 msgr2=0x7f42e4072a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 shutdown_connections 2024-09-17T15:32:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.164+0000 7f42eb260640 1 -- 172.21.15.86:0/1323032083 wait complete. 2024-09-17T15:32:15.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:15.194 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:32:15.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:32:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 10 monclient: get_monmap_and_config 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 10 monclient: build_initial_monmap 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 10 monclient: monmap: 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:15.322649+0000 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:15.322649+0000 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:15.323 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 1 Processor -- start 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 1 -- start start 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.316+0000 7fdfbe96b640 10 monclient: init 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdfb8138280 0x7fdfb8138660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-c con 0x7fdfb8138280 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfb8138ba0 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfb806fb00 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb806dd30 con 0x7fdfb8138ba0 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb806e190 con 0x7fdfb806fb00 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb80a48e0 con 0x7fdfb8138280 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 10 monclient(hunting): authenticate will time out at 1268.288547s 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdfb8138ba0 0x7fdfb80a2e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43426/0 (socket says 172.21.15.86:43426) 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 1 -- 172.21.15.86:0/4265256627 learned_addr learned my addr 172.21.15.86:0/4265256627 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/4265256627 >> v2:172.21.15.203:3300/0 conn(0x7fdfb8138280 0x7fdfb8138660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request con 0x7fdfb8138280 auth_method 0 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 1 -- 172.21.15.86:0/4265256627 >> v1:172.21.15.86:6789/0 conn(0x7fdfb8138ba0 legacy=0x7fdfb80a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573067393 0 0) 0x7fdfa0001460 con 0x7fdfb8138ba0 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:15.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): my global_id is 4767 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa4003530 con 0x7fdfb8138ba0 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 173562526 0 0) 0x7fdfa4003530 con 0x7fdfb8138ba0 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 >> v2:172.21.15.203:3300/0 conn(0x7fdfb8138280 msgr2=0x7fdfb8138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 --2- 172.21.15.86:0/4265256627 >> v2:172.21.15.203:3300/0 conn(0x7fdfb8138280 0x7fdfb8138660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 >> v1:172.21.15.203:6789/0 conn(0x7fdfb806fb00 legacy=0x7fdfb80a3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb806df70 con 0x7fdfb8138ba0 2024-09-17T15:32:15.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 5 monclient: authenticate success, global_id 4767 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fdfa0001b80 con 0x7fdfb8138ba0 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: dump: 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.332 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 >> v1:172.21.15.86:6789/0 conn(0x7fdfb8138ba0 legacy=0x7fdfb80a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 0x7fdfa4003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): picked mon.a con 0x7fdfa4003ad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4004810 0x7fdfa4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): picked mon.c con 0x7fdfa4004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa40052f0 0x7fdfa400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): picked mon.b con 0x7fdfa40052f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80a48e0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa400c1f0 con 0x7fdfa40052f0 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa400c3f0 con 0x7fdfa4004810 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbe96b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 0x7fdfa4003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request con 0x7fdfa4003ad0 auth_method 0 2024-09-17T15:32:15.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4004810 0x7fdfa4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): get_auth_request con 0x7fdfa4004810 auth_method 0 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_done global_id 4767 payload 293 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 -- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4004810 msgr2=0x7fdfa4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4004810 0x7fdfa4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 -- 172.21.15.86:0/4265256627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa40052f0 msgr2=0x7fdfa400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa40052f0 0x7fdfa400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 monclient: found mon.a 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfbcf08640 1 -- 172.21.15.86:0/4265256627 reap_dead start 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 1 -- 172.21.15.86:0/4265256627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa4003530 con 0x7fdfa4003ad0 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient: _finish_auth 0 2024-09-17T15:32:15.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:15.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.320+0000 7fdfb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:15.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb7fff640 1 -- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 msgr2=0x7fdfa4003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 0x7fdfa4003ed0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009fc0 tx=0x7fdfb001cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c2a1488ece84a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00b2070 con 0x7fdfa4003ad0 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: dump: 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.342 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb00a5b70 con 0x7fdfa4003ad0 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 1 -- 172.21.15.86:0/4265256627 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0009c10 con 0x7fdfa4003ad0 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfb6ffd640 10 monclient: dump: 2024-09-17T15:32:15.343 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 4 set_mon_vals no callback set 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 10 monclient: shutdown 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:15.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 msgr2=0x7fdfa4003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 0x7fdfa4003ed0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009fc0 tx=0x7fdfb001cc70 comp rx=0 tx=0).stop 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- 172.21.15.86:0/4265256627 shutdown_connections 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/4265256627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfa4003ad0 0x7fdfa4003ed0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- 172.21.15.86:0/4265256627 >> 172.21.15.86:0/4265256627 conn(0x7fdfb80a8470 msgr2=0x7fdfb80a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- 172.21.15.86:0/4265256627 shutdown_connections 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- 172.21.15.86:0/4265256627 wait complete. 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 10 monclient: build_initial_monmap 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 10 monclient: monmap: 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:15.345 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 Processor -- start 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 1 -- start start 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.328+0000 7fdfbe96b640 10 monclient: init 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-c con 0x7fdfa4003ad0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:15.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfb8137320 0x7fdfb81c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-b con 0x7fdfb8137320 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb81c33b0 0x7fdfb81c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): picked mon.noname-a con 0x7fdfb81c33b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80a3a10 con 0x7fdfb81c33b0 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80a3890 con 0x7fdfb8137320 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb80a3b90 con 0x7fdfa4003ad0 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient(hunting): authenticate will time out at 1268.300412s 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59724/0 (socket says 172.21.15.86:59724) 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 -- 172.21.15.86:0/1764311995 learned_addr learned my addr 172.21.15.86:0/1764311995 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request con 0x7fdfa4003ad0 auth_method 0 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb81c33b0 0x7fdfb81c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): get_auth_request con 0x7fdfb81c33b0 auth_method 0 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbcf08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient(hunting): handle_auth_done global_id 44213 payload 824 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb77fe640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfb8137320 0x7fdfb81c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 -- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfb8137320 msgr2=0x7fdfb81c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfb8137320 0x7fdfb81c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 -- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb81c33b0 msgr2=0x7fdfb81c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb81c33b0 0x7fdfb81c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 monclient: found mon.noname-c 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:32:15.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 -- 172.21.15.86:0/1764311995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb81378e0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 -- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 msgr2=0x7fdfb8136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb7fff640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfb001c6e0 tx=0x7fdfb0009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00bc020 con 0x7fdfa4003ad0 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: dump: 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.350 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb00ad940 con 0x7fdfa4003ad0 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 5 monclient: authenticate success, global_id 44213 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _renew_subs 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb81249a0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:15.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00b2040 con 0x7fdfa4003ad0 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb4ff9640 10 monclient: dump: 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:15.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfb67fc640 4 set_mon_vals no callback set 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _renew_subs 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.332+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb809f660 con 0x7fdfa4003ad0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfbe96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfbe96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfbe96b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdfb00bf020 con 0x7fdfa4003ad0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb80a3ec0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb4ff9640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 0x7fdf9803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(155..155 src has 1..155) ==== 4771+0+0 (secure 0 0 0) 0x7fdfb00eacb0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb77fe640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 0x7fdf9803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:15.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb77fe640 10 monclient: get_auth_request con 0x7fdf9803bb60 auth_method 0 2024-09-17T15:32:15.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.336+0000 7fdfb77fe640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 0x7fdf9803e020 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8137f90 tx=0x7fdfa0006040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:15.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.340+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdfb00c6070 con 0x7fdfa4003ad0 2024-09-17T15:32:15.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.340+0000 7fdfb4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:15.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.340+0000 7fdfb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:15.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfbe96b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:32:15.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfbe96b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:32:15.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfbe96b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:32:15.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fdfb80a45d0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=1730599210,config=3541143443,health=2204782128,kv=3591778543,logm=873099556}) 2024-09-17T15:32:15.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=1730599210,config=3541143443,health=3887269300,kv=215610202,logm=675780429}) 2024-09-17T15:32:15.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=1730599210,config=3541143443,health=2204782128,kv=3591778543,logm=873099556}) 2024-09-17T15:32:15.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.580+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2676758194,config=3541143443,health=3887269301,kv=3591778543,logm=1487065441}) 2024-09-17T15:32:15.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfb4ff9640 1 -- 172.21.15.86:0/1764311995 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fdfb00169b0 con 0x7fdfa4003ad0 2024-09-17T15:32:15.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfb4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:32:15.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.580+0000 7fdfb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 msgr2=0x7fdf9803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 0x7fdf9803e020 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8137f90 tx=0x7fdfa0006040 comp rx=0 tx=0).stop 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 10 monclient: shutdown 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 msgr2=0x7fdfb8136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fdfb001c6e0 tx=0x7fdfb0009750 comp rx=0 tx=0).stop 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 shutdown_connections 2024-09-17T15:32:15.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdf9803bb60 0x7fdf9803e020 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8137f90 tx=0x7fdfa0006040 comp rx=0 tx=0).stop 2024-09-17T15:32:15.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdfb81c33b0 0x7fdfb81c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfb8137320 0x7fdfb81c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 --2- 172.21.15.86:0/1764311995 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdfa4003ad0 0x7fdfb8136de0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:15.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 >> 172.21.15.86:0/1764311995 conn(0x7fdfb80a8470 msgr2=0x7fdfb80a0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:15.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 shutdown_connections 2024-09-17T15:32:15.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1564079468}) 2024-09-17T15:32:15.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4171862073}) 2024-09-17T15:32:15.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:15.584+0000 7fdfbe96b640 1 -- 172.21.15.86:0/1764311995 wait complete. 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1564079468}) 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.584+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=157432908}) 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485003539}) 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445191596}) 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485003539}) 2024-09-17T15:32:15.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755409809}) 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123450654}) 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859197250}) 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123450654}) 2024-09-17T15:32:15.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.588+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966816795}) 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620032929}) 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397900193}) 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620032929}) 2024-09-17T15:32:15.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1122257984}) 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239172606}) 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432103481}) 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=239172606}) 2024-09-17T15:32:15.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662228670}) 2024-09-17T15:32:15.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018547055}) 2024-09-17T15:32:15.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413716643}) 2024-09-17T15:32:15.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018547055}) 2024-09-17T15:32:15.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.592+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450111982}) 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136944158}) 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=883281181}) 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136944158}) 2024-09-17T15:32:15.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477044329}) 2024-09-17T15:32:15.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335267787}) 2024-09-17T15:32:15.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478133359}) 2024-09-17T15:32:15.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3335267787}) 2024-09-17T15:32:15.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.596+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497636449}) 2024-09-17T15:32:15.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379329789}) 2024-09-17T15:32:15.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735449105}) 2024-09-17T15:32:15.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379329789}) 2024-09-17T15:32:15.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270775140}) 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967171447}) 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=357153451}) 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967171447}) 2024-09-17T15:32:15.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.600+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582761775}) 2024-09-17T15:32:15.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:32:15.607 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=980961727,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=406423704,mon_config_key=3033277032}) 2024-09-17T15:32:15.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9} crc {logm=2546377647,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3520996381,mon_config_key=1807879747}) 2024-09-17T15:32:15.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=980961727,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=406423704,mon_config_key=3033277032}) 2024-09-17T15:32:15.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11} crc {logm=3945113450,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4165379462,mon_config_key=2520878402}) 2024-09-17T15:32:15.609 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.609 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=52} crc {mon_config_key=67567533,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4098176492,osd_snap=1799241852,osdmap=466826541}) 2024-09-17T15:32:15.609 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=27,osdmap=56} crc {mon_config_key=67567533,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3274135206,osd_pg_creating=4098176492,osd_snap=3786981048,osdmap=3821944252}) 2024-09-17T15:32:15.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=52} crc {mon_config_key=67567533,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4098176492,osd_snap=1799241852,osdmap=466826541}) 2024-09-17T15:32:15.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.604+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=55} crc {mon_config_key=46345141,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4098176492,osd_snap=292863401,osdmap=1318517985}) 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3924437887}) 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3864912429}) 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3924437887}) 2024-09-17T15:32:15.612 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.608+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=990297653}) 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=92} crc {osdmap=108162158}) 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=93} crc {osdmap=2613559871}) 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=92} crc {osdmap=108162158}) 2024-09-17T15:32:15.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:32:15.616+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=91} crc {osdmap=2634507732}) 2024-09-17T15:32:15.783 INFO:tasks.rados.rados.0.smithi203.stdout:652: done (1 left) 2024-09-17T15:32:15.783 INFO:tasks.rados.rados.0.smithi203.stdout:655: done (0 left) 2024-09-17T15:32:15.783 INFO:tasks.rados.rados.0.smithi203.stdout:656: write oid 25 current snap is 70 2024-09-17T15:32:15.783 INFO:tasks.rados.rados.0.smithi203.stdout:656: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-17T15:32:15.796 INFO:tasks.rados.rados.0.smithi203.stdout:656: writing smithi20328086-25 from 453855 to 1150842 tid 1 2024-09-17T15:32:15.811 INFO:tasks.rados.rados.0.smithi203.stdout:656: writing smithi20328086-25 from 1890787 to 2438870 tid 2 2024-09-17T15:32:15.811 INFO:tasks.rados.rados.0.smithi203.stdout:657: write oid 47 current snap is 70 2024-09-17T15:32:15.811 INFO:tasks.rados.rados.0.smithi203.stdout:656: finishing write tid 1 to smithi20328086-25 2024-09-17T15:32:15.812 INFO:tasks.rados.rados.0.smithi203.stdout:657: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-17T15:32:15.825 INFO:tasks.rados.rados.0.smithi203.stdout:657: writing smithi20328086-47 from 470662 to 1242898 tid 1 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:657: writing smithi20328086-47 from 1832916 to 2241010 tid 2 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:658: delete oid 10 current snap is 70 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: finishing write tid 2 to smithi20328086-25 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: finishing write tid 3 to smithi20328086-25 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: finishing write tid 4 to smithi20328086-25 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: oid 25 updating version 0 to 172 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: oid 25 version 172 is already newer than 171 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: oid 25 version 172 is already newer than 170 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 172 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-09-17T15:32:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:656: left oid 25 (ObjNum 265 snap 70 seq_num 265) 2024-09-17T15:32:15.844 INFO:tasks.rados.rados.0.smithi203.stdout:657: finishing write tid 1 to smithi20328086-47 2024-09-17T15:32:15.844 INFO:tasks.rados.rados.0.smithi203.stdout:657: finishing write tid 2 to smithi20328086-47 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:656: done (2 left) 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:658: done (1 left) 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:659: rollback oid 42 current snap is 70 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 62 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:660: write oid 46 current snap is 70 2024-09-17T15:32:15.845 INFO:tasks.rados.rados.0.smithi203.stdout:660: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-17T15:32:15.857 INFO:tasks.rados.rados.0.smithi203.stdout:660: writing smithi20328086-46 from 504276 to 1143363 tid 1 2024-09-17T15:32:15.870 INFO:tasks.rados.rados.0.smithi203.stdout:660: writing smithi20328086-46 from 1666233 to 2361643 tid 2 2024-09-17T15:32:15.871 INFO:tasks.rados.rados.0.smithi203.stdout:661: write oid 14 current snap is 70 2024-09-17T15:32:15.871 INFO:tasks.rados.rados.0.smithi203.stdout:661: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-17T15:32:15.882 INFO:tasks.rados.rados.0.smithi203.stdout:661: writing smithi20328086-14 from 521083 to 1235419 tid 1 2024-09-17T15:32:15.891 INFO:tasks.rados.rados.0.smithi203.stdout:661: writing smithi20328086-14 from 1724715 to 2280136 tid 2 2024-09-17T15:32:15.891 INFO:tasks.rados.rados.0.smithi203.stdout:662: write (excl) oid 34 current snap is 70 2024-09-17T15:32:15.891 INFO:tasks.rados.rados.0.smithi203.stdout:662: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-17T15:32:15.901 INFO:tasks.rados.rados.0.smithi203.stdout:662: writing smithi20328086-34 from 537890 to 1327475 tid 1 2024-09-17T15:32:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:662: writing smithi20328086-34 from 1783197 to 2482276 tid 2 2024-09-17T15:32:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:663: write oid 37 current snap is 70 2024-09-17T15:32:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:663: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-17T15:32:15.915 INFO:tasks.rados.rados.0.smithi203.stdout:663: writing smithi20328086-37 from 554697 to 1019531 tid 1 2024-09-17T15:32:15.923 INFO:tasks.rados.rados.0.smithi203.stdout:663: writing smithi20328086-37 from 1725326 to 2284416 tid 2 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:664: read oid 7 snap -1 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:664: expect (ObjNum 250 snap 65 seq_num 250) 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:665: delete oid 26 current snap is 70 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: finishing write tid 3 to smithi20328086-47 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: finishing write tid 4 to smithi20328086-47 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: oid 47 updating version 0 to 138 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: oid 47 updating version 138 to 139 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: oid 47 version 139 is already newer than 137 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 139 (ObjNum 266 snap 70 seq_num 266) dirty exists 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:657: left oid 47 (ObjNum 266 snap 70 seq_num 266) 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:659: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:659: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:32:15.924 INFO:tasks.rados.rados.0.smithi203.stdout:659: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 156 (ObjNum 126 snap 26 seq_num 126) dirty exists 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: finishing write tid 1 to smithi20328086-46 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: finishing write tid 2 to smithi20328086-46 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: finishing write tid 3 to smithi20328086-46 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: finishing write tid 4 to smithi20328086-46 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: oid 46 updating version 0 to 158 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: oid 46 updating version 158 to 159 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: oid 46 version 159 is already newer than 157 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 159 (ObjNum 268 snap 70 seq_num 268) dirty exists 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:660: left oid 46 (ObjNum 268 snap 70 seq_num 268) 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:661: finishing write tid 1 to smithi20328086-14 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:661: finishing write tid 2 to smithi20328086-14 2024-09-17T15:32:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:661: finishing write tid 3 to smithi20328086-14 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:661: finishing write tid 4 to smithi20328086-14 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:661: oid 14 updating version 0 to 155 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:661: oid 14 updating version 155 to 156 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:661: oid 14 version 156 is already newer than 154 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 156 (ObjNum 269 snap 70 seq_num 269) dirty exists 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:661: left oid 14 (ObjNum 269 snap 70 seq_num 269) 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: finishing write tid 1 to smithi20328086-34 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: finishing write tid 2 to smithi20328086-34 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: finishing write tid 3 to smithi20328086-34 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: finishing write tid 4 to smithi20328086-34 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: oid 34 updating version 0 to 90 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: oid 34 updating version 90 to 91 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:662: oid 34 updating version 91 to 92 2024-09-17T15:32:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 92 (ObjNum 270 snap 70 seq_num 270) dirty exists 2024-09-17T15:32:15.927 INFO:tasks.rados.rados.0.smithi203.stdout:662: left oid 34 (ObjNum 270 snap 70 seq_num 270) 2024-09-17T15:32:15.931 INFO:tasks.rados.rados.0.smithi203.stdout:663: finishing write tid 1 to smithi20328086-37 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:663: finishing write tid 2 to smithi20328086-37 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:657: done (7 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:659: done (6 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:660: done (5 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:661: done (4 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:662: done (3 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:665: done (2 left) 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:666: copy_from oid 36 from oid 40 current snap is 70 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:667: read oid 6 snap -1 2024-09-17T15:32:15.932 INFO:tasks.rados.rados.0.smithi203.stdout:667: expect (ObjNum 193 snap 48 seq_num 193) 2024-09-17T15:32:15.933 INFO:tasks.rados.rados.0.smithi203.stdout:668: read oid 9 snap -1 2024-09-17T15:32:15.933 INFO:tasks.rados.rados.0.smithi203.stdout:668: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:32:15.933 INFO:tasks.rados.rados.0.smithi203.stdout:669: snap_remove snap 69 2024-09-17T15:32:16.608 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:32:16.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: get_monmap_and_config 2024-09-17T15:32:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: build_initial_monmap 2024-09-17T15:32:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: monmap: 2024-09-17T15:32:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:16.734236+0000 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:16.734236+0000 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:16.734 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 Processor -- start 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 -- start start 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: init 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7eff8c0689d0 0x7eff8c09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-a con 0x7eff8c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff8c068db0 0x7eff8c0691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-c con 0x7eff8c068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-b con 0x7eff8c134a40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7eff8c09d770 con 0x7eff8c0689d0 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7eff8c09ed00 con 0x7eff8c068db0 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff8c09efe0 con 0x7eff8c134a40 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff93eda640 10 monclient(hunting): authenticate will time out at 1269.699611s 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c134ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7eff8c0689d0 0x7eff8c09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c134ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59728/0 (socket says 172.21.15.86:59728) 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 1 -- 172.21.15.86:0/167754559 learned_addr learned my addr 172.21.15.86:0/167754559 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): get_auth_request con 0x7eff8c134a40 auth_method 0 2024-09-17T15:32:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): get_auth_request con 0x7eff8c0689d0 auth_method 0 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.728+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91475640 1 --2- 172.21.15.86:0/167754559 >> v2:172.21.15.203:3300/0 conn(0x7eff8c068db0 0x7eff8c0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91475640 10 monclient(hunting): get_auth_request con 0x7eff8c068db0 auth_method 0 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91475640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91475640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91475640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 10 monclient(hunting): handle_auth_done global_id 4773 payload 274 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 -- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 msgr2=0x7eff8c134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 --2- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c134ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 -- 172.21.15.86:0/167754559 >> v2:172.21.15.203:3300/0 conn(0x7eff8c068db0 msgr2=0x7eff8c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 --2- 172.21.15.86:0/167754559 >> v2:172.21.15.203:3300/0 conn(0x7eff8c068db0 0x7eff8c0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 10 monclient: _finish_hunting 0 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 monclient: found mon.noname-a 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:16.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.732+0000 7eff91c76640 1 -- 172.21.15.86:0/167754559 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c0a04e0 con 0x7eff8c0689d0 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 -- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 msgr2=0x7eff8c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 --2- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c09ffa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7eff8c06ed90 tx=0x7eff8801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dc1183b0950829 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 1 -- 172.21.15.86:0/167754559 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff880b2070 con 0x7eff8c0689d0 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: dump: 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: _finish_auth 0 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: _check_auth_tickets 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 1 -- 172.21.15.86:0/167754559 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff880ad640 con 0x7eff8c0689d0 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 5 monclient: authenticate success, global_id 4773 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 1 -- 172.21.15.86:0/167754559 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff880ada10 con 0x7eff8c0689d0 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff90c74640 10 monclient: dump: 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:16.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 4 set_mon_vals no callback set 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: shutdown 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 msgr2=0x7eff8c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c09ffa0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7eff8c06ed90 tx=0x7eff8801c3d0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- 172.21.15.86:0/167754559 shutdown_connections 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c134ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- 172.21.15.86:0/167754559 >> v2:172.21.15.203:3300/0 conn(0x7eff8c068db0 0x7eff8c0691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- 172.21.15.86:0/167754559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c09ffa0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- 172.21.15.86:0/167754559 >> 172.21.15.86:0/167754559 conn(0x7eff8c0a6440 msgr2=0x7eff8c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- 172.21.15.86:0/167754559 shutdown_connections 2024-09-17T15:32:16.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- 172.21.15.86:0/167754559 wait complete. 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: build_initial_monmap 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: monmap: 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 Processor -- start 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- start start 2024-09-17T15:32:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: init 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-a con 0x7eff8c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff8c068db0 0x7eff8c11d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-b con 0x7eff8c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): picked mon.noname-c con 0x7eff8c134a40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c13e8b0 con 0x7eff8c0689d0 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff8c13e730 con 0x7eff8c068db0 2024-09-17T15:32:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff8c13ea30 con 0x7eff8c134a40 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff93eda640 10 monclient(hunting): authenticate will time out at 1269.706931s 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39102/0 (socket says 172.21.15.86:39102) 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 -- 172.21.15.86:0/1771360705 learned_addr learned my addr 172.21.15.86:0/1771360705 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): get_auth_request con 0x7eff8c0689d0 auth_method 0 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): get_auth_request con 0x7eff8c134a40 auth_method 0 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff92477640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient(hunting): handle_auth_done global_id 4776 payload 824 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 -- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 msgr2=0x7eff8c11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c11d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91475640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff8c068db0 0x7eff8c11d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 -- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff8c068db0 msgr2=0x7eff8c11d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff8c068db0 0x7eff8c11d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient: _finish_hunting 0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 monclient: found mon.noname-a 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.736+0000 7eff91c76640 1 -- 172.21.15.86:0/1771360705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8c120c50 con 0x7eff8c0689d0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff91c76640 1 -- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 msgr2=0x7eff8c11ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff91c76640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7eff880b0040 tx=0x7eff88019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff880ba020 con 0x7eff8c0689d0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: dump: 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: _finish_auth 0 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: _check_auth_tickets 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 5 monclient: authenticate success, global_id 4776 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff880b6620 con 0x7eff8c0689d0 2024-09-17T15:32:16.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _renew_subs 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8c09d460 con 0x7eff8c0689d0 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff880b2040 con 0x7eff8c0689d0 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 10 monclient: dump: 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:16.755 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff83fff640 4 set_mon_vals no callback set 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _renew_subs 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8c11e8f0 con 0x7eff8c0689d0 2024-09-17T15:32:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8c0691c0 con 0x7eff8c0689d0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7eff880bd020 con 0x7eff8c0689d0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 0x7eff7003e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(156..156 src has 1..156) ==== 4771+0+0 (secure 0 0 0) 0x7eff880e9930 con 0x7eff8c0689d0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff91475640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 0x7eff7003e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.740+0000 7eff91475640 10 monclient: get_auth_request con 0x7eff7003bc10 auth_method 0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.744+0000 7eff91475640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 0x7eff7003e0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009790 tx=0x7eff7c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.744+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff880c4070 con 0x7eff8c0689d0 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.744+0000 7eff827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.744+0000 7eff827fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:16.780 INFO:tasks.rados.rados.0.smithi203.stdout:669: done (5 left) 2024-09-17T15:32:16.780 INFO:tasks.rados.rados.0.smithi203.stdout:670: delete oid 39 current snap is 70 2024-09-17T15:32:16.780 INFO:tasks.rados.rados.0.smithi203.stdout:664: expect (ObjNum 250 snap 65 seq_num 250) 2024-09-17T15:32:16.871 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:16.902 INFO:tasks.rados.rados.0.smithi203.stdout:663: finishing write tid 3 to smithi20328086-37 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:663: finishing write tid 4 to smithi20328086-37 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:663: oid 37 updating version 0 to 159 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:663: oid 37 version 159 is already newer than 158 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:663: oid 37 updating version 159 to 160 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 160 (ObjNum 271 snap 70 seq_num 271) dirty exists 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:663: left oid 37 (ObjNum 271 snap 70 seq_num 271) 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:666: finishing copy_from to smithi20328086-36 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:666: got expected ENOENT (src dne) 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:666: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:668: expect (ObjNum 135 snap 29 seq_num 135) 2024-09-17T15:32:16.903 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 61698 2024-09-17T15:32:16.955 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 743380 2024-09-17T15:32:16.970 INFO:tasks.rados.rados.0.smithi203.stdout:667: expect (ObjNum 193 snap 48 seq_num 193) 2024-09-17T15:32:16.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff93eda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:16.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff93eda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:16.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff93eda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:16.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff8c074530 con 0x7eff8c0689d0 2024-09-17T15:32:16.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff827fc640 1 -- 172.21.15.86:0/1771360705 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7eff88016050 con 0x7eff8c0689d0 2024-09-17T15:32:16.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:16.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.992+0000 7eff827fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:16.998 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:17.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 msgr2=0x7eff7003e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 0x7eff7003e0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009790 tx=0x7eff7c008040 comp rx=0 tx=0).stop 2024-09-17T15:32:17.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 10 monclient: shutdown 2024-09-17T15:32:17.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 msgr2=0x7eff8c11ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7eff880b0040 tx=0x7eff88019170 comp rx=0 tx=0).stop 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 shutdown_connections 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff7003bc10 0x7eff7003e0d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8c134a40 0x7eff8c11d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff8c068db0 0x7eff8c11d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 --2- 172.21.15.86:0/1771360705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff8c0689d0 0x7eff8c11ce30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 >> 172.21.15.86:0/1771360705 conn(0x7eff8c0a6440 msgr2=0x7eff8c071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 shutdown_connections 2024-09-17T15:32:17.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:16.996+0000 7eff93eda640 1 -- 172.21.15.86:0/1771360705 wait complete. 2024-09-17T15:32:17.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:32:17.017 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:32:17.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:32:17.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:32:17.023 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:17.085 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: get_monmap_and_config 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: monmap: 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:17.140146+0000 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:17.140146+0000 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:17.140 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 Processor -- start 2024-09-17T15:32:17.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 -- start start 2024-09-17T15:32:17.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: init 2024-09-17T15:32:17.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6b40a50e0 0x7fc6b409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6b40a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6b40a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6b413d2f0 0x7fc6b413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-a con 0x7fc6b413d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b40a6280 con 0x7fc6b40a54c0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b409d2a0 con 0x7fc6b413d2f0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b409c0a0 con 0x7fc6b40a50e0 2024-09-17T15:32:17.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b93f8640 10 monclient(hunting): authenticate will time out at 1270.105619s 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6b413d2f0 0x7fc6b413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6b413d2f0 0x7fc6b413f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39114/0 (socket says 172.21.15.86:39114) 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 -- 172.21.15.86:0/3970937383 learned_addr learned my addr 172.21.15.86:0/3970937383 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): get_auth_request con 0x7fc6b413d2f0 auth_method 0 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 1 --2- 172.21.15.86:0/3970937383 >> v2:172.21.15.203:3300/0 conn(0x7fc6b40a50e0 0x7fc6b409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request con 0x7fc6b40a50e0 auth_method 0 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_done global_id 4779 payload 274 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 -- 172.21.15.86:0/3970937383 >> v2:172.21.15.203:3300/0 conn(0x7fc6b40a50e0 msgr2=0x7fc6b409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 --2- 172.21.15.86:0/3970937383 >> v2:172.21.15.203:3300/0 conn(0x7fc6b40a50e0 0x7fc6b409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 -- 172.21.15.86:0/3970937383 >> v1:172.21.15.203:6789/0 conn(0x7fc6b40a54c0 legacy=0x7fc6b40a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.136+0000 7fc6b37fe640 1 -- 172.21.15.86:0/3970937383 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b409c380 con 0x7fc6b413d2f0 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b37fe640 1 -- 172.21.15.86:0/3970937383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 msgr2=0x7fc6b413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b37fe640 1 --2- 172.21.15.86:0/3970937383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b413f6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40aa620 tx=0x7fc6ac01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220505f9e7830cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 1 -- 172.21.15.86:0/3970937383 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0b0070 con 0x7fc6b413d2f0 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: dump: 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 1 -- 172.21.15.86:0/3970937383 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac0ab600 con 0x7fc6b413d2f0 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 1 -- 172.21.15.86:0/3970937383 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0ab9d0 con 0x7fc6b413d2f0 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 5 monclient: authenticate success, global_id 4779 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 4 set_mon_vals no callback set 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b1ffb640 10 monclient: dump: 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 10 monclient: shutdown 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 -- 172.21.15.86:0/3970937383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 msgr2=0x7fc6b413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/3970937383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b413f6e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc6b40aa620 tx=0x7fc6ac01c3b0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 -- 172.21.15.86:0/3970937383 shutdown_connections 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/3970937383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b413f6e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/3970937383 >> v2:172.21.15.203:3300/0 conn(0x7fc6b40a50e0 0x7fc6b409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 -- 172.21.15.86:0/3970937383 >> 172.21.15.86:0/3970937383 conn(0x7fc6b40aaae0 msgr2=0x7fc6b40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 -- 172.21.15.86:0/3970937383 shutdown_connections 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.140+0000 7fc6b93f8640 1 -- 172.21.15.86:0/3970937383 wait complete. 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: monmap: 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 Processor -- start 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 -- start start 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: init 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b40a50e0 0x7fc6b409fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6b40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b40a54c0 0x7fc6b4121290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-b con 0x7fc6b40a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): picked mon.noname-a con 0x7fc6b413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b4142650 con 0x7fc6b413d2f0 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b40a6ad0 con 0x7fc6b40a54c0 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b41427d0 con 0x7fc6b40a50e0 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient(hunting): authenticate will time out at 1270.113013s 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b40a50e0 0x7fc6b409fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39124/0 (socket says 172.21.15.86:39124) 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 learned_addr learned my addr 172.21.15.86:0/829698607 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): get_auth_request con 0x7fc6b413d2f0 auth_method 0 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request con 0x7fc6b40a50e0 auth_method 0 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b27fc640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b40a54c0 0x7fc6b4121290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b27fc640 10 monclient(hunting): get_auth_request con 0x7fc6b40a54c0 auth_method 0 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient(hunting): handle_auth_done global_id 4782 payload 824 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b40a50e0 msgr2=0x7fc6b409fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b40a50e0 0x7fc6b409fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b40a54c0 msgr2=0x7fc6b4121290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b40a54c0 0x7fc6b4121290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b4138da0 con 0x7fc6b413d2f0 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:663: done (5 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:664: done (4 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:666: done (3 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:667: done (2 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:668: done (1 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:670: done (0 left) 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:671: copy_from oid 36 from oid 33 current snap is 70 2024-09-17T15:32:17.159 INFO:tasks.rados.rados.0.smithi203.stdout:672: snap_remove snap 59 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 msgr2=0x7fc6b4138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b37fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0ae040 tx=0x7fc6ac019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0b9020 con 0x7fc6b413d2f0 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 10 monclient: dump: 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.164 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 5 monclient: authenticate success, global_id 4782 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac0b1040 con 0x7fc6b413d2f0 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: _renew_subs 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.144+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b40a1110 con 0x7fc6b413d2f0 2024-09-17T15:32:17.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0b0070 con 0x7fc6b413d2f0 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: dump: 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b17fa640 4 set_mon_vals no callback set 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 10 monclient: _renew_subs 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b4120d30 con 0x7fc6b413d2f0 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 0x7fc6b41cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b406a0e0 con 0x7fc6b4138640 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 0x7fc6b41cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient: get_auth_request con 0x7fc6b4138640 auth_method 0 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_done global_id 4785 payload 824 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6b409cd60 con 0x7fc6b4138640 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc6ac0bc020 con 0x7fc6b413d2f0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 4815+0+0 (secure 0 0 0) 0x7fc6ac0e9d90 con 0x7fc6b413d2f0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b27fc640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b27fc640 10 monclient: get_auth_request con 0x7fc69403bb60 auth_method 0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 msgr2=0x7fc6b41cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6b2ffd640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 0x7fc6b41cfa00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0119a0 tx=0x7fc69c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c019070 con 0x7fc6b4138640 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: dump: 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc69c00f950 con 0x7fc6b4138640 2024-09-17T15:32:17.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 msgr2=0x7fc6b41cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 0x7fc6b41cfa00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0119a0 tx=0x7fc69c011e60 comp rx=0 tx=0).stop 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 msgr2=0x7fc6b41cfa00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.148+0000 7fc6ab7fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b4138640 0x7fc6b41cfa00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.152+0000 7fc6b27fc640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6b41d0ac0 tx=0x7fc6a4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b93f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 0x7fc6b41d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b93f8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b41d0890 con 0x7fc6b41cff40 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 0x7fc6b41d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient: get_auth_request con 0x7fc6b41cff40 auth_method 0 2024-09-17T15:32:17.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 10 monclient(hunting): handle_auth_done global_id 4788 payload 824 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.156+0000 7fc6b2ffd640 1 -- 172.21.15.86:0/829698607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6b4074be0 con 0x7fc6b41cff40 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b2ffd640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 msgr2=0x7fc6b41d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b2ffd640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 0x7fc6b41d0350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0119a0 tx=0x7fc69c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c019040 con 0x7fc6b41cff40 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 10 monclient: dump: 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fc69c01e660 con 0x7fc6b41cff40 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 msgr2=0x7fc6b41d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 0x7fc6b41d0350 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0119a0 tx=0x7fc69c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 msgr2=0x7fc6b41d0350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6ab7fe640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b41cff40 0x7fc6b41d0350 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0119a0 tx=0x7fc69c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:32:17.173 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 309804, 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.174 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.175 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:17.176 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:17.177 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:17.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.180 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.182 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.183 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.184 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:32:17.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:17.186 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:17.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 msgr2=0x7fc69403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6b41d0ac0 tx=0x7fc6a4009040 comp rx=0 tx=0).stop 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 10 monclient: shutdown 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 msgr2=0x7fc6b4138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0ae040 tx=0x7fc6ac019150 comp rx=0 tx=0).stop 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b37fe640 1 -- 172.21.15.86:0/829698607 reap_dead start 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 shutdown_connections 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 --2- 172.21.15.86:0/829698607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b413d2f0 0x7fc6b4138100 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 >> 172.21.15.86:0/829698607 conn(0x7fc6b40aaae0 msgr2=0x7fc6b40a3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 shutdown_connections 2024-09-17T15:32:17.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.160+0000 7fc6b93f8640 1 -- 172.21.15.86:0/829698607 wait complete. 2024-09-17T15:32:17.189 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:17.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:32:17.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:32:17.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: get_monmap_and_config 2024-09-17T15:32:17.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: monmap: 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:17.304550+0000 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:17.304550+0000 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:17.304 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 Processor -- start 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 -- start start 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: init 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2e3c06dee0 0x7f2e3c06f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c06dee0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e3c0a2e70 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e3c0a3990 0x7f2e3c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-a con 0x7f2e3c0a3990 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:17.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e3c0a4950 con 0x7f2e3c0a2e70 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2e3c06e2c0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2e3c06ec00 con 0x7f2e3c06dee0 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e43375640 10 monclient(hunting): authenticate will time out at 1270.269969s 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e3c0a3990 0x7f2e3c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e3c0a3990 0x7f2e3c13f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39162/0 (socket says 172.21.15.86:39162) 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 -- 172.21.15.86:0/3913139396 learned_addr learned my addr 172.21.15.86:0/3913139396 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): get_auth_request con 0x7f2e3c0a3990 auth_method 0 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41111640 1 --2- 172.21.15.86:0/3913139396 >> v2:172.21.15.203:3300/0 conn(0x7f2e3c06dee0 0x7f2e3c06f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41111640 10 monclient(hunting): get_auth_request con 0x7f2e3c06dee0 auth_method 0 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41111640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41111640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient(hunting): handle_auth_done global_id 4791 payload 274 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 -- 172.21.15.86:0/3913139396 >> v2:172.21.15.203:3300/0 conn(0x7f2e3c06dee0 msgr2=0x7f2e3c06f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 --2- 172.21.15.86:0/3913139396 >> v2:172.21.15.203:3300/0 conn(0x7f2e3c06dee0 0x7f2e3c06f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 -- 172.21.15.86:0/3913139396 >> v1:172.21.15.203:6789/0 conn(0x7f2e3c0a2e70 legacy=0x7f2e3c0a3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.300+0000 7f2e41912640 1 -- 172.21.15.86:0/3913139396 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3c06eee0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e41912640 1 -- 172.21.15.86:0/3913139396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 msgr2=0x7f2e3c13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e41912640 1 --2- 172.21.15.86:0/3913139396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c13f430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0a90b0 tx=0x7f2e2c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291977bae9a1fa83 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 1 -- 172.21.15.86:0/3913139396 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0a5640 con 0x7f2e3c0a3990 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: dump: 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 1 -- 172.21.15.86:0/3913139396 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e2c0a57e0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 5 monclient: authenticate success, global_id 4791 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 1 -- 172.21.15.86:0/3913139396 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0a5bb0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 4 set_mon_vals no callback set 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e3bfff640 10 monclient: dump: 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.314 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 10 monclient: shutdown 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 -- 172.21.15.86:0/3913139396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 msgr2=0x7f2e3c13f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 --2- 172.21.15.86:0/3913139396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c13f430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c0a90b0 tx=0x7f2e2c01c8b0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 -- 172.21.15.86:0/3913139396 shutdown_connections 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 --2- 172.21.15.86:0/3913139396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c13f430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 --2- 172.21.15.86:0/3913139396 >> v2:172.21.15.203:3300/0 conn(0x7f2e3c06dee0 0x7f2e3c06f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 -- 172.21.15.86:0/3913139396 >> 172.21.15.86:0/3913139396 conn(0x7f2e3c09bc70 msgr2=0x7f2e3c0a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 -- 172.21.15.86:0/3913139396 shutdown_connections 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.304+0000 7f2e43375640 1 -- 172.21.15.86:0/3913139396 wait complete. 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient: monmap: 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 Processor -- start 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 -- start start 2024-09-17T15:32:17.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient: init 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c06dee0 0x7f2e3c0a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-c con 0x7f2e3c06dee0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e3c0a2e70 0x7f2e3c0a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-b con 0x7f2e3c0a2e70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:17.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): picked mon.noname-a con 0x7f2e3c0a3990 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c142520 con 0x7f2e3c0a3990 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3c1423a0 con 0x7f2e3c0a2e70 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3c1426a0 con 0x7f2e3c06dee0 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e43375640 10 monclient(hunting): authenticate will time out at 1270.277625s 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c06dee0 0x7f2e3c0a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39178/0 (socket says 172.21.15.86:39178) 2024-09-17T15:32:17.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 learned_addr learned my addr 172.21.15.86:0/2236376271 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): get_auth_request con 0x7f2e3c0a3990 auth_method 0 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): get_auth_request con 0x7f2e3c06dee0 auth_method 0 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e40910640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e3c0a2e70 0x7f2e3c0a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient(hunting): handle_auth_done global_id 4794 payload 824 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c06dee0 msgr2=0x7f2e3c0a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c06dee0 0x7f2e3c0a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e3c0a2e70 msgr2=0x7f2e3c0a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e3c0a2e70 0x7f2e3c0a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e3c13b090 con 0x7f2e3c0a3990 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e41111640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c06dee0 0x7f2e3c0a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:32:17.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e40910640 10 monclient: get_auth_request con 0x7f2e3c0a2e70 auth_method 0 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.308+0000 7f2e40910640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e3c0a2e70 0x7f2e3c0a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 msgr2=0x7f2e3c0a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41912640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0af040 tx=0x7f2e2c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0bc020 con 0x7f2e3c0a3990 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: dump: 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.327 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e2c0b8ab0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 5 monclient: authenticate success, global_id 4794 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 monclient: _renew_subs 2024-09-17T15:32:17.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e3c13b370 con 0x7f2e3c0a3990 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e2c0a5640 con 0x7f2e3c0a3990 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 10 monclient: dump: 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.329 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e3b7fe640 4 set_mon_vals no callback set 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 monclient: _renew_subs 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e3c0738a0 con 0x7f2e3c0a3990 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 0x7f2e3c1cff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3c06a0e0 con 0x7f2e3c1cfb90 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 0x7f2e3c1cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient: get_auth_request con 0x7f2e3c1cfb90 auth_method 0 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2e2c0bf020 con 0x7f2e3c0a3990 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 10 monclient(hunting): handle_auth_done global_id 44228 payload 824 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e41111640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e3c09f410 con 0x7f2e3c1cfb90 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bbe0 0x7f2e2803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(157..157 src has 1..157) ==== 4815+0+0 (secure 0 0 0) 0x7f2e2c0eb110 con 0x7f2e3c0a3990 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e40910640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bbe0 0x7f2e2803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.312+0000 7f2e40910640 10 monclient: get_auth_request con 0x7f2e2803bbe0 auth_method 0 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e41111640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 msgr2=0x7f2e3c1cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e41111640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 0x7f2e3c1cff70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e3400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34018020 con 0x7f2e3c1cfb90 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 10 monclient: dump: 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.332 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e34010e90 con 0x7f2e3c1cfb90 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 msgr2=0x7f2e3c1cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 0x7f2e3c1cff70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e3400e5a0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 msgr2=0x7f2e3c1cff70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e39ffb640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1cfb90 0x7f2e3c1cff70 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e3400e5a0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.316+0000 7f2e40910640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bbe0 0x7f2e2803e0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c13a3f0 tx=0x7f2e30009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e43375640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e43375640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 0x7f2e3c09c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e43375640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3c09f660 con 0x7f2e3c1d04b0 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 0x7f2e3c09c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient: get_auth_request con 0x7f2e3c1d04b0 auth_method 0 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 10 monclient(hunting): handle_auth_done global_id 44231 payload 824 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.320+0000 7f2e41111640 1 -- 172.21.15.86:0/2236376271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e3c06f8c0 con 0x7f2e3c1d04b0 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e41111640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 msgr2=0x7f2e3c09c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e41111640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 0x7f2e3c09c370 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e340046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34018030 con 0x7f2e3c1d04b0 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 10 monclient: dump: 2024-09-17T15:32:17.336 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f2e3401f270 con 0x7f2e3c1d04b0 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:17.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 msgr2=0x7f2e3c09c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 0x7f2e3c09c370 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e340046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 msgr2=0x7f2e3c09c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e39ffb640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e3c1d04b0 0x7f2e3c09c370 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e34010b90 tx=0x7f2e340046d0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 66278, 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:17.338 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.339 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.340 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:17.341 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:17.342 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:17.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.344 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.345 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.347 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:17.348 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.350 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:17.351 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bbe0 msgr2=0x7f2e2803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bbe0 0x7f2e2803e0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c13a3f0 tx=0x7f2e30009040 comp rx=0 tx=0).stop 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 10 monclient: shutdown 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 msgr2=0x7f2e3c0a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0af040 tx=0x7f2e2c019650 comp rx=0 tx=0).stop 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e41912640 1 -- 172.21.15.86:0/2236376271 reap_dead start 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 shutdown_connections 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 --2- 172.21.15.86:0/2236376271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e3c0a3990 0x7f2e3c0a1660 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 >> 172.21.15.86:0/2236376271 conn(0x7f2e3c09bc70 msgr2=0x7f2e3c0a81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 shutdown_connections 2024-09-17T15:32:17.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.324+0000 7f2e43375640 1 -- 172.21.15.86:0/2236376271 wait complete. 2024-09-17T15:32:17.354 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:17.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:32:17.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:32:17.440 INFO:tasks.rados.rados.0.smithi203.stdout:672: done (1 left) 2024-09-17T15:32:17.440 INFO:tasks.rados.rados.0.smithi203.stdout:673: copy_from oid 5 from oid 8 current snap is 70 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:671: finishing copy_from to smithi20328086-36 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:671: got expected ENOENT (src dne) 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:671: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:671: done (1 left) 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:674: write oid 33 current snap is 70 2024-09-17T15:32:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:674: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-17T15:32:17.458 INFO:tasks.rados.rados.0.smithi203.stdout:674: writing smithi20328086-33 from 605118 to 1012052 tid 1 2024-09-17T15:32:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.476+0000 7f81da398640 10 monclient: get_monmap_and_config 2024-09-17T15:32:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.476+0000 7f81da398640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.476+0000 7f81da398640 10 monclient: monmap: 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:17.483643+0000 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:17.483643+0000 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:17.483 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.485 INFO:tasks.rados.rados.0.smithi203.stdout:674: writing smithi20328086-33 from 1617125 to 2323542 tid 2 2024-09-17T15:32:17.485 INFO:tasks.rados.rados.0.smithi203.stdout:675: read oid 26 snap -1 2024-09-17T15:32:17.485 INFO:tasks.rados.rados.0.smithi203.stdout:675: expect deleted 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:676: copy_from oid 49 from oid 39 current snap is 70 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:677: read oid 34 snap -1 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:677: expect (ObjNum 270 snap 70 seq_num 270) 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:678: read oid 24 snap -1 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:678: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:32:17.486 INFO:tasks.rados.rados.0.smithi203.stdout:679: snap_remove snap 63 2024-09-17T15:32:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 Processor -- start 2024-09-17T15:32:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 -- start start 2024-09-17T15:32:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient: init 2024-09-17T15:32:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): picked mon.noname-b con 0x7f81d40a2d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81d40a3170 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 0x7f81d413d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): picked mon.noname-a con 0x7f81d413afa0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d409cfb0 con 0x7f81d40a3170 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f81d40a4960 con 0x7f81d413afa0 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f81d409d460 con 0x7f81d40a2d90 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81da398640 10 monclient(hunting): authenticate will time out at 1270.449510s 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 0x7f81d413d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59786/0 (socket says 172.21.15.86:59786) 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 0x7f81d413d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39182/0 (socket says 172.21.15.86:39182) 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 -- 172.21.15.86:0/4260852562 learned_addr learned my addr 172.21.15.86:0/4260852562 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): get_auth_request con 0x7f81d40a2d90 auth_method 0 2024-09-17T15:32:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): get_auth_request con 0x7f81d413afa0 auth_method 0 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d8935640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient(hunting): handle_auth_done global_id 44234 payload 274 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 -- 172.21.15.86:0/4260852562 >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 msgr2=0x7f81d413d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 --2- 172.21.15.86:0/4260852562 >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 0x7f81d413d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 -- 172.21.15.86:0/4260852562 >> v1:172.21.15.86:6789/0 conn(0x7f81d40a3170 legacy=0x7f81d40a3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 monclient: found mon.noname-b 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:32:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.480+0000 7f81d3fff640 1 -- 172.21.15.86:0/4260852562 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d409d740 con 0x7f81d40a2d90 2024-09-17T15:32:17.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d3fff640 1 -- 172.21.15.86:0/4260852562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 msgr2=0x7f81d409e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d3fff640 1 --2- 172.21.15.86:0/4260852562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009870 tx=0x7f81c401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83c04edefdf53c26 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 1 -- 172.21.15.86:0/4260852562 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40b0070 con 0x7f81d40a2d90 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: dump: 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 1 -- 172.21.15.86:0/4260852562 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c40ab600 con 0x7f81d40a2d90 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 1 -- 172.21.15.86:0/4260852562 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40ab9d0 con 0x7f81d40a2d90 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 5 monclient: authenticate success, global_id 44234 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 4 set_mon_vals no callback set 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81d2ffd640 10 monclient: dump: 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 10 monclient: shutdown 2024-09-17T15:32:17.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 -- 172.21.15.86:0/4260852562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 msgr2=0x7f81d409e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 --2- 172.21.15.86:0/4260852562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009870 tx=0x7f81c401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 -- 172.21.15.86:0/4260852562 shutdown_connections 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 --2- 172.21.15.86:0/4260852562 >> v2:172.21.15.86:3300/0 conn(0x7f81d413afa0 0x7f81d413d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 --2- 172.21.15.86:0/4260852562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a2d90 0x7f81d409e460 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 -- 172.21.15.86:0/4260852562 >> 172.21.15.86:0/4260852562 conn(0x7f81d40a8390 msgr2=0x7f81d40a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 -- 172.21.15.86:0/4260852562 shutdown_connections 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.484+0000 7f81da398640 1 -- 172.21.15.86:0/4260852562 wait complete. 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient: monmap: 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.498 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 Processor -- start 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 -- start start 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient: init 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 0x7f81d411f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): picked mon.noname-a con 0x7f81d40a2d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a3170 0x7f81d40a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): picked mon.noname-c con 0x7f81d40a3170 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d413afa0 0x7f81d4135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): picked mon.noname-b con 0x7f81d413afa0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f81d4140510 con 0x7f81d40a2d90 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81d4140390 con 0x7f81d413afa0 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f81d4140690 con 0x7f81d40a3170 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81da398640 10 monclient(hunting): authenticate will time out at 1270.457416s 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 0x7f81d411f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 0x7f81d411f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39192/0 (socket says 172.21.15.86:39192) 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 -- 172.21.15.86:0/1864614791 learned_addr learned my addr 172.21.15.86:0/1864614791 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): get_auth_request con 0x7f81d40a2d90 auth_method 0 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a3170 0x7f81d40a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): get_auth_request con 0x7f81d40a3170 auth_method 0 2024-09-17T15:32:17.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient(hunting): handle_auth_done global_id 4800 payload 824 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a3170 msgr2=0x7f81d40a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81d40a3170 0x7f81d40a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d413afa0 msgr2=0x7f81d4135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d413afa0 0x7f81d4135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.488+0000 7f81d3fff640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d413d3a0 con 0x7f81d40a2d90 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d3fff640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 msgr2=0x7f81d411f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d3fff640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 0x7f81d411f080 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f81c40ae040 tx=0x7f81c401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40ba020 con 0x7f81d40a2d90 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: dump: 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c40b6630 con 0x7f81d40a2d90 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40b0040 con 0x7f81d40a2d90 2024-09-17T15:32:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 10 monclient: dump: 2024-09-17T15:32:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.510 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.510 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.510 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.510 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.510 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d27fc640 4 set_mon_vals no callback set 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 5 monclient: authenticate success, global_id 4800 2024-09-17T15:32:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 monclient: _renew_subs 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d40a1fa0 con 0x7f81d40a2d90 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 monclient: _renew_subs 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d41205a0 con 0x7f81d40a2d90 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 0x7f81d41cda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81d409f580 con 0x7f81d4136430 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f81c40bd020 con 0x7f81d40a2d90 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 0x7f81d41cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient: get_auth_request con 0x7f81d4136430 auth_method 0 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f81c003bbe0 0x7f81c003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f81c40ea200 con 0x7f81d40a2d90 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d8935640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f81c003bbe0 0x7f81c003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d8935640 10 monclient: get_auth_request con 0x7f81c003bbe0 auth_method 0 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 10 monclient(hunting): handle_auth_done global_id 54121 payload 824 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.492+0000 7f81d37fe640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81d409e460 con 0x7f81d4136430 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d37fe640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 msgr2=0x7f81d41cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d37fe640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 0x7f81d41cda60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81d411eb40 tx=0x7f81c800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d8935640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f81c003bbe0 0x7f81c003e0a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81d4072210 tx=0x7f81cc00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8018020 con 0x7f81d4136430 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 10 monclient: dump: 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.515 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f81c800d980 con 0x7f81d4136430 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 msgr2=0x7f81d41cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 0x7f81d41cda60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81d411eb40 tx=0x7f81c800fc00 comp rx=0 tx=0).stop 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 msgr2=0x7f81d41cda60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.496+0000 7f81d0ff9640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d4136430 0x7f81d41cda60 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f81d411eb40 tx=0x7f81c800fc00 comp rx=0 tx=0).stop 2024-09-17T15:32:17.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81da398640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:17.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81da398640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 0x7f81d41ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81da398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81d406a0e0 con 0x7f81d41cdfa0 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 0x7f81d41ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient: get_auth_request con 0x7f81d41cdfa0 auth_method 0 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 10 monclient(hunting): handle_auth_done global_id 54124 payload 824 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.504+0000 7f81d37fe640 1 -- 172.21.15.86:0/1864614791 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f81d41ce8c0 con 0x7f81d41cdfa0 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d37fe640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 msgr2=0x7f81d41ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d37fe640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 0x7f81d41ce380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f81c8019000 tx=0x7f81c8003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8018070 con 0x7f81d41cdfa0 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 10 monclient: dump: 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f81c8007030 con 0x7f81d41cdfa0 2024-09-17T15:32:17.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 msgr2=0x7f81d41ce380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 0x7f81d41ce380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f81c8019000 tx=0x7f81c8003770 comp rx=0 tx=0).stop 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 msgr2=0x7f81d41ce380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d0ff9640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81d41cdfa0 0x7f81d41ce380 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f81c8019000 tx=0x7f81c8003770 comp rx=0 tx=0).stop 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:17.520 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 56, 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12943, 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:32:17.521 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.522 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.524 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:17.525 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:17.526 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:17.527 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:17.528 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:17.529 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.530 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:17.531 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.532 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.533 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:17.534 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:17.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f81c003bbe0 msgr2=0x7f81c003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f81c003bbe0 0x7f81c003e0a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f81d4072210 tx=0x7f81cc00b040 comp rx=0 tx=0).stop 2024-09-17T15:32:17.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 10 monclient: shutdown 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 msgr2=0x7f81d411f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 --2- 172.21.15.86:0/1864614791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81d40a2d90 0x7f81d411f080 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f81c40ae040 tx=0x7f81c401ac20 comp rx=0 tx=0).stop 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d8935640 1 -- 172.21.15.86:0/1864614791 reap_dead start 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81d8935640 1 -- 172.21.15.86:0/1864614791 reap_dead start 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 shutdown_connections 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 >> 172.21.15.86:0/1864614791 conn(0x7f81d40a8390 msgr2=0x7f81d40742e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 shutdown_connections 2024-09-17T15:32:17.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.508+0000 7f81da398640 1 -- 172.21.15.86:0/1864614791 wait complete. 2024-09-17T15:32:17.537 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:32:17.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:32:17.537 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:32:17.537 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:32:17.537 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:32:17.537 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:32:17.537 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:32:17.548 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:32:17.548 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:32:17.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: get_monmap_and_config 2024-09-17T15:32:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: monmap: 2024-09-17T15:32:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:17.705088+0000 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:17.705088+0000 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:17.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 Processor -- start 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 -- start start 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: init 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa1380a50e0 0x7fa13809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-c con 0x7fa1380a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa1380a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-b con 0x7fa1380a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa138137ea0 0x7fa138138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-a con 0x7fa138137ea0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa13809bdc0 con 0x7fa138137ea0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa13809d350 con 0x7fa1380a50e0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380a6530 con 0x7fa1380a54c0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa13dbd7640 10 monclient(hunting): authenticate will time out at 1270.670532s 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa136ffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 msgr2=0x7fa1380a58d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa1380a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa138137ea0 0x7fa138138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa138137ea0 0x7fa138138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39194/0 (socket says 172.21.15.86:39194) 2024-09-17T15:32:17.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 -- 172.21.15.86:0/157257619 learned_addr learned my addr 172.21.15.86:0/157257619 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): get_auth_request con 0x7fa138137ea0 auth_method 0 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 1 --2- 172.21.15.86:0/157257619 >> v2:172.21.15.203:3300/0 conn(0x7fa1380a50e0 0x7fa13809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): get_auth_request con 0x7fa1380a50e0 auth_method 0 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa1377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient(hunting): handle_auth_done global_id 4803 payload 274 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 -- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 msgr2=0x7fa1380a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 --2- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa1380a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 -- 172.21.15.86:0/157257619 >> v2:172.21.15.203:3300/0 conn(0x7fa1380a50e0 msgr2=0x7fa13809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 --2- 172.21.15.86:0/157257619 >> v2:172.21.15.203:3300/0 conn(0x7fa1380a50e0 0x7fa13809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.700+0000 7fa137fff640 1 -- 172.21.15.86:0/157257619 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13809c150 con 0x7fa138137ea0 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa137fff640 1 -- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 msgr2=0x7fa138138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa137fff640 1 --2- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138138330 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa1380a8280 tx=0x7fa13001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a8183864ffbb07 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 1 -- 172.21.15.86:0/157257619 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300a5640 con 0x7fa138137ea0 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: dump: 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 1 -- 172.21.15.86:0/157257619 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1300a57e0 con 0x7fa138137ea0 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 1 -- 172.21.15.86:0/157257619 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300a5bb0 con 0x7fa138137ea0 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa13dbd7640 5 monclient: authenticate success, global_id 4803 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa13dbd7640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa13dbd7640 4 set_mon_vals no callback set 2024-09-17T15:32:17.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.704+0000 7fa1367fc640 10 monclient: dump: 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: shutdown 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 msgr2=0x7fa138138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138138330 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa1380a8280 tx=0x7fa13001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- 172.21.15.86:0/157257619 shutdown_connections 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138138330 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/157257619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa1380a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/157257619 >> v2:172.21.15.203:3300/0 conn(0x7fa1380a50e0 0x7fa13809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- 172.21.15.86:0/157257619 >> 172.21.15.86:0/157257619 conn(0x7fa1380aaae0 msgr2=0x7fa1380aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- 172.21.15.86:0/157257619 shutdown_connections 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- 172.21.15.86:0/157257619 wait complete. 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: build_initial_monmap 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: monmap: 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 Processor -- start 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- start start 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: init 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1380a50e0 0x7fa138121190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-b con 0x7fa1380a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa13813d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-c con 0x7fa1380a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): picked mon.noname-a con 0x7fa138137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa138141b20 con 0x7fa138137ea0 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa1380a5f00 con 0x7fa1380a50e0 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa138141ca0 con 0x7fa1380a54c0 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa13dbd7640 10 monclient(hunting): authenticate will time out at 1270.677675s 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa136ffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 msgr2=0x7fa13813d620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:17.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa136ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa13813d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39208/0 (socket says 172.21.15.86:39208) 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 -- 172.21.15.86:0/630596279 learned_addr learned my addr 172.21.15.86:0/630596279 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): get_auth_request con 0x7fa138137ea0 auth_method 0 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa1377fe640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1380a50e0 0x7fa138121190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa1377fe640 10 monclient(hunting): get_auth_request con 0x7fa1380a50e0 auth_method 0 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa1377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:17.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa1377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient(hunting): handle_auth_done global_id 4806 payload 824 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 -- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 msgr2=0x7fa13813d620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa13813d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 -- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1380a50e0 msgr2=0x7fa138121190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1380a50e0 0x7fa138121190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 monclient: found mon.noname-a 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.708+0000 7fa137fff640 1 -- 172.21.15.86:0/630596279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13813e5e0 con 0x7fa138137ea0 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa137fff640 1 -- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 msgr2=0x7fa138136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa137fff640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa13001d170 tx=0x7fa130019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300bc020 con 0x7fa138137ea0 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: dump: 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.724 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: _finish_auth 0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1300b8880 con 0x7fa138137ea0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 5 monclient: authenticate success, global_id 4806 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _renew_subs 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13813ec00 con 0x7fa138137ea0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300a5640 con 0x7fa138137ea0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 10 monclient: dump: 2024-09-17T15:32:17.725 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _renew_subs 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa1357fa640 4 set_mon_vals no callback set 2024-09-17T15:32:17.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1381220f0 con 0x7fa138137ea0 2024-09-17T15:32:17.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1380a58d0 con 0x7fa138137ea0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa1300bf020 con 0x7fa138137ea0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 0x7fa11803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7fa1300eaef0 con 0x7fa138137ea0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa1377fe640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 0x7fa11803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.712+0000 7fa1377fe640 10 monclient: get_auth_request con 0x7fa11803bb10 auth_method 0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.716+0000 7fa1377fe640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 0x7fa11803dfd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa120009a70 tx=0x7fa120008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:17.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.716+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa1300c6030 con 0x7fa138137ea0 2024-09-17T15:32:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.716+0000 7fa12ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:17.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.716+0000 7fa12ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:17.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa13dbd7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:17.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa13dbd7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:17.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa13dbd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:17.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa138074be0 con 0x7fa138137ea0 2024-09-17T15:32:17.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa12ffff640 1 -- 172.21.15.86:0/630596279 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa1300165e0 con 0x7fa138137ea0 2024-09-17T15:32:17.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa12ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:17.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.944+0000 7fa12ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:17.950 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 msgr2=0x7fa11803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 0x7fa11803dfd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa120009a70 tx=0x7fa120008040 comp rx=0 tx=0).stop 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 10 monclient: shutdown 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 msgr2=0x7fa138136ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa13001d170 tx=0x7fa130019670 comp rx=0 tx=0).stop 2024-09-17T15:32:17.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 shutdown_connections 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa11803bb10 0x7fa11803dfd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa138137ea0 0x7fa138136ff0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380a54c0 0x7fa13813d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 --2- 172.21.15.86:0/630596279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa1380a50e0 0x7fa138121190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 >> 172.21.15.86:0/630596279 conn(0x7fa1380aaae0 msgr2=0x7fa13809e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 shutdown_connections 2024-09-17T15:32:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:17.948+0000 7fa13dbd7640 1 -- 172.21.15.86:0/630596279 wait complete. 2024-09-17T15:32:20.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: get_monmap_and_config 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: build_initial_monmap 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: monmap: 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:21.097834+0000 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:21.097834+0000 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:21.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:21.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 Processor -- start 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 -- start start 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: init 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-17T15:32:21.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 0x7f8c5809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c580a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c580a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c580a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c58137e30 0x7f8c58138210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c58137e30 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8c580a6f40 con 0x7f8c58137e30 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8c580a65e0 con 0x7f8c580a5070 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8c5809c2a0 con 0x7f8c580a5450 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5fa3d640 10 monclient(hunting): authenticate will time out at 1274.063299s 2024-09-17T15:32:21.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5cfd8640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 msgr2=0x7f8c580a5860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5cfd8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c580a5860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c58137e30 0x7f8c58138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c58137e30 0x7f8c58138210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58116/0 (socket says 172.21.15.86:58116) 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 1 -- 172.21.15.86:0/1733015338 learned_addr learned my addr 172.21.15.86:0/1733015338 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): get_auth_request con 0x7f8c58137e30 auth_method 0 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1733015338 >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 0x7f8c5809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5d7d9640 10 monclient(hunting): get_auth_request con 0x7f8c580a5070 auth_method 0 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5d7d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:21.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5d7d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5d7d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 10 monclient(hunting): handle_auth_done global_id 4809 payload 274 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 1 -- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 msgr2=0x7f8c580a5860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.092+0000 7f8c5dfda640 1 --2- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c580a5860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 1 -- 172.21.15.86:0/1733015338 >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 msgr2=0x7f8c5809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 1 --2- 172.21.15.86:0/1733015338 >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 0x7f8c5809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 10 monclient: _finish_hunting 0 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 1 monclient: found mon.noname-a 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5dfda640 1 -- 172.21.15.86:0/1733015338 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5809c580 con 0x7f8c58137e30 2024-09-17T15:32:21.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.096+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1733015338 >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 0x7f8c5809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5dfda640 1 -- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c58137e30 msgr2=0x7f8c58138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5dfda640 1 --2- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c58137e30 0x7f8c58138210 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c5806e200 tx=0x7f8c4801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16463aafe0555f60 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 1 -- 172.21.15.86:0/1733015338 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480b0070 con 0x7f8c58137e30 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: dump: 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:21.105 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: _finish_auth 0 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 1 -- 172.21.15.86:0/1733015338 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c480ab600 con 0x7f8c58137e30 2024-09-17T15:32:21.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 5 monclient: authenticate success, global_id 4809 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 1 -- 172.21.15.86:0/1733015338 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c480ab9d0 con 0x7f8c58137e30 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 4 set_mon_vals no callback set 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c57fff640 10 monclient: dump: 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.107 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: shutdown 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c58137e30 msgr2=0x7f8c58138210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c58137e30 0x7f8c58138210 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8c5806e200 tx=0x7f8c4801c400 comp rx=0 tx=0).stop 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1733015338 shutdown_connections 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c58137e30 0x7f8c58138210 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1733015338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c580a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1733015338 >> v2:172.21.15.203:3300/0 conn(0x7f8c580a5070 0x7f8c5809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1733015338 >> 172.21.15.86:0/1733015338 conn(0x7f8c580742a0 msgr2=0x7f8c580746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1733015338 shutdown_connections 2024-09-17T15:32:21.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1733015338 wait complete. 2024-09-17T15:32:21.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: build_initial_monmap 2024-09-17T15:32:21.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: monmap: 2024-09-17T15:32:21.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:21.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:21.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 Processor -- start 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- start start 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: init 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c580a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:21.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c58137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c580a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c58137e30 0x7f8c5813c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c58137e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8c580a69e0 con 0x7f8c580a5070 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8c580a6860 con 0x7f8c58137e30 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8c58141ab0 con 0x7f8c580a5450 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5fa3d640 10 monclient(hunting): authenticate will time out at 1274.071623s 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5cfd8640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 msgr2=0x7f8c58137030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:21.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5cfd8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c58137030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58126/0 (socket says 172.21.15.86:58126) 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 -- 172.21.15.86:0/1131221085 learned_addr learned my addr 172.21.15.86:0/1131221085 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): get_auth_request con 0x7f8c580a5070 auth_method 0 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 -- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 msgr2=0x7f8c58137030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c58137030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 -- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c58137e30 msgr2=0x7f8c5813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:21.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c58137e30 0x7f8c5813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 10 monclient: _finish_hunting 0 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 1 monclient: found mon.noname-a 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.100+0000 7f8c5d7d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5d7d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5d7d9640 1 -- 172.21.15.86:0/1131221085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c5813dc50 con 0x7f8c580a5070 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5d7d9640 1 -- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 msgr2=0x7f8c581cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000d100 tx=0x7f8c5000d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5d7d9640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000d100 tx=0x7f8c5000d5c0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f8c50003ba0 config(0 keys) 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5001a070 con 0x7f8c580a5070 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: dump: 2024-09-17T15:32:21.116 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c50003ba0 con 0x7f8c580a5070 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:21.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 5 monclient: authenticate success, global_id 4812 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _renew_subs 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c58122150 con 0x7f8c580a5070 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c577fe640 4 set_mon_vals no callback set 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _renew_subs 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c581211f0 con 0x7f8c580a5070 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c580a5860 con 0x7f8c580a5070 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c50004030 con 0x7f8c580a5070 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:21.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 10 monclient: dump: 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8c5001d070 con 0x7f8c580a5070 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.104+0000 7f8c55ffb640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 0x7f8c4403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.108+0000 7f8c5cfd8640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 0x7f8c4403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:21.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.108+0000 7f8c5cfd8640 10 monclient: get_auth_request con 0x7f8c4403bb10 auth_method 0 2024-09-17T15:32:21.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.108+0000 7f8c5cfd8640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 0x7f8c4403dfd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c006fd0 tx=0x7f8c4c006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:21.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.112+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f8c500504b0 con 0x7f8c580a5070 2024-09-17T15:32:21.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.728+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8c50010070 con 0x7f8c580a5070 2024-09-17T15:32:21.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.728+0000 7f8c55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:21.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.728+0000 7f8c55ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c5fa3d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c5fa3d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c5fa3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8c58069920 con 0x7f8c580a5070 2024-09-17T15:32:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c55ffb640 1 -- 172.21.15.86:0/1131221085 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8c5004ee60 con 0x7f8c580a5070 2024-09-17T15:32:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.968+0000 7f8c55ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:21.974 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 msgr2=0x7f8c4403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 0x7f8c4403dfd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8c4c006fd0 tx=0x7f8c4c006d40 comp rx=0 tx=0).stop 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 10 monclient: shutdown 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 msgr2=0x7f8c581cfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000d100 tx=0x7f8c5000d5c0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 shutdown_connections 2024-09-17T15:32:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8c4403bb10 0x7f8c4403dfd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c58137e30 0x7f8c5813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c580a5450 0x7f8c58137030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 --2- 172.21.15.86:0/1131221085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c580a5070 0x7f8c581cfd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 >> 172.21.15.86:0/1131221085 conn(0x7f8c580742a0 msgr2=0x7f8c580a37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 shutdown_connections 2024-09-17T15:32:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:21.972+0000 7f8c5fa3d640 1 -- 172.21.15.86:0/1131221085 wait complete. 2024-09-17T15:32:24.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 10 monclient: get_monmap_and_config 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 10 monclient: build_initial_monmap 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 10 monclient: monmap: 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:25.123024+0000 2024-09-17T15:32:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:25.123024+0000 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:25.123 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 1 Processor -- start 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 1 -- start start 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.116+0000 7f87a90b3640 10 monclient: init 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f87a40689d0 0x7f87a406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f87a40689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f87a4068db0 0x7f87a4069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f87a4068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87a4135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a409c8b0 con 0x7f87a4135ae0 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f87a409cb90 con 0x7f87a40689d0 2024-09-17T15:32:25.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f87a409ce70 con 0x7f87a4068db0 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a90b3640 10 monclient(hunting): authenticate will time out at 1278.088636s 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f87a40689d0 0x7f87a406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f87a40689d0 0x7f87a406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58138/0 (socket says 172.21.15.86:58138) 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 -- 172.21.15.86:0/2639614504 learned_addr learned my addr 172.21.15.86:0/2639614504 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): get_auth_request con 0x7f87a40689d0 auth_method 0 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:25.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient(hunting): handle_auth_done global_id 4815 payload 274 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 -- 172.21.15.86:0/2639614504 >> v2:172.21.15.203:3300/0 conn(0x7f87a4068db0 msgr2=0x7f87a4069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 --2- 172.21.15.86:0/2639614504 >> v2:172.21.15.203:3300/0 conn(0x7f87a4068db0 0x7f87a4069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 -- 172.21.15.86:0/2639614504 >> v1:172.21.15.203:6789/0 conn(0x7f87a4135ae0 legacy=0x7f87a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.120+0000 7f87a2d9d640 1 -- 172.21.15.86:0/2639614504 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a4069740 con 0x7f87a40689d0 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a2d9d640 1 -- 172.21.15.86:0/2639614504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a40689d0 msgr2=0x7f87a406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a2d9d640 1 --2- 172.21.15.86:0/2639614504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a40689d0 0x7f87a406f840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f87a40a05c0 tx=0x7f879401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d8bfa024b7f5fd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 1 -- 172.21.15.86:0/2639614504 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87940b0070 con 0x7f87a40689d0 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: dump: 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.130 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: _finish_auth 0 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 5 monclient: authenticate success, global_id 4815 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 1 -- 172.21.15.86:0/2639614504 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87940ab600 con 0x7f87a40689d0 2024-09-17T15:32:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 1 -- 172.21.15.86:0/2639614504 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87940ab9d0 con 0x7f87a40689d0 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 4 set_mon_vals no callback set 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a1d9b640 10 monclient: dump: 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:25.133 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:25.133 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 10 monclient: shutdown 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 -- 172.21.15.86:0/2639614504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a40689d0 msgr2=0x7f87a406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 --2- 172.21.15.86:0/2639614504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a40689d0 0x7f87a406f840 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f87a40a05c0 tx=0x7f879401c470 comp rx=0 tx=0).stop 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 -- 172.21.15.86:0/2639614504 shutdown_connections 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 --2- 172.21.15.86:0/2639614504 >> v2:172.21.15.203:3300/0 conn(0x7f87a4068db0 0x7f87a4069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 --2- 172.21.15.86:0/2639614504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a40689d0 0x7f87a406f840 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 -- 172.21.15.86:0/2639614504 >> 172.21.15.86:0/2639614504 conn(0x7f87a40a6440 msgr2=0x7f87a40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 -- 172.21.15.86:0/2639614504 shutdown_connections 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.124+0000 7f87a90b3640 1 -- 172.21.15.86:0/2639614504 wait complete. 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient: build_initial_monmap 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient: monmap: 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:25.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 Processor -- start 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 -- start start 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient: init 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 0x7f87a4130de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f87a40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87a4068db0 0x7f87a4131320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-b con 0x7f87a4068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f87a4135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f87a4140a40 con 0x7f87a4135ae0 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f87a409c2d0 con 0x7f87a4068db0 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f87a4140bc0 con 0x7f87a40689d0 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a90b3640 10 monclient(hunting): authenticate will time out at 1278.097226s 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a2d9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 msgr2=0x7f87a4130de0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a2d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 0x7f87a4130de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58148/0 (socket says 172.21.15.86:58148) 2024-09-17T15:32:25.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 -- 172.21.15.86:0/1547820843 learned_addr learned my addr 172.21.15.86:0/1547820843 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): get_auth_request con 0x7f87a4135ae0 auth_method 0 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a259c640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87a4068db0 0x7f87a4131320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a259c640 10 monclient(hunting): get_auth_request con 0x7f87a4068db0 auth_method 0 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:25.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient(hunting): handle_auth_done global_id 4818 payload 824 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 -- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 msgr2=0x7f87a4130de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 0x7f87a4130de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 -- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87a4068db0 msgr2=0x7f87a4131320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87a4068db0 0x7f87a4131320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient: _finish_hunting 0 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 monclient: found mon.noname-a 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.128+0000 7f87a359e640 1 -- 172.21.15.86:0/1547820843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a4139cc0 con 0x7f87a4135ae0 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a359e640 1 -- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 msgr2=0x7f87a41ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a359e640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f879c00c780 tx=0x7f879c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c017070 con 0x7f87a4135ae0 2024-09-17T15:32:25.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: dump: 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:25.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879c005070 con 0x7f87a4135ae0 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c005460 con 0x7f87a4135ae0 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 10 monclient: dump: 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:25.144 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 5 monclient: authenticate success, global_id 4818 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _renew_subs 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a159a640 4 set_mon_vals no callback set 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a4131860 con 0x7f87a4135ae0 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _renew_subs 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a413c020 con 0x7f87a4135ae0 2024-09-17T15:32:25.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a406f840 con 0x7f87a4135ae0 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f879c005680 con 0x7f87a4135ae0 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 0x7f879003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f879c04f460 con 0x7f87a4135ae0 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a2d9d640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 0x7f879003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:25.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.132+0000 7f87a2d9d640 10 monclient: get_auth_request con 0x7f879003bb10 auth_method 0 2024-09-17T15:32:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.136+0000 7f87a2d9d640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 0x7f879003dfd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f87a40a0620 tx=0x7f8794008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.136+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f879c04de10 con 0x7f87a4135ae0 2024-09-17T15:32:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.136+0000 7f878b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:25.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.136+0000 7f878b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:25.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.360+0000 7f87a90b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:25.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:25.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:25.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87a413a820 con 0x7f87a4135ae0 2024-09-17T15:32:25.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f878b7fe640 1 -- 172.21.15.86:0/1547820843 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f879c04f100 con 0x7f87a4135ae0 2024-09-17T15:32:25.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f878b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:25.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f878b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:25.368 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:25.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 msgr2=0x7f879003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 0x7f879003dfd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f87a40a0620 tx=0x7f8794008040 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 10 monclient: shutdown 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 msgr2=0x7f87a41ce0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f879c00c780 tx=0x7f879c00cc40 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 shutdown_connections 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f879003bb10 0x7f879003dfd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87a4135ae0 0x7f87a41ce0a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87a4068db0 0x7f87a4131320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 --2- 172.21.15.86:0/1547820843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87a40689d0 0x7f87a4130de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 >> 172.21.15.86:0/1547820843 conn(0x7f87a40a6440 msgr2=0x7f87a40a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:25.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 shutdown_connections 2024-09-17T15:32:25.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:25.364+0000 7f87a90b3640 1 -- 172.21.15.86:0/1547820843 wait complete. 2024-09-17T15:32:28.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: get_monmap_and_config 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: build_initial_monmap 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: monmap: 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:28.513621+0000 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:28.513621+0000 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:28.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 Processor -- start 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 -- start start 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: init 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae80689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ae80a2b80 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae80a2f90 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae80a4110 con 0x7f5ae80a2b80 2024-09-17T15:32:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae8068db0 con 0x7f5ae80a2f90 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5ae809c520 con 0x7f5ae80689d0 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5af039e640 10 monclient(hunting): authenticate will time out at 1281.479247s 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35506/0 (socket says 172.21.15.86:35506) 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 learned_addr learned my addr 172.21.15.86:0/1676386912 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ae80a2f90 auth_method 0 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:28.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aed939640 1 --1- 172.21.15.86:0/1676386912 >> v1:172.21.15.86:6789/0 conn(0x7f5ae80a2b80 0x7f5ae813a430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58366/0 (socket says 172.21.15.86:58366) 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ae80689d0 auth_method 0 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.508+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.512+0000 7f5aed939640 1 -- 172.21.15.86:0/1676386912 >> v1:172.21.15.86:6789/0 conn(0x7f5ae80a2b80 legacy=0x7f5ae813a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.512+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:32:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.512+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.512+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.512+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ae80689d0 auth_method 0 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:28.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.712+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:32:28.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:28.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ae80a2f90 auth_method 0 2024-09-17T15:32:28.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:28.716+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ae80689d0 auth_method 0 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:29.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.116+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ae80a2f90 auth_method 0 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:29.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.120+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:32:29.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ae80689d0 auth_method 0 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:29.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.920+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:32:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ae80a2f90 auth_method 0 2024-09-17T15:32:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:32:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:32:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:29.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:29.924+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): tick 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 msgr2=0x7f5ae80a2640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.203:3300/0 conn(0x7f5ae80689d0 0x7f5ae80a2640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 -- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 msgr2=0x7f5ae813db60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 --2- 172.21.15.86:0/1676386912 >> v2:172.21.15.86:3300/0 conn(0x7f5ae80a2f90 0x7f5ae813db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 -- 172.21.15.86:0/1676386912 >> v1:172.21.15.86:6789/0 conn(0x7f5ae80a2b80 legacy=0x7f5ae813a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient: _add_conns ranks=[1,2] 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5acc003700 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 0x7f5acc009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): picked mon.noname-b con 0x7f5acc007350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 -- 172.21.15.86:0/1676386912 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae80a3440 con 0x7f5acc003700 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 1 -- 172.21.15.86:0/1676386912 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae8068db0 con 0x7f5acc007350 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5adffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5aed939640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 msgr2=0x7f5acc009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:31.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5aed939640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 0x7f5acc009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:31.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1676386912 >> v1:172.21.15.203:6789/0 conn(0x7f5acc003700 legacy=0x7f5acc005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:31.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.716+0000 7f5aed939640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 msgr2=0x7f5acc009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:31.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:31.716+0000 7f5aed939640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 0x7f5acc009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:32:32.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.120+0000 7f5aed939640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 msgr2=0x7f5acc009810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:32.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.120+0000 7f5aed939640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 0x7f5acc009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:32:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1927678911 0 0) 0x7f5ae80a3440 con 0x7f5acc003700 2024-09-17T15:32:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): _negotiate 2024-09-17T15:32:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): my global_id is 54139 2024-09-17T15:32:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad4003530 con 0x7f5acc003700 2024-09-17T15:32:32.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1905195704 0 0) 0x7f5ad4003530 con 0x7f5acc003700 2024-09-17T15:32:32.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 msgr2=0x7f5acc009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:32.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5acc007350 0x7f5acc009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _finish_hunting 0 2024-09-17T15:32:32.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5acc00aff0 con 0x7f5acc003700 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _finish_auth 0 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _check_auth_tickets 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f5ae400f040 con 0x7f5acc003700 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: dump: 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 >> v1:172.21.15.203:6789/0 conn(0x7f5acc003700 legacy=0x7f5acc005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 reap_dead start 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:32.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ad4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): picked mon.a con 0x7f5ad4003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ad4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): picked mon.c con 0x7f5ad4004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ad400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): picked mon.b con 0x7f5ad4005420 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae8068db0 con 0x7f5ad4003a30 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed939640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 msgr2=0x7f5ad4004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed939640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ad4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ad4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ad400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad400c330 con 0x7f5ad4005420 2024-09-17T15:32:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad400c530 con 0x7f5ad4004810 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aed138640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ad4003a30 auth_method 0 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ad4005420 auth_method 0 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.496+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 10 monclient(hunting): handle_auth_done global_id 54139 payload 293 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 msgr2=0x7f5ad4004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ad4004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 msgr2=0x7f5ad4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ad4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 10 monclient: _finish_hunting 0 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 monclient: found mon.b 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:32:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad4003530 con 0x7f5ad4005420 2024-09-17T15:32:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 10 monclient: _finish_auth 0 2024-09-17T15:32:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 10 monclient: _check_auth_tickets 2024-09-17T15:32:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5aee93b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5af039e640 5 monclient: authenticate success, global_id 54139 2024-09-17T15:32:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.500+0000 7f5af039e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aee93b640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 msgr2=0x7f5ad400ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ad400ba50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ae809c370 tx=0x7f5ae001d0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=164fc49944accd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00ab070 con 0x7f5ad4005420 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: dump: 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae00a5be0 con 0x7f5ad4005420 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5af039e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5af039e640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 1 -- 172.21.15.86:0/1676386912 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0009700 con 0x7f5ad4005420 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5af039e640 4 set_mon_vals no callback set 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.504+0000 7f5aed138640 10 monclient: dump: 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: shutdown 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 msgr2=0x7f5ad400ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ad400ba50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5ae809c370 tx=0x7f5ae001d0f0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- 172.21.15.86:0/1676386912 shutdown_connections 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ad400ba50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ad4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- 172.21.15.86:0/1676386912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ad4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- 172.21.15.86:0/1676386912 >> 172.21.15.86:0/1676386912 conn(0x7f5ae806d170 msgr2=0x7f5ae80a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:32.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- 172.21.15.86:0/1676386912 shutdown_connections 2024-09-17T15:32:32.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- 172.21.15.86:0/1676386912 wait complete. 2024-09-17T15:32:32.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: build_initial_monmap 2024-09-17T15:32:32.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: monmap: 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 Processor -- start 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- start start 2024-09-17T15:32:32.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: init 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad4003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ae81359c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad4004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ae81cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad4005420 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae80744b0 con 0x7f5ad4003a30 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae80a0420 con 0x7f5ad4005420 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae8138c30 con 0x7f5ad4004810 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5af039e640 10 monclient(hunting): authenticate will time out at 1285.479886s 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5aed939640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 msgr2=0x7f5ae81359c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.508+0000 7f5aed939640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ae81359c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35568/0 (socket says 172.21.15.86:35568) 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1022330131 learned_addr learned my addr 172.21.15.86:0/1022330131 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): get_auth_request con 0x7f5ad4003a30 auth_method 0 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ae81cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): get_auth_request con 0x7f5ad4005420 auth_method 0 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee93b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient(hunting): handle_auth_done global_id 4821 payload 824 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 msgr2=0x7f5ae81359c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:32.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ae81359c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 msgr2=0x7f5ae81cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ae81cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient: _finish_hunting 0 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 monclient: found mon.noname-a 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1022330131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae81253c0 con 0x7f5ad4003a30 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 -- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 msgr2=0x7f5ae81cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aee13a640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400e1f0 tx=0x7f5ae40146c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4004450 con 0x7f5ad4003a30 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: dump: 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: _finish_auth 0 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:32.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae4003620 con 0x7f5ad4003a30 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40039d0 con 0x7f5ad4003a30 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5adeffd640 10 monclient: dump: 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.526 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5aec937640 4 set_mon_vals no callback set 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5af039e640 5 monclient: authenticate success, global_id 4821 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5af039e640 10 monclient: _renew_subs 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5af039e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.512+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae80705e0 con 0x7f5ad4003a30 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 10 monclient: _renew_subs 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ae81250e0 con 0x7f5ad4003a30 2024-09-17T15:32:32.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:32.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae80a3400 con 0x7f5ad4003a30 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5ae4003b70 con 0x7f5ad4003a30 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5adeffd640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 0x7f5acc03e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f5ae40510e0 con 0x7f5ad4003a30 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5aed939640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 0x7f5acc03e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.516+0000 7f5aed939640 10 monclient: get_auth_request con 0x7f5acc03c450 auth_method 0 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.520+0000 7f5aed939640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 0x7f5acc03e910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ae81238a0 tx=0x7f5ad8007a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.520+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5ae4017c40 con 0x7f5ad4003a30 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.520+0000 7f5adeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:32.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.520+0000 7f5adeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:32.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5af039e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:32.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5af039e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5af039e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ae8123f00 con 0x7f5ad4003a30 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5adeffd640 1 -- 172.21.15.86:0/1022330131 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f5ae401f3a0 con 0x7f5ad4003a30 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5adeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.752+0000 7f5adeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:32.758 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 msgr2=0x7f5acc03e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 0x7f5acc03e910 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5ae81238a0 tx=0x7f5ad8007a90 comp rx=0 tx=0).stop 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 10 monclient: shutdown 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 msgr2=0x7f5ae81cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400e1f0 tx=0x7f5ae40146c0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 shutdown_connections 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5acc03c450 0x7f5acc03e910 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ad4005420 0x7f5ae81cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ad4004810 0x7f5ae81359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 --2- 172.21.15.86:0/1022330131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ad4003a30 0x7f5ae81cfac0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 >> 172.21.15.86:0/1022330131 conn(0x7f5ae806d170 msgr2=0x7f5ae80736a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:32.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 shutdown_connections 2024-09-17T15:32:32.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.756+0000 7f5af039e640 1 -- 172.21.15.86:0/1022330131 wait complete. 2024-09-17T15:32:32.776 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:32:32.776 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:32:32.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:32:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: get_monmap_and_config 2024-09-17T15:32:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: build_initial_monmap 2024-09-17T15:32:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: monmap: 2024-09-17T15:32:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:32.884 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:32.885243+0000 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:32.885243+0000 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:32.885 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 Processor -- start 2024-09-17T15:32:32.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 -- start start 2024-09-17T15:32:32.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: init 2024-09-17T15:32:32.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f020809d770 0x7f020809db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f020809d770 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f020809e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0208134f80 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f020809ff00 con 0x7f0208134f80 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f020809e7f0 con 0x7f020809d770 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f02080a2110 con 0x7f020809e090 2024-09-17T15:32:32.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f020d9f0640 10 monclient(hunting): authenticate will time out at 1285.850779s 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f02067fc640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 msgr2=0x7f0208134a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f02067fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208134a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f020809d770 0x7f020809db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f020809d770 0x7f020809db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:59302/0 (socket says 172.21.15.86:59302) 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 1 -- 172.21.15.86:0/2739216112 learned_addr learned my addr 172.21.15.86:0/2739216112 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): get_auth_request con 0x7f020809d770 auth_method 0 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.880+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:32.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f02077fe640 1 -- 172.21.15.86:0/2739216112 >> v1:172.21.15.203:6789/0 conn(0x7f0208134f80 legacy=0x7f020813bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:32.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 10 monclient(hunting): handle_auth_done global_id 54145 payload 274 2024-09-17T15:32:32.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 1 -- 172.21.15.86:0/2739216112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 msgr2=0x7f0208134a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:32.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 1 --2- 172.21.15.86:0/2739216112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208134a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 1 -- 172.21.15.86:0/2739216112 >> v1:172.21.15.203:6789/0 conn(0x7f0208134f80 legacy=0x7f020813bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 1 monclient: found mon.noname-c 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:32:32.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.884+0000 7f0206ffd640 1 -- 172.21.15.86:0/2739216112 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020809ec30 con 0x7f020809d770 2024-09-17T15:32:32.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0206ffd640 1 -- 172.21.15.86:0/2739216112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 msgr2=0x7f020809db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:32.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0206ffd640 1 --2- 172.21.15.86:0/2739216112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f020809db50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01f8009ed0 tx=0x7f01f801c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9120c1f536bad03a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 1 -- 172.21.15.86:0/2739216112 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80a56e0 con 0x7f020809d770 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: dump: 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: _finish_auth 0 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 1 -- 172.21.15.86:0/2739216112 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f80a5880 con 0x7f020809d770 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:32.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 1 -- 172.21.15.86:0/2739216112 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80a5c50 con 0x7f020809d770 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f0205ffb640 10 monclient: dump: 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.897 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 5 monclient: authenticate success, global_id 54145 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 4 set_mon_vals no callback set 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 10 monclient: shutdown 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 -- 172.21.15.86:0/2739216112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 msgr2=0x7f020809db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 --2- 172.21.15.86:0/2739216112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f020809db50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f01f8009ed0 tx=0x7f01f801c990 comp rx=0 tx=0).stop 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 -- 172.21.15.86:0/2739216112 shutdown_connections 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 --2- 172.21.15.86:0/2739216112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 --2- 172.21.15.86:0/2739216112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f020809db50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 -- 172.21.15.86:0/2739216112 >> 172.21.15.86:0/2739216112 conn(0x7f02080a2f40 msgr2=0x7f02080a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 -- 172.21.15.86:0/2739216112 shutdown_connections 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.888+0000 7f020d9f0640 1 -- 172.21.15.86:0/2739216112 wait complete. 2024-09-17T15:32:32.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: build_initial_monmap 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: monmap: 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 Processor -- start 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- start start 2024-09-17T15:32:32.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: init 2024-09-17T15:32:32.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:32.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:32:32.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f02081ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-b con 0x7f020809d770 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208130e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-c con 0x7f020809e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 0x7f0208139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): picked mon.noname-a con 0x7f0208134f80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f020809cce0 con 0x7f0208134f80 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f020809e4c0 con 0x7f020809d770 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f020809ce60 con 0x7f020809e090 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:32.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient(hunting): authenticate will time out at 1285.860853s 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02067fc640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 msgr2=0x7f0208130e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02067fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208130e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 0x7f0208139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 0x7f0208139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35580/0 (socket says 172.21.15.86:35580) 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 learned_addr learned my addr 172.21.15.86:0/3297474863 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): get_auth_request con 0x7f0208134f80 auth_method 0 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:32.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f02081ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient(hunting): handle_auth_done global_id 4824 payload 824 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 msgr2=0x7f0208130e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f020809e090 0x7f0208130e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 msgr2=0x7f02081ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f02081ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient: _finish_hunting 0 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 monclient: found mon.noname-a 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020809d460 con 0x7f0208134f80 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f0206ffd640 10 monclient: get_auth_request con 0x7f020809d770 auth_method 0 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f020809d770 0x7f02081ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 msgr2=0x7f0208139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:32.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02077fe640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 0x7f0208139090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c3c0 tx=0x7f01f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0017070 con 0x7f0208134f80 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: dump: 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.907 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: _finish_auth 0 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: _check_auth_tickets 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f00057d0 con 0x7f0208134f80 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 5 monclient: authenticate success, global_id 4824 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _renew_subs 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020813cbd0 con 0x7f0208134f80 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0005bc0 con 0x7f0208134f80 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:32.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f01fffff640 10 monclient: dump: 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f02057fa640 4 set_mon_vals no callback set 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _renew_subs 2024-09-17T15:32:32.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.892+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f020813c8f0 con 0x7f0208134f80 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f020d9f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f020d9f0640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f02080716f0 con 0x7f02081313d0 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 10 monclient: get_auth_request con 0x7f02081313d0 auth_method 0 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 msgr2=0x7f02081395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 msgr2=0x7f02081395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:32.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f01f001a070 con 0x7f0208134f80 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f01fffff640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01e803bb40 0x7f01e803e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f01f004f330 con 0x7f0208134f80 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f02067fc640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01e803bb40 0x7f01e803e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f02067fc640 10 monclient: get_auth_request con 0x7f01e803bb40 auth_method 0 2024-09-17T15:32:32.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:32.896+0000 7f02067fc640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01e803bb40 0x7f01e803e000 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0200006fd0 tx=0x7f0200006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient: get_auth_request con 0x7f02081313d0 auth_method 0 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 10 monclient(hunting): handle_auth_done global_id 4827 payload 824 2024-09-17T15:32:33.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f020809db50 con 0x7f02081313d0 2024-09-17T15:32:33.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 msgr2=0x7f02081395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.100+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f020809f280 tx=0x7f01f8019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80b7810 con 0x7f02081313d0 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 10 monclient: dump: 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f01f80b7810 con 0x7f02081313d0 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 msgr2=0x7f02081395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f020809f280 tx=0x7f01f8019730 comp rx=0 tx=0).stop 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 msgr2=0x7f02081395d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:33.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.104+0000 7f01fffff640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02081313d0 0x7f02081395d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f020d9f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:33.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f020d9f0640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 0x7f0208072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f020d9f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0208139f20 con 0x7f0208139b10 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 0x7f0208072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient: get_auth_request con 0x7f0208139b10 auth_method 0 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 10 monclient(hunting): handle_auth_done global_id 4830 payload 824 2024-09-17T15:32:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.112+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02080a1a40 con 0x7f0208139b10 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f0206ffd640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 msgr2=0x7f0208072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f0206ffd640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 0x7f0208072160 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f01f801a5c0 tx=0x7f01f8009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f80ca070 con 0x7f0208139b10 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 10 monclient: dump: 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.120 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f01f8004450 con 0x7f0208139b10 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 msgr2=0x7f0208072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 0x7f0208072160 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f01f801a5c0 tx=0x7f01f8009810 comp rx=0 tx=0).stop 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 msgr2=0x7f0208072160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f01fffff640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208139b10 0x7f0208072160 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f01f801a5c0 tx=0x7f01f8009810 comp rx=0 tx=0).stop 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:33.122 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 60, 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 325759, 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.123 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.124 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.125 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:33.126 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.127 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:33.128 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.130 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:33.131 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.132 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.133 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.134 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:33.135 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01e803bb40 msgr2=0x7f01e803e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f01e803bb40 0x7f01e803e000 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0200006fd0 tx=0x7f0200006d40 comp rx=0 tx=0).stop 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 10 monclient: shutdown 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 msgr2=0x7f0208139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 --2- 172.21.15.86:0/3297474863 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0208134f80 0x7f0208139090 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f01f000c3c0 tx=0x7f01f000c880 comp rx=0 tx=0).stop 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 reap_dead start 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f02077fe640 1 -- 172.21.15.86:0/3297474863 reap_dead start 2024-09-17T15:32:33.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 shutdown_connections 2024-09-17T15:32:33.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 >> 172.21.15.86:0/3297474863 conn(0x7f02080a2f40 msgr2=0x7f02080a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:33.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 shutdown_connections 2024-09-17T15:32:33.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.116+0000 7f020d9f0640 1 -- 172.21.15.86:0/3297474863 wait complete. 2024-09-17T15:32:33.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:32:33.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:32:33.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:32:33.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: get_monmap_and_config 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: build_initial_monmap 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: monmap: 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:33.268607+0000 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:33.268607+0000 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:33.268 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:33.269 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 Processor -- start 2024-09-17T15:32:33.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 -- start start 2024-09-17T15:32:33.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: init 2024-09-17T15:32:33.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8eb40a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8eb40a1760 0x7f8eb413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb40a1760 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb413d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb413b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8eb40a2130 con 0x7f8eb40a0c70 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8eb40a2410 con 0x7f8eb40a1760 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb40a26f0 con 0x7f8eb413b450 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb887f640 10 monclient(hunting): authenticate will time out at 1286.234243s 2024-09-17T15:32:33.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb2d9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 msgr2=0x7f8eb413d840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb2d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb413d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8eb40a1760 0x7f8eb413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8eb40a1760 0x7f8eb413af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:59320/0 (socket says 172.21.15.86:59320) 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/819044393 learned_addr learned my addr 172.21.15.86:0/819044393 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): get_auth_request con 0x7f8eb40a1760 auth_method 0 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.264+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb259c640 1 -- 172.21.15.86:0/819044393 >> v1:172.21.15.203:6789/0 conn(0x7f8eb40a0c70 legacy=0x7f8eb40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_done global_id 54148 payload 274 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/819044393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 msgr2=0x7f8eb413d840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 1 --2- 172.21.15.86:0/819044393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb413d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/819044393 >> v1:172.21.15.203:6789/0 conn(0x7f8eb40a0c70 legacy=0x7f8eb40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 1 monclient: found mon.noname-c 2024-09-17T15:32:33.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:33.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:32:33.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.268+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/819044393 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb409f9f0 con 0x7f8eb40a1760 2024-09-17T15:32:33.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/819044393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a1760 msgr2=0x7f8eb413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb1d9b640 1 --2- 172.21.15.86:0/819044393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a1760 0x7f8eb413af10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80098f0 tx=0x7f8ea801c4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=630e015d4cc0a9bb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 1 -- 172.21.15.86:0/819044393 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80a6040 con 0x7f8eb40a1760 2024-09-17T15:32:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:32:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: dump: 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: _finish_auth 0 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: _check_auth_tickets 2024-09-17T15:32:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 5 monclient: authenticate success, global_id 54148 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 1 -- 172.21.15.86:0/819044393 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea80af070 con 0x7f8eb40a1760 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 4 set_mon_vals no callback set 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 1 -- 172.21.15.86:0/819044393 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80aa940 con 0x7f8eb40a1760 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb159a640 10 monclient: dump: 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 10 monclient: shutdown 2024-09-17T15:32:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 -- 172.21.15.86:0/819044393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a1760 msgr2=0x7f8eb413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 --2- 172.21.15.86:0/819044393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a1760 0x7f8eb413af10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80098f0 tx=0x7f8ea801c4c0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 -- 172.21.15.86:0/819044393 shutdown_connections 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 --2- 172.21.15.86:0/819044393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 --2- 172.21.15.86:0/819044393 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a1760 0x7f8eb413af10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 -- 172.21.15.86:0/819044393 >> 172.21.15.86:0/819044393 conn(0x7f8eb40a2f40 msgr2=0x7f8eb40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 -- 172.21.15.86:0/819044393 shutdown_connections 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 1 -- 172.21.15.86:0/819044393 wait complete. 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 10 monclient: build_initial_monmap 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.272+0000 7f8eb887f640 10 monclient: monmap: 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:33.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 Processor -- start 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- start start 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: init 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a0c70 0x7f8eb412ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-b con 0x7f8eb40a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 0x7f8eb412f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-a con 0x7f8eb40a1760 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb41cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): picked mon.noname-c con 0x7f8eb413b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb4140720 con 0x7f8eb40a1760 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41405a0 con 0x7f8eb40a0c70 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8eb41408a0 con 0x7f8eb413b450 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient(hunting): authenticate will time out at 1286.244706s 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb2d9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 msgr2=0x7f8eb41cfcc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 0x7f8eb412f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb2d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb41cfcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 0x7f8eb412f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35620/0 (socket says 172.21.15.86:35620) 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/1999669094 learned_addr learned my addr 172.21.15.86:0/1999669094 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): get_auth_request con 0x7f8eb40a1760 auth_method 0 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb259c640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a0c70 0x7f8eb412ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb259c640 10 monclient(hunting): get_auth_request con 0x7f8eb40a0c70 auth_method 0 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient(hunting): handle_auth_done global_id 4833 payload 824 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 msgr2=0x7f8eb41cfcc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb413b450 0x7f8eb41cfcc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a0c70 msgr2=0x7f8eb412ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb40a0c70 0x7f8eb412ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 monclient: found mon.noname-a 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb4134010 con 0x7f8eb40a1760 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 msgr2=0x7f8eb412f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb1d9b640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 0x7f8eb412f270 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ea801a0f0 tx=0x7f8ea801ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80bd5f0 con 0x7f8eb40a1760 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: dump: 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea80a6030 con 0x7f8eb40a1760 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 5 monclient: authenticate success, global_id 4833 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _renew_subs 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb409d460 con 0x7f8eb40a1760 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80bd9d0 con 0x7f8eb40a1760 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea2ffd640 10 monclient: dump: 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _renew_subs 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8ea3fff640 4 set_mon_vals no callback set 2024-09-17T15:32:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.276+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb4135ac0 con 0x7f8eb40a1760 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb887f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb887f640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 0x7f8eb41d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb40a1050 con 0x7f8eb4136240 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8ea80bdb70 con 0x7f8eb40a1760 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 0x7f8eb41d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient: get_auth_request con 0x7f8eb4136240 auth_method 0 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8ea2ffd640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8e9403bb10 0x7f8e9403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(158..158 src has 1..158) ==== 4859+0+0 (secure 0 0 0) 0x7f8ea80ea4a0 con 0x7f8eb40a1760 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb2d9d640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8e9403bb10 0x7f8e9403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 10 monclient(hunting): handle_auth_done global_id 54154 payload 824 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eb41352e0 con 0x7f8eb4136240 2024-09-17T15:32:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb2d9d640 10 monclient: get_auth_request con 0x7f8e9403bb10 auth_method 0 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 msgr2=0x7f8eb41d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.280+0000 7f8eb259c640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 0x7f8eb41d0200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4001ec0 tx=0x7f8ea400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4019070 con 0x7f8eb4136240 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 10 monclient: dump: 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8eb2d9d640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8e9403bb10 0x7f8e9403dfd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8eac003d10 tx=0x7f8eac0091f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ea40074f0 con 0x7f8eb4136240 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 msgr2=0x7f8eb41d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 0x7f8eb41d0200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4001ec0 tx=0x7f8ea400e960 comp rx=0 tx=0).stop 2024-09-17T15:32:33.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 msgr2=0x7f8eb41d0200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.284+0000 7f8ea2ffd640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb4136240 0x7f8eb41d0200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb887f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb887f640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 0x7f8eb4136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb887f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41d0b50 con 0x7f8eb41d0740 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 0x7f8eb4136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient: get_auth_request con 0x7f8eb41d0740 auth_method 0 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 10 monclient(hunting): handle_auth_done global_id 54157 payload 824 2024-09-17T15:32:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.288+0000 7f8eb259c640 1 -- 172.21.15.86:0/1999669094 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8eb412f7b0 con 0x7f8eb41d0740 2024-09-17T15:32:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb259c640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 msgr2=0x7f8eb4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb259c640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 0x7f8eb4136ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009010 tx=0x7f8ea40014c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4019040 con 0x7f8eb41d0740 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 10 monclient: dump: 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4533 (secure 0 0 0) 0x7f8ea401b620 con 0x7f8eb41d0740 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 msgr2=0x7f8eb4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 0x7f8eb4136ef0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009010 tx=0x7f8ea40014c0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 msgr2=0x7f8eb4136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8ea2ffd640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8eb41d0740 0x7f8eb4136ef0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009010 tx=0x7f8ea40014c0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.301 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 60, 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28727, 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:32:33.302 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:32:33.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:32:33.305 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:32:33.306 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:32:33.308 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:32:33.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:32:33.313 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:32:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8e9403bb10 msgr2=0x7f8e9403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8e9403bb10 0x7f8e9403dfd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8eac003d10 tx=0x7f8eac0091f0 comp rx=0 tx=0).stop 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 10 monclient: shutdown 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 msgr2=0x7f8eb412f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 --2- 172.21.15.86:0/1999669094 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb40a1760 0x7f8eb412f270 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f8ea801a0f0 tx=0x7f8ea801ad00 comp rx=0 tx=0).stop 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb2d9d640 1 -- 172.21.15.86:0/1999669094 reap_dead start 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb2d9d640 1 -- 172.21.15.86:0/1999669094 reap_dead start 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 shutdown_connections 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 >> 172.21.15.86:0/1999669094 conn(0x7f8eb40a2f40 msgr2=0x7f8eb40a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 shutdown_connections 2024-09-17T15:32:33.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:33.292+0000 7f8eb887f640 1 -- 172.21.15.86:0/1999669094 wait complete. 2024-09-17T15:32:33.315 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:32:33.316 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:32:33.490 INFO:tasks.rados.rados.0.smithi203.stdout:679: done (6 left) 2024-09-17T15:32:33.490 INFO:tasks.rados.rados.0.smithi203.stdout:680: read oid 48 snap 68 2024-09-17T15:32:33.490 INFO:tasks.rados.rados.0.smithi203.stdout:680: expect (ObjNum 236 snap 58 seq_num 236) 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:681: read oid 23 snap -1 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:681: expect (ObjNum 260 snap 68 seq_num 260) 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:682: read oid 37 snap 65 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:682: expect (ObjNum 222 snap 56 seq_num 222) 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:683: write (excl) oid 32 current snap is 70 2024-09-17T15:32:33.491 INFO:tasks.rados.rados.0.smithi203.stdout:683: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-17T15:32:33.497 INFO:tasks.rados.rados.0.smithi203.stdout:683: writing smithi20328086-32 from 621925 to 1104108 tid 1 2024-09-17T15:32:33.504 INFO:tasks.rados.rados.0.smithi203.stdout:683: writing smithi20328086-32 from 1559254 to 2125682 tid 2 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:684: snap_create 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:676: finishing copy_from to smithi20328086-49 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:676: got expected ENOENT (src dne) 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:676: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:673: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:674: finishing write tid 1 to smithi20328086-33 2024-09-17T15:32:33.507 INFO:tasks.rados.rados.0.smithi203.stdout:678: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:32:33.528 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2378140 2024-09-17T15:32:33.605 INFO:tasks.rados.rados.0.smithi203.stdout:677: expect (ObjNum 270 snap 70 seq_num 270) 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:673: finishing copy_from to smithi20328086-5 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 119 (ObjNum 205 snap 51 seq_num 205) dirty exists 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: finishing write tid 2 to smithi20328086-33 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: finishing write tid 3 to smithi20328086-33 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: finishing write tid 4 to smithi20328086-33 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: oid 33 updating version 0 to 107 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: oid 33 updating version 107 to 120 2024-09-17T15:32:33.787 INFO:tasks.rados.rados.0.smithi203.stdout:674: oid 33 updating version 120 to 121 2024-09-17T15:32:33.788 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 121 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:32:33.788 INFO:tasks.rados.rados.0.smithi203.stdout:674: left oid 33 (ObjNum 274 snap 70 seq_num 274) 2024-09-17T15:32:33.788 INFO:tasks.rados.rados.0.smithi203.stdout:680: expect (ObjNum 236 snap 58 seq_num 236) 2024-09-17T15:32:33.957 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1480952 2024-09-17T15:32:34.017 INFO:tasks.rados.rados.0.smithi203.stdout:681: expect (ObjNum 260 snap 68 seq_num 260) 2024-09-17T15:32:34.017 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:34.042 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:34.068 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:34.094 INFO:tasks.rados.rados.0.smithi203.stdout:683: finishing write tid 1 to smithi20328086-32 2024-09-17T15:32:34.094 INFO:tasks.rados.rados.0.smithi203.stdout:683: finishing write tid 2 to smithi20328086-32 2024-09-17T15:32:34.094 INFO:tasks.rados.rados.0.smithi203.stdout:683: finishing write tid 3 to smithi20328086-32 2024-09-17T15:32:34.094 INFO:tasks.rados.rados.0.smithi203.stdout:682: expect (ObjNum 222 snap 56 seq_num 222) 2024-09-17T15:32:34.169 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:34.322 INFO:tasks.rados.rados.0.smithi203.stdout:683: finishing write tid 4 to smithi20328086-32 2024-09-17T15:32:34.322 INFO:tasks.rados.rados.0.smithi203.stdout:683: oid 32 updating version 0 to 92 2024-09-17T15:32:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:683: oid 32 version 92 is already newer than 91 2024-09-17T15:32:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:683: oid 32 updating version 92 to 93 2024-09-17T15:32:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:32:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:683: left oid 32 (ObjNum 275 snap 70 seq_num 275) 2024-09-17T15:32:34.497 INFO:tasks.rados.rados.0.smithi203.stdout:673: done (10 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:674: done (9 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:675: done (8 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:676: done (7 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:677: done (6 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:678: done (5 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:680: done (4 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:681: done (3 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:682: done (2 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:683: done (1 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:684: done (0 left) 2024-09-17T15:32:34.498 INFO:tasks.rados.rados.0.smithi203.stdout:685: snap_remove snap 60 2024-09-17T15:32:35.505 INFO:tasks.rados.rados.0.smithi203.stdout:685: done (0 left) 2024-09-17T15:32:35.505 INFO:tasks.rados.rados.0.smithi203.stdout:686: read oid 48 snap 70 2024-09-17T15:32:35.505 INFO:tasks.rados.rados.0.smithi203.stdout:686: expect (ObjNum 262 snap 69 seq_num 262) 2024-09-17T15:32:35.506 INFO:tasks.rados.rados.0.smithi203.stdout:687: read oid 16 snap 62 2024-09-17T15:32:35.506 INFO:tasks.rados.rados.0.smithi203.stdout:687: expect deleted 2024-09-17T15:32:35.506 INFO:tasks.rados.rados.0.smithi203.stdout:688: write oid 7 current snap is 71 2024-09-17T15:32:35.506 INFO:tasks.rados.rados.0.smithi203.stdout:688: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-17T15:32:35.530 INFO:tasks.rados.rados.0.smithi203.stdout:688: writing smithi20328086-7 from 638732 to 1196164 tid 1 2024-09-17T15:32:35.550 INFO:tasks.rados.rados.0.smithi203.stdout:688: writing smithi20328086-7 from 1617736 to 2327822 tid 2 2024-09-17T15:32:35.550 INFO:tasks.rados.rados.0.smithi203.stdout:689: rollback oid 39 current snap is 71 2024-09-17T15:32:35.550 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 39 to 70 2024-09-17T15:32:35.550 INFO:tasks.rados.rados.0.smithi203.stdout:686: expect (ObjNum 262 snap 69 seq_num 262) 2024-09-17T15:32:35.619 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2023773 2024-09-17T15:32:35.648 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:35.707 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 387268 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:688: finishing write tid 1 to smithi20328086-7 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:689: finishing rollback tid 1 to smithi20328086-39 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 162 (ObjNum 289 snap 0 seq_num 3083233184) dirty dne 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:688: finishing write tid 2 to smithi20328086-7 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:688: finishing write tid 3 to smithi20328086-7 2024-09-17T15:32:35.763 INFO:tasks.rados.rados.0.smithi203.stdout:688: finishing write tid 4 to smithi20328086-7 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:688: oid 7 updating version 0 to 129 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:688: oid 7 updating version 129 to 130 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:688: oid 7 updating version 130 to 131 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 131 (ObjNum 276 snap 71 seq_num 276) dirty exists 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:688: left oid 7 (ObjNum 276 snap 71 seq_num 276) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:686: done (3 left) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:687: done (2 left) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:688: done (1 left) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:689: done (0 left) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:690: read oid 28 snap -1 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:690: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:32:35.764 INFO:tasks.rados.rados.0.smithi203.stdout:691: snap_create 2024-09-17T15:32:35.776 INFO:tasks.rados.rados.0.smithi203.stdout:690: expect (ObjNum 175 snap 44 seq_num 175) 2024-09-17T15:32:35.776 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 86329 2024-09-17T15:32:35.915 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:36.513 INFO:tasks.rados.rados.0.smithi203.stdout:690: done (1 left) 2024-09-17T15:32:36.513 INFO:tasks.rados.rados.0.smithi203.stdout:691: done (0 left) 2024-09-17T15:32:36.513 INFO:tasks.rados.rados.0.smithi203.stdout:692: write oid 17 current snap is 72 2024-09-17T15:32:36.514 INFO:tasks.rados.rados.0.smithi203.stdout:692: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-17T15:32:36.534 INFO:tasks.rados.rados.0.smithi203.stdout:692: writing smithi20328086-17 from 655539 to 1288220 tid 1 2024-09-17T15:32:36.557 INFO:tasks.rados.rados.0.smithi203.stdout:692: writing smithi20328086-17 from 2076218 to 2646315 tid 2 2024-09-17T15:32:36.558 INFO:tasks.rados.rados.0.smithi203.stdout:693: write oid 36 current snap is 72 2024-09-17T15:32:36.558 INFO:tasks.rados.rados.0.smithi203.stdout:693: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-17T15:32:36.576 INFO:tasks.rados.rados.0.smithi203.stdout:693: writing smithi20328086-36 from 672346 to 1380276 tid 1 2024-09-17T15:32:36.589 INFO:tasks.rados.rados.0.smithi203.stdout:693: writing smithi20328086-36 from 2134700 to 2672346 tid 2 2024-09-17T15:32:36.589 INFO:tasks.rados.rados.0.smithi203.stdout:694: write oid 44 current snap is 72 2024-09-17T15:32:36.589 INFO:tasks.rados.rados.0.smithi203.stdout:694: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-17T15:32:36.601 INFO:tasks.rados.rados.0.smithi203.stdout:694: writing smithi20328086-44 from 689153 to 1472332 tid 1 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:694: writing smithi20328086-44 from 2076829 to 2534242 tid 2 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:695: rollback oid 20 current snap is 72 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 62 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: finishing write tid 1 to smithi20328086-17 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: finishing write tid 2 to smithi20328086-17 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: finishing write tid 3 to smithi20328086-17 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:696: snap_create 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: finishing write tid 4 to smithi20328086-17 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: oid 17 updating version 0 to 185 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: oid 17 version 185 is already newer than 183 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: oid 17 version 185 is already newer than 184 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 185 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-09-17T15:32:36.611 INFO:tasks.rados.rados.0.smithi203.stdout:692: left oid 17 (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: finishing write tid 1 to smithi20328086-36 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: finishing write tid 2 to smithi20328086-36 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: finishing write tid 3 to smithi20328086-36 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: finishing write tid 4 to smithi20328086-36 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: oid 36 updating version 0 to 293 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: oid 36 version 293 is already newer than 292 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: oid 36 version 293 is already newer than 291 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:693: left oid 36 (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:32:36.612 INFO:tasks.rados.rados.0.smithi203.stdout:694: finishing write tid 1 to smithi20328086-44 2024-09-17T15:32:36.617 INFO:tasks.rados.rados.0.smithi203.stdout:694: finishing write tid 2 to smithi20328086-44 2024-09-17T15:32:36.618 INFO:tasks.rados.rados.0.smithi203.stdout:694: finishing write tid 3 to smithi20328086-44 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:694: finishing write tid 4 to smithi20328086-44 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:694: oid 44 updating version 0 to 187 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:694: oid 44 updating version 187 to 189 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:694: oid 44 version 189 is already newer than 188 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 189 (ObjNum 279 snap 72 seq_num 279) dirty exists 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:694: left oid 44 (ObjNum 279 snap 72 seq_num 279) 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:695: finishing rollback tid 0 to smithi20328086-20 2024-09-17T15:32:36.619 INFO:tasks.rados.rados.0.smithi203.stdout:695: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:32:36.620 INFO:tasks.rados.rados.0.smithi203.stdout:695: finishing rollback tid 2 to smithi20328086-20 2024-09-17T15:32:36.620 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 180 (ObjNum 179 snap 44 seq_num 179) dirty exists 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:692: done (4 left) 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:693: done (3 left) 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:694: done (2 left) 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:695: done (1 left) 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:696: done (0 left) 2024-09-17T15:32:37.444 INFO:tasks.rados.rados.0.smithi203.stdout:697: copy_from oid 34 from oid 39 current snap is 73 2024-09-17T15:32:37.445 INFO:tasks.rados.rados.0.smithi203.stdout:698: write (excl) oid 35 current snap is 73 2024-09-17T15:32:37.445 INFO:tasks.rados.rados.0.smithi203.stdout:698: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-17T15:32:37.454 INFO:tasks.rados.rados.0.smithi203.stdout:698: writing smithi20328086-35 from 705960 to 1164388 tid 1 2024-09-17T15:32:37.462 INFO:tasks.rados.rados.0.smithi203.stdout:698: writing smithi20328086-35 from 1735311 to 2452735 tid 2 2024-09-17T15:32:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:699: write (excl) oid 38 current snap is 73 2024-09-17T15:32:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:699: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-17T15:32:37.469 INFO:tasks.rados.rados.0.smithi203.stdout:699: writing smithi20328086-38 from 722767 to 1256444 tid 1 2024-09-17T15:32:37.475 INFO:tasks.rados.rados.0.smithi203.stdout:699: writing smithi20328086-38 from 1793793 to 2254875 tid 2 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:700: delete oid 15 current snap is 73 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:697: finishing copy_from to smithi20328086-34 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:697: got expected ENOENT (src dne) 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:697: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: finishing write tid 1 to smithi20328086-35 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: finishing write tid 2 to smithi20328086-35 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: finishing write tid 3 to smithi20328086-35 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: finishing write tid 4 to smithi20328086-35 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: oid 35 updating version 0 to 102 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: oid 35 updating version 102 to 104 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:698: oid 35 version 104 is already newer than 103 2024-09-17T15:32:37.476 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 104 (ObjNum 280 snap 73 seq_num 280) dirty exists 2024-09-17T15:32:37.477 INFO:tasks.rados.rados.0.smithi203.stdout:698: left oid 35 (ObjNum 280 snap 73 seq_num 280) 2024-09-17T15:32:37.484 INFO:tasks.rados.rados.0.smithi203.stdout:697: done (3 left) 2024-09-17T15:32:37.484 INFO:tasks.rados.rados.0.smithi203.stdout:698: done (2 left) 2024-09-17T15:32:37.484 INFO:tasks.rados.rados.0.smithi203.stdout:700: done (1 left) 2024-09-17T15:32:37.484 INFO:tasks.rados.rados.0.smithi203.stdout:701: copy_from oid 23 from oid 41 current snap is 73 2024-09-17T15:32:37.484 INFO:tasks.rados.rados.0.smithi203.stdout:702: snap_create 2024-09-17T15:32:37.485 INFO:tasks.rados.rados.0.smithi203.stdout:701: finishing copy_from to smithi20328086-23 2024-09-17T15:32:37.485 INFO:tasks.rados.rados.0.smithi203.stdout:701: got expected ENOENT (src dne) 2024-09-17T15:32:37.485 INFO:tasks.rados.rados.0.smithi203.stdout:701: finishing copy_from racing read to smithi20328086-23 2024-09-17T15:32:37.487 INFO:tasks.rados.rados.0.smithi203.stdout:699: finishing write tid 1 to smithi20328086-38 2024-09-17T15:32:37.488 INFO:tasks.rados.rados.0.smithi203.stdout:699: finishing write tid 2 to smithi20328086-38 2024-09-17T15:32:37.488 INFO:tasks.rados.rados.0.smithi203.stdout:699: finishing write tid 3 to smithi20328086-38 2024-09-17T15:32:37.488 INFO:tasks.rados.rados.0.smithi203.stdout:699: finishing write tid 4 to smithi20328086-38 2024-09-17T15:32:37.488 INFO:tasks.rados.rados.0.smithi203.stdout:699: oid 38 updating version 0 to 145 2024-09-17T15:32:37.488 INFO:tasks.rados.rados.0.smithi203.stdout:699: oid 38 updating version 145 to 147 2024-09-17T15:32:37.489 INFO:tasks.rados.rados.0.smithi203.stdout:699: oid 38 version 147 is already newer than 146 2024-09-17T15:32:37.489 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 147 (ObjNum 281 snap 73 seq_num 281) dirty exists 2024-09-17T15:32:37.489 INFO:tasks.rados.rados.0.smithi203.stdout:699: left oid 38 (ObjNum 281 snap 73 seq_num 281) 2024-09-17T15:32:38.453 INFO:tasks.rados.rados.0.smithi203.stdout:699: done (2 left) 2024-09-17T15:32:38.454 INFO:tasks.rados.rados.0.smithi203.stdout:701: done (1 left) 2024-09-17T15:32:38.454 INFO:tasks.rados.rados.0.smithi203.stdout:702: done (0 left) 2024-09-17T15:32:38.454 INFO:tasks.rados.rados.0.smithi203.stdout:703: snap_remove snap 62 2024-09-17T15:32:39.458 INFO:tasks.rados.rados.0.smithi203.stdout:703: done (0 left) 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:704: rollback oid 50 current snap is 74 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 72 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:705: rollback oid 17 current snap is 74 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 73 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:706: read oid 42 snap -1 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:706: expect (ObjNum 126 snap 26 seq_num 126) 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:707: copy_from oid 30 from oid 18 current snap is 74 2024-09-17T15:32:39.459 INFO:tasks.rados.rados.0.smithi203.stdout:708: delete oid 31 current snap is 74 2024-09-17T15:32:39.472 INFO:tasks.rados.rados.0.smithi203.stdout:706: expect (ObjNum 126 snap 26 seq_num 126) 2024-09-17T15:32:39.482 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 559415 2024-09-17T15:32:39.601 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1850613 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:705: finishing rollback tid 0 to smithi20328086-17 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:705: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:705: finishing rollback tid 2 to smithi20328086-17 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 197 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:704: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:704: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:704: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 97 (ObjNum 255 snap 68 seq_num 255) dirty exists 2024-09-17T15:32:39.604 INFO:tasks.rados.rados.0.smithi203.stdout:707: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:707: finishing copy_from to smithi20328086-30 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 103 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:704: done (4 left) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:705: done (3 left) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:706: done (2 left) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:707: done (1 left) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:708: done (0 left) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:709: read oid 10 snap 68 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:709: expect (ObjNum 162 snap 39 seq_num 162) 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:710: copy_from oid 43 from oid 20 current snap is 74 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:711: copy_from oid 2 from oid 13 current snap is 74 2024-09-17T15:32:39.605 INFO:tasks.rados.rados.0.smithi203.stdout:712: snap_create 2024-09-17T15:32:39.617 INFO:tasks.rados.rados.0.smithi203.stdout:709: expect (ObjNum 162 snap 39 seq_num 162) 2024-09-17T15:32:39.617 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:710: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:710: finishing copy_from to smithi20328086-43 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 180 (ObjNum 179 snap 44 seq_num 179) dirty exists 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:711: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:711: finishing copy_from to smithi20328086-2 2024-09-17T15:32:39.808 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 127 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:709: done (3 left) 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:710: done (2 left) 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:711: done (1 left) 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:712: done (0 left) 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:713: write (excl) oid 29 current snap is 75 2024-09-17T15:32:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:713: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-17T15:32:40.484 INFO:tasks.rados.rados.0.smithi203.stdout:713: writing smithi20328086-29 from 773188 to 1248965 tid 1 2024-09-17T15:32:40.506 INFO:tasks.rados.rados.0.smithi203.stdout:713: writing smithi20328086-29 from 1969239 to 2694001 tid 2 2024-09-17T15:32:40.506 INFO:tasks.rados.rados.0.smithi203.stdout:714: read oid 50 snap -1 2024-09-17T15:32:40.506 INFO:tasks.rados.rados.0.smithi203.stdout:714: expect (ObjNum 255 snap 68 seq_num 255) 2024-09-17T15:32:40.506 INFO:tasks.rados.rados.0.smithi203.stdout:715: delete oid 1 current snap is 75 2024-09-17T15:32:40.506 INFO:tasks.rados.rados.0.smithi203.stdout:713: finishing write tid 1 to smithi20328086-29 2024-09-17T15:32:40.514 INFO:tasks.rados.rados.0.smithi203.stdout:715: done (2 left) 2024-09-17T15:32:40.514 INFO:tasks.rados.rados.0.smithi203.stdout:716: snap_create 2024-09-17T15:32:40.516 INFO:tasks.rados.rados.0.smithi203.stdout:714: expect (ObjNum 255 snap 68 seq_num 255) 2024-09-17T15:32:40.552 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 586366 2024-09-17T15:32:40.608 INFO:tasks.rados.rados.0.smithi203.stdout:713: finishing write tid 2 to smithi20328086-29 2024-09-17T15:32:40.608 INFO:tasks.rados.rados.0.smithi203.stdout:713: finishing write tid 3 to smithi20328086-29 2024-09-17T15:32:40.608 INFO:tasks.rados.rados.0.smithi203.stdout:713: finishing write tid 4 to smithi20328086-29 2024-09-17T15:32:40.608 INFO:tasks.rados.rados.0.smithi203.stdout:713: oid 29 updating version 0 to 174 2024-09-17T15:32:40.609 INFO:tasks.rados.rados.0.smithi203.stdout:713: oid 29 version 174 is already newer than 173 2024-09-17T15:32:40.609 INFO:tasks.rados.rados.0.smithi203.stdout:713: oid 29 updating version 174 to 175 2024-09-17T15:32:40.609 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 175 (ObjNum 284 snap 75 seq_num 284) dirty exists 2024-09-17T15:32:40.609 INFO:tasks.rados.rados.0.smithi203.stdout:713: left oid 29 (ObjNum 284 snap 75 seq_num 284) 2024-09-17T15:32:41.467 INFO:tasks.rados.rados.0.smithi203.stdout:713: done (2 left) 2024-09-17T15:32:41.468 INFO:tasks.rados.rados.0.smithi203.stdout:714: done (1 left) 2024-09-17T15:32:41.468 INFO:tasks.rados.rados.0.smithi203.stdout:716: done (0 left) 2024-09-17T15:32:41.468 INFO:tasks.rados.rados.0.smithi203.stdout:717: read oid 14 snap 70 2024-09-17T15:32:41.468 INFO:tasks.rados.rados.0.smithi203.stdout:717: expect (ObjNum 269 snap 70 seq_num 269) 2024-09-17T15:32:41.468 INFO:tasks.rados.rados.0.smithi203.stdout:718: snap_create 2024-09-17T15:32:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:717: expect (ObjNum 269 snap 70 seq_num 269) 2024-09-17T15:32:41.551 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:41.604 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:42.445 INFO:tasks.rados.rados.0.smithi203.stdout:717: done (1 left) 2024-09-17T15:32:42.445 INFO:tasks.rados.rados.0.smithi203.stdout:718: done (0 left) 2024-09-17T15:32:42.445 INFO:tasks.rados.rados.0.smithi203.stdout:719: read oid 27 snap -1 2024-09-17T15:32:42.445 INFO:tasks.rados.rados.0.smithi203.stdout:719: expect (ObjNum 237 snap 58 seq_num 237) 2024-09-17T15:32:42.445 INFO:tasks.rados.rados.0.smithi203.stdout:720: snap_remove snap 65 2024-09-17T15:32:43.449 INFO:tasks.rados.rados.0.smithi203.stdout:720: done (1 left) 2024-09-17T15:32:43.449 INFO:tasks.rados.rados.0.smithi203.stdout:721: copy_from oid 7 from oid 14 current snap is 77 2024-09-17T15:32:43.450 INFO:tasks.rados.rados.0.smithi203.stdout:719: expect (ObjNum 237 snap 58 seq_num 237) 2024-09-17T15:32:43.532 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3783213 2024-09-17T15:32:43.688 INFO:tasks.rados.rados.0.smithi203.stdout:721: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:32:43.688 INFO:tasks.rados.rados.0.smithi203.stdout:721: finishing copy_from to smithi20328086-7 2024-09-17T15:32:43.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 156 (ObjNum 269 snap 70 seq_num 269) dirty exists 2024-09-17T15:32:43.688 INFO:tasks.rados.rados.0.smithi203.stdout:719: done (1 left) 2024-09-17T15:32:43.688 INFO:tasks.rados.rados.0.smithi203.stdout:721: done (0 left) 2024-09-17T15:32:43.689 INFO:tasks.rados.rados.0.smithi203.stdout:722: delete oid 19 current snap is 77 2024-09-17T15:32:43.690 INFO:tasks.rados.rados.0.smithi203.stdout:722: done (0 left) 2024-09-17T15:32:43.690 INFO:tasks.rados.rados.0.smithi203.stdout:723: delete oid 41 current snap is 77 2024-09-17T15:32:43.693 INFO:tasks.rados.rados.0.smithi203.stdout:723: done (0 left) 2024-09-17T15:32:43.693 INFO:tasks.rados.rados.0.smithi203.stdout:724: write (excl) oid 15 current snap is 77 2024-09-17T15:32:43.693 INFO:tasks.rados.rados.0.smithi203.stdout:724: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-17T15:32:43.713 INFO:tasks.rados.rados.0.smithi203.stdout:724: writing smithi20328086-15 from 440416 to 1217189 tid 1 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:724: writing smithi20328086-15 from 1686814 to 2418914 tid 2 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:725: rollback oid 44 current snap is 77 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 75 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:724: finishing write tid 1 to smithi20328086-15 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:726: read oid 39 snap 75 2024-09-17T15:32:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:726: expect deleted 2024-09-17T15:32:43.734 INFO:tasks.rados.rados.0.smithi203.stdout:727: read oid 43 snap -1 2024-09-17T15:32:43.734 INFO:tasks.rados.rados.0.smithi203.stdout:727: expect (ObjNum 179 snap 44 seq_num 179) 2024-09-17T15:32:43.734 INFO:tasks.rados.rados.0.smithi203.stdout:728: write (excl) oid 45 current snap is 77 2024-09-17T15:32:43.734 INFO:tasks.rados.rados.0.smithi203.stdout:728: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-17T15:32:43.740 INFO:tasks.rados.rados.0.smithi203.stdout:728: writing smithi20328086-45 from 457223 to 1025598 tid 1 2024-09-17T15:32:43.746 INFO:tasks.rados.rados.0.smithi203.stdout:728: writing smithi20328086-45 from 1461649 to 1937407 tid 2 2024-09-17T15:32:43.751 INFO:tasks.rados.rados.0.smithi203.stdout:728: writing smithi20328086-45 from 2420186 to 2857223 tid 3 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:729: read oid 7 snap -1 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:729: expect (ObjNum 269 snap 70 seq_num 269) 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:730: read oid 35 snap -1 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:730: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:731: write (excl) oid 6 current snap is 77 2024-09-17T15:32:43.752 INFO:tasks.rados.rados.0.smithi203.stdout:731: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-17T15:32:43.758 INFO:tasks.rados.rados.0.smithi203.stdout:731: writing smithi20328086-6 from 474030 to 1117654 tid 1 2024-09-17T15:32:43.768 INFO:tasks.rados.rados.0.smithi203.stdout:731: writing smithi20328086-6 from 1520131 to 2255900 tid 2 2024-09-17T15:32:43.768 INFO:tasks.rados.rados.0.smithi203.stdout:727: expect (ObjNum 179 snap 44 seq_num 179) 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: finishing write tid 2 to smithi20328086-15 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: finishing write tid 3 to smithi20328086-15 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: finishing write tid 4 to smithi20328086-15 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: oid 15 updating version 0 to 110 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: oid 15 updating version 110 to 112 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: oid 15 version 112 is already newer than 111 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 112 (ObjNum 288 snap 77 seq_num 288) dirty exists 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:724: left oid 15 (ObjNum 288 snap 77 seq_num 288) 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:725: finishing rollback tid 0 to smithi20328086-44 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:725: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:32:43.872 INFO:tasks.rados.rados.0.smithi203.stdout:732: read oid 31 snap -1 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:732: expect deleted 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:724: done (8 left) 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:726: done (7 left) 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:727: done (6 left) 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:733: write (excl) oid 43 current snap is 77 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:725: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 207 (ObjNum 279 snap 72 seq_num 279) dirty exists 2024-09-17T15:32:43.873 INFO:tasks.rados.rados.0.smithi203.stdout:733: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-17T15:32:43.878 INFO:tasks.rados.rados.0.smithi203.stdout:733: writing smithi20328086-43 from 490837 to 1209710 tid 1 2024-09-17T15:32:43.885 INFO:tasks.rados.rados.0.smithi203.stdout:733: writing smithi20328086-43 from 1862260 to 2341687 tid 2 2024-09-17T15:32:43.888 INFO:tasks.rados.rados.0.smithi203.stdout:733: writing smithi20328086-43 from 2875032 to 2890837 tid 3 2024-09-17T15:32:43.888 INFO:tasks.rados.rados.0.smithi203.stdout:725: done (6 left) 2024-09-17T15:32:43.888 INFO:tasks.rados.rados.0.smithi203.stdout:734: snap_create 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: finishing write tid 1 to smithi20328086-45 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: finishing write tid 2 to smithi20328086-45 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: finishing write tid 3 to smithi20328086-45 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: finishing write tid 4 to smithi20328086-45 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: finishing write tid 5 to smithi20328086-45 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: oid 45 updating version 0 to 82 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: oid 45 version 82 is already newer than 80 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: oid 45 version 82 is already newer than 81 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: oid 45 version 82 is already newer than 79 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 82 (ObjNum 289 snap 77 seq_num 289) dirty exists 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:728: left oid 45 (ObjNum 289 snap 77 seq_num 289) 2024-09-17T15:32:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:729: expect (ObjNum 269 snap 70 seq_num 269) 2024-09-17T15:32:43.994 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 685678 2024-09-17T15:32:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:730: expect (ObjNum 280 snap 73 seq_num 280) 2024-09-17T15:32:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:44.116 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: finishing write tid 1 to smithi20328086-6 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: finishing write tid 2 to smithi20328086-6 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: finishing write tid 3 to smithi20328086-6 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: finishing write tid 4 to smithi20328086-6 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: oid 6 updating version 0 to 199 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: oid 6 updating version 199 to 200 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: oid 6 updating version 200 to 201 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 201 (ObjNum 290 snap 77 seq_num 290) dirty exists 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:731: left oid 6 (ObjNum 290 snap 77 seq_num 290) 2024-09-17T15:32:44.159 INFO:tasks.rados.rados.0.smithi203.stdout:733: finishing write tid 1 to smithi20328086-43 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: finishing write tid 2 to smithi20328086-43 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: finishing write tid 3 to smithi20328086-43 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: finishing write tid 4 to smithi20328086-43 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: finishing write tid 5 to smithi20328086-43 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: oid 43 updating version 0 to 184 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: oid 43 version 184 is already newer than 182 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: oid 43 version 184 is already newer than 181 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: oid 43 version 184 is already newer than 183 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 184 (ObjNum 291 snap 77 seq_num 291) dirty exists 2024-09-17T15:32:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:733: left oid 43 (ObjNum 291 snap 77 seq_num 291) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:728: done (6 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:729: done (5 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:730: done (4 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:731: done (3 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:732: done (2 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:733: done (1 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:734: done (0 left) 2024-09-17T15:32:44.460 INFO:tasks.rados.rados.0.smithi203.stdout:735: delete oid 20 current snap is 78 2024-09-17T15:32:44.469 INFO:tasks.rados.rados.0.smithi203.stdout:735: done (0 left) 2024-09-17T15:32:44.469 INFO:tasks.rados.rados.0.smithi203.stdout:736: snap_remove snap 76 2024-09-17T15:32:45.470 INFO:tasks.rados.rados.0.smithi203.stdout:736: done (0 left) 2024-09-17T15:32:45.470 INFO:tasks.rados.rados.0.smithi203.stdout:737: snap_create 2024-09-17T15:32:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:737: done (0 left) 2024-09-17T15:32:46.472 INFO:tasks.rados.rados.0.smithi203.stdout:738: snap_create 2024-09-17T15:32:47.449 INFO:tasks.rados.rados.0.smithi203.stdout:738: done (0 left) 2024-09-17T15:32:47.449 INFO:tasks.rados.rados.0.smithi203.stdout:739: write oid 7 current snap is 80 2024-09-17T15:32:47.449 INFO:tasks.rados.rados.0.smithi203.stdout:739: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-17T15:32:47.465 INFO:tasks.rados.rados.0.smithi203.stdout:739: writing smithi20328086-7 from 524451 to 993822 tid 1 2024-09-17T15:32:47.484 INFO:tasks.rados.rados.0.smithi203.stdout:739: writing smithi20328086-7 from 1579224 to 2062320 tid 2 2024-09-17T15:32:47.498 INFO:tasks.rados.rados.0.smithi203.stdout:739: writing smithi20328086-7 from 2529878 to 2924451 tid 3 2024-09-17T15:32:47.498 INFO:tasks.rados.rados.0.smithi203.stdout:740: read oid 43 snap -1 2024-09-17T15:32:47.498 INFO:tasks.rados.rados.0.smithi203.stdout:740: expect (ObjNum 291 snap 77 seq_num 291) 2024-09-17T15:32:47.498 INFO:tasks.rados.rados.0.smithi203.stdout:741: snap_remove snap 70 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:741: done (2 left) 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:742: copy_from oid 16 from oid 25 current snap is 80 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: finishing write tid 1 to smithi20328086-7 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: finishing write tid 2 to smithi20328086-7 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: finishing write tid 3 to smithi20328086-7 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: finishing write tid 4 to smithi20328086-7 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: finishing write tid 5 to smithi20328086-7 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: oid 7 updating version 0 to 159 2024-09-17T15:32:48.490 INFO:tasks.rados.rados.0.smithi203.stdout:739: oid 7 version 159 is already newer than 158 2024-09-17T15:32:48.491 INFO:tasks.rados.rados.0.smithi203.stdout:739: oid 7 updating version 159 to 160 2024-09-17T15:32:48.491 INFO:tasks.rados.rados.0.smithi203.stdout:739: oid 7 version 160 is already newer than 157 2024-09-17T15:32:48.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 160 (ObjNum 293 snap 80 seq_num 293) dirty exists 2024-09-17T15:32:48.491 INFO:tasks.rados.rados.0.smithi203.stdout:739: left oid 7 (ObjNum 293 snap 80 seq_num 293) 2024-09-17T15:32:48.491 INFO:tasks.rados.rados.0.smithi203.stdout:740: expect (ObjNum 291 snap 77 seq_num 291) 2024-09-17T15:32:48.587 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1039712 2024-09-17T15:32:48.668 INFO:tasks.rados.rados.0.smithi203.stdout:742: finishing copy_from racing read to smithi20328086-16 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:742: finishing copy_from to smithi20328086-16 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 172 (ObjNum 265 snap 70 seq_num 265) dirty exists 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:739: done (2 left) 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:740: done (1 left) 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:742: done (0 left) 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:743: read oid 31 snap -1 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:743: expect deleted 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:744: rollback oid 11 current snap is 80 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 77 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:745: read oid 30 snap -1 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:745: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:746: rollback oid 33 current snap is 80 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 79 2024-09-17T15:32:48.669 INFO:tasks.rados.rados.0.smithi203.stdout:747: snap_create 2024-09-17T15:32:48.675 INFO:tasks.rados.rados.0.smithi203.stdout:744: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:32:48.678 INFO:tasks.rados.rados.0.smithi203.stdout:744: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:32:48.679 INFO:tasks.rados.rados.0.smithi203.stdout:746: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:32:48.679 INFO:tasks.rados.rados.0.smithi203.stdout:745: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:32:48.752 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:48.876 INFO:tasks.rados.rados.0.smithi203.stdout:744: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:32:48.876 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 217 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:32:48.877 INFO:tasks.rados.rados.0.smithi203.stdout:746: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:32:48.877 INFO:tasks.rados.rados.0.smithi203.stdout:746: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:32:48.877 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 124 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:743: done (4 left) 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:744: done (3 left) 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:745: done (2 left) 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:746: done (1 left) 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:747: done (0 left) 2024-09-17T15:32:49.503 INFO:tasks.rados.rados.0.smithi203.stdout:748: delete oid 2 current snap is 81 2024-09-17T15:32:49.513 INFO:tasks.rados.rados.0.smithi203.stdout:748: done (0 left) 2024-09-17T15:32:49.513 INFO:tasks.rados.rados.0.smithi203.stdout:749: rollback oid 30 current snap is 81 2024-09-17T15:32:49.513 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 30 to 77 2024-09-17T15:32:49.513 INFO:tasks.rados.rados.0.smithi203.stdout:750: read oid 37 snap -1 2024-09-17T15:32:49.514 INFO:tasks.rados.rados.0.smithi203.stdout:750: expect (ObjNum 271 snap 70 seq_num 271) 2024-09-17T15:32:49.514 INFO:tasks.rados.rados.0.smithi203.stdout:751: snap_remove snap 78 2024-09-17T15:32:50.510 INFO:tasks.rados.rados.0.smithi203.stdout:751: done (2 left) 2024-09-17T15:32:50.510 INFO:tasks.rados.rados.0.smithi203.stdout:752: rollback oid 7 current snap is 81 2024-09-17T15:32:50.510 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 80 2024-09-17T15:32:50.510 INFO:tasks.rados.rados.0.smithi203.stdout:750: expect (ObjNum 271 snap 70 seq_num 271) 2024-09-17T15:32:50.695 INFO:tasks.rados.rados.0.smithi203.stdout:749: finishing rollback tid 0 to smithi20328086-30 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:749: finishing rollback tid 1 to smithi20328086-30 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:749: finishing rollback tid 2 to smithi20328086-30 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 120 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:749: done (2 left) 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:750: done (1 left) 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:753: read oid 47 snap -1 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:753: expect (ObjNum 266 snap 70 seq_num 266) 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:754: rollback oid 9 current snap is 81 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 73 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:752: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:752: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:32:50.696 INFO:tasks.rados.rados.0.smithi203.stdout:752: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 164 (ObjNum 293 snap 80 seq_num 293) dirty exists 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:752: done (2 left) 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:755: read oid 17 snap -1 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:755: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:756: copy_from oid 28 from oid 39 current snap is 81 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:757: read oid 11 snap -1 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:757: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:758: rollback oid 27 current snap is 81 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 68 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:759: rollback oid 3 current snap is 81 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 77 2024-09-17T15:32:50.697 INFO:tasks.rados.rados.0.smithi203.stdout:760: snap_create 2024-09-17T15:32:50.698 INFO:tasks.rados.rados.0.smithi203.stdout:756: finishing copy_from to smithi20328086-28 2024-09-17T15:32:50.698 INFO:tasks.rados.rados.0.smithi203.stdout:756: got expected ENOENT (src dne) 2024-09-17T15:32:50.698 INFO:tasks.rados.rados.0.smithi203.stdout:756: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:32:50.711 INFO:tasks.rados.rados.0.smithi203.stdout:754: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:32:50.712 INFO:tasks.rados.rados.0.smithi203.stdout:754: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:32:50.713 INFO:tasks.rados.rados.0.smithi203.stdout:754: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:32:50.713 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 178 (ObjNum 135 snap 29 seq_num 135) dirty exists 2024-09-17T15:32:50.717 INFO:tasks.rados.rados.0.smithi203.stdout:753: expect (ObjNum 266 snap 70 seq_num 266) 2024-09-17T15:32:50.717 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:50.768 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:50.818 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:50.867 INFO:tasks.rados.rados.0.smithi203.stdout:758: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:32:50.867 INFO:tasks.rados.rados.0.smithi203.stdout:758: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:32:50.867 INFO:tasks.rados.rados.0.smithi203.stdout:755: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:32:50.967 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:758: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 191 (ObjNum 237 snap 58 seq_num 237) dirty exists 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:759: finishing rollback tid 0 to smithi20328086-3 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:759: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:759: finishing rollback tid 2 to smithi20328086-3 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 195 (ObjNum 245 snap 63 seq_num 245) dirty exists 2024-09-17T15:32:51.023 INFO:tasks.rados.rados.0.smithi203.stdout:757: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:32:51.060 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 461428 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:753: done (7 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:754: done (6 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:755: done (5 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:756: done (4 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:757: done (3 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:758: done (2 left) 2024-09-17T15:32:51.511 INFO:tasks.rados.rados.0.smithi203.stdout:759: done (1 left) 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:760: done (0 left) 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:761: read oid 50 snap -1 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:761: expect (ObjNum 255 snap 68 seq_num 255) 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:762: copy_from oid 19 from oid 32 current snap is 82 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:763: read oid 44 snap 75 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:763: expect (ObjNum 279 snap 72 seq_num 279) 2024-09-17T15:32:51.512 INFO:tasks.rados.rados.0.smithi203.stdout:764: snap_remove snap 71 2024-09-17T15:32:52.449 INFO:tasks.rados.rados.0.smithi203.stdout:764: done (3 left) 2024-09-17T15:32:52.450 INFO:tasks.rados.rados.0.smithi203.stdout:765: delete oid 49 current snap is 82 2024-09-17T15:32:52.450 INFO:tasks.rados.rados.0.smithi203.stdout:761: expect (ObjNum 255 snap 68 seq_num 255) 2024-09-17T15:32:52.478 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:52.504 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:52.527 INFO:tasks.rados.rados.0.smithi203.stdout:763: expect (ObjNum 279 snap 72 seq_num 279) 2024-09-17T15:32:52.649 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2595844 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:762: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:762: finishing copy_from to smithi20328086-19 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:761: done (3 left) 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:762: done (2 left) 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:763: done (1 left) 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:765: done (0 left) 2024-09-17T15:32:52.727 INFO:tasks.rados.rados.0.smithi203.stdout:766: snap_create 2024-09-17T15:32:53.316 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:32:53.317 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:32:53.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:32:53.320 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:32:53.320 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:32:53.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: get_monmap_and_config 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: build_initial_monmap 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: monmap: 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:53.453583+0000 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:53.453583+0000 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 Processor -- start 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 -- start start 2024-09-17T15:32:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: init 2024-09-17T15:32:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-17T15:32:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31bc0689d0 0x7f31bc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31bc068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31bc13aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc09c8b0 con 0x7f31bc068db0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc09cb90 con 0x7f31bc13aff0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f31bc074500 con 0x7f31bc0689d0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c2c44640 10 monclient(hunting): authenticate will time out at 1306.418817s 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31bc0689d0 0x7f31bc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31bc0689d0 0x7f31bc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33060/0 (socket says 172.21.15.86:33060) 2024-09-17T15:32:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31bbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f31bc068db0 0x7f31bc069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49942/0 (socket says 172.21.15.86:49942) 2024-09-17T15:32:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 1 -- 172.21.15.86:0/2504609580 learned_addr learned my addr 172.21.15.86:0/2504609580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 10 monclient(hunting): get_auth_request con 0x7f31bc0689d0 auth_method 0 2024-09-17T15:32:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:53.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.451+0000 7f31c09e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31bbfff640 1 -- 172.21.15.86:0/2504609580 >> v1:172.21.15.86:6789/0 conn(0x7f31bc068db0 legacy=0x7f31bc069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient(hunting): handle_auth_done global_id 4836 payload 274 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 1 -- 172.21.15.86:0/2504609580 >> v1:172.21.15.203:6789/0 conn(0x7f31bc13aff0 legacy=0x7f31bc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 1 -- 172.21.15.86:0/2504609580 >> v1:172.21.15.86:6789/0 conn(0x7f31bc068db0 legacy=0x7f31bc069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient: _finish_hunting 0 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 1 monclient: found mon.noname-a 2024-09-17T15:32:53.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:53.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.455+0000 7f31c09e0640 1 -- 172.21.15.86:0/2504609580 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc09ce70 con 0x7f31bc0689d0 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c09e0640 1 -- 172.21.15.86:0/2504609580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc0689d0 msgr2=0x7f31bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c09e0640 1 --2- 172.21.15.86:0/2504609580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc0689d0 0x7f31bc06f840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0098f0 tx=0x7f31ac01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d510900a10822bee server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 1 -- 172.21.15.86:0/2504609580 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc09c2d0 con 0x7f31bc0689d0 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: dump: 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 1 -- 172.21.15.86:0/2504609580 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31ac0af070 con 0x7f31bc0689d0 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 5 monclient: authenticate success, global_id 4836 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 4 set_mon_vals no callback set 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 1 -- 172.21.15.86:0/2504609580 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0aa920 con 0x7f31bc0689d0 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31bb7fe640 10 monclient: dump: 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:53.464 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: shutdown 2024-09-17T15:32:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- 172.21.15.86:0/2504609580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc0689d0 msgr2=0x7f31bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 --2- 172.21.15.86:0/2504609580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc0689d0 0x7f31bc06f840 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0098f0 tx=0x7f31ac01c4a0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- 172.21.15.86:0/2504609580 shutdown_connections 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 --2- 172.21.15.86:0/2504609580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc0689d0 0x7f31bc06f840 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- 172.21.15.86:0/2504609580 >> 172.21.15.86:0/2504609580 conn(0x7f31bc0a6440 msgr2=0x7f31bc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- 172.21.15.86:0/2504609580 shutdown_connections 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- 172.21.15.86:0/2504609580 wait complete. 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: build_initial_monmap 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: monmap: 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 Processor -- start 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 -- start start 2024-09-17T15:32:53.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: init 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 0x7f31bc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-b con 0x7f31bc0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.459+0000 7f31c2c44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 0x7f31bc135b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-a con 0x7f31bc068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): picked mon.noname-c con 0x7f31bc13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc141c00 con 0x7f31bc068db0 2024-09-17T15:32:53.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc141a80 con 0x7f31bc0689d0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f31bc141d80 con 0x7f31bc13aff0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient(hunting): authenticate will time out at 1306.428040s 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 0x7f31bc135b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56220/0 (socket says 172.21.15.86:56220) 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 -- 172.21.15.86:0/814818857 learned_addr learned my addr 172.21.15.86:0/814818857 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 10 monclient(hunting): get_auth_request con 0x7f31bc13aff0 auth_method 0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): get_auth_request con 0x7f31bc068db0 auth_method 0 2024-09-17T15:32:53.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 msgr2=0x7f31bc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 msgr2=0x7f31bc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c11e1640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 0x7f31bc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 10 monclient(hunting): get_auth_request con 0x7f31bc0689d0 auth_method 0 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:53.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient(hunting): handle_auth_done global_id 4839 payload 824 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 msgr2=0x7f31bc136080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 msgr2=0x7f31bc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 0x7f31bc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 monclient: found mon.noname-a 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 -- 172.21.15.86:0/814818857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc13e000 con 0x7f31bc068db0 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c09e0640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 0x7f31bc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 msgr2=0x7f31bc135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31bbfff640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 0x7f31bc135b40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c430 tx=0x7f31b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00047f0 con 0x7f31bc068db0 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: dump: 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: _finish_auth 0 2024-09-17T15:32:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b00049b0 con 0x7f31bc068db0 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b0004da0 con 0x7f31bc068db0 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31b97fa640 10 monclient: dump: 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:53.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 5 monclient: authenticate success, global_id 4839 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient: _renew_subs 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31bc09d460 con 0x7f31bc068db0 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31baffd640 4 set_mon_vals no callback set 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient: _renew_subs 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.463+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31bc121500 con 0x7f31bc068db0 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c2c44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c2c44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c2c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31bc06f840 con 0x7f31bc068db0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f31b00123f0 con 0x7f31bc068db0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31b97fa640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 0x7f31a803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(178..178 src has 1..178) ==== 4827+0+0 (secure 0 0 0) 0x7f31b004f4b0 con 0x7f31bc068db0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c09e0640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 0x7f31a803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.467+0000 7f31c09e0640 10 monclient: get_auth_request con 0x7f31a803bb90 auth_method 0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.471+0000 7f31c09e0640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 0x7f31a803e050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0ab000 tx=0x7f31ac0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.471+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f31b00113a0 con 0x7f31bc068db0 2024-09-17T15:32:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.471+0000 7f31b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.471+0000 7f31b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:53.528 INFO:tasks.rados.rados.0.smithi203.stdout:766: done (0 left) 2024-09-17T15:32:53.528 INFO:tasks.rados.rados.0.smithi203.stdout:767: snap_remove snap 72 2024-09-17T15:32:53.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f31bc06d4f0 con 0x7f31bc068db0 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31b97fa640 1 -- 172.21.15.86:0/814818857 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f31b0010300 con 0x7f31bc068db0 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:53.709 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 msgr2=0x7f31a803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 0x7f31a803e050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0ab000 tx=0x7f31ac0a7000 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 10 monclient: shutdown 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 msgr2=0x7f31bc135b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 0x7f31bc135b40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f31b000c430 tx=0x7f31b000c8f0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 shutdown_connections 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a803bb90 0x7f31a803e050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31bc13aff0 0x7f31bc136080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31bc068db0 0x7f31bc135b40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 --2- 172.21.15.86:0/814818857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31bc0689d0 0x7f31bc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:53.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 >> 172.21.15.86:0/814818857 conn(0x7f31bc0a6440 msgr2=0x7f31bc13c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:53.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 shutdown_connections 2024-09-17T15:32:53.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:53.707+0000 7f31c2c44640 1 -- 172.21.15.86:0/814818857 wait complete. 2024-09-17T15:32:54.536 INFO:tasks.rados.rados.0.smithi203.stdout:767: done (0 left) 2024-09-17T15:32:54.537 INFO:tasks.rados.rados.0.smithi203.stdout:768: snap_remove snap 80 2024-09-17T15:32:55.538 INFO:tasks.rados.rados.0.smithi203.stdout:768: done (0 left) 2024-09-17T15:32:55.538 INFO:tasks.rados.rados.0.smithi203.stdout:769: snap_remove snap 81 2024-09-17T15:32:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:769: done (0 left) 2024-09-17T15:32:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:770: rollback oid 1 current snap is 83 2024-09-17T15:32:56.541 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 73 2024-09-17T15:32:56.542 INFO:tasks.rados.rados.0.smithi203.stdout:771: snap_remove snap 75 2024-09-17T15:32:56.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: get_monmap_and_config 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: build_initial_monmap 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: monmap: 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:32:56.853712+0000 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:32:56.853712+0000 2024-09-17T15:32:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:32:56.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 Processor -- start 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 -- start start 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: init 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bd00689d0 0x7f1bd006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd00689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bd0068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1bd013aff0 0x7f1bd013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd009c8b0 con 0x7f1bd0068db0 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd0074500 con 0x7f1bd00689d0 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:56.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd009cb90 con 0x7f1bd013aff0 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd8481640 10 monclient(hunting): authenticate will time out at 1309.819440s 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bd00689d0 0x7f1bd006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bd00689d0 0x7f1bd006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33118/0 (socket says 172.21.15.86:33118) 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 -- 172.21.15.86:0/368068309 learned_addr learned my addr 172.21.15.86:0/368068309 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): get_auth_request con 0x7f1bd00689d0 auth_method 0 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient(hunting): handle_auth_done global_id 4842 payload 274 2024-09-17T15:32:56.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 -- 172.21.15.86:0/368068309 >> v2:172.21.15.203:3300/0 conn(0x7f1bd013aff0 msgr2=0x7f1bd013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 --2- 172.21.15.86:0/368068309 >> v2:172.21.15.203:3300/0 conn(0x7f1bd013aff0 0x7f1bd013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 -- 172.21.15.86:0/368068309 >> v1:172.21.15.203:6789/0 conn(0x7f1bd0068db0 legacy=0x7f1bd0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient: _finish_hunting 0 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 monclient: found mon.noname-a 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:56.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.851+0000 7f1bd621d640 1 -- 172.21.15.86:0/368068309 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd009ce70 con 0x7f1bd00689d0 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 -- 172.21.15.86:0/368068309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd00689d0 msgr2=0x7f1bd006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 --2- 172.21.15.86:0/368068309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd00689d0 0x7f1bd006f840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0069a70 tx=0x7f1bcc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcbd23b5d809026 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 1 -- 172.21.15.86:0/368068309 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0b1070 con 0x7f1bd00689d0 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: dump: 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: _finish_auth 0 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: _check_auth_tickets 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 1 -- 172.21.15.86:0/368068309 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bcc0ac620 con 0x7f1bd00689d0 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 5 monclient: authenticate success, global_id 4842 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:32:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: get_monmap_and_config success 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 1 -- 172.21.15.86:0/368068309 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bd009c2d0 con 0x7f1bd00689d0 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 4 set_mon_vals no callback set 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd521b640 10 monclient: dump: 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: shutdown 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- 172.21.15.86:0/368068309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd00689d0 msgr2=0x7f1bd006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- 172.21.15.86:0/368068309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd00689d0 0x7f1bd006f840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1bd0069a70 tx=0x7f1bcc01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- 172.21.15.86:0/368068309 shutdown_connections 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- 172.21.15.86:0/368068309 >> v2:172.21.15.203:3300/0 conn(0x7f1bd013aff0 0x7f1bd013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- 172.21.15.86:0/368068309 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd00689d0 0x7f1bd006f840 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- 172.21.15.86:0/368068309 >> 172.21.15.86:0/368068309 conn(0x7f1bd00a6440 msgr2=0x7f1bd00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- 172.21.15.86:0/368068309 shutdown_connections 2024-09-17T15:32:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- 172.21.15.86:0/368068309 wait complete. 2024-09-17T15:32:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: build_initial_monmap 2024-09-17T15:32:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: monmap: 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 Processor -- start 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- start start 2024-09-17T15:32:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: init 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: _reopen_session rank -1 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-c con 0x7f1bd00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-a con 0x7f1bd0068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd013aff0 0x7f1bd0071b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): picked mon.noname-b con 0x7f1bd013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01405f0 con 0x7f1bd0068db0 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd0140470 con 0x7f1bd013aff0 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): start opening mon connection 2024-09-17T15:32:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd0140770 con 0x7f1bd00689d0 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): _renew_subs 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd8481640 10 monclient(hunting): authenticate will time out at 1309.826526s 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33134/0 (socket says 172.21.15.86:33134) 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 -- 172.21.15.86:0/3555530326 learned_addr learned my addr 172.21.15.86:0/3555530326 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 10 monclient(hunting): get_auth_request con 0x7f1bd00689d0 auth_method 0 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): get_auth_request con 0x7f1bd0068db0 auth_method 0 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:56.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 msgr2=0x7f1bd0135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 msgr2=0x7f1bd0135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd621d640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd6a1e640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd013aff0 0x7f1bd0071b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd6a1e640 10 monclient(hunting): get_auth_request con 0x7f1bd013aff0 auth_method 0 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd6a1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd6a1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd6a1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient(hunting): handle_auth_done global_id 4845 payload 824 2024-09-17T15:32:56.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 msgr2=0x7f1bd0135d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd013aff0 msgr2=0x7f1bd0071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd013aff0 0x7f1bd0071b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient: _finish_hunting 0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 monclient: found mon.noname-a 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.859+0000 7f1bd5a1c640 1 -- 172.21.15.86:0/3555530326 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd0073640 con 0x7f1bd0068db0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd5a1c640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 msgr2=0x7f1bd01362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd5a1c640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c490 tx=0x7f1bc0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0017070 con 0x7f1bd0068db0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:32:56.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: dump: 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:32:56.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 5 monclient: authenticate success, global_id 4845 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc0003820 con 0x7f1bd0068db0 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0003c10 con 0x7f1bd0068db0 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _renew_subs 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd013d580 con 0x7f1bd0068db0 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd4a1a640 4 set_mon_vals no callback set 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 10 monclient: dump: 2024-09-17T15:32:56.874 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _renew_subs 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd0122710 con 0x7f1bd0068db0 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:56.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd006f840 con 0x7f1bd0068db0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1bc001a070 con 0x7f1bd0068db0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 0x7f1bb403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 4799+0+0 (secure 0 0 0) 0x7f1bc004f420 con 0x7f1bd0068db0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd621d640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 0x7f1bb403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.863+0000 7f1bd621d640 10 monclient: get_auth_request con 0x7f1bb403bb60 auth_method 0 2024-09-17T15:32:56.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.867+0000 7f1bd621d640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 0x7f1bb403e020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006fdb0 tx=0x7f1bcc0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:32:56.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.867+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1bc001bde0 con 0x7f1bd0068db0 2024-09-17T15:32:56.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.867+0000 7f1bc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:32:56.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:56.867+0000 7f1bc6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:32:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:32:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:32:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1bd0121f30 con 0x7f1bd0068db0 2024-09-17T15:32:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bc6ffd640 1 -- 172.21.15.86:0/3555530326 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1bc004f080 con 0x7f1bd0068db0 2024-09-17T15:32:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:32:57.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bc6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:32:57.101 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":60,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 msgr2=0x7f1bb403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 0x7f1bb403e020 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1bd006fdb0 tx=0x7f1bcc0a7000 comp rx=0 tx=0).stop 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 10 monclient: shutdown 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 msgr2=0x7f1bd01362a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.099+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c490 tx=0x7f1bc0001e80 comp rx=0 tx=0).stop 2024-09-17T15:32:57.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 shutdown_connections 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1bb403bb60 0x7f1bb403e020 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bd013aff0 0x7f1bd0071b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd0068db0 0x7f1bd01362a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 --2- 172.21.15.86:0/3555530326 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd00689d0 0x7f1bd0135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 >> 172.21.15.86:0/3555530326 conn(0x7f1bd00a6440 msgr2=0x7f1bd009d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 shutdown_connections 2024-09-17T15:32:57.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:32:57.103+0000 7f1bd8481640 1 -- 172.21.15.86:0/3555530326 wait complete. 2024-09-17T15:32:57.452 INFO:tasks.rados.rados.0.smithi203.stdout:771: done (1 left) 2024-09-17T15:32:57.452 INFO:tasks.rados.rados.0.smithi203.stdout:772: delete oid 33 current snap is 83 2024-09-17T15:32:57.452 INFO:tasks.rados.rados.0.smithi203.stdout:770: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:32:57.452 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:32:57.464 INFO:tasks.rados.rados.0.smithi203.stdout:770: done (1 left) 2024-09-17T15:32:57.464 INFO:tasks.rados.rados.0.smithi203.stdout:772: done (0 left) 2024-09-17T15:32:57.464 INFO:tasks.rados.rados.0.smithi203.stdout:773: write (excl) oid 20 current snap is 83 2024-09-17T15:32:57.464 INFO:tasks.rados.rados.0.smithi203.stdout:773: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-17T15:32:57.482 INFO:tasks.rados.rados.0.smithi203.stdout:773: writing smithi20328086-20 from 591679 to 1078399 tid 1 2024-09-17T15:32:57.503 INFO:tasks.rados.rados.0.smithi203.stdout:773: writing smithi20328086-20 from 1813152 to 2303586 tid 2 2024-09-17T15:32:57.503 INFO:tasks.rados.rados.0.smithi203.stdout:774: snap_create 2024-09-17T15:32:57.503 INFO:tasks.rados.rados.0.smithi203.stdout:773: finishing write tid 1 to smithi20328086-20 2024-09-17T15:32:57.507 INFO:tasks.rados.rados.0.smithi203.stdout:773: finishing write tid 2 to smithi20328086-20 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: finishing write tid 3 to smithi20328086-20 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: finishing write tid 4 to smithi20328086-20 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: oid 20 updating version 0 to 224 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: oid 20 updating version 224 to 226 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: oid 20 version 226 is already newer than 225 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 226 (ObjNum 297 snap 83 seq_num 297) dirty exists 2024-09-17T15:32:57.508 INFO:tasks.rados.rados.0.smithi203.stdout:773: left oid 20 (ObjNum 297 snap 83 seq_num 297) 2024-09-17T15:32:58.578 INFO:tasks.rados.rados.0.smithi203.stdout:773: done (1 left) 2024-09-17T15:32:58.578 INFO:tasks.rados.rados.0.smithi203.stdout:774: done (0 left) 2024-09-17T15:32:58.578 INFO:tasks.rados.rados.0.smithi203.stdout:775: write oid 45 current snap is 84 2024-09-17T15:32:58.578 INFO:tasks.rados.rados.0.smithi203.stdout:775: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-17T15:32:58.597 INFO:tasks.rados.rados.0.smithi203.stdout:775: writing smithi20328086-45 from 608486 to 1170455 tid 1 2024-09-17T15:32:58.619 INFO:tasks.rados.rados.0.smithi203.stdout:775: writing smithi20328086-45 from 1871634 to 2505726 tid 2 2024-09-17T15:32:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:775: writing smithi20328086-45 from 2950640 to 3008486 tid 3 2024-09-17T15:32:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:776: write oid 49 current snap is 84 2024-09-17T15:32:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:775: finishing write tid 1 to smithi20328086-45 2024-09-17T15:32:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:775: finishing write tid 2 to smithi20328086-45 2024-09-17T15:32:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:776: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-17T15:32:58.645 INFO:tasks.rados.rados.0.smithi203.stdout:776: writing smithi20328086-49 from 625293 to 1262511 tid 1 2024-09-17T15:32:58.659 INFO:tasks.rados.rados.0.smithi203.stdout:776: writing smithi20328086-49 from 1813763 to 2307866 tid 2 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:776: writing smithi20328086-49 from 2978063 to 3025293 tid 3 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:777: copy_from oid 21 from oid 39 current snap is 84 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: finishing write tid 3 to smithi20328086-45 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: finishing write tid 4 to smithi20328086-45 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:778: delete oid 23 current snap is 84 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: finishing write tid 5 to smithi20328086-45 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: oid 45 updating version 0 to 95 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: oid 45 updating version 95 to 96 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: oid 45 version 96 is already newer than 94 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:775: oid 45 updating version 96 to 97 2024-09-17T15:32:58.668 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 97 (ObjNum 298 snap 84 seq_num 298) dirty exists 2024-09-17T15:32:58.669 INFO:tasks.rados.rados.0.smithi203.stdout:775: left oid 45 (ObjNum 298 snap 84 seq_num 298) 2024-09-17T15:32:58.669 INFO:tasks.rados.rados.0.smithi203.stdout:776: finishing write tid 1 to smithi20328086-49 2024-09-17T15:32:58.669 INFO:tasks.rados.rados.0.smithi203.stdout:776: finishing write tid 2 to smithi20328086-49 2024-09-17T15:32:58.672 INFO:tasks.rados.rados.0.smithi203.stdout:776: finishing write tid 3 to smithi20328086-49 2024-09-17T15:32:58.672 INFO:tasks.rados.rados.0.smithi203.stdout:777: finishing copy_from to smithi20328086-21 2024-09-17T15:32:58.672 INFO:tasks.rados.rados.0.smithi203.stdout:777: got expected ENOENT (src dne) 2024-09-17T15:32:58.677 INFO:tasks.rados.rados.0.smithi203.stdout:775: done (3 left) 2024-09-17T15:32:58.677 INFO:tasks.rados.rados.0.smithi203.stdout:778: done (2 left) 2024-09-17T15:32:58.678 INFO:tasks.rados.rados.0.smithi203.stdout:779: delete oid 28 current snap is 84 2024-09-17T15:32:58.678 INFO:tasks.rados.rados.0.smithi203.stdout:776: finishing write tid 4 to smithi20328086-49 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: finishing write tid 5 to smithi20328086-49 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: oid 49 updating version 0 to 324 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: oid 49 updating version 324 to 325 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: oid 49 updating version 325 to 326 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: oid 49 updating version 326 to 327 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 327 (ObjNum 299 snap 84 seq_num 299) dirty exists 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:776: left oid 49 (ObjNum 299 snap 84 seq_num 299) 2024-09-17T15:32:58.686 INFO:tasks.rados.rados.0.smithi203.stdout:777: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:32:58.690 INFO:tasks.rados.rados.0.smithi203.stdout:776: done (2 left) 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:777: done (1 left) 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:779: done (0 left) 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:780: read oid 8 snap 79 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:780: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:781: read oid 26 snap -1 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:781: expect deleted 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:782: read oid 16 snap 82 2024-09-17T15:32:58.691 INFO:tasks.rados.rados.0.smithi203.stdout:782: expect (ObjNum 265 snap 70 seq_num 265) 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:783: read oid 3 snap 83 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:783: expect (ObjNum 245 snap 63 seq_num 245) 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:784: read oid 5 snap -1 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:784: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:785: write oid 47 current snap is 84 2024-09-17T15:32:58.692 INFO:tasks.rados.rados.0.smithi203.stdout:785: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-17T15:32:58.705 INFO:tasks.rados.rados.0.smithi203.stdout:785: writing smithi20328086-47 from 675714 to 1138679 tid 1 2024-09-17T15:32:58.717 INFO:tasks.rados.rados.0.smithi203.stdout:785: writing smithi20328086-47 from 1589209 to 2230639 tid 2 2024-09-17T15:32:58.725 INFO:tasks.rados.rados.0.smithi203.stdout:785: writing smithi20328086-47 from 2943979 to 3075714 tid 3 2024-09-17T15:32:58.725 INFO:tasks.rados.rados.0.smithi203.stdout:786: write oid 4 current snap is 84 2024-09-17T15:32:58.725 INFO:tasks.rados.rados.0.smithi203.stdout:786: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-17T15:32:58.736 INFO:tasks.rados.rados.0.smithi203.stdout:786: writing smithi20328086-4 from 692521 to 1230735 tid 1 2024-09-17T15:32:58.748 INFO:tasks.rados.rados.0.smithi203.stdout:786: writing smithi20328086-4 from 1931338 to 2716426 tid 2 2024-09-17T15:32:58.748 INFO:tasks.rados.rados.0.smithi203.stdout:787: write oid 15 current snap is 84 2024-09-17T15:32:58.748 INFO:tasks.rados.rados.0.smithi203.stdout:787: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-17T15:32:58.758 INFO:tasks.rados.rados.0.smithi203.stdout:787: writing smithi20328086-15 from 709328 to 1322791 tid 1 2024-09-17T15:32:58.766 INFO:tasks.rados.rados.0.smithi203.stdout:787: writing smithi20328086-15 from 1989820 to 2634919 tid 2 2024-09-17T15:32:58.766 INFO:tasks.rados.rados.0.smithi203.stdout:788: copy_from oid 40 from oid 32 current snap is 84 2024-09-17T15:32:58.766 INFO:tasks.rados.rados.0.smithi203.stdout:782: expect (ObjNum 265 snap 70 seq_num 265) 2024-09-17T15:32:58.807 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2386488 2024-09-17T15:32:58.808 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:58.861 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 820669 2024-09-17T15:32:58.880 INFO:tasks.rados.rados.0.smithi203.stdout:780: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:32:58.881 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:58.978 INFO:tasks.rados.rados.0.smithi203.stdout:783: expect (ObjNum 245 snap 63 seq_num 245) 2024-09-17T15:32:58.982 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 769102 2024-09-17T15:32:59.031 INFO:tasks.rados.rados.0.smithi203.stdout:784: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:32:59.032 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:59.069 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:59.106 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: finishing write tid 1 to smithi20328086-47 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: finishing write tid 2 to smithi20328086-47 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: finishing write tid 3 to smithi20328086-47 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: finishing write tid 4 to smithi20328086-47 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: finishing write tid 5 to smithi20328086-47 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: oid 47 updating version 0 to 157 2024-09-17T15:32:59.144 INFO:tasks.rados.rados.0.smithi203.stdout:785: oid 47 updating version 157 to 159 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:785: oid 47 version 159 is already newer than 158 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:785: oid 47 updating version 159 to 160 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 160 (ObjNum 302 snap 84 seq_num 302) dirty exists 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:785: left oid 47 (ObjNum 302 snap 84 seq_num 302) 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: finishing write tid 1 to smithi20328086-4 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: finishing write tid 2 to smithi20328086-4 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: finishing write tid 3 to smithi20328086-4 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: finishing write tid 4 to smithi20328086-4 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: oid 4 updating version 0 to 99 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: oid 4 updating version 99 to 100 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: oid 4 updating version 100 to 101 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 101 (ObjNum 303 snap 84 seq_num 303) dirty exists 2024-09-17T15:32:59.145 INFO:tasks.rados.rados.0.smithi203.stdout:786: left oid 4 (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:780: done (8 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:781: done (7 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:782: done (6 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:783: done (5 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:784: done (4 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:785: done (3 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:786: done (2 left) 2024-09-17T15:32:59.146 INFO:tasks.rados.rados.0.smithi203.stdout:789: snap_remove snap 77 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:789: done (2 left) 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:790: delete oid 16 current snap is 84 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: finishing write tid 1 to smithi20328086-15 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: finishing write tid 2 to smithi20328086-15 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: finishing write tid 3 to smithi20328086-15 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: finishing write tid 4 to smithi20328086-15 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: oid 15 updating version 0 to 138 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: oid 15 version 138 is already newer than 137 2024-09-17T15:32:59.581 INFO:tasks.rados.rados.0.smithi203.stdout:787: oid 15 version 138 is already newer than 136 2024-09-17T15:32:59.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 138 (ObjNum 304 snap 84 seq_num 304) dirty exists 2024-09-17T15:32:59.582 INFO:tasks.rados.rados.0.smithi203.stdout:787: left oid 15 (ObjNum 304 snap 84 seq_num 304) 2024-09-17T15:32:59.582 INFO:tasks.rados.rados.0.smithi203.stdout:788: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:32:59.582 INFO:tasks.rados.rados.0.smithi203.stdout:788: finishing copy_from to smithi20328086-40 2024-09-17T15:32:59.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:32:59.592 INFO:tasks.rados.rados.0.smithi203.stdout:787: done (2 left) 2024-09-17T15:32:59.592 INFO:tasks.rados.rados.0.smithi203.stdout:788: done (1 left) 2024-09-17T15:32:59.592 INFO:tasks.rados.rados.0.smithi203.stdout:790: done (0 left) 2024-09-17T15:32:59.592 INFO:tasks.rados.rados.0.smithi203.stdout:791: read oid 41 snap -1 2024-09-17T15:32:59.592 INFO:tasks.rados.rados.0.smithi203.stdout:791: expect deleted 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:792: rollback oid 49 current snap is 84 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 82 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:793: read oid 43 snap 68 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:793: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:794: read oid 29 snap -1 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:794: expect (ObjNum 284 snap 75 seq_num 284) 2024-09-17T15:32:59.593 INFO:tasks.rados.rados.0.smithi203.stdout:795: snap_remove snap 73 2024-09-17T15:33:00.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: get_monmap_and_config 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: build_initial_monmap 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: monmap: 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:00.245799+0000 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:00.245799+0000 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:00.246 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 Processor -- start 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 -- start start 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: init 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 0x7f2f3806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-b con 0x7f2f380689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f38068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f3813aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f38074470 con 0x7f2f3813aff0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3809c8b0 con 0x7f2f38068db0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3809cbe0 con 0x7f2f380689d0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f40777640 10 monclient(hunting): authenticate will time out at 1313.211282s 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3ed14640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2f3813aff0 0x7f2f3813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:57378/0 (socket says 172.21.15.86:57378) 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3ed14640 1 -- 172.21.15.86:0/2120081207 learned_addr learned my addr 172.21.15.86:0/2120081207 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 1 --2- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 0x7f2f3806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): get_auth_request con 0x7f2f380689d0 auth_method 0 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient(hunting): handle_auth_done global_id 64100 payload 274 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 1 -- 172.21.15.86:0/2120081207 >> v1:172.21.15.203:6789/0 conn(0x7f2f38068db0 legacy=0x7f2f380691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 1 -- 172.21.15.86:0/2120081207 >> v1:172.21.15.86:6789/0 conn(0x7f2f3813aff0 legacy=0x7f2f3813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient: _finish_hunting 0 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 1 monclient: found mon.noname-b 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.243+0000 7f2f3e513640 1 -- 172.21.15.86:0/2120081207 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3809ce70 con 0x7f2f380689d0 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 1 -- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 msgr2=0x7f2f3806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 1 --2- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 0x7f2f3806f840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f280097f0 tx=0x7f2f2801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83e3508bf1ef3f4c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 1 -- 172.21.15.86:0/2120081207 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f280b0070 con 0x7f2f380689d0 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: dump: 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: _finish_auth 0 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: _check_auth_tickets 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 1 -- 172.21.15.86:0/2120081207 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f280ab600 con 0x7f2f380689d0 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:00.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 1 -- 172.21.15.86:0/2120081207 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3809c2d0 con 0x7f2f380689d0 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 5 monclient: authenticate success, global_id 64100 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 4 set_mon_vals no callback set 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3d511640 10 monclient: dump: 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.254 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: shutdown 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 msgr2=0x7f2f3806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 --2- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 0x7f2f3806f840 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f280097f0 tx=0x7f2f2801c3a0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- 172.21.15.86:0/2120081207 shutdown_connections 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 --2- 172.21.15.86:0/2120081207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f380689d0 0x7f2f3806f840 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- 172.21.15.86:0/2120081207 >> 172.21.15.86:0/2120081207 conn(0x7f2f380a6440 msgr2=0x7f2f380a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- 172.21.15.86:0/2120081207 shutdown_connections 2024-09-17T15:33:00.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- 172.21.15.86:0/2120081207 wait complete. 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: build_initial_monmap 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: monmap: 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:00.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 Processor -- start 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- start start 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: init 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f380689d0 0x7f2f38135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-b con 0x7f2f380689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-a con 0x7f2f38068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 0x7f2f3811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38141ca0 con 0x7f2f38068db0 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f38141b20 con 0x7f2f380689d0 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f38141e20 con 0x7f2f3813aff0 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f40777640 10 monclient(hunting): authenticate will time out at 1313.218737s 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55284/0 (socket says 172.21.15.86:55284) 2024-09-17T15:33:00.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 0x7f2f3811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 1 -- 172.21.15.86:0/2540927120 learned_addr learned my addr 172.21.15.86:0/2540927120 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): get_auth_request con 0x7f2f38068db0 auth_method 0 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): get_auth_request con 0x7f2f3813aff0 auth_method 0 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f380689d0 0x7f2f38135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 10 monclient(hunting): get_auth_request con 0x7f2f380689d0 auth_method 0 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3e513640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient(hunting): handle_auth_done global_id 64103 payload 824 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 -- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f380689d0 msgr2=0x7f2f38135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f380689d0 0x7f2f38135630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 -- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 msgr2=0x7f2f38135b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient: _finish_hunting 0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 monclient: found mon.noname-c 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3ed14640 1 -- 172.21.15.86:0/2540927120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f38073700 con 0x7f2f3813aff0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.251+0000 7f2f3dd12640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f3ed14640 1 -- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 msgr2=0x7f2f3811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f3ed14640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 0x7f2f3811c8c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000cb20 tx=0x7f2f3000cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30017070 con 0x7f2f3813aff0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: dump: 2024-09-17T15:33:00.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: _finish_auth 0 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:00.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f3000ade0 con 0x7f2f3813aff0 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f30012640 con 0x7f2f3813aff0 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 10 monclient: dump: 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2ffff640 4 set_mon_vals no callback set 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 5 monclient: authenticate success, global_id 64103 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _renew_subs 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3809d460 con 0x7f2f3813aff0 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _renew_subs 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f381360b0 con 0x7f2f3813aff0 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2f3001a070 con 0x7f2f3813aff0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f3806f840 con 0x7f2f3813aff0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 0x7f2f1c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(185..185 src has 1..185) ==== 4815+0+0 (secure 0 0 0) 0x7f2f3004fc90 con 0x7f2f3813aff0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f3e513640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 0x7f2f1c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.255+0000 7f2f3e513640 10 monclient: get_auth_request con 0x7f2f1c03bbb0 auth_method 0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.259+0000 7f2f3e513640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 0x7f2f1c03e070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f280ab000 tx=0x7f2f28008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.259+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2f30056080 con 0x7f2f3813aff0 2024-09-17T15:33:00.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.259+0000 7f2f2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:00.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.259+0000 7f2f2effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:00.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f40777640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:00.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f40777640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:00.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f40777640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2f3806d4f0 con 0x7f2f3813aff0 2024-09-17T15:33:00.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f2effd640 1 -- 172.21.15.86:0/2540927120 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f2f3004e670 con 0x7f2f3813aff0 2024-09-17T15:33:00.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:00.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.495+0000 7f2f2effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:00.498 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:00.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 msgr2=0x7f2f1c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 0x7f2f1c03e070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f280ab000 tx=0x7f2f28008040 comp rx=0 tx=0).stop 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 10 monclient: shutdown 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 msgr2=0x7f2f3811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 0x7f2f3811c8c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2f3000cb20 tx=0x7f2f3000cf20 comp rx=0 tx=0).stop 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 shutdown_connections 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bbb0 0x7f2f1c03e070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3813aff0 0x7f2f3811c8c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f38068db0 0x7f2f38135b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 --2- 172.21.15.86:0/2540927120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f380689d0 0x7f2f38135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 >> 172.21.15.86:0/2540927120 conn(0x7f2f380a6440 msgr2=0x7f2f3813c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:00.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 shutdown_connections 2024-09-17T15:33:00.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.499+0000 7f2f40777640 1 -- 172.21.15.86:0/2540927120 wait complete. 2024-09-17T15:33:00.516 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:33:00.516 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:33:00.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:33:00.606 INFO:tasks.rados.rados.0.smithi203.stdout:795: done (4 left) 2024-09-17T15:33:00.606 INFO:tasks.rados.rados.0.smithi203.stdout:796: snap_remove snap 74 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: get_monmap_and_config 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: build_initial_monmap 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: monmap: 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.645 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:00.646177+0000 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:00.646177+0000 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:00.646 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 Processor -- start 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 -- start start 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: init 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f9c0a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9c0a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c13f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9c13d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9c0a68d0 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7f9c0a64a0 con 0x7f7f9c0a5350 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9c09c230 con 0x7f7f9c13d180 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7fa187b640 10 monclient(hunting): authenticate will time out at 1313.611412s 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c13f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55302/0 (socket says 172.21.15.86:55302) 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 1 -- 172.21.15.86:0/513328087 learned_addr learned my addr 172.21.15.86:0/513328087 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f9c13d180 auth_method 0 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): get_auth_request con 0x7f7f9c0a5350 auth_method 0 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_done global_id 64106 payload 274 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 -- 172.21.15.86:0/513328087 >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 msgr2=0x7f7f9c0a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 --2- 172.21.15.86:0/513328087 >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 -- 172.21.15.86:0/513328087 >> v1:172.21.15.203:6789/0 conn(0x7f7f9c0a4f70 legacy=0x7f7f9c09cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 monclient: found mon.noname-b 2024-09-17T15:33:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:00.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9b7fe640 1 -- 172.21.15.86:0/513328087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9c09c510 con 0x7f7f9c13d180 2024-09-17T15:33:00.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.643+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/513328087 >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:00.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 1 -- 172.21.15.86:0/513328087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 msgr2=0x7f7f9c13f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 1 --2- 172.21.15.86:0/513328087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c13f570 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f940097c0 tx=0x7f7f9401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d099a854eda611f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 1 -- 172.21.15.86:0/513328087 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f940b0070 con 0x7f7f9c13d180 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: dump: 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.660 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 1 -- 172.21.15.86:0/513328087 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f940ab600 con 0x7f7f9c13d180 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 1 -- 172.21.15.86:0/513328087 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f940ab9d0 con 0x7f7f9c13d180 2024-09-17T15:33:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f99ffb640 10 monclient: dump: 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.662 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 5 monclient: authenticate success, global_id 64106 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 4 set_mon_vals no callback set 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: shutdown 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- 172.21.15.86:0/513328087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 msgr2=0x7f7f9c13f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- 172.21.15.86:0/513328087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c13f570 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7f940097c0 tx=0x7f7f9401c360 comp rx=0 tx=0).stop 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- 172.21.15.86:0/513328087 shutdown_connections 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- 172.21.15.86:0/513328087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c13f570 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- 172.21.15.86:0/513328087 >> v2:172.21.15.86:3300/0 conn(0x7f7f9c0a5350 0x7f7f9c0a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- 172.21.15.86:0/513328087 >> 172.21.15.86:0/513328087 conn(0x7f7f9c0a7240 msgr2=0x7f7f9c0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- 172.21.15.86:0/513328087 shutdown_connections 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- 172.21.15.86:0/513328087 wait complete. 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: build_initial_monmap 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: monmap: 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:00.664 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 Processor -- start 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- start start 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: init 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9c0a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9c0a5350 0x7f7f9c132760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9c0a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c1cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9c13d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1426f0 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c142570 con 0x7f7f9c0a5350 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9c142870 con 0x7f7f9c13d180 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:00.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7fa187b640 10 monclient(hunting): authenticate will time out at 1313.619351s 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c1cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55306/0 (socket says 172.21.15.86:55306) 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 -- 172.21.15.86:0/28651354 learned_addr learned my addr 172.21.15.86:0/28651354 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c1cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47660/0 (socket says 172.21.15.86:47660) 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): get_auth_request con 0x7f7f9c0a4f70 auth_method 0 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f7f9c13d180 auth_method 0 2024-09-17T15:33:00.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient(hunting): handle_auth_done global_id 4854 payload 824 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 msgr2=0x7f7f9c1cfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f9c13d180 0x7f7f9c1cfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9c0a5350 msgr2=0x7f7f9c132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9c0a5350 0x7f7f9c132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f9c0a5350 0x7f7f9c132760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient: _finish_hunting 0 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 monclient: found mon.noname-a 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.651+0000 7f7f9affd640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9c09f2f0 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9affd640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 msgr2=0x7f7f9c132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9affd640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c450 tx=0x7f7f9000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90012a40 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: dump: 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.670 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: _finish_auth 0 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f90017070 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 5 monclient: authenticate success, global_id 4854 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 monclient: _renew_subs 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f9c09fd90 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:00.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f90012e40 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 10 monclient: dump: 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.672 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f997fa640 4 set_mon_vals no callback set 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 monclient: _renew_subs 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9c09ee30 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 0x7f7f9c1d0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c0a0290 con 0x7f7f9c0a2dd0 2024-09-17T15:33:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 0x7f7f9c1d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient: get_auth_request con 0x7f7f9c0a2dd0 auth_method 0 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7f9001c460 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_done global_id 4857 payload 824 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9a7fc640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f9c0a5750 con 0x7f7f9c0a2dd0 2024-09-17T15:33:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7f8803bb60 0x7f7f8803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 4815+0+0 (secure 0 0 0) 0x7f7f9004fda0 con 0x7f7f9c0a4f70 2024-09-17T15:33:00.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9b7fe640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7f8803bb60 0x7f7f8803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.655+0000 7f7f9b7fe640 10 monclient: get_auth_request con 0x7f7f8803bb60 auth_method 0 2024-09-17T15:33:00.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f9a7fc640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 msgr2=0x7f7f9c1d0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 0x7f7f9c1d0220 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c01c070 con 0x7f7f9c0a2dd0 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 10 monclient: dump: 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.678 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f8c017640 con 0x7f7f9c0a2dd0 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 msgr2=0x7f7f9c1d0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 0x7f7f9c1d0220 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c00f4a0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 msgr2=0x7f7f9c1d0220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f83fff640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a2dd0 0x7f7f9c1d0220 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c00f4a0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.659+0000 7f7f9b7fe640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7f8803bb60 0x7f7f8803e020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ab000 tx=0x7f7f940a4f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7fa187b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7fa187b640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 0x7f7f9c1361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7fa187b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9c1d0b70 con 0x7f7f9c1d0760 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 0x7f7f9c1361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient: get_auth_request con 0x7f7f9c1d0760 auth_method 0 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.663+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f9a7fc640 10 monclient(hunting): handle_auth_done global_id 4860 payload 824 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f9a7fc640 1 -- 172.21.15.86:0/28651354 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f9c09e510 con 0x7f7f9c1d0760 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f9a7fc640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 msgr2=0x7f7f9c1361b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f9a7fc640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 0x7f7f9c1361b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c01c050 con 0x7f7f9c1d0760 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 10 monclient: dump: 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:00.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f7f8c01d040 con 0x7f7f9c1d0760 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 msgr2=0x7f7f9c1361b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 0x7f7f9c1361b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c004a50 comp rx=0 tx=0).stop 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 msgr2=0x7f7f9c1361b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f83fff640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c1d0760 0x7f7f9c1361b0 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c09fa10 tx=0x7f7f8c004a50 comp rx=0 tx=0).stop 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:33:00.683 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 353311, 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:00.684 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:00.685 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:00.686 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:00.687 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.689 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:00.691 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:00.693 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:00.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:00.695 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:00.696 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.697 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:00.698 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:00.699 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7f8803bb60 msgr2=0x7f7f8803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7f8803bb60 0x7f7f8803e020 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ab000 tx=0x7f7f940a4f90 comp rx=0 tx=0).stop 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 10 monclient: shutdown 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 msgr2=0x7f7f9c132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c450 tx=0x7f7f9000c910 comp rx=0 tx=0).stop 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7f9b7fe640 1 -- 172.21.15.86:0/28651354 reap_dead start 2024-09-17T15:33:00.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 shutdown_connections 2024-09-17T15:33:00.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 --2- 172.21.15.86:0/28651354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f9c0a4f70 0x7f7f9c132220 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 >> 172.21.15.86:0/28651354 conn(0x7f7f9c0a7240 msgr2=0x7f7f9c13c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:00.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 shutdown_connections 2024-09-17T15:33:00.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.667+0000 7f7fa187b640 1 -- 172.21.15.86:0/28651354 wait complete. 2024-09-17T15:33:00.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:00.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:33:00.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:33:00.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: get_monmap_and_config 2024-09-17T15:33:00.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: build_initial_monmap 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: monmap: 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:00.820353+0000 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:00.820353+0000 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:00.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:00.821 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:00.821 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:00.821 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:00.821 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:00.821 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:00.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 Processor -- start 2024-09-17T15:33:00.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 -- start start 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: init 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-b con 0x7f06080a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06080a1590 0x7f0608135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-a con 0x7f06080a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0608135f90 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06080a2130 con 0x7f0608135f90 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f06080a2410 con 0x7f06080a1590 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f06080a26f0 con 0x7f06080a0c70 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060eb2c640 10 monclient(hunting): authenticate will time out at 1313.785791s 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06080a1590 0x7f0608135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06080a1590 0x7f0608135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55328/0 (socket says 172.21.15.86:55328) 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 1 -- 172.21.15.86:0/3470586201 learned_addr learned my addr 172.21.15.86:0/3470586201 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): get_auth_request con 0x7f06080a1590 auth_method 0 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): get_auth_request con 0x7f06080a0c70 auth_method 0 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient(hunting): handle_auth_done global_id 64112 payload 274 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 >> v2:172.21.15.86:3300/0 conn(0x7f06080a1590 msgr2=0x7f0608135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> v2:172.21.15.86:3300/0 conn(0x7f06080a1590 0x7f0608135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 >> v1:172.21.15.86:6789/0 conn(0x7f0608135f90 legacy=0x7f060813cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient: _finish_hunting 0 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 monclient: found mon.noname-b 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:00.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.819+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060809f9f0 con 0x7f06080a0c70 2024-09-17T15:33:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.823+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 msgr2=0x7f06080a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:33:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.823+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 msgr2=0x7f06080a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.823+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f05f00097c0 tx=0x7f05f001c360 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:00.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:00.823+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:01.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f060c8c8640 10 monclient: get_auth_request con 0x7f06080a0c70 auth_method 0 2024-09-17T15:33:01.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f060c8c8640 0 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T15:33:01.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f060c8c8640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 msgr2=0x7f06080a1050 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06080a0c70 0x7f06080a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient: _add_conns ranks=[1,4] 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05f4003510 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 1 --2- 172.21.15.86:0/3470586201 >> v2:172.21.15.203:3300/0 conn(0x7f05f40074a0 0x7f05f4009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient(hunting): picked mon.noname-c con 0x7f05f40074a0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05f400a270 con 0x7f05f4003510 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f05f400b470 con 0x7f05f40074a0 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f06077fe640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f0607fff640 1 --2- 172.21.15.86:0/3470586201 >> v2:172.21.15.203:3300/0 conn(0x7f05f40074a0 0x7f05f4009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f0607fff640 10 monclient(hunting): get_auth_request con 0x7f05f40074a0 auth_method 0 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f0607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f0607fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.027+0000 7f0607fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f060c8c8640 1 -- 172.21.15.86:0/3470586201 >> v1:172.21.15.203:6789/0 conn(0x7f05f4003510 legacy=0x7f05f4005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient(hunting): handle_auth_done global_id 64112 payload 293 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 1 -- 172.21.15.86:0/3470586201 >> v1:172.21.15.203:6789/0 conn(0x7f05f4003510 legacy=0x7f05f4005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 1 monclient: found mon.noname-c 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:33:01.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 1 -- 172.21.15.86:0/3470586201 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f400b730 con 0x7f05f40074a0 2024-09-17T15:33:01.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 1 -- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 msgr2=0x7f05f4009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.031+0000 7f0607fff640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 0x7f05f4009960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f0009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=191a6fb648cf09a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f0607fff640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 0x7f05f4009960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f0009190 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f05f8017070 mon_map magic: 0 2024-09-17T15:33:01.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f8017070 con 0x7f05f40074a0 2024-09-17T15:33:01.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: dump: 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f800ee50 con 0x7f05f40074a0 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 5 monclient: authenticate success, global_id 64112 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:01.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 4 set_mon_vals no callback set 2024-09-17T15:33:01.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: shutdown 2024-09-17T15:33:01.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 msgr2=0x7f05f4009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 0x7f05f4009960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f0009190 comp rx=0 tx=0).stop 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 1 -- 172.21.15.86:0/3470586201 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f05f8009760 con 0x7f05f40074a0 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f06077fe640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060d0c9640 1 -- 172.21.15.86:0/3470586201 reap_dead start 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- 172.21.15.86:0/3470586201 shutdown_connections 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 --2- 172.21.15.86:0/3470586201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05f40074a0 0x7f05f4009960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- 172.21.15.86:0/3470586201 >> 172.21.15.86:0/3470586201 conn(0x7f06080a2f40 msgr2=0x7f06080a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- 172.21.15.86:0/3470586201 shutdown_connections 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- 172.21.15.86:0/3470586201 wait complete. 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: build_initial_monmap 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: monmap: 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:01.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 Processor -- start 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- start start 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: init 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 0x7f06081d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-c con 0x7f05f40074a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 0x7f06081d0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-a con 0x7f06081d05b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f060812e9e0 0x7f060812ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): picked mon.noname-b con 0x7f060812e9e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:01.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f060809ce60 con 0x7f06081d05b0 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f060809cce0 con 0x7f060812e9e0 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f060809ff50 con 0x7f05f40074a0 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060eb2c640 10 monclient(hunting): authenticate will time out at 1314.003797s 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f060c8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 0x7f06081d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.035+0000 7f0607fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 0x7f06081d0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 0x7f06081d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47688/0 (socket says 172.21.15.86:47688) 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 0x7f06081d0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55344/0 (socket says 172.21.15.86:55344) 2024-09-17T15:33:01.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 -- 172.21.15.86:0/2841129672 learned_addr learned my addr 172.21.15.86:0/2841129672 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): get_auth_request con 0x7f05f40074a0 auth_method 0 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): get_auth_request con 0x7f06081d05b0 auth_method 0 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f060812e9e0 0x7f060812ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): get_auth_request con 0x7f060812e9e0 auth_method 0 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060d0c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient(hunting): handle_auth_done global_id 64115 payload 824 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f060812e9e0 msgr2=0x7f060812ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f060812e9e0 0x7f060812ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 msgr2=0x7f06081d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 0x7f06081d0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 monclient: found mon.noname-c 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060813cfb0 con 0x7f05f40074a0 2024-09-17T15:33:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0607fff640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06081d05b0 0x7f06081d0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 msgr2=0x7f06081d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060c8c8640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 0x7f06081d0070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c10 tx=0x7f05f0019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f00b2020 con 0x7f05f40074a0 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: dump: 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.051 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 5 monclient: authenticate success, global_id 64115 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 10 monclient: _renew_subs 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f0003b00 con 0x7f05f40074a0 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06080700c0 con 0x7f05f40074a0 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f00ae400 con 0x7f05f40074a0 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f06057fa640 10 monclient: dump: 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.053 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f0606ffd640 4 set_mon_vals no callback set 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 10 monclient: _renew_subs 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.039+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f060806fb90 con 0x7f05f40074a0 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060eb2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060eb2c640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 0x7f06080713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f06080a1050 con 0x7f060812f380 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 0x7f06080713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient: get_auth_request con 0x7f060812f380 auth_method 0 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 10 monclient(hunting): handle_auth_done global_id 64118 payload 824 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f060813cd90 con 0x7f060812f380 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f05f00b5020 con 0x7f05f40074a0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05e803bb60 0x7f05e803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(186..186 src has 1..186) ==== 4815+0+0 (secure 0 0 0) 0x7f05f00e97e0 con 0x7f05f40074a0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060d0c9640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05e803bb60 0x7f05e803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f060d0c9640 10 monclient: get_auth_request con 0x7f05e803bb60 auth_method 0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 msgr2=0x7f06080713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f0607fff640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 0x7f06080713c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f80046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f802b070 con 0x7f060812f380 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 10 monclient: dump: 2024-09-17T15:33:01.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05f800ee50 con 0x7f060812f380 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 msgr2=0x7f06080713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 0x7f06080713c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f80046d0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 msgr2=0x7f06080713c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:01.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.043+0000 7f06057fa640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060812f380 0x7f06080713c0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b60 tx=0x7f05f80046d0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.047+0000 7f060d0c9640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05e803bb60 0x7f05e803e020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f060806e2e0 tx=0x7f0600003a70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f060eb2c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:01.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f060eb2c640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 0x7f0608071d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f060eb2c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f060812f760 con 0x7f0608071900 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 0x7f0608071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient: get_auth_request con 0x7f0608071900 auth_method 0 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 10 monclient(hunting): handle_auth_done global_id 64121 payload 824 2024-09-17T15:33:01.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.051+0000 7f0607fff640 1 -- 172.21.15.86:0/2841129672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f060806d4f0 con 0x7f0608071900 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f0607fff640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 msgr2=0x7f0608071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f0607fff640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 0x7f0608071d10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c10 tx=0x7f05f8004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f802b070 con 0x7f0608071900 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 10 monclient: dump: 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.061 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f05f802c040 con 0x7f0608071900 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 msgr2=0x7f0608071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 0x7f0608071d10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c10 tx=0x7f05f8004740 comp rx=0 tx=0).stop 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 msgr2=0x7f0608071d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:01.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f06057fa640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0608071900 0x7f0608071d10 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c10 tx=0x7f05f8004740 comp rx=0 tx=0).stop 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7655, 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.065 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:01.066 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.067 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.068 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.069 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:01.071 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:01.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:01.075 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:01.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05e803bb60 msgr2=0x7f05e803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05e803bb60 0x7f05e803e020 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f060806e2e0 tx=0x7f0600003a70 comp rx=0 tx=0).stop 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 10 monclient: shutdown 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 msgr2=0x7f06081d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 --2- 172.21.15.86:0/2841129672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05f40074a0 0x7f06081d0070 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f05f0000c10 tx=0x7f05f0019100 comp rx=0 tx=0).stop 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060d0c9640 1 -- 172.21.15.86:0/2841129672 reap_dead start 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060d0c9640 1 -- 172.21.15.86:0/2841129672 reap_dead start 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 shutdown_connections 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 >> 172.21.15.86:0/2841129672 conn(0x7f06080a2f40 msgr2=0x7f0608131db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 shutdown_connections 2024-09-17T15:33:01.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.055+0000 7f060eb2c640 1 -- 172.21.15.86:0/2841129672 wait complete. 2024-09-17T15:33:01.078 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:01.078 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:33:01.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: get_monmap_and_config 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: build_initial_monmap 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: monmap: 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.199 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:01.200186+0000 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:01.200186+0000 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:01.200 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 Processor -- start 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 -- start start 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: init 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc3c0a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-b con 0x7efc3c0a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efc3c13d2f0 0x7efc3c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-c con 0x7efc3c13d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc3c0a6280 con 0x7efc3c0a50e0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7efc3c09d2a0 con 0x7efc3c13d2f0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c09c0a0 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc4170b640 10 monclient(hunting): authenticate will time out at 1314.165726s 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47714/0 (socket says 172.21.15.86:47714) 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 -- 172.21.15.86:0/2727325299 learned_addr learned my addr 172.21.15.86:0/2727325299 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): get_auth_request con 0x7efc3c0a5bd0 auth_method 0 2024-09-17T15:33:01.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_done global_id 64124 payload 274 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 -- 172.21.15.86:0/2727325299 >> v2:172.21.15.203:3300/0 conn(0x7efc3c13d2f0 msgr2=0x7efc3c13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 --2- 172.21.15.86:0/2727325299 >> v2:172.21.15.203:3300/0 conn(0x7efc3c13d2f0 0x7efc3c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 -- 172.21.15.86:0/2727325299 >> v1:172.21.15.203:6789/0 conn(0x7efc3c0a50e0 legacy=0x7efc3c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 monclient: found mon.noname-b 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.199+0000 7efc3a7fc640 1 -- 172.21.15.86:0/2727325299 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c09c380 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc3a7fc640 1 -- 172.21.15.86:0/2727325299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 msgr2=0x7efc3c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc3a7fc640 1 --2- 172.21.15.86:0/2727325299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc24009870 tx=0x7efc2401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f93ec0372883c858 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 1 -- 172.21.15.86:0/2727325299 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc240b0070 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: dump: 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 5 monclient: authenticate success, global_id 64124 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 1 -- 172.21.15.86:0/2727325299 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc240ab600 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 4 set_mon_vals no callback set 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 1 -- 172.21.15.86:0/2727325299 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc240ab9d0 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc39ffb640 10 monclient: dump: 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.208 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 10 monclient: shutdown 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 -- 172.21.15.86:0/2727325299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 msgr2=0x7efc3c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 --2- 172.21.15.86:0/2727325299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc24009870 tx=0x7efc2401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 -- 172.21.15.86:0/2727325299 shutdown_connections 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 --2- 172.21.15.86:0/2727325299 >> v2:172.21.15.203:3300/0 conn(0x7efc3c13d2f0 0x7efc3c13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 --2- 172.21.15.86:0/2727325299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c0a5bd0 0x7efc3c09cd60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 -- 172.21.15.86:0/2727325299 >> 172.21.15.86:0/2727325299 conn(0x7efc3c0aaae0 msgr2=0x7efc3c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 -- 172.21.15.86:0/2727325299 shutdown_connections 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.203+0000 7efc4170b640 1 -- 172.21.15.86:0/2727325299 wait complete. 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient: build_initial_monmap 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient: monmap: 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:01.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 Processor -- start 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 -- start start 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient: init 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c0a50e0 0x7efc3c1cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-b con 0x7efc3c0a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:01.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 0x7efc3c1d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-a con 0x7efc3c0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): picked mon.noname-c con 0x7efc3c13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c142650 con 0x7efc3c0a5bd0 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c0a6b00 con 0x7efc3c0a50e0 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efc3c1427d0 con 0x7efc3c13d2f0 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc4170b640 10 monclient(hunting): authenticate will time out at 1314.173394s 2024-09-17T15:33:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 0x7efc3c1d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 0x7efc3c1d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55354/0 (socket says 172.21.15.86:55354) 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47730/0 (socket says 172.21.15.86:47730) 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 1 -- 172.21.15.86:0/4230331879 learned_addr learned my addr 172.21.15.86:0/4230331879 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): get_auth_request con 0x7efc3c0a5bd0 auth_method 0 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): get_auth_request con 0x7efc3c13d2f0 auth_method 0 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c0a50e0 0x7efc3c1cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient(hunting): handle_auth_done global_id 64127 payload 824 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c0a50e0 msgr2=0x7efc3c1cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c0a50e0 0x7efc3c1cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 msgr2=0x7efc3c1d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 0x7efc3c1d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3b7fe640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc3c13ac30 con 0x7efc3c13d2f0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3a7fc640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc3c0a5bd0 0x7efc3c1d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3affd640 10 monclient: get_auth_request con 0x7efc3c0a50e0 auth_method 0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.207+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c0a50e0 0x7efc3c1cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3b7fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 msgr2=0x7efc3c1d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3b7fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0a8520 tx=0x7efc2c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c015020 con 0x7efc3c13d2f0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: dump: 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2c011a40 con 0x7efc3c13d2f0 2024-09-17T15:33:01.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 5 monclient: authenticate success, global_id 64127 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 monclient: _renew_subs 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc3c0a18d0 con 0x7efc3c13d2f0 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc2c011e10 con 0x7efc3c13d2f0 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 10 monclient: dump: 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc397fa640 4 set_mon_vals no callback set 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 monclient: _renew_subs 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc3c13b1d0 con 0x7efc3c13d2f0 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 0x7efc3c135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c09cd60 con 0x7efc3c134e70 2024-09-17T15:33:01.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7efc2c018020 con 0x7efc3c13d2f0 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 0x7efc3c135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient: get_auth_request con 0x7efc3c134e70 auth_method 0 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efc1c03bbb0 0x7efc1c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(186..186 src has 1..186) ==== 4815+0+0 (secure 0 0 0) 0x7efc2c04fd30 con 0x7efc3c13d2f0 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3a7fc640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efc1c03bbb0 0x7efc1c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 10 monclient(hunting): handle_auth_done global_id 54172 payload 824 2024-09-17T15:33:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3affd640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc3c13afb0 con 0x7efc3c134e70 2024-09-17T15:33:01.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.211+0000 7efc3a7fc640 10 monclient: get_auth_request con 0x7efc1c03bbb0 auth_method 0 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc3affd640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 msgr2=0x7efc3c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 0x7efc3c135250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc3400d860 tx=0x7efc3400dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34006d60 con 0x7efc3c134e70 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 10 monclient: dump: 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.226 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc3a7fc640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efc1c03bbb0 0x7efc1c03e070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc24000c10 tx=0x7efc240a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc34006d60 con 0x7efc3c134e70 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 msgr2=0x7efc3c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 0x7efc3c135250 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc3400d860 tx=0x7efc3400dd20 comp rx=0 tx=0).stop 2024-09-17T15:33:01.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 msgr2=0x7efc3c135250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.215+0000 7efc337fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c134e70 0x7efc3c135250 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc3400d860 tx=0x7efc3400dd20 comp rx=0 tx=0).stop 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.219+0000 7efc4170b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.219+0000 7efc4170b640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 0x7efc3c09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.219+0000 7efc4170b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.219+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc3c1399d0 con 0x7efc3c135790 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 0x7efc3c09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient: get_auth_request con 0x7efc3c135790 auth_method 0 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 10 monclient(hunting): handle_auth_done global_id 54175 payload 824 2024-09-17T15:33:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 1 -- 172.21.15.86:0/4230331879 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc3c074be0 con 0x7efc3c135790 2024-09-17T15:33:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 msgr2=0x7efc3c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc3affd640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 0x7efc3c09e5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc34017000 tx=0x7efc34003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34028020 con 0x7efc3c135790 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.223+0000 7efc337fe640 10 monclient: dump: 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.230 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7efc340123f0 con 0x7efc3c135790 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 msgr2=0x7efc3c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 0x7efc3c09e5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efc34017000 tx=0x7efc34003af0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 msgr2=0x7efc3c09e5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc337fe640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc3c135790 0x7efc3c09e5e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:01.231 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 56654, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:01.232 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:01.233 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.234 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:01.235 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.236 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.238 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.239 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:01.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:01.241 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:01.242 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:01.243 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:01.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efc1c03bbb0 msgr2=0x7efc1c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efc1c03bbb0 0x7efc1c03e070 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7efc24000c10 tx=0x7efc240a6000 comp rx=0 tx=0).stop 2024-09-17T15:33:01.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 10 monclient: shutdown 2024-09-17T15:33:01.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 msgr2=0x7efc3c1d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efc3c0a8520 tx=0x7efc2c009f90 comp rx=0 tx=0).stop 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc3b7fe640 1 -- 172.21.15.86:0/4230331879 reap_dead start 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 shutdown_connections 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 --2- 172.21.15.86:0/4230331879 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc3c13d2f0 0x7efc3c1d0620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 >> 172.21.15.86:0/4230331879 conn(0x7efc3c0aaae0 msgr2=0x7efc3c13f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 shutdown_connections 2024-09-17T15:33:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.227+0000 7efc4170b640 1 -- 172.21.15.86:0/4230331879 wait complete. 2024-09-17T15:33:01.245 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:01.245 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:33:01.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.355+0000 7f88e31dd640 10 monclient: get_monmap_and_config 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.355+0000 7f88e31dd640 10 monclient: build_initial_monmap 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient: monmap: 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:01.360020+0000 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:01.360020+0000 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:01.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:01.360 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 Processor -- start 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 -- start start 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient: init 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88dc0689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f88dc068db0 0x7f88dc0691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-c con 0x7f88dc068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:01.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88dc139f00 0x7f88dc13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-a con 0x7f88dc139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88dc09d770 con 0x7f88dc0689d0 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f88dc09ed00 con 0x7f88dc139f00 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f88dc09efe0 con 0x7f88dc068db0 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e31dd640 10 monclient(hunting): authenticate will time out at 1314.325430s 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88dc139f00 0x7f88dc13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88dc139f00 0x7f88dc13c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55366/0 (socket says 172.21.15.86:55366) 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 -- 172.21.15.86:0/3476221158 learned_addr learned my addr 172.21.15.86:0/3476221158 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): get_auth_request con 0x7f88dc139f00 auth_method 0 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88dbfff640 1 --2- 172.21.15.86:0/3476221158 >> v2:172.21.15.203:3300/0 conn(0x7f88dc068db0 0x7f88dc0691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88dbfff640 10 monclient(hunting): get_auth_request con 0x7f88dc068db0 auth_method 0 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient(hunting): handle_auth_done global_id 4872 payload 274 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 -- 172.21.15.86:0/3476221158 >> v2:172.21.15.203:3300/0 conn(0x7f88dc068db0 msgr2=0x7f88dc0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 --2- 172.21.15.86:0/3476221158 >> v2:172.21.15.203:3300/0 conn(0x7f88dc068db0 0x7f88dc0691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 -- 172.21.15.86:0/3476221158 >> v1:172.21.15.203:6789/0 conn(0x7f88dc0689d0 legacy=0x7f88dc09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 monclient: found mon.noname-a 2024-09-17T15:33:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.359+0000 7f88e177a640 1 -- 172.21.15.86:0/3476221158 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc069700 con 0x7f88dc139f00 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e177a640 1 -- 172.21.15.86:0/3476221158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc139f00 msgr2=0x7f88dc13c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e177a640 1 --2- 172.21.15.86:0/3476221158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc139f00 0x7f88dc13c2f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0a06b0 tx=0x7f88d401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d634c8c8ee10a07 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 1 -- 172.21.15.86:0/3476221158 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d40b0070 con 0x7f88dc139f00 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: dump: 2024-09-17T15:33:01.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 1 -- 172.21.15.86:0/3476221158 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d40ab600 con 0x7f88dc139f00 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 1 -- 172.21.15.86:0/3476221158 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d40ab9d0 con 0x7f88dc139f00 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 5 monclient: authenticate success, global_id 4872 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 4 set_mon_vals no callback set 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88db7fe640 10 monclient: dump: 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 10 monclient: shutdown 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 -- 172.21.15.86:0/3476221158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc139f00 msgr2=0x7f88dc13c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 --2- 172.21.15.86:0/3476221158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc139f00 0x7f88dc13c2f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f88dc0a06b0 tx=0x7f88d401c3c0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 -- 172.21.15.86:0/3476221158 shutdown_connections 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 --2- 172.21.15.86:0/3476221158 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc139f00 0x7f88dc13c2f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 --2- 172.21.15.86:0/3476221158 >> v2:172.21.15.203:3300/0 conn(0x7f88dc068db0 0x7f88dc0691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 -- 172.21.15.86:0/3476221158 >> 172.21.15.86:0/3476221158 conn(0x7f88dc0a6440 msgr2=0x7f88dc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 -- 172.21.15.86:0/3476221158 shutdown_connections 2024-09-17T15:33:01.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 1 -- 172.21.15.86:0/3476221158 wait complete. 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 10 monclient: build_initial_monmap 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.363+0000 7f88e31dd640 10 monclient: monmap: 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:01.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 Processor -- start 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- start start 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: init 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-a con 0x7f88dc0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:01.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 0x7f88dc131b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-c con 0x7f88dc068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88dc139f00 0x7f88dc070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): picked mon.noname-b con 0x7f88dc139f00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc09c610 con 0x7f88dc0689d0 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc09c490 con 0x7f88dc139f00 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f88dc09c790 con 0x7f88dc068db0 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient(hunting): authenticate will time out at 1314.332620s 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 0x7f88dc131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55378/0 (socket says 172.21.15.86:55378) 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 0x7f88dc131b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47738/0 (socket says 172.21.15.86:47738) 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 -- 172.21.15.86:0/2243084567 learned_addr learned my addr 172.21.15.86:0/2243084567 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): get_auth_request con 0x7f88dc0689d0 auth_method 0 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): get_auth_request con 0x7f88dc068db0 auth_method 0 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient(hunting): handle_auth_done global_id 4875 payload 824 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e177a640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88dc139f00 0x7f88dc070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 -- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 msgr2=0x7f88dc131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 0x7f88dc131b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 -- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88dc139f00 msgr2=0x7f88dc070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88dc139f00 0x7f88dc070730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient: _finish_hunting 0 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 monclient: found mon.noname-a 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 -- 172.21.15.86:0/2243084567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc073f90 con 0x7f88dc0689d0 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 -- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 msgr2=0x7f88dc1cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e0f79640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f88cc00c7f0 tx=0x7f88cc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc016020 con 0x7f88dc0689d0 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: dump: 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: _finish_auth 0 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: _check_auth_tickets 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88cc00ac00 con 0x7f88dc0689d0 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc012420 con 0x7f88dc0689d0 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 5 monclient: authenticate success, global_id 4875 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88d97fa640 10 monclient: dump: 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:01.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88da7fc640 4 set_mon_vals no callback set 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _renew_subs 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88dc072690 con 0x7f88dc0689d0 2024-09-17T15:33:01.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _renew_subs 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.367+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88dc13a3e0 con 0x7f88dc0689d0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88e31dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88e31dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88e31dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88dc09d460 con 0x7f88dc0689d0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f88cc019020 con 0x7f88dc0689d0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88d97fa640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 0x7f88bc03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 4815+0+0 (secure 0 0 0) 0x7f88cc04ffe0 con 0x7f88dc0689d0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88dbfff640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 0x7f88bc03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.371+0000 7f88dbfff640 10 monclient: get_auth_request con 0x7f88bc03bb90 auth_method 0 2024-09-17T15:33:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.375+0000 7f88dbfff640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 0x7f88bc03e050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f88dc074930 tx=0x7f88c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.375+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f88cc020030 con 0x7f88dc0689d0 2024-09-17T15:33:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.375+0000 7f88d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.375+0000 7f88d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.611+0000 7f88e31dd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:33:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.611+0000 7f88e31dd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:33:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.611+0000 7f88e31dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.611+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f88dc1ce860 con 0x7f88dc0689d0 2024-09-17T15:33:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88d97fa640 1 -- 172.21.15.86:0/2243084567 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f88cc010cf0 con 0x7f88dc0689d0 2024-09-17T15:33:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:33:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 msgr2=0x7f88bc03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 0x7f88bc03e050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f88dc074930 tx=0x7f88c4008040 comp rx=0 tx=0).stop 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 10 monclient: shutdown 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 msgr2=0x7f88dc1cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f88cc00c7f0 tx=0x7f88cc00ccb0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 shutdown_connections 2024-09-17T15:33:01.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f88bc03bb90 0x7f88bc03e050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88dc139f00 0x7f88dc070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88dc068db0 0x7f88dc131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 --2- 172.21.15.86:0/2243084567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88dc0689d0 0x7f88dc1cd9e0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 >> 172.21.15.86:0/2243084567 conn(0x7f88dc0a6440 msgr2=0x7f88dc134c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 shutdown_connections 2024-09-17T15:33:01.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:01.615+0000 7f88e31dd640 1 -- 172.21.15.86:0/2243084567 wait complete. 2024-09-17T15:33:01.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:01.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=9,logm=66} crc {auth=3366757788,config=3541143443,health=1228072951,kv=3400921214,logm=812214591}) 2024-09-17T15:33:01.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=11,kv=10,logm=65} crc {auth=3745831156,config=3541143443,health=2069616219,kv=1466466540,logm=171667369}) 2024-09-17T15:33:01.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:01.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=9,logm=66} crc {auth=3366757788,config=3541143443,health=1228072951,kv=3400921214,logm=812214591}) 2024-09-17T15:33:01.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.619+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=10,logm=64} crc {auth=733273072,config=3541143443,health=3837507830,kv=3885551602,logm=1828136043}) 2024-09-17T15:33:01.629 INFO:tasks.rados.rados.0.smithi203.stdout:796: done (4 left) 2024-09-17T15:33:01.630 INFO:tasks.rados.rados.0.smithi203.stdout:792: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:33:01.630 INFO:tasks.rados.rados.0.smithi203.stdout:792: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:33:01.630 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 331 (ObjNum 129 snap 0 seq_num 3606078688) dirty dne 2024-09-17T15:33:01.630 INFO:tasks.rados.rados.0.smithi203.stdout:794: expect (ObjNum 284 snap 75 seq_num 284) 2024-09-17T15:33:01.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:01.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642456229}) 2024-09-17T15:33:01.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775916210}) 2024-09-17T15:33:01.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:01.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642456229}) 2024-09-17T15:33:01.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:01.627+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452706964}) 2024-09-17T15:33:01.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:33:01.712 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:01.754 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:01.791 INFO:tasks.rados.rados.0.smithi203.stdout:793: expect (ObjNum 76 snap 5 seq_num 76) 2024-09-17T15:33:01.791 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:01.936 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:797: read oid 26 snap 83 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:797: expect deleted 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:791: done (4 left) 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:792: done (3 left) 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:793: done (2 left) 2024-09-17T15:33:02.009 INFO:tasks.rados.rados.0.smithi203.stdout:794: done (1 left) 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:798: copy_from oid 14 from oid 24 current snap is 84 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:799: copy_from oid 25 from oid 21 current snap is 84 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:800: copy_from oid 30 from oid 19 current snap is 84 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:801: read oid 38 snap -1 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:801: expect (ObjNum 281 snap 73 seq_num 281) 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:802: write (excl) oid 1 current snap is 84 2024-09-17T15:33:02.010 INFO:tasks.rados.rados.0.smithi203.stdout:802: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-17T15:33:02.032 INFO:tasks.rados.rados.0.smithi203.stdout:802: writing smithi20328086-1 from 742942 to 1223256 tid 1 2024-09-17T15:33:02.055 INFO:tasks.rados.rados.0.smithi203.stdout:802: writing smithi20328086-1 from 1823137 to 2471905 tid 2 2024-09-17T15:33:02.055 INFO:tasks.rados.rados.0.smithi203.stdout:803: read oid 31 snap 68 2024-09-17T15:33:02.055 INFO:tasks.rados.rados.0.smithi203.stdout:803: expect (ObjNum 254 snap 68 seq_num 254) 2024-09-17T15:33:02.055 INFO:tasks.rados.rados.0.smithi203.stdout:804: read oid 37 snap -1 2024-09-17T15:33:02.055 INFO:tasks.rados.rados.0.smithi203.stdout:804: expect (ObjNum 271 snap 70 seq_num 271) 2024-09-17T15:33:02.056 INFO:tasks.rados.rados.0.smithi203.stdout:805: snap_create 2024-09-17T15:33:02.056 INFO:tasks.rados.rados.0.smithi203.stdout:800: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:33:02.056 INFO:tasks.rados.rados.0.smithi203.stdout:801: expect (ObjNum 281 snap 73 seq_num 281) 2024-09-17T15:33:02.056 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 404249 2024-09-17T15:33:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:02.138 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414530643}) 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015066492}) 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414530643}) 2024-09-17T15:33:02.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.159+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275143351}) 2024-09-17T15:33:02.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843417372}) 2024-09-17T15:33:02.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512314192}) 2024-09-17T15:33:02.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843417372}) 2024-09-17T15:33:02.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044058768}) 2024-09-17T15:33:02.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831301727}) 2024-09-17T15:33:02.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3943331526}) 2024-09-17T15:33:02.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831301727}) 2024-09-17T15:33:02.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.163+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645963430}) 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519529201}) 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825165303}) 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519529201}) 2024-09-17T15:33:02.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350220895}) 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695226537}) 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851326057}) 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695226537}) 2024-09-17T15:33:02.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.167+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594890613}) 2024-09-17T15:33:02.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170452666}) 2024-09-17T15:33:02.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153863555}) 2024-09-17T15:33:02.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=170452666}) 2024-09-17T15:33:02.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267826823}) 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197167032}) 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838506055}) 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.171+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197167032}) 2024-09-17T15:33:02.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278740048}) 2024-09-17T15:33:02.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593494661}) 2024-09-17T15:33:02.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3470926818}) 2024-09-17T15:33:02.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1593494661}) 2024-09-17T15:33:02.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.175+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716856880}) 2024-09-17T15:33:02.179 INFO:tasks.rados.rados.0.smithi203.stdout:800: finishing copy_from to smithi20328086-30 2024-09-17T15:33:02.179 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:33:02.179 INFO:tasks.rados.rados.0.smithi203.stdout:799: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:33:02.179 INFO:tasks.rados.rados.0.smithi203.stdout:798: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:33:02.179 INFO:tasks.rados.rados.0.smithi203.stdout:802: finishing write tid 1 to smithi20328086-1 2024-09-17T15:33:02.180 INFO:tasks.rados.rados.0.smithi203.stdout:799: finishing copy_from to smithi20328086-25 2024-09-17T15:33:02.180 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 261 (ObjNum 257 snap 68 seq_num 257) dirty exists 2024-09-17T15:33:02.180 INFO:tasks.rados.rados.0.smithi203.stdout:804: expect (ObjNum 271 snap 70 seq_num 271) 2024-09-17T15:33:02.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259096833}) 2024-09-17T15:33:02.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=572716347}) 2024-09-17T15:33:02.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2259096833}) 2024-09-17T15:33:02.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.179+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547280957}) 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659222217}) 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=850229952}) 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659222217}) 2024-09-17T15:33:02.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686341952}) 2024-09-17T15:33:02.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723289372}) 2024-09-17T15:33:02.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035706684}) 2024-09-17T15:33:02.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1723289372}) 2024-09-17T15:33:02.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.183+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313163487}) 2024-09-17T15:33:02.190 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.190 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3506302978,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2201730280}) 2024-09-17T15:33:02.190 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=2} crc {logm=2089457691,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1103556260,mon_config_key=1147139052}) 2024-09-17T15:33:02.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=3506302978,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2201730280}) 2024-09-17T15:33:02.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.187+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=7} crc {logm=4157246470,mdsmap=2815901553,mgr=1842696677,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2058650013,mon_config_key=2017621969}) 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=36} crc {mon_config_key=639531838,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=869782279,osd_snap=3428816185,osdmap=830473445}) 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=38,osdmap=40} crc {mon_config_key=1184047778,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3734937928,osd_pg_creating=869782279,osd_snap=3336922814,osdmap=3553872891}) 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=38,osdmap=36} crc {mon_config_key=639531838,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=869782279,osd_snap=3428816185,osdmap=830473445}) 2024-09-17T15:33:02.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.191+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=38,osdmap=37} crc {mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=2154151236,osd_pg_creating=869782279,osd_snap=3336922814,osdmap=750657665}) 2024-09-17T15:33:02.197 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.197 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1436088235}) 2024-09-17T15:33:02.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2201648841}) 2024-09-17T15:33:02.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1436088235}) 2024-09-17T15:33:02.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.195+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3147925219}) 2024-09-17T15:33:02.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1578366101}) 2024-09-17T15:33:02.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1482771656}) 2024-09-17T15:33:02.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1578366101}) 2024-09-17T15:33:02.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.199+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1580488635}) 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3594838690}) 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=50} crc {osdmap=2302583999}) 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=3594838690}) 2024-09-17T15:33:02.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:02.215+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=3182481026}) 2024-09-17T15:33:02.222 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1938200 2024-09-17T15:33:02.236 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:02.359 INFO:tasks.rados.rados.0.smithi203.stdout:803: expect (ObjNum 254 snap 68 seq_num 254) 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:798: finishing copy_from to smithi20328086-14 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 169 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: finishing write tid 2 to smithi20328086-1 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: finishing write tid 3 to smithi20328086-1 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: finishing write tid 4 to smithi20328086-1 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: oid 1 updating version 0 to 248 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: oid 1 version 248 is already newer than 245 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: oid 1 updating version 248 to 249 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 249 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:33:02.461 INFO:tasks.rados.rados.0.smithi203.stdout:802: left oid 1 (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:797: done (8 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:798: done (7 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:799: done (6 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:800: done (5 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:801: done (4 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:802: done (3 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:803: done (2 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:804: done (1 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:805: done (0 left) 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:806: rollback oid 24 current snap is 85 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 79 2024-09-17T15:33:02.462 INFO:tasks.rados.rados.0.smithi203.stdout:807: delete oid 42 current snap is 85 2024-09-17T15:33:02.470 INFO:tasks.rados.rados.0.smithi203.stdout:807: done (1 left) 2024-09-17T15:33:02.470 INFO:tasks.rados.rados.0.smithi203.stdout:808: copy_from oid 8 from oid 49 current snap is 85 2024-09-17T15:33:02.470 INFO:tasks.rados.rados.0.smithi203.stdout:809: snap_create 2024-09-17T15:33:02.471 INFO:tasks.rados.rados.0.smithi203.stdout:806: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:33:02.472 INFO:tasks.rados.rados.0.smithi203.stdout:806: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:33:02.472 INFO:tasks.rados.rados.0.smithi203.stdout:808: finishing copy_from to smithi20328086-8 2024-09-17T15:33:02.472 INFO:tasks.rados.rados.0.smithi203.stdout:808: got expected ENOENT (src dne) 2024-09-17T15:33:02.473 INFO:tasks.rados.rados.0.smithi203.stdout:806: finishing rollback tid 2 to smithi20328086-24 2024-09-17T15:33:02.473 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 264 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:33:02.473 INFO:tasks.rados.rados.0.smithi203.stdout:808: finishing copy_from racing read to smithi20328086-8 2024-09-17T15:33:02.637 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:33:02.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 10 monclient: get_monmap_and_config 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 10 monclient: build_initial_monmap 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 10 monclient: monmap: 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:02.766959+0000 2024-09-17T15:33:02.766 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:02.766959+0000 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:02.767 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 1 Processor -- start 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 1 -- start start 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.763+0000 7fc300a78640 10 monclient: init 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-b con 0x7fc2fc0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2fc068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2fc139f00 0x7fc2fc13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-c con 0x7fc2fc139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2fc09d770 con 0x7fc2fc068db0 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc2fc09ed00 con 0x7fc2fc139f00 2024-09-17T15:33:02.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2fc09efe0 con 0x7fc2fc0689d0 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc300a78640 10 monclient(hunting): authenticate will time out at 1315.732437s 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47742/0 (socket says 172.21.15.86:47742) 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 -- 172.21.15.86:0/2227145358 learned_addr learned my addr 172.21.15.86:0/2227145358 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request con 0x7fc2fc0689d0 auth_method 0 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:02.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fad9d640 1 --2- 172.21.15.86:0/2227145358 >> v2:172.21.15.203:3300/0 conn(0x7fc2fc139f00 0x7fc2fc13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fad9d640 10 monclient(hunting): get_auth_request con 0x7fc2fc139f00 auth_method 0 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_done global_id 64133 payload 274 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 -- 172.21.15.86:0/2227145358 >> v2:172.21.15.203:3300/0 conn(0x7fc2fc139f00 msgr2=0x7fc2fc13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 --2- 172.21.15.86:0/2227145358 >> v2:172.21.15.203:3300/0 conn(0x7fc2fc139f00 0x7fc2fc13c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 -- 172.21.15.86:0/2227145358 >> v1:172.21.15.203:6789/0 conn(0x7fc2fc068db0 legacy=0x7fc2fc069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient: _finish_hunting 0 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 monclient: found mon.noname-b 2024-09-17T15:33:02.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:02.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:02.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.767+0000 7fc2fa59c640 1 -- 172.21.15.86:0/2227145358 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc0a04e0 con 0x7fc2fc0689d0 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2fa59c640 1 -- 172.21.15.86:0/2227145358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 msgr2=0x7fc2fc09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2fa59c640 1 --2- 172.21.15.86:0/2227145358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0a0770 tx=0x7fc2ec01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=149a80d884985ede server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 1 -- 172.21.15.86:0/2227145358 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0b0070 con 0x7fc2fc0689d0 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:02.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: dump: 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: _finish_auth 0 2024-09-17T15:33:02.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 1 -- 172.21.15.86:0/2227145358 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec0ab600 con 0x7fc2fc0689d0 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 1 -- 172.21.15.86:0/2227145358 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec0ab9d0 con 0x7fc2fc0689d0 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 5 monclient: authenticate success, global_id 64133 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 4 set_mon_vals no callback set 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc2f9d9b640 10 monclient: dump: 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.776 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 10 monclient: shutdown 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 -- 172.21.15.86:0/2227145358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 msgr2=0x7fc2fc09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 --2- 172.21.15.86:0/2227145358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0a0770 tx=0x7fc2ec01c490 comp rx=0 tx=0).stop 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 -- 172.21.15.86:0/2227145358 shutdown_connections 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 --2- 172.21.15.86:0/2227145358 >> v2:172.21.15.203:3300/0 conn(0x7fc2fc139f00 0x7fc2fc13c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 --2- 172.21.15.86:0/2227145358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc0689d0 0x7fc2fc09ffa0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:02.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 -- 172.21.15.86:0/2227145358 >> 172.21.15.86:0/2227145358 conn(0x7fc2fc0a6440 msgr2=0x7fc2fc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:02.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 -- 172.21.15.86:0/2227145358 shutdown_connections 2024-09-17T15:33:02.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 1 -- 172.21.15.86:0/2227145358 wait complete. 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 10 monclient: build_initial_monmap 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.771+0000 7fc300a78640 10 monclient: monmap: 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:02.781 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 Processor -- start 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- start start 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: init 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-a con 0x7fc2fc0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 0x7fc2fc134080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-c con 0x7fc2fc068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 0x7fc2fc11de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): picked mon.noname-b con 0x7fc2fc139f00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc13f3e0 con 0x7fc2fc0689d0 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc13f260 con 0x7fc2fc139f00 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2fc13f560 con 0x7fc2fc068db0 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient(hunting): authenticate will time out at 1315.740732s 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 0x7fc2fc134080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55380/0 (socket says 172.21.15.86:55380) 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 1 -- 172.21.15.86:0/1015078575 learned_addr learned my addr 172.21.15.86:0/1015078575 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): get_auth_request con 0x7fc2fc068db0 auth_method 0 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:02.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request con 0x7fc2fc0689d0 auth_method 0 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fad9d640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 0x7fc2fc11de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient(hunting): handle_auth_done global_id 64136 payload 824 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 -- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 msgr2=0x7fc2fc11de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 0x7fc2fc11de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:02.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 -- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 msgr2=0x7fc2fc1cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 monclient: found mon.noname-c 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 -- 172.21.15.86:0/1015078575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc1214a0 con 0x7fc2fc068db0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fad9d640 10 monclient: get_auth_request con 0x7fc2fc139f00 auth_method 0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fad9d640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 0x7fc2fc11de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2fa59c640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 -- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 msgr2=0x7fc2fc134080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f1d9b640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 0x7fc2fc134080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c410 tx=0x7fc2e400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4017070 con 0x7fc2fc068db0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:02.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: dump: 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:02.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e40069e0 con 0x7fc2fc068db0 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 5 monclient: authenticate success, global_id 64136 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _renew_subs 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc121780 con 0x7fc2fc068db0 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4006dd0 con 0x7fc2fc068db0 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f37fe640 10 monclient: dump: 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.787 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _renew_subs 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc2f959a640 4 set_mon_vals no callback set 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.775+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc13bac0 con 0x7fc2fc068db0 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc300a78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc300a78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:02.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc300a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc09ffa0 con 0x7fc2fc068db0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc2e401a070 con 0x7fc2fc068db0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc2f37fe640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 0x7fc2dc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(188..188 src has 1..188) ==== 4815+0+0 (secure 0 0 0) 0x7fc2e404f300 con 0x7fc2fc068db0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc2fa59c640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 0x7fc2dc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.779+0000 7fc2fa59c640 10 monclient: get_auth_request con 0x7fc2dc03bb60 auth_method 0 2024-09-17T15:33:02.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.783+0000 7fc2fa59c640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 0x7fc2dc03e020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec01c4d0 tx=0x7fc2ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:02.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.783+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc2e40117a0 con 0x7fc2fc068db0 2024-09-17T15:33:02.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.783+0000 7fc2f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:02.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:02.783+0000 7fc2f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:03.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.015+0000 7fc300a78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:03.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.015+0000 7fc300a78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:03.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.015+0000 7fc300a78640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.015+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc2fc06d4f0 con 0x7fc2fc068db0 2024-09-17T15:33:03.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc2f37fe640 1 -- 172.21.15.86:0/1015078575 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fc2e404f0d0 con 0x7fc2fc068db0 2024-09-17T15:33:03.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc2f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:03.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc2f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:03.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 msgr2=0x7fc2dc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 0x7fc2dc03e020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec01c4d0 tx=0x7fc2ec008040 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 10 monclient: shutdown 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 msgr2=0x7fc2fc134080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 0x7fc2fc134080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400c410 tx=0x7fc2e400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 shutdown_connections 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2dc03bb60 0x7fc2dc03e020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2fc139f00 0x7fc2fc11de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2fc068db0 0x7fc2fc134080 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 --2- 172.21.15.86:0/1015078575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2fc0689d0 0x7fc2fc1cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 >> 172.21.15.86:0/1015078575 conn(0x7fc2fc0a6440 msgr2=0x7fc2fc139010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 shutdown_connections 2024-09-17T15:33:03.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.019+0000 7fc300a78640 1 -- 172.21.15.86:0/1015078575 wait complete. 2024-09-17T15:33:03.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:33:03.038 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:33:03.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:33:03.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.155+0000 7f9c509f7640 10 monclient: get_monmap_and_config 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.155+0000 7f9c509f7640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient: monmap: 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:03.160021+0000 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:03.160021+0000 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:03.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:03.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:03.160 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:03.160 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:03.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 Processor -- start 2024-09-17T15:33:03.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 -- start start 2024-09-17T15:33:03.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient: init 2024-09-17T15:33:03.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c4c0a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 0x7f9c4c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-b con 0x7f9c4c0a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9c4c13b450 0x7f9c4c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-c con 0x7f9c4c13b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c4c0a2130 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9c4c0a2410 con 0x7f9c4c13b450 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c0a26f0 con 0x7f9c4c0a1760 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c509f7640 10 monclient(hunting): authenticate will time out at 1316.125470s 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4a59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9c4c0a0c70 0x7f9c4c0a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:57394/0 (socket says 172.21.15.86:57394) 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 0x7f9c4c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4a59c640 1 -- 172.21.15.86:0/300853492 learned_addr learned my addr 172.21.15.86:0/300853492 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request con 0x7f9c4c0a1760 auth_method 0 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4ad9d640 1 --2- 172.21.15.86:0/300853492 >> v2:172.21.15.203:3300/0 conn(0x7f9c4c13b450 0x7f9c4c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4ad9d640 10 monclient(hunting): get_auth_request con 0x7f9c4c13b450 auth_method 0 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_done global_id 64139 payload 274 2024-09-17T15:33:03.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 -- 172.21.15.86:0/300853492 >> v2:172.21.15.203:3300/0 conn(0x7f9c4c13b450 msgr2=0x7f9c4c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/300853492 >> v2:172.21.15.203:3300/0 conn(0x7f9c4c13b450 0x7f9c4c13d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 -- 172.21.15.86:0/300853492 >> v1:172.21.15.86:6789/0 conn(0x7f9c4c0a0c70 legacy=0x7f9c4c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 monclient: found mon.noname-b 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.159+0000 7f9c49d9b640 1 -- 172.21.15.86:0/300853492 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c09f9f0 con 0x7f9c4c0a1760 2024-09-17T15:33:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c49d9b640 1 -- 172.21.15.86:0/300853492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 msgr2=0x7f9c4c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/300853492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 0x7f9c4c13af10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c400097c0 tx=0x7f9c4001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f96be8c851706e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 1 -- 172.21.15.86:0/300853492 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c400b0070 con 0x7f9c4c0a1760 2024-09-17T15:33:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: dump: 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 1 -- 172.21.15.86:0/300853492 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c400ab600 con 0x7f9c4c0a1760 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 1 -- 172.21.15.86:0/300853492 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c400ab9d0 con 0x7f9c4c0a1760 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 5 monclient: authenticate success, global_id 64139 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 4 set_mon_vals no callback set 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c4959a640 10 monclient: dump: 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.250 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 10 monclient: shutdown 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 -- 172.21.15.86:0/300853492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 msgr2=0x7f9c4c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 --2- 172.21.15.86:0/300853492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 0x7f9c4c13af10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9c400097c0 tx=0x7f9c4001c360 comp rx=0 tx=0).stop 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 -- 172.21.15.86:0/300853492 shutdown_connections 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 --2- 172.21.15.86:0/300853492 >> v2:172.21.15.203:3300/0 conn(0x7f9c4c13b450 0x7f9c4c13d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 --2- 172.21.15.86:0/300853492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a1760 0x7f9c4c13af10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 -- 172.21.15.86:0/300853492 >> 172.21.15.86:0/300853492 conn(0x7f9c4c0a2f40 msgr2=0x7f9c4c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 -- 172.21.15.86:0/300853492 shutdown_connections 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.163+0000 7f9c509f7640 1 -- 172.21.15.86:0/300853492 wait complete. 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient: monmap: 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 Processor -- start 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 -- start start 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient: init 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:33:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 0x7f9c4c1cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-c con 0x7f9c4c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c13b450 0x7f9c4c13a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-a con 0x7f9c4c13b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c4c13a900 0x7f9c4c070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): picked mon.noname-b con 0x7f9c4c13a900 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c140720 con 0x7f9c4c13b450 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c1405a0 con 0x7f9c4c13a900 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4c1408a0 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c509f7640 10 monclient(hunting): authenticate will time out at 1316.133360s 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 0x7f9c4c1cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 0x7f9c4c1cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47764/0 (socket says 172.21.15.86:47764) 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 -- 172.21.15.86:0/2148676643 learned_addr learned my addr 172.21.15.86:0/2148676643 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): get_auth_request con 0x7f9c4c0a0c70 auth_method 0 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c13b450 0x7f9c4c13a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request con 0x7f9c4c13b450 auth_method 0 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4ad9d640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c4c13a900 0x7f9c4c070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4ad9d640 10 monclient(hunting): get_auth_request con 0x7f9c4c13a900 auth_method 0 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient(hunting): handle_auth_done global_id 64142 payload 824 2024-09-17T15:33:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c4c13a900 msgr2=0x7f9c4c070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c4c13a900 0x7f9c4c070ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c13b450 msgr2=0x7f9c4c13a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c13b450 0x7f9c4c13a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 monclient: found mon.noname-c 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c4a59c640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c4c13ad60 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.167+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c13b450 0x7f9c4c13a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c4a59c640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 msgr2=0x7f9c4c1cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c4a59c640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 0x7f9c4c1cdca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c4a0 tx=0x7f9c3c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c017070 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: dump: 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c3c007070 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c007460 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 10 monclient: dump: 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 5 monclient: authenticate success, global_id 64142 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 monclient: _renew_subs 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c48d99640 4 set_mon_vals no callback set 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c4c13c180 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 monclient: _renew_subs 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c4c137e80 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9c3c01a070 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 0x7f9c4c02d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c0a1050 con 0x7f9c4c02d450 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 0x7f9c4c02d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient: get_auth_request con 0x7f9c4c02d450 auth_method 0 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9c2c03bbb0 0x7f9c2c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(188..188 src has 1..188) ==== 4815+0+0 (secure 0 0 0) 0x7f9c3c04f4e0 con 0x7f9c4c0a0c70 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_done global_id 4884 payload 824 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c4c1348e0 con 0x7f9c4c02d450 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c4ad9d640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9c2c03bbb0 0x7f9c2c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c4ad9d640 10 monclient: get_auth_request con 0x7f9c2c03bbb0 auth_method 0 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 msgr2=0x7f9c4c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.171+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 0x7f9c4c02d830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c06d160 tx=0x7f9c40019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c400ba070 con 0x7f9c4c02d450 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 10 monclient: dump: 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c400abe20 con 0x7f9c4c02d450 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 msgr2=0x7f9c4c02d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 0x7f9c4c02d830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c06d160 tx=0x7f9c40019100 comp rx=0 tx=0).stop 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 msgr2=0x7f9c4c02d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c3affd640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02d450 0x7f9c4c02d830 secure :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c06d160 tx=0x7f9c40019100 comp rx=0 tx=0).stop 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.175+0000 7f9c4ad9d640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9c2c03bbb0 0x7f9c2c03e070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009fd0 tx=0x7f9c4400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c509f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c509f7640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 0x7f9c4c02e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c509f7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4c073090 con 0x7f9c4c02dd70 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 0x7f9c4c02e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient: get_auth_request con 0x7f9c4c02dd70 auth_method 0 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 10 monclient(hunting): handle_auth_done global_id 4887 payload 824 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.179+0000 7f9c49d9b640 1 -- 172.21.15.86:0/2148676643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c4c09d3e0 con 0x7f9c4c02dd70 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c49d9b640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 msgr2=0x7f9c4c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c49d9b640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 0x7f9c4c02e180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9c4001cc00 tx=0x7f9c400a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c400cd020 con 0x7f9c4c02dd70 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 10 monclient: dump: 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f9c400b6450 con 0x7f9c4c02dd70 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 msgr2=0x7f9c4c02e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 0x7f9c4c02e180 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9c4001cc00 tx=0x7f9c400a6000 comp rx=0 tx=0).stop 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 msgr2=0x7f9c4c02e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c3affd640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c4c02dd70 0x7f9c4c02e180 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9c4001cc00 tx=0x7f9c400a6000 comp rx=0 tx=0).stop 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 355827, 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:03.267 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.268 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.269 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.270 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.271 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:03.272 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.273 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.274 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.275 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.276 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.277 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.278 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.279 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:03.280 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9c2c03bbb0 msgr2=0x7f9c2c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9c2c03bbb0 0x7f9c2c03e070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009fd0 tx=0x7f9c4400b040 comp rx=0 tx=0).stop 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 10 monclient: shutdown 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 msgr2=0x7f9c4c1cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 --2- 172.21.15.86:0/2148676643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c4c0a0c70 0x7f9c4c1cdca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c00c4a0 tx=0x7f9c3c00c960 comp rx=0 tx=0).stop 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c4ad9d640 1 -- 172.21.15.86:0/2148676643 reap_dead start 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c4ad9d640 1 -- 172.21.15.86:0/2148676643 reap_dead start 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 shutdown_connections 2024-09-17T15:33:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 >> 172.21.15.86:0/2148676643 conn(0x7f9c4c0a2f40 msgr2=0x7f9c4c131d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 shutdown_connections 2024-09-17T15:33:03.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.183+0000 7f9c509f7640 1 -- 172.21.15.86:0/2148676643 wait complete. 2024-09-17T15:33:03.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:03.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:33:03.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:33:03.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: get_monmap_and_config 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: monmap: 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:03.412448+0000 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:03.412448+0000 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:03.412 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:03.413 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:03.413 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:03.413 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 Processor -- start 2024-09-17T15:33:03.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 -- start start 2024-09-17T15:33:03.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: init 2024-09-17T15:33:03.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ce40a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ce40a1760 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7ce413c630 0x7f7ce413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce413c630 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce40a2130 con 0x7f7ce40a0c70 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ce40a26f0 con 0x7f7ce40a1760 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7ce40a2410 con 0x7f7ce413c630 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ceaee3640 10 monclient(hunting): authenticate will time out at 1316.378099s 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce8c7f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7ce40a0c70 0x7f7ce40a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:57398/0 (socket says 172.21.15.86:57398) 2024-09-17T15:33:03.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1706844485 learned_addr learned my addr 172.21.15.86:0/1706844485 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 1 --2- 172.21.15.86:0/1706844485 >> v2:172.21.15.203:3300/0 conn(0x7f7ce413c630 0x7f7ce413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): get_auth_request con 0x7f7ce413c630 auth_method 0 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.411+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1706844485 >> v1:172.21.15.86:6789/0 conn(0x7f7ce40a0c70 legacy=0x7f7ce40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 1 -- 172.21.15.86:0/1706844485 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077789645 0 0) 0x7f7ce40a2130 con 0x7f7ce40a0c70 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 10 monclient(hunting): _negotiate 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 10 monclient(hunting): my global_id is 4890 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce3fff640 1 -- 172.21.15.86:0/1706844485 >> v1:172.21.15.203:6789/0 conn(0x7f7ce40a1760 legacy=0x7f7ce413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce37fe640 1 -- 172.21.15.86:0/1706844485 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ccc003530 con 0x7f7ce40a0c70 2024-09-17T15:33:03.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 10 monclient(hunting): handle_auth_done global_id 54190 payload 274 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 1 -- 172.21.15.86:0/1706844485 >> v1:172.21.15.203:6789/0 conn(0x7f7ce40a1760 legacy=0x7f7ce413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 1 -- 172.21.15.86:0/1706844485 >> v1:172.21.15.86:6789/0 conn(0x7f7ce40a0c70 legacy=0x7f7ce40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 1 monclient: found mon.noname-c 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:33:03.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.415+0000 7f7ce9480640 1 -- 172.21.15.86:0/1706844485 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce409f9f0 con 0x7f7ce413c630 2024-09-17T15:33:03.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce9480640 1 -- 172.21.15.86:0/1706844485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce413c630 msgr2=0x7f7ce413ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce9480640 1 --2- 172.21.15.86:0/1706844485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce413c630 0x7f7ce413ea20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009e90 tx=0x7f7cdc01c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84fa8dfba7171125 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 1 -- 172.21.15.86:0/1706844485 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0b2070 con 0x7f7ce413c630 2024-09-17T15:33:03.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:03.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: dump: 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 1 -- 172.21.15.86:0/1706844485 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cdc0a5a10 con 0x7f7ce413c630 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 5 monclient: authenticate success, global_id 54190 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 1 -- 172.21.15.86:0/1706844485 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0a5de0 con 0x7f7ce413c630 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 4 set_mon_vals no callback set 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ce37fe640 10 monclient: dump: 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 10 monclient: shutdown 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1706844485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce413c630 msgr2=0x7f7ce413ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1706844485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce413c630 0x7f7ce413ea20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009e90 tx=0x7f7cdc01c930 comp rx=0 tx=0).stop 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1706844485 shutdown_connections 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1706844485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce413c630 0x7f7ce413ea20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1706844485 >> 172.21.15.86:0/1706844485 conn(0x7f7ce40a2f40 msgr2=0x7f7ce40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1706844485 shutdown_connections 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1706844485 wait complete. 2024-09-17T15:33:03.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.419+0000 7f7ceaee3640 10 monclient: monmap: 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 Processor -- start 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- start start 2024-09-17T15:33:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: init 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce40a0c70 0x7f7ce411c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-c con 0x7f7ce40a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce40a1760 0x7f7ce411ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-b con 0x7f7ce40a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 0x7f7ce411d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): picked mon.noname-a con 0x7f7ce413c630 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4141900 con 0x7f7ce413c630 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce4141780 con 0x7f7ce40a1760 2024-09-17T15:33:03.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce4141a80 con 0x7f7ce40a0c70 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient(hunting): authenticate will time out at 1316.388820s 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce40a0c70 0x7f7ce411c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 0x7f7ce411d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 0x7f7ce411d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55404/0 (socket says 172.21.15.86:55404) 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 learned_addr learned my addr 172.21.15.86:0/1103870835 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce40a0c70 0x7f7ce411c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47768/0 (socket says 172.21.15.86:47768) 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): get_auth_request con 0x7f7ce413c630 auth_method 0 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): get_auth_request con 0x7f7ce40a0c70 auth_method 0 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce3fff640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce40a1760 0x7f7ce411ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce3fff640 10 monclient(hunting): get_auth_request con 0x7f7ce40a1760 auth_method 0 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce40a0c70 msgr2=0x7f7ce411c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce40a0c70 0x7f7ce411c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce40a1760 msgr2=0x7f7ce411ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ce40a1760 0x7f7ce411ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 monclient: found mon.noname-a 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce411df70 con 0x7f7ce413c630 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 msgr2=0x7f7ce411d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce9480640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 0x7f7ce411d340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01a560 tx=0x7f7cdc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0bb020 con 0x7f7ce413c630 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: dump: 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cdc0ad850 con 0x7f7ce413c630 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0b2040 con 0x7f7ce413c630 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 5 monclient: authenticate success, global_id 4893 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _renew_subs 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce411f130 con 0x7f7ce413c630 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce2ffd640 4 set_mon_vals no callback set 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ce17fa640 10 monclient: dump: 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _renew_subs 2024-09-17T15:33:03.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.423+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce413bff0 con 0x7f7ce413c630 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ceaee3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 0x7f7ce41355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce413ca10 con 0x7f7ce4135200 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 0x7f7ce41355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient: get_auth_request con 0x7f7ce4135200 auth_method 0 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7cdc0be020 con 0x7f7ce413c630 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_done global_id 64148 payload 824 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ce40a1050 con 0x7f7ce4135200 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd003bb60 0x7f7cd003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(188..188 src has 1..188) ==== 4815+0+0 (secure 0 0 0) 0x7f7cdc0ea870 con 0x7f7ce413c630 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce3fff640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd003bb60 0x7f7cd003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce3fff640 10 monclient: get_auth_request con 0x7f7cd003bb60 auth_method 0 2024-09-17T15:33:03.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 msgr2=0x7f7ce41355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce8c7f640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 0x7f7ce41355e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4007410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4011870 con 0x7f7ce4135200 2024-09-17T15:33:03.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 10 monclient: dump: 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7cd4018070 con 0x7f7ce4135200 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 msgr2=0x7f7ce41355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 0x7f7ce41355e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4007410 comp rx=0 tx=0).stop 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 msgr2=0x7f7ce41355e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.427+0000 7f7ce17fa640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135200 0x7f7ce41355e0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4007410 comp rx=0 tx=0).stop 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.431+0000 7f7ce3fff640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd003bb60 0x7f7cd003e020 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ce411dba0 tx=0x7f7cd8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ceaee3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 0x7f7ce4136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ceaee3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce409d3e0 con 0x7f7ce4135b20 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 0x7f7ce4136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient: get_auth_request con 0x7f7ce4135b20 auth_method 0 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 10 monclient(hunting): handle_auth_done global_id 64151 payload 824 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.435+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1103870835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ce406d4f0 con 0x7f7ce4135b20 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce8c7f640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 msgr2=0x7f7ce4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce8c7f640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 0x7f7ce4136ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4001510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40047f0 con 0x7f7ce4135b20 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 10 monclient: dump: 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f7cd40047f0 con 0x7f7ce4135b20 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 msgr2=0x7f7ce4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 0x7f7ce4136ef0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4001510 comp rx=0 tx=0).stop 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 msgr2=0x7f7ce4136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce17fa640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ce4135b20 0x7f7ce4136ef0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40014e0 tx=0x7f7cd4001510 comp rx=0 tx=0).stop 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10039, 2024-09-17T15:33:03.446 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.447 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.448 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.449 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.450 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:03.451 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.452 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.453 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.454 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.456 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.457 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:03.458 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:03.459 INFO:tasks.rados.rados.0.smithi203.stdout:806: done (2 left) 2024-09-17T15:33:03.460 INFO:tasks.rados.rados.0.smithi203.stdout:808: done (1 left) 2024-09-17T15:33:03.460 INFO:tasks.rados.rados.0.smithi203.stdout:809: done (0 left) 2024-09-17T15:33:03.460 INFO:tasks.rados.rados.0.smithi203.stdout:810: read oid 39 snap -1 2024-09-17T15:33:03.460 INFO:tasks.rados.rados.0.smithi203.stdout:810: expect deleted 2024-09-17T15:33:03.460 INFO:tasks.rados.rados.0.smithi203.stdout:811: snap_create 2024-09-17T15:33:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd003bb60 msgr2=0x7f7cd003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd003bb60 0x7f7cd003e020 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7ce411dba0 tx=0x7f7cd8009040 comp rx=0 tx=0).stop 2024-09-17T15:33:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 10 monclient: shutdown 2024-09-17T15:33:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 msgr2=0x7f7ce411d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 --2- 172.21.15.86:0/1103870835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ce413c630 0x7f7ce411d340 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc01a560 tx=0x7f7cdc0196d0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 reap_dead start 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ce9480640 1 -- 172.21.15.86:0/1103870835 reap_dead start 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 shutdown_connections 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 >> 172.21.15.86:0/1103870835 conn(0x7f7ce40a2f40 msgr2=0x7f7ce40a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 shutdown_connections 2024-09-17T15:33:03.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.439+0000 7f7ceaee3640 1 -- 172.21.15.86:0/1103870835 wait complete. 2024-09-17T15:33:03.461 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:03.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:33:03.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:33:03.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: get_monmap_and_config 2024-09-17T15:33:03.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: monmap: 2024-09-17T15:33:03.575 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:03.576045+0000 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:03.576045+0000 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:03.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 Processor -- start 2024-09-17T15:33:03.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 -- start start 2024-09-17T15:33:03.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: init 2024-09-17T15:33:03.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7aec0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 0x7f7aec09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-b con 0x7f7aec0a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7aec13d2f0 0x7f7aec13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-a con 0x7f7aec13d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aec0a6280 con 0x7f7aec0a50e0 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7aec09d2a0 con 0x7f7aec13d2f0 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7aec09c0a0 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af3554640 10 monclient(hunting): authenticate will time out at 1316.541671s 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af12f0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7aec0a50e0 0x7f7aec0a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:57410/0 (socket says 172.21.15.86:57410) 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af12f0640 1 -- 172.21.15.86:0/2444773823 learned_addr learned my addr 172.21.15.86:0/2444773823 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af1af1640 1 --2- 172.21.15.86:0/2444773823 >> v2:172.21.15.86:3300/0 conn(0x7f7aec13d2f0 0x7f7aec13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 --2- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 0x7f7aec09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af1af1640 10 monclient(hunting): get_auth_request con 0x7f7aec13d2f0 auth_method 0 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af1af1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af1af1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af1af1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): get_auth_request con 0x7f7aec0a5bd0 auth_method 0 2024-09-17T15:33:03.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient(hunting): handle_auth_done global_id 64154 payload 274 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 -- 172.21.15.86:0/2444773823 >> v2:172.21.15.86:3300/0 conn(0x7f7aec13d2f0 msgr2=0x7f7aec13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 --2- 172.21.15.86:0/2444773823 >> v2:172.21.15.86:3300/0 conn(0x7f7aec13d2f0 0x7f7aec13f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 -- 172.21.15.86:0/2444773823 >> v1:172.21.15.86:6789/0 conn(0x7f7aec0a50e0 legacy=0x7f7aec0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 monclient: found mon.noname-b 2024-09-17T15:33:03.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.575+0000 7f7af0aef640 1 -- 172.21.15.86:0/2444773823 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec09c380 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af0aef640 1 -- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 msgr2=0x7f7aec09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af0aef640 1 --2- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 0x7f7aec09cd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0097c0 tx=0x7f7adc01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9301bdf6e2b8b0bc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 1 -- 172.21.15.86:0/2444773823 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0b0070 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: dump: 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.586 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 1 -- 172.21.15.86:0/2444773823 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc0ab600 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 5 monclient: authenticate success, global_id 64154 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 1 -- 172.21.15.86:0/2444773823 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc0a4e60 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 4 set_mon_vals no callback set 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7ae3fff640 10 monclient: dump: 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 10 monclient: shutdown 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 -- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 msgr2=0x7f7aec09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 --2- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 0x7f7aec09cd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0097c0 tx=0x7f7adc01c330 comp rx=0 tx=0).stop 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 -- 172.21.15.86:0/2444773823 shutdown_connections 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 --2- 172.21.15.86:0/2444773823 >> v2:172.21.15.86:3300/0 conn(0x7f7aec13d2f0 0x7f7aec13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 --2- 172.21.15.86:0/2444773823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec0a5bd0 0x7f7aec09cd60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 -- 172.21.15.86:0/2444773823 >> 172.21.15.86:0/2444773823 conn(0x7f7aec0aaae0 msgr2=0x7f7aec0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 -- 172.21.15.86:0/2444773823 shutdown_connections 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.579+0000 7f7af3554640 1 -- 172.21.15.86:0/2444773823 wait complete. 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient: monmap: 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:03.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 Processor -- start 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 -- start start 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient: init 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec0a50e0 0x7f7aec1213f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-b con 0x7f7aec0a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aec0a5bd0 0x7f7aec09fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-a con 0x7f7aec0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 0x7f7aec138260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): picked mon.noname-c con 0x7f7aec13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec142650 con 0x7f7aec0a5bd0 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec0a6b00 con 0x7f7aec0a50e0 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7aec1427d0 con 0x7f7aec13d2f0 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af3554640 10 monclient(hunting): authenticate will time out at 1316.549373s 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 0x7f7aec138260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 0x7f7aec138260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47796/0 (socket says 172.21.15.86:47796) 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 learned_addr learned my addr 172.21.15.86:0/4054058934 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): get_auth_request con 0x7f7aec13d2f0 auth_method 0 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aec0a5bd0 0x7f7aec09fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 10 monclient(hunting): get_auth_request con 0x7f7aec0a5bd0 auth_method 0 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af12f0640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec0a50e0 0x7f7aec1213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af12f0640 10 monclient(hunting): get_auth_request con 0x7f7aec0a50e0 auth_method 0 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af12f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af12f0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af12f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient(hunting): handle_auth_done global_id 64157 payload 824 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec0a50e0 msgr2=0x7f7aec1213f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec0a50e0 0x7f7aec1213f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aec0a5bd0 msgr2=0x7f7aec09fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aec0a5bd0 0x7f7aec09fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 monclient: found mon.noname-c 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec0a12b0 con 0x7f7aec13d2f0 2024-09-17T15:33:03.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.583+0000 7f7af0aef640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7aec0a5bd0 0x7f7aec09fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:33:03.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 msgr2=0x7f7aec138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af1af1640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 0x7f7aec138260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4007e60 tx=0x7f7ae400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4014620 con 0x7f7aec13d2f0 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: dump: 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.598 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 5 monclient: authenticate success, global_id 64157 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae4019070 con 0x7f7aec13d2f0 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 monclient: _renew_subs 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4014a00 con 0x7f7aec13d2f0 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec0a0d00 con 0x7f7aec13d2f0 2024-09-17T15:33:03.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae37fe640 4 set_mon_vals no callback set 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 10 monclient: dump: 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.600 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 monclient: _renew_subs 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec13d940 con 0x7f7aec13d2f0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 0x7f7aec1cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec06a0e0 con 0x7f7aec1387a0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7ae4014ba0 con 0x7f7aec13d2f0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 0x7f7aec1cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient: get_auth_request con 0x7f7aec1387a0 auth_method 0 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ad003bbb0 0x7f7ad003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7f7ae4050870 con 0x7f7aec13d2f0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af0aef640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ad003bbb0 0x7f7ad003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af0aef640 10 monclient: get_auth_request con 0x7f7ad003bbb0 auth_method 0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 10 monclient(hunting): handle_auth_done global_id 54202 payload 824 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.587+0000 7f7af12f0640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7aec09cd60 con 0x7f7aec1387a0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7af12f0640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 msgr2=0x7f7aec1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7af12f0640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 0x7f7aec1cfba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e500 tx=0x7f7ae800e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8004e40 con 0x7f7aec1387a0 2024-09-17T15:33:03.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 10 monclient: dump: 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7af0aef640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ad003bbb0 0x7f7ad003e070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0aa620 tx=0x7f7adc0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ae8018070 con 0x7f7aec1387a0 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 msgr2=0x7f7aec1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 0x7f7aec1cfba0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e500 tx=0x7f7ae800e9c0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 msgr2=0x7f7aec1cfba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.591+0000 7f7ae1ffb640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1387a0 0x7f7aec1cfba0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e500 tx=0x7f7ae800e9c0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af3554640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af3554640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 0x7f7aec1d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af3554640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec1d0a00 con 0x7f7aec1d00e0 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af12f0640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 0x7f7aec1d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af12f0640 10 monclient: get_auth_request con 0x7f7aec1d00e0 auth_method 0 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af12f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af12f0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.595+0000 7f7af12f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 10 monclient(hunting): handle_auth_done global_id 54205 payload 824 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 1 -- 172.21.15.86:0/4054058934 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7aec122350 con 0x7f7aec1d00e0 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 msgr2=0x7f7aec1d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7af12f0640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 0x7f7aec1d04c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e4d0 tx=0x7f7ae8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80064e0 con 0x7f7aec1d00e0 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 10 monclient: dump: 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f7ae80064e0 con 0x7f7aec1d00e0 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 msgr2=0x7f7aec1d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 0x7f7aec1d04c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e4d0 tx=0x7f7ae8004950 comp rx=0 tx=0).stop 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 msgr2=0x7f7aec1d04c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:03.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.599+0000 7f7ae1ffb640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7aec1d00e0 0x7f7aec1d04c0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e4d0 tx=0x7f7ae8004950 comp rx=0 tx=0).stop 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 62, 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 59030, 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.609 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.610 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.611 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:03.612 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:03.613 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.614 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.615 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.616 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:03.617 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.618 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.619 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:03.620 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:03.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ad003bbb0 msgr2=0x7f7ad003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ad003bbb0 0x7f7ad003e070 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0aa620 tx=0x7f7adc0a6000 comp rx=0 tx=0).stop 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 10 monclient: shutdown 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 msgr2=0x7f7aec138260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 --2- 172.21.15.86:0/4054058934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7aec13d2f0 0x7f7aec138260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4007e60 tx=0x7f7ae400e4b0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 reap_dead start 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af1af1640 1 -- 172.21.15.86:0/4054058934 reap_dead start 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 shutdown_connections 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 >> 172.21.15.86:0/4054058934 conn(0x7f7aec0aaae0 msgr2=0x7f7aec0a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 shutdown_connections 2024-09-17T15:33:03.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.603+0000 7f7af3554640 1 -- 172.21.15.86:0/4054058934 wait complete. 2024-09-17T15:33:03.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:03.623 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:33:03.623 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:33:03.623 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:33:03.623 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:33:03.624 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:33:03.624 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:33:03.635 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:33:03.635 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:33:03.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: get_monmap_and_config 2024-09-17T15:33:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: monmap: 2024-09-17T15:33:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:03.792206+0000 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:03.792206+0000 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:03.792 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 Processor -- start 2024-09-17T15:33:03.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 -- start start 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: init 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b6809d740 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b680a11f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b6813e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6b68135e80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b68069530 con 0x7f6b680a11f0 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b68069810 con 0x7f6b6809d740 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6809e230 con 0x7f6b68135e80 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6f8c8640 10 monclient(hunting): authenticate will time out at 1316.757836s 2024-09-17T15:33:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6de65640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 msgr2=0x7f6b6813e430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:03.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6de65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b6813e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:03.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6ce63640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6b680a11f0 0x7f6b6813ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:57420/0 (socket says 172.21.15.86:57420) 2024-09-17T15:33:03.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.791+0000 7f6b6ce63640 1 -- 172.21.15.86:0/3033288195 learned_addr learned my addr 172.21.15.86:0/3033288195 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6ce63640 1 -- 172.21.15.86:0/3033288195 >> v1:172.21.15.86:6789/0 conn(0x7f6b680a11f0 legacy=0x7f6b6813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2244428150 0 0) 0x7f6b68069530 con 0x7f6b680a11f0 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): _negotiate 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): my global_id is 4902 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b54003530 con 0x7f6b680a11f0 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 1 -- 172.21.15.86:0/3033288195 >> v1:172.21.15.203:6789/0 conn(0x7f6b6809d740 legacy=0x7f6b6809db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507961156 0 0) 0x7f6b54003530 con 0x7f6b680a11f0 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 msgr2=0x7f6b6813e430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:03.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b6813e430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 >> v1:172.21.15.203:6789/0 conn(0x7f6b6809d740 legacy=0x7f6b6809db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b680a1890 con 0x7f6b680a11f0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f6b580019b0 con 0x7f6b680a11f0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6f8c8640 5 monclient: authenticate success, global_id 4902 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6f8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: dump: 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:33:03.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 >> v1:172.21.15.86:6789/0 conn(0x7f6b680a11f0 legacy=0x7f6b6813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b54003ab0 0x7f6b54003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): picked mon.b con 0x7f6b54003ab0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b54004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): picked mon.c con 0x7f6b68135e80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 0x7f6b5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): picked mon.a con 0x7f6b54004fb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6809e230 con 0x7f6b54004fb0 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5400bf10 con 0x7f6b54003ab0 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5400c090 con 0x7f6b68135e80 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b5ffff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6ce63640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 msgr2=0x7f6b54004850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6ce63640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b54004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 0x7f6b5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): get_auth_request con 0x7f6b54004fb0 auth_method 0 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6f8c8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b54003ab0 0x7f6b54003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 10 monclient(hunting): get_auth_request con 0x7f6b54003ab0 auth_method 0 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6d664640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient(hunting): handle_auth_done global_id 4902 payload 293 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 msgr2=0x7f6b54004850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b68135e80 0x7f6b54004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b54003ab0 msgr2=0x7f6b54003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b54003ab0 0x7f6b54003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 monclient: found mon.a 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 1 -- 172.21.15.86:0/3033288195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b54003530 con 0x7f6b54004fb0 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.795+0000 7f6b6de65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 msgr2=0x7f6b5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 0x7f6b5400b9d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009e30 tx=0x7f6b6001caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc1aec8d671271e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60009a30 con 0x7f6b54004fb0 2024-09-17T15:33:03.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: dump: 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b600a9070 con 0x7f6b54004fb0 2024-09-17T15:33:03.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 1 -- 172.21.15.86:0/3033288195 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b600a5b20 con 0x7f6b54004fb0 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b5ffff640 10 monclient: dump: 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.811 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 4 set_mon_vals no callback set 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: shutdown 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 msgr2=0x7f6b5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/3033288195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b54004fb0 0x7f6b5400b9d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b60009e30 tx=0x7f6b6001caf0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 -- 172.21.15.86:0/3033288195 reap_dead start 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/3033288195 shutdown_connections 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/3033288195 >> 172.21.15.86:0/3033288195 conn(0x7f6b680a7460 msgr2=0x7f6b680a7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/3033288195 shutdown_connections 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/3033288195 wait complete. 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: build_initial_monmap 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: monmap: 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 Processor -- start 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- start start 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: init 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 0x7f6b6809e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f6b54004fb0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:03.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b6809eea0 0x7f6b6809f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f6b6809eea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6b6809f860 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6813f090 con 0x7f6b6809f860 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6d664640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 msgr2=0x7f6b6809e960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6813ef10 con 0x7f6b6809eea0 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6813f210 con 0x7f6b54004fb0 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6d664640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 0x7f6b6809e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6f8c8640 10 monclient(hunting): authenticate will time out at 1316.767812s 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55434/0 (socket says 172.21.15.86:55434) 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 1 -- 172.21.15.86:0/820671910 learned_addr learned my addr 172.21.15.86:0/820671910 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 10 monclient(hunting): get_auth_request con 0x7f6b6809f860 auth_method 0 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.799+0000 7f6b6de65640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6ce63640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b6809eea0 0x7f6b6809f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6ce63640 10 monclient(hunting): get_auth_request con 0x7f6b6809eea0 auth_method 0 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6ce63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6ce63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6ce63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:03.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient(hunting): handle_auth_done global_id 4905 payload 824 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 -- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 msgr2=0x7f6b6809e960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 0x7f6b6809e960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 -- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b6809eea0 msgr2=0x7f6b6809f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b6809eea0 0x7f6b6809f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient: _finish_hunting 0 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 monclient: found mon.noname-a 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 -- 172.21.15.86:0/820671910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b68139d40 con 0x7f6b6809f860 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 -- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 msgr2=0x7f6b681d6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6de65640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6b6001d390 tx=0x7f6b54005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b60009a30 con 0x7f6b6809f860 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:03.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: dump: 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b600aea90 con 0x7f6b6809f860 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b600aee40 con 0x7f6b6809f860 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5dffb640 10 monclient: dump: 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:03.818 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b5f7fe640 4 set_mon_vals no callback set 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 5 monclient: authenticate success, global_id 4905 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 10 monclient: _renew_subs 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b68139920 con 0x7f6b6809f860 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 10 monclient: _renew_subs 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.803+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b68138700 con 0x7f6b6809f860 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6f8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6f8c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6f8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6809db20 con 0x7f6b6809f860 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6b600b6070 con 0x7f6b6809f860 2024-09-17T15:33:03.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b5dffb640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 0x7f6b4c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7f6b600bd070 con 0x7f6b6809f860 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6d664640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 0x7f6b4c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6d664640 10 monclient: get_auth_request con 0x7f6b4c03bb60 auth_method 0 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.807+0000 7f6b6d664640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 0x7f6b4c03e020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b680a0b40 tx=0x7f6b64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.811+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6b68139d40 con 0x7f6b6809f860 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.811+0000 7f6b5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:03.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:03.811+0000 7f6b5dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:04.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b6f8c8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:04.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b6f8c8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:04.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b6f8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:04.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6b6809fcf0 con 0x7f6b6809f860 2024-09-17T15:33:04.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b5dffb640 1 -- 172.21.15.86:0/820671910 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6b600b4040 con 0x7f6b6809f860 2024-09-17T15:33:04.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:04.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.043+0000 7f6b5dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:04.046 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:04.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 msgr2=0x7f6b4c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 0x7f6b4c03e020 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b680a0b40 tx=0x7f6b64009040 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 10 monclient: shutdown 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 msgr2=0x7f6b681d6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6b6001d390 tx=0x7f6b54005840 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 shutdown_connections 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6b4c03bb60 0x7f6b4c03e020 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b6809f860 0x7f6b681d6420 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b6809eea0 0x7f6b6809f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 --2- 172.21.15.86:0/820671910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b54004fb0 0x7f6b6809e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 >> 172.21.15.86:0/820671910 conn(0x7f6b680a7460 msgr2=0x7f6b6809b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:04.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 shutdown_connections 2024-09-17T15:33:04.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:04.047+0000 7f6b6f8c8640 1 -- 172.21.15.86:0/820671910 wait complete. 2024-09-17T15:33:07.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:07.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: get_monmap_and_config 2024-09-17T15:33:07.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: build_initial_monmap 2024-09-17T15:33:07.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: monmap: 2024-09-17T15:33:07.196 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:07.196 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:07.197266+0000 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:07.197266+0000 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:07.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 Processor -- start 2024-09-17T15:33:07.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 -- start start 2024-09-17T15:33:07.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: init 2024-09-17T15:33:07.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c09ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-b con 0x7ff07c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff07c0a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff07c13d2f0 0x7ff07c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-c con 0x7ff07c13d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff07c09d350 con 0x7ff07c0a54c0 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff07c09bdc0 con 0x7ff07c13d2f0 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff07c0a6580 con 0x7ff07c0a50e0 2024-09-17T15:33:07.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff080d93640 10 monclient(hunting): authenticate will time out at 1320.163444s 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07a59c640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 msgr2=0x7ff07c09ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c09ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff07c13d2f0 0x7ff07c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff07c13d2f0 0x7ff07c13f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:33314/0 (socket says 172.21.15.86:33314) 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 1 -- 172.21.15.86:0/1175981613 learned_addr learned my addr 172.21.15.86:0/1175981613 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request con 0x7ff07c13d2f0 auth_method 0 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.195+0000 7ff07ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:07.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:07.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff079d9b640 1 -- 172.21.15.86:0/1175981613 >> v1:172.21.15.203:6789/0 conn(0x7ff07c0a54c0 legacy=0x7ff07c0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:07.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient(hunting): handle_auth_done global_id 54211 payload 274 2024-09-17T15:33:07.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 1 -- 172.21.15.86:0/1175981613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 msgr2=0x7ff07c09ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 1 --2- 172.21.15.86:0/1175981613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c09ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 1 -- 172.21.15.86:0/1175981613 >> v1:172.21.15.203:6789/0 conn(0x7ff07c0a54c0 legacy=0x7ff07c0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 1 monclient: found mon.noname-c 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:33:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.199+0000 7ff07ad9d640 1 -- 172.21.15.86:0/1175981613 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c09c150 con 0x7ff07c13d2f0 2024-09-17T15:33:07.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07ad9d640 1 -- 172.21.15.86:0/1175981613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 msgr2=0x7ff07c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:07.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07ad9d640 1 --2- 172.21.15.86:0/1175981613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c13f6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff074009f00 tx=0x7ff07401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9057e9d2b6dab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 1 -- 172.21.15.86:0/1175981613 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0740a5740 con 0x7ff07c13d2f0 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: dump: 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:07.207 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: _finish_auth 0 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: _check_auth_tickets 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 1 -- 172.21.15.86:0/1175981613 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0740a58e0 con 0x7ff07c13d2f0 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 1 -- 172.21.15.86:0/1175981613 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0740a5cb0 con 0x7ff07c13d2f0 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff080d93640 5 monclient: authenticate success, global_id 54211 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff080d93640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff080d93640 4 set_mon_vals no callback set 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:07.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.203+0000 7ff07959a640 10 monclient: dump: 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:07.209 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: shutdown 2024-09-17T15:33:07.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:07.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- 172.21.15.86:0/1175981613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 msgr2=0x7ff07c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:07.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- 172.21.15.86:0/1175981613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c13f6e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff074009f00 tx=0x7ff07401c970 comp rx=0 tx=0).stop 2024-09-17T15:33:07.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- 172.21.15.86:0/1175981613 shutdown_connections 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- 172.21.15.86:0/1175981613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c13f6e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- 172.21.15.86:0/1175981613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c09ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- 172.21.15.86:0/1175981613 >> 172.21.15.86:0/1175981613 conn(0x7ff07c0aaae0 msgr2=0x7ff07c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- 172.21.15.86:0/1175981613 shutdown_connections 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- 172.21.15.86:0/1175981613 wait complete. 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: build_initial_monmap 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: monmap: 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 Processor -- start 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- start start 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: init 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c137f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-c con 0x7ff07c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-a con 0x7ff07c0a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:07.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c0a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): picked mon.noname-b con 0x7ff07c13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c142880 con 0x7ff07c0a54c0 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c142700 con 0x7ff07c13d2f0 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff07c142a00 con 0x7ff07c0a50e0 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff07a59c640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 msgr2=0x7ff07c137f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff080d93640 10 monclient(hunting): authenticate will time out at 1320.174147s 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff07a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c137f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:07.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55444/0 (socket says 172.21.15.86:55444) 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 -- 172.21.15.86:0/3835087916 learned_addr learned my addr 172.21.15.86:0/3835087916 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): get_auth_request con 0x7ff07c0a54c0 auth_method 0 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff07ad9d640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c0a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient(hunting): handle_auth_done global_id 4908 payload 824 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 -- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 msgr2=0x7ff07c137f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c137f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 -- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 msgr2=0x7ff07c0a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c0a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 monclient: found mon.noname-a 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff079d9b640 1 -- 172.21.15.86:0/3835087916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c122820 con 0x7ff07c0a54c0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff07ad9d640 10 monclient: get_auth_request con 0x7ff07c13d2f0 auth_method 0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.207+0000 7ff07ad9d640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c0a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff079d9b640 1 -- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 msgr2=0x7ff07c1384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff079d9b640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff070001ec0 tx=0x7ff07000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070017070 con 0x7ff07c0a54c0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:07.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: dump: 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: _finish_auth 0 2024-09-17T15:33:07.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700057d0 con 0x7ff07c0a54c0 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 5 monclient: authenticate success, global_id 4908 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _renew_subs 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c0a18d0 con 0x7ff07c0a54c0 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff070005bc0 con 0x7ff07c0a54c0 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 10 monclient: dump: 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:07.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06bfff640 4 set_mon_vals no callback set 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _renew_subs 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:07.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c13f600 con 0x7ff07c0a54c0 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07c09ce10 con 0x7ff07c0a54c0 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff07001a070 con 0x7ff07c0a54c0 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 0x7ff05c03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:07.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7ff07004f380 con 0x7ff07c0a54c0 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff07a59c640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 0x7ff05c03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.211+0000 7ff07a59c640 10 monclient: get_auth_request con 0x7ff05c03bb40 auth_method 0 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.215+0000 7ff07a59c640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 0x7ff05c03e000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.215+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff070005de0 con 0x7ff07c0a54c0 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.215+0000 7ff06affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:07.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.215+0000 7ff06affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff080d93640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff080d93640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff080d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff07c122600 con 0x7ff07c0a54c0 2024-09-17T15:33:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff06affd640 1 -- 172.21.15.86:0/3835087916 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff07004f080 con 0x7ff07c0a54c0 2024-09-17T15:33:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff06affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.439+0000 7ff06affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:07.443 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 msgr2=0x7ff05c03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 0x7ff05c03e000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c006d40 comp rx=0 tx=0).stop 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 10 monclient: shutdown 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 msgr2=0x7ff07c1384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff070001ec0 tx=0x7ff07000c890 comp rx=0 tx=0).stop 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 shutdown_connections 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff05c03bb40 0x7ff05c03e000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff07c13d2f0 0x7ff07c0a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff07c0a54c0 0x7ff07c1384b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 --2- 172.21.15.86:0/3835087916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff07c0a50e0 0x7ff07c137f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 >> 172.21.15.86:0/3835087916 conn(0x7ff07c0aaae0 msgr2=0x7ff07c13ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 shutdown_connections 2024-09-17T15:33:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:07.443+0000 7ff080d93640 1 -- 172.21.15.86:0/3835087916 wait complete. 2024-09-17T15:33:10.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: get_monmap_and_config 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: build_initial_monmap 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: monmap: 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:10.593137+0000 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:10.593137+0000 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:10.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 Processor -- start 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 -- start start 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: init 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe618074c30 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6180690e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-b con 0x7fe61813b190 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61809cb80 con 0x7fe618074c30 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61809e410 con 0x7fe6180690e0 2024-09-17T15:33:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe61809d150 con 0x7fe61813b190 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61fe67640 10 monclient(hunting): authenticate will time out at 1323.558589s 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61e404640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 msgr2=0x7fe61813d580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61e404640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61dc03640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe618074c30 0x7fe6180689d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48504/0 (socket says 172.21.15.86:48504) 2024-09-17T15:33:10.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.591+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 learned_addr learned my addr 172.21.15.86:0/2555733142 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:10.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.595+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.86:6789/0 conn(0x7fe618074c30 legacy=0x7fe6180689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:10.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.595+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.203:6789/0 conn(0x7fe6180690e0 legacy=0x7fe618139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.795+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 msgr2=0x7fe61813d580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:10.795+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:33:11.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:11.199+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 msgr2=0x7fe61813d580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:11.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:11.199+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:33:12.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:12.003+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 msgr2=0x7fe61813d580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:12.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:12.003+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): tick 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 msgr2=0x7fe61813d580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe61813b190 0x7fe61813d580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.203:6789/0 conn(0x7fe6180690e0 legacy=0x7fe618139a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.86:6789/0 conn(0x7fe618074c30 legacy=0x7fe6180689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient: _add_conns ranks=[4,3] 2024-09-17T15:33:13.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fe5fc005a30 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fe5fc0085a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe61809d150 con 0x7fe5fc0085a0 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe5fc00b120 con 0x7fe5fc005a30 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe60f7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 10 monclient(hunting): get_auth_request con 0x7fe5fc0085a0 auth_method 0 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:13.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe5fc005a30 auth_method 0 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:13.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.591+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 10 monclient(hunting): get_auth_request con 0x7fe5fc0085a0 auth_method 0 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:13.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe5fc005a30 auth_method 0 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:13.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:13.795+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:33:14.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:14.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 10 monclient(hunting): get_auth_request con 0x7fe5fc0085a0 auth_method 0 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe5fc005a30 auth_method 0 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:14.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:14.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:14.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:14.199+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:33:15.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:15.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 10 monclient(hunting): get_auth_request con 0x7fe5fc0085a0 auth_method 0 2024-09-17T15:33:15.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:15.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe5fc005a30 auth_method 0 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:15.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:15.003+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): tick 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 msgr2=0x7fe5fc007eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.203:3300/0 conn(0x7fe5fc005a30 0x7fe5fc007eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 msgr2=0x7fe5fc00aa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc0085a0 0x7fe5fc00aa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 reap_dead start 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5fc005ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fe5fc00a170 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fe5fc00aa90 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe61809e6f0 con 0x7fe5fc005ea0 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:16.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe61809d150 con 0x7fe5fc00aa90 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 1 -- 172.21.15.86:0/2555733142 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe5fc00b120 con 0x7fe5fc00a170 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe60f7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe5fc00a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 10 monclient(hunting): get_auth_request con 0x7fe5fc00aa90 auth_method 0 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.591+0000 7fe61e404640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.595+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.595+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:16.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.595+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:16.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.595+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:16.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.595+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.86:6789/0 conn(0x7fe5fc005ea0 legacy=0x7fe5fc008a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:16.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.795+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe5fc00a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:16.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.795+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:33:16.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:16.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 10 monclient(hunting): get_auth_request con 0x7fe5fc00aa90 auth_method 0 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:16.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:16.799+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:33:17.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.199+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe5fc00a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:17.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.199+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 10 monclient(hunting): get_auth_request con 0x7fe5fc00aa90 auth_method 0 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:17.203+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:33:18.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.003+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe5fc00a550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.003+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 10 monclient(hunting): get_auth_request con 0x7fe5fc00aa90 auth_method 0 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:18.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.007+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472408280 0 0) 0x7fe61809e6f0 con 0x7fe5fc005ea0 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 10 monclient(hunting): _negotiate 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 10 monclient(hunting): my global_id is 4911 2024-09-17T15:33:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.503+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe604003530 con 0x7fe5fc005ea0 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1829930090 0 0) 0x7fe604003530 con 0x7fe5fc005ea0 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe5fc00a550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe5fc00a550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 msgr2=0x7fe5fc00f590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 --2- 172.21.15.86:0/2555733142 >> v2:172.21.15.86:3300/0 conn(0x7fe5fc00aa90 0x7fe5fc00f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5fc010cf0 con 0x7fe5fc005ea0 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61fe67640 5 monclient: authenticate success, global_id 4911 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61fe67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fe6140018f0 con 0x7fe5fc005ea0 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: dump: 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.511 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 >> v1:172.21.15.86:6789/0 conn(0x7fe5fc005ea0 legacy=0x7fe5fc008a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 0x7fe604003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): picked mon.b con 0x7fe604003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe604004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): picked mon.c con 0x7fe5fc00a170 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe604004da0 0x7fe604005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): picked mon.a con 0x7fe604004da0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe61809d150 con 0x7fe604004da0 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5fc00b120 con 0x7fe604003a30 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe60400bcd0 con 0x7fe5fc00a170 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61cc01640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61d402640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe604004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61fe67640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61d402640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe604004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe604004da0 0x7fe604005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): get_auth_request con 0x7fe604004da0 auth_method 0 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 0x7fe604003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe604003a30 auth_method 0 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:18.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient(hunting): handle_auth_done global_id 4911 payload 293 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 msgr2=0x7fe604004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5fc00a170 0x7fe604004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe604004da0 msgr2=0x7fe604005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe604004da0 0x7fe604005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 monclient: found mon.b 2024-09-17T15:33:18.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe604003530 con 0x7fe604003a30 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61dc03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.507+0000 7fe61e404640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe604004da0 0x7fe604005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:18.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.515+0000 7fe61dc03640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 msgr2=0x7fe604003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.515+0000 7fe61dc03640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 0x7fe604003e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe614007c10 tx=0x7fe61401d7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cabeb7d1b334b783 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140b4070 con 0x7fe604003a30 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: dump: 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.519 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140a9030 con 0x7fe604003a30 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 4 set_mon_vals no callback set 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 1 -- 172.21.15.86:0/2555733142 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140058c0 con 0x7fe604003a30 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:18.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61cc01640 10 monclient: dump: 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 10 monclient: shutdown 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 msgr2=0x7fe604003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 --2- 172.21.15.86:0/2555733142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe604003a30 0x7fe604003e50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe614007c10 tx=0x7fe61401d7f0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61e404640 1 -- 172.21.15.86:0/2555733142 reap_dead start 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- 172.21.15.86:0/2555733142 shutdown_connections 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- 172.21.15.86:0/2555733142 >> 172.21.15.86:0/2555733142 conn(0x7fe6180a6440 msgr2=0x7fe6180a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- 172.21.15.86:0/2555733142 shutdown_connections 2024-09-17T15:33:18.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- 172.21.15.86:0/2555733142 wait complete. 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 10 monclient: build_initial_monmap 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 10 monclient: monmap: 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 Processor -- start 2024-09-17T15:33:18.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 1 -- start start 2024-09-17T15:33:18.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.519+0000 7fe61fe67640 10 monclient: init 2024-09-17T15:33:18.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:18.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:18.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 0x7fe618121590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-b con 0x7fe6181211b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:18.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-a con 0x7fe618121ad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 0x7fe6181d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): picked mon.noname-c con 0x7fe6181d08c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe61809c400 con 0x7fe618121ad0 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe61809c280 con 0x7fe6181211b0 2024-09-17T15:33:18.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe61809c580 con 0x7fe6181d08c0 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient(hunting): authenticate will time out at 1331.488535s 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61e404640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 msgr2=0x7fe6181d0d50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61e404640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 0x7fe6181d0d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37042/0 (socket says 172.21.15.86:37042) 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 -- 172.21.15.86:0/1541772013 learned_addr learned my addr 172.21.15.86:0/1541772013 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): get_auth_request con 0x7fe618121ad0 auth_method 0 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 0x7fe618121590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 10 monclient(hunting): get_auth_request con 0x7fe6181211b0 auth_method 0 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient(hunting): handle_auth_done global_id 4914 payload 824 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 -- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 msgr2=0x7fe6181d0d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 0x7fe6181d0d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 -- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 msgr2=0x7fe618121590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 0x7fe618121590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 monclient: found mon.noname-a 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 -- 172.21.15.86:0/1541772013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe618124cf0 con 0x7fe618121ad0 2024-09-17T15:33:18.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61dc03640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 0x7fe618121590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 -- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 msgr2=0x7fe6181d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61d402640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe608012660 tx=0x7fe608012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe608019070 con 0x7fe618121ad0 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: dump: 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 5 monclient: authenticate success, global_id 4914 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _renew_subs 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe618124fd0 con 0x7fe618121ad0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _renew_subs 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6080072e0 con 0x7fe618121ad0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61809d460 con 0x7fe618121ad0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.523+0000 7fe60e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6080057d0 con 0x7fe618121ad0 2024-09-17T15:33:18.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 10 monclient: dump: 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60ffff640 4 set_mon_vals no callback set 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61fe67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61fe67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61fe67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe618068de0 con 0x7fe618121ad0 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe60801c070 con 0x7fe618121ad0 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 0x7fe5fc03f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7fe6080505b0 con 0x7fe618121ad0 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61dc03640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 0x7fe5fc03f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.527+0000 7fe61dc03640 10 monclient: get_auth_request con 0x7fe5fc03cf60 auth_method 0 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.531+0000 7fe61dc03640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 0x7fe5fc03f420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ae000 tx=0x7fe6140014e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.531+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe6080143a0 con 0x7fe618121ad0 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.531+0000 7fe60e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:18.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.531+0000 7fe60e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:18.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe61fe67640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:18.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe61fe67640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:18.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe61fe67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe61811dfd0 con 0x7fe618121ad0 2024-09-17T15:33:18.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe60e7fc640 1 -- 172.21.15.86:0/1541772013 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fe608027e10 con 0x7fe618121ad0 2024-09-17T15:33:18.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe60e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:18.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.775+0000 7fe60e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:18.778 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 msgr2=0x7fe5fc03f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 0x7fe5fc03f420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ae000 tx=0x7fe6140014e0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 10 monclient: shutdown 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 msgr2=0x7fe6181d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe608012660 tx=0x7fe608012b20 comp rx=0 tx=0).stop 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 shutdown_connections 2024-09-17T15:33:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5fc03cf60 0x7fe5fc03f420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6181d08c0 0x7fe6181d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe618121ad0 0x7fe6181d0380 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 --2- 172.21.15.86:0/1541772013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6181211b0 0x7fe618121590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 >> 172.21.15.86:0/1541772013 conn(0x7fe6180a6440 msgr2=0x7fe618073b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 shutdown_connections 2024-09-17T15:33:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.779+0000 7fe61fe67640 1 -- 172.21.15.86:0/1541772013 wait complete. 2024-09-17T15:33:18.797 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:33:18.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:33:18.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:33:18.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: get_monmap_and_config 2024-09-17T15:33:18.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: build_initial_monmap 2024-09-17T15:33:18.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: monmap: 2024-09-17T15:33:18.919 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:18.920452+0000 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:18.920452+0000 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:18.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 Processor -- start 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 -- start start 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: init 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff0bc0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-b con 0x7ff0bc0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0bc0a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0bc136160 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc0a2130 con 0x7ff0bc0a1590 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0bc0a26f0 con 0x7ff0bc136160 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff0bc0a2410 con 0x7ff0bc0a0c70 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0c2395640 10 monclient(hunting): authenticate will time out at 1331.885901s 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0bbfff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 msgr2=0x7ff0bc0a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0bbfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff0bc0a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0bb7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff0bc0a1590 0x7ff0bc135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:55728/0 (socket says 172.21.15.86:55728) 2024-09-17T15:33:18.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.919+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/230849389 learned_addr learned my addr 172.21.15.86:0/230849389 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/230849389 >> v1:172.21.15.86:6789/0 conn(0x7ff0bc0a1590 legacy=0x7ff0bc135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1883170045 0 0) 0x7ff0bc0a2130 con 0x7ff0bc0a1590 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): _negotiate 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): my global_id is 4917 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff09c003530 con 0x7ff0bc0a1590 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c0932640 1 -- 172.21.15.86:0/230849389 >> v1:172.21.15.203:6789/0 conn(0x7ff0bc136160 legacy=0x7ff0bc13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3031053442 0 0) 0x7ff09c003530 con 0x7ff0bc0a1590 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 msgr2=0x7ff0bc0a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff0bc0a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 >> v1:172.21.15.203:6789/0 conn(0x7ff0bc136160 legacy=0x7ff0bc13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc09f9f0 con 0x7ff0bc0a1590 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c2395640 5 monclient: authenticate success, global_id 4917 2024-09-17T15:33:18.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7ff0b4001d30 con 0x7ff0bc0a1590 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: dump: 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 >> v1:172.21.15.86:6789/0 conn(0x7ff0bc0a1590 legacy=0x7ff0bc135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 0x7ff09c003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): picked mon.a con 0x7ff09c003ab0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff09c004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): picked mon.c con 0x7ff0bc0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c004fb0 0x7ff09c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): picked mon.b con 0x7ff09c004fb0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc0a2410 con 0x7ff09c003ab0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c00bf10 con 0x7ff09c004fb0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff09c00c090 con 0x7ff0bc0a0c70 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0baffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c2395640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 msgr2=0x7ff09c004850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bb7fe640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff09c004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 0x7ff09c003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request con 0x7ff09c003ab0 auth_method 0 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c0932640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c004fb0 0x7ff09c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_done global_id 4917 payload 293 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 msgr2=0x7ff09c004850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc0a0c70 0x7ff09c004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c004fb0 msgr2=0x7ff09c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c004fb0 0x7ff09c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 monclient: found mon.a 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 1 -- 172.21.15.86:0/230849389 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c003530 con 0x7ff09c003ab0 2024-09-17T15:33:18.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c0932640 10 monclient: get_auth_request con 0x7ff09c004fb0 auth_method 0 2024-09-17T15:33:18.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.923+0000 7ff0c0932640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c004fb0 0x7ff09c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:18.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bbfff640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 msgr2=0x7ff09c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 0x7ff09c003ed0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4003990 tx=0x7ff0a401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d53e07f66565d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a4009520 con 0x7ff09c003ab0 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 10 monclient: dump: 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.939 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 0x7ff09c003ed0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4003990 tx=0x7ff0a401c420 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7ff0a4003bc0 config(0 keys) 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0a4003bc0 con 0x7ff09c003ab0 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 4 set_mon_vals no callback set 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: shutdown 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 msgr2=0x7ff09c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 --2- 172.21.15.86:0/230849389 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff09c003ab0 0x7ff09c003ed0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4003990 tx=0x7ff0a401c420 comp rx=0 tx=0).stop 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 1 -- 172.21.15.86:0/230849389 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7ff0a4005070 con 0x7ff09c003ab0 2024-09-17T15:33:18.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0baffd640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c0932640 1 -- 172.21.15.86:0/230849389 reap_dead start 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- 172.21.15.86:0/230849389 shutdown_connections 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- 172.21.15.86:0/230849389 >> 172.21.15.86:0/230849389 conn(0x7ff0bc0a2f40 msgr2=0x7ff0bc0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- 172.21.15.86:0/230849389 shutdown_connections 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- 172.21.15.86:0/230849389 wait complete. 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: build_initial_monmap 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: monmap: 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 Processor -- start 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- start start 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: init 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0bc13e720 0x7ff0bc09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-b con 0x7ff0bc13e720 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-a con 0x7ff0bc09dcb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc1d4dd0 0x7ff0bc1d71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): picked mon.noname-c con 0x7ff0bc1d4dd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc0a00d0 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc09ff50 con 0x7ff0bc13e720 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff0bc13fe60 con 0x7ff0bc1d4dd0 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c2395640 10 monclient(hunting): authenticate will time out at 1331.895723s 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c0932640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc1d4dd0 msgr2=0x7ff0bc1d71c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0c0932640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc1d4dd0 0x7ff0bc1d71c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37048/0 (socket says 172.21.15.86:37048) 2024-09-17T15:33:18.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/1950226190 learned_addr learned my addr 172.21.15.86:0/1950226190 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff0bc09dcb0 auth_method 0 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.927+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0bc13e720 0x7ff0bc09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request con 0x7ff0bc13e720 auth_method 0 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 10 monclient(hunting): handle_auth_done global_id 4920 payload 824 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc1d4dd0 msgr2=0x7ff0bc1d71c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0bc1d4dd0 0x7ff0bc1d71c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0bc13e720 msgr2=0x7ff0bc09d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0bc13e720 0x7ff0bc09d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 monclient: found mon.noname-a 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0bc11dc50 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 msgr2=0x7ff0bc09e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400cdf0 tx=0x7ff0b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bb7fe640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400cdf0 tx=0x7ff0b400a430 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7ff0b4018070 mon_map magic: 0 2024-09-17T15:33:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4018070 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: dump: 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: _finish_auth 0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b40072d0 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 5 monclient: authenticate success, global_id 4920 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 monclient: _renew_subs 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0bc09d460 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0ba7fc640 4 set_mon_vals no callback set 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 monclient: _renew_subs 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0bc09e980 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 0x7ff0bc0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc135a50 con 0x7ff0bc071100 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 0x7ff0bc0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient: get_auth_request con 0x7ff0bc071100 auth_method 0 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b40076c0 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 10 monclient: dump: 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff0b401b070 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_done global_id 4923 payload 824 2024-09-17T15:33:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.931+0000 7ff0bbfff640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0bc0008e0 con 0x7ff0bc071100 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0a003bb40 0x7ff0a003e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0c0932640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0a003bb40 0x7ff0a003e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0c0932640 10 monclient: get_auth_request con 0x7ff0a003bb40 auth_method 0 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0bbfff640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 msgr2=0x7ff0bc0714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 0x7ff0bc0714e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a40191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40be070 con 0x7ff0bc071100 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 10 monclient: dump: 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.950 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0a40052b0 con 0x7ff0bc071100 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 msgr2=0x7ff0bc0714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 0x7ff0bc0714e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a40191c0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 msgr2=0x7ff0bc0714e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0b8ff9640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071100 0x7ff0bc0714e0 secure :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a40191c0 comp rx=0 tx=0).stop 2024-09-17T15:33:18.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.935+0000 7ff0c0932640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0a003bb40 0x7ff0a003e000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0a1970 tx=0x7ff0ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0c2395640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0c2395640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 0x7ff0bc1db720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0c2395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff0bc11d8d0 con 0x7ff0bc071a20 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 0x7ff0bc1db720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient: get_auth_request con 0x7ff0bc071a20 auth_method 0 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 10 monclient(hunting): handle_auth_done global_id 4926 payload 824 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 1 -- 172.21.15.86:0/1950226190 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0bc11da80 con 0x7ff0bc071a20 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 msgr2=0x7ff0bc1db720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0bbfff640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 0x7ff0bc1db720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff0a401c460 tx=0x7ff09c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40be070 con 0x7ff0bc071a20 2024-09-17T15:33:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 10 monclient: dump: 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7ff0a40bf040 con 0x7ff0bc071a20 2024-09-17T15:33:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 msgr2=0x7ff0bc1db720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 0x7ff0bc1db720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff0a401c460 tx=0x7ff09c005840 comp rx=0 tx=0).stop 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 msgr2=0x7ff0bc1db720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.943+0000 7ff0b8ff9640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc071a20 0x7ff0bc1db720 secure :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff0a401c460 tx=0x7ff09c005840 comp rx=0 tx=0).stop 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:33:18.956 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 66, 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 371587, 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:18.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:18.958 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:18.959 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:18.960 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.961 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:18.963 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.965 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:18.966 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.967 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:18.968 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0a003bb40 msgr2=0x7ff0a003e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0a003bb40 0x7ff0a003e000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0a1970 tx=0x7ff0ac009040 comp rx=0 tx=0).stop 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 10 monclient: shutdown 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 msgr2=0x7ff0bc09e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 --2- 172.21.15.86:0/1950226190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0bc09dcb0 0x7ff0bc09e130 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400cdf0 tx=0x7ff0b400a430 comp rx=0 tx=0).stop 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c0932640 1 -- 172.21.15.86:0/1950226190 reap_dead start 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0b8ff9640 1 -- 172.21.15.86:0/1950226190 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (unknown 0 0 0) 0x7ff0b4050010 con 0x7ff0bc09dcb0 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c0932640 1 -- 172.21.15.86:0/1950226190 reap_dead start 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 shutdown_connections 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 >> 172.21.15.86:0/1950226190 conn(0x7ff0bc0a2f40 msgr2=0x7ff0bc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 shutdown_connections 2024-09-17T15:33:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:18.947+0000 7ff0c2395640 1 -- 172.21.15.86:0/1950226190 wait complete. 2024-09-17T15:33:18.971 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:33:18.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:33:18.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:33:19.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.079+0000 7fc6d5354640 10 monclient: get_monmap_and_config 2024-09-17T15:33:19.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.079+0000 7fc6d5354640 10 monclient: build_initial_monmap 2024-09-17T15:33:19.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.079+0000 7fc6d5354640 10 monclient: monmap: 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:19.083697+0000 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:19.083697+0000 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:19.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 Processor -- start 2024-09-17T15:33:19.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 -- start start 2024-09-17T15:33:19.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient: init 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6d00a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d00a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6d013d180 0x7fc6d013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d013d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d00a68d0 con 0x7fc6d00a4f70 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d00a64a0 con 0x7fc6d013d180 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d009c230 con 0x7fc6d00a5350 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:19.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6d5354640 10 monclient(hunting): authenticate will time out at 1332.049048s 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6c67fc640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 msgr2=0x7fc6d00a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6c67fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d00a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6ceffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc6d00a4f70 0x7fc6d009cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:55732/0 (socket says 172.21.15.86:55732) 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6ceffd640 1 -- 172.21.15.86:0/576724365 learned_addr learned my addr 172.21.15.86:0/576724365 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/576724365 >> v2:172.21.15.203:3300/0 conn(0x7fc6d013d180 0x7fc6d013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): get_auth_request con 0x7fc6d013d180 auth_method 0 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:19.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_done global_id 54223 payload 274 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/576724365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 msgr2=0x7fc6d00a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/576724365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d00a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/576724365 >> v1:172.21.15.86:6789/0 conn(0x7fc6d00a4f70 legacy=0x7fc6d009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 monclient: found mon.noname-c 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:33:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.083+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/576724365 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d009c510 con 0x7fc6d013d180 2024-09-17T15:33:19.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/576724365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d013d180 msgr2=0x7fc6d013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:19.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/576724365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d013d180 0x7fc6d013f570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009d90 tx=0x7fc6c801c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8b3ef9ff7d66e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 1 -- 172.21.15.86:0/576724365 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80b2070 con 0x7fc6d013d180 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: dump: 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: _finish_auth 0 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 1 -- 172.21.15.86:0/576724365 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c80a5890 con 0x7fc6d013d180 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 1 -- 172.21.15.86:0/576724365 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80a5c60 con 0x7fc6d013d180 2024-09-17T15:33:19.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6ce7fc640 10 monclient: dump: 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 5 monclient: authenticate success, global_id 54223 2024-09-17T15:33:19.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 4 set_mon_vals no callback set 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 10 monclient: shutdown 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 -- 172.21.15.86:0/576724365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d013d180 msgr2=0x7fc6d013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 --2- 172.21.15.86:0/576724365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d013d180 0x7fc6d013f570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009d90 tx=0x7fc6c801c7f0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 -- 172.21.15.86:0/576724365 shutdown_connections 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 --2- 172.21.15.86:0/576724365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d013d180 0x7fc6d013f570 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 --2- 172.21.15.86:0/576724365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d00a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.087+0000 7fc6d5354640 1 -- 172.21.15.86:0/576724365 >> 172.21.15.86:0/576724365 conn(0x7fc6d00a7240 msgr2=0x7fc6d00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- 172.21.15.86:0/576724365 shutdown_connections 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- 172.21.15.86:0/576724365 wait complete. 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient: build_initial_monmap 2024-09-17T15:33:19.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient: monmap: 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:19.095 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 Processor -- start 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- start start 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient: init 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d00a4f70 0x7fc6d0135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-b con 0x7fc6d00a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d0135590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-c con 0x7fc6d00a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 0x7fc6d01cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): picked mon.noname-a con 0x7fc6d013d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0142710 con 0x7fc6d013d180 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0142590 con 0x7fc6d00a4f70 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d0142890 con 0x7fc6d00a5350 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6d5354640 10 monclient(hunting): authenticate will time out at 1332.057566s 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6c67fc640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 msgr2=0x7fc6d0135590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6c67fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d0135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 0x7fc6d01cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 0x7fc6d01cfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37082/0 (socket says 172.21.15.86:37082) 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 learned_addr learned my addr 172.21.15.86:0/2202247381 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:19.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): get_auth_request con 0x7fc6d013d180 auth_method 0 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d00a4f70 0x7fc6d0135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient(hunting): handle_auth_done global_id 4929 payload 824 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 msgr2=0x7fc6d0135590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6d00a5350 0x7fc6d0135590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d00a4f70 msgr2=0x7fc6d0135050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d00a4f70 0x7fc6d0135050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:19.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 monclient: found mon.noname-a 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d00a0350 con 0x7fc6d013d180 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6ceffd640 10 monclient: get_auth_request con 0x7fc6d00a4f70 auth_method 0 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.091+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d00a4f70 0x7fc6d0135050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 msgr2=0x7fc6d01cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6cf7fe640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 0x7fc6d01cfc80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80b0040 tx=0x7fc6c80195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80bd020 con 0x7fc6d013d180 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: dump: 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: _finish_auth 0 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c80b8880 con 0x7fc6d013d180 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 5 monclient: authenticate success, global_id 4929 2024-09-17T15:33:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 monclient: _renew_subs 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d013da00 con 0x7fc6d013d180 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80b2040 con 0x7fc6d013d180 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6cdffb640 4 set_mon_vals no callback set 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 10 monclient: dump: 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.105 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 monclient: _renew_subs 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d00a1760 con 0x7fc6d013d180 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 0x7fc6d01d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d00a5750 con 0x7fc6d009ff50 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc6c80c0020 con 0x7fc6d013d180 2024-09-17T15:33:19.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc6b003bb70 0x7fc6b003e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 0x7fc6d01d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient: get_auth_request con 0x7fc6d009ff50 auth_method 0 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7fc6c80ead70 con 0x7fc6d013d180 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c67fc640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc6b003bb70 0x7fc6b003e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6c67fc640 10 monclient: get_auth_request con 0x7fc6b003bb70 auth_method 0 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_done global_id 54226 payload 824 2024-09-17T15:33:19.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.095+0000 7fc6ceffd640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6d00a5ff0 con 0x7fc6d009ff50 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c67fc640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc6b003bb70 0x7fc6b003e030 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc6d009d9c0 tx=0x7fc6b8006d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6ceffd640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 msgr2=0x7fc6d01d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 0x7fc6d01d01c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d4c0 tx=0x7fc6c000d980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0007070 con 0x7fc6d009ff50 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 10 monclient: dump: 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6c0017070 con 0x7fc6d009ff50 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 msgr2=0x7fc6d01d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 0x7fc6d01d01c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d4c0 tx=0x7fc6c000d980 comp rx=0 tx=0).stop 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 msgr2=0x7fc6d01d01c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.099+0000 7fc6c7fff640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d009ff50 0x7fc6d01d01c0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d4c0 tx=0x7fc6c000d980 comp rx=0 tx=0).stop 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.103+0000 7fc6d5354640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.103+0000 7fc6d5354640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 0x7fc6d00a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.103+0000 7fc6d5354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.103+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d01d0b10 con 0x7fc6d01d0700 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 0x7fc6d00a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient: get_auth_request con 0x7fc6d01d0700 auth_method 0 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 10 monclient(hunting): handle_auth_done global_id 54229 payload 824 2024-09-17T15:33:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 1 -- 172.21.15.86:0/2202247381 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6d0069920 con 0x7fc6d01d0700 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 msgr2=0x7fc6d00a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.107+0000 7fc6ceffd640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 0x7fc6d00a1930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0009010 tx=0x7fc6c0004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0007710 con 0x7fc6d01d0700 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 10 monclient: dump: 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fc6c0007710 con 0x7fc6d01d0700 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 msgr2=0x7fc6d00a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 0x7fc6d00a1930 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0009010 tx=0x7fc6c0004950 comp rx=0 tx=0).stop 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 msgr2=0x7fc6d00a1930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6c7fff640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6d01d0700 0x7fc6d00a1930 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0009010 tx=0x7fc6c0004950 comp rx=0 tx=0).stop 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 66, 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 74542, 2024-09-17T15:33:19.114 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:19.115 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:19.116 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:19.117 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:19.118 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:19.119 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.120 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:19.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:19.122 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.124 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:19.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:19.126 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc6b003bb70 msgr2=0x7fc6b003e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc6b003bb70 0x7fc6b003e030 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc6d009d9c0 tx=0x7fc6b8006d10 comp rx=0 tx=0).stop 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 10 monclient: shutdown 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 msgr2=0x7fc6d01cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 --2- 172.21.15.86:0/2202247381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6d013d180 0x7fc6d01cfc80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80b0040 tx=0x7fc6c80195f0 comp rx=0 tx=0).stop 2024-09-17T15:33:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 reap_dead start 2024-09-17T15:33:19.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6cf7fe640 1 -- 172.21.15.86:0/2202247381 reap_dead start 2024-09-17T15:33:19.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 shutdown_connections 2024-09-17T15:33:19.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 >> 172.21.15.86:0/2202247381 conn(0x7fc6d00a7240 msgr2=0x7fc6d013c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:19.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 shutdown_connections 2024-09-17T15:33:19.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:19.111+0000 7fc6d5354640 1 -- 172.21.15.86:0/2202247381 wait complete. 2024-09-17T15:33:19.129 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:33:19.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:33:19.496 INFO:tasks.rados.rados.0.smithi203.stdout:810: done (1 left) 2024-09-17T15:33:19.496 INFO:tasks.rados.rados.0.smithi203.stdout:811: done (0 left) 2024-09-17T15:33:19.496 INFO:tasks.rados.rados.0.smithi203.stdout:812: rollback oid 14 current snap is 87 2024-09-17T15:33:19.496 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 84 2024-09-17T15:33:19.496 INFO:tasks.rados.rados.0.smithi203.stdout:813: delete oid 27 current snap is 87 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:813: done (1 left) 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:814: read oid 34 snap -1 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:814: expect (ObjNum 270 snap 70 seq_num 270) 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:815: rollback oid 37 current snap is 87 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 79 2024-09-17T15:33:19.506 INFO:tasks.rados.rados.0.smithi203.stdout:816: snap_create 2024-09-17T15:33:19.509 INFO:tasks.rados.rados.0.smithi203.stdout:812: finishing rollback tid 0 to smithi20328086-14 2024-09-17T15:33:19.515 INFO:tasks.rados.rados.0.smithi203.stdout:814: expect (ObjNum 270 snap 70 seq_num 270) 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:812: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:812: finishing rollback tid 2 to smithi20328086-14 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 253 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:815: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:815: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:815: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:33:19.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 210 (ObjNum 271 snap 70 seq_num 271) dirty exists 2024-09-17T15:33:20.508 INFO:tasks.rados.rados.0.smithi203.stdout:812: done (3 left) 2024-09-17T15:33:20.508 INFO:tasks.rados.rados.0.smithi203.stdout:814: done (2 left) 2024-09-17T15:33:20.508 INFO:tasks.rados.rados.0.smithi203.stdout:815: done (1 left) 2024-09-17T15:33:20.508 INFO:tasks.rados.rados.0.smithi203.stdout:816: done (0 left) 2024-09-17T15:33:20.508 INFO:tasks.rados.rados.0.smithi203.stdout:817: delete oid 14 current snap is 88 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:817: done (0 left) 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:818: copy_from oid 19 from oid 22 current snap is 88 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:819: read oid 7 snap -1 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:819: expect (ObjNum 293 snap 80 seq_num 293) 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:820: rollback oid 16 current snap is 88 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 87 2024-09-17T15:33:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:821: snap_create 2024-09-17T15:33:20.522 INFO:tasks.rados.rados.0.smithi203.stdout:820: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:33:20.522 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 206 (ObjNum 1185 snap 0 seq_num 4108902640) dirty dne 2024-09-17T15:33:20.529 INFO:tasks.rados.rados.0.smithi203.stdout:819: expect (ObjNum 293 snap 80 seq_num 293) 2024-09-17T15:33:20.529 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:20.626 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:20.756 INFO:tasks.rados.rados.0.smithi203.stdout:818: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:33:20.756 INFO:tasks.rados.rados.0.smithi203.stdout:818: finishing copy_from to smithi20328086-19 2024-09-17T15:33:20.756 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:33:21.510 INFO:tasks.rados.rados.0.smithi203.stdout:818: done (3 left) 2024-09-17T15:33:21.510 INFO:tasks.rados.rados.0.smithi203.stdout:819: done (2 left) 2024-09-17T15:33:21.511 INFO:tasks.rados.rados.0.smithi203.stdout:820: done (1 left) 2024-09-17T15:33:21.511 INFO:tasks.rados.rados.0.smithi203.stdout:821: done (0 left) 2024-09-17T15:33:21.511 INFO:tasks.rados.rados.0.smithi203.stdout:822: snap_remove snap 68 2024-09-17T15:33:22.455 INFO:tasks.rados.rados.0.smithi203.stdout:822: done (0 left) 2024-09-17T15:33:22.455 INFO:tasks.rados.rados.0.smithi203.stdout:823: delete oid 34 current snap is 89 2024-09-17T15:33:22.485 INFO:tasks.rados.rados.0.smithi203.stdout:823: done (0 left) 2024-09-17T15:33:22.485 INFO:tasks.rados.rados.0.smithi203.stdout:824: snap_create 2024-09-17T15:33:23.458 INFO:tasks.rados.rados.0.smithi203.stdout:824: done (0 left) 2024-09-17T15:33:23.458 INFO:tasks.rados.rados.0.smithi203.stdout:825: read oid 22 snap 89 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:825: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:826: copy_from oid 24 from oid 25 current snap is 90 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:827: rollback oid 16 current snap is 90 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 86 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:828: copy_from oid 15 from oid 1 current snap is 90 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:829: copy_from oid 48 from oid 18 current snap is 90 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:830: write oid 23 current snap is 90 2024-09-17T15:33:23.459 INFO:tasks.rados.rados.0.smithi203.stdout:830: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-17T15:33:23.475 INFO:tasks.rados.rados.0.smithi203.stdout:830: writing smithi20328086-23 from 426977 to 883536 tid 1 2024-09-17T15:33:23.500 INFO:tasks.rados.rados.0.smithi203.stdout:830: writing smithi20328086-23 from 1482841 to 2282605 tid 2 2024-09-17T15:33:23.515 INFO:tasks.rados.rados.0.smithi203.stdout:830: writing smithi20328086-23 from 2958080 to 3226977 tid 3 2024-09-17T15:33:23.515 INFO:tasks.rados.rados.0.smithi203.stdout:831: delete oid 49 current snap is 90 2024-09-17T15:33:23.515 INFO:tasks.rados.rados.0.smithi203.stdout:825: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:23.520 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1420734 2024-09-17T15:33:23.580 INFO:tasks.rados.rados.0.smithi203.stdout:827: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:33:23.580 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 206 (ObjNum 129 snap 0 seq_num 3605662832) dirty dne 2024-09-17T15:33:23.580 INFO:tasks.rados.rados.0.smithi203.stdout:830: finishing write tid 1 to smithi20328086-23 2024-09-17T15:33:23.580 INFO:tasks.rados.rados.0.smithi203.stdout:826: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:826: finishing copy_from to smithi20328086-24 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 261 (ObjNum 257 snap 68 seq_num 257) dirty exists 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:828: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:828: finishing copy_from to smithi20328086-15 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 249 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:829: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: finishing write tid 2 to smithi20328086-23 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:829: finishing copy_from to smithi20328086-48 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 103 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: finishing write tid 3 to smithi20328086-23 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: finishing write tid 4 to smithi20328086-23 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: finishing write tid 5 to smithi20328086-23 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: oid 23 updating version 0 to 267 2024-09-17T15:33:23.581 INFO:tasks.rados.rados.0.smithi203.stdout:830: oid 23 version 267 is already newer than 266 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:830: oid 23 version 267 is already newer than 265 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:830: oid 23 version 267 is already newer than 264 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 267 (ObjNum 311 snap 90 seq_num 311) dirty exists 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:830: left oid 23 (ObjNum 311 snap 90 seq_num 311) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:825: done (6 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:826: done (5 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:827: done (4 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:828: done (3 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:829: done (2 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:830: done (1 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:831: done (0 left) 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:832: write (excl) oid 40 current snap is 90 2024-09-17T15:33:23.582 INFO:tasks.rados.rados.0.smithi203.stdout:832: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-17T15:33:23.597 INFO:tasks.rados.rados.0.smithi203.stdout:832: writing smithi20328086-40 from 460591 to 1184001 tid 1 2024-09-17T15:33:23.610 INFO:tasks.rados.rados.0.smithi203.stdout:832: writing smithi20328086-40 from 1716158 to 2119591 tid 2 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:832: writing smithi20328086-40 from 2729279 to 3169419 tid 3 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:833: read oid 19 snap 89 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:833: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:834: delete oid 16 current snap is 90 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:832: finishing write tid 1 to smithi20328086-40 2024-09-17T15:33:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:832: finishing write tid 2 to smithi20328086-40 2024-09-17T15:33:23.630 INFO:tasks.rados.rados.0.smithi203.stdout:832: finishing write tid 3 to smithi20328086-40 2024-09-17T15:33:23.631 INFO:tasks.rados.rados.0.smithi203.stdout:832: finishing write tid 4 to smithi20328086-40 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: finishing write tid 5 to smithi20328086-40 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: oid 40 updating version 0 to 256 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: oid 40 updating version 256 to 257 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: oid 40 updating version 257 to 258 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: oid 40 updating version 258 to 259 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 259 (ObjNum 313 snap 90 seq_num 313) dirty exists 2024-09-17T15:33:23.632 INFO:tasks.rados.rados.0.smithi203.stdout:832: left oid 40 (ObjNum 313 snap 90 seq_num 313) 2024-09-17T15:33:23.633 INFO:tasks.rados.rados.0.smithi203.stdout:833: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:23.633 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 375281 2024-09-17T15:33:23.656 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 677045 2024-09-17T15:33:23.677 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 217238 2024-09-17T15:33:23.702 INFO:tasks.rados.rados.0.smithi203.stdout:832: done (2 left) 2024-09-17T15:33:23.702 INFO:tasks.rados.rados.0.smithi203.stdout:833: done (1 left) 2024-09-17T15:33:23.702 INFO:tasks.rados.rados.0.smithi203.stdout:834: done (0 left) 2024-09-17T15:33:23.703 INFO:tasks.rados.rados.0.smithi203.stdout:835: read oid 22 snap 87 2024-09-17T15:33:23.703 INFO:tasks.rados.rados.0.smithi203.stdout:835: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:23.703 INFO:tasks.rados.rados.0.smithi203.stdout:836: snap_create 2024-09-17T15:33:23.706 INFO:tasks.rados.rados.0.smithi203.stdout:835: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:24.467 INFO:tasks.rados.rados.0.smithi203.stdout:835: done (1 left) 2024-09-17T15:33:24.467 INFO:tasks.rados.rados.0.smithi203.stdout:836: done (0 left) 2024-09-17T15:33:24.467 INFO:tasks.rados.rados.0.smithi203.stdout:837: rollback oid 24 current snap is 91 2024-09-17T15:33:24.467 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 88 2024-09-17T15:33:24.467 INFO:tasks.rados.rados.0.smithi203.stdout:838: read oid 5 snap 90 2024-09-17T15:33:24.468 INFO:tasks.rados.rados.0.smithi203.stdout:838: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:33:24.468 INFO:tasks.rados.rados.0.smithi203.stdout:839: read oid 11 snap -1 2024-09-17T15:33:24.468 INFO:tasks.rados.rados.0.smithi203.stdout:839: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:33:24.468 INFO:tasks.rados.rados.0.smithi203.stdout:840: snap_create 2024-09-17T15:33:24.484 INFO:tasks.rados.rados.0.smithi203.stdout:837: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:33:24.484 INFO:tasks.rados.rados.0.smithi203.stdout:837: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:33:24.484 INFO:tasks.rados.rados.0.smithi203.stdout:837: finishing rollback tid 2 to smithi20328086-24 2024-09-17T15:33:24.484 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 267 (ObjNum 226 snap 57 seq_num 226) dirty exists 2024-09-17T15:33:24.486 INFO:tasks.rados.rados.0.smithi203.stdout:839: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:33:24.548 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:24.590 INFO:tasks.rados.rados.0.smithi203.stdout:838: expect (ObjNum 205 snap 51 seq_num 205) 2024-09-17T15:33:24.590 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:24.661 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:25.471 INFO:tasks.rados.rados.0.smithi203.stdout:837: done (3 left) 2024-09-17T15:33:25.471 INFO:tasks.rados.rados.0.smithi203.stdout:838: done (2 left) 2024-09-17T15:33:25.471 INFO:tasks.rados.rados.0.smithi203.stdout:839: done (1 left) 2024-09-17T15:33:25.471 INFO:tasks.rados.rados.0.smithi203.stdout:840: done (0 left) 2024-09-17T15:33:25.471 INFO:tasks.rados.rados.0.smithi203.stdout:841: delete oid 39 current snap is 92 2024-09-17T15:33:25.483 INFO:tasks.rados.rados.0.smithi203.stdout:841: done (0 left) 2024-09-17T15:33:25.483 INFO:tasks.rados.rados.0.smithi203.stdout:842: snap_create 2024-09-17T15:33:26.489 INFO:tasks.rados.rados.0.smithi203.stdout:842: done (0 left) 2024-09-17T15:33:26.489 INFO:tasks.rados.rados.0.smithi203.stdout:843: rollback oid 28 current snap is 93 2024-09-17T15:33:26.489 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 28 to 89 2024-09-17T15:33:26.489 INFO:tasks.rados.rados.0.smithi203.stdout:844: delete oid 38 current snap is 93 2024-09-17T15:33:26.496 INFO:tasks.rados.rados.0.smithi203.stdout:843: finishing rollback tid 1 to smithi20328086-28 2024-09-17T15:33:26.496 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 329 (ObjNum 929 snap 0 seq_num 3605701152) dirty dne 2024-09-17T15:33:26.500 INFO:tasks.rados.rados.0.smithi203.stdout:843: done (1 left) 2024-09-17T15:33:26.500 INFO:tasks.rados.rados.0.smithi203.stdout:844: done (0 left) 2024-09-17T15:33:26.501 INFO:tasks.rados.rados.0.smithi203.stdout:845: copy_from oid 39 from oid 14 current snap is 93 2024-09-17T15:33:26.501 INFO:tasks.rados.rados.0.smithi203.stdout:846: copy_from oid 18 from oid 2 current snap is 93 2024-09-17T15:33:26.501 INFO:tasks.rados.rados.0.smithi203.stdout:847: copy_from oid 45 from oid 30 current snap is 93 2024-09-17T15:33:26.501 INFO:tasks.rados.rados.0.smithi203.stdout:848: rollback oid 29 current snap is 93 2024-09-17T15:33:26.501 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 29 to 82 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:849: read oid 15 snap -1 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:849: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:850: copy_from oid 37 from oid 36 current snap is 93 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:851: read oid 11 snap 92 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:851: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:852: read oid 17 snap -1 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:852: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:33:26.502 INFO:tasks.rados.rados.0.smithi203.stdout:853: delete oid 8 current snap is 93 2024-09-17T15:33:26.503 INFO:tasks.rados.rados.0.smithi203.stdout:845: finishing copy_from to smithi20328086-39 2024-09-17T15:33:26.503 INFO:tasks.rados.rados.0.smithi203.stdout:845: got expected ENOENT (src dne) 2024-09-17T15:33:26.503 INFO:tasks.rados.rados.0.smithi203.stdout:845: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:33:26.507 INFO:tasks.rados.rados.0.smithi203.stdout:846: finishing copy_from to smithi20328086-18 2024-09-17T15:33:26.507 INFO:tasks.rados.rados.0.smithi203.stdout:846: got expected ENOENT (src dne) 2024-09-17T15:33:26.508 INFO:tasks.rados.rados.0.smithi203.stdout:846: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:33:26.512 INFO:tasks.rados.rados.0.smithi203.stdout:849: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:26.512 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:26.562 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1889407 2024-09-17T15:33:26.608 INFO:tasks.rados.rados.0.smithi203.stdout:848: finishing rollback tid 0 to smithi20328086-29 2024-09-17T15:33:26.608 INFO:tasks.rados.rados.0.smithi203.stdout:848: finishing rollback tid 1 to smithi20328086-29 2024-09-17T15:33:26.609 INFO:tasks.rados.rados.0.smithi203.stdout:848: finishing rollback tid 2 to smithi20328086-29 2024-09-17T15:33:26.609 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 270 (ObjNum 284 snap 75 seq_num 284) dirty exists 2024-09-17T15:33:26.609 INFO:tasks.rados.rados.0.smithi203.stdout:851: expect (ObjNum 226 snap 57 seq_num 226) 2024-09-17T15:33:26.687 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:26.726 INFO:tasks.rados.rados.0.smithi203.stdout:852: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:33:26.836 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:850: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:847: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:847: finishing copy_from to smithi20328086-45 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:850: finishing copy_from to smithi20328086-37 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:845: done (8 left) 2024-09-17T15:33:26.892 INFO:tasks.rados.rados.0.smithi203.stdout:846: done (7 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:847: done (6 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:848: done (5 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:849: done (4 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:850: done (3 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:851: done (2 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:852: done (1 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:853: done (0 left) 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:854: write (excl) oid 20 current snap is 93 2024-09-17T15:33:26.893 INFO:tasks.rados.rados.0.smithi203.stdout:854: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-17T15:33:26.898 INFO:tasks.rados.rados.0.smithi203.stdout:854: writing smithi20328086-20 from 544626 to 1244281 tid 1 2024-09-17T15:33:26.905 INFO:tasks.rados.rados.0.smithi203.stdout:854: writing smithi20328086-20 from 1892215 to 2446644 tid 2 2024-09-17T15:33:26.911 INFO:tasks.rados.rados.0.smithi203.stdout:854: writing smithi20328086-20 from 3150041 to 3344626 tid 3 2024-09-17T15:33:26.911 INFO:tasks.rados.rados.0.smithi203.stdout:855: write oid 47 current snap is 93 2024-09-17T15:33:26.911 INFO:tasks.rados.rados.0.smithi203.stdout:855: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-17T15:33:26.919 INFO:tasks.rados.rados.0.smithi203.stdout:855: writing smithi20328086-47 from 561433 to 1336337 tid 1 2024-09-17T15:33:26.926 INFO:tasks.rados.rados.0.smithi203.stdout:855: writing smithi20328086-47 from 1950697 to 2365137 tid 2 2024-09-17T15:33:26.932 INFO:tasks.rados.rados.0.smithi203.stdout:855: writing smithi20328086-47 from 2777464 to 3361433 tid 3 2024-09-17T15:33:26.932 INFO:tasks.rados.rados.0.smithi203.stdout:856: write oid 31 current snap is 93 2024-09-17T15:33:26.932 INFO:tasks.rados.rados.0.smithi203.stdout:856: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-17T15:33:26.939 INFO:tasks.rados.rados.0.smithi203.stdout:856: writing smithi20328086-31 from 578240 to 1144746 tid 1 2024-09-17T15:33:26.946 INFO:tasks.rados.rados.0.smithi203.stdout:856: writing smithi20328086-31 from 1609179 to 2167277 tid 2 2024-09-17T15:33:26.953 INFO:tasks.rados.rados.0.smithi203.stdout:856: writing smithi20328086-31 from 2804887 to 3356637 tid 3 2024-09-17T15:33:26.953 INFO:tasks.rados.rados.0.smithi203.stdout:857: read oid 48 snap -1 2024-09-17T15:33:26.953 INFO:tasks.rados.rados.0.smithi203.stdout:857: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:33:26.954 INFO:tasks.rados.rados.0.smithi203.stdout:858: write oid 45 current snap is 93 2024-09-17T15:33:26.954 INFO:tasks.rados.rados.0.smithi203.stdout:858: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-17T15:33:26.961 INFO:tasks.rados.rados.0.smithi203.stdout:858: writing smithi20328086-45 from 595047 to 1236802 tid 1 2024-09-17T15:33:26.966 INFO:tasks.rados.rados.0.smithi203.stdout:858: writing smithi20328086-45 from 1667661 to 2085770 tid 2 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:858: writing smithi20328086-45 from 2548663 to 3311685 tid 3 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:859: delete oid 27 current snap is 93 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:854: finishing write tid 1 to smithi20328086-20 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:854: finishing write tid 2 to smithi20328086-20 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:854: finishing write tid 3 to smithi20328086-20 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:854: finishing write tid 4 to smithi20328086-20 2024-09-17T15:33:26.974 INFO:tasks.rados.rados.0.smithi203.stdout:854: finishing write tid 5 to smithi20328086-20 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:854: oid 20 updating version 0 to 270 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:854: oid 20 updating version 270 to 272 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:854: oid 20 version 272 is already newer than 271 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:854: oid 20 version 272 is already newer than 269 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 272 (ObjNum 318 snap 93 seq_num 318) dirty exists 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:854: left oid 20 (ObjNum 318 snap 93 seq_num 318) 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:855: finishing write tid 1 to smithi20328086-47 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:855: finishing write tid 2 to smithi20328086-47 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:855: finishing write tid 3 to smithi20328086-47 2024-09-17T15:33:26.975 INFO:tasks.rados.rados.0.smithi203.stdout:855: finishing write tid 4 to smithi20328086-47 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: finishing write tid 5 to smithi20328086-47 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: oid 47 updating version 0 to 172 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: oid 47 updating version 172 to 175 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: oid 47 version 175 is already newer than 174 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: oid 47 version 175 is already newer than 173 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 175 (ObjNum 319 snap 93 seq_num 319) dirty exists 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:855: left oid 47 (ObjNum 319 snap 93 seq_num 319) 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:856: finishing write tid 1 to smithi20328086-31 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:856: finishing write tid 2 to smithi20328086-31 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:856: finishing write tid 3 to smithi20328086-31 2024-09-17T15:33:26.976 INFO:tasks.rados.rados.0.smithi203.stdout:857: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:33:27.163 INFO:tasks.rados.rados.0.smithi203.stdout:856: finishing write tid 4 to smithi20328086-31 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: finishing write tid 5 to smithi20328086-31 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: oid 31 updating version 0 to 100 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: oid 31 updating version 100 to 101 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: oid 31 updating version 101 to 103 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: oid 31 version 103 is already newer than 102 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 103 (ObjNum 320 snap 93 seq_num 320) dirty exists 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: left oid 31 (ObjNum 320 snap 93 seq_num 320) 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:858: finishing write tid 1 to smithi20328086-45 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:854: done (5 left) 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:855: done (4 left) 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:856: done (3 left) 2024-09-17T15:33:27.164 INFO:tasks.rados.rados.0.smithi203.stdout:857: done (2 left) 2024-09-17T15:33:27.165 INFO:tasks.rados.rados.0.smithi203.stdout:859: done (1 left) 2024-09-17T15:33:27.165 INFO:tasks.rados.rados.0.smithi203.stdout:860: read oid 16 snap -1 2024-09-17T15:33:27.165 INFO:tasks.rados.rados.0.smithi203.stdout:860: expect deleted 2024-09-17T15:33:27.165 INFO:tasks.rados.rados.0.smithi203.stdout:861: write (excl) oid 38 current snap is 93 2024-09-17T15:33:27.165 INFO:tasks.rados.rados.0.smithi203.stdout:861: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-17T15:33:27.171 INFO:tasks.rados.rados.0.smithi203.stdout:861: writing smithi20328086-38 from 628661 to 1420914 tid 1 2024-09-17T15:33:27.178 INFO:tasks.rados.rados.0.smithi203.stdout:861: writing smithi20328086-38 from 2184625 to 2606403 tid 2 2024-09-17T15:33:27.182 INFO:tasks.rados.rados.0.smithi203.stdout:861: writing smithi20328086-38 from 3403509 to 3428661 tid 3 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:862: snap_create 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:858: finishing write tid 2 to smithi20328086-45 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:858: finishing write tid 3 to smithi20328086-45 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:858: finishing write tid 4 to smithi20328086-45 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:858: finishing write tid 5 to smithi20328086-45 2024-09-17T15:33:27.183 INFO:tasks.rados.rados.0.smithi203.stdout:858: oid 45 updating version 0 to 123 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:858: oid 45 version 123 is already newer than 122 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:858: oid 45 version 123 is already newer than 121 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:858: oid 45 version 123 is already newer than 120 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 123 (ObjNum 321 snap 93 seq_num 321) dirty exists 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:858: left oid 45 (ObjNum 321 snap 93 seq_num 321) 2024-09-17T15:33:27.184 INFO:tasks.rados.rados.0.smithi203.stdout:861: finishing write tid 1 to smithi20328086-38 2024-09-17T15:33:27.186 INFO:tasks.rados.rados.0.smithi203.stdout:861: finishing write tid 2 to smithi20328086-38 2024-09-17T15:33:27.186 INFO:tasks.rados.rados.0.smithi203.stdout:861: finishing write tid 3 to smithi20328086-38 2024-09-17T15:33:27.186 INFO:tasks.rados.rados.0.smithi203.stdout:861: finishing write tid 4 to smithi20328086-38 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: finishing write tid 5 to smithi20328086-38 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: oid 38 updating version 0 to 172 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: oid 38 updating version 172 to 175 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: oid 38 version 175 is already newer than 173 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: oid 38 version 175 is already newer than 174 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 175 (ObjNum 323 snap 93 seq_num 323) dirty exists 2024-09-17T15:33:27.187 INFO:tasks.rados.rados.0.smithi203.stdout:861: left oid 38 (ObjNum 323 snap 93 seq_num 323) 2024-09-17T15:33:27.456 INFO:tasks.rados.rados.0.smithi203.stdout:858: done (3 left) 2024-09-17T15:33:27.456 INFO:tasks.rados.rados.0.smithi203.stdout:860: done (2 left) 2024-09-17T15:33:27.456 INFO:tasks.rados.rados.0.smithi203.stdout:861: done (1 left) 2024-09-17T15:33:27.457 INFO:tasks.rados.rados.0.smithi203.stdout:862: done (0 left) 2024-09-17T15:33:27.457 INFO:tasks.rados.rados.0.smithi203.stdout:863: delete oid 30 current snap is 94 2024-09-17T15:33:27.468 INFO:tasks.rados.rados.0.smithi203.stdout:863: done (0 left) 2024-09-17T15:33:27.468 INFO:tasks.rados.rados.0.smithi203.stdout:864: snap_remove snap 83 2024-09-17T15:33:28.466 INFO:tasks.rados.rados.0.smithi203.stdout:864: done (0 left) 2024-09-17T15:33:28.466 INFO:tasks.rados.rados.0.smithi203.stdout:865: copy_from oid 50 from oid 18 current snap is 94 2024-09-17T15:33:28.466 INFO:tasks.rados.rados.0.smithi203.stdout:866: write (excl) oid 44 current snap is 94 2024-09-17T15:33:28.466 INFO:tasks.rados.rados.0.smithi203.stdout:866: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-17T15:33:28.478 INFO:tasks.rados.rados.0.smithi203.stdout:866: writing smithi20328086-44 from 662275 to 1205026 tid 1 2024-09-17T15:33:28.490 INFO:tasks.rados.rados.0.smithi203.stdout:866: writing smithi20328086-44 from 1785236 to 2210683 tid 2 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:866: writing smithi20328086-44 from 2942002 to 3462275 tid 3 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:867: copy_from oid 33 from oid 38 current snap is 94 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:866: finishing write tid 1 to smithi20328086-44 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:868: copy_from oid 11 from oid 8 current snap is 94 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:869: copy_from oid 23 from oid 32 current snap is 94 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:870: write oid 21 current snap is 94 2024-09-17T15:33:28.503 INFO:tasks.rados.rados.0.smithi203.stdout:870: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-17T15:33:28.518 INFO:tasks.rados.rados.0.smithi203.stdout:870: writing smithi20328086-21 from 679082 to 1297082 tid 1 2024-09-17T15:33:28.529 INFO:tasks.rados.rados.0.smithi203.stdout:870: writing smithi20328086-21 from 1843718 to 2412823 tid 2 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:870: writing smithi20328086-21 from 2853072 to 3479082 tid 3 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:871: rollback oid 26 current snap is 94 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 92 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:866: finishing write tid 2 to smithi20328086-44 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:865: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:33:28.538 INFO:tasks.rados.rados.0.smithi203.stdout:868: finishing copy_from to smithi20328086-11 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:868: got expected ENOENT (src dne) 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:868: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:868: done (6 left) 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:872: snap_create 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:865: finishing copy_from to smithi20328086-50 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 103 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: finishing write tid 3 to smithi20328086-44 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: finishing write tid 4 to smithi20328086-44 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: finishing write tid 5 to smithi20328086-44 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: oid 44 updating version 0 to 263 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: oid 44 version 263 is already newer than 261 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: oid 44 updating version 263 to 264 2024-09-17T15:33:28.539 INFO:tasks.rados.rados.0.smithi203.stdout:866: oid 44 version 264 is already newer than 262 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 264 (ObjNum 325 snap 94 seq_num 325) dirty exists 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:866: left oid 44 (ObjNum 325 snap 94 seq_num 325) 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:870: finishing write tid 1 to smithi20328086-21 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:869: finishing copy_from racing read to smithi20328086-23 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:869: finishing copy_from to smithi20328086-23 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 93 (ObjNum 275 snap 70 seq_num 275) dirty exists 2024-09-17T15:33:28.540 INFO:tasks.rados.rados.0.smithi203.stdout:870: finishing write tid 2 to smithi20328086-21 2024-09-17T15:33:28.541 INFO:tasks.rados.rados.0.smithi203.stdout:867: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:867: finishing copy_from to smithi20328086-33 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 175 (ObjNum 323 snap 93 seq_num 323) dirty exists 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:871: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 0 (ObjNum 33 snap 0 seq_num 3606075104) dirty dne 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:870: finishing write tid 3 to smithi20328086-21 2024-09-17T15:33:28.544 INFO:tasks.rados.rados.0.smithi203.stdout:870: finishing write tid 4 to smithi20328086-21 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: finishing write tid 5 to smithi20328086-21 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: oid 21 updating version 0 to 357 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: oid 21 updating version 357 to 360 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: oid 21 version 360 is already newer than 358 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: oid 21 version 360 is already newer than 359 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 360 (ObjNum 326 snap 94 seq_num 326) dirty exists 2024-09-17T15:33:28.545 INFO:tasks.rados.rados.0.smithi203.stdout:870: left oid 21 (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:33:29.514 INFO:tasks.rados.rados.0.smithi203.stdout:865: done (6 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:866: done (5 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:867: done (4 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:869: done (3 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:870: done (2 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:871: done (1 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:872: done (0 left) 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:873: write oid 5 current snap is 95 2024-09-17T15:33:29.515 INFO:tasks.rados.rados.0.smithi203.stdout:873: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-17T15:33:29.522 INFO:tasks.rados.rados.0.smithi203.stdout:873: writing smithi20328086-5 from 695889 to 1105491 tid 1 2024-09-17T15:33:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:873: writing smithi20328086-5 from 1618553 to 2331316 tid 2 2024-09-17T15:33:29.540 INFO:tasks.rados.rados.0.smithi203.stdout:873: writing smithi20328086-5 from 2996848 to 3495889 tid 3 2024-09-17T15:33:29.540 INFO:tasks.rados.rados.0.smithi203.stdout:874: write oid 11 current snap is 95 2024-09-17T15:33:29.540 INFO:tasks.rados.rados.0.smithi203.stdout:873: finishing write tid 1 to smithi20328086-5 2024-09-17T15:33:29.541 INFO:tasks.rados.rados.0.smithi203.stdout:874: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-17T15:33:29.549 INFO:tasks.rados.rados.0.smithi203.stdout:874: writing smithi20328086-11 from 712696 to 1197547 tid 1 2024-09-17T15:33:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:874: writing smithi20328086-11 from 1960682 to 2533456 tid 2 2024-09-17T15:33:29.565 INFO:tasks.rados.rados.0.smithi203.stdout:874: writing smithi20328086-11 from 3307918 to 3512696 tid 3 2024-09-17T15:33:29.565 INFO:tasks.rados.rados.0.smithi203.stdout:875: snap_create 2024-09-17T15:33:29.565 INFO:tasks.rados.rados.0.smithi203.stdout:873: finishing write tid 2 to smithi20328086-5 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: finishing write tid 3 to smithi20328086-5 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: finishing write tid 4 to smithi20328086-5 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: finishing write tid 5 to smithi20328086-5 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: oid 5 updating version 0 to 177 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: oid 5 version 177 is already newer than 176 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: oid 5 updating version 177 to 178 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: oid 5 updating version 178 to 179 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 179 (ObjNum 327 snap 95 seq_num 327) dirty exists 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:873: left oid 5 (ObjNum 327 snap 95 seq_num 327) 2024-09-17T15:33:29.566 INFO:tasks.rados.rados.0.smithi203.stdout:874: finishing write tid 1 to smithi20328086-11 2024-09-17T15:33:29.570 INFO:tasks.rados.rados.0.smithi203.stdout:874: finishing write tid 2 to smithi20328086-11 2024-09-17T15:33:29.570 INFO:tasks.rados.rados.0.smithi203.stdout:874: finishing write tid 3 to smithi20328086-11 2024-09-17T15:33:29.572 INFO:tasks.rados.rados.0.smithi203.stdout:874: finishing write tid 4 to smithi20328086-11 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:874: finishing write tid 5 to smithi20328086-11 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:874: oid 11 updating version 0 to 267 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:874: oid 11 updating version 267 to 268 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:874: oid 11 version 268 is already newer than 266 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:874: oid 11 updating version 268 to 269 2024-09-17T15:33:29.573 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 269 (ObjNum 328 snap 95 seq_num 328) dirty exists 2024-09-17T15:33:29.574 INFO:tasks.rados.rados.0.smithi203.stdout:874: left oid 11 (ObjNum 328 snap 95 seq_num 328) 2024-09-17T15:33:30.522 INFO:tasks.rados.rados.0.smithi203.stdout:873: done (2 left) 2024-09-17T15:33:30.523 INFO:tasks.rados.rados.0.smithi203.stdout:874: done (1 left) 2024-09-17T15:33:30.523 INFO:tasks.rados.rados.0.smithi203.stdout:875: done (0 left) 2024-09-17T15:33:30.523 INFO:tasks.rados.rados.0.smithi203.stdout:876: copy_from oid 30 from oid 27 current snap is 96 2024-09-17T15:33:30.523 INFO:tasks.rados.rados.0.smithi203.stdout:877: delete oid 3 current snap is 96 2024-09-17T15:33:30.528 INFO:tasks.rados.rados.0.smithi203.stdout:877: done (1 left) 2024-09-17T15:33:30.528 INFO:tasks.rados.rados.0.smithi203.stdout:878: write oid 32 current snap is 96 2024-09-17T15:33:30.528 INFO:tasks.rados.rados.0.smithi203.stdout:878: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-17T15:33:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:878: writing smithi20328086-32 from 746310 to 1381659 tid 1 2024-09-17T15:33:30.556 INFO:tasks.rados.rados.0.smithi203.stdout:878: writing smithi20328086-32 from 2077646 to 2654089 tid 2 2024-09-17T15:33:30.565 INFO:tasks.rados.rados.0.smithi203.stdout:878: writing smithi20328086-32 from 3362764 to 3546310 tid 3 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:879: copy_from oid 20 from oid 37 current snap is 96 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:876: finishing copy_from to smithi20328086-30 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:876: got expected ENOENT (src dne) 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:876: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:878: finishing write tid 1 to smithi20328086-32 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:878: finishing write tid 2 to smithi20328086-32 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:876: done (2 left) 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:880: rollback oid 33 current snap is 96 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 79 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:881: copy_from oid 45 from oid 5 current snap is 96 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:882: copy_from oid 8 from oid 22 current snap is 96 2024-09-17T15:33:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:883: snap_remove snap 88 2024-09-17T15:33:31.524 INFO:tasks.rados.rados.0.smithi203.stdout:883: done (5 left) 2024-09-17T15:33:31.524 INFO:tasks.rados.rados.0.smithi203.stdout:884: read oid 15 snap -1 2024-09-17T15:33:31.524 INFO:tasks.rados.rados.0.smithi203.stdout:884: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: finishing write tid 3 to smithi20328086-32 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: finishing write tid 4 to smithi20328086-32 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: finishing write tid 5 to smithi20328086-32 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: oid 32 updating version 0 to 113 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: oid 32 updating version 113 to 116 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: oid 32 version 116 is already newer than 115 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: oid 32 version 116 is already newer than 114 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 116 (ObjNum 330 snap 96 seq_num 330) dirty exists 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:878: left oid 32 (ObjNum 330 snap 96 seq_num 330) 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:882: finishing copy_from racing read to smithi20328086-8 2024-09-17T15:33:31.525 INFO:tasks.rados.rados.0.smithi203.stdout:880: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:880: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:880: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 182 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:882: finishing copy_from to smithi20328086-8 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:879: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:879: finishing copy_from to smithi20328086-20 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:881: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:881: finishing copy_from to smithi20328086-45 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 179 (ObjNum 327 snap 95 seq_num 327) dirty exists 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:878: done (5 left) 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:879: done (4 left) 2024-09-17T15:33:31.526 INFO:tasks.rados.rados.0.smithi203.stdout:880: done (3 left) 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:881: done (2 left) 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:882: done (1 left) 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:885: read oid 5 snap -1 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:885: expect (ObjNum 327 snap 95 seq_num 327) 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:886: read oid 28 snap -1 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:886: expect deleted 2024-09-17T15:33:31.527 INFO:tasks.rados.rados.0.smithi203.stdout:887: delete oid 24 current snap is 96 2024-09-17T15:33:31.546 INFO:tasks.rados.rados.0.smithi203.stdout:885: expect (ObjNum 327 snap 95 seq_num 327) 2024-09-17T15:33:31.672 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3133670 2024-09-17T15:33:31.725 INFO:tasks.rados.rados.0.smithi203.stdout:884: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:31.756 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:31.791 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:31.826 INFO:tasks.rados.rados.0.smithi203.stdout:884: done (3 left) 2024-09-17T15:33:31.826 INFO:tasks.rados.rados.0.smithi203.stdout:885: done (2 left) 2024-09-17T15:33:31.826 INFO:tasks.rados.rados.0.smithi203.stdout:886: done (1 left) 2024-09-17T15:33:31.826 INFO:tasks.rados.rados.0.smithi203.stdout:887: done (0 left) 2024-09-17T15:33:31.826 INFO:tasks.rados.rados.0.smithi203.stdout:888: copy_from oid 38 from oid 27 current snap is 96 2024-09-17T15:33:31.827 INFO:tasks.rados.rados.0.smithi203.stdout:889: snap_create 2024-09-17T15:33:31.828 INFO:tasks.rados.rados.0.smithi203.stdout:888: finishing copy_from to smithi20328086-38 2024-09-17T15:33:31.828 INFO:tasks.rados.rados.0.smithi203.stdout:888: got expected ENOENT (src dne) 2024-09-17T15:33:31.828 INFO:tasks.rados.rados.0.smithi203.stdout:888: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:33:32.459 INFO:tasks.rados.rados.0.smithi203.stdout:888: done (1 left) 2024-09-17T15:33:32.459 INFO:tasks.rados.rados.0.smithi203.stdout:889: done (0 left) 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:890: read oid 31 snap -1 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:890: expect (ObjNum 320 snap 93 seq_num 320) 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:891: copy_from oid 7 from oid 15 current snap is 97 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:892: read oid 3 snap 79 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:892: expect (ObjNum 245 snap 63 seq_num 245) 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:893: read oid 48 snap -1 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:893: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:33:32.460 INFO:tasks.rados.rados.0.smithi203.stdout:894: snap_create 2024-09-17T15:33:32.475 INFO:tasks.rados.rados.0.smithi203.stdout:893: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:33:32.481 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 460554 2024-09-17T15:33:32.536 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 145486 2024-09-17T15:33:32.656 INFO:tasks.rados.rados.0.smithi203.stdout:892: expect (ObjNum 245 snap 63 seq_num 245) 2024-09-17T15:33:32.685 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:32.706 INFO:tasks.rados.rados.0.smithi203.stdout:890: expect (ObjNum 320 snap 93 seq_num 320) 2024-09-17T15:33:32.738 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:32.791 INFO:tasks.rados.rados.0.smithi203.stdout:891: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:33:32.791 INFO:tasks.rados.rados.0.smithi203.stdout:891: finishing copy_from to smithi20328086-7 2024-09-17T15:33:32.791 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 249 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:890: done (4 left) 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:891: done (3 left) 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:892: done (2 left) 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:893: done (1 left) 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:894: done (0 left) 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:895: write (excl) oid 2 current snap is 98 2024-09-17T15:33:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:895: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-17T15:33:33.486 INFO:tasks.rados.rados.0.smithi203.stdout:895: writing smithi20328086-2 from 779924 to 1565771 tid 1 2024-09-17T15:33:33.504 INFO:tasks.rados.rados.0.smithi203.stdout:895: writing smithi20328086-2 from 2078257 to 2658369 tid 2 2024-09-17T15:33:33.518 INFO:tasks.rados.rados.0.smithi203.stdout:895: writing smithi20328086-2 from 3301257 to 3579924 tid 3 2024-09-17T15:33:33.518 INFO:tasks.rados.rados.0.smithi203.stdout:896: write (excl) oid 28 current snap is 98 2024-09-17T15:33:33.518 INFO:tasks.rados.rados.0.smithi203.stdout:895: finishing write tid 1 to smithi20328086-2 2024-09-17T15:33:33.518 INFO:tasks.rados.rados.0.smithi203.stdout:895: finishing write tid 2 to smithi20328086-2 2024-09-17T15:33:33.518 INFO:tasks.rados.rados.0.smithi203.stdout:896: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-17T15:33:33.533 INFO:tasks.rados.rados.0.smithi203.stdout:896: writing smithi20328086-28 from 796731 to 1257827 tid 1 2024-09-17T15:33:33.549 INFO:tasks.rados.rados.0.smithi203.stdout:896: writing smithi20328086-28 from 1736739 to 2460509 tid 2 2024-09-17T15:33:33.561 INFO:tasks.rados.rados.0.smithi203.stdout:896: writing smithi20328086-28 from 2928680 to 3596731 tid 3 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:897: read oid 30 snap 95 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:897: expect deleted 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:898: rollback oid 18 current snap is 98 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 84 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: finishing write tid 3 to smithi20328086-2 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: finishing write tid 4 to smithi20328086-2 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: finishing write tid 5 to smithi20328086-2 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: oid 2 updating version 0 to 250 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: oid 2 updating version 250 to 253 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: oid 2 version 253 is already newer than 252 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: oid 2 version 253 is already newer than 251 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 253 (ObjNum 332 snap 98 seq_num 332) dirty exists 2024-09-17T15:33:33.562 INFO:tasks.rados.rados.0.smithi203.stdout:895: left oid 2 (ObjNum 332 snap 98 seq_num 332) 2024-09-17T15:33:33.563 INFO:tasks.rados.rados.0.smithi203.stdout:896: finishing write tid 1 to smithi20328086-28 2024-09-17T15:33:33.563 INFO:tasks.rados.rados.0.smithi203.stdout:896: finishing write tid 2 to smithi20328086-28 2024-09-17T15:33:33.563 INFO:tasks.rados.rados.0.smithi203.stdout:895: done (3 left) 2024-09-17T15:33:33.563 INFO:tasks.rados.rados.0.smithi203.stdout:899: snap_create 2024-09-17T15:33:33.568 INFO:tasks.rados.rados.0.smithi203.stdout:896: finishing write tid 3 to smithi20328086-28 2024-09-17T15:33:33.568 INFO:tasks.rados.rados.0.smithi203.stdout:896: finishing write tid 4 to smithi20328086-28 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:896: finishing write tid 5 to smithi20328086-28 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:896: oid 28 updating version 0 to 365 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:896: oid 28 updating version 365 to 366 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:896: oid 28 version 366 is already newer than 364 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:896: oid 28 version 366 is already newer than 363 2024-09-17T15:33:33.569 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 366 (ObjNum 333 snap 98 seq_num 333) dirty exists 2024-09-17T15:33:33.570 INFO:tasks.rados.rados.0.smithi203.stdout:896: left oid 28 (ObjNum 333 snap 98 seq_num 333) 2024-09-17T15:33:33.575 INFO:tasks.rados.rados.0.smithi203.stdout:898: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:33:33.575 INFO:tasks.rados.rados.0.smithi203.stdout:898: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:33:33.578 INFO:tasks.rados.rados.0.smithi203.stdout:898: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:33:33.578 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 188 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:896: done (3 left) 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:897: done (2 left) 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:898: done (1 left) 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:899: done (0 left) 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:900: write oid 25 current snap is 99 2024-09-17T15:33:34.471 INFO:tasks.rados.rados.0.smithi203.stdout:900: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-17T15:33:34.481 INFO:tasks.rados.rados.0.smithi203.stdout:900: writing smithi20328086-25 from 413538 to 949883 tid 1 2024-09-17T15:33:34.491 INFO:tasks.rados.rados.0.smithi203.stdout:900: writing smithi20328086-25 from 1395221 to 1979002 tid 2 2024-09-17T15:33:34.502 INFO:tasks.rados.rados.0.smithi203.stdout:900: writing smithi20328086-25 from 2556103 to 3214720 tid 3 2024-09-17T15:33:34.502 INFO:tasks.rados.rados.0.smithi203.stdout:901: snap_remove snap 82 2024-09-17T15:33:35.481 INFO:tasks.rados.rados.0.smithi203.stdout:901: done (1 left) 2024-09-17T15:33:35.481 INFO:tasks.rados.rados.0.smithi203.stdout:902: snap_remove snap 84 2024-09-17T15:33:36.487 INFO:tasks.rados.rados.0.smithi203.stdout:902: done (1 left) 2024-09-17T15:33:36.488 INFO:tasks.rados.rados.0.smithi203.stdout:903: write (excl) oid 11 current snap is 99 2024-09-17T15:33:36.488 INFO:tasks.rados.rados.0.smithi203.stdout:903: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-17T15:33:36.504 INFO:tasks.rados.rados.0.smithi203.stdout:903: writing smithi20328086-11 from 430345 to 1158292 tid 1 2024-09-17T15:33:36.519 INFO:tasks.rados.rados.0.smithi203.stdout:903: writing smithi20328086-11 from 1570056 to 2297495 tid 2 2024-09-17T15:33:36.532 INFO:tasks.rados.rados.0.smithi203.stdout:903: writing smithi20328086-11 from 2699879 to 3169768 tid 3 2024-09-17T15:33:36.533 INFO:tasks.rados.rados.0.smithi203.stdout:904: write oid 29 current snap is 99 2024-09-17T15:33:36.533 INFO:tasks.rados.rados.0.smithi203.stdout:904: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-17T15:33:36.543 INFO:tasks.rados.rados.0.smithi203.stdout:904: writing smithi20328086-29 from 447152 to 850348 tid 1 2024-09-17T15:33:36.557 INFO:tasks.rados.rados.0.smithi203.stdout:904: writing smithi20328086-29 from 1512185 to 2099635 tid 2 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:904: writing smithi20328086-29 from 2727302 to 3408463 tid 3 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:905: delete oid 9 current snap is 99 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: finishing write tid 1 to smithi20328086-25 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: finishing write tid 2 to smithi20328086-25 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: finishing write tid 3 to smithi20328086-25 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: finishing write tid 4 to smithi20328086-25 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: finishing write tid 5 to smithi20328086-25 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: oid 25 updating version 0 to 274 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: oid 25 version 274 is already newer than 273 2024-09-17T15:33:36.570 INFO:tasks.rados.rados.0.smithi203.stdout:900: oid 25 updating version 274 to 275 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:900: oid 25 version 275 is already newer than 272 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 275 (ObjNum 334 snap 99 seq_num 334) dirty exists 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:900: left oid 25 (ObjNum 334 snap 99 seq_num 334) 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: finishing write tid 1 to smithi20328086-11 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: finishing write tid 2 to smithi20328086-11 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: finishing write tid 3 to smithi20328086-11 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: finishing write tid 4 to smithi20328086-11 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: finishing write tid 5 to smithi20328086-11 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: oid 11 updating version 0 to 288 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: oid 11 updating version 288 to 289 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: oid 11 version 289 is already newer than 287 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: oid 11 updating version 289 to 290 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 290 (ObjNum 335 snap 99 seq_num 335) dirty exists 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:903: left oid 11 (ObjNum 335 snap 99 seq_num 335) 2024-09-17T15:33:36.571 INFO:tasks.rados.rados.0.smithi203.stdout:904: finishing write tid 1 to smithi20328086-29 2024-09-17T15:33:36.572 INFO:tasks.rados.rados.0.smithi203.stdout:904: finishing write tid 2 to smithi20328086-29 2024-09-17T15:33:36.577 INFO:tasks.rados.rados.0.smithi203.stdout:904: finishing write tid 3 to smithi20328086-29 2024-09-17T15:33:36.577 INFO:tasks.rados.rados.0.smithi203.stdout:904: finishing write tid 4 to smithi20328086-29 2024-09-17T15:33:36.578 INFO:tasks.rados.rados.0.smithi203.stdout:904: finishing write tid 5 to smithi20328086-29 2024-09-17T15:33:36.578 INFO:tasks.rados.rados.0.smithi203.stdout:904: oid 29 updating version 0 to 279 2024-09-17T15:33:36.578 INFO:tasks.rados.rados.0.smithi203.stdout:904: oid 29 version 279 is already newer than 276 2024-09-17T15:33:36.578 INFO:tasks.rados.rados.0.smithi203.stdout:904: oid 29 version 279 is already newer than 277 2024-09-17T15:33:36.578 INFO:tasks.rados.rados.0.smithi203.stdout:904: oid 29 version 279 is already newer than 278 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 279 (ObjNum 336 snap 99 seq_num 336) dirty exists 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:904: left oid 29 (ObjNum 336 snap 99 seq_num 336) 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:900: done (3 left) 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:903: done (2 left) 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:904: done (1 left) 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:905: done (0 left) 2024-09-17T15:33:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:906: delete oid 49 current snap is 99 2024-09-17T15:33:36.581 INFO:tasks.rados.rados.0.smithi203.stdout:906: done (0 left) 2024-09-17T15:33:36.581 INFO:tasks.rados.rados.0.smithi203.stdout:907: write (excl) oid 40 current snap is 99 2024-09-17T15:33:36.581 INFO:tasks.rados.rados.0.smithi203.stdout:907: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-17T15:33:36.590 INFO:tasks.rados.rados.0.smithi203.stdout:907: writing smithi20328086-40 from 497573 to 1126516 tid 1 2024-09-17T15:33:36.599 INFO:tasks.rados.rados.0.smithi203.stdout:907: writing smithi20328086-40 from 1687631 to 2422408 tid 2 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:907: writing smithi20328086-40 from 3093218 to 3697573 tid 3 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:908: read oid 33 snap 92 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:908: expect deleted 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:909: copy_from oid 45 from oid 7 current snap is 99 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:907: finishing write tid 1 to smithi20328086-40 2024-09-17T15:33:36.610 INFO:tasks.rados.rados.0.smithi203.stdout:910: snap_remove snap 90 2024-09-17T15:33:37.462 INFO:tasks.rados.rados.0.smithi203.stdout:910: done (3 left) 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:911: copy_from oid 49 from oid 17 current snap is 99 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: finishing write tid 2 to smithi20328086-40 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: finishing write tid 3 to smithi20328086-40 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: finishing write tid 4 to smithi20328086-40 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: finishing write tid 5 to smithi20328086-40 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: oid 40 updating version 0 to 293 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: oid 40 updating version 293 to 295 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: oid 40 version 295 is already newer than 294 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: oid 40 version 295 is already newer than 292 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 295 (ObjNum 339 snap 99 seq_num 339) dirty exists 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:907: left oid 40 (ObjNum 339 snap 99 seq_num 339) 2024-09-17T15:33:37.463 INFO:tasks.rados.rados.0.smithi203.stdout:909: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:909: finishing copy_from to smithi20328086-45 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 249 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:907: done (3 left) 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:908: done (2 left) 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:909: done (1 left) 2024-09-17T15:33:37.464 INFO:tasks.rados.rados.0.smithi203.stdout:912: delete oid 24 current snap is 99 2024-09-17T15:33:37.482 INFO:tasks.rados.rados.0.smithi203.stdout:912: done (1 left) 2024-09-17T15:33:37.482 INFO:tasks.rados.rados.0.smithi203.stdout:913: write oid 48 current snap is 99 2024-09-17T15:33:37.482 INFO:tasks.rados.rados.0.smithi203.stdout:913: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-17T15:33:37.493 INFO:tasks.rados.rados.0.smithi203.stdout:913: writing smithi20328086-48 from 531187 to 1310628 tid 1 2024-09-17T15:33:37.507 INFO:tasks.rados.rados.0.smithi203.stdout:913: writing smithi20328086-48 from 2088242 to 2826688 tid 2 2024-09-17T15:33:37.514 INFO:tasks.rados.rados.0.smithi203.stdout:913: writing smithi20328086-48 from 3431711 to 3731187 tid 3 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:914: snap_create 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:911: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:911: finishing copy_from to smithi20328086-49 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 197 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:913: finishing write tid 1 to smithi20328086-48 2024-09-17T15:33:37.515 INFO:tasks.rados.rados.0.smithi203.stdout:913: finishing write tid 2 to smithi20328086-48 2024-09-17T15:33:37.520 INFO:tasks.rados.rados.0.smithi203.stdout:913: finishing write tid 3 to smithi20328086-48 2024-09-17T15:33:37.520 INFO:tasks.rados.rados.0.smithi203.stdout:913: finishing write tid 4 to smithi20328086-48 2024-09-17T15:33:37.520 INFO:tasks.rados.rados.0.smithi203.stdout:913: finishing write tid 5 to smithi20328086-48 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:913: oid 48 updating version 0 to 297 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:913: oid 48 version 297 is already newer than 294 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:913: oid 48 version 297 is already newer than 296 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:913: oid 48 version 297 is already newer than 295 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 297 (ObjNum 341 snap 99 seq_num 341) dirty exists 2024-09-17T15:33:37.521 INFO:tasks.rados.rados.0.smithi203.stdout:913: left oid 48 (ObjNum 341 snap 99 seq_num 341) 2024-09-17T15:33:38.664 INFO:tasks.rados.rados.0.smithi203.stdout:911: done (2 left) 2024-09-17T15:33:38.664 INFO:tasks.rados.rados.0.smithi203.stdout:913: done (1 left) 2024-09-17T15:33:38.664 INFO:tasks.rados.rados.0.smithi203.stdout:914: done (0 left) 2024-09-17T15:33:38.665 INFO:tasks.rados.rados.0.smithi203.stdout:915: write (excl) oid 13 current snap is 100 2024-09-17T15:33:38.665 INFO:tasks.rados.rados.0.smithi203.stdout:915: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-17T15:33:38.673 INFO:tasks.rados.rados.0.smithi203.stdout:915: writing smithi20328086-13 from 547994 to 1002684 tid 1 2024-09-17T15:33:38.684 INFO:tasks.rados.rados.0.smithi203.stdout:915: writing smithi20328086-13 from 1746724 to 2228828 tid 2 2024-09-17T15:33:38.692 INFO:tasks.rados.rados.0.smithi203.stdout:915: writing smithi20328086-13 from 2659134 to 3124280 tid 3 2024-09-17T15:33:38.697 INFO:tasks.rados.rados.0.smithi203.stdout:915: writing smithi20328086-13 from 3605976 to 3747994 tid 4 2024-09-17T15:33:38.698 INFO:tasks.rados.rados.0.smithi203.stdout:916: delete oid 29 current snap is 100 2024-09-17T15:33:38.698 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 1 to smithi20328086-13 2024-09-17T15:33:38.698 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 2 to smithi20328086-13 2024-09-17T15:33:38.701 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 3 to smithi20328086-13 2024-09-17T15:33:38.702 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 4 to smithi20328086-13 2024-09-17T15:33:38.702 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 5 to smithi20328086-13 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: finishing write tid 6 to smithi20328086-13 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: oid 13 updating version 0 to 188 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: oid 13 updating version 188 to 191 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: oid 13 version 191 is already newer than 187 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: oid 13 version 191 is already newer than 190 2024-09-17T15:33:38.703 INFO:tasks.rados.rados.0.smithi203.stdout:915: oid 13 version 191 is already newer than 189 2024-09-17T15:33:38.704 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 191 (ObjNum 342 snap 100 seq_num 342) dirty exists 2024-09-17T15:33:38.704 INFO:tasks.rados.rados.0.smithi203.stdout:915: left oid 13 (ObjNum 342 snap 100 seq_num 342) 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:915: done (1 left) 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:916: done (0 left) 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:917: rollback oid 17 current snap is 100 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 79 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:918: read oid 30 snap 89 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:918: expect (ObjNum 275 snap 70 seq_num 275) 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:919: read oid 15 snap -1 2024-09-17T15:33:38.711 INFO:tasks.rados.rados.0.smithi203.stdout:919: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:38.712 INFO:tasks.rados.rados.0.smithi203.stdout:920: snap_create 2024-09-17T15:33:38.721 INFO:tasks.rados.rados.0.smithi203.stdout:917: finishing rollback tid 0 to smithi20328086-17 2024-09-17T15:33:38.721 INFO:tasks.rados.rados.0.smithi203.stdout:917: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:33:38.723 INFO:tasks.rados.rados.0.smithi203.stdout:917: finishing rollback tid 2 to smithi20328086-17 2024-09-17T15:33:38.723 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 305 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-09-17T15:33:38.727 INFO:tasks.rados.rados.0.smithi203.stdout:918: expect (ObjNum 275 snap 70 seq_num 275) 2024-09-17T15:33:38.846 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:38.885 INFO:tasks.rados.rados.0.smithi203.stdout:919: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:33:38.885 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:38.924 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1337639 2024-09-17T15:33:38.947 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:39.130 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:33:39.130 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:33:39.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:33:39.134 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:33:39.134 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:33:39.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: get_monmap_and_config 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: build_initial_monmap 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: monmap: 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:39.252216+0000 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:39.252216+0000 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:39.252 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:39.252 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:39.252 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:39.252 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:39.252 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 Processor -- start 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 -- start start 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: init 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-17T15:33:39.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c380a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 0x7f6c380a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c380a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c3813d2f0 0x7f6c3813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c380a6530 con 0x7f6c380a50e0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3809c150 con 0x7f6c3813d2f0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3809c430 con 0x7f6c380a54c0 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3f1b4640 10 monclient(hunting): authenticate will time out at 1352.217503s 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3cf50640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6c380a50e0 0x7f6c3809ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36298/0 (socket says 172.21.15.86:36298) 2024-09-17T15:33:39.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4241047379 learned_addr learned my addr 172.21.15.86:0/4241047379 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 --2- 172.21.15.86:0/4241047379 >> v2:172.21.15.86:3300/0 conn(0x7f6c3813d2f0 0x7f6c3813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): get_auth_request con 0x7f6c3813d2f0 auth_method 0 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:39.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4241047379 >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 0x7f6c380a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient(hunting): handle_auth_done global_id 4932 payload 274 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 -- 172.21.15.86:0/4241047379 >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 msgr2=0x7f6c380a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 --2- 172.21.15.86:0/4241047379 >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 0x7f6c380a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 -- 172.21.15.86:0/4241047379 >> v1:172.21.15.86:6789/0 conn(0x7f6c380a50e0 legacy=0x7f6c3809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient: _finish_hunting 0 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 monclient: found mon.noname-a 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c3d751640 1 -- 172.21.15.86:0/4241047379 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c380a6940 con 0x7f6c3813d2f0 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c37fff640 10 monclient: get_auth_request con 0x7f6c380a54c0 auth_method 0 2024-09-17T15:33:39.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.251+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4241047379 >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 0x7f6c380a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3d751640 1 -- 172.21.15.86:0/4241047379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c3813d2f0 msgr2=0x7f6c3813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3d751640 1 --2- 172.21.15.86:0/4241047379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c3813d2f0 0x7f6c3813f6e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6c380a6220 tx=0x7f6c2801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef76ceba8424f92d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 1 -- 172.21.15.86:0/4241047379 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c280b2070 con 0x7f6c3813d2f0 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: dump: 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:39.261 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: _finish_auth 0 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: _check_auth_tickets 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 1 -- 172.21.15.86:0/4241047379 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c280a5970 con 0x7f6c3813d2f0 2024-09-17T15:33:39.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 1 -- 172.21.15.86:0/4241047379 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c280a5d40 con 0x7f6c3813d2f0 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 5 monclient: authenticate success, global_id 4932 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 4 set_mon_vals no callback set 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c377fe640 10 monclient: dump: 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.263 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 10 monclient: shutdown 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4241047379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c3813d2f0 msgr2=0x7f6c3813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4241047379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c3813d2f0 0x7f6c3813f6e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6c380a6220 tx=0x7f6c2801c7f0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4241047379 shutdown_connections 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4241047379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c3813d2f0 0x7f6c3813f6e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4241047379 >> v2:172.21.15.203:3300/0 conn(0x7f6c380a54c0 0x7f6c380a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4241047379 >> 172.21.15.86:0/4241047379 conn(0x7f6c380aaae0 msgr2=0x7f6c380aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4241047379 shutdown_connections 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4241047379 wait complete. 2024-09-17T15:33:39.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 10 monclient: build_initial_monmap 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 10 monclient: monmap: 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.255+0000 7f6c3f1b4640 1 Processor -- start 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- start start 2024-09-17T15:33:39.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: init 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c380a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c381212d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c380a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c38138140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c381427d0 con 0x7f6c380a50e0 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c38142650 con 0x7f6c3813d2f0 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c38142950 con 0x7f6c380a54c0 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient(hunting): authenticate will time out at 1352.224555s 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c381212d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49018/0 (socket says 172.21.15.86:49018) 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 learned_addr learned my addr 172.21.15.86:0/4266792815 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): get_auth_request con 0x7f6c380a50e0 auth_method 0 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 10 monclient(hunting): get_auth_request con 0x7f6c380a54c0 auth_method 0 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 msgr2=0x7f6c381212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 msgr2=0x7f6c381212d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c381212d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c381212d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3d751640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c38138140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3d751640 10 monclient(hunting): get_auth_request con 0x7f6c3813d2f0 auth_method 0 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3d751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3d751640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3d751640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_done global_id 4935 payload 824 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 msgr2=0x7f6c381212d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c381212d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 msgr2=0x7f6c38138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c38138140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient: _finish_hunting 0 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 monclient: found mon.noname-a 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:39.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c380a4670 con 0x7f6c380a50e0 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 msgr2=0x7f6c380a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c470 tx=0x7f6c3000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30007070 con 0x7f6c380a50e0 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: dump: 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:39.274 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: _finish_auth 0 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: _check_auth_tickets 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c30007210 con 0x7f6c380a50e0 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c30007600 con 0x7f6c380a50e0 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 5 monclient: authenticate success, global_id 4935 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _renew_subs 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:39.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c380a7c00 con 0x7f6c380a50e0 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c36ffd640 4 set_mon_vals no callback set 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: dump: 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:39.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _renew_subs 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c380a3f40 con 0x7f6c380a50e0 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 msgr2=0x7f6c380a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 msgr2=0x7f6c380a30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c470 tx=0x7f6c3000c930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000c470 tx=0x7f6c3000c930 comp rx=0 tx=0).stop 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 msgr2=0x7f6c380a30a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c380a50e0 0x7f6c380a30a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:39.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c180058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): picked mon.c con 0x7f6c380a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c18008310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): picked mon.b con 0x7f6c3813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c18008d20 0x7f6c1800b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): picked mon.a con 0x7f6c18008d20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1800bc60 con 0x7f6c18008d20 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1800bde0 con 0x7f6c3813d2f0 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.259+0000 7f6c357fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1800bf60 con 0x7f6c380a54c0 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c180058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): get_auth_request con 0x7f6c380a54c0 auth_method 0 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c18008d20 0x7f6c1800b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): get_auth_request con 0x7f6c18008d20 auth_method 0 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3d751640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:33:39.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c18008310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c37fff640 10 monclient(hunting): get_auth_request con 0x7f6c3813d2f0 auth_method 0 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c37fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c37fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient(hunting): handle_auth_done global_id 4935 payload 293 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 msgr2=0x7f6c18008310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c18008310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c18008d20 msgr2=0x7f6c1800b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c18008d20 0x7f6c1800b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient: _finish_hunting 0 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 monclient: found mon.c 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:39.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f6c1800c280 con 0x7f6c380a54c0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient: _finish_auth 0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 10 monclient: _check_auth_tickets 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3f1b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3f1b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3f1b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c380a58d0 con 0x7f6c380a54c0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 msgr2=0x7f6c180058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c3cf50640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c180058d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6c38122290 tx=0x7f6c30004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3001b480 con 0x7f6c380a54c0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:39.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 10 monclient: dump: 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3001b9d0 con 0x7f6c380a54c0 2024-09-17T15:33:39.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c367fc640 4 set_mon_vals no callback set 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6c3001bb80 con 0x7f6c380a54c0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.263+0000 7f6c357fa640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 0x7f6c18046060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.267+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 4 ==== osd_map(189..189 src has 1..189) ==== 4755+0+0 (secure 0 0 0) 0x7f6c30052530 con 0x7f6c380a54c0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.267+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 0x7f6c18046060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.267+0000 7f6c37fff640 10 monclient: get_auth_request con 0x7f6c380a50e0 auth_method 0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.267+0000 7f6c37fff640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 0x7f6c18046060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006fd0 tx=0x7f6c20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.271+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6c3004ff90 con 0x7f6c380a54c0 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.271+0000 7f6c357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:39.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.271+0000 7f6c357fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:39.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.503+0000 7f6c3f1b4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:39.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.503+0000 7f6c3f1b4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:39.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.503+0000 7f6c3f1b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:39.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:39.503+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6c38074be0 con 0x7f6c380a54c0 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:917: done (3 left) 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:918: done (2 left) 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:919: done (1 left) 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:920: done (0 left) 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:921: read oid 28 snap 97 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:921: expect deleted 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:922: rollback oid 25 current snap is 101 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 100 2024-09-17T15:33:39.672 INFO:tasks.rados.rados.0.smithi203.stdout:923: snap_remove snap 91 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:923: done (2 left) 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:924: rollback oid 41 current snap is 101 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 85 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:922: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:922: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:922: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 284 (ObjNum 334 snap 99 seq_num 334) dirty exists 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:921: done (2 left) 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:922: done (1 left) 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:925: copy_from oid 32 from oid 20 current snap is 101 2024-09-17T15:33:40.681 INFO:tasks.rados.rados.0.smithi203.stdout:926: snap_create 2024-09-17T15:33:40.685 INFO:tasks.rados.rados.0.smithi203.stdout:924: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:33:40.685 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3605725936) dirty dne 2024-09-17T15:33:40.721 INFO:tasks.rados.rados.0.smithi203.stdout:925: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:33:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:925: finishing copy_from to smithi20328086-32 2024-09-17T15:33:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:33:41.684 INFO:tasks.rados.rados.0.smithi203.stdout:924: done (2 left) 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:925: done (1 left) 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:926: done (0 left) 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:927: copy_from oid 14 from oid 36 current snap is 102 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:928: rollback oid 41 current snap is 102 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 93 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:929: write oid 39 current snap is 102 2024-09-17T15:33:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:929: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-17T15:33:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:929: writing smithi20328086-39 from 581608 to 1303149 tid 1 2024-09-17T15:33:41.724 INFO:tasks.rados.rados.0.smithi203.stdout:929: writing smithi20328086-39 from 1863688 to 2349461 tid 2 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:929: writing smithi20328086-39 from 3113980 to 3718023 tid 3 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:930: read oid 8 snap -1 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:930: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:931: write (excl) oid 34 current snap is 102 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:928: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:33:41.744 INFO:tasks.rados.rados.0.smithi203.stdout:929: finishing write tid 1 to smithi20328086-39 2024-09-17T15:33:41.745 INFO:tasks.rados.rados.0.smithi203.stdout:927: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:33:41.745 INFO:tasks.rados.rados.0.smithi203.stdout:931: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-17T15:33:41.760 INFO:tasks.rados.rados.0.smithi203.stdout:931: writing smithi20328086-34 from 598415 to 1395205 tid 1 2024-09-17T15:33:41.775 INFO:tasks.rados.rados.0.smithi203.stdout:931: writing smithi20328086-34 from 1922170 to 2667954 tid 2 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:931: writing smithi20328086-34 from 3141403 to 3556718 tid 3 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:928: done (4 left) 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:932: rollback oid 18 current snap is 102 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 94 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:927: finishing copy_from to smithi20328086-14 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:927: done (4 left) 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:933: write (excl) oid 1 current snap is 102 2024-09-17T15:33:41.785 INFO:tasks.rados.rados.0.smithi203.stdout:933: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-17T15:33:41.794 INFO:tasks.rados.rados.0.smithi203.stdout:933: writing smithi20328086-1 from 615222 to 1087261 tid 1 2024-09-17T15:33:41.802 INFO:tasks.rados.rados.0.smithi203.stdout:933: writing smithi20328086-1 from 1580652 to 2070094 tid 2 2024-09-17T15:33:41.812 INFO:tasks.rados.rados.0.smithi203.stdout:933: writing smithi20328086-1 from 2768826 to 3395413 tid 3 2024-09-17T15:33:41.812 INFO:tasks.rados.rados.0.smithi203.stdout:934: snap_remove snap 85 2024-09-17T15:33:42.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:42.259+0000 7f6c2ffff640 10 monclient: tick 2024-09-17T15:33:42.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:42.259+0000 7f6c2ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:42.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:42.259+0000 7f6c2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:42.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:42.259+0000 7f6c2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:934: done (5 left) 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:935: delete oid 35 current snap is 102 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: finishing write tid 2 to smithi20328086-39 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: finishing write tid 3 to smithi20328086-39 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: finishing write tid 4 to smithi20328086-39 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: finishing write tid 5 to smithi20328086-39 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: oid 39 updating version 0 to 300 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: oid 39 version 300 is already newer than 298 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: oid 39 version 300 is already newer than 299 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:929: oid 39 updating version 300 to 301 2024-09-17T15:33:42.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 301 (ObjNum 344 snap 102 seq_num 344) dirty exists 2024-09-17T15:33:42.904 INFO:tasks.rados.rados.0.smithi203.stdout:929: left oid 39 (ObjNum 344 snap 102 seq_num 344) 2024-09-17T15:33:42.904 INFO:tasks.rados.rados.0.smithi203.stdout:930: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:33:42.999 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 111331 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: finishing write tid 1 to smithi20328086-34 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: finishing write tid 2 to smithi20328086-34 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: finishing write tid 3 to smithi20328086-34 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: finishing write tid 4 to smithi20328086-34 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: finishing write tid 5 to smithi20328086-34 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: oid 34 updating version 0 to 134 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: oid 34 updating version 134 to 135 2024-09-17T15:33:43.027 INFO:tasks.rados.rados.0.smithi203.stdout:931: oid 34 version 135 is already newer than 133 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:931: oid 34 updating version 135 to 136 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 136 (ObjNum 345 snap 102 seq_num 345) dirty exists 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:931: left oid 34 (ObjNum 345 snap 102 seq_num 345) 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:932: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:932: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:932: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 216 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: finishing write tid 1 to smithi20328086-1 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: finishing write tid 2 to smithi20328086-1 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: finishing write tid 3 to smithi20328086-1 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: finishing write tid 4 to smithi20328086-1 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: finishing write tid 5 to smithi20328086-1 2024-09-17T15:33:43.028 INFO:tasks.rados.rados.0.smithi203.stdout:933: oid 1 updating version 0 to 303 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:933: oid 1 updating version 303 to 304 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:933: oid 1 updating version 304 to 306 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:933: oid 1 version 306 is already newer than 305 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 306 (ObjNum 346 snap 102 seq_num 346) dirty exists 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:933: left oid 1 (ObjNum 346 snap 102 seq_num 346) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:929: done (5 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:930: done (4 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:931: done (3 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:932: done (2 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:933: done (1 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:935: done (0 left) 2024-09-17T15:33:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:936: snap_remove snap 93 2024-09-17T15:33:43.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:43.259+0000 7f6c2ffff640 10 monclient: tick 2024-09-17T15:33:43.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:43.259+0000 7f6c2ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:43.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:43.259+0000 7f6c2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:936: done (0 left) 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:937: read oid 16 snap 95 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:937: expect deleted 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:938: rollback oid 34 current snap is 102 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 92 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:939: read oid 44 snap 96 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:939: expect (ObjNum 325 snap 94 seq_num 325) 2024-09-17T15:33:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:940: copy_from oid 2 from oid 13 current snap is 102 2024-09-17T15:33:43.939 INFO:tasks.rados.rados.0.smithi203.stdout:941: delete oid 4 current snap is 102 2024-09-17T15:33:43.951 INFO:tasks.rados.rados.0.smithi203.stdout:938: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:33:43.952 INFO:tasks.rados.rados.0.smithi203.stdout:938: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:33:43.952 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 142 (ObjNum 129 snap 0 seq_num 3606628000) dirty dne 2024-09-17T15:33:43.962 INFO:tasks.rados.rados.0.smithi203.stdout:937: done (4 left) 2024-09-17T15:33:43.962 INFO:tasks.rados.rados.0.smithi203.stdout:938: done (3 left) 2024-09-17T15:33:43.962 INFO:tasks.rados.rados.0.smithi203.stdout:941: done (2 left) 2024-09-17T15:33:43.962 INFO:tasks.rados.rados.0.smithi203.stdout:942: delete oid 34 current snap is 102 2024-09-17T15:33:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:939: expect (ObjNum 325 snap 94 seq_num 325) 2024-09-17T15:33:44.116 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1644532 2024-09-17T15:33:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.119+0000 7f6c357fa640 1 -- 172.21.15.86:0/4266792815 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6c30050310 con 0x7f6c380a54c0 2024-09-17T15:33:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.119+0000 7f6c357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.119+0000 7f6c357fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:44.122 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 msgr2=0x7f6c18046060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 0x7f6c18046060 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006fd0 tx=0x7f6c20008040 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 10 monclient: shutdown 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 msgr2=0x7f6c180058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c180058d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6c38122290 tx=0x7f6c30004a20 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 shutdown_connections 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c380a50e0 0x7f6c18046060 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c18008d20 0x7f6c1800b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3813d2f0 0x7f6c18008310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 --2- 172.21.15.86:0/4266792815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c380a54c0 0x7f6c180058d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 >> 172.21.15.86:0/4266792815 conn(0x7f6c380aaae0 msgr2=0x7f6c380a08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 shutdown_connections 2024-09-17T15:33:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.123+0000 7f6c3f1b4640 1 -- 172.21.15.86:0/4266792815 wait complete. 2024-09-17T15:33:44.145 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:33:44.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:33:44.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:33:44.180 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:33:44.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: get_monmap_and_config 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: monmap: 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:44.268151+0000 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:44.268151+0000 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:44.268 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:44.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 Processor -- start 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 -- start start 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: init 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bb80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a5bd0 0x7f4bb809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bb813d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:44.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bb80a6280 con 0x7f4bb80a50e0 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bb809c0a0 con 0x7f4bb813d2f0 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4bb809d2a0 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bbea20640 10 monclient(hunting): authenticate will time out at 1357.233711s 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb7fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4bb80a50e0 0x7f4bb80a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36302/0 (socket says 172.21.15.86:36302) 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb7fff640 1 -- 172.21.15.86:0/3352394687 learned_addr learned my addr 172.21.15.86:0/3352394687 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/3352394687 >> v2:172.21.15.203:3300/0 conn(0x7f4bb80a5bd0 0x7f4bb809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request con 0x7f4bb80a5bd0 auth_method 0 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_done global_id 54235 payload 274 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3352394687 >> v1:172.21.15.203:6789/0 conn(0x7f4bb813d2f0 legacy=0x7f4bb813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3352394687 >> v1:172.21.15.86:6789/0 conn(0x7f4bb80a50e0 legacy=0x7f4bb80a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 1 monclient: found mon.noname-c 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:33:44.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.267+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3352394687 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb809c3d0 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb77fe640 1 -- 172.21.15.86:0/3352394687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a5bd0 msgr2=0x7f4bb809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/3352394687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb809cd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0097c0 tx=0x7f4bac01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e989ad36f5969dc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3352394687 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0b0070 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: dump: 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.274 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3352394687 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bac0ab600 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 1 -- 172.21.15.86:0/3352394687 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0ab9d0 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bb6ffd640 10 monclient: dump: 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bbea20640 5 monclient: authenticate success, global_id 54235 2024-09-17T15:33:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bbea20640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.271+0000 7f4bbea20640 4 set_mon_vals no callback set 2024-09-17T15:33:44.278 INFO:tasks.rados.rados.0.smithi203.stdout:940: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:33:44.278 INFO:tasks.rados.rados.0.smithi203.stdout:940: finishing copy_from to smithi20328086-2 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 191 (ObjNum 342 snap 100 seq_num 342) dirty exists 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:939: done (2 left) 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:940: done (1 left) 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:942: done (0 left) 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:943: rollback oid 13 current snap is 102 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 96 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:944: write oid 34 current snap is 102 2024-09-17T15:33:44.279 INFO:tasks.rados.rados.0.smithi203.stdout:944: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: shutdown 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- 172.21.15.86:0/3352394687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a5bd0 msgr2=0x7f4bb809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 --2- 172.21.15.86:0/3352394687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb809cd60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0097c0 tx=0x7f4bac01c360 comp rx=0 tx=0).stop 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- 172.21.15.86:0/3352394687 shutdown_connections 2024-09-17T15:33:44.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 --2- 172.21.15.86:0/3352394687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb809cd60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- 172.21.15.86:0/3352394687 >> 172.21.15.86:0/3352394687 conn(0x7f4bb80aaae0 msgr2=0x7f4bb80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- 172.21.15.86:0/3352394687 shutdown_connections 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- 172.21.15.86:0/3352394687 wait complete. 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: monmap: 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:44.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 Processor -- start 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- start start 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: init 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 0x7f4bb81cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-c con 0x7f4bb80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb8138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-a con 0x7f4bb80a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:44.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb813d2f0 0x7f4bb809e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): picked mon.noname-b con 0x7f4bb813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb81439b0 con 0x7f4bb80a5bd0 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb8143830 con 0x7f4bb813d2f0 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb8143b30 con 0x7f4bb80a50e0 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbea20640 10 monclient(hunting): authenticate will time out at 1357.242239s 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 0x7f4bb81cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 0x7f4bb81cfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58622/0 (socket says 172.21.15.86:58622) 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 -- 172.21.15.86:0/2328606561 learned_addr learned my addr 172.21.15.86:0/2328606561 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): get_auth_request con 0x7f4bb80a50e0 auth_method 0 2024-09-17T15:33:44.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb8138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request con 0x7f4bb80a5bd0 auth_method 0 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb813d2f0 0x7f4bb809e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 10 monclient(hunting): get_auth_request con 0x7f4bb813d2f0 auth_method 0 2024-09-17T15:33:44.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient(hunting): handle_auth_done global_id 74099 payload 824 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb813d2f0 msgr2=0x7f4bb809e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb813d2f0 0x7f4bb809e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a5bd0 msgr2=0x7f4bb8138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb80a5bd0 0x7f4bb8138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 monclient: found mon.noname-c 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bb7fff640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb80a0e40 con 0x7f4bb80a50e0 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.275+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4bb813d2f0 0x7f4bb809e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb7fff640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 msgr2=0x7f4bb81cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb7fff640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 0x7f4bb81cfef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c4a0 tx=0x7f4ba800c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8017070 con 0x7f4bb80a50e0 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: dump: 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8007070 con 0x7f4bb80a50e0 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8007460 con 0x7f4bb80a50e0 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 5 monclient: authenticate success, global_id 74099 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 monclient: _renew_subs 2024-09-17T15:33:44.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb80a18d0 con 0x7f4bb80a50e0 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb5ffb640 4 set_mon_vals no callback set 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 monclient: _renew_subs 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb80a0240 con 0x7f4bb80a50e0 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 10 monclient: dump: 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4ba801a070 con 0x7f4bb80a50e0 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bb60 0x7f4ba403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4799+0+0 (secure 0 0 0) 0x7f4ba804f3d0 con 0x7f4bb80a50e0 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 0x7f4bb809ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb809cd60 con 0x7f4bb8121800 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bb60 0x7f4ba403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bb77fe640 10 monclient: get_auth_request con 0x7f4ba403bb60 auth_method 0 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 0x7f4bb809ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient: get_auth_request con 0x7f4bb8121800 auth_method 0 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.279+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_done global_id 4941 payload 824 2024-09-17T15:33:44.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bb8138ea0 con 0x7f4bb8121800 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb77fe640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bb60 0x7f4ba403e020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80aa620 tx=0x7f4bac0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 msgr2=0x7f4bb809ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 0x7f4bb809ee00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e7d0 tx=0x7f4bb000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000cb50 con 0x7f4bb8121800 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 10 monclient: dump: 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.297 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4bb001a070 con 0x7f4bb8121800 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:44.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 msgr2=0x7f4bb809ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 0x7f4bb809ee00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e7d0 tx=0x7f4bb000ec90 comp rx=0 tx=0).stop 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 msgr2=0x7f4bb809ee00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.283+0000 7f4bb4ff9640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb8121800 0x7f4bb809ee00 secure :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e7d0 tx=0x7f4bb000ec90 comp rx=0 tx=0).stop 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbea20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbea20640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 0x7f4bb81392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb809f720 con 0x7f4bb809f340 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 0x7f4bb81392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient: get_auth_request con 0x7f4bb809f340 auth_method 0 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 10 monclient(hunting): handle_auth_done global_id 4944 payload 824 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4bb8120ec0 con 0x7f4bb809f340 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 msgr2=0x7f4bb81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bbcfbd640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 0x7f4bb81392d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e7d0 tx=0x7f4bb0003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0025070 con 0x7f4bb809f340 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 10 monclient: dump: 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f4bb0006b60 con 0x7f4bb809f340 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 msgr2=0x7f4bb81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 0x7f4bb81392d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e7d0 tx=0x7f4bb0003af0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 msgr2=0x7f4bb81392d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.291+0000 7f4bb4ff9640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb809f340 0x7f4bb81392d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 396934, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:33:44.302 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.303 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.304 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:44.305 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.306 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:44.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:44.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.310 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.312 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.313 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:44.314 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:44.315 INFO:tasks.rados.rados.0.smithi203.stdout:944: writing smithi20328086-34 from 682450 to 1171838 tid 1 2024-09-17T15:33:44.316 INFO:tasks.rados.rados.0.smithi203.stdout:944: writing smithi20328086-34 from 1814580 to 2311360 tid 2 2024-09-17T15:33:44.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bb60 msgr2=0x7f4ba403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4ba403bb60 0x7f4ba403e020 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80aa620 tx=0x7f4bac0a6000 comp rx=0 tx=0).stop 2024-09-17T15:33:44.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 10 monclient: shutdown 2024-09-17T15:33:44.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 msgr2=0x7f4bb81cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 --2- 172.21.15.86:0/2328606561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb80a50e0 0x7f4bb81cfef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c4a0 tx=0x7f4ba800c960 comp rx=0 tx=0).stop 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 reap_dead start 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbcfbd640 1 -- 172.21.15.86:0/2328606561 reap_dead start 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 shutdown_connections 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 >> 172.21.15.86:0/2328606561 conn(0x7f4bb80aaae0 msgr2=0x7f4bb80a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 shutdown_connections 2024-09-17T15:33:44.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.295+0000 7f4bbea20640 1 -- 172.21.15.86:0/2328606561 wait complete. 2024-09-17T15:33:44.317 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:44.317 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:33:44.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:33:44.334 INFO:tasks.rados.rados.0.smithi203.stdout:944: writing smithi20328086-34 from 2878518 to 3666546 tid 3 2024-09-17T15:33:44.334 INFO:tasks.rados.rados.0.smithi203.stdout:945: write oid 2 current snap is 102 2024-09-17T15:33:44.334 INFO:tasks.rados.rados.0.smithi203.stdout:945: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-17T15:33:44.348 INFO:tasks.rados.rados.0.smithi203.stdout:945: writing smithi20328086-2 from 699257 to 1263894 tid 1 2024-09-17T15:33:44.359 INFO:tasks.rados.rados.0.smithi203.stdout:945: writing smithi20328086-2 from 1873062 to 2513500 tid 2 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:945: writing smithi20328086-2 from 3305941 to 3899257 tid 3 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:946: copy_from oid 42 from oid 41 current snap is 102 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:943: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:943: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:943: finishing rollback tid 2 to smithi20328086-13 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 205 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:33:44.370 INFO:tasks.rados.rados.0.smithi203.stdout:944: finishing write tid 1 to smithi20328086-34 2024-09-17T15:33:44.371 INFO:tasks.rados.rados.0.smithi203.stdout:943: done (3 left) 2024-09-17T15:33:44.371 INFO:tasks.rados.rados.0.smithi203.stdout:947: write oid 36 current snap is 102 2024-09-17T15:33:44.371 INFO:tasks.rados.rados.0.smithi203.stdout:947: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-17T15:33:44.381 INFO:tasks.rados.rados.0.smithi203.stdout:947: writing smithi20328086-36 from 716064 to 1355950 tid 1 2024-09-17T15:33:44.389 INFO:tasks.rados.rados.0.smithi203.stdout:947: writing smithi20328086-36 from 1815191 to 2315640 tid 2 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:947: writing smithi20328086-36 from 2817011 to 3343936 tid 3 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:948: rollback oid 38 current snap is 102 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 94 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:944: finishing write tid 2 to smithi20328086-34 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:944: finishing write tid 3 to smithi20328086-34 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:944: finishing write tid 4 to smithi20328086-34 2024-09-17T15:33:44.399 INFO:tasks.rados.rados.0.smithi203.stdout:944: finishing write tid 5 to smithi20328086-34 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: oid 34 updating version 0 to 148 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: oid 34 version 148 is already newer than 146 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: oid 34 version 148 is already newer than 147 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: oid 34 updating version 148 to 149 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 149 (ObjNum 350 snap 102 seq_num 350) dirty exists 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: left oid 34 (ObjNum 350 snap 102 seq_num 350) 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:944: done (4 left) 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:949: copy_from oid 45 from oid 33 current snap is 102 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:945: finishing write tid 1 to smithi20328086-2 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:945: finishing write tid 2 to smithi20328086-2 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:946: finishing copy_from to smithi20328086-42 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:946: got expected ENOENT (src dne) 2024-09-17T15:33:44.400 INFO:tasks.rados.rados.0.smithi203.stdout:946: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: finishing write tid 3 to smithi20328086-2 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: finishing write tid 4 to smithi20328086-2 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:946: done (4 left) 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:950: delete oid 23 current snap is 102 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: finishing write tid 5 to smithi20328086-2 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: oid 2 updating version 0 to 255 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: oid 2 version 255 is already newer than 254 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: oid 2 updating version 255 to 257 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: oid 2 version 257 is already newer than 256 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 257 (ObjNum 351 snap 102 seq_num 351) dirty exists 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:945: left oid 2 (ObjNum 351 snap 102 seq_num 351) 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:947: finishing write tid 1 to smithi20328086-36 2024-09-17T15:33:44.401 INFO:tasks.rados.rados.0.smithi203.stdout:947: finishing write tid 2 to smithi20328086-36 2024-09-17T15:33:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:948: finishing rollback tid 0 to smithi20328086-38 2024-09-17T15:33:44.403 INFO:tasks.rados.rados.0.smithi203.stdout:948: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:33:44.404 INFO:tasks.rados.rados.0.smithi203.stdout:947: finishing write tid 3 to smithi20328086-36 2024-09-17T15:33:44.404 INFO:tasks.rados.rados.0.smithi203.stdout:948: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:33:44.404 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 209 (ObjNum 323 snap 93 seq_num 323) dirty exists 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: finishing write tid 4 to smithi20328086-36 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: finishing write tid 5 to smithi20328086-36 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: oid 36 updating version 0 to 387 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: oid 36 version 387 is already newer than 386 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: oid 36 version 387 is already newer than 384 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: oid 36 version 387 is already newer than 385 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 387 (ObjNum 352 snap 102 seq_num 352) dirty exists 2024-09-17T15:33:44.407 INFO:tasks.rados.rados.0.smithi203.stdout:947: left oid 36 (ObjNum 352 snap 102 seq_num 352) 2024-09-17T15:33:44.408 INFO:tasks.rados.rados.0.smithi203.stdout:945: done (4 left) 2024-09-17T15:33:44.408 INFO:tasks.rados.rados.0.smithi203.stdout:947: done (3 left) 2024-09-17T15:33:44.408 INFO:tasks.rados.rados.0.smithi203.stdout:948: done (2 left) 2024-09-17T15:33:44.408 INFO:tasks.rados.rados.0.smithi203.stdout:950: done (1 left) 2024-09-17T15:33:44.408 INFO:tasks.rados.rados.0.smithi203.stdout:951: snap_create 2024-09-17T15:33:44.415 INFO:tasks.rados.rados.0.smithi203.stdout:949: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:33:44.416 INFO:tasks.rados.rados.0.smithi203.stdout:949: finishing copy_from to smithi20328086-45 2024-09-17T15:33:44.416 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 182 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.435+0000 7f583c4ea640 10 monclient: get_monmap_and_config 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.435+0000 7f583c4ea640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.435+0000 7f583c4ea640 10 monclient: monmap: 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:44.439335+0000 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:44.439335+0000 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:44.439 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:44.439 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 Processor -- start 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 -- start start 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient: init 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f583406dee0 0x7f583406f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-a con 0x7f583406dee0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58340a2e70 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58340a3990 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58340a4950 con 0x7f58340a3990 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f583406ec00 con 0x7f58340a2e70 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f583406e2c0 con 0x7f583406dee0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583c4ea640 10 monclient(hunting): authenticate will time out at 1357.404824s 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583aa87640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f58340a3990 0x7f583413f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36308/0 (socket says 172.21.15.86:36308) 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583aa87640 1 -- 172.21.15.86:0/2281633008 learned_addr learned my addr 172.21.15.86:0/2281633008 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 1 --2- 172.21.15.86:0/2281633008 >> v2:172.21.15.86:3300/0 conn(0x7f583406dee0 0x7f583406f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): get_auth_request con 0x7f583406dee0 auth_method 0 2024-09-17T15:33:44.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient(hunting): handle_auth_done global_id 4947 payload 274 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 1 -- 172.21.15.86:0/2281633008 >> v1:172.21.15.203:6789/0 conn(0x7f58340a2e70 legacy=0x7f58340a3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 1 -- 172.21.15.86:0/2281633008 >> v1:172.21.15.86:6789/0 conn(0x7f58340a3990 legacy=0x7f583413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 1 monclient: found mon.noname-a 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:44.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.439+0000 7f583a286640 1 -- 172.21.15.86:0/2281633008 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583406eee0 con 0x7f583406dee0 2024-09-17T15:33:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583a286640 1 -- 172.21.15.86:0/2281633008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583406dee0 msgr2=0x7f583406f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583a286640 1 --2- 172.21.15.86:0/2281633008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583406dee0 0x7f583406f8c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f582c0097c0 tx=0x7f582c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3314b3105bc9429 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 1 -- 172.21.15.86:0/2281633008 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c0b0070 con 0x7f583406dee0 2024-09-17T15:33:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: dump: 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 5 monclient: authenticate success, global_id 4947 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 1 -- 172.21.15.86:0/2281633008 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f582c0ab600 con 0x7f583406dee0 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 1 -- 172.21.15.86:0/2281633008 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c0ab9d0 con 0x7f583406dee0 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 4 set_mon_vals no callback set 2024-09-17T15:33:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f5839284640 10 monclient: dump: 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 10 monclient: shutdown 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- 172.21.15.86:0/2281633008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583406dee0 msgr2=0x7f583406f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 --2- 172.21.15.86:0/2281633008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583406dee0 0x7f583406f8c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f582c0097c0 tx=0x7f582c01c360 comp rx=0 tx=0).stop 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- 172.21.15.86:0/2281633008 shutdown_connections 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 --2- 172.21.15.86:0/2281633008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f583406dee0 0x7f583406f8c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- 172.21.15.86:0/2281633008 >> 172.21.15.86:0/2281633008 conn(0x7f583409bc70 msgr2=0x7f58340a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- 172.21.15.86:0/2281633008 shutdown_connections 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- 172.21.15.86:0/2281633008 wait complete. 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 10 monclient: monmap: 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:44.452 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 Processor -- start 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 1 -- start start 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.443+0000 7f583c4ea640 10 monclient: init 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 0x7f58341d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-c con 0x7f583406dee0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58340a2e70 0x7f583413f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-b con 0x7f58340a2e70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 0x7f583413fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): picked mon.noname-a con 0x7f58340a3990 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5834143b10 con 0x7f58340a3990 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5834143990 con 0x7f58340a2e70 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5834143c90 con 0x7f583406dee0 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient(hunting): authenticate will time out at 1357.412256s 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 0x7f58341d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 0x7f583413fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 0x7f583413fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49108/0 (socket says 172.21.15.86:49108) 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 1 -- 172.21.15.86:0/1207290221 learned_addr learned my addr 172.21.15.86:0/1207290221 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): get_auth_request con 0x7f58340a3990 auth_method 0 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): get_auth_request con 0x7f583406dee0 auth_method 0 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58340a2e70 0x7f583413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient(hunting): handle_auth_done global_id 74102 payload 824 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58340a2e70 msgr2=0x7f583413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58340a2e70 0x7f583413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 msgr2=0x7f583413fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 0x7f583413fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 monclient: found mon.noname-c 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5834140590 con 0x7f583406dee0 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5839a85640 10 monclient: get_auth_request con 0x7f58340a2e70 auth_method 0 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58340a2e70 0x7f583413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583aa87640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f58340a3990 0x7f583413fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 msgr2=0x7f58341d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583a286640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 0x7f58341d0180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ab000 tx=0x7f582c01aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c0b0050 con 0x7f583406dee0 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: dump: 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f582c0bc070 con 0x7f583406dee0 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 5 monclient: authenticate success, global_id 74102 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _renew_subs 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583409f660 con 0x7f583406dee0 2024-09-17T15:33:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f582c0aad30 con 0x7f583406dee0 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5832ffd640 10 monclient: dump: 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.459 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f5838a83640 4 set_mon_vals no callback set 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _renew_subs 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f583413bdc0 con 0x7f583406dee0 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.447+0000 7f583c4ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583c4ea640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 0x7f583413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f583413dda0 con 0x7f583413a300 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 0x7f583413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f582c0c1460 con 0x7f583406dee0 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient: get_auth_request con 0x7f583413a300 auth_method 0 2024-09-17T15:33:44.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 10 monclient(hunting): handle_auth_done global_id 74105 payload 824 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f581803bb60 0x7f581803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f583406f8c0 con 0x7f583413a300 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4799+0+0 (secure 0 0 0) 0x7f582c0eaab0 con 0x7f583406dee0 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583aa87640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f581803bb60 0x7f581803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583aa87640 10 monclient: get_auth_request con 0x7f581803bb60 auth_method 0 2024-09-17T15:33:44.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 msgr2=0x7f583413a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 0x7f583413a6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f583413c8b0 tx=0x7f582800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828006d60 con 0x7f583413a300 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 10 monclient: dump: 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.464 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5828006d60 con 0x7f583413a300 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 msgr2=0x7f583413a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 0x7f583413a6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f583413c8b0 tx=0x7f582800dcc0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 msgr2=0x7f583413a6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f5832ffd640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413a300 0x7f583413a6e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f583413c8b0 tx=0x7f582800dcc0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.451+0000 7f583aa87640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f581803bb60 0x7f581803e020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5820007d60 tx=0x7f5820007cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f583c4ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f583c4ea640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 0x7f583409c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f583c4ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f583406a0e0 con 0x7f583413ac20 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 0x7f583409c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient: get_auth_request con 0x7f583413ac20 auth_method 0 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 10 monclient(hunting): handle_auth_done global_id 74108 payload 824 2024-09-17T15:33:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 1 -- 172.21.15.86:0/1207290221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f583413c920 con 0x7f583413ac20 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 msgr2=0x7f583409c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.459+0000 7f5839a85640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 0x7f583409c370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5828017000 tx=0x7f5828003af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828028020 con 0x7f583413ac20 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 10 monclient: dump: 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f582801a5f0 con 0x7f583413ac20 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 msgr2=0x7f583409c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 0x7f583409c370 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5828017000 tx=0x7f5828003af0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 msgr2=0x7f583409c370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f5832ffd640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583413ac20 0x7f583409c370 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5828017000 tx=0x7f5828003af0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 5251, 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:44.469 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.470 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.471 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.472 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.473 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.474 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.475 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:44.476 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.479 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.480 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:44.481 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:44.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f581803bb60 msgr2=0x7f581803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f581803bb60 0x7f581803e020 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5820007d60 tx=0x7f5820007cb0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 10 monclient: shutdown 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 msgr2=0x7f58341d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 --2- 172.21.15.86:0/1207290221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f583406dee0 0x7f58341d0180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f582c0ab000 tx=0x7f582c01aba0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583aa87640 1 -- 172.21.15.86:0/1207290221 reap_dead start 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583aa87640 1 -- 172.21.15.86:0/1207290221 reap_dead start 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 shutdown_connections 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 >> 172.21.15.86:0/1207290221 conn(0x7f583409bc70 msgr2=0x7f58340a1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 shutdown_connections 2024-09-17T15:33:44.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.463+0000 7f583c4ea640 1 -- 172.21.15.86:0/1207290221 wait complete. 2024-09-17T15:33:44.484 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:44.484 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:33:44.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:33:44.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: get_monmap_and_config 2024-09-17T15:33:44.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: monmap: 2024-09-17T15:33:44.599 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.599 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:44.600146+0000 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:44.600146+0000 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:44.600 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 Processor -- start 2024-09-17T15:33:44.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 -- start start 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: init 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6180a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-b con 0x7fd6180a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd61813d2f0 0x7fd61813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-c con 0x7fd61813d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6180a6280 con 0x7fd6180a50e0 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd61809d2a0 con 0x7fd61813d2f0 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd61809c0a0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61fcf2640 10 monclient(hunting): authenticate will time out at 1357.565731s 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58672/0 (socket says 172.21.15.86:58672) 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 -- 172.21.15.86:0/503812630 learned_addr learned my addr 172.21.15.86:0/503812630 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): get_auth_request con 0x7fd6180a5bd0 auth_method 0 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61e28f640 1 --2- 172.21.15.86:0/503812630 >> v2:172.21.15.203:3300/0 conn(0x7fd61813d2f0 0x7fd61813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61e28f640 10 monclient(hunting): get_auth_request con 0x7fd61813d2f0 auth_method 0 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61e28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61e28f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61e28f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient(hunting): handle_auth_done global_id 74111 payload 274 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 -- 172.21.15.86:0/503812630 >> v2:172.21.15.203:3300/0 conn(0x7fd61813d2f0 msgr2=0x7fd61813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 --2- 172.21.15.86:0/503812630 >> v2:172.21.15.203:3300/0 conn(0x7fd61813d2f0 0x7fd61813f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 -- 172.21.15.86:0/503812630 >> v1:172.21.15.203:6789/0 conn(0x7fd6180a50e0 legacy=0x7fd6180a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 monclient: found mon.noname-b 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.599+0000 7fd61d28d640 1 -- 172.21.15.86:0/503812630 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61809c380 con 0x7fd6180a5bd0 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61d28d640 1 -- 172.21.15.86:0/503812630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 msgr2=0x7fd61809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61d28d640 1 --2- 172.21.15.86:0/503812630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd614009870 tx=0x7fd61401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=228817d99f8bd817 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 1 -- 172.21.15.86:0/503812630 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b0070 con 0x7fd6180a5bd0 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: dump: 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 5 monclient: authenticate success, global_id 74111 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 1 -- 172.21.15.86:0/503812630 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140ab600 con 0x7fd6180a5bd0 2024-09-17T15:33:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 1 -- 172.21.15.86:0/503812630 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140ab9d0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 4 set_mon_vals no callback set 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61ca8c640 10 monclient: dump: 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 10 monclient: shutdown 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 -- 172.21.15.86:0/503812630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 msgr2=0x7fd61809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/503812630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd614009870 tx=0x7fd61401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 -- 172.21.15.86:0/503812630 shutdown_connections 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/503812630 >> v2:172.21.15.203:3300/0 conn(0x7fd61813d2f0 0x7fd61813f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/503812630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd61809cd60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 -- 172.21.15.86:0/503812630 >> 172.21.15.86:0/503812630 conn(0x7fd6180aaae0 msgr2=0x7fd6180aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 -- 172.21.15.86:0/503812630 shutdown_connections 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 1 -- 172.21.15.86:0/503812630 wait complete. 2024-09-17T15:33:44.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.603+0000 7fd61fcf2640 10 monclient: monmap: 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 Processor -- start 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- start start 2024-09-17T15:33:44.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: init 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a50e0 0x7fd6181cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-b con 0x7fd6180a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd618138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-c con 0x7fd6180a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61813d2f0 0x7fd61809f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): picked mon.noname-a con 0x7fd61813d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd618142650 con 0x7fd61813d2f0 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6180a6b00 con 0x7fd6180a50e0 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6181427d0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient(hunting): authenticate will time out at 1357.572870s 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd618138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61813d2f0 0x7fd61809f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd618138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58680/0 (socket says 172.21.15.86:58680) 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 -- 172.21.15.86:0/2899663627 learned_addr learned my addr 172.21.15.86:0/2899663627 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): get_auth_request con 0x7fd6180a5bd0 auth_method 0 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): get_auth_request con 0x7fd61813d2f0 auth_method 0 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61e28f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61da8e640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a50e0 0x7fd6181cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61da8e640 10 monclient(hunting): get_auth_request con 0x7fd6180a50e0 auth_method 0 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61da8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61da8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient(hunting): handle_auth_done global_id 74114 payload 824 2024-09-17T15:33:44.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a50e0 msgr2=0x7fd6181cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a50e0 0x7fd6181cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61813d2f0 msgr2=0x7fd61809f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd61813d2f0 0x7fd61809f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 monclient: found mon.noname-c 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd618122bd0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 msgr2=0x7fd618138240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61d28d640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd618138240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6140ae040 tx=0x7fd614019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b0040 con 0x7fd6180a5bd0 2024-09-17T15:33:44.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: dump: 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 5 monclient: authenticate success, global_id 74114 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _renew_subs 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6140b9070 con 0x7fd6180a5bd0 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6180a1290 con 0x7fd6180a5bd0 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6140b37d0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60e7fc640 10 monclient: dump: 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.622 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd60ffff640 4 set_mon_vals no callback set 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _renew_subs 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.607+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6180a18d0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61fcf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 0x7fd61809fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd6140b39f0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd61806a0e0 con 0x7fd618138780 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd60e7fc640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd5fc03bb60 0x7fd5fc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4799+0+0 (secure 0 0 0) 0x7fd6140e9df0 con 0x7fd6180a5bd0 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 0x7fd61809fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient: get_auth_request con 0x7fd618138780 auth_method 0 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61e28f640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd5fc03bb60 0x7fd5fc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61e28f640 10 monclient: get_auth_request con 0x7fd5fc03bb60 auth_method 0 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 10 monclient(hunting): handle_auth_done global_id 54247 payload 824 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.611+0000 7fd61da8e640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd618122570 con 0x7fd618138780 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd61da8e640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 msgr2=0x7fd61809fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd61da8e640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 0x7fd61809fdb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd60800e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd61e28f640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd5fc03bb60 0x7fd5fc03e020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6180a8520 tx=0x7fd610009820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608007450 con 0x7fd618138780 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 10 monclient: dump: 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.625 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd608007450 con 0x7fd618138780 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 msgr2=0x7fd61809fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 0x7fd61809fdb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd60800e910 comp rx=0 tx=0).stop 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 msgr2=0x7fd61809fdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.615+0000 7fd60e7fc640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd618138780 0x7fd61809fdb0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd60800e910 comp rx=0 tx=0).stop 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.619+0000 7fd61fcf2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.619+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 0x7fd6181392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.619+0000 7fd61fcf2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.619+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6180a0700 con 0x7fd6180a02f0 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 0x7fd6181392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient: get_auth_request con 0x7fd6180a02f0 auth_method 0 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 10 monclient(hunting): handle_auth_done global_id 54250 payload 824 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 1 -- 172.21.15.86:0/2899663627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd61809cd60 con 0x7fd6180a02f0 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 msgr2=0x7fd6181392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd61da8e640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 0x7fd6181392d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd608003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60801eb70 con 0x7fd6180a02f0 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 10 monclient: dump: 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fd60801eb70 con 0x7fd6180a02f0 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.623+0000 7fd60e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd60e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 msgr2=0x7fd6181392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd60e7fc640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 0x7fd6181392d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd608003770 comp rx=0 tx=0).stop 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd60e7fc640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 msgr2=0x7fd6181392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd60e7fc640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6180a02f0 0x7fd6181392d0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd60800e450 tx=0x7fd608003770 comp rx=0 tx=0).stop 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 100054, 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.632 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.633 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.634 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:44.635 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:44.636 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:44.637 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.638 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.640 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:44.641 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.642 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.643 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:44.644 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:44.645 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd5fc03bb60 msgr2=0x7fd5fc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd5fc03bb60 0x7fd5fc03e020 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd6180a8520 tx=0x7fd610009820 comp rx=0 tx=0).stop 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 10 monclient: shutdown 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 msgr2=0x7fd618138240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 --2- 172.21.15.86:0/2899663627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6180a5bd0 0x7fd618138240 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd6140ae040 tx=0x7fd614019190 comp rx=0 tx=0).stop 2024-09-17T15:33:44.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61e28f640 1 -- 172.21.15.86:0/2899663627 reap_dead start 2024-09-17T15:33:44.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61e28f640 1 -- 172.21.15.86:0/2899663627 reap_dead start 2024-09-17T15:33:44.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 shutdown_connections 2024-09-17T15:33:44.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 >> 172.21.15.86:0/2899663627 conn(0x7fd6180aaae0 msgr2=0x7fd61813de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 shutdown_connections 2024-09-17T15:33:44.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.627+0000 7fd61fcf2640 1 -- 172.21.15.86:0/2899663627 wait complete. 2024-09-17T15:33:44.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:44.647 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:33:44.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:33:44.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 10 monclient: get_monmap_and_config 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 10 monclient: monmap: 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:44.762943+0000 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:44.762943+0000 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:44.762 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:44.763 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 1 Processor -- start 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 1 -- start start 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.759+0000 7f9aba00b640 10 monclient: init 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9ab40689d0 0x7f9ab406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab40689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9ab4068db0 0x7f9ab4069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab4068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ab4135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab409c8b0 con 0x7f9ab4135ae0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab409cb90 con 0x7f9ab4068db0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab409ce70 con 0x7f9ab40689d0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aba00b640 10 monclient(hunting): authenticate will time out at 1357.728372s 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9ab4068db0 0x7f9ab4069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9ab4068db0 0x7f9ab4069200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49122/0 (socket says 172.21.15.86:49122) 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 -- 172.21.15.86:0/1581482462 learned_addr learned my addr 172.21.15.86:0/1581482462 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): get_auth_request con 0x7f9ab4068db0 auth_method 0 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9ab37fe640 1 --2- 172.21.15.86:0/1581482462 >> v2:172.21.15.203:3300/0 conn(0x7f9ab40689d0 0x7f9ab406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9ab37fe640 10 monclient(hunting): get_auth_request con 0x7f9ab40689d0 auth_method 0 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9ab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9ab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_done global_id 4956 payload 274 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 -- 172.21.15.86:0/1581482462 >> v2:172.21.15.203:3300/0 conn(0x7f9ab40689d0 msgr2=0x7f9ab406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 --2- 172.21.15.86:0/1581482462 >> v2:172.21.15.203:3300/0 conn(0x7f9ab40689d0 0x7f9ab406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 -- 172.21.15.86:0/1581482462 >> v1:172.21.15.203:6789/0 conn(0x7f9ab4135ae0 legacy=0x7f9ab4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 monclient: found mon.noname-a 2024-09-17T15:33:44.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:44.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.763+0000 7f9aaaffd640 1 -- 172.21.15.86:0/1581482462 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab4069740 con 0x7f9ab4068db0 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 1 -- 172.21.15.86:0/1581482462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 msgr2=0x7f9ab4069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 1 --2- 172.21.15.86:0/1581482462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4069200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4069a70 tx=0x7f9a9c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f37f33947a36edb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 1 -- 172.21.15.86:0/1581482462 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0b0070 con 0x7f9ab4068db0 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: dump: 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.774 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 1 -- 172.21.15.86:0/1581482462 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c0ab600 con 0x7f9ab4068db0 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 5 monclient: authenticate success, global_id 4956 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 1 -- 172.21.15.86:0/1581482462 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c0ab9d0 con 0x7f9ab4068db0 2024-09-17T15:33:44.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 4 set_mon_vals no callback set 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab2ffd640 10 monclient: dump: 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: shutdown 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- 172.21.15.86:0/1581482462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 msgr2=0x7f9ab4069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- 172.21.15.86:0/1581482462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4069200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4069a70 tx=0x7f9a9c01c470 comp rx=0 tx=0).stop 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- 172.21.15.86:0/1581482462 shutdown_connections 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- 172.21.15.86:0/1581482462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4069200 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- 172.21.15.86:0/1581482462 >> v2:172.21.15.203:3300/0 conn(0x7f9ab40689d0 0x7f9ab406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- 172.21.15.86:0/1581482462 >> 172.21.15.86:0/1581482462 conn(0x7f9ab40a6440 msgr2=0x7f9ab40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- 172.21.15.86:0/1581482462 shutdown_connections 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- 172.21.15.86:0/1581482462 wait complete. 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: build_initial_monmap 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: monmap: 2024-09-17T15:33:44.777 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 Processor -- start 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- start start 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: init 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:44.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab40689d0 0x7f9ab413cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 0x7f9ab41d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab4135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab4140a40 con 0x7f9ab4068db0 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab409c2d0 con 0x7f9ab40689d0 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab4140bc0 con 0x7f9ab4135ae0 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aba00b640 10 monclient(hunting): authenticate will time out at 1357.735518s 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49138/0 (socket says 172.21.15.86:49138) 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 1 -- 172.21.15.86:0/2004989205 learned_addr learned my addr 172.21.15.86:0/2004989205 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): get_auth_request con 0x7f9ab4068db0 auth_method 0 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 0x7f9ab41d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): get_auth_request con 0x7f9ab4135ae0 auth_method 0 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9aaaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab37fe640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab40689d0 0x7f9ab413cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab37fe640 10 monclient(hunting): get_auth_request con 0x7f9ab40689d0 auth_method 0 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:44.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.767+0000 7f9ab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 10 monclient(hunting): handle_auth_done global_id 74117 payload 824 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 -- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab40689d0 msgr2=0x7f9ab413cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab40689d0 0x7f9ab413cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 -- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 msgr2=0x7f9ab4134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 monclient: found mon.noname-c 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 -- 172.21.15.86:0/2004989205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab4121600 con 0x7f9ab4135ae0 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aaaffd640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:44.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 -- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 msgr2=0x7f9ab41d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab3fff640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 0x7f9ab41d01a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c780 tx=0x7f9aac00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac016020 con 0x7f9ab4135ae0 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: dump: 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.783 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aac00ac60 con 0x7f9ab4135ae0 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac012660 con 0x7f9ab4135ae0 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 10 monclient: dump: 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 5 monclient: authenticate success, global_id 74117 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _renew_subs 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab40a2820 con 0x7f9ab4135ae0 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab27fc640 4 set_mon_vals no callback set 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _renew_subs 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab40736e0 con 0x7f9ab4135ae0 2024-09-17T15:33:44.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:44.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab406f840 con 0x7f9ab4135ae0 2024-09-17T15:33:44.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9aac019020 con 0x7f9ab4135ae0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.771+0000 7f9ab0ff9640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 0x7f9a9403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.775+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4799+0+0 (secure 0 0 0) 0x7f9aac04fda0 con 0x7f9ab4135ae0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.775+0000 7f9ab37fe640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 0x7f9a9403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.775+0000 7f9ab37fe640 10 monclient: get_auth_request con 0x7f9a9403bbb0 auth_method 0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.775+0000 7f9ab37fe640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 0x7f9a9403e070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400b9d0 tx=0x7f9aa4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.779+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9aac020030 con 0x7f9ab4135ae0 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.779+0000 7f9ab0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:44.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:44.779+0000 7f9ab0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:45.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.015+0000 7f9aba00b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:33:45.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.015+0000 7f9aba00b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:33:45.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.015+0000 7f9aba00b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:45.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.015+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9ab406d4f0 con 0x7f9ab4135ae0 2024-09-17T15:33:45.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9ab0ff9640 1 -- 172.21.15.86:0/2004989205 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9aac00ae10 con 0x7f9ab4135ae0 2024-09-17T15:33:45.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9ab0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:33:45.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9ab0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=12,kv=11,logm=62} crc {auth=3325915006,config=555582061,health=1068221574,kv=3591778543,logm=2682831399}) 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=1,health=12,kv=9,logm=65} crc {auth=2217208065,config=555582061,health=1068221574,kv=3210360986,logm=1369265261}) 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=1,health=12,kv=11,logm=62} crc {auth=3325915006,config=555582061,health=1068221574,kv=3591778543,logm=2682831399}) 2024-09-17T15:33:45.020 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=1,health=12,kv=11,logm=63} crc {auth=1024579655,config=555582061,health=1068221574,kv=3926014085,logm=2941334898}) 2024-09-17T15:33:45.021 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.021 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765161682}) 2024-09-17T15:33:45.021 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868843498}) 2024-09-17T15:33:45.022 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.022 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2765161682}) 2024-09-17T15:33:45.022 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.019+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473529132}) 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 msgr2=0x7f9a9403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 0x7f9a9403e070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400b9d0 tx=0x7f9aa4008040 comp rx=0 tx=0).stop 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 10 monclient: shutdown 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 msgr2=0x7f9ab41d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 0x7f9ab41d01a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9aac00c780 tx=0x7f9aac00cc40 comp rx=0 tx=0).stop 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 shutdown_connections 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a9403bbb0 0x7f9a9403e070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab4135ae0 0x7f9ab41d01a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:45.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab4068db0 0x7f9ab4134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:45.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 --2- 172.21.15.86:0/2004989205 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ab40689d0 0x7f9ab413cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:45.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 >> 172.21.15.86:0/2004989205 conn(0x7f9ab40a6440 msgr2=0x7f9ab40a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:45.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 shutdown_connections 2024-09-17T15:33:45.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:45.019+0000 7f9aba00b640 1 -- 172.21.15.86:0/2004989205 wait complete. 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702361095}) 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2989955312}) 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702361095}) 2024-09-17T15:33:45.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875059129}) 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853213441}) 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187451087}) 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853213441}) 2024-09-17T15:33:45.026 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.023+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740004779}) 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403204115}) 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498939533}) 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403204115}) 2024-09-17T15:33:45.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260857253}) 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479259728}) 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1575242570}) 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479259728}) 2024-09-17T15:33:45.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.027+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3799356667}) 2024-09-17T15:33:45.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962078734}) 2024-09-17T15:33:45.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1451509372}) 2024-09-17T15:33:45.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962078734}) 2024-09-17T15:33:45.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335378688}) 2024-09-17T15:33:45.032 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.032 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638907976}) 2024-09-17T15:33:45.033 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524007526}) 2024-09-17T15:33:45.033 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.033 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638907976}) 2024-09-17T15:33:45.033 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2055693383}) 2024-09-17T15:33:45.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934832745}) 2024-09-17T15:33:45.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3738670796}) 2024-09-17T15:33:45.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934832745}) 2024-09-17T15:33:45.035 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.031+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3551765141}) 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417049494}) 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352725813}) 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417049494}) 2024-09-17T15:33:45.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258136533}) 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811160694}) 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=205633237}) 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=811160694}) 2024-09-17T15:33:45.038 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.035+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3363671966}) 2024-09-17T15:33:45.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183174935}) 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937918875}) 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183174935}) 2024-09-17T15:33:45.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1583012117}) 2024-09-17T15:33:45.042 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.042 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976363343}) 2024-09-17T15:33:45.042 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013283127}) 2024-09-17T15:33:45.042 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.043 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976363343}) 2024-09-17T15:33:45.043 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.039+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734903195}) 2024-09-17T15:33:45.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.045 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366851772}) 2024-09-17T15:33:45.045 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1476250582}) 2024-09-17T15:33:45.045 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.045 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366851772}) 2024-09-17T15:33:45.045 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.043+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614506780}) 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917557367}) 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593421017}) 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917557367}) 2024-09-17T15:33:45.047 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4065483942}) 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485439282}) 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128140972}) 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=485439282}) 2024-09-17T15:33:45.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.047+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810732046}) 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25} crc {logm=3013058240,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2855625506,mon_config_key=2265748168,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=3616902286,osd_pg_creating=2524813190,osd_snap=930829469}) 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=28,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21} crc {logm=1598697913,mdsmap=3140514009,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2855625506,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2524813190,osd_snap=802320778}) 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25} crc {logm=3013058240,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2855625506,mon_config_key=2265748168,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=3616902286,osd_pg_creating=2524813190,osd_snap=930829469}) 2024-09-17T15:33:45.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=2967957487,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3467918178,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2524813190,osd_snap=917715605}) 2024-09-17T15:33:45.054 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.054 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=3229859071,osdmap=2320060383}) 2024-09-17T15:33:45.055 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=3229859071,osdmap=2820670834}) 2024-09-17T15:33:45.055 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.055 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.051+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=3229859071,osdmap=2320060383}) 2024-09-17T15:33:45.055 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=258256856,osdmap=4148577567}) 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2085039648}) 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3252227105}) 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2085039648}) 2024-09-17T15:33:45.057 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.055+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=992625662}) 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1487466578}) 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2319102289}) 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1487466578}) 2024-09-17T15:33:45.061 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.059+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1948614335}) 2024-09-17T15:33:45.069 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.069 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=3124665831}) 2024-09-17T15:33:45.069 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=220388785}) 2024-09-17T15:33:45.069 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:33:45.070 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=3124665831}) 2024-09-17T15:33:45.070 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:33:45.067+0000 7febaf6a4640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=2483056742}) 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:949: done (1 left) 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:951: done (0 left) 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:952: rollback oid 48 current snap is 103 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 101 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:953: read oid 31 snap 92 2024-09-17T15:33:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:953: expect deleted 2024-09-17T15:33:45.231 INFO:tasks.rados.rados.0.smithi203.stdout:954: snap_remove snap 96 2024-09-17T15:33:46.041 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:33:46.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 10 monclient: get_monmap_and_config 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 10 monclient: monmap: 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:46.170248+0000 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:46.170248+0000 2024-09-17T15:33:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:46.170 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 1 Processor -- start 2024-09-17T15:33:46.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 1 -- start start 2024-09-17T15:33:46.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.167+0000 7f69fb94d640 10 monclient: init 2024-09-17T15:33:46.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f69f40a50e0 0x7f69f409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-c con 0x7f69f40a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-b con 0x7f69f40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69f4137ea0 0x7f69f4138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-a con 0x7f69f4137ea0 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f69f409bdc0 con 0x7f69f4137ea0 2024-09-17T15:33:46.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f69f409d350 con 0x7f69f40a50e0 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69f40a6530 con 0x7f69f40a54c0 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69fb94d640 10 monclient(hunting): authenticate will time out at 1359.136367s 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69f4137ea0 0x7f69f4138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69f4137ea0 0x7f69f4138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49142/0 (socket says 172.21.15.86:49142) 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58690/0 (socket says 172.21.15.86:58690) 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 -- 172.21.15.86:0/1058333464 learned_addr learned my addr 172.21.15.86:0/1058333464 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): get_auth_request con 0x7f69f4137ea0 auth_method 0 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): get_auth_request con 0x7f69f40a54c0 auth_method 0 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 1 --2- 172.21.15.86:0/1058333464 >> v2:172.21.15.203:3300/0 conn(0x7f69f40a50e0 0x7f69f409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): get_auth_request con 0x7f69f40a50e0 auth_method 0 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient(hunting): handle_auth_done global_id 4962 payload 274 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 -- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 msgr2=0x7f69f40a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 -- 172.21.15.86:0/1058333464 >> v2:172.21.15.203:3300/0 conn(0x7f69f40a50e0 msgr2=0x7f69f409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 --2- 172.21.15.86:0/1058333464 >> v2:172.21.15.203:3300/0 conn(0x7f69f40a50e0 0x7f69f409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 monclient: found mon.noname-a 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f9eea640 1 -- 172.21.15.86:0/1058333464 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f409c150 con 0x7f69f4137ea0 2024-09-17T15:33:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.171+0000 7f69f8ee8640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69f9eea640 1 -- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f4137ea0 msgr2=0x7f69f4138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69f9eea640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f4137ea0 0x7f69f4138330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f69f40a8280 tx=0x7f69ec01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9889a7f408bed08a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 1 -- 172.21.15.86:0/1058333464 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0a5620 con 0x7f69f4137ea0 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: dump: 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.185 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 5 monclient: authenticate success, global_id 4962 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 1 -- 172.21.15.86:0/1058333464 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ec0a57c0 con 0x7f69f4137ea0 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 4 set_mon_vals no callback set 2024-09-17T15:33:46.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 1 -- 172.21.15.86:0/1058333464 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0a5b90 con 0x7f69f4137ea0 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69ebfff640 10 monclient: dump: 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: shutdown 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f4137ea0 msgr2=0x7f69f4138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f4137ea0 0x7f69f4138330 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f69f40a8280 tx=0x7f69ec01c8b0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- 172.21.15.86:0/1058333464 shutdown_connections 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f4137ea0 0x7f69f4138330 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- 172.21.15.86:0/1058333464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a54c0 0x7f69f40a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- 172.21.15.86:0/1058333464 >> v2:172.21.15.203:3300/0 conn(0x7f69f40a50e0 0x7f69f409ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- 172.21.15.86:0/1058333464 >> 172.21.15.86:0/1058333464 conn(0x7f69f40aaae0 msgr2=0x7f69f40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- 172.21.15.86:0/1058333464 shutdown_connections 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- 172.21.15.86:0/1058333464 wait complete. 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: monmap: 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 Processor -- start 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- start start 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: init 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-c con 0x7f69f40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f40a54c0 0x7f69f4136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-a con 0x7f69f40a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f4137ea0 0x7f69f413d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): picked mon.noname-b con 0x7f69f4137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f69f4141b20 con 0x7f69f40a54c0 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69f40a5f00 con 0x7f69f4137ea0 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69f4141ca0 con 0x7f69f40a50e0 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69fb94d640 10 monclient(hunting): authenticate will time out at 1359.143905s 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.175+0000 7f69f96e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58702/0 (socket says 172.21.15.86:58702) 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 -- 172.21.15.86:0/455900604 learned_addr learned my addr 172.21.15.86:0/455900604 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): get_auth_request con 0x7f69f40a50e0 auth_method 0 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f40a54c0 0x7f69f4136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): get_auth_request con 0x7f69f40a54c0 auth_method 0 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f9eea640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f4137ea0 0x7f69f413d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f9eea640 10 monclient(hunting): get_auth_request con 0x7f69f4137ea0 auth_method 0 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f9eea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f9eea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f9eea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f8ee8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient(hunting): handle_auth_done global_id 74123 payload 824 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 -- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f4137ea0 msgr2=0x7f69f413d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f4137ea0 0x7f69f413d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 -- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f40a54c0 msgr2=0x7f69f4136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f40a54c0 0x7f69f4136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 monclient: found mon.noname-c 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 -- 172.21.15.86:0/455900604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69f413ea50 con 0x7f69f40a50e0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 -- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 msgr2=0x7f69f41369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69f96e9640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69e400e960 tx=0x7f69e400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4017070 con 0x7f69f40a50e0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: dump: 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.195 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 5 monclient: authenticate success, global_id 74123 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69e400cbb0 con 0x7f69f40a50e0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _renew_subs 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4013470 con 0x7f69f40a50e0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69f4120d30 con 0x7f69f40a50e0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69eb7fe640 4 set_mon_vals no callback set 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 10 monclient: dump: 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _renew_subs 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69f4121f40 con 0x7f69f40a50e0 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69f40a58d0 con 0x7f69f40a50e0 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.179+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f69e401a070 con 0x7f69f40a50e0 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.183+0000 7f69e9ffb640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 0x7f69d803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.183+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(215..215 src has 1..215) ==== 4771+0+0 (secure 0 0 0) 0x7f69e4050000 con 0x7f69f40a50e0 2024-09-17T15:33:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.183+0000 7f69f8ee8640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 0x7f69d803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.183+0000 7f69f8ee8640 10 monclient: get_auth_request con 0x7f69d803bb60 auth_method 0 2024-09-17T15:33:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.183+0000 7f69f8ee8640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 0x7f69d803e020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009580 tx=0x7f69f0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.187+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f69e40123d0 con 0x7f69f40a50e0 2024-09-17T15:33:46.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.187+0000 7f69e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.187+0000 7f69e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:46.236 INFO:tasks.rados.rados.0.smithi203.stdout:954: done (2 left) 2024-09-17T15:33:46.236 INFO:tasks.rados.rados.0.smithi203.stdout:955: rollback oid 15 current snap is 103 2024-09-17T15:33:46.236 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 102 2024-09-17T15:33:46.236 INFO:tasks.rados.rados.0.smithi203.stdout:952: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:952: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:952: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 304 (ObjNum 341 snap 99 seq_num 341) dirty exists 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:952: done (2 left) 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:953: done (1 left) 2024-09-17T15:33:46.237 INFO:tasks.rados.rados.0.smithi203.stdout:956: snap_create 2024-09-17T15:33:46.252 INFO:tasks.rados.rados.0.smithi203.stdout:955: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:33:46.252 INFO:tasks.rados.rados.0.smithi203.stdout:955: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:33:46.252 INFO:tasks.rados.rados.0.smithi203.stdout:955: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:33:46.252 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 253 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.427+0000 7f69fb94d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.427+0000 7f69fb94d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.427+0000 7f69fb94d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.427+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f69f4074be0 con 0x7f69f40a50e0 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69e9ffb640 1 -- 172.21.15.86:0/455900604 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f69e40232f0 con 0x7f69f40a50e0 2024-09-17T15:33:46.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:33:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:46.431 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:33:46.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 msgr2=0x7f69d803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 0x7f69d803e020 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f69f0009580 tx=0x7f69f0009190 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 10 monclient: shutdown 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 msgr2=0x7f69f41369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f69e400e960 tx=0x7f69e400ee20 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 shutdown_connections 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69d803bb60 0x7f69d803e020 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69f4137ea0 0x7f69f413d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69f40a54c0 0x7f69f4136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 --2- 172.21.15.86:0/455900604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69f40a50e0 0x7f69f41369b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 >> 172.21.15.86:0/455900604 conn(0x7f69f40aaae0 msgr2=0x7f69f409fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 shutdown_connections 2024-09-17T15:33:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.431+0000 7f69fb94d640 1 -- 172.21.15.86:0/455900604 wait complete. 2024-09-17T15:33:46.447 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:33:46.448 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:33:46.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:33:46.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: get_monmap_and_config 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: monmap: 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:46.568337+0000 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:46.568337+0000 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.567 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:46.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 Processor -- start 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 -- start start 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: init 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f145009d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 0x7f1450139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f145009e260 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f145013a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145009ff00 con 0x7f145009d770 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f145009e7f0 con 0x7f145009e260 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f14500a2110 con 0x7f145013a440 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f1454ef8640 10 monclient(hunting): authenticate will time out at 1359.533779s 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 0x7f1450139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58714/0 (socket says 172.21.15.86:58714) 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 -- 172.21.15.86:0/3227317104 learned_addr learned my addr 172.21.15.86:0/3227317104 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): get_auth_request con 0x7f145013a440 auth_method 0 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): get_auth_request con 0x7f145009e260 auth_method 0 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient(hunting): handle_auth_done global_id 74126 payload 274 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 -- 172.21.15.86:0/3227317104 >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 msgr2=0x7f1450139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 --2- 172.21.15.86:0/3227317104 >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 0x7f1450139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 -- 172.21.15.86:0/3227317104 >> v1:172.21.15.203:6789/0 conn(0x7f145009d770 legacy=0x7f145009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 monclient: found mon.noname-b 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144ed9d640 1 -- 172.21.15.86:0/3227317104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145009ec30 con 0x7f145013a440 2024-09-17T15:33:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.567+0000 7f144dd9b640 1 --2- 172.21.15.86:0/3227317104 >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 0x7f1450139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:46.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144ed9d640 1 -- 172.21.15.86:0/3227317104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 msgr2=0x7f145013c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144ed9d640 1 --2- 172.21.15.86:0/3227317104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f144800d150 tx=0x7f144801eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea5a5695f899dbe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 1 -- 172.21.15.86:0/3227317104 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144800b2d0 con 0x7f145013a440 2024-09-17T15:33:46.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: dump: 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 1 -- 172.21.15.86:0/3227317104 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144800b470 con 0x7f145013a440 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 1 -- 172.21.15.86:0/3227317104 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14480a78d0 con 0x7f145013a440 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 5 monclient: authenticate success, global_id 74126 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 4 set_mon_vals no callback set 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f144d59a640 10 monclient: dump: 2024-09-17T15:33:46.578 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 10 monclient: shutdown 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 -- 172.21.15.86:0/3227317104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 msgr2=0x7f145013c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 --2- 172.21.15.86:0/3227317104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f144800d150 tx=0x7f144801eaf0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 -- 172.21.15.86:0/3227317104 shutdown_connections 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 --2- 172.21.15.86:0/3227317104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145013a440 0x7f145013c830 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 --2- 172.21.15.86:0/3227317104 >> v2:172.21.15.86:3300/0 conn(0x7f145009e260 0x7f1450139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 -- 172.21.15.86:0/3227317104 >> 172.21.15.86:0/3227317104 conn(0x7f14500a2f40 msgr2=0x7f14500a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 -- 172.21.15.86:0/3227317104 shutdown_connections 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 1 -- 172.21.15.86:0/3227317104 wait complete. 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.571+0000 7f1454ef8640 10 monclient: monmap: 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:46.580 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 Processor -- start 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- start start 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: init 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 0x7f1450072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-c con 0x7f145009d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f145009e260 0x7f14501cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-b con 0x7f145009e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 0x7f1450070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): picked mon.noname-a con 0x7f145013a440 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f14500a1850 con 0x7f145013a440 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14500a16d0 con 0x7f145009e260 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f14500a19d0 con 0x7f145009d770 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient(hunting): authenticate will time out at 1359.540713s 2024-09-17T15:33:46.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 0x7f1450070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 0x7f1450072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 0x7f1450070730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49168/0 (socket says 172.21.15.86:49168) 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 1 -- 172.21.15.86:0/1320674630 learned_addr learned my addr 172.21.15.86:0/1320674630 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): get_auth_request con 0x7f145013a440 auth_method 0 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): get_auth_request con 0x7f145009d770 auth_method 0 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f145009e260 0x7f14501cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient(hunting): handle_auth_done global_id 74129 payload 824 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f145009e260 msgr2=0x7f14501cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f145009e260 0x7f14501cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 msgr2=0x7f1450070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 0x7f1450070730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 monclient: found mon.noname-c 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145009d460 con 0x7f145009d770 2024-09-17T15:33:46.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144ed9d640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f145013a440 0x7f1450070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:33:46.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144dd9b640 10 monclient: get_auth_request con 0x7f145009e260 auth_method 0 2024-09-17T15:33:46.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f145009e260 0x7f14501cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 msgr2=0x7f1450072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144e59c640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 0x7f1450072a70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f144000b440 tx=0x7f144000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440015020 con 0x7f145009d770 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: dump: 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.590 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 5 monclient: authenticate success, global_id 74129 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1440011a40 con 0x7f145009d770 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _renew_subs 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440011e30 con 0x7f145009d770 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1450073f90 con 0x7f145009d770 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f144cd99640 4 set_mon_vals no callback set 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f143effd640 10 monclient: dump: 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.592 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _renew_subs 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.575+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1450073400 con 0x7f145009d770 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f1454ef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f1454ef8640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 0x7f1450070c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f145009c780 con 0x7f14501cc6c0 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 0x7f1450070c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient: get_auth_request con 0x7f14501cc6c0 auth_method 0 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1440018020 con 0x7f145009d770 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 10 monclient(hunting): handle_auth_done global_id 4974 payload 824 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1450132030 con 0x7f14501cc6c0 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f143003bb60 0x7f143003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 4815+0+0 (secure 0 0 0) 0x7f144004fac0 con 0x7f145009d770 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144ed9d640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f143003bb60 0x7f143003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144ed9d640 10 monclient: get_auth_request con 0x7f143003bb60 auth_method 0 2024-09-17T15:33:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 msgr2=0x7f1450070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 0x7f1450070c70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1450073be0 tx=0x7f1444010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144400f040 con 0x7f14501cc6c0 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 10 monclient: dump: 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.595 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1444004020 con 0x7f14501cc6c0 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 msgr2=0x7f1450070c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 0x7f1450070c70 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1450073be0 tx=0x7f1444010c20 comp rx=0 tx=0).stop 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 msgr2=0x7f1450070c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.579+0000 7f143effd640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14501cc6c0 0x7f1450070c70 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1450073be0 tx=0x7f1444010c20 comp rx=0 tx=0).stop 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.583+0000 7f144ed9d640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f143003bb60 0x7f143003e020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f144801f390 tx=0x7f14480a9070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f1454ef8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f1454ef8640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 0x7f14501363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f1454ef8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f14500715c0 con 0x7f14500711b0 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 0x7f14501363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient: get_auth_request con 0x7f14500711b0 auth_method 0 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 10 monclient(hunting): handle_auth_done global_id 4977 payload 824 2024-09-17T15:33:46.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 1 -- 172.21.15.86:0/1320674630 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f145006d4f0 con 0x7f14500711b0 2024-09-17T15:33:46.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 msgr2=0x7f14501363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.587+0000 7f144dd9b640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 0x7f14501363d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f144401a000 tx=0x7f14440046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444003e60 con 0x7f14500711b0 2024-09-17T15:33:46.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 10 monclient: dump: 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.600 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f1444003e60 con 0x7f14500711b0 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 msgr2=0x7f14501363d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 0x7f14501363d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f144401a000 tx=0x7f14440046d0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 msgr2=0x7f14501363d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f143effd640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14500711b0 0x7f14501363d0 secure :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f144401a000 tx=0x7f14440046d0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:46.601 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 399234, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:46.602 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.603 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.604 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:46.605 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.606 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:46.607 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.608 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.609 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.610 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:46.611 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.613 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.614 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:33:46.615 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.615 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.615 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:46.615 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:46.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f143003bb60 msgr2=0x7f143003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f143003bb60 0x7f143003e020 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f144801f390 tx=0x7f14480a9070 comp rx=0 tx=0).stop 2024-09-17T15:33:46.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 10 monclient: shutdown 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 msgr2=0x7f1450072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 0x7f1450072a70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f144000b440 tx=0x7f144000b900 comp rx=0 tx=0).stop 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f144ed9d640 1 -- 172.21.15.86:0/1320674630 reap_dead start 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 shutdown_connections 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 --2- 172.21.15.86:0/1320674630 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f145009d770 0x7f1450072a70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 >> 172.21.15.86:0/1320674630 conn(0x7f14500a2f40 msgr2=0x7f1450134cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 shutdown_connections 2024-09-17T15:33:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.591+0000 7f1454ef8640 1 -- 172.21.15.86:0/1320674630 wait complete. 2024-09-17T15:33:46.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:46.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:33:46.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:33:46.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.727+0000 7fb2ff933640 10 monclient: get_monmap_and_config 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.727+0000 7fb2ff933640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.727+0000 7fb2ff933640 10 monclient: monmap: 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:46.731851+0000 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:46.731851+0000 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:46.731 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:46.732 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:46.732 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 Processor -- start 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 -- start start 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient: init 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2f80a0c70 0x7fb2f80a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-a con 0x7fb2f80a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:33:46.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2f80a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb2f8136160 0x7fb2f813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-c con 0x7fb2f8136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2f80a2130 con 0x7fb2f80a1590 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f80a2410 con 0x7fb2f80a0c70 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb2f80a26f0 con 0x7fb2f8136160 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2ff933640 10 monclient(hunting): authenticate will time out at 1359.697258s 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2f80a0c70 0x7fb2f80a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fcece640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb2f80a1590 0x7fb2f8135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36324/0 (socket says 172.21.15.86:36324) 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fcece640 1 -- 172.21.15.86:0/1266664886 learned_addr learned my addr 172.21.15.86:0/1266664886 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:46.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request con 0x7fb2f80a0c70 auth_method 0 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fded0640 1 --2- 172.21.15.86:0/1266664886 >> v2:172.21.15.203:3300/0 conn(0x7fb2f8136160 0x7fb2f813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fded0640 10 monclient(hunting): get_auth_request con 0x7fb2f8136160 auth_method 0 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fded0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fded0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fded0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_done global_id 4980 payload 274 2024-09-17T15:33:46.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1266664886 >> v2:172.21.15.203:3300/0 conn(0x7fb2f8136160 msgr2=0x7fb2f813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1266664886 >> v2:172.21.15.203:3300/0 conn(0x7fb2f8136160 0x7fb2f813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1266664886 >> v1:172.21.15.86:6789/0 conn(0x7fb2f80a1590 legacy=0x7fb2f8135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 monclient: found mon.noname-a 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:46.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.731+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1266664886 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f809f9f0 con 0x7fb2f80a0c70 2024-09-17T15:33:46.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1266664886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a0c70 msgr2=0x7fb2f80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1266664886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a0c70 0x7fb2f80a1050 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb2f806f840 tx=0x7fb2e801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15eab38d6fbaae3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 1 -- 172.21.15.86:0/1266664886 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80b0070 con 0x7fb2f80a0c70 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: dump: 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 1 -- 172.21.15.86:0/1266664886 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2e80ab600 con 0x7fb2f80a0c70 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 1 -- 172.21.15.86:0/1266664886 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80ab9d0 con 0x7fb2f80a0c70 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 5 monclient: authenticate success, global_id 4980 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 4 set_mon_vals no callback set 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2effff640 10 monclient: dump: 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 10 monclient: shutdown 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- 172.21.15.86:0/1266664886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a0c70 msgr2=0x7fb2f80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1266664886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a0c70 0x7fb2f80a1050 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb2f806f840 tx=0x7fb2e801c340 comp rx=0 tx=0).stop 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- 172.21.15.86:0/1266664886 shutdown_connections 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1266664886 >> v2:172.21.15.203:3300/0 conn(0x7fb2f8136160 0x7fb2f813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1266664886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a0c70 0x7fb2f80a1050 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- 172.21.15.86:0/1266664886 >> 172.21.15.86:0/1266664886 conn(0x7fb2f80a2f40 msgr2=0x7fb2f80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- 172.21.15.86:0/1266664886 shutdown_connections 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- 172.21.15.86:0/1266664886 wait complete. 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 10 monclient: monmap: 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:46.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 Processor -- start 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 1 -- start start 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.735+0000 7fb2ff933640 10 monclient: init 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:33:46.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f80a0c70 0x7fb2f811e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-b con 0x7fb2f80a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 0x7fb2f81351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-a con 0x7fb2f80a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f8136160 0x7fb2f81cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): picked mon.noname-c con 0x7fb2f8136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f80a00d0 con 0x7fb2f80a1590 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f809ff50 con 0x7fb2f80a0c70 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f8140c90 con 0x7fb2f8136160 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): authenticate will time out at 1359.704143s 2024-09-17T15:33:46.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 0x7fb2f81351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f8136160 0x7fb2f81cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 0x7fb2f81351b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49196/0 (socket says 172.21.15.86:49196) 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 -- 172.21.15.86:0/1692119173 learned_addr learned my addr 172.21.15.86:0/1692119173 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): get_auth_request con 0x7fb2f80a1590 auth_method 0 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): get_auth_request con 0x7fb2f8136160 auth_method 0 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fded0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f80a0c70 0x7fb2f811e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request con 0x7fb2f80a0c70 auth_method 0 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient(hunting): handle_auth_done global_id 4983 payload 824 2024-09-17T15:33:46.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f8136160 msgr2=0x7fb2f81cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f8136160 0x7fb2f81cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f80a0c70 msgr2=0x7fb2f811e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2f80a0c70 0x7fb2f811e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 monclient: found mon.noname-a 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f806fab0 con 0x7fb2f80a1590 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 msgr2=0x7fb2f81351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fcece640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 0x7fb2f81351b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400cc40 tx=0x7fb2f400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4012400 con 0x7fb2f80a1590 2024-09-17T15:33:46.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: dump: 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2f40125a0 con 0x7fb2f80a1590 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 5 monclient: authenticate success, global_id 4983 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _renew_subs 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f811f2a0 con 0x7fb2f80a1590 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4012990 con 0x7fb2f80a1590 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2edffb640 10 monclient: dump: 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.754 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ef7fe640 4 set_mon_vals no callback set 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _renew_subs 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f8135e50 con 0x7fb2f80a1590 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 0x7fb2f81ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f80a1050 con 0x7fb2f81356f0 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.739+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 0x7fb2f81ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient: get_auth_request con 0x7fb2f81356f0 auth_method 0 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb2f4012bb0 con 0x7fb2f80a1590 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_done global_id 74135 payload 824 2024-09-17T15:33:46.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2f811f660 con 0x7fb2f81356f0 2024-09-17T15:33:46.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2dc03bb60 0x7fb2dc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 4815+0+0 (secure 0 0 0) 0x7fb2f4050670 con 0x7fb2f80a1590 2024-09-17T15:33:46.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fded0640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2dc03bb60 0x7fb2dc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fded0640 10 monclient: get_auth_request con 0x7fb2dc03bb60 auth_method 0 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 msgr2=0x7fb2f81ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 0x7fb2f81ce270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80ad000 tx=0x7fb2e80190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80bb070 con 0x7fb2f81356f0 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 10 monclient: dump: 2024-09-17T15:33:46.759 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb2e80abe50 con 0x7fb2f81356f0 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 msgr2=0x7fb2f81ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 0x7fb2f81ce270 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80ad000 tx=0x7fb2e80190e0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 msgr2=0x7fb2f81ce270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2edffb640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f81356f0 0x7fb2f81ce270 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80ad000 tx=0x7fb2e80190e0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.743+0000 7fb2fded0640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2dc03bb60 0x7fb2dc03e020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2f811f1d0 tx=0x7fb2f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2ff933640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 0x7fb2f81ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2ff933640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb2f806d4f0 con 0x7fb2f811c8c0 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 0x7fb2f81ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient: get_auth_request con 0x7fb2f811c8c0 auth_method 0 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 10 monclient(hunting): handle_auth_done global_id 74138 payload 824 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1692119173 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2f81d90c0 con 0x7fb2f811c8c0 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 msgr2=0x7fb2f81ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2fd6cf640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 0x7fb2f81ce7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2e801c380 tx=0x7fb2e8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e80cc020 con 0x7fb2f811c8c0 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.751+0000 7fb2edffb640 10 monclient: dump: 2024-09-17T15:33:46.762 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fb2e80c0450 con 0x7fb2f811c8c0 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 msgr2=0x7fb2f81ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 0x7fb2f81ce7b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2e801c380 tx=0x7fb2e8008040 comp rx=0 tx=0).stop 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 msgr2=0x7fb2f81ce7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2edffb640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f811c8c0 0x7fb2f81ce7b0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2e801c380 tx=0x7fb2e8008040 comp rx=0 tx=0).stop 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7539, 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:46.764 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.765 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.766 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:46.767 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:46.768 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.769 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.770 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.771 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.772 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.773 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.774 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.775 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:46.776 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:46.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2dc03bb60 msgr2=0x7fb2dc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb2dc03bb60 0x7fb2dc03e020 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2f811f1d0 tx=0x7fb2f0008040 comp rx=0 tx=0).stop 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 10 monclient: shutdown 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 msgr2=0x7fb2f81351b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 --2- 172.21.15.86:0/1692119173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80a1590 0x7fb2f81351b0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400cc40 tx=0x7fb2f400a430 comp rx=0 tx=0).stop 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2fded0640 1 -- 172.21.15.86:0/1692119173 reap_dead start 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2fded0640 1 -- 172.21.15.86:0/1692119173 reap_dead start 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 shutdown_connections 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 >> 172.21.15.86:0/1692119173 conn(0x7fb2f80a2f40 msgr2=0x7fb2f8071490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 shutdown_connections 2024-09-17T15:33:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.755+0000 7fb2ff933640 1 -- 172.21.15.86:0/1692119173 wait complete. 2024-09-17T15:33:46.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:46.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:33:46.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:33:46.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: get_monmap_and_config 2024-09-17T15:33:46.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: monmap: 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:33:46.892871+0000 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:33:46.892871+0000 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:33:46.892 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:33:46.893 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 Processor -- start 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 -- start start 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: init 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b900a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b900a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b900a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1b90136160 0x7f1b9013df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b90136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:33:46.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b900a2130 con 0x7f1b900a1590 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1b900a2410 con 0x7f1b90136160 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1b900a26f0 con 0x7f1b900a0c70 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b97c9f640 10 monclient(hunting): authenticate will time out at 1359.858309s 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9523a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1b900a1590 0x7f1b90135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36334/0 (socket says 172.21.15.86:36334) 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9523a640 1 -- 172.21.15.86:0/2705491089 learned_addr learned my addr 172.21.15.86:0/2705491089 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b900a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): get_auth_request con 0x7f1b900a0c70 auth_method 0 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 1 --2- 172.21.15.86:0/2705491089 >> v2:172.21.15.203:3300/0 conn(0x7f1b90136160 0x7f1b9013df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): get_auth_request con 0x7f1b90136160 auth_method 0 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_done global_id 74141 payload 274 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 -- 172.21.15.86:0/2705491089 >> v2:172.21.15.203:3300/0 conn(0x7f1b90136160 msgr2=0x7f1b9013df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/2705491089 >> v2:172.21.15.203:3300/0 conn(0x7f1b90136160 0x7f1b9013df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 -- 172.21.15.86:0/2705491089 >> v1:172.21.15.86:6789/0 conn(0x7f1b900a1590 legacy=0x7f1b90135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 monclient: found mon.noname-b 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.891+0000 7f1b95a3b640 1 -- 172.21.15.86:0/2705491089 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9009f9f0 con 0x7f1b900a0c70 2024-09-17T15:33:46.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b95a3b640 1 -- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 msgr2=0x7f1b900a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b900a1050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009d90 tx=0x7f1b8801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feefb8245c89bfb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 1 -- 172.21.15.86:0/2705491089 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b880b2070 con 0x7f1b900a0c70 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: dump: 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.903 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b97c9f640 5 monclient: authenticate success, global_id 74141 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b97c9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 1 -- 172.21.15.86:0/2705491089 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b880a5890 con 0x7f1b900a0c70 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b97c9f640 10 monclient: get_monmap_and_config success 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b97c9f640 4 set_mon_vals no callback set 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 1 -- 172.21.15.86:0/2705491089 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b880a5c60 con 0x7f1b900a0c70 2024-09-17T15:33:46.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.895+0000 7f1b94a39640 10 monclient: dump: 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.905 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: shutdown 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 msgr2=0x7f1b900a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b900a1050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b88009d90 tx=0x7f1b8801c830 comp rx=0 tx=0).stop 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- 172.21.15.86:0/2705491089 shutdown_connections 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/2705491089 >> v2:172.21.15.203:3300/0 conn(0x7f1b90136160 0x7f1b9013df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/2705491089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b900a1050 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- 172.21.15.86:0/2705491089 >> 172.21.15.86:0/2705491089 conn(0x7f1b900a2f40 msgr2=0x7f1b900a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- 172.21.15.86:0/2705491089 shutdown_connections 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- 172.21.15.86:0/2705491089 wait complete. 2024-09-17T15:33:46.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: build_initial_monmap 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: monmap: 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 Processor -- start 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- start start 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: init 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: _reopen_session rank -1 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b9009f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f1b900a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b900a1590 0x7f1b90135530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f1b900a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b90136160 0x7f1b9009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f1b90136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1b900a00d0 con 0x7f1b90136160 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9009ff50 con 0x7f1b900a1590 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1b90140c90 con 0x7f1b900a0c70 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): _renew_subs 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b97c9f640 10 monclient(hunting): authenticate will time out at 1359.865788s 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b9009f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b9009f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58782/0 (socket says 172.21.15.86:58782) 2024-09-17T15:33:46.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 -- 172.21.15.86:0/1477421017 learned_addr learned my addr 172.21.15.86:0/1477421017 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:33:46.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b90136160 0x7f1b9009d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): get_auth_request con 0x7f1b900a0c70 auth_method 0 2024-09-17T15:33:46.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): get_auth_request con 0x7f1b90136160 auth_method 0 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9623c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b900a1590 0x7f1b90135530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient(hunting): handle_auth_done global_id 74144 payload 824 2024-09-17T15:33:46.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b900a1590 msgr2=0x7f1b90135530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b900a1590 0x7f1b90135530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b90136160 msgr2=0x7f1b9009d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b90136160 0x7f1b9009d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient: _finish_hunting 0 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 monclient: found mon.noname-c 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b95a3b640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b9006fb90 con 0x7f1b900a0c70 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9523a640 10 monclient: get_auth_request con 0x7f1b900a1590 auth_method 0 2024-09-17T15:33:46.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.899+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b900a1590 0x7f1b90135530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b95a3b640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 msgr2=0x7f1b9009f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b95a3b640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b9009f3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b8801d0d0 tx=0x7f1b880195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b880bd020 con 0x7f1b900a0c70 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: dump: 2024-09-17T15:33:46.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 5 monclient: authenticate success, global_id 74144 2024-09-17T15:33:46.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b880b8680 con 0x7f1b900a0c70 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b880b2040 con 0x7f1b900a0c70 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 monclient: _renew_subs 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b90121f00 con 0x7f1b900a0c70 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 10 monclient: dump: 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.919 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8ffff640 4 set_mon_vals no callback set 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 monclient: _renew_subs 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b9009e850 con 0x7f1b900a0c70 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 0x7f1b9009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b900a1050 con 0x7f1b901d00d0 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1b880c0020 con 0x7f1b900a0c70 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 0x7f1b9009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient: get_auth_request con 0x7f1b901d00d0 auth_method 0 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b7c03bbb0 0x7f1b7c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 4815+0+0 (secure 0 0 0) 0x7f1b880eada0 con 0x7f1b900a0c70 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 10 monclient(hunting): handle_auth_done global_id 54274 payload 824 2024-09-17T15:33:46.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9623c640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b7c03bbb0 0x7f1b7c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9523a640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b900008e0 con 0x7f1b901d00d0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.903+0000 7f1b9623c640 10 monclient: get_auth_request con 0x7f1b7c03bbb0 auth_method 0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b9523a640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 msgr2=0x7f1b9009dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 0x7f1b9009dcb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b8400d980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84007070 con 0x7f1b901d00d0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 10 monclient: dump: 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.922 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b9623c640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b7c03bbb0 0x7f1b7c03e070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b9009f920 tx=0x7f1b80009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1b84017070 con 0x7f1b901d00d0 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 msgr2=0x7f1b9009dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 0x7f1b9009dcb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b8400d980 comp rx=0 tx=0).stop 2024-09-17T15:33:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 msgr2=0x7f1b9009dcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.907+0000 7f1b8e7fc640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b901d00d0 0x7f1b9009dcb0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b8400d980 comp rx=0 tx=0).stop 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b97c9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 0x7f1b9009e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b97c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1b901216d0 con 0x7f1b90135a70 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 0x7f1b9009e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient: get_auth_request con 0x7f1b90135a70 auth_method 0 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:33:46.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 10 monclient(hunting): handle_auth_done global_id 54277 payload 824 2024-09-17T15:33:46.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.915+0000 7f1b9523a640 1 -- 172.21.15.86:0/1477421017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1b90121880 con 0x7f1b90135a70 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b9523a640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 msgr2=0x7f1b9009e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b9523a640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 0x7f1b9009e1f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b84004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84007710 con 0x7f1b90135a70 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 10 monclient: dump: 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:33:46.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f1b84007710 con 0x7f1b90135a70 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 msgr2=0x7f1b9009e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 0x7f1b9009e1f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b84004950 comp rx=0 tx=0).stop 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 msgr2=0x7f1b9009e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:33:46.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b8e7fc640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b90135a70 0x7f1b9009e1f0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b8400d4c0 tx=0x7f1b84004950 comp rx=0 tx=0).stop 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 68, 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 102350, 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.928 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.929 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.930 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:33:46.931 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:33:46.932 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:33:46.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.934 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:33:46.936 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.939 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:33:46.940 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b7c03bbb0 msgr2=0x7f1b7c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b7c03bbb0 0x7f1b7c03e070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b9009f920 tx=0x7f1b80009190 comp rx=0 tx=0).stop 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 10 monclient: shutdown 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 msgr2=0x7f1b9009f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 --2- 172.21.15.86:0/1477421017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b900a0c70 0x7f1b9009f3b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1b8801d0d0 tx=0x7f1b880195d0 comp rx=0 tx=0).stop 2024-09-17T15:33:46.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b9623c640 1 -- 172.21.15.86:0/1477421017 reap_dead start 2024-09-17T15:33:46.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b9623c640 1 -- 172.21.15.86:0/1477421017 reap_dead start 2024-09-17T15:33:46.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 shutdown_connections 2024-09-17T15:33:46.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 >> 172.21.15.86:0/1477421017 conn(0x7f1b900a2f40 msgr2=0x7f1b90072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:33:46.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 shutdown_connections 2024-09-17T15:33:46.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:33:46.919+0000 7f1b97c9f640 1 -- 172.21.15.86:0/1477421017 wait complete. 2024-09-17T15:33:46.943 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:33:46.943 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:33:46.944 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-17T15:33:46.944 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:33:46.944 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:33:46.944 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:33:46.944 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:33:46.957 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:33:46.957 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:33:46.958 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:33:46.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:34:01.959 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:34:01.959 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:34:01.960 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:34:01.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:02.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: get_monmap_and_config 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: build_initial_monmap 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: monmap: 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:02.089159+0000 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:02.089159+0000 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:02.089 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:02.089 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:02.089 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:02.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 Processor -- start 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 -- start start 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: init 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee4c0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-a con 0x7fee4c0694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-c con 0x7fee4c13aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4c09c8b0 con 0x7fee4c0689d0 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fee4c074500 con 0x7fee4c0694c0 2024-09-17T15:34:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fee4c09cb90 con 0x7fee4c13aff0 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee517b7640 10 monclient(hunting): authenticate will time out at 1375.054653s 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4a7fc640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4affd640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fee4c0689d0 legacy=0x7fee4c068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4a7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:48904/0 (socket says 172.21.15.86:48904) 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 learned_addr learned my addr 172.21.15.86:0/4176413317 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request con 0x7fee4c13aff0 auth_method 0 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:34:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:02.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:02.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.086+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:02.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.86:6789/0 conn(0x7fee4c0689d0 legacy=0x7fee4c068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:34:02.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:02.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request con 0x7fee4c13aff0 auth_method 0 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.290+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:34:02.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.86:6789/0 conn(0x7fee4c0689d0 legacy=0x7fee4c068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:34:02.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:02.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:34:02.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:02.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request con 0x7fee4c13aff0 auth_method 0 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:02.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:02.694+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:34:03.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.86:6789/0 conn(0x7fee4c0689d0 legacy=0x7fee4c068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:34:03.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:03.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request con 0x7fee4c13aff0 auth_method 0 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:03.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:03.498+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): tick 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 msgr2=0x7fee4c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.203:3300/0 conn(0x7fee4c13aff0 0x7fee4c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 -- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 msgr2=0x7fee4c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 --2- 172.21.15.86:0/4176413317 >> v2:172.21.15.86:3300/0 conn(0x7fee4c0694c0 0x7fee4c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.86:6789/0 conn(0x7fee4c0689d0 legacy=0x7fee4c068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient: _add_conns ranks=[2,1] 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fee380036a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee38006150 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 -- 172.21.15.86:0/4176413317 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4c09ce70 con 0x7fee38006150 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 1 -- 172.21.15.86:0/4176413317 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c09cb90 con 0x7fee380036a0 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee48ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee380036a0 auth_method 0 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.086+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.090+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.203:6789/0 conn(0x7fee38006150 legacy=0x7fee38008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee380036a0 auth_method 0 2024-09-17T15:34:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:05.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:05.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:05.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:05.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:05.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.290+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee380036a0 auth_method 0 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:05.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:05.694+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee380036a0 auth_method 0 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:06.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:06.498+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1621841000 0 0) 0x7fee4c09ce70 con 0x7fee38006150 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 10 monclient(hunting): _negotiate 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 10 monclient(hunting): my global_id is 54280 2024-09-17T15:34:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.018+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee34003530 con 0x7fee38006150 2024-09-17T15:34:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2811919972 0 0) 0x7fee34003530 con 0x7fee38006150 2024-09-17T15:34:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 msgr2=0x7fee38005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee380036a0 0x7fee38005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c09c8b0 con 0x7fee38006150 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fee3c0043b0 con 0x7fee38006150 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: dump: 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.030 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 >> v1:172.21.15.203:6789/0 conn(0x7fee38006150 legacy=0x7fee38008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 reap_dead start 2024-09-17T15:34:07.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee34003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): picked mon.c con 0x7fee34003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 0x7fee34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): picked mon.b con 0x7fee34004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 0x7fee3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): picked mon.a con 0x7fee340052f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c09cb90 con 0x7fee340052f0 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee3400c210 con 0x7fee34004810 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fee3400c410 con 0x7fee34003a30 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee49ffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4b7fe640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 msgr2=0x7fee3400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 0x7fee3400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee34003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee34003a30 auth_method 0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 0x7fee34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): get_auth_request con 0x7fee34004810 auth_method 0 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient(hunting): handle_auth_done global_id 54280 payload 293 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 msgr2=0x7fee34003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee34003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 msgr2=0x7fee3400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 0x7fee3400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 monclient: found mon.b 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee34003530 con 0x7fee34004810 2024-09-17T15:34:07.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee517b7640 5 monclient: authenticate success, global_id 54280 2024-09-17T15:34:07.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee517b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:07.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.026+0000 7fee4affd640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee34003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:07.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee4a7fc640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 msgr2=0x7fee34004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 0x7fee34004c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee3c003990 tx=0x7fee3c01f3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4740c9f740151615 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c00c0e0 con 0x7fee34004810 2024-09-17T15:34:07.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: dump: 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3c006290 con 0x7fee34004810 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 1 -- 172.21.15.86:0/4176413317 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c007020 con 0x7fee34004810 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee49ffb640 10 monclient: dump: 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.038 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:07.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 4 set_mon_vals no callback set 2024-09-17T15:34:07.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 10 monclient: shutdown 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 -- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 msgr2=0x7fee34004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 0x7fee34004c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fee3c003990 tx=0x7fee3c01f3f0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 -- 172.21.15.86:0/4176413317 shutdown_connections 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee340052f0 0x7fee3400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee34004810 0x7fee34004c90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 --2- 172.21.15.86:0/4176413317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee34003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 -- 172.21.15.86:0/4176413317 >> 172.21.15.86:0/4176413317 conn(0x7fee4c0a6440 msgr2=0x7fee4c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 -- 172.21.15.86:0/4176413317 shutdown_connections 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.034+0000 7fee517b7640 1 -- 172.21.15.86:0/4176413317 wait complete. 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient: build_initial_monmap 2024-09-17T15:34:07.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient: monmap: 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:07.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 Processor -- start 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 -- start start 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient: init 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-c con 0x7fee34003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 0x7fee4c135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-a con 0x7fee34004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 0x7fee4c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): picked mon.noname-b con 0x7fee340052f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c09c240 con 0x7fee34004810 2024-09-17T15:34:07.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c09c0c0 con 0x7fee340052f0 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c06f840 con 0x7fee34003a30 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee517b7640 10 monclient(hunting): authenticate will time out at 1380.007721s 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4a7fc640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 msgr2=0x7fee4c135800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 0x7fee4c135800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58594/0 (socket says 172.21.15.86:58594) 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 1 -- 172.21.15.86:0/1599008759 learned_addr learned my addr 172.21.15.86:0/1599008759 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 10 monclient(hunting): get_auth_request con 0x7fee34003a30 auth_method 0 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.038+0000 7fee4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 0x7fee4c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request con 0x7fee340052f0 auth_method 0 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient(hunting): handle_auth_done global_id 74147 payload 824 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 -- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 msgr2=0x7fee4c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 0x7fee4c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 -- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 msgr2=0x7fee4c135800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 0x7fee4c135800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 monclient: found mon.noname-c 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 -- 172.21.15.86:0/1599008759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c1221b0 con 0x7fee34003a30 2024-09-17T15:34:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4b7fe640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 0x7fee4c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:07.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 -- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 msgr2=0x7fee4c1c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee4affd640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee40000eb0 tx=0x7fee400054e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4001c070 con 0x7fee34003a30 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: dump: 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 5 monclient: authenticate success, global_id 74147 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 10 monclient: _renew_subs 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee40018840 con 0x7fee34003a30 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c11fe50 con 0x7fee34003a30 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee40018bf0 con 0x7fee34003a30 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee337fe640 10 monclient: dump: 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee497fa640 4 set_mon_vals no callback set 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 10 monclient: _renew_subs 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.042+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c1c9020 con 0x7fee34003a30 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee517b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee517b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee517b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee4c09d460 con 0x7fee34003a30 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fee4001f070 con 0x7fee34003a30 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee337fe640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 0x7fee3803e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 4815+0+0 (secure 0 0 0) 0x7fee40051640 con 0x7fee34003a30 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 0x7fee3803e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.046+0000 7fee4a7fc640 10 monclient: get_auth_request con 0x7fee3803c330 auth_method 0 2024-09-17T15:34:07.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.050+0000 7fee4a7fc640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 0x7fee3803e7f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0045d0 tx=0x7fee3c0aa000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.054+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fee4001fe30 con 0x7fee34003a30 2024-09-17T15:34:07.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.054+0000 7fee337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.054+0000 7fee337fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:955: done (1 left) 2024-09-17T15:34:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:956: done (0 left) 2024-09-17T15:34:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:957: snap_remove snap 86 2024-09-17T15:34:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fee4c074470 con 0x7fee34003a30 2024-09-17T15:34:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee337fe640 1 -- 172.21.15.86:0/1599008759 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fee400173c0 con 0x7fee34003a30 2024-09-17T15:34:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee337fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:07.284 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":70,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 msgr2=0x7fee3803e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 0x7fee3803e7f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0045d0 tx=0x7fee3c0aa000 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 10 monclient: shutdown 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 msgr2=0x7fee4c1c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fee40000eb0 tx=0x7fee400054e0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 shutdown_connections 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee3803c330 0x7fee3803e7f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee340052f0 0x7fee4c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee34004810 0x7fee4c135800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 --2- 172.21.15.86:0/1599008759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee34003a30 0x7fee4c1c8570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 >> 172.21.15.86:0/1599008759 conn(0x7fee4c0a6440 msgr2=0x7fee4c13a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.282+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 shutdown_connections 2024-09-17T15:34:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.286+0000 7fee517b7640 1 -- 172.21.15.86:0/1599008759 wait complete. 2024-09-17T15:34:07.301 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:34:07.302 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:34:07.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:34:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: get_monmap_and_config 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: build_initial_monmap 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: monmap: 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:07.424252+0000 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:07.424252+0000 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:07.423 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:07.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:07.424 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:07.424 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:07.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 Processor -- start 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 -- start start 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: init 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-17T15:34:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 0x7efd7409f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-a con 0x7efd740a01a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd740a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-b con 0x7efd740a1380 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd74074e00 con 0x7efd740a0580 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7efd7409e350 con 0x7efd740a01a0 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efd7409e630 con 0x7efd740a1380 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd79b01640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 msgr2=0x7efd7409f010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7bd65640 10 monclient(hunting): authenticate will time out at 1380.389918s 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd79b01640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 0x7efd7409f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58610/0 (socket says 172.21.15.86:58610) 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 -- 172.21.15.86:0/1427873014 learned_addr learned my addr 172.21.15.86:0/1427873014 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): get_auth_request con 0x7efd740a1380 auth_method 0 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient(hunting): handle_auth_done global_id 74150 payload 274 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 -- 172.21.15.86:0/1427873014 >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 msgr2=0x7efd7409f010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 --2- 172.21.15.86:0/1427873014 >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 0x7efd7409f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 -- 172.21.15.86:0/1427873014 >> v1:172.21.15.203:6789/0 conn(0x7efd740a0580 legacy=0x7efd740a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 monclient: found mon.noname-b 2024-09-17T15:34:07.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.422+0000 7efd7a302640 1 -- 172.21.15.86:0/1427873014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7409e910 con 0x7efd740a1380 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7a302640 1 -- 172.21.15.86:0/1427873014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 msgr2=0x7efd7413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7a302640 1 --2- 172.21.15.86:0/1427873014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd640098a0 tx=0x7efd6401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a66775210add06f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 1 -- 172.21.15.86:0/1427873014 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd640b1070 con 0x7efd740a1380 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: dump: 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.431 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 1 -- 172.21.15.86:0/1427873014 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd640ac620 con 0x7efd740a1380 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 5 monclient: authenticate success, global_id 74150 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 1 -- 172.21.15.86:0/1427873014 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd640ac9f0 con 0x7efd740a1380 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 4 set_mon_vals no callback set 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd78aff640 10 monclient: dump: 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.433 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 10 monclient: shutdown 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 1 -- 172.21.15.86:0/1427873014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 msgr2=0x7efd7413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.426+0000 7efd7bd65640 1 --2- 172.21.15.86:0/1427873014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efd640098a0 tx=0x7efd6401c420 comp rx=0 tx=0).stop 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- 172.21.15.86:0/1427873014 shutdown_connections 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 --2- 172.21.15.86:0/1427873014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a1380 0x7efd7413ba80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 --2- 172.21.15.86:0/1427873014 >> v2:172.21.15.86:3300/0 conn(0x7efd740a01a0 0x7efd7409f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- 172.21.15.86:0/1427873014 >> 172.21.15.86:0/1427873014 conn(0x7efd740a2f40 msgr2=0x7efd740a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- 172.21.15.86:0/1427873014 shutdown_connections 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- 172.21.15.86:0/1427873014 wait complete. 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient: build_initial_monmap 2024-09-17T15:34:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient: monmap: 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:07.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 Processor -- start 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- start start 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient: init 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd740a01a0 0x7efd74122620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-a con 0x7efd740a01a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-c con 0x7efd740a0580 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd740a1380 0x7efd741230a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): picked mon.noname-b con 0x7efd740a1380 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7efd7413ebc0 con 0x7efd740a01a0 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd7413ea40 con 0x7efd740a1380 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efd7413ed40 con 0x7efd740a0580 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7bd65640 10 monclient(hunting): authenticate will time out at 1380.397545s 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79b01640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd740a01a0 msgr2=0x7efd74122620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79b01640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd740a01a0 0x7efd74122620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58612/0 (socket says 172.21.15.86:58612) 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 -- 172.21.15.86:0/2236513744 learned_addr learned my addr 172.21.15.86:0/2236513744 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): get_auth_request con 0x7efd740a0580 auth_method 0 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7a302640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd740a1380 0x7efd741230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient(hunting): handle_auth_done global_id 74153 payload 824 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd740a1380 msgr2=0x7efd741230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd740a1380 0x7efd741230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd740a01a0 msgr2=0x7efd74122620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd740a01a0 0x7efd74122620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 monclient: found mon.noname-c 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd79300640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd74073bd0 con 0x7efd740a0580 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7a302640 10 monclient: get_auth_request con 0x7efd740a1380 auth_method 0 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.430+0000 7efd7a302640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd740a1380 0x7efd741230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79300640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 msgr2=0x7efd74122b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79300640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c7d0 tx=0x7efd6800cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68016020 con 0x7efd740a0580 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: dump: 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd6800ac60 con 0x7efd740a0580 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 5 monclient: authenticate success, global_id 74153 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 monclient: _renew_subs 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd74128df0 con 0x7efd740a0580 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68012660 con 0x7efd740a0580 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 10 monclient: dump: 2024-09-17T15:34:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd737fe640 4 set_mon_vals no callback set 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 monclient: _renew_subs 2024-09-17T15:34:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd74123a70 con 0x7efd740a0580 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 0x7efd7402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efd7409f010 con 0x7efd7413ac60 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 0x7efd7402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient: get_auth_request con 0x7efd7413ac60 auth_method 0 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7efd68019020 con 0x7efd740a0580 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 10 monclient(hunting): handle_auth_done global_id 74156 payload 824 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd7411efe0 con 0x7efd7413ac60 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efd6003bb60 0x7efd6003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(217..217 src has 1..217) ==== 4787+0+0 (secure 0 0 0) 0x7efd68050040 con 0x7efd740a0580 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7a302640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efd6003bb60 0x7efd6003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd7a302640 10 monclient: get_auth_request con 0x7efd6003bb60 auth_method 0 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 msgr2=0x7efd7402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.434+0000 7efd79b01640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 0x7efd7402d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c005270 con 0x7efd7413ac60 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 10 monclient: dump: 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd6c015070 con 0x7efd7413ac60 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 msgr2=0x7efd7402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 0x7efd7402d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c00b910 comp rx=0 tx=0).stop 2024-09-17T15:34:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 msgr2=0x7efd7402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd727fc640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413ac60 0x7efd7402d450 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c00b910 comp rx=0 tx=0).stop 2024-09-17T15:34:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.438+0000 7efd7a302640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efd6003bb60 0x7efd6003e020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efd640ac000 tx=0x7efd64008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd7bd65640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd7bd65640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 0x7efd7402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd7bd65640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efd741237c0 con 0x7efd7413b040 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 0x7efd7402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient: get_auth_request con 0x7efd7413b040 auth_method 0 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 10 monclient(hunting): handle_auth_done global_id 74159 payload 824 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.442+0000 7efd79b01640 1 -- 172.21.15.86:0/2236513744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd7406d4f0 con 0x7efd7413b040 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd79b01640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 msgr2=0x7efd7402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd79b01640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 0x7efd7402d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c0059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c01ae60 con 0x7efd7413b040 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 10 monclient: dump: 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7efd6c010480 con 0x7efd7413b040 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 msgr2=0x7efd7402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 0x7efd7402d990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c0059e0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 msgr2=0x7efd7402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd727fc640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd7413b040 0x7efd7402d990 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7efd6c00b450 tx=0x7efd6c0059e0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.455 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 70, 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28235, 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:34:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:07.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.470 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:07.471 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efd6003bb60 msgr2=0x7efd6003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efd6003bb60 0x7efd6003e020 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efd640ac000 tx=0x7efd64008040 comp rx=0 tx=0).stop 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 10 monclient: shutdown 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 msgr2=0x7efd74122b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd6800c7d0 tx=0x7efd6800cc90 comp rx=0 tx=0).stop 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7a302640 1 -- 172.21.15.86:0/2236513744 reap_dead start 2024-09-17T15:34:07.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 shutdown_connections 2024-09-17T15:34:07.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 --2- 172.21.15.86:0/2236513744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd740a0580 0x7efd74122b60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 >> 172.21.15.86:0/2236513744 conn(0x7efd740a2f40 msgr2=0x7efd7409c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 shutdown_connections 2024-09-17T15:34:07.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.446+0000 7efd7bd65640 1 -- 172.21.15.86:0/2236513744 wait complete. 2024-09-17T15:34:07.473 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:34:07.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:34:07.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:34:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: get_monmap_and_config 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: build_initial_monmap 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: monmap: 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:07.588289+0000 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:07.588289+0000 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:07.588 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 Processor -- start 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 -- start start 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: init 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 0x7fa5200a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): picked mon.noname-a con 0x7fa5200a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:07.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa5200a1590 0x7fa520135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): picked mon.noname-c con 0x7fa5200a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa520135f90 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5200a2130 con 0x7fa520135f90 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa5200a2410 con 0x7fa5200a0c70 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa5200a26f0 con 0x7fa5200a1590 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa526583640 10 monclient(hunting): authenticate will time out at 1380.553768s 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51ffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 msgr2=0x7fa5200a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 0x7fa5200a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa5200a1590 0x7fa520135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa524b20640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa520135f90 0x7fa52013cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:47740/0 (socket says 172.21.15.86:47740) 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa524b20640 1 -- 172.21.15.86:0/184246801 learned_addr learned my addr 172.21.15.86:0/184246801 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): get_auth_request con 0x7fa5200a1590 auth_method 0 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_done global_id 54286 payload 274 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 -- 172.21.15.86:0/184246801 >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 msgr2=0x7fa5200a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/184246801 >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 0x7fa5200a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 -- 172.21.15.86:0/184246801 >> v1:172.21.15.203:6789/0 conn(0x7fa520135f90 legacy=0x7fa52013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:34:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:34:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.586+0000 7fa51f7fe640 1 -- 172.21.15.86:0/184246801 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa52009f9f0 con 0x7fa5200a1590 2024-09-17T15:34:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.590+0000 7fa51f7fe640 1 -- 172.21.15.86:0/184246801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 msgr2=0x7fa520135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.590+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/184246801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa520135a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa520074ae0 tx=0x7fa51401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7002d644d478fa16 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 1 -- 172.21.15.86:0/184246801 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5140b0070 con 0x7fa5200a1590 2024-09-17T15:34:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: dump: 2024-09-17T15:34:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 1 -- 172.21.15.86:0/184246801 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5140ab600 con 0x7fa5200a1590 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 1 -- 172.21.15.86:0/184246801 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5140ab9d0 con 0x7fa5200a1590 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 5 monclient: authenticate success, global_id 54286 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 4 set_mon_vals no callback set 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51effd640 10 monclient: dump: 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: shutdown 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- 172.21.15.86:0/184246801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 msgr2=0x7fa520135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- 172.21.15.86:0/184246801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa520135a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa520074ae0 tx=0x7fa51401c360 comp rx=0 tx=0).stop 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- 172.21.15.86:0/184246801 shutdown_connections 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- 172.21.15.86:0/184246801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa520135a50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- 172.21.15.86:0/184246801 >> v2:172.21.15.86:3300/0 conn(0x7fa5200a0c70 0x7fa5200a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- 172.21.15.86:0/184246801 >> 172.21.15.86:0/184246801 conn(0x7fa5200a2f40 msgr2=0x7fa5200a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- 172.21.15.86:0/184246801 shutdown_connections 2024-09-17T15:34:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- 172.21.15.86:0/184246801 wait complete. 2024-09-17T15:34:07.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: build_initial_monmap 2024-09-17T15:34:07.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: monmap: 2024-09-17T15:34:07.603 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:07.603 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 Processor -- start 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- start start 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: init 2024-09-17T15:34:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 0x7fa5201cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): picked mon.noname-c con 0x7fa5200a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa52012fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): picked mon.noname-b con 0x7fa5200a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa520135f90 0x7fa5201301d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): picked mon.noname-a con 0x7fa520135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa52009ce60 con 0x7fa520135f90 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa52009cce0 con 0x7fa5200a1590 2024-09-17T15:34:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa52009ff50 con 0x7fa5200a0c70 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa526583640 10 monclient(hunting): authenticate will time out at 1380.562699s 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa524b20640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa520135f90 msgr2=0x7fa5201301d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa524b20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa520135f90 0x7fa5201301d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 0x7fa5201cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 0x7fa5201cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58644/0 (socket says 172.21.15.86:58644) 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 -- 172.21.15.86:0/807227432 learned_addr learned my addr 172.21.15.86:0/807227432 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): get_auth_request con 0x7fa5200a0c70 auth_method 0 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa52012fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient(hunting): handle_auth_done global_id 74162 payload 824 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 msgr2=0x7fa52012fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa52012fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa520135f90 msgr2=0x7fa5201301d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa520135f90 0x7fa5201301d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 monclient: found mon.noname-c 2024-09-17T15:34:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51ffff640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa520073910 con 0x7fa5200a0c70 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51f7fe640 10 monclient: get_auth_request con 0x7fa5200a1590 auth_method 0 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.594+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5200a1590 0x7fa52012fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51ffff640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 msgr2=0x7fa5201cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51ffff640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 0x7fa5201cff90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa51000b580 tx=0x7fa51000ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100050f0 con 0x7fa5200a0c70 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: dump: 2024-09-17T15:34:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: _finish_auth 0 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa510015070 con 0x7fa5200a0c70 2024-09-17T15:34:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 5 monclient: authenticate success, global_id 74162 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 monclient: _renew_subs 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa52009d460 con 0x7fa5200a0c70 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100054d0 con 0x7fa5200a0c70 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 10 monclient: dump: 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51dffb640 4 set_mon_vals no callback set 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 monclient: _renew_subs 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa520073630 con 0x7fa5200a0c70 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 0x7fa5200703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5200a1050 con 0x7fa520072110 2024-09-17T15:34:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa510005670 con 0x7fa5200a0c70 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa50c03bb10 0x7fa50c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 0x7fa5200703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(217..217 src has 1..217) ==== 4787+0+0 (secure 0 0 0) 0x7fa51004ff40 con 0x7fa5200a0c70 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient: get_auth_request con 0x7fa520072110 auth_method 0 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa524b20640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa50c03bb10 0x7fa50c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa524b20640 10 monclient: get_auth_request con 0x7fa50c03bb10 auth_method 0 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_done global_id 54289 payload 824 2024-09-17T15:34:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.598+0000 7fa51f7fe640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5200008e0 con 0x7fa520072110 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51f7fe640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 msgr2=0x7fa5200703c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 0x7fa5200703c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa52006f840 tx=0x7fa51401aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa524b20640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa50c03bb10 0x7fa50c03dfd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa518004ba0 tx=0x7fa518005130 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5140bb020 con 0x7fa520072110 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 10 monclient: dump: 2024-09-17T15:34:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa5140b1040 con 0x7fa520072110 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 msgr2=0x7fa5200703c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 0x7fa5200703c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa52006f840 tx=0x7fa51401aba0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 msgr2=0x7fa5200703c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.602+0000 7fa51cff9640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa520072110 0x7fa5200703c0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa52006f840 tx=0x7fa51401aba0 comp rx=0 tx=0).stop 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa526583640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa526583640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 0x7fa52002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa526583640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa520132340 con 0x7fa52002d450 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 0x7fa52002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient: get_auth_request con 0x7fa52002d450 auth_method 0 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:07.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 10 monclient(hunting): handle_auth_done global_id 54292 payload 824 2024-09-17T15:34:07.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.610+0000 7fa51f7fe640 1 -- 172.21.15.86:0/807227432 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa520070900 con 0x7fa52002d450 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51f7fe640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 msgr2=0x7fa52002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51f7fe640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 0x7fa52002d860 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c10 tx=0x7fa514019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5140bb070 con 0x7fa52002d450 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 10 monclient: dump: 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:07.618 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7fa5140c2c70 con 0x7fa52002d450 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 msgr2=0x7fa52002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 0x7fa52002d860 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c10 tx=0x7fa514019100 comp rx=0 tx=0).stop 2024-09-17T15:34:07.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 msgr2=0x7fa52002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa51cff9640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa52002d450 0x7fa52002d860 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa514000c10 tx=0x7fa514019100 comp rx=0 tx=0).stop 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 70, 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 123045, 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:07.630 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.631 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.632 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:07.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:07.634 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:07.634 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa50c03bb10 msgr2=0x7fa50c03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa50c03bb10 0x7fa50c03dfd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa518004ba0 tx=0x7fa518005130 comp rx=0 tx=0).stop 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 10 monclient: shutdown 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 msgr2=0x7fa5201cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 --2- 172.21.15.86:0/807227432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5200a0c70 0x7fa5201cff90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa51000b580 tx=0x7fa51000ba40 comp rx=0 tx=0).stop 2024-09-17T15:34:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa524b20640 1 -- 172.21.15.86:0/807227432 reap_dead start 2024-09-17T15:34:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa524b20640 1 -- 172.21.15.86:0/807227432 reap_dead start 2024-09-17T15:34:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 shutdown_connections 2024-09-17T15:34:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 >> 172.21.15.86:0/807227432 conn(0x7fa5200a2f40 msgr2=0x7fa520132f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 shutdown_connections 2024-09-17T15:34:07.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:07.614+0000 7fa526583640 1 -- 172.21.15.86:0/807227432 wait complete. 2024-09-17T15:34:07.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:34:07.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:34:08.094 INFO:tasks.rados.rados.0.smithi203.stdout:957: done (0 left) 2024-09-17T15:34:08.094 INFO:tasks.rados.rados.0.smithi203.stdout:958: delete oid 46 current snap is 104 2024-09-17T15:34:08.114 INFO:tasks.rados.rados.0.smithi203.stdout:958: done (0 left) 2024-09-17T15:34:08.114 INFO:tasks.rados.rados.0.smithi203.stdout:959: rollback oid 41 current snap is 104 2024-09-17T15:34:08.115 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 99 2024-09-17T15:34:08.115 INFO:tasks.rados.rados.0.smithi203.stdout:960: read oid 42 snap 89 2024-09-17T15:34:08.115 INFO:tasks.rados.rados.0.smithi203.stdout:960: expect deleted 2024-09-17T15:34:08.117 INFO:tasks.rados.rados.0.smithi203.stdout:961: write (excl) oid 43 current snap is 104 2024-09-17T15:34:08.118 INFO:tasks.rados.rados.0.smithi203.stdout:961: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-17T15:34:08.135 INFO:tasks.rados.rados.0.smithi203.stdout:961: writing smithi20328086-43 from 766485 to 1232118 tid 1 2024-09-17T15:34:08.158 INFO:tasks.rados.rados.0.smithi203.stdout:961: writing smithi20328086-43 from 1990637 to 2638413 tid 2 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:961: writing smithi20328086-43 from 3299280 to 3966485 tid 3 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:962: delete oid 18 current snap is 104 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:959: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 0 (ObjNum 529 snap 0 seq_num 3605756736) dirty dne 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:961: finishing write tid 1 to smithi20328086-43 2024-09-17T15:34:08.175 INFO:tasks.rados.rados.0.smithi203.stdout:961: finishing write tid 2 to smithi20328086-43 2024-09-17T15:34:08.183 INFO:tasks.rados.rados.0.smithi203.stdout:961: finishing write tid 3 to smithi20328086-43 2024-09-17T15:34:08.183 INFO:tasks.rados.rados.0.smithi203.stdout:961: finishing write tid 4 to smithi20328086-43 2024-09-17T15:34:08.183 INFO:tasks.rados.rados.0.smithi203.stdout:961: finishing write tid 5 to smithi20328086-43 2024-09-17T15:34:08.183 INFO:tasks.rados.rados.0.smithi203.stdout:961: oid 43 updating version 0 to 187 2024-09-17T15:34:08.184 INFO:tasks.rados.rados.0.smithi203.stdout:961: oid 43 updating version 187 to 188 2024-09-17T15:34:08.184 INFO:tasks.rados.rados.0.smithi203.stdout:961: oid 43 version 188 is already newer than 186 2024-09-17T15:34:08.184 INFO:tasks.rados.rados.0.smithi203.stdout:961: oid 43 version 188 is already newer than 185 2024-09-17T15:34:08.184 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 188 (ObjNum 355 snap 104 seq_num 355) dirty exists 2024-09-17T15:34:08.184 INFO:tasks.rados.rados.0.smithi203.stdout:961: left oid 43 (ObjNum 355 snap 104 seq_num 355) 2024-09-17T15:34:08.185 INFO:tasks.rados.rados.0.smithi203.stdout:959: done (3 left) 2024-09-17T15:34:08.186 INFO:tasks.rados.rados.0.smithi203.stdout:960: done (2 left) 2024-09-17T15:34:08.186 INFO:tasks.rados.rados.0.smithi203.stdout:961: done (1 left) 2024-09-17T15:34:08.186 INFO:tasks.rados.rados.0.smithi203.stdout:962: done (0 left) 2024-09-17T15:34:08.186 INFO:tasks.rados.rados.0.smithi203.stdout:963: write oid 28 current snap is 104 2024-09-17T15:34:08.186 INFO:tasks.rados.rados.0.smithi203.stdout:963: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-17T15:34:08.196 INFO:tasks.rados.rados.0.smithi203.stdout:963: writing smithi20328086-28 from 400099 to 1016230 tid 1 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:963: writing smithi20328086-28 from 1591248 to 2000099 tid 2 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:964: rollback oid 34 current snap is 104 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 103 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:965: read oid 19 snap -1 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:965: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:966: write (excl) oid 33 current snap is 104 2024-09-17T15:34:08.208 INFO:tasks.rados.rados.0.smithi203.stdout:966: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-17T15:34:08.217 INFO:tasks.rados.rados.0.smithi203.stdout:966: writing smithi20328086-33 from 416906 to 824639 tid 1 2024-09-17T15:34:08.228 INFO:tasks.rados.rados.0.smithi203.stdout:966: writing smithi20328086-33 from 1366083 to 1877539 tid 2 2024-09-17T15:34:08.228 INFO:tasks.rados.rados.0.smithi203.stdout:967: read oid 12 snap -1 2024-09-17T15:34:08.228 INFO:tasks.rados.rados.0.smithi203.stdout:967: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:08.228 INFO:tasks.rados.rados.0.smithi203.stdout:968: delete oid 38 current snap is 104 2024-09-17T15:34:08.229 INFO:tasks.rados.rados.0.smithi203.stdout:963: finishing write tid 1 to smithi20328086-28 2024-09-17T15:34:08.229 INFO:tasks.rados.rados.0.smithi203.stdout:965: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:08.341 INFO:tasks.rados.rados.0.smithi203.stdout:963: finishing write tid 2 to smithi20328086-28 2024-09-17T15:34:08.341 INFO:tasks.rados.rados.0.smithi203.stdout:963: finishing write tid 3 to smithi20328086-28 2024-09-17T15:34:08.341 INFO:tasks.rados.rados.0.smithi203.stdout:964: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:34:08.341 INFO:tasks.rados.rados.0.smithi203.stdout:964: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:963: finishing write tid 4 to smithi20328086-28 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:963: oid 28 updating version 0 to 397 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:963: oid 28 version 397 is already newer than 395 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:963: oid 28 version 397 is already newer than 396 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 397 (ObjNum 357 snap 104 seq_num 357) dirty exists 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:963: left oid 28 (ObjNum 357 snap 104 seq_num 357) 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:964: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 159 (ObjNum 350 snap 102 seq_num 350) dirty exists 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:966: finishing write tid 1 to smithi20328086-33 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:966: finishing write tid 2 to smithi20328086-33 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:967: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:08.342 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:08.419 INFO:tasks.rados.rados.0.smithi203.stdout:966: finishing write tid 3 to smithi20328086-33 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: finishing write tid 4 to smithi20328086-33 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: oid 33 updating version 0 to 185 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: oid 33 version 185 is already newer than 184 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: oid 33 version 185 is already newer than 183 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 185 (ObjNum 358 snap 104 seq_num 358) dirty exists 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: left oid 33 (ObjNum 358 snap 104 seq_num 358) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:963: done (5 left) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:964: done (4 left) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:965: done (3 left) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:966: done (2 left) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:967: done (1 left) 2024-09-17T15:34:08.420 INFO:tasks.rados.rados.0.smithi203.stdout:968: done (0 left) 2024-09-17T15:34:08.421 INFO:tasks.rados.rados.0.smithi203.stdout:969: snap_create 2024-09-17T15:34:09.095 INFO:tasks.rados.rados.0.smithi203.stdout:969: done (0 left) 2024-09-17T15:34:09.096 INFO:tasks.rados.rados.0.smithi203.stdout:970: delete oid 43 current snap is 105 2024-09-17T15:34:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:970: done (0 left) 2024-09-17T15:34:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:971: read oid 5 snap 99 2024-09-17T15:34:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:971: expect (ObjNum 327 snap 95 seq_num 327) 2024-09-17T15:34:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:972: write (excl) oid 27 current snap is 105 2024-09-17T15:34:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:972: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-17T15:34:09.125 INFO:tasks.rados.rados.0.smithi203.stdout:972: writing smithi20328086-27 from 467327 to 1100807 tid 1 2024-09-17T15:34:09.141 INFO:tasks.rados.rados.0.smithi203.stdout:972: writing smithi20328086-27 from 1825176 to 2067327 tid 2 2024-09-17T15:34:09.141 INFO:tasks.rados.rados.0.smithi203.stdout:973: read oid 10 snap -1 2024-09-17T15:34:09.141 INFO:tasks.rados.rados.0.smithi203.stdout:973: expect deleted 2024-09-17T15:34:09.141 INFO:tasks.rados.rados.0.smithi203.stdout:974: delete oid 50 current snap is 105 2024-09-17T15:34:09.141 INFO:tasks.rados.rados.0.smithi203.stdout:971: expect (ObjNum 327 snap 95 seq_num 327) 2024-09-17T15:34:09.272 INFO:tasks.rados.rados.0.smithi203.stdout:972: finishing write tid 1 to smithi20328086-27 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: finishing write tid 2 to smithi20328086-27 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: finishing write tid 3 to smithi20328086-27 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: finishing write tid 4 to smithi20328086-27 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: oid 27 updating version 0 to 274 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: oid 27 updating version 274 to 275 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: oid 27 version 275 is already newer than 273 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 275 (ObjNum 361 snap 105 seq_num 361) dirty exists 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: left oid 27 (ObjNum 361 snap 105 seq_num 361) 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:971: done (3 left) 2024-09-17T15:34:09.273 INFO:tasks.rados.rados.0.smithi203.stdout:972: done (2 left) 2024-09-17T15:34:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:973: done (1 left) 2024-09-17T15:34:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:974: done (0 left) 2024-09-17T15:34:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:975: copy_from oid 28 from oid 14 current snap is 105 2024-09-17T15:34:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:976: delete oid 47 current snap is 105 2024-09-17T15:34:09.282 INFO:tasks.rados.rados.0.smithi203.stdout:976: done (1 left) 2024-09-17T15:34:09.282 INFO:tasks.rados.rados.0.smithi203.stdout:977: snap_remove snap 103 2024-09-17T15:34:09.599 INFO:tasks.rados.rados.0.smithi203.stdout:977: done (1 left) 2024-09-17T15:34:09.599 INFO:tasks.rados.rados.0.smithi203.stdout:978: snap_remove snap 98 2024-09-17T15:34:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:978: done (1 left) 2024-09-17T15:34:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:979: rollback oid 4 current snap is 105 2024-09-17T15:34:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 87 2024-09-17T15:34:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:975: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:34:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:975: finishing copy_from to smithi20328086-28 2024-09-17T15:34:10.600 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:34:10.600 INFO:tasks.rados.rados.0.smithi203.stdout:975: done (1 left) 2024-09-17T15:34:10.600 INFO:tasks.rados.rados.0.smithi203.stdout:980: write (excl) oid 44 current snap is 105 2024-09-17T15:34:10.600 INFO:tasks.rados.rados.0.smithi203.stdout:980: seq_num 364 ranges {517748=459227,1600622=406110} 2024-09-17T15:34:10.618 INFO:tasks.rados.rados.0.smithi203.stdout:980: writing smithi20328086-44 from 517748 to 976975 tid 1 2024-09-17T15:34:10.636 INFO:tasks.rados.rados.0.smithi203.stdout:980: writing smithi20328086-44 from 1600622 to 2006732 tid 2 2024-09-17T15:34:10.636 INFO:tasks.rados.rados.0.smithi203.stdout:981: snap_remove snap 89 2024-09-17T15:34:11.615 INFO:tasks.rados.rados.0.smithi203.stdout:981: done (2 left) 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:982: write oid 15 current snap is 105 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:979: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:979: finishing rollback tid 2 to smithi20328086-4 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 252 (ObjNum 303 snap 84 seq_num 303) dirty exists 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: finishing write tid 1 to smithi20328086-44 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: finishing write tid 2 to smithi20328086-44 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: finishing write tid 3 to smithi20328086-44 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: finishing write tid 4 to smithi20328086-44 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: oid 44 updating version 0 to 353 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: oid 44 version 353 is already newer than 352 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: oid 44 version 353 is already newer than 351 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 353 (ObjNum 364 snap 105 seq_num 364) dirty exists 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:980: left oid 44 (ObjNum 364 snap 105 seq_num 364) 2024-09-17T15:34:11.616 INFO:tasks.rados.rados.0.smithi203.stdout:982: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-17T15:34:11.624 INFO:tasks.rados.rados.0.smithi203.stdout:982: writing smithi20328086-15 from 534555 to 1185384 tid 1 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:982: writing smithi20328086-15 from 1659104 to 2134555 tid 2 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:979: done (2 left) 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:980: done (1 left) 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:983: copy_from oid 17 from oid 11 current snap is 105 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:984: read oid 8 snap -1 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:984: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:985: read oid 39 snap -1 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:985: expect (ObjNum 344 snap 102 seq_num 344) 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:986: read oid 14 snap 95 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:986: expect deleted 2024-09-17T15:34:11.631 INFO:tasks.rados.rados.0.smithi203.stdout:987: snap_remove snap 101 2024-09-17T15:34:12.624 INFO:tasks.rados.rados.0.smithi203.stdout:987: done (5 left) 2024-09-17T15:34:12.625 INFO:tasks.rados.rados.0.smithi203.stdout:988: copy_from oid 47 from oid 19 current snap is 105 2024-09-17T15:34:12.625 INFO:tasks.rados.rados.0.smithi203.stdout:985: expect (ObjNum 344 snap 102 seq_num 344) 2024-09-17T15:34:12.738 INFO:tasks.rados.rados.0.smithi203.stdout:982: finishing write tid 1 to smithi20328086-15 2024-09-17T15:34:12.738 INFO:tasks.rados.rados.0.smithi203.stdout:984: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-17T15:34:12.738 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 647978 2024-09-17T15:34:12.759 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:12.811 INFO:tasks.rados.rados.0.smithi203.stdout:982: finishing write tid 2 to smithi20328086-15 2024-09-17T15:34:12.811 INFO:tasks.rados.rados.0.smithi203.stdout:982: finishing write tid 3 to smithi20328086-15 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: finishing write tid 4 to smithi20328086-15 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: oid 15 updating version 0 to 255 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: oid 15 updating version 255 to 256 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: oid 15 version 256 is already newer than 254 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 256 (ObjNum 365 snap 105 seq_num 365) dirty exists 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: left oid 15 (ObjNum 365 snap 105 seq_num 365) 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:983: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:983: finishing copy_from to smithi20328086-17 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 290 (ObjNum 335 snap 99 seq_num 335) dirty exists 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:988: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:988: finishing copy_from to smithi20328086-47 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 47 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:34:12.812 INFO:tasks.rados.rados.0.smithi203.stdout:982: done (5 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:983: done (4 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:984: done (3 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:985: done (2 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:986: done (1 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:988: done (0 left) 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:989: copy_from oid 44 from oid 45 current snap is 105 2024-09-17T15:34:12.813 INFO:tasks.rados.rados.0.smithi203.stdout:990: delete oid 25 current snap is 105 2024-09-17T15:34:12.817 INFO:tasks.rados.rados.0.smithi203.stdout:990: done (1 left) 2024-09-17T15:34:12.817 INFO:tasks.rados.rados.0.smithi203.stdout:991: read oid 24 snap -1 2024-09-17T15:34:12.817 INFO:tasks.rados.rados.0.smithi203.stdout:991: expect deleted 2024-09-17T15:34:12.818 INFO:tasks.rados.rados.0.smithi203.stdout:992: copy_from oid 26 from oid 6 current snap is 105 2024-09-17T15:34:12.818 INFO:tasks.rados.rados.0.smithi203.stdout:993: delete oid 40 current snap is 105 2024-09-17T15:34:12.829 INFO:tasks.rados.rados.0.smithi203.stdout:991: done (3 left) 2024-09-17T15:34:12.829 INFO:tasks.rados.rados.0.smithi203.stdout:993: done (2 left) 2024-09-17T15:34:12.829 INFO:tasks.rados.rados.0.smithi203.stdout:994: copy_from oid 47 from oid 32 current snap is 105 2024-09-17T15:34:12.829 INFO:tasks.rados.rados.0.smithi203.stdout:995: copy_from oid 18 from oid 40 current snap is 105 2024-09-17T15:34:12.830 INFO:tasks.rados.rados.0.smithi203.stdout:996: copy_from oid 42 from oid 23 current snap is 105 2024-09-17T15:34:12.830 INFO:tasks.rados.rados.0.smithi203.stdout:997: snap_remove snap 87 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:997: done (5 left) 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:998: read oid 30 snap -1 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:998: expect deleted 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:999: read oid 21 snap 100 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:999: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:34:13.631 INFO:tasks.rados.rados.0.smithi203.stdout:1000: snap_remove snap 97 2024-09-17T15:34:14.598 INFO:tasks.rados.rados.0.smithi203.stdout:1000: done (7 left) 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:1001: write oid 34 current snap is 105 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:989: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:996: finishing copy_from to smithi20328086-42 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:996: got expected ENOENT (src dne) 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:992: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:34:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:1001: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-17T15:34:14.616 INFO:tasks.rados.rados.0.smithi203.stdout:1001: writing smithi20328086-34 from 584976 to 1061552 tid 1 2024-09-17T15:34:14.636 INFO:tasks.rados.rados.0.smithi203.stdout:1001: writing smithi20328086-34 from 1834550 to 2184976 tid 2 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:1002: rollback oid 46 current snap is 105 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 99 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:996: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:989: finishing copy_from to smithi20328086-44 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 182 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:992: finishing copy_from to smithi20328086-26 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 201 (ObjNum 290 snap 77 seq_num 290) dirty exists 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:995: finishing copy_from to smithi20328086-18 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:995: got expected ENOENT (src dne) 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:995: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:994: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:994: finishing copy_from to smithi20328086-47 2024-09-17T15:34:14.637 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:34:14.638 INFO:tasks.rados.rados.0.smithi203.stdout:999: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:34:14.699 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:14.803 INFO:tasks.rados.rados.0.smithi203.stdout:1001: finishing write tid 1 to smithi20328086-34 2024-09-17T15:34:14.803 INFO:tasks.rados.rados.0.smithi203.stdout:1002: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:34:14.803 INFO:tasks.rados.rados.0.smithi203.stdout:1002: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 293 (ObjNum 268 snap 70 seq_num 268) dirty exists 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: finishing write tid 2 to smithi20328086-34 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: finishing write tid 3 to smithi20328086-34 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: finishing write tid 4 to smithi20328086-34 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: oid 34 updating version 0 to 176 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: oid 34 updating version 176 to 177 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: oid 34 version 177 is already newer than 175 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 177 (ObjNum 368 snap 105 seq_num 368) dirty exists 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:1001: left oid 34 (ObjNum 368 snap 105 seq_num 368) 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:989: done (8 left) 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:992: done (7 left) 2024-09-17T15:34:14.804 INFO:tasks.rados.rados.0.smithi203.stdout:994: done (6 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:995: done (5 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:996: done (4 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:998: done (3 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:999: done (2 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:1001: done (1 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:1002: done (0 left) 2024-09-17T15:34:14.805 INFO:tasks.rados.rados.0.smithi203.stdout:1003: delete oid 3 current snap is 105 2024-09-17T15:34:14.807 INFO:tasks.rados.rados.0.smithi203.stdout:1003: done (0 left) 2024-09-17T15:34:14.807 INFO:tasks.rados.rados.0.smithi203.stdout:1004: snap_remove snap 102 2024-09-17T15:34:15.644 INFO:tasks.rados.rados.0.smithi203.stdout:1004: done (0 left) 2024-09-17T15:34:15.645 INFO:tasks.rados.rados.0.smithi203.stdout:1005: write oid 26 current snap is 105 2024-09-17T15:34:15.645 INFO:tasks.rados.rados.0.smithi203.stdout:1005: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-17T15:34:15.667 INFO:tasks.rados.rados.0.smithi203.stdout:1005: writing smithi20328086-26 from 618590 to 1245664 tid 1 2024-09-17T15:34:15.684 INFO:tasks.rados.rados.0.smithi203.stdout:1005: writing smithi20328086-26 from 1835161 to 2218590 tid 2 2024-09-17T15:34:15.684 INFO:tasks.rados.rados.0.smithi203.stdout:1006: write oid 40 current snap is 105 2024-09-17T15:34:15.684 INFO:tasks.rados.rados.0.smithi203.stdout:1005: finishing write tid 1 to smithi20328086-26 2024-09-17T15:34:15.684 INFO:tasks.rados.rados.0.smithi203.stdout:1006: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-17T15:34:15.706 INFO:tasks.rados.rados.0.smithi203.stdout:1006: writing smithi20328086-40 from 635397 to 1337720 tid 1 2024-09-17T15:34:15.719 INFO:tasks.rados.rados.0.smithi203.stdout:1006: writing smithi20328086-40 from 1893643 to 2235397 tid 2 2024-09-17T15:34:15.719 INFO:tasks.rados.rados.0.smithi203.stdout:1007: snap_create 2024-09-17T15:34:15.719 INFO:tasks.rados.rados.0.smithi203.stdout:1005: finishing write tid 2 to smithi20328086-26 2024-09-17T15:34:15.719 INFO:tasks.rados.rados.0.smithi203.stdout:1005: finishing write tid 3 to smithi20328086-26 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1005: finishing write tid 4 to smithi20328086-26 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1005: oid 26 updating version 0 to 423 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1005: oid 26 updating version 423 to 424 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1005: oid 26 updating version 424 to 425 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 425 (ObjNum 370 snap 105 seq_num 370) dirty exists 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1005: left oid 26 (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:34:15.720 INFO:tasks.rados.rados.0.smithi203.stdout:1006: finishing write tid 1 to smithi20328086-40 2024-09-17T15:34:15.725 INFO:tasks.rados.rados.0.smithi203.stdout:1006: finishing write tid 2 to smithi20328086-40 2024-09-17T15:34:15.725 INFO:tasks.rados.rados.0.smithi203.stdout:1006: finishing write tid 3 to smithi20328086-40 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:1006: finishing write tid 4 to smithi20328086-40 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:1006: oid 40 updating version 0 to 395 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:1006: oid 40 version 395 is already newer than 394 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:1006: oid 40 version 395 is already newer than 393 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 395 (ObjNum 371 snap 105 seq_num 371) dirty exists 2024-09-17T15:34:15.726 INFO:tasks.rados.rados.0.smithi203.stdout:1006: left oid 40 (ObjNum 371 snap 105 seq_num 371) 2024-09-17T15:34:16.651 INFO:tasks.rados.rados.0.smithi203.stdout:1005: done (2 left) 2024-09-17T15:34:16.651 INFO:tasks.rados.rados.0.smithi203.stdout:1006: done (1 left) 2024-09-17T15:34:16.651 INFO:tasks.rados.rados.0.smithi203.stdout:1007: done (0 left) 2024-09-17T15:34:16.652 INFO:tasks.rados.rados.0.smithi203.stdout:1008: copy_from oid 14 from oid 42 current snap is 106 2024-09-17T15:34:16.652 INFO:tasks.rados.rados.0.smithi203.stdout:1009: snap_create 2024-09-17T15:34:16.656 INFO:tasks.rados.rados.0.smithi203.stdout:1008: finishing copy_from to smithi20328086-14 2024-09-17T15:34:16.657 INFO:tasks.rados.rados.0.smithi203.stdout:1008: got expected ENOENT (src dne) 2024-09-17T15:34:16.657 INFO:tasks.rados.rados.0.smithi203.stdout:1008: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:34:17.660 INFO:tasks.rados.rados.0.smithi203.stdout:1008: done (1 left) 2024-09-17T15:34:17.660 INFO:tasks.rados.rados.0.smithi203.stdout:1009: done (0 left) 2024-09-17T15:34:17.660 INFO:tasks.rados.rados.0.smithi203.stdout:1010: copy_from oid 32 from oid 38 current snap is 107 2024-09-17T15:34:17.660 INFO:tasks.rados.rados.0.smithi203.stdout:1011: write (excl) oid 45 current snap is 107 2024-09-17T15:34:17.660 INFO:tasks.rados.rados.0.smithi203.stdout:1011: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-17T15:34:17.683 INFO:tasks.rados.rados.0.smithi203.stdout:1011: writing smithi20328086-45 from 652204 to 1429776 tid 1 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1011: writing smithi20328086-45 from 1952125 to 2252204 tid 2 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1012: delete oid 10 current snap is 107 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1010: finishing copy_from to smithi20328086-32 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1010: got expected ENOENT (src dne) 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1010: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:34:17.697 INFO:tasks.rados.rados.0.smithi203.stdout:1011: finishing write tid 1 to smithi20328086-45 2024-09-17T15:34:17.701 INFO:tasks.rados.rados.0.smithi203.stdout:1010: done (2 left) 2024-09-17T15:34:17.701 INFO:tasks.rados.rados.0.smithi203.stdout:1012: done (1 left) 2024-09-17T15:34:17.701 INFO:tasks.rados.rados.0.smithi203.stdout:1013: snap_create 2024-09-17T15:34:17.703 INFO:tasks.rados.rados.0.smithi203.stdout:1011: finishing write tid 2 to smithi20328086-45 2024-09-17T15:34:17.704 INFO:tasks.rados.rados.0.smithi203.stdout:1011: finishing write tid 3 to smithi20328086-45 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:1011: finishing write tid 4 to smithi20328086-45 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:1011: oid 45 updating version 0 to 255 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:1011: oid 45 version 255 is already newer than 253 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:1011: oid 45 version 255 is already newer than 254 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 255 (ObjNum 372 snap 107 seq_num 372) dirty exists 2024-09-17T15:34:17.705 INFO:tasks.rados.rados.0.smithi203.stdout:1011: left oid 45 (ObjNum 372 snap 107 seq_num 372) 2024-09-17T15:34:18.671 INFO:tasks.rados.rados.0.smithi203.stdout:1011: done (1 left) 2024-09-17T15:34:18.671 INFO:tasks.rados.rados.0.smithi203.stdout:1013: done (0 left) 2024-09-17T15:34:18.672 INFO:tasks.rados.rados.0.smithi203.stdout:1014: read oid 38 snap -1 2024-09-17T15:34:18.672 INFO:tasks.rados.rados.0.smithi203.stdout:1014: expect deleted 2024-09-17T15:34:18.672 INFO:tasks.rados.rados.0.smithi203.stdout:1015: write (excl) oid 9 current snap is 108 2024-09-17T15:34:18.672 INFO:tasks.rados.rados.0.smithi203.stdout:1015: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-17T15:34:18.693 INFO:tasks.rados.rados.0.smithi203.stdout:1015: writing smithi20328086-9 from 685818 to 1330241 tid 1 2024-09-17T15:34:18.709 INFO:tasks.rados.rados.0.smithi203.stdout:1015: writing smithi20328086-9 from 2069089 to 2285818 tid 2 2024-09-17T15:34:18.709 INFO:tasks.rados.rados.0.smithi203.stdout:1016: read oid 1 snap -1 2024-09-17T15:34:18.709 INFO:tasks.rados.rados.0.smithi203.stdout:1016: expect (ObjNum 346 snap 102 seq_num 346) 2024-09-17T15:34:18.709 INFO:tasks.rados.rados.0.smithi203.stdout:1017: write (excl) oid 40 current snap is 108 2024-09-17T15:34:18.709 INFO:tasks.rados.rados.0.smithi203.stdout:1017: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-17T15:34:18.730 INFO:tasks.rados.rados.0.smithi203.stdout:1017: writing smithi20328086-40 from 702625 to 1422297 tid 1 2024-09-17T15:34:18.740 INFO:tasks.rados.rados.0.smithi203.stdout:1017: writing smithi20328086-40 from 2127571 to 2302625 tid 2 2024-09-17T15:34:18.740 INFO:tasks.rados.rados.0.smithi203.stdout:1018: read oid 2 snap -1 2024-09-17T15:34:18.740 INFO:tasks.rados.rados.0.smithi203.stdout:1018: expect (ObjNum 351 snap 102 seq_num 351) 2024-09-17T15:34:18.740 INFO:tasks.rados.rados.0.smithi203.stdout:1019: snap_remove snap 106 2024-09-17T15:34:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:1019: done (5 left) 2024-09-17T15:34:19.601 INFO:tasks.rados.rados.0.smithi203.stdout:1020: snap_remove snap 94 2024-09-17T15:34:20.602 INFO:tasks.rados.rados.0.smithi203.stdout:1020: done (5 left) 2024-09-17T15:34:20.605 INFO:tasks.rados.rados.0.smithi203.stdout:1015: finishing write tid 1 to smithi20328086-9 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: finishing write tid 2 to smithi20328086-9 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: finishing write tid 3 to smithi20328086-9 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: finishing write tid 4 to smithi20328086-9 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: oid 9 updating version 0 to 324 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: oid 9 updating version 324 to 325 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: oid 9 version 325 is already newer than 323 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 325 (ObjNum 374 snap 108 seq_num 374) dirty exists 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1015: left oid 9 (ObjNum 374 snap 108 seq_num 374) 2024-09-17T15:34:20.606 INFO:tasks.rados.rados.0.smithi203.stdout:1016: expect (ObjNum 346 snap 102 seq_num 346) 2024-09-17T15:34:20.798 INFO:tasks.rados.rados.0.smithi203.stdout:1017: finishing write tid 1 to smithi20328086-40 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: finishing write tid 2 to smithi20328086-40 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: finishing write tid 3 to smithi20328086-40 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: finishing write tid 4 to smithi20328086-40 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: oid 40 updating version 0 to 399 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: oid 40 version 399 is already newer than 397 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: oid 40 version 399 is already newer than 398 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 399 (ObjNum 375 snap 108 seq_num 375) dirty exists 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1017: left oid 40 (ObjNum 375 snap 108 seq_num 375) 2024-09-17T15:34:20.799 INFO:tasks.rados.rados.0.smithi203.stdout:1018: expect (ObjNum 351 snap 102 seq_num 351) 2024-09-17T15:34:20.845 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2353411 2024-09-17T15:34:20.995 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1021: read oid 10 snap -1 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1021: expect deleted 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1014: done (5 left) 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1015: done (4 left) 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1016: done (3 left) 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1017: done (2 left) 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1018: done (1 left) 2024-09-17T15:34:21.099 INFO:tasks.rados.rados.0.smithi203.stdout:1022: copy_from oid 5 from oid 17 current snap is 108 2024-09-17T15:34:21.100 INFO:tasks.rados.rados.0.smithi203.stdout:1023: write (excl) oid 2 current snap is 108 2024-09-17T15:34:21.100 INFO:tasks.rados.rados.0.smithi203.stdout:1023: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-17T15:34:21.125 INFO:tasks.rados.rados.0.smithi203.stdout:1023: writing smithi20328086-2 from 719432 to 1514353 tid 1 2024-09-17T15:34:21.138 INFO:tasks.rados.rados.0.smithi203.stdout:1023: writing smithi20328086-2 from 2186053 to 2319432 tid 2 2024-09-17T15:34:21.139 INFO:tasks.rados.rados.0.smithi203.stdout:1024: read oid 18 snap -1 2024-09-17T15:34:21.139 INFO:tasks.rados.rados.0.smithi203.stdout:1024: expect deleted 2024-09-17T15:34:21.139 INFO:tasks.rados.rados.0.smithi203.stdout:1025: write (excl) oid 34 current snap is 108 2024-09-17T15:34:21.139 INFO:tasks.rados.rados.0.smithi203.stdout:1025: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-17T15:34:21.156 INFO:tasks.rados.rados.0.smithi203.stdout:1025: writing smithi20328086-34 from 736239 to 1206409 tid 1 2024-09-17T15:34:21.168 INFO:tasks.rados.rados.0.smithi203.stdout:1025: writing smithi20328086-34 from 1728182 to 2299964 tid 2 2024-09-17T15:34:21.168 INFO:tasks.rados.rados.0.smithi203.stdout:1026: write oid 41 current snap is 108 2024-09-17T15:34:21.169 INFO:tasks.rados.rados.0.smithi203.stdout:1026: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-17T15:34:21.180 INFO:tasks.rados.rados.0.smithi203.stdout:1026: writing smithi20328086-41 from 753046 to 1298465 tid 1 2024-09-17T15:34:21.188 INFO:tasks.rados.rados.0.smithi203.stdout:1026: writing smithi20328086-41 from 1786664 to 2218457 tid 2 2024-09-17T15:34:21.188 INFO:tasks.rados.rados.0.smithi203.stdout:1027: write oid 16 current snap is 108 2024-09-17T15:34:21.188 INFO:tasks.rados.rados.0.smithi203.stdout:1027: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-17T15:34:21.198 INFO:tasks.rados.rados.0.smithi203.stdout:1027: writing smithi20328086-16 from 769853 to 1390521 tid 1 2024-09-17T15:34:21.205 INFO:tasks.rados.rados.0.smithi203.stdout:1027: writing smithi20328086-16 from 1845146 to 2369853 tid 2 2024-09-17T15:34:21.205 INFO:tasks.rados.rados.0.smithi203.stdout:1028: delete oid 23 current snap is 108 2024-09-17T15:34:21.205 INFO:tasks.rados.rados.0.smithi203.stdout:1023: finishing write tid 1 to smithi20328086-2 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1022: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1022: finishing copy_from to smithi20328086-5 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 290 (ObjNum 335 snap 99 seq_num 335) dirty exists 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: finishing write tid 2 to smithi20328086-2 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: finishing write tid 3 to smithi20328086-2 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: finishing write tid 4 to smithi20328086-2 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: oid 2 updating version 0 to 260 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: oid 2 version 260 is already newer than 259 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: oid 2 version 260 is already newer than 258 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 260 (ObjNum 376 snap 108 seq_num 376) dirty exists 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1023: left oid 2 (ObjNum 376 snap 108 seq_num 376) 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1025: finishing write tid 1 to smithi20328086-34 2024-09-17T15:34:21.206 INFO:tasks.rados.rados.0.smithi203.stdout:1025: finishing write tid 2 to smithi20328086-34 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: finishing write tid 3 to smithi20328086-34 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: finishing write tid 4 to smithi20328086-34 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: oid 34 updating version 0 to 185 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: oid 34 version 185 is already newer than 184 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: oid 34 version 185 is already newer than 183 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 185 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1025: left oid 34 (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: finishing write tid 1 to smithi20328086-41 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: finishing write tid 2 to smithi20328086-41 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: finishing write tid 3 to smithi20328086-41 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: finishing write tid 4 to smithi20328086-41 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: oid 41 updating version 0 to 191 2024-09-17T15:34:21.207 INFO:tasks.rados.rados.0.smithi203.stdout:1026: oid 41 updating version 191 to 192 2024-09-17T15:34:21.208 INFO:tasks.rados.rados.0.smithi203.stdout:1026: oid 41 version 192 is already newer than 190 2024-09-17T15:34:21.208 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 192 (ObjNum 378 snap 108 seq_num 378) dirty exists 2024-09-17T15:34:21.208 INFO:tasks.rados.rados.0.smithi203.stdout:1026: left oid 41 (ObjNum 378 snap 108 seq_num 378) 2024-09-17T15:34:21.208 INFO:tasks.rados.rados.0.smithi203.stdout:1027: finishing write tid 1 to smithi20328086-16 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1021: done (7 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1022: done (6 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1023: done (5 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1024: done (4 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1025: done (3 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1026: done (2 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1028: done (1 left) 2024-09-17T15:34:21.210 INFO:tasks.rados.rados.0.smithi203.stdout:1029: delete oid 25 current snap is 108 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: finishing write tid 2 to smithi20328086-16 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: finishing write tid 3 to smithi20328086-16 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: finishing write tid 4 to smithi20328086-16 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: oid 16 updating version 0 to 309 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: oid 16 updating version 309 to 310 2024-09-17T15:34:21.212 INFO:tasks.rados.rados.0.smithi203.stdout:1027: oid 16 version 310 is already newer than 308 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 310 (ObjNum 379 snap 108 seq_num 379) dirty exists 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:1027: left oid 16 (ObjNum 379 snap 108 seq_num 379) 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:1027: done (1 left) 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:1029: done (0 left) 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:1030: write (excl) oid 2 current snap is 108 2024-09-17T15:34:21.213 INFO:tasks.rados.rados.0.smithi203.stdout:1030: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-17T15:34:21.218 INFO:tasks.rados.rados.0.smithi203.stdout:1030: writing smithi20328086-2 from 420274 to 983042 tid 1 2024-09-17T15:34:21.225 INFO:tasks.rados.rados.0.smithi203.stdout:1030: writing smithi20328086-2 from 1620592 to 2059723 tid 2 2024-09-17T15:34:21.225 INFO:tasks.rados.rados.0.smithi203.stdout:1031: write (excl) oid 8 current snap is 108 2024-09-17T15:34:21.225 INFO:tasks.rados.rados.0.smithi203.stdout:1031: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-17T15:34:21.232 INFO:tasks.rados.rados.0.smithi203.stdout:1031: writing smithi20328086-8 from 437081 to 1075098 tid 1 2024-09-17T15:34:21.239 INFO:tasks.rados.rados.0.smithi203.stdout:1031: writing smithi20328086-8 from 1679074 to 2261863 tid 2 2024-09-17T15:34:21.239 INFO:tasks.rados.rados.0.smithi203.stdout:1032: write oid 9 current snap is 108 2024-09-17T15:34:21.240 INFO:tasks.rados.rados.0.smithi203.stdout:1032: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-17T15:34:21.246 INFO:tasks.rados.rados.0.smithi203.stdout:1032: writing smithi20328086-9 from 453888 to 1167154 tid 1 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1032: writing smithi20328086-9 from 1737556 to 2180356 tid 2 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1033: copy_from oid 28 from oid 37 current snap is 108 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: finishing write tid 1 to smithi20328086-2 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: finishing write tid 2 to smithi20328086-2 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: finishing write tid 3 to smithi20328086-2 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: finishing write tid 4 to smithi20328086-2 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: oid 2 updating version 0 to 262 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: oid 2 version 262 is already newer than 261 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: oid 2 updating version 262 to 263 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 263 (ObjNum 382 snap 108 seq_num 382) dirty exists 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1030: left oid 2 (ObjNum 382 snap 108 seq_num 382) 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1031: finishing write tid 1 to smithi20328086-8 2024-09-17T15:34:21.254 INFO:tasks.rados.rados.0.smithi203.stdout:1031: finishing write tid 2 to smithi20328086-8 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: finishing write tid 3 to smithi20328086-8 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: finishing write tid 4 to smithi20328086-8 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: oid 8 updating version 0 to 296 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: oid 8 version 296 is already newer than 294 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: oid 8 version 296 is already newer than 295 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 296 (ObjNum 383 snap 108 seq_num 383) dirty exists 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: left oid 8 (ObjNum 383 snap 108 seq_num 383) 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1030: done (3 left) 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1031: done (2 left) 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1034: write oid 20 current snap is 108 2024-09-17T15:34:21.255 INFO:tasks.rados.rados.0.smithi203.stdout:1034: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-17T15:34:21.261 INFO:tasks.rados.rados.0.smithi203.stdout:1034: writing smithi20328086-20 from 470695 to 1259210 tid 1 2024-09-17T15:34:21.267 INFO:tasks.rados.rados.0.smithi203.stdout:1034: writing smithi20328086-20 from 1679685 to 2266143 tid 2 2024-09-17T15:34:21.267 INFO:tasks.rados.rados.0.smithi203.stdout:1035: write (excl) oid 40 current snap is 108 2024-09-17T15:34:21.267 INFO:tasks.rados.rados.0.smithi203.stdout:1032: finishing write tid 1 to smithi20328086-9 2024-09-17T15:34:21.267 INFO:tasks.rados.rados.0.smithi203.stdout:1035: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-17T15:34:21.273 INFO:tasks.rados.rados.0.smithi203.stdout:1035: writing smithi20328086-40 from 487502 to 951266 tid 1 2024-09-17T15:34:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:1035: writing smithi20328086-40 from 1738167 to 2184636 tid 2 2024-09-17T15:34:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:1036: copy_from oid 16 from oid 31 current snap is 108 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: finishing write tid 2 to smithi20328086-9 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: finishing write tid 3 to smithi20328086-9 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: finishing write tid 4 to smithi20328086-9 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: oid 9 updating version 0 to 335 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: oid 9 updating version 335 to 337 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: oid 9 version 337 is already newer than 336 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 337 (ObjNum 384 snap 108 seq_num 384) dirty exists 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: left oid 9 (ObjNum 384 snap 108 seq_num 384) 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1033: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1034: finishing write tid 1 to smithi20328086-20 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1034: finishing write tid 2 to smithi20328086-20 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1032: done (4 left) 2024-09-17T15:34:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:1037: write (excl) oid 19 current snap is 108 2024-09-17T15:34:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:1037: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-17T15:34:21.286 INFO:tasks.rados.rados.0.smithi203.stdout:1037: writing smithi20328086-19 from 504309 to 1043322 tid 1 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1037: writing smithi20328086-19 from 1796649 to 2386776 tid 2 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1038: rollback oid 50 current snap is 108 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 95 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1033: finishing copy_from to smithi20328086-28 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1034: finishing write tid 3 to smithi20328086-20 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1033: done (5 left) 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1039: write oid 25 current snap is 108 2024-09-17T15:34:21.295 INFO:tasks.rados.rados.0.smithi203.stdout:1039: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-17T15:34:21.302 INFO:tasks.rados.rados.0.smithi203.stdout:1039: writing smithi20328086-25 from 521116 to 1251731 tid 1 2024-09-17T15:34:21.309 INFO:tasks.rados.rados.0.smithi203.stdout:1039: writing smithi20328086-25 from 1971484 to 2521116 tid 2 2024-09-17T15:34:21.309 INFO:tasks.rados.rados.0.smithi203.stdout:1040: snap_remove snap 99 2024-09-17T15:34:21.679 INFO:tasks.rados.rados.0.smithi203.stdout:1040: done (6 left) 2024-09-17T15:34:21.679 INFO:tasks.rados.rados.0.smithi203.stdout:1041: snap_remove snap 107 2024-09-17T15:34:22.688 INFO:tasks.rados.rados.0.smithi203.stdout:1041: done (6 left) 2024-09-17T15:34:22.688 INFO:tasks.rados.rados.0.smithi203.stdout:1042: copy_from oid 18 from oid 34 current snap is 108 2024-09-17T15:34:22.688 INFO:tasks.rados.rados.0.smithi203.stdout:1034: finishing write tid 4 to smithi20328086-20 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1034: oid 20 updating version 0 to 361 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1034: oid 20 updating version 361 to 362 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1034: oid 20 updating version 362 to 363 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 363 (ObjNum 385 snap 108 seq_num 385) dirty exists 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1034: left oid 20 (ObjNum 385 snap 108 seq_num 385) 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: finishing write tid 1 to smithi20328086-40 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: finishing write tid 2 to smithi20328086-40 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: finishing write tid 3 to smithi20328086-40 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: finishing write tid 4 to smithi20328086-40 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: oid 40 updating version 0 to 412 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: oid 40 version 412 is already newer than 411 2024-09-17T15:34:22.689 INFO:tasks.rados.rados.0.smithi203.stdout:1035: oid 40 version 412 is already newer than 410 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 412 (ObjNum 386 snap 108 seq_num 386) dirty exists 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1035: left oid 40 (ObjNum 386 snap 108 seq_num 386) 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1038: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1038: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 296 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1036: finishing copy_from racing read to smithi20328086-16 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1037: finishing write tid 1 to smithi20328086-19 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1034: done (6 left) 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1035: done (5 left) 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1038: done (4 left) 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1043: write (excl) oid 29 current snap is 108 2024-09-17T15:34:22.690 INFO:tasks.rados.rados.0.smithi203.stdout:1043: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-17T15:34:22.694 INFO:tasks.rados.rados.0.smithi203.stdout:1043: writing smithi20328086-29 from 537923 to 943787 tid 1 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:1043: writing smithi20328086-29 from 1513613 to 2107409 tid 2 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:1044: read oid 26 snap -1 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:1044: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:1045: snap_create 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:1036: finishing copy_from to smithi20328086-16 2024-09-17T15:34:22.701 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 103 (ObjNum 320 snap 93 seq_num 320) dirty exists 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: finishing write tid 1 to smithi20328086-25 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: finishing write tid 2 to smithi20328086-19 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: finishing write tid 3 to smithi20328086-19 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: finishing write tid 4 to smithi20328086-19 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: oid 19 updating version 0 to 312 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: oid 19 updating version 312 to 314 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: oid 19 updating version 314 to 315 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 315 (ObjNum 387 snap 108 seq_num 387) dirty exists 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1037: left oid 19 (ObjNum 387 snap 108 seq_num 387) 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: finishing write tid 2 to smithi20328086-25 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: finishing write tid 3 to smithi20328086-25 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: finishing write tid 4 to smithi20328086-25 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: oid 25 updating version 0 to 339 2024-09-17T15:34:22.702 INFO:tasks.rados.rados.0.smithi203.stdout:1039: oid 25 updating version 339 to 340 2024-09-17T15:34:22.703 INFO:tasks.rados.rados.0.smithi203.stdout:1039: oid 25 version 340 is already newer than 338 2024-09-17T15:34:22.703 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 340 (ObjNum 388 snap 108 seq_num 388) dirty exists 2024-09-17T15:34:22.703 INFO:tasks.rados.rados.0.smithi203.stdout:1039: left oid 25 (ObjNum 388 snap 108 seq_num 388) 2024-09-17T15:34:22.705 INFO:tasks.rados.rados.0.smithi203.stdout:1043: finishing write tid 1 to smithi20328086-29 2024-09-17T15:34:22.710 INFO:tasks.rados.rados.0.smithi203.stdout:1043: finishing write tid 2 to smithi20328086-29 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: finishing write tid 3 to smithi20328086-29 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: finishing write tid 4 to smithi20328086-29 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: oid 29 updating version 0 to 345 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: oid 29 updating version 345 to 346 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: oid 29 updating version 346 to 347 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 347 (ObjNum 389 snap 108 seq_num 389) dirty exists 2024-09-17T15:34:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1043: left oid 29 (ObjNum 389 snap 108 seq_num 389) 2024-09-17T15:34:22.719 INFO:tasks.rados.rados.0.smithi203.stdout:1044: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:34:22.720 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:22.784 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 484488 2024-09-17T15:34:22.822 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 834012 2024-09-17T15:34:22.842 INFO:tasks.rados.rados.0.smithi203.stdout:1042: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:34:22.842 INFO:tasks.rados.rados.0.smithi203.stdout:1042: finishing copy_from to smithi20328086-18 2024-09-17T15:34:22.842 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 185 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1036: done (6 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1037: done (5 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1039: done (4 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1042: done (3 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1043: done (2 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1044: done (1 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1045: done (0 left) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1046: read oid 2 snap -1 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1046: expect (ObjNum 382 snap 108 seq_num 382) 2024-09-17T15:34:23.690 INFO:tasks.rados.rados.0.smithi203.stdout:1047: snap_create 2024-09-17T15:34:23.702 INFO:tasks.rados.rados.0.smithi203.stdout:1046: expect (ObjNum 382 snap 108 seq_num 382) 2024-09-17T15:34:23.766 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:23.824 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:1046: done (1 left) 2024-09-17T15:34:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:1047: done (0 left) 2024-09-17T15:34:24.603 INFO:tasks.rados.rados.0.smithi203.stdout:1048: write oid 37 current snap is 110 2024-09-17T15:34:24.603 INFO:tasks.rados.rados.0.smithi203.stdout:1048: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-17T15:34:24.613 INFO:tasks.rados.rados.0.smithi203.stdout:1048: writing smithi20328086-37 from 554730 to 1035843 tid 1 2024-09-17T15:34:24.626 INFO:tasks.rados.rados.0.smithi203.stdout:1048: writing smithi20328086-37 from 1572095 to 2309549 tid 2 2024-09-17T15:34:24.626 INFO:tasks.rados.rados.0.smithi203.stdout:1049: rollback oid 42 current snap is 110 2024-09-17T15:34:24.626 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 95 2024-09-17T15:34:24.626 INFO:tasks.rados.rados.0.smithi203.stdout:1050: write oid 1 current snap is 110 2024-09-17T15:34:24.626 INFO:tasks.rados.rados.0.smithi203.stdout:1050: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-17T15:34:24.637 INFO:tasks.rados.rados.0.smithi203.stdout:1050: writing smithi20328086-1 from 571537 to 1127899 tid 1 2024-09-17T15:34:24.649 INFO:tasks.rados.rados.0.smithi203.stdout:1050: writing smithi20328086-1 from 1630577 to 2228042 tid 2 2024-09-17T15:34:24.649 INFO:tasks.rados.rados.0.smithi203.stdout:1051: write (excl) oid 10 current snap is 110 2024-09-17T15:34:24.649 INFO:tasks.rados.rados.0.smithi203.stdout:1051: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-17T15:34:24.659 INFO:tasks.rados.rados.0.smithi203.stdout:1051: writing smithi20328086-10 from 588344 to 1219955 tid 1 2024-09-17T15:34:24.669 INFO:tasks.rados.rados.0.smithi203.stdout:1051: writing smithi20328086-10 from 1689059 to 2430182 tid 2 2024-09-17T15:34:24.669 INFO:tasks.rados.rados.0.smithi203.stdout:1052: write oid 28 current snap is 110 2024-09-17T15:34:24.669 INFO:tasks.rados.rados.0.smithi203.stdout:1052: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-17T15:34:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:1052: writing smithi20328086-28 from 605151 to 1312011 tid 1 2024-09-17T15:34:24.688 INFO:tasks.rados.rados.0.smithi203.stdout:1052: writing smithi20328086-28 from 2031188 to 2605151 tid 2 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1053: rollback oid 7 current snap is 110 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 109 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1049: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 225 (ObjNum 129 snap 0 seq_num 3605662832) dirty dne 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: finishing write tid 1 to smithi20328086-37 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: finishing write tid 2 to smithi20328086-37 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: finishing write tid 3 to smithi20328086-37 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: finishing write tid 4 to smithi20328086-37 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: oid 37 updating version 0 to 323 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: oid 37 updating version 323 to 325 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:1048: oid 37 version 325 is already newer than 324 2024-09-17T15:34:24.689 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 325 (ObjNum 390 snap 110 seq_num 390) dirty exists 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1048: left oid 37 (ObjNum 390 snap 110 seq_num 390) 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1050: finishing write tid 1 to smithi20328086-1 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1050: finishing write tid 2 to smithi20328086-1 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1048: done (5 left) 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1049: done (4 left) 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1054: write oid 30 current snap is 110 2024-09-17T15:34:24.690 INFO:tasks.rados.rados.0.smithi203.stdout:1054: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-17T15:34:24.697 INFO:tasks.rados.rados.0.smithi203.stdout:1054: writing smithi20328086-30 from 621958 to 1404067 tid 1 2024-09-17T15:34:24.704 INFO:tasks.rados.rados.0.smithi203.stdout:1054: writing smithi20328086-30 from 2089670 to 2621958 tid 2 2024-09-17T15:34:24.704 INFO:tasks.rados.rados.0.smithi203.stdout:1055: write (excl) oid 23 current snap is 110 2024-09-17T15:34:24.705 INFO:tasks.rados.rados.0.smithi203.stdout:1055: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-17T15:34:24.710 INFO:tasks.rados.rados.0.smithi203.stdout:1055: writing smithi20328086-23 from 638765 to 1096123 tid 1 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1055: writing smithi20328086-23 from 1748152 to 2352955 tid 2 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1056: snap_remove snap 105 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: finishing write tid 3 to smithi20328086-1 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: finishing write tid 4 to smithi20328086-1 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: oid 1 updating version 0 to 375 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: oid 1 version 375 is already newer than 374 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: oid 1 version 375 is already newer than 373 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 375 (ObjNum 391 snap 110 seq_num 391) dirty exists 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1050: left oid 1 (ObjNum 391 snap 110 seq_num 391) 2024-09-17T15:34:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:1051: finishing write tid 1 to smithi20328086-10 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: finishing write tid 2 to smithi20328086-10 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: finishing write tid 3 to smithi20328086-10 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: finishing write tid 4 to smithi20328086-10 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: oid 10 updating version 0 to 254 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: oid 10 version 254 is already newer than 253 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: oid 10 updating version 254 to 255 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 255 (ObjNum 392 snap 110 seq_num 392) dirty exists 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1051: left oid 10 (ObjNum 392 snap 110 seq_num 392) 2024-09-17T15:34:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:1052: finishing write tid 1 to smithi20328086-28 2024-09-17T15:34:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:1050: done (6 left) 2024-09-17T15:34:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:1051: done (5 left) 2024-09-17T15:34:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:1056: done (4 left) 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: finishing write tid 2 to smithi20328086-28 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: finishing write tid 3 to smithi20328086-28 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1053: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1053: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1053: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 266 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: finishing write tid 4 to smithi20328086-28 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: oid 28 updating version 0 to 437 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: oid 28 updating version 437 to 439 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: oid 28 version 439 is already newer than 438 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 439 (ObjNum 393 snap 110 seq_num 393) dirty exists 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1052: left oid 28 (ObjNum 393 snap 110 seq_num 393) 2024-09-17T15:34:25.712 INFO:tasks.rados.rados.0.smithi203.stdout:1054: finishing write tid 1 to smithi20328086-30 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: finishing write tid 2 to smithi20328086-30 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: finishing write tid 3 to smithi20328086-30 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: finishing write tid 4 to smithi20328086-30 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: oid 30 updating version 0 to 258 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: oid 30 version 258 is already newer than 257 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: oid 30 updating version 258 to 259 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 259 (ObjNum 394 snap 110 seq_num 394) dirty exists 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1054: left oid 30 (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: finishing write tid 1 to smithi20328086-23 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: finishing write tid 2 to smithi20328086-23 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: finishing write tid 3 to smithi20328086-23 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: finishing write tid 4 to smithi20328086-23 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: oid 23 updating version 0 to 376 2024-09-17T15:34:25.713 INFO:tasks.rados.rados.0.smithi203.stdout:1055: oid 23 updating version 376 to 377 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1055: oid 23 updating version 377 to 378 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 378 (ObjNum 395 snap 110 seq_num 395) dirty exists 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1055: left oid 23 (ObjNum 395 snap 110 seq_num 395) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1057: read oid 21 snap 108 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1057: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1052: done (4 left) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1053: done (3 left) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1054: done (2 left) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1055: done (1 left) 2024-09-17T15:34:25.714 INFO:tasks.rados.rados.0.smithi203.stdout:1058: snap_remove snap 95 2024-09-17T15:34:26.719 INFO:tasks.rados.rados.0.smithi203.stdout:1058: done (1 left) 2024-09-17T15:34:26.719 INFO:tasks.rados.rados.0.smithi203.stdout:1059: read oid 49 snap -1 2024-09-17T15:34:26.719 INFO:tasks.rados.rados.0.smithi203.stdout:1059: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:34:26.719 INFO:tasks.rados.rados.0.smithi203.stdout:1057: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:34:26.827 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:1057: done (1 left) 2024-09-17T15:34:26.880 INFO:tasks.rados.rados.0.smithi203.stdout:1060: snap_remove snap 79 2024-09-17T15:34:27.637 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:34:27.637 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:34:27.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:34:27.640 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:34:27.640 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:34:27.641 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:34:27.641 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:34:42.642 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:34:42.643 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:34:42.643 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:34:42.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:42.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: get_monmap_and_config 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: build_initial_monmap 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: monmap: 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:42.773488+0000 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:42.773488+0000 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:42.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:42.773 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:42.773 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:42.773 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:42.773 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:42.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 Processor -- start 2024-09-17T15:34:42.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 -- start start 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: init 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f480689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-a con 0x7f5f480694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f4813aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4809c8b0 con 0x7f5f480689d0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4809cb90 con 0x7f5f4813aff0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5f48074500 con 0x7f5f480694c0 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:42.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f4d72a640 10 monclient(hunting): authenticate will time out at 1415.738962s 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f46ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5f480689d0 0x7f5f48068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40144/0 (socket says 172.21.15.86:40144) 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f46ffd640 1 -- 172.21.15.86:0/2464122587 learned_addr learned my addr 172.21.15.86:0/2464122587 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f480694c0 auth_method 0 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:42.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.770+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:42.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.774+0000 7f5f46ffd640 1 -- 172.21.15.86:0/2464122587 >> v1:172.21.15.86:6789/0 conn(0x7f5f480689d0 legacy=0x7f5f48068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:42.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.774+0000 7f5f477fe640 1 -- 172.21.15.86:0/2464122587 >> v1:172.21.15.203:6789/0 conn(0x7f5f4813aff0 legacy=0x7f5f4813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:42.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f480694c0 auth_method 0 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:42.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:42.974+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f480694c0 auth_method 0 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:43.378+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f480694c0 auth_method 0 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:34:44.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:34:44.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:44.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:44.182+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:34:45.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): tick 2024-09-17T15:34:45.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 msgr2=0x7f5f4806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.86:3300/0 conn(0x7f5f480694c0 0x7f5f4806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 -- 172.21.15.86:0/2464122587 >> v1:172.21.15.203:6789/0 conn(0x7f5f4813aff0 legacy=0x7f5f4813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 -- 172.21.15.86:0/2464122587 >> v1:172.21.15.86:6789/0 conn(0x7f5f480689d0 legacy=0x7f5f48068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient: _add_conns ranks=[4,2] 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.203:3300/0 conn(0x7f5f340037f0 0x7f5f34005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f5f340037f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 --2- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 0x7f5f340087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f5f34006330 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:45.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 -- 172.21.15.86:0/2464122587 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5f34008ed0 con 0x7f5f340037f0 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 1 -- 172.21.15.86:0/2464122587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f34009080 con 0x7f5f34006330 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f44ff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 0x7f5f340087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f34006330 auth_method 0 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.203:3300/0 conn(0x7f5f340037f0 0x7f5f34005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:45.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): get_auth_request con 0x7f5f340037f0 auth_method 0 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient(hunting): handle_auth_done global_id 74165 payload 274 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> v2:172.21.15.203:3300/0 conn(0x7f5f340037f0 msgr2=0x7f5f34005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> v2:172.21.15.203:3300/0 conn(0x7f5f340037f0 0x7f5f34005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient: _finish_hunting 0 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 1 monclient: found mon.noname-b 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:45.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:45.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.774+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4809ce70 con 0x7f5f34006330 2024-09-17T15:34:45.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.778+0000 7f5f467fc640 1 -- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 msgr2=0x7f5f340087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:45.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.778+0000 7f5f467fc640 1 --2- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 0x7f5f340087d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f38000ee0 tx=0x7f5f3801d180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e503ab1913a0fd46 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 1 -- 172.21.15.86:0/2464122587 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f380b3070 con 0x7f5f34006330 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: dump: 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:45.783 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: _finish_auth 0 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 5 monclient: authenticate success, global_id 74165 2024-09-17T15:34:45.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 1 -- 172.21.15.86:0/2464122587 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f380a5e50 con 0x7f5f34006330 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 1 -- 172.21.15.86:0/2464122587 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4809c2d0 con 0x7f5f34006330 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 4 set_mon_vals no callback set 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f45ffb640 10 monclient: dump: 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:45.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:45.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:45.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:45.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:45.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 10 monclient: shutdown 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 -- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 msgr2=0x7f5f340087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/2464122587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34006330 0x7f5f340087d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f38000ee0 tx=0x7f5f3801d180 comp rx=0 tx=0).stop 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f477fe640 1 -- 172.21.15.86:0/2464122587 reap_dead start 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 -- 172.21.15.86:0/2464122587 shutdown_connections 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 -- 172.21.15.86:0/2464122587 >> 172.21.15.86:0/2464122587 conn(0x7f5f480a6440 msgr2=0x7f5f480a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 -- 172.21.15.86:0/2464122587 shutdown_connections 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 1 -- 172.21.15.86:0/2464122587 wait complete. 2024-09-17T15:34:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 10 monclient: build_initial_monmap 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.782+0000 7f5f4d72a640 10 monclient: monmap: 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 Processor -- start 2024-09-17T15:34:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 -- start start 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient: init 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f34006330 0x7f5f481359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-b con 0x7f5f34006330 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-a con 0x7f5f48135f20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): picked mon.noname-c con 0x7f5f4809dcb0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:45.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5f48141bd0 con 0x7f5f48135f20 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f48141a50 con 0x7f5f34006330 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f48141d50 con 0x7f5f4809dcb0 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f4d72a640 10 monclient(hunting): authenticate will time out at 1418.753666s 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59076/0 (socket says 172.21.15.86:59076) 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36922/0 (socket says 172.21.15.86:36922) 2024-09-17T15:34:45.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 -- 172.21.15.86:0/4223102303 learned_addr learned my addr 172.21.15.86:0/4223102303 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 10 monclient(hunting): get_auth_request con 0x7f5f48135f20 auth_method 0 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): get_auth_request con 0x7f5f4809dcb0 auth_method 0 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 msgr2=0x7f5f4809d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 msgr2=0x7f5f4809d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:45.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f467fc640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f46ffd640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f34006330 0x7f5f481359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f46ffd640 10 monclient(hunting): get_auth_request con 0x7f5f34006330 auth_method 0 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient(hunting): handle_auth_done global_id 74168 payload 824 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f34006330 msgr2=0x7f5f481359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:45.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f34006330 0x7f5f481359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 msgr2=0x7f5f4809d770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient: _finish_hunting 0 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 monclient: found mon.noname-c 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.786+0000 7f5f477fe640 1 -- 172.21.15.86:0/4223102303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f48121350 con 0x7f5f4809dcb0 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f477fe640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 msgr2=0x7f5f4809e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f477fe640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f48069970 tx=0x7f5f4000cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40017070 con 0x7f5f4809dcb0 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: dump: 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:45.799 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4000a610 con 0x7f5f4809dcb0 2024-09-17T15:34:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4000aa00 con 0x7f5f4809dcb0 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 10 monclient: dump: 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 5 monclient: authenticate success, global_id 74168 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _renew_subs 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f457fa640 4 set_mon_vals no callback set 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4813b540 con 0x7f5f4809dcb0 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _renew_subs 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f48136a30 con 0x7f5f4809dcb0 2024-09-17T15:34:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5f4001a070 con 0x7f5f4809dcb0 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4806f800 con 0x7f5f4809dcb0 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 0x7f5f3403f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(237..237 src has 1..237) ==== 4847+0+0 (secure 0 0 0) 0x7f5f4004f1e0 con 0x7f5f4809dcb0 2024-09-17T15:34:45.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f46ffd640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 0x7f5f3403f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:45.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.790+0000 7f5f46ffd640 10 monclient: get_auth_request con 0x7f5f3403ce20 auth_method 0 2024-09-17T15:34:45.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.794+0000 7f5f46ffd640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 0x7f5f3403f2e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5f480a0690 tx=0x7f5f3c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:45.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.798+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5f40010300 con 0x7f5f4809dcb0 2024-09-17T15:34:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.798+0000 7f5f2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:45.798+0000 7f5f2f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:46.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:46.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:46.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5f48120fd0 con 0x7f5f4809dcb0 2024-09-17T15:34:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f2f7fe640 1 -- 172.21.15.86:0/4223102303 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f5f4004dba0 con 0x7f5f4809dcb0 2024-09-17T15:34:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:46.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f2f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:46.028 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 msgr2=0x7f5f3403f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 0x7f5f3403f2e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5f480a0690 tx=0x7f5f3c009040 comp rx=0 tx=0).stop 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.026+0000 7f5f4d72a640 10 monclient: shutdown 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 msgr2=0x7f5f4809e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5f48069970 tx=0x7f5f4000cb90 comp rx=0 tx=0).stop 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 shutdown_connections 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403ce20 0x7f5f3403f2e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f4809dcb0 0x7f5f4809e130 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f48135f20 0x7f5f4809d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 --2- 172.21.15.86:0/4223102303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f34006330 0x7f5f481359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 >> 172.21.15.86:0/4223102303 conn(0x7f5f480a6440 msgr2=0x7f5f4813ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 shutdown_connections 2024-09-17T15:34:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.030+0000 7f5f4d72a640 1 -- 172.21.15.86:0/4223102303 wait complete. 2024-09-17T15:34:46.045 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:34:46.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:34:46.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.162+0000 7f393a504640 10 monclient: get_monmap_and_config 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.162+0000 7f393a504640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.162+0000 7f393a504640 10 monclient: monmap: 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:46.166860+0000 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:46.166860+0000 2024-09-17T15:34:46.165 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:46.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 Processor -- start 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 -- start start 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient: init 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 0x7f393409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): picked mon.noname-c con 0x7f39340a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): picked mon.noname-b con 0x7f39340a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3934137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39340a6280 con 0x7f3934137ea0 2024-09-17T15:34:46.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f393409c0a0 con 0x7f39340a50e0 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f393409c380 con 0x7f39340a54c0 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f393a504640 10 monclient(hunting): authenticate will time out at 1419.133093s 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36924/0 (socket says 172.21.15.86:36924) 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f3938aa1640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3934137ea0 0x7f39341382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40154/0 (socket says 172.21.15.86:40154) 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 -- 172.21.15.86:0/3852069305 learned_addr learned my addr 172.21.15.86:0/3852069305 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): get_auth_request con 0x7f39340a54c0 auth_method 0 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient(hunting): handle_auth_done global_id 74171 payload 274 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 -- 172.21.15.86:0/3852069305 >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 msgr2=0x7f393409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f3933fff640 1 --2- 172.21.15.86:0/3852069305 >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 0x7f393409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 --2- 172.21.15.86:0/3852069305 >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 0x7f393409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 -- 172.21.15.86:0/3852069305 >> v1:172.21.15.86:6789/0 conn(0x7f3934137ea0 legacy=0x7f39341382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 monclient: found mon.noname-b 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f39337fe640 1 -- 172.21.15.86:0/3852069305 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393409c660 con 0x7f39340a54c0 2024-09-17T15:34:46.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f3933fff640 10 monclient: get_auth_request con 0x7f39340a50e0 auth_method 0 2024-09-17T15:34:46.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.166+0000 7f3933fff640 1 --2- 172.21.15.86:0/3852069305 >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 0x7f393409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f39337fe640 1 -- 172.21.15.86:0/3852069305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 msgr2=0x7f39340a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f39337fe640 1 --2- 172.21.15.86:0/3852069305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3928009870 tx=0x7f392801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b499c2ac631351ce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 1 -- 172.21.15.86:0/3852069305 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280b0070 con 0x7f39340a54c0 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: dump: 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.176 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 5 monclient: authenticate success, global_id 74171 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 1 -- 172.21.15.86:0/3852069305 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39280a4e50 con 0x7f39340a54c0 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 1 -- 172.21.15.86:0/3852069305 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39280ab820 con 0x7f39340a54c0 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 4 set_mon_vals no callback set 2024-09-17T15:34:46.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f3932ffd640 10 monclient: dump: 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 10 monclient: shutdown 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- 172.21.15.86:0/3852069305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 msgr2=0x7f39340a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 --2- 172.21.15.86:0/3852069305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3928009870 tx=0x7f392801c4b0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- 172.21.15.86:0/3852069305 shutdown_connections 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 --2- 172.21.15.86:0/3852069305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f39340a54c0 0x7f39340a58f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 --2- 172.21.15.86:0/3852069305 >> v2:172.21.15.203:3300/0 conn(0x7f39340a50e0 0x7f393409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- 172.21.15.86:0/3852069305 >> 172.21.15.86:0/3852069305 conn(0x7f39340aaae0 msgr2=0x7f39340aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- 172.21.15.86:0/3852069305 shutdown_connections 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- 172.21.15.86:0/3852069305 wait complete. 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 10 monclient: monmap: 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 Processor -- start 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 1 -- start start 2024-09-17T15:34:46.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.170+0000 7f393a504640 10 monclient: init 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): picked mon.noname-a con 0x7f39340a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39340a54c0 0x7f393413c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): picked mon.noname-b con 0x7f39340a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 0x7f393413c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): picked mon.noname-c con 0x7f3934137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3934142cb0 con 0x7f39340a50e0 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3934142b30 con 0x7f39340a54c0 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3934142e30 con 0x7f3934137ea0 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): authenticate will time out at 1419.140213s 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 0x7f393413c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59088/0 (socket says 172.21.15.86:59088) 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 learned_addr learned my addr 172.21.15.86:0/2791087576 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 0x7f393413c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36928/0 (socket says 172.21.15.86:36928) 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 10 monclient(hunting): get_auth_request con 0x7f39340a50e0 auth_method 0 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): get_auth_request con 0x7f3934137ea0 auth_method 0 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 msgr2=0x7f3934136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 msgr2=0x7f3934136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f39337fe640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39340a54c0 0x7f393413c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient(hunting): handle_auth_done global_id 74174 payload 824 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39340a54c0 msgr2=0x7f393413c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f39340a54c0 0x7f393413c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 msgr2=0x7f3934136d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39340a50e0 0x7f3934136d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 monclient: found mon.noname-c 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3934121590 con 0x7f3934137ea0 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 msgr2=0x7f393413c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3938aa1640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 0x7f393413c740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f392c00c780 tx=0x7f392c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c017070 con 0x7f3934137ea0 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: dump: 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 5 monclient: authenticate success, global_id 74174 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _renew_subs 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f392c00a840 con 0x7f3934137ea0 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3934121870 con 0x7f3934137ea0 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392c00ac30 con 0x7f3934137ea0 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f39327fc640 4 set_mon_vals no callback set 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f3930ff9640 10 monclient: dump: 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _renew_subs 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39340a45d0 con 0x7f3934137ea0 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.174+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f393406a0e0 con 0x7f39341372b0 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 10 monclient: get_auth_request con 0x7f39341372b0 auth_method 0 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f392c01a070 con 0x7f3934137ea0 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 msgr2=0x7f393413cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 msgr2=0x7f393413cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:46.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3930ff9640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f392003bbe0 0x7f392003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(237..237 src has 1..237) ==== 4847+0+0 (secure 0 0 0) 0x7f392c04f5a0 con 0x7f3934137ea0 2024-09-17T15:34:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f39337fe640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f392003bbe0 0x7f392003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f39337fe640 10 monclient: get_auth_request con 0x7f392003bbe0 auth_method 0 2024-09-17T15:34:46.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.178+0000 7f39337fe640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f392003bbe0 0x7f392003e0a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f393406f8c0 tx=0x7f3928008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient: get_auth_request con 0x7f39341372b0 auth_method 0 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:34:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 10 monclient(hunting): handle_auth_done global_id 84099 payload 824 2024-09-17T15:34:46.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.382+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f393409cd60 con 0x7f39341372b0 2024-09-17T15:34:46.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.386+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 msgr2=0x7f393413cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.386+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f3924010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392401a070 con 0x7f39341372b0 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 10 monclient: dump: 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.391 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f392400ebe0 con 0x7f39341372b0 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:46.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 msgr2=0x7f393413cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f3924010d20 comp rx=0 tx=0).stop 2024-09-17T15:34:46.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 msgr2=0x7f393413cc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.390+0000 7f3930ff9640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39341372b0 0x7f393413cc80 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f3924010d20 comp rx=0 tx=0).stop 2024-09-17T15:34:46.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f393a504640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:46.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f393a504640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 0x7f39340a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f393a504640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f393413d5d0 con 0x7f393413d1c0 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 0x7f39340a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient: get_auth_request con 0x7f393413d1c0 auth_method 0 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 10 monclient(hunting): handle_auth_done global_id 84102 payload 824 2024-09-17T15:34:46.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3934074be0 con 0x7f393413d1c0 2024-09-17T15:34:46.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 msgr2=0x7f39340a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3933fff640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 0x7f39340a85c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f39240046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f392401fa70 con 0x7f393413d1c0 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 10 monclient: dump: 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.402 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f3924016e50 con 0x7f393413d1c0 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 msgr2=0x7f39340a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 0x7f39340a85c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f39240046d0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 msgr2=0x7f39340a85c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.398+0000 7f3930ff9640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393413d1c0 0x7f39340a85c0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3924010860 tx=0x7f39240046d0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.403 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18675, 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.405 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.406 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:46.407 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.408 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.409 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.411 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.413 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.414 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.415 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.416 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:46.417 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f392003bbe0 msgr2=0x7f392003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f392003bbe0 0x7f392003e0a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f393406f8c0 tx=0x7f3928008040 comp rx=0 tx=0).stop 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 10 monclient: shutdown 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 msgr2=0x7f393413c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 --2- 172.21.15.86:0/2791087576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3934137ea0 0x7f393413c740 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f392c00c780 tx=0x7f392c00cc40 comp rx=0 tx=0).stop 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 reap_dead start 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f3938aa1640 1 -- 172.21.15.86:0/2791087576 reap_dead start 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 shutdown_connections 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 >> 172.21.15.86:0/2791087576 conn(0x7f39340aaae0 msgr2=0x7f39340a3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 shutdown_connections 2024-09-17T15:34:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.402+0000 7f393a504640 1 -- 172.21.15.86:0/2791087576 wait complete. 2024-09-17T15:34:46.421 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:46.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:34:46.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:34:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: get_monmap_and_config 2024-09-17T15:34:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: monmap: 2024-09-17T15:34:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:46.540244+0000 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:46.540244+0000 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:46.539 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 Processor -- start 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 -- start start 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: init 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-a con 0x7fccd40a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fccd40a54c0 0x7fccd40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-c con 0x7fccd40a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-b con 0x7fccd4137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fccd40a6280 con 0x7fccd40a50e0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fccd409c0a0 con 0x7fccd40a54c0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fccd409c380 con 0x7fccd4137ea0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccdc25c640 10 monclient(hunting): authenticate will time out at 1419.505680s 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36932/0 (socket says 172.21.15.86:36932) 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59132/0 (socket says 172.21.15.86:59132) 2024-09-17T15:34:46.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 -- 172.21.15.86:0/3032765249 learned_addr learned my addr 172.21.15.86:0/3032765249 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): get_auth_request con 0x7fccd4137ea0 auth_method 0 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): get_auth_request con 0x7fccd40a50e0 auth_method 0 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd97f7640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.203:3300/0 conn(0x7fccd40a54c0 0x7fccd40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd97f7640 10 monclient(hunting): get_auth_request con 0x7fccd40a54c0 auth_method 0 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd97f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd97f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd97f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient(hunting): handle_auth_done global_id 74177 payload 274 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 -- 172.21.15.86:0/3032765249 >> v2:172.21.15.203:3300/0 conn(0x7fccd40a54c0 msgr2=0x7fccd40a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.203:3300/0 conn(0x7fccd40a54c0 0x7fccd40a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 -- 172.21.15.86:0/3032765249 >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 msgr2=0x7fccd409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 monclient: found mon.noname-b 2024-09-17T15:34:46.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccda7f9640 1 -- 172.21.15.86:0/3032765249 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd409c660 con 0x7fccd4137ea0 2024-09-17T15:34:46.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.538+0000 7fccd9ff8640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:46.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccda7f9640 1 -- 172.21.15.86:0/3032765249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 msgr2=0x7fccd4138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccda7f9640 1 --2- 172.21.15.86:0/3032765249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccd406cc20 tx=0x7fcccc01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d51b8aecb214f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 1 -- 172.21.15.86:0/3032765249 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0a5750 con 0x7fccd4137ea0 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: dump: 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 1 -- 172.21.15.86:0/3032765249 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcccc0a58f0 con 0x7fccd4137ea0 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 1 -- 172.21.15.86:0/3032765249 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc0a5cc0 con 0x7fccd4137ea0 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 5 monclient: authenticate success, global_id 74177 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 4 set_mon_vals no callback set 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccd8ff6640 10 monclient: dump: 2024-09-17T15:34:46.554 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 10 monclient: shutdown 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 -- 172.21.15.86:0/3032765249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 msgr2=0x7fccd4138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 --2- 172.21.15.86:0/3032765249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccd406cc20 tx=0x7fcccc01c8b0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 -- 172.21.15.86:0/3032765249 shutdown_connections 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 --2- 172.21.15.86:0/3032765249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137ea0 0x7fccd4138350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.203:3300/0 conn(0x7fccd40a54c0 0x7fccd40a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 --2- 172.21.15.86:0/3032765249 >> v2:172.21.15.86:3300/0 conn(0x7fccd40a50e0 0x7fccd409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 -- 172.21.15.86:0/3032765249 >> 172.21.15.86:0/3032765249 conn(0x7fccd40aaae0 msgr2=0x7fccd40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 -- 172.21.15.86:0/3032765249 shutdown_connections 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 1 -- 172.21.15.86:0/3032765249 wait complete. 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.542+0000 7fccdc25c640 10 monclient: monmap: 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:46.556 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 Processor -- start 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- start start 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: init 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 0x7fccd41cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-c con 0x7fccd40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccd40a54c0 0x7fccd4136f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-b con 0x7fccd40a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 0x7fccd413c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): picked mon.noname-a con 0x7fccd4137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fccd40a6ae0 con 0x7fccd4137ea0 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fccd40a6960 con 0x7fccd40a54c0 2024-09-17T15:34:46.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fccd4141b20 con 0x7fccd40a50e0 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient(hunting): authenticate will time out at 1419.512720s 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 0x7fccd41cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 0x7fccd413c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 0x7fccd41cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36934/0 (socket says 172.21.15.86:36934) 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 0x7fccd413c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59138/0 (socket says 172.21.15.86:59138) 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 -- 172.21.15.86:0/97240872 learned_addr learned my addr 172.21.15.86:0/97240872 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): get_auth_request con 0x7fccd40a50e0 auth_method 0 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): get_auth_request con 0x7fccd4137ea0 auth_method 0 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd97f7640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccd40a54c0 0x7fccd4136f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient(hunting): handle_auth_done global_id 74180 payload 824 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccd40a54c0 msgr2=0x7fccd4136f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fccd40a54c0 0x7fccd4136f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 msgr2=0x7fccd413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 0x7fccd413c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 monclient: found mon.noname-c 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd40a18d0 con 0x7fccd40a50e0 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccda7f9640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccd4137ea0 0x7fccd413c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 msgr2=0x7fccd41cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccd9ff8640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 0x7fccd41cfcb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd000e9b0 tx=0x7fccd000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd0017070 con 0x7fccd40a50e0 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: dump: 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccd000cc10 con 0x7fccd40a50e0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 5 monclient: authenticate success, global_id 74180 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _renew_subs 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd413db50 con 0x7fccd40a50e0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccd0013470 con 0x7fccd40a50e0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccca7fc640 10 monclient: dump: 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fcccbfff640 4 set_mon_vals no callback set 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _renew_subs 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.546+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd41210c0 con 0x7fccd40a50e0 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccdc25c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccdc25c640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 0x7fccd413c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fccd406a0e0 con 0x7fccd4137460 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 0x7fccd413c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient: get_auth_request con 0x7fccd4137460 auth_method 0 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 10 monclient(hunting): handle_auth_done global_id 74183 payload 824 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fccd001a070 con 0x7fccd40a50e0 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccd41d0c30 con 0x7fccd4137460 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fccb803bb90 0x7fccb803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(237..237 src has 1..237) ==== 4847+0+0 (secure 0 0 0) 0x7fccd004feb0 con 0x7fccd40a50e0 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccda7f9640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fccb803bb90 0x7fccb803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccda7f9640 10 monclient: get_auth_request con 0x7fccb803bb90 auth_method 0 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 msgr2=0x7fccd413c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccd97f7640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 0x7fccd413c6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc400e9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017070 con 0x7fccd4137460 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 10 monclient: dump: 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.569 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fccc4013900 con 0x7fccd4137460 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 msgr2=0x7fccd413c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 0x7fccd413c6d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc400e9d0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 msgr2=0x7fccd413c6d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.550+0000 7fccca7fc640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd4137460 0x7fccd413c6d0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc400e9d0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.554+0000 7fccda7f9640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fccb803bb90 0x7fccb803e050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fccd40aa620 tx=0x7fcccc0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccdc25c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccdc25c640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 0x7fccd40a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccdc25c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fccd413d020 con 0x7fccd413cc10 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 0x7fccd40a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient: get_auth_request con 0x7fccd413cc10 auth_method 0 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 10 monclient(hunting): handle_auth_done global_id 74186 payload 824 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.558+0000 7fccd97f7640 1 -- 172.21.15.86:0/97240872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fccd4122490 con 0x7fccd413cc10 2024-09-17T15:34:46.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccd97f7640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 msgr2=0x7fccd40a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccd97f7640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 0x7fccd40a2af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc4004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc400f040 con 0x7fccd413cc10 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 10 monclient: dump: 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fccc4012370 con 0x7fccd413cc10 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 msgr2=0x7fccd40a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 0x7fccd40a2af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc4004740 comp rx=0 tx=0).stop 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 msgr2=0x7fccd40a2af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccca7fc640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd413cc10 0x7fccd40a2af0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fccc400e510 tx=0x7fccc4004740 comp rx=0 tx=0).stop 2024-09-17T15:34:46.573 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 67351, 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.574 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.575 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.577 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:46.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.585 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:46.586 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fccb803bb90 msgr2=0x7fccb803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fccb803bb90 0x7fccb803e050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fccd40aa620 tx=0x7fcccc0a7070 comp rx=0 tx=0).stop 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 10 monclient: shutdown 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 msgr2=0x7fccd41cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 --2- 172.21.15.86:0/97240872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccd40a50e0 0x7fccd41cfcb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fccd000e9b0 tx=0x7fccd000ee70 comp rx=0 tx=0).stop 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccda7f9640 1 -- 172.21.15.86:0/97240872 reap_dead start 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccda7f9640 1 -- 172.21.15.86:0/97240872 reap_dead start 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 shutdown_connections 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 >> 172.21.15.86:0/97240872 conn(0x7fccd40aaae0 msgr2=0x7fccd40a1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 shutdown_connections 2024-09-17T15:34:46.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.562+0000 7fccdc25c640 1 -- 172.21.15.86:0/97240872 wait complete. 2024-09-17T15:34:46.589 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:46.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:34:46.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:34:46.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 10 monclient: get_monmap_and_config 2024-09-17T15:34:46.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 10 monclient: monmap: 2024-09-17T15:34:46.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:46.695154+0000 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:46.695154+0000 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:46.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 1 Processor -- start 2024-09-17T15:34:46.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 1 -- start start 2024-09-17T15:34:46.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.690+0000 7fd90b22b640 10 monclient: init 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-b con 0x7fd90409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd90409e090 0x7fd904134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-c con 0x7fd90409e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 0x7fd904135360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-a con 0x7fd904134f80 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd90409e7f0 con 0x7fd904134f80 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd9041358a0 con 0x7fd90409e090 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd9040a2110 con 0x7fd90409d770 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd90b22b640 10 monclient(hunting): authenticate will time out at 1419.660526s 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 0x7fd904135360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36974/0 (socket says 172.21.15.86:36974) 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 -- 172.21.15.86:0/3823979999 learned_addr learned my addr 172.21.15.86:0/3823979999 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): get_auth_request con 0x7fd90409d770 auth_method 0 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): get_auth_request con 0x7fd904134f80 auth_method 0 2024-09-17T15:34:46.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient(hunting): handle_auth_done global_id 74189 payload 274 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 -- 172.21.15.86:0/3823979999 >> v2:172.21.15.203:3300/0 conn(0x7fd90409e090 msgr2=0x7fd904134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3823979999 >> v2:172.21.15.203:3300/0 conn(0x7fd90409e090 0x7fd904134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 -- 172.21.15.86:0/3823979999 >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 msgr2=0x7fd904135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3823979999 >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 0x7fd904135360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 monclient: found mon.noname-b 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd908fc7640 1 -- 172.21.15.86:0/3823979999 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd90409ff00 con 0x7fd90409d770 2024-09-17T15:34:46.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.694+0000 7fd9097c8640 1 --2- 172.21.15.86:0/3823979999 >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 0x7fd904135360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:46.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 -- 172.21.15.86:0/3823979999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 msgr2=0x7fd90409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3823979999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd90406e2e0 tx=0x7fd8f801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a6ba23e975b56f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 1 -- 172.21.15.86:0/3823979999 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80b0070 con 0x7fd90409d770 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: dump: 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 1 -- 172.21.15.86:0/3823979999 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f80ab600 con 0x7fd90409d770 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 5 monclient: authenticate success, global_id 74189 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 4 set_mon_vals no callback set 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 1 -- 172.21.15.86:0/3823979999 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80ab9d0 con 0x7fd90409d770 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9037fe640 10 monclient: dump: 2024-09-17T15:34:46.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: shutdown 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- 172.21.15.86:0/3823979999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 msgr2=0x7fd90409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3823979999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd90406e2e0 tx=0x7fd8f801c3e0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- 172.21.15.86:0/3823979999 shutdown_connections 2024-09-17T15:34:46.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3823979999 >> v2:172.21.15.86:3300/0 conn(0x7fd904134f80 0x7fd904135360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3823979999 >> v2:172.21.15.203:3300/0 conn(0x7fd90409e090 0x7fd904134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3823979999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd90409db50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- 172.21.15.86:0/3823979999 >> 172.21.15.86:0/3823979999 conn(0x7fd9040a2f40 msgr2=0x7fd9040a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- 172.21.15.86:0/3823979999 shutdown_connections 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- 172.21.15.86:0/3823979999 wait complete. 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: monmap: 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 Processor -- start 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- start start 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: init 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd904134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-c con 0x7fd90409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 0x7fd90413aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-a con 0x7fd90409e090 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd904134f80 0x7fd90413b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): picked mon.noname-b con 0x7fd904134f80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd90409ce60 con 0x7fd90409e090 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd90409cce0 con 0x7fd904134f80 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd9040a1480 con 0x7fd90409d770 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd90b22b640 10 monclient(hunting): authenticate will time out at 1419.667491s 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd904134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 0x7fd90413aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 0x7fd90413aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59154/0 (socket says 172.21.15.86:59154) 2024-09-17T15:34:46.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): get_auth_request con 0x7fd90409d770 auth_method 0 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 1 -- 172.21.15.86:0/3421827882 learned_addr learned my addr 172.21.15.86:0/3421827882 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): get_auth_request con 0x7fd90409e090 auth_method 0 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9097c8640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd904134f80 0x7fd90413b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9097c8640 10 monclient(hunting): get_auth_request con 0x7fd904134f80 auth_method 0 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9097c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9097c8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd9097c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient(hunting): handle_auth_done global_id 74192 payload 824 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd904134f80 msgr2=0x7fd90413b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd904134f80 0x7fd90413b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 msgr2=0x7fd90413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 0x7fd90413aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 monclient: found mon.noname-c 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd908fc7640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9041c5300 con 0x7fd90409d770 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.698+0000 7fd903fff640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd90409e090 0x7fd90413aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd908fc7640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 msgr2=0x7fd904134110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd908fc7640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd904134110 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80ad000 tx=0x7fd8f8019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80b0070 con 0x7fd90409d770 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: dump: 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f80b1040 con 0x7fd90409d770 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80aa9e0 con 0x7fd90409d770 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 5 monclient: authenticate success, global_id 74192 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 monclient: _renew_subs 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd904121590 con 0x7fd90409d770 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 10 monclient: dump: 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.718 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd902ffd640 4 set_mon_vals no callback set 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 monclient: _renew_subs 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd90409c500 con 0x7fd90409d770 2024-09-17T15:34:46.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 0x7fd90402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd90409d460 con 0x7fd90413c0e0 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd8f80aab80 con 0x7fd90409d770 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 0x7fd90402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient: get_auth_request con 0x7fd90413c0e0 auth_method 0 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd8f003bbe0 0x7fd8f003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(237..237 src has 1..237) ==== 4847+0+0 (secure 0 0 0) 0x7fd8f80e9d50 con 0x7fd90409d770 2024-09-17T15:34:46.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9097c8640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd8f003bbe0 0x7fd8f003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.702+0000 7fd9097c8640 10 monclient: get_auth_request con 0x7fd8f003bbe0 auth_method 0 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd903fff640 10 monclient(hunting): handle_auth_done global_id 54307 payload 824 2024-09-17T15:34:46.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd903fff640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9040a1850 con 0x7fd90413c0e0 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd903fff640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 msgr2=0x7fd90402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd903fff640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 0x7fd90402d450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400eac0 tx=0x7fd8f400ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9097c8640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd8f003bbe0 0x7fd8f003e0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009bd0 tx=0x7fd8fc00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4017070 con 0x7fd90413c0e0 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 10 monclient: dump: 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.724 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd8f400ce80 con 0x7fd90413c0e0 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 msgr2=0x7fd90402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 0x7fd90402d450 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400eac0 tx=0x7fd8f400ef80 comp rx=0 tx=0).stop 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 msgr2=0x7fd90402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.706+0000 7fd9017fa640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90413c0e0 0x7fd90402d450 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400eac0 tx=0x7fd8f400ef80 comp rx=0 tx=0).stop 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd90b22b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 0x7fd90402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd90b22b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9041375f0 con 0x7fd90402d990 2024-09-17T15:34:46.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 0x7fd90402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient: get_auth_request con 0x7fd90402d990 auth_method 0 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 10 monclient(hunting): handle_auth_done global_id 54310 payload 824 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.714+0000 7fd903fff640 1 -- 172.21.15.86:0/3421827882 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd90406d4f0 con 0x7fd90402d990 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd903fff640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 msgr2=0x7fd90402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd903fff640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 0x7fd90402dda0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4019000 tx=0x7fd8f40047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4017070 con 0x7fd90402d990 2024-09-17T15:34:46.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 10 monclient: dump: 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fd8f4006890 con 0x7fd90402d990 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:46.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 msgr2=0x7fd90402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 0x7fd90402dda0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4019000 tx=0x7fd8f40047e0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 msgr2=0x7fd90402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9017fa640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd90402d990 0x7fd90402dda0 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4019000 tx=0x7fd8f40047e0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:46.728 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 162149, 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.729 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.730 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:46.731 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:46.732 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:46.733 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.734 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:46.735 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.736 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:46.737 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:46.738 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:46.739 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:46.740 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:34:46.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:46.742 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:46.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd8f003bbe0 msgr2=0x7fd8f003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd8f003bbe0 0x7fd8f003e0a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009bd0 tx=0x7fd8fc00b040 comp rx=0 tx=0).stop 2024-09-17T15:34:46.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 10 monclient: shutdown 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 msgr2=0x7fd904134110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 --2- 172.21.15.86:0/3421827882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd90409d770 0x7fd904134110 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80ad000 tx=0x7fd8f8019180 comp rx=0 tx=0).stop 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9097c8640 1 -- 172.21.15.86:0/3421827882 reap_dead start 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd9097c8640 1 -- 172.21.15.86:0/3421827882 reap_dead start 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 shutdown_connections 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 >> 172.21.15.86:0/3421827882 conn(0x7fd9040a2f40 msgr2=0x7fd904072e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 shutdown_connections 2024-09-17T15:34:46.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.718+0000 7fd90b22b640 1 -- 172.21.15.86:0/3421827882 wait complete. 2024-09-17T15:34:46.744 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:46.745 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:34:46.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:34:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: get_monmap_and_config 2024-09-17T15:34:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: monmap: 2024-09-17T15:34:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:46.853312+0000 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:46.853312+0000 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:46.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 Processor -- start 2024-09-17T15:34:46.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 -- start start 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: init 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83a8135ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83a809e500 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83a809e910 0x7f83a813e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-a con 0x7f83a809e910 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a8136990 con 0x7f83a8135ea0 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a809c540 con 0x7f83a809e500 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f83a80a4bb0 con 0x7f83a809e910 2024-09-17T15:34:46.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83b004a640 10 monclient(hunting): authenticate will time out at 1419.818603s 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83adde6640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f83a8135ea0 0x7f83a8136280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40164/0 (socket says 172.21.15.86:40164) 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83adde6640 1 -- 172.21.15.86:0/3201232166 learned_addr learned my addr 172.21.15.86:0/3201232166 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 1 --2- 172.21.15.86:0/3201232166 >> v2:172.21.15.86:3300/0 conn(0x7f83a809e910 0x7f83a813e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): get_auth_request con 0x7f83a809e910 auth_method 0 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_done global_id 84117 payload 274 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 1 -- 172.21.15.86:0/3201232166 >> v1:172.21.15.203:6789/0 conn(0x7f83a809e500 legacy=0x7f83a813ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 1 -- 172.21.15.86:0/3201232166 >> v1:172.21.15.86:6789/0 conn(0x7f83a8135ea0 legacy=0x7f83a8136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 1 monclient: found mon.noname-a 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:46.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.850+0000 7f83ae5e7640 1 -- 172.21.15.86:0/3201232166 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a809ecf0 con 0x7f83a809e910 2024-09-17T15:34:46.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83ae5e7640 1 -- 172.21.15.86:0/3201232166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 msgr2=0x7f83a813e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83ae5e7640 1 --2- 172.21.15.86:0/3201232166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a813e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009d90 tx=0x7f83a001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69644583468fd537 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 1 -- 172.21.15.86:0/3201232166 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00b2070 con 0x7f83a809e910 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: dump: 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.860 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 1 -- 172.21.15.86:0/3201232166 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a00a5840 con 0x7f83a809e910 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 1 -- 172.21.15.86:0/3201232166 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00a5c10 con 0x7f83a809e910 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83acde4640 10 monclient: dump: 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 5 monclient: authenticate success, global_id 84117 2024-09-17T15:34:46.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:46.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 4 set_mon_vals no callback set 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1060: done (1 left) 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1061: read oid 3 snap -1 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1061: expect deleted 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1062: read oid 31 snap 104 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1062: expect (ObjNum 320 snap 93 seq_num 320) 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1063: write (excl) oid 43 current snap is 110 2024-09-17T15:34:46.864 INFO:tasks.rados.rados.0.smithi203.stdout:1063: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-17T15:34:46.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 10 monclient: shutdown 2024-09-17T15:34:46.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 1 -- 172.21.15.86:0/3201232166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 msgr2=0x7f83a813e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.858+0000 7f83b004a640 1 --2- 172.21.15.86:0/3201232166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a813e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009d90 tx=0x7f83a001c830 comp rx=0 tx=0).stop 2024-09-17T15:34:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 -- 172.21.15.86:0/3201232166 shutdown_connections 2024-09-17T15:34:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 --2- 172.21.15.86:0/3201232166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a813e4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 -- 172.21.15.86:0/3201232166 >> 172.21.15.86:0/3201232166 conn(0x7f83a80a8390 msgr2=0x7f83a80a8770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:46.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 -- 172.21.15.86:0/3201232166 shutdown_connections 2024-09-17T15:34:46.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 -- 172.21.15.86:0/3201232166 wait complete. 2024-09-17T15:34:46.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 10 monclient: build_initial_monmap 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 10 monclient: monmap: 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 Processor -- start 2024-09-17T15:34:46.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 1 -- start start 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.862+0000 7f83b004a640 10 monclient: init 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a809e500 0x7f83a8073580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-b con 0x7f83a809e500 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a8073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-a con 0x7f83a809e910 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): picked mon.noname-c con 0x7f83a8135ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83a806e100 con 0x7f83a809e910 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83a806df80 con 0x7f83a809e500 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83a806e280 con 0x7f83a8135ea0 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient(hunting): authenticate will time out at 1419.832363s 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36986/0 (socket says 172.21.15.86:36986) 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 -- 172.21.15.86:0/4176250376 learned_addr learned my addr 172.21.15.86:0/4176250376 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): get_auth_request con 0x7f83a8135ea0 auth_method 0 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a8073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): get_auth_request con 0x7f83a809e910 auth_method 0 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient(hunting): handle_auth_done global_id 74195 payload 824 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 -- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a809e500 msgr2=0x7f83a8073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a809e500 0x7f83a8073580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 -- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 msgr2=0x7f83a8073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a8073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient: _finish_hunting 0 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 monclient: found mon.noname-c 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 -- 172.21.15.86:0/4176250376 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a81317e0 con 0x7f83a8135ea0 2024-09-17T15:34:46.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ad5e5640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a8073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 -- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 msgr2=0x7f83a8074000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83ae5e7640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f83a001c660 tx=0x7f83a00195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00bd020 con 0x7f83a8135ea0 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: dump: 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83a00b3040 con 0x7f83a8135ea0 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 5 monclient: authenticate success, global_id 74195 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _renew_subs 2024-09-17T15:34:46.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a8132080 con 0x7f83a8135ea0 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00b2070 con 0x7f83a8135ea0 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a67fc640 10 monclient: dump: 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:46.876 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _renew_subs 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a8131da0 con 0x7f83a8135ea0 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.866+0000 7f83a7fff640 4 set_mon_vals no callback set 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83b004a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83b004a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83b004a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a8136280 con 0x7f83a8135ea0 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f83a00c0020 con 0x7f83a8135ea0 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83a67fc640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 0x7f839403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:46.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(238..238 src has 1..238) ==== 4863+0+0 (secure 0 0 0) 0x7f83a00eaf40 con 0x7f83a8135ea0 2024-09-17T15:34:46.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83adde6640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 0x7f839403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:46.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.870+0000 7f83adde6640 10 monclient: get_auth_request con 0x7f839403bb60 auth_method 0 2024-09-17T15:34:46.878 INFO:tasks.rados.rados.0.smithi203.stdout:1063: writing smithi20328086-43 from 655572 to 1304532 tid 1 2024-09-17T15:34:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.874+0000 7f83adde6640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 0x7f839403e020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8398004fa0 tx=0x7f8398007ab0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.874+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f83a00c7030 con 0x7f83a8135ea0 2024-09-17T15:34:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.874+0000 7f83a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:46.874+0000 7f83a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:46.891 INFO:tasks.rados.rados.0.smithi203.stdout:1063: writing smithi20328086-43 from 1922987 to 2655572 tid 2 2024-09-17T15:34:46.891 INFO:tasks.rados.rados.0.smithi203.stdout:1064: rollback oid 38 current snap is 110 2024-09-17T15:34:46.891 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 108 2024-09-17T15:34:46.891 INFO:tasks.rados.rados.0.smithi203.stdout:1059: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:34:46.891 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:46.997 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 424584 2024-09-17T15:34:47.105 INFO:tasks.rados.rados.0.smithi203.stdout:1064: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:34:47.105 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 219 (ObjNum 33 snap 0 seq_num 3605851904) dirty dne 2024-09-17T15:34:47.105 INFO:tasks.rados.rados.0.smithi203.stdout:1063: finishing write tid 1 to smithi20328086-43 2024-09-17T15:34:47.105 INFO:tasks.rados.rados.0.smithi203.stdout:1062: expect (ObjNum 320 snap 93 seq_num 320) 2024-09-17T15:34:47.105 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:47.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83b004a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:34:47.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83b004a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:34:47.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83b004a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:47.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f83a8130b50 con 0x7f83a8135ea0 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1325019408,config=3541143443,health=4058548626,kv=3088674189,logm=369516990}) 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=11,logm=61} crc {auth=1328274543,config=3541143443,health=1367206252,kv=3591778543,logm=2105331533}) 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=12,kv=9,logm=63} crc {auth=1325019408,config=3541143443,health=4058548626,kv=3088674189,logm=369516990}) 2024-09-17T15:34:47.109 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=11,logm=62} crc {auth=1325019408,config=3541143443,health=1367206252,kv=3591778544,logm=3014605086}) 2024-09-17T15:34:47.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83a67fc640 1 -- 172.21.15.86:0/4176250376 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f83a0016540 con 0x7f83a8135ea0 2024-09-17T15:34:47.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:34:47.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.106+0000 7f83a67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:47.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.112 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412709794}) 2024-09-17T15:34:47.112 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679137487}) 2024-09-17T15:34:47.112 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.112 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412709794}) 2024-09-17T15:34:47.112 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926399910}) 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 msgr2=0x7f839403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 0x7f839403e020 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8398004fa0 tx=0x7f8398007ab0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 10 monclient: shutdown 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 msgr2=0x7f83a8074000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f83a001c660 tx=0x7f83a00195d0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 shutdown_connections 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f839403bb60 0x7f839403e020 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a8135ea0 0x7f83a8074000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a809e910 0x7f83a8073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 --2- 172.21.15.86:0/4176250376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a809e500 0x7f83a8073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 >> 172.21.15.86:0/4176250376 conn(0x7f83a80a8390 msgr2=0x7f83a806e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 shutdown_connections 2024-09-17T15:34:47.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f83b004a640 1 -- 172.21.15.86:0/4176250376 wait complete. 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109918862}) 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135349153}) 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109918862}) 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.110+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026924515}) 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984310745}) 2024-09-17T15:34:47.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=387946128}) 2024-09-17T15:34:47.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984310745}) 2024-09-17T15:34:47.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435646153}) 2024-09-17T15:34:47.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294833897}) 2024-09-17T15:34:47.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344462577}) 2024-09-17T15:34:47.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294833897}) 2024-09-17T15:34:47.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915443989}) 2024-09-17T15:34:47.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706465496}) 2024-09-17T15:34:47.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000491950}) 2024-09-17T15:34:47.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706465496}) 2024-09-17T15:34:47.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.114+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312100515}) 2024-09-17T15:34:47.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965740350}) 2024-09-17T15:34:47.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=624761488}) 2024-09-17T15:34:47.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965740350}) 2024-09-17T15:34:47.120 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963387586}) 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562559021}) 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2180988781}) 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562559021}) 2024-09-17T15:34:47.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.118+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671013288}) 2024-09-17T15:34:47.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195664304}) 2024-09-17T15:34:47.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512543281}) 2024-09-17T15:34:47.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3195664304}) 2024-09-17T15:34:47.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1175877870}) 2024-09-17T15:34:47.124 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.124 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885826110}) 2024-09-17T15:34:47.124 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2420913095}) 2024-09-17T15:34:47.124 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885826110}) 2024-09-17T15:34:47.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2817531756}) 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390991336}) 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637104455}) 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390991336}) 2024-09-17T15:34:47.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.122+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468929942}) 2024-09-17T15:34:47.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915143251}) 2024-09-17T15:34:47.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268139720}) 2024-09-17T15:34:47.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915143251}) 2024-09-17T15:34:47.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80114775}) 2024-09-17T15:34:47.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:34:47.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123862742}) 2024-09-17T15:34:47.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2134586515}) 2024-09-17T15:34:47.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123862742}) 2024-09-17T15:34:47.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.126+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354329668}) 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4154299311}) 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=821430608}) 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4154299311}) 2024-09-17T15:34:47.131 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149397398}) 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057176044}) 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668132172}) 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057176044}) 2024-09-17T15:34:47.133 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.130+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076580483}) 2024-09-17T15:34:47.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975649417}) 2024-09-17T15:34:47.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162781482}) 2024-09-17T15:34:47.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975649417}) 2024-09-17T15:34:47.135 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271331640}) 2024-09-17T15:34:47.135 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340129665}) 2024-09-17T15:34:47.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682758564}) 2024-09-17T15:34:47.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340129665}) 2024-09-17T15:34:47.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213834381}) 2024-09-17T15:34:47.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914885517}) 2024-09-17T15:34:47.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471139707}) 2024-09-17T15:34:47.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914885517}) 2024-09-17T15:34:47.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.134+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559039048}) 2024-09-17T15:34:47.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=13,mon_config_key=10} crc {logm=4184165555,mdsmap=2815901553,mgr=4190274005,mgr_command_descs=1770323759,mgrstat=1596076116,mon_config_key=3416188778}) 2024-09-17T15:34:47.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11} crc {logm=4015622471,mdsmap=1228700967,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1754930408,mon_config_key=2746304775}) 2024-09-17T15:34:47.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=66,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=13,mon_config_key=10} crc {logm=4184165555,mdsmap=2815901553,mgr=4190274005,mgr_command_descs=1770323759,mgrstat=1596076116,mon_config_key=3416188778}) 2024-09-17T15:34:47.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10} crc {logm=1647955819,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2675995821,mon_config_key=3416188778}) 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=49,osdmap=34} crc {mon_config_key=4067755550,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=1119462942,osd_snap=3200971927,osdmap=403475565}) 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=51,osdmap=32} crc {mon_config_key=4067755550,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=1119462942,osd_snap=3509314778,osdmap=2263050094}) 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=49,osdmap=34} crc {mon_config_key=4067755550,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=1119462942,osd_snap=3200971927,osdmap=403475565}) 2024-09-17T15:34:47.141 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.138+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=51,osdmap=32} crc {mon_config_key=4067755550,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=1119462942,osd_snap=3509314778,osdmap=3952049816}) 2024-09-17T15:34:47.143 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.143 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=242298461}) 2024-09-17T15:34:47.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2592176939}) 2024-09-17T15:34:47.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=242298461}) 2024-09-17T15:34:47.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1528846009}) 2024-09-17T15:34:47.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1742848703}) 2024-09-17T15:34:47.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3171556001}) 2024-09-17T15:34:47.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1742848703}) 2024-09-17T15:34:47.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.142+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=901785034}) 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1862890171}) 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3005198074}) 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1862890171}) 2024-09-17T15:34:47.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.146+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1379844988}) 2024-09-17T15:34:47.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=235907514}) 2024-09-17T15:34:47.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=2392955845}) 2024-09-17T15:34:47.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:34:47.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=26} crc {osdmap=235907514}) 2024-09-17T15:34:47.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:34:47.154+0000 7f9b83b9b640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=2628924022}) 2024-09-17T15:34:47.177 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: finishing write tid 2 to smithi20328086-43 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: finishing write tid 3 to smithi20328086-43 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: finishing write tid 4 to smithi20328086-43 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: oid 43 updating version 0 to 195 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: oid 43 version 195 is already newer than 193 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: oid 43 version 195 is already newer than 194 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 195 (ObjNum 396 snap 110 seq_num 396) dirty exists 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1063: left oid 43 (ObjNum 396 snap 110 seq_num 396) 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1059: done (4 left) 2024-09-17T15:34:47.213 INFO:tasks.rados.rados.0.smithi203.stdout:1061: done (3 left) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1062: done (2 left) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1063: done (1 left) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1064: done (0 left) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1065: rollback oid 23 current snap is 110 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 108 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1066: read oid 33 snap -1 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1066: expect (ObjNum 358 snap 104 seq_num 358) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1067: read oid 32 snap -1 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1067: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1068: copy_from oid 34 from oid 3 current snap is 110 2024-09-17T15:34:47.214 INFO:tasks.rados.rados.0.smithi203.stdout:1069: snap_create 2024-09-17T15:34:47.215 INFO:tasks.rados.rados.0.smithi203.stdout:1068: finishing copy_from to smithi20328086-34 2024-09-17T15:34:47.215 INFO:tasks.rados.rados.0.smithi203.stdout:1068: got expected ENOENT (src dne) 2024-09-17T15:34:47.216 INFO:tasks.rados.rados.0.smithi203.stdout:1068: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:34:47.216 INFO:tasks.rados.rados.0.smithi203.stdout:1065: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:34:47.220 INFO:tasks.rados.rados.0.smithi203.stdout:1065: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:34:47.220 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 394 (ObjNum 145 snap 0 seq_num 3606646432) dirty dne 2024-09-17T15:34:47.222 INFO:tasks.rados.rados.0.smithi203.stdout:1066: expect (ObjNum 358 snap 104 seq_num 358) 2024-09-17T15:34:47.222 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:47.281 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:47.312 INFO:tasks.rados.rados.0.smithi203.stdout:1067: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:34:47.432 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1065: done (4 left) 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1066: done (3 left) 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1067: done (2 left) 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1068: done (1 left) 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1069: done (0 left) 2024-09-17T15:34:47.739 INFO:tasks.rados.rados.0.smithi203.stdout:1070: write oid 24 current snap is 111 2024-09-17T15:34:47.740 INFO:tasks.rados.rados.0.smithi203.stdout:1070: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-17T15:34:47.761 INFO:tasks.rados.rados.0.smithi203.stdout:1070: writing smithi20328086-24 from 672379 to 1396588 tid 1 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1070: writing smithi20328086-24 from 1865116 to 2473588 tid 2 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1071: read oid 38 snap 100 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1071: expect (ObjNum 323 snap 93 seq_num 323) 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1072: copy_from oid 37 from oid 33 current snap is 111 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1070: finishing write tid 1 to smithi20328086-24 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1073: write (excl) oid 31 current snap is 111 2024-09-17T15:34:47.780 INFO:tasks.rados.rados.0.smithi203.stdout:1073: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-17T15:34:47.801 INFO:tasks.rados.rados.0.smithi203.stdout:1073: writing smithi20328086-31 from 689186 to 1488644 tid 1 2024-09-17T15:34:47.813 INFO:tasks.rados.rados.0.smithi203.stdout:1073: writing smithi20328086-31 from 1923598 to 2675728 tid 2 2024-09-17T15:34:47.813 INFO:tasks.rados.rados.0.smithi203.stdout:1074: snap_remove snap 109 2024-09-17T15:34:48.130 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:34:48.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:48.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: get_monmap_and_config 2024-09-17T15:34:48.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: monmap: 2024-09-17T15:34:48.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:48.261507+0000 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:48.261507+0000 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:48.261 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:48.262 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:48.262 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:48.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 Processor -- start 2024-09-17T15:34:48.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 -- start start 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: init 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10200a2ee0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-b con 0x7f102009fc40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 0x7f102013d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-c con 0x7f10200a39d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10200a1200 con 0x7f10200a2ee0 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f10200a4db0 con 0x7f10200a39d0 2024-09-17T15:34:48.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f10200a4b30 con 0x7f102009fc40 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f1025a35640 10 monclient(hunting): authenticate will time out at 1421.226998s 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48778/0 (socket says 172.21.15.86:48778) 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 -- 172.21.15.86:0/3160191416 learned_addr learned my addr 172.21.15.86:0/3160191416 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): get_auth_request con 0x7f102009fc40 auth_method 0 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101f7fe640 1 --2- 172.21.15.86:0/3160191416 >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 0x7f102013d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient(hunting): handle_auth_done global_id 74198 payload 274 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 -- 172.21.15.86:0/3160191416 >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 msgr2=0x7f102013d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 --2- 172.21.15.86:0/3160191416 >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 0x7f102013d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 -- 172.21.15.86:0/3160191416 >> v1:172.21.15.86:6789/0 conn(0x7f10200a2ee0 legacy=0x7f10200a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 monclient: found mon.noname-b 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101e7fc640 1 -- 172.21.15.86:0/3160191416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10200a0eb0 con 0x7f102009fc40 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101f7fe640 10 monclient: get_auth_request con 0x7f10200a39d0 auth_method 0 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.258+0000 7f101f7fe640 1 --2- 172.21.15.86:0/3160191416 >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 0x7f102013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.262+0000 7f101e7fc640 1 -- 172.21.15.86:0/3160191416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 msgr2=0x7f102013b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.262+0000 7f101e7fc640 1 --2- 172.21.15.86:0/3160191416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1014009870 tx=0x7f101401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7c5fc91ef7dca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 1 -- 172.21.15.86:0/3160191416 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140b0070 con 0x7f102009fc40 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: dump: 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.273 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 1 -- 172.21.15.86:0/3160191416 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10140ab600 con 0x7f102009fc40 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 1 -- 172.21.15.86:0/3160191416 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140ab9d0 con 0x7f102009fc40 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101dffb640 10 monclient: dump: 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.274 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 5 monclient: authenticate success, global_id 74198 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 4 set_mon_vals no callback set 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: shutdown 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- 172.21.15.86:0/3160191416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 msgr2=0x7f102013b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- 172.21.15.86:0/3160191416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1014009870 tx=0x7f101401c420 comp rx=0 tx=0).stop 2024-09-17T15:34:48.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- 172.21.15.86:0/3160191416 shutdown_connections 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- 172.21.15.86:0/3160191416 >> v2:172.21.15.203:3300/0 conn(0x7f10200a39d0 0x7f102013d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- 172.21.15.86:0/3160191416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f102009fc40 0x7f102013b0f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- 172.21.15.86:0/3160191416 >> 172.21.15.86:0/3160191416 conn(0x7f10200a86b0 msgr2=0x7f10200a8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- 172.21.15.86:0/3160191416 shutdown_connections 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- 172.21.15.86:0/3160191416 wait complete. 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: monmap: 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.276 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 Processor -- start 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- start start 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: init 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 0x7f1020137e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-b con 0x7f102009fc40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-a con 0x7f10200a2ee0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10200a39d0 0x7f10201388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): picked mon.noname-c con 0x7f10200a39d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f102013e720 con 0x7f10200a2ee0 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f102013e5a0 con 0x7f102009fc40 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f102013e8a0 con 0x7f10200a39d0 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f1025a35640 10 monclient(hunting): authenticate will time out at 1421.234527s 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10200a39d0 0x7f10201388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59662/0 (socket says 172.21.15.86:59662) 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 -- 172.21.15.86:0/3282654376 learned_addr learned my addr 172.21.15.86:0/3282654376 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): get_auth_request con 0x7f10200a2ee0 auth_method 0 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): get_auth_request con 0x7f10200a39d0 auth_method 0 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101effd640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 0x7f1020137e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient(hunting): handle_auth_done global_id 84123 payload 824 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 -- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10200a39d0 msgr2=0x7f10201388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10200a39d0 0x7f10201388e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 -- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 msgr2=0x7f1020137e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 0x7f1020137e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 monclient: found mon.noname-a 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101e7fc640 1 -- 172.21.15.86:0/3282654376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1020139920 con 0x7f10200a2ee0 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101effd640 10 monclient: get_auth_request con 0x7f102009fc40 auth_method 0 2024-09-17T15:34:48.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.266+0000 7f101effd640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 0x7f1020137e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f101e7fc640 1 -- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 msgr2=0x7f10201383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f101e7fc640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f10140ae040 tx=0x7f101401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140ba020 con 0x7f10200a2ee0 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: dump: 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10140b6630 con 0x7f10200a2ee0 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 5 monclient: authenticate success, global_id 84123 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _renew_subs 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102013bc80 con 0x7f10200a2ee0 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140b0040 con 0x7f10200a2ee0 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 10 monclient: dump: 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.284 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _renew_subs 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f101d7fa640 4 set_mon_vals no callback set 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102013a650 con 0x7f10200a2ee0 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10200a32c0 con 0x7f10200a2ee0 2024-09-17T15:34:48.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f10140bd020 con 0x7f10200a2ee0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 0x7f100c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 4847+0+0 (secure 0 0 0) 0x7f10140ea210 con 0x7f10200a2ee0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f101effd640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 0x7f100c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.270+0000 7f101effd640 10 monclient: get_auth_request con 0x7f100c03bb60 auth_method 0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.274+0000 7f101effd640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 0x7f100c03e020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1010003e40 tx=0x7f1010009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.274+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f102013a650 con 0x7f10200a2ee0 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.274+0000 7f1007fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.274+0000 7f1007fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f102006d670 con 0x7f10200a2ee0 2024-09-17T15:34:48.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1007fff640 1 -- 172.21.15.86:0/3282654376 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f10140c4030 con 0x7f10200a2ee0 2024-09-17T15:34:48.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1007fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:48.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1007fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:48.515 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 msgr2=0x7f100c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 0x7f100c03e020 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1010003e40 tx=0x7f1010009040 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 10 monclient: shutdown 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 msgr2=0x7f10201383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f10140ae040 tx=0x7f101401ac60 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 shutdown_connections 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f100c03bb60 0x7f100c03e020 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10200a39d0 0x7f10201388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10200a2ee0 0x7f10201383a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 --2- 172.21.15.86:0/3282654376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f102009fc40 0x7f1020137e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 >> 172.21.15.86:0/3282654376 conn(0x7f10200a86b0 msgr2=0x7f102009e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 shutdown_connections 2024-09-17T15:34:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.514+0000 7f1025a35640 1 -- 172.21.15.86:0/3282654376 wait complete. 2024-09-17T15:34:48.533 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:34:48.533 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:34:48.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:34:48.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:34:48.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: get_monmap_and_config 2024-09-17T15:34:48.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: monmap: 2024-09-17T15:34:48.654 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:48.656308+0000 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:48.656308+0000 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:48.655 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 Processor -- start 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 -- start start 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: init 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 0x7f7e3809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-c con 0x7f7e380a4f70 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e380a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-b con 0x7f7e380a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7e38137d30 0x7f7e381381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-a con 0x7f7e38137d30 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7e3809d430 con 0x7f7e38137d30 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7e380a68d0 con 0x7f7e380a4f70 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e380a64a0 con 0x7f7e380a5350 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3d224640 10 monclient(hunting): authenticate will time out at 1421.621795s 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e380a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e380a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48796/0 (socket says 172.21.15.86:48796) 2024-09-17T15:34:48.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7e38137d30 0x7f7e381381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 1 -- 172.21.15.86:0/2068282349 learned_addr learned my addr 172.21.15.86:0/2068282349 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): get_auth_request con 0x7f7e380a5350 auth_method 0 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): get_auth_request con 0x7f7e38137d30 auth_method 0 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e36d9d640 1 --2- 172.21.15.86:0/2068282349 >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 0x7f7e3809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient(hunting): handle_auth_done global_id 84126 payload 274 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 -- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 msgr2=0x7f7e380a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 --2- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e380a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 -- 172.21.15.86:0/2068282349 >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 msgr2=0x7f7e3809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 --2- 172.21.15.86:0/2068282349 >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 0x7f7e3809cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 monclient: found mon.noname-a 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e3759e640 1 -- 172.21.15.86:0/2068282349 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3809c230 con 0x7f7e38137d30 2024-09-17T15:34:48.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e36d9d640 10 monclient: get_auth_request con 0x7f7e380a4f70 auth_method 0 2024-09-17T15:34:48.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.654+0000 7f7e36d9d640 1 --2- 172.21.15.86:0/2068282349 >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 0x7f7e3809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3759e640 1 -- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38137d30 msgr2=0x7f7e381381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3759e640 1 --2- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38137d30 0x7f7e381381a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e3806e190 tx=0x7f7e2001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f82ffab04b4c72e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 1 -- 172.21.15.86:0/2068282349 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200b0070 con 0x7f7e38137d30 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: dump: 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.668 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 1 -- 172.21.15.86:0/2068282349 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e200ab600 con 0x7f7e38137d30 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 1 -- 172.21.15.86:0/2068282349 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e200ab9d0 con 0x7f7e38137d30 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 5 monclient: authenticate success, global_id 84126 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 4 set_mon_vals no callback set 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e35d9b640 10 monclient: dump: 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.670 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 10 monclient: shutdown 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 -- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38137d30 msgr2=0x7f7e381381a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 --2- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38137d30 0x7f7e381381a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7e3806e190 tx=0x7f7e2001c3d0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 -- 172.21.15.86:0/2068282349 shutdown_connections 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 --2- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38137d30 0x7f7e381381a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 --2- 172.21.15.86:0/2068282349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e380a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 --2- 172.21.15.86:0/2068282349 >> v2:172.21.15.203:3300/0 conn(0x7f7e380a4f70 0x7f7e3809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 -- 172.21.15.86:0/2068282349 >> 172.21.15.86:0/2068282349 conn(0x7f7e380a7240 msgr2=0x7f7e380a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 -- 172.21.15.86:0/2068282349 shutdown_connections 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.658+0000 7f7e3d224640 1 -- 172.21.15.86:0/2068282349 wait complete. 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient: monmap: 2024-09-17T15:34:48.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 Processor -- start 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 -- start start 2024-09-17T15:34:48.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient: init 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 0x7f7e381cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-a con 0x7f7e380a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e381350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-c con 0x7f7e380a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38137d30 0x7f7e3813aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): picked mon.noname-b con 0x7f7e38137d30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3806f9c0 con 0x7f7e380a4f70 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3806f840 con 0x7f7e38137d30 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e381419b0 con 0x7f7e380a5350 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3d224640 10 monclient(hunting): authenticate will time out at 1421.628932s 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 0x7f7e381cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 0x7f7e381cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59680/0 (socket says 172.21.15.86:59680) 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 -- 172.21.15.86:0/419262934 learned_addr learned my addr 172.21.15.86:0/419262934 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): get_auth_request con 0x7f7e380a4f70 auth_method 0 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e381350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): get_auth_request con 0x7f7e380a5350 auth_method 0 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3759e640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38137d30 0x7f7e3813aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient(hunting): handle_auth_done global_id 84129 payload 824 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 msgr2=0x7f7e381350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e381350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38137d30 msgr2=0x7f7e3813aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38137d30 0x7f7e3813aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 monclient: found mon.noname-a 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3813e0f0 con 0x7f7e380a4f70 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3759e640 10 monclient: get_auth_request con 0x7f7e38137d30 auth_method 0 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e380a5350 0x7f7e381350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3759e640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38137d30 0x7f7e3813aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:48.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 msgr2=0x7f7e381cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e36d9d640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 0x7f7e381cfee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000cee0 tx=0x7f7e3000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3001a070 con 0x7f7e380a4f70 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: dump: 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.677 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e3000da50 con 0x7f7e380a4f70 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3000de20 con 0x7f7e380a4f70 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e2f7fe640 10 monclient: dump: 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.678 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.662+0000 7f7e3559a640 4 set_mon_vals no callback set 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 5 monclient: authenticate success, global_id 84129 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 monclient: _renew_subs 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3806ef90 con 0x7f7e380a4f70 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 monclient: _renew_subs 2024-09-17T15:34:48.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e3813ceb0 con 0x7f7e380a4f70 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 0x7f7e3813af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7e380a5750 con 0x7f7e38135620 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 0x7f7e3813af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient: get_auth_request con 0x7f7e38135620 auth_method 0 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7e3001d070 con 0x7f7e380a4f70 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 10 monclient(hunting): handle_auth_done global_id 84132 payload 824 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e380a5ff0 con 0x7f7e38135620 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e1803bbb0 0x7f7e1803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(239..239 src has 1..239) ==== 4847+0+0 (secure 0 0 0) 0x7f7e300507b0 con 0x7f7e380a4f70 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3759e640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e1803bbb0 0x7f7e1803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3759e640 10 monclient: get_auth_request con 0x7f7e1803bbb0 auth_method 0 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 msgr2=0x7f7e3813af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 0x7f7e3813af40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e3813dd40 tx=0x7f7e2800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28014650 con 0x7f7e38135620 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 10 monclient: dump: 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e28014650 con 0x7f7e38135620 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 msgr2=0x7f7e3813af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 0x7f7e3813af40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e3813dd40 tx=0x7f7e2800f610 comp rx=0 tx=0).stop 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 msgr2=0x7f7e3813af40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.666+0000 7f7e2f7fe640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e38135620 0x7f7e3813af40 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.670+0000 7f7e3759e640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e1803bbb0 0x7f7e1803e070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e200ab000 tx=0x7f7e200a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3d224640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3d224640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 0x7f7e3809f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3d224640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3813b860 con 0x7f7e3813b480 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 0x7f7e3809f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient: get_auth_request con 0x7f7e3813b480 auth_method 0 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 10 monclient(hunting): handle_auth_done global_id 84135 payload 824 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.674+0000 7f7e3659c640 1 -- 172.21.15.86:0/419262934 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e3813ca30 con 0x7f7e3813b480 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3659c640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 msgr2=0x7f7e3809f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3659c640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 0x7f7e3809f720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e28019000 tx=0x7f7e28004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28003620 con 0x7f7e3813b480 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 10 monclient: dump: 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f7e28003620 con 0x7f7e3813b480 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 msgr2=0x7f7e3809f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 0x7f7e3809f720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e28019000 tx=0x7f7e28004740 comp rx=0 tx=0).stop 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 msgr2=0x7f7e3809f720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e2f7fe640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e3813b480 0x7f7e3809f720 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e28019000 tx=0x7f7e28004740 comp rx=0 tx=0).stop 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 20955, 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:48.689 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.690 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.691 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:48.692 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.693 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:48.694 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:48.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:48.696 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:48.699 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.700 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:48.701 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e1803bbb0 msgr2=0x7f7e1803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e1803bbb0 0x7f7e1803e070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7e200ab000 tx=0x7f7e200a6000 comp rx=0 tx=0).stop 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 10 monclient: shutdown 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 msgr2=0x7f7e381cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 --2- 172.21.15.86:0/419262934 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e380a4f70 0x7f7e381cfee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7e3000cee0 tx=0x7f7e3000a710 comp rx=0 tx=0).stop 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3759e640 1 -- 172.21.15.86:0/419262934 reap_dead start 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3759e640 1 -- 172.21.15.86:0/419262934 reap_dead start 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 shutdown_connections 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 >> 172.21.15.86:0/419262934 conn(0x7f7e380a7240 msgr2=0x7f7e380a9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 shutdown_connections 2024-09-17T15:34:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.678+0000 7f7e3d224640 1 -- 172.21.15.86:0/419262934 wait complete. 2024-09-17T15:34:48.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:48.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:34:48.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:34:48.752 INFO:tasks.rados.rados.0.smithi203.stdout:1074: done (4 left) 2024-09-17T15:34:48.752 INFO:tasks.rados.rados.0.smithi203.stdout:1075: copy_from oid 17 from oid 23 current snap is 111 2024-09-17T15:34:48.752 INFO:tasks.rados.rados.0.smithi203.stdout:1070: finishing write tid 2 to smithi20328086-24 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: finishing write tid 3 to smithi20328086-24 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: finishing write tid 4 to smithi20328086-24 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: oid 24 updating version 0 to 365 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: oid 24 updating version 365 to 366 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: oid 24 version 366 is already newer than 364 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 366 (ObjNum 397 snap 111 seq_num 397) dirty exists 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: left oid 24 (ObjNum 397 snap 111 seq_num 397) 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1072: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1070: done (4 left) 2024-09-17T15:34:48.753 INFO:tasks.rados.rados.0.smithi203.stdout:1076: snap_remove snap 92 2024-09-17T15:34:48.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: get_monmap_and_config 2024-09-17T15:34:48.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: monmap: 2024-09-17T15:34:48.818 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:48.820116+0000 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:48.820116+0000 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:48.820 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:48.820 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 Processor -- start 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 -- start start 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: init 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:34:48.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2ec0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2ec0a5bd0 0x7fc2ec09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-c con 0x7fc2ec0a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 0x7fc2ec13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-b con 0x7fc2ec13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2ec0a6280 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc2ec09d2a0 con 0x7fc2ec0a5bd0 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec09c0a0 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2f1e92640 10 monclient(hunting): authenticate will time out at 1421.785553s 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2eb7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc2ec0a50e0 0x7fc2ec0a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47614/0 (socket says 172.21.15.86:47614) 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 0x7fc2ec13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3571686132 learned_addr learned my addr 172.21.15.86:0/3571686132 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): get_auth_request con 0x7fc2ec13d2f0 auth_method 0 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2eaffd640 1 --2- 172.21.15.86:0/3571686132 >> v2:172.21.15.203:3300/0 conn(0x7fc2ec0a5bd0 0x7fc2ec09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient(hunting): handle_auth_done global_id 74210 payload 274 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3571686132 >> v2:172.21.15.203:3300/0 conn(0x7fc2ec0a5bd0 msgr2=0x7fc2ec09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 --2- 172.21.15.86:0/3571686132 >> v2:172.21.15.203:3300/0 conn(0x7fc2ec0a5bd0 0x7fc2ec09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3571686132 >> v1:172.21.15.86:6789/0 conn(0x7fc2ec0a50e0 legacy=0x7fc2ec0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 monclient: found mon.noname-b 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.818+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3571686132 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec09c380 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3571686132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 msgr2=0x7fc2ec13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ebfff640 1 --2- 172.21.15.86:0/3571686132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 0x7fc2ec13f6e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40097c0 tx=0x7fc2d401c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a3cc110cc5cf57c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 1 -- 172.21.15.86:0/3571686132 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40b0070 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.831 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 1 -- 172.21.15.86:0/3571686132 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d40ab600 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 5 monclient: authenticate success, global_id 74210 2024-09-17T15:34:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 1 -- 172.21.15.86:0/3571686132 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40ab9d0 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 4 set_mon_vals no callback set 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2ea7fc640 10 monclient: dump: 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 10 monclient: shutdown 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3571686132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 msgr2=0x7fc2ec13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3571686132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 0x7fc2ec13f6e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc2d40097c0 tx=0x7fc2d401c2d0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3571686132 shutdown_connections 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3571686132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec13d2f0 0x7fc2ec13f6e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3571686132 >> v2:172.21.15.203:3300/0 conn(0x7fc2ec0a5bd0 0x7fc2ec09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3571686132 >> 172.21.15.86:0/3571686132 conn(0x7fc2ec0aaae0 msgr2=0x7fc2ec0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3571686132 shutdown_connections 2024-09-17T15:34:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3571686132 wait complete. 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 10 monclient: monmap: 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 Processor -- start 2024-09-17T15:34:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 1 -- start start 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.822+0000 7fc2f1e92640 10 monclient: init 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 0x7fc2ec121410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-c con 0x7fc2ec0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2ec0a5bd0 0x7fc2ec09fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-a con 0x7fc2ec0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2ec13d2f0 0x7fc2ec138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): picked mon.noname-b con 0x7fc2ec13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ec142650 con 0x7fc2ec0a5bd0 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc2ec0a6b00 con 0x7fc2ec13d2f0 2024-09-17T15:34:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec1427d0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): authenticate will time out at 1421.792386s 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 0x7fc2ec121410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 0x7fc2ec121410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48838/0 (socket says 172.21.15.86:48838) 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2ec0a5bd0 0x7fc2ec09fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3279144585 learned_addr learned my addr 172.21.15.86:0/3279144585 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc2ec0a50e0 auth_method 0 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): get_auth_request con 0x7fc2ec0a5bd0 auth_method 0 2024-09-17T15:34:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient(hunting): handle_auth_done global_id 74213 payload 824 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2ec13d2f0 msgr2=0x7fc2ec138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2ec13d2f0 0x7fc2ec138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2ec0a5bd0 msgr2=0x7fc2ec09fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2ec0a5bd0 0x7fc2ec09fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 monclient: found mon.noname-c 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2ec0a12d0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 msgr2=0x7fc2ec121410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2eb7fe640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 0x7fc2ec121410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400cbc0 tx=0x7fc2e400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4012400 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: dump: 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e40125a0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4012990 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 5 monclient: authenticate success, global_id 74213 2024-09-17T15:34:48.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _renew_subs 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2ec122c10 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e9ffb640 4 set_mon_vals no callback set 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2e3fff640 10 monclient: dump: 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.841 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _renew_subs 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2ec0a0bd0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 0x7fc2ec1cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.826+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec06a0e0 con 0x7fc2ec1387c0 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 0x7fc2ec1cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient: get_auth_request con 0x7fc2ec1387c0 auth_method 0 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc2e4012bb0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_done global_id 74216 payload 824 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc2ec0a18d0 con 0x7fc2ec1387c0 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2d003bb60 0x7fc2d003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7fc2e40505f0 con 0x7fc2ec0a50e0 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2ebfff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2d003bb60 0x7fc2d003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2ebfff640 10 monclient: get_auth_request con 0x7fc2d003bb60 auth_method 0 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 msgr2=0x7fc2ec1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2eaffd640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 0x7fc2ec1cfba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc00fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00de30 con 0x7fc2ec1387c0 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 10 monclient: dump: 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2dc00de30 con 0x7fc2ec1387c0 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 msgr2=0x7fc2ec1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 0x7fc2ec1cfba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc00fca0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 msgr2=0x7fc2ec1cfba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2e3fff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1387c0 0x7fc2ec1cfba0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc00fca0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.830+0000 7fc2ebfff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2d003bb60 0x7fc2d003e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0aa620 tx=0x7fc2d4004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2f1e92640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 0x7fc2ec1d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2f1e92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc2ec1d0a00 con 0x7fc2ec1d00e0 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 0x7fc2ec1d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient: get_auth_request con 0x7fc2ec1d00e0 auth_method 0 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 10 monclient(hunting): handle_auth_done global_id 74219 payload 824 2024-09-17T15:34:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 1 -- 172.21.15.86:0/3279144585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc2ec09cd60 con 0x7fc2ec1d00e0 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 msgr2=0x7fc2ec1d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2eaffd640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 0x7fc2ec1d04c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc003830 con 0x7fc2ec1d00e0 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 10 monclient: dump: 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fc2dc003830 con 0x7fc2ec1d00e0 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 msgr2=0x7fc2ec1d04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 0x7fc2ec1d04c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 msgr2=0x7fc2ec1d04c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.838+0000 7fc2e3fff640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec1d00e0 0x7fc2ec1d04c0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00f7e0 tx=0x7fc2dc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.850 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 69627, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:48.851 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.852 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.853 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:48.854 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.856 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.857 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:48.859 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.860 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:48.861 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:48.862 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:48.863 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2d003bb60 msgr2=0x7fc2d003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2d003bb60 0x7fc2d003e020 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec0aa620 tx=0x7fc2d4004740 comp rx=0 tx=0).stop 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 10 monclient: shutdown 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 msgr2=0x7fc2ec121410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 --2- 172.21.15.86:0/3279144585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2ec0a50e0 0x7fc2ec121410 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc2e400cbc0 tx=0x7fc2e400a430 comp rx=0 tx=0).stop 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3279144585 reap_dead start 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2ebfff640 1 -- 172.21.15.86:0/3279144585 reap_dead start 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 shutdown_connections 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 >> 172.21.15.86:0/3279144585 conn(0x7fc2ec0aaae0 msgr2=0x7fc2ec13dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 shutdown_connections 2024-09-17T15:34:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.842+0000 7fc2f1e92640 1 -- 172.21.15.86:0/3279144585 wait complete. 2024-09-17T15:34:48.866 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:48.866 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:34:48.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: get_monmap_and_config 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: monmap: 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:48.976928+0000 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:48.976928+0000 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:48.976 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:48.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 Processor -- start 2024-09-17T15:34:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 -- start start 2024-09-17T15:34:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: init 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95c8134b10 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95c8134650 0x7f95c813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-a con 0x7f95c8134650 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c813d410 0x7f95c813f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-b con 0x7f95c813d410 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95c80a1010 con 0x7f95c8134b10 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f95c80a0010 con 0x7f95c8134650 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f95c806ffd0 con 0x7f95c813d410 2024-09-17T15:34:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95cdc87640 10 monclient(hunting): authenticate will time out at 1421.942479s 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95c8134650 0x7f95c813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f95c8134b10 0x7f95c8134ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47628/0 (socket says 172.21.15.86:47628) 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95c8134650 0x7f95c813ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59716/0 (socket says 172.21.15.86:59716) 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c77fe640 1 -- 172.21.15.86:0/1099719852 learned_addr learned my addr 172.21.15.86:0/1099719852 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c813d410 0x7f95c813f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): get_auth_request con 0x7f95c8134650 auth_method 0 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): get_auth_request con 0x7f95c813d410 auth_method 0 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_done global_id 84141 payload 274 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 -- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c813d410 msgr2=0x7f95c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c813d410 0x7f95c813f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 -- 172.21.15.86:0/1099719852 >> v1:172.21.15.86:6789/0 conn(0x7f95c8134b10 legacy=0x7f95c8134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:48.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 monclient: found mon.noname-a 2024-09-17T15:34:48.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.974+0000 7f95c6ffd640 1 -- 172.21.15.86:0/1099719852 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c80702b0 con 0x7f95c8134650 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c6ffd640 1 -- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134650 msgr2=0x7f95c813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c6ffd640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134650 0x7f95c813ced0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f95b00097c0 tx=0x7f95b001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45aefa8aaeca999c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 1 -- 172.21.15.86:0/1099719852 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00b0070 con 0x7f95c8134650 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: dump: 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 1 -- 172.21.15.86:0/1099719852 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95b00ab600 con 0x7f95c8134650 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95cdc87640 5 monclient: authenticate success, global_id 84141 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95cdc87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95cdc87640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 1 -- 172.21.15.86:0/1099719852 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00ab9d0 con 0x7f95c8134650 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95cdc87640 4 set_mon_vals no callback set 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.978+0000 7f95c67fc640 10 monclient: dump: 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: shutdown 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134650 msgr2=0x7f95c813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134650 0x7f95c813ced0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f95b00097c0 tx=0x7f95b001c360 comp rx=0 tx=0).stop 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- 172.21.15.86:0/1099719852 shutdown_connections 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c813d410 0x7f95c813f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- 172.21.15.86:0/1099719852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134650 0x7f95c813ced0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- 172.21.15.86:0/1099719852 >> 172.21.15.86:0/1099719852 conn(0x7f95c80a7210 msgr2=0x7f95c80a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- 172.21.15.86:0/1099719852 shutdown_connections 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- 172.21.15.86:0/1099719852 wait complete. 2024-09-17T15:34:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: build_initial_monmap 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: monmap: 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 Processor -- start 2024-09-17T15:34:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- start start 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: init 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c8134650 0x7f95c81cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-c con 0x7f95c8134650 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 0x7f95c81d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-a con 0x7f95c8134b10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813d410 0x7f95c81d0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): picked mon.noname-b con 0x7f95c813d410 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f95c81428f0 con 0x7f95c8134b10 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95c8142770 con 0x7f95c813d410 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f95c8142a70 con 0x7f95c8134650 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95cdc87640 10 monclient(hunting): authenticate will time out at 1421.949981s 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 0x7f95c81d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c8134650 0x7f95c81cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 0x7f95c81d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59728/0 (socket says 172.21.15.86:59728) 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 -- 172.21.15.86:0/2875757498 learned_addr learned my addr 172.21.15.86:0/2875757498 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): get_auth_request con 0x7f95c8134b10 auth_method 0 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): get_auth_request con 0x7f95c8134650 auth_method 0 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient(hunting): handle_auth_done global_id 84144 payload 824 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c8134650 msgr2=0x7f95c81cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95c8134650 0x7f95c81cfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813d410 msgr2=0x7f95c81d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813d410 0x7f95c81d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 monclient: found mon.noname-a 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.982+0000 7f95c6ffd640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c806ef00 con 0x7f95c8134b10 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c6ffd640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 msgr2=0x7f95c81d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c6ffd640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 0x7f95c81d00f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f95b001cc00 tx=0x7f95b001aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00b0050 con 0x7f95c8134b10 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: dump: 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: _finish_auth 0 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95b00bc070 con 0x7f95c8134b10 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b00b5a40 con 0x7f95c8134b10 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 5 monclient: authenticate success, global_id 84144 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 monclient: _renew_subs 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c813be10 con 0x7f95c8134b10 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c5ffb640 4 set_mon_vals no callback set 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 10 monclient: dump: 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 monclient: _renew_subs 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c813c870 con 0x7f95c8134b10 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 0x7f95c8131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95c813f8c0 con 0x7f95c813a8f0 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f95b00b5c60 con 0x7f95c8134b10 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 0x7f95c8131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95a803bbb0 0x7f95a803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient: get_auth_request con 0x7f95c813a8f0 auth_method 0 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7f95b00ea680 con 0x7f95c8134b10 2024-09-17T15:34:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c7fff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95a803bbb0 0x7f95a803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c7fff640 10 monclient: get_auth_request con 0x7f95a803bbb0 auth_method 0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 10 monclient(hunting): handle_auth_done global_id 54313 payload 824 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.986+0000 7f95c77fe640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95c8139d90 con 0x7f95c813a8f0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95c77fe640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 msgr2=0x7f95c8131b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95c77fe640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 0x7f95c8131b60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95c000eaa0 tx=0x7f95c000eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95c7fff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95a803bbb0 0x7f95a803e070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95c813b780 tx=0x7f95b800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c000ee60 con 0x7f95c813a8f0 2024-09-17T15:34:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 10 monclient: dump: 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95c0019070 con 0x7f95c813a8f0 2024-09-17T15:34:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 msgr2=0x7f95c8131b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 0x7f95c8131b60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95c000eaa0 tx=0x7f95c000eb90 comp rx=0 tx=0).stop 2024-09-17T15:34:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 msgr2=0x7f95c8131b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.990+0000 7f95bffff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c813a8f0 0x7f95c8131b60 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95c000eaa0 tx=0x7f95c000eb90 comp rx=0 tx=0).stop 2024-09-17T15:34:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95cdc87640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:34:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95cdc87640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 0x7f95c802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95cdc87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95c8134ef0 con 0x7f95c802d450 2024-09-17T15:34:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 0x7f95c802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient: get_auth_request con 0x7f95c802d450 auth_method 0 2024-09-17T15:34:49.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 10 monclient(hunting): handle_auth_done global_id 54316 payload 824 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:48.998+0000 7f95c77fe640 1 -- 172.21.15.86:0/2875757498 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95c813f120 con 0x7f95c802d450 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95c77fe640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 msgr2=0x7f95c802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95c77fe640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 0x7f95c802d860 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95c00070c0 tx=0x7f95c0004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c001ea40 con 0x7f95c802d450 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:49.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 10 monclient: dump: 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.009 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f95c0014480 con 0x7f95c802d450 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 msgr2=0x7f95c802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 0x7f95c802d860 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95c00070c0 tx=0x7f95c0004950 comp rx=0 tx=0).stop 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 msgr2=0x7f95c802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:34:49.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95bffff640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c802d450 0x7f95c802d860 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95c00070c0 tx=0x7f95c0004950 comp rx=0 tx=0).stop 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 76, 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 164433, 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:34:49.011 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:49.012 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:49.013 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:34:49.014 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:34:49.015 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:34:49.016 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:34:49.018 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:49.019 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:49.020 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:49.023 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:34:49.024 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:34:49.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:34:49.025 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:34:49.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:34:49.025 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:34:49.025 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:34:49.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95a803bbb0 msgr2=0x7f95a803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95a803bbb0 0x7f95a803e070 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f95c813b780 tx=0x7f95b800b040 comp rx=0 tx=0).stop 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 10 monclient: shutdown 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 msgr2=0x7f95c81d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 --2- 172.21.15.86:0/2875757498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95c8134b10 0x7f95c81d00f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f95b001cc00 tx=0x7f95b001aba0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95c7fff640 1 -- 172.21.15.86:0/2875757498 reap_dead start 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95c7fff640 1 -- 172.21.15.86:0/2875757498 reap_dead start 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 shutdown_connections 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 >> 172.21.15.86:0/2875757498 conn(0x7f95c80a7210 msgr2=0x7f95c80a5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 shutdown_connections 2024-09-17T15:34:49.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.002+0000 7f95cdc87640 1 -- 172.21.15.86:0/2875757498 wait complete. 2024-09-17T15:34:49.028 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:34:49.028 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:34:49.028 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:34:49.028 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:34:49.028 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:34:49.028 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:34:49.028 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:34:49.038 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:34:49.039 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:34:49.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: get_monmap_and_config 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: build_initial_monmap 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: monmap: 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:49.197028+0000 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:49.197028+0000 2024-09-17T15:34:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:49.196 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 Processor -- start 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 -- start start 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: init 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9ee0068db0 0x7f9ee00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee0068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ee0135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee009c8b0 con 0x7f9ee0135ae0 2024-09-17T15:34:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee0074500 con 0x7f9ee0068db0 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee009cbe0 con 0x7f9ee00689d0 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee7190640 10 monclient(hunting): authenticate will time out at 1422.162464s 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48892/0 (socket says 172.21.15.86:48892) 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 -- 172.21.15.86:0/2893507328 learned_addr learned my addr 172.21.15.86:0/2893507328 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): get_auth_request con 0x7f9ee00689d0 auth_method 0 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:49.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9edffff640 1 --2- 172.21.15.86:0/2893507328 >> v2:172.21.15.203:3300/0 conn(0x7f9ee0068db0 0x7f9ee00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9edffff640 10 monclient(hunting): get_auth_request con 0x7f9ee0068db0 auth_method 0 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9edffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9edffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient(hunting): handle_auth_done global_id 74228 payload 274 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 -- 172.21.15.86:0/2893507328 >> v2:172.21.15.203:3300/0 conn(0x7f9ee0068db0 msgr2=0x7f9ee00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/2893507328 >> v2:172.21.15.203:3300/0 conn(0x7f9ee0068db0 0x7f9ee00691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 -- 172.21.15.86:0/2893507328 >> v1:172.21.15.203:6789/0 conn(0x7f9ee0135ae0 legacy=0x7f9ee0135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient: _finish_hunting 0 2024-09-17T15:34:49.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 monclient: found mon.noname-b 2024-09-17T15:34:49.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:49.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.194+0000 7f9ee4f2c640 1 -- 172.21.15.86:0/2893507328 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee009ce70 con 0x7f9ee00689d0 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9ee4f2c640 1 -- 172.21.15.86:0/2893507328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 msgr2=0x7f9ee006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/2893507328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009e50 tx=0x7f9ed001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=728a07d3bde97d04 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9edf7fe640 1 -- 172.21.15.86:0/2893507328 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00a56f0 con 0x7f9ee00689d0 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9edf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9edf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.198+0000 7f9edf7fe640 10 monclient: dump: 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 1 -- 172.21.15.86:0/2893507328 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed00a5890 con 0x7f9ee00689d0 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 1 -- 172.21.15.86:0/2893507328 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00a5c60 con 0x7f9ee00689d0 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edf7fe640 10 monclient: dump: 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 5 monclient: authenticate success, global_id 74228 2024-09-17T15:34:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 4 set_mon_vals no callback set 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: shutdown 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- 172.21.15.86:0/2893507328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 msgr2=0x7f9ee006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- 172.21.15.86:0/2893507328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009e50 tx=0x7f9ed001c890 comp rx=0 tx=0).stop 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- 172.21.15.86:0/2893507328 shutdown_connections 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- 172.21.15.86:0/2893507328 >> v2:172.21.15.203:3300/0 conn(0x7f9ee0068db0 0x7f9ee00691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- 172.21.15.86:0/2893507328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee00689d0 0x7f9ee006f840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- 172.21.15.86:0/2893507328 >> 172.21.15.86:0/2893507328 conn(0x7f9ee00a6440 msgr2=0x7f9ee00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- 172.21.15.86:0/2893507328 shutdown_connections 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- 172.21.15.86:0/2893507328 wait complete. 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: build_initial_monmap 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: monmap: 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 Processor -- start 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- start start 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: init 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 0x7f9ee013c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee00689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 0x7f9ee011d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee0135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee01409b0 con 0x7f9ee0135ae0 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee0140830 con 0x7f9ee00689d0 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee0140b30 con 0x7f9ee0068db0 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee7190640 10 monclient(hunting): authenticate will time out at 1422.170242s 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee572d640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 msgr2=0x7f9ee011d5b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee572d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 0x7f9ee011d5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48900/0 (socket says 172.21.15.86:48900) 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 -- 172.21.15.86:0/1328066976 learned_addr learned my addr 172.21.15.86:0/1328066976 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): get_auth_request con 0x7f9ee0068db0 auth_method 0 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 0x7f9ee013c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient(hunting): handle_auth_done global_id 74231 payload 824 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 -- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 msgr2=0x7f9ee013c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 0x7f9ee013c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 -- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 msgr2=0x7f9ee011d5b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 0x7f9ee011d5b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient: _finish_hunting 0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 monclient: found mon.noname-c 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9edffff640 1 -- 172.21.15.86:0/1328066976 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee011f640 con 0x7f9ee0068db0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee4f2c640 10 monclient: get_auth_request con 0x7f9ee00689d0 auth_method 0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.202+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 0x7f9ee013c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edffff640 1 -- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 msgr2=0x7f9ee011d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edffff640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800d830 tx=0x7f9ec800dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800b8d0 con 0x7f9ee0068db0 2024-09-17T15:34:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: dump: 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ec8019070 con 0x7f9ee0068db0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800bc70 con 0x7f9ee0068db0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 5 monclient: authenticate success, global_id 74231 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _renew_subs 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee0073680 con 0x7f9ee0068db0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 10 monclient: dump: 2024-09-17T15:34:49.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edeffd640 4 set_mon_vals no callback set 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _renew_subs 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee011f360 con 0x7f9ee0068db0 2024-09-17T15:34:49.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee006f840 con 0x7f9ee0068db0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9ec801e460 con 0x7f9ee0068db0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 0x7f9ec003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7f9ec8050410 con 0x7f9ee0068db0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 0x7f9ec003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.206+0000 7f9ee4f2c640 10 monclient: get_auth_request con 0x7f9ec003bb60 auth_method 0 2024-09-17T15:34:49.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.210+0000 7f9ee4f2c640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 0x7f9ec003e020 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ee006fdb0 tx=0x7f9ed00a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:49.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.210+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:34:49.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.210+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9ec800be10 con 0x7f9ee0068db0 2024-09-17T15:34:49.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.214+0000 7f9edd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:49.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.214+0000 7f9edd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:49.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9ee7190640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:49.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9ee7190640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:49.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9ee7190640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:49.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9ee006d4f0 con 0x7f9ee0068db0 2024-09-17T15:34:49.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9edd7fa640 1 -- 172.21.15.86:0/1328066976 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9ec801cd70 con 0x7f9ee0068db0 2024-09-17T15:34:49.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9edd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:49.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.446+0000 7f9edd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:49.450 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 msgr2=0x7f9ec003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 0x7f9ec003e020 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9ee006fdb0 tx=0x7f9ed00a7070 comp rx=0 tx=0).stop 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 10 monclient: shutdown 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 msgr2=0x7f9ee011d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:49.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9ec800d830 tx=0x7f9ec800dcf0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 shutdown_connections 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ec003bb60 0x7f9ec003e020 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ee0135ae0 0x7f9ee011d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ee0068db0 0x7f9ee011d070 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 --2- 172.21.15.86:0/1328066976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ee00689d0 0x7f9ee013c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 >> 172.21.15.86:0/1328066976 conn(0x7f9ee00a6440 msgr2=0x7f9ee00a2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 shutdown_connections 2024-09-17T15:34:49.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:49.450+0000 7f9ee7190640 1 -- 172.21.15.86:0/1328066976 wait complete. 2024-09-17T15:34:52.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: get_monmap_and_config 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: build_initial_monmap 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: monmap: 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:52.597867+0000 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:52.597867+0000 2024-09-17T15:34:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:52.597 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 Processor -- start 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 -- start start 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: init 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 0x7f7d7c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:34:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d7c135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d7c09c8b0 con 0x7f7d7c135ae0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7c074500 con 0x7f7d7c0689d0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c09cbe0 con 0x7f7d7c068db0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d819e6640 10 monclient(hunting): authenticate will time out at 1425.563474s 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7affd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 msgr2=0x7f7d7c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7affd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 0x7f7d7c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48902/0 (socket says 172.21.15.86:48902) 2024-09-17T15:34:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 -- 172.21.15.86:0/2902333171 learned_addr learned my addr 172.21.15.86:0/2902333171 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): get_auth_request con 0x7f7d7c068db0 auth_method 0 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 10 monclient(hunting): handle_auth_done global_id 74234 payload 274 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 -- 172.21.15.86:0/2902333171 >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 msgr2=0x7f7d7c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 --2- 172.21.15.86:0/2902333171 >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 0x7f7d7c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.594+0000 7f7d7a7fc640 1 -- 172.21.15.86:0/2902333171 >> v1:172.21.15.203:6789/0 conn(0x7f7d7c135ae0 legacy=0x7f7d7c135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:34:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.598+0000 7f7d7a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:34:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.598+0000 7f7d7a7fc640 1 monclient: found mon.noname-b 2024-09-17T15:34:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.598+0000 7f7d7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.598+0000 7f7d7a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:34:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.598+0000 7f7d7a7fc640 1 -- 172.21.15.86:0/2902333171 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c09ce70 con 0x7f7d7c068db0 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7a7fc640 1 -- 172.21.15.86:0/2902333171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 msgr2=0x7f7d7c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7a7fc640 1 --2- 172.21.15.86:0/2902333171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009870 tx=0x7f7d6c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c7ffe94bc4634f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 1 -- 172.21.15.86:0/2902333171 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0b2070 con 0x7f7d7c068db0 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: dump: 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: _finish_auth 0 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 1 -- 172.21.15.86:0/2902333171 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d6c0ad680 con 0x7f7d7c068db0 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 1 -- 172.21.15.86:0/2902333171 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0ada50 con 0x7f7d7c068db0 2024-09-17T15:34:52.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 5 monclient: authenticate success, global_id 74234 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 4 set_mon_vals no callback set 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d79ffb640 10 monclient: dump: 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:52.606 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: shutdown 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- 172.21.15.86:0/2902333171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 msgr2=0x7f7d7c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- 172.21.15.86:0/2902333171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009870 tx=0x7f7d6c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- 172.21.15.86:0/2902333171 shutdown_connections 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- 172.21.15.86:0/2902333171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c068db0 0x7f7d7c0691d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- 172.21.15.86:0/2902333171 >> v2:172.21.15.86:3300/0 conn(0x7f7d7c0689d0 0x7f7d7c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- 172.21.15.86:0/2902333171 >> 172.21.15.86:0/2902333171 conn(0x7f7d7c0a6440 msgr2=0x7f7d7c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- 172.21.15.86:0/2902333171 shutdown_connections 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- 172.21.15.86:0/2902333171 wait complete. 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: build_initial_monmap 2024-09-17T15:34:52.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: monmap: 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 Processor -- start 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- start start 2024-09-17T15:34:52.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: init 2024-09-17T15:34:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:34:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 0x7f7d7c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7d7c0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 0x7f7d7c11dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7d7c068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): picked mon.noname-c con 0x7f7d7c135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c1409b0 con 0x7f7d7c068db0 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7c140830 con 0x7f7d7c0689d0 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7c140b30 con 0x7f7d7c135ae0 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d819e6640 10 monclient(hunting): authenticate will time out at 1425.570887s 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7a7fc640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 msgr2=0x7f7d7c11dd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7a7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 0x7f7d7c11dd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48916/0 (socket says 172.21.15.86:48916) 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 -- 172.21.15.86:0/1601904461 learned_addr learned my addr 172.21.15.86:0/1601904461 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): get_auth_request con 0x7f7d7c135ae0 auth_method 0 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7affd640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 0x7f7d7c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient(hunting): handle_auth_done global_id 74237 payload 824 2024-09-17T15:34:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 -- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 msgr2=0x7f7d7c11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 0x7f7d7c11c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 -- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 msgr2=0x7f7d7c11dd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 0x7f7d7c11dd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7b7fe640 1 -- 172.21.15.86:0/1601904461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7c135860 con 0x7f7d7c135ae0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7affd640 10 monclient: get_auth_request con 0x7f7d7c0689d0 auth_method 0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.602+0000 7f7d7affd640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 0x7f7d7c11c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d7b7fe640 1 -- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 msgr2=0x7f7d7c134bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d7b7fe640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c780 tx=0x7f7d7400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d7b7fe640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c780 tx=0x7f7d7400cc40 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f7d7400ae00 config(0 keys) 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400ac60 con 0x7f7d7c135ae0 2024-09-17T15:34:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: dump: 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: _finish_auth 0 2024-09-17T15:34:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d7400ae00 con 0x7f7d7c135ae0 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 5 monclient: authenticate success, global_id 74237 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _renew_subs 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d7c13d930 con 0x7f7d7c135ae0 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d797fa640 4 set_mon_vals no callback set 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _renew_subs 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7c11ecb0 con 0x7f7d7c135ae0 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7c06f840 con 0x7f7d7c135ae0 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74004bb0 con 0x7f7d7c135ae0 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 10 monclient: dump: 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7d74006b60 con 0x7f7d7c135ae0 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.606+0000 7f7d63fff640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 0x7f7d6803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.610+0000 7f7d7affd640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 0x7f7d6803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.610+0000 7f7d7affd640 10 monclient: get_auth_request con 0x7f7d6803bb10 auth_method 0 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.610+0000 7f7d7affd640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 0x7f7d6803dfd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003990 tx=0x7f7d70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:52.614+0000 7f7d7b7fe640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c780 tx=0x7f7d7400cc40 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f7d74020070 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-17T15:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.230+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7f7d74050af0 con 0x7f7d7c135ae0 2024-09-17T15:34:53.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.442+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7d74020070 con 0x7f7d7c135ae0 2024-09-17T15:34:53.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.442+0000 7f7d63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:53.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.442+0000 7f7d63fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:53.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.686+0000 7f7d819e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.686+0000 7f7d819e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.686+0000 7f7d819e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.686+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d7c06d4f0 con 0x7f7d7c135ae0 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d63fff640 1 -- 172.21.15.86:0/1601904461 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7d74004dd0 con 0x7f7d7c135ae0 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d63fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:53.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d63fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:53.691 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 msgr2=0x7f7d6803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 0x7f7d6803dfd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7d70003990 tx=0x7f7d70008040 comp rx=0 tx=0).stop 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 10 monclient: shutdown 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 msgr2=0x7f7d7c134bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c780 tx=0x7f7d7400cc40 comp rx=0 tx=0).stop 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 shutdown_connections 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7d6803bb10 0x7f7d6803dfd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d7c135ae0 0x7f7d7c134bc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d7c068db0 0x7f7d7c11dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 --2- 172.21.15.86:0/1601904461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d7c0689d0 0x7f7d7c11c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 >> 172.21.15.86:0/1601904461 conn(0x7f7d7c0a6440 msgr2=0x7f7d7c0a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 shutdown_connections 2024-09-17T15:34:53.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:53.690+0000 7f7d819e6640 1 -- 172.21.15.86:0/1601904461 wait complete. 2024-09-17T15:34:56.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:34:56.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: get_monmap_and_config 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: build_initial_monmap 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: monmap: 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:34:56.841558+0000 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:34:56.841558+0000 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:34:56.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:34:56.841 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:34:56.841 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:34:56.841 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:34:56.841 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 Processor -- start 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 -- start start 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: init 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0dc0a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0dc0a5b60 addr v1:172.21.15.203:6789/0 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa0dc09cf60 0x7fa0dc1409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-c con 0x7fa0dc09cf60 addr v2:172.21.15.203:3300/0 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0dc0a6f40 con 0x7fa0dc0a5070 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0dc0a65e0 con 0x7fa0dc0a5b60 2024-09-17T15:34:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa0dc09d390 con 0x7fa0dc09cf60 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0e2203640 10 monclient(hunting): authenticate will time out at 1429.807079s 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0db7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fa0dc0a5070 legacy=0x7fa0dc0a5450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa0dc09cf60 0x7fa0dc1409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa0dc09cf60 0x7fa0dc1409b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:48716/0 (socket says 172.21.15.86:48716) 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 1 -- 172.21.15.86:0/703484870 learned_addr learned my addr 172.21.15.86:0/703484870 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): get_auth_request con 0x7fa0dc09cf60 auth_method 0 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:56.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:56.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.838+0000 7fa0dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:56.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0daffd640 1 -- 172.21.15.86:0/703484870 >> v1:172.21.15.203:6789/0 conn(0x7fa0dc0a5b60 legacy=0x7fa0dc13d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:56.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 10 monclient(hunting): handle_auth_done global_id 54322 payload 274 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 1 -- 172.21.15.86:0/703484870 >> v1:172.21.15.203:6789/0 conn(0x7fa0dc0a5b60 legacy=0x7fa0dc13d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 1 -- 172.21.15.86:0/703484870 >> v1:172.21.15.86:6789/0 conn(0x7fa0dc0a5070 legacy=0x7fa0dc0a5450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 1 monclient: found mon.noname-c 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:34:56.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.842+0000 7fa0dbfff640 1 -- 172.21.15.86:0/703484870 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc09c2a0 con 0x7fa0dc09cf60 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0dbfff640 1 -- 172.21.15.86:0/703484870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 msgr2=0x7fa0dc1409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0dbfff640 1 --2- 172.21.15.86:0/703484870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1409b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009ed0 tx=0x7fa0d401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2821271c90999033 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 1 -- 172.21.15.86:0/703484870 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40a5800 con 0x7fa0dc09cf60 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: dump: 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:56.850 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: _finish_auth 0 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0e2203640 5 monclient: authenticate success, global_id 54322 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 1 -- 172.21.15.86:0/703484870 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d40a59a0 con 0x7fa0dc09cf60 2024-09-17T15:34:56.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0e2203640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0e2203640 10 monclient: get_monmap_and_config success 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 1 -- 172.21.15.86:0/703484870 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d40a5d70 con 0x7fa0dc09cf60 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0e2203640 4 set_mon_vals no callback set 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.846+0000 7fa0da7fc640 10 monclient: dump: 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:56.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:56.853 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: shutdown 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- 172.21.15.86:0/703484870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 msgr2=0x7fa0dc1409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 --2- 172.21.15.86:0/703484870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1409b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4009ed0 tx=0x7fa0d401c940 comp rx=0 tx=0).stop 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- 172.21.15.86:0/703484870 shutdown_connections 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 --2- 172.21.15.86:0/703484870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1409b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:56.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- 172.21.15.86:0/703484870 >> 172.21.15.86:0/703484870 conn(0x7fa0dc0742a0 msgr2=0x7fa0dc0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- 172.21.15.86:0/703484870 shutdown_connections 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- 172.21.15.86:0/703484870 wait complete. 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: build_initial_monmap 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: monmap: 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:34:56.860 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 Processor -- start 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- start start 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: init 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: _reopen_session rank -1 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-b con 0x7fa0dc09cf60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-c con 0x7fa0dc0a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 0x7fa0dc13d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): picked mon.noname-a con 0x7fa0dc0a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:56.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc143890 con 0x7fa0dc0a5b60 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc143710 con 0x7fa0dc09cf60 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): start opening mon connection 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa0dc143a10 con 0x7fa0dc0a5070 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): _renew_subs 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0e2203640 10 monclient(hunting): authenticate will time out at 1429.818127s 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0dbfff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 msgr2=0x7fa0dc13d470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0dbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 0x7fa0dc13d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48932/0 (socket says 172.21.15.86:48932) 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 -- 172.21.15.86:0/214911025 learned_addr learned my addr 172.21.15.86:0/214911025 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): get_auth_request con 0x7fa0dc0a5070 auth_method 0 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:56.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0db7fe640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0db7fe640 10 monclient(hunting): get_auth_request con 0x7fa0dc09cf60 auth_method 0 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient(hunting): handle_auth_done global_id 74240 payload 824 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 -- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 msgr2=0x7fa0dc1cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 -- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 msgr2=0x7fa0dc13d470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:34:56.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 0x7fa0dc13d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient: _finish_hunting 0 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 monclient: found mon.noname-c 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.850+0000 7fa0daffd640 1 -- 172.21.15.86:0/214911025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc13faf0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0daffd640 1 -- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 msgr2=0x7fa0dc138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0daffd640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c500 tx=0x7fa0d000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0017070 con 0x7fa0dc0a5070 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: dump: 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:56.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: _finish_auth 0 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:34:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 5 monclient: authenticate success, global_id 74240 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d00057d0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _renew_subs 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d0005bc0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0dc1215e0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 10 monclient: dump: 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0d97fa640 4 set_mon_vals no callback set 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _renew_subs 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0dc13eef0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:56.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa0d001a070 con 0x7fa0dc0a5070 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0dc120fb0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 0x7fa0c803e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7fa0d004f360 con 0x7fa0dc0a5070 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0db7fe640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 0x7fa0c803e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.854+0000 7fa0db7fe640 10 monclient: get_auth_request con 0x7fa0c803bb40 auth_method 0 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.858+0000 7fa0db7fe640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 0x7fa0c803e000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc006fd0 tx=0x7fa0cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:34:56.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.858+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa0dc13faf0 con 0x7fa0dc0a5070 2024-09-17T15:34:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.858+0000 7fa0c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:34:56.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:56.858+0000 7fa0c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:34:57.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.086+0000 7fa0e2203640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:34:57.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.086+0000 7fa0e2203640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:57.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.086+0000 7fa0e2203640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:34:57.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.086+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa0dc121820 con 0x7fa0dc0a5070 2024-09-17T15:34:57.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0c3fff640 1 -- 172.21.15.86:0/214911025 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fa0dc121820 con 0x7fa0dc0a5070 2024-09-17T15:34:57.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:34:57.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:34:57.091 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":76,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:34:57.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 msgr2=0x7fa0c803e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:57.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 0x7fa0c803e000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc006fd0 tx=0x7fa0cc008040 comp rx=0 tx=0).stop 2024-09-17T15:34:57.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 10 monclient: shutdown 2024-09-17T15:34:57.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:34:57.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 msgr2=0x7fa0dc138220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa0d000c500 tx=0x7fa0d000c9c0 comp rx=0 tx=0).stop 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 shutdown_connections 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa0c803bb40 0x7fa0c803e000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0dc0a5b60 0x7fa0dc13d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0dc0a5070 0x7fa0dc138220 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 --2- 172.21.15.86:0/214911025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0dc09cf60 0x7fa0dc1cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 >> 172.21.15.86:0/214911025 conn(0x7fa0dc0742a0 msgr2=0x7fa0dc0a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 shutdown_connections 2024-09-17T15:34:57.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:34:57.090+0000 7fa0e2203640 1 -- 172.21.15.86:0/214911025 wait complete. 2024-09-17T15:35:00.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:00.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: get_monmap_and_config 2024-09-17T15:35:00.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: build_initial_monmap 2024-09-17T15:35:00.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: monmap: 2024-09-17T15:35:00.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:00.241521+0000 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:00.241521+0000 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:00.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 Processor -- start 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 -- start start 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: init 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0100689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): picked mon.noname-a con 0x7fb0100694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): picked mon.noname-b con 0x7fb01013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb010074470 con 0x7fb0100689d0 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb01009c900 con 0x7fb0100694c0 2024-09-17T15:35:00.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb01009cb90 con 0x7fb01013aff0 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00effd640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fb0100689d0 legacy=0x7fb010068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00e7fc640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 msgr2=0x7fb01006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb015596640 10 monclient(hunting): authenticate will time out at 1433.207107s 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00e7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50994/0 (socket says 172.21.15.86:50994) 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 learned_addr learned my addr 172.21.15.86:0/599211255 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request con 0x7fb01013aff0 auth_method 0 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:00.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:00.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.238+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:00.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.86:6789/0 conn(0x7fb0100689d0 legacy=0x7fb010068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:00.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 msgr2=0x7fb01006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:00.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:00.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:00.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request con 0x7fb01013aff0 auth_method 0 2024-09-17T15:35:00.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:00.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.442+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.86:6789/0 conn(0x7fb0100689d0 legacy=0x7fb010068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 msgr2=0x7fb01006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request con 0x7fb01013aff0 auth_method 0 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:00.846+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:01.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.86:6789/0 conn(0x7fb0100689d0 legacy=0x7fb010068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 msgr2=0x7fb01006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request con 0x7fb01013aff0 auth_method 0 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:01.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:01.650+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:03.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): tick 2024-09-17T15:35:03.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 monclient(hunting): continuing hunt 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 msgr2=0x7fb01013d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb01013aff0 0x7fb01013d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 msgr2=0x7fb01006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.86:3300/0 conn(0x7fb0100694c0 0x7fb01006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.86:6789/0 conn(0x7fb0100689d0 legacy=0x7fb010068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient: _add_conns ranks=[4,1] 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): picked mon.noname-c con 0x7faff00036b0 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faff0006220 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:03.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 -- 172.21.15.86:0/599211255 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01009ce70 con 0x7faff0006220 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 1 -- 172.21.15.86:0/599211255 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb01009c900 con 0x7faff00036b0 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00cff9640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 10 monclient(hunting): get_auth_request con 0x7faff00036b0 auth_method 0 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:03.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:03.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.242+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.203:6789/0 conn(0x7faff0006220 legacy=0x7faff00086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 10 monclient(hunting): get_auth_request con 0x7faff00036b0 auth_method 0 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:03.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:03.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:03.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.446+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:03.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 10 monclient(hunting): get_auth_request con 0x7faff00036b0 auth_method 0 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:03.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.850+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1672256383 0 0) 0x7fb01009ce70 con 0x7faff0006220 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 10 monclient(hunting): _negotiate 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 10 monclient(hunting): my global_id is 54328 2024-09-17T15:35:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.910+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faffc003530 con 0x7faff0006220 2024-09-17T15:35:03.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1941683145 0 0) 0x7faffc003530 con 0x7faff0006220 2024-09-17T15:35:03.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 msgr2=0x7faff0005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:03.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 --2- 172.21.15.86:0/599211255 >> v2:172.21.15.203:3300/0 conn(0x7faff00036b0 0x7faff0005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _finish_hunting 0 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010074470 con 0x7faff0006220 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _finish_auth 0 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fb000004660 con 0x7faff0006220 2024-09-17T15:35:03.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: dump: 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.921 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 >> v1:172.21.15.203:6789/0 conn(0x7faff0006220 legacy=0x7faff00086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 reap_dead start 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7faffc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient(hunting): picked mon.b con 0x7faffc003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 0x7faffc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient(hunting): picked mon.c con 0x7faffc004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 0x7faffc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient(hunting): picked mon.a con 0x7faffc0053d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.914+0000 7fb00dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb01009c900 con 0x7faffc0053d0 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faffc00c260 con 0x7faffc003a30 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faffc00c460 con 0x7faffc004810 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00dffb640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:35:03.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00f7fe640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 msgr2=0x7faffc00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 0x7faffc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 0x7faffc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7faffc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): get_auth_request con 0x7faffc004810 auth_method 0 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): get_auth_request con 0x7faffc003a30 auth_method 0 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient(hunting): handle_auth_done global_id 54328 payload 293 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 msgr2=0x7faffc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7faffc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 msgr2=0x7faffc00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:03.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 0x7faffc00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 monclient: found mon.c 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faffc003530 con 0x7faffc004810 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb00e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb015596640 5 monclient: authenticate success, global_id 54328 2024-09-17T15:35:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.918+0000 7fb015596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00e7fc640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 msgr2=0x7faffc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00e7fc640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 0x7faffc004c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb000004880 tx=0x7fb00001f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b0d6f8af1fa32e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00000c240 con 0x7faffc004810 2024-09-17T15:35:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:03.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: dump: 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:03.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb000006bc0 con 0x7faffc004810 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 4 set_mon_vals no callback set 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 1 -- 172.21.15.86:0/599211255 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000004930 con 0x7faffc004810 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb00dffb640 10 monclient: dump: 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:03.930 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:03.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 10 monclient: shutdown 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 -- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 msgr2=0x7faffc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 0x7faffc004c90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb000004880 tx=0x7fb00001f6a0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 -- 172.21.15.86:0/599211255 shutdown_connections 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc0053d0 0x7faffc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc004810 0x7faffc004c90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 --2- 172.21.15.86:0/599211255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7faffc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 -- 172.21.15.86:0/599211255 >> 172.21.15.86:0/599211255 conn(0x7fb0100a6440 msgr2=0x7fb0100a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 -- 172.21.15.86:0/599211255 shutdown_connections 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.926+0000 7fb015596640 1 -- 172.21.15.86:0/599211255 wait complete. 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient: build_initial_monmap 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient: monmap: 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:03.939 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 Processor -- start 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 -- start start 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient: init 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7fb0101ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): picked mon.noname-b con 0x7faffc003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 0x7fb010135530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): picked mon.noname-a con 0x7faffc004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:03.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): picked mon.noname-c con 0x7faffc0053d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb01009c240 con 0x7faffc004810 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb01009c0c0 con 0x7faffc003a30 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00e7fc640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 msgr2=0x7fb010135530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb01006f840 con 0x7faffc0053d0 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00e7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 0x7fb010135530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:03.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb015596640 10 monclient(hunting): authenticate will time out at 1436.899118s 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51044/0 (socket says 172.21.15.86:51044) 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 -- 172.21.15.86:0/901645997 learned_addr learned my addr 172.21.15.86:0/901645997 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request con 0x7faffc0053d0 auth_method 0 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:03.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00effd640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7fb0101ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00effd640 10 monclient(hunting): get_auth_request con 0x7faffc003a30 auth_method 0 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient(hunting): handle_auth_done global_id 74243 payload 824 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 -- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 msgr2=0x7fb0101ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7fb0101ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 -- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 msgr2=0x7fb010135530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 0x7fb010135530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:03.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.930+0000 7fb00f7fe640 1 -- 172.21.15.86:0/901645997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010121c00 con 0x7faffc0053d0 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb00f7fe640 1 -- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 msgr2=0x7fb010135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb00f7fe640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faff0006190 tx=0x7fb008012a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00801b030 con 0x7faffc0053d0 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: dump: 2024-09-17T15:35:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:03.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb008016970 con 0x7faffc0053d0 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00800f040 con 0x7faffc0053d0 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 5 monclient: authenticate success, global_id 74243 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _renew_subs 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb00d7fa640 4 set_mon_vals no callback set 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb01011fb80 con 0x7faffc0053d0 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 10 monclient: dump: 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _renew_subs 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb010120d50 con 0x7faffc0053d0 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb00801e030 con 0x7faffc0053d0 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.934+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0101217b0 con 0x7faffc0053d0 2024-09-17T15:35:03.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.938+0000 7fb0077fe640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 0x7faff003e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:03.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.938+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(240..240 src has 1..240) ==== 4783+0+0 (secure 0 0 0) 0x7fb0080505d0 con 0x7faffc0053d0 2024-09-17T15:35:03.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.938+0000 7fb00effd640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 0x7faff003e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:03.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.938+0000 7fb00effd640 10 monclient: get_auth_request con 0x7faff003c2e0 auth_method 0 2024-09-17T15:35:03.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.938+0000 7fb00effd640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 0x7faff003e7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faff8003b20 tx=0x7faff8005040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:03.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.942+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb0101217b0 con 0x7faffc0053d0 2024-09-17T15:35:03.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.942+0000 7fb0077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:03.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:03.942+0000 7fb0077fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:03.976 INFO:tasks.rados.rados.0.smithi203.stdout:1076: done (4 left) 2024-09-17T15:35:03.977 INFO:tasks.rados.rados.0.smithi203.stdout:1077: delete oid 20 current snap is 111 2024-09-17T15:35:03.977 INFO:tasks.rados.rados.0.smithi203.stdout:1072: finishing copy_from to smithi20328086-37 2024-09-17T15:35:03.977 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 185 (ObjNum 358 snap 104 seq_num 358) dirty exists 2024-09-17T15:35:03.977 INFO:tasks.rados.rados.0.smithi203.stdout:1071: expect (ObjNum 323 snap 93 seq_num 323) 2024-09-17T15:35:03.977 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 586595 2024-09-17T15:35:04.029 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 158200 2024-09-17T15:35:04.108 INFO:tasks.rados.rados.0.smithi203.stdout:1073: finishing write tid 1 to smithi20328086-31 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: finishing write tid 2 to smithi20328086-31 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: finishing write tid 3 to smithi20328086-31 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: finishing write tid 4 to smithi20328086-31 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: oid 31 updating version 0 to 107 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: oid 31 version 107 is already newer than 106 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: oid 31 version 107 is already newer than 105 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 107 (ObjNum 398 snap 111 seq_num 398) dirty exists 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1073: left oid 31 (ObjNum 398 snap 111 seq_num 398) 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1075: finishing copy_from to smithi20328086-17 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1075: got expected ENOENT (src dne) 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1075: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:35:04.109 INFO:tasks.rados.rados.0.smithi203.stdout:1071: done (4 left) 2024-09-17T15:35:04.110 INFO:tasks.rados.rados.0.smithi203.stdout:1072: done (3 left) 2024-09-17T15:35:04.110 INFO:tasks.rados.rados.0.smithi203.stdout:1073: done (2 left) 2024-09-17T15:35:04.110 INFO:tasks.rados.rados.0.smithi203.stdout:1075: done (1 left) 2024-09-17T15:35:04.110 INFO:tasks.rados.rados.0.smithi203.stdout:1077: done (0 left) 2024-09-17T15:35:04.110 INFO:tasks.rados.rados.0.smithi203.stdout:1078: snap_create 2024-09-17T15:35:04.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb015596640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:04.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb015596640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:04.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb015596640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb010068db0 con 0x7faffc0053d0 2024-09-17T15:35:04.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb0077fe640 1 -- 172.21.15.86:0/901645997 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fb008025030 con 0x7faffc0053d0 2024-09-17T15:35:04.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb0077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:04.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.198+0000 7fb0077fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:04.203 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 msgr2=0x7faff003e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 0x7faff003e7a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7faff8003b20 tx=0x7faff8005040 comp rx=0 tx=0).stop 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 10 monclient: shutdown 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 msgr2=0x7fb010135a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faff0006190 tx=0x7fb008012a30 comp rx=0 tx=0).stop 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 shutdown_connections 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faff003c2e0 0x7faff003e7a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faffc0053d0 0x7fb010135a70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faffc004810 0x7fb010135530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 --2- 172.21.15.86:0/901645997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc003a30 0x7fb0101ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 >> 172.21.15.86:0/901645997 conn(0x7fb0100a6440 msgr2=0x7fb01013a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:04.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 shutdown_connections 2024-09-17T15:35:04.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.202+0000 7fb015596640 1 -- 172.21.15.86:0/901645997 wait complete. 2024-09-17T15:35:04.221 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:35:04.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:35:04.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:35:04.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: get_monmap_and_config 2024-09-17T15:35:04.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: build_initial_monmap 2024-09-17T15:35:04.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: monmap: 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:04.344526+0000 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:04.344526+0000 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:04.343 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 Processor -- start 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 -- start start 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: init 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): picked mon.noname-b con 0x7f130c0a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f130c0a5350 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 0x7f130c13f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): picked mon.noname-a con 0x7f130c13d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130c09d430 con 0x7f130c0a5350 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f130c09c7f0 con 0x7f130c13d180 2024-09-17T15:35:04.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f130c0a68d0 con 0x7f130c0a4f70 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f131198e640 10 monclient(hunting): authenticate will time out at 1437.309993s 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130b7fe640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 msgr2=0x7f130c13f570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 0x7f130c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130a7fc640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f130c0a5350 legacy=0x7f130c0a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51060/0 (socket says 172.21.15.86:51060) 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 -- 172.21.15.86:0/2455310434 learned_addr learned my addr 172.21.15.86:0/2455310434 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): get_auth_request con 0x7f130c0a4f70 auth_method 0 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient(hunting): handle_auth_done global_id 74246 payload 274 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 -- 172.21.15.86:0/2455310434 >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 msgr2=0x7f130c13f570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 --2- 172.21.15.86:0/2455310434 >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 0x7f130c13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 -- 172.21.15.86:0/2455310434 >> v1:172.21.15.86:6789/0 conn(0x7f130c0a5350 legacy=0x7f130c0a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient: _finish_hunting 0 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 monclient: found mon.noname-b 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.342+0000 7f130affd640 1 -- 172.21.15.86:0/2455310434 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c0a64a0 con 0x7f130c0a4f70 2024-09-17T15:35:04.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f130affd640 1 -- 172.21.15.86:0/2455310434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 msgr2=0x7f130c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f130affd640 1 --2- 172.21.15.86:0/2455310434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f130c09e420 tx=0x7f12fc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f334e9bae0ea0ca9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 1 -- 172.21.15.86:0/2455310434 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0b0070 con 0x7f130c0a4f70 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: dump: 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: _finish_auth 0 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 1 -- 172.21.15.86:0/2455310434 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12fc0ab600 con 0x7f130c0a4f70 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 1 -- 172.21.15.86:0/2455310434 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0ab9d0 con 0x7f130c0a4f70 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f131198e640 5 monclient: authenticate success, global_id 74246 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f131198e640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f131198e640 4 set_mon_vals no callback set 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.346+0000 7f1309ffb640 10 monclient: dump: 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.352 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: shutdown 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- 172.21.15.86:0/2455310434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 msgr2=0x7f130c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- 172.21.15.86:0/2455310434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f130c09e420 tx=0x7f12fc01c3f0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- 172.21.15.86:0/2455310434 shutdown_connections 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- 172.21.15.86:0/2455310434 >> v2:172.21.15.86:3300/0 conn(0x7f130c13d180 0x7f130c13f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- 172.21.15.86:0/2455310434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c0a4f70 0x7f130c09cef0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- 172.21.15.86:0/2455310434 >> 172.21.15.86:0/2455310434 conn(0x7f130c0a7240 msgr2=0x7f130c0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- 172.21.15.86:0/2455310434 shutdown_connections 2024-09-17T15:35:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- 172.21.15.86:0/2455310434 wait complete. 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: build_initial_monmap 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: monmap: 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:04.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 Processor -- start 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- start start 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: init 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130c0a4f70 0x7f130c0a4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): picked mon.noname-a con 0x7f130c0a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f130c0a5350 0x7f130c1cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): picked mon.noname-b con 0x7f130c0a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 0x7f130c1ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): picked mon.noname-c con 0x7f130c13d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f130c1424e0 con 0x7f130c0a4f70 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f130c06f9a0 con 0x7f130c0a5350 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f130c142660 con 0x7f130c13d180 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f131198e640 10 monclient(hunting): authenticate will time out at 1437.317870s 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130affd640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130c0a4f70 msgr2=0x7f130c0a4310 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130c0a4f70 0x7f130c0a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 0x7f130c1ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 0x7f130c1ccdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51074/0 (socket says 172.21.15.86:51074) 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 learned_addr learned my addr 172.21.15.86:0/2276732583 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): get_auth_request con 0x7f130c13d180 auth_method 0 2024-09-17T15:35:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient(hunting): handle_auth_done global_id 74249 payload 824 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f130c0a5350 msgr2=0x7f130c1cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f130c0a5350 0x7f130c1cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130c0a4f70 msgr2=0x7f130c0a4310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f130c0a4f70 0x7f130c0a4310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:35:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.350+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c13d8d0 con 0x7f130c13d180 2024-09-17T15:35:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 msgr2=0x7f130c1ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130b7fe640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 0x7f130c1ccdd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f130400be60 tx=0x7f130400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304016630 con 0x7f130c13d180 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: dump: 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: _finish_auth 0 2024-09-17T15:35:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1304010030 con 0x7f130c13d180 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304016a10 con 0x7f130c13d180 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 5 monclient: authenticate success, global_id 74249 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 monclient: _renew_subs 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c0a32f0 con 0x7f130c13d180 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f13097fa640 4 set_mon_vals no callback set 2024-09-17T15:35:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 10 monclient: dump: 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 monclient: _renew_subs 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c0a25d0 con 0x7f130c13d180 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 0x7f130c131cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f130c1cd6f0 con 0x7f130c1cd310 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 0x7f130c131cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient: get_auth_request con 0x7f130c1cd310 auth_method 0 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 10 monclient(hunting): handle_auth_done global_id 74252 payload 824 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130affd640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f130c09cef0 con 0x7f130c1cd310 2024-09-17T15:35:04.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f13040153e0 con 0x7f130c13d180 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f12f803bb40 0x7f12f803e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(241..241 src has 1..241) ==== 4799+0+0 (secure 0 0 0) 0x7f13040509d0 con 0x7f130c13d180 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130a7fc640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f12f803bb40 0x7f12f803e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.354+0000 7f130a7fc640 10 monclient: get_auth_request con 0x7f12f803bb40 auth_method 0 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f130affd640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 msgr2=0x7f130c131cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f130affd640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 0x7f130c131cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f130c069c50 tx=0x7f12fc01ac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0bb020 con 0x7f130c1cd310 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 10 monclient: dump: 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f12fc0ab8f0 con 0x7f130c1cd310 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 msgr2=0x7f130c131cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 0x7f130c131cd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f130c069c50 tx=0x7f12fc01ac30 comp rx=0 tx=0).stop 2024-09-17T15:35:04.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 msgr2=0x7f130c131cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:04.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f12f3fff640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c1cd310 0x7f130c131cd0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f130c069c50 tx=0x7f12fc01ac30 comp rx=0 tx=0).stop 2024-09-17T15:35:04.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.358+0000 7f130a7fc640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f12f803bb40 0x7f12f803e000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f130c0a3190 tx=0x7f1300006c70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f131198e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f131198e640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 0x7f130c132620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f131198e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f130c132b60 con 0x7f130c132210 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 0x7f130c132620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient: get_auth_request con 0x7f130c132210 auth_method 0 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 10 monclient(hunting): handle_auth_done global_id 74255 payload 824 2024-09-17T15:35:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.362+0000 7f130affd640 1 -- 172.21.15.86:0/2276732583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f130c0a5ff0 con 0x7f130c132210 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f130affd640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 msgr2=0x7f130c132620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f130affd640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 0x7f130c132620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01c220 tx=0x7f12fc019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0bb050 con 0x7f130c132210 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 10 monclient: dump: 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f12fc003620 con 0x7f130c132210 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 msgr2=0x7f130c132620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 0x7f130c132620 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01c220 tx=0x7f12fc019190 comp rx=0 tx=0).stop 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 msgr2=0x7f130c132620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f12f3fff640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c132210 0x7f130c132620 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12fc01c220 tx=0x7f12fc019190 comp rx=0 tx=0).stop 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 78, 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 85154, 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:04.380 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.381 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.382 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:04.383 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.384 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:04.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:04.387 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.389 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.391 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.392 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:04.393 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f12f803bb40 msgr2=0x7f12f803e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f12f803bb40 0x7f12f803e000 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f130c0a3190 tx=0x7f1300006c70 comp rx=0 tx=0).stop 2024-09-17T15:35:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 10 monclient: shutdown 2024-09-17T15:35:04.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 msgr2=0x7f130c1ccdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 --2- 172.21.15.86:0/2276732583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f130c13d180 0x7f130c1ccdd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f130400be60 tx=0x7f130400bf50 comp rx=0 tx=0).stop 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 reap_dead start 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f130b7fe640 1 -- 172.21.15.86:0/2276732583 reap_dead start 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 shutdown_connections 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 >> 172.21.15.86:0/2276732583 conn(0x7f130c0a7240 msgr2=0x7f130c13c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 shutdown_connections 2024-09-17T15:35:04.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.366+0000 7f131198e640 1 -- 172.21.15.86:0/2276732583 wait complete. 2024-09-17T15:35:04.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:35:04.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:35:04.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:35:04.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: get_monmap_and_config 2024-09-17T15:35:04.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: build_initial_monmap 2024-09-17T15:35:04.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: monmap: 2024-09-17T15:35:04.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:04.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:04.512217+0000 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:04.512217+0000 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:04.512 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:04.513 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:04.513 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:04.513 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 Processor -- start 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 -- start start 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: init 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8308073c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8308134cd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83080a2d30 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8308070160 con 0x7f83080a2d30 2024-09-17T15:35:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8308070440 con 0x7f8308134cd0 2024-09-17T15:35:04.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8308135710 con 0x7f8308073c70 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830f1fb640 10 monclient(hunting): authenticate will time out at 1437.477887s 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830d798640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f83080a2d30 legacy=0x7f8308140060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51096/0 (socket says 172.21.15.86:51096) 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 -- 172.21.15.86:0/1844449772 learned_addr learned my addr 172.21.15.86:0/1844449772 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:04.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): get_auth_request con 0x7f8308073c70 auth_method 0 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient(hunting): handle_auth_done global_id 74258 payload 274 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 -- 172.21.15.86:0/1844449772 >> v1:172.21.15.203:6789/0 conn(0x7f8308134cd0 legacy=0x7f83080a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 -- 172.21.15.86:0/1844449772 >> v1:172.21.15.86:6789/0 conn(0x7f83080a2d30 legacy=0x7f8308140060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient: _finish_hunting 0 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 monclient: found mon.noname-b 2024-09-17T15:35:04.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:04.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.510+0000 7f830cf97640 1 -- 172.21.15.86:0/1844449772 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8308070770 con 0x7f8308073c70 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830cf97640 1 -- 172.21.15.86:0/1844449772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 msgr2=0x7f8308074050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830cf97640 1 --2- 172.21.15.86:0/1844449772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8300009eb0 tx=0x7f830001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d0519d93fbe879 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 1 -- 172.21.15.86:0/1844449772 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000a64e0 con 0x7f8308073c70 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:35:04.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: dump: 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:04.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 1 -- 172.21.15.86:0/1844449772 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83000a6680 con 0x7f8308073c70 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 5 monclient: authenticate success, global_id 74258 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 1 -- 172.21.15.86:0/1844449772 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000a6a50 con 0x7f8308073c70 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 4 set_mon_vals no callback set 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f83077fe640 10 monclient: dump: 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 10 monclient: shutdown 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 -- 172.21.15.86:0/1844449772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 msgr2=0x7f8308074050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 --2- 172.21.15.86:0/1844449772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8300009eb0 tx=0x7f830001c8f0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 -- 172.21.15.86:0/1844449772 shutdown_connections 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 --2- 172.21.15.86:0/1844449772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308073c70 0x7f8308074050 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 -- 172.21.15.86:0/1844449772 >> 172.21.15.86:0/1844449772 conn(0x7f830806d230 msgr2=0x7f83080a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.514+0000 7f830f1fb640 1 -- 172.21.15.86:0/1844449772 shutdown_connections 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 -- 172.21.15.86:0/1844449772 wait complete. 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient: build_initial_monmap 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient: monmap: 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:04.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 Processor -- start 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 -- start start 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient: init 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308073c70 0x7f8308131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8308073c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83080a2d30 0x7f8308131c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-a con 0x7f83080a2d30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8308134cd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8308143fa0 con 0x7f83080a2d30 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83080a0320 con 0x7f8308073c70 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8308144120 con 0x7f8308134cd0 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830f1fb640 10 monclient(hunting): authenticate will time out at 1437.485386s 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f8307fff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83080a2d30 msgr2=0x7f8308131c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f8307fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83080a2d30 0x7f8308131c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51110/0 (socket says 172.21.15.86:51110) 2024-09-17T15:35:04.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 learned_addr learned my addr 172.21.15.86:0/3674309403 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): get_auth_request con 0x7f8308134cd0 auth_method 0 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830cf97640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308073c70 0x7f8308131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830cf97640 10 monclient(hunting): get_auth_request con 0x7f8308073c70 auth_method 0 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830cf97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830cf97640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830cf97640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient(hunting): handle_auth_done global_id 74261 payload 824 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308073c70 msgr2=0x7f8308131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8308073c70 0x7f8308131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83080a2d30 msgr2=0x7f8308131c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83080a2d30 0x7f8308131c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient: _finish_hunting 0 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 monclient: found mon.noname-c 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.518+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f830813e6a0 con 0x7f8308134cd0 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 msgr2=0x7f830813a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830d798640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b410 tx=0x7f82f800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f8017070 con 0x7f8308134cd0 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: dump: 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: _finish_auth 0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: _check_auth_tickets 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f80060d0 con 0x7f8308134cd0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 5 monclient: authenticate success, global_id 74261 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 monclient: _renew_subs 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f830813c340 con 0x7f8308134cd0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f80064a0 con 0x7f8308134cd0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:04.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 10 monclient: dump: 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 monclient: _renew_subs 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:04.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f8306ffd640 4 set_mon_vals no callback set 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8308132180 con 0x7f8308134cd0 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 0x7f830813b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f830813df10 con 0x7f830813f0d0 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f82f801a070 con 0x7f8308134cd0 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 0x7f830813b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82e803bb60 0x7f82e803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient: get_auth_request con 0x7f830813f0d0 auth_method 0 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(241..241 src has 1..241) ==== 4799+0+0 (secure 0 0 0) 0x7f82f804f190 con 0x7f8308134cd0 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f8307fff640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82e803bb60 0x7f82e803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f8307fff640 10 monclient: get_auth_request con 0x7f82e803bb60 auth_method 0 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 10 monclient(hunting): handle_auth_done global_id 54337 payload 824 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.522+0000 7f830cf97640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8308074050 con 0x7f830813f0d0 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f830cf97640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 msgr2=0x7f830813b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f830cf97640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 0x7f830813b580 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009e80 tx=0x7f8300019690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f8307fff640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82e803bb60 0x7f82e803e020 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f830813d7f0 tx=0x7f82f0007a20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000be020 con 0x7f830813f0d0 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 10 monclient: dump: 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83000a6ca0 con 0x7f830813f0d0 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 msgr2=0x7f830813b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 0x7f830813b580 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009e80 tx=0x7f8300019690 comp rx=0 tx=0).stop 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 msgr2=0x7f830813b580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.526+0000 7f83057fa640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830813f0d0 0x7f830813b580 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009e80 tx=0x7f8300019690 comp rx=0 tx=0).stop 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.530+0000 7f830f1fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:04.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.530+0000 7f830f1fb640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 0x7f830802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.530+0000 7f830f1fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.530+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83080a6670 con 0x7f830802d450 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 0x7f830802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient: get_auth_request con 0x7f830802d450 auth_method 0 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 10 monclient(hunting): handle_auth_done global_id 54340 payload 824 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 1 -- 172.21.15.86:0/3674309403 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8308136fa0 con 0x7f830802d450 2024-09-17T15:35:04.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 msgr2=0x7f830802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f830cf97640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 0x7f830802d860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8300000f90 tx=0x7f83000a9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83000be040 con 0x7f830802d450 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f83057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f83057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.534+0000 7f83057fa640 10 monclient: dump: 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:04.545 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f83000c9070 con 0x7f830802d450 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 msgr2=0x7f830802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 0x7f830802d860 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8300000f90 tx=0x7f83000a9000 comp rx=0 tx=0).stop 2024-09-17T15:35:04.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 msgr2=0x7f830802d860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f83057fa640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f830802d450 0x7f830802d860 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 78, 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 179969, 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:04.547 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.548 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:04.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.550 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:04.551 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:04.552 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:04.553 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:04.553 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:04.553 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:04.556 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:04.556 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:04.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:04.558 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:04.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.563 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:35:04.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:04.565 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:04.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82e803bb60 msgr2=0x7f82e803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82e803bb60 0x7f82e803e020 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f830813d7f0 tx=0x7f82f0007a20 comp rx=0 tx=0).stop 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 10 monclient: shutdown 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 msgr2=0x7f830813a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f82f800b410 tx=0x7f82f800b8d0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830d798640 1 -- 172.21.15.86:0/3674309403 reap_dead start 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 shutdown_connections 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 --2- 172.21.15.86:0/3674309403 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8308134cd0 0x7f830813a880 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 >> 172.21.15.86:0/3674309403 conn(0x7f830806d230 msgr2=0x7f8308072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 shutdown_connections 2024-09-17T15:35:04.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:04.538+0000 7f830f1fb640 1 -- 172.21.15.86:0/3674309403 wait complete. 2024-09-17T15:35:04.568 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:35:04.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:35:04.982 INFO:tasks.rados.rados.0.smithi203.stdout:1078: done (0 left) 2024-09-17T15:35:04.982 INFO:tasks.rados.rados.0.smithi203.stdout:1079: write oid 45 current snap is 112 2024-09-17T15:35:04.982 INFO:tasks.rados.rados.0.smithi203.stdout:1079: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-17T15:35:05.000 INFO:tasks.rados.rados.0.smithi203.stdout:1079: writing smithi20328086-45 from 722800 to 1272756 tid 1 2024-09-17T15:35:05.024 INFO:tasks.rados.rados.0.smithi203.stdout:1079: writing smithi20328086-45 from 2040562 to 2722800 tid 2 2024-09-17T15:35:05.024 INFO:tasks.rados.rados.0.smithi203.stdout:1080: copy_from oid 43 from oid 30 current snap is 112 2024-09-17T15:35:05.024 INFO:tasks.rados.rados.0.smithi203.stdout:1079: finishing write tid 1 to smithi20328086-45 2024-09-17T15:35:05.024 INFO:tasks.rados.rados.0.smithi203.stdout:1081: write oid 40 current snap is 112 2024-09-17T15:35:05.024 INFO:tasks.rados.rados.0.smithi203.stdout:1081: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-17T15:35:05.043 INFO:tasks.rados.rados.0.smithi203.stdout:1081: writing smithi20328086-40 from 739607 to 1364812 tid 1 2024-09-17T15:35:05.055 INFO:tasks.rados.rados.0.smithi203.stdout:1081: writing smithi20328086-40 from 1982691 to 2482148 tid 2 2024-09-17T15:35:05.055 INFO:tasks.rados.rados.0.smithi203.stdout:1082: snap_remove snap 111 2024-09-17T15:35:05.055 INFO:tasks.rados.rados.0.smithi203.stdout:1079: finishing write tid 2 to smithi20328086-45 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1082: done (3 left) 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1083: snap_create 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1079: finishing write tid 3 to smithi20328086-45 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1079: finishing write tid 4 to smithi20328086-45 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1079: oid 45 updating version 0 to 256 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1079: oid 45 updating version 256 to 257 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:1079: oid 45 updating version 257 to 258 2024-09-17T15:35:05.998 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 258 (ObjNum 400 snap 112 seq_num 400) dirty exists 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1079: left oid 45 (ObjNum 400 snap 112 seq_num 400) 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1080: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: finishing write tid 1 to smithi20328086-40 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1080: finishing copy_from to smithi20328086-43 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 259 (ObjNum 394 snap 110 seq_num 394) dirty exists 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: finishing write tid 2 to smithi20328086-40 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: finishing write tid 3 to smithi20328086-40 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: finishing write tid 4 to smithi20328086-40 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: oid 40 updating version 0 to 448 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: oid 40 version 448 is already newer than 446 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:1081: oid 40 version 448 is already newer than 447 2024-09-17T15:35:05.999 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 448 (ObjNum 401 snap 112 seq_num 401) dirty exists 2024-09-17T15:35:06.000 INFO:tasks.rados.rados.0.smithi203.stdout:1081: left oid 40 (ObjNum 401 snap 112 seq_num 401) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1079: done (3 left) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1080: done (2 left) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1081: done (1 left) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1083: done (0 left) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1084: read oid 45 snap -1 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1084: expect (ObjNum 400 snap 112 seq_num 400) 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1085: write oid 6 current snap is 113 2024-09-17T15:35:07.001 INFO:tasks.rados.rados.0.smithi203.stdout:1085: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-17T15:35:07.018 INFO:tasks.rados.rados.0.smithi203.stdout:1085: writing smithi20328086-6 from 756414 to 1456868 tid 1 2024-09-17T15:35:07.034 INFO:tasks.rados.rados.0.smithi203.stdout:1085: writing smithi20328086-6 from 2041173 to 2756414 tid 2 2024-09-17T15:35:07.034 INFO:tasks.rados.rados.0.smithi203.stdout:1086: write oid 11 current snap is 113 2024-09-17T15:35:07.034 INFO:tasks.rados.rados.0.smithi203.stdout:1086: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-17T15:35:07.050 INFO:tasks.rados.rados.0.smithi203.stdout:1086: writing smithi20328086-11 from 773221 to 1265277 tid 1 2024-09-17T15:35:07.061 INFO:tasks.rados.rados.0.smithi203.stdout:1086: writing smithi20328086-11 from 1816008 to 2319134 tid 2 2024-09-17T15:35:07.061 INFO:tasks.rados.rados.0.smithi203.stdout:1087: read oid 30 snap -1 2024-09-17T15:35:07.061 INFO:tasks.rados.rados.0.smithi203.stdout:1087: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:35:07.061 INFO:tasks.rados.rados.0.smithi203.stdout:1088: copy_from oid 48 from oid 15 current snap is 113 2024-09-17T15:35:07.061 INFO:tasks.rados.rados.0.smithi203.stdout:1084: expect (ObjNum 400 snap 112 seq_num 400) 2024-09-17T15:35:07.182 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: finishing write tid 1 to smithi20328086-6 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1084: done (4 left) 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1089: snap_create 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: finishing write tid 2 to smithi20328086-6 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: finishing write tid 3 to smithi20328086-6 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: finishing write tid 4 to smithi20328086-6 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: oid 6 updating version 0 to 407 2024-09-17T15:35:07.231 INFO:tasks.rados.rados.0.smithi203.stdout:1085: oid 6 updating version 407 to 408 2024-09-17T15:35:07.232 INFO:tasks.rados.rados.0.smithi203.stdout:1085: oid 6 version 408 is already newer than 406 2024-09-17T15:35:07.232 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 408 (ObjNum 402 snap 113 seq_num 402) dirty exists 2024-09-17T15:35:07.232 INFO:tasks.rados.rados.0.smithi203.stdout:1085: left oid 6 (ObjNum 402 snap 113 seq_num 402) 2024-09-17T15:35:07.232 INFO:tasks.rados.rados.0.smithi203.stdout:1086: finishing write tid 1 to smithi20328086-11 2024-09-17T15:35:07.232 INFO:tasks.rados.rados.0.smithi203.stdout:1087: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:35:07.234 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1859958 2024-09-17T15:35:07.275 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: finishing write tid 2 to smithi20328086-11 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: finishing write tid 3 to smithi20328086-11 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: finishing write tid 4 to smithi20328086-11 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: oid 11 updating version 0 to 453 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: oid 11 version 453 is already newer than 452 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: oid 11 updating version 453 to 454 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 454 (ObjNum 403 snap 113 seq_num 403) dirty exists 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1086: left oid 11 (ObjNum 403 snap 113 seq_num 403) 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1088: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:35:07.303 INFO:tasks.rados.rados.0.smithi203.stdout:1088: finishing copy_from to smithi20328086-48 2024-09-17T15:35:07.304 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 256 (ObjNum 365 snap 105 seq_num 365) dirty exists 2024-09-17T15:35:08.015 INFO:tasks.rados.rados.0.smithi203.stdout:1085: done (4 left) 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1086: done (3 left) 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1087: done (2 left) 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1088: done (1 left) 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1089: done (0 left) 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1090: write (excl) oid 42 current snap is 114 2024-09-17T15:35:08.016 INFO:tasks.rados.rados.0.smithi203.stdout:1090: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-17T15:35:08.031 INFO:tasks.rados.rados.0.smithi203.stdout:1090: writing smithi20328086-42 from 790028 to 1357333 tid 1 2024-09-17T15:35:08.046 INFO:tasks.rados.rados.0.smithi203.stdout:1090: writing smithi20328086-42 from 1874490 to 2637627 tid 2 2024-09-17T15:35:08.047 INFO:tasks.rados.rados.0.smithi203.stdout:1091: rollback oid 39 current snap is 114 2024-09-17T15:35:08.047 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 39 to 112 2024-09-17T15:35:08.047 INFO:tasks.rados.rados.0.smithi203.stdout:1090: finishing write tid 1 to smithi20328086-42 2024-09-17T15:35:08.047 INFO:tasks.rados.rados.0.smithi203.stdout:1092: snap_remove snap 108 2024-09-17T15:35:09.022 INFO:tasks.rados.rados.0.smithi203.stdout:1092: done (2 left) 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1093: read oid 24 snap 104 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1093: expect deleted 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1094: write (excl) oid 36 current snap is 114 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: finishing write tid 2 to smithi20328086-42 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: finishing write tid 3 to smithi20328086-42 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: finishing write tid 4 to smithi20328086-42 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: oid 42 updating version 0 to 321 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: oid 42 version 321 is already newer than 320 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: oid 42 updating version 321 to 322 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 322 (ObjNum 404 snap 114 seq_num 404) dirty exists 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1090: left oid 42 (ObjNum 404 snap 114 seq_num 404) 2024-09-17T15:35:09.023 INFO:tasks.rados.rados.0.smithi203.stdout:1091: finishing rollback tid 0 to smithi20328086-39 2024-09-17T15:35:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:1091: finishing rollback tid 1 to smithi20328086-39 2024-09-17T15:35:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:1091: finishing rollback tid 2 to smithi20328086-39 2024-09-17T15:35:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 355 (ObjNum 344 snap 102 seq_num 344) dirty exists 2024-09-17T15:35:09.024 INFO:tasks.rados.rados.0.smithi203.stdout:1094: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-17T15:35:09.040 INFO:tasks.rados.rados.0.smithi203.stdout:1094: writing smithi20328086-36 from 406835 to 1049389 tid 1 2024-09-17T15:35:09.059 INFO:tasks.rados.rados.0.smithi203.stdout:1094: writing smithi20328086-36 from 1532972 to 2039767 tid 2 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1094: writing smithi20328086-36 from 2772312 to 2806835 tid 3 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1090: done (3 left) 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1091: done (2 left) 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1095: read oid 18 snap 113 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1095: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1096: read oid 21 snap -1 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1096: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1097: write oid 19 current snap is 114 2024-09-17T15:35:09.073 INFO:tasks.rados.rados.0.smithi203.stdout:1097: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-17T15:35:09.087 INFO:tasks.rados.rados.0.smithi203.stdout:1097: writing smithi20328086-19 from 423642 to 1141445 tid 1 2024-09-17T15:35:09.102 INFO:tasks.rados.rados.0.smithi203.stdout:1097: writing smithi20328086-19 from 1875101 to 2641907 tid 2 2024-09-17T15:35:09.102 INFO:tasks.rados.rados.0.smithi203.stdout:1098: snap_remove snap 110 2024-09-17T15:35:09.615 INFO:tasks.rados.rados.0.smithi203.stdout:1098: done (5 left) 2024-09-17T15:35:09.615 INFO:tasks.rados.rados.0.smithi203.stdout:1099: rollback oid 22 current snap is 114 2024-09-17T15:35:09.615 INFO:tasks.rados.rados.0.smithi203.stdout:1094: finishing write tid 1 to smithi20328086-36 2024-09-17T15:35:09.615 INFO:tasks.rados.rados.0.smithi203.stdout:1094: finishing write tid 2 to smithi20328086-36 2024-09-17T15:35:09.615 INFO:tasks.rados.rados.0.smithi203.stdout:1095: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:35:09.671 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 963343 2024-09-17T15:35:09.695 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:09.733 INFO:tasks.rados.rados.0.smithi203.stdout:1094: finishing write tid 3 to smithi20328086-36 2024-09-17T15:35:09.733 INFO:tasks.rados.rados.0.smithi203.stdout:1096: expect (ObjNum 326 snap 94 seq_num 326) 2024-09-17T15:35:09.733 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:09.849 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2694098 2024-09-17T15:35:09.861 INFO:tasks.rados.rados.0.smithi203.stdout:1094: finishing write tid 4 to smithi20328086-36 2024-09-17T15:35:09.861 INFO:tasks.rados.rados.0.smithi203.stdout:1094: finishing write tid 5 to smithi20328086-36 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1094: oid 36 updating version 0 to 463 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1094: oid 36 version 463 is already newer than 462 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1094: oid 36 updating version 463 to 464 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1094: oid 36 version 464 is already newer than 459 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 464 (ObjNum 405 snap 114 seq_num 405) dirty exists 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1094: left oid 36 (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: finishing write tid 1 to smithi20328086-19 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: finishing write tid 2 to smithi20328086-19 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: finishing write tid 3 to smithi20328086-19 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: finishing write tid 4 to smithi20328086-19 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: oid 19 updating version 0 to 364 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: oid 19 updating version 364 to 365 2024-09-17T15:35:09.862 INFO:tasks.rados.rados.0.smithi203.stdout:1097: oid 19 version 365 is already newer than 363 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 365 (ObjNum 406 snap 114 seq_num 406) dirty exists 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1097: left oid 19 (ObjNum 406 snap 114 seq_num 406) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 22 to 112 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1093: done (5 left) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1094: done (4 left) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1095: done (3 left) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1096: done (2 left) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1097: done (1 left) 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1100: write oid 24 current snap is 114 2024-09-17T15:35:09.863 INFO:tasks.rados.rados.0.smithi203.stdout:1100: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-17T15:35:09.880 INFO:tasks.rados.rados.0.smithi203.stdout:1100: writing smithi20328086-24 from 440449 to 1233501 tid 1 2024-09-17T15:35:09.899 INFO:tasks.rados.rados.0.smithi203.stdout:1100: writing smithi20328086-24 from 1933583 to 2444047 tid 2 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1101: rollback oid 15 current snap is 114 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 100 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1099: finishing rollback tid 0 to smithi20328086-22 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1099: finishing rollback tid 1 to smithi20328086-22 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1099: finishing rollback tid 2 to smithi20328086-22 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 117 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1100: finishing write tid 1 to smithi20328086-24 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1099: done (2 left) 2024-09-17T15:35:09.900 INFO:tasks.rados.rados.0.smithi203.stdout:1102: snap_create 2024-09-17T15:35:09.906 INFO:tasks.rados.rados.0.smithi203.stdout:1101: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:35:09.907 INFO:tasks.rados.rados.0.smithi203.stdout:1101: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:35:09.910 INFO:tasks.rados.rados.0.smithi203.stdout:1100: finishing write tid 2 to smithi20328086-24 2024-09-17T15:35:09.910 INFO:tasks.rados.rados.0.smithi203.stdout:1100: finishing write tid 3 to smithi20328086-24 2024-09-17T15:35:09.910 INFO:tasks.rados.rados.0.smithi203.stdout:1100: finishing write tid 4 to smithi20328086-24 2024-09-17T15:35:09.910 INFO:tasks.rados.rados.0.smithi203.stdout:1100: oid 24 updating version 0 to 376 2024-09-17T15:35:09.910 INFO:tasks.rados.rados.0.smithi203.stdout:1100: oid 24 updating version 376 to 377 2024-09-17T15:35:09.911 INFO:tasks.rados.rados.0.smithi203.stdout:1100: oid 24 updating version 377 to 378 2024-09-17T15:35:09.911 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 378 (ObjNum 407 snap 114 seq_num 407) dirty exists 2024-09-17T15:35:09.911 INFO:tasks.rados.rados.0.smithi203.stdout:1100: left oid 24 (ObjNum 407 snap 114 seq_num 407) 2024-09-17T15:35:09.912 INFO:tasks.rados.rados.0.smithi203.stdout:1101: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:35:09.912 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 262 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:35:10.620 INFO:tasks.rados.rados.0.smithi203.stdout:1100: done (2 left) 2024-09-17T15:35:10.620 INFO:tasks.rados.rados.0.smithi203.stdout:1101: done (1 left) 2024-09-17T15:35:10.620 INFO:tasks.rados.rados.0.smithi203.stdout:1102: done (0 left) 2024-09-17T15:35:10.620 INFO:tasks.rados.rados.0.smithi203.stdout:1103: snap_remove snap 100 2024-09-17T15:35:11.623 INFO:tasks.rados.rados.0.smithi203.stdout:1103: done (0 left) 2024-09-17T15:35:11.623 INFO:tasks.rados.rados.0.smithi203.stdout:1104: rollback oid 20 current snap is 115 2024-09-17T15:35:11.623 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 114 2024-09-17T15:35:11.623 INFO:tasks.rados.rados.0.smithi203.stdout:1105: snap_remove snap 113 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:1105: done (1 left) 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:1106: copy_from oid 12 from oid 44 current snap is 115 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:1104: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 398 (ObjNum 129 snap 0 seq_num 3605771176) dirty dne 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:1104: done (1 left) 2024-09-17T15:35:12.626 INFO:tasks.rados.rados.0.smithi203.stdout:1107: delete oid 37 current snap is 115 2024-09-17T15:35:12.641 INFO:tasks.rados.rados.0.smithi203.stdout:1107: done (1 left) 2024-09-17T15:35:12.641 INFO:tasks.rados.rados.0.smithi203.stdout:1108: rollback oid 42 current snap is 115 2024-09-17T15:35:12.641 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 112 2024-09-17T15:35:12.641 INFO:tasks.rados.rados.0.smithi203.stdout:1109: write (excl) oid 21 current snap is 115 2024-09-17T15:35:12.641 INFO:tasks.rados.rados.0.smithi203.stdout:1109: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-17T15:35:12.658 INFO:tasks.rados.rados.0.smithi203.stdout:1109: writing smithi20328086-21 from 474063 to 1017613 tid 1 2024-09-17T15:35:12.676 INFO:tasks.rados.rados.0.smithi203.stdout:1109: writing smithi20328086-21 from 1650547 to 2164680 tid 2 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1109: writing smithi20328086-21 from 2765651 to 2874063 tid 3 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1110: snap_create 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1108: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1108: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 327 (ObjNum 129 snap 0 seq_num 3605870968) dirty dne 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1106: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:1106: finishing copy_from to smithi20328086-12 2024-09-17T15:35:12.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 182 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:35:12.689 INFO:tasks.rados.rados.0.smithi203.stdout:1109: finishing write tid 1 to smithi20328086-21 2024-09-17T15:35:12.689 INFO:tasks.rados.rados.0.smithi203.stdout:1109: finishing write tid 2 to smithi20328086-21 2024-09-17T15:35:12.693 INFO:tasks.rados.rados.0.smithi203.stdout:1109: finishing write tid 3 to smithi20328086-21 2024-09-17T15:35:12.694 INFO:tasks.rados.rados.0.smithi203.stdout:1109: finishing write tid 4 to smithi20328086-21 2024-09-17T15:35:12.694 INFO:tasks.rados.rados.0.smithi203.stdout:1109: finishing write tid 5 to smithi20328086-21 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1109: oid 21 updating version 0 to 474 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1109: oid 21 updating version 474 to 476 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1109: oid 21 version 476 is already newer than 475 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1109: oid 21 updating version 476 to 477 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 477 (ObjNum 409 snap 115 seq_num 409) dirty exists 2024-09-17T15:35:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1109: left oid 21 (ObjNum 409 snap 115 seq_num 409) 2024-09-17T15:35:13.648 INFO:tasks.rados.rados.0.smithi203.stdout:1106: done (3 left) 2024-09-17T15:35:13.648 INFO:tasks.rados.rados.0.smithi203.stdout:1108: done (2 left) 2024-09-17T15:35:13.648 INFO:tasks.rados.rados.0.smithi203.stdout:1109: done (1 left) 2024-09-17T15:35:13.648 INFO:tasks.rados.rados.0.smithi203.stdout:1110: done (0 left) 2024-09-17T15:35:13.649 INFO:tasks.rados.rados.0.smithi203.stdout:1111: copy_from oid 46 from oid 31 current snap is 116 2024-09-17T15:35:13.649 INFO:tasks.rados.rados.0.smithi203.stdout:1112: read oid 1 snap 115 2024-09-17T15:35:13.649 INFO:tasks.rados.rados.0.smithi203.stdout:1112: expect (ObjNum 391 snap 110 seq_num 391) 2024-09-17T15:35:13.649 INFO:tasks.rados.rados.0.smithi203.stdout:1113: snap_create 2024-09-17T15:35:13.661 INFO:tasks.rados.rados.0.smithi203.stdout:1112: expect (ObjNum 391 snap 110 seq_num 391) 2024-09-17T15:35:13.786 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 685016 2024-09-17T15:35:13.826 INFO:tasks.rados.rados.0.smithi203.stdout:1111: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:35:13.827 INFO:tasks.rados.rados.0.smithi203.stdout:1111: finishing copy_from to smithi20328086-46 2024-09-17T15:35:13.827 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 107 (ObjNum 398 snap 111 seq_num 398) dirty exists 2024-09-17T15:35:14.657 INFO:tasks.rados.rados.0.smithi203.stdout:1111: done (2 left) 2024-09-17T15:35:14.657 INFO:tasks.rados.rados.0.smithi203.stdout:1112: done (1 left) 2024-09-17T15:35:14.657 INFO:tasks.rados.rados.0.smithi203.stdout:1113: done (0 left) 2024-09-17T15:35:14.657 INFO:tasks.rados.rados.0.smithi203.stdout:1114: snap_remove snap 116 2024-09-17T15:35:15.659 INFO:tasks.rados.rados.0.smithi203.stdout:1114: done (0 left) 2024-09-17T15:35:15.659 INFO:tasks.rados.rados.0.smithi203.stdout:1115: copy_from oid 34 from oid 26 current snap is 117 2024-09-17T15:35:15.660 INFO:tasks.rados.rados.0.smithi203.stdout:1116: copy_from oid 41 from oid 15 current snap is 117 2024-09-17T15:35:15.665 INFO:tasks.rados.rados.0.smithi203.stdout:1117: write oid 45 current snap is 117 2024-09-17T15:35:15.665 INFO:tasks.rados.rados.0.smithi203.stdout:1117: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-17T15:35:15.679 INFO:tasks.rados.rados.0.smithi203.stdout:1117: writing smithi20328086-45 from 490870 to 1109669 tid 1 2024-09-17T15:35:15.700 INFO:tasks.rados.rados.0.smithi203.stdout:1117: writing smithi20328086-45 from 1592676 to 2366820 tid 2 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1117: writing smithi20328086-45 from 2793074 to 2890870 tid 3 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1118: delete oid 20 current snap is 117 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1115: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1116: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1115: finishing copy_from to smithi20328086-34 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 425 (ObjNum 370 snap 105 seq_num 370) dirty exists 2024-09-17T15:35:15.709 INFO:tasks.rados.rados.0.smithi203.stdout:1116: finishing copy_from to smithi20328086-41 2024-09-17T15:35:15.710 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 262 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:35:15.710 INFO:tasks.rados.rados.0.smithi203.stdout:1117: finishing write tid 1 to smithi20328086-45 2024-09-17T15:35:15.710 INFO:tasks.rados.rados.0.smithi203.stdout:1117: finishing write tid 2 to smithi20328086-45 2024-09-17T15:35:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:1115: done (3 left) 2024-09-17T15:35:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:1116: done (2 left) 2024-09-17T15:35:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:1118: done (1 left) 2024-09-17T15:35:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:1119: snap_remove snap 115 2024-09-17T15:35:16.674 INFO:tasks.rados.rados.0.smithi203.stdout:1119: done (1 left) 2024-09-17T15:35:16.674 INFO:tasks.rados.rados.0.smithi203.stdout:1120: snap_remove snap 112 2024-09-17T15:35:17.678 INFO:tasks.rados.rados.0.smithi203.stdout:1120: done (1 left) 2024-09-17T15:35:17.678 INFO:tasks.rados.rados.0.smithi203.stdout:1121: read oid 49 snap -1 2024-09-17T15:35:17.678 INFO:tasks.rados.rados.0.smithi203.stdout:1121: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:35:17.678 INFO:tasks.rados.rados.0.smithi203.stdout:1122: write (excl) oid 39 current snap is 117 2024-09-17T15:35:17.678 INFO:tasks.rados.rados.0.smithi203.stdout:1122: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-17T15:35:17.694 INFO:tasks.rados.rados.0.smithi203.stdout:1122: writing smithi20328086-39 from 524484 to 1010134 tid 1 2024-09-17T15:35:17.712 INFO:tasks.rados.rados.0.smithi203.stdout:1122: writing smithi20328086-39 from 1425993 to 2087453 tid 2 2024-09-17T15:35:17.725 INFO:tasks.rados.rados.0.smithi203.stdout:1122: writing smithi20328086-39 from 2847920 to 2924484 tid 3 2024-09-17T15:35:17.726 INFO:tasks.rados.rados.0.smithi203.stdout:1123: rollback oid 7 current snap is 117 2024-09-17T15:35:17.726 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 104 2024-09-17T15:35:17.726 INFO:tasks.rados.rados.0.smithi203.stdout:1117: finishing write tid 3 to smithi20328086-45 2024-09-17T15:35:17.726 INFO:tasks.rados.rados.0.smithi203.stdout:1117: finishing write tid 4 to smithi20328086-45 2024-09-17T15:35:17.726 INFO:tasks.rados.rados.0.smithi203.stdout:1117: finishing write tid 5 to smithi20328086-45 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1117: oid 45 updating version 0 to 260 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1117: oid 45 version 260 is already newer than 259 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1117: oid 45 updating version 260 to 262 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1117: oid 45 version 262 is already newer than 261 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 262 (ObjNum 410 snap 117 seq_num 410) dirty exists 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1117: left oid 45 (ObjNum 410 snap 117 seq_num 410) 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:1121: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:35:17.727 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:17.928 INFO:tasks.rados.rados.0.smithi203.stdout:1122: finishing write tid 1 to smithi20328086-39 2024-09-17T15:35:17.928 INFO:tasks.rados.rados.0.smithi203.stdout:1122: finishing write tid 2 to smithi20328086-39 2024-09-17T15:35:17.928 INFO:tasks.rados.rados.0.smithi203.stdout:1123: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1123: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1123: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 271 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: finishing write tid 3 to smithi20328086-39 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: finishing write tid 4 to smithi20328086-39 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: finishing write tid 5 to smithi20328086-39 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: oid 39 updating version 0 to 387 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: oid 39 updating version 387 to 392 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: oid 39 updating version 392 to 397 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: oid 39 updating version 397 to 398 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 398 (ObjNum 412 snap 117 seq_num 412) dirty exists 2024-09-17T15:35:17.929 INFO:tasks.rados.rados.0.smithi203.stdout:1122: left oid 39 (ObjNum 412 snap 117 seq_num 412) 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1117: done (3 left) 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1121: done (2 left) 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1122: done (1 left) 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1123: done (0 left) 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1124: copy_from oid 44 from oid 19 current snap is 117 2024-09-17T15:35:17.930 INFO:tasks.rados.rados.0.smithi203.stdout:1125: snap_create 2024-09-17T15:35:17.951 INFO:tasks.rados.rados.0.smithi203.stdout:1124: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:35:17.956 INFO:tasks.rados.rados.0.smithi203.stdout:1124: finishing copy_from to smithi20328086-44 2024-09-17T15:35:17.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 365 (ObjNum 406 snap 114 seq_num 406) dirty exists 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1124: done (1 left) 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1125: done (0 left) 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1126: rollback oid 3 current snap is 118 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 114 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1127: read oid 26 snap -1 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1127: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:35:18.694 INFO:tasks.rados.rados.0.smithi203.stdout:1128: snap_create 2024-09-17T15:35:18.695 INFO:tasks.rados.rados.0.smithi203.stdout:1126: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:35:18.695 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 0 (ObjNum 481 snap 0 seq_num 4108902064) dirty dne 2024-09-17T15:35:18.700 INFO:tasks.rados.rados.0.smithi203.stdout:1127: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:35:18.768 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1194142 2024-09-17T15:35:19.618 INFO:tasks.rados.rados.0.smithi203.stdout:1126: done (2 left) 2024-09-17T15:35:19.619 INFO:tasks.rados.rados.0.smithi203.stdout:1127: done (1 left) 2024-09-17T15:35:19.619 INFO:tasks.rados.rados.0.smithi203.stdout:1128: done (0 left) 2024-09-17T15:35:19.619 INFO:tasks.rados.rados.0.smithi203.stdout:1129: delete oid 34 current snap is 119 2024-09-17T15:35:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:1129: done (0 left) 2024-09-17T15:35:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:1130: write oid 32 current snap is 119 2024-09-17T15:35:19.632 INFO:tasks.rados.rados.0.smithi203.stdout:1130: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-17T15:35:19.651 INFO:tasks.rados.rados.0.smithi203.stdout:1130: writing smithi20328086-32 from 558098 to 1194246 tid 1 2024-09-17T15:35:19.672 INFO:tasks.rados.rados.0.smithi203.stdout:1130: writing smithi20328086-32 from 1826604 to 2491733 tid 2 2024-09-17T15:35:19.672 INFO:tasks.rados.rados.0.smithi203.stdout:1131: write oid 40 current snap is 119 2024-09-17T15:35:19.673 INFO:tasks.rados.rados.0.smithi203.stdout:1131: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-17T15:35:19.690 INFO:tasks.rados.rados.0.smithi203.stdout:1131: writing smithi20328086-40 from 574905 to 1286302 tid 1 2024-09-17T15:35:19.702 INFO:tasks.rados.rados.0.smithi203.stdout:1131: writing smithi20328086-40 from 1885086 to 2410226 tid 2 2024-09-17T15:35:19.708 INFO:tasks.rados.rados.0.smithi203.stdout:1131: writing smithi20328086-40 from 2930189 to 2974905 tid 3 2024-09-17T15:35:19.708 INFO:tasks.rados.rados.0.smithi203.stdout:1132: snap_remove snap 104 2024-09-17T15:35:20.703 INFO:tasks.rados.rados.0.smithi203.stdout:1132: done (2 left) 2024-09-17T15:35:20.703 INFO:tasks.rados.rados.0.smithi203.stdout:1133: rollback oid 18 current snap is 119 2024-09-17T15:35:20.703 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 114 2024-09-17T15:35:20.703 INFO:tasks.rados.rados.0.smithi203.stdout:1130: finishing write tid 1 to smithi20328086-32 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: finishing write tid 2 to smithi20328086-32 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: finishing write tid 3 to smithi20328086-32 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: finishing write tid 4 to smithi20328086-32 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: oid 32 updating version 0 to 297 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: oid 32 updating version 297 to 299 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: oid 32 version 299 is already newer than 298 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 299 (ObjNum 414 snap 119 seq_num 414) dirty exists 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1130: left oid 32 (ObjNum 414 snap 119 seq_num 414) 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1131: finishing write tid 1 to smithi20328086-40 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1131: finishing write tid 2 to smithi20328086-40 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1131: finishing write tid 3 to smithi20328086-40 2024-09-17T15:35:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1131: finishing write tid 4 to smithi20328086-40 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: finishing write tid 5 to smithi20328086-40 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: oid 40 updating version 0 to 478 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: oid 40 version 478 is already newer than 476 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: oid 40 version 478 is already newer than 477 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: oid 40 updating version 478 to 479 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 479 (ObjNum 415 snap 119 seq_num 415) dirty exists 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: left oid 40 (ObjNum 415 snap 119 seq_num 415) 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1130: done (2 left) 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1131: done (1 left) 2024-09-17T15:35:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1134: snap_create 2024-09-17T15:35:20.732 INFO:tasks.rados.rados.0.smithi203.stdout:1133: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:35:20.740 INFO:tasks.rados.rados.0.smithi203.stdout:1133: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:35:20.740 INFO:tasks.rados.rados.0.smithi203.stdout:1133: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:35:20.740 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 315 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-17T15:35:21.704 INFO:tasks.rados.rados.0.smithi203.stdout:1133: done (1 left) 2024-09-17T15:35:21.705 INFO:tasks.rados.rados.0.smithi203.stdout:1134: done (0 left) 2024-09-17T15:35:21.705 INFO:tasks.rados.rados.0.smithi203.stdout:1135: copy_from oid 27 from oid 50 current snap is 120 2024-09-17T15:35:21.705 INFO:tasks.rados.rados.0.smithi203.stdout:1136: snap_create 2024-09-17T15:35:21.736 INFO:tasks.rados.rados.0.smithi203.stdout:1135: finishing copy_from racing read to smithi20328086-27 2024-09-17T15:35:21.742 INFO:tasks.rados.rados.0.smithi203.stdout:1135: finishing copy_from to smithi20328086-27 2024-09-17T15:35:21.742 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 296 (ObjNum 216 snap 55 seq_num 216) dirty exists 2024-09-17T15:35:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1135: done (1 left) 2024-09-17T15:35:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1136: done (0 left) 2024-09-17T15:35:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1137: read oid 31 snap -1 2024-09-17T15:35:22.711 INFO:tasks.rados.rados.0.smithi203.stdout:1137: expect (ObjNum 398 snap 111 seq_num 398) 2024-09-17T15:35:22.712 INFO:tasks.rados.rados.0.smithi203.stdout:1138: snap_create 2024-09-17T15:35:22.719 INFO:tasks.rados.rados.0.smithi203.stdout:1137: expect (ObjNum 398 snap 111 seq_num 398) 2024-09-17T15:35:22.758 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1715991 2024-09-17T15:35:22.792 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 200937 2024-09-17T15:35:22.837 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 75635 2024-09-17T15:35:23.710 INFO:tasks.rados.rados.0.smithi203.stdout:1137: done (1 left) 2024-09-17T15:35:23.711 INFO:tasks.rados.rados.0.smithi203.stdout:1138: done (0 left) 2024-09-17T15:35:23.711 INFO:tasks.rados.rados.0.smithi203.stdout:1139: write oid 38 current snap is 122 2024-09-17T15:35:23.711 INFO:tasks.rados.rados.0.smithi203.stdout:1139: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-17T15:35:23.731 INFO:tasks.rados.rados.0.smithi203.stdout:1139: writing smithi20328086-38 from 591712 to 1378358 tid 1 2024-09-17T15:35:23.750 INFO:tasks.rados.rados.0.smithi203.stdout:1139: writing smithi20328086-38 from 1943568 to 2612366 tid 2 2024-09-17T15:35:23.750 INFO:tasks.rados.rados.0.smithi203.stdout:1140: read oid 43 snap 117 2024-09-17T15:35:23.750 INFO:tasks.rados.rados.0.smithi203.stdout:1140: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:35:23.750 INFO:tasks.rados.rados.0.smithi203.stdout:1141: write oid 4 current snap is 122 2024-09-17T15:35:23.751 INFO:tasks.rados.rados.0.smithi203.stdout:1139: finishing write tid 1 to smithi20328086-38 2024-09-17T15:35:23.751 INFO:tasks.rados.rados.0.smithi203.stdout:1141: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-17T15:35:23.766 INFO:tasks.rados.rados.0.smithi203.stdout:1141: writing smithi20328086-4 from 608519 to 1070414 tid 1 2024-09-17T15:35:23.783 INFO:tasks.rados.rados.0.smithi203.stdout:1141: writing smithi20328086-4 from 1602050 to 2130859 tid 2 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1141: writing smithi20328086-4 from 2585035 to 3008519 tid 3 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1142: snap_create 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: finishing write tid 2 to smithi20328086-38 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: finishing write tid 3 to smithi20328086-38 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: finishing write tid 4 to smithi20328086-38 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: oid 38 updating version 0 to 275 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: oid 38 version 275 is already newer than 273 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:1139: oid 38 version 275 is already newer than 274 2024-09-17T15:35:23.793 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 275 (ObjNum 416 snap 122 seq_num 416) dirty exists 2024-09-17T15:35:23.794 INFO:tasks.rados.rados.0.smithi203.stdout:1139: left oid 38 (ObjNum 416 snap 122 seq_num 416) 2024-09-17T15:35:23.794 INFO:tasks.rados.rados.0.smithi203.stdout:1140: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:35:23.794 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:23.903 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1903930 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: finishing write tid 1 to smithi20328086-4 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: finishing write tid 2 to smithi20328086-4 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: finishing write tid 3 to smithi20328086-4 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: finishing write tid 4 to smithi20328086-4 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: finishing write tid 5 to smithi20328086-4 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: oid 4 updating version 0 to 265 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: oid 4 version 265 is already newer than 263 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: oid 4 updating version 265 to 266 2024-09-17T15:35:23.918 INFO:tasks.rados.rados.0.smithi203.stdout:1141: oid 4 version 266 is already newer than 264 2024-09-17T15:35:23.919 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 266 (ObjNum 417 snap 122 seq_num 417) dirty exists 2024-09-17T15:35:23.919 INFO:tasks.rados.rados.0.smithi203.stdout:1141: left oid 4 (ObjNum 417 snap 122 seq_num 417) 2024-09-17T15:35:24.569 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:35:24.569 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:35:24.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:35:24.572 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:35:24.572 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:35:24.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: get_monmap_and_config 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: build_initial_monmap 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: monmap: 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:24.697188+0000 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:24.697188+0000 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:24.695 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:24.696 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 Processor -- start 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 -- start start 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: init 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T15:35:24.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82a80a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f82a80a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f82a813e300 0x7f82a81406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f82a813e300 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a80a6530 con 0x7f82a80a54c0 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a809c150 con 0x7f82a80a50e0 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f82a809bdc0 con 0x7f82a813e300 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:24.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82afdf4640 10 monclient(hunting): authenticate will time out at 1457.662378s 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ad38f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f82a80a54c0 0x7f82a80a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:33032/0 (socket says 172.21.15.86:33032) 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ad38f640 1 -- 172.21.15.86:0/1553115864 learned_addr learned my addr 172.21.15.86:0/1553115864 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 1 --2- 172.21.15.86:0/1553115864 >> v2:172.21.15.203:3300/0 conn(0x7f82a813e300 0x7f82a81406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): get_auth_request con 0x7f82a813e300 auth_method 0 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient(hunting): handle_auth_done global_id 54343 payload 274 2024-09-17T15:35:24.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 1 -- 172.21.15.86:0/1553115864 >> v1:172.21.15.203:6789/0 conn(0x7f82a80a50e0 legacy=0x7f82a809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 1 -- 172.21.15.86:0/1553115864 >> v1:172.21.15.86:6789/0 conn(0x7f82a80a54c0 legacy=0x7f82a80a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient: _finish_hunting 0 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 1 monclient: found mon.noname-c 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.693+0000 7f82ae391640 1 -- 172.21.15.86:0/1553115864 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a809c430 con 0x7f82a813e300 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.697+0000 7f82ae391640 1 -- 172.21.15.86:0/1553115864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a813e300 msgr2=0x7f82a81406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:24.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.697+0000 7f82ae391640 1 --2- 172.21.15.86:0/1553115864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a813e300 0x7f82a81406f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009d90 tx=0x7f82a001c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ed36136d3f4efc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 1 -- 172.21.15.86:0/1553115864 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a00b2070 con 0x7f82a813e300 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: dump: 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.702 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: _finish_auth 0 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: _check_auth_tickets 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 1 -- 172.21.15.86:0/1553115864 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a00a5970 con 0x7f82a813e300 2024-09-17T15:35:24.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 1 -- 172.21.15.86:0/1553115864 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a00a5d40 con 0x7f82a813e300 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82acb8e640 10 monclient: dump: 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:24.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:24.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:24.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 5 monclient: authenticate success, global_id 54343 2024-09-17T15:35:24.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:24.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 4 set_mon_vals no callback set 2024-09-17T15:35:24.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: shutdown 2024-09-17T15:35:24.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- 172.21.15.86:0/1553115864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a813e300 msgr2=0x7f82a81406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 --2- 172.21.15.86:0/1553115864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a813e300 0x7f82a81406f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009d90 tx=0x7f82a001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- 172.21.15.86:0/1553115864 shutdown_connections 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 --2- 172.21.15.86:0/1553115864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a813e300 0x7f82a81406f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- 172.21.15.86:0/1553115864 >> 172.21.15.86:0/1553115864 conn(0x7f82a80aaae0 msgr2=0x7f82a80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- 172.21.15.86:0/1553115864 shutdown_connections 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- 172.21.15.86:0/1553115864 wait complete. 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: build_initial_monmap 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: monmap: 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:24.710 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 Processor -- start 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- start start 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: init 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 0x7f82a8138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-b con 0x7f82a80a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:24.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 0x7f82a813b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f82a80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): picked mon.noname-a con 0x7f82a813e300 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f82a8143980 con 0x7f82a813e300 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82a8143800 con 0x7f82a80a50e0 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f82a8143b00 con 0x7f82a80a54c0 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82afdf4640 10 monclient(hunting): authenticate will time out at 1457.670439s 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:24.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40832/0 (socket says 172.21.15.86:40832) 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 -- 172.21.15.86:0/146742037 learned_addr learned my addr 172.21.15.86:0/146742037 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 10 monclient(hunting): get_auth_request con 0x7f82a813e300 auth_method 0 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 0x7f82a813b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): get_auth_request con 0x7f82a80a54c0 auth_method 0 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 msgr2=0x7f82a813b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 msgr2=0x7f82a813b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:24.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ae391640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 0x7f82a8138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 10 monclient(hunting): get_auth_request con 0x7f82a80a50e0 auth_method 0 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient(hunting): handle_auth_done global_id 74264 payload 824 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 msgr2=0x7f82a8138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 0x7f82a8138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 msgr2=0x7f82a813b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:24.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient: _finish_hunting 0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 monclient: found mon.noname-c 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82ad38f640 1 -- 172.21.15.86:0/146742037 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a813e6e0 con 0x7f82a80a54c0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.701+0000 7f82adb90640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 0x7f82a8138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82ad38f640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 msgr2=0x7f82a813b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82ad38f640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 0x7f82a813b1d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f829800c410 tx=0x7f829800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298017070 con 0x7f82a80a54c0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: dump: 2024-09-17T15:35:24.715 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:24.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 5 monclient: authenticate success, global_id 74264 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82980070d0 con 0x7f82a80a54c0 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82980074c0 con 0x7f82a80a54c0 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _renew_subs 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a80a18d0 con 0x7f82a80a54c0 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 10 monclient: dump: 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:24.717 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829f7fe640 4 set_mon_vals no callback set 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _renew_subs 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a8121660 con 0x7f82a80a54c0 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a809ce10 con 0x7f82a80a54c0 2024-09-17T15:35:24.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f829801a070 con 0x7f82a80a54c0 2024-09-17T15:35:24.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 0x7f828c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:24.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(261..261 src has 213..261) ==== 4771+0+0 (secure 0 0 0) 0x7f829804f420 con 0x7f82a80a54c0 2024-09-17T15:35:24.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82adb90640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 0x7f828c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:24.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.705+0000 7f82adb90640 10 monclient: get_auth_request con 0x7f828c03bb60 auth_method 0 2024-09-17T15:35:24.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.709+0000 7f82adb90640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 0x7f828c03e020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82a4003ec0 tx=0x7f82a4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:24.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.709+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82a80a18d0 con 0x7f82a80a54c0 2024-09-17T15:35:24.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.709+0000 7f829e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:24.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.709+0000 7f829e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:24.734 INFO:tasks.rados.rados.0.smithi203.stdout:1139: done (3 left) 2024-09-17T15:35:24.735 INFO:tasks.rados.rados.0.smithi203.stdout:1140: done (2 left) 2024-09-17T15:35:24.735 INFO:tasks.rados.rados.0.smithi203.stdout:1141: done (1 left) 2024-09-17T15:35:24.735 INFO:tasks.rados.rados.0.smithi203.stdout:1142: done (0 left) 2024-09-17T15:35:24.735 INFO:tasks.rados.rados.0.smithi203.stdout:1143: write (excl) oid 34 current snap is 123 2024-09-17T15:35:24.735 INFO:tasks.rados.rados.0.smithi203.stdout:1143: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-17T15:35:24.747 INFO:tasks.rados.rados.0.smithi203.stdout:1143: writing smithi20328086-34 from 625326 to 1162470 tid 1 2024-09-17T15:35:24.761 INFO:tasks.rados.rados.0.smithi203.stdout:1143: writing smithi20328086-34 from 1944179 to 2616646 tid 2 2024-09-17T15:35:24.761 INFO:tasks.rados.rados.0.smithi203.stdout:1144: read oid 45 snap -1 2024-09-17T15:35:24.761 INFO:tasks.rados.rados.0.smithi203.stdout:1144: expect (ObjNum 410 snap 117 seq_num 410) 2024-09-17T15:35:24.761 INFO:tasks.rados.rados.0.smithi203.stdout:1145: snap_remove snap 114 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f82afdf4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f82afdf4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f82afdf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f82a813d4a0 con 0x7f82a80a54c0 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f829e7fc640 1 -- 172.21.15.86:0/146742037 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f829804de20 con 0x7f82a80a54c0 2024-09-17T15:35:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f829e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:24.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.941+0000 7f829e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:24.944 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 msgr2=0x7f828c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 0x7f828c03e020 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82a4003ec0 tx=0x7f82a4009040 comp rx=0 tx=0).stop 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 10 monclient: shutdown 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 msgr2=0x7f82a813b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 0x7f82a813b1d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f829800c410 tx=0x7f829800c8d0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 shutdown_connections 2024-09-17T15:35:24.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f828c03bb60 0x7f828c03e020 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82a813e300 0x7f82a813b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82a80a54c0 0x7f82a813b1d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 --2- 172.21.15.86:0/146742037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82a80a50e0 0x7f82a8138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 >> 172.21.15.86:0/146742037 conn(0x7f82a80aaae0 msgr2=0x7f82a80a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 shutdown_connections 2024-09-17T15:35:24.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:24.945+0000 7f82afdf4640 1 -- 172.21.15.86:0/146742037 wait complete. 2024-09-17T15:35:25.738 INFO:tasks.rados.rados.0.smithi203.stdout:1145: done (2 left) 2024-09-17T15:35:25.739 INFO:tasks.rados.rados.0.smithi203.stdout:1146: read oid 18 snap 120 2024-09-17T15:35:25.739 INFO:tasks.rados.rados.0.smithi203.stdout:1146: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:35:25.739 INFO:tasks.rados.rados.0.smithi203.stdout:1147: write oid 25 current snap is 123 2024-09-17T15:35:25.739 INFO:tasks.rados.rados.0.smithi203.stdout:1147: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-17T15:35:25.756 INFO:tasks.rados.rados.0.smithi203.stdout:1147: writing smithi20328086-25 from 642133 to 1370879 tid 1 2024-09-17T15:35:25.771 INFO:tasks.rados.rados.0.smithi203.stdout:1147: writing smithi20328086-25 from 2119014 to 2651492 tid 2 2024-09-17T15:35:25.772 INFO:tasks.rados.rados.0.smithi203.stdout:1148: read oid 36 snap -1 2024-09-17T15:35:25.772 INFO:tasks.rados.rados.0.smithi203.stdout:1148: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:35:25.772 INFO:tasks.rados.rados.0.smithi203.stdout:1149: snap_remove snap 122 2024-09-17T15:35:27.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:28.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: get_monmap_and_config 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: build_initial_monmap 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: monmap: 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:28.093688+0000 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:28.093688+0000 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:28.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:28.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:28.093 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:28.093 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:28.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 Processor -- start 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 -- start start 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: init 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5540689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd554068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:28.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-a con 0x7fd554139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd55409d770 con 0x7fd554068db0 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd55409ed00 con 0x7fd554139f00 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd55409efe0 con 0x7fd5540689d0 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd559d9c640 10 monclient(hunting): authenticate will time out at 1461.059344s 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45564/0 (socket says 172.21.15.86:45564) 2024-09-17T15:35:28.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 learned_addr learned my addr 172.21.15.86:0/4409060 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd5540689d0 auth_method 0 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd554139f00 auth_method 0 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.089+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:28.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.093+0000 7fd552ffd640 1 -- 172.21.15.86:0/4409060 >> v1:172.21.15.203:6789/0 conn(0x7fd554068db0 legacy=0x7fd554069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd5540689d0 auth_method 0 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd554139f00 auth_method 0 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:28.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.293+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:28.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:28.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd5540689d0 auth_method 0 2024-09-17T15:35:28.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd554139f00 auth_method 0 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:28.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:28.697+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd5540689d0 auth_method 0 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:29.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd554139f00 auth_method 0 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:29.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:29.501+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893820181 0 0) 0x7fd55409d770 con 0x7fd554068db0 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 10 monclient(hunting): _negotiate 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 10 monclient(hunting): my global_id is 54349 2024-09-17T15:35:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.933+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd540005740 con 0x7fd554068db0 2024-09-17T15:35:30.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2113565552 0 0) 0x7fd540005740 con 0x7fd554068db0 2024-09-17T15:35:30.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd55409ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:30.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd55409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 msgr2=0x7fd55413c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 --2- 172.21.15.86:0/4409060 >> v2:172.21.15.86:3300/0 conn(0x7fd554139f00 0x7fd55413c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _finish_hunting 0 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5540a04e0 con 0x7fd554068db0 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fd53c001a00 con 0x7fd554068db0 2024-09-17T15:35:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: dump: 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 >> v1:172.21.15.203:6789/0 conn(0x7fd554068db0 legacy=0x7fd554069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 0x7fd540006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): picked mon.b con 0x7fd540005ce0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd540006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): picked mon.c con 0x7fd5540689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540006fb0 0x7fd540007430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): picked mon.a con 0x7fd540006fb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd55409ed00 con 0x7fd540006fb0 2024-09-17T15:35:30.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd55409efe0 con 0x7fd540005ce0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd54000dec0 con 0x7fd5540689d0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5527fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540006fb0 0x7fd540007430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd540006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 0x7fd540006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd540005ce0 auth_method 0 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): get_auth_request con 0x7fd5540689d0 auth_method 0 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd540006fb0 auth_method 0 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:30.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd553fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient(hunting): handle_auth_done global_id 54349 payload 293 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 msgr2=0x7fd540006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5540689d0 0x7fd540006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540006fb0 msgr2=0x7fd540007430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540006fb0 0x7fd540007430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 monclient: found mon.b 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:30.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:35:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd540005740 con 0x7fd540005ce0 2024-09-17T15:35:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:30.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd5537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.945+0000 7fd559d9c640 5 monclient: authenticate success, global_id 54349 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd559d9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5537fe640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 msgr2=0x7fd540006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5537fe640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 0x7fd540006120 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd5440129f0 tx=0x7fd54401d290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57b032bd02e6c22d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5440b3070 con 0x7fd540005ce0 2024-09-17T15:35:30.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: dump: 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5440ae410 con 0x7fd540005ce0 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 1 -- 172.21.15.86:0/4409060 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54400f040 con 0x7fd540005ce0 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd5527fc640 10 monclient: dump: 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd559d9c640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.949+0000 7fd559d9c640 4 set_mon_vals no callback set 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 10 monclient: shutdown 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 -- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 msgr2=0x7fd540006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 --2- 172.21.15.86:0/4409060 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd540005ce0 0x7fd540006120 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd5440129f0 tx=0x7fd54401d290 comp rx=0 tx=0).stop 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd553fff640 1 -- 172.21.15.86:0/4409060 reap_dead start 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 -- 172.21.15.86:0/4409060 shutdown_connections 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 -- 172.21.15.86:0/4409060 >> 172.21.15.86:0/4409060 conn(0x7fd5540a6440 msgr2=0x7fd5540a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 -- 172.21.15.86:0/4409060 shutdown_connections 2024-09-17T15:35:30.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 -- 172.21.15.86:0/4409060 wait complete. 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 10 monclient: build_initial_monmap 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 10 monclient: monmap: 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:30.962 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.953+0000 7fd559d9c640 1 Processor -- start 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 -- start start 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient: init 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-a con 0x7fd540005ce0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 0x7fd554121660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-c con 0x7fd554123c80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd554135230 0x7fd554121ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): picked mon.noname-b con 0x7fd554135230 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd5540a0120 con 0x7fd540005ce0 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd55409ffa0 con 0x7fd554135230 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd5540a02a0 con 0x7fd554123c80 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd559d9c640 10 monclient(hunting): authenticate will time out at 1463.925152s 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 0x7fd554121660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53118/0 (socket says 172.21.15.86:53118) 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): get_auth_request con 0x7fd554123c80 auth_method 0 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 1 -- 172.21.15.86:0/2540510226 learned_addr learned my addr 172.21.15.86:0/2540510226 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): get_auth_request con 0x7fd540005ce0 auth_method 0 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:30.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd553fff640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd554135230 0x7fd554121ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd553fff640 10 monclient(hunting): get_auth_request con 0x7fd554135230 auth_method 0 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd553fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient(hunting): handle_auth_done global_id 74267 payload 824 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 -- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd554135230 msgr2=0x7fd554121ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd554135230 0x7fd554121ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 -- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 msgr2=0x7fd554134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:30.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 monclient: found mon.noname-c 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd552ffd640 1 -- 172.21.15.86:0/2540510226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5540a3290 con 0x7fd554123c80 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.957+0000 7fd5537fe640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd552ffd640 1 -- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 msgr2=0x7fd554121660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd552ffd640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 0x7fd554121660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001c20 tx=0x7fd53c012630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c01c070 con 0x7fd554123c80 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: dump: 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:30.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: _finish_auth 0 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 5 monclient: authenticate success, global_id 74267 2024-09-17T15:35:30.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _renew_subs 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd53c014770 con 0x7fd554123c80 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55409d460 con 0x7fd554123c80 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c012ea0 con 0x7fd554123c80 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 10 monclient: dump: 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:30.969 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd551ffb640 4 set_mon_vals no callback set 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _renew_subs 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:30.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd554124be0 con 0x7fd554123c80 2024-09-17T15:35:30.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd53c01f070 con 0x7fd554123c80 2024-09-17T15:35:30.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:30.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:30.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554069190 con 0x7fd554123c80 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 0x7fd53403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(263..263 src has 213..263) ==== 4799+0+0 (secure 0 0 0) 0x7fd53c052610 con 0x7fd554123c80 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd5537fe640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 0x7fd53403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.961+0000 7fd5537fe640 10 monclient: get_auth_request con 0x7fd53403bb60 auth_method 0 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.965+0000 7fd5537fe640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 0x7fd53403e020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd544006c90 tx=0x7fd544004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.965+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd53c0132d0 con 0x7fd554123c80 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.965+0000 7fd54bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:30.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:30.965+0000 7fd54bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:31.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd559d9c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd559d9c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd559d9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd5540a2b80 con 0x7fd554123c80 2024-09-17T15:35:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd54bfff640 1 -- 172.21.15.86:0/2540510226 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd53c013df0 con 0x7fd554123c80 2024-09-17T15:35:31.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd54bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:31.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.197+0000 7fd54bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:31.201 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 msgr2=0x7fd53403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 0x7fd53403e020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd544006c90 tx=0x7fd544004740 comp rx=0 tx=0).stop 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 10 monclient: shutdown 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 msgr2=0x7fd554121660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 0x7fd554121660 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd53c001c20 tx=0x7fd53c012630 comp rx=0 tx=0).stop 2024-09-17T15:35:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 shutdown_connections 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd53403bb60 0x7fd53403e020 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd554135230 0x7fd554121ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd554123c80 0x7fd554121660 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 --2- 172.21.15.86:0/2540510226 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd540005ce0 0x7fd554134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 >> 172.21.15.86:0/2540510226 conn(0x7fd5540a6440 msgr2=0x7fd554073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 shutdown_connections 2024-09-17T15:35:31.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.201+0000 7fd559d9c640 1 -- 172.21.15.86:0/2540510226 wait complete. 2024-09-17T15:35:31.218 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:35:31.218 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:35:31.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: get_monmap_and_config 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: monmap: 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:31.337093+0000 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:31.337093+0000 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.335 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:31.336 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:31.336 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:31.336 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:31.336 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:31.336 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 Processor -- start 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 -- start start 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: init 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35f80a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:31.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35f80a1760 0x7f35f813af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-a con 0x7f35f80a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f35f813b450 0x7f35f813d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-c con 0x7f35f813b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f80a2130 con 0x7f35f80a0c70 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f35f80a2410 con 0x7f35f80a1760 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f35f80a26f0 con 0x7f35f813b450 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35fe22e640 10 monclient(hunting): authenticate will time out at 1464.302604s 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35f80a1760 0x7f35f813af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35f80a1760 0x7f35f813af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53122/0 (socket says 172.21.15.86:53122) 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 -- 172.21.15.86:0/452855290 learned_addr learned my addr 172.21.15.86:0/452855290 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request con 0x7f35f80a1760 auth_method 0 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f7fff640 1 --2- 172.21.15.86:0/452855290 >> v2:172.21.15.203:3300/0 conn(0x7f35f813b450 0x7f35f813d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f7fff640 10 monclient(hunting): get_auth_request con 0x7f35f813b450 auth_method 0 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_done global_id 94101 payload 274 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 -- 172.21.15.86:0/452855290 >> v2:172.21.15.203:3300/0 conn(0x7f35f813b450 msgr2=0x7f35f813d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/452855290 >> v2:172.21.15.203:3300/0 conn(0x7f35f813b450 0x7f35f813d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 -- 172.21.15.86:0/452855290 >> v1:172.21.15.203:6789/0 conn(0x7f35f80a0c70 legacy=0x7f35f80a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 monclient: found mon.noname-a 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.333+0000 7f35f6ffd640 1 -- 172.21.15.86:0/452855290 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f809f9f0 con 0x7f35f80a1760 2024-09-17T15:35:31.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.337+0000 7f35f6ffd640 1 -- 172.21.15.86:0/452855290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 msgr2=0x7f35f813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.337+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/452855290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f813af10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f35f806f840 tx=0x7f35e001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ee275514fb3916 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.341+0000 7f35f67fc640 1 -- 172.21.15.86:0/452855290 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00a5750 con 0x7f35f80a1760 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.341+0000 7f35f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: dump: 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.346 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 5 monclient: authenticate success, global_id 94101 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 1 -- 172.21.15.86:0/452855290 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e00a58f0 con 0x7f35f80a1760 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 1 -- 172.21.15.86:0/452855290 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00a5cc0 con 0x7f35f80a1760 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:31.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 4 set_mon_vals no callback set 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f67fc640 10 monclient: dump: 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.348 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: shutdown 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- 172.21.15.86:0/452855290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 msgr2=0x7f35f813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- 172.21.15.86:0/452855290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f813af10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f35f806f840 tx=0x7f35e001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- 172.21.15.86:0/452855290 shutdown_connections 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- 172.21.15.86:0/452855290 >> v2:172.21.15.203:3300/0 conn(0x7f35f813b450 0x7f35f813d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- 172.21.15.86:0/452855290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f813af10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- 172.21.15.86:0/452855290 >> 172.21.15.86:0/452855290 conn(0x7f35f80a2f40 msgr2=0x7f35f80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- 172.21.15.86:0/452855290 shutdown_connections 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- 172.21.15.86:0/452855290 wait complete. 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: monmap: 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 Processor -- start 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- start start 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: init 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-c con 0x7f35f80a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f8130140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-a con 0x7f35f80a1760 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f813b450 0x7f35f8130680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): picked mon.noname-b con 0x7f35f813b450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:31.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f35f8140720 con 0x7f35f80a1760 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81405a0 con 0x7f35f813b450 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f35f81408a0 con 0x7f35f80a0c70 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35fe22e640 10 monclient(hunting): authenticate will time out at 1464.314540s 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f8130140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45640/0 (socket says 172.21.15.86:45640) 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 -- 172.21.15.86:0/4239337548 learned_addr learned my addr 172.21.15.86:0/4239337548 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): get_auth_request con 0x7f35f80a0c70 auth_method 0 2024-09-17T15:35:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f8130140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53132/0 (socket says 172.21.15.86:53132) 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request con 0x7f35f80a1760 auth_method 0 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f7fff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f813b450 0x7f35f8130680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f7fff640 10 monclient(hunting): get_auth_request con 0x7f35f813b450 auth_method 0 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient(hunting): handle_auth_done global_id 74270 payload 824 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f813b450 msgr2=0x7f35f8130680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f813b450 0x7f35f8130680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 msgr2=0x7f35f8130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f8130140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 monclient: found mon.noname-c 2024-09-17T15:35:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f77fe640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f81335c0 con 0x7f35f80a0c70 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.345+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f80a1760 0x7f35f8130140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f77fe640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 msgr2=0x7f35f812fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f77fe640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c450 tx=0x7f35e800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8012a40 con 0x7f35f80a0c70 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: dump: 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e8017070 con 0x7f35f80a0c70 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 5 monclient: authenticate success, global_id 74270 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 monclient: _renew_subs 2024-09-17T15:35:31.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f813bdf0 con 0x7f35f80a0c70 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e8012e40 con 0x7f35f80a0c70 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 10 monclient: dump: 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f5ffb640 4 set_mon_vals no callback set 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 monclient: _renew_subs 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f81332e0 con 0x7f35f80a0c70 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 0x7f35f81cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f35f809d3e0 con 0x7f35f81cd8b0 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 0x7f35f81cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient: get_auth_request con 0x7f35f81cd8b0 auth_method 0 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f77fe640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c450 tx=0x7f35e800c910 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f35e804fd00 osd_map(264..264 src has 213..264) 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f35e801c460 con 0x7f35f80a0c70 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_done global_id 94107 payload 824 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35f80a1050 con 0x7f35f81cd8b0 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35effff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f35d803bbb0 0x7f35d803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f7fff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f35d803bbb0 0x7f35d803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.349+0000 7f35f7fff640 10 monclient: get_auth_request con 0x7f35d803bbb0 auth_method 0 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1149: done (5 left) 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1150: read oid 12 snap 121 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1150: expect (ObjNum 274 snap 70 seq_num 274) 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1151: write (excl) oid 17 current snap is 123 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1151: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-17T15:35:31.366 INFO:tasks.rados.rados.0.smithi203.stdout:1151: writing smithi20328086-17 from 658940 to 1062935 tid 1 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 msgr2=0x7f35f81cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 0x7f35f81cdc90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35e00ae000 tx=0x7f35e0019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00adde0 con 0x7f35f81cd8b0 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 10 monclient: dump: 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f35e00adde0 con 0x7f35f81cd8b0 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 msgr2=0x7f35f81cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 0x7f35f81cdc90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35e00ae000 tx=0x7f35e0019650 comp rx=0 tx=0).stop 2024-09-17T15:35:31.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 msgr2=0x7f35f81cdc90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35effff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81cd8b0 0x7f35f81cdc90 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35e00ae000 tx=0x7f35e0019650 comp rx=0 tx=0).stop 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.353+0000 7f35f7fff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f35d803bbb0 0x7f35d803e070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35f00079d0 tx=0x7f35f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35fe22e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35fe22e640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35fe22e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f35f81366e0 con 0x7f35f81ce1d0 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient: get_auth_request con 0x7f35f81ce1d0 auth_method 0 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 10 monclient(hunting): handle_auth_done global_id 94110 payload 824 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35f81ce5b0 con 0x7f35f81ce1d0 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 msgr2=0x7f35f806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 msgr2=0x7f35f806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f35e001af60 tx=0x7f35e0009810 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:31.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.361+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:31.381 INFO:tasks.rados.rados.0.smithi203.stdout:1151: writing smithi20328086-17 from 1777496 to 2453632 tid 2 2024-09-17T15:35:31.388 INFO:tasks.rados.rados.0.smithi203.stdout:1151: writing smithi20328086-17 from 2950951 to 3058940 tid 3 2024-09-17T15:35:31.388 INFO:tasks.rados.rados.0.smithi203.stdout:1152: snap_remove snap 117 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: get_auth_request con 0x7f35f81ce1d0 auth_method 0 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: _init_auth method 2 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: _init_auth already have auth, reseting 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: handle_auth_reply_more payload 9 2024-09-17T15:35:31.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:31.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.565+0000 7f35f6ffd640 10 monclient: handle_auth_done global_id 94110 payload 293 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35f6ffd640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 msgr2=0x7f35f806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35f6ffd640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f35e0000f90 tx=0x7f35e00adf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e00c62e0 con 0x7f35f81ce1d0 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 10 monclient: dump: 2024-09-17T15:35:31.572 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f35e00cc050 con 0x7f35f81ce1d0 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 msgr2=0x7f35f806f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f35e0000f90 tx=0x7f35e00adf90 comp rx=0 tx=0).stop 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 msgr2=0x7f35f806f900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35f81ce1d0 0x7f35f806f900 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 6923, 2024-09-17T15:35:31.574 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.575 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:31.577 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:31.578 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.579 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:31.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.585 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:31.586 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f35d803bbb0 msgr2=0x7f35d803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f35d803bbb0 0x7f35d803e070 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35f00079d0 tx=0x7f35f0008040 comp rx=0 tx=0).stop 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 10 monclient: shutdown 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 msgr2=0x7f35f812fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f35e800c450 tx=0x7f35e800c910 comp rx=0 tx=0).stop 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35f7fff640 1 -- 172.21.15.86:0/4239337548 reap_dead start 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35effff640 1 -- 172.21.15.86:0/4239337548 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(264..264 src has 213..264) ==== 4815+0+0 (unknown 0 0 0) 0x7f35e804fd00 con 0x7f35f80a0c70 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 shutdown_connections 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 --2- 172.21.15.86:0/4239337548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35f80a0c70 0x7f35f812fc00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 >> 172.21.15.86:0/4239337548 conn(0x7f35f80a2f40 msgr2=0x7f35f80a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 shutdown_connections 2024-09-17T15:35:31.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.569+0000 7f35fe22e640 1 -- 172.21.15.86:0/4239337548 wait complete. 2024-09-17T15:35:31.591 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:31.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:35:31.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:35:31.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: get_monmap_and_config 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: monmap: 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:31.712834+0000 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:31.712834+0000 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:31.711 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:31.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 Processor -- start 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 -- start start 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: init 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-b con 0x7f06000a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:31.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06000a1590 0x7f0600135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-a con 0x7f06000a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0600135f90 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06000a2130 con 0x7f0600135f90 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f06000a2410 con 0x7f06000a1590 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f06000a26f0 con 0x7f06000a0c70 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f06086c3640 10 monclient(hunting): authenticate will time out at 1464.678356s 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06000a1590 0x7f0600135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45656/0 (socket says 172.21.15.86:45656) 2024-09-17T15:35:31.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 1 -- 172.21.15.86:0/3184865267 learned_addr learned my addr 172.21.15.86:0/3184865267 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): get_auth_request con 0x7f06000a0c70 auth_method 0 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): get_auth_request con 0x7f06000a1590 auth_method 0 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient(hunting): handle_auth_done global_id 94113 payload 274 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 -- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 msgr2=0x7f06000a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 -- 172.21.15.86:0/3184865267 >> v1:172.21.15.203:6789/0 conn(0x7f0600135f90 legacy=0x7f060013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 monclient: found mon.noname-a 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f0605c5e640 1 -- 172.21.15.86:0/3184865267 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f060009f9f0 con 0x7f06000a1590 2024-09-17T15:35:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.709+0000 7f060645f640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:31.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f0605c5e640 1 -- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 msgr2=0x7f0600135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f0605c5e640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600135a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05f40097c0 tx=0x7f05f401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef425fbba0306641 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 1 -- 172.21.15.86:0/3184865267 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40b0070 con 0x7f06000a1590 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: dump: 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 1 -- 172.21.15.86:0/3184865267 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f40ab600 con 0x7f06000a1590 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 1 -- 172.21.15.86:0/3184865267 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40ab9d0 con 0x7f06000a1590 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f060545d640 10 monclient: dump: 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 5 monclient: authenticate success, global_id 94113 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 4 set_mon_vals no callback set 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 10 monclient: shutdown 2024-09-17T15:35:31.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 -- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 msgr2=0x7f0600135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600135a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05f40097c0 tx=0x7f05f401c360 comp rx=0 tx=0).stop 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 -- 172.21.15.86:0/3184865267 shutdown_connections 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600135a50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 --2- 172.21.15.86:0/3184865267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06000a0c70 0x7f06000a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 -- 172.21.15.86:0/3184865267 >> 172.21.15.86:0/3184865267 conn(0x7f06000a2f40 msgr2=0x7f06000a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 -- 172.21.15.86:0/3184865267 shutdown_connections 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.713+0000 7f06086c3640 1 -- 172.21.15.86:0/3184865267 wait complete. 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient: monmap: 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:31.724 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 Processor -- start 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 -- start start 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient: init 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:31.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06000a0c70 0x7f060012ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-b con 0x7f06000a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-a con 0x7f06000a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 0x7f0600137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): picked mon.noname-c con 0x7f0600135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f060009ce60 con 0x7f06000a1590 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f060009cce0 con 0x7f06000a0c70 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f060009ff50 con 0x7f0600135f90 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f06086c3640 10 monclient(hunting): authenticate will time out at 1464.685258s 2024-09-17T15:35:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 0x7f0600137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 0x7f0600137430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45658/0 (socket says 172.21.15.86:45658) 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 1 -- 172.21.15.86:0/3673089810 learned_addr learned my addr 172.21.15.86:0/3673089810 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): get_auth_request con 0x7f0600135f90 auth_method 0 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): get_auth_request con 0x7f06000a1590 auth_method 0 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f060645f640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06000a0c70 0x7f060012ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f060645f640 10 monclient(hunting): get_auth_request con 0x7f06000a0c70 auth_method 0 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f060645f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f060645f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f060645f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient(hunting): handle_auth_done global_id 94116 payload 824 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 msgr2=0x7f0600137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 0x7f0600137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06000a0c70 msgr2=0x7f060012ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06000a0c70 0x7f060012ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 monclient: found mon.noname-a 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0600139a10 con 0x7f06000a1590 2024-09-17T15:35:31.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0606c60640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0600135f90 0x7f0600137430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 msgr2=0x7f0600136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.717+0000 7f0605c5e640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600136ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05f40ad000 tx=0x7f05f4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40b8020 con 0x7f06000a1590 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: dump: 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.734 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f40b4850 con 0x7f06000a1590 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40b0040 con 0x7f06000a1590 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 10 monclient: dump: 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.735 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f0604c5c640 4 set_mon_vals no callback set 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 5 monclient: authenticate success, global_id 94116 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 monclient: _renew_subs 2024-09-17T15:35:31.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f060009d460 con 0x7f06000a1590 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 monclient: _renew_subs 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f060013ad50 con 0x7f06000a1590 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 0x7f0600137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f06000a1050 con 0x7f060013b6a0 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 0x7f0600137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f05f40bb020 con 0x7f06000a1590 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient: get_auth_request con 0x7f060013b6a0 auth_method 0 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 10 monclient(hunting): handle_auth_done global_id 74279 payload 824 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05ec03bbb0 0x7f05ec03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f060645f640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0600139550 con 0x7f060013b6a0 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(264..264 src has 213..264) ==== 4815+0+0 (secure 0 0 0) 0x7f05f40e9c40 con 0x7f06000a1590 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f0606c60640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05ec03bbb0 0x7f05ec03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.721+0000 7f0606c60640 10 monclient: get_auth_request con 0x7f05ec03bbb0 auth_method 0 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f060645f640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 msgr2=0x7f0600137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f060645f640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 0x7f0600137970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007d30 tx=0x7f05f0010590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0010ea0 con 0x7f060013b6a0 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 10 monclient: dump: 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.739 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05f0015850 con 0x7f060013b6a0 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 msgr2=0x7f0600137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 0x7f0600137970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007d30 tx=0x7f05f0010590 comp rx=0 tx=0).stop 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 msgr2=0x7f0600137970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f05feffd640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060013b6a0 0x7f0600137970 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.725+0000 7f0606c60640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05ec03bbb0 0x7f05ec03e070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f05f8004370 tx=0x7f05f8007dd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f06086c3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f06086c3640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 0x7f0600137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f06086c3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f060006d4f0 con 0x7f060012f4d0 2024-09-17T15:35:31.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 0x7f0600137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient: get_auth_request con 0x7f060012f4d0 auth_method 0 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 10 monclient(hunting): handle_auth_done global_id 74282 payload 824 2024-09-17T15:35:31.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.729+0000 7f060645f640 1 -- 172.21.15.86:0/3673089810 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06001d9290 con 0x7f060012f4d0 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f060645f640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 msgr2=0x7f0600137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f060645f640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 0x7f0600137eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007d30 tx=0x7f05f0004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f0020a00 con 0x7f060012f4d0 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 10 monclient: dump: 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f05f002a430 con 0x7f060012f4d0 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 msgr2=0x7f0600137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 0x7f0600137eb0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007d30 tx=0x7f05f0004a50 comp rx=0 tx=0).stop 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 msgr2=0x7f0600137eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f05feffd640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f060012f4d0 0x7f0600137eb0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f05f0007d30 tx=0x7f05f0004a50 comp rx=0 tx=0).stop 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 112522, 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:31.747 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.748 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.749 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.750 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.751 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:31.752 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.753 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.754 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.755 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:31.756 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.757 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:31.758 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.760 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:31.760 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05ec03bbb0 msgr2=0x7f05ec03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05ec03bbb0 0x7f05ec03e070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f05f8004370 tx=0x7f05f8007dd0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 10 monclient: shutdown 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 msgr2=0x7f0600136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 --2- 172.21.15.86:0/3673089810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06000a1590 0x7f0600136ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f05f40ad000 tx=0x7f05f4019100 comp rx=0 tx=0).stop 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f0606c60640 1 -- 172.21.15.86:0/3673089810 reap_dead start 2024-09-17T15:35:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f0606c60640 1 -- 172.21.15.86:0/3673089810 reap_dead start 2024-09-17T15:35:31.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 shutdown_connections 2024-09-17T15:35:31.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 >> 172.21.15.86:0/3673089810 conn(0x7f06000a2f40 msgr2=0x7f06000a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 shutdown_connections 2024-09-17T15:35:31.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.733+0000 7f06086c3640 1 -- 172.21.15.86:0/3673089810 wait complete. 2024-09-17T15:35:31.762 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:31.762 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:35:31.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:35:31.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: get_monmap_and_config 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: monmap: 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:31.868867+0000 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:31.868867+0000 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:31.867 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:31.868 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:31.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 Processor -- start 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 -- start start 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: init 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 0x7f3164132d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): picked mon.noname-b con 0x7f3164132950 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3164132420 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31640a2d30 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3164070940 con 0x7f3164132420 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3164073390 con 0x7f31640a2d30 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3164073930 con 0x7f3164132950 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f316c095640 10 monclient(hunting): authenticate will time out at 1464.834387s 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 0x7f3164132d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169630640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3164132420 0x7f31640a2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38824/0 (socket says 172.21.15.86:38824) 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169630640 1 -- 172.21.15.86:0/16031342 learned_addr learned my addr 172.21.15.86:0/16031342 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): get_auth_request con 0x7f3164132950 auth_method 0 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient(hunting): handle_auth_done global_id 74285 payload 274 2024-09-17T15:35:31.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 1 -- 172.21.15.86:0/16031342 >> v1:172.21.15.203:6789/0 conn(0x7f31640a2d30 legacy=0x7f316413dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 1 -- 172.21.15.86:0/16031342 >> v1:172.21.15.86:6789/0 conn(0x7f3164132420 legacy=0x7f31640a2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 1 monclient: found mon.noname-b 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:35:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.865+0000 7f3169e31640 1 -- 172.21.15.86:0/16031342 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3164074610 con 0x7f3164132950 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3169e31640 1 -- 172.21.15.86:0/16031342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 msgr2=0x7f3164132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3169e31640 1 --2- 172.21.15.86:0/16031342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 0x7f3164132d30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3164073c40 tx=0x7f316001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc863a9a4ecd6271 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 1 -- 172.21.15.86:0/16031342 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600b0070 con 0x7f3164132950 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: dump: 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 1 -- 172.21.15.86:0/16031342 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31600ab600 con 0x7f3164132950 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 5 monclient: authenticate success, global_id 74285 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 1 -- 172.21.15.86:0/16031342 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600ab9d0 con 0x7f3164132950 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 4 set_mon_vals no callback set 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f3168e2f640 10 monclient: dump: 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.876 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 10 monclient: shutdown 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 -- 172.21.15.86:0/16031342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 msgr2=0x7f3164132d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 --2- 172.21.15.86:0/16031342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 0x7f3164132d30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3164073c40 tx=0x7f316001c3f0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 -- 172.21.15.86:0/16031342 shutdown_connections 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 --2- 172.21.15.86:0/16031342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3164132950 0x7f3164132d30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 -- 172.21.15.86:0/16031342 >> 172.21.15.86:0/16031342 conn(0x7f31640a7210 msgr2=0x7f31640a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 -- 172.21.15.86:0/16031342 shutdown_connections 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.869+0000 7f316c095640 1 -- 172.21.15.86:0/16031342 wait complete. 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient: build_initial_monmap 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient: monmap: 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:31.877 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:31.878 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 Processor -- start 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 -- start start 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient: init 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:35:31.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 0x7f31641ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): picked mon.noname-c con 0x7f31640a2d30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 0x7f316413d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): picked mon.noname-a con 0x7f3164132420 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164132950 0x7f316413dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): picked mon.noname-b con 0x7f3164132950 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3164141d80 con 0x7f3164132420 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3164141c00 con 0x7f3164132950 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3164141f00 con 0x7f31640a2d30 2024-09-17T15:35:31.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316c095640 10 monclient(hunting): authenticate will time out at 1464.841323s 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 0x7f31641ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 0x7f316413d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 0x7f31641ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45688/0 (socket says 172.21.15.86:45688) 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 0x7f316413d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53158/0 (socket says 172.21.15.86:53158) 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 1 -- 172.21.15.86:0/1248812093 learned_addr learned my addr 172.21.15.86:0/1248812093 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): get_auth_request con 0x7f31640a2d30 auth_method 0 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): get_auth_request con 0x7f3164132420 auth_method 0 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316a632640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164132950 0x7f316413dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient(hunting): handle_auth_done global_id 94119 payload 824 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 msgr2=0x7f31641ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 0x7f31641ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164132950 msgr2=0x7f316413dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164132950 0x7f316413dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient: _finish_hunting 0 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 monclient: found mon.noname-a 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169630640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f316409c080 con 0x7f3164132420 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316a632640 10 monclient: get_auth_request con 0x7f3164132950 auth_method 0 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f316a632640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164132950 0x7f316413dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:35:31.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.873+0000 7f3169e31640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31640a2d30 0x7f31641ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169630640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 msgr2=0x7f316413d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169630640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 0x7f316413d9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f315400c410 tx=0x7f315400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154017070 con 0x7f3164132420 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: dump: 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.888 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31540070d0 con 0x7f3164132420 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 5 monclient: authenticate success, global_id 94119 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 monclient: _renew_subs 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f316413c470 con 0x7f3164132420 2024-09-17T15:35:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540074c0 con 0x7f3164132420 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 10 monclient: dump: 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.890 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315b7fe640 4 set_mon_vals no callback set 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 monclient: _renew_subs 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f316413a110 con 0x7f3164132420 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 0x7f316402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f316413be10 con 0x7f3164138e10 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f315401a070 con 0x7f3164132420 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 0x7f316402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient: get_auth_request con 0x7f3164138e10 auth_method 0 2024-09-17T15:35:31.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(264..264 src has 213..264) ==== 4815+0+0 (secure 0 0 0) 0x7f315404f440 con 0x7f3164132420 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316a632640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f316a632640 10 monclient: get_auth_request con 0x7f314803bb60 auth_method 0 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 10 monclient(hunting): handle_auth_done global_id 54364 payload 824 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.877+0000 7f3169e31640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3164132d30 con 0x7f3164138e10 2024-09-17T15:35:31.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f3169e31640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 msgr2=0x7f316402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f3169e31640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 0x7f316402d450 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3160009740 tx=0x7f3160019190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f316a632640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f315c007fd0 tx=0x7f315c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600b0040 con 0x7f3164138e10 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 10 monclient: dump: 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.893 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31600b7840 con 0x7f3164138e10 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 msgr2=0x7f316402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 0x7f316402d450 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3160009740 tx=0x7f3160019190 comp rx=0 tx=0).stop 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 msgr2=0x7f316402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.881+0000 7f315a7fc640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3164138e10 0x7f316402d450 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3160009740 tx=0x7f3160019190 comp rx=0 tx=0).stop 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.885+0000 7f316c095640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.885+0000 7f316c095640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 0x7f316402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.885+0000 7f316c095640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.885+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3164134c20 con 0x7f31641391f0 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.885+0000 7f3169e31640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 0x7f316402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:31.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient: get_auth_request con 0x7f31641391f0 auth_method 0 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 10 monclient(hunting): handle_auth_done global_id 54367 payload 824 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 1 -- 172.21.15.86:0/1248812093 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f316413ac70 con 0x7f31641391f0 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 msgr2=0x7f316402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f3169e31640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 0x7f316402d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3160000c10 tx=0x7f31600a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31600b0040 con 0x7f31641391f0 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 10 monclient: dump: 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:31.898 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f31600c1b20 con 0x7f31641391f0 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 msgr2=0x7f316402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 0x7f316402d990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3160000c10 tx=0x7f31600a8000 comp rx=0 tx=0).stop 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 msgr2=0x7f316402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.889+0000 7f315a7fc640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31641391f0 0x7f316402d990 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3160000c10 tx=0x7f31600a8000 comp rx=0 tx=0).stop 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:31.899 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 207325, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.900 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.902 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:31.903 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:31.904 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:31.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.908 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:31.910 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:31.911 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:31.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:31.913 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:31.914 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 msgr2=0x7f314803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f314803bb60 0x7f314803e020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f315c007fd0 tx=0x7f315c009040 comp rx=0 tx=0).stop 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 10 monclient: shutdown 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 msgr2=0x7f316413d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 --2- 172.21.15.86:0/1248812093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3164132420 0x7f316413d9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f315400c410 tx=0x7f315400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316a632640 1 -- 172.21.15.86:0/1248812093 reap_dead start 2024-09-17T15:35:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316a632640 1 -- 172.21.15.86:0/1248812093 reap_dead start 2024-09-17T15:35:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 shutdown_connections 2024-09-17T15:35:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 >> 172.21.15.86:0/1248812093 conn(0x7f31640a7210 msgr2=0x7f316409c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 shutdown_connections 2024-09-17T15:35:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:31.893+0000 7f316c095640 1 -- 172.21.15.86:0/1248812093 wait complete. 2024-09-17T15:35:31.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:31.917 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:35:31.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:35:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: get_monmap_and_config 2024-09-17T15:35:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: build_initial_monmap 2024-09-17T15:35:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: monmap: 2024-09-17T15:35:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:32.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:32.030500+0000 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:32.030500+0000 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:32.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 1 Processor -- start 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 1 -- start start 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: init 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93980689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93980694c0 0x7f939806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-c con 0x7f93980694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f939813aff0 0x7f939813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-a con 0x7f939813aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.025+0000 7f939eea9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f939809c8b0 con 0x7f93980689d0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9398074500 con 0x7f939813aff0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f939809cb90 con 0x7f93980694c0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939eea9640 10 monclient(hunting): authenticate will time out at 1464.996035s 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939cc45640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f93980689d0 0x7f9398068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38828/0 (socket says 172.21.15.86:38828) 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f939813aff0 0x7f939813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939cc45640 1 -- 172.21.15.86:0/1219931028 learned_addr learned my addr 172.21.15.86:0/1219931028 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): get_auth_request con 0x7f939813aff0 auth_method 0 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:32.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f9397fff640 1 --2- 172.21.15.86:0/1219931028 >> v2:172.21.15.203:3300/0 conn(0x7f93980694c0 0x7f939806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f9397fff640 10 monclient(hunting): get_auth_request con 0x7f93980694c0 auth_method 0 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f9397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f9397fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f9397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient(hunting): handle_auth_done global_id 94122 payload 274 2024-09-17T15:35:32.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 -- 172.21.15.86:0/1219931028 >> v2:172.21.15.203:3300/0 conn(0x7f93980694c0 msgr2=0x7f939806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 --2- 172.21.15.86:0/1219931028 >> v2:172.21.15.203:3300/0 conn(0x7f93980694c0 0x7f939806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 -- 172.21.15.86:0/1219931028 >> v1:172.21.15.86:6789/0 conn(0x7f93980689d0 legacy=0x7f9398068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient: _finish_hunting 0 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 monclient: found mon.noname-a 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.029+0000 7f939d446640 1 -- 172.21.15.86:0/1219931028 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939809ce70 con 0x7f939813aff0 2024-09-17T15:35:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939d446640 1 -- 172.21.15.86:0/1219931028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f939813aff0 msgr2=0x7f939813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939d446640 1 --2- 172.21.15.86:0/1219931028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f939813aff0 0x7f939813d3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9398069940 tx=0x7f938801c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3839254efe9cf2ce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 1 -- 172.21.15.86:0/1219931028 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93880b0070 con 0x7f939813aff0 2024-09-17T15:35:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: dump: 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 1 -- 172.21.15.86:0/1219931028 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93880ab600 con 0x7f939813aff0 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 5 monclient: authenticate success, global_id 94122 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 4 set_mon_vals no callback set 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 1 -- 172.21.15.86:0/1219931028 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f939809c2d0 con 0x7f939813aff0 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f93977fe640 10 monclient: dump: 2024-09-17T15:35:32.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 10 monclient: shutdown 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:32.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- 172.21.15.86:0/1219931028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f939813aff0 msgr2=0x7f939813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 --2- 172.21.15.86:0/1219931028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f939813aff0 0x7f939813d3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9398069940 tx=0x7f938801c300 comp rx=0 tx=0).stop 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- 172.21.15.86:0/1219931028 shutdown_connections 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 --2- 172.21.15.86:0/1219931028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f939813aff0 0x7f939813d3e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 --2- 172.21.15.86:0/1219931028 >> v2:172.21.15.203:3300/0 conn(0x7f93980694c0 0x7f939806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- 172.21.15.86:0/1219931028 >> 172.21.15.86:0/1219931028 conn(0x7f93980a6440 msgr2=0x7f93980a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- 172.21.15.86:0/1219931028 shutdown_connections 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- 172.21.15.86:0/1219931028 wait complete. 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 10 monclient: build_initial_monmap 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 10 monclient: monmap: 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:32.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 Processor -- start 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 1 -- start start 2024-09-17T15:35:32.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.033+0000 7f939eea9640 10 monclient: init 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980689d0 0x7f9398133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-b con 0x7f93980689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-a con 0x7f93980694c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:32.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f939813aff0 0x7f939809dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): picked mon.noname-c con 0x7f939813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f93981405c0 con 0x7f93980694c0 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9398140440 con 0x7f93980689d0 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9398140740 con 0x7f939813aff0 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient(hunting): authenticate will time out at 1465.004253s 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f939813aff0 0x7f939809dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53178/0 (socket says 172.21.15.86:53178) 2024-09-17T15:35:32.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 -- 172.21.15.86:0/4153289577 learned_addr learned my addr 172.21.15.86:0/4153289577 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): get_auth_request con 0x7f93980694c0 auth_method 0 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): get_auth_request con 0x7f939813aff0 auth_method 0 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939d446640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939cc45640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980689d0 0x7f9398133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939cc45640 10 monclient(hunting): get_auth_request con 0x7f93980689d0 auth_method 0 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939cc45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939cc45640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939cc45640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient(hunting): handle_auth_done global_id 94125 payload 824 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 -- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f939813aff0 msgr2=0x7f939809dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f939813aff0 0x7f939809dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 -- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980689d0 msgr2=0x7f9398133230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980689d0 0x7f9398133230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient: _finish_hunting 0 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 monclient: found mon.noname-a 2024-09-17T15:35:32.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 -- 172.21.15.86:0/4153289577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9398072510 con 0x7f93980694c0 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 -- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 msgr2=0x7f939809d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9397fff640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9390007ef0 tx=0x7f939000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9390013790 con 0x7f93980694c0 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: dump: 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.050 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: _finish_auth 0 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: _check_auth_tickets 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9390019070 con 0x7f93980694c0 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 5 monclient: authenticate success, global_id 94125 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _renew_subs 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9398071f50 con 0x7f93980694c0 2024-09-17T15:35:32.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9390013b70 con 0x7f93980694c0 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f93957fa640 10 monclient: dump: 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:32.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f9396ffd640 4 set_mon_vals no callback set 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _renew_subs 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.037+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93980736f0 con 0x7f93980694c0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939eea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939eea9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939eea9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9398073cc0 con 0x7f93980694c0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9390013d10 con 0x7f93980694c0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f93957fa640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 0x7f937803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(264..264 src has 213..264) ==== 4815+0+0 (secure 0 0 0) 0x7f9390050890 con 0x7f93980694c0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939cc45640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 0x7f937803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939cc45640 10 monclient: get_auth_request con 0x7f937803bb60 auth_method 0 2024-09-17T15:35:32.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.041+0000 7f939cc45640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 0x7f937803e020 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9380007fd0 tx=0x7f9380009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.045+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9398072510 con 0x7f93980694c0 2024-09-17T15:35:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.045+0000 7f93957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.045+0000 7f93957fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:32.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.269+0000 7f939eea9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:35:32.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.269+0000 7f939eea9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:35:32.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.269+0000 7f939eea9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:32.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.269+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f939806d4f0 con 0x7f93980694c0 2024-09-17T15:35:32.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f93957fa640 1 -- 172.21.15.86:0/4153289577 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f939001ee00 con 0x7f93980694c0 2024-09-17T15:35:32.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f93957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:35:32.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f93957fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:32.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=2997033417,config=3541143443,health=2925548024,kv=3591778544,logm=2322039398}) 2024-09-17T15:35:32.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=1,health=10,kv=10,logm=62} crc {auth=2997033417,config=555582061,health=3027944225,kv=2814306489,logm=682504512}) 2024-09-17T15:35:32.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=2997033417,config=3541143443,health=2925548024,kv=3591778544,logm=2322039398}) 2024-09-17T15:35:32.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=11,kv=11,logm=59} crc {auth=2997033417,config=3541143443,health=4009975670,kv=3591778543,logm=2702182668}) 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 msgr2=0x7f937803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 0x7f937803e020 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9380007fd0 tx=0x7f9380009040 comp rx=0 tx=0).stop 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 10 monclient: shutdown 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 msgr2=0x7f939809d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9390007ef0 tx=0x7f939000e4b0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 shutdown_connections 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f937803bb60 0x7f937803e020 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f939813aff0 0x7f939809dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93980694c0 0x7f939809d7a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.273+0000 7f939eea9640 1 --2- 172.21.15.86:0/4153289577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93980689d0 0x7f9398133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:32.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 >> 172.21.15.86:0/4153289577 conn(0x7f93980a6440 msgr2=0x7f93980a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:32.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 shutdown_connections 2024-09-17T15:35:32.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f939eea9640 1 -- 172.21.15.86:0/4153289577 wait complete. 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519761764}) 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=360911717}) 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519761764}) 2024-09-17T15:35:32.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.277+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179288178}) 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082993067}) 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216812697}) 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082993067}) 2024-09-17T15:35:32.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1234081335}) 2024-09-17T15:35:32.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458071912}) 2024-09-17T15:35:32.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847954587}) 2024-09-17T15:35:32.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458071912}) 2024-09-17T15:35:32.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605137387}) 2024-09-17T15:35:32.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524722560}) 2024-09-17T15:35:32.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=991720576}) 2024-09-17T15:35:32.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524722560}) 2024-09-17T15:35:32.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.281+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080186399}) 2024-09-17T15:35:32.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026882713}) 2024-09-17T15:35:32.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717175372}) 2024-09-17T15:35:32.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026882713}) 2024-09-17T15:35:32.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063117159}) 2024-09-17T15:35:32.289 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.289 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245029103}) 2024-09-17T15:35:32.289 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2754192020}) 2024-09-17T15:35:32.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245029103}) 2024-09-17T15:35:32.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.285+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898613418}) 2024-09-17T15:35:32.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708044797}) 2024-09-17T15:35:32.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493638442}) 2024-09-17T15:35:32.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708044797}) 2024-09-17T15:35:32.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840533338}) 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870281899}) 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=548713141}) 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1870281899}) 2024-09-17T15:35:32.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.289+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869865290}) 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128477062}) 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2233023271}) 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128477062}) 2024-09-17T15:35:32.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2971938894}) 2024-09-17T15:35:32.296 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3015181380}) 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274727142}) 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3015181380}) 2024-09-17T15:35:32.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.293+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369020109}) 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785260817}) 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=22672359}) 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2785260817}) 2024-09-17T15:35:32.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4209660777}) 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12012686}) 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3507611474}) 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12012686}) 2024-09-17T15:35:32.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.297+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050811239}) 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792110262}) 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=876000698}) 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792110262}) 2024-09-17T15:35:32.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728555151}) 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186764854}) 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1967730080}) 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186764854}) 2024-09-17T15:35:32.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.301+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907043621}) 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039302776}) 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591880927}) 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2039302776}) 2024-09-17T15:35:32.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4152712714}) 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407126155}) 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969188857}) 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407126155}) 2024-09-17T15:35:32.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.305+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732572134}) 2024-09-17T15:35:32.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142670706}) 2024-09-17T15:35:32.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=217858086}) 2024-09-17T15:35:32.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142670706}) 2024-09-17T15:35:32.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587349341}) 2024-09-17T15:35:32.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184558322}) 2024-09-17T15:35:32.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2466341631}) 2024-09-17T15:35:32.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4184558322}) 2024-09-17T15:35:32.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.309+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195554944}) 2024-09-17T15:35:32.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1288150480}) 2024-09-17T15:35:32.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078999707}) 2024-09-17T15:35:32.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1288150480}) 2024-09-17T15:35:32.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511824227}) 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814038073}) 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635750798}) 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=814038073}) 2024-09-17T15:35:32.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.313+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227941104}) 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3871756336,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1065246131,mon_config_key=1025766423,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3616902286,osd_pg_creating=1936079306,osd_snap=1075536015}) 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=18} crc {logm=2104289026,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1065246131,mon_config_key=915382266,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=1936079306,osd_snap=2995184957}) 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3871756336,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1065246131,mon_config_key=1025766423,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3616902286,osd_pg_creating=1936079306,osd_snap=1075536015}) 2024-09-17T15:35:32.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=12,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=723958863,mdsmap=2815901553,mgr=1375715255,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=520969005,mon_config_key=2888925928,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=1936079306,osd_snap=3244257427}) 2024-09-17T15:35:32.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=3756858613,osdmap=2243637804}) 2024-09-17T15:35:32.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=26,osdmap=74} crc {osd_snap=3665377089,osdmap=2936313390}) 2024-09-17T15:35:32.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=3756858613,osdmap=2243637804}) 2024-09-17T15:35:32.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.317+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=27,osdmap=73} crc {osd_snap=545757083,osdmap=3490167832}) 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=3971772784}) 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=276045484}) 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=3971772784}) 2024-09-17T15:35:32.330 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:35:32.329+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=4077652388}) 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1152: done (7 left) 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1153: delete oid 31 current snap is 123 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: finishing write tid 1 to smithi20328086-34 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: finishing write tid 2 to smithi20328086-34 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: finishing write tid 3 to smithi20328086-34 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: finishing write tid 4 to smithi20328086-34 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: oid 34 updating version 0 to 429 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: oid 34 version 429 is already newer than 428 2024-09-17T15:35:32.360 INFO:tasks.rados.rados.0.smithi203.stdout:1143: oid 34 version 429 is already newer than 427 2024-09-17T15:35:32.361 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 429 (ObjNum 418 snap 123 seq_num 418) dirty exists 2024-09-17T15:35:32.361 INFO:tasks.rados.rados.0.smithi203.stdout:1143: left oid 34 (ObjNum 418 snap 123 seq_num 418) 2024-09-17T15:35:32.361 INFO:tasks.rados.rados.0.smithi203.stdout:1144: expect (ObjNum 410 snap 117 seq_num 410) 2024-09-17T15:35:32.432 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2700183 2024-09-17T15:35:32.443 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 270036 2024-09-17T15:35:32.495 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:32.545 INFO:tasks.rados.rados.0.smithi203.stdout:1146: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:35:32.580 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:32.631 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1885687 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: finishing write tid 1 to smithi20328086-25 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: finishing write tid 2 to smithi20328086-25 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: finishing write tid 3 to smithi20328086-25 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: finishing write tid 4 to smithi20328086-25 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: oid 25 updating version 0 to 389 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: oid 25 version 389 is already newer than 388 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: oid 25 updating version 389 to 390 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 390 (ObjNum 419 snap 123 seq_num 419) dirty exists 2024-09-17T15:35:32.645 INFO:tasks.rados.rados.0.smithi203.stdout:1147: left oid 25 (ObjNum 419 snap 123 seq_num 419) 2024-09-17T15:35:32.646 INFO:tasks.rados.rados.0.smithi203.stdout:1148: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:35:32.706 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:32.829 INFO:tasks.rados.rados.0.smithi203.stdout:1150: expect (ObjNum 274 snap 70 seq_num 274) 2024-09-17T15:35:32.847 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:32.894 INFO:tasks.rados.rados.0.smithi203.stdout:1151: finishing write tid 1 to smithi20328086-17 2024-09-17T15:35:32.894 INFO:tasks.rados.rados.0.smithi203.stdout:1151: finishing write tid 2 to smithi20328086-17 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: finishing write tid 3 to smithi20328086-17 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: finishing write tid 4 to smithi20328086-17 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: finishing write tid 5 to smithi20328086-17 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: oid 17 updating version 0 to 494 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: oid 17 updating version 494 to 496 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: oid 17 version 496 is already newer than 495 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: oid 17 version 496 is already newer than 493 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 496 (ObjNum 420 snap 123 seq_num 420) dirty exists 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1151: left oid 17 (ObjNum 420 snap 123 seq_num 420) 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1143: done (7 left) 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1144: done (6 left) 2024-09-17T15:35:32.895 INFO:tasks.rados.rados.0.smithi203.stdout:1146: done (5 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1147: done (4 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1148: done (3 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1150: done (2 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1151: done (1 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1153: done (0 left) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1154: read oid 39 snap 120 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1154: expect (ObjNum 412 snap 117 seq_num 412) 2024-09-17T15:35:32.896 INFO:tasks.rados.rados.0.smithi203.stdout:1155: copy_from oid 29 from oid 7 current snap is 123 2024-09-17T15:35:32.897 INFO:tasks.rados.rados.0.smithi203.stdout:1156: write oid 27 current snap is 123 2024-09-17T15:35:32.897 INFO:tasks.rados.rados.0.smithi203.stdout:1156: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-17T15:35:32.916 INFO:tasks.rados.rados.0.smithi203.stdout:1156: writing smithi20328086-27 from 692554 to 1247047 tid 1 2024-09-17T15:35:32.935 INFO:tasks.rados.rados.0.smithi203.stdout:1156: writing smithi20328086-27 from 1778107 to 2457912 tid 2 2024-09-17T15:35:32.945 INFO:tasks.rados.rados.0.smithi203.stdout:1156: writing smithi20328086-27 from 2889444 to 3092554 tid 3 2024-09-17T15:35:32.945 INFO:tasks.rados.rados.0.smithi203.stdout:1157: snap_remove snap 118 2024-09-17T15:35:33.297 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:35:33.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:33.354 INFO:tasks.rados.rados.0.smithi203.stdout:1157: done (3 left) 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1158: read oid 36 snap 121 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1158: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1159: read oid 46 snap -1 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1159: expect (ObjNum 398 snap 111 seq_num 398) 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1160: write (excl) oid 25 current snap is 123 2024-09-17T15:35:33.355 INFO:tasks.rados.rados.0.smithi203.stdout:1160: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-17T15:35:33.377 INFO:tasks.rados.rados.0.smithi203.stdout:1160: writing smithi20328086-25 from 709361 to 1339103 tid 1 2024-09-17T15:35:33.393 INFO:tasks.rados.rados.0.smithi203.stdout:1160: writing smithi20328086-25 from 1836589 to 2376405 tid 2 2024-09-17T15:35:33.404 INFO:tasks.rados.rados.0.smithi203.stdout:1160: writing smithi20328086-25 from 3033220 to 3109361 tid 3 2024-09-17T15:35:33.404 INFO:tasks.rados.rados.0.smithi203.stdout:1161: snap_create 2024-09-17T15:35:33.405 INFO:tasks.rados.rados.0.smithi203.stdout:1154: expect (ObjNum 412 snap 117 seq_num 412) 2024-09-17T15:35:33.433 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1251607 2024-09-17T15:35:33.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: get_monmap_and_config 2024-09-17T15:35:33.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: monmap: 2024-09-17T15:35:33.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:33.437673+0000 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:33.437673+0000 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:33.436 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:33.437 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:33.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 Processor -- start 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 -- start start 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: init 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46f00a5070 0x7f46f009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-a con 0x7f46f00a5070 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 0x7f46f00a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-c con 0x7f46f00a5450 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46f0137e30 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46f00a6f40 con 0x7f46f0137e30 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f46f00a65e0 con 0x7f46f00a5070 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f46f009c2a0 con 0x7f46f00a5450 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f7590640 10 monclient(hunting): authenticate will time out at 1466.403118s 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46f00a5070 0x7f46f009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f5b2d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f46f0137e30 0x7f46f0138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38842/0 (socket says 172.21.15.86:38842) 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f5b2d640 1 -- 172.21.15.86:0/3054742787 learned_addr learned my addr 172.21.15.86:0/3054742787 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): get_auth_request con 0x7f46f00a5070 auth_method 0 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f4b2b640 1 --2- 172.21.15.86:0/3054742787 >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 0x7f46f00a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient(hunting): handle_auth_done global_id 94128 payload 274 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 -- 172.21.15.86:0/3054742787 >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 msgr2=0x7f46f00a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 --2- 172.21.15.86:0/3054742787 >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 0x7f46f00a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 -- 172.21.15.86:0/3054742787 >> v1:172.21.15.86:6789/0 conn(0x7f46f0137e30 legacy=0x7f46f0138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 monclient: found mon.noname-a 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f532c640 1 -- 172.21.15.86:0/3054742787 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f009c580 con 0x7f46f00a5070 2024-09-17T15:35:33.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f4b2b640 10 monclient: get_auth_request con 0x7f46f00a5450 auth_method 0 2024-09-17T15:35:33.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.433+0000 7f46f4b2b640 1 --2- 172.21.15.86:0/3054742787 >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 0x7f46f00a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f532c640 1 -- 172.21.15.86:0/3054742787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5070 msgr2=0x7f46f009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f532c640 1 --2- 172.21.15.86:0/3054742787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5070 0x7f46f009cf60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f46f00a17e0 tx=0x7f46ec01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3b943c1a16f160 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 1 -- 172.21.15.86:0/3054742787 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0b0070 con 0x7f46f00a5070 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: dump: 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 1 -- 172.21.15.86:0/3054742787 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46ec0ab600 con 0x7f46f00a5070 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f7590640 5 monclient: authenticate success, global_id 94128 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f7590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f7590640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:33.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 1 -- 172.21.15.86:0/3054742787 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0ab9d0 con 0x7f46f00a5070 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46f7590640 4 set_mon_vals no callback set 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.437+0000 7f46e7fff640 10 monclient: dump: 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: shutdown 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- 172.21.15.86:0/3054742787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5070 msgr2=0x7f46f009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- 172.21.15.86:0/3054742787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5070 0x7f46f009cf60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f46f00a17e0 tx=0x7f46ec01c340 comp rx=0 tx=0).stop 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- 172.21.15.86:0/3054742787 shutdown_connections 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- 172.21.15.86:0/3054742787 >> v2:172.21.15.203:3300/0 conn(0x7f46f00a5450 0x7f46f00a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- 172.21.15.86:0/3054742787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5070 0x7f46f009cf60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- 172.21.15.86:0/3054742787 >> 172.21.15.86:0/3054742787 conn(0x7f46f00742a0 msgr2=0x7f46f00746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- 172.21.15.86:0/3054742787 shutdown_connections 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- 172.21.15.86:0/3054742787 wait complete. 2024-09-17T15:35:33.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: monmap: 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 Processor -- start 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- start start 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: init 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46f00a5070 0x7f46f01cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-b con 0x7f46f00a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-a con 0x7f46f00a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46f0137e30 0x7f46f013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): picked mon.noname-c con 0x7f46f0137e30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f46f0142ac0 con 0x7f46f00a5450 2024-09-17T15:35:33.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46f00a6a70 con 0x7f46f00a5070 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f46f0142c40 con 0x7f46f0137e30 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f7590640 10 monclient(hunting): authenticate will time out at 1466.410169s 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53198/0 (socket says 172.21.15.86:53198) 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 -- 172.21.15.86:0/607476132 learned_addr learned my addr 172.21.15.86:0/607476132 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): get_auth_request con 0x7f46f00a5450 auth_method 0 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46f0137e30 0x7f46f013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): get_auth_request con 0x7f46f0137e30 auth_method 0 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f5b2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f532c640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46f00a5070 0x7f46f01cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f532c640 10 monclient(hunting): get_auth_request con 0x7f46f00a5070 auth_method 0 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f532c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f532c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f532c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient(hunting): handle_auth_done global_id 94131 payload 824 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 -- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46f0137e30 msgr2=0x7f46f013d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46f0137e30 0x7f46f013d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 -- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46f00a5070 msgr2=0x7f46f01cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46f00a5070 0x7f46f01cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 monclient: found mon.noname-a 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.441+0000 7f46f4b2b640 1 -- 172.21.15.86:0/607476132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0121340 con 0x7f46f00a5450 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f4b2b640 1 -- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 msgr2=0x7f46f0136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f4b2b640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c7c0 tx=0x7f46e000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0007c40 con 0x7f46f00a5450 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: dump: 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.456 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e0007de0 con 0x7f46f00a5450 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0011630 con 0x7f46f00a5450 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 5 monclient: authenticate success, global_id 94131 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _renew_subs 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f0121620 con 0x7f46f00a5450 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e6ffd640 4 set_mon_vals no callback set 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 10 monclient: dump: 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _renew_subs 2024-09-17T15:35:33.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f01d0d40 con 0x7f46f00a5450 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f009cf60 con 0x7f46f00a5450 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f46e00117d0 con 0x7f46f00a5450 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 0x7f46d403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f46e0050280 con 0x7f46f00a5450 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f532c640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 0x7f46d403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.445+0000 7f46f532c640 10 monclient: get_auth_request con 0x7f46d403bbb0 auth_method 0 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.449+0000 7f46f532c640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 0x7f46d403e070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46f00a8470 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.449+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f46e0015020 con 0x7f46f00a5450 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.449+0000 7f46e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.449+0000 7f46e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:33.512 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2464899 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1155: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1155: finishing copy_from to smithi20328086-29 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 271 (ObjNum 306 snap 84 seq_num 306) dirty exists 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: finishing write tid 1 to smithi20328086-27 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: finishing write tid 2 to smithi20328086-27 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: finishing write tid 3 to smithi20328086-27 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: finishing write tid 4 to smithi20328086-27 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: finishing write tid 5 to smithi20328086-27 2024-09-17T15:35:33.566 INFO:tasks.rados.rados.0.smithi203.stdout:1156: oid 27 updating version 0 to 348 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:1156: oid 27 version 348 is already newer than 347 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:1156: oid 27 version 348 is already newer than 345 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:1156: oid 27 version 348 is already newer than 346 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 348 (ObjNum 422 snap 123 seq_num 422) dirty exists 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:1156: left oid 27 (ObjNum 422 snap 123 seq_num 422) 2024-09-17T15:35:33.567 INFO:tasks.rados.rados.0.smithi203.stdout:1158: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:35:33.617 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:33.679 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:33.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46f7590640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:33.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46f7590640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:33.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46f7590640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f46f0137c10 con 0x7f46f00a5450 2024-09-17T15:35:33.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46e5ffb640 1 -- 172.21.15.86:0/607476132 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f46f0137c10 con 0x7f46f00a5450 2024-09-17T15:35:33.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:33.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.685+0000 7f46e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:33.688 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 msgr2=0x7f46d403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 0x7f46d403e070 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46f00a8470 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-09-17T15:35:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 10 monclient: shutdown 2024-09-17T15:35:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 msgr2=0x7f46f0136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c7c0 tx=0x7f46e000cc80 comp rx=0 tx=0).stop 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 shutdown_connections 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46d403bbb0 0x7f46d403e070 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46f0137e30 0x7f46f013d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46f00a5450 0x7f46f0136f70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 --2- 172.21.15.86:0/607476132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46f00a5070 0x7f46f01cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 >> 172.21.15.86:0/607476132 conn(0x7f46f00742a0 msgr2=0x7f46f00a9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 shutdown_connections 2024-09-17T15:35:33.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.689+0000 7f46f7590640 1 -- 172.21.15.86:0/607476132 wait complete. 2024-09-17T15:35:33.707 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:35:33.708 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:35:33.708 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:35:33.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:35:33.742 INFO:tasks.rados.rados.0.smithi203.stdout:1159: expect (ObjNum 398 snap 111 seq_num 398) 2024-09-17T15:35:33.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:33.808 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:33.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: get_monmap_and_config 2024-09-17T15:35:33.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: monmap: 2024-09-17T15:35:33.819 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:33.820466+0000 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:33.820466+0000 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:33.820 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 Processor -- start 2024-09-17T15:35:33.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 -- start start 2024-09-17T15:35:33.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: init 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3d1c0a0c70 0x7f3d1c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c0a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d1c0a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1c0a2130 con 0x7f3d1c0a1590 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1c0a2410 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1c0a26f0 con 0x7f3d1c136160 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d23281640 10 monclient(hunting): authenticate will time out at 1466.786444s 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45706/0 (socket says 172.21.15.86:45706) 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 -- 172.21.15.86:0/257998828 learned_addr learned my addr 172.21.15.86:0/257998828 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): get_auth_request con 0x7f3d1c136160 auth_method 0 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2101d640 1 --2- 172.21.15.86:0/257998828 >> v2:172.21.15.203:3300/0 conn(0x7f3d1c0a0c70 0x7f3d1c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2101d640 10 monclient(hunting): get_auth_request con 0x7f3d1c0a0c70 auth_method 0 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2101d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2101d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2101d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient(hunting): handle_auth_done global_id 74297 payload 274 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 -- 172.21.15.86:0/257998828 >> v2:172.21.15.203:3300/0 conn(0x7f3d1c0a0c70 msgr2=0x7f3d1c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 --2- 172.21.15.86:0/257998828 >> v2:172.21.15.203:3300/0 conn(0x7f3d1c0a0c70 0x7f3d1c0a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 -- 172.21.15.86:0/257998828 >> v1:172.21.15.203:6789/0 conn(0x7f3d1c0a1590 legacy=0x7f3d1c135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 monclient: found mon.noname-b 2024-09-17T15:35:33.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:35:33.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.817+0000 7f3d2181e640 1 -- 172.21.15.86:0/257998828 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c09f9f0 con 0x7f3d1c136160 2024-09-17T15:35:33.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d2181e640 1 -- 172.21.15.86:0/257998828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 msgr2=0x7f3d1c13df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d2181e640 1 --2- 172.21.15.86:0/257998828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009e50 tx=0x7f3d1401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ec4759fecfc480e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 1 -- 172.21.15.86:0/257998828 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d140a5750 con 0x7f3d1c136160 2024-09-17T15:35:33.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: dump: 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d23281640 5 monclient: authenticate success, global_id 74297 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d23281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 1 -- 172.21.15.86:0/257998828 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d140a58f0 con 0x7f3d1c136160 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d23281640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d23281640 4 set_mon_vals no callback set 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 1 -- 172.21.15.86:0/257998828 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d140a5cc0 con 0x7f3d1c136160 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.821+0000 7f3d13fff640 10 monclient: dump: 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.831 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: shutdown 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- 172.21.15.86:0/257998828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 msgr2=0x7f3d1c13df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- 172.21.15.86:0/257998828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009e50 tx=0x7f3d1401c8f0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- 172.21.15.86:0/257998828 shutdown_connections 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- 172.21.15.86:0/257998828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c13df30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- 172.21.15.86:0/257998828 >> v2:172.21.15.203:3300/0 conn(0x7f3d1c0a0c70 0x7f3d1c0a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- 172.21.15.86:0/257998828 >> 172.21.15.86:0/257998828 conn(0x7f3d1c0a2f40 msgr2=0x7f3d1c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- 172.21.15.86:0/257998828 shutdown_connections 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- 172.21.15.86:0/257998828 wait complete. 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: monmap: 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.833 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 Processor -- start 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- start start 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: init 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 0x7f3d1c11e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-a con 0x7f3d1c0a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d1c0a1590 0x7f3d1c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c0a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c135200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c0a00d0 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c09ff50 con 0x7f3d1c0a1590 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1c140c90 con 0x7f3d1c136160 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d23281640 10 monclient(hunting): authenticate will time out at 1466.794215s 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 0x7f3d1c11e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c135200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c135200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45722/0 (socket says 172.21.15.86:45722) 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 0x7f3d1c11e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53214/0 (socket says 172.21.15.86:53214) 2024-09-17T15:35:33.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 1 -- 172.21.15.86:0/3548789726 learned_addr learned my addr 172.21.15.86:0/3548789726 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): get_auth_request con 0x7f3d1c136160 auth_method 0 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): get_auth_request con 0x7f3d1c0a0c70 auth_method 0 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2081c640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d1c0a1590 0x7f3d1c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2081c640 10 monclient(hunting): get_auth_request con 0x7f3d1c0a1590 auth_method 0 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2081c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2081c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2081c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient(hunting): handle_auth_done global_id 94134 payload 824 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 msgr2=0x7f3d1c135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c135200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d1c0a1590 msgr2=0x7f3d1c11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d1c0a1590 0x7f3d1c11c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 monclient: found mon.noname-a 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2101d640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1c09d460 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.825+0000 7f3d2181e640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d1c136160 0x7f3d1c135200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:33.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2101d640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 msgr2=0x7f3d1c11e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2101d640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 0x7f3d1c11e390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00d600 tx=0x7f3d0c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c012800 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: dump: 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d0c019070 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 5 monclient: authenticate success, global_id 94134 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 monclient: _renew_subs 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1c11f2f0 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c012bc0 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 10 monclient: dump: 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.843 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 monclient: _renew_subs 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d12ffd640 4 set_mon_vals no callback set 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1c135ea0 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3d0c01e460 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 0x7f3d1c1cdda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c0a1050 con 0x7f3d1c135740 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bb60 0x7f3d0003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f3d0c0502c0 con 0x7f3d1c0a0c70 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 0x7f3d1c1cdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient: get_auth_request con 0x7f3d1c135740 auth_method 0 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2181e640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bb60 0x7f3d0003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2181e640 10 monclient: get_auth_request con 0x7f3d0003bb60 auth_method 0 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 10 monclient(hunting): handle_auth_done global_id 94137 payload 824 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.829+0000 7f3d2081c640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d1c11f680 con 0x7f3d1c135740 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d2081c640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 msgr2=0x7f3d1c1cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d2081c640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 0x7f3d1c1cdda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c13dc50 tx=0x7f3d1800e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18007450 con 0x7f3d1c135740 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 10 monclient: dump: 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d2181e640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bb60 0x7f3d0003e020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009040 tx=0x7f3d140a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d18007450 con 0x7f3d1c135740 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 msgr2=0x7f3d1c1cdda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 0x7f3d1c1cdda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c13dc50 tx=0x7f3d1800e8b0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 msgr2=0x7f3d1c1cdda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.833+0000 7f3d11ffb640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c135740 0x7f3d1c1cdda0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c13dc50 tx=0x7f3d1800e8b0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d23281640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d23281640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 0x7f3d1c1ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d23281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1c1cec30 con 0x7f3d1c1ce2e0 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 0x7f3d1c1ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient: get_auth_request con 0x7f3d1c1ce2e0 auth_method 0 2024-09-17T15:35:33.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 10 monclient(hunting): handle_auth_done global_id 94140 payload 824 2024-09-17T15:35:33.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.837+0000 7f3d2081c640 1 -- 172.21.15.86:0/3548789726 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d1c06d4f0 con 0x7f3d1c1ce2e0 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: finishing write tid 1 to smithi20328086-25 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: finishing write tid 2 to smithi20328086-25 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: finishing write tid 3 to smithi20328086-25 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: finishing write tid 4 to smithi20328086-25 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: finishing write tid 5 to smithi20328086-25 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: oid 25 updating version 0 to 404 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: oid 25 version 404 is already newer than 403 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: oid 25 version 404 is already newer than 401 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: oid 25 version 404 is already newer than 402 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 404 (ObjNum 423 snap 123 seq_num 423) dirty exists 2024-09-17T15:35:33.850 INFO:tasks.rados.rados.0.smithi203.stdout:1160: left oid 25 (ObjNum 423 snap 123 seq_num 423) 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d2081c640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 msgr2=0x7f3d1c1ce6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d2081c640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 0x7f3d1c1ce6f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c13dc50 tx=0x7f3d18003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1801e960 con 0x7f3d1c1ce2e0 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 10 monclient: dump: 2024-09-17T15:35:33.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f3d18028460 con 0x7f3d1c1ce2e0 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:33.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 msgr2=0x7f3d1c1ce6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 0x7f3d1c1ce6f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c13dc50 tx=0x7f3d18003770 comp rx=0 tx=0).stop 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 msgr2=0x7f3d1c1ce6f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d11ffb640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c1ce2e0 0x7f3d1c1ce6f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:33.853 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9195, 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:33.854 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:33.855 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:33.856 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:33.857 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:33.858 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.859 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.860 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.861 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:33.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:33.863 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.864 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:33.865 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:33.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bb60 msgr2=0x7f3d0003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bb60 0x7f3d0003e020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009040 tx=0x7f3d140a7070 comp rx=0 tx=0).stop 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 10 monclient: shutdown 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 msgr2=0x7f3d1c11e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 --2- 172.21.15.86:0/3548789726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d1c0a0c70 0x7f3d1c11e390 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00d600 tx=0x7f3d0c00dac0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d2181e640 1 -- 172.21.15.86:0/3548789726 reap_dead start 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d2181e640 1 -- 172.21.15.86:0/3548789726 reap_dead start 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 shutdown_connections 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 >> 172.21.15.86:0/3548789726 conn(0x7f3d1c0a2f40 msgr2=0x7f3d1c071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 shutdown_connections 2024-09-17T15:35:33.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.841+0000 7f3d23281640 1 -- 172.21.15.86:0/3548789726 wait complete. 2024-09-17T15:35:33.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:33.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:35:33.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:35:33.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.969+0000 7f696deee640 10 monclient: get_monmap_and_config 2024-09-17T15:35:33.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.969+0000 7f696deee640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.969+0000 7f696deee640 10 monclient: monmap: 2024-09-17T15:35:33.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:33.975809+0000 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:33.975809+0000 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:33.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 Processor -- start 2024-09-17T15:35:33.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 -- start start 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient: init 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69680a4f70 0x7f696809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): picked mon.noname-a con 0x7f69680a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69680a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f696813d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696809d430 con 0x7f696813d180 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69680a68d0 con 0x7f69680a5350 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f696809c7f0 con 0x7f69680a4f70 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f696deee640 10 monclient(hunting): authenticate will time out at 1466.941333s 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69680a4f70 0x7f696809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69680a4f70 0x7f696809cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53242/0 (socket says 172.21.15.86:53242) 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 -- 172.21.15.86:0/1032736100 learned_addr learned my addr 172.21.15.86:0/1032736100 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): get_auth_request con 0x7f69680a4f70 auth_method 0 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f6967fff640 1 --1- 172.21.15.86:0/1032736100 >> v1:172.21.15.86:6789/0 conn(0x7f696813d180 0x7f696813f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38844/0 (socket says 172.21.15.86:38844) 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient(hunting): handle_auth_done global_id 94143 payload 274 2024-09-17T15:35:33.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 -- 172.21.15.86:0/1032736100 >> v1:172.21.15.203:6789/0 conn(0x7f69680a5350 legacy=0x7f69680a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 -- 172.21.15.86:0/1032736100 >> v1:172.21.15.86:6789/0 conn(0x7f696813d180 legacy=0x7f696813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 monclient: found mon.noname-a 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:33.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.973+0000 7f69677fe640 1 -- 172.21.15.86:0/1032736100 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69680a64a0 con 0x7f69680a4f70 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69677fe640 1 -- 172.21.15.86:0/1032736100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69680a4f70 msgr2=0x7f696809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69677fe640 1 --2- 172.21.15.86:0/1032736100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69680a4f70 0x7f696809cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f695c009870 tx=0x7f695c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8986571f7fa0734 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 1 -- 172.21.15.86:0/1032736100 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0b0070 con 0x7f69680a4f70 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: dump: 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 1 -- 172.21.15.86:0/1032736100 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c0ab600 con 0x7f69680a4f70 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 1 -- 172.21.15.86:0/1032736100 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0ab9d0 con 0x7f69680a4f70 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 5 monclient: authenticate success, global_id 94143 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 4 set_mon_vals no callback set 2024-09-17T15:35:33.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f69667fc640 10 monclient: dump: 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 10 monclient: shutdown 2024-09-17T15:35:33.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- 172.21.15.86:0/1032736100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69680a4f70 msgr2=0x7f696809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 --2- 172.21.15.86:0/1032736100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69680a4f70 0x7f696809cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f695c009870 tx=0x7f695c01c420 comp rx=0 tx=0).stop 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- 172.21.15.86:0/1032736100 shutdown_connections 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 --2- 172.21.15.86:0/1032736100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69680a4f70 0x7f696809cef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- 172.21.15.86:0/1032736100 >> 172.21.15.86:0/1032736100 conn(0x7f69680a7240 msgr2=0x7f69680a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- 172.21.15.86:0/1032736100 shutdown_connections 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- 172.21.15.86:0/1032736100 wait complete. 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 10 monclient: build_initial_monmap 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 10 monclient: monmap: 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:33.985 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.985 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:33.985 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:33.985 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:33.985 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 Processor -- start 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 1 -- start start 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.977+0000 7f696deee640 10 monclient: init 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:35:33.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 0x7f696813ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): picked mon.noname-c con 0x7f69680a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69680a5350 0x7f6968139a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): picked mon.noname-b con 0x7f69680a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f696813d180 0x7f69681382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): picked mon.noname-a con 0x7f696813d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6968143930 con 0x7f696813d180 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69681437b0 con 0x7f69680a5350 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6968143ab0 con 0x7f69680a4f70 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:33.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient(hunting): authenticate will time out at 1466.948528s 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 0x7f696813ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f696813d180 0x7f69681382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 0x7f696813ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45728/0 (socket says 172.21.15.86:45728) 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 -- 172.21.15.86:0/2157077639 learned_addr learned my addr 172.21.15.86:0/2157077639 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): get_auth_request con 0x7f69680a4f70 auth_method 0 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): get_auth_request con 0x7f696813d180 auth_method 0 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6966ffd640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69680a5350 0x7f6968139a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6966ffd640 10 monclient(hunting): get_auth_request con 0x7f69680a5350 auth_method 0 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient(hunting): handle_auth_done global_id 74303 payload 824 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69680a5350 msgr2=0x7f6968139a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69680a5350 0x7f6968139a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f696813d180 msgr2=0x7f69681382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f696813d180 0x7f69681382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 monclient: found mon.noname-c 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69680a1760 con 0x7f69680a4f70 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6967fff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f696813d180 0x7f69681382b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 msgr2=0x7f696813ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f69677fe640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 0x7f696813ba40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f695c01ccc0 tx=0x7f695c01ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0ba020 con 0x7f69680a4f70 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: dump: 2024-09-17T15:35:33.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: _finish_auth 0 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 5 monclient: authenticate success, global_id 74303 2024-09-17T15:35:33.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _renew_subs 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f695c0b6630 con 0x7f69680a4f70 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6968121390 con 0x7f69680a4f70 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f695c0b0040 con 0x7f69680a4f70 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f694ffff640 10 monclient: dump: 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.996 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _renew_subs 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f6965ffb640 4 set_mon_vals no callback set 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.981+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6968139f90 con 0x7f69680a4f70 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f696deee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f696deee640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 0x7f69681d0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:33.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69681408d0 con 0x7f69681387f0 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 0x7f69681d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient: get_auth_request con 0x7f69681387f0 auth_method 0 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f695c0bd020 con 0x7f69680a4f70 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 10 monclient(hunting): handle_auth_done global_id 74306 payload 824 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6968138bd0 con 0x7f69681387f0 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f695403bb60 0x7f695403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f695c0ea300 con 0x7f69680a4f70 2024-09-17T15:35:33.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6967fff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f695403bb60 0x7f695403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6967fff640 10 monclient: get_auth_request con 0x7f695403bb60 auth_method 0 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 msgr2=0x7f69681d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f6966ffd640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 0x7f69681d0000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f695800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958007450 con 0x7f69681387f0 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 10 monclient: dump: 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:33.999 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6958007450 con 0x7f69681387f0 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 msgr2=0x7f69681d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 0x7f69681d0000 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f695800e910 comp rx=0 tx=0).stop 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 msgr2=0x7f69681d0000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.985+0000 7f694ffff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681387f0 0x7f69681d0000 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f695800e910 comp rx=0 tx=0).stop 2024-09-17T15:35:34.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.989+0000 7f6967fff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f695403bb60 0x7f695403e020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f696809cce0 tx=0x7f6960004f40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f696deee640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f696deee640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 0x7f69681d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f696deee640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69681d0e90 con 0x7f69681d0540 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 0x7f69681d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient: get_auth_request con 0x7f69681d0540 auth_method 0 2024-09-17T15:35:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 10 monclient(hunting): handle_auth_done global_id 74309 payload 824 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 1 -- 172.21.15.86:0/2157077639 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69680a5ff0 con 0x7f69681d0540 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 msgr2=0x7f69681d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.993+0000 7f6966ffd640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 0x7f69681d0950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f6958003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6958006e60 con 0x7f69681d0540 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 10 monclient: dump: 2024-09-17T15:35:34.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f6958013420 con 0x7f69681d0540 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:34.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 msgr2=0x7f69681d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 0x7f69681d0950 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f6958003770 comp rx=0 tx=0).stop 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 msgr2=0x7f69681d0950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f694ffff640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69681d0540 0x7f69681d0950 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f695800e450 tx=0x7f6958003770 comp rx=0 tx=0).stop 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:34.005 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 114786, 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.006 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.007 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:34.008 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:34.009 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.010 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:34.011 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.012 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.016 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:34.017 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.018 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.019 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.020 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:34.021 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:34.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f695403bb60 msgr2=0x7f695403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f695403bb60 0x7f695403e020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f696809cce0 tx=0x7f6960004f40 comp rx=0 tx=0).stop 2024-09-17T15:35:34.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 10 monclient: shutdown 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 msgr2=0x7f696813ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 --2- 172.21.15.86:0/2157077639 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69680a4f70 0x7f696813ba40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f695c01ccc0 tx=0x7f695c01ac60 comp rx=0 tx=0).stop 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f6967fff640 1 -- 172.21.15.86:0/2157077639 reap_dead start 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f6967fff640 1 -- 172.21.15.86:0/2157077639 reap_dead start 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 shutdown_connections 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 >> 172.21.15.86:0/2157077639 conn(0x7f69680a7240 msgr2=0x7f69680a07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 shutdown_connections 2024-09-17T15:35:34.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:33.997+0000 7f696deee640 1 -- 172.21.15.86:0/2157077639 wait complete. 2024-09-17T15:35:34.023 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:34.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:35:34.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:35:34.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.125+0000 7fbd3be89640 10 monclient: get_monmap_and_config 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.125+0000 7fbd3be89640 10 monclient: build_initial_monmap 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.125+0000 7fbd3be89640 10 monclient: monmap: 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:34.130964+0000 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:34.130964+0000 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:34.130 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:34.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 Processor -- start 2024-09-17T15:35:34.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 -- start start 2024-09-17T15:35:34.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient: init 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-a con 0x7fbd340a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 0x7fbd34135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-b con 0x7fbd340a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbd34135f90 0x7fbd34136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-c con 0x7fbd34135f90 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbd340a2410 con 0x7fbd340a0c70 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbd340a2130 con 0x7fbd34135f90 2024-09-17T15:35:34.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbd340a26f0 con 0x7fbd340a1590 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3be89640 10 monclient(hunting): authenticate will time out at 1467.096908s 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 0x7fbd34135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53254/0 (socket says 172.21.15.86:53254) 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 1 -- 172.21.15.86:0/2663729674 learned_addr learned my addr 172.21.15.86:0/2663729674 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): get_auth_request con 0x7fbd340a0c70 auth_method 0 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): get_auth_request con 0x7fbd340a1590 auth_method 0 2024-09-17T15:35:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3a426640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.203:3300/0 conn(0x7fbd34135f90 0x7fbd34136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3a426640 10 monclient(hunting): get_auth_request con 0x7fbd34135f90 auth_method 0 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3a426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3a426640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd3a426640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient(hunting): handle_auth_done global_id 74312 payload 274 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 -- 172.21.15.86:0/2663729674 >> v2:172.21.15.203:3300/0 conn(0x7fbd34135f90 msgr2=0x7fbd34136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.203:3300/0 conn(0x7fbd34135f90 0x7fbd34136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 -- 172.21.15.86:0/2663729674 >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 msgr2=0x7fbd340a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient: _finish_hunting 0 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 monclient: found mon.noname-b 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39424640 1 -- 172.21.15.86:0/2663729674 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3409f9f0 con 0x7fbd340a1590 2024-09-17T15:35:34.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.129+0000 7fbd39c25640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:35:34.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 1 -- 172.21.15.86:0/2663729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 msgr2=0x7fbd34135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 1 --2- 172.21.15.86:0/2663729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 0x7fbd34135a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd340a40b0 tx=0x7fbd2401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bc1a523bd7b96f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 1 -- 172.21.15.86:0/2663729674 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240b0070 con 0x7fbd340a1590 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: dump: 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: _finish_auth 0 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: _check_auth_tickets 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 5 monclient: authenticate success, global_id 74312 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 1 -- 172.21.15.86:0/2663729674 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd240ab600 con 0x7fbd340a1590 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 1 -- 172.21.15.86:0/2663729674 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240ab9d0 con 0x7fbd340a1590 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd38c23640 10 monclient: dump: 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.145 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 4 set_mon_vals no callback set 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: shutdown 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:34.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- 172.21.15.86:0/2663729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 msgr2=0x7fbd34135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- 172.21.15.86:0/2663729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 0x7fbd34135a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbd340a40b0 tx=0x7fbd2401c340 comp rx=0 tx=0).stop 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- 172.21.15.86:0/2663729674 shutdown_connections 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.203:3300/0 conn(0x7fbd34135f90 0x7fbd34136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- 172.21.15.86:0/2663729674 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd340a1590 0x7fbd34135a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- 172.21.15.86:0/2663729674 >> v2:172.21.15.86:3300/0 conn(0x7fbd340a0c70 0x7fbd340a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- 172.21.15.86:0/2663729674 >> 172.21.15.86:0/2663729674 conn(0x7fbd340a2f40 msgr2=0x7fbd340a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- 172.21.15.86:0/2663729674 shutdown_connections 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- 172.21.15.86:0/2663729674 wait complete. 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: build_initial_monmap 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: monmap: 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:34.147 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 Processor -- start 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- start start 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: init 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd340a0c70 0x7fbd341cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-b con 0x7fbd340a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 0x7fbd34133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-a con 0x7fbd340a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd34135f90 0x7fbd341343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): picked mon.noname-c con 0x7fbd34135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbd340a00d0 con 0x7fbd340a1590 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3409ff50 con 0x7fbd340a0c70 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3413f740 con 0x7fbd34135f90 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3be89640 10 monclient(hunting): authenticate will time out at 1467.103828s 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd34135f90 0x7fbd341343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 0x7fbd34133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 0x7fbd34133e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53270/0 (socket says 172.21.15.86:53270) 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 1 -- 172.21.15.86:0/4081136908 learned_addr learned my addr 172.21.15.86:0/4081136908 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 10 monclient(hunting): get_auth_request con 0x7fbd340a1590 auth_method 0 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd34135f90 0x7fbd341343a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45780/0 (socket says 172.21.15.86:45780) 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd39424640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 10 monclient(hunting): get_auth_request con 0x7fbd34135f90 auth_method 0 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.133+0000 7fbd3a426640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3a426640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3a426640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3a426640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39c25640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd340a0c70 0x7fbd341cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39c25640 10 monclient(hunting): get_auth_request con 0x7fbd340a0c70 auth_method 0 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39c25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient(hunting): handle_auth_done global_id 94152 payload 824 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd34135f90 msgr2=0x7fbd341343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd34135f90 0x7fbd341343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd340a0c70 msgr2=0x7fbd341cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd340a0c70 0x7fbd341cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient: _finish_hunting 0 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 monclient: found mon.noname-a 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3411f4c0 con 0x7fbd340a1590 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 msgr2=0x7fbd34133e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd39424640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 0x7fbd34133e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd2401cbe0 tx=0x7fbd240190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240b0040 con 0x7fbd340a1590 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: dump: 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.152 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd240b9070 con 0x7fbd340a1590 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd240aaa10 con 0x7fbd340a1590 2024-09-17T15:35:34.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 10 monclient: dump: 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 5 monclient: authenticate success, global_id 94152 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2bfff640 4 set_mon_vals no callback set 2024-09-17T15:35:34.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 monclient: _renew_subs 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3409d460 con 0x7fbd340a1590 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 monclient: _renew_subs 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd341205e0 con 0x7fbd340a1590 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbd240ab4c0 con 0x7fbd340a1590 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 0x7fbd3411de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3409c1c0 con 0x7fbd3411da50 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbd1803bbb0 0x7fbd1803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.137+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7fbd240e9cb0 con 0x7fbd340a1590 2024-09-17T15:35:34.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 0x7fbd3411de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient: get_auth_request con 0x7fbd3411da50 auth_method 0 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd3a426640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbd1803bbb0 0x7fbd1803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd3a426640 10 monclient: get_auth_request con 0x7fbd1803bbb0 auth_method 0 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 10 monclient(hunting): handle_auth_done global_id 54394 payload 824 2024-09-17T15:35:34.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd34135a50 con 0x7fbd3411da50 2024-09-17T15:35:34.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd3a426640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbd1803bbb0 0x7fbd1803e070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd3411f8f0 tx=0x7fbd2c009770 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 msgr2=0x7fbd3411de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd39c25640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 0x7fbd3411de30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300119c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30015b60 con 0x7fbd3411da50 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.141+0000 7fbd2a7fc640 10 monclient: dump: 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbd30015b60 con 0x7fbd3411da50 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 msgr2=0x7fbd3411de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 0x7fbd3411de30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300119c0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 msgr2=0x7fbd3411de30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.145+0000 7fbd2a7fc640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411da50 0x7fbd3411de30 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300119c0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd3be89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd3be89640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 0x7fbd3406f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd3be89640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd341d01e0 con 0x7fbd3411e370 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 0x7fbd3406f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient: get_auth_request con 0x7fbd3411e370 auth_method 0 2024-09-17T15:35:34.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 10 monclient(hunting): handle_auth_done global_id 54397 payload 824 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.149+0000 7fbd39c25640 1 -- 172.21.15.86:0/4081136908 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd3406d4f0 con 0x7fbd3411e370 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd39c25640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 msgr2=0x7fbd3406f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd39c25640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 0x7fbd3406f900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30026070 con 0x7fbd3411e370 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 10 monclient: dump: 2024-09-17T15:35:34.161 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fbd300066d0 con 0x7fbd3411e370 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 msgr2=0x7fbd3406f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 0x7fbd3406f900 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300046d0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 msgr2=0x7fbd3406f900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd2a7fc640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd3411e370 0x7fbd3406f900 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd30011500 tx=0x7fbd300046d0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 82, 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 209585, 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:34.163 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.164 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.165 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:34.166 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:34.167 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:34.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.169 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:34.170 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.172 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:34.173 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.174 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.175 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:34.176 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:34.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbd1803bbb0 msgr2=0x7fbd1803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbd1803bbb0 0x7fbd1803e070 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbd3411f8f0 tx=0x7fbd2c009770 comp rx=0 tx=0).stop 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 10 monclient: shutdown 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 msgr2=0x7fbd34133e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 --2- 172.21.15.86:0/4081136908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd340a1590 0x7fbd34133e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd2401cbe0 tx=0x7fbd240190e0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3a426640 1 -- 172.21.15.86:0/4081136908 reap_dead start 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3a426640 1 -- 172.21.15.86:0/4081136908 reap_dead start 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 shutdown_connections 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 >> 172.21.15.86:0/4081136908 conn(0x7fbd340a2f40 msgr2=0x7fbd3413b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 shutdown_connections 2024-09-17T15:35:34.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.153+0000 7fbd3be89640 1 -- 172.21.15.86:0/4081136908 wait complete. 2024-09-17T15:35:34.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:35:34.179 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:35:34.179 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:35:34.179 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:35:34.179 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:35:34.179 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:35:34.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:35:34.191 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:35:34.191 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:35:34.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:34.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: get_monmap_and_config 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: build_initial_monmap 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: monmap: 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:34.357024+0000 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:34.357024+0000 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:34.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:34.356 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:34.356 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:34.356 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 Processor -- start 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 -- start start 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: init 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5de80a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5de80a54c0 0x7f5de80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-c con 0x7f5de80a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:34.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5de813d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de80a6530 con 0x7f5de813d2f0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5de809c150 con 0x7f5de80a50e0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5de809c430 con 0x7f5de80a54c0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5deed7e640 10 monclient(hunting): authenticate will time out at 1467.323340s 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ded31b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5de813d2f0 0x7f5de813f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38852/0 (socket says 172.21.15.86:38852) 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ded31b640 1 -- 172.21.15.86:0/1841583966 learned_addr learned my addr 172.21.15.86:0/1841583966 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 1 --2- 172.21.15.86:0/1841583966 >> v2:172.21.15.203:3300/0 conn(0x7f5de80a54c0 0x7f5de80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): get_auth_request con 0x7f5de80a54c0 auth_method 0 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.353+0000 7f5ddffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 10 monclient(hunting): handle_auth_done global_id 54400 payload 274 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 1 -- 172.21.15.86:0/1841583966 >> v1:172.21.15.203:6789/0 conn(0x7f5de80a50e0 legacy=0x7f5de809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 1 -- 172.21.15.86:0/1841583966 >> v1:172.21.15.86:6789/0 conn(0x7f5de813d2f0 legacy=0x7f5de813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 10 monclient: _finish_hunting 0 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 1 monclient: found mon.noname-c 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:35:34.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.357+0000 7f5ddffff640 1 -- 172.21.15.86:0/1841583966 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de80a6940 con 0x7f5de80a54c0 2024-09-17T15:35:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddffff640 1 -- 172.21.15.86:0/1841583966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a54c0 msgr2=0x7f5de80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddffff640 1 --2- 172.21.15.86:0/1841583966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a54c0 0x7f5de80a58d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80097c0 tx=0x7f5dd801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cb6c04cd13c62f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 1 -- 172.21.15.86:0/1841583966 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80b0070 con 0x7f5de80a54c0 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: dump: 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.363 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 5 monclient: authenticate success, global_id 54400 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 1 -- 172.21.15.86:0/1841583966 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd80ab600 con 0x7f5de80a54c0 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 1 -- 172.21.15.86:0/1841583966 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd80ab9d0 con 0x7f5de80a54c0 2024-09-17T15:35:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddf7fe640 10 monclient: dump: 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:34.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 4 set_mon_vals no callback set 2024-09-17T15:35:34.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: shutdown 2024-09-17T15:35:34.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- 172.21.15.86:0/1841583966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a54c0 msgr2=0x7f5de80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 --2- 172.21.15.86:0/1841583966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a54c0 0x7f5de80a58d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5dd80097c0 tx=0x7f5dd801c3f0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- 172.21.15.86:0/1841583966 shutdown_connections 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 --2- 172.21.15.86:0/1841583966 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a54c0 0x7f5de80a58d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- 172.21.15.86:0/1841583966 >> 172.21.15.86:0/1841583966 conn(0x7f5de80aaae0 msgr2=0x7f5de80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- 172.21.15.86:0/1841583966 shutdown_connections 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- 172.21.15.86:0/1841583966 wait complete. 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: build_initial_monmap 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: monmap: 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:34.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 Processor -- start 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- start start 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: init 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:34.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a50e0 0x7f5de81cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-b con 0x7f5de80a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 0x7f5de81381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-c con 0x7f5de80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): picked mon.noname-a con 0x7f5de813d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81439f0 con 0x7f5de813d2f0 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5de8143870 con 0x7f5de80a50e0 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5de8143b70 con 0x7f5de80a54c0 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5deed7e640 10 monclient(hunting): authenticate will time out at 1467.331984s 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.361+0000 7f5ddffff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 msgr2=0x7f5de81381e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 0x7f5de81381e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53310/0 (socket says 172.21.15.86:53310) 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 -- 172.21.15.86:0/907492004 learned_addr learned my addr 172.21.15.86:0/907492004 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): get_auth_request con 0x7f5de813d2f0 auth_method 0 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5decb1a640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a50e0 0x7f5de81cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5decb1a640 10 monclient(hunting): get_auth_request con 0x7f5de80a50e0 auth_method 0 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5decb1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5decb1a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5decb1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient(hunting): handle_auth_done global_id 94155 payload 824 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 -- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 msgr2=0x7f5de81381e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 0x7f5de81381e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 -- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a50e0 msgr2=0x7f5de81cfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a50e0 0x7f5de81cfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient: _finish_hunting 0 2024-09-17T15:35:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 monclient: found mon.noname-a 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 -- 172.21.15.86:0/907492004 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de8121880 con 0x7f5de813d2f0 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 -- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 msgr2=0x7f5de80a1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ded31b640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5de000cac0 tx=0x7f5de000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de000ae40 con 0x7f5de813d2f0 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: dump: 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.376 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:35:34.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de0012400 con 0x7f5de813d2f0 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 5 monclient: authenticate success, global_id 94155 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _renew_subs 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de8121b60 con 0x7f5de813d2f0 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de00127f0 con 0x7f5de813d2f0 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 10 monclient: dump: 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:34.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddeffd640 4 set_mon_vals no callback set 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _renew_subs 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de80a7ce0 con 0x7f5de813d2f0 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de80a18d0 con 0x7f5de813d2f0 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.365+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5de0012a10 con 0x7f5de813d2f0 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.369+0000 7f5ddd7fa640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 0x7f5dd403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.369+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f5de0050610 con 0x7f5de813d2f0 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.369+0000 7f5decb1a640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 0x7f5dd403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:34.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.369+0000 7f5decb1a640 10 monclient: get_auth_request con 0x7f5dd403bb40 auth_method 0 2024-09-17T15:35:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.369+0000 7f5decb1a640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 0x7f5dd403e000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5de4004e30 tx=0x7f5de4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.373+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5de0012d00 con 0x7f5de813d2f0 2024-09-17T15:35:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.373+0000 7f5ddd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:34.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.373+0000 7f5ddd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:34.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5deed7e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5deed7e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5deed7e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5de80a58d0 con 0x7f5de813d2f0 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5ddd7fa640 1 -- 172.21.15.86:0/907492004 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f5de001a360 con 0x7f5de813d2f0 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5ddd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:34.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.609+0000 7f5ddd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:34.613 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:34.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 msgr2=0x7f5dd403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 0x7f5dd403e000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5de4004e30 tx=0x7f5de4009040 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 10 monclient: shutdown 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 msgr2=0x7f5de80a1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5de000cac0 tx=0x7f5de000cf80 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 shutdown_connections 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5dd403bb40 0x7f5dd403e000 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5de813d2f0 0x7f5de80a1f00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5de80a54c0 0x7f5de81381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 --2- 172.21.15.86:0/907492004 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de80a50e0 0x7f5de81cfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 >> 172.21.15.86:0/907492004 conn(0x7f5de80aaae0 msgr2=0x7f5de80a0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 shutdown_connections 2024-09-17T15:35:34.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:34.613+0000 7f5deed7e640 1 -- 172.21.15.86:0/907492004 wait complete. 2024-09-17T15:35:37.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: get_monmap_and_config 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: build_initial_monmap 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: monmap: 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:37.765386+0000 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:37.765386+0000 2024-09-17T15:35:37.763 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:37.764 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 Processor -- start 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 -- start start 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: init 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 0x7fbc1409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-b con 0x7fbc140a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbc140a5450 0x7fbc140a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-a con 0x7fbc140a5450 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc14137e30 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc140a6f40 con 0x7fbc14137e30 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbc140a65e0 con 0x7fbc140a5450 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1409c2a0 con 0x7fbc140a5070 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc1ae87640 10 monclient(hunting): authenticate will time out at 1470.730844s 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc18c23640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 msgr2=0x7fbc1409cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc18c23640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 0x7fbc1409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbc140a5450 0x7fbc140a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbc140a5450 0x7fbc140a5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53320/0 (socket says 172.21.15.86:53320) 2024-09-17T15:35:37.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 -- 172.21.15.86:0/2115965809 learned_addr learned my addr 172.21.15.86:0/2115965809 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): get_auth_request con 0x7fbc140a5450 auth_method 0 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient(hunting): handle_auth_done global_id 94158 payload 274 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 -- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 msgr2=0x7fbc1409cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 --2- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 0x7fbc1409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 -- 172.21.15.86:0/2115965809 >> v1:172.21.15.203:6789/0 conn(0x7fbc14137e30 legacy=0x7fbc14138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient: _finish_hunting 0 2024-09-17T15:35:37.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 monclient: found mon.noname-a 2024-09-17T15:35:37.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:37.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:37.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.761+0000 7fbc13fff640 1 -- 172.21.15.86:0/2115965809 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1409c580 con 0x7fbc140a5450 2024-09-17T15:35:37.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc13fff640 1 -- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc140a5450 msgr2=0x7fbc140a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc13fff640 1 --2- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc140a5450 0x7fbc140a5830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009870 tx=0x7fbc0401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f390481cbc9f7b5b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 1 -- 172.21.15.86:0/2115965809 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040b0070 con 0x7fbc140a5450 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: dump: 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:37.776 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: _finish_auth 0 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: _check_auth_tickets 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc1ae87640 5 monclient: authenticate success, global_id 94158 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 1 -- 172.21.15.86:0/2115965809 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc040ab600 con 0x7fbc140a5450 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc1ae87640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:37.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc1ae87640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc1ae87640 4 set_mon_vals no callback set 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 1 -- 172.21.15.86:0/2115965809 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc040ab9d0 con 0x7fbc140a5450 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.765+0000 7fbc137fe640 10 monclient: dump: 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:37.778 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: shutdown 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc140a5450 msgr2=0x7fbc140a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc140a5450 0x7fbc140a5830 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009870 tx=0x7fbc0401c440 comp rx=0 tx=0).stop 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- 172.21.15.86:0/2115965809 shutdown_connections 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc140a5450 0x7fbc140a5830 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/2115965809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5070 0x7fbc1409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- 172.21.15.86:0/2115965809 >> 172.21.15.86:0/2115965809 conn(0x7fbc140742a0 msgr2=0x7fbc140746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- 172.21.15.86:0/2115965809 shutdown_connections 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- 172.21.15.86:0/2115965809 wait complete. 2024-09-17T15:35:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: build_initial_monmap 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: monmap: 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 Processor -- start 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- start start 2024-09-17T15:35:37.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: init 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc140a5070 0x7fbc141d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-b con 0x7fbc140a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 0x7fbc14136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-c con 0x7fbc140a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): picked mon.noname-a con 0x7fbc14137e30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbc14142ac0 con 0x7fbc14137e30 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:37.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc140a6a70 con 0x7fbc140a5070 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbc14142c40 con 0x7fbc140a5450 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc1ae87640 10 monclient(hunting): authenticate will time out at 1470.737901s 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc13fff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 msgr2=0x7fbc14136fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc13fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 0x7fbc14136fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53334/0 (socket says 172.21.15.86:53334) 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 -- 172.21.15.86:0/4125818610 learned_addr learned my addr 172.21.15.86:0/4125818610 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): get_auth_request con 0x7fbc14137e30 auth_method 0 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc18c23640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc140a5070 0x7fbc141d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc18c23640 10 monclient(hunting): get_auth_request con 0x7fbc140a5070 auth_method 0 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc18c23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc18c23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc18c23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient(hunting): handle_auth_done global_id 94161 payload 824 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 -- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 msgr2=0x7fbc14136fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 0x7fbc14136fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 -- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc140a5070 msgr2=0x7fbc141d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc140a5070 0x7fbc141d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient: _finish_hunting 0 2024-09-17T15:35:37.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 monclient: found mon.noname-a 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.769+0000 7fbc19424640 1 -- 172.21.15.86:0/4125818610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc140a1860 con 0x7fbc14137e30 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc19424640 1 -- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 msgr2=0x7fbc1413d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc19424640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c7b0 tx=0x7fbc0c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c00ac60 con 0x7fbc14137e30 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: dump: 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: _finish_auth 0 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: _check_auth_tickets 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc0c00ae00 con 0x7fbc14137e30 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:37.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c004bb0 con 0x7fbc14137e30 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 10 monclient: dump: 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:37.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc127fc640 4 set_mon_vals no callback set 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 5 monclient: authenticate success, global_id 94161 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _renew_subs 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1413ebc0 con 0x7fbc14137e30 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _renew_subs 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1413f720 con 0x7fbc14137e30 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1409cfe0 con 0x7fbc14137e30 2024-09-17T15:35:37.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbc0c005030 con 0x7fbc14137e30 2024-09-17T15:35:37.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 0x7fbc0003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:37.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7fbc0c050570 con 0x7fbc14137e30 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc18c23640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 0x7fbc0003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.773+0000 7fbc18c23640 10 monclient: get_auth_request con 0x7fbc0003bb10 auth_method 0 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.777+0000 7fbc18c23640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 0x7fbc0003dfd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006fd0 tx=0x7fbc08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.777+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbc0c005600 con 0x7fbc14137e30 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.777+0000 7fbc117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:37.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:37.777+0000 7fbc117fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:38.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc1ae87640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc1ae87640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc1ae87640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc14069920 con 0x7fbc14137e30 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc117fa640 1 -- 172.21.15.86:0/4125818610 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fbc0c01acc0 con 0x7fbc14137e30 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.013+0000 7fbc117fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:38.018 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 msgr2=0x7fbc0003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 0x7fbc0003dfd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbc08006fd0 tx=0x7fbc08008040 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 10 monclient: shutdown 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 msgr2=0x7fbc1413d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c7b0 tx=0x7fbc0c00cc70 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 shutdown_connections 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc0003bb10 0x7fbc0003dfd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc14137e30 0x7fbc1413d0d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc140a5450 0x7fbc14136fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 --2- 172.21.15.86:0/4125818610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc140a5070 0x7fbc141d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 >> 172.21.15.86:0/4125818610 conn(0x7fbc140742a0 msgr2=0x7fbc140a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 shutdown_connections 2024-09-17T15:35:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:38.017+0000 7fbc1ae87640 1 -- 172.21.15.86:0/4125818610 wait complete. 2024-09-17T15:35:41.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: get_monmap_and_config 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: build_initial_monmap 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: monmap: 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:41.169203+0000 2024-09-17T15:35:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:41.169203+0000 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:41.168 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 Processor -- start 2024-09-17T15:35:41.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 -- start start 2024-09-17T15:35:41.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: init 2024-09-17T15:35:41.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97100689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f97100694c0 0x7f971006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): picked mon.noname-a con 0x7f97100694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f971013aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971009c8b0 con 0x7f97100689d0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971009cb90 con 0x7f971013aff0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9710074500 con 0x7f97100694c0 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:41.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9718759640 10 monclient(hunting): authenticate will time out at 1474.134683s 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f97100694c0 0x7f971006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f97164f5640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f97100689d0 0x7f9710068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34228/0 (socket says 172.21.15.86:34228) 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f97164f5640 1 -- 172.21.15.86:0/3872016797 learned_addr learned my addr 172.21.15.86:0/3872016797 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): get_auth_request con 0x7f97100694c0 auth_method 0 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:41.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient(hunting): handle_auth_done global_id 94164 payload 274 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 1 -- 172.21.15.86:0/3872016797 >> v1:172.21.15.203:6789/0 conn(0x7f971013aff0 legacy=0x7f971013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 1 -- 172.21.15.86:0/3872016797 >> v1:172.21.15.86:6789/0 conn(0x7f97100689d0 legacy=0x7f9710068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient: _finish_hunting 0 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 1 monclient: found mon.noname-a 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.165+0000 7f9715cf4640 1 -- 172.21.15.86:0/3872016797 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971009ce70 con 0x7f97100694c0 2024-09-17T15:35:41.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f9715cf4640 1 -- 172.21.15.86:0/3872016797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97100694c0 msgr2=0x7f971006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f9715cf4640 1 --2- 172.21.15.86:0/3872016797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97100694c0 0x7f971006f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9710069940 tx=0x7f970401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c181df71674b2f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 1 -- 172.21.15.86:0/3872016797 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97040b0070 con 0x7f97100694c0 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: dump: 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.178 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: _finish_auth 0 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: _check_auth_tickets 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 1 -- 172.21.15.86:0/3872016797 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97040ab600 con 0x7f97100694c0 2024-09-17T15:35:41.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 1 -- 172.21.15.86:0/3872016797 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971009c2d0 con 0x7f97100694c0 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f97154f3640 10 monclient: dump: 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:41.180 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f9718759640 5 monclient: authenticate success, global_id 94164 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f9718759640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.169+0000 7f9718759640 4 set_mon_vals no callback set 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: shutdown 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- 172.21.15.86:0/3872016797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97100694c0 msgr2=0x7f971006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 --2- 172.21.15.86:0/3872016797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97100694c0 0x7f971006f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9710069940 tx=0x7f970401c340 comp rx=0 tx=0).stop 2024-09-17T15:35:41.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- 172.21.15.86:0/3872016797 shutdown_connections 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 --2- 172.21.15.86:0/3872016797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97100694c0 0x7f971006f840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- 172.21.15.86:0/3872016797 >> 172.21.15.86:0/3872016797 conn(0x7f97100a6440 msgr2=0x7f97100a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- 172.21.15.86:0/3872016797 shutdown_connections 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- 172.21.15.86:0/3872016797 wait complete. 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: build_initial_monmap 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: monmap: 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 Processor -- start 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- start start 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: init 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 0x7f971011db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): picked mon.noname-b con 0x7f97100689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 0x7f97101359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:41.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): picked mon.noname-c con 0x7f97100694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): picked mon.noname-a con 0x7f971013aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9710141c30 con 0x7f971013aff0 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9710141ab0 con 0x7f97100689d0 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9710141db0 con 0x7f97100694c0 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:41.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9718759640 10 monclient(hunting): authenticate will time out at 1474.141862s 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9715cf4640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 msgr2=0x7f97101359d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9715cf4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 0x7f97101359d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33568/0 (socket says 172.21.15.86:33568) 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 -- 172.21.15.86:0/1853903438 learned_addr learned my addr 172.21.15.86:0/1853903438 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): get_auth_request con 0x7f971013aff0 auth_method 0 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:41.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 0x7f971011db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 10 monclient(hunting): get_auth_request con 0x7f97100689d0 auth_method 0 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient(hunting): handle_auth_done global_id 94167 payload 824 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 -- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 msgr2=0x7f97101359d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 0x7f97101359d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 -- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 msgr2=0x7f971011db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:41.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 0x7f971011db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient: _finish_hunting 0 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 monclient: found mon.noname-a 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f9716cf6640 1 -- 172.21.15.86:0/1853903438 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971011f350 con 0x7f971013aff0 2024-09-17T15:35:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.173+0000 7f97164f5640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 0x7f971011db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:35:41.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9716cf6640 1 -- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 msgr2=0x7f971011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:41.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9716cf6640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f970800c780 tx=0x7f970800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970800ac00 con 0x7f971013aff0 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: dump: 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: _finish_auth 0 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 5 monclient: authenticate success, global_id 94167 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f970800ada0 con 0x7f971013aff0 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _renew_subs 2024-09-17T15:35:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f971013b580 con 0x7f971013aff0 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708004bb0 con 0x7f971013aff0 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 10 monclient: dump: 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:41.192 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _renew_subs 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9710073510 con 0x7f971013aff0 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9714cf2640 4 set_mon_vals no callback set 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:41.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971006f840 con 0x7f971013aff0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9708005030 con 0x7f971013aff0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 0x7f96f403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f9708050180 con 0x7f971013aff0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f97164f5640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 0x7f96f403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.177+0000 7f97164f5640 10 monclient: get_auth_request con 0x7f96f403bb90 auth_method 0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.181+0000 7f97164f5640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 0x7f96f403e050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f970c007fd0 tx=0x7f970c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.181+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9708020070 con 0x7f971013aff0 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.181+0000 7f9702ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:41.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.181+0000 7f9702ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:41.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:41.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:41.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:41.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f971006d4f0 con 0x7f971013aff0 2024-09-17T15:35:41.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9702ffd640 1 -- 172.21.15.86:0/1853903438 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9708004e10 con 0x7f971013aff0 2024-09-17T15:35:41.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9702ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:41.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9702ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:41.419 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 msgr2=0x7f96f403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 0x7f96f403e050 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f970c007fd0 tx=0x7f970c009040 comp rx=0 tx=0).stop 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 10 monclient: shutdown 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 msgr2=0x7f971011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:41.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f970800c780 tx=0x7f970800cc40 comp rx=0 tx=0).stop 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 shutdown_connections 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f96f403bb90 0x7f96f403e050 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971013aff0 0x7f971011c8c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97100694c0 0x7f97101359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 --2- 172.21.15.86:0/1853903438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97100689d0 0x7f971011db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 >> 172.21.15.86:0/1853903438 conn(0x7f97100a6440 msgr2=0x7f97100a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 shutdown_connections 2024-09-17T15:35:41.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:41.417+0000 7f9718759640 1 -- 172.21.15.86:0/1853903438 wait complete. 2024-09-17T15:35:44.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: get_monmap_and_config 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: build_initial_monmap 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: monmap: 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:44.569411+0000 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:44.569411+0000 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:44.567 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:44.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 Processor -- start 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 -- start start 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: init 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:35:44.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-c con 0x7f4f080689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f08068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-a con 0x7f4f0813aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0809c8b0 con 0x7f4f08068db0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4f08074500 con 0x7f4f0813aff0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4f0809cb90 con 0x7f4f080689d0 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f10726640 10 monclient(hunting): authenticate will time out at 1477.534882s 2024-09-17T15:35:44.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33572/0 (socket says 172.21.15.86:33572) 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 learned_addr learned my addr 172.21.15.86:0/1176657245 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request con 0x7f4f0813aff0 auth_method 0 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request con 0x7f4f080689d0 auth_method 0 2024-09-17T15:35:44.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.565+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.569+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> v1:172.21.15.203:6789/0 conn(0x7f4f08068db0 legacy=0x7f4f08069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request con 0x7f4f0813aff0 auth_method 0 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:44.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request con 0x7f4f080689d0 auth_method 0 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:44.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:44.769+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:45.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request con 0x7f4f0813aff0 auth_method 0 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request con 0x7f4f080689d0 auth_method 0 2024-09-17T15:35:45.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.173+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:45.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request con 0x7f4f0813aff0 auth_method 0 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request con 0x7f4f080689d0 auth_method 0 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:45.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:45.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:45.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:35:45.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:35:45.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:35:45.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:45.977+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:35:47.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): tick 2024-09-17T15:35:47.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:35:47.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 monclient(hunting): continuing hunt 2024-09-17T15:35:47.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:35:47.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 msgr2=0x7f4f0806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.203:3300/0 conn(0x7f4f080689d0 0x7f4f0806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 -- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 msgr2=0x7f4f0813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 --2- 172.21.15.86:0/1176657245 >> v2:172.21.15.86:3300/0 conn(0x7f4f0813aff0 0x7f4f0813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 -- 172.21.15.86:0/1176657245 >> v1:172.21.15.203:6789/0 conn(0x7f4f08068db0 legacy=0x7f4f08069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient: _add_conns ranks=[0,2] 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4eec003720 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 0x7f4eec009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f4eec0073f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 -- 172.21.15.86:0/1176657245 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f0809ce70 con 0x7f4eec003720 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 1 -- 172.21.15.86:0/1176657245 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f0809cb90 con 0x7f4eec0073f0 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4efbfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 msgr2=0x7f4eec009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 0x7f4eec009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.569+0000 7f4f0e4c2640 1 -- 172.21.15.86:0/1176657245 >> v1:172.21.15.86:6789/0 conn(0x7f4eec003720 legacy=0x7f4eec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:47.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.773+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 msgr2=0x7f4eec009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:47.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:47.773+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 0x7f4eec009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:35:48.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.177+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 msgr2=0x7f4eec009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.177+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 0x7f4eec009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:35:48.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891203437 0 0) 0x7f4f0809ce70 con 0x7f4eec003720 2024-09-17T15:35:48.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 10 monclient(hunting): _negotiate 2024-09-17T15:35:48.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 10 monclient(hunting): my global_id is 94170 2024-09-17T15:35:48.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.413+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ef4003530 con 0x7f4eec003720 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1553878973 0 0) 0x7f4ef4003530 con 0x7f4eec003720 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 msgr2=0x7f4eec009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eec0073f0 0x7f4eec009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:35:48.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eec00b0d0 con 0x7f4eec003720 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f4f0400f040 con 0x7f4eec003720 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: dump: 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 >> v1:172.21.15.86:6789/0 conn(0x7f4eec003720 legacy=0x7f4eec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 reap_dead start 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4003a30 0x7f4ef4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): picked mon.a con 0x7f4ef4003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:48.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 0x7f4ef4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): picked mon.b con 0x7f4ef4004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 0x7f4ef400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): picked mon.c con 0x7f4ef4005420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0809cb90 con 0x7f4ef4003a30 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef400c330 con 0x7f4ef4004810 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef400c530 con 0x7f4ef4005420 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0d4c0640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0ecc3640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 msgr2=0x7f4ef400ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 0x7f4ef400ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 0x7f4ef4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): get_auth_request con 0x7f4ef4004810 auth_method 0 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_done global_id 94170 payload 293 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 msgr2=0x7f4ef400ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 0x7f4ef400ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4003a30 msgr2=0x7f4ef4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4003a30 0x7f4ef4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 monclient: found mon.b 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef4003530 con 0x7f4ef4004810 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f0dcc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f10726640 5 monclient: authenticate success, global_id 94170 2024-09-17T15:35:48.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.417+0000 7f4f10726640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.425+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 msgr2=0x7f4ef4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.425+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 0x7f4ef4004c90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f080a0690 tx=0x7f4efc01c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c12cc2adbf74d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0ad070 con 0x7f4ef4004810 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: dump: 2024-09-17T15:35:48.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4efc007270 con 0x7f4ef4004810 2024-09-17T15:35:48.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 1 -- 172.21.15.86:0/1176657245 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc007970 con 0x7f4ef4004810 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 4 set_mon_vals no callback set 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f0d4c0640 10 monclient: dump: 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.433 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.433 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.433 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.433 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 10 monclient: shutdown 2024-09-17T15:35:48.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:48.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 -- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 msgr2=0x7f4ef4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 0x7f4ef4004c90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4f080a0690 tx=0x7f4efc01c790 comp rx=0 tx=0).stop 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 -- 172.21.15.86:0/1176657245 shutdown_connections 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4005420 0x7f4ef400ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4004810 0x7f4ef4004c90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 --2- 172.21.15.86:0/1176657245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4003a30 0x7f4ef4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 -- 172.21.15.86:0/1176657245 >> 172.21.15.86:0/1176657245 conn(0x7f4f080a6440 msgr2=0x7f4f080a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 -- 172.21.15.86:0/1176657245 shutdown_connections 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 1 -- 172.21.15.86:0/1176657245 wait complete. 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 10 monclient: build_initial_monmap 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.429+0000 7f4f10726640 10 monclient: monmap: 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:48.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 Processor -- start 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 -- start start 2024-09-17T15:35:48.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient: init 2024-09-17T15:35:48.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:35:48.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 0x7f4f081ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef4003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef4004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 0x7f4f0811f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef4005420 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4f081362f0 con 0x7f4ef4004810 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f0809c2d0 con 0x7f4ef4005420 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f08136470 con 0x7f4ef4003a30 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0e4c2640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 msgr2=0x7f4f081ce490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f10726640 10 monclient(hunting): authenticate will time out at 1481.402048s 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0e4c2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 0x7f4f081ce490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46228/0 (socket says 172.21.15.86:46228) 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/2598744063 learned_addr learned my addr 172.21.15.86:0/2598744063 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): get_auth_request con 0x7f4ef4004810 auth_method 0 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient(hunting): handle_auth_done global_id 94173 payload 824 2024-09-17T15:35:48.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 0x7f4f0811f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 msgr2=0x7f4f081ce490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 0x7f4f081ce490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 msgr2=0x7f4f0811f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 0x7f4f0811f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 monclient: found mon.noname-a 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/2598744063 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f08133660 con 0x7f4ef4004810 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0ecc3640 10 monclient: get_auth_request con 0x7f4ef4005420 auth_method 0 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.433+0000 7f4f0ecc3640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 0x7f4f0811f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f0dcc1640 1 -- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 msgr2=0x7f4f081ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f0dcc1640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0ab040 tx=0x7f4ef4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0b7070 con 0x7f4ef4004810 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: dump: 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4efc0070c0 con 0x7f4ef4004810 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4efc0ad040 con 0x7f4ef4004810 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 10 monclient: dump: 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 5 monclient: authenticate success, global_id 94173 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _renew_subs 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f081227f0 con 0x7f4ef4004810 2024-09-17T15:35:48.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f0ccbf640 4 set_mon_vals no callback set 2024-09-17T15:35:48.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _renew_subs 2024-09-17T15:35:48.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f08121890 con 0x7f4ef4004810 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f08136850 con 0x7f4ef4004810 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4efc0b3470 con 0x7f4ef4004810 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 0x7f4eec03edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7f4efc0e99d0 con 0x7f4ef4004810 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 0x7f4eec03edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.437+0000 7f4f0e4c2640 10 monclient: get_auth_request con 0x7f4eec03c930 auth_method 0 2024-09-17T15:35:48.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.441+0000 7f4f0e4c2640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 0x7f4eec03edf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f04004710 tx=0x7f4f04006c40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.445+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4efc0c1020 con 0x7f4ef4004810 2024-09-17T15:35:48.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.445+0000 7f4efaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:48.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.445+0000 7f4efaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:48.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f08133880 con 0x7f4ef4004810 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4efaffd640 1 -- 172.21.15.86:0/2598744063 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f4f081227f0 con 0x7f4ef4004810 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4efaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:35:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4efaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:48.683 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":86,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:35:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 msgr2=0x7f4eec03edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 0x7f4eec03edf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4f04004710 tx=0x7f4f04006c40 comp rx=0 tx=0).stop 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 10 monclient: shutdown 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 msgr2=0x7f4f081ce9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4efc0ab040 tx=0x7f4ef4005840 comp rx=0 tx=0).stop 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 shutdown_connections 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4eec03c930 0x7f4eec03edf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ef4005420 0x7f4f0811f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ef4004810 0x7f4f081ce9d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 --2- 172.21.15.86:0/2598744063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ef4003a30 0x7f4f081ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 >> 172.21.15.86:0/2598744063 conn(0x7f4f080a6440 msgr2=0x7f4f080a3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 shutdown_connections 2024-09-17T15:35:48.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.681+0000 7f4f10726640 1 -- 172.21.15.86:0/2598744063 wait complete. 2024-09-17T15:35:48.702 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:35:48.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:35:48.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:35:48.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: get_monmap_and_config 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: build_initial_monmap 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: monmap: 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:48.828149+0000 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:48.828149+0000 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:48.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:48.827 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:48.827 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:48.827 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 Processor -- start 2024-09-17T15:35:48.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 -- start start 2024-09-17T15:35:48.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: init 2024-09-17T15:35:48.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc300a4970 addr v1:172.21.15.203:6789/0 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc300a34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3009ae20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdc3013d530 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3009b980 con 0x7fdc3013d530 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3009bf40 con 0x7fdc300a4970 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3006d1d0 con 0x7fdc3009ae20 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc378a1640 10 monclient(hunting): authenticate will time out at 1481.793600s 2024-09-17T15:35:48.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc34e3c640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 msgr2=0x7fdc300a34a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc34e3c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc300a34a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc35e3e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdc3013d530 0x7fdc3013f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51470/0 (socket says 172.21.15.86:51470) 2024-09-17T15:35:48.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc35e3e640 1 -- 172.21.15.86:0/2685178127 learned_addr learned my addr 172.21.15.86:0/2685178127 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.825+0000 7fdc35e3e640 1 -- 172.21.15.86:0/2685178127 >> v1:172.21.15.86:6789/0 conn(0x7fdc3013d530 legacy=0x7fdc3013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2238339053 0 0) 0x7fdc3009b980 con 0x7fdc3013d530 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): _negotiate 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): my global_id is 94176 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc1c003530 con 0x7fdc3013d530 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 -- 172.21.15.86:0/2685178127 >> v1:172.21.15.203:6789/0 conn(0x7fdc300a4970 legacy=0x7fdc300a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3677817128 0 0) 0x7fdc1c003530 con 0x7fdc3013d530 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 msgr2=0x7fdc300a34a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:48.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc300a34a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 >> v1:172.21.15.203:6789/0 conn(0x7fdc300a4970 legacy=0x7fdc300a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3006cb70 con 0x7fdc3013d530 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fdc28001d30 con 0x7fdc3013d530 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:35:48.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: dump: 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 >> v1:172.21.15.86:6789/0 conn(0x7fdc3013d530 legacy=0x7fdc3013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 0x7fdc1c003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): picked mon.a con 0x7fdc1c003ab0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c004810 0x7fdc1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): picked mon.b con 0x7fdc1c004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc1c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): picked mon.c con 0x7fdc3009ae20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3006d1d0 con 0x7fdc1c003ab0 2024-09-17T15:35:48.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c00bf10 con 0x7fdc1c004810 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdc1c00c090 con 0x7fdc3009ae20 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc27fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc35e3e640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 msgr2=0x7fdc1c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc35e3e640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc1c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 0x7fdc1c003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): get_auth_request con 0x7fdc1c003ab0 auth_method 0 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:35:48.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc34e3c640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c004810 0x7fdc1c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient(hunting): handle_auth_done global_id 94176 payload 293 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 msgr2=0x7fdc1c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc3009ae20 0x7fdc1c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c004810 msgr2=0x7fdc1c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c004810 0x7fdc1c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 monclient: found mon.a 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 1 -- 172.21.15.86:0/2685178127 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c003530 con 0x7fdc1c003ab0 2024-09-17T15:35:48.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc3563d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc378a1640 5 monclient: authenticate success, global_id 94176 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc378a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc34e3c640 10 monclient: get_auth_request con 0x7fdc1c004810 auth_method 0 2024-09-17T15:35:48.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.829+0000 7fdc34e3c640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c004810 0x7fdc1c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 msgr2=0x7fdc1c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 0x7fdc1c003ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc200019f0 tx=0x7fdc2001c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c399602e06aad89 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 0x7fdc1c003ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc200019f0 tx=0x7fdc2001c580 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fdc200016c0 mon_map magic: 0 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200016c0 con 0x7fdc1c003ab0 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:48.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 10 monclient: dump: 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc20007350 con 0x7fdc1c003ab0 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:48.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 4 set_mon_vals no callback set 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: shutdown 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 msgr2=0x7fdc1c003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 --2- 172.21.15.86:0/2685178127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc1c003ab0 0x7fdc1c003ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdc200019f0 tx=0x7fdc2001c580 comp rx=0 tx=0).stop 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc35e3e640 1 -- 172.21.15.86:0/2685178127 reap_dead start 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 1 -- 172.21.15.86:0/2685178127 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fdc200057d0 con 0x7fdc1c003ab0 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc27fff640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- 172.21.15.86:0/2685178127 shutdown_connections 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- 172.21.15.86:0/2685178127 >> 172.21.15.86:0/2685178127 conn(0x7fdc300a3e80 msgr2=0x7fdc300a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- 172.21.15.86:0/2685178127 shutdown_connections 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- 172.21.15.86:0/2685178127 wait complete. 2024-09-17T15:35:48.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: build_initial_monmap 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: monmap: 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 Processor -- start 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- start start 2024-09-17T15:35:48.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: init 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c003ab0 0x7fdc30074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-b con 0x7fdc1c003ab0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 0x7fdc30091850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-a con 0x7fdc30091410 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc30091d90 0x7fdc301d6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): picked mon.noname-c con 0x7fdc30091d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc30141a60 con 0x7fdc30091410 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3006cd40 con 0x7fdc1c003ab0 2024-09-17T15:35:48.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdc30141be0 con 0x7fdc30091d90 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc378a1640 10 monclient(hunting): authenticate will time out at 1481.803565s 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc35e3e640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc30091d90 msgr2=0x7fdc301d6d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc35e3e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc30091d90 0x7fdc301d6d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 0x7fdc30091850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 0x7fdc30091850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46238/0 (socket says 172.21.15.86:46238) 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 1 -- 172.21.15.86:0/1103062705 learned_addr learned my addr 172.21.15.86:0/1103062705 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): get_auth_request con 0x7fdc30091410 auth_method 0 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc34e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c003ab0 0x7fdc30074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 10 monclient(hunting): get_auth_request con 0x7fdc1c003ab0 auth_method 0 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.833+0000 7fdc3563d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 10 monclient(hunting): handle_auth_done global_id 94179 payload 824 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc30091d90 msgr2=0x7fdc301d6d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc30091d90 0x7fdc301d6d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c003ab0 msgr2=0x7fdc30074350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc1c003ab0 0x7fdc30074350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 10 monclient: _finish_hunting 0 2024-09-17T15:35:48.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 monclient: found mon.noname-a 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc300944c0 con 0x7fdc30091410 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 msgr2=0x7fdc30091850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc34e3c640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 0x7fdc30091850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00c7f0 tx=0x7fdc2c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c003da0 con 0x7fdc30091410 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: dump: 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: _finish_auth 0 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc2c016070 con 0x7fdc30091410 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 5 monclient: authenticate success, global_id 94179 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 monclient: _renew_subs 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3009a310 con 0x7fdc30091410 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc2c005270 con 0x7fdc30091410 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 10 monclient: dump: 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.855 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc277fe640 4 set_mon_vals no callback set 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 monclient: _renew_subs 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc30140c60 con 0x7fdc30091410 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 0x7fdc300713f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3009e570 con 0x7fdc30071010 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 0x7fdc300713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient: get_auth_request con 0x7fdc30071010 auth_method 0 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdc2c005450 con 0x7fdc30091410 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 10 monclient(hunting): handle_auth_done global_id 94182 payload 824 2024-09-17T15:35:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc3563d640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc300a2e10 con 0x7fdc30071010 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc1403bb40 0x7fdc1403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.837+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7fdc2c04ff40 con 0x7fdc30091410 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc35e3e640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc1403bb40 0x7fdc1403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc35e3e640 10 monclient: get_auth_request con 0x7fdc1403bb40 auth_method 0 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc3563d640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 msgr2=0x7fdc300713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc3563d640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 0x7fdc300713f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c005730 tx=0x7fdc200193c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200bd070 con 0x7fdc30071010 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 10 monclient: dump: 2024-09-17T15:35:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdc20007540 con 0x7fdc30071010 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:48.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 msgr2=0x7fdc300713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 0x7fdc300713f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c005730 tx=0x7fdc200193c0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 msgr2=0x7fdc300713f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:48.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc25ffb640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071010 0x7fdc300713f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.841+0000 7fdc35e3e640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc1403bb40 0x7fdc1403e000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdc3006f390 tx=0x7fdc28001780 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 0x7fdc301db260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdc30092190 con 0x7fdc30071930 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 0x7fdc301db260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient: get_auth_request con 0x7fdc30071930 auth_method 0 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 10 monclient(hunting): handle_auth_done global_id 94185 payload 824 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 1 -- 172.21.15.86:0/1103062705 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdc3006e600 con 0x7fdc30071930 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 msgr2=0x7fdc301db260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc3563d640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 0x7fdc301db260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc20001120 tx=0x7fdc200a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:48.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200bd050 con 0x7fdc30071930 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 10 monclient: dump: 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:48.864 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fdc200be040 con 0x7fdc30071930 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 msgr2=0x7fdc301db260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 0x7fdc301db260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc20001120 tx=0x7fdc200a6000 comp rx=0 tx=0).stop 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 msgr2=0x7fdc301db260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc25ffb640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30071930 0x7fdc301db260 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc20001120 tx=0x7fdc200a6000 comp rx=0 tx=0).stop 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:48.865 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 86, 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 24203, 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:48.866 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:48.867 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:48.868 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:48.869 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:48.870 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.871 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:48.872 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:48.873 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.874 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:48.875 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:48.876 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:48.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:48.878 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc1403bb40 msgr2=0x7fdc1403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdc1403bb40 0x7fdc1403e000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdc3006f390 tx=0x7fdc28001780 comp rx=0 tx=0).stop 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 10 monclient: shutdown 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 msgr2=0x7fdc30091850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:48.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 --2- 172.21.15.86:0/1103062705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc30091410 0x7fdc30091850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c00c7f0 tx=0x7fdc2c00ccb0 comp rx=0 tx=0).stop 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc35e3e640 1 -- 172.21.15.86:0/1103062705 reap_dead start 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc35e3e640 1 -- 172.21.15.86:0/1103062705 reap_dead start 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 shutdown_connections 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 >> 172.21.15.86:0/1103062705 conn(0x7fdc300a3e80 msgr2=0x7fdc300a0d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.849+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 shutdown_connections 2024-09-17T15:35:48.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.853+0000 7fdc378a1640 1 -- 172.21.15.86:0/1103062705 wait complete. 2024-09-17T15:35:48.881 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:35:48.881 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:35:48.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: get_monmap_and_config 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: build_initial_monmap 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: monmap: 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:35:48.996298+0000 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:35:48.996298+0000 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:35:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:35:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:35:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:35:48.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 Processor -- start 2024-09-17T15:35:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 -- start start 2024-09-17T15:35:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: init 2024-09-17T15:35:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb1a00a4f70 0x7fb1a009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a00a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 0x7fb1a00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a00a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1a0137d30 addr v1:172.21.15.86:6789/0 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a009d430 con 0x7fb1a0137d30 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a00a68d0 con 0x7fb1a00a4f70 2024-09-17T15:35:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a00a64a0 con 0x7fb1a00a5350 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb1a4f6d640 10 monclient(hunting): authenticate will time out at 1481.961856s 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19dd9b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 msgr2=0x7fb1a00a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 0x7fb1a00a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19ed9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb1a0137d30 0x7fb1a01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51478/0 (socket says 172.21.15.86:51478) 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb1a00a4f70 0x7fb1a009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19ed9d640 1 -- 172.21.15.86:0/4244347749 learned_addr learned my addr 172.21.15.86:0/4244347749 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): get_auth_request con 0x7fb1a00a4f70 auth_method 0 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient(hunting): handle_auth_done global_id 94188 payload 274 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 -- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 msgr2=0x7fb1a00a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 --2- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 0x7fb1a00a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 -- 172.21.15.86:0/4244347749 >> v1:172.21.15.86:6789/0 conn(0x7fb1a0137d30 legacy=0x7fb1a01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 monclient: found mon.noname-a 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.993+0000 7fb19e59c640 1 -- 172.21.15.86:0/4244347749 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a009c230 con 0x7fb1a00a4f70 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19e59c640 1 -- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 msgr2=0x7fb1a009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19e59c640 1 --2- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a009cef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1a006e190 tx=0x7fb19401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b09c3546fde362 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 1 -- 172.21.15.86:0/4244347749 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940b0070 con 0x7fb1a00a4f70 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: dump: 2024-09-17T15:35:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: _finish_auth 0 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 1 -- 172.21.15.86:0/4244347749 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1940ab600 con 0x7fb1a00a4f70 2024-09-17T15:35:49.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 1 -- 172.21.15.86:0/4244347749 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940ab9d0 con 0x7fb1a00a4f70 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 5 monclient: authenticate success, global_id 94188 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 10 monclient: get_monmap_and_config success 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 4 set_mon_vals no callback set 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb19d59a640 10 monclient: dump: 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 10 monclient: shutdown 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 msgr2=0x7fb1a009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a009cef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb1a006e190 tx=0x7fb19401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/4244347749 shutdown_connections 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a00a5350 0x7fb1a00a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/4244347749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a009cef0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/4244347749 >> 172.21.15.86:0/4244347749 conn(0x7fb1a00a7240 msgr2=0x7fb1a00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/4244347749 shutdown_connections 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:48.997+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/4244347749 wait complete. 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient: build_initial_monmap 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient: monmap: 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:35:49.006 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:35:49.007 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:35:49.007 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 Processor -- start 2024-09-17T15:35:49.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 -- start start 2024-09-17T15:35:49.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient: init 2024-09-17T15:35:49.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient: _reopen_session rank -1 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a01d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a00a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a00a5350 0x7fb1a0136fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a00a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a0137d30 0x7fb1a013d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a0137d30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0142b40 con 0x7fb1a00a4f70 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a01429c0 con 0x7fb1a00a5350 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:49.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a0142cc0 con 0x7fb1a0137d30 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): _renew_subs 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb1a4f6d640 10 monclient(hunting): authenticate will time out at 1481.969651s 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19ed9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a0137d30 msgr2=0x7fb1a013d230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a0137d30 0x7fb1a013d230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a01d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a01d01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46272/0 (socket says 172.21.15.86:46272) 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 -- 172.21.15.86:0/3271941835 learned_addr learned my addr 172.21.15.86:0/3271941835 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): get_auth_request con 0x7fb1a00a4f70 auth_method 0 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a00a5350 0x7fb1a0136fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient(hunting): handle_auth_done global_id 94191 payload 824 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a0137d30 msgr2=0x7fb1a013d230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1a0137d30 0x7fb1a013d230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a00a5350 msgr2=0x7fb1a0136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a00a5350 0x7fb1a0136fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 monclient: found mon.noname-a 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19e59c640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a013f750 con 0x7fb1a00a4f70 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19dd9b640 10 monclient: get_auth_request con 0x7fb1a00a5350 auth_method 0 2024-09-17T15:35:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.001+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a00a5350 0x7fb1a0136fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19e59c640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 msgr2=0x7fb1a01d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19e59c640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a01d01a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ae040 tx=0x7fb19401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940b0050 con 0x7fb1a00a4f70 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: dump: 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.015 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: _finish_auth 0 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: _check_auth_tickets 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 5 monclient: authenticate success, global_id 94191 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1940bc070 con 0x7fb1a00a4f70 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 monclient: _renew_subs 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1940b5c60 con 0x7fb1a00a4f70 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:49.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19cd99640 4 set_mon_vals no callback set 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a00a1760 con 0x7fb1a00a4f70 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 10 monclient: dump: 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.017 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 monclient: _renew_subs 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a01210d0 con 0x7fb1a00a4f70 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 0x7fb1a013db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a013e090 con 0x7fb1a013d770 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb1940c1460 con 0x7fb1a00a4f70 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb18003bb10 0x7fb18003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(266..266 src has 213..266) ==== 4815+0+0 (secure 0 0 0) 0x7fb1940eaa70 con 0x7fb1a00a4f70 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 0x7fb1a013db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient: get_auth_request con 0x7fb1a013d770 auth_method 0 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:49.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19ed9d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb18003bb10 0x7fb18003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19ed9d640 10 monclient: get_auth_request con 0x7fb18003bb10 auth_method 0 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_done global_id 54418 payload 824 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.005+0000 7fb19dd9b640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1a009cef0 con 0x7fb1a013d770 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb19dd9b640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 msgr2=0x7fb1a013db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 0x7fb1a013db50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c6f0 tx=0x7fb19000cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb19ed9d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb18003bb10 0x7fb18003dfd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb1a013f4e0 tx=0x7fb198004fe0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190003da0 con 0x7fb1a013d770 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 10 monclient: dump: 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb190003da0 con 0x7fb1a013d770 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:35:49.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:35:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 msgr2=0x7fb1a013db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 0x7fb1a013db50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c6f0 tx=0x7fb19000cbb0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 msgr2=0x7fb1a013db50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.009+0000 7fb18effd640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a013d770 0x7fb1a013db50 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c6f0 tx=0x7fb19000cbb0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb1a4f6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 0x7fb1a0139540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb1a4f6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a00a5ff0 con 0x7fb1a0139160 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 0x7fb1a0139540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient: get_auth_request con 0x7fb1a0139160 auth_method 0 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 10 monclient(hunting): handle_auth_done global_id 54421 payload 824 2024-09-17T15:35:49.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 1 -- 172.21.15.86:0/3271941835 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1a013ebf0 con 0x7fb1a0139160 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 msgr2=0x7fb1a0139540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb19dd9b640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 0x7fb1a0139540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c6f0 tx=0x7fb1900059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19001c110 con 0x7fb1a0139160 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb18effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb18effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.017+0000 7fb18effd640 10 monclient: dump: 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:35:49.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7fb19001c110 con 0x7fb1a0139160 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 msgr2=0x7fb1a0139540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 0x7fb1a0139540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c6f0 tx=0x7fb1900059b0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 msgr2=0x7fb1a0139540 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:35:49.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb18effd640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1a0139160 0x7fb1a0139540 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 86, 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 224453, 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.026 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:49.027 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:49.028 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:35:49.029 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:35:49.030 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:35:49.031 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:35:49.033 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:49.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:35:49.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:35:49.036 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.037 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:35:49.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:35:49.039 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:35:49.039 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb18003bb10 msgr2=0x7fb18003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb18003bb10 0x7fb18003dfd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb1a013f4e0 tx=0x7fb198004fe0 comp rx=0 tx=0).stop 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 10 monclient: shutdown 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 msgr2=0x7fb1a01d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 --2- 172.21.15.86:0/3271941835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb1a00a4f70 0x7fb1a01d01a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb1940ae040 tx=0x7fb19401ac30 comp rx=0 tx=0).stop 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb19ed9d640 1 -- 172.21.15.86:0/3271941835 reap_dead start 2024-09-17T15:35:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb19ed9d640 1 -- 172.21.15.86:0/3271941835 reap_dead start 2024-09-17T15:35:49.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 shutdown_connections 2024-09-17T15:35:49.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 >> 172.21.15.86:0/3271941835 conn(0x7fb1a00a7240 msgr2=0x7fb1a00a8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:35:49.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 shutdown_connections 2024-09-17T15:35:49.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:35:49.021+0000 7fb1a4f6d640 1 -- 172.21.15.86:0/3271941835 wait complete. 2024-09-17T15:35:49.041 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:35:49.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1154: done (6 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1155: done (5 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1156: done (4 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1158: done (3 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1159: done (2 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1160: done (1 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1161: done (0 left) 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1162: write (excl) oid 21 current snap is 124 2024-09-17T15:35:49.408 INFO:tasks.rados.rados.0.smithi203.stdout:1162: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-17T15:35:49.426 INFO:tasks.rados.rados.0.smithi203.stdout:1162: writing smithi20328086-21 from 726168 to 1431159 tid 1 2024-09-17T15:35:49.443 INFO:tasks.rados.rados.0.smithi203.stdout:1162: writing smithi20328086-21 from 1895071 to 2578545 tid 2 2024-09-17T15:35:49.443 INFO:tasks.rados.rados.0.smithi203.stdout:1163: read oid 2 snap 120 2024-09-17T15:35:49.444 INFO:tasks.rados.rados.0.smithi203.stdout:1163: expect (ObjNum 382 snap 108 seq_num 382) 2024-09-17T15:35:49.444 INFO:tasks.rados.rados.0.smithi203.stdout:1164: snap_remove snap 119 2024-09-17T15:35:49.666 INFO:tasks.rados.rados.0.smithi203.stdout:1164: done (2 left) 2024-09-17T15:35:49.666 INFO:tasks.rados.rados.0.smithi203.stdout:1165: write oid 43 current snap is 124 2024-09-17T15:35:49.666 INFO:tasks.rados.rados.0.smithi203.stdout:1165: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-17T15:35:49.689 INFO:tasks.rados.rados.0.smithi203.stdout:1165: writing smithi20328086-43 from 742975 to 1523215 tid 1 2024-09-17T15:35:49.700 INFO:tasks.rados.rados.0.smithi203.stdout:1165: writing smithi20328086-43 from 1953553 to 2380685 tid 2 2024-09-17T15:35:49.710 INFO:tasks.rados.rados.0.smithi203.stdout:1165: writing smithi20328086-43 from 2971713 to 3142975 tid 3 2024-09-17T15:35:49.711 INFO:tasks.rados.rados.0.smithi203.stdout:1166: write oid 46 current snap is 124 2024-09-17T15:35:49.711 INFO:tasks.rados.rados.0.smithi203.stdout:1162: finishing write tid 1 to smithi20328086-21 2024-09-17T15:35:49.711 INFO:tasks.rados.rados.0.smithi203.stdout:1166: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-17T15:35:49.728 INFO:tasks.rados.rados.0.smithi203.stdout:1166: writing smithi20328086-46 from 759782 to 1331624 tid 1 2024-09-17T15:35:49.744 INFO:tasks.rados.rados.0.smithi203.stdout:1166: writing smithi20328086-46 from 2012035 to 2699178 tid 2 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1167: delete oid 19 current snap is 124 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: finishing write tid 2 to smithi20328086-21 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: finishing write tid 3 to smithi20328086-21 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: finishing write tid 4 to smithi20328086-21 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: oid 21 updating version 0 to 492 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: oid 21 version 492 is already newer than 491 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: oid 21 updating version 492 to 493 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 493 (ObjNum 424 snap 124 seq_num 424) dirty exists 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1162: left oid 21 (ObjNum 424 snap 124 seq_num 424) 2024-09-17T15:35:49.745 INFO:tasks.rados.rados.0.smithi203.stdout:1163: expect (ObjNum 382 snap 108 seq_num 382) 2024-09-17T15:35:49.801 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:49.909 INFO:tasks.rados.rados.0.smithi203.stdout:1165: finishing write tid 1 to smithi20328086-43 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: finishing write tid 2 to smithi20328086-43 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: finishing write tid 3 to smithi20328086-43 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: finishing write tid 4 to smithi20328086-43 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: finishing write tid 5 to smithi20328086-43 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: oid 43 updating version 0 to 264 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: oid 43 updating version 264 to 266 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: oid 43 version 266 is already newer than 263 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: oid 43 version 266 is already newer than 265 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 266 (ObjNum 425 snap 124 seq_num 425) dirty exists 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1165: left oid 43 (ObjNum 425 snap 124 seq_num 425) 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1166: finishing write tid 1 to smithi20328086-46 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1166: finishing write tid 2 to smithi20328086-46 2024-09-17T15:35:49.910 INFO:tasks.rados.rados.0.smithi203.stdout:1166: finishing write tid 3 to smithi20328086-46 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:1166: finishing write tid 4 to smithi20328086-46 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:1166: oid 46 updating version 0 to 356 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:1166: oid 46 version 356 is already newer than 355 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:1166: oid 46 version 356 is already newer than 354 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 356 (ObjNum 426 snap 124 seq_num 426) dirty exists 2024-09-17T15:35:49.911 INFO:tasks.rados.rados.0.smithi203.stdout:1166: left oid 46 (ObjNum 426 snap 124 seq_num 426) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1162: done (4 left) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1163: done (3 left) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1165: done (2 left) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1166: done (1 left) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1167: done (0 left) 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:1168: rollback oid 19 current snap is 124 2024-09-17T15:35:49.917 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 120 2024-09-17T15:35:49.918 INFO:tasks.rados.rados.0.smithi203.stdout:1169: write oid 42 current snap is 124 2024-09-17T15:35:49.918 INFO:tasks.rados.rados.0.smithi203.stdout:1169: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-17T15:35:49.940 INFO:tasks.rados.rados.0.smithi203.stdout:1169: writing smithi20328086-42 from 793396 to 1515736 tid 1 2024-09-17T15:35:49.960 INFO:tasks.rados.rados.0.smithi203.stdout:1169: writing smithi20328086-42 from 2128999 to 2819811 tid 2 2024-09-17T15:35:49.960 INFO:tasks.rados.rados.0.smithi203.stdout:1170: write (excl) oid 2 current snap is 124 2024-09-17T15:35:49.961 INFO:tasks.rados.rados.0.smithi203.stdout:1170: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-17T15:35:49.975 INFO:tasks.rados.rados.0.smithi203.stdout:1170: writing smithi20328086-2 from 410203 to 1207792 tid 1 2024-09-17T15:35:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1170: writing smithi20328086-2 from 1787481 to 2221951 tid 2 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1170: writing smithi20328086-2 from 2681405 to 3210203 tid 3 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1171: snap_create 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1168: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1168: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 414 (ObjNum 406 snap 114 seq_num 406) dirty exists 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1169: finishing write tid 1 to smithi20328086-42 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1169: finishing write tid 2 to smithi20328086-42 2024-09-17T15:35:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1169: finishing write tid 3 to smithi20328086-42 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1169: finishing write tid 4 to smithi20328086-42 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1169: oid 42 updating version 0 to 358 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1169: oid 42 updating version 358 to 359 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1169: oid 42 version 359 is already newer than 357 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 359 (ObjNum 428 snap 124 seq_num 428) dirty exists 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1169: left oid 42 (ObjNum 428 snap 124 seq_num 428) 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1170: finishing write tid 1 to smithi20328086-2 2024-09-17T15:35:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1170: finishing write tid 2 to smithi20328086-2 2024-09-17T15:35:50.002 INFO:tasks.rados.rados.0.smithi203.stdout:1170: finishing write tid 3 to smithi20328086-2 2024-09-17T15:35:50.002 INFO:tasks.rados.rados.0.smithi203.stdout:1170: finishing write tid 4 to smithi20328086-2 2024-09-17T15:35:50.002 INFO:tasks.rados.rados.0.smithi203.stdout:1170: finishing write tid 5 to smithi20328086-2 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:1170: oid 2 updating version 0 to 282 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:1170: oid 2 version 282 is already newer than 280 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:1170: oid 2 version 282 is already newer than 279 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:1170: oid 2 version 282 is already newer than 281 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 282 (ObjNum 429 snap 124 seq_num 429) dirty exists 2024-09-17T15:35:50.003 INFO:tasks.rados.rados.0.smithi203.stdout:1170: left oid 2 (ObjNum 429 snap 124 seq_num 429) 2024-09-17T15:35:50.681 INFO:tasks.rados.rados.0.smithi203.stdout:1168: done (3 left) 2024-09-17T15:35:50.682 INFO:tasks.rados.rados.0.smithi203.stdout:1169: done (2 left) 2024-09-17T15:35:50.682 INFO:tasks.rados.rados.0.smithi203.stdout:1170: done (1 left) 2024-09-17T15:35:50.682 INFO:tasks.rados.rados.0.smithi203.stdout:1171: done (0 left) 2024-09-17T15:35:50.682 INFO:tasks.rados.rados.0.smithi203.stdout:1172: snap_remove snap 123 2024-09-17T15:35:51.693 INFO:tasks.rados.rados.0.smithi203.stdout:1172: done (0 left) 2024-09-17T15:35:51.693 INFO:tasks.rados.rados.0.smithi203.stdout:1173: rollback oid 12 current snap is 125 2024-09-17T15:35:51.693 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 120 2024-09-17T15:35:51.693 INFO:tasks.rados.rados.0.smithi203.stdout:1174: write oid 19 current snap is 125 2024-09-17T15:35:51.693 INFO:tasks.rados.rados.0.smithi203.stdout:1174: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-17T15:35:51.702 INFO:tasks.rados.rados.0.smithi203.stdout:1174: writing smithi20328086-19 from 427010 to 899848 tid 1 2024-09-17T15:35:51.712 INFO:tasks.rados.rados.0.smithi203.stdout:1174: writing smithi20328086-19 from 1329610 to 2024091 tid 2 2024-09-17T15:35:51.722 INFO:tasks.rados.rados.0.smithi203.stdout:1174: writing smithi20328086-19 from 2708828 to 3227010 tid 3 2024-09-17T15:35:51.722 INFO:tasks.rados.rados.0.smithi203.stdout:1175: write (excl) oid 28 current snap is 125 2024-09-17T15:35:51.722 INFO:tasks.rados.rados.0.smithi203.stdout:1173: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:35:51.722 INFO:tasks.rados.rados.0.smithi203.stdout:1175: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-17T15:35:51.730 INFO:tasks.rados.rados.0.smithi203.stdout:1175: writing smithi20328086-28 from 443817 to 991904 tid 1 2024-09-17T15:35:51.740 INFO:tasks.rados.rados.0.smithi203.stdout:1175: writing smithi20328086-28 from 1788092 to 2226231 tid 2 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1175: writing smithi20328086-28 from 3019898 to 3243817 tid 3 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1176: rollback oid 37 current snap is 125 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 120 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1173: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1173: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 265 (ObjNum 274 snap 70 seq_num 274) dirty exists 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1174: finishing write tid 1 to smithi20328086-19 2024-09-17T15:35:51.750 INFO:tasks.rados.rados.0.smithi203.stdout:1174: finishing write tid 2 to smithi20328086-19 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1173: done (3 left) 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1177: copy_from oid 48 from oid 1 current snap is 125 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: finishing write tid 3 to smithi20328086-19 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: finishing write tid 4 to smithi20328086-19 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: finishing write tid 5 to smithi20328086-19 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: oid 19 updating version 0 to 421 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: oid 19 version 421 is already newer than 420 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: oid 19 version 421 is already newer than 419 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: oid 19 version 421 is already newer than 418 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 421 (ObjNum 430 snap 125 seq_num 430) dirty exists 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: left oid 19 (ObjNum 430 snap 125 seq_num 430) 2024-09-17T15:35:51.751 INFO:tasks.rados.rados.0.smithi203.stdout:1174: done (3 left) 2024-09-17T15:35:51.752 INFO:tasks.rados.rados.0.smithi203.stdout:1178: snap_create 2024-09-17T15:35:51.752 INFO:tasks.rados.rados.0.smithi203.stdout:1175: finishing write tid 1 to smithi20328086-28 2024-09-17T15:35:51.752 INFO:tasks.rados.rados.0.smithi203.stdout:1176: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:35:51.752 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 0 (ObjNum 33 snap 0 seq_num 3605769008) dirty dne 2024-09-17T15:35:51.752 INFO:tasks.rados.rados.0.smithi203.stdout:1175: finishing write tid 2 to smithi20328086-28 2024-09-17T15:35:51.753 INFO:tasks.rados.rados.0.smithi203.stdout:1175: finishing write tid 3 to smithi20328086-28 2024-09-17T15:35:51.753 INFO:tasks.rados.rados.0.smithi203.stdout:1175: finishing write tid 4 to smithi20328086-28 2024-09-17T15:35:51.754 INFO:tasks.rados.rados.0.smithi203.stdout:1175: finishing write tid 5 to smithi20328086-28 2024-09-17T15:35:51.754 INFO:tasks.rados.rados.0.smithi203.stdout:1175: oid 28 updating version 0 to 501 2024-09-17T15:35:51.754 INFO:tasks.rados.rados.0.smithi203.stdout:1175: oid 28 updating version 501 to 502 2024-09-17T15:35:51.754 INFO:tasks.rados.rados.0.smithi203.stdout:1175: oid 28 version 502 is already newer than 498 2024-09-17T15:35:51.755 INFO:tasks.rados.rados.0.smithi203.stdout:1175: oid 28 version 502 is already newer than 497 2024-09-17T15:35:51.755 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 502 (ObjNum 431 snap 125 seq_num 431) dirty exists 2024-09-17T15:35:51.755 INFO:tasks.rados.rados.0.smithi203.stdout:1175: left oid 28 (ObjNum 431 snap 125 seq_num 431) 2024-09-17T15:35:51.768 INFO:tasks.rados.rados.0.smithi203.stdout:1177: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:35:51.770 INFO:tasks.rados.rados.0.smithi203.stdout:1177: finishing copy_from to smithi20328086-48 2024-09-17T15:35:51.770 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 375 (ObjNum 391 snap 110 seq_num 391) dirty exists 2024-09-17T15:35:53.089 INFO:tasks.rados.rados.0.smithi203.stdout:1175: done (3 left) 2024-09-17T15:35:53.089 INFO:tasks.rados.rados.0.smithi203.stdout:1176: done (2 left) 2024-09-17T15:35:53.090 INFO:tasks.rados.rados.0.smithi203.stdout:1177: done (1 left) 2024-09-17T15:35:53.090 INFO:tasks.rados.rados.0.smithi203.stdout:1178: done (0 left) 2024-09-17T15:35:53.090 INFO:tasks.rados.rados.0.smithi203.stdout:1179: write oid 32 current snap is 126 2024-09-17T15:35:53.090 INFO:tasks.rados.rados.0.smithi203.stdout:1179: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-17T15:35:53.099 INFO:tasks.rados.rados.0.smithi203.stdout:1179: writing smithi20328086-32 from 460624 to 1083960 tid 1 2024-09-17T15:35:53.112 INFO:tasks.rados.rados.0.smithi203.stdout:1179: writing smithi20328086-32 from 1846574 to 2544724 tid 2 2024-09-17T15:35:53.118 INFO:tasks.rados.rados.0.smithi203.stdout:1179: writing smithi20328086-32 from 3163674 to 3260624 tid 3 2024-09-17T15:35:53.118 INFO:tasks.rados.rados.0.smithi203.stdout:1180: read oid 17 snap -1 2024-09-17T15:35:53.118 INFO:tasks.rados.rados.0.smithi203.stdout:1180: expect (ObjNum 420 snap 123 seq_num 420) 2024-09-17T15:35:53.119 INFO:tasks.rados.rados.0.smithi203.stdout:1181: write (excl) oid 33 current snap is 126 2024-09-17T15:35:53.119 INFO:tasks.rados.rados.0.smithi203.stdout:1181: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-17T15:35:53.128 INFO:tasks.rados.rados.0.smithi203.stdout:1181: writing smithi20328086-33 from 477431 to 1176016 tid 1 2024-09-17T15:35:53.139 INFO:tasks.rados.rados.0.smithi203.stdout:1181: writing smithi20328086-33 from 1905056 to 2346864 tid 2 2024-09-17T15:35:53.147 INFO:tasks.rados.rados.0.smithi203.stdout:1181: writing smithi20328086-33 from 3074744 to 3277431 tid 3 2024-09-17T15:35:53.147 INFO:tasks.rados.rados.0.smithi203.stdout:1182: write (excl) oid 31 current snap is 126 2024-09-17T15:35:53.147 INFO:tasks.rados.rados.0.smithi203.stdout:1182: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-17T15:35:53.155 INFO:tasks.rados.rados.0.smithi203.stdout:1182: writing smithi20328086-31 from 494238 to 984425 tid 1 2024-09-17T15:35:53.164 INFO:tasks.rados.rados.0.smithi203.stdout:1182: writing smithi20328086-31 from 1563538 to 2265357 tid 2 2024-09-17T15:35:53.170 INFO:tasks.rados.rados.0.smithi203.stdout:1182: writing smithi20328086-31 from 2818520 to 3294238 tid 3 2024-09-17T15:35:53.170 INFO:tasks.rados.rados.0.smithi203.stdout:1183: read oid 11 snap -1 2024-09-17T15:35:53.170 INFO:tasks.rados.rados.0.smithi203.stdout:1183: expect (ObjNum 403 snap 113 seq_num 403) 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1184: snap_create 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: finishing write tid 1 to smithi20328086-32 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: finishing write tid 2 to smithi20328086-32 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: finishing write tid 3 to smithi20328086-32 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: finishing write tid 4 to smithi20328086-32 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: finishing write tid 5 to smithi20328086-32 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: oid 32 updating version 0 to 303 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: oid 32 version 303 is already newer than 301 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: oid 32 version 303 is already newer than 302 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: oid 32 version 303 is already newer than 300 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 303 (ObjNum 432 snap 126 seq_num 432) dirty exists 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1179: left oid 32 (ObjNum 432 snap 126 seq_num 432) 2024-09-17T15:35:53.171 INFO:tasks.rados.rados.0.smithi203.stdout:1180: expect (ObjNum 420 snap 123 seq_num 420) 2024-09-17T15:35:53.172 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:53.413 INFO:tasks.rados.rados.0.smithi203.stdout:1181: finishing write tid 1 to smithi20328086-33 2024-09-17T15:35:53.413 INFO:tasks.rados.rados.0.smithi203.stdout:1181: finishing write tid 2 to smithi20328086-33 2024-09-17T15:35:53.413 INFO:tasks.rados.rados.0.smithi203.stdout:1181: finishing write tid 3 to smithi20328086-33 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: finishing write tid 4 to smithi20328086-33 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: finishing write tid 5 to smithi20328086-33 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: oid 33 updating version 0 to 291 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: oid 33 updating version 291 to 293 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: oid 33 updating version 293 to 294 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: oid 33 version 294 is already newer than 292 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 294 (ObjNum 433 snap 126 seq_num 433) dirty exists 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1181: left oid 33 (ObjNum 433 snap 126 seq_num 433) 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1182: finishing write tid 1 to smithi20328086-31 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1182: finishing write tid 2 to smithi20328086-31 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1182: finishing write tid 3 to smithi20328086-31 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1182: finishing write tid 4 to smithi20328086-31 2024-09-17T15:35:53.414 INFO:tasks.rados.rados.0.smithi203.stdout:1182: finishing write tid 5 to smithi20328086-31 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1182: oid 31 updating version 0 to 139 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1182: oid 31 version 139 is already newer than 136 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1182: oid 31 version 139 is already newer than 138 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1182: oid 31 version 139 is already newer than 137 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 139 (ObjNum 434 snap 126 seq_num 434) dirty exists 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1182: left oid 31 (ObjNum 434 snap 126 seq_num 434) 2024-09-17T15:35:53.415 INFO:tasks.rados.rados.0.smithi203.stdout:1183: expect (ObjNum 403 snap 113 seq_num 403) 2024-09-17T15:35:53.474 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:35:53.546 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1208301 2024-09-17T15:35:53.839 INFO:tasks.rados.rados.0.smithi203.stdout:1179: done (5 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1180: done (4 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1181: done (3 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1182: done (2 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1183: done (1 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1184: done (0 left) 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1185: copy_from oid 2 from oid 34 current snap is 127 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1186: copy_from oid 38 from oid 40 current snap is 127 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1187: read oid 23 snap -1 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1187: expect deleted 2024-09-17T15:35:53.840 INFO:tasks.rados.rados.0.smithi203.stdout:1188: snap_remove snap 124 2024-09-17T15:35:54.667 INFO:tasks.rados.rados.0.smithi203.stdout:1188: done (3 left) 2024-09-17T15:35:54.667 INFO:tasks.rados.rados.0.smithi203.stdout:1189: write oid 43 current snap is 127 2024-09-17T15:35:54.667 INFO:tasks.rados.rados.0.smithi203.stdout:1189: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-17T15:35:54.676 INFO:tasks.rados.rados.0.smithi203.stdout:1189: writing smithi20328086-43 from 511045 to 1076481 tid 1 2024-09-17T15:35:54.684 INFO:tasks.rados.rados.0.smithi203.stdout:1189: writing smithi20328086-43 from 1622020 to 2067497 tid 2 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:1189: writing smithi20328086-43 from 2729590 to 3286445 tid 3 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:1190: snap_create 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:1186: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:1185: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:1186: finishing copy_from to smithi20328086-38 2024-09-17T15:35:54.694 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 479 (ObjNum 415 snap 119 seq_num 415) dirty exists 2024-09-17T15:35:54.695 INFO:tasks.rados.rados.0.smithi203.stdout:1185: finishing copy_from to smithi20328086-2 2024-09-17T15:35:54.695 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 429 (ObjNum 418 snap 123 seq_num 418) dirty exists 2024-09-17T15:35:54.695 INFO:tasks.rados.rados.0.smithi203.stdout:1189: finishing write tid 1 to smithi20328086-43 2024-09-17T15:35:54.695 INFO:tasks.rados.rados.0.smithi203.stdout:1189: finishing write tid 2 to smithi20328086-43 2024-09-17T15:35:54.701 INFO:tasks.rados.rados.0.smithi203.stdout:1189: finishing write tid 3 to smithi20328086-43 2024-09-17T15:35:54.701 INFO:tasks.rados.rados.0.smithi203.stdout:1189: finishing write tid 4 to smithi20328086-43 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: finishing write tid 5 to smithi20328086-43 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: oid 43 updating version 0 to 267 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: oid 43 updating version 267 to 269 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: oid 43 updating version 269 to 270 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: oid 43 version 270 is already newer than 268 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 270 (ObjNum 435 snap 127 seq_num 435) dirty exists 2024-09-17T15:35:54.702 INFO:tasks.rados.rados.0.smithi203.stdout:1189: left oid 43 (ObjNum 435 snap 127 seq_num 435) 2024-09-17T15:35:55.670 INFO:tasks.rados.rados.0.smithi203.stdout:1185: done (4 left) 2024-09-17T15:35:55.670 INFO:tasks.rados.rados.0.smithi203.stdout:1186: done (3 left) 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1187: done (2 left) 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1189: done (1 left) 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1190: done (0 left) 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1191: read oid 26 snap 126 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1191: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1192: write oid 16 current snap is 128 2024-09-17T15:35:55.671 INFO:tasks.rados.rados.0.smithi203.stdout:1192: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-17T15:35:55.681 INFO:tasks.rados.rados.0.smithi203.stdout:1192: writing smithi20328086-16 from 527852 to 1168537 tid 1 2024-09-17T15:35:55.691 INFO:tasks.rados.rados.0.smithi203.stdout:1192: writing smithi20328086-16 from 1680502 to 2269637 tid 2 2024-09-17T15:35:55.701 INFO:tasks.rados.rados.0.smithi203.stdout:1192: writing smithi20328086-16 from 2757013 to 3327852 tid 3 2024-09-17T15:35:55.701 INFO:tasks.rados.rados.0.smithi203.stdout:1193: snap_create 2024-09-17T15:35:55.701 INFO:tasks.rados.rados.0.smithi203.stdout:1191: expect (ObjNum 370 snap 105 seq_num 370) 2024-09-17T15:35:55.850 INFO:tasks.rados.rados.0.smithi203.stdout:1192: finishing write tid 1 to smithi20328086-16 2024-09-17T15:35:55.850 INFO:tasks.rados.rados.0.smithi203.stdout:1192: finishing write tid 2 to smithi20328086-16 2024-09-17T15:35:55.850 INFO:tasks.rados.rados.0.smithi203.stdout:1192: finishing write tid 3 to smithi20328086-16 2024-09-17T15:35:55.850 INFO:tasks.rados.rados.0.smithi203.stdout:1192: finishing write tid 4 to smithi20328086-16 2024-09-17T15:35:55.850 INFO:tasks.rados.rados.0.smithi203.stdout:1192: finishing write tid 5 to smithi20328086-16 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:1192: oid 16 updating version 0 to 430 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:1192: oid 16 updating version 430 to 431 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:1192: oid 16 updating version 431 to 432 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:1192: oid 16 version 432 is already newer than 429 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 432 (ObjNum 436 snap 128 seq_num 436) dirty exists 2024-09-17T15:35:55.851 INFO:tasks.rados.rados.0.smithi203.stdout:1192: left oid 16 (ObjNum 436 snap 128 seq_num 436) 2024-09-17T15:35:56.682 INFO:tasks.rados.rados.0.smithi203.stdout:1191: done (2 left) 2024-09-17T15:35:56.682 INFO:tasks.rados.rados.0.smithi203.stdout:1192: done (1 left) 2024-09-17T15:35:56.682 INFO:tasks.rados.rados.0.smithi203.stdout:1193: done (0 left) 2024-09-17T15:35:56.683 INFO:tasks.rados.rados.0.smithi203.stdout:1194: write (excl) oid 38 current snap is 129 2024-09-17T15:35:56.683 INFO:tasks.rados.rados.0.smithi203.stdout:1194: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-17T15:35:56.694 INFO:tasks.rados.rados.0.smithi203.stdout:1194: writing smithi20328086-38 from 544659 to 1260593 tid 1 2024-09-17T15:35:56.703 INFO:tasks.rados.rados.0.smithi203.stdout:1194: writing smithi20328086-38 from 1738984 to 2188130 tid 2 2024-09-17T15:35:56.714 INFO:tasks.rados.rados.0.smithi203.stdout:1194: writing smithi20328086-38 from 2784436 to 3344659 tid 3 2024-09-17T15:35:56.714 INFO:tasks.rados.rados.0.smithi203.stdout:1195: delete oid 6 current snap is 129 2024-09-17T15:35:56.714 INFO:tasks.rados.rados.0.smithi203.stdout:1194: finishing write tid 1 to smithi20328086-38 2024-09-17T15:35:56.715 INFO:tasks.rados.rados.0.smithi203.stdout:1194: finishing write tid 2 to smithi20328086-38 2024-09-17T15:35:56.721 INFO:tasks.rados.rados.0.smithi203.stdout:1194: finishing write tid 3 to smithi20328086-38 2024-09-17T15:35:56.721 INFO:tasks.rados.rados.0.smithi203.stdout:1194: finishing write tid 4 to smithi20328086-38 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:1194: finishing write tid 5 to smithi20328086-38 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:1194: oid 38 updating version 0 to 483 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:1194: oid 38 version 483 is already newer than 480 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:1194: oid 38 version 483 is already newer than 482 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:1194: oid 38 version 483 is already newer than 481 2024-09-17T15:35:56.722 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 483 (ObjNum 437 snap 129 seq_num 437) dirty exists 2024-09-17T15:35:56.723 INFO:tasks.rados.rados.0.smithi203.stdout:1194: left oid 38 (ObjNum 437 snap 129 seq_num 437) 2024-09-17T15:35:56.727 INFO:tasks.rados.rados.0.smithi203.stdout:1194: done (1 left) 2024-09-17T15:35:56.727 INFO:tasks.rados.rados.0.smithi203.stdout:1195: done (0 left) 2024-09-17T15:35:56.727 INFO:tasks.rados.rados.0.smithi203.stdout:1196: snap_create 2024-09-17T15:35:57.689 INFO:tasks.rados.rados.0.smithi203.stdout:1196: done (0 left) 2024-09-17T15:35:57.689 INFO:tasks.rados.rados.0.smithi203.stdout:1197: snap_create 2024-09-17T15:35:58.695 INFO:tasks.rados.rados.0.smithi203.stdout:1197: done (0 left) 2024-09-17T15:35:58.696 INFO:tasks.rados.rados.0.smithi203.stdout:1198: read oid 10 snap -1 2024-09-17T15:35:58.696 INFO:tasks.rados.rados.0.smithi203.stdout:1198: expect (ObjNum 392 snap 110 seq_num 392) 2024-09-17T15:35:58.696 INFO:tasks.rados.rados.0.smithi203.stdout:1199: snap_remove snap 128 2024-09-17T15:35:59.671 INFO:tasks.rados.rados.0.smithi203.stdout:1199: done (1 left) 2024-09-17T15:35:59.671 INFO:tasks.rados.rados.0.smithi203.stdout:1200: write (excl) oid 31 current snap is 131 2024-09-17T15:35:59.671 INFO:tasks.rados.rados.0.smithi203.stdout:1200: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-17T15:35:59.687 INFO:tasks.rados.rados.0.smithi203.stdout:1200: writing smithi20328086-31 from 578273 to 1044705 tid 1 2024-09-17T15:35:59.704 INFO:tasks.rados.rados.0.smithi203.stdout:1200: writing smithi20328086-31 from 1739595 to 2192410 tid 2 2024-09-17T15:35:59.721 INFO:tasks.rados.rados.0.smithi203.stdout:1200: writing smithi20328086-31 from 2722929 to 3378273 tid 3 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1201: read oid 20 snap -1 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1201: expect deleted 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1202: read oid 15 snap -1 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1202: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1203: read oid 29 snap -1 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1203: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:35:59.722 INFO:tasks.rados.rados.0.smithi203.stdout:1204: snap_remove snap 126 2024-09-17T15:36:00.708 INFO:tasks.rados.rados.0.smithi203.stdout:1204: done (5 left) 2024-09-17T15:36:00.708 INFO:tasks.rados.rados.0.smithi203.stdout:1205: snap_create 2024-09-17T15:36:00.708 INFO:tasks.rados.rados.0.smithi203.stdout:1198: expect (ObjNum 392 snap 110 seq_num 392) 2024-09-17T15:36:00.761 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:00.810 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 698720 2024-09-17T15:36:00.830 INFO:tasks.rados.rados.0.smithi203.stdout:1200: finishing write tid 1 to smithi20328086-31 2024-09-17T15:36:00.830 INFO:tasks.rados.rados.0.smithi203.stdout:1200: finishing write tid 2 to smithi20328086-31 2024-09-17T15:36:00.830 INFO:tasks.rados.rados.0.smithi203.stdout:1200: finishing write tid 3 to smithi20328086-31 2024-09-17T15:36:00.830 INFO:tasks.rados.rados.0.smithi203.stdout:1200: finishing write tid 4 to smithi20328086-31 2024-09-17T15:36:00.830 INFO:tasks.rados.rados.0.smithi203.stdout:1200: finishing write tid 5 to smithi20328086-31 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1200: oid 31 updating version 0 to 141 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1200: oid 31 updating version 141 to 144 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1200: oid 31 version 144 is already newer than 142 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1200: oid 31 version 144 is already newer than 143 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 144 (ObjNum 439 snap 131 seq_num 439) dirty exists 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1200: left oid 31 (ObjNum 439 snap 131 seq_num 439) 2024-09-17T15:36:00.831 INFO:tasks.rados.rados.0.smithi203.stdout:1202: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:00.902 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1729250 2024-09-17T15:36:00.922 INFO:tasks.rados.rados.0.smithi203.stdout:1203: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:00.922 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 251854 2024-09-17T15:36:00.956 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:01.000 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2826564 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1198: done (5 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1200: done (4 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1201: done (3 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1202: done (2 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1203: done (1 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1205: done (0 left) 2024-09-17T15:36:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1206: snap_create 2024-09-17T15:36:02.725 INFO:tasks.rados.rados.0.smithi203.stdout:1206: done (0 left) 2024-09-17T15:36:02.725 INFO:tasks.rados.rados.0.smithi203.stdout:1207: rollback oid 49 current snap is 133 2024-09-17T15:36:02.726 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 129 2024-09-17T15:36:02.726 INFO:tasks.rados.rados.0.smithi203.stdout:1208: copy_from oid 38 from oid 17 current snap is 133 2024-09-17T15:36:02.726 INFO:tasks.rados.rados.0.smithi203.stdout:1209: snap_create 2024-09-17T15:36:02.739 INFO:tasks.rados.rados.0.smithi203.stdout:1207: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:36:02.739 INFO:tasks.rados.rados.0.smithi203.stdout:1207: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:36:02.741 INFO:tasks.rados.rados.0.smithi203.stdout:1207: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:36:02.741 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 508 (ObjNum 277 snap 72 seq_num 277) dirty exists 2024-09-17T15:36:02.754 INFO:tasks.rados.rados.0.smithi203.stdout:1208: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:36:02.756 INFO:tasks.rados.rados.0.smithi203.stdout:1208: finishing copy_from to smithi20328086-38 2024-09-17T15:36:02.756 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 496 (ObjNum 420 snap 123 seq_num 420) dirty exists 2024-09-17T15:36:03.732 INFO:tasks.rados.rados.0.smithi203.stdout:1207: done (2 left) 2024-09-17T15:36:03.733 INFO:tasks.rados.rados.0.smithi203.stdout:1208: done (1 left) 2024-09-17T15:36:03.733 INFO:tasks.rados.rados.0.smithi203.stdout:1209: done (0 left) 2024-09-17T15:36:03.733 INFO:tasks.rados.rados.0.smithi203.stdout:1210: read oid 6 snap -1 2024-09-17T15:36:03.733 INFO:tasks.rados.rados.0.smithi203.stdout:1210: expect deleted 2024-09-17T15:36:03.733 INFO:tasks.rados.rados.0.smithi203.stdout:1211: snap_create 2024-09-17T15:36:04.674 INFO:tasks.rados.rados.0.smithi203.stdout:1210: done (1 left) 2024-09-17T15:36:04.674 INFO:tasks.rados.rados.0.smithi203.stdout:1211: done (0 left) 2024-09-17T15:36:04.674 INFO:tasks.rados.rados.0.smithi203.stdout:1212: read oid 44 snap -1 2024-09-17T15:36:04.674 INFO:tasks.rados.rados.0.smithi203.stdout:1212: expect (ObjNum 406 snap 114 seq_num 406) 2024-09-17T15:36:04.674 INFO:tasks.rados.rados.0.smithi203.stdout:1213: delete oid 37 current snap is 135 2024-09-17T15:36:04.680 INFO:tasks.rados.rados.0.smithi203.stdout:1213: done (1 left) 2024-09-17T15:36:04.681 INFO:tasks.rados.rados.0.smithi203.stdout:1214: delete oid 48 current snap is 135 2024-09-17T15:36:04.688 INFO:tasks.rados.rados.0.smithi203.stdout:1214: done (1 left) 2024-09-17T15:36:04.688 INFO:tasks.rados.rados.0.smithi203.stdout:1215: rollback oid 35 current snap is 135 2024-09-17T15:36:04.688 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 121 2024-09-17T15:36:04.688 INFO:tasks.rados.rados.0.smithi203.stdout:1216: snap_remove snap 131 2024-09-17T15:36:05.675 INFO:tasks.rados.rados.0.smithi203.stdout:1216: done (2 left) 2024-09-17T15:36:05.675 INFO:tasks.rados.rados.0.smithi203.stdout:1217: snap_remove snap 132 2024-09-17T15:36:06.687 INFO:tasks.rados.rados.0.smithi203.stdout:1217: done (2 left) 2024-09-17T15:36:06.687 INFO:tasks.rados.rados.0.smithi203.stdout:1218: read oid 18 snap -1 2024-09-17T15:36:06.687 INFO:tasks.rados.rados.0.smithi203.stdout:1218: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:36:06.687 INFO:tasks.rados.rados.0.smithi203.stdout:1219: read oid 2 snap -1 2024-09-17T15:36:06.688 INFO:tasks.rados.rados.0.smithi203.stdout:1219: expect (ObjNum 418 snap 123 seq_num 418) 2024-09-17T15:36:06.688 INFO:tasks.rados.rados.0.smithi203.stdout:1220: write oid 39 current snap is 135 2024-09-17T15:36:06.688 INFO:tasks.rados.rados.0.smithi203.stdout:1220: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-17T15:36:06.704 INFO:tasks.rados.rados.0.smithi203.stdout:1220: writing smithi20328086-39 from 628694 to 1037226 tid 1 2024-09-17T15:36:06.721 INFO:tasks.rados.rados.0.smithi203.stdout:1220: writing smithi20328086-39 from 1515041 to 2115183 tid 2 2024-09-17T15:36:06.739 INFO:tasks.rados.rados.0.smithi203.stdout:1220: writing smithi20328086-39 from 2805198 to 3357310 tid 3 2024-09-17T15:36:06.739 INFO:tasks.rados.rados.0.smithi203.stdout:1221: read oid 49 snap 127 2024-09-17T15:36:06.739 INFO:tasks.rados.rados.0.smithi203.stdout:1221: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:36:06.739 INFO:tasks.rados.rados.0.smithi203.stdout:1222: write oid 27 current snap is 135 2024-09-17T15:36:06.739 INFO:tasks.rados.rados.0.smithi203.stdout:1222: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-17T15:36:06.751 INFO:tasks.rados.rados.0.smithi203.stdout:1222: writing smithi20328086-27 from 645501 to 1129282 tid 1 2024-09-17T15:36:06.760 INFO:tasks.rados.rados.0.smithi203.stdout:1222: writing smithi20328086-27 from 1573523 to 2033676 tid 2 2024-09-17T15:36:06.770 INFO:tasks.rados.rados.0.smithi203.stdout:1222: writing smithi20328086-27 from 2548974 to 3312358 tid 3 2024-09-17T15:36:06.770 INFO:tasks.rados.rados.0.smithi203.stdout:1223: snap_create 2024-09-17T15:36:06.770 INFO:tasks.rados.rados.0.smithi203.stdout:1212: expect (ObjNum 406 snap 114 seq_num 406) 2024-09-17T15:36:06.933 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1531595 2024-09-17T15:36:06.970 INFO:tasks.rados.rados.0.smithi203.stdout:1215: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:36:06.971 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 0 (ObjNum 33 snap 0 seq_num 3605851904) dirty dne 2024-09-17T15:36:06.971 INFO:tasks.rados.rados.0.smithi203.stdout:1218: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:36:06.977 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1659423 2024-09-17T15:36:07.066 INFO:tasks.rados.rados.0.smithi203.stdout:1219: expect (ObjNum 418 snap 123 seq_num 418) 2024-09-17T15:36:07.066 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:07.102 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1613425 2024-09-17T15:36:07.155 INFO:tasks.rados.rados.0.smithi203.stdout:1220: finishing write tid 1 to smithi20328086-39 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: finishing write tid 2 to smithi20328086-39 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: finishing write tid 3 to smithi20328086-39 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: finishing write tid 4 to smithi20328086-39 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: finishing write tid 5 to smithi20328086-39 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: oid 39 updating version 0 to 445 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: oid 39 version 445 is already newer than 444 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: oid 39 version 445 is already newer than 443 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:1220: oid 39 updating version 445 to 446 2024-09-17T15:36:07.156 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 446 (ObjNum 442 snap 135 seq_num 442) dirty exists 2024-09-17T15:36:07.157 INFO:tasks.rados.rados.0.smithi203.stdout:1220: left oid 39 (ObjNum 442 snap 135 seq_num 442) 2024-09-17T15:36:07.157 INFO:tasks.rados.rados.0.smithi203.stdout:1221: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:36:07.216 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: finishing write tid 1 to smithi20328086-27 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: finishing write tid 2 to smithi20328086-27 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: finishing write tid 3 to smithi20328086-27 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: finishing write tid 4 to smithi20328086-27 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: finishing write tid 5 to smithi20328086-27 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: oid 27 updating version 0 to 363 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: oid 27 updating version 363 to 364 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: oid 27 updating version 364 to 366 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: oid 27 version 366 is already newer than 365 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 366 (ObjNum 443 snap 135 seq_num 443) dirty exists 2024-09-17T15:36:07.336 INFO:tasks.rados.rados.0.smithi203.stdout:1222: left oid 27 (ObjNum 443 snap 135 seq_num 443) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1212: done (7 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1215: done (6 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1218: done (5 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1219: done (4 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1220: done (3 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1221: done (2 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1222: done (1 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1223: done (0 left) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1224: read oid 1 snap -1 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1224: expect (ObjNum 391 snap 110 seq_num 391) 2024-09-17T15:36:07.686 INFO:tasks.rados.rados.0.smithi203.stdout:1225: write (excl) oid 12 current snap is 136 2024-09-17T15:36:07.687 INFO:tasks.rados.rados.0.smithi203.stdout:1225: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-17T15:36:07.697 INFO:tasks.rados.rados.0.smithi203.stdout:1225: writing smithi20328086-12 from 662308 to 1221338 tid 1 2024-09-17T15:36:07.705 INFO:tasks.rados.rados.0.smithi203.stdout:1225: writing smithi20328086-12 from 1632005 to 2235816 tid 2 2024-09-17T15:36:07.715 INFO:tasks.rados.rados.0.smithi203.stdout:1225: writing smithi20328086-12 from 2860044 to 3462308 tid 3 2024-09-17T15:36:07.715 INFO:tasks.rados.rados.0.smithi203.stdout:1226: delete oid 8 current snap is 136 2024-09-17T15:36:07.715 INFO:tasks.rados.rados.0.smithi203.stdout:1224: expect (ObjNum 391 snap 110 seq_num 391) 2024-09-17T15:36:07.715 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:07.813 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 129383 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: finishing write tid 1 to smithi20328086-12 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: finishing write tid 2 to smithi20328086-12 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: finishing write tid 3 to smithi20328086-12 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: finishing write tid 4 to smithi20328086-12 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: finishing write tid 5 to smithi20328086-12 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: oid 12 updating version 0 to 268 2024-09-17T15:36:07.861 INFO:tasks.rados.rados.0.smithi203.stdout:1225: oid 12 updating version 268 to 269 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1225: oid 12 version 269 is already newer than 266 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1225: oid 12 version 269 is already newer than 267 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 269 (ObjNum 444 snap 136 seq_num 444) dirty exists 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1225: left oid 12 (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1224: done (2 left) 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1225: done (1 left) 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1226: done (0 left) 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1227: write (excl) oid 24 current snap is 136 2024-09-17T15:36:07.862 INFO:tasks.rados.rados.0.smithi203.stdout:1227: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-17T15:36:07.876 INFO:tasks.rados.rados.0.smithi203.stdout:1227: writing smithi20328086-24 from 695922 to 1405450 tid 1 2024-09-17T15:36:07.888 INFO:tasks.rados.rados.0.smithi203.stdout:1227: writing smithi20328086-24 from 2032616 to 2640096 tid 2 2024-09-17T15:36:07.896 INFO:tasks.rados.rados.0.smithi203.stdout:1227: writing smithi20328086-24 from 3198537 to 3495922 tid 3 2024-09-17T15:36:07.896 INFO:tasks.rados.rados.0.smithi203.stdout:1228: snap_remove snap 125 2024-09-17T15:36:07.896 INFO:tasks.rados.rados.0.smithi203.stdout:1227: finishing write tid 1 to smithi20328086-24 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1228: done (1 left) 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1229: read oid 50 snap -1 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1229: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1230: write (excl) oid 2 current snap is 136 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1227: finishing write tid 2 to smithi20328086-24 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1227: finishing write tid 3 to smithi20328086-24 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1227: finishing write tid 4 to smithi20328086-24 2024-09-17T15:36:08.705 INFO:tasks.rados.rados.0.smithi203.stdout:1230: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-17T15:36:08.723 INFO:tasks.rados.rados.0.smithi203.stdout:1230: writing smithi20328086-2 from 712729 to 1497506 tid 1 2024-09-17T15:36:08.736 INFO:tasks.rados.rados.0.smithi203.stdout:1230: writing smithi20328086-2 from 2091098 to 2558589 tid 2 2024-09-17T15:36:08.747 INFO:tasks.rados.rados.0.smithi203.stdout:1230: writing smithi20328086-2 from 3342313 to 3512729 tid 3 2024-09-17T15:36:08.747 INFO:tasks.rados.rados.0.smithi203.stdout:1231: write oid 1 current snap is 136 2024-09-17T15:36:08.747 INFO:tasks.rados.rados.0.smithi203.stdout:1231: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-17T15:36:08.760 INFO:tasks.rados.rados.0.smithi203.stdout:1231: writing smithi20328086-1 from 729536 to 1189562 tid 1 2024-09-17T15:36:08.770 INFO:tasks.rados.rados.0.smithi203.stdout:1231: writing smithi20328086-1 from 1749580 to 2360729 tid 2 2024-09-17T15:36:08.778 INFO:tasks.rados.rados.0.smithi203.stdout:1231: writing smithi20328086-1 from 2853383 to 3305833 tid 3 2024-09-17T15:36:08.778 INFO:tasks.rados.rados.0.smithi203.stdout:1232: snap_create 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: finishing write tid 5 to smithi20328086-24 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: oid 24 updating version 0 to 413 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: oid 24 version 413 is already newer than 412 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: oid 24 version 413 is already newer than 411 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: oid 24 version 413 is already newer than 410 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 413 (ObjNum 446 snap 136 seq_num 446) dirty exists 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1227: left oid 24 (ObjNum 446 snap 136 seq_num 446) 2024-09-17T15:36:08.779 INFO:tasks.rados.rados.0.smithi203.stdout:1229: expect (ObjNum 216 snap 55 seq_num 216) 2024-09-17T15:36:08.883 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:08.945 INFO:tasks.rados.rados.0.smithi203.stdout:1230: finishing write tid 1 to smithi20328086-2 2024-09-17T15:36:08.945 INFO:tasks.rados.rados.0.smithi203.stdout:1230: finishing write tid 2 to smithi20328086-2 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: finishing write tid 3 to smithi20328086-2 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: finishing write tid 4 to smithi20328086-2 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: finishing write tid 5 to smithi20328086-2 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: oid 2 updating version 0 to 430 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: oid 2 updating version 430 to 432 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: oid 2 version 432 is already newer than 431 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: oid 2 updating version 432 to 433 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 433 (ObjNum 447 snap 136 seq_num 447) dirty exists 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1230: left oid 2 (ObjNum 447 snap 136 seq_num 447) 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1231: finishing write tid 1 to smithi20328086-1 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1231: finishing write tid 2 to smithi20328086-1 2024-09-17T15:36:08.946 INFO:tasks.rados.rados.0.smithi203.stdout:1231: finishing write tid 3 to smithi20328086-1 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: finishing write tid 4 to smithi20328086-1 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: finishing write tid 5 to smithi20328086-1 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: oid 1 updating version 0 to 449 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: oid 1 version 449 is already newer than 448 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: oid 1 version 449 is already newer than 446 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: oid 1 version 449 is already newer than 447 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 449 (ObjNum 448 snap 136 seq_num 448) dirty exists 2024-09-17T15:36:08.947 INFO:tasks.rados.rados.0.smithi203.stdout:1231: left oid 1 (ObjNum 448 snap 136 seq_num 448) 2024-09-17T15:36:09.042 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:36:09.042 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:36:09.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:36:09.045 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:36:09.045 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:36:09.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:09.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: get_monmap_and_config 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: build_initial_monmap 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: monmap: 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:09.165279+0000 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:09.165279+0000 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:09.163 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:09.164 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 Processor -- start 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 -- start start 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: init 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa918138460 0x7fa918138840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa918138460 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa91809fe80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa918138d80 0x7fa9180a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa918138d80 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa91806f770 con 0x7fa918138460 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa91806dda0 con 0x7fa918138d80 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa9180a4b10 con 0x7fa91809fe80 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91f2d3640 10 monclient(hunting): authenticate will time out at 1502.130537s 2024-09-17T15:36:09.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa918138460 0x7fa918138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa918138460 0x7fa918138840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43278/0 (socket says 172.21.15.86:43278) 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 -- 172.21.15.86:0/330308513 learned_addr learned my addr 172.21.15.86:0/330308513 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): get_auth_request con 0x7fa918138460 auth_method 0 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 10 monclient(hunting): get_auth_request con 0x7fa91809fe80 auth_method 0 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:09.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 1 -- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 msgr2=0x7fa9180a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 1 -- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 msgr2=0x7fa9180a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91c86e640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient(hunting): handle_auth_done global_id 94194 payload 274 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d870640 1 --2- 172.21.15.86:0/330308513 >> v2:172.21.15.203:3300/0 conn(0x7fa918138d80 0x7fa9180a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 -- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 msgr2=0x7fa9180a2e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:09.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 -- 172.21.15.86:0/330308513 >> v2:172.21.15.203:3300/0 conn(0x7fa918138d80 msgr2=0x7fa9180a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 --2- 172.21.15.86:0/330308513 >> v2:172.21.15.203:3300/0 conn(0x7fa918138d80 0x7fa9180a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient: _finish_hunting 0 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 monclient: found mon.noname-a 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.161+0000 7fa91d06f640 1 -- 172.21.15.86:0/330308513 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91806ee40 con 0x7fa918138460 2024-09-17T15:36:09.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91d06f640 1 -- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 msgr2=0x7fa918138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:09.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91d06f640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918138840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa91806dd10 tx=0x7fa91401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb24c1a5e7bab923 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 1 -- 172.21.15.86:0/330308513 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b2070 con 0x7fa918138460 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: dump: 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:09.180 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: _finish_auth 0 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 1 -- 172.21.15.86:0/330308513 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140ad640 con 0x7fa918138460 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 1 -- 172.21.15.86:0/330308513 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140ada10 con 0x7fa918138460 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 5 monclient: authenticate success, global_id 94194 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 4 set_mon_vals no callback set 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:09.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa90ffff640 10 monclient: dump: 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 10 monclient: shutdown 2024-09-17T15:36:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 -- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 msgr2=0x7fa918138840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918138840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa91806dd10 tx=0x7fa91401c3b0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 -- 172.21.15.86:0/330308513 shutdown_connections 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/330308513 >> v2:172.21.15.203:3300/0 conn(0x7fa918138d80 0x7fa9180a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa91809fe80 0x7fa9180a2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/330308513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918138840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.165+0000 7fa91f2d3640 1 -- 172.21.15.86:0/330308513 >> 172.21.15.86:0/330308513 conn(0x7fa9180a8470 msgr2=0x7fa9180a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- 172.21.15.86:0/330308513 shutdown_connections 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- 172.21.15.86:0/330308513 wait complete. 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient: build_initial_monmap 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient: monmap: 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:09.183 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 Processor -- start 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- start start 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient: init 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:36:09.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa91809fe80 0x7fa9181363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-b con 0x7fa91809fe80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-a con 0x7fa918138460 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): picked mon.noname-c con 0x7fa918138d80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9181419a0 con 0x7fa918138460 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa918141820 con 0x7fa91809fe80 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa918141b20 con 0x7fa918138d80 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:09.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91f2d3640 10 monclient(hunting): authenticate will time out at 1502.137412s 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43290/0 (socket says 172.21.15.86:43290) 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 -- 172.21.15.86:0/1216030090 learned_addr learned my addr 172.21.15.86:0/1216030090 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): get_auth_request con 0x7fa918138460 auth_method 0 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 10 monclient(hunting): get_auth_request con 0x7fa918138d80 auth_method 0 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 msgr2=0x7fa9181cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 msgr2=0x7fa9181cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91d870640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient(hunting): handle_auth_done global_id 94197 payload 824 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 msgr2=0x7fa9181cfd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa91809fe80 msgr2=0x7fa9181363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa91809fe80 0x7fa9181363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient: _finish_hunting 0 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 monclient: found mon.noname-a 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.169+0000 7fa91c86e640 1 -- 172.21.15.86:0/1216030090 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9180a7e10 con 0x7fa918138460 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91c86e640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 msgr2=0x7fa918136920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91c86e640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa90800e7e0 tx=0x7fa90800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa908017020 con 0x7fa918138460 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:09.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: dump: 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: _finish_auth 0 2024-09-17T15:36:09.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 5 monclient: authenticate success, global_id 94197 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _renew_subs 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90800c870 con 0x7fa918138460 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91813ad80 con 0x7fa918138460 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90800f040 con 0x7fa918138460 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 10 monclient: dump: 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90f7fe640 4 set_mon_vals no callback set 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _renew_subs 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91813c840 con 0x7fa918138460 2024-09-17T15:36:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa918138840 con 0x7fa918138460 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa90801a020 con 0x7fa918138460 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 0x7fa8fc03e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(287..287 src has 213..287) ==== 4799+0+0 (secure 0 0 0) 0x7fa90804fe70 con 0x7fa918138460 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91d06f640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 0x7fa8fc03e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.173+0000 7fa91d06f640 10 monclient: get_auth_request con 0x7fa8fc03bbc0 auth_method 0 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.177+0000 7fa91d06f640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 0x7fa8fc03e080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa9140ac000 tx=0x7fa914004740 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.177+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa908021020 con 0x7fa918138460 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.177+0000 7fa90dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:09.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.177+0000 7fa90dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:09.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa91f2d3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:09.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa91f2d3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa91f2d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa9180699d0 con 0x7fa918138460 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa90dffb640 1 -- 172.21.15.86:0/1216030090 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa908012390 con 0x7fa918138460 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa90dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.409+0000 7fa90dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:09.412 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":86,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:09.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 msgr2=0x7fa8fc03e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:09.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 0x7fa8fc03e080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa9140ac000 tx=0x7fa914004740 comp rx=0 tx=0).stop 2024-09-17T15:36:09.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 10 monclient: shutdown 2024-09-17T15:36:09.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:09.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 msgr2=0x7fa918136920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa90800e7e0 tx=0x7fa90800eca0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 shutdown_connections 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa8fc03bbc0 0x7fa8fc03e080 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa918138d80 0x7fa9181cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa918138460 0x7fa918136920 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 --2- 172.21.15.86:0/1216030090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa91809fe80 0x7fa9181363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 >> 172.21.15.86:0/1216030090 conn(0x7fa9180a8470 msgr2=0x7fa9180a1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 shutdown_connections 2024-09-17T15:36:09.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:09.413+0000 7fa91f2d3640 1 -- 172.21.15.86:0/1216030090 wait complete. 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1227: done (4 left) 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1229: done (3 left) 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1230: done (2 left) 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1231: done (1 left) 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1232: done (0 left) 2024-09-17T15:36:09.681 INFO:tasks.rados.rados.0.smithi203.stdout:1233: snap_remove snap 133 2024-09-17T15:36:10.693 INFO:tasks.rados.rados.0.smithi203.stdout:1233: done (0 left) 2024-09-17T15:36:10.693 INFO:tasks.rados.rados.0.smithi203.stdout:1234: write (excl) oid 2 current snap is 137 2024-09-17T15:36:10.693 INFO:tasks.rados.rados.0.smithi203.stdout:1234: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-17T15:36:10.707 INFO:tasks.rados.rados.0.smithi203.stdout:1234: writing smithi20328086-2 from 746343 to 1397971 tid 1 2024-09-17T15:36:10.721 INFO:tasks.rados.rados.0.smithi203.stdout:1234: writing smithi20328086-2 from 1924415 to 2679222 tid 2 2024-09-17T15:36:10.729 INFO:tasks.rados.rados.0.smithi203.stdout:1234: writing smithi20328086-2 from 3397159 to 3546343 tid 3 2024-09-17T15:36:10.729 INFO:tasks.rados.rados.0.smithi203.stdout:1235: write oid 21 current snap is 137 2024-09-17T15:36:10.729 INFO:tasks.rados.rados.0.smithi203.stdout:1234: finishing write tid 1 to smithi20328086-2 2024-09-17T15:36:10.729 INFO:tasks.rados.rados.0.smithi203.stdout:1235: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-17T15:36:10.744 INFO:tasks.rados.rados.0.smithi203.stdout:1235: writing smithi20328086-21 from 763150 to 1490027 tid 1 2024-09-17T15:36:10.756 INFO:tasks.rados.rados.0.smithi203.stdout:1235: writing smithi20328086-21 from 1982897 to 2597715 tid 2 2024-09-17T15:36:10.764 INFO:tasks.rados.rados.0.smithi203.stdout:1235: writing smithi20328086-21 from 3024582 to 3499576 tid 3 2024-09-17T15:36:10.764 INFO:tasks.rados.rados.0.smithi203.stdout:1236: write (excl) oid 34 current snap is 137 2024-09-17T15:36:10.764 INFO:tasks.rados.rados.0.smithi203.stdout:1234: finishing write tid 2 to smithi20328086-2 2024-09-17T15:36:10.764 INFO:tasks.rados.rados.0.smithi203.stdout:1236: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-17T15:36:10.774 INFO:tasks.rados.rados.0.smithi203.stdout:1236: writing smithi20328086-34 from 779957 to 1182083 tid 1 2024-09-17T15:36:10.786 INFO:tasks.rados.rados.0.smithi203.stdout:1236: writing smithi20328086-34 from 1925026 to 2683502 tid 2 2024-09-17T15:36:10.793 INFO:tasks.rados.rados.0.smithi203.stdout:1236: writing smithi20328086-34 from 3335652 to 3579957 tid 3 2024-09-17T15:36:10.793 INFO:tasks.rados.rados.0.smithi203.stdout:1237: write (excl) oid 42 current snap is 137 2024-09-17T15:36:10.793 INFO:tasks.rados.rados.0.smithi203.stdout:1237: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-17T15:36:10.802 INFO:tasks.rados.rados.0.smithi203.stdout:1237: writing smithi20328086-42 from 796764 to 1274139 tid 1 2024-09-17T15:36:10.811 INFO:tasks.rados.rados.0.smithi203.stdout:1237: writing smithi20328086-42 from 1983508 to 2601995 tid 2 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:1237: writing smithi20328086-42 from 3363075 to 3596764 tid 3 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:1238: rollback oid 4 current snap is 137 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 120 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:1234: finishing write tid 3 to smithi20328086-2 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:1234: finishing write tid 4 to smithi20328086-2 2024-09-17T15:36:10.818 INFO:tasks.rados.rados.0.smithi203.stdout:1234: finishing write tid 5 to smithi20328086-2 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: oid 2 updating version 0 to 434 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: oid 2 updating version 434 to 437 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: oid 2 version 437 is already newer than 436 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: oid 2 version 437 is already newer than 435 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 437 (ObjNum 449 snap 137 seq_num 449) dirty exists 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: left oid 2 (ObjNum 449 snap 137 seq_num 449) 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1235: finishing write tid 1 to smithi20328086-21 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1235: finishing write tid 2 to smithi20328086-21 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1234: done (4 left) 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1239: write oid 40 current snap is 137 2024-09-17T15:36:10.819 INFO:tasks.rados.rados.0.smithi203.stdout:1239: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-17T15:36:10.824 INFO:tasks.rados.rados.0.smithi203.stdout:1239: writing smithi20328086-40 from 413571 to 966195 tid 1 2024-09-17T15:36:10.833 INFO:tasks.rados.rados.0.smithi203.stdout:1239: writing smithi20328086-40 from 1641990 to 2404135 tid 2 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1239: writing smithi20328086-40 from 2990498 to 3415661 tid 3 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1240: read oid 10 snap 136 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1240: expect (ObjNum 392 snap 110 seq_num 392) 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1241: read oid 7 snap -1 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1241: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:10.840 INFO:tasks.rados.rados.0.smithi203.stdout:1242: snap_remove snap 134 2024-09-17T15:36:11.716 INFO:tasks.rados.rados.0.smithi203.stdout:1242: done (7 left) 2024-09-17T15:36:11.716 INFO:tasks.rados.rados.0.smithi203.stdout:1243: write oid 9 current snap is 137 2024-09-17T15:36:11.716 INFO:tasks.rados.rados.0.smithi203.stdout:1243: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-17T15:36:11.722 INFO:tasks.rados.rados.0.smithi203.stdout:1243: writing smithi20328086-9 from 430378 to 1058251 tid 1 2024-09-17T15:36:11.730 INFO:tasks.rados.rados.0.smithi203.stdout:1243: writing smithi20328086-9 from 1700472 to 2322628 tid 2 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1243: writing smithi20328086-9 from 3017921 to 3630378 tid 3 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1244: delete oid 2 current snap is 137 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: finishing write tid 3 to smithi20328086-21 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: finishing write tid 4 to smithi20328086-21 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: finishing write tid 5 to smithi20328086-21 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: oid 21 updating version 0 to 518 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: oid 21 version 518 is already newer than 517 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: oid 21 updating version 518 to 519 2024-09-17T15:36:11.739 INFO:tasks.rados.rados.0.smithi203.stdout:1235: oid 21 version 519 is already newer than 516 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 519 (ObjNum 450 snap 137 seq_num 450) dirty exists 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1235: left oid 21 (ObjNum 450 snap 137 seq_num 450) 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: finishing write tid 1 to smithi20328086-34 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: finishing write tid 2 to smithi20328086-34 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: finishing write tid 3 to smithi20328086-34 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: finishing write tid 4 to smithi20328086-34 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: finishing write tid 5 to smithi20328086-34 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: oid 34 updating version 0 to 431 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: oid 34 version 431 is already newer than 430 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: oid 34 updating version 431 to 432 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: oid 34 updating version 432 to 433 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 433 (ObjNum 451 snap 137 seq_num 451) dirty exists 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1236: left oid 34 (ObjNum 451 snap 137 seq_num 451) 2024-09-17T15:36:11.740 INFO:tasks.rados.rados.0.smithi203.stdout:1237: finishing write tid 1 to smithi20328086-42 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: finishing write tid 2 to smithi20328086-42 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: finishing write tid 3 to smithi20328086-42 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1238: finishing rollback tid 0 to smithi20328086-4 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: finishing write tid 4 to smithi20328086-42 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: finishing write tid 5 to smithi20328086-42 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: oid 42 updating version 0 to 374 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: oid 42 version 374 is already newer than 373 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: oid 42 updating version 374 to 375 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: oid 42 version 375 is already newer than 372 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 375 (ObjNum 452 snap 137 seq_num 452) dirty exists 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1237: left oid 42 (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1238: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:1238: finishing rollback tid 2 to smithi20328086-4 2024-09-17T15:36:11.741 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 269 (ObjNum 303 snap 84 seq_num 303) dirty exists 2024-09-17T15:36:11.742 INFO:tasks.rados.rados.0.smithi203.stdout:1239: finishing write tid 1 to smithi20328086-40 2024-09-17T15:36:11.742 INFO:tasks.rados.rados.0.smithi203.stdout:1240: expect (ObjNum 392 snap 110 seq_num 392) 2024-09-17T15:36:11.806 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1768005 2024-09-17T15:36:11.827 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1172254 2024-09-17T15:36:11.844 INFO:tasks.rados.rados.0.smithi203.stdout:1241: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:11.844 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:11.885 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1119843 2024-09-17T15:36:11.928 INFO:tasks.rados.rados.0.smithi203.stdout:1239: finishing write tid 2 to smithi20328086-40 2024-09-17T15:36:11.928 INFO:tasks.rados.rados.0.smithi203.stdout:1239: finishing write tid 3 to smithi20328086-40 2024-09-17T15:36:11.928 INFO:tasks.rados.rados.0.smithi203.stdout:1239: finishing write tid 4 to smithi20328086-40 2024-09-17T15:36:11.928 INFO:tasks.rados.rados.0.smithi203.stdout:1239: finishing write tid 5 to smithi20328086-40 2024-09-17T15:36:11.928 INFO:tasks.rados.rados.0.smithi203.stdout:1239: oid 40 updating version 0 to 511 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1239: oid 40 version 511 is already newer than 510 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1239: oid 40 version 511 is already newer than 508 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1239: oid 40 version 511 is already newer than 509 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 511 (ObjNum 453 snap 137 seq_num 453) dirty exists 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1239: left oid 40 (ObjNum 453 snap 137 seq_num 453) 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: finishing write tid 1 to smithi20328086-9 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: finishing write tid 2 to smithi20328086-9 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: finishing write tid 3 to smithi20328086-9 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: finishing write tid 4 to smithi20328086-9 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: finishing write tid 5 to smithi20328086-9 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: oid 9 updating version 0 to 423 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: oid 9 version 423 is already newer than 422 2024-09-17T15:36:11.929 INFO:tasks.rados.rados.0.smithi203.stdout:1243: oid 9 updating version 423 to 424 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1243: oid 9 version 424 is already newer than 421 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 424 (ObjNum 454 snap 137 seq_num 454) dirty exists 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1243: left oid 9 (ObjNum 454 snap 137 seq_num 454) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1235: done (8 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1236: done (7 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1237: done (6 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1238: done (5 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1239: done (4 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1240: done (3 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1241: done (2 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1243: done (1 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1244: done (0 left) 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:1245: rollback oid 45 current snap is 137 2024-09-17T15:36:11.930 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 136 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:1246: read oid 39 snap 136 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:1246: expect (ObjNum 442 snap 135 seq_num 442) 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:1247: rollback oid 46 current snap is 137 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 120 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:1248: rollback oid 3 current snap is 137 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 136 2024-09-17T15:36:11.931 INFO:tasks.rados.rados.0.smithi203.stdout:1249: snap_remove snap 135 2024-09-17T15:36:12.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:12.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: get_monmap_and_config 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: build_initial_monmap 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: monmap: 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:12.561399+0000 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:12.561399+0000 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:12.559 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:12.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:12.560 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:12.560 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:12.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 Processor -- start 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 -- start start 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: init 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07540a5070 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-b con 0x7f07540a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:12.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f075413d280 0x7f075413f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-c con 0x7f075413d280 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07540a6f40 con 0x7f07540a5070 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f07540a65e0 con 0x7f075413d280 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f075409c2a0 con 0x7f07540a5b60 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f075b63a640 10 monclient(hunting): authenticate will time out at 1505.527666s 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33346/0 (socket says 172.21.15.86:33346) 2024-09-17T15:36:12.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 -- 172.21.15.86:0/3516410953 learned_addr learned my addr 172.21.15.86:0/3516410953 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 10 monclient(hunting): get_auth_request con 0x7f07540a5b60 auth_method 0 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 -- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 msgr2=0x7f075409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 -- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 msgr2=0x7f075409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.557+0000 7f0758bd5640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0758bd5640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 --2- 172.21.15.86:0/3516410953 >> v2:172.21.15.203:3300/0 conn(0x7f075413d280 0x7f075413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): get_auth_request con 0x7f075413d280 auth_method 0 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:12.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient(hunting): handle_auth_done global_id 54424 payload 274 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 -- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 msgr2=0x7f075409cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 -- 172.21.15.86:0/3516410953 >> v1:172.21.15.203:6789/0 conn(0x7f07540a5070 legacy=0x7f07540a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient: _finish_hunting 0 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 monclient: found mon.noname-c 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:36:12.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.561+0000 7f0759bd7640 1 -- 172.21.15.86:0/3516410953 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075409c580 con 0x7f075413d280 2024-09-17T15:36:12.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f0759bd7640 1 -- 172.21.15.86:0/3516410953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 msgr2=0x7f075413f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:12.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f0759bd7640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413f670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f074c009d90 tx=0x7f074c01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b242a0979f72b655 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:12.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 1 -- 172.21.15.86:0/3516410953 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c0ac070 con 0x7f075413d280 2024-09-17T15:36:12.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: dump: 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: _finish_auth 0 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 1 -- 172.21.15.86:0/3516410953 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c0a6680 con 0x7f075413d280 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 5 monclient: authenticate success, global_id 54424 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 1 -- 172.21.15.86:0/3516410953 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c0a6a50 con 0x7f075413d280 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 4 set_mon_vals no callback set 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f074bfff640 10 monclient: dump: 2024-09-17T15:36:12.569 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:12.570 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 10 monclient: shutdown 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 -- 172.21.15.86:0/3516410953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 msgr2=0x7f075413f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413f670 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f074c009d90 tx=0x7f074c01c830 comp rx=0 tx=0).stop 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 -- 172.21.15.86:0/3516410953 shutdown_connections 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413f670 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 --2- 172.21.15.86:0/3516410953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5b60 0x7f075409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 -- 172.21.15.86:0/3516410953 >> 172.21.15.86:0/3516410953 conn(0x7f07540742a0 msgr2=0x7f07540746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:12.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 -- 172.21.15.86:0/3516410953 shutdown_connections 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 1 -- 172.21.15.86:0/3516410953 wait complete. 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 10 monclient: build_initial_monmap 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.565+0000 7f075b63a640 10 monclient: monmap: 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:12.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 Processor -- start 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 -- start start 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient: init 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-c con 0x7f07540a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-a con 0x7f07540a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): picked mon.noname-b con 0x7f075413d280 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0754142760 con 0x7f07540a5b60 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f07541425e0 con 0x7f075413d280 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f07541428e0 con 0x7f07540a5070 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f075b63a640 10 monclient(hunting): authenticate will time out at 1505.537016s 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43318/0 (socket says 172.21.15.86:43318) 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33354/0 (socket says 172.21.15.86:33354) 2024-09-17T15:36:12.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 -- 172.21.15.86:0/1193261359 learned_addr learned my addr 172.21.15.86:0/1193261359 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): get_auth_request con 0x7f07540a5b60 auth_method 0 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 10 monclient(hunting): get_auth_request con 0x7f07540a5070 auth_method 0 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 msgr2=0x7f0754137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 msgr2=0x7f0754137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:12.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f07593d6640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0759bd7640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0759bd7640 10 monclient(hunting): get_auth_request con 0x7f075413d280 auth_method 0 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0759bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0759bd7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0759bd7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient(hunting): handle_auth_done global_id 94200 payload 824 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 msgr2=0x7f0754137e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 msgr2=0x7f075413b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient: _finish_hunting 0 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 monclient: found mon.noname-a 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:12.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 -- 172.21.15.86:0/1193261359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07541220e0 con 0x7f07540a5b60 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 msgr2=0x7f075413b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0758bd5640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f074400e840 tx=0x7f074400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.569+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744017070 con 0x7f07540a5b60 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.581 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: _finish_auth 0 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0744004020 con 0x7f07540a5b60 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 5 monclient: authenticate success, global_id 94200 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _renew_subs 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075413d940 con 0x7f07540a5b60 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074400f040 con 0x7f07540a5b60 2024-09-17T15:36:12.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 10 monclient: dump: 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f074b7fe640 4 set_mon_vals no callback set 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _renew_subs 2024-09-17T15:36:12.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07541226a0 con 0x7f07540a5b60 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075409cf60 con 0x7f07540a5b60 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f074400c880 con 0x7f07540a5b60 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 0x7f073803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(290..290 src has 213..290) ==== 4783+0+0 (secure 0 0 0) 0x7f074404fbf0 con 0x7f07540a5b60 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f07593d6640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 0x7f073803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.573+0000 7f07593d6640 10 monclient: get_auth_request con 0x7f073803bb60 auth_method 0 2024-09-17T15:36:12.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.577+0000 7f07593d6640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 0x7f073803e020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0750007fd0 tx=0x7f0750009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:12.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.577+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0744003620 con 0x7f07540a5b60 2024-09-17T15:36:12.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.577+0000 7f0749ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:12.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.577+0000 7f0749ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:12.726 INFO:tasks.rados.rados.0.smithi203.stdout:1249: done (4 left) 2024-09-17T15:36:12.726 INFO:tasks.rados.rados.0.smithi203.stdout:1250: read oid 12 snap -1 2024-09-17T15:36:12.726 INFO:tasks.rados.rados.0.smithi203.stdout:1250: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:12.727 INFO:tasks.rados.rados.0.smithi203.stdout:1251: snap_create 2024-09-17T15:36:12.727 INFO:tasks.rados.rados.0.smithi203.stdout:1245: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:36:12.727 INFO:tasks.rados.rados.0.smithi203.stdout:1246: expect (ObjNum 442 snap 135 seq_num 442) 2024-09-17T15:36:12.766 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3346412 2024-09-17T15:36:12.777 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:12.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0754069920 con 0x7f07540a5b60 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f0749ffb640 1 -- 172.21.15.86:0/1193261359 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f074400cb70 con 0x7f07540a5b60 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f0749ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:12.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f0749ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:12.823 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":86,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 msgr2=0x7f073803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 0x7f073803e020 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0750007fd0 tx=0x7f0750009040 comp rx=0 tx=0).stop 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 10 monclient: shutdown 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 msgr2=0x7f075413b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f074400e840 tx=0x7f074400ed00 comp rx=0 tx=0).stop 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 shutdown_connections 2024-09-17T15:36:12.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f073803bb60 0x7f073803e020 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f075413d280 0x7f075413b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07540a5b60 0x7f075413b110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 --2- 172.21.15.86:0/1193261359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07540a5070 0x7f0754137e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.821+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 >> 172.21.15.86:0/1193261359 conn(0x7f07540742a0 msgr2=0x7f07540a0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.825+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 shutdown_connections 2024-09-17T15:36:12.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:12.825+0000 7f075b63a640 1 -- 172.21.15.86:0/1193261359 wait complete. 2024-09-17T15:36:12.842 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:12.908 INFO:tasks.rados.rados.0.smithi203.stdout:1247: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:36:12.908 INFO:tasks.rados.rados.0.smithi203.stdout:1245: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:1245: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 272 (ObjNum 410 snap 117 seq_num 410) dirty exists 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:1247: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:1247: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 383 (ObjNum 398 snap 111 seq_num 398) dirty exists 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:1248: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:36:12.909 INFO:tasks.rados.rados.0.smithi203.stdout:1250: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:12.957 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1245: done (5 left) 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1246: done (4 left) 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1247: done (3 left) 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1248: done (2 left) 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1250: done (1 left) 2024-09-17T15:36:13.734 INFO:tasks.rados.rados.0.smithi203.stdout:1251: done (0 left) 2024-09-17T15:36:13.735 INFO:tasks.rados.rados.0.smithi203.stdout:1252: write oid 44 current snap is 138 2024-09-17T15:36:13.735 INFO:tasks.rados.rados.0.smithi203.stdout:1252: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-17T15:36:13.754 INFO:tasks.rados.rados.0.smithi203.stdout:1252: writing smithi20328086-44 from 463992 to 1242363 tid 1 2024-09-17T15:36:13.774 INFO:tasks.rados.rados.0.smithi203.stdout:1252: writing smithi20328086-44 from 1701083 to 2326908 tid 2 2024-09-17T15:36:13.792 INFO:tasks.rados.rados.0.smithi203.stdout:1252: writing smithi20328086-44 from 2956414 to 3615393 tid 3 2024-09-17T15:36:13.792 INFO:tasks.rados.rados.0.smithi203.stdout:1253: read oid 29 snap -1 2024-09-17T15:36:13.792 INFO:tasks.rados.rados.0.smithi203.stdout:1253: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:13.793 INFO:tasks.rados.rados.0.smithi203.stdout:1254: snap_create 2024-09-17T15:36:13.793 INFO:tasks.rados.rados.0.smithi203.stdout:1252: finishing write tid 1 to smithi20328086-44 2024-09-17T15:36:13.793 INFO:tasks.rados.rados.0.smithi203.stdout:1252: finishing write tid 2 to smithi20328086-44 2024-09-17T15:36:13.803 INFO:tasks.rados.rados.0.smithi203.stdout:1252: finishing write tid 3 to smithi20328086-44 2024-09-17T15:36:13.803 INFO:tasks.rados.rados.0.smithi203.stdout:1252: finishing write tid 4 to smithi20328086-44 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:1252: finishing write tid 5 to smithi20328086-44 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:1252: oid 44 updating version 0 to 518 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:1252: oid 44 updating version 518 to 519 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:1252: oid 44 version 519 is already newer than 517 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:1252: oid 44 updating version 519 to 520 2024-09-17T15:36:13.804 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 520 (ObjNum 456 snap 138 seq_num 456) dirty exists 2024-09-17T15:36:13.805 INFO:tasks.rados.rados.0.smithi203.stdout:1252: left oid 44 (ObjNum 456 snap 138 seq_num 456) 2024-09-17T15:36:13.805 INFO:tasks.rados.rados.0.smithi203.stdout:1253: expect (ObjNum 306 snap 84 seq_num 306) 2024-09-17T15:36:13.841 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:14.676 INFO:tasks.rados.rados.0.smithi203.stdout:1252: done (2 left) 2024-09-17T15:36:14.676 INFO:tasks.rados.rados.0.smithi203.stdout:1253: done (1 left) 2024-09-17T15:36:14.676 INFO:tasks.rados.rados.0.smithi203.stdout:1254: done (0 left) 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:1255: read oid 4 snap 129 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:1255: expect (ObjNum 417 snap 122 seq_num 417) 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:1256: rollback oid 40 current snap is 139 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 129 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:1257: write oid 6 current snap is 139 2024-09-17T15:36:14.677 INFO:tasks.rados.rados.0.smithi203.stdout:1257: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-17T15:36:14.689 INFO:tasks.rados.rados.0.smithi203.stdout:1257: writing smithi20328086-6 from 480799 to 1050772 tid 1 2024-09-17T15:36:14.704 INFO:tasks.rados.rados.0.smithi203.stdout:1257: writing smithi20328086-6 from 1475918 to 2245401 tid 2 2024-09-17T15:36:14.716 INFO:tasks.rados.rados.0.smithi203.stdout:1257: writing smithi20328086-6 from 2700190 to 3286794 tid 3 2024-09-17T15:36:14.716 INFO:tasks.rados.rados.0.smithi203.stdout:1258: rollback oid 48 current snap is 139 2024-09-17T15:36:14.717 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 138 2024-09-17T15:36:14.717 INFO:tasks.rados.rados.0.smithi203.stdout:1255: expect (ObjNum 417 snap 122 seq_num 417) 2024-09-17T15:36:14.859 INFO:tasks.rados.rados.0.smithi203.stdout:1256: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1256: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1256: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 524 (ObjNum 415 snap 119 seq_num 415) dirty exists 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: finishing write tid 1 to smithi20328086-6 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: finishing write tid 2 to smithi20328086-6 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1258: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 437 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: finishing write tid 3 to smithi20328086-6 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: finishing write tid 4 to smithi20328086-6 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: finishing write tid 5 to smithi20328086-6 2024-09-17T15:36:14.860 INFO:tasks.rados.rados.0.smithi203.stdout:1257: oid 6 updating version 0 to 459 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1257: oid 6 version 459 is already newer than 458 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1257: oid 6 version 459 is already newer than 456 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1257: oid 6 version 459 is already newer than 457 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 459 (ObjNum 457 snap 139 seq_num 457) dirty exists 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1257: left oid 6 (ObjNum 457 snap 139 seq_num 457) 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1255: done (3 left) 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1256: done (2 left) 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1257: done (1 left) 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1258: done (0 left) 2024-09-17T15:36:14.861 INFO:tasks.rados.rados.0.smithi203.stdout:1259: snap_create 2024-09-17T15:36:15.751 INFO:tasks.rados.rados.0.smithi203.stdout:1259: done (0 left) 2024-09-17T15:36:15.752 INFO:tasks.rados.rados.0.smithi203.stdout:1260: read oid 31 snap -1 2024-09-17T15:36:15.752 INFO:tasks.rados.rados.0.smithi203.stdout:1260: expect (ObjNum 439 snap 131 seq_num 439) 2024-09-17T15:36:15.752 INFO:tasks.rados.rados.0.smithi203.stdout:1261: delete oid 41 current snap is 140 2024-09-17T15:36:15.765 INFO:tasks.rados.rados.0.smithi203.stdout:1261: done (1 left) 2024-09-17T15:36:15.765 INFO:tasks.rados.rados.0.smithi203.stdout:1262: delete oid 47 current snap is 140 2024-09-17T15:36:15.773 INFO:tasks.rados.rados.0.smithi203.stdout:1262: done (1 left) 2024-09-17T15:36:15.773 INFO:tasks.rados.rados.0.smithi203.stdout:1263: write oid 46 current snap is 140 2024-09-17T15:36:15.773 INFO:tasks.rados.rados.0.smithi203.stdout:1263: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-17T15:36:15.780 INFO:tasks.rados.rados.0.smithi203.stdout:1263: writing smithi20328086-46 from 531220 to 1326940 tid 1 2024-09-17T15:36:15.787 INFO:tasks.rados.rados.0.smithi203.stdout:1263: writing smithi20328086-46 from 1935011 to 2451821 tid 2 2024-09-17T15:36:15.793 INFO:tasks.rados.rados.0.smithi203.stdout:1263: writing smithi20328086-46 from 3066106 to 3486526 tid 3 2024-09-17T15:36:15.793 INFO:tasks.rados.rados.0.smithi203.stdout:1264: read oid 24 snap -1 2024-09-17T15:36:15.793 INFO:tasks.rados.rados.0.smithi203.stdout:1264: expect (ObjNum 446 snap 136 seq_num 446) 2024-09-17T15:36:15.793 INFO:tasks.rados.rados.0.smithi203.stdout:1265: snap_remove snap 130 2024-09-17T15:36:15.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:15.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 10 monclient: get_monmap_and_config 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 10 monclient: build_initial_monmap 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 10 monclient: monmap: 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:15.974418+0000 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:15.974418+0000 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:15.972 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:15.973 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:15.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 1 Processor -- start 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 1 -- start start 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 10 monclient: init 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.969+0000 7f36dab44640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36d40a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f36d40a54c0 0x7f36d40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-c con 0x7f36d40a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-b con 0x7f36d413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36d40a6530 con 0x7f36d40a50e0 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f36d409bdf0 con 0x7f36d40a54c0 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f36d409c1a0 con 0x7f36d413d2f0 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36dab44640 10 monclient(hunting): authenticate will time out at 1508.940141s 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33384/0 (socket says 172.21.15.86:33384) 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 -- 172.21.15.86:0/3347254336 learned_addr learned my addr 172.21.15.86:0/3347254336 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 10 monclient(hunting): get_auth_request con 0x7f36d413d2f0 auth_method 0 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 -- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 msgr2=0x7f36d413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 -- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 msgr2=0x7f36d413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d90e1640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 --2- 172.21.15.86:0/3347254336 >> v2:172.21.15.203:3300/0 conn(0x7f36d40a54c0 0x7f36d40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): get_auth_request con 0x7f36d40a54c0 auth_method 0 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient(hunting): handle_auth_done global_id 54430 payload 274 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 -- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 msgr2=0x7f36d413f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 -- 172.21.15.86:0/3347254336 >> v1:172.21.15.203:6789/0 conn(0x7f36d40a50e0 legacy=0x7f36d409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 monclient: found mon.noname-c 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:15.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:36:15.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.973+0000 7f36d3fff640 1 -- 172.21.15.86:0/3347254336 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d409c460 con 0x7f36d40a54c0 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d3fff640 1 -- 172.21.15.86:0/3347254336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a54c0 msgr2=0x7f36d40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d3fff640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a54c0 0x7f36d40a58d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36c80097c0 tx=0x7f36c801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf6a810c74d6185c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 1 -- 172.21.15.86:0/3347254336 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c80b0070 con 0x7f36d40a54c0 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: dump: 2024-09-17T15:36:15.981 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: _finish_auth 0 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:15.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 1 -- 172.21.15.86:0/3347254336 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36c80ab600 con 0x7f36d40a54c0 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 5 monclient: authenticate success, global_id 54430 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 1 -- 172.21.15.86:0/3347254336 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c80ab9d0 con 0x7f36d40a54c0 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 4 set_mon_vals no callback set 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36d37fe640 10 monclient: dump: 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:15.983 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 10 monclient: shutdown 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 -- 172.21.15.86:0/3347254336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a54c0 msgr2=0x7f36d40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a54c0 0x7f36d40a58d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f36c80097c0 tx=0x7f36c801c360 comp rx=0 tx=0).stop 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 -- 172.21.15.86:0/3347254336 shutdown_connections 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d413d2f0 0x7f36d413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 --2- 172.21.15.86:0/3347254336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a54c0 0x7f36d40a58d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 -- 172.21.15.86:0/3347254336 >> 172.21.15.86:0/3347254336 conn(0x7f36d40aaae0 msgr2=0x7f36d40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:15.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 -- 172.21.15.86:0/3347254336 shutdown_connections 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.977+0000 7f36dab44640 1 -- 172.21.15.86:0/3347254336 wait complete. 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient: build_initial_monmap 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient: monmap: 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:15.985 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 Processor -- start 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 -- start start 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient: init 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a50e0 0x7f36d41351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-b con 0x7f36d40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-c con 0x7f36d40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 0x7f36d41d00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): picked mon.noname-a con 0x7f36d413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36d4142820 con 0x7f36d413d2f0 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36d41426a0 con 0x7f36d40a50e0 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f36d41429a0 con 0x7f36d40a54c0 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36dab44640 10 monclient(hunting): authenticate will time out at 1508.949498s 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33396/0 (socket says 172.21.15.86:33396) 2024-09-17T15:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 -- 172.21.15.86:0/2098433907 learned_addr learned my addr 172.21.15.86:0/2098433907 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 10 monclient(hunting): get_auth_request con 0x7f36d40a54c0 auth_method 0 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 0x7f36d41d00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): get_auth_request con 0x7f36d413d2f0 auth_method 0 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 msgr2=0x7f36d41356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 msgr2=0x7f36d41356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:15.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d3fff640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d88e0640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a50e0 0x7f36d41351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d88e0640 10 monclient(hunting): get_auth_request con 0x7f36d40a50e0 auth_method 0 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d88e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d88e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d88e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient(hunting): handle_auth_done global_id 94203 payload 824 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 msgr2=0x7f36d41356e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:15.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a50e0 msgr2=0x7f36d41351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a50e0 0x7f36d41351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient: _finish_hunting 0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 monclient: found mon.noname-a 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.981+0000 7f36d90e1640 1 -- 172.21.15.86:0/2098433907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36d413b2f0 con 0x7f36d413d2f0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d90e1640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 msgr2=0x7f36d41d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d90e1640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 0x7f36d41d00a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0096d0 tx=0x7f36cc007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc01a070 con 0x7f36d413d2f0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: dump: 2024-09-17T15:36:15.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: _finish_auth 0 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36cc00ddc0 con 0x7f36d413d2f0 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc015640 con 0x7f36d413d2f0 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 5 monclient: authenticate success, global_id 94203 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _renew_subs 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36d413db20 con 0x7f36d413d2f0 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d2ffd640 4 set_mon_vals no callback set 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 10 monclient: dump: 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _renew_subs 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36d4139b20 con 0x7f36d413d2f0 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:15.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36d40a58d0 con 0x7f36d413d2f0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f36cc01d070 con 0x7f36d413d2f0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 0x7f36c003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(294..294 src has 213..294) ==== 4755+0+0 (secure 0 0 0) 0x7f36cc04ff20 con 0x7f36d413d2f0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d88e0640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 0x7f36c003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.985+0000 7f36d88e0640 10 monclient: get_auth_request con 0x7f36c003bb60 auth_method 0 2024-09-17T15:36:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.989+0000 7f36d88e0640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 0x7f36c003e020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36c4004e30 tx=0x7f36c4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.989+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f36cc010070 con 0x7f36d413d2f0 2024-09-17T15:36:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.989+0000 7f36d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:15.989+0000 7f36d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f36d413cdd0 con 0x7f36d413d2f0 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36d17fa640 1 -- 172.21.15.86:0/2098433907 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f36cc01eda0 con 0x7f36d413d2f0 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:16.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:16.227 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 msgr2=0x7f36c003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 0x7f36c003e020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f36c4004e30 tx=0x7f36c4009040 comp rx=0 tx=0).stop 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 10 monclient: shutdown 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 msgr2=0x7f36d41d00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 0x7f36d41d00a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0096d0 tx=0x7f36cc007ef0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 shutdown_connections 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f36c003bb60 0x7f36c003e020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36d413d2f0 0x7f36d41d00a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36d40a54c0 0x7f36d41356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 --2- 172.21.15.86:0/2098433907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36d40a50e0 0x7f36d41351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 >> 172.21.15.86:0/2098433907 conn(0x7f36d40aaae0 msgr2=0x7f36d413c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:16.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 shutdown_connections 2024-09-17T15:36:16.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.225+0000 7f36dab44640 1 -- 172.21.15.86:0/2098433907 wait complete. 2024-09-17T15:36:16.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:36:16.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:36:16.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:36:16.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: get_monmap_and_config 2024-09-17T15:36:16.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: build_initial_monmap 2024-09-17T15:36:16.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: monmap: 2024-09-17T15:36:16.369 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:16.369 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:16.372066+0000 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:16.372066+0000 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:16.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 Processor -- start 2024-09-17T15:36:16.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 -- start start 2024-09-17T15:36:16.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: init 2024-09-17T15:36:16.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8448134d90 0x7f84480a1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8448134d90 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8448135170 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8448135550 0x7f8448138120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8448135550 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f84480a21b0 con 0x7f8448135550 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8448073b10 con 0x7f8448134d90 2024-09-17T15:36:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f84480a1ed0 con 0x7f8448135170 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844eb3d640 10 monclient(hunting): authenticate will time out at 1509.337563s 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8448135550 0x7f8448138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8448135550 0x7f8448138120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43332/0 (socket says 172.21.15.86:43332) 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 -- 172.21.15.86:0/3531405533 learned_addr learned my addr 172.21.15.86:0/3531405533 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33406/0 (socket says 172.21.15.86:33406) 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 10 monclient(hunting): get_auth_request con 0x7f8448135170 auth_method 0 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): get_auth_request con 0x7f8448135550 auth_method 0 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 -- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 msgr2=0x7f8448137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 -- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 msgr2=0x7f8448137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f8447fff640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844c8d9640 1 --2- 172.21.15.86:0/3531405533 >> v2:172.21.15.203:3300/0 conn(0x7f8448134d90 0x7f84480a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844c8d9640 10 monclient(hunting): get_auth_request con 0x7f8448134d90 auth_method 0 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844c8d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient(hunting): handle_auth_done global_id 94206 payload 274 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 -- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 msgr2=0x7f8448137be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 -- 172.21.15.86:0/3531405533 >> v2:172.21.15.203:3300/0 conn(0x7f8448134d90 msgr2=0x7f84480a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 --2- 172.21.15.86:0/3531405533 >> v2:172.21.15.203:3300/0 conn(0x7f8448134d90 0x7f84480a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient: _finish_hunting 0 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 monclient: found mon.noname-a 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.369+0000 7f844d0da640 1 -- 172.21.15.86:0/3531405533 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84480a2490 con 0x7f8448135550 2024-09-17T15:36:16.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844d0da640 1 -- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448135550 msgr2=0x7f8448138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844d0da640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448135550 0x7f8448138120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f844806d160 tx=0x7f844001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7080862bc9d1085d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 1 -- 172.21.15.86:0/3531405533 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400a5530 con 0x7f8448135550 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: dump: 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: _finish_auth 0 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: _check_auth_tickets 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 1 -- 172.21.15.86:0/3531405533 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84400a56d0 con 0x7f8448135550 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 5 monclient: authenticate success, global_id 94206 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 4 set_mon_vals no callback set 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 1 -- 172.21.15.86:0/3531405533 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400a5aa0 con 0x7f8448135550 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f84477fe640 10 monclient: dump: 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 10 monclient: shutdown 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 -- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448135550 msgr2=0x7f8448138120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448135550 0x7f8448138120 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f844806d160 tx=0x7f844001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 -- 172.21.15.86:0/3531405533 shutdown_connections 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448135550 0x7f8448138120 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3531405533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135170 0x7f8448137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3531405533 >> v2:172.21.15.203:3300/0 conn(0x7f8448134d90 0x7f84480a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 -- 172.21.15.86:0/3531405533 >> 172.21.15.86:0/3531405533 conn(0x7f84480a2f40 msgr2=0x7f84480a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 -- 172.21.15.86:0/3531405533 shutdown_connections 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 1 -- 172.21.15.86:0/3531405533 wait complete. 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 10 monclient: build_initial_monmap 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.373+0000 7f844eb3d640 10 monclient: monmap: 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:16.383 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 Processor -- start 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- start start 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: init 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 0x7f84481319a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8448134d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8448135170 0x7f8448131ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8448135170 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 0x7f844813ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8448135550 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f84481419a0 con 0x7f8448134d90 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8448138a70 con 0x7f8448135170 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8448141b20 con 0x7f8448135550 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient(hunting): authenticate will time out at 1509.344733s 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 0x7f84481319a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 0x7f844813ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 0x7f84481319a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43348/0 (socket says 172.21.15.86:43348) 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 -- 172.21.15.86:0/3261910556 learned_addr learned my addr 172.21.15.86:0/3261910556 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): get_auth_request con 0x7f8448134d90 auth_method 0 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 10 monclient(hunting): get_auth_request con 0x7f8448135550 auth_method 0 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 msgr2=0x7f844813ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:36:16.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 msgr2=0x7f844813ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 0x7f844813ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844d0da640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 0x7f844813ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8448135170 0x7f8448131ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient(hunting): handle_auth_done global_id 94209 payload 824 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 msgr2=0x7f844813ba50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8448135550 0x7f844813ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8448135170 msgr2=0x7f8448131ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8448135170 0x7f8448131ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient: _finish_hunting 0 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 monclient: found mon.noname-a 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844811cdf0 con 0x7f8448134d90 2024-09-17T15:36:16.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f8447fff640 10 monclient: get_auth_request con 0x7f8448135170 auth_method 0 2024-09-17T15:36:16.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8448135170 0x7f8448131ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:36:16.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 msgr2=0x7f84481319a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844c8d9640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 0x7f84481319a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d830 tx=0x7f843c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c018020 con 0x7f8448134d90 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: dump: 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: _finish_auth 0 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: _check_auth_tickets 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f843c0098b0 con 0x7f8448134d90 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 5 monclient: authenticate success, global_id 94209 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _renew_subs 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844811d0d0 con 0x7f8448134d90 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:16.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f843c009c60 con 0x7f8448134d90 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f84457fa640 10 monclient: dump: 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _renew_subs 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f8446ffd640 4 set_mon_vals no callback set 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.377+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f844811cb10 con 0x7f8448134d90 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844eb3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 0x7f844802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f84480a1130 con 0x7f844813c750 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 0x7f844802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient: get_auth_request con 0x7f844813c750 auth_method 0 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f843c01b020 con 0x7f8448134d90 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 10 monclient(hunting): handle_auth_done global_id 94212 payload 824 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8448134340 con 0x7f844813c750 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843403bb60 0x7f843403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(294..294 src has 213..294) ==== 4755+0+0 (secure 0 0 0) 0x7f843c0505a0 con 0x7f8448134d90 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844d0da640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843403bb60 0x7f843403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f844d0da640 10 monclient: get_auth_request con 0x7f843403bb60 auth_method 0 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 msgr2=0x7f844802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 0x7f844802d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f844813db80 tx=0x7f843800f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438017020 con 0x7f844813c750 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 10 monclient: dump: 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8438003820 con 0x7f844813c750 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 msgr2=0x7f844802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 0x7f844802d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f844813db80 tx=0x7f843800f9e0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 msgr2=0x7f844802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.381+0000 7f84457fa640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813c750 0x7f844802d450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.385+0000 7f844d0da640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843403bb60 0x7f843403e020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f84400ab000 tx=0x7f84400a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f844eb3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 0x7f844802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f844eb3d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f844806d4f0 con 0x7f844813dc50 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 0x7f844802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient: get_auth_request con 0x7f844813dc50 auth_method 0 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 10 monclient(hunting): handle_auth_done global_id 94215 payload 824 2024-09-17T15:36:16.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.389+0000 7f8447fff640 1 -- 172.21.15.86:0/3261910556 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84481dafb0 con 0x7f844813dc50 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f8447fff640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 msgr2=0x7f844802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f8447fff640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 0x7f844802d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f844813db80 tx=0x7f84380046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438017030 con 0x7f844813dc50 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 10 monclient: dump: 2024-09-17T15:36:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f843801c630 con 0x7f844813dc50 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 msgr2=0x7f844802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 0x7f844802d990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f844813db80 tx=0x7f84380046d0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 msgr2=0x7f844802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f84457fa640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f844813dc50 0x7f844802d990 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f844813db80 tx=0x7f84380046d0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 51747, 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:16.416 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:16.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843403bb60 msgr2=0x7f843403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843403bb60 0x7f843403e020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f84400ab000 tx=0x7f84400a7070 comp rx=0 tx=0).stop 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 10 monclient: shutdown 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 msgr2=0x7f84481319a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 --2- 172.21.15.86:0/3261910556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8448134d90 0x7f84481319a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f843c00d830 tx=0x7f843c00dcf0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844d0da640 1 -- 172.21.15.86:0/3261910556 reap_dead start 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844d0da640 1 -- 172.21.15.86:0/3261910556 reap_dead start 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 shutdown_connections 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 >> 172.21.15.86:0/3261910556 conn(0x7f84480a2f40 msgr2=0x7f8448137970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 shutdown_connections 2024-09-17T15:36:16.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.393+0000 7f844eb3d640 1 -- 172.21.15.86:0/3261910556 wait complete. 2024-09-17T15:36:16.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:16.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:36:16.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:36:16.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: get_monmap_and_config 2024-09-17T15:36:16.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: build_initial_monmap 2024-09-17T15:36:16.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: monmap: 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:16.536231+0000 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:16.536231+0000 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:16.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 Processor -- start 2024-09-17T15:36:16.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 -- start start 2024-09-17T15:36:16.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: init 2024-09-17T15:36:16.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:16.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e480a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2e480a1760 0x7f2e4813af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-c con 0x7f2e480a1760 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-b con 0x7f2e4813b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e480a2130 con 0x7f2e480a0c70 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2e480a2410 con 0x7f2e480a1760 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e480a26f0 con 0x7f2e4813b450 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:16.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e4df27640 10 monclient(hunting): authenticate will time out at 1509.501894s 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e477fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e480a0c70 0x7f2e480a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49540/0 (socket says 172.21.15.86:49540) 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e477fe640 1 -- 172.21.15.86:0/2739697174 learned_addr learned my addr 172.21.15.86:0/2739697174 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 10 monclient(hunting): get_auth_request con 0x7f2e4813b450 auth_method 0 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 1 -- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 msgr2=0x7f2e4813d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 1 -- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 msgr2=0x7f2e4813d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e47fff640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/2739697174 >> v2:172.21.15.203:3300/0 conn(0x7f2e480a1760 0x7f2e4813af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): get_auth_request con 0x7f2e480a1760 auth_method 0 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient(hunting): handle_auth_done global_id 54439 payload 274 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 -- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 msgr2=0x7f2e4813d840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 -- 172.21.15.86:0/2739697174 >> v1:172.21.15.86:6789/0 conn(0x7f2e480a0c70 legacy=0x7f2e480a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 monclient: found mon.noname-c 2024-09-17T15:36:16.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:16.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:36:16.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.533+0000 7f2e46ffd640 1 -- 172.21.15.86:0/2739697174 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4809f9f0 con 0x7f2e480a1760 2024-09-17T15:36:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e46ffd640 1 -- 172.21.15.86:0/2739697174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 msgr2=0x7f2e4813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e4813af10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097c0 tx=0x7f2e3001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c3dbfe0e3460ad server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 1 -- 172.21.15.86:0/2739697174 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300b0070 con 0x7f2e480a1760 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: dump: 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.543 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: _finish_auth 0 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: _check_auth_tickets 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e4df27640 5 monclient: authenticate success, global_id 54439 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e4df27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 1 -- 172.21.15.86:0/2739697174 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e300ab600 con 0x7f2e480a1760 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 1 -- 172.21.15.86:0/2739697174 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300ab9d0 con 0x7f2e480a1760 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e467fc640 10 monclient: dump: 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e4df27640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.537+0000 7f2e4df27640 4 set_mon_vals no callback set 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: shutdown 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- 172.21.15.86:0/2739697174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 msgr2=0x7f2e4813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e4813af10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2e300097c0 tx=0x7f2e3001c360 comp rx=0 tx=0).stop 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- 172.21.15.86:0/2739697174 shutdown_connections 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4813b450 0x7f2e4813d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- 172.21.15.86:0/2739697174 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e4813af10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- 172.21.15.86:0/2739697174 >> 172.21.15.86:0/2739697174 conn(0x7f2e480a2f40 msgr2=0x7f2e480a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- 172.21.15.86:0/2739697174 shutdown_connections 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- 172.21.15.86:0/2739697174 wait complete. 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: build_initial_monmap 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: monmap: 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:16.547 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 Processor -- start 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- start start 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: init 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-c con 0x7f2e480a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e48135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-b con 0x7f2e480a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:16.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 0x7f2e48136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): picked mon.noname-a con 0x7f2e4813b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48140720 con 0x7f2e4813b450 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e481405a0 con 0x7f2e480a1760 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e481408a0 con 0x7f2e480a0c70 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e4df27640 10 monclient(hunting): authenticate will time out at 1509.510768s 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 0x7f2e48136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33440/0 (socket says 172.21.15.86:33440) 2024-09-17T15:36:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): get_auth_request con 0x7f2e4813b450 auth_method 0 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 learned_addr learned my addr 172.21.15.86:0/1166109782 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 10 monclient(hunting): get_auth_request con 0x7f2e480a0c70 auth_method 0 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 msgr2=0x7f2e481cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:36:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 msgr2=0x7f2e481cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e48135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e46ffd640 10 monclient(hunting): get_auth_request con 0x7f2e480a1760 auth_method 0 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient(hunting): handle_auth_done global_id 94218 payload 824 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 msgr2=0x7f2e481cdcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e480a0c70 0x7f2e481cdcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 msgr2=0x7f2e48135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e480a1760 0x7f2e48135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 monclient: found mon.noname-a 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.541+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e48131560 con 0x7f2e4813b450 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 msgr2=0x7f2e48136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e47fff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 0x7f2e48136340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e38000ee0 tx=0x7f2e3800e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38017020 con 0x7f2e4813b450 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: dump: 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: _finish_auth 0 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e38013860 con 0x7f2e4813b450 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800f040 con 0x7f2e4813b450 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 10 monclient: dump: 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 5 monclient: authenticate success, global_id 94218 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 monclient: _renew_subs 2024-09-17T15:36:16.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e45ffb640 4 set_mon_vals no callback set 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4813ba30 con 0x7f2e4813b450 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 monclient: _renew_subs 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e48136880 con 0x7f2e4813b450 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e480a1050 con 0x7f2e4812fc00 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e4812fc00 auth_method 0 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2e3801a020 con 0x7f2e4813b450 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bb60 0x7f2e2803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(294..294 src has 213..294) ==== 4755+0+0 (secure 0 0 0) 0x7f2e3804fc90 con 0x7f2e4813b450 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bb60 0x7f2e2803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.545+0000 7f2e46ffd640 10 monclient: get_auth_request con 0x7f2e2803bb60 auth_method 0 2024-09-17T15:36:16.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.549+0000 7f2e46ffd640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bb60 0x7f2e2803e020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e4806d160 tx=0x7f2e300a4f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e4812fc00 auth_method 0 2024-09-17T15:36:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:36:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:16.749+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:36:16.772 INFO:tasks.rados.rados.0.smithi203.stdout:1265: done (3 left) 2024-09-17T15:36:16.772 INFO:tasks.rados.rados.0.smithi203.stdout:1266: copy_from oid 18 from oid 2 current snap is 140 2024-09-17T15:36:16.772 INFO:tasks.rados.rados.0.smithi203.stdout:1260: expect (ObjNum 439 snap 131 seq_num 439) 2024-09-17T15:36:16.887 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:16.934 INFO:tasks.rados.rados.0.smithi203.stdout:1263: finishing write tid 1 to smithi20328086-46 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: finishing write tid 2 to smithi20328086-46 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: finishing write tid 3 to smithi20328086-46 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: finishing write tid 4 to smithi20328086-46 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: finishing write tid 5 to smithi20328086-46 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: oid 46 updating version 0 to 393 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: oid 46 version 393 is already newer than 391 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: oid 46 version 393 is already newer than 390 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: oid 46 version 393 is already newer than 392 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 393 (ObjNum 460 snap 140 seq_num 460) dirty exists 2024-09-17T15:36:16.935 INFO:tasks.rados.rados.0.smithi203.stdout:1263: left oid 46 (ObjNum 460 snap 140 seq_num 460) 2024-09-17T15:36:16.936 INFO:tasks.rados.rados.0.smithi203.stdout:1264: expect (ObjNum 446 snap 136 seq_num 446) 2024-09-17T15:36:16.936 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:17.132 INFO:tasks.rados.rados.0.smithi203.stdout:1266: finishing copy_from to smithi20328086-18 2024-09-17T15:36:17.132 INFO:tasks.rados.rados.0.smithi203.stdout:1266: got expected ENOENT (src dne) 2024-09-17T15:36:17.132 INFO:tasks.rados.rados.0.smithi203.stdout:1266: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:36:17.132 INFO:tasks.rados.rados.0.smithi203.stdout:1260: done (3 left) 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1263: done (2 left) 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1264: done (1 left) 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1266: done (0 left) 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1267: read oid 21 snap -1 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1267: expect (ObjNum 450 snap 137 seq_num 450) 2024-09-17T15:36:17.133 INFO:tasks.rados.rados.0.smithi203.stdout:1268: delete oid 3 current snap is 140 2024-09-17T15:36:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:1268: done (1 left) 2024-09-17T15:36:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:1269: snap_remove snap 127 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e4812fc00 auth_method 0 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:36:17.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:17.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.153+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:36:17.772 INFO:tasks.rados.rados.0.smithi203.stdout:1269: done (1 left) 2024-09-17T15:36:17.772 INFO:tasks.rados.rados.0.smithi203.stdout:1270: copy_from oid 34 from oid 18 current snap is 140 2024-09-17T15:36:17.772 INFO:tasks.rados.rados.0.smithi203.stdout:1267: expect (ObjNum 450 snap 137 seq_num 450) 2024-09-17T15:36:17.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:17.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:17.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e4812fc00 auth_method 0 2024-09-17T15:36:17.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:17.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:17.957+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:36:18.053 INFO:tasks.rados.rados.0.smithi203.stdout:1270: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1270: finishing copy_from to smithi20328086-34 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 315 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1267: done (1 left) 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1270: done (0 left) 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1271: copy_from oid 40 from oid 21 current snap is 140 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1272: write (excl) oid 45 current snap is 140 2024-09-17T15:36:18.058 INFO:tasks.rados.rados.0.smithi203.stdout:1272: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-17T15:36:18.072 INFO:tasks.rados.rados.0.smithi203.stdout:1272: writing smithi20328086-45 from 564834 to 1111052 tid 1 2024-09-17T15:36:18.083 INFO:tasks.rados.rados.0.smithi203.stdout:1272: writing smithi20328086-45 from 1651975 to 2172454 tid 2 2024-09-17T15:36:18.089 INFO:tasks.rados.rados.0.smithi203.stdout:1272: writing smithi20328086-45 from 2720952 to 3280269 tid 3 2024-09-17T15:36:18.089 INFO:tasks.rados.rados.0.smithi203.stdout:1273: copy_from oid 50 from oid 47 current snap is 140 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:1271: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:1271: finishing copy_from to smithi20328086-40 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 519 (ObjNum 450 snap 137 seq_num 450) dirty exists 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:1272: finishing write tid 1 to smithi20328086-45 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:1272: finishing write tid 2 to smithi20328086-45 2024-09-17T15:36:18.090 INFO:tasks.rados.rados.0.smithi203.stdout:1271: done (2 left) 2024-09-17T15:36:18.091 INFO:tasks.rados.rados.0.smithi203.stdout:1274: read oid 4 snap -1 2024-09-17T15:36:18.091 INFO:tasks.rados.rados.0.smithi203.stdout:1274: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:18.091 INFO:tasks.rados.rados.0.smithi203.stdout:1275: snap_remove snap 137 2024-09-17T15:36:18.803 INFO:tasks.rados.rados.0.smithi203.stdout:1275: done (3 left) 2024-09-17T15:36:18.804 INFO:tasks.rados.rados.0.smithi203.stdout:1276: write (excl) oid 8 current snap is 140 2024-09-17T15:36:18.804 INFO:tasks.rados.rados.0.smithi203.stdout:1276: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-17T15:36:18.820 INFO:tasks.rados.rados.0.smithi203.stdout:1276: writing smithi20328086-8 from 581641 to 1203108 tid 1 2024-09-17T15:36:18.840 INFO:tasks.rados.rados.0.smithi203.stdout:1276: writing smithi20328086-8 from 1994104 to 2774594 tid 2 2024-09-17T15:36:18.853 INFO:tasks.rados.rados.0.smithi203.stdout:1276: writing smithi20328086-8 from 3432022 to 3781641 tid 3 2024-09-17T15:36:18.854 INFO:tasks.rados.rados.0.smithi203.stdout:1277: read oid 32 snap -1 2024-09-17T15:36:18.854 INFO:tasks.rados.rados.0.smithi203.stdout:1277: expect (ObjNum 432 snap 126 seq_num 432) 2024-09-17T15:36:18.854 INFO:tasks.rados.rados.0.smithi203.stdout:1278: write (excl) oid 30 current snap is 140 2024-09-17T15:36:18.854 INFO:tasks.rados.rados.0.smithi203.stdout:1278: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-17T15:36:18.866 INFO:tasks.rados.rados.0.smithi203.stdout:1278: writing smithi20328086-30 from 598448 to 1011517 tid 1 2024-09-17T15:36:18.876 INFO:tasks.rados.rados.0.smithi203.stdout:1278: writing smithi20328086-30 from 1768939 to 2293087 tid 2 2024-09-17T15:36:18.885 INFO:tasks.rados.rados.0.smithi203.stdout:1278: writing smithi20328086-30 from 2775798 to 3357659 tid 3 2024-09-17T15:36:18.888 INFO:tasks.rados.rados.0.smithi203.stdout:1278: writing smithi20328086-30 from 3781488 to 3798448 tid 4 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1279: write oid 38 current snap is 140 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1273: finishing copy_from to smithi20328086-50 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1273: got expected ENOENT (src dne) 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1273: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1272: finishing write tid 3 to smithi20328086-45 2024-09-17T15:36:18.889 INFO:tasks.rados.rados.0.smithi203.stdout:1279: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-17T15:36:18.896 INFO:tasks.rados.rados.0.smithi203.stdout:1279: writing smithi20328086-38 from 615255 to 1103573 tid 1 2024-09-17T15:36:18.907 INFO:tasks.rados.rados.0.smithi203.stdout:1279: writing smithi20328086-38 from 1827421 to 2611580 tid 2 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1279: writing smithi20328086-38 from 3203221 to 3815255 tid 3 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1273: done (6 left) 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1280: snap_create 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1272: finishing write tid 4 to smithi20328086-45 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1272: finishing write tid 5 to smithi20328086-45 2024-09-17T15:36:18.916 INFO:tasks.rados.rados.0.smithi203.stdout:1272: oid 45 updating version 0 to 276 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:1272: oid 45 version 276 is already newer than 274 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:1272: oid 45 version 276 is already newer than 273 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:1272: oid 45 version 276 is already newer than 275 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 276 (ObjNum 462 snap 140 seq_num 462) dirty exists 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:1272: left oid 45 (ObjNum 462 snap 140 seq_num 462) 2024-09-17T15:36:18.917 INFO:tasks.rados.rados.0.smithi203.stdout:1274: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:18.949 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2227579 2024-09-17T15:36:18.981 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 700688 2024-09-17T15:36:19.026 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 694841 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: finishing write tid 1 to smithi20328086-8 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: finishing write tid 2 to smithi20328086-8 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: finishing write tid 3 to smithi20328086-8 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: finishing write tid 4 to smithi20328086-8 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: finishing write tid 5 to smithi20328086-8 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: oid 8 updating version 0 to 351 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: oid 8 version 351 is already newer than 348 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: oid 8 version 351 is already newer than 350 2024-09-17T15:36:19.063 INFO:tasks.rados.rados.0.smithi203.stdout:1276: oid 8 version 351 is already newer than 349 2024-09-17T15:36:19.064 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 351 (ObjNum 463 snap 140 seq_num 463) dirty exists 2024-09-17T15:36:19.064 INFO:tasks.rados.rados.0.smithi203.stdout:1276: left oid 8 (ObjNum 463 snap 140 seq_num 463) 2024-09-17T15:36:19.064 INFO:tasks.rados.rados.0.smithi203.stdout:1277: expect (ObjNum 432 snap 126 seq_num 432) 2024-09-17T15:36:19.130 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2226541 2024-09-17T15:36:19.235 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3109745 2024-09-17T15:36:19.346 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 1 to smithi20328086-30 2024-09-17T15:36:19.346 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 2 to smithi20328086-30 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 3 to smithi20328086-30 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 4 to smithi20328086-30 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 5 to smithi20328086-30 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: finishing write tid 6 to smithi20328086-30 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: oid 30 updating version 0 to 272 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: oid 30 updating version 272 to 276 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: oid 30 version 276 is already newer than 273 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: oid 30 version 276 is already newer than 274 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: oid 30 version 276 is already newer than 275 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 276 (ObjNum 464 snap 140 seq_num 464) dirty exists 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1278: left oid 30 (ObjNum 464 snap 140 seq_num 464) 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1279: finishing write tid 1 to smithi20328086-38 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1279: finishing write tid 2 to smithi20328086-38 2024-09-17T15:36:19.347 INFO:tasks.rados.rados.0.smithi203.stdout:1279: finishing write tid 3 to smithi20328086-38 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: finishing write tid 4 to smithi20328086-38 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: finishing write tid 5 to smithi20328086-38 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: oid 38 updating version 0 to 499 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: oid 38 version 499 is already newer than 497 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: oid 38 updating version 499 to 500 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: oid 38 version 500 is already newer than 498 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 500 (ObjNum 465 snap 140 seq_num 465) dirty exists 2024-09-17T15:36:19.348 INFO:tasks.rados.rados.0.smithi203.stdout:1279: left oid 38 (ObjNum 465 snap 140 seq_num 465) 2024-09-17T15:36:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.541+0000 7f2e3effd640 10 monclient: tick 2024-09-17T15:36:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.541+0000 7f2e3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.541+0000 7f2e3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.541+0000 7f2e3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e4812fc00 auth_method 0 2024-09-17T15:36:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:19.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 10 monclient(hunting): handle_auth_done global_id 104099 payload 824 2024-09-17T15:36:19.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e481320c0 con 0x7f2e4812fc00 2024-09-17T15:36:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.561+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2e400052b0 tx=0x7f2e400156c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40010040 con 0x7f2e4812fc00 2024-09-17T15:36:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 10 monclient: dump: 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e40019860 con 0x7f2e4812fc00 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2e400052b0 tx=0x7f2e400156c0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 msgr2=0x7f2e4812ffe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:19.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.565+0000 7f2e3ffff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e4812fc00 0x7f2e4812ffe0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2e400052b0 tx=0x7f2e400156c0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e4df27640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e4df27640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 0x7f2e48136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e4df27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4806d4f0 con 0x7f2e48130520 2024-09-17T15:36:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 0x7f2e48136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient: get_auth_request con 0x7f2e48130520 auth_method 0 2024-09-17T15:36:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 10 monclient(hunting): handle_auth_done global_id 104102 payload 824 2024-09-17T15:36:19.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e48130930 con 0x7f2e48130520 2024-09-17T15:36:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 msgr2=0x7f2e48136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e477fe640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 0x7f2e48136ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e40015700 tx=0x7f2e40005190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40026c50 con 0x7f2e48130520 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.573+0000 7f2e3ffff640 10 monclient: dump: 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.578 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f2e40026c50 con 0x7f2e48130520 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 msgr2=0x7f2e48136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 0x7f2e48136ef0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e40015700 tx=0x7f2e40005190 comp rx=0 tx=0).stop 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 msgr2=0x7f2e48136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e3ffff640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e48130520 0x7f2e48136ef0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:19.579 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10447, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:19.580 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.581 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.582 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:19.583 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.584 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.585 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.587 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.588 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.590 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.591 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:19.592 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bb60 msgr2=0x7f2e2803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e2803bb60 0x7f2e2803e020 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e4806d160 tx=0x7f2e300a4f90 comp rx=0 tx=0).stop 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 10 monclient: shutdown 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 msgr2=0x7f2e48136340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 --2- 172.21.15.86:0/1166109782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e4813b450 0x7f2e48136340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2e38000ee0 tx=0x7f2e3800e900 comp rx=0 tx=0).stop 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 reap_dead start 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e47fff640 1 -- 172.21.15.86:0/1166109782 reap_dead start 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 shutdown_connections 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 >> 172.21.15.86:0/1166109782 conn(0x7f2e480a2f40 msgr2=0x7f2e480a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 shutdown_connections 2024-09-17T15:36:19.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.577+0000 7f2e4df27640 1 -- 172.21.15.86:0/1166109782 wait complete. 2024-09-17T15:36:19.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:19.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:36:19.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:36:19.675 INFO:tasks.rados.rados.0.smithi203.stdout:1272: done (6 left) 2024-09-17T15:36:19.675 INFO:tasks.rados.rados.0.smithi203.stdout:1274: done (5 left) 2024-09-17T15:36:19.675 INFO:tasks.rados.rados.0.smithi203.stdout:1276: done (4 left) 2024-09-17T15:36:19.675 INFO:tasks.rados.rados.0.smithi203.stdout:1277: done (3 left) 2024-09-17T15:36:19.675 INFO:tasks.rados.rados.0.smithi203.stdout:1278: done (2 left) 2024-09-17T15:36:19.676 INFO:tasks.rados.rados.0.smithi203.stdout:1279: done (1 left) 2024-09-17T15:36:19.676 INFO:tasks.rados.rados.0.smithi203.stdout:1280: done (0 left) 2024-09-17T15:36:19.676 INFO:tasks.rados.rados.0.smithi203.stdout:1281: snap_remove snap 129 2024-09-17T15:36:19.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.737+0000 7f5d8dced640 10 monclient: get_monmap_and_config 2024-09-17T15:36:19.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.737+0000 7f5d8dced640 10 monclient: build_initial_monmap 2024-09-17T15:36:19.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient: monmap: 2024-09-17T15:36:19.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:19.744034+0000 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:19.744034+0000 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:19.746 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:19.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 Processor -- start 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 -- start start 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient: init 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d880a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d880a5350 0x7f5d880a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-a con 0x7f5d880a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5d8813d180 0x7f5d8813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-c con 0x7f5d8813d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d880a68d0 con 0x7f5d880a4f70 2024-09-17T15:36:19.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5d880a64a0 con 0x7f5d880a5350 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5d8809c230 con 0x7f5d8813d180 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d8dced640 10 monclient(hunting): authenticate will time out at 1512.709617s 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d880a5350 0x7f5d880a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d880a5350 0x7f5d880a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37336/0 (socket says 172.21.15.86:37336) 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 -- 172.21.15.86:0/2962171486 learned_addr learned my addr 172.21.15.86:0/2962171486 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request con 0x7f5d880a5350 auth_method 0 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_done global_id 94221 payload 274 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d87fff640 1 --2- 172.21.15.86:0/2962171486 >> v2:172.21.15.203:3300/0 conn(0x7f5d8813d180 0x7f5d8813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 -- 172.21.15.86:0/2962171486 >> v2:172.21.15.203:3300/0 conn(0x7f5d8813d180 msgr2=0x7f5d8813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/2962171486 >> v2:172.21.15.203:3300/0 conn(0x7f5d8813d180 0x7f5d8813f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 -- 172.21.15.86:0/2962171486 >> v1:172.21.15.203:6789/0 conn(0x7f5d880a4f70 legacy=0x7f5d8809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 monclient: found mon.noname-a 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.741+0000 7f5d86ffd640 1 -- 172.21.15.86:0/2962171486 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d8809c510 con 0x7f5d880a5350 2024-09-17T15:36:19.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d86ffd640 1 -- 172.21.15.86:0/2962171486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d880a5350 msgr2=0x7f5d880a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/2962171486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d880a5350 0x7f5d880a5750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d880a6380 tx=0x7f5d7801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6817cb3444152e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 1 -- 172.21.15.86:0/2962171486 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d780b1070 con 0x7f5d880a5350 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: dump: 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: _finish_auth 0 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: _check_auth_tickets 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 1 -- 172.21.15.86:0/2962171486 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d780ac620 con 0x7f5d880a5350 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 1 -- 172.21.15.86:0/2962171486 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d780ac9f0 con 0x7f5d880a5350 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 5 monclient: authenticate success, global_id 94221 2024-09-17T15:36:19.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d867fc640 10 monclient: dump: 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.760 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 4 set_mon_vals no callback set 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 10 monclient: shutdown 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 -- 172.21.15.86:0/2962171486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d880a5350 msgr2=0x7f5d880a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 --2- 172.21.15.86:0/2962171486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d880a5350 0x7f5d880a5750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d880a6380 tx=0x7f5d7801c3e0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 -- 172.21.15.86:0/2962171486 shutdown_connections 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 --2- 172.21.15.86:0/2962171486 >> v2:172.21.15.203:3300/0 conn(0x7f5d8813d180 0x7f5d8813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 --2- 172.21.15.86:0/2962171486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d880a5350 0x7f5d880a5750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 -- 172.21.15.86:0/2962171486 >> 172.21.15.86:0/2962171486 conn(0x7f5d880a7240 msgr2=0x7f5d880a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:19.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 -- 172.21.15.86:0/2962171486 shutdown_connections 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.745+0000 7f5d8dced640 1 -- 172.21.15.86:0/2962171486 wait complete. 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient: build_initial_monmap 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient: monmap: 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:19.762 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 Processor -- start 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 -- start start 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient: init 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-c con 0x7f5d880a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d880a5350 0x7f5d88132560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-b con 0x7f5d880a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d8813d180 0x7f5d881cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): picked mon.noname-a con 0x7f5d8813d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5d88142660 con 0x7f5d8813d180 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d881424e0 con 0x7f5d880a5350 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5d881427e0 con 0x7f5d880a4f70 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d8dced640 10 monclient(hunting): authenticate will time out at 1512.716985s 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d8813d180 0x7f5d881cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51338/0 (socket says 172.21.15.86:51338) 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 -- 172.21.15.86:0/1068364196 learned_addr learned my addr 172.21.15.86:0/1068364196 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): get_auth_request con 0x7f5d880a4f70 auth_method 0 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): get_auth_request con 0x7f5d8813d180 auth_method 0 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d880a5350 0x7f5d88132560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request con 0x7f5d880a5350 auth_method 0 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient(hunting): handle_auth_done global_id 104105 payload 824 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d880a5350 msgr2=0x7f5d88132560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d880a5350 0x7f5d88132560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d8813d180 msgr2=0x7f5d881cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d8813d180 0x7f5d881cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient: _finish_hunting 0 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 monclient: found mon.noname-c 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:19.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d880a3b90 con 0x7f5d880a4f70 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d87fff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d8813d180 0x7f5d881cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 msgr2=0x7f5d88132020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.749+0000 7f5d877fe640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00c480 tx=0x7f5d7c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c012a40 con 0x7f5d880a4f70 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: dump: 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.767 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: _finish_auth 0 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 5 monclient: authenticate success, global_id 104105 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 monclient: _renew_subs 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d880a3e70 con 0x7f5d880a4f70 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d7c017070 con 0x7f5d880a4f70 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7c012e40 con 0x7f5d880a4f70 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 10 monclient: dump: 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 monclient: _renew_subs 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d85ffb640 4 set_mon_vals no callback set 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d880a1760 con 0x7f5d880a4f70 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 0x7f5d881d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d881d0e00 con 0x7f5d881d04e0 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5d7c01c460 con 0x7f5d880a4f70 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 0x7f5d881d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient: get_auth_request con 0x7f5d881d04e0 auth_method 0 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d7403bb60 0x7f5d7403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(298..298 src has 213..298) ==== 4815+0+0 (secure 0 0 0) 0x7f5d7c04fdd0 con 0x7f5d880a4f70 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d87fff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d7403bb60 0x7f5d7403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_done global_id 54448 payload 824 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d86ffd640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d880a5750 con 0x7f5d881d04e0 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.753+0000 7f5d87fff640 10 monclient: get_auth_request con 0x7f5d7403bb60 auth_method 0 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d86ffd640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 msgr2=0x7f5d881d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 0x7f5d881d08c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae000 tx=0x7f5d78019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d87fff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d7403bb60 0x7f5d7403e020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5d880a1df0 tx=0x7f5d800076f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d780bc070 con 0x7f5d881d04e0 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 10 monclient: dump: 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.774 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d780acd20 con 0x7f5d881d04e0 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 msgr2=0x7f5d881d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 0x7f5d881d08c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae000 tx=0x7f5d78019180 comp rx=0 tx=0).stop 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 msgr2=0x7f5d881d08c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.757+0000 7f5d6ffff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881d04e0 0x7f5d881d08c0 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5d780ae000 tx=0x7f5d78019180 comp rx=0 tx=0).stop 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.761+0000 7f5d8dced640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.761+0000 7f5d8dced640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 0x7f5d881366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.761+0000 7f5d8dced640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.761+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d880a2670 con 0x7f5d881362e0 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 0x7f5d881366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient: get_auth_request con 0x7f5d881362e0 auth_method 0 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 10 monclient(hunting): handle_auth_done global_id 54451 payload 824 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 1 -- 172.21.15.86:0/1068364196 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d880a5ff0 con 0x7f5d881362e0 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 msgr2=0x7f5d881366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.765+0000 7f5d86ffd640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 0x7f5d881366c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5d7801c420 tx=0x7f5d78008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d780b1040 con 0x7f5d881362e0 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 10 monclient: dump: 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f5d780cb430 con 0x7f5d881362e0 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 msgr2=0x7f5d881366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 0x7f5d881366c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5d7801c420 tx=0x7f5d78008040 comp rx=0 tx=0).stop 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 msgr2=0x7f5d881366c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d6ffff640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d881362e0 0x7f5d881366c0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5d7801c420 tx=0x7f5d78008040 comp rx=0 tx=0).stop 2024-09-17T15:36:19.778 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 255204, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:19.779 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.780 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.781 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.781 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:19.781 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:19.781 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.781 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:19.782 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:19.783 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:19.784 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:19.785 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:19.786 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.788 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.789 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:19.790 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:19.793 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:19.794 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:19.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.796 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.797 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:36:19.798 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:19.799 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:19.800 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:19.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d7403bb60 msgr2=0x7f5d7403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d7403bb60 0x7f5d7403e020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5d880a1df0 tx=0x7f5d800076f0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 10 monclient: shutdown 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 msgr2=0x7f5d88132020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00c480 tx=0x7f5d7c00c940 comp rx=0 tx=0).stop 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d87fff640 1 -- 172.21.15.86:0/1068364196 reap_dead start 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 shutdown_connections 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 --2- 172.21.15.86:0/1068364196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d880a4f70 0x7f5d88132020 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 >> 172.21.15.86:0/1068364196 conn(0x7f5d880a7240 msgr2=0x7f5d8813c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 shutdown_connections 2024-09-17T15:36:19.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.769+0000 7f5d8dced640 1 -- 172.21.15.86:0/1068364196 wait complete. 2024-09-17T15:36:19.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:19.803 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:36:19.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:36:19.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: get_monmap_and_config 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: build_initial_monmap 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: monmap: 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:19.921341+0000 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:19.921341+0000 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:19.919 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:19.920 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:19.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 Processor -- start 2024-09-17T15:36:19.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 -- start start 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: init 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 0x7f180c0689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-b con 0x7f180c074c30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f180c068f10 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 0x7f180c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-a con 0x7f180c0692f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180c09cb80 con 0x7f180c068f10 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f180c09d150 con 0x7f180c0692f0 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f180c09ce80 con 0x7f180c074c30 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f1811f76640 10 monclient(hunting): authenticate will time out at 1512.887648s 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 0x7f180c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 0x7f180c0689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 0x7f180c13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37354/0 (socket says 172.21.15.86:37354) 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 1 -- 172.21.15.86:0/4112799920 learned_addr learned my addr 172.21.15.86:0/4112799920 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): get_auth_request con 0x7f180c074c30 auth_method 0 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): get_auth_request con 0x7f180c0692f0 auth_method 0 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.917+0000 7f180bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 10 monclient(hunting): handle_auth_done global_id 104108 payload 274 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 1 -- 172.21.15.86:0/4112799920 >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 msgr2=0x7f180c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 1 --2- 172.21.15.86:0/4112799920 >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 0x7f180c13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 1 -- 172.21.15.86:0/4112799920 >> v1:172.21.15.203:6789/0 conn(0x7f180c068f10 legacy=0x7f180c138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 1 monclient: found mon.noname-b 2024-09-17T15:36:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:19.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:19.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.921+0000 7f180b7fe640 1 -- 172.21.15.86:0/4112799920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c09e410 con 0x7f180c074c30 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 -- 172.21.15.86:0/4112799920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 msgr2=0x7f180c0689d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 --2- 172.21.15.86:0/4112799920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 0x7f180c0689d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0097c0 tx=0x7f17fc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2f6b41fd9827b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 1 -- 172.21.15.86:0/4112799920 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0b0070 con 0x7f180c074c30 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: dump: 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: _finish_auth 0 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 1 -- 172.21.15.86:0/4112799920 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17fc0ab600 con 0x7f180c074c30 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 5 monclient: authenticate success, global_id 104108 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 1 -- 172.21.15.86:0/4112799920 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0ab9d0 con 0x7f180c074c30 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 4 set_mon_vals no callback set 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180affd640 10 monclient: dump: 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: shutdown 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- 172.21.15.86:0/4112799920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 msgr2=0x7f180c0689d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- 172.21.15.86:0/4112799920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 0x7f180c0689d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0097c0 tx=0x7f17fc01c3f0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- 172.21.15.86:0/4112799920 shutdown_connections 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- 172.21.15.86:0/4112799920 >> v2:172.21.15.86:3300/0 conn(0x7f180c0692f0 0x7f180c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- 172.21.15.86:0/4112799920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c074c30 0x7f180c0689d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- 172.21.15.86:0/4112799920 >> 172.21.15.86:0/4112799920 conn(0x7f180c0a6440 msgr2=0x7f180c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- 172.21.15.86:0/4112799920 shutdown_connections 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- 172.21.15.86:0/4112799920 wait complete. 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: build_initial_monmap 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: monmap: 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:19.934 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 Processor -- start 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- start start 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: init 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-c con 0x7f180c068f10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 0x7f180c13b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-a con 0x7f180c0692f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f180c074c30 0x7f180c1cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): picked mon.noname-b con 0x7f180c074c30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f180c13f300 con 0x7f180c0692f0 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f180c13f180 con 0x7f180c074c30 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f180c13f480 con 0x7f180c068f10 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1811f76640 10 monclient(hunting): authenticate will time out at 1512.894699s 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 0x7f180c13b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51368/0 (socket says 172.21.15.86:51368) 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 -- 172.21.15.86:0/311927753 learned_addr learned my addr 172.21.15.86:0/311927753 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:19.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): get_auth_request con 0x7f180c068f10 auth_method 0 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): get_auth_request con 0x7f180c0692f0 auth_method 0 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:19.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient(hunting): handle_auth_done global_id 94230 payload 824 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180bfff640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f180c074c30 0x7f180c1cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 -- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 msgr2=0x7f180c125aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 -- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f180c074c30 msgr2=0x7f180c1cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f180c074c30 0x7f180c1cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 monclient: found mon.noname-a 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f1802ffd640 1 -- 172.21.15.86:0/311927753 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c09d460 con 0x7f180c0692f0 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.925+0000 7f180b7fe640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:19.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1802ffd640 1 -- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 msgr2=0x7f180c13b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1802ffd640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 0x7f180c13b900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c3e0 tx=0x7f17f400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f40119a0 con 0x7f180c0692f0 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: dump: 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.939 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: _finish_auth 0 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 5 monclient: authenticate success, global_id 94230 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _renew_subs 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f4017070 con 0x7f180c0692f0 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180c123e10 con 0x7f180c0692f0 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f4011d80 con 0x7f180c0692f0 2024-09-17T15:36:19.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 10 monclient: dump: 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _renew_subs 2024-09-17T15:36:19.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f180a7fc640 4 set_mon_vals no callback set 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180c1272a0 con 0x7f180c0692f0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f180c0689d0 con 0x7f180c0692f0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f17f401c460 con 0x7f180c0692f0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 0x7f17ec03e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(298..298 src has 213..298) ==== 4815+0+0 (secure 0 0 0) 0x7f17f404fca0 con 0x7f180c0692f0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f180b7fe640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 0x7f17ec03e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:19.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.929+0000 7f180b7fe640 10 monclient: get_auth_request con 0x7f17ec03bc10 auth_method 0 2024-09-17T15:36:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.933+0000 7f180b7fe640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 0x7f17ec03e0d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17fc01c430 tx=0x7f17fc0a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.933+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f17f4010cf0 con 0x7f180c0692f0 2024-09-17T15:36:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.933+0000 7f1808ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:19.933+0000 7f1808ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.169+0000 7f1811f76640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:36:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.169+0000 7f1811f76640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:36:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.169+0000 7f1811f76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:20.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.169+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f180c06d4f0 con 0x7f180c0692f0 2024-09-17T15:36:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1808ff9640 1 -- 172.21.15.86:0/311927753 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f17f4012420 con 0x7f180c0692f0 2024-09-17T15:36:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1808ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:36:20.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1808ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 msgr2=0x7f17ec03e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 0x7f17ec03e0d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17fc01c430 tx=0x7f17fc0a6000 comp rx=0 tx=0).stop 2024-09-17T15:36:20.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 10 monclient: shutdown 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 msgr2=0x7f180c13b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.173+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 0x7f180c13b900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f17f400c3e0 tx=0x7f17f400c8a0 comp rx=0 tx=0).stop 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 shutdown_connections 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f17ec03bc10 0x7f17ec03e0d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f180c074c30 0x7f180c1cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f180c0692f0 0x7f180c13b900 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 --2- 172.21.15.86:0/311927753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f180c068f10 0x7f180c125aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 >> 172.21.15.86:0/311927753 conn(0x7f180c0a6440 msgr2=0x7f180c09c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 shutdown_connections 2024-09-17T15:36:20.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:20.177+0000 7f1811f76640 1 -- 172.21.15.86:0/311927753 wait complete. 2024-09-17T15:36:20.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=1228945672,config=3541143443,health=2925548024,kv=3591778543,logm=365442547}) 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=11,kv=10,logm=60} crc {auth=1369086949,config=3541143443,health=1676433854,kv=2898202873,logm=3520641142}) 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=1228945672,config=3541143443,health=2925548024,kv=3591778543,logm=365442547}) 2024-09-17T15:36:20.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=38496017,config=3541143443,health=2925548024,kv=3591778543,logm=3678499650}) 2024-09-17T15:36:20.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.201+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581911213}) 2024-09-17T15:36:20.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872984181}) 2024-09-17T15:36:20.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581911213}) 2024-09-17T15:36:20.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601756680}) 2024-09-17T15:36:20.207 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292486255}) 2024-09-17T15:36:20.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725032351}) 2024-09-17T15:36:20.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292486255}) 2024-09-17T15:36:20.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.205+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3937779717}) 2024-09-17T15:36:20.209 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603247452}) 2024-09-17T15:36:20.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125134999}) 2024-09-17T15:36:20.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603247452}) 2024-09-17T15:36:20.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717323237}) 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058086981}) 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650004679}) 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058086981}) 2024-09-17T15:36:20.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.209+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1524666886}) 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431790199}) 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3487933296}) 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431790199}) 2024-09-17T15:36:20.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027256920}) 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778758711}) 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3971982308}) 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778758711}) 2024-09-17T15:36:20.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.213+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1930018522}) 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415793114}) 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2724334913}) 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415793114}) 2024-09-17T15:36:20.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953638007}) 2024-09-17T15:36:20.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476248396}) 2024-09-17T15:36:20.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792738539}) 2024-09-17T15:36:20.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476248396}) 2024-09-17T15:36:20.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.217+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4221891333}) 2024-09-17T15:36:20.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872861326}) 2024-09-17T15:36:20.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2464519623}) 2024-09-17T15:36:20.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872861326}) 2024-09-17T15:36:20.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835271363}) 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293503117}) 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=403365478}) 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293503117}) 2024-09-17T15:36:20.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.221+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645529505}) 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757801552}) 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555382336}) 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=757801552}) 2024-09-17T15:36:20.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005263993}) 2024-09-17T15:36:20.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336084084}) 2024-09-17T15:36:20.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133330868}) 2024-09-17T15:36:20.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.225+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336084084}) 2024-09-17T15:36:20.230 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126954990}) 2024-09-17T15:36:20.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623759056}) 2024-09-17T15:36:20.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921189228}) 2024-09-17T15:36:20.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=623759056}) 2024-09-17T15:36:20.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.229+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298133065}) 2024-09-17T15:36:20.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953084829}) 2024-09-17T15:36:20.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517860119}) 2024-09-17T15:36:20.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953084829}) 2024-09-17T15:36:20.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1342109116}) 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021187679}) 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708827407}) 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021187679}) 2024-09-17T15:36:20.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.233+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738971448}) 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988995654}) 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212210734}) 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988995654}) 2024-09-17T15:36:20.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.237+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2349164353}) 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021918345}) 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=644351411}) 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1021918345}) 2024-09-17T15:36:20.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.677+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480857311}) 2024-09-17T15:36:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116332751}) 2024-09-17T15:36:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333157388}) 2024-09-17T15:36:20.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116332751}) 2024-09-17T15:36:20.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.681+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=311018178}) 2024-09-17T15:36:20.699 INFO:tasks.rados.rados.0.smithi203.stdout:1281: done (0 left) 2024-09-17T15:36:20.699 INFO:tasks.rados.rados.0.smithi203.stdout:1282: read oid 30 snap 139 2024-09-17T15:36:20.699 INFO:tasks.rados.rados.0.smithi203.stdout:1282: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:36:20.699 INFO:tasks.rados.rados.0.smithi203.stdout:1283: write oid 28 current snap is 141 2024-09-17T15:36:20.700 INFO:tasks.rados.rados.0.smithi203.stdout:1283: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-17T15:36:20.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507997887}) 2024-09-17T15:36:20.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=300354064}) 2024-09-17T15:36:20.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507997887}) 2024-09-17T15:36:20.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.697+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2667162770}) 2024-09-17T15:36:20.713 INFO:tasks.rados.rados.0.smithi203.stdout:1283: writing smithi20328086-28 from 632062 to 1195629 tid 1 2024-09-17T15:36:20.728 INFO:tasks.rados.rados.0.smithi203.stdout:1283: writing smithi20328086-28 from 1885903 to 2413720 tid 2 2024-09-17T15:36:20.740 INFO:tasks.rados.rados.0.smithi203.stdout:1283: writing smithi20328086-28 from 2830644 to 3551402 tid 3 2024-09-17T15:36:20.741 INFO:tasks.rados.rados.0.smithi203.stdout:1284: snap_remove snap 140 2024-09-17T15:36:20.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425974640}) 2024-09-17T15:36:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248103432}) 2024-09-17T15:36:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425974640}) 2024-09-17T15:36:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.745+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421158741}) 2024-09-17T15:36:20.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2531061799}) 2024-09-17T15:36:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496535684}) 2024-09-17T15:36:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2531061799}) 2024-09-17T15:36:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=998047874}) 2024-09-17T15:36:20.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753034059}) 2024-09-17T15:36:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=846261266}) 2024-09-17T15:36:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753034059}) 2024-09-17T15:36:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.749+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508758601}) 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030536393}) 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401897972}) 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030536393}) 2024-09-17T15:36:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669870846}) 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=6,osd_snap=50} crc {logm=2633362676,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3418781978,mon_config_key=578014828,monmap=4191199434,nvmeofgw=3140514009,osd_metadata=4043941930,osd_snap=1687972062}) 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=49} crc {logm=2633362676,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1318100172,mon_config_key=1453524256,monmap=3140514009,nvmeofgw=598337686,osd_metadata=2564601247,osd_pg_creating=3798571723,osd_snap=3406705579}) 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=6,osd_snap=50} crc {logm=2633362676,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3418781978,mon_config_key=578014828,monmap=4191199434,nvmeofgw=3140514009,osd_metadata=4043941930,osd_snap=1687972062}) 2024-09-17T15:36:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.753+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=4,osd_pg_creating=1,osd_snap=50} crc {logm=2633362676,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2568572441,mon_config_key=639531838,monmap=4191199434,nvmeofgw=3140514009,osd_metadata=2356047796,osd_pg_creating=3798571723,osd_snap=1923195661}) 2024-09-17T15:36:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=127657245,osdmap=1924133787}) 2024-09-17T15:36:20.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=127657245,osdmap=1574255846}) 2024-09-17T15:36:20.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=127657245,osdmap=1924133787}) 2024-09-17T15:36:20.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.757+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=127657245,osdmap=3245333900}) 2024-09-17T15:36:20.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=1090152989}) 2024-09-17T15:36:20.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=3690023189}) 2024-09-17T15:36:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:36:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=1090152989}) 2024-09-17T15:36:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:36:20.769+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=44} crc {osdmap=3433635611}) 2024-09-17T15:36:21.194 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:36:21.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: get_monmap_and_config 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: monmap: 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:21.325438+0000 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:21.325438+0000 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:21.323 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:21.324 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:21.324 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:21.324 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:21.324 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 Processor -- start 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 -- start start 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: init 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 0x7ffb2c09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb2c068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffb2c139f00 0x7ffb2c13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb2c09d770 con 0x7ffb2c068db0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ffb2c09ed00 con 0x7ffb2c139f00 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ffb2c09efe0 con 0x7ffb2c0689d0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb30a0d640 10 monclient(hunting): authenticate will time out at 1514.291006s 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb29d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ffb2c068db0 0x7ffb2c069190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59484/0 (socket says 172.21.15.86:59484) 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 0x7ffb2c09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb29d9b640 1 -- 172.21.15.86:0/4281308153 learned_addr learned my addr 172.21.15.86:0/4281308153 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2ad9d640 1 --2- 172.21.15.86:0/4281308153 >> v2:172.21.15.86:3300/0 conn(0x7ffb2c139f00 0x7ffb2c13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): get_auth_request con 0x7ffb2c0689d0 auth_method 0 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2ad9d640 10 monclient(hunting): get_auth_request con 0x7ffb2c139f00 auth_method 0 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient(hunting): handle_auth_done global_id 104114 payload 274 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 -- 172.21.15.86:0/4281308153 >> v2:172.21.15.86:3300/0 conn(0x7ffb2c139f00 msgr2=0x7ffb2c13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 --2- 172.21.15.86:0/4281308153 >> v2:172.21.15.86:3300/0 conn(0x7ffb2c139f00 0x7ffb2c13c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 -- 172.21.15.86:0/4281308153 >> v1:172.21.15.86:6789/0 conn(0x7ffb2c068db0 legacy=0x7ffb2c069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 monclient: found mon.noname-b 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.321+0000 7ffb2a59c640 1 -- 172.21.15.86:0/4281308153 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2c0a04e0 con 0x7ffb2c0689d0 2024-09-17T15:36:21.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2a59c640 1 -- 172.21.15.86:0/4281308153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 msgr2=0x7ffb2c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2a59c640 1 --2- 172.21.15.86:0/4281308153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 0x7ffb2c09ffa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0a0770 tx=0x7ffb1c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d12dd686efbc3f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 1 -- 172.21.15.86:0/4281308153 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c0b0070 con 0x7ffb2c0689d0 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: dump: 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.332 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 1 -- 172.21.15.86:0/4281308153 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb1c0ab600 con 0x7ffb2c0689d0 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 1 -- 172.21.15.86:0/4281308153 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c0ab9d0 con 0x7ffb2c0689d0 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 5 monclient: authenticate success, global_id 104114 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 4 set_mon_vals no callback set 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb2959a640 10 monclient: dump: 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.334 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 10 monclient: shutdown 2024-09-17T15:36:21.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 -- 172.21.15.86:0/4281308153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 msgr2=0x7ffb2c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/4281308153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 0x7ffb2c09ffa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c0a0770 tx=0x7ffb1c01c3f0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 -- 172.21.15.86:0/4281308153 shutdown_connections 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/4281308153 >> v2:172.21.15.86:3300/0 conn(0x7ffb2c139f00 0x7ffb2c13c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/4281308153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c0689d0 0x7ffb2c09ffa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 -- 172.21.15.86:0/4281308153 >> 172.21.15.86:0/4281308153 conn(0x7ffb2c0a6440 msgr2=0x7ffb2c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 -- 172.21.15.86:0/4281308153 shutdown_connections 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.329+0000 7ffb30a0d640 1 -- 172.21.15.86:0/4281308153 wait complete. 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient: monmap: 2024-09-17T15:36:21.340 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 Processor -- start 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 -- start start 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient: init 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb2c068db0 0x7ffb2c073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb2c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb2c139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 0x7ffb2c1d4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb2c073d10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c09be00 con 0x7ffb2c073d10 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c09bc80 con 0x7ffb2c068db0 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ffb2c09bf80 con 0x7ffb2c139f00 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb30a0d640 10 monclient(hunting): authenticate will time out at 1514.301298s 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51390/0 (socket says 172.21.15.86:51390) 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 -- 172.21.15.86:0/588630556 learned_addr learned my addr 172.21.15.86:0/588630556 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): get_auth_request con 0x7ffb2c139f00 auth_method 0 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 0x7ffb2c1d4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): get_auth_request con 0x7ffb2c073d10 auth_method 0 2024-09-17T15:36:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2a59c640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb2c068db0 0x7ffb2c073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient(hunting): handle_auth_done global_id 104117 payload 824 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 -- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb2c068db0 msgr2=0x7ffb2c073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb2c068db0 0x7ffb2c073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 -- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 msgr2=0x7ffb2c1d4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 0x7ffb2c1d4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 monclient: found mon.noname-c 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb29d9b640 1 -- 172.21.15.86:0/588630556 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2c11f010 con 0x7ffb2c139f00 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.333+0000 7ffb2ad9d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 0x7ffb2c1d4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb29d9b640 1 -- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 msgr2=0x7ffb2c0737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb29d9b640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c4a0 tx=0x7ffb2000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20017070 con 0x7ffb2c139f00 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: dump: 2024-09-17T15:36:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb20007070 con 0x7ffb2c139f00 2024-09-17T15:36:21.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 5 monclient: authenticate success, global_id 104117 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _renew_subs 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2c0a27d0 con 0x7ffb2c139f00 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20007460 con 0x7ffb2c139f00 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 10 monclient: dump: 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.347 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb28d99640 4 set_mon_vals no callback set 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _renew_subs 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2c11ccb0 con 0x7ffb2c139f00 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb2c09ffa0 con 0x7ffb2c139f00 2024-09-17T15:36:21.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ffb2001a070 con 0x7ffb2c139f00 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 0x7ffb0c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(299..299 src has 213..299) ==== 4799+0+0 (secure 0 0 0) 0x7ffb2004f450 con 0x7ffb2c139f00 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb2a59c640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 0x7ffb0c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.337+0000 7ffb2a59c640 10 monclient: get_auth_request con 0x7ffb0c03bbb0 auth_method 0 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.341+0000 7ffb2a59c640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 0x7ffb0c03e070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c06f6b0 tx=0x7ffb1c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.341+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffb20007640 con 0x7ffb2c139f00 2024-09-17T15:36:21.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.341+0000 7ffb1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.341+0000 7ffb1affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:21.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb30a0d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:21.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb30a0d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:21.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb30a0d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffb2c06d4f0 con 0x7ffb2c139f00 2024-09-17T15:36:21.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb1affd640 1 -- 172.21.15.86:0/588630556 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ffb2004f0d0 con 0x7ffb2c139f00 2024-09-17T15:36:21.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:21.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.557+0000 7ffb1affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:21.560 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:21.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 msgr2=0x7ffb0c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 0x7ffb0c03e070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c06f6b0 tx=0x7ffb1c008040 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 10 monclient: shutdown 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 msgr2=0x7ffb2c0737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000c4a0 tx=0x7ffb2000c960 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 shutdown_connections 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffb0c03bbb0 0x7ffb0c03e070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb2c073d10 0x7ffb2c1d4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb2c139f00 0x7ffb2c0737d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 --2- 172.21.15.86:0/588630556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb2c068db0 0x7ffb2c073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 >> 172.21.15.86:0/588630556 conn(0x7ffb2c0a6440 msgr2=0x7ffb2c13bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 shutdown_connections 2024-09-17T15:36:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.561+0000 7ffb30a0d640 1 -- 172.21.15.86:0/588630556 wait complete. 2024-09-17T15:36:21.577 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:36:21.577 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:36:21.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:36:21.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:36:21.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: get_monmap_and_config 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: monmap: 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:21.685450+0000 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:21.685450+0000 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:21.683 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 Processor -- start 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 -- start start 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: init 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97800a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 0x7f97800a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): picked mon.noname-b con 0x7f97800a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): picked mon.noname-a con 0x7f978013d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97800a68d0 con 0x7f97800a4f70 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f97800a64a0 con 0x7f978013d180 2024-09-17T15:36:21.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f978009c230 con 0x7f97800a5350 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9784859640 10 monclient(hunting): authenticate will time out at 1514.650966s 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 0x7f97800a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37412/0 (socket says 172.21.15.86:37412) 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): get_auth_request con 0x7f97800a5350 auth_method 0 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 1 -- 172.21.15.86:0/3685221993 learned_addr learned my addr 172.21.15.86:0/3685221993 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): get_auth_request con 0x7f978013d180 auth_method 0 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient(hunting): handle_auth_done global_id 104120 payload 274 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 -- 172.21.15.86:0/3685221993 >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 msgr2=0x7f978013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 --2- 172.21.15.86:0/3685221993 >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 -- 172.21.15.86:0/3685221993 >> v1:172.21.15.203:6789/0 conn(0x7f97800a4f70 legacy=0x7f978009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 monclient: found mon.noname-b 2024-09-17T15:36:21.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f9777fff640 1 -- 172.21.15.86:0/3685221993 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978009c510 con 0x7f97800a5350 2024-09-17T15:36:21.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.681+0000 7f977ed9d640 1 --2- 172.21.15.86:0/3685221993 >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:21.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.685+0000 7f9777fff640 1 -- 172.21.15.86:0/3685221993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 msgr2=0x7f97800a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.685+0000 7f9777fff640 1 --2- 172.21.15.86:0/3685221993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 0x7f97800a5750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97700097c0 tx=0x7f977001c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba1b62c5cf81e6b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.685+0000 7f977dd9b640 1 -- 172.21.15.86:0/3685221993 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700b0070 con 0x7f97800a5350 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.685+0000 7f977dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: dump: 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 5 monclient: authenticate success, global_id 104120 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 1 -- 172.21.15.86:0/3685221993 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97700ab600 con 0x7f97800a5350 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 4 set_mon_vals no callback set 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 1 -- 172.21.15.86:0/3685221993 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700ab9d0 con 0x7f97800a5350 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977dd9b640 10 monclient: dump: 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: shutdown 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- 172.21.15.86:0/3685221993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 msgr2=0x7f97800a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- 172.21.15.86:0/3685221993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 0x7f97800a5750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97700097c0 tx=0x7f977001c300 comp rx=0 tx=0).stop 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- 172.21.15.86:0/3685221993 shutdown_connections 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- 172.21.15.86:0/3685221993 >> v2:172.21.15.86:3300/0 conn(0x7f978013d180 0x7f978013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- 172.21.15.86:0/3685221993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97800a5350 0x7f97800a5750 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- 172.21.15.86:0/3685221993 >> 172.21.15.86:0/3685221993 conn(0x7f97800a7240 msgr2=0x7f97800a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- 172.21.15.86:0/3685221993 shutdown_connections 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- 172.21.15.86:0/3685221993 wait complete. 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: monmap: 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 Processor -- start 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- start start 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: init 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97800a4f70 0x7f97801cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): picked mon.noname-b con 0x7f97800a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 0x7f97801d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): picked mon.noname-a con 0x7f97800a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f978013d180 0x7f97801d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): picked mon.noname-c con 0x7f978013d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9780142710 con 0x7f97800a5350 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9780142590 con 0x7f97800a4f70 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9780142890 con 0x7f978013d180 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9784859640 10 monclient(hunting): authenticate will time out at 1514.657973s 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 0x7f97801d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f978013d180 0x7f97801d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f978013d180 0x7f97801d0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51398/0 (socket says 172.21.15.86:51398) 2024-09-17T15:36:21.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 1 -- 172.21.15.86:0/625566993 learned_addr learned my addr 172.21.15.86:0/625566993 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): get_auth_request con 0x7f97800a5350 auth_method 0 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): get_auth_request con 0x7f978013d180 auth_method 0 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977e59c640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97800a4f70 0x7f97801cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977e59c640 10 monclient(hunting): get_auth_request con 0x7f97800a4f70 auth_method 0 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f977e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient(hunting): handle_auth_done global_id 94242 payload 824 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f978013d180 msgr2=0x7f97801d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f978013d180 0x7f97801d0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97800a4f70 msgr2=0x7f97801cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97800a4f70 0x7f97801cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 monclient: found mon.noname-a 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.689+0000 7f9777fff640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978013ae60 con 0x7f97800a5350 2024-09-17T15:36:21.701 INFO:tasks.rados.rados.0.smithi203.stdout:1284: done (2 left) 2024-09-17T15:36:21.701 INFO:tasks.rados.rados.0.smithi203.stdout:1285: snap_remove snap 136 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9777fff640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 msgr2=0x7f97801d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9777fff640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 0x7f97801d0240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97700ab000 tx=0x7f9770019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700b8020 con 0x7f97800a5350 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: dump: 2024-09-17T15:36:21.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97700b4620 con 0x7f97800a5350 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97700b0040 con 0x7f97800a5350 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 10 monclient: dump: 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 5 monclient: authenticate success, global_id 94242 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 monclient: _renew_subs 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f977d59a640 4 set_mon_vals no callback set 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978013b140 con 0x7f97800a5350 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 monclient: _renew_subs 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978013ab80 con 0x7f97800a5350 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f97700bb020 con 0x7f97800a5350 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 0x7f97801350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.693+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f978013da90 con 0x7f9780134d00 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f976003bbb0 0x7f976003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 0x7f97801350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient: get_auth_request con 0x7f9780134d00 auth_method 0 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f97700e9a60 con 0x7f97800a5350 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 10 monclient(hunting): handle_auth_done global_id 94245 payload 824 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97800a5750 con 0x7f9780134d00 2024-09-17T15:36:21.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977ed9d640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f976003bbb0 0x7f976003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977ed9d640 10 monclient: get_auth_request con 0x7f976003bbb0 auth_method 0 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 msgr2=0x7f97801350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f977e59c640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 0x7f97801350e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f976800e450 tx=0x7f976800e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9768007450 con 0x7f9780134d00 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 10 monclient: dump: 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9768007450 con 0x7f9780134d00 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 msgr2=0x7f97801350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 0x7f97801350e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f976800e450 tx=0x7f976800e910 comp rx=0 tx=0).stop 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 msgr2=0x7f97801350e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.697+0000 7f9776ffd640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780134d00 0x7f97801350e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.701+0000 7f977ed9d640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f976003bbb0 0x7f976003e070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9778009c90 tx=0x7f977800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9784859640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9784859640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 0x7f97800a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9784859640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f97800a5ff0 con 0x7f9780135620 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 0x7f97800a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient: get_auth_request con 0x7f9780135620 auth_method 0 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 10 monclient(hunting): handle_auth_done global_id 94248 payload 824 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 1 -- 172.21.15.86:0/625566993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f978013c040 con 0x7f9780135620 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 msgr2=0x7f97800a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f977e59c640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 0x7f97800a1970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f976800e450 tx=0x7f9768003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f976801eb70 con 0x7f9780135620 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 10 monclient: dump: 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f976801eb70 con 0x7f9780135620 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 msgr2=0x7f97800a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 0x7f97800a1970 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f976800e450 tx=0x7f9768003770 comp rx=0 tx=0).stop 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 msgr2=0x7f97800a1970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:21.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.705+0000 7f9776ffd640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9780135620 0x7f97800a1970 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f976800e450 tx=0x7f9768003770 comp rx=0 tx=0).stop 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 57059, 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:21.715 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 6, 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.716 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.717 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:21.718 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.719 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:21.720 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:21.722 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:21.724 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:21.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.726 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:21.727 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:21.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f976003bbb0 msgr2=0x7f976003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f976003bbb0 0x7f976003e070 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9778009c90 tx=0x7f977800b040 comp rx=0 tx=0).stop 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 10 monclient: shutdown 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 msgr2=0x7f97801d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 --2- 172.21.15.86:0/625566993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800a5350 0x7f97801d0240 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f97700ab000 tx=0x7f9770019100 comp rx=0 tx=0).stop 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f977ed9d640 1 -- 172.21.15.86:0/625566993 reap_dead start 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f977ed9d640 1 -- 172.21.15.86:0/625566993 reap_dead start 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 shutdown_connections 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 >> 172.21.15.86:0/625566993 conn(0x7f97800a7240 msgr2=0x7f978013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 shutdown_connections 2024-09-17T15:36:21.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.709+0000 7f9784859640 1 -- 172.21.15.86:0/625566993 wait complete. 2024-09-17T15:36:21.730 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:21.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:36:21.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: get_monmap_and_config 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: monmap: 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:21.845664+0000 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:21.845664+0000 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:21.844 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:21.844 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:21.844 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 Processor -- start 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 -- start start 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: init 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94b40a4900 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94b406d280 0x7f94b413d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-a con 0x7f94b406d280 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-b con 0x7f94b406d660 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b409bf70 con 0x7f94b40a4900 2024-09-17T15:36:21.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f94b40a7180 con 0x7f94b406d280 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94b40a81b0 con 0x7f94b406d660 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b9050640 10 monclient(hunting): authenticate will time out at 1514.811781s 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94b406d280 0x7f94b413d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51404/0 (socket says 172.21.15.86:51404) 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 1 -- 172.21.15.86:0/3310964845 learned_addr learned my addr 172.21.15.86:0/3310964845 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 10 monclient(hunting): get_auth_request con 0x7f94b406d660 auth_method 0 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.841+0000 7f94b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b259c640 10 monclient(hunting): get_auth_request con 0x7f94b406d280 auth_method 0 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient(hunting): handle_auth_done global_id 104126 payload 274 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 1 -- 172.21.15.86:0/3310964845 >> v2:172.21.15.86:3300/0 conn(0x7f94b406d280 msgr2=0x7f94b413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 1 --2- 172.21.15.86:0/3310964845 >> v2:172.21.15.86:3300/0 conn(0x7f94b406d280 0x7f94b413d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 1 -- 172.21.15.86:0/3310964845 >> v1:172.21.15.86:6789/0 conn(0x7f94b40a4900 legacy=0x7f94b406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 1 monclient: found mon.noname-b 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:21.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.845+0000 7f94b359e640 1 -- 172.21.15.86:0/3310964845 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b409b3f0 con 0x7f94b406d660 2024-09-17T15:36:21.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 1 -- 172.21.15.86:0/3310964845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 msgr2=0x7f94b413fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 1 --2- 172.21.15.86:0/3310964845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009e50 tx=0x7f94ac01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6805dd75122dd20 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 1 -- 172.21.15.86:0/3310964845 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0a5530 con 0x7f94b406d660 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: dump: 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.850 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 1 -- 172.21.15.86:0/3310964845 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ac0a56d0 con 0x7f94b406d660 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 1 -- 172.21.15.86:0/3310964845 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0a5aa0 con 0x7f94b406d660 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 5 monclient: authenticate success, global_id 104126 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 4 set_mon_vals no callback set 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b1d9b640 10 monclient: dump: 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: shutdown 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- 172.21.15.86:0/3310964845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 msgr2=0x7f94b413fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- 172.21.15.86:0/3310964845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009e50 tx=0x7f94ac01c8f0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- 172.21.15.86:0/3310964845 shutdown_connections 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- 172.21.15.86:0/3310964845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b406d660 0x7f94b413fa10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- 172.21.15.86:0/3310964845 >> v2:172.21.15.86:3300/0 conn(0x7f94b406d280 0x7f94b413d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- 172.21.15.86:0/3310964845 >> 172.21.15.86:0/3310964845 conn(0x7f94b40a3e10 msgr2=0x7f94b40a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- 172.21.15.86:0/3310964845 shutdown_connections 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- 172.21.15.86:0/3310964845 wait complete. 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: monmap: 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:21.858 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 Processor -- start 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- start start 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: init 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 0x7f94b41cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-a con 0x7f94b406d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94b406d660 0x7f94b41d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-b con 0x7f94b406d660 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b40a4900 0x7f94b41d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): picked mon.noname-c con 0x7f94b40a4900 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94b409c2c0 con 0x7f94b406d280 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94b406ed80 con 0x7f94b406d660 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94b40a4ce0 con 0x7f94b40a4900 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b9050640 10 monclient(hunting): authenticate will time out at 1514.818562s 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 0x7f94b41cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 0x7f94b41cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37472/0 (socket says 172.21.15.86:37472) 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b40a4900 0x7f94b41d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 -- 172.21.15.86:0/3098513202 learned_addr learned my addr 172.21.15.86:0/3098513202 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): get_auth_request con 0x7f94b406d280 auth_method 0 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): get_auth_request con 0x7f94b40a4900 auth_method 0 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_done global_id 94254 payload 824 2024-09-17T15:36:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b40a4900 msgr2=0x7f94b41d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b40a4900 0x7f94b41d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94b406d660 msgr2=0x7f94b41d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94b406d660 0x7f94b41d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b2d9d640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b413e0d0 con 0x7f94b406d280 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.849+0000 7f94b359e640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b40a4900 0x7f94b41d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b2d9d640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 msgr2=0x7f94b41cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b2d9d640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 0x7f94b41cfb40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94a400c450 tx=0x7f94a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a40119a0 con 0x7f94b406d280 2024-09-17T15:36:21.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: dump: 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:36:21.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 5 monclient: authenticate success, global_id 94254 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a4017070 con 0x7f94b406d280 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 monclient: _renew_subs 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a4011da0 con 0x7f94b406d280 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b413e3b0 con 0x7f94b406d280 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 10 monclient: dump: 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b159a640 4 set_mon_vals no callback set 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 monclient: _renew_subs 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b4074630 con 0x7f94b406d280 2024-09-17T15:36:21.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 0x7f94b4137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94b413ea60 con 0x7f94b413f3a0 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 0x7f94b4137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient: get_auth_request con 0x7f94b413f3a0 auth_method 0 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f94a401c460 con 0x7f94b406d280 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 10 monclient(hunting): handle_auth_done global_id 104132 payload 824 2024-09-17T15:36:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b259c640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94b406cb70 con 0x7f94b413f3a0 2024-09-17T15:36:21.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f94a404fe20 con 0x7f94b406d280 2024-09-17T15:36:21.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b359e640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.853+0000 7f94b359e640 10 monclient: get_auth_request con 0x7f94a003bbb0 auth_method 0 2024-09-17T15:36:21.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f94b259c640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 msgr2=0x7f94b4137c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f94b259c640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 0x7f94b4137c40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94a800d500 tx=0x7f94a800d9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8018030 con 0x7f94b413f3a0 2024-09-17T15:36:21.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 10 monclient: dump: 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f94a8004020 con 0x7f94b413f3a0 2024-09-17T15:36:21.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 msgr2=0x7f94b4137c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 0x7f94b4137c40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94a800d500 tx=0x7f94a800d9c0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 msgr2=0x7f94b4137c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f949b7fe640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b413f3a0 0x7f94b4137c40 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94a800d500 tx=0x7f94a800d9c0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.857+0000 7f94b359e640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0ab000 tx=0x7f94ac0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b9050640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b9050640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 0x7f94b402d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b9050640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94b413b510 con 0x7f94b402d450 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 0x7f94b402d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient: get_auth_request con 0x7f94b402d450 auth_method 0 2024-09-17T15:36:21.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 10 monclient(hunting): handle_auth_done global_id 104135 payload 824 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.861+0000 7f94b259c640 1 -- 172.21.15.86:0/3098513202 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94b406e5b0 con 0x7f94b402d450 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b259c640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 msgr2=0x7f94b402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b259c640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 0x7f94b402d860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f94a8018000 tx=0x7f94a80046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a801ea20 con 0x7f94b402d450 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:21.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 10 monclient: dump: 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f94a801ea20 con 0x7f94b402d450 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 msgr2=0x7f94b402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 0x7f94b402d860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f94a8018000 tx=0x7f94a80046d0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 msgr2=0x7f94b402d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f949b7fe640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94b402d450 0x7f94b402d860 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f94a8018000 tx=0x7f94a80046d0 comp rx=0 tx=0).stop 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:21.874 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12739, 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.875 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.876 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:21.877 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:21.878 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:21.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:21.880 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:21.883 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:21.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:21.885 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:21.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:21.886 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:21.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:21.886 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:21.886 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 msgr2=0x7f94a003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0ab000 tx=0x7f94ac0a7070 comp rx=0 tx=0).stop 2024-09-17T15:36:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 10 monclient: shutdown 2024-09-17T15:36:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 msgr2=0x7f94b41cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 --2- 172.21.15.86:0/3098513202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94b406d280 0x7f94b41cfb40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f94a400c450 tx=0x7f94a400c910 comp rx=0 tx=0).stop 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b359e640 1 -- 172.21.15.86:0/3098513202 reap_dead start 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b359e640 1 -- 172.21.15.86:0/3098513202 reap_dead start 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 shutdown_connections 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 >> 172.21.15.86:0/3098513202 conn(0x7f94b40a3e10 msgr2=0x7f94b41386e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 shutdown_connections 2024-09-17T15:36:21.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.865+0000 7f94b9050640 1 -- 172.21.15.86:0/3098513202 wait complete. 2024-09-17T15:36:21.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:21.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:36:21.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: get_monmap_and_config 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: build_initial_monmap 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: monmap: 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:22.000837+0000 2024-09-17T15:36:21.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:22.000837+0000 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:21.999 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 Processor -- start 2024-09-17T15:36:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 -- start start 2024-09-17T15:36:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: init 2024-09-17T15:36:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-b con 0x7f20f00a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 0x7f20f00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20f0137d30 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f009d430 con 0x7f20f0137d30 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f20f00a68d0 con 0x7f20f00a5350 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00a64a0 con 0x7f20f00a4f70 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f7c8c640 10 monclient(hunting): authenticate will time out at 1514.966254s 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 0x7f20f00a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 0x7f20f00a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37480/0 (socket says 172.21.15.86:37480) 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 1 -- 172.21.15.86:0/4220903666 learned_addr learned my addr 172.21.15.86:0/4220903666 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51442/0 (socket says 172.21.15.86:51442) 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): get_auth_request con 0x7f20f00a5350 auth_method 0 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f6229640 1 --1- 172.21.15.86:0/4220903666 >> v1:172.21.15.86:6789/0 conn(0x7f20f0137d30 0x7f20f01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59498/0 (socket says 172.21.15.86:59498) 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): get_auth_request con 0x7f20f00a4f70 auth_method 0 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient(hunting): handle_auth_done global_id 104138 payload 274 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 -- 172.21.15.86:0/4220903666 >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 msgr2=0x7f20f00a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 --2- 172.21.15.86:0/4220903666 >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 0x7f20f00a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 -- 172.21.15.86:0/4220903666 >> v1:172.21.15.86:6789/0 conn(0x7f20f0137d30 legacy=0x7f20f01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient: _finish_hunting 0 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 monclient: found mon.noname-b 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:21.997+0000 7f20f5a28640 1 -- 172.21.15.86:0/4220903666 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f009c230 con 0x7f20f00a4f70 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f5a28640 1 -- 172.21.15.86:0/4220903666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 msgr2=0x7f20f009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f5a28640 1 --2- 172.21.15.86:0/4220903666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20f0069c50 tx=0x7f20e801c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1190c178a6d8ea00 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 1 -- 172.21.15.86:0/4220903666 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e80b2070 con 0x7f20f00a4f70 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: dump: 2024-09-17T15:36:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: _finish_auth 0 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: _check_auth_tickets 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 1 -- 172.21.15.86:0/4220903666 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e80a57d0 con 0x7f20f00a4f70 2024-09-17T15:36:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 5 monclient: authenticate success, global_id 104138 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 4 set_mon_vals no callback set 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 1 -- 172.21.15.86:0/4220903666 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e80a5ba0 con 0x7f20f00a4f70 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f4a26640 10 monclient: dump: 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 10 monclient: shutdown 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 -- 172.21.15.86:0/4220903666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 msgr2=0x7f20f009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/4220903666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20f0069c50 tx=0x7f20e801c880 comp rx=0 tx=0).stop 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 -- 172.21.15.86:0/4220903666 shutdown_connections 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/4220903666 >> v2:172.21.15.86:3300/0 conn(0x7f20f00a5350 0x7f20f00a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/4220903666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f00a4f70 0x7f20f009cef0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 -- 172.21.15.86:0/4220903666 >> 172.21.15.86:0/4220903666 conn(0x7f20f00a7240 msgr2=0x7f20f00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 -- 172.21.15.86:0/4220903666 shutdown_connections 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.001+0000 7f20f7c8c640 1 -- 172.21.15.86:0/4220903666 wait complete. 2024-09-17T15:36:22.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient: build_initial_monmap 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient: monmap: 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 Processor -- start 2024-09-17T15:36:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 -- start start 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient: init 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 0x7f20f01d0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f20f00a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f00a5350 0x7f20f01d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-b con 0x7f20f00a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): picked mon.noname-c con 0x7f20f0137d30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f20f0142b40 con 0x7f20f00a4f70 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20f01429c0 con 0x7f20f00a5350 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f20f0142cc0 con 0x7f20f0137d30 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f7c8c640 10 monclient(hunting): authenticate will time out at 1514.973111s 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 0x7f20f01d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 0x7f20f01d0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37494/0 (socket says 172.21.15.86:37494) 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51446/0 (socket says 172.21.15.86:51446) 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 learned_addr learned my addr 172.21.15.86:0/1692961765 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): get_auth_request con 0x7f20f0137d30 auth_method 0 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): get_auth_request con 0x7f20f00a4f70 auth_method 0 2024-09-17T15:36:22.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient(hunting): handle_auth_done global_id 104141 payload 824 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5227640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f00a5350 0x7f20f01d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f00a5350 msgr2=0x7f20f01d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f00a5350 0x7f20f01d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 msgr2=0x7f20f01d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 0x7f20f01d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient: _finish_hunting 0 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 monclient: found mon.noname-c 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00a1760 con 0x7f20f0137d30 2024-09-17T15:36:22.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f5a28640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f00a4f70 0x7f20f01d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 msgr2=0x7f20f013c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20f6229640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20e000c3e0 tx=0x7f20e000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0017070 con 0x7f20f0137d30 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.005+0000 7f20ee7fc640 10 monclient: dump: 2024-09-17T15:36:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: _finish_auth 0 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 5 monclient: authenticate success, global_id 104141 2024-09-17T15:36:22.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e0005800 con 0x7f20f0137d30 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 monclient: _renew_subs 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f00a0f20 con 0x7f20f0137d30 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0005bf0 con 0x7f20f0137d30 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 10 monclient: dump: 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20effff640 4 set_mon_vals no callback set 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 monclient: _renew_subs 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f013e910 con 0x7f20f0137d30 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 0x7f20f013c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20f009cef0 con 0x7f20f01d0f80 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f20e001a070 con 0x7f20f0137d30 2024-09-17T15:36:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f20dc03bb90 0x7f20dc03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 0x7f20f013c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient: get_auth_request con 0x7f20f01d0f80 auth_method 0 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f20e004f3f0 con 0x7f20f0137d30 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5227640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f20dc03bb90 0x7f20dc03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5227640 10 monclient: get_auth_request con 0x7f20dc03bb90 auth_method 0 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 10 monclient(hunting): handle_auth_done global_id 54457 payload 824 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.009+0000 7f20f5a28640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20f00a5ff0 con 0x7f20f01d0f80 2024-09-17T15:36:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20f5a28640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 msgr2=0x7f20f013c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20f5a28640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 0x7f20f013c980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20e80af000 tx=0x7f20e8019620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20f5227640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f20dc03bb90 0x7f20dc03e050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f20f009ef80 tx=0x7f20e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e80b2040 con 0x7f20f01d0f80 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 10 monclient: dump: 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20e80b3040 con 0x7f20f01d0f80 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 msgr2=0x7f20f013c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 0x7f20f013c980 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20e80af000 tx=0x7f20e8019620 comp rx=0 tx=0).stop 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 msgr2=0x7f20f013c980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:22.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.013+0000 7f20ee7fc640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f01d0f80 0x7f20f013c980 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f20e80af000 tx=0x7f20e8019620 comp rx=0 tx=0).stop 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f7c8c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 0x7f20f0139160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f7c8c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f20f013d2a0 con 0x7f20f013cec0 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f5a28640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 0x7f20f0139160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f5a28640 10 monclient: get_auth_request con 0x7f20f013cec0 auth_method 0 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.017+0000 7f20f5a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 10 monclient(hunting): handle_auth_done global_id 54460 payload 824 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 1 -- 172.21.15.86:0/1692961765 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20f0121930 con 0x7f20f013cec0 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 msgr2=0x7f20f0139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20f5a28640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 0x7f20f0139160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20e801c6b0 tx=0x7f20e80a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e80b2070 con 0x7f20f013cec0 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 10 monclient: dump: 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f20e80c4c60 con 0x7f20f013cec0 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 msgr2=0x7f20f0139160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 0x7f20f0139160 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20e801c6b0 tx=0x7f20e80a7000 comp rx=0 tx=0).stop 2024-09-17T15:36:22.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 msgr2=0x7f20f0139160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.021+0000 7f20ee7fc640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20f013cec0 0x7f20f0139160 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f20e801c6b0 tx=0x7f20e80a7000 comp rx=0 tx=0).stop 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 88, 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 257460, 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:22.034 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:22.035 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:22.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:22.038 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:22.039 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.040 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.041 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:22.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:22.043 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:22.043 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f20dc03bb90 msgr2=0x7f20dc03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f20dc03bb90 0x7f20dc03e050 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f20f009ef80 tx=0x7f20e4008040 comp rx=0 tx=0).stop 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 10 monclient: shutdown 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 msgr2=0x7f20f013c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f6229640 1 -- 172.21.15.86:0/1692961765 reap_dead start 2024-09-17T15:36:22.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20e000c3e0 tx=0x7f20e000c8a0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 shutdown_connections 2024-09-17T15:36:22.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 --2- 172.21.15.86:0/1692961765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0137d30 0x7f20f013c440 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 >> 172.21.15.86:0/1692961765 conn(0x7f20f00a7240 msgr2=0x7f20f013f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:22.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 shutdown_connections 2024-09-17T15:36:22.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.025+0000 7f20f7c8c640 1 -- 172.21.15.86:0/1692961765 wait complete. 2024-09-17T15:36:22.045 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:36:22.045 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:36:22.045 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:36:22.045 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:36:22.046 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:36:22.046 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:36:22.046 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:36:22.078 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:36:22.078 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:36:22.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:22.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: get_monmap_and_config 2024-09-17T15:36:22.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: build_initial_monmap 2024-09-17T15:36:22.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: monmap: 2024-09-17T15:36:22.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:22.193101+0000 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:22.193101+0000 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:22.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 Processor -- start 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 -- start start 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: init 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:36:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d600a0c10 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 0x7f9d60135df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6006cb70 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9d6013ac70 0x7f9d6013d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6013ac70 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6006d7e0 con 0x7f9d600a0c10 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9d60136860 con 0x7f9d6013ac70 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9d6006d220 con 0x7f9d6006cb70 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d66057640 10 monclient(hunting): authenticate will time out at 1515.158500s 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9d6013ac70 0x7f9d6013d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5effd640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 msgr2=0x7f9d60135df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9d6013ac70 0x7f9d6013d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37506/0 (socket says 172.21.15.86:37506) 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9d600a0c10 0x7f9d600a0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59504/0 (socket says 172.21.15.86:59504) 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 -- 172.21.15.86:0/4193108211 learned_addr learned my addr 172.21.15.86:0/4193108211 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5effd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 0x7f9d60135df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): get_auth_request con 0x7f9d6013ac70 auth_method 0 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5e7fc640 1 -- 172.21.15.86:0/4193108211 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient(hunting): handle_auth_done global_id 94263 payload 274 2024-09-17T15:36:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 -- 172.21.15.86:0/4193108211 >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 msgr2=0x7f9d60135df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 --2- 172.21.15.86:0/4193108211 >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 0x7f9d60135df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 -- 172.21.15.86:0/4193108211 >> v1:172.21.15.86:6789/0 conn(0x7f9d600a0c10 legacy=0x7f9d600a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 monclient: found mon.noname-a 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.189+0000 7f9d5ffff640 1 -- 172.21.15.86:0/4193108211 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6006d500 con 0x7f9d6013ac70 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5ffff640 1 -- 172.21.15.86:0/4193108211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6013ac70 msgr2=0x7f9d6013d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5ffff640 1 --2- 172.21.15.86:0/4193108211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6013ac70 0x7f9d6013d060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9d60075100 tx=0x7f9d5001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d79cf379a4b577 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 1 -- 172.21.15.86:0/4193108211 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500a5550 con 0x7f9d6013ac70 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: dump: 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 5 monclient: authenticate success, global_id 94263 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 1 -- 172.21.15.86:0/4193108211 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d500a56f0 con 0x7f9d6013ac70 2024-09-17T15:36:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 1 -- 172.21.15.86:0/4193108211 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500a5ac0 con 0x7f9d6013ac70 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 4 set_mon_vals no callback set 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d5e7fc640 10 monclient: dump: 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.202 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 10 monclient: shutdown 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 1 -- 172.21.15.86:0/4193108211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6013ac70 msgr2=0x7f9d6013d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.193+0000 7f9d66057640 1 --2- 172.21.15.86:0/4193108211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6013ac70 0x7f9d6013d060 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9d60075100 tx=0x7f9d5001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- 172.21.15.86:0/4193108211 shutdown_connections 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 --2- 172.21.15.86:0/4193108211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6013ac70 0x7f9d6013d060 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 --2- 172.21.15.86:0/4193108211 >> v2:172.21.15.203:3300/0 conn(0x7f9d6006cb70 0x7f9d60135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- 172.21.15.86:0/4193108211 >> 172.21.15.86:0/4193108211 conn(0x7f9d600a3e80 msgr2=0x7f9d600a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- 172.21.15.86:0/4193108211 shutdown_connections 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- 172.21.15.86:0/4193108211 wait complete. 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient: build_initial_monmap 2024-09-17T15:36:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient: monmap: 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 Processor -- start 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- start start 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient: init 2024-09-17T15:36:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6006cb70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d600a0c10 0x7f9d6009b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600a0c10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 0x7f9d6009b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6013ac70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9d601402f0 con 0x7f9d6006cb70 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d60140170 con 0x7f9d6013ac70 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9d60140470 con 0x7f9d600a0c10 2024-09-17T15:36:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d66057640 10 monclient(hunting): authenticate will time out at 1515.165453s 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:37512/0 (socket says 172.21.15.86:37512) 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 -- 172.21.15.86:0/1899048873 learned_addr learned my addr 172.21.15.86:0/1899048873 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): get_auth_request con 0x7f9d6006cb70 auth_method 0 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d600a0c10 0x7f9d6009b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): get_auth_request con 0x7f9d600a0c10 auth_method 0 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5ffff640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 msgr2=0x7f9d6009b930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5ffff640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 0x7f9d6009b930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient(hunting): handle_auth_done global_id 94266 payload 824 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d600a0c10 msgr2=0x7f9d6009b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d600a0c10 0x7f9d6009b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 msgr2=0x7f9d6009b930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 0x7f9d6009b930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:36:22.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 monclient: found mon.noname-a 2024-09-17T15:36:22.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:22.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.197+0000 7f9d5f7fe640 1 -- 172.21.15.86:0/1899048873 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d600a6cf0 con 0x7f9d6006cb70 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d5f7fe640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 msgr2=0x7f9d60134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d5f7fe640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c490 tx=0x7f9d5800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58017070 con 0x7f9d6006cb70 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: dump: 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: _finish_auth 0 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d58005820 con 0x7f9d6006cb70 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 5 monclient: authenticate success, global_id 94266 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _renew_subs 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6009e570 con 0x7f9d6006cb70 2024-09-17T15:36:22.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58005c10 con 0x7f9d6006cb70 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 10 monclient: dump: 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:22.212 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d5dffb640 4 set_mon_vals no callback set 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _renew_subs 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d60138ea0 con 0x7f9d6006cb70 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9d5801a070 con 0x7f9d6006cb70 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d60135df0 con 0x7f9d6006cb70 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 0x7f9d4003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f9d5804f390 con 0x7f9d6006cb70 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d5effd640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 0x7f9d4003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:22.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.201+0000 7f9d5effd640 10 monclient: get_auth_request con 0x7f9d4003bb60 auth_method 0 2024-09-17T15:36:22.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.205+0000 7f9d5effd640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 0x7f9d4003e020 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9d480093d0 tx=0x7f9d48009360 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:22.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.205+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9d5804dd70 con 0x7f9d6006cb70 2024-09-17T15:36:22.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.205+0000 7f9d57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:22.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.205+0000 7f9d57fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d66057640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d66057640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d66057640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:22.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9d601394f0 con 0x7f9d6006cb70 2024-09-17T15:36:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d57fff640 1 -- 172.21.15.86:0/1899048873 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9d5804f080 con 0x7f9d6006cb70 2024-09-17T15:36:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:22.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.437+0000 7f9d57fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:22.441 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:22.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 msgr2=0x7f9d4003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 0x7f9d4003e020 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9d480093d0 tx=0x7f9d48009360 comp rx=0 tx=0).stop 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 10 monclient: shutdown 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 msgr2=0x7f9d60134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800c490 tx=0x7f9d5800c950 comp rx=0 tx=0).stop 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 shutdown_connections 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9d4003bb60 0x7f9d4003e020 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d6013ac70 0x7f9d6009b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d600a0c10 0x7f9d6009b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 --2- 172.21.15.86:0/1899048873 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d6006cb70 0x7f9d60134c30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:22.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 >> 172.21.15.86:0/1899048873 conn(0x7f9d600a3e80 msgr2=0x7f9d600a2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:22.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 shutdown_connections 2024-09-17T15:36:22.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:22.441+0000 7f9d66057640 1 -- 172.21.15.86:0/1899048873 wait complete. 2024-09-17T15:36:25.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: get_monmap_and_config 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: build_initial_monmap 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: monmap: 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:25.589762+0000 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:25.589762+0000 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:25.588 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 1 Processor -- start 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 1 -- start start 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: init 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 0x7eff8409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient(hunting): picked mon.noname-b con 0x7eff840a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff840a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 0x7eff8413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.585+0000 7eff88853640 10 monclient(hunting): picked mon.noname-c con 0x7eff8413d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8409d350 con 0x7eff840a54c0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7eff8409bdc0 con 0x7eff8413d2f0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff840a6580 con 0x7eff840a50e0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff88853640 10 monclient(hunting): authenticate will time out at 1518.556041s 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 0x7eff8409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff81d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7eff840a54c0 0x7eff840a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59510/0 (socket says 172.21.15.86:59510) 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff81d9b640 1 -- 172.21.15.86:0/314715876 learned_addr learned my addr 172.21.15.86:0/314715876 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:36:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): get_auth_request con 0x7eff840a50e0 auth_method 0 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff82d9d640 1 -- 172.21.15.86:0/314715876 >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 msgr2=0x7eff8413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff82d9d640 1 --2- 172.21.15.86:0/314715876 >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 0x7eff8413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient(hunting): handle_auth_done global_id 104147 payload 274 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 -- 172.21.15.86:0/314715876 >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 msgr2=0x7eff8413f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 --2- 172.21.15.86:0/314715876 >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 0x7eff8413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 -- 172.21.15.86:0/314715876 >> v1:172.21.15.86:6789/0 conn(0x7eff840a54c0 legacy=0x7eff840a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient: _finish_hunting 0 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 monclient: found mon.noname-b 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.589+0000 7eff8259c640 1 -- 172.21.15.86:0/314715876 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8409c150 con 0x7eff840a50e0 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8259c640 1 -- 172.21.15.86:0/314715876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 msgr2=0x7eff8409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8259c640 1 --2- 172.21.15.86:0/314715876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 0x7eff8409ce10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff8409c770 tx=0x7eff7401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c84eb02e2fa0b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 1 -- 172.21.15.86:0/314715876 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff740b0070 con 0x7eff840a50e0 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: dump: 2024-09-17T15:36:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: _finish_auth 0 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: _check_auth_tickets 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 1 -- 172.21.15.86:0/314715876 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff740ab600 con 0x7eff840a50e0 2024-09-17T15:36:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 5 monclient: authenticate success, global_id 104147 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 4 set_mon_vals no callback set 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 1 -- 172.21.15.86:0/314715876 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff740ab9d0 con 0x7eff840a50e0 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8159a640 10 monclient: dump: 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:25.597 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:25.597 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: shutdown 2024-09-17T15:36:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- 172.21.15.86:0/314715876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 msgr2=0x7eff8409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- 172.21.15.86:0/314715876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 0x7eff8409ce10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7eff8409c770 tx=0x7eff7401c360 comp rx=0 tx=0).stop 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- 172.21.15.86:0/314715876 shutdown_connections 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- 172.21.15.86:0/314715876 >> v2:172.21.15.203:3300/0 conn(0x7eff8413d2f0 0x7eff8413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- 172.21.15.86:0/314715876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff840a50e0 0x7eff8409ce10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- 172.21.15.86:0/314715876 >> 172.21.15.86:0/314715876 conn(0x7eff840aaae0 msgr2=0x7eff840aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- 172.21.15.86:0/314715876 shutdown_connections 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- 172.21.15.86:0/314715876 wait complete. 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: build_initial_monmap 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: monmap: 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 Processor -- start 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- start start 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: init 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:36:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 0x7eff84121410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): picked mon.noname-b con 0x7eff840a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff840a54c0 0x7eff840a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): picked mon.noname-a con 0x7eff840a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): picked mon.noname-c con 0x7eff8413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7eff84142880 con 0x7eff840a54c0 2024-09-17T15:36:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff84142700 con 0x7eff840a50e0 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff84142a00 con 0x7eff8413d2f0 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff88853640 10 monclient(hunting): authenticate will time out at 1518.563260s 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51460/0 (socket says 172.21.15.86:51460) 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 -- 172.21.15.86:0/657983868 learned_addr learned my addr 172.21.15.86:0/657983868 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): get_auth_request con 0x7eff8413d2f0 auth_method 0 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:25.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8259c640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 msgr2=0x7eff84121410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff8259c640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 0x7eff84121410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff840a54c0 0x7eff840a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): get_auth_request con 0x7eff840a54c0 auth_method 0 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff81d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient(hunting): handle_auth_done global_id 104150 payload 824 2024-09-17T15:36:25.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 msgr2=0x7eff84121410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 0x7eff84121410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff840a54c0 msgr2=0x7eff840a31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff840a54c0 0x7eff840a31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 monclient: found mon.noname-c 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.593+0000 7eff82d9d640 1 -- 172.21.15.86:0/657983868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff840a7db0 con 0x7eff8413d2f0 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff82d9d640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 msgr2=0x7eff84138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff82d9d640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00be50 tx=0x7eff7c00be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c016630 con 0x7eff8413d2f0 2024-09-17T15:36:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: dump: 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:25.612 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: _finish_auth 0 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c010030 con 0x7eff8413d2f0 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 5 monclient: authenticate success, global_id 104150 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _renew_subs 2024-09-17T15:36:25.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff840a18d0 con 0x7eff8413d2f0 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c016a10 con 0x7eff8413d2f0 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 10 monclient: dump: 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:25.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff80d99640 4 set_mon_vals no callback set 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _renew_subs 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff84122370 con 0x7eff8413d2f0 2024-09-17T15:36:25.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8409ce10 con 0x7eff8413d2f0 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.597+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7eff7c0153e0 con 0x7eff8413d2f0 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff72ffd640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 0x7eff6403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7eff7c050c20 con 0x7eff8413d2f0 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff8259c640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 0x7eff6403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff8259c640 10 monclient: get_auth_request con 0x7eff6403bb40 auth_method 0 2024-09-17T15:36:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:36:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.601+0000 7eff8259c640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 0x7eff6403e000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7eff8409c770 tx=0x7eff740a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.605+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff7c050e20 con 0x7eff8413d2f0 2024-09-17T15:36:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.605+0000 7eff72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.605+0000 7eff72ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:25.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff88853640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:25.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff88853640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:25.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff88853640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:25.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff84074be0 con 0x7eff8413d2f0 2024-09-17T15:36:25.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff72ffd640 1 -- 172.21.15.86:0/657983868 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7eff7c007cf0 con 0x7eff8413d2f0 2024-09-17T15:36:25.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:25.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.841+0000 7eff72ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:25.843 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:25.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 msgr2=0x7eff6403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 0x7eff6403e000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7eff8409c770 tx=0x7eff740a6000 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 10 monclient: shutdown 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 msgr2=0x7eff84138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7eff7c00be50 tx=0x7eff7c00be80 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 shutdown_connections 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff6403bb40 0x7eff6403e000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff8413d2f0 0x7eff84138280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff840a54c0 0x7eff840a31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 --2- 172.21.15.86:0/657983868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff840a50e0 0x7eff84121410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 >> 172.21.15.86:0/657983868 conn(0x7eff840aaae0 msgr2=0x7eff840a0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:25.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 shutdown_connections 2024-09-17T15:36:25.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:25.845+0000 7eff88853640 1 -- 172.21.15.86:0/657983868 wait complete. 2024-09-17T15:36:28.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: get_monmap_and_config 2024-09-17T15:36:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: build_initial_monmap 2024-09-17T15:36:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: monmap: 2024-09-17T15:36:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:28.993966+0000 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:28.993966+0000 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:28.992 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 Processor -- start 2024-09-17T15:36:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 -- start start 2024-09-17T15:36:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: init 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38e00a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38e00a5b60 0x7f38e009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-a con 0x7f38e00a5b60 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 0x7f38e013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-c con 0x7f38e013d280 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38e00a6f40 con 0x7f38e00a5070 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f38e00a65e0 con 0x7f38e00a5b60 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f38e009c2a0 con 0x7f38e013d280 2024-09-17T15:36:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e79d1640 10 monclient(hunting): authenticate will time out at 1521.959495s 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38e00a5b60 0x7f38e009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f38e00a5b60 0x7f38e009cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40644/0 (socket says 172.21.15.86:40644) 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 1 -- 172.21.15.86:0/3815731287 learned_addr learned my addr 172.21.15.86:0/3815731287 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): get_auth_request con 0x7f38e00a5b60 auth_method 0 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e5f6e640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 msgr2=0x7f38e013f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e576d640 1 --1- 172.21.15.86:0/3815731287 >> v1:172.21.15.86:6789/0 conn(0x7f38e00a5070 0x7f38e00a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51026/0 (socket says 172.21.15.86:51026) 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e5f6e640 1 --2- 172.21.15.86:0/3815731287 >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 0x7f38e013f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 10 monclient(hunting): handle_auth_done global_id 94272 payload 274 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 1 -- 172.21.15.86:0/3815731287 >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 msgr2=0x7f38e013f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.988+0000 7f38e4f6c640 1 --2- 172.21.15.86:0/3815731287 >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 0x7f38e013f670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 1 -- 172.21.15.86:0/3815731287 >> v1:172.21.15.86:6789/0 conn(0x7f38e00a5070 legacy=0x7f38e00a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 10 monclient: _finish_hunting 0 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 1 monclient: found mon.noname-a 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.992+0000 7f38e4f6c640 1 -- 172.21.15.86:0/3815731287 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e009c580 con 0x7f38e00a5b60 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e4f6c640 1 -- 172.21.15.86:0/3815731287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5b60 msgr2=0x7f38e009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e4f6c640 1 --2- 172.21.15.86:0/3815731287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5b60 0x7f38e009cf60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f38e00a8470 tx=0x7f38dc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ab332916ad1567 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 1 -- 172.21.15.86:0/3815731287 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0b0070 con 0x7f38e00a5b60 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: dump: 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: _finish_auth 0 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 5 monclient: authenticate success, global_id 94272 2024-09-17T15:36:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 1 -- 172.21.15.86:0/3815731287 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38dc0a4e50 con 0x7f38e00a5b60 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 4 set_mon_vals no callback set 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 1 -- 172.21.15.86:0/3815731287 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0ab820 con 0x7f38e00a5b60 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38d7fff640 10 monclient: dump: 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: shutdown 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- 172.21.15.86:0/3815731287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5b60 msgr2=0x7f38e009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- 172.21.15.86:0/3815731287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5b60 0x7f38e009cf60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f38e00a8470 tx=0x7f38dc01c400 comp rx=0 tx=0).stop 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- 172.21.15.86:0/3815731287 shutdown_connections 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- 172.21.15.86:0/3815731287 >> v2:172.21.15.203:3300/0 conn(0x7f38e013d280 0x7f38e013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- 172.21.15.86:0/3815731287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5b60 0x7f38e009cf60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- 172.21.15.86:0/3815731287 >> 172.21.15.86:0/3815731287 conn(0x7f38e00742a0 msgr2=0x7f38e00746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- 172.21.15.86:0/3815731287 shutdown_connections 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- 172.21.15.86:0/3815731287 wait complete. 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: build_initial_monmap 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: monmap: 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 Processor -- start 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- start start 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: init 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-a con 0x7f38e00a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 0x7f38e01381d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-b con 0x7f38e00a5b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38e013d280 0x7f38e00a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): picked mon.noname-c con 0x7f38e013d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:29.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0142760 con 0x7f38e00a5070 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38e01425e0 con 0x7f38e00a5b60 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f38e01428e0 con 0x7f38e013d280 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e79d1640 10 monclient(hunting): authenticate will time out at 1521.966547s 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e4f6c640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 msgr2=0x7f38e01381d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e4f6c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 0x7f38e01381d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38e013d280 0x7f38e00a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40650/0 (socket says 172.21.15.86:40650) 2024-09-17T15:36:29.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 -- 172.21.15.86:0/1075637013 learned_addr learned my addr 172.21.15.86:0/1075637013 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): get_auth_request con 0x7f38e00a5070 auth_method 0 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): get_auth_request con 0x7f38e013d280 auth_method 0 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:29.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e5f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient(hunting): handle_auth_done global_id 94275 payload 824 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 -- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38e013d280 msgr2=0x7f38e00a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38e013d280 0x7f38e00a2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 -- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 msgr2=0x7f38e01381d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 0x7f38e01381d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient: _finish_hunting 0 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 monclient: found mon.noname-a 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:29.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:28.996+0000 7f38e576d640 1 -- 172.21.15.86:0/1075637013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e0122880 con 0x7f38e00a5070 2024-09-17T15:36:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e576d640 1 -- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 msgr2=0x7f38e01cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e576d640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f38d000c7f0 tx=0x7f38d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d0017070 con 0x7f38e00a5070 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: dump: 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38d000a860 con 0x7f38e00a5070 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 5 monclient: authenticate success, global_id 94275 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _renew_subs 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e0122b60 con 0x7f38e00a5070 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d000ac50 con 0x7f38e00a5070 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 10 monclient: dump: 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d6ffd640 4 set_mon_vals no callback set 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _renew_subs 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e0120ec0 con 0x7f38e00a5070 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38e009cfe0 con 0x7f38e00a5070 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f38d001a070 con 0x7f38e00a5070 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 0x7f38cc03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f38d004f6d0 con 0x7f38e00a5070 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e4f6c640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 0x7f38cc03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.000+0000 7f38e4f6c640 10 monclient: get_auth_request con 0x7f38cc03bb40 auth_method 0 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.004+0000 7f38e4f6c640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 0x7f38cc03e000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0ab000 tx=0x7f38dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:29.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.004+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f38d000adf0 con 0x7f38e00a5070 2024-09-17T15:36:29.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.004+0000 7f38d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:29.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.004+0000 7f38d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f38e0121f30 con 0x7f38e00a5070 2024-09-17T15:36:29.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38d5ffb640 1 -- 172.21.15.86:0/1075637013 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f38d00078c0 con 0x7f38e00a5070 2024-09-17T15:36:29.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:29.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:29.231 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 msgr2=0x7f38cc03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 0x7f38cc03e000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0ab000 tx=0x7f38dc008040 comp rx=0 tx=0).stop 2024-09-17T15:36:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 10 monclient: shutdown 2024-09-17T15:36:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 msgr2=0x7f38e01cfb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f38d000c7f0 tx=0x7f38d000ccb0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 shutdown_connections 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f38cc03bb40 0x7f38cc03e000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38e013d280 0x7f38e00a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38e00a5b60 0x7f38e01381d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 --2- 172.21.15.86:0/1075637013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38e00a5070 0x7f38e01cfb90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.228+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 >> 172.21.15.86:0/1075637013 conn(0x7f38e00742a0 msgr2=0x7f38e00a0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.232+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 shutdown_connections 2024-09-17T15:36:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:29.232+0000 7f38e79d1640 1 -- 172.21.15.86:0/1075637013 wait complete. 2024-09-17T15:36:32.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:32.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: get_monmap_and_config 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: build_initial_monmap 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: monmap: 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:32.381292+0000 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:32.381292+0000 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:32.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 Processor -- start 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 -- start start 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: init 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f35b00a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f35b00a5450 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f35b0137e30 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f35b00a6f40 con 0x7f35b00a5450 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f35b00a65e0 con 0x7f35b0137e30 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f35b009c2a0 con 0x7f35b00a5070 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b6fc7640 10 monclient(hunting): authenticate will time out at 1525.347033s 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40674/0 (socket says 172.21.15.86:40674) 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 learned_addr learned my addr 172.21.15.86:0/1645783637 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f35b00a5450 auth_method 0 2024-09-17T15:36:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 msgr2=0x7f35b0138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b5564640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f35b00a5070 auth_method 0 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:36:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.376+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:32.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 msgr2=0x7f35b0138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b5564640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f35b00a5070 auth_method 0 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f35b00a5450 auth_method 0 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:36:32.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:36:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.580+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:36:32.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:32.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f35b00a5450 auth_method 0 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f35b00a5070 auth_method 0 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:36:32.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:36:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 msgr2=0x7f35b0138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:32.984+0000 7f35b5564640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:36:33.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:33.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f35b00a5450 auth_method 0 2024-09-17T15:36:33.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 msgr2=0x7f35b0138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:33.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b5564640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f35b00a5070 auth_method 0 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:36:33.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:36:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:36:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:36:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:33.788+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:36:35.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): tick 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 monclient(hunting): continuing hunt 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 msgr2=0x7f35b009cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35b00a5070 0x7f35b009cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 msgr2=0x7f35b0138240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.203:3300/0 conn(0x7f35b0137e30 0x7f35b0138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 -- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 msgr2=0x7f35b00a5830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 --2- 172.21.15.86:0/1645783637 >> v2:172.21.15.86:3300/0 conn(0x7f35b00a5450 0x7f35b00a5830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient: _add_conns ranks=[1,0] 2024-09-17T15:36:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35900036a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35900073f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 -- 172.21.15.86:0/1645783637 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b009c580 con 0x7f35900073f0 2024-09-17T15:36:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 1 -- 172.21.15.86:0/1645783637 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f359000c110 con 0x7f35900036a0 2024-09-17T15:36:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35ae7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:35.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.376+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.203:6789/0 conn(0x7f35900036a0 legacy=0x7f3590005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:36:35.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.380+0000 7f35affff640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.86:6789/0 conn(0x7f35900073f0 legacy=0x7f3590009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:35.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.580+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.203:6789/0 conn(0x7f35900036a0 legacy=0x7f3590005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:36:35.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:35.984+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.203:6789/0 conn(0x7f35900036a0 legacy=0x7f3590005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939217119 0 0) 0x7f35b009c580 con 0x7f35900073f0 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 10 monclient(hunting): my global_id is 94278 2024-09-17T15:36:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.772+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f359c003530 con 0x7f35900073f0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1575230852 0 0) 0x7f359c003530 con 0x7f35900073f0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.203:6789/0 conn(0x7f35900036a0 legacy=0x7f3590005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359000d370 con 0x7f35900073f0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _finish_auth 0 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:36:36.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:36.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f359800f040 con 0x7f35900073f0 2024-09-17T15:36:36.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:36.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:36:36.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: dump: 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:36.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 >> v1:172.21.15.86:6789/0 conn(0x7f35900073f0 legacy=0x7f3590009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 reap_dead start 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f359c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): picked mon.a con 0x7f359c003a60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f359c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): picked mon.c con 0x7f359c004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f359c00b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): picked mon.b con 0x7f359c0053b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f359c00c2a0 con 0x7f359c003a60 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f359c00c450 con 0x7f359c0053b0 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f359c00c650 con 0x7f359c004810 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35af7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b5564640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 msgr2=0x7f359c00b9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f359c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:36.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f359c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b5564640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f359c00b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f359c003a60 auth_method 0 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f359c004810 auth_method 0 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:36.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient(hunting): handle_auth_done global_id 94278 payload 293 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 msgr2=0x7f359c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f359c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 msgr2=0x7f359c00b9f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f359c00b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient: _finish_hunting 0 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 monclient: found mon.a 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:36.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c003530 con 0x7f359c003a60 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient: _finish_auth 0 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 10 monclient: _check_auth_tickets 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b4d63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b6fc7640 5 monclient: authenticate success, global_id 94278 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35b6fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:36.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.780+0000 7f35affff640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f359c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b4d63640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 msgr2=0x7f359c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b4d63640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f359c003e40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009b00 tx=0x7f35a801d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d64337b25502889 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a80b3070 con 0x7f359c003a60 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: dump: 2024-09-17T15:36:36.793 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a80ae410 con 0x7f359c003a60 2024-09-17T15:36:36.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 4 set_mon_vals no callback set 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 1 -- 172.21.15.86:0/1645783637 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a80097e0 con 0x7f359c003a60 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35af7fe640 10 monclient: dump: 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:36.795 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 10 monclient: shutdown 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 -- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 msgr2=0x7f359c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f359c003e40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f35a8009b00 tx=0x7f35a801d5f0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 -- 172.21.15.86:0/1645783637 shutdown_connections 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f359c00b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f359c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/1645783637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f359c003e40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 -- 172.21.15.86:0/1645783637 >> 172.21.15.86:0/1645783637 conn(0x7f35b00742a0 msgr2=0x7f35b00746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 -- 172.21.15.86:0/1645783637 shutdown_connections 2024-09-17T15:36:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.788+0000 7f35b6fc7640 1 -- 172.21.15.86:0/1645783637 wait complete. 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient: build_initial_monmap 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient: monmap: 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:36.802 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 Processor -- start 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 -- start start 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient: init 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f35b01cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-a con 0x7f359c003a60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-c con 0x7f359c004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f35b01374f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): picked mon.noname-b con 0x7f359c0053b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f35b01385d0 con 0x7f359c003a60 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f35b0138450 con 0x7f359c0053b0 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f35b0138750 con 0x7f359c004810 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b6fc7640 10 monclient(hunting): authenticate will time out at 1529.762864s 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f35b01cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39218/0 (socket says 172.21.15.86:39218) 2024-09-17T15:36:36.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 -- 172.21.15.86:0/3694153125 learned_addr learned my addr 172.21.15.86:0/3694153125 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): get_auth_request con 0x7f359c004810 auth_method 0 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): get_auth_request con 0x7f359c003a60 auth_method 0 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b5564640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 msgr2=0x7f35b01374f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b5564640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f35b01374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35b4d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient(hunting): handle_auth_done global_id 104156 payload 824 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 msgr2=0x7f35b01374f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f35b01374f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 msgr2=0x7f35b01cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f35b01cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 monclient: found mon.noname-c 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:36.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.792+0000 7f35affff640 1 -- 172.21.15.86:0/3694153125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b00a1c00 con 0x7f359c004810 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35affff640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 msgr2=0x7f35b0136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35affff640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3598016e00 tx=0x7f3598014720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359801c070 con 0x7f359c004810 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: dump: 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:36.807 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: _finish_auth 0 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 5 monclient: authenticate success, global_id 104156 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3598004240 con 0x7f359c004810 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _renew_subs 2024-09-17T15:36:36.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359800f040 con 0x7f359c004810 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b0124690 con 0x7f359c004810 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35aeffd640 4 set_mon_vals no callback set 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 10 monclient: dump: 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:36.809 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _renew_subs 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b01243b0 con 0x7f359c004810 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f359801f070 con 0x7f359c004810 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b00a65e0 con 0x7f359c004810 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 0x7f359003e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:36.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.796+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f3598050d00 con 0x7f359c004810 2024-09-17T15:36:36.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.800+0000 7f35b4d63640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 0x7f359003e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:36.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.800+0000 7f35b4d63640 10 monclient: get_auth_request con 0x7f359003c3f0 auth_method 0 2024-09-17T15:36:36.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.800+0000 7f35b4d63640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 0x7f359003e8b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f35a80ad000 tx=0x7f35a8004ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:36.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.804+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f35b00a1c00 con 0x7f359c004810 2024-09-17T15:36:36.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.804+0000 7f35ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:36.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:36.804+0000 7f35ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:37.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.048+0000 7f35b6fc7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:37.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:37.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f35b00a2130 con 0x7f359c004810 2024-09-17T15:36:37.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35ad7fa640 1 -- 172.21.15.86:0/3694153125 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f359800cdb0 con 0x7f359c004810 2024-09-17T15:36:37.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:37.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:37.054 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 msgr2=0x7f359003e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 0x7f359003e8b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f35a80ad000 tx=0x7f35a8004ac0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 10 monclient: shutdown 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 msgr2=0x7f35b0136fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3598016e00 tx=0x7f3598014720 comp rx=0 tx=0).stop 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 shutdown_connections 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f359003c3f0 0x7f359003e8b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f359c0053b0 0x7f35b01374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f359c004810 0x7f35b0136fb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 --2- 172.21.15.86:0/3694153125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f359c003a60 0x7f35b01cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 >> 172.21.15.86:0/3694153125 conn(0x7f35b00742a0 msgr2=0x7f35b00a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:37.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 shutdown_connections 2024-09-17T15:36:37.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.052+0000 7f35b6fc7640 1 -- 172.21.15.86:0/3694153125 wait complete. 2024-09-17T15:36:37.076 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:36:37.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:36:37.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 10 monclient: get_monmap_and_config 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 10 monclient: build_initial_monmap 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 10 monclient: monmap: 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:37.198719+0000 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:37.198719+0000 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.196 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:37.197 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:37.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 1 Processor -- start 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 1 -- start start 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.192+0000 7f3816508640 10 monclient: init 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): picked mon.noname-b con 0x7f381009d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f381009e090 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 0x7f381013cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): picked mon.noname-c con 0x7f3810135150 addr v2:172.21.15.203:3300/0 2024-09-17T15:36:37.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381009ff00 con 0x7f381009e090 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f381009e7f0 con 0x7f3810135150 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f38100a2110 con 0x7f381009d770 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3816508640 10 monclient(hunting): authenticate will time out at 1530.164168s 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380f7fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f381009e090 legacy=0x7f3810134a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:36:37.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39230/0 (socket says 172.21.15.86:39230) 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3814aa5640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 msgr2=0x7f381013cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 -- 172.21.15.86:0/3642610427 learned_addr learned my addr 172.21.15.86:0/3642610427 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): get_auth_request con 0x7f381009d770 auth_method 0 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f3814aa5640 1 --2- 172.21.15.86:0/3642610427 >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 0x7f381013cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient(hunting): handle_auth_done global_id 104159 payload 274 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 -- 172.21.15.86:0/3642610427 >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 msgr2=0x7f381013cf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:37.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 --2- 172.21.15.86:0/3642610427 >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 0x7f381013cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 -- 172.21.15.86:0/3642610427 >> v1:172.21.15.203:6789/0 conn(0x7f381009e090 legacy=0x7f3810134a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 monclient: found mon.noname-b 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.196+0000 7f380ffff640 1 -- 172.21.15.86:0/3642610427 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381009ec30 con 0x7f381009d770 2024-09-17T15:36:37.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380ffff640 1 -- 172.21.15.86:0/3642610427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 msgr2=0x7f381009db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380ffff640 1 --2- 172.21.15.86:0/3642610427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3800009e50 tx=0x7f380001c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87cf893e753e124 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 1 -- 172.21.15.86:0/3642610427 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000a5550 con 0x7f381009d770 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: dump: 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: _finish_auth 0 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: _check_auth_tickets 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 5 monclient: authenticate success, global_id 104159 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 1 -- 172.21.15.86:0/3642610427 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38000a56f0 con 0x7f381009d770 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 1 -- 172.21.15.86:0/3642610427 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000a5ac0 con 0x7f381009d770 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 4 set_mon_vals no callback set 2024-09-17T15:36:37.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380effd640 10 monclient: dump: 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: shutdown 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- 172.21.15.86:0/3642610427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 msgr2=0x7f381009db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- 172.21.15.86:0/3642610427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3800009e50 tx=0x7f380001c910 comp rx=0 tx=0).stop 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- 172.21.15.86:0/3642610427 shutdown_connections 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- 172.21.15.86:0/3642610427 >> v2:172.21.15.203:3300/0 conn(0x7f3810135150 0x7f381013cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- 172.21.15.86:0/3642610427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009d770 0x7f381009db50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- 172.21.15.86:0/3642610427 >> 172.21.15.86:0/3642610427 conn(0x7f38100a2f40 msgr2=0x7f38100a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- 172.21.15.86:0/3642610427 shutdown_connections 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- 172.21.15.86:0/3642610427 wait complete. 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: build_initial_monmap 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: monmap: 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:37.209 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 Processor -- start 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- start start 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: init 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:36:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381009d770 0x7f381011e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): picked mon.noname-b con 0x7f381009d770 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 0x7f381011ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): picked mon.noname-c con 0x7f381009e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3810135150 0x7f381011f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): picked mon.noname-a con 0x7f3810135150 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f381009cce0 con 0x7f3810135150 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f381009ce60 con 0x7f381009d770 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f38100a0c70 con 0x7f381009e090 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3816508640 10 monclient(hunting): authenticate will time out at 1530.171374s 2024-09-17T15:36:37.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 0x7f381011ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 0x7f381011ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39236/0 (socket says 172.21.15.86:39236) 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3810135150 0x7f381011f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 -- 172.21.15.86:0/2222828770 learned_addr learned my addr 172.21.15.86:0/2222828770 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): get_auth_request con 0x7f381009e090 auth_method 0 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380ffff640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381009d770 msgr2=0x7f381011e8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380ffff640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381009d770 0x7f381011e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): get_auth_request con 0x7f3810135150 auth_method 0 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient(hunting): handle_auth_done global_id 104162 payload 824 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381009d770 msgr2=0x7f381011e8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f381009d770 0x7f381011e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3810135150 msgr2=0x7f381011f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3810135150 0x7f381011f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:36:37.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:36:37.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:37.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f380f7fe640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381009d460 con 0x7f381009e090 2024-09-17T15:36:37.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.200+0000 7f3814aa5640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3810135150 0x7f381011f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380f7fe640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 msgr2=0x7f381011ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380f7fe640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 0x7f381011ede0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f37f800b530 tx=0x7f37f800b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8013020 con 0x7f381009e090 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: dump: 2024-09-17T15:36:37.218 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: _finish_auth 0 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37f8005070 con 0x7f381009e090 2024-09-17T15:36:37.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 5 monclient: authenticate success, global_id 104162 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 monclient: _renew_subs 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38101223e0 con 0x7f381009e090 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f8005460 con 0x7f381009e090 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 10 monclient: dump: 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.220 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380e7fc640 4 set_mon_vals no callback set 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 monclient: _renew_subs 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3810122100 con 0x7f381009e090 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 0x7f381002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f38100a07a0 con 0x7f38101d1210 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f37f8016020 con 0x7f381009e090 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 0x7f381002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient: get_auth_request con 0x7f38101d1210 auth_method 0 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f37f003bb60 0x7f37f003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f37f804fb50 con 0x7f381009e090 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 10 monclient(hunting): handle_auth_done global_id 94287 payload 824 2024-09-17T15:36:37.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.204+0000 7f380ffff640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3810122bc0 con 0x7f38101d1210 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f3814aa5640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f37f003bb60 0x7f37f003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f3814aa5640 10 monclient: get_auth_request con 0x7f37f003bb60 auth_method 0 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380ffff640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 msgr2=0x7f381002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380ffff640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 0x7f381002d450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38000ae000 tx=0x7f38000196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000bd070 con 0x7f38101d1210 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 10 monclient: dump: 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.223 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3800009af0 con 0x7f38101d1210 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 msgr2=0x7f381002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 0x7f381002d450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38000ae000 tx=0x7f38000196b0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 msgr2=0x7f381002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:37.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f380cff9640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d1210 0x7f381002d450 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38000ae000 tx=0x7f38000196b0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.208+0000 7f3814aa5640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f37f003bb60 0x7f37f003e020 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3810120790 tx=0x7f3808003800 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 0x7f381002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f381009db50 con 0x7f38101d15f0 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 0x7f381002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient: get_auth_request con 0x7f38101d15f0 auth_method 0 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 10 monclient(hunting): handle_auth_done global_id 94290 payload 824 2024-09-17T15:36:37.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 1 -- 172.21.15.86:0/2222828770 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38100744d0 con 0x7f38101d15f0 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 msgr2=0x7f381002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380ffff640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 0x7f381002d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f380001c740 tx=0x7f3800009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38000b9d70 con 0x7f38101d15f0 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 10 monclient: dump: 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f38000b9d70 con 0x7f38101d15f0 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 msgr2=0x7f381002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 0x7f381002d990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f380001c740 tx=0x7f3800009810 comp rx=0 tx=0).stop 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 msgr2=0x7f381002d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:37.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f380cff9640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38101d15f0 0x7f381002d990 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f380001c740 tx=0x7f3800009810 comp rx=0 tx=0).stop 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 92, 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72571, 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.229 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.230 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.231 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:37.232 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.233 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.234 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.235 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:37.237 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.239 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.240 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.241 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:37.242 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:37.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f37f003bb60 msgr2=0x7f37f003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f37f003bb60 0x7f37f003e020 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3810120790 tx=0x7f3808003800 comp rx=0 tx=0).stop 2024-09-17T15:36:37.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 10 monclient: shutdown 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 msgr2=0x7f381011ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 --2- 172.21.15.86:0/2222828770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f381009e090 0x7f381011ede0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f37f800b530 tx=0x7f37f800b9f0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3814aa5640 1 -- 172.21.15.86:0/2222828770 reap_dead start 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3814aa5640 1 -- 172.21.15.86:0/2222828770 reap_dead start 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 shutdown_connections 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.216+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 >> 172.21.15.86:0/2222828770 conn(0x7f38100a2f40 msgr2=0x7f38100a4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.220+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 shutdown_connections 2024-09-17T15:36:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.220+0000 7f3816508640 1 -- 172.21.15.86:0/2222828770 wait complete. 2024-09-17T15:36:37.244 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:36:37.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:36:37.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 10 monclient: get_monmap_and_config 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 10 monclient: build_initial_monmap 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 10 monclient: monmap: 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:37.362981+0000 2024-09-17T15:36:37.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:37.362981+0000 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:37.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 1 Processor -- start 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 1 -- start start 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.356+0000 7f57bdb56640 10 monclient: init 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57b80a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 0x7f57b809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-a con 0x7f57b80a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-b con 0x7f57b813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b80a6280 con 0x7f57b80a50e0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f57b809d2a0 con 0x7f57b80a5bd0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b809c0a0 con 0x7f57b813d2f0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57bdb56640 10 monclient(hunting): authenticate will time out at 1530.328522s 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 0x7f57b809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b77fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f57b80a50e0 legacy=0x7f57b80a54c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39238/0 (socket says 172.21.15.86:39238) 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 -- 172.21.15.86:0/4280207140 learned_addr learned my addr 172.21.15.86:0/4280207140 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:37.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): get_auth_request con 0x7f57b813d2f0 auth_method 0 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): get_auth_request con 0x7f57b80a5bd0 auth_method 0 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient(hunting): handle_auth_done global_id 104165 payload 274 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 -- 172.21.15.86:0/4280207140 >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 msgr2=0x7f57b809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 --2- 172.21.15.86:0/4280207140 >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 0x7f57b809cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 -- 172.21.15.86:0/4280207140 >> v1:172.21.15.203:6789/0 conn(0x7f57b80a50e0 legacy=0x7f57b80a54c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 monclient: found mon.noname-b 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b7fff640 1 -- 172.21.15.86:0/4280207140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b809c380 con 0x7f57b813d2f0 2024-09-17T15:36:37.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.360+0000 7f57b6ffd640 1 --2- 172.21.15.86:0/4280207140 >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 0x7f57b809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 -- 172.21.15.86:0/4280207140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 msgr2=0x7f57b813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 --2- 172.21.15.86:0/4280207140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e50 tx=0x7f57b001c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bf67d2d3750e32d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 1 -- 172.21.15.86:0/4280207140 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a5600 con 0x7f57b813d2f0 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: dump: 2024-09-17T15:36:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: _finish_auth 0 2024-09-17T15:36:37.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 5 monclient: authenticate success, global_id 104165 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 1 -- 172.21.15.86:0/4280207140 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b00a57a0 con 0x7f57b813d2f0 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 4 set_mon_vals no callback set 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 1 -- 172.21.15.86:0/4280207140 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a5b70 con 0x7f57b813d2f0 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b67fc640 10 monclient: dump: 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: shutdown 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- 172.21.15.86:0/4280207140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 msgr2=0x7f57b813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- 172.21.15.86:0/4280207140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57b0009e50 tx=0x7f57b001c860 comp rx=0 tx=0).stop 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- 172.21.15.86:0/4280207140 shutdown_connections 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- 172.21.15.86:0/4280207140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b813f6e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- 172.21.15.86:0/4280207140 >> v2:172.21.15.86:3300/0 conn(0x7f57b80a5bd0 0x7f57b809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- 172.21.15.86:0/4280207140 >> 172.21.15.86:0/4280207140 conn(0x7f57b80aaae0 msgr2=0x7f57b80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- 172.21.15.86:0/4280207140 shutdown_connections 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- 172.21.15.86:0/4280207140 wait complete. 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: build_initial_monmap 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: monmap: 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:37.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 Processor -- start 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- start start 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: init 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b80a50e0 0x7f57b81cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-b con 0x7f57b80a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:37.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 0x7f57b8138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-a con 0x7f57b80a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b809f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): picked mon.noname-c con 0x7f57b813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f57b8142650 con 0x7f57b80a5bd0 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57b80a6b00 con 0x7f57b80a50e0 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b81427d0 con 0x7f57b813d2f0 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:37.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57bdb56640 10 monclient(hunting): authenticate will time out at 1530.335485s 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 0x7f57b8138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b809f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 0x7f57b8138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:40770/0 (socket says 172.21.15.86:40770) 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 1 -- 172.21.15.86:0/1748837092 learned_addr learned my addr 172.21.15.86:0/1748837092 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): get_auth_request con 0x7f57b80a5bd0 auth_method 0 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): get_auth_request con 0x7f57b813d2f0 auth_method 0 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b77fe640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b80a50e0 msgr2=0x7f57b81cfb50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b77fe640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b80a50e0 0x7f57b81cfb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient(hunting): handle_auth_done global_id 104168 payload 824 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b80a50e0 msgr2=0x7f57b81cfb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b80a50e0 0x7f57b81cfb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 msgr2=0x7f57b8138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 0x7f57b8138240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 monclient: found mon.noname-c 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b7fff640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b8122bd0 con 0x7f57b813d2f0 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.364+0000 7f57b6ffd640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b80a5bd0 0x7f57b8138240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b7fff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 msgr2=0x7f57b809f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b7fff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b809f7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57b00ab000 tx=0x7f57b0019660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00bc070 con 0x7f57b813d2f0 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: dump: 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: _finish_auth 0 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b00b8850 con 0x7f57b813d2f0 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 5 monclient: authenticate success, global_id 104168 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a5600 con 0x7f57b813d2f0 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 10 monclient: dump: 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b5ffb640 4 set_mon_vals no callback set 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 monclient: _renew_subs 2024-09-17T15:36:37.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b80a1260 con 0x7f57b813d2f0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 monclient: _renew_subs 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b80a18d0 con 0x7f57b813d2f0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 0x7f57b809fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b806a0e0 con 0x7f57b8138780 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 0x7f57b809fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient: get_auth_request con 0x7f57b8138780 auth_method 0 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f57b00b8cc0 con 0x7f57b813d2f0 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 10 monclient(hunting): handle_auth_done global_id 104171 payload 824 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f57b77fe640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57b809cd60 con 0x7f57b8138780 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f57a403bb10 0x7f57a403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.368+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(300..300 src has 213..300) ==== 4815+0+0 (secure 0 0 0) 0x7f57b00eaab0 con 0x7f57b813d2f0 2024-09-17T15:36:37.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f57b6ffd640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f57a403bb10 0x7f57a403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f57b6ffd640 10 monclient: get_auth_request con 0x7f57a403bb10 auth_method 0 2024-09-17T15:36:37.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f57b77fe640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 msgr2=0x7f57b809fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f57b77fe640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 0x7f57b809fce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00b450 tx=0x7f57ac00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac005270 con 0x7f57b8138780 2024-09-17T15:36:37.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 10 monclient: dump: 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f57ac015070 con 0x7f57b8138780 2024-09-17T15:36:37.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 msgr2=0x7f57b809fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 0x7f57b809fce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00b450 tx=0x7f57ac00b910 comp rx=0 tx=0).stop 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 msgr2=0x7f57b809fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f579ffff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b8138780 0x7f57b809fce0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.372+0000 7f57b6ffd640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f57a403bb10 0x7f57a403dfd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57b8120e90 tx=0x7f57a8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 0x7f57b81392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b80a0630 con 0x7f57b80a0220 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 0x7f57b81392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:37.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient: get_auth_request con 0x7f57b80a0220 auth_method 0 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 10 monclient(hunting): handle_auth_done global_id 104174 payload 824 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 1 -- 172.21.15.86:0/1748837092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57b8120f30 con 0x7f57b80a0220 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 msgr2=0x7f57b81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b77fe640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 0x7f57b81392d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00b450 tx=0x7f57ac0059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac010690 con 0x7f57b80a0220 2024-09-17T15:36:37.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 10 monclient: dump: 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:37.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f57ac010690 con 0x7f57b80a0220 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 msgr2=0x7f57b81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 0x7f57b81392d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00b450 tx=0x7f57ac0059e0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 msgr2=0x7f57b81392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f579ffff640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b80a0220 0x7f57b81392d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:36:37.396 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 92, 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28255, 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:36:37.397 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:36:37.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:36:37.400 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:36:37.401 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:36:37.402 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.404 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:36:37.405 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:36:37.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:36:37.407 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.408 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:36:37.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:36:37.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:36:37.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:36:37.410 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:36:37.410 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:36:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f57a403bb10 msgr2=0x7f57a403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f57a403bb10 0x7f57a403dfd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f57b8120e90 tx=0x7f57a8009190 comp rx=0 tx=0).stop 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 10 monclient: shutdown 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 msgr2=0x7f57b809f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b809f7a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57b00ab000 tx=0x7f57b0019660 comp rx=0 tx=0).stop 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57b7fff640 1 -- 172.21.15.86:0/1748837092 reap_dead start 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 shutdown_connections 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 --2- 172.21.15.86:0/1748837092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b813d2f0 0x7f57b809f7a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.380+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 >> 172.21.15.86:0/1748837092 conn(0x7f57b80aaae0 msgr2=0x7f57b813dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.384+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 shutdown_connections 2024-09-17T15:36:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:37.384+0000 7f57bdb56640 1 -- 172.21.15.86:0/1748837092 wait complete. 2024-09-17T15:36:37.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:36:37.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:36:37.763 INFO:tasks.rados.rados.0.smithi203.stdout:1285: done (2 left) 2024-09-17T15:36:37.763 INFO:tasks.rados.rados.0.smithi203.stdout:1286: write (excl) oid 23 current snap is 141 2024-09-17T15:36:37.763 INFO:tasks.rados.rados.0.smithi203.stdout:1286: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-17T15:36:37.778 INFO:tasks.rados.rados.0.smithi203.stdout:1286: writing smithi20328086-23 from 648869 to 1287685 tid 1 2024-09-17T15:36:37.793 INFO:tasks.rados.rados.0.smithi203.stdout:1286: writing smithi20328086-23 from 1828032 to 2615860 tid 2 2024-09-17T15:36:37.804 INFO:tasks.rados.rados.0.smithi203.stdout:1286: writing smithi20328086-23 from 3141714 to 3673744 tid 3 2024-09-17T15:36:37.804 INFO:tasks.rados.rados.0.smithi203.stdout:1287: copy_from oid 21 from oid 32 current snap is 141 2024-09-17T15:36:37.805 INFO:tasks.rados.rados.0.smithi203.stdout:1282: expect (ObjNum 394 snap 110 seq_num 394) 2024-09-17T15:36:37.889 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2476538 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: finishing write tid 1 to smithi20328086-28 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: finishing write tid 2 to smithi20328086-28 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: finishing write tid 3 to smithi20328086-28 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: finishing write tid 4 to smithi20328086-28 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: finishing write tid 5 to smithi20328086-28 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: oid 28 updating version 0 to 537 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: oid 28 updating version 537 to 538 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: oid 28 updating version 538 to 539 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: oid 28 updating version 539 to 540 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 540 (ObjNum 466 snap 141 seq_num 466) dirty exists 2024-09-17T15:36:37.909 INFO:tasks.rados.rados.0.smithi203.stdout:1283: left oid 28 (ObjNum 466 snap 141 seq_num 466) 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: finishing write tid 1 to smithi20328086-23 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: finishing write tid 2 to smithi20328086-23 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: finishing write tid 3 to smithi20328086-23 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: finishing write tid 4 to smithi20328086-23 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: finishing write tid 5 to smithi20328086-23 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: oid 23 updating version 0 to 468 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: oid 23 updating version 468 to 469 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: oid 23 updating version 469 to 470 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: oid 23 updating version 470 to 471 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 471 (ObjNum 467 snap 141 seq_num 467) dirty exists 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1286: left oid 23 (ObjNum 467 snap 141 seq_num 467) 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1287: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:36:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:1287: finishing copy_from to smithi20328086-21 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 303 (ObjNum 432 snap 126 seq_num 432) dirty exists 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:1282: done (3 left) 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:1283: done (2 left) 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:1286: done (1 left) 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:1287: done (0 left) 2024-09-17T15:36:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:1288: snap_remove snap 121 2024-09-17T15:36:38.762 INFO:tasks.rados.rados.0.smithi203.stdout:1288: done (0 left) 2024-09-17T15:36:38.762 INFO:tasks.rados.rados.0.smithi203.stdout:1289: rollback oid 11 current snap is 141 2024-09-17T15:36:38.762 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 120 2024-09-17T15:36:38.762 INFO:tasks.rados.rados.0.smithi203.stdout:1290: write oid 33 current snap is 141 2024-09-17T15:36:38.762 INFO:tasks.rados.rados.0.smithi203.stdout:1290: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-17T15:36:38.786 INFO:tasks.rados.rados.0.smithi203.stdout:1290: writing smithi20328086-33 from 665676 to 1379741 tid 1 2024-09-17T15:36:38.801 INFO:tasks.rados.rados.0.smithi203.stdout:1290: writing smithi20328086-33 from 1886514 to 2418000 tid 2 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1290: writing smithi20328086-33 from 3169137 to 3865676 tid 3 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1291: read oid 12 snap -1 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1291: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1292: snap_create 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1289: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1289: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1289: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 550 (ObjNum 403 snap 113 seq_num 403) dirty exists 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1290: finishing write tid 1 to smithi20328086-33 2024-09-17T15:36:38.821 INFO:tasks.rados.rados.0.smithi203.stdout:1290: finishing write tid 2 to smithi20328086-33 2024-09-17T15:36:38.828 INFO:tasks.rados.rados.0.smithi203.stdout:1290: finishing write tid 3 to smithi20328086-33 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: finishing write tid 4 to smithi20328086-33 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: finishing write tid 5 to smithi20328086-33 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: oid 33 updating version 0 to 295 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: oid 33 updating version 295 to 296 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: oid 33 updating version 296 to 298 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: oid 33 version 298 is already newer than 297 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 298 (ObjNum 468 snap 141 seq_num 468) dirty exists 2024-09-17T15:36:38.829 INFO:tasks.rados.rados.0.smithi203.stdout:1290: left oid 33 (ObjNum 468 snap 141 seq_num 468) 2024-09-17T15:36:38.835 INFO:tasks.rados.rados.0.smithi203.stdout:1291: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:38.835 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:38.911 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:38.961 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1289: done (3 left) 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1290: done (2 left) 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1291: done (1 left) 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1292: done (0 left) 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1293: write (excl) oid 12 current snap is 142 2024-09-17T15:36:39.678 INFO:tasks.rados.rados.0.smithi203.stdout:1293: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-17T15:36:39.694 INFO:tasks.rados.rados.0.smithi203.stdout:1293: writing smithi20328086-12 from 682483 to 1471797 tid 1 2024-09-17T15:36:39.708 INFO:tasks.rados.rados.0.smithi203.stdout:1293: writing smithi20328086-12 from 1944996 to 2736493 tid 2 2024-09-17T15:36:39.720 INFO:tasks.rados.rados.0.smithi203.stdout:1293: writing smithi20328086-12 from 3196560 to 3751134 tid 3 2024-09-17T15:36:39.720 INFO:tasks.rados.rados.0.smithi203.stdout:1294: read oid 34 snap -1 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1294: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1295: rollback oid 42 current snap is 142 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 141 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1293: finishing write tid 1 to smithi20328086-12 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1293: finishing write tid 2 to smithi20328086-12 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1296: read oid 41 snap -1 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1296: expect deleted 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1297: rollback oid 8 current snap is 142 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 138 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1298: rollback oid 31 current snap is 142 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 138 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:1299: rollback oid 20 current snap is 142 2024-09-17T15:36:39.721 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 139 2024-09-17T15:36:39.722 INFO:tasks.rados.rados.0.smithi203.stdout:1300: write oid 16 current snap is 142 2024-09-17T15:36:39.722 INFO:tasks.rados.rados.0.smithi203.stdout:1300: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-17T15:36:39.733 INFO:tasks.rados.rados.0.smithi203.stdout:1300: writing smithi20328086-16 from 699290 to 1163853 tid 1 2024-09-17T15:36:39.745 INFO:tasks.rados.rados.0.smithi203.stdout:1300: writing smithi20328086-16 from 1603478 to 2138633 tid 2 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:1300: writing smithi20328086-16 from 2823983 to 3589829 tid 3 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:1301: snap_create 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:1299: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:1293: finishing write tid 3 to smithi20328086-12 2024-09-17T15:36:39.761 INFO:tasks.rados.rados.0.smithi203.stdout:1293: finishing write tid 4 to smithi20328086-12 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: finishing write tid 5 to smithi20328086-12 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: oid 12 updating version 0 to 289 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: oid 12 updating version 289 to 290 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: oid 12 updating version 290 to 291 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: oid 12 version 291 is already newer than 288 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 291 (ObjNum 469 snap 142 seq_num 469) dirty exists 2024-09-17T15:36:39.762 INFO:tasks.rados.rados.0.smithi203.stdout:1293: left oid 12 (ObjNum 469 snap 142 seq_num 469) 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:1298: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:1298: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:1298: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 158 (ObjNum 439 snap 131 seq_num 439) dirty exists 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:1297: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:1297: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:36:39.763 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 364 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:36:39.764 INFO:tasks.rados.rados.0.smithi203.stdout:1300: finishing write tid 1 to smithi20328086-16 2024-09-17T15:36:39.764 INFO:tasks.rados.rados.0.smithi203.stdout:1295: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:36:39.764 INFO:tasks.rados.rados.0.smithi203.stdout:1294: expect (ObjNum 377 snap 108 seq_num 377) 2024-09-17T15:36:39.764 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:39.845 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:39.887 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1295: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1295: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 426 (ObjNum 452 snap 137 seq_num 452) dirty exists 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: finishing write tid 2 to smithi20328086-16 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: finishing write tid 3 to smithi20328086-16 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: finishing write tid 4 to smithi20328086-16 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: finishing write tid 5 to smithi20328086-16 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: oid 16 updating version 0 to 487 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: oid 16 version 487 is already newer than 485 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: oid 16 updating version 487 to 488 2024-09-17T15:36:39.925 INFO:tasks.rados.rados.0.smithi203.stdout:1300: oid 16 version 488 is already newer than 486 2024-09-17T15:36:39.926 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 488 (ObjNum 470 snap 142 seq_num 470) dirty exists 2024-09-17T15:36:39.926 INFO:tasks.rados.rados.0.smithi203.stdout:1300: left oid 16 (ObjNum 470 snap 142 seq_num 470) 2024-09-17T15:36:40.773 INFO:tasks.rados.rados.0.smithi203.stdout:1293: done (8 left) 2024-09-17T15:36:40.773 INFO:tasks.rados.rados.0.smithi203.stdout:1294: done (7 left) 2024-09-17T15:36:40.773 INFO:tasks.rados.rados.0.smithi203.stdout:1295: done (6 left) 2024-09-17T15:36:40.773 INFO:tasks.rados.rados.0.smithi203.stdout:1296: done (5 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1297: done (4 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1298: done (3 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1299: done (2 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1300: done (1 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1301: done (0 left) 2024-09-17T15:36:40.774 INFO:tasks.rados.rados.0.smithi203.stdout:1302: snap_create 2024-09-17T15:36:41.776 INFO:tasks.rados.rados.0.smithi203.stdout:1302: done (0 left) 2024-09-17T15:36:41.776 INFO:tasks.rados.rados.0.smithi203.stdout:1303: write oid 7 current snap is 144 2024-09-17T15:36:41.776 INFO:tasks.rados.rados.0.smithi203.stdout:1303: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-17T15:36:41.790 INFO:tasks.rados.rados.0.smithi203.stdout:1303: writing smithi20328086-7 from 716097 to 1255909 tid 1 2024-09-17T15:36:41.806 INFO:tasks.rados.rados.0.smithi203.stdout:1303: writing smithi20328086-7 from 1945607 to 2740773 tid 2 2024-09-17T15:36:41.819 INFO:tasks.rados.rados.0.smithi203.stdout:1303: writing smithi20328086-7 from 3535053 to 3916097 tid 3 2024-09-17T15:36:41.819 INFO:tasks.rados.rados.0.smithi203.stdout:1304: snap_remove snap 141 2024-09-17T15:36:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:1304: done (1 left) 2024-09-17T15:36:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:1305: snap_remove snap 143 2024-09-17T15:36:43.799 INFO:tasks.rados.rados.0.smithi203.stdout:1305: done (1 left) 2024-09-17T15:36:43.799 INFO:tasks.rados.rados.0.smithi203.stdout:1306: write (excl) oid 41 current snap is 144 2024-09-17T15:36:43.799 INFO:tasks.rados.rados.0.smithi203.stdout:1303: finishing write tid 1 to smithi20328086-7 2024-09-17T15:36:43.799 INFO:tasks.rados.rados.0.smithi203.stdout:1306: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-17T15:36:43.823 INFO:tasks.rados.rados.0.smithi203.stdout:1306: writing smithi20328086-41 from 732904 to 1464318 tid 1 2024-09-17T15:36:43.842 INFO:tasks.rados.rados.0.smithi203.stdout:1306: writing smithi20328086-41 from 2120442 to 2659266 tid 2 2024-09-17T15:36:43.858 INFO:tasks.rados.rados.0.smithi203.stdout:1306: writing smithi20328086-41 from 3278829 to 3783572 tid 3 2024-09-17T15:36:43.858 INFO:tasks.rados.rados.0.smithi203.stdout:1307: delete oid 27 current snap is 144 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: finishing write tid 2 to smithi20328086-7 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: finishing write tid 3 to smithi20328086-7 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: finishing write tid 4 to smithi20328086-7 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: finishing write tid 5 to smithi20328086-7 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: oid 7 updating version 0 to 440 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: oid 7 updating version 440 to 442 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: oid 7 version 442 is already newer than 441 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: oid 7 version 442 is already newer than 439 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 442 (ObjNum 471 snap 144 seq_num 471) dirty exists 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1303: left oid 7 (ObjNum 471 snap 144 seq_num 471) 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1306: finishing write tid 1 to smithi20328086-41 2024-09-17T15:36:43.859 INFO:tasks.rados.rados.0.smithi203.stdout:1306: finishing write tid 2 to smithi20328086-41 2024-09-17T15:36:43.864 INFO:tasks.rados.rados.0.smithi203.stdout:1306: finishing write tid 3 to smithi20328086-41 2024-09-17T15:36:43.865 INFO:tasks.rados.rados.0.smithi203.stdout:1306: finishing write tid 4 to smithi20328086-41 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: finishing write tid 5 to smithi20328086-41 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: oid 41 updating version 0 to 275 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: oid 41 version 275 is already newer than 272 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: oid 41 version 275 is already newer than 273 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: oid 41 version 275 is already newer than 274 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 275 (ObjNum 472 snap 144 seq_num 472) dirty exists 2024-09-17T15:36:43.866 INFO:tasks.rados.rados.0.smithi203.stdout:1306: left oid 41 (ObjNum 472 snap 144 seq_num 472) 2024-09-17T15:36:43.874 INFO:tasks.rados.rados.0.smithi203.stdout:1303: done (2 left) 2024-09-17T15:36:43.874 INFO:tasks.rados.rados.0.smithi203.stdout:1306: done (1 left) 2024-09-17T15:36:43.874 INFO:tasks.rados.rados.0.smithi203.stdout:1307: done (0 left) 2024-09-17T15:36:43.874 INFO:tasks.rados.rados.0.smithi203.stdout:1308: write oid 48 current snap is 144 2024-09-17T15:36:43.874 INFO:tasks.rados.rados.0.smithi203.stdout:1308: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-09-17T15:36:43.889 INFO:tasks.rados.rados.0.smithi203.stdout:1308: writing smithi20328086-48 from 766518 to 1248430 tid 1 2024-09-17T15:36:43.902 INFO:tasks.rados.rados.0.smithi203.stdout:1308: writing smithi20328086-48 from 1837406 to 2379899 tid 2 2024-09-17T15:36:43.914 INFO:tasks.rados.rados.0.smithi203.stdout:1308: writing smithi20328086-48 from 2933675 to 3460962 tid 3 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1308: writing smithi20328086-48 from 3876584 to 3966518 tid 4 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1309: write (excl) oid 29 current snap is 144 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 1 to smithi20328086-48 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 2 to smithi20328086-48 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 3 to smithi20328086-48 2024-09-17T15:36:43.920 INFO:tasks.rados.rados.0.smithi203.stdout:1309: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-17T15:36:43.932 INFO:tasks.rados.rados.0.smithi203.stdout:1309: writing smithi20328086-29 from 783325 to 1340486 tid 1 2024-09-17T15:36:43.943 INFO:tasks.rados.rados.0.smithi203.stdout:1309: writing smithi20328086-29 from 1779535 to 2465686 tid 2 2024-09-17T15:36:43.954 INFO:tasks.rados.rados.0.smithi203.stdout:1309: writing smithi20328086-29 from 3244745 to 3699657 tid 3 2024-09-17T15:36:43.954 INFO:tasks.rados.rados.0.smithi203.stdout:1310: snap_create 2024-09-17T15:36:43.954 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 4 to smithi20328086-48 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 5 to smithi20328086-48 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: finishing write tid 6 to smithi20328086-48 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: oid 48 updating version 0 to 493 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: oid 48 updating version 493 to 495 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: oid 48 version 495 is already newer than 491 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: oid 48 version 495 is already newer than 494 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: oid 48 version 495 is already newer than 492 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 495 (ObjNum 474 snap 144 seq_num 474) dirty exists 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1308: left oid 48 (ObjNum 474 snap 144 seq_num 474) 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1309: finishing write tid 1 to smithi20328086-29 2024-09-17T15:36:43.955 INFO:tasks.rados.rados.0.smithi203.stdout:1309: finishing write tid 2 to smithi20328086-29 2024-09-17T15:36:43.959 INFO:tasks.rados.rados.0.smithi203.stdout:1309: finishing write tid 3 to smithi20328086-29 2024-09-17T15:36:43.959 INFO:tasks.rados.rados.0.smithi203.stdout:1309: finishing write tid 4 to smithi20328086-29 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: finishing write tid 5 to smithi20328086-29 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: oid 29 updating version 0 to 455 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: oid 29 version 455 is already newer than 452 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: oid 29 version 455 is already newer than 454 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: oid 29 version 455 is already newer than 453 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 455 (ObjNum 475 snap 144 seq_num 475) dirty exists 2024-09-17T15:36:43.960 INFO:tasks.rados.rados.0.smithi203.stdout:1309: left oid 29 (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1308: done (2 left) 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1309: done (1 left) 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1310: done (0 left) 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1311: read oid 8 snap -1 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1311: expect deleted 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1312: copy_from oid 19 from oid 23 current snap is 145 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1313: rollback oid 26 current snap is 145 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 120 2024-09-17T15:36:44.680 INFO:tasks.rados.rados.0.smithi203.stdout:1314: write oid 41 current snap is 145 2024-09-17T15:36:44.681 INFO:tasks.rados.rados.0.smithi203.stdout:1314: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-17T15:36:44.688 INFO:tasks.rados.rados.0.smithi203.stdout:1314: writing smithi20328086-41 from 400132 to 1032542 tid 1 2024-09-17T15:36:44.696 INFO:tasks.rados.rados.0.smithi203.stdout:1314: writing smithi20328086-41 from 1438017 to 1984179 tid 2 2024-09-17T15:36:44.696 INFO:tasks.rados.rados.0.smithi203.stdout:1315: read oid 43 snap -1 2024-09-17T15:36:44.696 INFO:tasks.rados.rados.0.smithi203.stdout:1315: expect (ObjNum 435 snap 127 seq_num 435) 2024-09-17T15:36:44.696 INFO:tasks.rados.rados.0.smithi203.stdout:1316: write oid 2 current snap is 145 2024-09-17T15:36:44.696 INFO:tasks.rados.rados.0.smithi203.stdout:1316: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-17T15:36:44.705 INFO:tasks.rados.rados.0.smithi203.stdout:1316: writing smithi20328086-2 from 416939 to 1124598 tid 1 2024-09-17T15:36:44.713 INFO:tasks.rados.rados.0.smithi203.stdout:1316: writing smithi20328086-2 from 1896499 to 2016939 tid 2 2024-09-17T15:36:44.713 INFO:tasks.rados.rados.0.smithi203.stdout:1317: copy_from oid 34 from oid 16 current snap is 145 2024-09-17T15:36:44.713 INFO:tasks.rados.rados.0.smithi203.stdout:1313: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1313: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1313: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 558 (ObjNum 370 snap 105 seq_num 370) dirty exists 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: finishing write tid 1 to smithi20328086-41 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: finishing write tid 2 to smithi20328086-41 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: finishing write tid 3 to smithi20328086-41 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: finishing write tid 4 to smithi20328086-41 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: oid 41 updating version 0 to 278 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: oid 41 version 278 is already newer than 276 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: oid 41 version 278 is already newer than 277 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 278 (ObjNum 476 snap 145 seq_num 476) dirty exists 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1314: left oid 41 (ObjNum 476 snap 145 seq_num 476) 2024-09-17T15:36:44.714 INFO:tasks.rados.rados.0.smithi203.stdout:1311: done (6 left) 2024-09-17T15:36:44.715 INFO:tasks.rados.rados.0.smithi203.stdout:1313: done (5 left) 2024-09-17T15:36:44.715 INFO:tasks.rados.rados.0.smithi203.stdout:1314: done (4 left) 2024-09-17T15:36:44.715 INFO:tasks.rados.rados.0.smithi203.stdout:1318: write oid 30 current snap is 145 2024-09-17T15:36:44.715 INFO:tasks.rados.rados.0.smithi203.stdout:1318: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-17T15:36:44.723 INFO:tasks.rados.rados.0.smithi203.stdout:1318: writing smithi20328086-30 from 433746 to 1216654 tid 1 2024-09-17T15:36:44.730 INFO:tasks.rados.rados.0.smithi203.stdout:1318: writing smithi20328086-30 from 1954981 to 2033746 tid 2 2024-09-17T15:36:44.730 INFO:tasks.rados.rados.0.smithi203.stdout:1319: write (excl) oid 20 current snap is 145 2024-09-17T15:36:44.731 INFO:tasks.rados.rados.0.smithi203.stdout:1319: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-17T15:36:44.738 INFO:tasks.rados.rados.0.smithi203.stdout:1319: writing smithi20328086-20 from 450553 to 1025063 tid 1 2024-09-17T15:36:44.746 INFO:tasks.rados.rados.0.smithi203.stdout:1319: writing smithi20328086-20 from 1613463 to 2050553 tid 2 2024-09-17T15:36:44.746 INFO:tasks.rados.rados.0.smithi203.stdout:1320: delete oid 50 current snap is 145 2024-09-17T15:36:44.746 INFO:tasks.rados.rados.0.smithi203.stdout:1312: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:36:44.746 INFO:tasks.rados.rados.0.smithi203.stdout:1316: finishing write tid 1 to smithi20328086-2 2024-09-17T15:36:44.746 INFO:tasks.rados.rados.0.smithi203.stdout:1315: expect (ObjNum 435 snap 127 seq_num 435) 2024-09-17T15:36:44.801 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2092831 2024-09-17T15:36:44.848 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 143314 2024-09-17T15:36:44.971 INFO:tasks.rados.rados.0.smithi203.stdout:1316: finishing write tid 2 to smithi20328086-2 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: finishing write tid 3 to smithi20328086-2 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: finishing write tid 4 to smithi20328086-2 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: oid 2 updating version 0 to 445 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: oid 2 version 445 is already newer than 444 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: oid 2 version 445 is already newer than 443 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 445 (ObjNum 477 snap 145 seq_num 477) dirty exists 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1316: left oid 2 (ObjNum 477 snap 145 seq_num 477) 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1312: finishing copy_from to smithi20328086-19 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 471 (ObjNum 467 snap 141 seq_num 467) dirty exists 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1318: finishing write tid 1 to smithi20328086-30 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1318: finishing write tid 2 to smithi20328086-30 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1318: finishing write tid 3 to smithi20328086-30 2024-09-17T15:36:44.972 INFO:tasks.rados.rados.0.smithi203.stdout:1318: finishing write tid 4 to smithi20328086-30 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1318: oid 30 updating version 0 to 297 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1318: oid 30 version 297 is already newer than 295 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1318: oid 30 version 297 is already newer than 296 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 297 (ObjNum 478 snap 145 seq_num 478) dirty exists 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1318: left oid 30 (ObjNum 478 snap 145 seq_num 478) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1319: finishing write tid 1 to smithi20328086-20 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1312: done (6 left) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1315: done (5 left) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1316: done (4 left) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1318: done (3 left) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1320: done (2 left) 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1321: copy_from oid 17 from oid 50 current snap is 145 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1319: finishing write tid 2 to smithi20328086-20 2024-09-17T15:36:44.973 INFO:tasks.rados.rados.0.smithi203.stdout:1319: finishing write tid 3 to smithi20328086-20 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: finishing write tid 4 to smithi20328086-20 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: oid 20 updating version 0 to 477 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: oid 20 version 477 is already newer than 476 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: oid 20 updating version 477 to 478 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 478 (ObjNum 479 snap 145 seq_num 479) dirty exists 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: left oid 20 (ObjNum 479 snap 145 seq_num 479) 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1317: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1317: finishing copy_from to smithi20328086-34 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 488 (ObjNum 470 snap 142 seq_num 470) dirty exists 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1317: done (2 left) 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1319: done (1 left) 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1322: read oid 2 snap -1 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1322: expect (ObjNum 477 snap 145 seq_num 477) 2024-09-17T15:36:44.974 INFO:tasks.rados.rados.0.smithi203.stdout:1323: write (excl) oid 22 current snap is 145 2024-09-17T15:36:44.975 INFO:tasks.rados.rados.0.smithi203.stdout:1323: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-17T15:36:44.980 INFO:tasks.rados.rados.0.smithi203.stdout:1323: writing smithi20328086-22 from 484167 to 1209175 tid 1 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1323: writing smithi20328086-22 from 1730427 to 2084167 tid 2 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1324: read oid 5 snap -1 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1324: expect (ObjNum 335 snap 99 seq_num 335) 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1325: read oid 16 snap 144 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1325: expect (ObjNum 470 snap 142 seq_num 470) 2024-09-17T15:36:44.987 INFO:tasks.rados.rados.0.smithi203.stdout:1326: snap_remove snap 142 2024-09-17T15:36:45.813 INFO:tasks.rados.rados.0.smithi203.stdout:1326: done (5 left) 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1327: read oid 14 snap -1 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1327: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1328: read oid 49 snap -1 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1328: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1321: finishing copy_from to smithi20328086-17 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1321: got expected ENOENT (src dne) 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1321: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:36:45.814 INFO:tasks.rados.rados.0.smithi203.stdout:1322: expect (ObjNum 477 snap 145 seq_num 477) 2024-09-17T15:36:45.817 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1362577 2024-09-17T15:36:45.843 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 690102 2024-09-17T15:36:45.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:45.892 INFO:tasks.rados.rados.0.smithi203.stdout:1325: expect (ObjNum 470 snap 142 seq_num 470) 2024-09-17T15:36:45.966 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2517687 2024-09-17T15:36:46.046 INFO:tasks.rados.rados.0.smithi203.stdout:1324: expect (ObjNum 335 snap 99 seq_num 335) 2024-09-17T15:36:46.146 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2541633 2024-09-17T15:36:46.249 INFO:tasks.rados.rados.0.smithi203.stdout:1323: finishing write tid 1 to smithi20328086-22 2024-09-17T15:36:46.249 INFO:tasks.rados.rados.0.smithi203.stdout:1323: finishing write tid 2 to smithi20328086-22 2024-09-17T15:36:46.249 INFO:tasks.rados.rados.0.smithi203.stdout:1323: finishing write tid 3 to smithi20328086-22 2024-09-17T15:36:46.249 INFO:tasks.rados.rados.0.smithi203.stdout:1323: finishing write tid 4 to smithi20328086-22 2024-09-17T15:36:46.249 INFO:tasks.rados.rados.0.smithi203.stdout:1323: oid 22 updating version 0 to 163 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1323: oid 22 version 163 is already newer than 162 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1323: oid 22 updating version 163 to 164 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 164 (ObjNum 481 snap 145 seq_num 481) dirty exists 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1323: left oid 22 (ObjNum 481 snap 145 seq_num 481) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1321: done (6 left) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1322: done (5 left) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1323: done (4 left) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1324: done (3 left) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1325: done (2 left) 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1329: write (excl) oid 10 current snap is 145 2024-09-17T15:36:46.250 INFO:tasks.rados.rados.0.smithi203.stdout:1329: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-17T15:36:46.254 INFO:tasks.rados.rados.0.smithi203.stdout:1329: writing smithi20328086-10 from 500974 to 901231 tid 1 2024-09-17T15:36:46.260 INFO:tasks.rados.rados.0.smithi203.stdout:1329: writing smithi20328086-10 from 1388909 to 1946078 tid 2 2024-09-17T15:36:46.260 INFO:tasks.rados.rados.0.smithi203.stdout:1330: read oid 42 snap -1 2024-09-17T15:36:46.260 INFO:tasks.rados.rados.0.smithi203.stdout:1330: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:36:46.260 INFO:tasks.rados.rados.0.smithi203.stdout:1331: snap_create 2024-09-17T15:36:46.260 INFO:tasks.rados.rados.0.smithi203.stdout:1327: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:36:46.310 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2135801 2024-09-17T15:36:46.447 INFO:tasks.rados.rados.0.smithi203.stdout:1328: expect (ObjNum 277 snap 72 seq_num 277) 2024-09-17T15:36:46.503 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:46.559 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:46.614 INFO:tasks.rados.rados.0.smithi203.stdout:1330: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:36:46.679 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3182657 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: finishing write tid 1 to smithi20328086-10 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: finishing write tid 2 to smithi20328086-10 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: finishing write tid 3 to smithi20328086-10 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: finishing write tid 4 to smithi20328086-10 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: oid 10 updating version 0 to 502 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: oid 10 updating version 502 to 503 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:1329: oid 10 version 503 is already newer than 501 2024-09-17T15:36:46.729 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 503 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:36:46.730 INFO:tasks.rados.rados.0.smithi203.stdout:1329: left oid 10 (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:36:46.814 INFO:tasks.rados.rados.0.smithi203.stdout:1327: done (4 left) 2024-09-17T15:36:46.814 INFO:tasks.rados.rados.0.smithi203.stdout:1328: done (3 left) 2024-09-17T15:36:46.814 INFO:tasks.rados.rados.0.smithi203.stdout:1329: done (2 left) 2024-09-17T15:36:46.814 INFO:tasks.rados.rados.0.smithi203.stdout:1330: done (1 left) 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1331: done (0 left) 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1332: rollback oid 1 current snap is 146 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 138 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1333: read oid 31 snap -1 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1333: expect (ObjNum 439 snap 131 seq_num 439) 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1334: read oid 36 snap -1 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1334: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1335: rollback oid 34 current snap is 146 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 145 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1336: copy_from oid 2 from oid 8 current snap is 146 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1337: write (excl) oid 7 current snap is 146 2024-09-17T15:36:46.815 INFO:tasks.rados.rados.0.smithi203.stdout:1337: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-17T15:36:46.832 INFO:tasks.rados.rados.0.smithi203.stdout:1337: writing smithi20328086-7 from 517781 to 993287 tid 1 2024-09-17T15:36:46.851 INFO:tasks.rados.rados.0.smithi203.stdout:1337: writing smithi20328086-7 from 1731038 to 2117781 tid 2 2024-09-17T15:36:46.851 INFO:tasks.rados.rados.0.smithi203.stdout:1338: snap_create 2024-09-17T15:36:46.851 INFO:tasks.rados.rados.0.smithi203.stdout:1333: expect (ObjNum 439 snap 131 seq_num 439) 2024-09-17T15:36:46.892 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 632190 2024-09-17T15:36:46.954 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3255616 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1332: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1336: finishing copy_from to smithi20328086-2 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1336: got expected ENOENT (src dne) 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1335: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1332: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1336: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1335: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:1332: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:36:46.955 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 482 (ObjNum 448 snap 136 seq_num 448) dirty exists 2024-09-17T15:36:46.956 INFO:tasks.rados.rados.0.smithi203.stdout:1335: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:36:46.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 491 (ObjNum 470 snap 142 seq_num 470) dirty exists 2024-09-17T15:36:46.956 INFO:tasks.rados.rados.0.smithi203.stdout:1334: expect (ObjNum 405 snap 114 seq_num 405) 2024-09-17T15:36:47.066 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:47.124 INFO:tasks.rados.rados.0.smithi203.stdout:1337: finishing write tid 1 to smithi20328086-7 2024-09-17T15:36:47.124 INFO:tasks.rados.rados.0.smithi203.stdout:1337: finishing write tid 2 to smithi20328086-7 2024-09-17T15:36:47.124 INFO:tasks.rados.rados.0.smithi203.stdout:1337: finishing write tid 3 to smithi20328086-7 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:1337: finishing write tid 4 to smithi20328086-7 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:1337: oid 7 updating version 0 to 447 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:1337: oid 7 version 447 is already newer than 446 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:1337: oid 7 updating version 447 to 448 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 448 (ObjNum 483 snap 146 seq_num 483) dirty exists 2024-09-17T15:36:47.125 INFO:tasks.rados.rados.0.smithi203.stdout:1337: left oid 7 (ObjNum 483 snap 146 seq_num 483) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1332: done (6 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1333: done (5 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1334: done (4 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1335: done (3 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1336: done (2 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1337: done (1 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1338: done (0 left) 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1339: write oid 24 current snap is 147 2024-09-17T15:36:47.833 INFO:tasks.rados.rados.0.smithi203.stdout:1339: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-17T15:36:47.850 INFO:tasks.rados.rados.0.smithi203.stdout:1339: writing smithi20328086-24 from 534588 to 1085343 tid 1 2024-09-17T15:36:47.868 INFO:tasks.rados.rados.0.smithi203.stdout:1339: writing smithi20328086-24 from 1789520 to 2134588 tid 2 2024-09-17T15:36:47.868 INFO:tasks.rados.rados.0.smithi203.stdout:1340: snap_remove snap 145 2024-09-17T15:36:48.842 INFO:tasks.rados.rados.0.smithi203.stdout:1340: done (1 left) 2024-09-17T15:36:48.842 INFO:tasks.rados.rados.0.smithi203.stdout:1341: write (excl) oid 28 current snap is 147 2024-09-17T15:36:48.842 INFO:tasks.rados.rados.0.smithi203.stdout:1341: seq_num 485 ranges {551395=626004,1848002=303393} 2024-09-17T15:36:48.861 INFO:tasks.rados.rados.0.smithi203.stdout:1341: writing smithi20328086-28 from 551395 to 1177399 tid 1 2024-09-17T15:36:48.877 INFO:tasks.rados.rados.0.smithi203.stdout:1341: writing smithi20328086-28 from 1848002 to 2151395 tid 2 2024-09-17T15:36:48.877 INFO:tasks.rados.rados.0.smithi203.stdout:1342: write (excl) oid 37 current snap is 147 2024-09-17T15:36:48.878 INFO:tasks.rados.rados.0.smithi203.stdout:1342: seq_num 486 ranges {568202=701253,1906484=261718} 2024-09-17T15:36:48.897 INFO:tasks.rados.rados.0.smithi203.stdout:1342: writing smithi20328086-37 from 568202 to 1269455 tid 1 2024-09-17T15:36:48.908 INFO:tasks.rados.rados.0.smithi203.stdout:1342: writing smithi20328086-37 from 1906484 to 2168202 tid 2 2024-09-17T15:36:48.908 INFO:tasks.rados.rados.0.smithi203.stdout:1343: read oid 12 snap -1 2024-09-17T15:36:48.908 INFO:tasks.rados.rados.0.smithi203.stdout:1343: expect (ObjNum 469 snap 142 seq_num 469) 2024-09-17T15:36:48.908 INFO:tasks.rados.rados.0.smithi203.stdout:1344: snap_remove snap 120 2024-09-17T15:36:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1344: done (4 left) 2024-09-17T15:36:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1345: read oid 4 snap -1 2024-09-17T15:36:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1345: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1346: snap_create 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: finishing write tid 1 to smithi20328086-24 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: finishing write tid 2 to smithi20328086-24 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: finishing write tid 3 to smithi20328086-24 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: finishing write tid 4 to smithi20328086-24 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: oid 24 updating version 0 to 460 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: oid 24 version 460 is already newer than 459 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: oid 24 updating version 460 to 461 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 461 (ObjNum 484 snap 147 seq_num 484) dirty exists 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1339: left oid 24 (ObjNum 484 snap 147 seq_num 484) 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1341: finishing write tid 1 to smithi20328086-28 2024-09-17T15:36:49.687 INFO:tasks.rados.rados.0.smithi203.stdout:1341: finishing write tid 2 to smithi20328086-28 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: finishing write tid 3 to smithi20328086-28 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: finishing write tid 4 to smithi20328086-28 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: oid 28 updating version 0 to 562 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: oid 28 updating version 562 to 563 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: oid 28 updating version 563 to 564 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 564 (ObjNum 485 snap 147 seq_num 485) dirty exists 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: left oid 28 (ObjNum 485 snap 147 seq_num 485) 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: finishing write tid 1 to smithi20328086-37 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: finishing write tid 2 to smithi20328086-37 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: finishing write tid 3 to smithi20328086-37 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: finishing write tid 4 to smithi20328086-37 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: oid 37 updating version 0 to 500 2024-09-17T15:36:49.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: oid 37 updating version 500 to 501 2024-09-17T15:36:49.689 INFO:tasks.rados.rados.0.smithi203.stdout:1342: oid 37 updating version 501 to 502 2024-09-17T15:36:49.689 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 502 (ObjNum 486 snap 147 seq_num 486) dirty exists 2024-09-17T15:36:49.689 INFO:tasks.rados.rados.0.smithi203.stdout:1342: left oid 37 (ObjNum 486 snap 147 seq_num 486) 2024-09-17T15:36:49.689 INFO:tasks.rados.rados.0.smithi203.stdout:1343: expect (ObjNum 469 snap 142 seq_num 469) 2024-09-17T15:36:49.796 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1331869 2024-09-17T15:36:49.848 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:49.921 INFO:tasks.rados.rados.0.smithi203.stdout:1345: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1339: done (5 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1341: done (4 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1342: done (3 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1343: done (2 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1345: done (1 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1346: done (0 left) 2024-09-17T15:36:50.688 INFO:tasks.rados.rados.0.smithi203.stdout:1347: delete oid 15 current snap is 148 2024-09-17T15:36:50.693 INFO:tasks.rados.rados.0.smithi203.stdout:1347: done (0 left) 2024-09-17T15:36:50.693 INFO:tasks.rados.rados.0.smithi203.stdout:1348: delete oid 37 current snap is 148 2024-09-17T15:36:50.699 INFO:tasks.rados.rados.0.smithi203.stdout:1348: done (0 left) 2024-09-17T15:36:50.699 INFO:tasks.rados.rados.0.smithi203.stdout:1349: snap_remove snap 147 2024-09-17T15:36:51.694 INFO:tasks.rados.rados.0.smithi203.stdout:1349: done (0 left) 2024-09-17T15:36:51.694 INFO:tasks.rados.rados.0.smithi203.stdout:1350: read oid 23 snap -1 2024-09-17T15:36:51.694 INFO:tasks.rados.rados.0.smithi203.stdout:1350: expect (ObjNum 467 snap 141 seq_num 467) 2024-09-17T15:36:51.694 INFO:tasks.rados.rados.0.smithi203.stdout:1351: write (excl) oid 41 current snap is 148 2024-09-17T15:36:51.694 INFO:tasks.rados.rados.0.smithi203.stdout:1351: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-17T15:36:51.699 INFO:tasks.rados.rados.0.smithi203.stdout:1351: writing smithi20328086-41 from 618623 to 1261976 tid 1 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1351: writing smithi20328086-41 from 1681930 to 2218623 tid 2 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1352: read oid 16 snap 139 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1352: expect (ObjNum 436 snap 128 seq_num 436) 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1353: read oid 20 snap -1 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1353: expect (ObjNum 479 snap 145 seq_num 479) 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1354: read oid 6 snap -1 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1354: expect (ObjNum 457 snap 139 seq_num 457) 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1355: write (excl) oid 43 current snap is 148 2024-09-17T15:36:51.706 INFO:tasks.rados.rados.0.smithi203.stdout:1355: seq_num 490 ranges {635430=718602,2140412=95018} 2024-09-17T15:36:51.714 INFO:tasks.rados.rados.0.smithi203.stdout:1355: writing smithi20328086-43 from 635430 to 1354032 tid 1 2024-09-17T15:36:51.720 INFO:tasks.rados.rados.0.smithi203.stdout:1355: writing smithi20328086-43 from 2140412 to 2235430 tid 2 2024-09-17T15:36:51.720 INFO:tasks.rados.rados.0.smithi203.stdout:1356: read oid 4 snap -1 2024-09-17T15:36:51.720 INFO:tasks.rados.rados.0.smithi203.stdout:1356: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:51.720 INFO:tasks.rados.rados.0.smithi203.stdout:1357: rollback oid 25 current snap is 148 2024-09-17T15:36:51.720 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 138 2024-09-17T15:36:51.721 INFO:tasks.rados.rados.0.smithi203.stdout:1350: expect (ObjNum 467 snap 141 seq_num 467) 2024-09-17T15:36:51.728 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1765228 2024-09-17T15:36:51.822 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:51.862 INFO:tasks.rados.rados.0.smithi203.stdout:1353: expect (ObjNum 479 snap 145 seq_num 479) 2024-09-17T15:36:51.862 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:51.918 INFO:tasks.rados.rados.0.smithi203.stdout:1351: finishing write tid 1 to smithi20328086-41 2024-09-17T15:36:51.918 INFO:tasks.rados.rados.0.smithi203.stdout:1351: finishing write tid 2 to smithi20328086-41 2024-09-17T15:36:51.918 INFO:tasks.rados.rados.0.smithi203.stdout:1351: finishing write tid 3 to smithi20328086-41 2024-09-17T15:36:51.918 INFO:tasks.rados.rados.0.smithi203.stdout:1351: finishing write tid 4 to smithi20328086-41 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:1351: oid 41 updating version 0 to 279 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:1351: oid 41 updating version 279 to 280 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:1351: oid 41 updating version 280 to 281 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 281 (ObjNum 489 snap 148 seq_num 489) dirty exists 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:1351: left oid 41 (ObjNum 489 snap 148 seq_num 489) 2024-09-17T15:36:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:1352: expect (ObjNum 436 snap 128 seq_num 436) 2024-09-17T15:36:51.961 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:52.038 INFO:tasks.rados.rados.0.smithi203.stdout:1354: expect (ObjNum 457 snap 139 seq_num 457) 2024-09-17T15:36:52.072 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:52.110 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:52.148 INFO:tasks.rados.rados.0.smithi203.stdout:1356: expect (ObjNum 303 snap 84 seq_num 303) 2024-09-17T15:36:52.172 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2574575 2024-09-17T15:36:52.184 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: finishing write tid 1 to smithi20328086-43 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: finishing write tid 2 to smithi20328086-43 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: finishing write tid 3 to smithi20328086-43 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: finishing write tid 4 to smithi20328086-43 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: oid 43 updating version 0 to 282 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: oid 43 updating version 282 to 283 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:1355: oid 43 updating version 283 to 284 2024-09-17T15:36:52.275 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 284 (ObjNum 490 snap 148 seq_num 490) dirty exists 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1355: left oid 43 (ObjNum 490 snap 148 seq_num 490) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1357: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1357: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1357: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 475 (ObjNum 423 snap 123 seq_num 423) dirty exists 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1350: done (7 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1351: done (6 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1352: done (5 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1353: done (4 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1354: done (3 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1355: done (2 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1356: done (1 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1357: done (0 left) 2024-09-17T15:36:52.276 INFO:tasks.rados.rados.0.smithi203.stdout:1358: rollback oid 12 current snap is 148 2024-09-17T15:36:52.277 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 138 2024-09-17T15:36:52.277 INFO:tasks.rados.rados.0.smithi203.stdout:1359: snap_remove snap 139 2024-09-17T15:36:52.702 INFO:tasks.rados.rados.0.smithi203.stdout:1359: done (1 left) 2024-09-17T15:36:52.702 INFO:tasks.rados.rados.0.smithi203.stdout:1360: write (excl) oid 36 current snap is 148 2024-09-17T15:36:52.702 INFO:tasks.rados.rados.0.smithi203.stdout:1360: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-17T15:36:52.730 INFO:tasks.rados.rados.0.smithi203.stdout:1360: writing smithi20328086-36 from 652237 to 1446088 tid 1 2024-09-17T15:36:52.743 INFO:tasks.rados.rados.0.smithi203.stdout:1360: writing smithi20328086-36 from 2198894 to 2252237 tid 2 2024-09-17T15:36:52.743 INFO:tasks.rados.rados.0.smithi203.stdout:1361: write oid 21 current snap is 148 2024-09-17T15:36:52.743 INFO:tasks.rados.rados.0.smithi203.stdout:1361: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-17T15:36:52.759 INFO:tasks.rados.rados.0.smithi203.stdout:1361: writing smithi20328086-21 from 669044 to 1138144 tid 1 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1361: writing smithi20328086-21 from 1741023 to 2200184 tid 2 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1362: delete oid 39 current snap is 148 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1358: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1358: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1358: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 311 (ObjNum 444 snap 136 seq_num 444) dirty exists 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1360: finishing write tid 1 to smithi20328086-36 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1360: finishing write tid 2 to smithi20328086-36 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1360: finishing write tid 3 to smithi20328086-36 2024-09-17T15:36:52.772 INFO:tasks.rados.rados.0.smithi203.stdout:1360: finishing write tid 4 to smithi20328086-36 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:1360: oid 36 updating version 0 to 581 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:1360: oid 36 updating version 581 to 582 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:1360: oid 36 version 582 is already newer than 580 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 582 (ObjNum 491 snap 148 seq_num 491) dirty exists 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:1360: left oid 36 (ObjNum 491 snap 148 seq_num 491) 2024-09-17T15:36:52.773 INFO:tasks.rados.rados.0.smithi203.stdout:1361: finishing write tid 1 to smithi20328086-21 2024-09-17T15:36:52.779 INFO:tasks.rados.rados.0.smithi203.stdout:1361: finishing write tid 2 to smithi20328086-21 2024-09-17T15:36:52.780 INFO:tasks.rados.rados.0.smithi203.stdout:1361: finishing write tid 3 to smithi20328086-21 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:1361: finishing write tid 4 to smithi20328086-21 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:1361: oid 21 updating version 0 to 584 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:1361: oid 21 updating version 584 to 586 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:1361: oid 21 version 586 is already newer than 585 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 586 (ObjNum 492 snap 148 seq_num 492) dirty exists 2024-09-17T15:36:52.781 INFO:tasks.rados.rados.0.smithi203.stdout:1361: left oid 21 (ObjNum 492 snap 148 seq_num 492) 2024-09-17T15:36:52.782 INFO:tasks.rados.rados.0.smithi203.stdout:1358: done (3 left) 2024-09-17T15:36:52.782 INFO:tasks.rados.rados.0.smithi203.stdout:1360: done (2 left) 2024-09-17T15:36:52.782 INFO:tasks.rados.rados.0.smithi203.stdout:1361: done (1 left) 2024-09-17T15:36:52.782 INFO:tasks.rados.rados.0.smithi203.stdout:1362: done (0 left) 2024-09-17T15:36:52.783 INFO:tasks.rados.rados.0.smithi203.stdout:1363: write oid 46 current snap is 148 2024-09-17T15:36:52.783 INFO:tasks.rados.rados.0.smithi203.stdout:1363: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-17T15:36:52.795 INFO:tasks.rados.rados.0.smithi203.stdout:1363: writing smithi20328086-46 from 702658 to 1322256 tid 1 2024-09-17T15:36:52.805 INFO:tasks.rados.rados.0.smithi203.stdout:1363: writing smithi20328086-46 from 1857987 to 2302658 tid 2 2024-09-17T15:36:52.805 INFO:tasks.rados.rados.0.smithi203.stdout:1364: copy_from oid 16 from oid 4 current snap is 148 2024-09-17T15:36:52.805 INFO:tasks.rados.rados.0.smithi203.stdout:1365: write (excl) oid 33 current snap is 148 2024-09-17T15:36:52.805 INFO:tasks.rados.rados.0.smithi203.stdout:1365: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-17T15:36:52.816 INFO:tasks.rados.rados.0.smithi203.stdout:1365: writing smithi20328086-33 from 719465 to 1130665 tid 1 2024-09-17T15:36:52.830 INFO:tasks.rados.rados.0.smithi203.stdout:1365: writing smithi20328086-33 from 1632822 to 2319465 tid 2 2024-09-17T15:36:52.830 INFO:tasks.rados.rados.0.smithi203.stdout:1366: rollback oid 23 current snap is 148 2024-09-17T15:36:52.830 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 144 2024-09-17T15:36:52.830 INFO:tasks.rados.rados.0.smithi203.stdout:1363: finishing write tid 1 to smithi20328086-46 2024-09-17T15:36:52.830 INFO:tasks.rados.rados.0.smithi203.stdout:1367: write (excl) oid 3 current snap is 148 2024-09-17T15:36:52.831 INFO:tasks.rados.rados.0.smithi203.stdout:1367: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-17T15:36:52.840 INFO:tasks.rados.rados.0.smithi203.stdout:1367: writing smithi20328086-3 from 736272 to 1222721 tid 1 2024-09-17T15:36:52.849 INFO:tasks.rados.rados.0.smithi203.stdout:1367: writing smithi20328086-3 from 1974951 to 2336272 tid 2 2024-09-17T15:36:52.849 INFO:tasks.rados.rados.0.smithi203.stdout:1368: write oid 41 current snap is 148 2024-09-17T15:36:52.849 INFO:tasks.rados.rados.0.smithi203.stdout:1368: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-17T15:36:52.858 INFO:tasks.rados.rados.0.smithi203.stdout:1368: writing smithi20328086-41 from 753079 to 1314777 tid 1 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1368: writing smithi20328086-41 from 2033433 to 2353079 tid 2 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1369: delete oid 11 current snap is 148 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1363: finishing write tid 2 to smithi20328086-46 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1363: finishing write tid 3 to smithi20328086-46 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1363: finishing write tid 4 to smithi20328086-46 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1363: oid 46 updating version 0 to 446 2024-09-17T15:36:52.866 INFO:tasks.rados.rados.0.smithi203.stdout:1363: oid 46 version 446 is already newer than 445 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:1363: oid 46 version 446 is already newer than 444 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 446 (ObjNum 494 snap 148 seq_num 494) dirty exists 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:1363: left oid 46 (ObjNum 494 snap 148 seq_num 494) 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:1364: finishing copy_from racing read to smithi20328086-16 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:1365: finishing write tid 1 to smithi20328086-33 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:1364: finishing copy_from to smithi20328086-16 2024-09-17T15:36:52.867 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 269 (ObjNum 303 snap 84 seq_num 303) dirty exists 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1365: finishing write tid 2 to smithi20328086-33 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1366: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1365: finishing write tid 3 to smithi20328086-33 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1365: finishing write tid 4 to smithi20328086-33 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1365: oid 33 updating version 0 to 300 2024-09-17T15:36:52.868 INFO:tasks.rados.rados.0.smithi203.stdout:1365: oid 33 version 300 is already newer than 299 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1365: oid 33 updating version 300 to 301 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 301 (ObjNum 495 snap 148 seq_num 495) dirty exists 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1365: left oid 33 (ObjNum 495 snap 148 seq_num 495) 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1366: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: finishing write tid 1 to smithi20328086-3 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1366: finishing rollback tid 2 to smithi20328086-23 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 494 (ObjNum 467 snap 141 seq_num 467) dirty exists 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: finishing write tid 2 to smithi20328086-3 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: finishing write tid 3 to smithi20328086-3 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: finishing write tid 4 to smithi20328086-3 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: oid 3 updating version 0 to 448 2024-09-17T15:36:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:1367: oid 3 updating version 448 to 449 2024-09-17T15:36:52.870 INFO:tasks.rados.rados.0.smithi203.stdout:1367: oid 3 version 449 is already newer than 447 2024-09-17T15:36:52.870 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 449 (ObjNum 496 snap 148 seq_num 496) dirty exists 2024-09-17T15:36:52.870 INFO:tasks.rados.rados.0.smithi203.stdout:1367: left oid 3 (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:36:52.870 INFO:tasks.rados.rados.0.smithi203.stdout:1368: finishing write tid 1 to smithi20328086-41 2024-09-17T15:36:52.871 INFO:tasks.rados.rados.0.smithi203.stdout:1363: done (6 left) 2024-09-17T15:36:52.871 INFO:tasks.rados.rados.0.smithi203.stdout:1364: done (5 left) 2024-09-17T15:36:52.871 INFO:tasks.rados.rados.0.smithi203.stdout:1365: done (4 left) 2024-09-17T15:36:52.871 INFO:tasks.rados.rados.0.smithi203.stdout:1366: done (3 left) 2024-09-17T15:36:52.871 INFO:tasks.rados.rados.0.smithi203.stdout:1367: done (2 left) 2024-09-17T15:36:52.872 INFO:tasks.rados.rados.0.smithi203.stdout:1369: done (1 left) 2024-09-17T15:36:52.872 INFO:tasks.rados.rados.0.smithi203.stdout:1370: write oid 33 current snap is 148 2024-09-17T15:36:52.872 INFO:tasks.rados.rados.0.smithi203.stdout:1370: seq_num 499 ranges {786693=712196,2150397=236296} 2024-09-17T15:36:52.881 INFO:tasks.rados.rados.0.smithi203.stdout:1370: writing smithi20328086-33 from 786693 to 1498889 tid 1 2024-09-17T15:36:52.887 INFO:tasks.rados.rados.0.smithi203.stdout:1370: writing smithi20328086-33 from 2150397 to 2386693 tid 2 2024-09-17T15:36:52.887 INFO:tasks.rados.rados.0.smithi203.stdout:1371: snap_remove snap 138 2024-09-17T15:36:53.702 INFO:tasks.rados.rados.0.smithi203.stdout:1371: done (2 left) 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: finishing write tid 2 to smithi20328086-41 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: finishing write tid 3 to smithi20328086-41 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: finishing write tid 4 to smithi20328086-41 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: oid 41 updating version 0 to 286 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: oid 41 updating version 286 to 287 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: oid 41 version 287 is already newer than 285 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 287 (ObjNum 497 snap 148 seq_num 497) dirty exists 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1368: left oid 41 (ObjNum 497 snap 148 seq_num 497) 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1370: finishing write tid 1 to smithi20328086-33 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1370: finishing write tid 2 to smithi20328086-33 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1370: finishing write tid 3 to smithi20328086-33 2024-09-17T15:36:53.703 INFO:tasks.rados.rados.0.smithi203.stdout:1370: finishing write tid 4 to smithi20328086-33 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1370: oid 33 updating version 0 to 303 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1370: oid 33 version 303 is already newer than 302 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1370: oid 33 updating version 303 to 304 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 304 (ObjNum 499 snap 148 seq_num 499) dirty exists 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1370: left oid 33 (ObjNum 499 snap 148 seq_num 499) 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1372: read oid 48 snap -1 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1372: expect (ObjNum 474 snap 144 seq_num 474) 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1368: done (2 left) 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1370: done (1 left) 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1373: rollback oid 24 current snap is 148 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 146 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1374: rollback oid 2 current snap is 148 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 146 2024-09-17T15:36:53.704 INFO:tasks.rados.rados.0.smithi203.stdout:1375: read oid 12 snap -1 2024-09-17T15:36:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:1375: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:53.705 INFO:tasks.rados.rados.0.smithi203.stdout:1376: delete oid 11 current snap is 148 2024-09-17T15:36:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:1373: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:36:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:1373: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:36:53.719 INFO:tasks.rados.rados.0.smithi203.stdout:1372: expect (ObjNum 474 snap 144 seq_num 474) 2024-09-17T15:36:53.820 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1346047 2024-09-17T15:36:53.852 INFO:tasks.rados.rados.0.smithi203.stdout:1373: finishing rollback tid 2 to smithi20328086-24 2024-09-17T15:36:53.853 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 488 (ObjNum 446 snap 136 seq_num 446) dirty exists 2024-09-17T15:36:53.853 INFO:tasks.rados.rados.0.smithi203.stdout:1375: expect (ObjNum 444 snap 136 seq_num 444) 2024-09-17T15:36:53.853 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:36:54.000 INFO:tasks.rados.rados.0.smithi203.stdout:1374: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:36:54.000 INFO:tasks.rados.rados.0.smithi203.stdout:1374: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:36:54.000 INFO:tasks.rados.rados.0.smithi203.stdout:1374: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 451 (ObjNum 477 snap 145 seq_num 477) dirty exists 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1372: done (4 left) 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1373: done (3 left) 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1374: done (2 left) 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1375: done (1 left) 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1376: done (0 left) 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1377: read oid 8 snap -1 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1377: expect deleted 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1378: write (excl) oid 2 current snap is 148 2024-09-17T15:36:54.001 INFO:tasks.rados.rados.0.smithi203.stdout:1378: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-17T15:36:54.013 INFO:tasks.rados.rados.0.smithi203.stdout:1378: writing smithi20328086-2 from 420307 to 883001 tid 1 2024-09-17T15:36:54.029 INFO:tasks.rados.rados.0.smithi203.stdout:1378: writing smithi20328086-2 from 1351008 to 1968503 tid 2 2024-09-17T15:36:54.030 INFO:tasks.rados.rados.0.smithi203.stdout:1379: write (excl) oid 47 current snap is 148 2024-09-17T15:36:54.030 INFO:tasks.rados.rados.0.smithi203.stdout:1379: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-17T15:36:54.046 INFO:tasks.rados.rados.0.smithi203.stdout:1379: writing smithi20328086-47 from 437114 to 1091410 tid 1 2024-09-17T15:36:54.057 INFO:tasks.rados.rados.0.smithi203.stdout:1379: writing smithi20328086-47 from 1525843 to 2003349 tid 2 2024-09-17T15:36:54.058 INFO:tasks.rados.rados.0.smithi203.stdout:1380: snap_remove snap 144 2024-09-17T15:36:54.690 INFO:tasks.rados.rados.0.smithi203.stdout:1380: done (3 left) 2024-09-17T15:36:54.691 INFO:tasks.rados.rados.0.smithi203.stdout:1381: snap_remove snap 146 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1381: done (3 left) 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1382: snap_create 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: finishing write tid 1 to smithi20328086-2 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: finishing write tid 2 to smithi20328086-2 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: finishing write tid 3 to smithi20328086-2 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: finishing write tid 4 to smithi20328086-2 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: oid 2 updating version 0 to 453 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: oid 2 version 453 is already newer than 452 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:1378: oid 2 updating version 453 to 454 2024-09-17T15:36:55.686 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 454 (ObjNum 501 snap 148 seq_num 501) dirty exists 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1378: left oid 2 (ObjNum 501 snap 148 seq_num 501) 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: finishing write tid 1 to smithi20328086-47 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: finishing write tid 2 to smithi20328086-47 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: finishing write tid 3 to smithi20328086-47 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: finishing write tid 4 to smithi20328086-47 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: oid 47 updating version 0 to 375 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: oid 47 updating version 375 to 377 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: oid 47 version 377 is already newer than 376 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 377 (ObjNum 502 snap 148 seq_num 502) dirty exists 2024-09-17T15:36:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1379: left oid 47 (ObjNum 502 snap 148 seq_num 502) 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1377: done (3 left) 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1378: done (2 left) 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1379: done (1 left) 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1382: done (0 left) 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1383: write (excl) oid 5 current snap is 149 2024-09-17T15:36:56.693 INFO:tasks.rados.rados.0.smithi203.stdout:1383: seq_num 503 ranges {453921=729545,1584325=621164} 2024-09-17T15:36:56.712 INFO:tasks.rados.rados.0.smithi203.stdout:1383: writing smithi20328086-5 from 453921 to 1183466 tid 1 2024-09-17T15:36:56.729 INFO:tasks.rados.rados.0.smithi203.stdout:1383: writing smithi20328086-5 from 1584325 to 2205489 tid 2 2024-09-17T15:36:56.730 INFO:tasks.rados.rados.0.smithi203.stdout:1384: write (excl) oid 32 current snap is 149 2024-09-17T15:36:56.730 INFO:tasks.rados.rados.0.smithi203.stdout:1384: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-17T15:36:56.744 INFO:tasks.rados.rados.0.smithi203.stdout:1384: writing smithi20328086-32 from 470728 to 875522 tid 1 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1384: writing smithi20328086-32 from 1526454 to 2007629 tid 2 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1385: delete oid 28 current snap is 149 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: finishing write tid 1 to smithi20328086-5 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: finishing write tid 2 to smithi20328086-5 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: finishing write tid 3 to smithi20328086-5 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: finishing write tid 4 to smithi20328086-5 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: oid 5 updating version 0 to 306 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: oid 5 updating version 306 to 307 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:1383: oid 5 version 307 is already newer than 305 2024-09-17T15:36:56.758 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 307 (ObjNum 503 snap 149 seq_num 503) dirty exists 2024-09-17T15:36:56.759 INFO:tasks.rados.rados.0.smithi203.stdout:1383: left oid 5 (ObjNum 503 snap 149 seq_num 503) 2024-09-17T15:36:56.759 INFO:tasks.rados.rados.0.smithi203.stdout:1384: finishing write tid 1 to smithi20328086-32 2024-09-17T15:36:56.765 INFO:tasks.rados.rados.0.smithi203.stdout:1383: done (2 left) 2024-09-17T15:36:56.765 INFO:tasks.rados.rados.0.smithi203.stdout:1385: done (1 left) 2024-09-17T15:36:56.765 INFO:tasks.rados.rados.0.smithi203.stdout:1386: snap_create 2024-09-17T15:36:56.768 INFO:tasks.rados.rados.0.smithi203.stdout:1384: finishing write tid 2 to smithi20328086-32 2024-09-17T15:36:56.770 INFO:tasks.rados.rados.0.smithi203.stdout:1384: finishing write tid 3 to smithi20328086-32 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:1384: finishing write tid 4 to smithi20328086-32 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:1384: oid 32 updating version 0 to 306 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:1384: oid 32 version 306 is already newer than 305 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:1384: oid 32 updating version 306 to 307 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 307 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:36:56.771 INFO:tasks.rados.rados.0.smithi203.stdout:1384: left oid 32 (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:36:57.414 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:36:57.414 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:36:57.415 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:36:57.418 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:36:57.418 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:36:57.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:36:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: get_monmap_and_config 2024-09-17T15:36:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: build_initial_monmap 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: monmap: 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:36:57.549520+0000 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:36:57.549520+0000 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:36:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:36:57.548 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 Processor -- start 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 -- start start 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: init 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a4138280 addr v1:172.21.15.86:6789/0 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6a406fa90 0x7fd6a40a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a406fa90 addr v2:172.21.15.86:3300/0 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 0x7fd6a413f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a413d260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a40a4b30 con 0x7fd6a4138280 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd6a406f700 con 0x7fd6a406fa90 2024-09-17T15:36:57.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a406dd30 con 0x7fd6a413d260 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6aba0b640 10 monclient(hunting): authenticate will time out at 1550.514958s 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a97a7640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd6a4138280 0x7fd6a4138660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59310/0 (socket says 172.21.15.86:59310) 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 0x7fd6a413f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a97a7640 1 -- 172.21.15.86:0/1200199410 learned_addr learned my addr 172.21.15.86:0/1200199410 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6a406fa90 0x7fd6a40a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): get_auth_request con 0x7fd6a413d260 auth_method 0 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:57.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): get_auth_request con 0x7fd6a406fa90 auth_method 0 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:57.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_done global_id 94299 payload 274 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 -- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 msgr2=0x7fd6a413f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 0x7fd6a413f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 -- 172.21.15.86:0/1200199410 >> v1:172.21.15.86:6789/0 conn(0x7fd6a4138280 legacy=0x7fd6a4138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient: _finish_hunting 0 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 monclient: found mon.noname-a 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a8fa6640 1 -- 172.21.15.86:0/1200199410 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a406edd0 con 0x7fd6a406fa90 2024-09-17T15:36:57.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.544+0000 7fd6a9fa8640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 0x7fd6a413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6a8fa6640 1 -- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a406fa90 msgr2=0x7fd6a40a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6a8fa6640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a406fa90 0x7fd6a40a3e80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009d90 tx=0x7fd6a001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53792abda1e6af7f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 1 -- 172.21.15.86:0/1200199410 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00b2070 con 0x7fd6a406fa90 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: dump: 2024-09-17T15:36:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: _finish_auth 0 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:36:57.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 1 -- 172.21.15.86:0/1200199410 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a00a5890 con 0x7fd6a406fa90 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 5 monclient: authenticate success, global_id 94299 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: get_monmap_and_config success 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 1 -- 172.21.15.86:0/1200199410 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00a5c60 con 0x7fd6a406fa90 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 4 set_mon_vals no callback set 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd69bfff640 10 monclient: dump: 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:57.561 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: shutdown 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a406fa90 msgr2=0x7fd6a40a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a406fa90 0x7fd6a40a3e80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0009d90 tx=0x7fd6a001c830 comp rx=0 tx=0).stop 2024-09-17T15:36:57.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- 172.21.15.86:0/1200199410 shutdown_connections 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a413d260 0x7fd6a413f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/1200199410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a406fa90 0x7fd6a40a3e80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- 172.21.15.86:0/1200199410 >> 172.21.15.86:0/1200199410 conn(0x7fd6a40a8470 msgr2=0x7fd6a40a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- 172.21.15.86:0/1200199410 shutdown_connections 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- 172.21.15.86:0/1200199410 wait complete. 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: build_initial_monmap 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: monmap: 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:36:57.563 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 Processor -- start 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 1 -- start start 2024-09-17T15:36:57.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: init 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: _reopen_session rank -1 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.552+0000 7fd6aba0b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a4138280 0x7fd6a41ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a4138280 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 0x7fd6a41cd160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a413d260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a4074be0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:57.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4142800 con 0x7fd6a413d260 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4142680 con 0x7fd6a4138280 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a4142980 con 0x7fd6a4074be0 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): _renew_subs 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient(hunting): authenticate will time out at 1550.524072s 2024-09-17T15:36:57.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36852/0 (socket says 172.21.15.86:36852) 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 -- 172.21.15.86:0/2819354570 learned_addr learned my addr 172.21.15.86:0/2819354570 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): get_auth_request con 0x7fd6a4074be0 auth_method 0 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 0x7fd6a41cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): get_auth_request con 0x7fd6a413d260 auth_method 0 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:57.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a97a7640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a4138280 0x7fd6a41ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a97a7640 10 monclient(hunting): get_auth_request con 0x7fd6a4138280 auth_method 0 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a97a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a97a7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a97a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient(hunting): handle_auth_done global_id 104180 payload 824 2024-09-17T15:36:57.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 -- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a4138280 msgr2=0x7fd6a41ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a4138280 0x7fd6a41ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 -- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 msgr2=0x7fd6a41cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 0x7fd6a41cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient: _finish_hunting 0 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 monclient: found mon.noname-c 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 -- 172.21.15.86:0/2819354570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a4072f30 con 0x7fd6a4074be0 2024-09-17T15:36:57.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a8fa6640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 0x7fd6a41cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:36:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 -- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 msgr2=0x7fd6a41cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:36:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6a9fa8640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00e920 tx=0x7fd69c00ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c017070 con 0x7fd6a4074be0 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: dump: 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: _finish_auth 0 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:36:57.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd69c00cbb0 con 0x7fd6a4074be0 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 5 monclient: authenticate success, global_id 104180 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _renew_subs 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a40a76d0 con 0x7fd6a4074be0 2024-09-17T15:36:57.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c013470 con 0x7fd6a4074be0 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd699ffb640 10 monclient: dump: 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:36:57.580 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd69affd640 4 set_mon_vals no callback set 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _renew_subs 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a4073bc0 con 0x7fd6a4074be0 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.556+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a4138660 con 0x7fd6a4074be0 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd69c01a070 con 0x7fd6a4074be0 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd699ffb640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 0x7fd68803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:36:57.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(320..320 src has 213..320) ==== 4771+0+0 (secure 0 0 0) 0x7fd69c04fff0 con 0x7fd6a4074be0 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd6a97a7640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 0x7fd68803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd6a97a7640 10 monclient: get_auth_request con 0x7fd68803bb60 auth_method 0 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.560+0000 7fd6a97a7640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 0x7fd68803e020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd694007fd0 tx=0x7fd694009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.564+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd69c009340 con 0x7fd6a4074be0 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.564+0000 7fd699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:36:57.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.564+0000 7fd699ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:36:57.699 INFO:tasks.rados.rados.0.smithi203.stdout:1384: done (1 left) 2024-09-17T15:36:57.699 INFO:tasks.rados.rados.0.smithi203.stdout:1386: done (0 left) 2024-09-17T15:36:57.700 INFO:tasks.rados.rados.0.smithi203.stdout:1387: snap_create 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6a4072c60 con 0x7fd6a4074be0 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd699ffb640 1 -- 172.21.15.86:0/2819354570 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fd69c013bb0 con 0x7fd6a4074be0 2024-09-17T15:36:57.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:36:57.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd699ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:36:57.811 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 msgr2=0x7fd68803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 0x7fd68803e020 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd694007fd0 tx=0x7fd694009040 comp rx=0 tx=0).stop 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 10 monclient: shutdown 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 msgr2=0x7fd6a41cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:36:57.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00e920 tx=0x7fd69c00ede0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 shutdown_connections 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68803bb60 0x7fd68803e020 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a4074be0 0x7fd6a41cd6a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a413d260 0x7fd6a41cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 --2- 172.21.15.86:0/2819354570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a4138280 0x7fd6a41ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.808+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 >> 172.21.15.86:0/2819354570 conn(0x7fd6a40a8470 msgr2=0x7fd6a413c510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.812+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 shutdown_connections 2024-09-17T15:36:57.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:36:57.812+0000 7fd6aba0b640 1 -- 172.21.15.86:0/2819354570 wait complete. 2024-09-17T15:36:58.703 INFO:tasks.rados.rados.0.smithi203.stdout:1387: done (0 left) 2024-09-17T15:36:58.703 INFO:tasks.rados.rados.0.smithi203.stdout:1388: write oid 19 current snap is 151 2024-09-17T15:36:58.704 INFO:tasks.rados.rados.0.smithi203.stdout:1388: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-17T15:36:58.720 INFO:tasks.rados.rados.0.smithi203.stdout:1388: writing smithi20328086-19 from 504342 to 1059634 tid 1 2024-09-17T15:36:58.739 INFO:tasks.rados.rados.0.smithi203.stdout:1388: writing smithi20328086-19 from 1643418 to 2128262 tid 2 2024-09-17T15:36:58.739 INFO:tasks.rados.rados.0.smithi203.stdout:1389: rollback oid 35 current snap is 151 2024-09-17T15:36:58.740 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 148 2024-09-17T15:36:58.740 INFO:tasks.rados.rados.0.smithi203.stdout:1388: finishing write tid 1 to smithi20328086-19 2024-09-17T15:36:58.740 INFO:tasks.rados.rados.0.smithi203.stdout:1390: delete oid 47 current snap is 151 2024-09-17T15:36:58.747 INFO:tasks.rados.rados.0.smithi203.stdout:1390: done (2 left) 2024-09-17T15:36:58.747 INFO:tasks.rados.rados.0.smithi203.stdout:1391: rollback oid 18 current snap is 151 2024-09-17T15:36:58.748 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 148 2024-09-17T15:36:58.748 INFO:tasks.rados.rados.0.smithi203.stdout:1392: snap_remove snap 150 2024-09-17T15:36:59.687 INFO:tasks.rados.rados.0.smithi203.stdout:1392: done (3 left) 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1393: delete oid 4 current snap is 151 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: finishing write tid 2 to smithi20328086-19 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: finishing write tid 3 to smithi20328086-19 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1389: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 0 (ObjNum 289 snap 0 seq_num 3083322650) dirty dne 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: finishing write tid 4 to smithi20328086-19 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: oid 19 updating version 0 to 541 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: oid 19 updating version 541 to 542 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: oid 19 version 542 is already newer than 540 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 542 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:36:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:1388: left oid 19 (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:36:59.689 INFO:tasks.rados.rados.0.smithi203.stdout:1391: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:36:59.689 INFO:tasks.rados.rados.0.smithi203.stdout:1391: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:36:59.689 INFO:tasks.rados.rados.0.smithi203.stdout:1391: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:36:59.689 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 383 (ObjNum 377 snap 108 seq_num 377) dirty exists 2024-09-17T15:36:59.696 INFO:tasks.rados.rados.0.smithi203.stdout:1388: done (3 left) 2024-09-17T15:36:59.696 INFO:tasks.rados.rados.0.smithi203.stdout:1389: done (2 left) 2024-09-17T15:36:59.696 INFO:tasks.rados.rados.0.smithi203.stdout:1391: done (1 left) 2024-09-17T15:36:59.696 INFO:tasks.rados.rados.0.smithi203.stdout:1393: done (0 left) 2024-09-17T15:36:59.696 INFO:tasks.rados.rados.0.smithi203.stdout:1394: snap_remove snap 148 2024-09-17T15:37:00.707 INFO:tasks.rados.rados.0.smithi203.stdout:1394: done (0 left) 2024-09-17T15:37:00.708 INFO:tasks.rados.rados.0.smithi203.stdout:1395: snap_remove snap 149 2024-09-17T15:37:00.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:00.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 10 monclient: get_monmap_and_config 2024-09-17T15:37:00.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 10 monclient: build_initial_monmap 2024-09-17T15:37:00.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 10 monclient: monmap: 2024-09-17T15:37:00.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:00.958206+0000 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:00.958206+0000 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:00.956 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 1 Processor -- start 2024-09-17T15:37:00.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 1 -- start start 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.952+0000 7fbbf64af640 10 monclient: init 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbbf00689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbf0068db0 0x7fbbf00691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf0068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbf0139f00 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf009d770 con 0x7fbbf0139f00 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf009ed00 con 0x7fbbf00689d0 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf009efe0 con 0x7fbbf0068db0 2024-09-17T15:37:00.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf64af640 10 monclient(hunting): authenticate will time out at 1553.924235s 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbf0068db0 0x7fbbf00691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbf0068db0 0x7fbbf00691c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36048/0 (socket says 172.21.15.86:36048) 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbf4a4c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbbf0139f00 0x7fbbf013c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43276/0 (socket says 172.21.15.86:43276) 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 -- 172.21.15.86:0/3425754971 learned_addr learned my addr 172.21.15.86:0/3425754971 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): get_auth_request con 0x7fbbf0068db0 auth_method 0 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:00.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_done global_id 94305 payload 274 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 -- 172.21.15.86:0/3425754971 >> v1:172.21.15.203:6789/0 conn(0x7fbbf00689d0 legacy=0x7fbbf009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 -- 172.21.15.86:0/3425754971 >> v1:172.21.15.86:6789/0 conn(0x7fbbf0139f00 legacy=0x7fbbf013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 monclient: found mon.noname-a 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:00.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.956+0000 7fbbef7fe640 1 -- 172.21.15.86:0/3425754971 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf0069700 con 0x7fbbf0068db0 2024-09-17T15:37:00.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbef7fe640 1 -- 172.21.15.86:0/3425754971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf0068db0 msgr2=0x7fbbf00691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbef7fe640 1 --2- 172.21.15.86:0/3425754971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf0068db0 0x7fbbf00691c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0069aa0 tx=0x7fbbe401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1360d1602524d41 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 1 -- 172.21.15.86:0/3425754971 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40b0070 con 0x7fbbf0068db0 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: dump: 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:00.964 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 1 -- 172.21.15.86:0/3425754971 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe40ab600 con 0x7fbbf0068db0 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 5 monclient: authenticate success, global_id 94305 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 1 -- 172.21.15.86:0/3425754971 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40ab9d0 con 0x7fbbf0068db0 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 4 set_mon_vals no callback set 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeeffd640 10 monclient: dump: 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: shutdown 2024-09-17T15:37:00.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- 172.21.15.86:0/3425754971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf0068db0 msgr2=0x7fbbf00691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 --2- 172.21.15.86:0/3425754971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf0068db0 0x7fbbf00691c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0069aa0 tx=0x7fbbe401c400 comp rx=0 tx=0).stop 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- 172.21.15.86:0/3425754971 shutdown_connections 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 --2- 172.21.15.86:0/3425754971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf0068db0 0x7fbbf00691c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- 172.21.15.86:0/3425754971 >> 172.21.15.86:0/3425754971 conn(0x7fbbf00a6440 msgr2=0x7fbbf00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- 172.21.15.86:0/3425754971 shutdown_connections 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- 172.21.15.86:0/3425754971 wait complete. 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: build_initial_monmap 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: monmap: 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:00.967 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 Processor -- start 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- start start 2024-09-17T15:37:00.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: init 2024-09-17T15:37:00.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:00.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:00.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbf0068db0 0x7fbbf0133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf0068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbf0139f00 0x7fbbf0134020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf0139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf01409d0 con 0x7fbbf00689d0 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0140850 con 0x7fbbf0068db0 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf0140b50 con 0x7fbbf0139f00 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:00.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf64af640 10 monclient(hunting): authenticate will time out at 1553.931616s 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36062/0 (socket says 172.21.15.86:36062) 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 1 -- 172.21.15.86:0/718235486 learned_addr learned my addr 172.21.15.86:0/718235486 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): get_auth_request con 0x7fbbf00689d0 auth_method 0 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbf0139f00 0x7fbbf0134020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): get_auth_request con 0x7fbbf0139f00 auth_method 0 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:00.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.960+0000 7fbbf4a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 10 monclient(hunting): handle_auth_done global_id 94308 payload 824 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 -- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbf0139f00 msgr2=0x7fbbf0134020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbf0139f00 0x7fbbf0134020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbef7fe640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbf0068db0 0x7fbbf0133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 -- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbf0068db0 msgr2=0x7fbbf0133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbf0068db0 0x7fbbf0133ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 monclient: found mon.noname-a 2024-09-17T15:37:00.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 -- 172.21.15.86:0/718235486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf011fef0 con 0x7fbbf00689d0 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 -- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 msgr2=0x7fbbf01cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbeffff640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000c450 tx=0x7fbbe000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0012a40 con 0x7fbbf00689d0 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: dump: 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.976 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: _finish_auth 0 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: _check_auth_tickets 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0017070 con 0x7fbbf00689d0 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0012e40 con 0x7fbbf00689d0 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 5 monclient: authenticate success, global_id 94308 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _renew_subs 2024-09-17T15:37:00.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf01201d0 con 0x7fbbf00689d0 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbee7fc640 4 set_mon_vals no callback set 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 10 monclient: dump: 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:00.978 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _renew_subs 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf011e150 con 0x7fbbf00689d0 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf00691c0 con 0x7fbbf00689d0 2024-09-17T15:37:00.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbbe001c460 con 0x7fbbf00689d0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 0x7fbbdc03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.964+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(324..324 src has 213..324) ==== 4799+0+0 (secure 0 0 0) 0x7fbbe004fc80 con 0x7fbbf00689d0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.968+0000 7fbbef7fe640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 0x7fbbdc03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.968+0000 7fbbef7fe640 10 monclient: get_auth_request con 0x7fbbdc03bbe0 auth_method 0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.968+0000 7fbbef7fe640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 0x7fbbdc03e0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40ab000 tx=0x7fbbe4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.972+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbbe001ac60 con 0x7fbbf00689d0 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.972+0000 7fbbecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:00.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:00.972+0000 7fbbecff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:01.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbbf011ecb0 con 0x7fbbf00689d0 2024-09-17T15:37:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbecff9640 1 -- 172.21.15.86:0/718235486 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fbbe0010cf0 con 0x7fbbf00689d0 2024-09-17T15:37:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:01.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbecff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:01.207 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":92,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:01.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 msgr2=0x7fbbdc03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:01.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 0x7fbbdc03e0a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbbe40ab000 tx=0x7fbbe4008040 comp rx=0 tx=0).stop 2024-09-17T15:37:01.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 10 monclient: shutdown 2024-09-17T15:37:01.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:01.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 msgr2=0x7fbbf01cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000c450 tx=0x7fbbe000c910 comp rx=0 tx=0).stop 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 shutdown_connections 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbbdc03bbe0 0x7fbbdc03e0a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbf0139f00 0x7fbbf0134020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbf0068db0 0x7fbbf0133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 --2- 172.21.15.86:0/718235486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbf00689d0 0x7fbbf01cfd40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 >> 172.21.15.86:0/718235486 conn(0x7fbbf00a6440 msgr2=0x7fbbf01391b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.204+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 shutdown_connections 2024-09-17T15:37:01.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:01.208+0000 7fbbf64af640 1 -- 172.21.15.86:0/718235486 wait complete. 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1395: done (0 left) 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1396: rollback oid 6 current snap is 151 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1396: done (0 left) 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1397: read oid 3 snap -1 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1397: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:01.708 INFO:tasks.rados.rados.0.smithi203.stdout:1398: read oid 35 snap -1 2024-09-17T15:37:01.709 INFO:tasks.rados.rados.0.smithi203.stdout:1398: expect deleted 2024-09-17T15:37:01.709 INFO:tasks.rados.rados.0.smithi203.stdout:1399: write oid 40 current snap is 151 2024-09-17T15:37:01.709 INFO:tasks.rados.rados.0.smithi203.stdout:1399: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-17T15:37:01.733 INFO:tasks.rados.rados.0.smithi203.stdout:1399: writing smithi20328086-40 from 554763 to 1335802 tid 1 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:1399: writing smithi20328086-40 from 2102511 to 2554763 tid 2 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:1400: read oid 44 snap -1 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:1400: expect (ObjNum 456 snap 138 seq_num 456) 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:1401: snap_create 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:1397: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:01.749 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:01.846 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:01.874 INFO:tasks.rados.rados.0.smithi203.stdout:1399: finishing write tid 1 to smithi20328086-40 2024-09-17T15:37:01.874 INFO:tasks.rados.rados.0.smithi203.stdout:1400: expect (ObjNum 456 snap 138 seq_num 456) 2024-09-17T15:37:01.941 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:02.018 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 42684 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: finishing write tid 2 to smithi20328086-40 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: finishing write tid 3 to smithi20328086-40 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: finishing write tid 4 to smithi20328086-40 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: oid 40 updating version 0 to 580 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: oid 40 updating version 580 to 582 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: oid 40 version 582 is already newer than 581 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 582 (ObjNum 509 snap 151 seq_num 509) dirty exists 2024-09-17T15:37:02.096 INFO:tasks.rados.rados.0.smithi203.stdout:1399: left oid 40 (ObjNum 509 snap 151 seq_num 509) 2024-09-17T15:37:02.870 INFO:tasks.rados.rados.0.smithi203.stdout:1397: done (4 left) 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1398: done (3 left) 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1399: done (2 left) 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1400: done (1 left) 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1401: done (0 left) 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1402: copy_from oid 1 from oid 17 current snap is 152 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1403: write oid 46 current snap is 152 2024-09-17T15:37:02.871 INFO:tasks.rados.rados.0.smithi203.stdout:1403: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-17T15:37:02.888 INFO:tasks.rados.rados.0.smithi203.stdout:1403: writing smithi20328086-46 from 571570 to 1144211 tid 1 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:1403: writing smithi20328086-46 from 1877346 to 2571570 tid 2 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:1404: snap_create 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:1402: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:1403: finishing write tid 1 to smithi20328086-46 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:1402: finishing copy_from to smithi20328086-1 2024-09-17T15:37:02.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 496 (ObjNum 420 snap 123 seq_num 420) dirty exists 2024-09-17T15:37:02.920 INFO:tasks.rados.rados.0.smithi203.stdout:1403: finishing write tid 2 to smithi20328086-46 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:1403: finishing write tid 3 to smithi20328086-46 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:1403: finishing write tid 4 to smithi20328086-46 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:1403: oid 46 updating version 0 to 462 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:1403: oid 46 updating version 462 to 463 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:1403: oid 46 version 463 is already newer than 461 2024-09-17T15:37:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 463 (ObjNum 510 snap 152 seq_num 510) dirty exists 2024-09-17T15:37:02.922 INFO:tasks.rados.rados.0.smithi203.stdout:1403: left oid 46 (ObjNum 510 snap 152 seq_num 510) 2024-09-17T15:37:03.879 INFO:tasks.rados.rados.0.smithi203.stdout:1402: done (2 left) 2024-09-17T15:37:03.880 INFO:tasks.rados.rados.0.smithi203.stdout:1403: done (1 left) 2024-09-17T15:37:03.880 INFO:tasks.rados.rados.0.smithi203.stdout:1404: done (0 left) 2024-09-17T15:37:03.880 INFO:tasks.rados.rados.0.smithi203.stdout:1405: copy_from oid 28 from oid 45 current snap is 153 2024-09-17T15:37:03.880 INFO:tasks.rados.rados.0.smithi203.stdout:1406: delete oid 47 current snap is 153 2024-09-17T15:37:03.883 INFO:tasks.rados.rados.0.smithi203.stdout:1406: done (1 left) 2024-09-17T15:37:03.883 INFO:tasks.rados.rados.0.smithi203.stdout:1407: write oid 30 current snap is 153 2024-09-17T15:37:03.884 INFO:tasks.rados.rados.0.smithi203.stdout:1407: seq_num 512 ranges {605184=723139,1877957=727227} 2024-09-17T15:37:03.906 INFO:tasks.rados.rados.0.smithi203.stdout:1407: writing smithi20328086-30 from 605184 to 1328323 tid 1 2024-09-17T15:37:03.923 INFO:tasks.rados.rados.0.smithi203.stdout:1407: writing smithi20328086-30 from 1877957 to 2605184 tid 2 2024-09-17T15:37:03.923 INFO:tasks.rados.rados.0.smithi203.stdout:1408: snap_remove snap 152 2024-09-17T15:37:04.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: get_monmap_and_config 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: monmap: 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:04.357798+0000 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:04.357798+0000 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:04.356 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 Processor -- start 2024-09-17T15:37:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 -- start start 2024-09-17T15:37:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: init 2024-09-17T15:37:04.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a200689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a200694c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 0x7f3a2013e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2006f840 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a20074470 con 0x7f3a200689d0 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a2009c8b0 con 0x7f3a200694c0 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3a2009cb90 con 0x7f3a2006f840 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a25b9d640 10 monclient(hunting): authenticate will time out at 1557.323350s 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3a200689d0 0x7f3a20068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43282/0 (socket says 172.21.15.86:43282) 2024-09-17T15:37:04.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1f7fe640 1 -- 172.21.15.86:0/1342387321 learned_addr learned my addr 172.21.15.86:0/1342387321 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 0x7f3a2013e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request con 0x7f3a2006f840 auth_method 0 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.352+0000 7f3a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 msgr2=0x7f3a2013e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 msgr2=0x7f3a2013e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 0x7f3a2013e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 0x7f3a2013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 1 -- 172.21.15.86:0/1342387321 >> v1:172.21.15.86:6789/0 conn(0x7f3a200689d0 legacy=0x7f3a20068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3566405033 0 0) 0x7f3a20074470 con 0x7f3a200689d0 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): my global_id is 94311 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a0c003530 con 0x7f3a200689d0 2024-09-17T15:37:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 1 -- 172.21.15.86:0/1342387321 >> v1:172.21.15.203:6789/0 conn(0x7f3a200694c0 legacy=0x7f3a2013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2061899343 0 0) 0x7f3a0c003530 con 0x7f3a200689d0 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 msgr2=0x7f3a2013e720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 --2- 172.21.15.86:0/1342387321 >> v2:172.21.15.203:3300/0 conn(0x7f3a2006f840 0x7f3a2013e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 >> v1:172.21.15.203:6789/0 conn(0x7f3a200694c0 legacy=0x7f3a2013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2009ce70 con 0x7f3a200689d0 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f3a10001d30 con 0x7f3a200689d0 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a25b9d640 5 monclient: authenticate success, global_id 94311 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a25b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: dump: 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 >> v1:172.21.15.86:6789/0 conn(0x7f3a200689d0 legacy=0x7f3a20068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a0c003b20 0x7f3a0c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): picked mon.c con 0x7f3a0c003b20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): picked mon.b con 0x7f3a0c004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 0x7f3a0c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): picked mon.a con 0x7f3a0c0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2009cb90 con 0x7f3a0c0099c0 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c00c3d0 con 0x7f3a0c004810 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c00c5d0 con 0x7f3a0c003b20 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a25b9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 0x7f3a0c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a0c003b20 0x7f3a0c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f3a0c003b20 auth_method 0 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request con 0x7f3a0c0099c0 auth_method 0 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:04.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): get_auth_request con 0x7f3a0c004810 auth_method 0 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_done global_id 94311 payload 293 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a0c003b20 msgr2=0x7f3a0c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a0c003b20 0x7f3a0c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c004810 msgr2=0x7f3a0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c004810 0x7f3a0c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 monclient: found mon.a 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c003530 con 0x7f3a0c0099c0 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.356+0000 7f3a1f7fe640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a0c003b20 0x7f3a0c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:04.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.360+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 reap_dead start 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.360+0000 7f3a1ffff640 1 -- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 msgr2=0x7f3a0c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.360+0000 7f3a1ffff640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 0x7f3a0c00bdb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c820 tx=0x7f3a1801c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12730c9cb4b60f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a180b1070 con 0x7f3a0c0099c0 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: dump: 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a180ac410 con 0x7f3a0c0099c0 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 1 -- 172.21.15.86:0/1342387321 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1000a690 con 0x7f3a0c0099c0 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 4 set_mon_vals no callback set 2024-09-17T15:37:04.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1e7fc640 10 monclient: dump: 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: shutdown 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 msgr2=0x7f3a0c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 0x7f3a0c00bdb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800c820 tx=0x7f3a1801c580 comp rx=0 tx=0).stop 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- 172.21.15.86:0/1342387321 shutdown_connections 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/1342387321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a0c0099c0 0x7f3a0c00bdb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- 172.21.15.86:0/1342387321 >> 172.21.15.86:0/1342387321 conn(0x7f3a200a6440 msgr2=0x7f3a200a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- 172.21.15.86:0/1342387321 shutdown_connections 2024-09-17T15:37:04.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- 172.21.15.86:0/1342387321 wait complete. 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: monmap: 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 Processor -- start 2024-09-17T15:37:04.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- start start 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: init 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c0099c0 0x7f3a201cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c0099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a201351d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a20135650 0x7f3a201ca910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3a20135650 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20069140 con 0x7f3a201351d0 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20068fc0 con 0x7f3a0c0099c0 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3a200692c0 con 0x7f3a20135650 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a25b9d640 10 monclient(hunting): authenticate will time out at 1557.334870s 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a20135650 0x7f3a201ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36082/0 (socket says 172.21.15.86:36082) 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 -- 172.21.15.86:0/2649024078 learned_addr learned my addr 172.21.15.86:0/2649024078 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): get_auth_request con 0x7f3a201351d0 auth_method 0 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request con 0x7f3a20135650 auth_method 0 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1f7fe640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c0099c0 0x7f3a201cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f3a0c0099c0 auth_method 0 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient(hunting): handle_auth_done global_id 94314 payload 824 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 -- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a20135650 msgr2=0x7f3a201ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a20135650 0x7f3a201ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 -- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c0099c0 msgr2=0x7f3a201cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c0099c0 0x7f3a201cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 monclient: found mon.noname-a 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.364+0000 7f3a1effd640 1 -- 172.21.15.86:0/2649024078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2013e580 con 0x7f3a201351d0 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a1effd640 1 -- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 msgr2=0x7f3a201ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a1effd640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a080076d0 tx=0x7f3a080019d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a08017020 con 0x7f3a201351d0 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: dump: 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.387 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0800f870 con 0x7f3a201351d0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 5 monclient: authenticate success, global_id 94314 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _renew_subs 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2009d460 con 0x7f3a201351d0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0800fc40 con 0x7f3a201351d0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 10 monclient: dump: 2024-09-17T15:37:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a1dffb640 4 set_mon_vals no callback set 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _renew_subs 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2011e2b0 con 0x7f3a201351d0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3a0801a020 con 0x7f3a201351d0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 0x7f3a0003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a20135ae0 con 0x7f3a201351d0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(327..327 src has 213..327) ==== 4755+0+0 (secure 0 0 0) 0x7f3a0804fe40 con 0x7f3a201351d0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a1f7fe640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 0x7f3a0003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.368+0000 7f3a1f7fe640 10 monclient: get_auth_request con 0x7f3a0003bbb0 auth_method 0 2024-09-17T15:37:04.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.372+0000 7f3a1f7fe640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 0x7f3a0003e070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3a10001830 tx=0x7f3a10006040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.376+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3a08021020 con 0x7f3a201351d0 2024-09-17T15:37:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.376+0000 7f3a17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.376+0000 7f3a17fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.612+0000 7f3a25b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.612+0000 7f3a25b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.612+0000 7f3a25b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.612+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3a2006d4f0 con 0x7f3a201351d0 2024-09-17T15:37:04.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.612+0000 7f3a17fff640 1 -- 172.21.15.86:0/2649024078 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3a2013e580 con 0x7f3a201351d0 2024-09-17T15:37:04.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:04.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a17fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:04.617 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:04.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 msgr2=0x7f3a0003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 0x7f3a0003e070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3a10001830 tx=0x7f3a10006040 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 10 monclient: shutdown 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 msgr2=0x7f3a201ca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3a080076d0 tx=0x7f3a080019d0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 shutdown_connections 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a0003bbb0 0x7f3a0003e070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a20135650 0x7f3a201ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a201351d0 0x7f3a201ca3d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 --2- 172.21.15.86:0/2649024078 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a0c0099c0 0x7f3a201cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 >> 172.21.15.86:0/2649024078 conn(0x7f3a200a6440 msgr2=0x7f3a200a2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 shutdown_connections 2024-09-17T15:37:04.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.616+0000 7f3a25b9d640 1 -- 172.21.15.86:0/2649024078 wait complete. 2024-09-17T15:37:04.635 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:37:04.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:37:04.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1408: done (2 left) 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1409: copy_from oid 18 from oid 1 current snap is 153 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1405: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1407: finishing write tid 1 to smithi20328086-30 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1405: finishing copy_from to smithi20328086-28 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 276 (ObjNum 462 snap 140 seq_num 462) dirty exists 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1407: finishing write tid 2 to smithi20328086-30 2024-09-17T15:37:04.686 INFO:tasks.rados.rados.0.smithi203.stdout:1407: finishing write tid 3 to smithi20328086-30 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: finishing write tid 4 to smithi20328086-30 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: oid 30 updating version 0 to 335 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: oid 30 updating version 335 to 337 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: oid 30 version 337 is already newer than 336 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 337 (ObjNum 512 snap 153 seq_num 512) dirty exists 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: left oid 30 (ObjNum 512 snap 153 seq_num 512) 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1405: done (2 left) 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1407: done (1 left) 2024-09-17T15:37:04.687 INFO:tasks.rados.rados.0.smithi203.stdout:1410: snap_remove snap 151 2024-09-17T15:37:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: get_monmap_and_config 2024-09-17T15:37:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: monmap: 2024-09-17T15:37:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:04.760282+0000 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:04.760282+0000 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:04.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 Processor -- start 2024-09-17T15:37:04.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 -- start start 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: init 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05a80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 0x7f05a809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-c con 0x7f05a80a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 0x7f05a813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-b con 0x7f05a813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a80a6280 con 0x7f05a80a50e0 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f05a809d2a0 con 0x7f05a80a5bd0 2024-09-17T15:37:04.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f05a809c0a0 con 0x7f05a813d2f0 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05aef3f640 10 monclient(hunting): authenticate will time out at 1557.725640s 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05accdb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f05a80a50e0 0x7f05a80a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43288/0 (socket says 172.21.15.86:43288) 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 0x7f05a813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05accdb640 1 -- 172.21.15.86:0/2855043719 learned_addr learned my addr 172.21.15.86:0/2855043719 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): get_auth_request con 0x7f05a813d2f0 auth_method 0 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05a7fff640 1 --2- 172.21.15.86:0/2855043719 >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 0x7f05a809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient(hunting): handle_auth_done global_id 104189 payload 274 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 -- 172.21.15.86:0/2855043719 >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 msgr2=0x7f05a809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/2855043719 >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 0x7f05a809cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 -- 172.21.15.86:0/2855043719 >> v1:172.21.15.86:6789/0 conn(0x7f05a80a50e0 legacy=0x7f05a80a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 monclient: found mon.noname-b 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05ad4dc640 1 -- 172.21.15.86:0/2855043719 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a809c380 con 0x7f05a813d2f0 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05a7fff640 10 monclient: get_auth_request con 0x7f05a80a5bd0 auth_method 0 2024-09-17T15:37:04.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.756+0000 7f05a7fff640 1 --2- 172.21.15.86:0/2855043719 >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 0x7f05a809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05ad4dc640 1 -- 172.21.15.86:0/2855043719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 msgr2=0x7f05a813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/2855043719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 0x7f05a813f6e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009d90 tx=0x7f05a001c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f61128f5e8f7fe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 1 -- 172.21.15.86:0/2855043719 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a00b2070 con 0x7f05a813d2f0 2024-09-17T15:37:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: dump: 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 1 -- 172.21.15.86:0/2855043719 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a00a5820 con 0x7f05a813d2f0 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 1 -- 172.21.15.86:0/2855043719 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a00a5bf0 con 0x7f05a813d2f0 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 5 monclient: authenticate success, global_id 104189 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 4 set_mon_vals no callback set 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:04.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05a77fe640 10 monclient: dump: 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 10 monclient: shutdown 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 -- 172.21.15.86:0/2855043719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 msgr2=0x7f05a813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 --2- 172.21.15.86:0/2855043719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 0x7f05a813f6e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05a0009d90 tx=0x7f05a001c7a0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 -- 172.21.15.86:0/2855043719 shutdown_connections 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 --2- 172.21.15.86:0/2855043719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a813d2f0 0x7f05a813f6e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 --2- 172.21.15.86:0/2855043719 >> v2:172.21.15.203:3300/0 conn(0x7f05a80a5bd0 0x7f05a809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 -- 172.21.15.86:0/2855043719 >> 172.21.15.86:0/2855043719 conn(0x7f05a80aaae0 msgr2=0x7f05a80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 -- 172.21.15.86:0/2855043719 shutdown_connections 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 1 -- 172.21.15.86:0/2855043719 wait complete. 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.760+0000 7f05aef3f640 10 monclient: monmap: 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:04.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 Processor -- start 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 -- start start 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient: init 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:37:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-a con 0x7f05a80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a80a5bd0 0x7f05a8138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-c con 0x7f05a80a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a813d2f0 0x7f05a81cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): picked mon.noname-b con 0x7f05a813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8142650 con 0x7f05a80a50e0 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05a80a6b00 con 0x7f05a813d2f0 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f05a81427d0 con 0x7f05a80a5bd0 2024-09-17T15:37:04.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 10 monclient(hunting): authenticate will time out at 1557.732957s 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a80a5bd0 0x7f05a8138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36092/0 (socket says 172.21.15.86:36092) 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 -- 172.21.15.86:0/3526745776 learned_addr learned my addr 172.21.15.86:0/3526745776 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): get_auth_request con 0x7f05a80a50e0 auth_method 0 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): get_auth_request con 0x7f05a80a5bd0 auth_method 0 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient(hunting): handle_auth_done global_id 94317 payload 824 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a813d2f0 0x7f05a81cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a80a5bd0 msgr2=0x7f05a8138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a80a5bd0 0x7f05a8138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a813d2f0 msgr2=0x7f05a81cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a813d2f0 0x7f05a81cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 monclient: found mon.noname-a 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a8122bb0 con 0x7f05a80a50e0 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05ad4dc640 10 monclient: get_auth_request con 0x7f05a813d2f0 auth_method 0 2024-09-17T15:37:04.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a813d2f0 0x7f05a81cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 msgr2=0x7f05a81213b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05accdb640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f059800c470 tx=0x7f059800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598007070 con 0x7f05a80a50e0 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: dump: 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0598007210 con 0x7f05a80a50e0 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05aef3f640 5 monclient: authenticate success, global_id 94317 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598007600 con 0x7f05a80a50e0 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a57fa640 10 monclient: dump: 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.764+0000 7f05a6ffd640 4 set_mon_vals no callback set 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 monclient: _renew_subs 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a80a0be0 con 0x7f05a80a50e0 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 monclient: _renew_subs 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a80a18d0 con 0x7f05a80a50e0 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 0x7f05a81d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05a806a0e0 con 0x7f05a8138760 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 0x7f05a81d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient: get_auth_request con 0x7f05a8138760 auth_method 0 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f059801b400 con 0x7f05a80a50e0 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 10 monclient(hunting): handle_auth_done global_id 94320 payload 824 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05a809cd60 con 0x7f05a8138760 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb60 0x7f059403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(328..328 src has 213..328) ==== 4827+0+0 (secure 0 0 0) 0x7f059804fd20 con 0x7f05a80a50e0 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb60 0x7f059403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05ad4dc640 10 monclient: get_auth_request con 0x7f059403bb60 auth_method 0 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 msgr2=0x7f05a81d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 0x7f05a81d0070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f059c00f850 tx=0x7f059c00fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c00de30 con 0x7f05a8138760 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 10 monclient: dump: 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.785 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f059c00de30 con 0x7f05a8138760 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 msgr2=0x7f05a81d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 0x7f05a81d0070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f059c00f850 tx=0x7f059c00fd10 comp rx=0 tx=0).stop 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 msgr2=0x7f05a81d0070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.768+0000 7f05a57fa640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8138760 0x7f05a81d0070 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f059c00f850 tx=0x7f059c00fd10 comp rx=0 tx=0).stop 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.772+0000 7f05ad4dc640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb60 0x7f059403e020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f05a80aa620 tx=0x7f05a00a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05aef3f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05aef3f640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 0x7f05a81392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05aef3f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8122510 con 0x7f05a81d05b0 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 0x7f05a81392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient: get_auth_request con 0x7f05a81d05b0 auth_method 0 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 10 monclient(hunting): handle_auth_done global_id 94323 payload 824 2024-09-17T15:37:04.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.776+0000 7f05a7fff640 1 -- 172.21.15.86:0/3526745776 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05a81d09c0 con 0x7f05a81d05b0 2024-09-17T15:37:04.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a7fff640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 msgr2=0x7f05a81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 0x7f05a81392d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f059c00f850 tx=0x7f059c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c003a40 con 0x7f05a81d05b0 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 10 monclient: dump: 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f059c003a40 con 0x7f05a81d05b0 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 msgr2=0x7f05a81392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 0x7f05a81392d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f059c00f850 tx=0x7f059c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 msgr2=0x7f05a81392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05a57fa640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a81d05b0 0x7f05a81392d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.790 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 100134, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:37:04.791 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.792 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.793 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:04.794 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.795 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.796 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:04.797 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.798 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:04.799 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:04.800 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.801 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:37:04.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.803 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:04.804 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb60 msgr2=0x7f059403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb60 0x7f059403e020 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f05a80aa620 tx=0x7f05a00a7070 comp rx=0 tx=0).stop 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 10 monclient: shutdown 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 msgr2=0x7f05a81213b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f059800c470 tx=0x7f059800c930 comp rx=0 tx=0).stop 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05ad4dc640 1 -- 172.21.15.86:0/3526745776 reap_dead start 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 shutdown_connections 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 --2- 172.21.15.86:0/3526745776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a80a50e0 0x7f05a81213b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 >> 172.21.15.86:0/3526745776 conn(0x7f05a80aaae0 msgr2=0x7f05a813dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 shutdown_connections 2024-09-17T15:37:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.780+0000 7f05aef3f640 1 -- 172.21.15.86:0/3526745776 wait complete. 2024-09-17T15:37:04.807 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:04.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:37:04.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:37:04.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.920+0000 7f1ecc3da640 10 monclient: get_monmap_and_config 2024-09-17T15:37:04.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.920+0000 7f1ecc3da640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.920+0000 7f1ecc3da640 10 monclient: monmap: 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:04.927574+0000 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:04.927574+0000 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:04.925 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 Processor -- start 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 -- start start 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient: init 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 0x7f1ec409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ec40a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ec413d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec40a6280 con 0x7f1ec40a54c0 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec409c0a0 con 0x7f1ec413d2f0 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec409d2a0 con 0x7f1ec40a50e0 2024-09-17T15:37:04.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ecc3da640 10 monclient(hunting): authenticate will time out at 1557.893032s 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 0x7f1ec409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ec9975640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1ec40a54c0 0x7f1ec40a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43296/0 (socket says 172.21.15.86:43296) 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1ec9975640 1 -- 172.21.15.86:0/2798360791 learned_addr learned my addr 172.21.15.86:0/2798360791 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): get_auth_request con 0x7f1ec40a50e0 auth_method 0 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient(hunting): handle_auth_done global_id 104195 payload 274 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 1 -- 172.21.15.86:0/2798360791 >> v1:172.21.15.203:6789/0 conn(0x7f1ec413d2f0 legacy=0x7f1ec413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 1 -- 172.21.15.86:0/2798360791 >> v1:172.21.15.86:6789/0 conn(0x7f1ec40a54c0 legacy=0x7f1ec40a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 1 monclient: found mon.noname-b 2024-09-17T15:37:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.924+0000 7f1eca176640 1 -- 172.21.15.86:0/2798360791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec409c380 con 0x7f1ec40a50e0 2024-09-17T15:37:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1eca176640 1 -- 172.21.15.86:0/2798360791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 msgr2=0x7f1ec409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1eca176640 1 --2- 172.21.15.86:0/2798360791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 0x7f1ec409cd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00097f0 tx=0x7f1ec001c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911de43fbe731ae6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 1 -- 172.21.15.86:0/2798360791 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00b0070 con 0x7f1ec40a50e0 2024-09-17T15:37:04.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: dump: 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 1 -- 172.21.15.86:0/2798360791 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec00ab600 con 0x7f1ec40a50e0 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 5 monclient: authenticate success, global_id 104195 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 1 -- 172.21.15.86:0/2798360791 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00ab9d0 con 0x7f1ec40a50e0 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 4 set_mon_vals no callback set 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ec9174640 10 monclient: dump: 2024-09-17T15:37:04.933 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.934 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 10 monclient: shutdown 2024-09-17T15:37:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 -- 172.21.15.86:0/2798360791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 msgr2=0x7f1ec409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/2798360791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 0x7f1ec409cd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00097f0 tx=0x7f1ec001c370 comp rx=0 tx=0).stop 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 -- 172.21.15.86:0/2798360791 shutdown_connections 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/2798360791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec40a50e0 0x7f1ec409cd60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 -- 172.21.15.86:0/2798360791 >> 172.21.15.86:0/2798360791 conn(0x7f1ec40aaae0 msgr2=0x7f1ec40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 -- 172.21.15.86:0/2798360791 shutdown_connections 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 1 -- 172.21.15.86:0/2798360791 wait complete. 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 10 monclient: build_initial_monmap 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.928+0000 7f1ecc3da640 10 monclient: monmap: 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:04.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 Processor -- start 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- start start 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: init 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ec40a50e0 0x7f1ec41380a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:04.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 0x7f1ec41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec40a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 0x7f1ec4139810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec4143830 con 0x7f1ec40a54c0 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec41439b0 con 0x7f1ec40a50e0 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec4143b30 con 0x7f1ec413d2f0 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient(hunting): authenticate will time out at 1557.900753s 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 0x7f1ec4139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 0x7f1ec41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 0x7f1ec4139810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55680/0 (socket says 172.21.15.86:55680) 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 1 -- 172.21.15.86:0/3947221362 learned_addr learned my addr 172.21.15.86:0/3947221362 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): get_auth_request con 0x7f1ec413d2f0 auth_method 0 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): get_auth_request con 0x7f1ec40a54c0 auth_method 0 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca176640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ec40a50e0 0x7f1ec41380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca176640 10 monclient(hunting): get_auth_request con 0x7f1ec40a50e0 auth_method 0 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca176640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca176640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient(hunting): handle_auth_done global_id 94326 payload 824 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 msgr2=0x7f1ec4139810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 0x7f1ec4139810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ec40a50e0 msgr2=0x7f1ec41380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ec40a50e0 0x7f1ec41380a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient: _finish_hunting 0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 monclient: found mon.noname-a 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec413bf30 con 0x7f1ec40a54c0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1eca977640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec413d2f0 0x7f1ec4139810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 msgr2=0x7f1ec41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec9975640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 0x7f1ec41392d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c410 tx=0x7f1eb400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4017070 con 0x7f1ec40a54c0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: dump: 2024-09-17T15:37:04.945 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:04.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb40070d0 con 0x7f1ec40a54c0 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 5 monclient: authenticate success, global_id 94326 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _renew_subs 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec413acd0 con 0x7f1ec40a54c0 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40074c0 con 0x7f1ec40a54c0 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ebaffd640 10 monclient: dump: 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.947 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ec8973640 4 set_mon_vals no callback set 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _renew_subs 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.932+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec4120d30 con 0x7f1ec40a54c0 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ecc3da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 0x7f1ec413a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec4121a50 con 0x7f1ec4139d50 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 0x7f1ec413a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient: get_auth_request con 0x7f1ec4139d50 auth_method 0 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1eb401a070 con 0x7f1ec40a54c0 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 10 monclient(hunting): handle_auth_done global_id 104201 payload 824 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ec41212a0 con 0x7f1ec4139d50 2024-09-17T15:37:04.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ea803bb60 0x7f1ea803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(328..328 src has 213..328) ==== 4827+0+0 (secure 0 0 0) 0x7f1eb404f450 con 0x7f1ec40a54c0 2024-09-17T15:37:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca977640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ea803bb60 0x7f1ea803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca977640 10 monclient: get_auth_request con 0x7f1ea803bb60 auth_method 0 2024-09-17T15:37:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 msgr2=0x7f1ec413a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1eca176640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 0x7f1ec413a130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00ad000 tx=0x7f1ec0019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00bb070 con 0x7f1ec4139d50 2024-09-17T15:37:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 10 monclient: dump: 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ec00c0450 con 0x7f1ec4139d50 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 msgr2=0x7f1ec413a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 0x7f1ec413a130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00ad000 tx=0x7f1ec0019110 comp rx=0 tx=0).stop 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 msgr2=0x7f1ec413a130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.936+0000 7f1ebaffd640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec4139d50 0x7f1ec413a130 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00ad000 tx=0x7f1ec0019110 comp rx=0 tx=0).stop 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.940+0000 7f1eca977640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ea803bb60 0x7f1ea803e020 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ec413b100 tx=0x7f1ebc003dd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1ecc3da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 0x7f1ec409e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1ecc3da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec41216a0 con 0x7f1ec409e5e0 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 0x7f1ec409e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient: get_auth_request con 0x7f1ec409e5e0 auth_method 0 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 10 monclient(hunting): handle_auth_done global_id 104204 payload 824 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.944+0000 7f1eca176640 1 -- 172.21.15.86:0/3947221362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ec409cd60 con 0x7f1ec409e5e0 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1eca176640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 msgr2=0x7f1ec409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1eca176640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 0x7f1ec409e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00097f0 tx=0x7f1ec0009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00b0040 con 0x7f1ec409e5e0 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:04.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 10 monclient: dump: 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f1ec0003620 con 0x7f1ec409e5e0 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 msgr2=0x7f1ec409e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 0x7f1ec409e9f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00097f0 tx=0x7f1ec0009190 comp rx=0 tx=0).stop 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 msgr2=0x7f1ec409e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ebaffd640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ec409e5e0 0x7f1ec409e9f0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1ec00097f0 tx=0x7f1ec0009190 comp rx=0 tx=0).stop 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:04.959 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 55823, 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.960 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.961 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:04.962 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:04.963 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:04.964 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:04.972 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:04.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ea803bb60 msgr2=0x7f1ea803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ea803bb60 0x7f1ea803e020 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ec413b100 tx=0x7f1ebc003dd0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 10 monclient: shutdown 2024-09-17T15:37:04.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:04.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 msgr2=0x7f1ec41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 --2- 172.21.15.86:0/3947221362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ec40a54c0 0x7f1ec41392d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c410 tx=0x7f1eb400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1eca977640 1 -- 172.21.15.86:0/3947221362 reap_dead start 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1eca977640 1 -- 172.21.15.86:0/3947221362 reap_dead start 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 shutdown_connections 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 >> 172.21.15.86:0/3947221362 conn(0x7f1ec40aaae0 msgr2=0x7f1ec40a31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 shutdown_connections 2024-09-17T15:37:04.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:04.948+0000 7f1ecc3da640 1 -- 172.21.15.86:0/3947221362 wait complete. 2024-09-17T15:37:04.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:04.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:37:04.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.084+0000 7f988d267640 10 monclient: get_monmap_and_config 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.084+0000 7f988d267640 10 monclient: build_initial_monmap 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.084+0000 7f988d267640 10 monclient: monmap: 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:05.091728+0000 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:05.091728+0000 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:05.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 Processor -- start 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 -- start start 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient: init 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f988809d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f988809e260 0x7f9888139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): picked mon.noname-c con 0x7f988809e260 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): picked mon.noname-b con 0x7f988813a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988809ff00 con 0x7f988809d770 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f988809e7f0 con 0x7f988809e260 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f98880a2110 con 0x7f988813a440 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f988d267640 10 monclient(hunting): authenticate will time out at 1558.057125s 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55690/0 (socket says 172.21.15.86:55690) 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 -- 172.21.15.86:0/1353022685 learned_addr learned my addr 172.21.15.86:0/1353022685 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): get_auth_request con 0x7f988813a440 auth_method 0 2024-09-17T15:37:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98867fc640 1 --2- 172.21.15.86:0/1353022685 >> v2:172.21.15.203:3300/0 conn(0x7f988809e260 0x7f9888139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98867fc640 10 monclient(hunting): get_auth_request con 0x7f988809e260 auth_method 0 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient(hunting): handle_auth_done global_id 104207 payload 274 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 -- 172.21.15.86:0/1353022685 >> v2:172.21.15.203:3300/0 conn(0x7f988809e260 msgr2=0x7f9888139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 --2- 172.21.15.86:0/1353022685 >> v2:172.21.15.203:3300/0 conn(0x7f988809e260 0x7f9888139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 -- 172.21.15.86:0/1353022685 >> v1:172.21.15.203:6789/0 conn(0x7f988809d770 legacy=0x7f988809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 monclient: found mon.noname-b 2024-09-17T15:37:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.088+0000 7f98877fe640 1 -- 172.21.15.86:0/1353022685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988809ec30 con 0x7f988813a440 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f98877fe640 1 -- 172.21.15.86:0/1353022685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 msgr2=0x7f988813c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f98877fe640 1 --2- 172.21.15.86:0/1353022685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9880009e50 tx=0x7f988001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b9f84195cb5fc54 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 1 -- 172.21.15.86:0/1353022685 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98800a64e0 con 0x7f988813a440 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: dump: 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: _finish_auth 0 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 1 -- 172.21.15.86:0/1353022685 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98800a6680 con 0x7f988813a440 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 5 monclient: authenticate success, global_id 104207 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 1 -- 172.21.15.86:0/1353022685 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98800a6a50 con 0x7f988813a440 2024-09-17T15:37:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 4 set_mon_vals no callback set 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f9885ffb640 10 monclient: dump: 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 10 monclient: shutdown 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- 172.21.15.86:0/1353022685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 msgr2=0x7f988813c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 --2- 172.21.15.86:0/1353022685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9880009e50 tx=0x7f988001c8f0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- 172.21.15.86:0/1353022685 shutdown_connections 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 --2- 172.21.15.86:0/1353022685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988813a440 0x7f988813c830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 --2- 172.21.15.86:0/1353022685 >> v2:172.21.15.203:3300/0 conn(0x7f988809e260 0x7f9888139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- 172.21.15.86:0/1353022685 >> 172.21.15.86:0/1353022685 conn(0x7f98880a2f40 msgr2=0x7f98880a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- 172.21.15.86:0/1353022685 shutdown_connections 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- 172.21.15.86:0/1353022685 wait complete. 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 10 monclient: build_initial_monmap 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 10 monclient: monmap: 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 Processor -- start 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 1 -- start start 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.092+0000 7f988d267640 10 monclient: init 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 0x7f98881cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): picked mon.noname-c con 0x7f988809d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988809e260 0x7f9888134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): picked mon.noname-b con 0x7f988809e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f988813a440 0x7f98881353c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): picked mon.noname-a con 0x7f988813a440 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f988813f710 con 0x7f988813a440 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f988813f590 con 0x7f988809e260 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f988813f890 con 0x7f988809d770 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): authenticate will time out at 1558.064271s 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 0x7f98881cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f988813a440 0x7f98881353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 0x7f98881cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55704/0 (socket says 172.21.15.86:55704) 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 -- 172.21.15.86:0/3322966157 learned_addr learned my addr 172.21.15.86:0/3322966157 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): get_auth_request con 0x7f988809d770 auth_method 0 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): get_auth_request con 0x7f988813a440 auth_method 0 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988809e260 0x7f9888134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient(hunting): handle_auth_done global_id 104210 payload 824 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988809e260 msgr2=0x7f9888134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988809e260 0x7f9888134e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f988813a440 msgr2=0x7f98881353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f988813a440 0x7f98881353c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 monclient: found mon.noname-c 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:05.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98881d0220 con 0x7f988809d770 2024-09-17T15:37:05.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98867fc640 10 monclient: get_auth_request con 0x7f988809e260 auth_method 0 2024-09-17T15:37:05.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988809e260 0x7f9888134e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:05.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 msgr2=0x7f98881cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9886ffd640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 0x7f98881cfce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f987c00c450 tx=0x7f987c00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c012a40 con 0x7f988809d770 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: dump: 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f987c017070 con 0x7f988809d770 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987c012e40 con 0x7f988809d770 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 5 monclient: authenticate success, global_id 104210 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _renew_subs 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9888122280 con 0x7f988809d770 2024-09-17T15:37:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f9884ff9640 4 set_mon_vals no callback set 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f986f7fe640 10 monclient: dump: 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _renew_subs 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9888121040 con 0x7f988809d770 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.096+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9888069db0 con 0x7f9888120520 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f987c01c460 con 0x7f988809d770 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient: get_auth_request con 0x7f9888120520 auth_method 0 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f987403bbe0 0x7f987403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(328..328 src has 213..328) ==== 4827+0+0 (secure 0 0 0) 0x7f987c04fd10 con 0x7f988809d770 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98877fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f987403bbe0 0x7f987403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98877fe640 10 monclient: get_auth_request con 0x7f987403bbe0 auth_method 0 2024-09-17T15:37:05.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 10 monclient(hunting): handle_auth_done global_id 114109 payload 824 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f988809db50 con 0x7f9888120520 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 msgr2=0x7f988811f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 msgr2=0x7f988811f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f987800e710 tx=0x7f987800ebd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.100+0000 7f98877fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f987403bbe0 0x7f987403e0a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9880009e50 tx=0x7f98800a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: get_auth_request con 0x7f9888120520 auth_method 0 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: _init_auth method 2 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: _init_auth already have auth, reseting 2024-09-17T15:37:05.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: handle_auth_reply_more payload 9 2024-09-17T15:37:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:05.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 10 monclient: handle_auth_done global_id 114109 payload 293 2024-09-17T15:37:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 msgr2=0x7f988811f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98780095a0 tx=0x7f9878008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987800f040 con 0x7f9888120520 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 10 monclient: dump: 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.307 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f987800ca30 con 0x7f9888120520 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 msgr2=0x7f988811f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98780095a0 tx=0x7f9878008040 comp rx=0 tx=0).stop 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 msgr2=0x7f988811f3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:05.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.304+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9888120520 0x7f988811f3c0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98780095a0 tx=0x7f9878008040 comp rx=0 tx=0).stop 2024-09-17T15:37:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f988d267640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f988d267640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 0x7f988802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f988d267640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9888072e00 con 0x7f988802d450 2024-09-17T15:37:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 0x7f988802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient: get_auth_request con 0x7f988802d450 auth_method 0 2024-09-17T15:37:05.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 10 monclient(hunting): handle_auth_done global_id 114112 payload 824 2024-09-17T15:37:05.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.312+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9888122000 con 0x7f988802d450 2024-09-17T15:37:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f98867fc640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 msgr2=0x7f988802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f98867fc640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 0x7f988802d860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f987801d000 tx=0x7f98780046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878027070 con 0x7f988802d450 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 10 monclient: dump: 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f9878000e50 con 0x7f988802d450 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 msgr2=0x7f988802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 0x7f988802d860 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f987801d000 tx=0x7f98780046d0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 msgr2=0x7f988802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f986f7fe640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f988802d450 0x7f988802d860 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f987801d000 tx=0x7f98780046d0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:05.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7815, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:05.322 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:05.323 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:05.328 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:05.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.332 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:05.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f987403bbe0 msgr2=0x7f987403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f987403bbe0 0x7f987403e0a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9880009e50 tx=0x7f98800a7000 comp rx=0 tx=0).stop 2024-09-17T15:37:05.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 10 monclient: shutdown 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 msgr2=0x7f98881cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 --2- 172.21.15.86:0/3322966157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f988809d770 0x7f98881cfce0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f987c00c450 tx=0x7f987c00c910 comp rx=0 tx=0).stop 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f98877fe640 1 -- 172.21.15.86:0/3322966157 reap_dead start 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f98877fe640 1 -- 172.21.15.86:0/3322966157 reap_dead start 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 shutdown_connections 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 >> 172.21.15.86:0/3322966157 conn(0x7f98880a2f40 msgr2=0x7f98880a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 shutdown_connections 2024-09-17T15:37:05.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.316+0000 7f988d267640 1 -- 172.21.15.86:0/3322966157 wait complete. 2024-09-17T15:37:05.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:05.343 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:37:05.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:37:05.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 10 monclient: get_monmap_and_config 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 10 monclient: build_initial_monmap 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 10 monclient: monmap: 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:05.470642+0000 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:05.470642+0000 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:05.468 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:05.469 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:05.469 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 1 Processor -- start 2024-09-17T15:37:05.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 1 -- start start 2024-09-17T15:37:05.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.464+0000 7eff9f143640 10 monclient: init 2024-09-17T15:37:05.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 0x7eff9806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-b con 0x7eff980689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff98068db0 0x7eff980691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-c con 0x7eff98068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff98135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9809c8b0 con 0x7eff98135ae0 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7eff98074500 con 0x7eff98068db0 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff9809cbe0 con 0x7eff980689d0 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:05.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9f143640 10 monclient(hunting): authenticate will time out at 1558.436422s 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 0x7eff9806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9d6e0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7eff98135ae0 0x7eff98135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43302/0 (socket says 172.21.15.86:43302) 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9d6e0640 1 -- 172.21.15.86:0/3349126584 learned_addr learned my addr 172.21.15.86:0/3349126584 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): get_auth_request con 0x7eff980689d0 auth_method 0 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient(hunting): handle_auth_done global_id 104213 payload 274 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 -- 172.21.15.86:0/3349126584 >> v2:172.21.15.203:3300/0 conn(0x7eff98068db0 msgr2=0x7eff980691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:05.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff97fff640 1 --2- 172.21.15.86:0/3349126584 >> v2:172.21.15.203:3300/0 conn(0x7eff98068db0 0x7eff980691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 --2- 172.21.15.86:0/3349126584 >> v2:172.21.15.203:3300/0 conn(0x7eff98068db0 0x7eff980691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 -- 172.21.15.86:0/3349126584 >> v1:172.21.15.86:6789/0 conn(0x7eff98135ae0 legacy=0x7eff98135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient: _finish_hunting 0 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 monclient: found mon.noname-b 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:05.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.468+0000 7eff9cedf640 1 -- 172.21.15.86:0/3349126584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9809ce70 con 0x7eff980689d0 2024-09-17T15:37:05.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9cedf640 1 -- 172.21.15.86:0/3349126584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 msgr2=0x7eff9806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9cedf640 1 --2- 172.21.15.86:0/3349126584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 0x7eff9806f840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097c0 tx=0x7eff8c01c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8634a4abbb7db147 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 1 -- 172.21.15.86:0/3349126584 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0b0070 con 0x7eff980689d0 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: dump: 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.476 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: _finish_auth 0 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: _check_auth_tickets 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 1 -- 172.21.15.86:0/3349126584 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8c0ab600 con 0x7eff980689d0 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 1 -- 172.21.15.86:0/3349126584 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0ab9d0 con 0x7eff980689d0 2024-09-17T15:37:05.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 5 monclient: authenticate success, global_id 104213 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 4 set_mon_vals no callback set 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff977fe640 10 monclient: dump: 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.478 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: shutdown 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- 172.21.15.86:0/3349126584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 msgr2=0x7eff9806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 --2- 172.21.15.86:0/3349126584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 0x7eff9806f840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097c0 tx=0x7eff8c01c300 comp rx=0 tx=0).stop 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- 172.21.15.86:0/3349126584 shutdown_connections 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 --2- 172.21.15.86:0/3349126584 >> v2:172.21.15.203:3300/0 conn(0x7eff98068db0 0x7eff980691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 --2- 172.21.15.86:0/3349126584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff980689d0 0x7eff9806f840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- 172.21.15.86:0/3349126584 >> 172.21.15.86:0/3349126584 conn(0x7eff980a6440 msgr2=0x7eff980a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- 172.21.15.86:0/3349126584 shutdown_connections 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- 172.21.15.86:0/3349126584 wait complete. 2024-09-17T15:37:05.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: build_initial_monmap 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: monmap: 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 Processor -- start 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 -- start start 2024-09-17T15:37:05.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: init 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.472+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-a con 0x7eff980689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff98068db0 0x7eff98134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-b con 0x7eff98068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:05.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 0x7eff981cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): picked mon.noname-c con 0x7eff98135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7eff981409b0 con 0x7eff980689d0 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff98140830 con 0x7eff98068db0 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff98140b30 con 0x7eff98135ae0 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient(hunting): authenticate will time out at 1558.444063s 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36158/0 (socket says 172.21.15.86:36158) 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 -- 172.21.15.86:0/3801618054 learned_addr learned my addr 172.21.15.86:0/3801618054 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:05.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): get_auth_request con 0x7eff980689d0 auth_method 0 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 0x7eff981cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): get_auth_request con 0x7eff98135ae0 auth_method 0 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:05.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff97fff640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff98068db0 0x7eff98134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff97fff640 10 monclient(hunting): get_auth_request con 0x7eff98068db0 auth_method 0 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient(hunting): handle_auth_done global_id 94332 payload 824 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 -- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 msgr2=0x7eff981cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 0x7eff981cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 -- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff98068db0 msgr2=0x7eff98134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff98068db0 0x7eff98134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient: _finish_hunting 0 2024-09-17T15:37:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 monclient: found mon.noname-a 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 -- 172.21.15.86:0/3801618054 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9811f510 con 0x7eff980689d0 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9d6e0640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 0x7eff981cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 -- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 msgr2=0x7eff981347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9cedf640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff8c01cba0 tx=0x7eff8c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0b9020 con 0x7eff980689d0 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: dump: 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: _finish_auth 0 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: _check_auth_tickets 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8c0b5690 con 0x7eff980689d0 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0b0040 con 0x7eff980689d0 2024-09-17T15:37:05.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff957fa640 10 monclient: dump: 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff96ffd640 4 set_mon_vals no callback set 2024-09-17T15:37:05.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 5 monclient: authenticate success, global_id 94332 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient: _renew_subs 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff98073610 con 0x7eff980689d0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient: _renew_subs 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.476+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9813d8f0 con 0x7eff980689d0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff9f143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff9f143640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff9f143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9806f840 con 0x7eff980689d0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7eff8c0bc020 con 0x7eff980689d0 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff957fa640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 0x7eff8403e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:05.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(328..328 src has 213..328) ==== 4827+0+0 (secure 0 0 0) 0x7eff8c0ea1d0 con 0x7eff980689d0 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff97fff640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 0x7eff8403e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff97fff640 10 monclient: get_auth_request con 0x7eff8403bbc0 auth_method 0 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.480+0000 7eff97fff640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 0x7eff8403e080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff9813deb0 tx=0x7eff88008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.484+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff8c0c3030 con 0x7eff980689d0 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.484+0000 7eff957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:05.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.484+0000 7eff957fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:05.690 INFO:tasks.rados.rados.0.smithi203.stdout:1409: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:37:05.690 INFO:tasks.rados.rados.0.smithi203.stdout:1409: finishing copy_from to smithi20328086-18 2024-09-17T15:37:05.690 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 496 (ObjNum 420 snap 123 seq_num 420) dirty exists 2024-09-17T15:37:05.690 INFO:tasks.rados.rados.0.smithi203.stdout:1409: done (1 left) 2024-09-17T15:37:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:1410: done (0 left) 2024-09-17T15:37:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:1411: write (excl) oid 36 current snap is 153 2024-09-17T15:37:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:1411: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-17T15:37:05.715 INFO:tasks.rados.rados.0.smithi203.stdout:1411: writing smithi20328086-36 from 621991 to 1420379 tid 1 2024-09-17T15:37:05.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.716+0000 7eff9f143640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:37:05.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.716+0000 7eff9f143640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:37:05.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.716+0000 7eff9f143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:05.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.716+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7eff9806d4f0 con 0x7eff980689d0 2024-09-17T15:37:05.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.720+0000 7eff957fa640 1 -- 172.21.15.86:0/3801618054 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7eff8c016070 con 0x7eff980689d0 2024-09-17T15:37:05.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.720+0000 7eff957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:37:05.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.720+0000 7eff957fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:05.723 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=2150178526,config=3541143443,health=3160483489,kv=3885551602,logm=3818272088}) 2024-09-17T15:37:05.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=3620093678,config=3541143443,health=185112578,kv=3591778543,logm=1263418158}) 2024-09-17T15:37:05.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=2150178526,config=3541143443,health=3160483489,kv=3885551602,logm=3818272088}) 2024-09-17T15:37:05.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.720+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=1066679490,config=3541143443,health=3334251417,kv=3591778543,logm=2547957211}) 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 msgr2=0x7eff8403e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 0x7eff8403e080 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff9813deb0 tx=0x7eff88008040 comp rx=0 tx=0).stop 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 10 monclient: shutdown 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 msgr2=0x7eff981347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff8c01cba0 tx=0x7eff8c019100 comp rx=0 tx=0).stop 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 shutdown_connections 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff8403bbc0 0x7eff8403e080 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff98135ae0 0x7eff981cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff98068db0 0x7eff98134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 --2- 172.21.15.86:0/3801618054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff980689d0 0x7eff981347d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:05.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 >> 172.21.15.86:0/3801618054 conn(0x7eff980a6440 msgr2=0x7eff980a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:05.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 shutdown_connections 2024-09-17T15:37:05.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:05.724+0000 7eff9f143640 1 -- 172.21.15.86:0/3801618054 wait complete. 2024-09-17T15:37:05.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752795463}) 2024-09-17T15:37:05.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3265327466}) 2024-09-17T15:37:05.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752795463}) 2024-09-17T15:37:05.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.724+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428702134}) 2024-09-17T15:37:05.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663534998}) 2024-09-17T15:37:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864518655}) 2024-09-17T15:37:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663534998}) 2024-09-17T15:37:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=418832147}) 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899863984}) 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714402247}) 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=899863984}) 2024-09-17T15:37:05.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.728+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534074607}) 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771642007}) 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3176067147}) 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=771642007}) 2024-09-17T15:37:05.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270521511}) 2024-09-17T15:37:05.735 INFO:tasks.rados.rados.0.smithi203.stdout:1411: writing smithi20328086-36 from 1936439 to 2575948 tid 2 2024-09-17T15:37:05.735 INFO:tasks.rados.rados.0.smithi203.stdout:1412: write (excl) oid 12 current snap is 153 2024-09-17T15:37:05.735 INFO:tasks.rados.rados.0.smithi203.stdout:1411: finishing write tid 1 to smithi20328086-36 2024-09-17T15:37:05.735 INFO:tasks.rados.rados.0.smithi203.stdout:1412: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382790030}) 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2544344702}) 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382790030}) 2024-09-17T15:37:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.732+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3336109413}) 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761421880}) 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=335732570}) 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=761421880}) 2024-09-17T15:37:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1330629746}) 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742710087}) 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=885501212}) 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742710087}) 2024-09-17T15:37:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463498482}) 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=926083218}) 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915275283}) 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=926083218}) 2024-09-17T15:37:05.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.736+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1934450893}) 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004796698}) 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774434312}) 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004796698}) 2024-09-17T15:37:05.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=73837986}) 2024-09-17T15:37:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574993534}) 2024-09-17T15:37:05.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705282096}) 2024-09-17T15:37:05.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2574993534}) 2024-09-17T15:37:05.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.740+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762559469}) 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792543480}) 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264476811}) 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792543480}) 2024-09-17T15:37:05.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=295179643}) 2024-09-17T15:37:05.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768314228}) 2024-09-17T15:37:05.748 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284567724}) 2024-09-17T15:37:05.748 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.748 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3768314228}) 2024-09-17T15:37:05.748 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.744+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556319960}) 2024-09-17T15:37:05.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:37:05.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562710554}) 2024-09-17T15:37:05.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=830804622}) 2024-09-17T15:37:05.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562710554}) 2024-09-17T15:37:05.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890474613}) 2024-09-17T15:37:05.751 INFO:tasks.rados.rados.0.smithi203.stdout:1412: writing smithi20328086-12 from 638798 to 1112435 tid 1 2024-09-17T15:37:05.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4051522463}) 2024-09-17T15:37:05.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253723459}) 2024-09-17T15:37:05.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4051522463}) 2024-09-17T15:37:05.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532486608}) 2024-09-17T15:37:05.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222414285}) 2024-09-17T15:37:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013354363}) 2024-09-17T15:37:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222414285}) 2024-09-17T15:37:05.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.748+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911758944}) 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399724711}) 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=492975357}) 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3399724711}) 2024-09-17T15:37:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284347408}) 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094538435}) 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341486782}) 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094538435}) 2024-09-17T15:37:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.752+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1427486803}) 2024-09-17T15:37:05.758 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.758 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945804907}) 2024-09-17T15:37:05.758 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2868654348}) 2024-09-17T15:37:05.758 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945804907}) 2024-09-17T15:37:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946617052}) 2024-09-17T15:37:05.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964274140}) 2024-09-17T15:37:05.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885017609}) 2024-09-17T15:37:05.760 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964274140}) 2024-09-17T15:37:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.756+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495179254}) 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1665675831}) 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015731823}) 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1665675831}) 2024-09-17T15:37:05.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597644172}) 2024-09-17T15:37:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2890496298}) 2024-09-17T15:37:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4262612440}) 2024-09-17T15:37:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2890496298}) 2024-09-17T15:37:05.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.760+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481233833}) 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175638784}) 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2707694794}) 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175638784}) 2024-09-17T15:37:05.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419276661}) 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717858323}) 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835246833}) 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717858323}) 2024-09-17T15:37:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.764+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352860191}) 2024-09-17T15:37:05.769 INFO:tasks.rados.rados.0.smithi203.stdout:1412: writing smithi20328086-12 from 1594921 to 2378088 tid 2 2024-09-17T15:37:05.769 INFO:tasks.rados.rados.0.smithi203.stdout:1413: snap_create 2024-09-17T15:37:05.769 INFO:tasks.rados.rados.0.smithi203.stdout:1411: finishing write tid 2 to smithi20328086-36 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: finishing write tid 3 to smithi20328086-36 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: finishing write tid 4 to smithi20328086-36 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: oid 36 updating version 0 to 615 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: oid 36 version 615 is already newer than 614 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: oid 36 updating version 615 to 616 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 616 (ObjNum 513 snap 153 seq_num 513) dirty exists 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1411: left oid 36 (ObjNum 513 snap 153 seq_num 513) 2024-09-17T15:37:05.770 INFO:tasks.rados.rados.0.smithi203.stdout:1412: finishing write tid 1 to smithi20328086-12 2024-09-17T15:37:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100516710}) 2024-09-17T15:37:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374408269}) 2024-09-17T15:37:05.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100516710}) 2024-09-17T15:37:05.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808157957}) 2024-09-17T15:37:05.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018656363}) 2024-09-17T15:37:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807346872}) 2024-09-17T15:37:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2018656363}) 2024-09-17T15:37:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.768+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327983329}) 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2655542712,mdsmap=2815901553,mgr=2751267415}) 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1795713045,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=269949400}) 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=2655542712,mdsmap=2815901553,mgr=2751267415}) 2024-09-17T15:37:05.774 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1991423479,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=269949400}) 2024-09-17T15:37:05.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=56,osdmap=5} crc {mgr=1747058506,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3323608917,mon_config_key=639531838,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=291412896,osd_snap=2652993790,osdmap=1543218801}) 2024-09-17T15:37:05.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=56,osdmap=7} crc {mgr=1747058506,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2551229617,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=291412896,osd_snap=346290915,osdmap=3038300629}) 2024-09-17T15:37:05.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=56,osdmap=5} crc {mgr=1747058506,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3323608917,mon_config_key=639531838,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=291412896,osd_snap=2652993790,osdmap=1543218801}) 2024-09-17T15:37:05.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.772+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=56,osdmap=7} crc {mgr=1747058506,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=736630645,mon_config_key=1293761627,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=291412896,osd_snap=1647365181,osdmap=3038300629}) 2024-09-17T15:37:05.777 INFO:tasks.rados.rados.0.smithi203.stdout:1412: finishing write tid 2 to smithi20328086-12 2024-09-17T15:37:05.777 INFO:tasks.rados.rados.0.smithi203.stdout:1412: finishing write tid 3 to smithi20328086-12 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:1412: finishing write tid 4 to smithi20328086-12 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:1412: oid 12 updating version 0 to 343 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:1412: oid 12 updating version 343 to 344 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:1412: oid 12 version 344 is already newer than 342 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 344 (ObjNum 514 snap 153 seq_num 514) dirty exists 2024-09-17T15:37:05.778 INFO:tasks.rados.rados.0.smithi203.stdout:1412: left oid 12 (ObjNum 514 snap 153 seq_num 514) 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=875806086}) 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=375605204}) 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=875806086}) 2024-09-17T15:37:05.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.776+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3104437708}) 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=2314048706}) 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=77} crc {osdmap=3801146024}) 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=2314048706}) 2024-09-17T15:37:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:37:05.784+0000 7f4051a22640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=2248779273}) 2024-09-17T15:37:06.697 INFO:tasks.rados.rados.0.smithi203.stdout:1411: done (2 left) 2024-09-17T15:37:06.697 INFO:tasks.rados.rados.0.smithi203.stdout:1412: done (1 left) 2024-09-17T15:37:06.697 INFO:tasks.rados.rados.0.smithi203.stdout:1413: done (0 left) 2024-09-17T15:37:06.698 INFO:tasks.rados.rados.0.smithi203.stdout:1414: delete oid 41 current snap is 154 2024-09-17T15:37:06.709 INFO:tasks.rados.rados.0.smithi203.stdout:1414: done (0 left) 2024-09-17T15:37:06.709 INFO:tasks.rados.rados.0.smithi203.stdout:1415: read oid 25 snap 153 2024-09-17T15:37:06.709 INFO:tasks.rados.rados.0.smithi203.stdout:1415: expect (ObjNum 423 snap 123 seq_num 423) 2024-09-17T15:37:06.709 INFO:tasks.rados.rados.0.smithi203.stdout:1416: write oid 11 current snap is 154 2024-09-17T15:37:06.709 INFO:tasks.rados.rados.0.smithi203.stdout:1416: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-17T15:37:06.730 INFO:tasks.rados.rados.0.smithi203.stdout:1416: writing smithi20328086-11 from 672412 to 1296547 tid 1 2024-09-17T15:37:06.750 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:37:06.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:06.753 INFO:tasks.rados.rados.0.smithi203.stdout:1416: writing smithi20328086-11 from 1995532 to 2672412 tid 2 2024-09-17T15:37:06.753 INFO:tasks.rados.rados.0.smithi203.stdout:1417: snap_create 2024-09-17T15:37:06.753 INFO:tasks.rados.rados.0.smithi203.stdout:1415: expect (ObjNum 423 snap 123 seq_num 423) 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: get_monmap_and_config 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: build_initial_monmap 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: monmap: 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:06.877911+0000 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:06.877911+0000 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.875 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:06.876 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:06.876 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:06.876 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:06.876 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:06.876 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 Processor -- start 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 -- start start 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: init 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae240689d0 0x7fae2406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-a con 0x7fae240689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae24068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae2413aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2409c8b0 con 0x7fae24068db0 2024-09-17T15:37:06.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae2409cb90 con 0x7fae2413aff0 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fae24074500 con 0x7fae240689d0 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae2ad98640 10 monclient(hunting): authenticate will time out at 1559.843364s 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae240689d0 0x7fae2406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae23fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fae24068db0 0x7fae24069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43306/0 (socket says 172.21.15.86:43306) 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): get_auth_request con 0x7fae240689d0 auth_method 0 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae23fff640 1 -- 172.21.15.86:0/132007972 learned_addr learned my addr 172.21.15.86:0/132007972 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:06.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient(hunting): handle_auth_done global_id 94335 payload 274 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 1 -- 172.21.15.86:0/132007972 >> v1:172.21.15.203:6789/0 conn(0x7fae2413aff0 legacy=0x7fae2413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 1 -- 172.21.15.86:0/132007972 >> v1:172.21.15.86:6789/0 conn(0x7fae24068db0 legacy=0x7fae24069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient: _finish_hunting 0 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 1 monclient: found mon.noname-a 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:06.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.872+0000 7fae28b34640 1 -- 172.21.15.86:0/132007972 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2409ce70 con 0x7fae240689d0 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 1 -- 172.21.15.86:0/132007972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae240689d0 msgr2=0x7fae2406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 1 --2- 172.21.15.86:0/132007972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae240689d0 0x7fae2406f840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae24069940 tx=0x7fae1001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc53b9708b7ba server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 1 -- 172.21.15.86:0/132007972 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae100b0070 con 0x7fae240689d0 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: dump: 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: _finish_auth 0 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: _check_auth_tickets 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 5 monclient: authenticate success, global_id 94335 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 1 -- 172.21.15.86:0/132007972 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae100ab600 con 0x7fae240689d0 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 4 set_mon_vals no callback set 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 1 -- 172.21.15.86:0/132007972 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2409c2d0 con 0x7fae240689d0 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae237fe640 10 monclient: dump: 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:06.884 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: shutdown 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- 172.21.15.86:0/132007972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae240689d0 msgr2=0x7fae2406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 --2- 172.21.15.86:0/132007972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae240689d0 0x7fae2406f840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae24069940 tx=0x7fae1001c340 comp rx=0 tx=0).stop 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- 172.21.15.86:0/132007972 shutdown_connections 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 --2- 172.21.15.86:0/132007972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae240689d0 0x7fae2406f840 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- 172.21.15.86:0/132007972 >> 172.21.15.86:0/132007972 conn(0x7fae240a6440 msgr2=0x7fae240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- 172.21.15.86:0/132007972 shutdown_connections 2024-09-17T15:37:06.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- 172.21.15.86:0/132007972 wait complete. 2024-09-17T15:37:06.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: build_initial_monmap 2024-09-17T15:37:06.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: monmap: 2024-09-17T15:37:06.889 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:06.889 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 Processor -- start 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- start start 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: init 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:06.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae240689d0 0x7fae24072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-b con 0x7fae240689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-a con 0x7fae24068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 0x7fae2409d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): picked mon.noname-c con 0x7fae2413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae24141c30 con 0x7fae24068db0 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae24141ab0 con 0x7fae240689d0 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:06.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fae24141db0 con 0x7fae2413aff0 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae2ad98640 10 monclient(hunting): authenticate will time out at 1559.850770s 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36164/0 (socket says 172.21.15.86:36164) 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 -- 172.21.15.86:0/3659660633 learned_addr learned my addr 172.21.15.86:0/3659660633 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): get_auth_request con 0x7fae24068db0 auth_method 0 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 0x7fae2409d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): get_auth_request con 0x7fae2413aff0 auth_method 0 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:06.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae240689d0 0x7fae24072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 10 monclient(hunting): get_auth_request con 0x7fae240689d0 auth_method 0 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae28b34640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient(hunting): handle_auth_done global_id 94338 payload 824 2024-09-17T15:37:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 -- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 msgr2=0x7fae2409d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 0x7fae2409d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 -- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae240689d0 msgr2=0x7fae24072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae240689d0 0x7fae24072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 monclient: found mon.noname-a 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae23fff640 1 -- 172.21.15.86:0/3659660633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae24073960 con 0x7fae24068db0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.880+0000 7fae29335640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 0x7fae2409d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae23fff640 1 -- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 msgr2=0x7fae24135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae23fff640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c4a0 tx=0x7fae1800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18017070 con 0x7fae24068db0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: dump: 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: _finish_auth 0 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: _check_auth_tickets 2024-09-17T15:37:06.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae18007070 con 0x7fae24068db0 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae18007460 con 0x7fae24068db0 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 5 monclient: authenticate success, global_id 94338 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _renew_subs 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae24121200 con 0x7fae24068db0 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 10 monclient: dump: 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:06.896 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae22ffd640 4 set_mon_vals no callback set 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _renew_subs 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae24073ba0 con 0x7fae24068db0 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:06.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2406f840 con 0x7fae24068db0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fae1801a070 con 0x7fae24068db0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 0x7fae0803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7fae1804f440 con 0x7fae24068db0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae28b34640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 0x7fae0803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.884+0000 7fae28b34640 10 monclient: get_auth_request con 0x7fae0803bb60 auth_method 0 2024-09-17T15:37:06.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.888+0000 7fae28b34640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 0x7fae0803e020 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae2406fdb0 tx=0x7fae100a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:06.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.888+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fae18007680 con 0x7fae24068db0 2024-09-17T15:37:06.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.888+0000 7fae217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:06.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:06.888+0000 7fae217fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:06.928 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1624339 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: finishing write tid 1 to smithi20328086-11 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: finishing write tid 2 to smithi20328086-11 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: finishing write tid 3 to smithi20328086-11 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: finishing write tid 4 to smithi20328086-11 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: oid 11 updating version 0 to 585 2024-09-17T15:37:06.967 INFO:tasks.rados.rados.0.smithi203.stdout:1416: oid 11 version 585 is already newer than 583 2024-09-17T15:37:06.968 INFO:tasks.rados.rados.0.smithi203.stdout:1416: oid 11 version 585 is already newer than 584 2024-09-17T15:37:06.968 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 585 (ObjNum 516 snap 154 seq_num 516) dirty exists 2024-09-17T15:37:06.968 INFO:tasks.rados.rados.0.smithi203.stdout:1416: left oid 11 (ObjNum 516 snap 154 seq_num 516) 2024-09-17T15:37:07.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae2ad98640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:07.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae2ad98640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae2ad98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fae2409e470 con 0x7fae24068db0 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae217fa640 1 -- 172.21.15.86:0/3659660633 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fae1804f0d0 con 0x7fae24068db0 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.128+0000 7fae217fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:07.132 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:07.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 msgr2=0x7fae0803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 0x7fae0803e020 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fae2406fdb0 tx=0x7fae100a6000 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 10 monclient: shutdown 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 msgr2=0x7fae24135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae1800c4a0 tx=0x7fae1800c960 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 shutdown_connections 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae0803bb60 0x7fae0803e020 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae2413aff0 0x7fae2409d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae24068db0 0x7fae24135b20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 --2- 172.21.15.86:0/3659660633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae240689d0 0x7fae24072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 >> 172.21.15.86:0/3659660633 conn(0x7fae240a6440 msgr2=0x7fae240a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 shutdown_connections 2024-09-17T15:37:07.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.132+0000 7fae2ad98640 1 -- 172.21.15.86:0/3659660633 wait complete. 2024-09-17T15:37:07.151 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:37:07.151 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:37:07.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:37:07.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: get_monmap_and_config 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: monmap: 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:07.269065+0000 2024-09-17T15:37:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:07.269065+0000 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:07.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 Processor -- start 2024-09-17T15:37:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 -- start start 2024-09-17T15:37:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: init 2024-09-17T15:37:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 0x7f2f9c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f9c0a1590 0x7f2f9c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c0a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 0x7f2f9c136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c135f90 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9c0a2410 con 0x7f2f9c0a1590 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9c0a2130 con 0x7f2f9c135f90 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c0a26f0 con 0x7f2f9c0a0c70 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2fa1a7f640 10 monclient(hunting): authenticate will time out at 1560.234588s 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f9c0a1590 0x7f2f9c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 0x7f2f9c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f9c0a1590 0x7f2f9c135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36180/0 (socket says 172.21.15.86:36180) 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 0x7f2f9c0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55736/0 (socket says 172.21.15.86:55736) 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 -- 172.21.15.86:0/1149237077 learned_addr learned my addr 172.21.15.86:0/1149237077 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): get_auth_request con 0x7f2f9c0a1590 auth_method 0 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2f9c0a0c70 auth_method 0 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9bfff640 1 --2- 172.21.15.86:0/1149237077 >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 0x7f2f9c136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient(hunting): handle_auth_done global_id 94341 payload 274 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 -- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 msgr2=0x7f2f9c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 0x7f2f9c0a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 -- 172.21.15.86:0/1149237077 >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 msgr2=0x7f2f9c136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 --2- 172.21.15.86:0/1149237077 >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 0x7f2f9c136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 monclient: found mon.noname-a 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9affd640 1 -- 172.21.15.86:0/1149237077 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c09f9f0 con 0x7f2f9c0a1590 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9bfff640 10 monclient: get_auth_request con 0x7f2f9c135f90 auth_method 0 2024-09-17T15:37:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.264+0000 7f2f9bfff640 1 --2- 172.21.15.86:0/1149237077 >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 0x7f2f9c136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9affd640 1 -- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 msgr2=0x7f2f9c135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9affd640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c135a50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0a40b0 tx=0x7f2f8c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdb59b28c5112fb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 1 -- 172.21.15.86:0/1149237077 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0b1070 con 0x7f2f9c0a1590 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: dump: 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 1 -- 172.21.15.86:0/1149237077 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0ac620 con 0x7f2f9c0a1590 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 1 -- 172.21.15.86:0/1149237077 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0ac9f0 con 0x7f2f9c0a1590 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2fa1a7f640 5 monclient: authenticate success, global_id 94341 2024-09-17T15:37:07.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2fa1a7f640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2fa1a7f640 4 set_mon_vals no callback set 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.268+0000 7f2f9a7fc640 10 monclient: dump: 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.281 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: shutdown 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 msgr2=0x7f2f9c135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c135a50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0a40b0 tx=0x7f2f8c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/1149237077 shutdown_connections 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/1149237077 >> v2:172.21.15.203:3300/0 conn(0x7f2f9c135f90 0x7f2f9c136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c135a50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/1149237077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c0a0c70 0x7f2f9c0a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/1149237077 >> 172.21.15.86:0/1149237077 conn(0x7f2f9c0a2f40 msgr2=0x7f2f9c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/1149237077 shutdown_connections 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/1149237077 wait complete. 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: monmap: 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 Processor -- start 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- start start 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: init 2024-09-17T15:37:07.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f9c0a0c70 0x7f2f9c1cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c0a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c11ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c0a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 0x7f2f9c134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c0a00d0 con 0x7f2f9c0a1590 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c09ff50 con 0x7f2f9c0a0c70 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f9c13f740 con 0x7f2f9c135f90 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2fa1a7f640 10 monclient(hunting): authenticate will time out at 1560.241675s 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c11ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 0x7f2f9c134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c11ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36182/0 (socket says 172.21.15.86:36182) 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 1 -- 172.21.15.86:0/572494874 learned_addr learned my addr 172.21.15.86:0/572494874 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): get_auth_request con 0x7f2f9c0a1590 auth_method 0 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): get_auth_request con 0x7f2f9c135f90 auth_method 0 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f9c0a0c70 0x7f2f9c1cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient(hunting): handle_auth_done global_id 104225 payload 824 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f9c0a0c70 msgr2=0x7f2f9c1cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f9c0a0c70 0x7f2f9c1cb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 msgr2=0x7f2f9c11ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c0a1590 0x7f2f9c11ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 monclient: found mon.noname-c 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c09c2e0 con 0x7f2f9c135f90 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9b7fe640 10 monclient: get_auth_request con 0x7f2f9c0a0c70 auth_method 0 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.272+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f9c0a0c70 0x7f2f9c1cb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 msgr2=0x7f2f9c134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9bfff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 0x7f2f9c134c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c5c0 tx=0x7f2f9400ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94015070 con 0x7f2f9c135f90 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:07.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: dump: 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f94011a30 con 0x7f2f9c135f90 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94011e20 con 0x7f2f9c135f90 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 5 monclient: authenticate success, global_id 104225 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 monclient: _renew_subs 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c073530 con 0x7f2f9c135f90 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f99ffb640 4 set_mon_vals no callback set 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 10 monclient: dump: 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.289 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 monclient: _renew_subs 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c120330 con 0x7f2f9c135f90 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 0x7f2f9c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c135a50 con 0x7f2f9c135190 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 0x7f2f9c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient: get_auth_request con 0x7f2f9c135190 auth_method 0 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2f94018070 con 0x7f2f9c135f90 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_done global_id 94347 payload 824 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9b7fe640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f9c0008e0 con 0x7f2f9c135190 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f7c03bb60 0x7f2f7c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7f2f9404fe40 con 0x7f2f9c135f90 2024-09-17T15:37:07.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9affd640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f7c03bb60 0x7f2f7c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.276+0000 7f2f9affd640 10 monclient: get_auth_request con 0x7f2f7c03bb60 auth_method 0 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f9b7fe640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 msgr2=0x7f2f9c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 0x7f2f9c11c8c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f8400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840039a0 con 0x7f2f9c135190 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 10 monclient: dump: 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.294 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f840039a0 con 0x7f2f9c135190 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 msgr2=0x7f2f9c11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 0x7f2f9c11c8c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f8400f520 comp rx=0 tx=0).stop 2024-09-17T15:37:07.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 msgr2=0x7f2f9c11c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f93fff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c135190 0x7f2f9c11c8c0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f8400f520 comp rx=0 tx=0).stop 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.280+0000 7f2f9affd640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f7c03bb60 0x7f2f7c03e020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c01c420 tx=0x7f2f8c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2fa1a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 0x7f2f9c11d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2fa1a7f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c11d720 con 0x7f2f9c11ce00 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 0x7f2f9c11d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient: get_auth_request con 0x7f2f9c11ce00 auth_method 0 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 10 monclient(hunting): handle_auth_done global_id 94350 payload 824 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.284+0000 7f2f9b7fe640 1 -- 172.21.15.86:0/572494874 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f9c06d4f0 con 0x7f2f9c11ce00 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f9b7fe640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 msgr2=0x7f2f9c11d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f9b7fe640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 0x7f2f9c11d1e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f84004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840147e0 con 0x7f2f9c11ce00 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 10 monclient: dump: 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.297 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f2f840147e0 con 0x7f2f9c11ce00 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 msgr2=0x7f2f9c11d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 0x7f2f9c11d1e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f84004a50 comp rx=0 tx=0).stop 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 msgr2=0x7f2f9c11d1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f93fff640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f9c11ce00 0x7f2f9c11d1e0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009f80 tx=0x7f2f84004a50 comp rx=0 tx=0).stop 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 102642, 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:07.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.300 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.301 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.302 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.303 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.304 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.306 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.307 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.308 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.309 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:07.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.311 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.312 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.313 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:07.313 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:07.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f7c03bb60 msgr2=0x7f2f7c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f7c03bb60 0x7f2f7c03e020 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c01c420 tx=0x7f2f8c008040 comp rx=0 tx=0).stop 2024-09-17T15:37:07.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 10 monclient: shutdown 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 msgr2=0x7f2f9c134c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 --2- 172.21.15.86:0/572494874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f9c135f90 0x7f2f9c134c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400c5c0 tx=0x7f2f9400ca80 comp rx=0 tx=0).stop 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 reap_dead start 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2f9bfff640 1 -- 172.21.15.86:0/572494874 reap_dead start 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 shutdown_connections 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 >> 172.21.15.86:0/572494874 conn(0x7f2f9c0a2f40 msgr2=0x7f2f9c074b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 shutdown_connections 2024-09-17T15:37:07.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.288+0000 7f2fa1a7f640 1 -- 172.21.15.86:0/572494874 wait complete. 2024-09-17T15:37:07.315 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:07.315 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:37:07.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: get_monmap_and_config 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: monmap: 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:07.424149+0000 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:07.424149+0000 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:07.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 Processor -- start 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 -- start start 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: init 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4a00a50e0 0x7fc4a009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a00a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a00a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a00a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc4a0137ea0 0x7fc4a01382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a0137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a00a6280 con 0x7fc4a00a50e0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a009c0a0 con 0x7fc4a0137ea0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a009c380 con 0x7fc4a00a54c0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a7cdc640 10 monclient(hunting): authenticate will time out at 1560.389605s 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4a00a50e0 0x7fc4a009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4a00a50e0 0x7fc4a009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36206/0 (socket says 172.21.15.86:36206) 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 -- 172.21.15.86:0/3158246250 learned_addr learned my addr 172.21.15.86:0/3158246250 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): get_auth_request con 0x7fc4a00a50e0 auth_method 0 2024-09-17T15:37:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a00a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): get_auth_request con 0x7fc4a00a54c0 auth_method 0 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a6279640 1 --2- 172.21.15.86:0/3158246250 >> v2:172.21.15.203:3300/0 conn(0x7fc4a0137ea0 0x7fc4a01382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a6279640 10 monclient(hunting): get_auth_request con 0x7fc4a0137ea0 auth_method 0 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a6279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a6279640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a6279640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_done global_id 94353 payload 274 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 -- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 msgr2=0x7fc4a00a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a00a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 -- 172.21.15.86:0/3158246250 >> v2:172.21.15.203:3300/0 conn(0x7fc4a0137ea0 msgr2=0x7fc4a01382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/3158246250 >> v2:172.21.15.203:3300/0 conn(0x7fc4a0137ea0 0x7fc4a01382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 monclient: found mon.noname-a 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.420+0000 7fc4a5a78640 1 -- 172.21.15.86:0/3158246250 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a009c660 con 0x7fc4a00a50e0 2024-09-17T15:37:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a5a78640 1 -- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 msgr2=0x7fc4a009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a009cd60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4a006f8c0 tx=0x7fc49801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784a69ef59b9f15d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 1 -- 172.21.15.86:0/3158246250 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980a56f0 con 0x7fc4a00a50e0 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: dump: 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 1 -- 172.21.15.86:0/3158246250 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4980a5890 con 0x7fc4a00a50e0 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 1 -- 172.21.15.86:0/3158246250 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980a5c60 con 0x7fc4a00a50e0 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 5 monclient: authenticate success, global_id 94353 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 4 set_mon_vals no callback set 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a4a76640 10 monclient: dump: 2024-09-17T15:37:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 10 monclient: shutdown 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 msgr2=0x7fc4a009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a009cd60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc4a006f8c0 tx=0x7fc49801c8b0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/3158246250 shutdown_connections 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/3158246250 >> v2:172.21.15.203:3300/0 conn(0x7fc4a0137ea0 0x7fc4a01382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a00a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/3158246250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a009cd60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/3158246250 >> 172.21.15.86:0/3158246250 conn(0x7fc4a00aaae0 msgr2=0x7fc4a00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/3158246250 shutdown_connections 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/3158246250 wait complete. 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.424+0000 7fc4a7cdc640 10 monclient: monmap: 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:07.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 Processor -- start 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- start start 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: init 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a01cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a00a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a01370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a00a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4a0137ea0 0x7fc4a013d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a0137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00a6ae0 con 0x7fc4a00a50e0 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00a6960 con 0x7fc4a0137ea0 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a0141b20 con 0x7fc4a00a54c0 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient(hunting): authenticate will time out at 1560.396551s 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a01cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a01370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a01cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36220/0 (socket says 172.21.15.86:36220) 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 -- 172.21.15.86:0/28493255 learned_addr learned my addr 172.21.15.86:0/28493255 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): get_auth_request con 0x7fc4a00a50e0 auth_method 0 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): get_auth_request con 0x7fc4a00a54c0 auth_method 0 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient(hunting): handle_auth_done global_id 94356 payload 824 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 msgr2=0x7fc4a01370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a00a54c0 0x7fc4a01370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4a0137ea0 msgr2=0x7fc4a013d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a6279640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4a0137ea0 0x7fc4a013d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4a0137ea0 0x7fc4a013d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 monclient: found mon.noname-a 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a00a3d70 con 0x7fc4a00a50e0 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 msgr2=0x7fc4a01cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a5a78640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a01cfe30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc49801af60 tx=0x7fc498019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980bc020 con 0x7fc4a00a50e0 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: dump: 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4980b8880 con 0x7fc4a00a50e0 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 5 monclient: authenticate success, global_id 94356 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _renew_subs 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a01d0cb0 con 0x7fc4a00a50e0 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4980a56f0 con 0x7fc4a00a50e0 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4967fc640 10 monclient: dump: 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc497fff640 4 set_mon_vals no callback set 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _renew_subs 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.428+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a00a79f0 con 0x7fc4a00a50e0 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a7cdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 0x7fc4a01221c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a006a0e0 con 0x7fc4a01375e0 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 0x7fc4a01221c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient: get_auth_request con 0x7fc4a01375e0 auth_method 0 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc4980bf020 con 0x7fc4a00a50e0 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 10 monclient(hunting): handle_auth_done global_id 104234 payload 824 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc4a00a58f0 con 0x7fc4a01375e0 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc48403bbe0 0x7fc48403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7fc4980eae20 con 0x7fc4a00a50e0 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a6279640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc48403bbe0 0x7fc48403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a6279640 10 monclient: get_auth_request con 0x7fc48403bbe0 auth_method 0 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 msgr2=0x7fc4a01221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a5277640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 0x7fc4a01221c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c011e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c019070 con 0x7fc4a01375e0 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 10 monclient: dump: 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.450 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc49c00f950 con 0x7fc4a01375e0 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 msgr2=0x7fc4a01221c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 0x7fc4a01221c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c011e60 comp rx=0 tx=0).stop 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 msgr2=0x7fc4a01221c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4967fc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a01375e0 0x7fc4a01221c0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c011e60 comp rx=0 tx=0).stop 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.432+0000 7fc4a6279640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc48403bbe0 0x7fc48403e0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc490009690 tx=0x7fc490008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a7cdc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 0x7fc4a013db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a7cdc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a013e060 con 0x7fc4a013d740 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 0x7fc4a013db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient: get_auth_request con 0x7fc4a013d740 auth_method 0 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 10 monclient(hunting): handle_auth_done global_id 104237 payload 824 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 1 -- 172.21.15.86:0/28493255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc4a0121630 con 0x7fc4a013d740 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 msgr2=0x7fc4a013db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4a5277640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 0x7fc4a013db20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c019040 con 0x7fc4a013d740 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.440+0000 7fc4967fc640 10 monclient: dump: 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.453 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fc49c003e00 con 0x7fc4a013d740 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 msgr2=0x7fc4a013db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 0x7fc4a013db20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 msgr2=0x7fc4a013db20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4967fc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4a013d740 0x7fc4a013db20 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc49c0119a0 tx=0x7fc49c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 58315, 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:07.455 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.456 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.457 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:07.458 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.459 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.460 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.461 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.462 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.463 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.465 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:07.466 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:37:07.467 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.468 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:07.468 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc48403bbe0 msgr2=0x7fc48403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc48403bbe0 0x7fc48403e0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc490009690 tx=0x7fc490008040 comp rx=0 tx=0).stop 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 10 monclient: shutdown 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 msgr2=0x7fc4a01cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 --2- 172.21.15.86:0/28493255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4a00a50e0 0x7fc4a01cfe30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc49801af60 tx=0x7fc498019650 comp rx=0 tx=0).stop 2024-09-17T15:37:07.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a6279640 1 -- 172.21.15.86:0/28493255 reap_dead start 2024-09-17T15:37:07.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a6279640 1 -- 172.21.15.86:0/28493255 reap_dead start 2024-09-17T15:37:07.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 shutdown_connections 2024-09-17T15:37:07.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 >> 172.21.15.86:0/28493255 conn(0x7fc4a00aaae0 msgr2=0x7fc4a00a3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 shutdown_connections 2024-09-17T15:37:07.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.444+0000 7fc4a7cdc640 1 -- 172.21.15.86:0/28493255 wait complete. 2024-09-17T15:37:07.470 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:07.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:37:07.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: get_monmap_and_config 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: monmap: 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:07.585005+0000 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:07.585005+0000 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.582 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:07.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 Processor -- start 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 -- start start 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: init 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5cc0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5cc09ea70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5cc13a4a0 0x7fe5cc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5cc13a4a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:07.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5cc09d410 con 0x7fe5cc0689d0 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe5cc0747b0 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc0a4540 con 0x7fe5cc09ea70 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5d0d5d640 10 monclient(hunting): authenticate will time out at 1560.551433s 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5cc13a4a0 0x7fe5cc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55792/0 (socket says 172.21.15.86:55792) 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5ca59c640 1 --1- 172.21.15.86:0/1260822858 >> v1:172.21.15.86:6789/0 conn(0x7fe5cc0689d0 0x7fe5cc0a26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43322/0 (socket says 172.21.15.86:43322) 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 1 -- 172.21.15.86:0/1260822858 learned_addr learned my addr 172.21.15.86:0/1260822858 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): get_auth_request con 0x7fe5cc09ea70 auth_method 0 2024-09-17T15:37:07.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): get_auth_request con 0x7fe5cc13a4a0 auth_method 0 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.580+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 10 monclient(hunting): handle_auth_done global_id 94359 payload 274 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 1 -- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 msgr2=0x7fe5cc0a2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 1 -- 172.21.15.86:0/1260822858 >> v1:172.21.15.86:6789/0 conn(0x7fe5cc0689d0 legacy=0x7fe5cc0a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 1 monclient: found mon.noname-a 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5cad9d640 1 -- 172.21.15.86:0/1260822858 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc0a4300 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.584+0000 7fe5c3fff640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 1 -- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc13a4a0 msgr2=0x7fe5cc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc13c890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009d90 tx=0x7fe5c401c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc968b48cccf178 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 1 -- 172.21.15.86:0/1260822858 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c40ac070 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: dump: 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 1 -- 172.21.15.86:0/1260822858 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c40a6620 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 5 monclient: authenticate success, global_id 94359 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 1 -- 172.21.15.86:0/1260822858 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c40a69f0 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 4 set_mon_vals no callback set 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c9d9b640 10 monclient: dump: 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: shutdown 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc13a4a0 msgr2=0x7fe5cc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc13c890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009d90 tx=0x7fe5c401c790 comp rx=0 tx=0).stop 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/1260822858 shutdown_connections 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc13c890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/1260822858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc0a2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/1260822858 >> 172.21.15.86:0/1260822858 conn(0x7fe5cc06d170 msgr2=0x7fe5cc0a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/1260822858 shutdown_connections 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/1260822858 wait complete. 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: monmap: 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 Processor -- start 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- start start 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: init 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc0689d0 0x7fe5cc1cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5cc0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc11eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe5cc09ea70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc134d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5cc13a4a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:07.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc09ba70 con 0x7fe5cc0689d0 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc0a3510 con 0x7fe5cc13a4a0 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe5cc09bbf0 con 0x7fe5cc09ea70 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5d0d5d640 10 monclient(hunting): authenticate will time out at 1560.558521s 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc11eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc11eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55796/0 (socket says 172.21.15.86:55796) 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 -- 172.21.15.86:0/106310602 learned_addr learned my addr 172.21.15.86:0/106310602 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): get_auth_request con 0x7fe5cc09ea70 auth_method 0 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc0689d0 0x7fe5cc1cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): get_auth_request con 0x7fe5cc0689d0 auth_method 0 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 10 monclient(hunting): get_auth_request con 0x7fe5cc13a4a0 auth_method 0 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient(hunting): handle_auth_done global_id 104243 payload 824 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc13a4a0 msgr2=0x7fe5cc134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc13a4a0 0x7fe5cc134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc0689d0 msgr2=0x7fe5cc1cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5cc0689d0 0x7fe5cc1cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 monclient: found mon.noname-c 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.588+0000 7fe5c3fff640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5cc0a2d00 con 0x7fe5cc09ea70 2024-09-17T15:37:07.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c3fff640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 msgr2=0x7fe5cc11eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c3fff640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc11eec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400e9b0 tx=0x7fe5b400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4017070 con 0x7fe5cc09ea70 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: dump: 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5b400cc10 con 0x7fe5cc09ea70 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5b4013470 con 0x7fe5cc09ea70 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 10 monclient: dump: 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c959a640 4 set_mon_vals no callback set 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 5 monclient: authenticate success, global_id 104243 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 monclient: _renew_subs 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5cc06efd0 con 0x7fe5cc09ea70 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 monclient: _renew_subs 2024-09-17T15:37:07.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5cc1203e0 con 0x7fe5cc09ea70 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 0x7fe5cc11d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc120060 con 0x7fe5cc135240 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe5b401a070 con 0x7fe5cc09ea70 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 0x7fe5cc11d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5ac03bb90 0x7fe5ac03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient: get_auth_request con 0x7fe5cc135240 auth_method 0 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7fe5b404ff00 con 0x7fe5cc09ea70 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5cad9d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5ac03bb90 0x7fe5ac03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.592+0000 7fe5cad9d640 10 monclient: get_auth_request con 0x7fe5ac03bb90 auth_method 0 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_done global_id 114127 payload 824 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5ca59c640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5cc09bfd0 con 0x7fe5cc135240 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5ca59c640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 msgr2=0x7fe5cc11d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5ca59c640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 0x7fe5cc11d2f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00e730 tx=0x7fe5bc00ebf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5cad9d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5ac03bb90 0x7fe5ac03e050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06db80 tx=0x7fe5c40a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc004d60 con 0x7fe5cc135240 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 10 monclient: dump: 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5bc019070 con 0x7fe5cc135240 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 msgr2=0x7fe5cc11d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 0x7fe5cc11d2f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00e730 tx=0x7fe5bc00ebf0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 msgr2=0x7fe5cc11d2f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.596+0000 7fe5c2ffd640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc135240 0x7fe5cc11d2f0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00e730 tx=0x7fe5bc00ebf0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5d0d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 0x7fe5cc11dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5d0d5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe5cc11e150 con 0x7fe5cc11d830 2024-09-17T15:37:07.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 0x7fe5cc11dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient: get_auth_request con 0x7fe5cc11d830 auth_method 0 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 10 monclient(hunting): handle_auth_done global_id 114130 payload 824 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.604+0000 7fe5ca59c640 1 -- 172.21.15.86:0/106310602 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe5cc06de20 con 0x7fe5cc11d830 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5ca59c640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 msgr2=0x7fe5cc11dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5ca59c640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 0x7fe5cc11dc10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00f010 tx=0x7fe5bc004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc01e230 con 0x7fe5cc11d830 2024-09-17T15:37:07.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 10 monclient: dump: 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fe5bc01e230 con 0x7fe5cc11d830 2024-09-17T15:37:07.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 msgr2=0x7fe5cc11dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 0x7fe5cc11dc10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00f010 tx=0x7fe5bc004940 comp rx=0 tx=0).stop 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 msgr2=0x7fe5cc11dc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5c2ffd640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5cc11d830 0x7fe5cc11dc10 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00f010 tx=0x7fe5bc004940 comp rx=0 tx=0).stop 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:37:07.615 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 94, 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10107, 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.616 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.617 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:07.618 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:07.619 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:07.623 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:07.624 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:07.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:07.627 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:07.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5ac03bb90 msgr2=0x7fe5ac03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe5ac03bb90 0x7fe5ac03e050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc06db80 tx=0x7fe5c40a7000 comp rx=0 tx=0).stop 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 10 monclient: shutdown 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 msgr2=0x7fe5cc11eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 --2- 172.21.15.86:0/106310602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5cc09ea70 0x7fe5cc11eec0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe5b400e9b0 tx=0x7fe5b400ee70 comp rx=0 tx=0).stop 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5cad9d640 1 -- 172.21.15.86:0/106310602 reap_dead start 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5cad9d640 1 -- 172.21.15.86:0/106310602 reap_dead start 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 shutdown_connections 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 >> 172.21.15.86:0/106310602 conn(0x7fe5cc06d170 msgr2=0x7fe5cc139bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 shutdown_connections 2024-09-17T15:37:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.608+0000 7fe5d0d5d640 1 -- 172.21.15.86:0/106310602 wait complete. 2024-09-17T15:37:07.630 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:37:07.630 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:37:07.630 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:37:07.630 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:37:07.630 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:37:07.630 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:37:07.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:37:07.642 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:37:07.643 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:37:07.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: get_monmap_and_config 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: monmap: 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:07.800976+0000 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:07.800976+0000 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:07.799 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 Processor -- start 2024-09-17T15:37:07.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 -- start start 2024-09-17T15:37:07.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: init 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3ccc0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ccc068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 0x7f3ccc13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-a con 0x7f3ccc13aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ccc074470 con 0x7f3ccc068db0 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3ccc09c8b0 con 0x7f3ccc13aff0 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3ccc09cb90 con 0x7f3ccc0689d0 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd40ec640 10 monclient(hunting): authenticate will time out at 1560.766525s 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1687640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f3ccc068db0 legacy=0x7f3ccc0691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd2689640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 msgr2=0x7f3ccc13d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd2689640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 0x7f3ccc13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55838/0 (socket says 172.21.15.86:55838) 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 -- 172.21.15.86:0/1464348272 learned_addr learned my addr 172.21.15.86:0/1464348272 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): get_auth_request con 0x7f3ccc0689d0 auth_method 0 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_done global_id 104246 payload 274 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 -- 172.21.15.86:0/1464348272 >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 msgr2=0x7f3ccc13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 --2- 172.21.15.86:0/1464348272 >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 0x7f3ccc13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 -- 172.21.15.86:0/1464348272 >> v1:172.21.15.86:6789/0 conn(0x7f3ccc068db0 legacy=0x7f3ccc0691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 monclient: found mon.noname-b 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.796+0000 7f3cd1e88640 1 -- 172.21.15.86:0/1464348272 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ccc09ce70 con 0x7f3ccc0689d0 2024-09-17T15:37:07.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 -- 172.21.15.86:0/1464348272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 msgr2=0x7f3ccc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 --2- 172.21.15.86:0/1464348272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009870 tx=0x7f3cc001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=967213c05d179d02 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 1 -- 172.21.15.86:0/1464348272 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00b1070 con 0x7f3ccc0689d0 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: dump: 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.807 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 5 monclient: authenticate success, global_id 104246 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 1 -- 172.21.15.86:0/1464348272 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc00a8050 con 0x7f3ccc0689d0 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 4 set_mon_vals no callback set 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 1 -- 172.21.15.86:0/1464348272 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc09c2d0 con 0x7f3ccc0689d0 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd0e86640 10 monclient: dump: 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: shutdown 2024-09-17T15:37:07.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- 172.21.15.86:0/1464348272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 msgr2=0x7f3ccc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/1464348272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0009870 tx=0x7f3cc001c440 comp rx=0 tx=0).stop 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- 172.21.15.86:0/1464348272 shutdown_connections 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/1464348272 >> v2:172.21.15.86:3300/0 conn(0x7f3ccc13aff0 0x7f3ccc13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/1464348272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc06f840 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- 172.21.15.86:0/1464348272 >> 172.21.15.86:0/1464348272 conn(0x7f3ccc0a6440 msgr2=0x7f3ccc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- 172.21.15.86:0/1464348272 shutdown_connections 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- 172.21.15.86:0/1464348272 wait complete. 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: build_initial_monmap 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: monmap: 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:07.810 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 Processor -- start 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- start start 2024-09-17T15:37:07.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: init 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-c con 0x7f3ccc0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 0x7f3ccc09d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-a con 0x7f3ccc068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ccc13aff0 0x7f3ccc09dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3ccc13aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:07.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc140670 con 0x7f3ccc068db0 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc1404f0 con 0x7f3ccc13aff0 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3ccc1407f0 con 0x7f3ccc0689d0 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1687640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 msgr2=0x7f3ccc09d7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd40ec640 10 monclient(hunting): authenticate will time out at 1560.774400s 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1687640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 0x7f3ccc09d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55840/0 (socket says 172.21.15.86:55840) 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 -- 172.21.15.86:0/626690809 learned_addr learned my addr 172.21.15.86:0/626690809 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): get_auth_request con 0x7f3ccc0689d0 auth_method 0 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient(hunting): handle_auth_done global_id 104249 payload 824 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 -- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ccc13aff0 msgr2=0x7f3ccc09dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ccc13aff0 0x7f3ccc09dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 -- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 msgr2=0x7f3ccc09d7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 0x7f3ccc09d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient: _finish_hunting 0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 monclient: found mon.noname-c 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.804+0000 7f3cd1e88640 1 -- 172.21.15.86:0/626690809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ccc0721a0 con 0x7f3ccc0689d0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd1e88640 1 -- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 msgr2=0x7f3ccc072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd1e88640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cc001cce0 tx=0x7f3cc001ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00bb020 con 0x7f3ccc0689d0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: dump: 2024-09-17T15:37:07.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc00b7620 con 0x7f3ccc0689d0 2024-09-17T15:37:07.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00b1040 con 0x7f3ccc0689d0 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 10 monclient: dump: 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:07.819 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbf7fe640 4 set_mon_vals no callback set 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 5 monclient: authenticate success, global_id 104249 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _renew_subs 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ccc071c20 con 0x7f3ccc0689d0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _renew_subs 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ccc13d2f0 con 0x7f3ccc0689d0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ccc09d460 con 0x7f3ccc0689d0 2024-09-17T15:37:07.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3cc00be020 con 0x7f3ccc0689d0 2024-09-17T15:37:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 0x7f3cb003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7f3cc00ea330 con 0x7f3ccc0689d0 2024-09-17T15:37:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd1687640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 0x7f3cb003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:07.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.808+0000 7f3cd1687640 10 monclient: get_auth_request con 0x7f3cb003bb10 auth_method 0 2024-09-17T15:37:07.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.812+0000 7f3cd1687640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 0x7f3cb003dfd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006fd0 tx=0x7f3cc8006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:07.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.812+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3cc00c5030 con 0x7f3ccc0689d0 2024-09-17T15:37:07.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.812+0000 7f3cbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:07.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:07.812+0000 7f3cbe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:08.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:08.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:08.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:08.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3ccc06f840 con 0x7f3ccc0689d0 2024-09-17T15:37:08.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cbe7fc640 1 -- 172.21.15.86:0/626690809 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3cc0016150 con 0x7f3ccc0689d0 2024-09-17T15:37:08.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:08.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cbe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:08.055 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 msgr2=0x7f3cb003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 0x7f3cb003dfd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006fd0 tx=0x7f3cc8006d40 comp rx=0 tx=0).stop 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 10 monclient: shutdown 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 msgr2=0x7f3ccc072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:08.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3cc001cce0 tx=0x7f3cc001ac80 comp rx=0 tx=0).stop 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 shutdown_connections 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3cb003bb10 0x7f3cb003dfd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ccc13aff0 0x7f3ccc09dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ccc068db0 0x7f3ccc09d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 --2- 172.21.15.86:0/626690809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ccc0689d0 0x7f3ccc072eb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.052+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 >> 172.21.15.86:0/626690809 conn(0x7f3ccc0a6440 msgr2=0x7f3ccc135c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.056+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 shutdown_connections 2024-09-17T15:37:08.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:08.056+0000 7f3cd40ec640 1 -- 172.21.15.86:0/626690809 wait complete. 2024-09-17T15:37:11.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: get_monmap_and_config 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: build_initial_monmap 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: monmap: 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:11.209121+0000 2024-09-17T15:37:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:11.209121+0000 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:11.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 Processor -- start 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 -- start start 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: init 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 0x7f136409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f13640689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1364068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1364139f00 0x7f136413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1364139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f136409d770 con 0x7f1364068db0 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f136409ed00 con 0x7f13640689d0 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:11.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f136409efe0 con 0x7f1364139f00 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136c1e8640 10 monclient(hunting): authenticate will time out at 1564.174814s 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f1369f84640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 msgr2=0x7f136409ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f1369f84640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 0x7f136409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1364139f00 0x7f136413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1364139f00 0x7f136413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:47022/0 (socket says 172.21.15.86:47022) 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 -- 172.21.15.86:0/2709859490 learned_addr learned my addr 172.21.15.86:0/2709859490 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): get_auth_request con 0x7f1364139f00 auth_method 0 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f1369783640 1 --1- 172.21.15.86:0/2709859490 >> v1:172.21.15.203:6789/0 conn(0x7f1364068db0 0x7f13640691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:37502/0 (socket says 172.21.15.86:37502) 2024-09-17T15:37:11.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient(hunting): handle_auth_done global_id 114133 payload 274 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 -- 172.21.15.86:0/2709859490 >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 msgr2=0x7f136409ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 --2- 172.21.15.86:0/2709859490 >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 0x7f136409ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 -- 172.21.15.86:0/2709859490 >> v1:172.21.15.203:6789/0 conn(0x7f1364068db0 legacy=0x7f13640691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient: _finish_hunting 0 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 monclient: found mon.noname-c 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:37:11.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.204+0000 7f136a785640 1 -- 172.21.15.86:0/2709859490 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13640a04e0 con 0x7f1364139f00 2024-09-17T15:37:11.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136a785640 1 -- 172.21.15.86:0/2709859490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364139f00 msgr2=0x7f136413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:11.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136a785640 1 --2- 172.21.15.86:0/2709859490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364139f00 0x7f136413c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f136409f2c0 tx=0x7f135401c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f171ab9475ac6039 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 1 -- 172.21.15.86:0/2709859490 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13540b0070 con 0x7f1364139f00 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: dump: 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:11.215 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: _finish_auth 0 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: _check_auth_tickets 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 1 -- 172.21.15.86:0/2709859490 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13540ab600 con 0x7f1364139f00 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:11.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 1 -- 172.21.15.86:0/2709859490 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13540a4e60 con 0x7f1364139f00 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1368f82640 10 monclient: dump: 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.217 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:11.218 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:11.218 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:11.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 5 monclient: authenticate success, global_id 114133 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 4 set_mon_vals no callback set 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: shutdown 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- 172.21.15.86:0/2709859490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364139f00 msgr2=0x7f136413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- 172.21.15.86:0/2709859490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364139f00 0x7f136413c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f136409f2c0 tx=0x7f135401c400 comp rx=0 tx=0).stop 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- 172.21.15.86:0/2709859490 shutdown_connections 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- 172.21.15.86:0/2709859490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364139f00 0x7f136413c2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- 172.21.15.86:0/2709859490 >> v2:172.21.15.86:3300/0 conn(0x7f13640689d0 0x7f136409ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- 172.21.15.86:0/2709859490 >> 172.21.15.86:0/2709859490 conn(0x7f13640a6440 msgr2=0x7f13640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- 172.21.15.86:0/2709859490 shutdown_connections 2024-09-17T15:37:11.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- 172.21.15.86:0/2709859490 wait complete. 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: build_initial_monmap 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: monmap: 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.226 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 Processor -- start 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- start start 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: init 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f13640689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364068db0 0x7f13640709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f1364068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:11.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 0x7f1364071210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1364139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f136409c610 con 0x7f1364139f00 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f136409c490 con 0x7f1364068db0 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f136409c790 con 0x7f13640689d0 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136c1e8640 10 monclient(hunting): authenticate will time out at 1564.183821s 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136a785640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 msgr2=0x7f1364071210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f136a785640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 0x7f1364071210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:11.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50742/0 (socket says 172.21.15.86:50742) 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 1 -- 172.21.15.86:0/3721717749 learned_addr learned my addr 172.21.15.86:0/3721717749 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 10 monclient(hunting): get_auth_request con 0x7f13640689d0 auth_method 0 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.212+0000 7f1369f84640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364068db0 0x7f13640709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): get_auth_request con 0x7f1364068db0 auth_method 0 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:11.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369783640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient(hunting): handle_auth_done global_id 104252 payload 824 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 -- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364068db0 msgr2=0x7f13640709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364068db0 0x7f13640709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 -- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 msgr2=0x7f1364071210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 0x7f1364071210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient: _finish_hunting 0 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 monclient: found mon.noname-c 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 -- 172.21.15.86:0/3721717749 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1364071f40 con 0x7f13640689d0 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 -- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 msgr2=0x7f1364131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f1369f84640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f135c00b550 tx=0x7f135c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0050f0 con 0x7f13640689d0 2024-09-17T15:37:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: dump: 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f135c015070 con 0x7f13640689d0 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 5 monclient: authenticate success, global_id 104252 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f135c0054d0 con 0x7f13640689d0 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:11.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13627fc640 10 monclient: dump: 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 10 monclient: _renew_subs 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1364073ce0 con 0x7f13640689d0 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f13637fe640 4 set_mon_vals no callback set 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 10 monclient: _renew_subs 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.216+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1364072d00 con 0x7f13640689d0 2024-09-17T15:37:11.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f136c1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f136c1e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f136c1e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136409ffa0 con 0x7f13640689d0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f135c0056b0 con 0x7f13640689d0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f13627fc640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 0x7f135003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7f135c04fd00 con 0x7f13640689d0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f1369783640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 0x7f135003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.220+0000 7f1369783640 10 monclient: get_auth_request con 0x7f135003bb60 auth_method 0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.224+0000 7f1369783640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 0x7f135003e020 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1364072a30 tx=0x7f1358009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.224+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f135c010dd0 con 0x7f13640689d0 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.224+0000 7f13627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:11.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.224+0000 7f13627fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:11.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f136c1e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:11.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f136c1e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:11.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f136c1e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:11.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f136406d4f0 con 0x7f13640689d0 2024-09-17T15:37:11.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f13627fc640 1 -- 172.21.15.86:0/3721717749 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f135c019cc0 con 0x7f13640689d0 2024-09-17T15:37:11.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f13627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:11.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.460+0000 7f13627fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:11.465 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 msgr2=0x7f135003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 0x7f135003e020 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1364072a30 tx=0x7f1358009040 comp rx=0 tx=0).stop 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 10 monclient: shutdown 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 msgr2=0x7f1364131e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f135c00b550 tx=0x7f135c00ba10 comp rx=0 tx=0).stop 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 shutdown_connections 2024-09-17T15:37:11.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f135003bb60 0x7f135003e020 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1364139f00 0x7f1364071210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1364068db0 0x7f13640709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 --2- 172.21.15.86:0/3721717749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13640689d0 0x7f1364131e40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 >> 172.21.15.86:0/3721717749 conn(0x7f13640a6440 msgr2=0x7f1364134c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 shutdown_connections 2024-09-17T15:37:11.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:11.464+0000 7f136c1e8640 1 -- 172.21.15.86:0/3721717749 wait complete. 2024-09-17T15:37:14.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:14.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 10 monclient: get_monmap_and_config 2024-09-17T15:37:14.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 10 monclient: build_initial_monmap 2024-09-17T15:37:14.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 10 monclient: monmap: 2024-09-17T15:37:14.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:14.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:14.614022+0000 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:14.614022+0000 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:14.612 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 1 Processor -- start 2024-09-17T15:37:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 1 -- start start 2024-09-17T15:37:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.608+0000 7fdd3d64b640 10 monclient: init 2024-09-17T15:37:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T15:37:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd380a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd380a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:14.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdd3813e300 0x7fdd381406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:14.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd3813e300 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:14.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd380a6530 con 0x7fdd380a54c0 2024-09-17T15:37:14.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3809c150 con 0x7fdd380a50e0 2024-09-17T15:37:14.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:14.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3809bdc0 con 0x7fdd3813e300 2024-09-17T15:37:14.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:14.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd3d64b640 10 monclient(hunting): authenticate will time out at 1567.580196s 2024-09-17T15:37:14.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd2e7fc640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fdd380a54c0 legacy=0x7fdd380a58d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:14.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdd3813e300 0x7fdd381406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:14.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdd3813e300 0x7fdd381406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:47052/0 (socket says 172.21.15.86:47052) 2024-09-17T15:37:14.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 -- 172.21.15.86:0/1326138561 learned_addr learned my addr 172.21.15.86:0/1326138561 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:14.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): get_auth_request con 0x7fdd3813e300 auth_method 0 2024-09-17T15:37:14.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:14.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient(hunting): handle_auth_done global_id 114139 payload 274 2024-09-17T15:37:14.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 -- 172.21.15.86:0/1326138561 >> v1:172.21.15.203:6789/0 conn(0x7fdd380a50e0 legacy=0x7fdd3809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:14.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 -- 172.21.15.86:0/1326138561 >> v1:172.21.15.86:6789/0 conn(0x7fdd380a54c0 legacy=0x7fdd380a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:14.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:14.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 monclient: found mon.noname-c 2024-09-17T15:37:14.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:14.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:37:14.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.612+0000 7fdd377fe640 1 -- 172.21.15.86:0/1326138561 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3809c430 con 0x7fdd3813e300 2024-09-17T15:37:14.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd377fe640 1 -- 172.21.15.86:0/1326138561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3813e300 msgr2=0x7fdd381406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:14.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd377fe640 1 --2- 172.21.15.86:0/1326138561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3813e300 0x7fdd381406f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009e50 tx=0x7fdd3001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=767aaabd84d6150d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:14.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 1 -- 172.21.15.86:0/1326138561 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300a5750 con 0x7fdd3813e300 2024-09-17T15:37:14.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:14.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: dump: 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:14.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 1 -- 172.21.15.86:0/1326138561 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd300a58f0 con 0x7fdd3813e300 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 1 -- 172.21.15.86:0/1326138561 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300a5cc0 con 0x7fdd3813e300 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 5 monclient: authenticate success, global_id 114139 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 4 set_mon_vals no callback set 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd367fc640 10 monclient: dump: 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.628 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 10 monclient: shutdown 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- 172.21.15.86:0/1326138561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3813e300 msgr2=0x7fdd381406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/1326138561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3813e300 0x7fdd381406f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009e50 tx=0x7fdd3001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- 172.21.15.86:0/1326138561 shutdown_connections 2024-09-17T15:37:14.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/1326138561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd3813e300 0x7fdd381406f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- 172.21.15.86:0/1326138561 >> 172.21.15.86:0/1326138561 conn(0x7fdd380aaae0 msgr2=0x7fdd380aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- 172.21.15.86:0/1326138561 shutdown_connections 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- 172.21.15.86:0/1326138561 wait complete. 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 10 monclient: build_initial_monmap 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 10 monclient: monmap: 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:14.630 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 Processor -- start 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 1 -- start start 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.616+0000 7fdd3d64b640 10 monclient: init 2024-09-17T15:37:14.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380a50e0 0x7fdd38137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-b con 0x7fdd380a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 0x7fdd38137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-a con 0x7fdd380a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): picked mon.noname-c con 0x7fdd3813e300 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38143980 con 0x7fdd380a54c0 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdd38143800 con 0x7fdd380a50e0 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdd38143b00 con 0x7fdd3813e300 2024-09-17T15:37:14.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient(hunting): authenticate will time out at 1567.588149s 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2e7fc640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 msgr2=0x7fdd38137e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2e7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 0x7fdd38137e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50744/0 (socket says 172.21.15.86:50744) 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 -- 172.21.15.86:0/2894816938 learned_addr learned my addr 172.21.15.86:0/2894816938 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:14.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): get_auth_request con 0x7fdd3813e300 auth_method 0 2024-09-17T15:37:14.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd36ffd640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380a50e0 0x7fdd38137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd36ffd640 10 monclient(hunting): get_auth_request con 0x7fdd380a50e0 auth_method 0 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient(hunting): handle_auth_done global_id 104255 payload 824 2024-09-17T15:37:14.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 -- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380a50e0 msgr2=0x7fdd38137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380a50e0 0x7fdd38137900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 -- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 msgr2=0x7fdd38137e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 0x7fdd38137e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 monclient: found mon.noname-c 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 -- 172.21.15.86:0/2894816938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd38120d30 con 0x7fdd3813e300 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 -- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 msgr2=0x7fdd3813b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd377fe640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd300af040 tx=0x7fdd30019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300bc020 con 0x7fdd3813e300 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:14.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: dump: 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:14.637 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: _finish_auth 0 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd300b8880 con 0x7fdd3813e300 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 5 monclient: authenticate success, global_id 104255 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _renew_subs 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd38121290 con 0x7fdd3813e300 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd300b8c70 con 0x7fdd3813e300 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:14.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd2ffff640 10 monclient: dump: 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd357fa640 4 set_mon_vals no callback set 2024-09-17T15:37:14.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _renew_subs 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.620+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd38120fb0 con 0x7fdd3813e300 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdd300bf020 con 0x7fdd3813e300 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd3d64b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd3d64b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd3d64b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd3809ce10 con 0x7fdd3813e300 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd2ffff640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 0x7fdd1803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7fdd300eb1f0 con 0x7fdd3813e300 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd36ffd640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 0x7fdd1803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:14.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd36ffd640 10 monclient: get_auth_request con 0x7fdd1803bb10 auth_method 0 2024-09-17T15:37:14.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.624+0000 7fdd36ffd640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 0x7fdd1803dfd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd28004e30 tx=0x7fdd28009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:14.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.628+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdd38120d30 con 0x7fdd3813e300 2024-09-17T15:37:14.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.628+0000 7fdd2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:14.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.628+0000 7fdd2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:14.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdd38074be0 con 0x7fdd3813e300 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd2ffff640 1 -- 172.21.15.86:0/2894816938 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fdd300165c0 con 0x7fdd3813e300 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:14.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd2ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:14.867 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:14.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 msgr2=0x7fdd1803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:14.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 0x7fdd1803dfd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd28004e30 tx=0x7fdd28009040 comp rx=0 tx=0).stop 2024-09-17T15:37:14.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 10 monclient: shutdown 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 msgr2=0x7fdd3813b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdd300af040 tx=0x7fdd30019650 comp rx=0 tx=0).stop 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 shutdown_connections 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdd1803bb10 0x7fdd1803dfd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd3813e300 0x7fdd3813b200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd380a54c0 0x7fdd38137e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 --2- 172.21.15.86:0/2894816938 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd380a50e0 0x7fdd38137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 >> 172.21.15.86:0/2894816938 conn(0x7fdd380aaae0 msgr2=0x7fdd380a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 shutdown_connections 2024-09-17T15:37:14.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:14.864+0000 7fdd3d64b640 1 -- 172.21.15.86:0/2894816938 wait complete. 2024-09-17T15:37:17.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: get_monmap_and_config 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: build_initial_monmap 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: monmap: 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:18.017768+0000 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:18.017768+0000 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:18.015 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:18.016 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 Processor -- start 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 -- start start 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: init 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-b con 0x7fcb540689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-a con 0x7fcb54068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb54135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409c8b0 con 0x7fcb54135ae0 2024-09-17T15:37:18.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcb54074500 con 0x7fcb54068db0 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5409cbe0 con 0x7fcb540689d0 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb58c9c640 10 monclient(hunting): authenticate will time out at 1570.983397s 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb52d9d640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fcb54135ae0 legacy=0x7fcb54135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb51d9b640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb540691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb51d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50760/0 (socket says 172.21.15.86:50760) 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 learned_addr learned my addr 172.21.15.86:0/3482587363 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:18.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:18.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:18.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.012+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb54135ae0 legacy=0x7fcb54135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb540691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:18.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:18.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:18.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:18.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:18.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.216+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb540691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb54135ae0 legacy=0x7fcb54135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:18.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:18.620+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb540691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:19.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb54135ae0 legacy=0x7fcb54135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:19.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:37:19.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:19.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:19.424+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:37:21.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): tick 2024-09-17T15:37:21.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:37:21.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb5406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb5406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb540691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb540691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb54135ae0 legacy=0x7fcb54135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient: _add_conns ranks=[1,4] 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb340036b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-c con 0x7fcb34007400 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409ce70 con 0x7fcb340036b0 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:21.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5409cbe0 con 0x7fcb34007400 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb43fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request con 0x7fcb34007400 auth_method 0 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:21.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.016+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.203:6789/0 conn(0x7fcb340036b0 legacy=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request con 0x7fcb34007400 auth_method 0 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.220+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request con 0x7fcb34007400 auth_method 0 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:21.624+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:22.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request con 0x7fcb34007400 auth_method 0 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:22.428+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 10 monclient(hunting): tick 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 msgr2=0x7fcb340098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34007400 0x7fcb340098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:24.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.203:6789/0 conn(0x7fcb340036b0 legacy=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb43fff640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.012+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 reap_dead start 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb3400a0d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-a con 0x7fcb54068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-b con 0x7fcb540689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:24.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5409c8b0 con 0x7fcb3400a0d0 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5409cbe0 con 0x7fcb54068db0 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcb340060c0 con 0x7fcb540689d0 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb43fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:24.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb34010180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:24.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:24.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.016+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.203:6789/0 conn(0x7fcb3400a0d0 legacy=0x7fcb3400a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:24.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb34010180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:24.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:24.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.220+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb34010180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:24.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:24.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:24.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:24.624+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:25.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:25.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb540689d0 auth_method 0 2024-09-17T15:37:25.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb34010180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:25.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:25.428+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): tick 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 msgr2=0x7fcb34005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb540689d0 0x7fcb34005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 msgr2=0x7fcb34010180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.86:3300/0 conn(0x7fcb54068db0 0x7fcb34010180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.203:6789/0 conn(0x7fcb3400a0d0 legacy=0x7fcb3400a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient: _add_conns ranks=[4,0] 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-c con 0x7fcb34006300 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb34006c80 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb340078b0 con 0x7fcb34006c80 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 1 -- 172.21.15.86:0/3482587363 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5409cbe0 con 0x7fcb34006300 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb43fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb34006c80 legacy=0x7fcb340126d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb34006300 auth_method 0 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.020+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:27.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb34006c80 legacy=0x7fcb340126d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:27.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:27.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb34006300 auth_method 0 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:27.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.223+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:37:27.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb34006c80 legacy=0x7fcb340126d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb34006300 auth_method 0 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 msgr2=0x7fcb34006740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:27.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:27.627+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb51d9b640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb34006c80 legacy=0x7fcb340126d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> v2:172.21.15.203:3300/0 conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): get_auth_request con 0x7fcb34006300 auth_method 0 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:28.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient(hunting): handle_auth_done global_id 114145 payload 274 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> v1:172.21.15.86:6789/0 conn(0x7fcb34006c80 legacy=0x7fcb340126d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient: _finish_hunting 0 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 1 monclient: found mon.noname-c 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:28.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:37:28.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.431+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb34009b70 con 0x7fcb34006300 2024-09-17T15:37:28.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.439+0000 7fcb5259c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb34006300 msgr2=0x7fcb34006740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.439+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb34006300 0x7fcb34006740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c300 tx=0x7fcb4801dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4004de0bff095a16 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 1 -- 172.21.15.86:0/3482587363 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb480ac070 con 0x7fcb34006300 2024-09-17T15:37:28.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: dump: 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: _finish_auth 0 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: _check_auth_tickets 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 1 -- 172.21.15.86:0/3482587363 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb480a6c30 con 0x7fcb34006300 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 1 -- 172.21.15.86:0/3482587363 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800f040 con 0x7fcb34006300 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 5 monclient: authenticate success, global_id 114145 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 4 set_mon_vals no callback set 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb5159a640 10 monclient: dump: 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 10 monclient: shutdown 2024-09-17T15:37:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3482587363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb34006300 msgr2=0x7fcb34006740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb34006300 0x7fcb34006740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c300 tx=0x7fcb4801dde0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3482587363 reap_dead start 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3482587363 shutdown_connections 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3482587363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb34006300 0x7fcb34006740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3482587363 >> 172.21.15.86:0/3482587363 conn(0x7fcb540a6440 msgr2=0x7fcb540a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3482587363 shutdown_connections 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.443+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3482587363 wait complete. 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient: build_initial_monmap 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient: monmap: 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.446 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 Processor -- start 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 -- start start 2024-09-17T15:37:28.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient: init 2024-09-17T15:37:28.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:28.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:28.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 0x7fcb541ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-a con 0x7fcb34006300 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:28.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 0x7fcb54123460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-b con 0x7fcb541ce830 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): picked mon.noname-c con 0x7fcb541cec90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54136700 con 0x7fcb34006300 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5409c2d0 con 0x7fcb541ce830 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcb54136880 con 0x7fcb541cec90 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb58c9c640 10 monclient(hunting): authenticate will time out at 1581.413600s 2024-09-17T15:37:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb5259c640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 msgr2=0x7fcb541ce2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb5259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 0x7fcb541ce2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47144/0 (socket says 172.21.15.86:47144) 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3937925900 learned_addr learned my addr 172.21.15.86:0/3937925900 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request con 0x7fcb541cec90 auth_method 0 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 0x7fcb54123460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request con 0x7fcb541ce830 auth_method 0 2024-09-17T15:37:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient(hunting): handle_auth_done global_id 104258 payload 824 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 msgr2=0x7fcb54123460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 0x7fcb54123460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 msgr2=0x7fcb541ce2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 0x7fcb541ce2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 monclient: found mon.noname-c 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3937925900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb54135830 con 0x7fcb541cec90 2024-09-17T15:37:28.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.447+0000 7fcb51d9b640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 0x7fcb54123460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:37:28.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb52d9d640 1 -- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 msgr2=0x7fcb541239a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb52d9d640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c003c20 tx=0x7fcb4c014b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c01d3f0 con 0x7fcb541cec90 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: dump: 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb4c009040 con 0x7fcb541cec90 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 5 monclient: authenticate success, global_id 104258 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _renew_subs 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb54124f20 con 0x7fcb541cec90 2024-09-17T15:37:28.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c01d7f0 con 0x7fcb541cec90 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 10 monclient: dump: 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.457 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb50d99640 4 set_mon_vals no callback set 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _renew_subs 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb54127280 con 0x7fcb541cec90 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcb4c01c3a0 con 0x7fcb541cec90 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb54139070 con 0x7fcb541cec90 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 0x7fcb3403fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7fcb4c050db0 con 0x7fcb541cec90 2024-09-17T15:37:28.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 0x7fcb3403fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.451+0000 7fcb5259c640 10 monclient: get_auth_request con 0x7fcb3403d9e0 auth_method 0 2024-09-17T15:37:28.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.455+0000 7fcb5259c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 0x7fcb3403fea0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb480ae000 tx=0x7fcb480a8070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.459+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcb4c019070 con 0x7fcb541cec90 2024-09-17T15:37:28.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.459+0000 7fcb42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:28.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.459+0000 7fcb42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb58c9c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb58c9c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb58c9c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcb54125a80 con 0x7fcb541cec90 2024-09-17T15:37:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb42ffd640 1 -- 172.21.15.86:0/3937925900 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fcb4c04f7d0 con 0x7fcb541cec90 2024-09-17T15:37:28.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:28.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.683+0000 7fcb42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:28.685 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":96,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:28.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 msgr2=0x7fcb3403fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 0x7fcb3403fea0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcb480ae000 tx=0x7fcb480a8070 comp rx=0 tx=0).stop 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 10 monclient: shutdown 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 msgr2=0x7fcb541239a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c003c20 tx=0x7fcb4c014b80 comp rx=0 tx=0).stop 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 shutdown_connections 2024-09-17T15:37:28.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcb3403d9e0 0x7fcb3403fea0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb541cec90 0x7fcb541239a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb541ce830 0x7fcb54123460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 --2- 172.21.15.86:0/3937925900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb34006300 0x7fcb541ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 >> 172.21.15.86:0/3937925900 conn(0x7fcb540a6440 msgr2=0x7fcb540a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:28.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 shutdown_connections 2024-09-17T15:37:28.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.687+0000 7fcb58c9c640 1 -- 172.21.15.86:0/3937925900 wait complete. 2024-09-17T15:37:28.702 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:37:28.703 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:37:28.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:37:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: get_monmap_and_config 2024-09-17T15:37:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: build_initial_monmap 2024-09-17T15:37:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: monmap: 2024-09-17T15:37:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:28.828329+0000 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:28.828329+0000 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:28.827 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 Processor -- start 2024-09-17T15:37:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 -- start start 2024-09-17T15:37:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: init 2024-09-17T15:37:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-b con 0x7f607c09d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f607c09e090 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 0x7f607c13cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-a con 0x7f607c135150 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f607c09ff00 con 0x7f607c09e090 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f607c09e7f0 con 0x7f607c135150 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f607c0a2110 con 0x7f607c09d770 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6080d30640 10 monclient(hunting): authenticate will time out at 1581.793777s 2024-09-17T15:37:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607ad9d640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 msgr2=0x7f607c13cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f6079d9b640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f607c09e090 legacy=0x7f607c134a40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607ad9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 0x7f607c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47154/0 (socket says 172.21.15.86:47154) 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 -- 172.21.15.86:0/1445838506 learned_addr learned my addr 172.21.15.86:0/1445838506 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): get_auth_request con 0x7f607c09d770 auth_method 0 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient(hunting): handle_auth_done global_id 104261 payload 274 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 -- 172.21.15.86:0/1445838506 >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 msgr2=0x7f607c13cf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 --2- 172.21.15.86:0/1445838506 >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 0x7f607c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 -- 172.21.15.86:0/1445838506 >> v1:172.21.15.86:6789/0 conn(0x7f607c09e090 legacy=0x7f607c134a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 monclient: found mon.noname-b 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.827+0000 7f607a59c640 1 -- 172.21.15.86:0/1445838506 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c09ec30 con 0x7f607c09d770 2024-09-17T15:37:28.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607a59c640 1 -- 172.21.15.86:0/1445838506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 msgr2=0x7f607c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607a59c640 1 --2- 172.21.15.86:0/1445838506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6064009870 tx=0x7f606401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbb5887ac753845a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 1 -- 172.21.15.86:0/1445838506 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640b0070 con 0x7f607c09d770 2024-09-17T15:37:28.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:28.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: dump: 2024-09-17T15:37:28.847 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: _finish_auth 0 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: _check_auth_tickets 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 1 -- 172.21.15.86:0/1445838506 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60640a4e50 con 0x7f607c09d770 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 1 -- 172.21.15.86:0/1445838506 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640ab820 con 0x7f607c09d770 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f607959a640 10 monclient: dump: 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 5 monclient: authenticate success, global_id 104261 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 4 set_mon_vals no callback set 2024-09-17T15:37:28.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 10 monclient: shutdown 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 -- 172.21.15.86:0/1445838506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 msgr2=0x7f607c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 --2- 172.21.15.86:0/1445838506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6064009870 tx=0x7f606401c440 comp rx=0 tx=0).stop 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 -- 172.21.15.86:0/1445838506 shutdown_connections 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 --2- 172.21.15.86:0/1445838506 >> v2:172.21.15.86:3300/0 conn(0x7f607c135150 0x7f607c13cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 --2- 172.21.15.86:0/1445838506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c09d770 0x7f607c09db50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 -- 172.21.15.86:0/1445838506 >> 172.21.15.86:0/1445838506 conn(0x7f607c0a2f40 msgr2=0x7f607c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 -- 172.21.15.86:0/1445838506 shutdown_connections 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.831+0000 7f6080d30640 1 -- 172.21.15.86:0/1445838506 wait complete. 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient: build_initial_monmap 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient: monmap: 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:28.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 Processor -- start 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 -- start start 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient: init 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:28.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f607c11d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-a con 0x7f607c09d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f607c13bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-b con 0x7f607c09e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): picked mon.noname-c con 0x7f607c135150 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f607c09cce0 con 0x7f607c09d770 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f607c09ce60 con 0x7f607c09e090 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f607c13fc80 con 0x7f607c135150 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6080d30640 10 monclient(hunting): authenticate will time out at 1581.801201s 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607a59c640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 msgr2=0x7f607c11d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f607c11d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47166/0 (socket says 172.21.15.86:47166) 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 learned_addr learned my addr 172.21.15.86:0/1314393704 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): get_auth_request con 0x7f607c135150 auth_method 0 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient(hunting): handle_auth_done global_id 104264 payload 824 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 msgr2=0x7f607c13bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f607c13bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 msgr2=0x7f607c11d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f607c11d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 monclient: found mon.noname-c 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c13cc60 con 0x7f607c135150 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 msgr2=0x7f607c134360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f607400be90 tx=0x7f607400bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074016630 con 0x7f607c135150 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6072ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.835+0000 7f6072ffd640 10 monclient: dump: 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.856 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 5 monclient: authenticate success, global_id 104264 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6074010030 con 0x7f607c135150 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 monclient: _renew_subs 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c06ff40 con 0x7f607c135150 2024-09-17T15:37:28.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074016a10 con 0x7f607c135150 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: dump: 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.858 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 monclient: _renew_subs 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6078d99640 4 set_mon_vals no callback set 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c11e450 con 0x7f607c135150 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 0x7f607c1cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f607c0a07a0 con 0x7f607c1348a0 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 0x7f607c1cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient: get_auth_request con 0x7f607c1348a0 auth_method 0 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_done global_id 104267 payload 824 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f607c09db50 con 0x7f607c1348a0 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 msgr2=0x7f607c134360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 msgr2=0x7f607c134360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f607400be90 tx=0x7f607400bf80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f607400be90 tx=0x7f607400bf80 comp rx=0 tx=0).stop 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:28.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 msgr2=0x7f607c134360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c135150 0x7f607c134360 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605c003820 0x7f605c005c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): picked mon.c con 0x7f605c003820 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f605c008730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): picked mon.a con 0x7f607c09d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f605c00b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): picked mon.b con 0x7f607c09e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f605c00bbc0 con 0x7f607c09d770 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f605c00bd40 con 0x7f607c09e090 2024-09-17T15:37:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f605c00bec0 con 0x7f605c003820 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (unknown 0 0 0) 0x7f6074006150 con 0x7f607c135150 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 msgr2=0x7f605c008730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f605c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605c003820 0x7f605c005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 10 monclient(hunting): get_auth_request con 0x7f605c003820 auth_method 0 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f605c0449c0 0x7f605c046e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f605c00b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): get_auth_request con 0x7f607c09e090 auth_method 0 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f605c0449c0 0x7f605c046e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6079d9b640 10 monclient(hunting): get_auth_request con 0x7f605c0449c0 auth_method 0 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient(hunting): handle_auth_done global_id 104264 payload 293 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605c003820 msgr2=0x7f605c005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605c003820 0x7f605c005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 msgr2=0x7f605c008730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:28.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f607c09d770 0x7f605c008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 monclient: found mon.b 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.203:3300/0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f605c00c1f0 con 0x7f607c09e090 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient: _finish_auth 0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 10 monclient: _check_auth_tickets 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 msgr2=0x7f607c1cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 0x7f607c1cd0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60640ad000 tx=0x7f606401ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640b0040 con 0x7f607c1348a0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: dump: 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60640ba040 con 0x7f607c1348a0 2024-09-17T15:37:28.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 msgr2=0x7f607c1cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 0x7f607c1cd0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60640ad000 tx=0x7f606401ac80 comp rx=0 tx=0).stop 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 msgr2=0x7f607c1cd0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.839+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c1348a0 0x7f607c1cd0c0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f60640ad000 tx=0x7f606401ac80 comp rx=0 tx=0).stop 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f607a59c640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 msgr2=0x7f605c00b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f607a59c640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f605c00b1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60640ab000 tx=0x7f60740095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6079d9b640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f605c0449c0 0x7f605c046e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f607c0709b0 tx=0x7f606c0059f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60640b0070 con 0x7f607c09e090 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:28.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 10 monclient: dump: 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.867 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6064003620 con 0x7f607c09e090 2024-09-17T15:37:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6078d99640 4 set_mon_vals no callback set 2024-09-17T15:37:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f60640b3070 con 0x7f607c09e090 2024-09-17T15:37:28.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.843+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.1 v2:172.21.15.203:3300/0 4 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7f60640ea1e0 con 0x7f607c09e090 2024-09-17T15:37:28.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f6080d30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f6080d30640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 0x7f607c13bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f6080d30640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f607c06d4f0 con 0x7f607c134360 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 0x7f607c13bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient: get_auth_request con 0x7f607c134360 auth_method 0 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:28.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 10 monclient(hunting): handle_auth_done global_id 104270 payload 824 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.847+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f607c1d03d0 con 0x7f607c134360 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 msgr2=0x7f607c13bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f607ad9d640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 0x7f607c13bc30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6074013a10 tx=0x7f60740136f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074013cb0 con 0x7f607c134360 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 10 monclient: dump: 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:28.872 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f6074013cb0 con 0x7f607c134360 2024-09-17T15:37:28.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 msgr2=0x7f607c13bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 0x7f607c13bc30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6074013a10 tx=0x7f60740136f0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 msgr2=0x7f607c13bc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6072ffd640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f607c134360 0x7f607c13bc30 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6074013a10 tx=0x7f60740136f0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:28.874 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 96, 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 79726, 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:28.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:28.886 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:28.887 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:37:28.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.889 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:37:28.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:28.891 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f605c0449c0 msgr2=0x7f605c046e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f605c0449c0 0x7f605c046e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f607c0709b0 tx=0x7f606c0059f0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 10 monclient: shutdown 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 msgr2=0x7f605c00b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f607ad9d640 1 -- 172.21.15.86:0/1314393704 reap_dead start 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f605c00b1b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60640ab000 tx=0x7f60740095e0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 shutdown_connections 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 --2- 172.21.15.86:0/1314393704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f607c09e090 0x7f605c00b1b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:28.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 >> 172.21.15.86:0/1314393704 conn(0x7f607c0a2f40 msgr2=0x7f607c073320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:28.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 shutdown_connections 2024-09-17T15:37:28.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:28.851+0000 7f6080d30640 1 -- 172.21.15.86:0/1314393704 wait complete. 2024-09-17T15:37:28.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:37:28.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:37:28.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:37:29.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: get_monmap_and_config 2024-09-17T15:37:29.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: build_initial_monmap 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: monmap: 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:29.008861+0000 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:29.008861+0000 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:29.011 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 Processor -- start 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 -- start start 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: init 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 0x7f55e013c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-a con 0x7f55e013c590 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55e00a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-b con 0x7f55e00a1790 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e00a20a0 con 0x7f55e00a0c70 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f55e00a2be0 con 0x7f55e013c590 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f55e009de50 con 0x7f55e00a1790 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55e5376640 10 monclient(hunting): authenticate will time out at 1581.974083s 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55deffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 msgr2=0x7f55e013c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55deffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 0x7f55e013c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47204/0 (socket says 172.21.15.86:47204) 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 -- 172.21.15.86:0/2422842066 learned_addr learned my addr 172.21.15.86:0/2422842066 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): get_auth_request con 0x7f55e00a1790 auth_method 0 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:29.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient(hunting): handle_auth_done global_id 104273 payload 274 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 -- 172.21.15.86:0/2422842066 >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 msgr2=0x7f55e013c970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 --2- 172.21.15.86:0/2422842066 >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 0x7f55e013c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 -- 172.21.15.86:0/2422842066 >> v1:172.21.15.203:6789/0 conn(0x7f55e00a0c70 legacy=0x7f55e00a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 monclient: found mon.noname-b 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:29.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.007+0000 7f55df7fe640 1 -- 172.21.15.86:0/2422842066 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e009db70 con 0x7f55e00a1790 2024-09-17T15:37:29.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55df7fe640 1 -- 172.21.15.86:0/2422842066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 msgr2=0x7f55e01444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:29.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55df7fe640 1 --2- 172.21.15.86:0/2422842066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009870 tx=0x7f55d801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e750ea08c529bf5d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 1 -- 172.21.15.86:0/2422842066 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80b0070 con 0x7f55e00a1790 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: dump: 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.021 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 1 -- 172.21.15.86:0/2422842066 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d80a4e50 con 0x7f55e00a1790 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 5 monclient: authenticate success, global_id 104273 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 1 -- 172.21.15.86:0/2422842066 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80ab820 con 0x7f55e00a1790 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 4 set_mon_vals no callback set 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55ddffb640 10 monclient: dump: 2024-09-17T15:37:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 10 monclient: shutdown 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 -- 172.21.15.86:0/2422842066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 msgr2=0x7f55e01444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 --2- 172.21.15.86:0/2422842066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009870 tx=0x7f55d801c440 comp rx=0 tx=0).stop 2024-09-17T15:37:29.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 -- 172.21.15.86:0/2422842066 shutdown_connections 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 --2- 172.21.15.86:0/2422842066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e00a1790 0x7f55e01444a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 --2- 172.21.15.86:0/2422842066 >> v2:172.21.15.86:3300/0 conn(0x7f55e013c590 0x7f55e013c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 -- 172.21.15.86:0/2422842066 >> 172.21.15.86:0/2422842066 conn(0x7f55e00a2f40 msgr2=0x7f55e00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 -- 172.21.15.86:0/2422842066 shutdown_connections 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.011+0000 7f55e5376640 1 -- 172.21.15.86:0/2422842066 wait complete. 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient: build_initial_monmap 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient: monmap: 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 Processor -- start 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 -- start start 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient: init 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e00a0c70 0x7f55e013b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-b con 0x7f55e00a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55e00a1790 0x7f55e0140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-a con 0x7f55e00a1790 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 0x7f55e0141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): picked mon.noname-c con 0x7f55e013c590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55e00a1bd0 con 0x7f55e00a1790 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55e009efe0 con 0x7f55e00a0c70 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f55e00a1d50 con 0x7f55e013c590 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55e5376640 10 monclient(hunting): authenticate will time out at 1581.981396s 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55de7fc640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55e00a1790 msgr2=0x7f55e0140fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 0x7f55e0141500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55de7fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55e00a1790 0x7f55e0140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 0x7f55e0141500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47216/0 (socket says 172.21.15.86:47216) 2024-09-17T15:37:29.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 learned_addr learned my addr 172.21.15.86:0/705175560 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): get_auth_request con 0x7f55e013c590 auth_method 0 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e00a0c70 0x7f55e013b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient(hunting): handle_auth_done global_id 104276 payload 824 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e00a0c70 msgr2=0x7f55e013b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e00a0c70 0x7f55e013b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55e00a1790 msgr2=0x7f55e0140fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55e00a1790 0x7f55e0140fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 monclient: found mon.noname-c 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e0125990 con 0x7f55e013c590 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55deffd640 10 monclient: get_auth_request con 0x7f55e00a0c70 auth_method 0 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.015+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e00a0c70 0x7f55e013b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 msgr2=0x7f55e0141500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55df7fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 0x7f55e0141500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f55d801c270 tx=0x7f55d80191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80b9020 con 0x7f55e013c590 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: dump: 2024-09-17T15:37:29.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: _finish_auth 0 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 5 monclient: authenticate success, global_id 104276 2024-09-17T15:37:29.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d80b1040 con 0x7f55e013c590 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80b0050 con 0x7f55e013c590 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 monclient: _renew_subs 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e0125c70 con 0x7f55e013c590 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55dd7fa640 4 set_mon_vals no callback set 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 10 monclient: dump: 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.030 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 monclient: _renew_subs 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e009d460 con 0x7f55e013c590 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 0x7f55e0141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55e00a51f0 con 0x7f55e013b740 2024-09-17T15:37:29.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f55d80bc020 con 0x7f55e013c590 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 0x7f55e0141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient: get_auth_request con 0x7f55e013b740 auth_method 0 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55cc03bb70 0x7f55cc03e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(330..330 src has 213..330) ==== 4739+0+0 (secure 0 0 0) 0x7f55d80e9ec0 con 0x7f55e013c590 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55de7fc640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55cc03bb70 0x7f55cc03e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55de7fc640 10 monclient: get_auth_request con 0x7f55cc03bb70 auth_method 0 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 10 monclient(hunting): handle_auth_done global_id 114151 payload 824 2024-09-17T15:37:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.019+0000 7f55deffd640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55e013c970 con 0x7f55e013b740 2024-09-17T15:37:29.033 INFO:tasks.rados.rados.0.smithi203.stdout:1415: done (2 left) 2024-09-17T15:37:29.033 INFO:tasks.rados.rados.0.smithi203.stdout:1416: done (1 left) 2024-09-17T15:37:29.034 INFO:tasks.rados.rados.0.smithi203.stdout:1417: done (0 left) 2024-09-17T15:37:29.034 INFO:tasks.rados.rados.0.smithi203.stdout:1418: copy_from oid 47 from oid 35 current snap is 155 2024-09-17T15:37:29.034 INFO:tasks.rados.rados.0.smithi203.stdout:1419: snap_create 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55deffd640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 msgr2=0x7f55e0141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 0x7f55e0141a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d400cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55de7fc640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55cc03bb70 0x7f55cc03e030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55d0007fd0 tx=0x7f55d0007da0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d4003dd0 con 0x7f55e013b740 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 10 monclient: dump: 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.036 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f55d4003dd0 con 0x7f55e013b740 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 msgr2=0x7f55e0141a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 0x7f55e0141a40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d400cbb0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 msgr2=0x7f55e0141a40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:29.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.023+0000 7f55c77fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e013b740 0x7f55e0141a40 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d400cbb0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55e5376640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55e5376640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 0x7f55e013d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55e5376640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55e01250d0 con 0x7f55e0141f80 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 0x7f55e013d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient: get_auth_request con 0x7f55e0141f80 auth_method 0 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:29.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 10 monclient(hunting): handle_auth_done global_id 114154 payload 824 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.031+0000 7f55deffd640 1 -- 172.21.15.86:0/705175560 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f55e0142390 con 0x7f55e0141f80 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55deffd640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 msgr2=0x7f55e013d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55deffd640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 0x7f55e013d1d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d40059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d401ce00 con 0x7f55e0141f80 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 10 monclient: dump: 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:29.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7f55d401c320 con 0x7f55e0141f80 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 msgr2=0x7f55e013d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 0x7f55e013d1d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d40059e0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 msgr2=0x7f55e013d1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:37:29.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55c77fe640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55e0141f80 0x7f55e013d1d0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f55d400c6f0 tx=0x7f55d40059e0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 96, 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 31531, 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:37:29.041 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:29.042 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:29.043 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:37:29.044 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:37:29.045 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:29.046 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:29.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.048 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:37:29.049 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:29.051 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:37:29.052 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:37:29.053 INFO:tasks.rados.rados.0.smithi203.stdout:1418: finishing copy_from to smithi20328086-47 2024-09-17T15:37:29.053 INFO:tasks.rados.rados.0.smithi203.stdout:1418: got expected ENOENT (src dne) 2024-09-17T15:37:29.053 INFO:tasks.rados.rados.0.smithi203.stdout:1418: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:37:29.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55cc03bb70 msgr2=0x7f55cc03e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55cc03bb70 0x7f55cc03e030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f55d0007fd0 tx=0x7f55d0007da0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 10 monclient: shutdown 2024-09-17T15:37:29.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:29.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 msgr2=0x7f55e0141500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 --2- 172.21.15.86:0/705175560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55e013c590 0x7f55e0141500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f55d801c270 tx=0x7f55d80191e0 comp rx=0 tx=0).stop 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 reap_dead start 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55df7fe640 1 -- 172.21.15.86:0/705175560 reap_dead start 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 shutdown_connections 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 >> 172.21.15.86:0/705175560 conn(0x7f55e00a2f40 msgr2=0x7f55e00a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 shutdown_connections 2024-09-17T15:37:29.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:29.035+0000 7f55e5376640 1 -- 172.21.15.86:0/705175560 wait complete. 2024-09-17T15:37:29.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:37:29.055 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:37:30.037 INFO:tasks.rados.rados.0.smithi203.stdout:1418: done (1 left) 2024-09-17T15:37:30.037 INFO:tasks.rados.rados.0.smithi203.stdout:1419: done (0 left) 2024-09-17T15:37:30.037 INFO:tasks.rados.rados.0.smithi203.stdout:1420: snap_create 2024-09-17T15:37:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:1420: done (0 left) 2024-09-17T15:37:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:1421: snap_remove snap 154 2024-09-17T15:37:32.196 INFO:tasks.rados.rados.0.smithi203.stdout:1421: done (0 left) 2024-09-17T15:37:32.196 INFO:tasks.rados.rados.0.smithi203.stdout:1422: read oid 32 snap -1 2024-09-17T15:37:32.196 INFO:tasks.rados.rados.0.smithi203.stdout:1422: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:37:32.197 INFO:tasks.rados.rados.0.smithi203.stdout:1423: delete oid 7 current snap is 157 2024-09-17T15:37:32.208 INFO:tasks.rados.rados.0.smithi203.stdout:1423: done (1 left) 2024-09-17T15:37:32.208 INFO:tasks.rados.rados.0.smithi203.stdout:1424: delete oid 8 current snap is 157 2024-09-17T15:37:32.211 INFO:tasks.rados.rados.0.smithi203.stdout:1422: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:37:32.378 INFO:tasks.rados.rados.0.smithi203.stdout:1422: done (1 left) 2024-09-17T15:37:32.378 INFO:tasks.rados.rados.0.smithi203.stdout:1424: done (0 left) 2024-09-17T15:37:32.378 INFO:tasks.rados.rados.0.smithi203.stdout:1425: write oid 37 current snap is 157 2024-09-17T15:37:32.378 INFO:tasks.rados.rados.0.smithi203.stdout:1425: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-17T15:37:32.398 INFO:tasks.rados.rados.0.smithi203.stdout:1425: writing smithi20328086-37 from 722833 to 1289068 tid 1 2024-09-17T15:37:32.420 INFO:tasks.rados.rados.0.smithi203.stdout:1425: writing smithi20328086-37 from 1887331 to 2537847 tid 2 2024-09-17T15:37:32.420 INFO:tasks.rados.rados.0.smithi203.stdout:1426: snap_remove snap 153 2024-09-17T15:37:32.516 INFO:tasks.rados.rados.0.smithi203.stdout:1426: done (1 left) 2024-09-17T15:37:32.517 INFO:tasks.rados.rados.0.smithi203.stdout:1427: write oid 26 current snap is 157 2024-09-17T15:37:32.517 INFO:tasks.rados.rados.0.smithi203.stdout:1427: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-17T15:37:32.539 INFO:tasks.rados.rados.0.smithi203.stdout:1427: writing smithi20328086-26 from 739640 to 1381124 tid 1 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1427: writing smithi20328086-26 from 1829460 to 2623634 tid 2 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1428: rollback oid 48 current snap is 157 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 155 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: finishing write tid 1 to smithi20328086-37 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: finishing write tid 2 to smithi20328086-37 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: finishing write tid 3 to smithi20328086-37 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: finishing write tid 4 to smithi20328086-37 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: oid 37 updating version 0 to 549 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: oid 37 updating version 549 to 550 2024-09-17T15:37:32.562 INFO:tasks.rados.rados.0.smithi203.stdout:1425: oid 37 updating version 550 to 551 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 551 (ObjNum 519 snap 157 seq_num 519) dirty exists 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:1425: left oid 37 (ObjNum 519 snap 157 seq_num 519) 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:1427: finishing write tid 1 to smithi20328086-26 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:1425: done (2 left) 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:1429: write (excl) oid 30 current snap is 157 2024-09-17T15:37:32.563 INFO:tasks.rados.rados.0.smithi203.stdout:1429: seq_num 521 ranges {756447=716733,1887942=537832} 2024-09-17T15:37:32.579 INFO:tasks.rados.rados.0.smithi203.stdout:1429: writing smithi20328086-30 from 756447 to 1473180 tid 1 2024-09-17T15:37:32.589 INFO:tasks.rados.rados.0.smithi203.stdout:1429: writing smithi20328086-30 from 1887942 to 2425774 tid 2 2024-09-17T15:37:32.589 INFO:tasks.rados.rados.0.smithi203.stdout:1430: write oid 33 current snap is 157 2024-09-17T15:37:32.589 INFO:tasks.rados.rados.0.smithi203.stdout:1428: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:37:32.589 INFO:tasks.rados.rados.0.smithi203.stdout:1427: finishing write tid 2 to smithi20328086-26 2024-09-17T15:37:32.589 INFO:tasks.rados.rados.0.smithi203.stdout:1430: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-17T15:37:32.602 INFO:tasks.rados.rados.0.smithi203.stdout:1430: writing smithi20328086-33 from 773254 to 1565236 tid 1 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1430: writing smithi20328086-33 from 2346424 to 2773254 tid 2 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1431: copy_from oid 17 from oid 24 current snap is 157 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: finishing write tid 3 to smithi20328086-26 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: finishing write tid 4 to smithi20328086-26 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: oid 26 updating version 0 to 618 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: oid 26 updating version 618 to 619 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: oid 26 updating version 619 to 620 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 620 (ObjNum 520 snap 157 seq_num 520) dirty exists 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: left oid 26 (ObjNum 520 snap 157 seq_num 520) 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1428: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1427: done (4 left) 2024-09-17T15:37:32.614 INFO:tasks.rados.rados.0.smithi203.stdout:1432: write (excl) oid 21 current snap is 157 2024-09-17T15:37:32.615 INFO:tasks.rados.rados.0.smithi203.stdout:1432: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-17T15:37:32.621 INFO:tasks.rados.rados.0.smithi203.stdout:1432: writing smithi20328086-21 from 790061 to 1257292 tid 1 2024-09-17T15:37:32.629 INFO:tasks.rados.rados.0.smithi203.stdout:1432: writing smithi20328086-21 from 2004906 to 2546407 tid 2 2024-09-17T15:37:32.629 INFO:tasks.rados.rados.0.smithi203.stdout:1433: write (excl) oid 44 current snap is 157 2024-09-17T15:37:32.630 INFO:tasks.rados.rados.0.smithi203.stdout:1433: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-17T15:37:32.636 INFO:tasks.rados.rados.0.smithi203.stdout:1433: writing smithi20328086-44 from 406868 to 949348 tid 1 2024-09-17T15:37:32.641 INFO:tasks.rados.rados.0.smithi203.stdout:1433: writing smithi20328086-44 from 1547035 to 1948547 tid 2 2024-09-17T15:37:32.647 INFO:tasks.rados.rados.0.smithi203.stdout:1433: writing smithi20328086-44 from 2690354 to 2806868 tid 3 2024-09-17T15:37:32.647 INFO:tasks.rados.rados.0.smithi203.stdout:1434: write (excl) oid 38 current snap is 157 2024-09-17T15:37:32.647 INFO:tasks.rados.rados.0.smithi203.stdout:1428: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:37:32.647 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 555 (ObjNum 474 snap 144 seq_num 474) dirty exists 2024-09-17T15:37:32.647 INFO:tasks.rados.rados.0.smithi203.stdout:1434: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2024-09-17T15:37:32.653 INFO:tasks.rados.rados.0.smithi203.stdout:1434: writing smithi20328086-38 from 423675 to 1157757 tid 1 2024-09-17T15:37:32.660 INFO:tasks.rados.rados.0.smithi203.stdout:1434: writing smithi20328086-38 from 1721870 to 2267040 tid 2 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1434: writing smithi20328086-38 from 2717777 to 2823675 tid 3 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1428: done (6 left) 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1435: rollback oid 45 current snap is 157 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 156 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1429: finishing write tid 1 to smithi20328086-30 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1429: finishing write tid 2 to smithi20328086-30 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1429: finishing write tid 3 to smithi20328086-30 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1429: finishing write tid 4 to smithi20328086-30 2024-09-17T15:37:32.664 INFO:tasks.rados.rados.0.smithi203.stdout:1429: oid 30 updating version 0 to 346 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1429: oid 30 updating version 346 to 347 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1429: oid 30 updating version 347 to 348 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 348 (ObjNum 521 snap 157 seq_num 521) dirty exists 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1429: left oid 30 (ObjNum 521 snap 157 seq_num 521) 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: finishing write tid 1 to smithi20328086-33 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: finishing write tid 2 to smithi20328086-33 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: finishing write tid 3 to smithi20328086-33 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: finishing write tid 4 to smithi20328086-33 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: oid 33 updating version 0 to 310 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: oid 33 version 310 is already newer than 309 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: oid 33 version 310 is already newer than 308 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 310 (ObjNum 522 snap 157 seq_num 522) dirty exists 2024-09-17T15:37:32.665 INFO:tasks.rados.rados.0.smithi203.stdout:1430: left oid 33 (ObjNum 522 snap 157 seq_num 522) 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1432: finishing write tid 1 to smithi20328086-21 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1431: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1429: done (6 left) 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1430: done (5 left) 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1436: read oid 30 snap -1 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1436: expect (ObjNum 521 snap 157 seq_num 521) 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1437: read oid 3 snap -1 2024-09-17T15:37:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:1437: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1438: write oid 47 current snap is 157 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: finishing write tid 2 to smithi20328086-21 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: finishing write tid 3 to smithi20328086-21 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: finishing write tid 4 to smithi20328086-21 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: oid 21 updating version 0 to 623 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: oid 21 updating version 623 to 624 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: oid 21 version 624 is already newer than 622 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 624 (ObjNum 523 snap 157 seq_num 523) dirty exists 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1432: left oid 21 (ObjNum 523 snap 157 seq_num 523) 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1431: finishing copy_from to smithi20328086-17 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 488 (ObjNum 446 snap 136 seq_num 446) dirty exists 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1433: finishing write tid 1 to smithi20328086-44 2024-09-17T15:37:32.667 INFO:tasks.rados.rados.0.smithi203.stdout:1438: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-17T15:37:32.668 INFO:tasks.rados.rados.0.smithi203.stdout:1438: writing smithi20328086-47 from 440482 to 849813 tid 1 2024-09-17T15:37:32.674 INFO:tasks.rados.rados.0.smithi203.stdout:1438: writing smithi20328086-47 from 1380352 to 1785533 tid 2 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1438: writing smithi20328086-47 from 2461553 to 2840482 tid 3 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1431: done (7 left) 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1432: done (6 left) 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1439: rollback oid 5 current snap is 157 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 156 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1433: finishing write tid 2 to smithi20328086-44 2024-09-17T15:37:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1433: finishing write tid 3 to smithi20328086-44 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: finishing write tid 4 to smithi20328086-44 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: finishing write tid 5 to smithi20328086-44 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: oid 44 updating version 0 to 590 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: oid 44 version 590 is already newer than 589 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: oid 44 updating version 590 to 592 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: oid 44 version 592 is already newer than 591 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 592 (ObjNum 524 snap 157 seq_num 524) dirty exists 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: left oid 44 (ObjNum 524 snap 157 seq_num 524) 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1433: done (6 left) 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1440: rollback oid 43 current snap is 157 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 155 2024-09-17T15:37:32.682 INFO:tasks.rados.rados.0.smithi203.stdout:1437: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:32.696 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 921151 2024-09-17T15:37:32.735 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:32.792 INFO:tasks.rados.rados.0.smithi203.stdout:1434: finishing write tid 1 to smithi20328086-38 2024-09-17T15:37:32.792 INFO:tasks.rados.rados.0.smithi203.stdout:1434: finishing write tid 2 to smithi20328086-38 2024-09-17T15:37:32.792 INFO:tasks.rados.rados.0.smithi203.stdout:1434: finishing write tid 3 to smithi20328086-38 2024-09-17T15:37:32.792 INFO:tasks.rados.rados.0.smithi203.stdout:1436: expect (ObjNum 521 snap 157 seq_num 521) 2024-09-17T15:37:32.792 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:32.847 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:32.902 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1435: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1435: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1438: finishing write tid 1 to smithi20328086-47 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: finishing write tid 4 to smithi20328086-38 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1435: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 308 (ObjNum 462 snap 140 seq_num 462) dirty exists 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: finishing write tid 5 to smithi20328086-38 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: oid 38 updating version 0 to 505 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: oid 38 updating version 505 to 507 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: oid 38 version 507 is already newer than 506 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:1434: oid 38 version 507 is already newer than 504 2024-09-17T15:37:32.958 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 507 (ObjNum 525 snap 157 seq_num 525) dirty exists 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1434: left oid 38 (ObjNum 525 snap 157 seq_num 525) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1438: finishing write tid 2 to smithi20328086-47 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1439: finishing rollback tid 0 to smithi20328086-5 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1439: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1439: finishing rollback tid 2 to smithi20328086-5 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 313 (ObjNum 503 snap 149 seq_num 503) dirty exists 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1434: done (6 left) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1435: done (5 left) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1436: done (4 left) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1437: done (3 left) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1439: done (2 left) 2024-09-17T15:37:32.959 INFO:tasks.rados.rados.0.smithi203.stdout:1441: snap_remove snap 156 2024-09-17T15:37:33.529 INFO:tasks.rados.rados.0.smithi203.stdout:1441: done (2 left) 2024-09-17T15:37:33.529 INFO:tasks.rados.rados.0.smithi203.stdout:1442: write (excl) oid 16 current snap is 157 2024-09-17T15:37:33.529 INFO:tasks.rados.rados.0.smithi203.stdout:1442: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-17T15:37:33.538 INFO:tasks.rados.rados.0.smithi203.stdout:1442: writing smithi20328086-16 from 457289 to 941869 tid 1 2024-09-17T15:37:33.549 INFO:tasks.rados.rados.0.smithi203.stdout:1442: writing smithi20328086-16 from 1438834 to 1987673 tid 2 2024-09-17T15:37:33.557 INFO:tasks.rados.rados.0.smithi203.stdout:1442: writing smithi20328086-16 from 2772623 to 2857289 tid 3 2024-09-17T15:37:33.557 INFO:tasks.rados.rados.0.smithi203.stdout:1443: write oid 31 current snap is 157 2024-09-17T15:37:33.558 INFO:tasks.rados.rados.0.smithi203.stdout:1443: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-17T15:37:33.564 INFO:tasks.rados.rados.0.smithi203.stdout:1443: writing smithi20328086-31 from 474096 to 1033925 tid 1 2024-09-17T15:37:33.570 INFO:tasks.rados.rados.0.smithi203.stdout:1443: writing smithi20328086-31 from 1780963 to 2189813 tid 2 2024-09-17T15:37:33.575 INFO:tasks.rados.rados.0.smithi203.stdout:1443: writing smithi20328086-31 from 2800046 to 2874096 tid 3 2024-09-17T15:37:33.575 INFO:tasks.rados.rados.0.smithi203.stdout:1444: write oid 18 current snap is 157 2024-09-17T15:37:33.576 INFO:tasks.rados.rados.0.smithi203.stdout:1440: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:37:33.576 INFO:tasks.rados.rados.0.smithi203.stdout:1438: finishing write tid 3 to smithi20328086-47 2024-09-17T15:37:33.576 INFO:tasks.rados.rados.0.smithi203.stdout:1438: finishing write tid 4 to smithi20328086-47 2024-09-17T15:37:33.576 INFO:tasks.rados.rados.0.smithi203.stdout:1444: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-17T15:37:33.581 INFO:tasks.rados.rados.0.smithi203.stdout:1444: writing smithi20328086-18 from 490903 to 1125981 tid 1 2024-09-17T15:37:33.588 INFO:tasks.rados.rados.0.smithi203.stdout:1444: writing smithi20328086-18 from 1839445 to 2391953 tid 2 2024-09-17T15:37:33.588 INFO:tasks.rados.rados.0.smithi203.stdout:1445: write (excl) oid 23 current snap is 157 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: finishing write tid 5 to smithi20328086-47 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: oid 47 updating version 0 to 497 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: oid 47 updating version 497 to 498 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: oid 47 updating version 498 to 499 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: oid 47 updating version 499 to 500 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 500 (ObjNum 526 snap 157 seq_num 526) dirty exists 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1438: left oid 47 (ObjNum 526 snap 157 seq_num 526) 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1440: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1440: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 291 (ObjNum 490 snap 148 seq_num 490) dirty exists 2024-09-17T15:37:33.589 INFO:tasks.rados.rados.0.smithi203.stdout:1445: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-09-17T15:37:33.595 INFO:tasks.rados.rados.0.smithi203.stdout:1445: writing smithi20328086-23 from 507710 to 1218037 tid 1 2024-09-17T15:37:33.602 INFO:tasks.rados.rados.0.smithi203.stdout:1445: writing smithi20328086-23 from 1897927 to 2310446 tid 2 2024-09-17T15:37:33.606 INFO:tasks.rados.rados.0.smithi203.stdout:1445: writing smithi20328086-23 from 2854892 to 2907710 tid 3 2024-09-17T15:37:33.606 INFO:tasks.rados.rados.0.smithi203.stdout:1438: done (5 left) 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1440: done (4 left) 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1446: delete oid 15 current snap is 157 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: finishing write tid 1 to smithi20328086-16 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: finishing write tid 2 to smithi20328086-16 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: finishing write tid 3 to smithi20328086-16 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: finishing write tid 4 to smithi20328086-16 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: finishing write tid 5 to smithi20328086-16 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: oid 16 updating version 0 to 560 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: oid 16 version 560 is already newer than 557 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: oid 16 updating version 560 to 562 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: oid 16 version 562 is already newer than 561 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 562 (ObjNum 527 snap 157 seq_num 527) dirty exists 2024-09-17T15:37:33.607 INFO:tasks.rados.rados.0.smithi203.stdout:1442: left oid 16 (ObjNum 527 snap 157 seq_num 527) 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: finishing write tid 1 to smithi20328086-31 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: finishing write tid 2 to smithi20328086-31 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: finishing write tid 3 to smithi20328086-31 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: finishing write tid 4 to smithi20328086-31 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: finishing write tid 5 to smithi20328086-31 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: oid 31 updating version 0 to 184 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: oid 31 version 184 is already newer than 183 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: oid 31 version 184 is already newer than 182 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: oid 31 updating version 184 to 185 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 185 (ObjNum 528 snap 157 seq_num 528) dirty exists 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1443: left oid 31 (ObjNum 528 snap 157 seq_num 528) 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1444: finishing write tid 1 to smithi20328086-18 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1444: finishing write tid 2 to smithi20328086-18 2024-09-17T15:37:33.608 INFO:tasks.rados.rados.0.smithi203.stdout:1444: finishing write tid 3 to smithi20328086-18 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1444: finishing write tid 4 to smithi20328086-18 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1444: oid 18 updating version 0 to 501 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1444: oid 18 updating version 501 to 502 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1444: oid 18 updating version 502 to 503 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 503 (ObjNum 529 snap 157 seq_num 529) dirty exists 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1444: left oid 18 (ObjNum 529 snap 157 seq_num 529) 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1445: finishing write tid 1 to smithi20328086-23 2024-09-17T15:37:33.609 INFO:tasks.rados.rados.0.smithi203.stdout:1442: done (4 left) 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1443: done (3 left) 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1444: done (2 left) 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1446: done (1 left) 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1447: delete oid 46 current snap is 157 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1445: finishing write tid 2 to smithi20328086-23 2024-09-17T15:37:33.610 INFO:tasks.rados.rados.0.smithi203.stdout:1445: finishing write tid 3 to smithi20328086-23 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: finishing write tid 4 to smithi20328086-23 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: finishing write tid 5 to smithi20328086-23 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: oid 23 updating version 0 to 509 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: oid 23 updating version 509 to 510 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: oid 23 version 510 is already newer than 508 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: oid 23 updating version 510 to 511 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 511 (ObjNum 530 snap 157 seq_num 530) dirty exists 2024-09-17T15:37:33.611 INFO:tasks.rados.rados.0.smithi203.stdout:1445: left oid 23 (ObjNum 530 snap 157 seq_num 530) 2024-09-17T15:37:33.615 INFO:tasks.rados.rados.0.smithi203.stdout:1445: done (1 left) 2024-09-17T15:37:33.615 INFO:tasks.rados.rados.0.smithi203.stdout:1447: done (0 left) 2024-09-17T15:37:33.615 INFO:tasks.rados.rados.0.smithi203.stdout:1448: delete oid 25 current snap is 157 2024-09-17T15:37:33.620 INFO:tasks.rados.rados.0.smithi203.stdout:1448: done (0 left) 2024-09-17T15:37:33.621 INFO:tasks.rados.rados.0.smithi203.stdout:1449: rollback oid 45 current snap is 157 2024-09-17T15:37:33.621 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 155 2024-09-17T15:37:33.621 INFO:tasks.rados.rados.0.smithi203.stdout:1450: snap_create 2024-09-17T15:37:33.627 INFO:tasks.rados.rados.0.smithi203.stdout:1449: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:37:33.628 INFO:tasks.rados.rados.0.smithi203.stdout:1449: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:37:33.628 INFO:tasks.rados.rados.0.smithi203.stdout:1449: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:37:33.628 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 313 (ObjNum 462 snap 140 seq_num 462) dirty exists 2024-09-17T15:37:34.532 INFO:tasks.rados.rados.0.smithi203.stdout:1449: done (1 left) 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1450: done (0 left) 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1451: rollback oid 20 current snap is 158 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 157 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1452: read oid 39 snap -1 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1452: expect deleted 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1453: copy_from oid 48 from oid 32 current snap is 158 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1454: write (excl) oid 6 current snap is 158 2024-09-17T15:37:34.533 INFO:tasks.rados.rados.0.smithi203.stdout:1454: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-17T15:37:34.544 INFO:tasks.rados.rados.0.smithi203.stdout:1454: writing smithi20328086-6 from 574938 to 1302614 tid 1 2024-09-17T15:37:34.553 INFO:tasks.rados.rados.0.smithi203.stdout:1454: writing smithi20328086-6 from 1731855 to 2435359 tid 2 2024-09-17T15:37:34.558 INFO:tasks.rados.rados.0.smithi203.stdout:1454: writing smithi20328086-6 from 2848231 to 2974938 tid 3 2024-09-17T15:37:34.558 INFO:tasks.rados.rados.0.smithi203.stdout:1455: write oid 25 current snap is 158 2024-09-17T15:37:34.558 INFO:tasks.rados.rados.0.smithi203.stdout:1455: seq_num 535 ranges {591745=402925,1790337=563515} 2024-09-17T15:37:34.566 INFO:tasks.rados.rados.0.smithi203.stdout:1455: writing smithi20328086-25 from 591745 to 994670 tid 1 2024-09-17T15:37:34.576 INFO:tasks.rados.rados.0.smithi203.stdout:1455: writing smithi20328086-25 from 1790337 to 2353852 tid 2 2024-09-17T15:37:34.576 INFO:tasks.rados.rados.0.smithi203.stdout:1456: write (excl) oid 37 current snap is 158 2024-09-17T15:37:34.576 INFO:tasks.rados.rados.0.smithi203.stdout:1456: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-17T15:37:34.583 INFO:tasks.rados.rados.0.smithi203.stdout:1456: writing smithi20328086-37 from 608552 to 1086726 tid 1 2024-09-17T15:37:34.594 INFO:tasks.rados.rados.0.smithi203.stdout:1456: writing smithi20328086-37 from 1848819 to 2555992 tid 2 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1457: copy_from oid 41 from oid 16 current snap is 158 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1458: rollback oid 13 current snap is 158 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 155 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1451: finishing rollback tid 0 to smithi20328086-20 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1451: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1451: finishing rollback tid 2 to smithi20328086-20 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 515 (ObjNum 479 snap 145 seq_num 479) dirty exists 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1453: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1454: finishing write tid 1 to smithi20328086-6 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1454: finishing write tid 2 to smithi20328086-6 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:1453: finishing copy_from to smithi20328086-48 2024-09-17T15:37:34.595 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 307 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: finishing write tid 3 to smithi20328086-6 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: finishing write tid 4 to smithi20328086-6 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: finishing write tid 5 to smithi20328086-6 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: oid 6 updating version 0 to 520 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: oid 6 version 520 is already newer than 519 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: oid 6 version 520 is already newer than 517 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: oid 6 version 520 is already newer than 518 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 520 (ObjNum 534 snap 158 seq_num 534) dirty exists 2024-09-17T15:37:34.596 INFO:tasks.rados.rados.0.smithi203.stdout:1454: left oid 6 (ObjNum 534 snap 158 seq_num 534) 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: finishing write tid 1 to smithi20328086-25 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: finishing write tid 2 to smithi20328086-25 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: finishing write tid 3 to smithi20328086-25 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: finishing write tid 4 to smithi20328086-25 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: oid 25 updating version 0 to 501 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: oid 25 updating version 501 to 502 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:1455: oid 25 updating version 502 to 503 2024-09-17T15:37:34.597 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 503 (ObjNum 535 snap 158 seq_num 535) dirty exists 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1455: left oid 25 (ObjNum 535 snap 158 seq_num 535) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1456: finishing write tid 1 to smithi20328086-37 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1451: done (7 left) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1452: done (6 left) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1453: done (5 left) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1454: done (4 left) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1455: done (3 left) 2024-09-17T15:37:34.598 INFO:tasks.rados.rados.0.smithi203.stdout:1459: snap_create 2024-09-17T15:37:34.605 INFO:tasks.rados.rados.0.smithi203.stdout:1456: finishing write tid 2 to smithi20328086-37 2024-09-17T15:37:34.606 INFO:tasks.rados.rados.0.smithi203.stdout:1456: finishing write tid 3 to smithi20328086-37 2024-09-17T15:37:34.606 INFO:tasks.rados.rados.0.smithi203.stdout:1456: finishing write tid 4 to smithi20328086-37 2024-09-17T15:37:34.606 INFO:tasks.rados.rados.0.smithi203.stdout:1456: oid 37 updating version 0 to 566 2024-09-17T15:37:34.606 INFO:tasks.rados.rados.0.smithi203.stdout:1456: oid 37 updating version 566 to 568 2024-09-17T15:37:34.607 INFO:tasks.rados.rados.0.smithi203.stdout:1456: oid 37 version 568 is already newer than 567 2024-09-17T15:37:34.607 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 568 (ObjNum 536 snap 158 seq_num 536) dirty exists 2024-09-17T15:37:34.607 INFO:tasks.rados.rados.0.smithi203.stdout:1456: left oid 37 (ObjNum 536 snap 158 seq_num 536) 2024-09-17T15:37:34.613 INFO:tasks.rados.rados.0.smithi203.stdout:1457: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:37:34.621 INFO:tasks.rados.rados.0.smithi203.stdout:1457: finishing copy_from to smithi20328086-41 2024-09-17T15:37:34.622 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 562 (ObjNum 527 snap 157 seq_num 527) dirty exists 2024-09-17T15:37:35.015 INFO:tasks.rados.rados.0.smithi203.stdout:1458: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:37:35.016 INFO:tasks.rados.rados.0.smithi203.stdout:1458: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:37:35.016 INFO:tasks.rados.rados.0.smithi203.stdout:1458: finishing rollback tid 2 to smithi20328086-13 2024-09-17T15:37:35.016 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 510 (ObjNum 236 snap 58 seq_num 236) dirty exists 2024-09-17T15:37:35.535 INFO:tasks.rados.rados.0.smithi203.stdout:1456: done (3 left) 2024-09-17T15:37:35.535 INFO:tasks.rados.rados.0.smithi203.stdout:1457: done (2 left) 2024-09-17T15:37:35.535 INFO:tasks.rados.rados.0.smithi203.stdout:1458: done (1 left) 2024-09-17T15:37:35.535 INFO:tasks.rados.rados.0.smithi203.stdout:1459: done (0 left) 2024-09-17T15:37:35.535 INFO:tasks.rados.rados.0.smithi203.stdout:1460: write (excl) oid 30 current snap is 159 2024-09-17T15:37:35.536 INFO:tasks.rados.rados.0.smithi203.stdout:1460: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-17T15:37:35.543 INFO:tasks.rados.rados.0.smithi203.stdout:1460: writing smithi20328086-30 from 625359 to 1178782 tid 1 2024-09-17T15:37:35.552 INFO:tasks.rados.rados.0.smithi203.stdout:1460: writing smithi20328086-30 from 1790948 to 2358132 tid 2 2024-09-17T15:37:35.556 INFO:tasks.rados.rados.0.smithi203.stdout:1460: writing smithi20328086-30 from 2930500 to 3025359 tid 3 2024-09-17T15:37:35.556 INFO:tasks.rados.rados.0.smithi203.stdout:1461: snap_remove snap 158 2024-09-17T15:37:36.553 INFO:tasks.rados.rados.0.smithi203.stdout:1461: done (1 left) 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1462: rollback oid 43 current snap is 159 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 155 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: finishing write tid 1 to smithi20328086-30 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: finishing write tid 2 to smithi20328086-30 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: finishing write tid 3 to smithi20328086-30 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: finishing write tid 4 to smithi20328086-30 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: finishing write tid 5 to smithi20328086-30 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: oid 30 updating version 0 to 353 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: oid 30 updating version 353 to 355 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: oid 30 updating version 355 to 356 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: oid 30 version 356 is already newer than 354 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 356 (ObjNum 537 snap 159 seq_num 537) dirty exists 2024-09-17T15:37:36.554 INFO:tasks.rados.rados.0.smithi203.stdout:1460: left oid 30 (ObjNum 537 snap 159 seq_num 537) 2024-09-17T15:37:36.555 INFO:tasks.rados.rados.0.smithi203.stdout:1460: done (1 left) 2024-09-17T15:37:36.555 INFO:tasks.rados.rados.0.smithi203.stdout:1463: snap_create 2024-09-17T15:37:36.566 INFO:tasks.rados.rados.0.smithi203.stdout:1462: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:37:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:1462: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:37:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:1462: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:37:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 565 (ObjNum 490 snap 148 seq_num 490) dirty exists 2024-09-17T15:37:37.518 INFO:tasks.rados.rados.0.smithi203.stdout:1462: done (1 left) 2024-09-17T15:37:37.519 INFO:tasks.rados.rados.0.smithi203.stdout:1463: done (0 left) 2024-09-17T15:37:37.519 INFO:tasks.rados.rados.0.smithi203.stdout:1464: copy_from oid 43 from oid 14 current snap is 160 2024-09-17T15:37:37.519 INFO:tasks.rados.rados.0.smithi203.stdout:1465: write oid 49 current snap is 160 2024-09-17T15:37:37.519 INFO:tasks.rados.rados.0.smithi203.stdout:1465: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-17T15:37:37.533 INFO:tasks.rados.rados.0.smithi203.stdout:1465: writing smithi20328086-49 from 642166 to 1270838 tid 1 2024-09-17T15:37:37.547 INFO:tasks.rados.rados.0.smithi203.stdout:1465: writing smithi20328086-49 from 1849430 to 2560272 tid 2 2024-09-17T15:37:37.548 INFO:tasks.rados.rados.0.smithi203.stdout:1466: read oid 23 snap 159 2024-09-17T15:37:37.548 INFO:tasks.rados.rados.0.smithi203.stdout:1466: expect (ObjNum 530 snap 157 seq_num 530) 2024-09-17T15:37:37.548 INFO:tasks.rados.rados.0.smithi203.stdout:1467: write (excl) oid 16 current snap is 160 2024-09-17T15:37:37.548 INFO:tasks.rados.rados.0.smithi203.stdout:1467: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-17T15:37:37.564 INFO:tasks.rados.rados.0.smithi203.stdout:1467: writing smithi20328086-16 from 658973 to 1362894 tid 1 2024-09-17T15:37:37.575 INFO:tasks.rados.rados.0.smithi203.stdout:1467: writing smithi20328086-16 from 1907912 to 2478765 tid 2 2024-09-17T15:37:37.580 INFO:tasks.rados.rados.0.smithi203.stdout:1467: writing smithi20328086-16 from 2985346 to 3058973 tid 3 2024-09-17T15:37:37.580 INFO:tasks.rados.rados.0.smithi203.stdout:1468: write (excl) oid 31 current snap is 160 2024-09-17T15:37:37.580 INFO:tasks.rados.rados.0.smithi203.stdout:1468: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-17T15:37:37.590 INFO:tasks.rados.rados.0.smithi203.stdout:1468: writing smithi20328086-31 from 675780 to 1171303 tid 1 2024-09-17T15:37:37.598 INFO:tasks.rados.rados.0.smithi203.stdout:1468: writing smithi20328086-31 from 1682747 to 2397258 tid 2 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1468: writing smithi20328086-31 from 3012769 to 3075780 tid 3 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1469: copy_from oid 26 from oid 33 current snap is 160 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1464: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1464: finishing copy_from to smithi20328086-43 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 293 (ObjNum 278 snap 72 seq_num 278) dirty exists 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: finishing write tid 1 to smithi20328086-49 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: finishing write tid 2 to smithi20328086-49 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: finishing write tid 3 to smithi20328086-49 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: finishing write tid 4 to smithi20328086-49 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: oid 49 updating version 0 to 630 2024-09-17T15:37:37.604 INFO:tasks.rados.rados.0.smithi203.stdout:1465: oid 49 updating version 630 to 632 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1465: oid 49 version 632 is already newer than 631 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 632 (ObjNum 538 snap 160 seq_num 538) dirty exists 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1465: left oid 49 (ObjNum 538 snap 160 seq_num 538) 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1464: done (5 left) 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1465: done (4 left) 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1470: read oid 1 snap -1 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1470: expect (ObjNum 420 snap 123 seq_num 420) 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1471: snap_create 2024-09-17T15:37:37.605 INFO:tasks.rados.rados.0.smithi203.stdout:1466: expect (ObjNum 530 snap 157 seq_num 530) 2024-09-17T15:37:37.632 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1797963 2024-09-17T15:37:37.685 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:37.735 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:37.779 INFO:tasks.rados.rados.0.smithi203.stdout:1467: finishing write tid 1 to smithi20328086-16 2024-09-17T15:37:37.779 INFO:tasks.rados.rados.0.smithi203.stdout:1467: finishing write tid 2 to smithi20328086-16 2024-09-17T15:37:37.779 INFO:tasks.rados.rados.0.smithi203.stdout:1467: finishing write tid 3 to smithi20328086-16 2024-09-17T15:37:37.779 INFO:tasks.rados.rados.0.smithi203.stdout:1467: finishing write tid 4 to smithi20328086-16 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: finishing write tid 5 to smithi20328086-16 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: oid 16 updating version 0 to 571 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: oid 16 updating version 571 to 573 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: oid 16 version 573 is already newer than 572 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: oid 16 version 573 is already newer than 570 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 573 (ObjNum 539 snap 160 seq_num 539) dirty exists 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1467: left oid 16 (ObjNum 539 snap 160 seq_num 539) 2024-09-17T15:37:37.780 INFO:tasks.rados.rados.0.smithi203.stdout:1470: expect (ObjNum 420 snap 123 seq_num 420) 2024-09-17T15:37:37.969 INFO:tasks.rados.rados.0.smithi203.stdout:1468: finishing write tid 1 to smithi20328086-31 2024-09-17T15:37:37.969 INFO:tasks.rados.rados.0.smithi203.stdout:1468: finishing write tid 2 to smithi20328086-31 2024-09-17T15:37:37.969 INFO:tasks.rados.rados.0.smithi203.stdout:1468: finishing write tid 3 to smithi20328086-31 2024-09-17T15:37:37.969 INFO:tasks.rados.rados.0.smithi203.stdout:1468: finishing write tid 4 to smithi20328086-31 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: finishing write tid 5 to smithi20328086-31 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: oid 31 updating version 0 to 188 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: oid 31 version 188 is already newer than 187 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: oid 31 updating version 188 to 189 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: oid 31 updating version 189 to 190 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 190 (ObjNum 540 snap 160 seq_num 540) dirty exists 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1468: left oid 31 (ObjNum 540 snap 160 seq_num 540) 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1469: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:1469: finishing copy_from to smithi20328086-26 2024-09-17T15:37:37.970 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 310 (ObjNum 522 snap 157 seq_num 522) dirty exists 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1466: done (5 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1467: done (4 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1468: done (3 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1469: done (2 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1470: done (1 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1471: done (0 left) 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1472: copy_from oid 22 from oid 47 current snap is 161 2024-09-17T15:37:39.140 INFO:tasks.rados.rados.0.smithi203.stdout:1473: delete oid 24 current snap is 161 2024-09-17T15:37:39.152 INFO:tasks.rados.rados.0.smithi203.stdout:1473: done (1 left) 2024-09-17T15:37:39.152 INFO:tasks.rados.rados.0.smithi203.stdout:1474: write oid 2 current snap is 161 2024-09-17T15:37:39.152 INFO:tasks.rados.rados.0.smithi203.stdout:1474: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-17T15:37:39.165 INFO:tasks.rados.rados.0.smithi203.stdout:1474: writing smithi20328086-2 from 709394 to 1355415 tid 1 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1474: writing smithi20328086-2 from 2083358 to 2801538 tid 2 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1475: read oid 15 snap 157 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1475: expect deleted 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1476: copy_from oid 6 from oid 10 current snap is 161 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1472: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1472: finishing copy_from to smithi20328086-22 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 500 (ObjNum 526 snap 157 seq_num 526) dirty exists 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1474: finishing write tid 1 to smithi20328086-2 2024-09-17T15:37:39.181 INFO:tasks.rados.rados.0.smithi203.stdout:1472: done (3 left) 2024-09-17T15:37:39.182 INFO:tasks.rados.rados.0.smithi203.stdout:1477: read oid 25 snap -1 2024-09-17T15:37:39.182 INFO:tasks.rados.rados.0.smithi203.stdout:1477: expect (ObjNum 535 snap 158 seq_num 535) 2024-09-17T15:37:39.182 INFO:tasks.rados.rados.0.smithi203.stdout:1478: read oid 33 snap -1 2024-09-17T15:37:39.182 INFO:tasks.rados.rados.0.smithi203.stdout:1478: expect (ObjNum 522 snap 157 seq_num 522) 2024-09-17T15:37:39.182 INFO:tasks.rados.rados.0.smithi203.stdout:1479: delete oid 44 current snap is 161 2024-09-17T15:37:39.185 INFO:tasks.rados.rados.0.smithi203.stdout:1477: expect (ObjNum 535 snap 158 seq_num 535) 2024-09-17T15:37:39.340 INFO:tasks.rados.rados.0.smithi203.stdout:1474: finishing write tid 2 to smithi20328086-2 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: finishing write tid 3 to smithi20328086-2 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: finishing write tid 4 to smithi20328086-2 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: oid 2 updating version 0 to 457 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: oid 2 version 457 is already newer than 456 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: oid 2 updating version 457 to 458 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 458 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1474: left oid 2 (ObjNum 542 snap 161 seq_num 542) 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:1478: expect (ObjNum 522 snap 157 seq_num 522) 2024-09-17T15:37:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:39.436 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1783255 2024-09-17T15:37:39.495 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1900650 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1476: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1476: finishing copy_from to smithi20328086-6 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 503 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1474: done (5 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1475: done (4 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1476: done (3 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1477: done (2 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1478: done (1 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1479: done (0 left) 2024-09-17T15:37:39.524 INFO:tasks.rados.rados.0.smithi203.stdout:1480: rollback oid 44 current snap is 161 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 159 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1481: rollback oid 45 current snap is 161 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 159 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1482: read oid 15 snap -1 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1482: expect deleted 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1483: read oid 6 snap 160 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1483: expect (ObjNum 534 snap 158 seq_num 534) 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1484: copy_from oid 48 from oid 46 current snap is 161 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1485: write (excl) oid 12 current snap is 161 2024-09-17T15:37:39.525 INFO:tasks.rados.rados.0.smithi203.stdout:1485: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-17T15:37:39.532 INFO:tasks.rados.rados.0.smithi203.stdout:1485: writing smithi20328086-12 from 743008 to 1539527 tid 1 2024-09-17T15:37:39.540 INFO:tasks.rados.rados.0.smithi203.stdout:1485: writing smithi20328086-12 from 2200322 to 2922171 tid 2 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1486: delete oid 40 current snap is 161 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1484: finishing copy_from to smithi20328086-48 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1484: got expected ENOENT (src dne) 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1484: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1480: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1480: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 600 (ObjNum 524 snap 157 seq_num 524) dirty exists 2024-09-17T15:37:39.541 INFO:tasks.rados.rados.0.smithi203.stdout:1483: expect (ObjNum 534 snap 158 seq_num 534) 2024-09-17T15:37:39.559 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1913436 2024-09-17T15:37:39.577 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1481: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1481: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1481: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 317 (ObjNum 462 snap 140 seq_num 462) dirty exists 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1480: done (6 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1481: done (5 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1482: done (4 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1483: done (3 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1484: done (2 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1486: done (1 left) 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1487: delete oid 35 current snap is 161 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1485: finishing write tid 1 to smithi20328086-12 2024-09-17T15:37:39.667 INFO:tasks.rados.rados.0.smithi203.stdout:1485: finishing write tid 2 to smithi20328086-12 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: finishing write tid 3 to smithi20328086-12 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: finishing write tid 4 to smithi20328086-12 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: oid 12 updating version 0 to 362 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: oid 12 version 362 is already newer than 360 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: oid 12 version 362 is already newer than 361 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 362 (ObjNum 544 snap 161 seq_num 544) dirty exists 2024-09-17T15:37:39.668 INFO:tasks.rados.rados.0.smithi203.stdout:1485: left oid 12 (ObjNum 544 snap 161 seq_num 544) 2024-09-17T15:37:39.671 INFO:tasks.rados.rados.0.smithi203.stdout:1485: done (1 left) 2024-09-17T15:37:39.671 INFO:tasks.rados.rados.0.smithi203.stdout:1487: done (0 left) 2024-09-17T15:37:39.671 INFO:tasks.rados.rados.0.smithi203.stdout:1488: rollback oid 26 current snap is 161 2024-09-17T15:37:39.671 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 155 2024-09-17T15:37:39.671 INFO:tasks.rados.rados.0.smithi203.stdout:1489: delete oid 16 current snap is 161 2024-09-17T15:37:39.679 INFO:tasks.rados.rados.0.smithi203.stdout:1489: done (1 left) 2024-09-17T15:37:39.679 INFO:tasks.rados.rados.0.smithi203.stdout:1490: write oid 39 current snap is 161 2024-09-17T15:37:39.679 INFO:tasks.rados.rados.0.smithi203.stdout:1490: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-17T15:37:39.692 INFO:tasks.rados.rados.0.smithi203.stdout:1490: writing smithi20328086-39 from 410236 to 824104 tid 1 2024-09-17T15:37:39.711 INFO:tasks.rados.rados.0.smithi203.stdout:1490: writing smithi20328086-39 from 1234250 to 1963437 tid 2 2024-09-17T15:37:39.729 INFO:tasks.rados.rados.0.smithi203.stdout:1490: writing smithi20328086-39 from 2715800 to 3138863 tid 3 2024-09-17T15:37:39.729 INFO:tasks.rados.rados.0.smithi203.stdout:1491: delete oid 38 current snap is 161 2024-09-17T15:37:39.729 INFO:tasks.rados.rados.0.smithi203.stdout:1488: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:37:39.729 INFO:tasks.rados.rados.0.smithi203.stdout:1488: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:37:39.729 INFO:tasks.rados.rados.0.smithi203.stdout:1488: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:37:39.730 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 638 (ObjNum 370 snap 105 seq_num 370) dirty exists 2024-09-17T15:37:39.730 INFO:tasks.rados.rados.0.smithi203.stdout:1490: finishing write tid 1 to smithi20328086-39 2024-09-17T15:37:39.730 INFO:tasks.rados.rados.0.smithi203.stdout:1490: finishing write tid 2 to smithi20328086-39 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: finishing write tid 3 to smithi20328086-39 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: finishing write tid 4 to smithi20328086-39 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: finishing write tid 5 to smithi20328086-39 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: oid 39 updating version 0 to 578 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: oid 39 version 578 is already newer than 577 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: oid 39 updating version 578 to 579 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: oid 39 version 579 is already newer than 576 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 579 (ObjNum 548 snap 161 seq_num 548) dirty exists 2024-09-17T15:37:39.734 INFO:tasks.rados.rados.0.smithi203.stdout:1490: left oid 39 (ObjNum 548 snap 161 seq_num 548) 2024-09-17T15:37:39.736 INFO:tasks.rados.rados.0.smithi203.stdout:1488: done (2 left) 2024-09-17T15:37:39.736 INFO:tasks.rados.rados.0.smithi203.stdout:1490: done (1 left) 2024-09-17T15:37:39.736 INFO:tasks.rados.rados.0.smithi203.stdout:1491: done (0 left) 2024-09-17T15:37:39.737 INFO:tasks.rados.rados.0.smithi203.stdout:1492: copy_from oid 22 from oid 48 current snap is 161 2024-09-17T15:37:39.737 INFO:tasks.rados.rados.0.smithi203.stdout:1493: write (excl) oid 16 current snap is 161 2024-09-17T15:37:39.737 INFO:tasks.rados.rados.0.smithi203.stdout:1493: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-09-17T15:37:39.750 INFO:tasks.rados.rados.0.smithi203.stdout:1493: writing smithi20328086-16 from 443850 to 1008216 tid 1 2024-09-17T15:37:39.767 INFO:tasks.rados.rados.0.smithi203.stdout:1493: writing smithi20328086-16 from 1634861 to 2367717 tid 2 2024-09-17T15:37:39.778 INFO:tasks.rados.rados.0.smithi203.stdout:1493: writing smithi20328086-16 from 2770646 to 3243850 tid 3 2024-09-17T15:37:39.778 INFO:tasks.rados.rados.0.smithi203.stdout:1494: snap_remove snap 160 2024-09-17T15:37:39.980 INFO:tasks.rados.rados.0.smithi203.stdout:1494: done (2 left) 2024-09-17T15:37:39.980 INFO:tasks.rados.rados.0.smithi203.stdout:1495: read oid 10 snap -1 2024-09-17T15:37:39.980 INFO:tasks.rados.rados.0.smithi203.stdout:1495: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1496: read oid 29 snap 157 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1496: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1497: read oid 43 snap -1 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1497: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1498: write (excl) oid 11 current snap is 161 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1492: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1493: finishing write tid 1 to smithi20328086-16 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1492: finishing copy_from to smithi20328086-22 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 307 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1493: finishing write tid 2 to smithi20328086-16 2024-09-17T15:37:39.981 INFO:tasks.rados.rados.0.smithi203.stdout:1493: finishing write tid 3 to smithi20328086-16 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: finishing write tid 4 to smithi20328086-16 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: finishing write tid 5 to smithi20328086-16 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: oid 16 updating version 0 to 582 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: oid 16 version 582 is already newer than 580 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: oid 16 version 582 is already newer than 581 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: oid 16 updating version 582 to 583 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 583 (ObjNum 550 snap 161 seq_num 550) dirty exists 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1493: left oid 16 (ObjNum 550 snap 161 seq_num 550) 2024-09-17T15:37:39.982 INFO:tasks.rados.rados.0.smithi203.stdout:1498: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-17T15:37:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:1498: writing smithi20328086-11 from 460657 to 1100272 tid 1 2024-09-17T15:37:40.008 INFO:tasks.rados.rados.0.smithi203.stdout:1498: writing smithi20328086-11 from 1693343 to 2169857 tid 2 2024-09-17T15:37:40.024 INFO:tasks.rados.rados.0.smithi203.stdout:1498: writing smithi20328086-11 from 2681716 to 3260657 tid 3 2024-09-17T15:37:40.024 INFO:tasks.rados.rados.0.smithi203.stdout:1492: done (5 left) 2024-09-17T15:37:40.024 INFO:tasks.rados.rados.0.smithi203.stdout:1493: done (4 left) 2024-09-17T15:37:40.024 INFO:tasks.rados.rados.0.smithi203.stdout:1499: delete oid 30 current snap is 161 2024-09-17T15:37:40.024 INFO:tasks.rados.rados.0.smithi203.stdout:1495: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:40.053 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:40.099 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1651080 2024-09-17T15:37:40.109 INFO:tasks.rados.rados.0.smithi203.stdout:1497: expect (ObjNum 278 snap 72 seq_num 278) 2024-09-17T15:37:40.306 INFO:tasks.rados.rados.0.smithi203.stdout:1496: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:40.306 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:40.365 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:40.459 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3753817 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: finishing write tid 1 to smithi20328086-11 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: finishing write tid 2 to smithi20328086-11 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: finishing write tid 3 to smithi20328086-11 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: finishing write tid 4 to smithi20328086-11 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: finishing write tid 5 to smithi20328086-11 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: oid 11 updating version 0 to 604 2024-09-17T15:37:40.465 INFO:tasks.rados.rados.0.smithi203.stdout:1498: oid 11 updating version 604 to 609 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1498: oid 11 updating version 609 to 610 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1498: oid 11 updating version 610 to 611 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1498: left oid 11 (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1495: done (4 left) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1496: done (3 left) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1497: done (2 left) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1498: done (1 left) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1499: done (0 left) 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1500: write (excl) oid 39 current snap is 161 2024-09-17T15:37:40.466 INFO:tasks.rados.rados.0.smithi203.stdout:1500: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-17T15:37:40.485 INFO:tasks.rados.rados.0.smithi203.stdout:1500: writing smithi20328086-39 from 494271 to 1284384 tid 1 2024-09-17T15:37:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:1500: writing smithi20328086-39 from 1693954 to 2174137 tid 2 2024-09-17T15:37:40.516 INFO:tasks.rados.rados.0.smithi203.stdout:1500: writing smithi20328086-39 from 2620209 to 3132338 tid 3 2024-09-17T15:37:40.516 INFO:tasks.rados.rados.0.smithi203.stdout:1501: snap_create 2024-09-17T15:37:40.516 INFO:tasks.rados.rados.0.smithi203.stdout:1500: finishing write tid 1 to smithi20328086-39 2024-09-17T15:37:40.516 INFO:tasks.rados.rados.0.smithi203.stdout:1500: finishing write tid 2 to smithi20328086-39 2024-09-17T15:37:40.523 INFO:tasks.rados.rados.0.smithi203.stdout:1500: finishing write tid 3 to smithi20328086-39 2024-09-17T15:37:40.524 INFO:tasks.rados.rados.0.smithi203.stdout:1500: finishing write tid 4 to smithi20328086-39 2024-09-17T15:37:40.524 INFO:tasks.rados.rados.0.smithi203.stdout:1500: finishing write tid 5 to smithi20328086-39 2024-09-17T15:37:40.524 INFO:tasks.rados.rados.0.smithi203.stdout:1500: oid 39 updating version 0 to 589 2024-09-17T15:37:40.524 INFO:tasks.rados.rados.0.smithi203.stdout:1500: oid 39 version 589 is already newer than 586 2024-09-17T15:37:40.524 INFO:tasks.rados.rados.0.smithi203.stdout:1500: oid 39 version 589 is already newer than 588 2024-09-17T15:37:40.525 INFO:tasks.rados.rados.0.smithi203.stdout:1500: oid 39 version 589 is already newer than 587 2024-09-17T15:37:40.525 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 589 (ObjNum 553 snap 161 seq_num 553) dirty exists 2024-09-17T15:37:40.525 INFO:tasks.rados.rados.0.smithi203.stdout:1500: left oid 39 (ObjNum 553 snap 161 seq_num 553) 2024-09-17T15:37:40.995 INFO:tasks.rados.rados.0.smithi203.stdout:1500: done (1 left) 2024-09-17T15:37:40.995 INFO:tasks.rados.rados.0.smithi203.stdout:1501: done (0 left) 2024-09-17T15:37:40.995 INFO:tasks.rados.rados.0.smithi203.stdout:1502: read oid 46 snap -1 2024-09-17T15:37:40.995 INFO:tasks.rados.rados.0.smithi203.stdout:1502: expect deleted 2024-09-17T15:37:40.995 INFO:tasks.rados.rados.0.smithi203.stdout:1503: rollback oid 32 current snap is 162 2024-09-17T15:37:40.996 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 161 2024-09-17T15:37:40.996 INFO:tasks.rados.rados.0.smithi203.stdout:1504: rollback oid 26 current snap is 162 2024-09-17T15:37:40.996 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 157 2024-09-17T15:37:40.996 INFO:tasks.rados.rados.0.smithi203.stdout:1505: snap_remove snap 159 2024-09-17T15:37:42.006 INFO:tasks.rados.rados.0.smithi203.stdout:1505: done (3 left) 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1506: rollback oid 41 current snap is 162 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 157 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1504: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1504: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1504: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 644 (ObjNum 520 snap 157 seq_num 520) dirty exists 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1503: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1503: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1503: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 310 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:37:42.007 INFO:tasks.rados.rados.0.smithi203.stdout:1502: done (3 left) 2024-09-17T15:37:42.008 INFO:tasks.rados.rados.0.smithi203.stdout:1503: done (2 left) 2024-09-17T15:37:42.008 INFO:tasks.rados.rados.0.smithi203.stdout:1504: done (1 left) 2024-09-17T15:37:42.008 INFO:tasks.rados.rados.0.smithi203.stdout:1507: write (excl) oid 34 current snap is 162 2024-09-17T15:37:42.008 INFO:tasks.rados.rados.0.smithi203.stdout:1507: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-17T15:37:42.020 INFO:tasks.rados.rados.0.smithi203.stdout:1507: writing smithi20328086-34 from 511078 to 976440 tid 1 2024-09-17T15:37:42.041 INFO:tasks.rados.rados.0.smithi203.stdout:1507: writing smithi20328086-34 from 1752436 to 2492630 tid 2 2024-09-17T15:37:42.054 INFO:tasks.rados.rados.0.smithi203.stdout:1507: writing smithi20328086-34 from 3163985 to 3311078 tid 3 2024-09-17T15:37:42.054 INFO:tasks.rados.rados.0.smithi203.stdout:1508: write oid 15 current snap is 162 2024-09-17T15:37:42.054 INFO:tasks.rados.rados.0.smithi203.stdout:1508: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-17T15:37:42.065 INFO:tasks.rados.rados.0.smithi203.stdout:1508: writing smithi20328086-15 from 527885 to 1184849 tid 1 2024-09-17T15:37:42.077 INFO:tasks.rados.rados.0.smithi203.stdout:1508: writing smithi20328086-15 from 1927271 to 2411123 tid 2 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1508: writing smithi20328086-15 from 3191408 to 3327885 tid 3 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1509: write (excl) oid 1 current snap is 162 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1506: finishing rollback tid 0 to smithi20328086-41 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1506: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 567 (ObjNum 145 snap 0 seq_num 4108901728) dirty dne 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1507: finishing write tid 1 to smithi20328086-34 2024-09-17T15:37:42.085 INFO:tasks.rados.rados.0.smithi203.stdout:1509: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-17T15:37:42.095 INFO:tasks.rados.rados.0.smithi203.stdout:1509: writing smithi20328086-1 from 544692 to 1276905 tid 1 2024-09-17T15:37:42.106 INFO:tasks.rados.rados.0.smithi203.stdout:1509: writing smithi20328086-1 from 1985753 to 2729616 tid 2 2024-09-17T15:37:42.111 INFO:tasks.rados.rados.0.smithi203.stdout:1509: writing smithi20328086-1 from 3335184 to 3344692 tid 3 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1506: done (3 left) 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1510: read oid 8 snap -1 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1510: expect deleted 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1511: snap_create 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: finishing write tid 2 to smithi20328086-34 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: finishing write tid 3 to smithi20328086-34 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: finishing write tid 4 to smithi20328086-34 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: finishing write tid 5 to smithi20328086-34 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: oid 34 updating version 0 to 494 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: oid 34 version 494 is already newer than 493 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: oid 34 version 494 is already newer than 492 2024-09-17T15:37:42.112 INFO:tasks.rados.rados.0.smithi203.stdout:1507: oid 34 updating version 494 to 495 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 495 (ObjNum 554 snap 162 seq_num 554) dirty exists 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1507: left oid 34 (ObjNum 554 snap 162 seq_num 554) 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: finishing write tid 1 to smithi20328086-15 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: finishing write tid 2 to smithi20328086-15 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: finishing write tid 3 to smithi20328086-15 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: finishing write tid 4 to smithi20328086-15 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: finishing write tid 5 to smithi20328086-15 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: oid 15 updating version 0 to 374 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: oid 15 version 374 is already newer than 373 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: oid 15 version 374 is already newer than 371 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: oid 15 version 374 is already newer than 372 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 374 (ObjNum 555 snap 162 seq_num 555) dirty exists 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1508: left oid 15 (ObjNum 555 snap 162 seq_num 555) 2024-09-17T15:37:42.113 INFO:tasks.rados.rados.0.smithi203.stdout:1509: finishing write tid 1 to smithi20328086-1 2024-09-17T15:37:42.117 INFO:tasks.rados.rados.0.smithi203.stdout:1509: finishing write tid 2 to smithi20328086-1 2024-09-17T15:37:42.119 INFO:tasks.rados.rados.0.smithi203.stdout:1509: finishing write tid 3 to smithi20328086-1 2024-09-17T15:37:42.120 INFO:tasks.rados.rados.0.smithi203.stdout:1509: finishing write tid 4 to smithi20328086-1 2024-09-17T15:37:42.120 INFO:tasks.rados.rados.0.smithi203.stdout:1509: finishing write tid 5 to smithi20328086-1 2024-09-17T15:37:42.120 INFO:tasks.rados.rados.0.smithi203.stdout:1509: oid 1 updating version 0 to 530 2024-09-17T15:37:42.121 INFO:tasks.rados.rados.0.smithi203.stdout:1509: oid 1 version 530 is already newer than 529 2024-09-17T15:37:42.121 INFO:tasks.rados.rados.0.smithi203.stdout:1509: oid 1 version 530 is already newer than 528 2024-09-17T15:37:42.121 INFO:tasks.rados.rados.0.smithi203.stdout:1509: oid 1 updating version 530 to 531 2024-09-17T15:37:42.121 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 531 (ObjNum 556 snap 162 seq_num 556) dirty exists 2024-09-17T15:37:42.121 INFO:tasks.rados.rados.0.smithi203.stdout:1509: left oid 1 (ObjNum 556 snap 162 seq_num 556) 2024-09-17T15:37:42.522 INFO:tasks.rados.rados.0.smithi203.stdout:1507: done (4 left) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1508: done (3 left) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1509: done (2 left) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1510: done (1 left) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1511: done (0 left) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1512: copy_from oid 10 from oid 16 current snap is 163 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1513: read oid 3 snap -1 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1513: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1514: read oid 5 snap -1 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1514: expect (ObjNum 503 snap 149 seq_num 503) 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1515: read oid 47 snap 162 2024-09-17T15:37:42.523 INFO:tasks.rados.rados.0.smithi203.stdout:1515: expect (ObjNum 526 snap 157 seq_num 526) 2024-09-17T15:37:42.524 INFO:tasks.rados.rados.0.smithi203.stdout:1516: snap_remove snap 155 2024-09-17T15:37:43.531 INFO:tasks.rados.rados.0.smithi203.stdout:1516: done (4 left) 2024-09-17T15:37:43.531 INFO:tasks.rados.rados.0.smithi203.stdout:1517: snap_create 2024-09-17T15:37:43.531 INFO:tasks.rados.rados.0.smithi203.stdout:1513: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:43.625 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2321633 2024-09-17T15:37:43.654 INFO:tasks.rados.rados.0.smithi203.stdout:1515: expect (ObjNum 526 snap 157 seq_num 526) 2024-09-17T15:37:43.661 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1903513 2024-09-17T15:37:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:1514: expect (ObjNum 503 snap 149 seq_num 503) 2024-09-17T15:37:43.802 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1477057 2024-09-17T15:37:43.891 INFO:tasks.rados.rados.0.smithi203.stdout:1512: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:37:43.891 INFO:tasks.rados.rados.0.smithi203.stdout:1512: finishing copy_from to smithi20328086-10 2024-09-17T15:37:43.891 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 583 (ObjNum 550 snap 161 seq_num 550) dirty exists 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1512: done (4 left) 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1513: done (3 left) 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1514: done (2 left) 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1515: done (1 left) 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1517: done (0 left) 2024-09-17T15:37:44.538 INFO:tasks.rados.rados.0.smithi203.stdout:1518: snap_create 2024-09-17T15:37:45.545 INFO:tasks.rados.rados.0.smithi203.stdout:1518: done (0 left) 2024-09-17T15:37:45.545 INFO:tasks.rados.rados.0.smithi203.stdout:1519: read oid 6 snap -1 2024-09-17T15:37:45.545 INFO:tasks.rados.rados.0.smithi203.stdout:1519: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:45.546 INFO:tasks.rados.rados.0.smithi203.stdout:1520: snap_create 2024-09-17T15:37:45.557 INFO:tasks.rados.rados.0.smithi203.stdout:1519: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:46.556 INFO:tasks.rados.rados.0.smithi203.stdout:1519: done (1 left) 2024-09-17T15:37:46.557 INFO:tasks.rados.rados.0.smithi203.stdout:1520: done (0 left) 2024-09-17T15:37:46.557 INFO:tasks.rados.rados.0.smithi203.stdout:1521: copy_from oid 43 from oid 15 current snap is 166 2024-09-17T15:37:46.557 INFO:tasks.rados.rados.0.smithi203.stdout:1522: snap_remove snap 157 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1522: done (1 left) 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1523: copy_from oid 45 from oid 12 current snap is 166 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1521: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1521: finishing copy_from to smithi20328086-43 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 374 (ObjNum 555 snap 162 seq_num 555) dirty exists 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1521: done (1 left) 2024-09-17T15:37:47.521 INFO:tasks.rados.rados.0.smithi203.stdout:1524: write oid 17 current snap is 166 2024-09-17T15:37:47.522 INFO:tasks.rados.rados.0.smithi203.stdout:1524: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-17T15:37:47.545 INFO:tasks.rados.rados.0.smithi203.stdout:1524: writing smithi20328086-17 from 561499 to 968961 tid 1 2024-09-17T15:37:47.566 INFO:tasks.rados.rados.0.smithi203.stdout:1524: writing smithi20328086-17 from 1527882 to 2015403 tid 2 2024-09-17T15:37:47.585 INFO:tasks.rados.rados.0.smithi203.stdout:1524: writing smithi20328086-17 from 2729901 to 3287118 tid 3 2024-09-17T15:37:47.585 INFO:tasks.rados.rados.0.smithi203.stdout:1525: read oid 37 snap 162 2024-09-17T15:37:47.585 INFO:tasks.rados.rados.0.smithi203.stdout:1525: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-17T15:37:47.585 INFO:tasks.rados.rados.0.smithi203.stdout:1526: write oid 10 current snap is 166 2024-09-17T15:37:47.585 INFO:tasks.rados.rados.0.smithi203.stdout:1526: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-09-17T15:37:47.597 INFO:tasks.rados.rados.0.smithi203.stdout:1526: writing smithi20328086-10 from 578306 to 1061017 tid 1 2024-09-17T15:37:47.608 INFO:tasks.rados.rados.0.smithi203.stdout:1526: writing smithi20328086-10 from 1586364 to 2217543 tid 2 2024-09-17T15:37:47.617 INFO:tasks.rados.rados.0.smithi203.stdout:1526: writing smithi20328086-10 from 2757324 to 3242166 tid 3 2024-09-17T15:37:47.617 INFO:tasks.rados.rados.0.smithi203.stdout:1527: write (excl) oid 30 current snap is 166 2024-09-17T15:37:47.617 INFO:tasks.rados.rados.0.smithi203.stdout:1523: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:37:47.617 INFO:tasks.rados.rados.0.smithi203.stdout:1527: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-17T15:37:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:1527: writing smithi20328086-30 from 595113 to 1153073 tid 1 2024-09-17T15:37:47.632 INFO:tasks.rados.rados.0.smithi203.stdout:1527: writing smithi20328086-30 from 1644846 to 2136036 tid 2 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:1527: writing smithi20328086-30 from 2784747 to 3395113 tid 3 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:1528: snap_create 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:1523: finishing copy_from to smithi20328086-45 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 362 (ObjNum 544 snap 161 seq_num 544) dirty exists 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:1524: finishing write tid 1 to smithi20328086-17 2024-09-17T15:37:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:1524: finishing write tid 2 to smithi20328086-17 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: finishing write tid 3 to smithi20328086-17 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: finishing write tid 4 to smithi20328086-17 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: finishing write tid 5 to smithi20328086-17 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: oid 17 updating version 0 to 634 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: oid 17 updating version 634 to 636 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: oid 17 version 636 is already newer than 635 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: oid 17 version 636 is already newer than 633 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 636 (ObjNum 557 snap 166 seq_num 557) dirty exists 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1524: left oid 17 (ObjNum 557 snap 166 seq_num 557) 2024-09-17T15:37:47.643 INFO:tasks.rados.rados.0.smithi203.stdout:1525: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-17T15:37:47.744 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 717608 2024-09-17T15:37:47.784 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 599305 2024-09-17T15:37:47.827 INFO:tasks.rados.rados.0.smithi203.stdout:1526: finishing write tid 1 to smithi20328086-10 2024-09-17T15:37:47.827 INFO:tasks.rados.rados.0.smithi203.stdout:1526: finishing write tid 2 to smithi20328086-10 2024-09-17T15:37:47.827 INFO:tasks.rados.rados.0.smithi203.stdout:1526: finishing write tid 3 to smithi20328086-10 2024-09-17T15:37:47.827 INFO:tasks.rados.rados.0.smithi203.stdout:1526: finishing write tid 4 to smithi20328086-10 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: finishing write tid 5 to smithi20328086-10 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: oid 10 updating version 0 to 586 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: oid 10 updating version 586 to 587 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: oid 10 version 587 is already newer than 585 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: oid 10 version 587 is already newer than 584 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 587 (ObjNum 558 snap 166 seq_num 558) dirty exists 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1526: left oid 10 (ObjNum 558 snap 166 seq_num 558) 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1527: finishing write tid 1 to smithi20328086-30 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1527: finishing write tid 2 to smithi20328086-30 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1527: finishing write tid 3 to smithi20328086-30 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1527: finishing write tid 4 to smithi20328086-30 2024-09-17T15:37:47.828 INFO:tasks.rados.rados.0.smithi203.stdout:1527: finishing write tid 5 to smithi20328086-30 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:1527: oid 30 updating version 0 to 385 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:1527: oid 30 version 385 is already newer than 383 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:1527: oid 30 version 385 is already newer than 384 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:1527: oid 30 updating version 385 to 386 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 386 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:37:47.829 INFO:tasks.rados.rados.0.smithi203.stdout:1527: left oid 30 (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:37:48.565 INFO:tasks.rados.rados.0.smithi203.stdout:1523: done (5 left) 2024-09-17T15:37:48.565 INFO:tasks.rados.rados.0.smithi203.stdout:1524: done (4 left) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1525: done (3 left) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1526: done (2 left) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1527: done (1 left) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1528: done (0 left) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1529: copy_from oid 35 from oid 2 current snap is 167 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1530: read oid 15 snap -1 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1530: expect (ObjNum 555 snap 162 seq_num 555) 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1531: rollback oid 8 current snap is 167 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 162 2024-09-17T15:37:48.566 INFO:tasks.rados.rados.0.smithi203.stdout:1532: delete oid 40 current snap is 167 2024-09-17T15:37:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:1531: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:37:48.570 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:37:48.572 INFO:tasks.rados.rados.0.smithi203.stdout:1531: done (3 left) 2024-09-17T15:37:48.572 INFO:tasks.rados.rados.0.smithi203.stdout:1532: done (2 left) 2024-09-17T15:37:48.572 INFO:tasks.rados.rados.0.smithi203.stdout:1533: delete oid 45 current snap is 167 2024-09-17T15:37:48.579 INFO:tasks.rados.rados.0.smithi203.stdout:1533: done (2 left) 2024-09-17T15:37:48.579 INFO:tasks.rados.rados.0.smithi203.stdout:1534: rollback oid 12 current snap is 167 2024-09-17T15:37:48.579 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 164 2024-09-17T15:37:48.579 INFO:tasks.rados.rados.0.smithi203.stdout:1535: read oid 42 snap 163 2024-09-17T15:37:48.579 INFO:tasks.rados.rados.0.smithi203.stdout:1535: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:37:48.580 INFO:tasks.rados.rados.0.smithi203.stdout:1536: write oid 39 current snap is 167 2024-09-17T15:37:48.580 INFO:tasks.rados.rados.0.smithi203.stdout:1536: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-17T15:37:48.600 INFO:tasks.rados.rados.0.smithi203.stdout:1536: writing smithi20328086-39 from 645534 to 1429241 tid 1 2024-09-17T15:37:48.614 INFO:tasks.rados.rados.0.smithi203.stdout:1536: writing smithi20328086-39 from 2103939 to 2742456 tid 2 2024-09-17T15:37:48.622 INFO:tasks.rados.rados.0.smithi203.stdout:1536: writing smithi20328086-39 from 3150663 to 3445534 tid 3 2024-09-17T15:37:48.623 INFO:tasks.rados.rados.0.smithi203.stdout:1537: write (excl) oid 31 current snap is 167 2024-09-17T15:37:48.623 INFO:tasks.rados.rados.0.smithi203.stdout:1530: expect (ObjNum 555 snap 162 seq_num 555) 2024-09-17T15:37:48.623 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:48.784 INFO:tasks.rados.rados.0.smithi203.stdout:1534: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:37:48.784 INFO:tasks.rados.rados.0.smithi203.stdout:1535: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:37:48.888 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1534: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1534: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 390 (ObjNum 544 snap 161 seq_num 544) dirty exists 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1529: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1529: finishing copy_from to smithi20328086-35 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 458 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: finishing write tid 1 to smithi20328086-39 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: finishing write tid 2 to smithi20328086-39 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: finishing write tid 3 to smithi20328086-39 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: finishing write tid 4 to smithi20328086-39 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: finishing write tid 5 to smithi20328086-39 2024-09-17T15:37:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:1536: oid 39 updating version 0 to 606 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:1536: oid 39 version 606 is already newer than 603 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:1536: oid 39 version 606 is already newer than 604 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:1536: oid 39 version 606 is already newer than 605 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 606 (ObjNum 562 snap 167 seq_num 562) dirty exists 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:1536: left oid 39 (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:37:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:1537: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-17T15:37:48.961 INFO:tasks.rados.rados.0.smithi203.stdout:1537: writing smithi20328086-31 from 662341 to 1237650 tid 1 2024-09-17T15:37:48.977 INFO:tasks.rados.rados.0.smithi203.stdout:1537: writing smithi20328086-31 from 1878774 to 2377302 tid 2 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1537: writing smithi20328086-31 from 2894439 to 3351994 tid 3 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1529: done (5 left) 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1530: done (4 left) 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1534: done (3 left) 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1535: done (2 left) 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1536: done (1 left) 2024-09-17T15:37:48.991 INFO:tasks.rados.rados.0.smithi203.stdout:1538: rollback oid 24 current snap is 167 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 166 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:1537: finishing write tid 1 to smithi20328086-31 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:1537: finishing write tid 2 to smithi20328086-31 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:1539: delete oid 28 current snap is 167 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:1538: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:37:48.992 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:37:48.998 INFO:tasks.rados.rados.0.smithi203.stdout:1537: finishing write tid 3 to smithi20328086-31 2024-09-17T15:37:48.999 INFO:tasks.rados.rados.0.smithi203.stdout:1538: done (2 left) 2024-09-17T15:37:48.999 INFO:tasks.rados.rados.0.smithi203.stdout:1539: done (1 left) 2024-09-17T15:37:48.999 INFO:tasks.rados.rados.0.smithi203.stdout:1540: read oid 49 snap -1 2024-09-17T15:37:48.999 INFO:tasks.rados.rados.0.smithi203.stdout:1540: expect (ObjNum 538 snap 160 seq_num 538) 2024-09-17T15:37:49.000 INFO:tasks.rados.rados.0.smithi203.stdout:1541: write (excl) oid 28 current snap is 167 2024-09-17T15:37:49.000 INFO:tasks.rados.rados.0.smithi203.stdout:1541: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-17T15:37:49.018 INFO:tasks.rados.rados.0.smithi203.stdout:1541: writing smithi20328086-28 from 695955 to 1421762 tid 1 2024-09-17T15:37:49.031 INFO:tasks.rados.rados.0.smithi203.stdout:1541: writing smithi20328086-28 from 1879385 to 2381582 tid 2 2024-09-17T15:37:49.042 INFO:tasks.rados.rados.0.smithi203.stdout:1541: writing smithi20328086-28 from 2949285 to 3429384 tid 3 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1542: read oid 6 snap -1 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1542: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1543: copy_from oid 38 from oid 41 current snap is 167 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: finishing write tid 4 to smithi20328086-31 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: finishing write tid 5 to smithi20328086-31 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: oid 31 updating version 0 to 504 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: oid 31 version 504 is already newer than 501 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: oid 31 version 504 is already newer than 503 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: oid 31 version 504 is already newer than 502 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 504 (ObjNum 563 snap 167 seq_num 563) dirty exists 2024-09-17T15:37:49.043 INFO:tasks.rados.rados.0.smithi203.stdout:1537: left oid 31 (ObjNum 563 snap 167 seq_num 563) 2024-09-17T15:37:49.044 INFO:tasks.rados.rados.0.smithi203.stdout:1540: expect (ObjNum 538 snap 160 seq_num 538) 2024-09-17T15:37:49.056 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:37:49.057 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:37:49.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:37:49.059 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:37:49.059 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:37:49.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:49.128 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: get_monmap_and_config 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: build_initial_monmap 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: monmap: 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:49.185435+0000 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:49.185435+0000 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:49.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 Processor -- start 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 -- start start 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: init 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f100a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 0x7f3f100a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-a con 0x7f3f100a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:49.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f1013d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f100a6530 con 0x7f3f100a50e0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f1009c150 con 0x7f3f1013d2f0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3f1009c430 con 0x7f3f100a54c0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f15d61640 10 monclient(hunting): authenticate will time out at 1602.150597s 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 0x7f3f100a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3f100a50e0 0x7f3f1009ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50368/0 (socket says 172.21.15.86:50368) 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/1936287412 learned_addr learned my addr 172.21.15.86:0/1936287412 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 10 monclient(hunting): get_auth_request con 0x7f3f100a54c0 auth_method 0 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:49.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 1 -- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 msgr2=0x7f3f100a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:49.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 1 -- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 msgr2=0x7f3f100a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:49.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 1 --2- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 0x7f3f100a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:49.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.183+0000 7f3f0effd640 1 --2- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 0x7f3f100a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:49.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/1936287412 >> v1:172.21.15.86:6789/0 conn(0x7f3f100a50e0 legacy=0x7f3f1009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:49.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 >> v1:172.21.15.203:6789/0 conn(0x7f3f1013d2f0 legacy=0x7f3f1013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:49.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3465602857 0 0) 0x7f3f1009c150 con 0x7f3f1013d2f0 2024-09-17T15:37:49.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): my global_id is 114157 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ef8003530 con 0x7f3f1013d2f0 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3672343243 0 0) 0x7f3ef8003530 con 0x7f3f1013d2f0 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 msgr2=0x7f3f100a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 --2- 172.21.15.86:0/1936287412 >> v2:172.21.15.86:3300/0 conn(0x7f3f100a54c0 0x7f3f100a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 >> v1:172.21.15.86:6789/0 conn(0x7f3f100a50e0 legacy=0x7f3f1009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:37:49.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f100a6940 con 0x7f3f1013d2f0 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f3f08001e40 con 0x7f3f1013d2f0 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f15d61640 5 monclient: authenticate success, global_id 114157 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: dump: 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:49.192 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 >> v1:172.21.15.203:6789/0 conn(0x7f3f1013d2f0 legacy=0x7f3f1013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef8003ad0 0x7f3ef8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): picked mon.a con 0x7f3ef8003ad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8004810 0x7f3ef8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): picked mon.b con 0x7f3ef8004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3ef800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): picked mon.c con 0x7f3ef8005420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1009c430 con 0x7f3ef8003ad0 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef800c360 con 0x7f3ef8004810 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef800c560 con 0x7f3ef8005420 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f15d61640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef8003ad0 0x7f3ef8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): get_auth_request con 0x7f3ef8003ad0 auth_method 0 2024-09-17T15:37:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3ef800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): get_auth_request con 0x7f3ef8005420 auth_method 0 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8004810 0x7f3ef8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:49.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): get_auth_request con 0x7f3ef8004810 auth_method 0 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient(hunting): handle_auth_done global_id 114157 payload 293 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8004810 msgr2=0x7f3ef8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ef8004810 0x7f3ef8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef8003ad0 msgr2=0x7f3ef8003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:49.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef8003ad0 0x7f3ef8003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 monclient: found mon.c 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef8003530 con 0x7f3ef8005420 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient: _finish_auth 0 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.187+0000 7f3f0f7fe640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef8003ad0 0x7f3ef8003ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:37:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 reap_dead start 2024-09-17T15:37:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0ffff640 1 -- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 msgr2=0x7f3ef800ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3ef800ba80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f1009cda0 tx=0x7f3f0801d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9f25da8c627de4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00001770 con 0x7f3ef8005420 2024-09-17T15:37:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:49.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: dump: 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:49.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f08005d90 con 0x7f3ef8005420 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 4 set_mon_vals no callback set 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 1 -- 172.21.15.86:0/1936287412 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800a3b0 con 0x7f3ef8005420 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f0e7fc640 10 monclient: dump: 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:49.205 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 10 monclient: shutdown 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 -- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 msgr2=0x7f3ef800ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3ef800ba80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f1009cda0 tx=0x7f3f0801d000 comp rx=0 tx=0).stop 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 -- 172.21.15.86:0/1936287412 shutdown_connections 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 --2- 172.21.15.86:0/1936287412 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3ef800ba80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 -- 172.21.15.86:0/1936287412 >> 172.21.15.86:0/1936287412 conn(0x7f3f100aaae0 msgr2=0x7f3f100aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 -- 172.21.15.86:0/1936287412 shutdown_connections 2024-09-17T15:37:49.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.191+0000 7f3f15d61640 1 -- 172.21.15.86:0/1936287412 wait complete. 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: build_initial_monmap 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: monmap: 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 Processor -- start 2024-09-17T15:37:49.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- start start 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: init 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef8005420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f10138720 0x7f3f10138b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-b con 0x7f3f10138720 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): picked mon.noname-a con 0x7f3f101cfa00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3f100a5c60 con 0x7f3f101cfa00 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f100a5ae0 con 0x7f3f10138720 2024-09-17T15:37:49.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1009ce10 con 0x7f3ef8005420 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient(hunting): authenticate will time out at 1602.160908s 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43228/0 (socket says 172.21.15.86:43228) 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/585866000 learned_addr learned my addr 172.21.15.86:0/585866000 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): get_auth_request con 0x7f3ef8005420 auth_method 0 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 10 monclient(hunting): get_auth_request con 0x7f3f101cfa00 auth_method 0 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f10138720 0x7f3f10138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 msgr2=0x7f3f101cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 msgr2=0x7f3f101cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): get_auth_request con 0x7f3f10138720 auth_method 0 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:49.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0ffff640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient(hunting): handle_auth_done global_id 104279 payload 824 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f10138720 msgr2=0x7f3f10138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f10138720 0x7f3f10138b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 msgr2=0x7f3f101cfe80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/585866000 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f101cd3f0 con 0x7f3ef8005420 2024-09-17T15:37:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 msgr2=0x7f3f101cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0f7fe640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f000076b0 tx=0x7f3f00001a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f00001d40 con 0x7f3ef8005420 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: dump: 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:49.212 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: _finish_auth 0 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 5 monclient: authenticate success, global_id 104279 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f00018070 con 0x7f3ef8005420 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _renew_subs 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f10121ae0 con 0x7f3ef8005420 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f0000fb60 con 0x7f3ef8005420 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3ef7fff640 10 monclient: dump: 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f0dffb640 4 set_mon_vals no callback set 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _renew_subs 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.195+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f100a1040 con 0x7f3ef8005420 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f15d61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f15d61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f15d61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f1009d1f0 con 0x7f3ef8005420 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3f0000fd00 con 0x7f3ef8005420 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3ef7fff640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 0x7f3efc03e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(351..351 src has 213..351) ==== 4755+0+0 (secure 0 0 0) 0x7f3f0004fc10 con 0x7f3ef8005420 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f0effd640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 0x7f3efc03e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f0effd640 10 monclient: get_auth_request con 0x7f3efc03bd00 auth_method 0 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.199+0000 7f3f0effd640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 0x7f3efc03e1c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3f1013d2a0 tx=0x7f3f04008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.203+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3f101cd3f0 con 0x7f3ef8005420 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.203+0000 7f3ef7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:49.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.203+0000 7f3ef7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:49.265 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2744278 2024-09-17T15:37:49.273 INFO:tasks.rados.rados.0.smithi203.stdout:1541: finishing write tid 1 to smithi20328086-28 2024-09-17T15:37:49.274 INFO:tasks.rados.rados.0.smithi203.stdout:1541: finishing write tid 2 to smithi20328086-28 2024-09-17T15:37:49.274 INFO:tasks.rados.rados.0.smithi203.stdout:1543: finishing copy_from to smithi20328086-38 2024-09-17T15:37:49.274 INFO:tasks.rados.rados.0.smithi203.stdout:1543: got expected ENOENT (src dne) 2024-09-17T15:37:49.274 INFO:tasks.rados.rados.0.smithi203.stdout:1543: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:37:49.274 INFO:tasks.rados.rados.0.smithi203.stdout:1542: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:37:49.307 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 675105 2024-09-17T15:37:49.362 INFO:tasks.rados.rados.0.smithi203.stdout:1541: finishing write tid 3 to smithi20328086-28 2024-09-17T15:37:49.362 INFO:tasks.rados.rados.0.smithi203.stdout:1541: finishing write tid 4 to smithi20328086-28 2024-09-17T15:37:49.362 INFO:tasks.rados.rados.0.smithi203.stdout:1541: finishing write tid 5 to smithi20328086-28 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: oid 28 updating version 0 to 663 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: oid 28 version 663 is already newer than 661 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: oid 28 version 663 is already newer than 662 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: oid 28 updating version 663 to 664 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 664 (ObjNum 565 snap 167 seq_num 565) dirty exists 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: left oid 28 (ObjNum 565 snap 167 seq_num 565) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1537: done (4 left) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1540: done (3 left) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1541: done (2 left) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1542: done (1 left) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1543: done (0 left) 2024-09-17T15:37:49.363 INFO:tasks.rados.rados.0.smithi203.stdout:1544: snap_create 2024-09-17T15:37:49.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:49.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:49.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:49.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3f1013cba0 con 0x7f3ef8005420 2024-09-17T15:37:49.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3ef7fff640 1 -- 172.21.15.86:0/585866000 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f3f00012d60 con 0x7f3ef8005420 2024-09-17T15:37:49.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3ef7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:49.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3ef7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:49.438 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":96,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:49.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 msgr2=0x7f3efc03e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 0x7f3efc03e1c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3f1013d2a0 tx=0x7f3f04008040 comp rx=0 tx=0).stop 2024-09-17T15:37:49.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 10 monclient: shutdown 2024-09-17T15:37:49.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 msgr2=0x7f3f101cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3f000076b0 tx=0x7f3f00001a00 comp rx=0 tx=0).stop 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 shutdown_connections 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3efc03bd00 0x7f3efc03e1c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f101cfa00 0x7f3f101cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f10138720 0x7f3f10138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 --2- 172.21.15.86:0/585866000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef8005420 0x7f3f101cc4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 >> 172.21.15.86:0/585866000 conn(0x7f3f100aaae0 msgr2=0x7f3f1009f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 shutdown_connections 2024-09-17T15:37:49.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:49.439+0000 7f3f15d61640 1 -- 172.21.15.86:0/585866000 wait complete. 2024-09-17T15:37:49.570 INFO:tasks.rados.rados.0.smithi203.stdout:1544: done (0 left) 2024-09-17T15:37:49.570 INFO:tasks.rados.rados.0.smithi203.stdout:1545: delete oid 50 current snap is 168 2024-09-17T15:37:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:1545: done (0 left) 2024-09-17T15:37:49.577 INFO:tasks.rados.rados.0.smithi203.stdout:1546: snap_remove snap 162 2024-09-17T15:37:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:1546: done (0 left) 2024-09-17T15:37:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:1547: write oid 28 current snap is 168 2024-09-17T15:37:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:1547: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-17T15:37:50.598 INFO:tasks.rados.rados.0.smithi203.stdout:1547: writing smithi20328086-28 from 729569 to 1205874 tid 1 2024-09-17T15:37:50.606 INFO:tasks.rados.rados.0.smithi203.stdout:1547: writing smithi20328086-28 from 1996349 to 2502215 tid 2 2024-09-17T15:37:50.612 INFO:tasks.rados.rados.0.smithi203.stdout:1547: writing smithi20328086-28 from 3004131 to 3529569 tid 3 2024-09-17T15:37:50.612 INFO:tasks.rados.rados.0.smithi203.stdout:1548: write oid 34 current snap is 168 2024-09-17T15:37:50.612 INFO:tasks.rados.rados.0.smithi203.stdout:1548: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-17T15:37:50.620 INFO:tasks.rados.rados.0.smithi203.stdout:1548: writing smithi20328086-34 from 746376 to 1297930 tid 1 2024-09-17T15:37:50.628 INFO:tasks.rados.rados.0.smithi203.stdout:1548: writing smithi20328086-34 from 2054831 to 2704355 tid 2 2024-09-17T15:37:50.633 INFO:tasks.rados.rados.0.smithi203.stdout:1548: writing smithi20328086-34 from 3315201 to 3546376 tid 3 2024-09-17T15:37:50.633 INFO:tasks.rados.rados.0.smithi203.stdout:1549: read oid 5 snap -1 2024-09-17T15:37:50.633 INFO:tasks.rados.rados.0.smithi203.stdout:1549: expect (ObjNum 503 snap 149 seq_num 503) 2024-09-17T15:37:50.633 INFO:tasks.rados.rados.0.smithi203.stdout:1550: copy_from oid 14 from oid 6 current snap is 168 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: finishing write tid 1 to smithi20328086-28 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: finishing write tid 2 to smithi20328086-28 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: finishing write tid 3 to smithi20328086-28 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1551: delete oid 10 current snap is 168 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: finishing write tid 4 to smithi20328086-28 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: finishing write tid 5 to smithi20328086-28 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: oid 28 updating version 0 to 671 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: oid 28 version 671 is already newer than 666 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: oid 28 version 671 is already newer than 670 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: oid 28 version 671 is already newer than 667 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 671 (ObjNum 567 snap 168 seq_num 567) dirty exists 2024-09-17T15:37:50.634 INFO:tasks.rados.rados.0.smithi203.stdout:1547: left oid 28 (ObjNum 567 snap 168 seq_num 567) 2024-09-17T15:37:50.637 INFO:tasks.rados.rados.0.smithi203.stdout:1549: expect (ObjNum 503 snap 149 seq_num 503) 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: finishing write tid 1 to smithi20328086-34 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: finishing write tid 2 to smithi20328086-34 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: finishing write tid 3 to smithi20328086-34 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1550: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: finishing write tid 4 to smithi20328086-34 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: finishing write tid 5 to smithi20328086-34 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: oid 34 updating version 0 to 498 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: oid 34 updating version 498 to 499 2024-09-17T15:37:50.856 INFO:tasks.rados.rados.0.smithi203.stdout:1548: oid 34 version 499 is already newer than 496 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1548: oid 34 version 499 is already newer than 497 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 499 (ObjNum 568 snap 168 seq_num 568) dirty exists 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1548: left oid 34 (ObjNum 568 snap 168 seq_num 568) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1550: finishing copy_from to smithi20328086-14 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 503 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1547: done (4 left) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1548: done (3 left) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1549: done (2 left) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1550: done (1 left) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1551: done (0 left) 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1552: write oid 8 current snap is 168 2024-09-17T15:37:50.857 INFO:tasks.rados.rados.0.smithi203.stdout:1552: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-09-17T15:37:50.866 INFO:tasks.rados.rados.0.smithi203.stdout:1552: writing smithi20328086-8 from 779990 to 1482042 tid 1 2024-09-17T15:37:50.875 INFO:tasks.rados.rados.0.smithi203.stdout:1552: writing smithi20328086-8 from 2055442 to 2708635 tid 2 2024-09-17T15:37:50.881 INFO:tasks.rados.rados.0.smithi203.stdout:1552: writing smithi20328086-8 from 3253694 to 3579990 tid 3 2024-09-17T15:37:50.882 INFO:tasks.rados.rados.0.smithi203.stdout:1553: copy_from oid 26 from oid 10 current snap is 168 2024-09-17T15:37:50.882 INFO:tasks.rados.rados.0.smithi203.stdout:1552: finishing write tid 1 to smithi20328086-8 2024-09-17T15:37:50.882 INFO:tasks.rados.rados.0.smithi203.stdout:1554: snap_create 2024-09-17T15:37:50.884 INFO:tasks.rados.rados.0.smithi203.stdout:1553: finishing copy_from to smithi20328086-26 2024-09-17T15:37:50.884 INFO:tasks.rados.rados.0.smithi203.stdout:1553: got expected ENOENT (src dne) 2024-09-17T15:37:50.884 INFO:tasks.rados.rados.0.smithi203.stdout:1553: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:37:50.887 INFO:tasks.rados.rados.0.smithi203.stdout:1552: finishing write tid 2 to smithi20328086-8 2024-09-17T15:37:50.888 INFO:tasks.rados.rados.0.smithi203.stdout:1552: finishing write tid 3 to smithi20328086-8 2024-09-17T15:37:50.888 INFO:tasks.rados.rados.0.smithi203.stdout:1552: finishing write tid 4 to smithi20328086-8 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: finishing write tid 5 to smithi20328086-8 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: oid 8 updating version 0 to 505 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: oid 8 updating version 505 to 506 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: oid 8 updating version 506 to 507 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: oid 8 version 507 is already newer than 504 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 507 (ObjNum 570 snap 168 seq_num 570) dirty exists 2024-09-17T15:37:50.889 INFO:tasks.rados.rados.0.smithi203.stdout:1552: left oid 8 (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:37:51.587 INFO:tasks.rados.rados.0.smithi203.stdout:1552: done (2 left) 2024-09-17T15:37:51.588 INFO:tasks.rados.rados.0.smithi203.stdout:1553: done (1 left) 2024-09-17T15:37:51.588 INFO:tasks.rados.rados.0.smithi203.stdout:1554: done (0 left) 2024-09-17T15:37:51.588 INFO:tasks.rados.rados.0.smithi203.stdout:1555: snap_create 2024-09-17T15:37:52.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:52.524 INFO:tasks.rados.rados.0.smithi203.stdout:1555: done (0 left) 2024-09-17T15:37:52.524 INFO:tasks.rados.rados.0.smithi203.stdout:1556: write oid 18 current snap is 170 2024-09-17T15:37:52.524 INFO:tasks.rados.rados.0.smithi203.stdout:1556: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2024-09-17T15:37:52.539 INFO:tasks.rados.rados.0.smithi203.stdout:1556: writing smithi20328086-18 from 796797 to 1290451 tid 1 2024-09-17T15:37:52.556 INFO:tasks.rados.rados.0.smithi203.stdout:1556: writing smithi20328086-18 from 1830277 to 2627128 tid 2 2024-09-17T15:37:52.568 INFO:tasks.rados.rados.0.smithi203.stdout:1556: writing smithi20328086-18 from 3397470 to 3596797 tid 3 2024-09-17T15:37:52.568 INFO:tasks.rados.rados.0.smithi203.stdout:1557: read oid 30 snap -1 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1557: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1558: read oid 29 snap -1 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1558: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1559: rollback oid 6 current snap is 170 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 168 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1556: finishing write tid 1 to smithi20328086-18 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1556: finishing write tid 2 to smithi20328086-18 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1560: read oid 4 snap 169 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1560: expect deleted 2024-09-17T15:37:52.569 INFO:tasks.rados.rados.0.smithi203.stdout:1561: delete oid 36 current snap is 170 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: finishing write tid 3 to smithi20328086-18 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: finishing write tid 4 to smithi20328086-18 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: finishing write tid 5 to smithi20328086-18 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: oid 18 updating version 0 to 509 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: oid 18 updating version 509 to 510 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: oid 18 version 510 is already newer than 508 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: oid 18 updating version 510 to 511 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 511 (ObjNum 571 snap 170 seq_num 571) dirty exists 2024-09-17T15:37:52.575 INFO:tasks.rados.rados.0.smithi203.stdout:1556: left oid 18 (ObjNum 571 snap 170 seq_num 571) 2024-09-17T15:37:52.581 INFO:tasks.rados.rados.0.smithi203.stdout:1558: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:52.581 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:52.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 10 monclient: get_monmap_and_config 2024-09-17T15:37:52.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 10 monclient: build_initial_monmap 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 10 monclient: monmap: 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:52.597946+0000 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:52.597946+0000 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:52.595 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 1 Processor -- start 2024-09-17T15:37:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 1 -- start start 2024-09-17T15:37:52.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.595+0000 7f4a832e6640 10 monclient: init 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a7c0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4a7c0a2dc0 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c13c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7c0a31f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:52.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7c0a4650 con 0x7f4a7c0689d0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4a7c068db0 con 0x7f4a7c0a2dc0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c09dc00 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a832e6640 10 monclient(hunting): authenticate will time out at 1605.564409s 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81082640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4a7c0689d0 0x7f4a7c0a26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50370/0 (socket says 172.21.15.86:50370) 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81082640 1 -- 172.21.15.86:0/1967120479 learned_addr learned my addr 172.21.15.86:0/1967120479 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 --2- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c13c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 10 monclient(hunting): get_auth_request con 0x7f4a7c0a2dc0 auth_method 0 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:52.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): get_auth_request con 0x7f4a7c0a31f0 auth_method 0 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 1 -- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c13a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 1 -- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c13a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 1 --2- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a80881640 1 --2- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient(hunting): handle_auth_done global_id 104282 payload 274 2024-09-17T15:37:52.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 -- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c13a4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 --2- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 -- 172.21.15.86:0/1967120479 >> v1:172.21.15.86:6789/0 conn(0x7f4a7c0689d0 legacy=0x7f4a7c0a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient: _finish_hunting 0 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 monclient: found mon.noname-b 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.599+0000 7f4a81883640 1 -- 172.21.15.86:0/1967120479 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c09e2d0 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 -- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 msgr2=0x7f4a7c13c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 --2- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c13c9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ff9a5773dda98c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 1 -- 172.21.15.86:0/1967120479 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740b2070 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: dump: 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.602 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: _finish_auth 0 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: _check_auth_tickets 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 1 -- 172.21.15.86:0/1967120479 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a740a5760 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 5 monclient: authenticate success, global_id 104282 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 1 -- 172.21.15.86:0/1967120479 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740a5b30 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 4 set_mon_vals no callback set 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a73fff640 10 monclient: dump: 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:52.604 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:52.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: shutdown 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 msgr2=0x7f4a7c13c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c13c9f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009d90 tx=0x7f4a7401c7d0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- 172.21.15.86:0/1967120479 shutdown_connections 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- 172.21.15.86:0/1967120479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c13c9f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- 172.21.15.86:0/1967120479 >> v2:172.21.15.86:3300/0 conn(0x7f4a7c0a2dc0 0x7f4a7c13a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- 172.21.15.86:0/1967120479 >> 172.21.15.86:0/1967120479 conn(0x7f4a7c0a72a0 msgr2=0x7f4a7c0a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:52.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- 172.21.15.86:0/1967120479 shutdown_connections 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- 172.21.15.86:0/1967120479 wait complete. 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: build_initial_monmap 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: monmap: 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:52.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 Processor -- start 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- start start 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: init 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a7c0689d0 0x7f4a7c122c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4a7c0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4a7c0a2dc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c123690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4a7c0a31f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c09be00 con 0x7f4a7c0a2dc0 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c09bc80 con 0x7f4a7c0689d0 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c09bf80 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a832e6640 10 monclient(hunting): authenticate will time out at 1605.571424s 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52940/0 (socket says 172.21.15.86:52940) 2024-09-17T15:37:52.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 -- 172.21.15.86:0/42057513 learned_addr learned my addr 172.21.15.86:0/42057513 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 10 monclient(hunting): get_auth_request con 0x7f4a7c0a2dc0 auth_method 0 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c123690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c123150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c123150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): get_auth_request con 0x7f4a7c0a31f0 auth_method 0 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a80881640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:52.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a7c0689d0 0x7f4a7c122c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): get_auth_request con 0x7f4a7c0689d0 auth_method 0 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81082640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient(hunting): handle_auth_done global_id 104285 payload 824 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a7c0689d0 msgr2=0x7f4a7c122c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:52.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a7c0689d0 0x7f4a7c122c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 msgr2=0x7f4a7c123150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient: _finish_hunting 0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 monclient: found mon.noname-c 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.603+0000 7f4a81883640 1 -- 172.21.15.86:0/42057513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c06f060 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a81883640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 msgr2=0x7f4a7c123690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a81883640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c123690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a740af000 tx=0x7f4a740195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740bb070 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:52.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: dump: 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: _finish_auth 0 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:52.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a740ad650 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 5 monclient: authenticate success, global_id 104285 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _renew_subs 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7c126700 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740b2040 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 10 monclient: dump: 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a737fe640 4 set_mon_vals no callback set 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _renew_subs 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7c0a52d0 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4a740ada40 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a7c0a26b0 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 0x7f4a6003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(355..355 src has 213..355) ==== 4783+0+0 (secure 0 0 0) 0x7f4a740ea510 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a81082640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 0x7f4a6003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.607+0000 7f4a81082640 10 monclient: get_auth_request con 0x7f4a6003bbb0 auth_method 0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.611+0000 7f4a81082640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 0x7f4a6003e070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a780092c0 tx=0x7f4a78009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.615+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4a7c06f060 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.615+0000 7f4a71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:52.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.615+0000 7f4a71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:52.704 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1809218 2024-09-17T15:37:52.806 INFO:tasks.rados.rados.0.smithi203.stdout:1559: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:37:52.806 INFO:tasks.rados.rados.0.smithi203.stdout:1559: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:37:52.807 INFO:tasks.rados.rados.0.smithi203.stdout:1559: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:37:52.807 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 551 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:37:52.807 INFO:tasks.rados.rados.0.smithi203.stdout:1557: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:37:52.833 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:37:52.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a832e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a832e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a832e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4a7c125b70 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a71ffb640 1 -- 172.21.15.86:0/42057513 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f4a74016540 con 0x7f4a7c0a31f0 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:52.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.851+0000 7f4a71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:52.852 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":96,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 msgr2=0x7f4a6003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 0x7f4a6003e070 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a780092c0 tx=0x7f4a78009250 comp rx=0 tx=0).stop 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 10 monclient: shutdown 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 msgr2=0x7f4a7c123690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c123690 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a740af000 tx=0x7f4a740195d0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 shutdown_connections 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a6003bbb0 0x7f4a6003e070 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a7c0a31f0 0x7f4a7c123690 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a7c0a2dc0 0x7f4a7c123150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 --2- 172.21.15.86:0/42057513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4a7c0689d0 0x7f4a7c122c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:52.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 >> 172.21.15.86:0/42057513 conn(0x7f4a7c0a72a0 msgr2=0x7f4a7c072770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:52.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 shutdown_connections 2024-09-17T15:37:52.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:52.855+0000 7f4a832e6640 1 -- 172.21.15.86:0/42057513 wait complete. 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1556: done (5 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1557: done (4 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1558: done (3 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1559: done (2 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1560: done (1 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1561: done (0 left) 2024-09-17T15:37:52.904 INFO:tasks.rados.rados.0.smithi203.stdout:1562: delete oid 16 current snap is 170 2024-09-17T15:37:52.910 INFO:tasks.rados.rados.0.smithi203.stdout:1562: done (0 left) 2024-09-17T15:37:52.910 INFO:tasks.rados.rados.0.smithi203.stdout:1563: read oid 16 snap -1 2024-09-17T15:37:52.911 INFO:tasks.rados.rados.0.smithi203.stdout:1563: expect deleted 2024-09-17T15:37:52.911 INFO:tasks.rados.rados.0.smithi203.stdout:1564: write oid 7 current snap is 170 2024-09-17T15:37:52.911 INFO:tasks.rados.rados.0.smithi203.stdout:1564: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-17T15:37:52.929 INFO:tasks.rados.rados.0.smithi203.stdout:1564: writing smithi20328086-7 from 447218 to 1166619 tid 1 2024-09-17T15:37:52.952 INFO:tasks.rados.rados.0.smithi203.stdout:1564: writing smithi20328086-7 from 1889370 to 2549901 tid 2 2024-09-17T15:37:52.967 INFO:tasks.rados.rados.0.smithi203.stdout:1564: writing smithi20328086-7 from 2963386 to 3577639 tid 3 2024-09-17T15:37:52.968 INFO:tasks.rados.rados.0.smithi203.stdout:1565: delete oid 32 current snap is 170 2024-09-17T15:37:52.968 INFO:tasks.rados.rados.0.smithi203.stdout:1564: finishing write tid 1 to smithi20328086-7 2024-09-17T15:37:52.968 INFO:tasks.rados.rados.0.smithi203.stdout:1564: finishing write tid 2 to smithi20328086-7 2024-09-17T15:37:52.974 INFO:tasks.rados.rados.0.smithi203.stdout:1563: done (2 left) 2024-09-17T15:37:52.974 INFO:tasks.rados.rados.0.smithi203.stdout:1565: done (1 left) 2024-09-17T15:37:52.974 INFO:tasks.rados.rados.0.smithi203.stdout:1566: snap_create 2024-09-17T15:37:52.975 INFO:tasks.rados.rados.0.smithi203.stdout:1564: finishing write tid 3 to smithi20328086-7 2024-09-17T15:37:52.975 INFO:tasks.rados.rados.0.smithi203.stdout:1564: finishing write tid 4 to smithi20328086-7 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: finishing write tid 5 to smithi20328086-7 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: oid 7 updating version 0 to 462 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: oid 7 version 462 is already newer than 461 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: oid 7 version 462 is already newer than 459 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: oid 7 version 462 is already newer than 460 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 462 (ObjNum 574 snap 170 seq_num 574) dirty exists 2024-09-17T15:37:52.976 INFO:tasks.rados.rados.0.smithi203.stdout:1564: left oid 7 (ObjNum 574 snap 170 seq_num 574) 2024-09-17T15:37:53.609 INFO:tasks.rados.rados.0.smithi203.stdout:1564: done (1 left) 2024-09-17T15:37:53.609 INFO:tasks.rados.rados.0.smithi203.stdout:1566: done (0 left) 2024-09-17T15:37:53.609 INFO:tasks.rados.rados.0.smithi203.stdout:1567: delete oid 26 current snap is 171 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1567: done (0 left) 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1568: copy_from oid 49 from oid 39 current snap is 171 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1569: read oid 29 snap 167 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1569: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1570: read oid 32 snap -1 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1570: expect deleted 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1571: rollback oid 38 current snap is 171 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 169 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1572: rollback oid 36 current snap is 171 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 163 2024-09-17T15:37:53.620 INFO:tasks.rados.rados.0.smithi203.stdout:1573: snap_remove snap 161 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:1573: done (5 left) 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:1574: copy_from oid 26 from oid 2 current snap is 171 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:1571: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:1572: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:1572: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:37:54.617 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 677 (ObjNum 513 snap 153 seq_num 513) dirty exists 2024-09-17T15:37:54.618 INFO:tasks.rados.rados.0.smithi203.stdout:1569: expect (ObjNum 475 snap 144 seq_num 475) 2024-09-17T15:37:54.747 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 936547 2024-09-17T15:37:54.791 INFO:tasks.rados.rados.0.smithi203.stdout:1568: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1568: finishing copy_from to smithi20328086-49 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 606 (ObjNum 562 snap 167 seq_num 562) dirty exists 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1568: done (5 left) 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1569: done (4 left) 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1570: done (3 left) 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1571: done (2 left) 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1572: done (1 left) 2024-09-17T15:37:54.792 INFO:tasks.rados.rados.0.smithi203.stdout:1575: snap_remove snap 168 2024-09-17T15:37:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:1575: done (1 left) 2024-09-17T15:37:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:1576: write (excl) oid 29 current snap is 171 2024-09-17T15:37:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:1574: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:37:55.627 INFO:tasks.rados.rados.0.smithi203.stdout:1574: finishing copy_from to smithi20328086-26 2024-09-17T15:37:55.628 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 458 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:37:55.628 INFO:tasks.rados.rados.0.smithi203.stdout:1576: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-17T15:37:55.647 INFO:tasks.rados.rados.0.smithi203.stdout:1576: writing smithi20328086-29 from 497639 to 1042787 tid 1 2024-09-17T15:37:55.663 INFO:tasks.rados.rados.0.smithi203.stdout:1576: writing smithi20328086-29 from 1548463 to 1956321 tid 2 2024-09-17T15:37:55.679 INFO:tasks.rados.rados.0.smithi203.stdout:1576: writing smithi20328086-29 from 2529302 to 3093724 tid 3 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1576: writing smithi20328086-29 from 3528143 to 3697639 tid 4 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1574: done (1 left) 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1577: snap_create 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 1 to smithi20328086-29 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 2 to smithi20328086-29 2024-09-17T15:37:55.687 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 3 to smithi20328086-29 2024-09-17T15:37:55.692 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 4 to smithi20328086-29 2024-09-17T15:37:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 5 to smithi20328086-29 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: finishing write tid 6 to smithi20328086-29 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: oid 29 updating version 0 to 519 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: oid 29 version 519 is already newer than 517 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: oid 29 updating version 519 to 521 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: oid 29 version 521 is already newer than 520 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: oid 29 version 521 is already newer than 518 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 521 (ObjNum 577 snap 171 seq_num 577) dirty exists 2024-09-17T15:37:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:1576: left oid 29 (ObjNum 577 snap 171 seq_num 577) 2024-09-17T15:37:55.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: get_monmap_and_config 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: build_initial_monmap 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: monmap: 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:56.009059+0000 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:56.009059+0000 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:56.007 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 Processor -- start 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 -- start start 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: init 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbe00689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 0x7fcbe006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe00694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcbe013aff0 0x7fcbe013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe0074470 con 0x7fcbe00689d0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe009c8b0 con 0x7fcbe013aff0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe009cb90 con 0x7fcbe00694c0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbe666f640 10 monclient(hunting): authenticate will time out at 1608.974661s 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 0x7fcbe006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcbe00689d0 0x7fcbe0068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50372/0 (socket says 172.21.15.86:50372) 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdffff640 1 -- 172.21.15.86:0/3446227533 learned_addr learned my addr 172.21.15.86:0/3446227533 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbe00694c0 auth_method 0 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:56.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient(hunting): handle_auth_done global_id 104288 payload 274 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3446227533 >> v2:172.21.15.203:3300/0 conn(0x7fcbe013aff0 msgr2=0x7fcbe013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/3446227533 >> v2:172.21.15.203:3300/0 conn(0x7fcbe013aff0 0x7fcbe013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3446227533 >> v1:172.21.15.86:6789/0 conn(0x7fcbe00689d0 legacy=0x7fcbe0068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 monclient: found mon.noname-b 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.007+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3446227533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe009ce70 con 0x7fcbe00694c0 2024-09-17T15:37:56.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdf7fe640 1 -- 172.21.15.86:0/3446227533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 msgr2=0x7fcbe006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:56.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/3446227533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 0x7fcbe006f840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40097c0 tx=0x7fcbd401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dae6d20d647242 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3446227533 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40b0070 con 0x7fcbe00694c0 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: dump: 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:56.014 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbe666f640 5 monclient: authenticate success, global_id 104288 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbe666f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3446227533 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd40ab600 con 0x7fcbe00694c0 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbe666f640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbe666f640 4 set_mon_vals no callback set 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 1 -- 172.21.15.86:0/3446227533 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe009c2d0 con 0x7fcbe00694c0 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:56.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.011+0000 7fcbdeffd640 10 monclient: dump: 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: shutdown 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:56.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- 172.21.15.86:0/3446227533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 msgr2=0x7fcbe006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- 172.21.15.86:0/3446227533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 0x7fcbe006f840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40097c0 tx=0x7fcbd401c360 comp rx=0 tx=0).stop 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- 172.21.15.86:0/3446227533 shutdown_connections 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- 172.21.15.86:0/3446227533 >> v2:172.21.15.203:3300/0 conn(0x7fcbe013aff0 0x7fcbe013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- 172.21.15.86:0/3446227533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe00694c0 0x7fcbe006f840 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- 172.21.15.86:0/3446227533 >> 172.21.15.86:0/3446227533 conn(0x7fcbe00a6440 msgr2=0x7fcbe00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- 172.21.15.86:0/3446227533 shutdown_connections 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- 172.21.15.86:0/3446227533 wait complete. 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: build_initial_monmap 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: monmap: 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:56.017 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:56.018 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 Processor -- start 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- start start 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: init 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00694c0 0x7fcbe0135fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe00694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 0x7fcbe01364e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:56.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe0140670 con 0x7fcbe00689d0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe01404f0 con 0x7fcbe00694c0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe01407f0 con 0x7fcbe013aff0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe666f640 10 monclient(hunting): authenticate will time out at 1608.982516s 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 0x7fcbe01364e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:52946/0 (socket says 172.21.15.86:52946) 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 -- 172.21.15.86:0/2087592652 learned_addr learned my addr 172.21.15.86:0/2087592652 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 10 monclient(hunting): get_auth_request con 0x7fcbe00689d0 auth_method 0 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:56.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): get_auth_request con 0x7fcbe013aff0 auth_method 0 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 msgr2=0x7fcbe01cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 msgr2=0x7fcbe01cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdffff640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:56.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdf7fe640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00694c0 0x7fcbe0135fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request con 0x7fcbe00694c0 auth_method 0 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient(hunting): handle_auth_done global_id 104291 payload 824 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00694c0 msgr2=0x7fcbe0135fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00694c0 0x7fcbe0135fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 msgr2=0x7fcbe01cfea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient: _finish_hunting 0 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 monclient: found mon.noname-c 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.015+0000 7fcbe4c0c640 1 -- 172.21.15.86:0/2087592652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe013d310 con 0x7fcbe013aff0 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe4c0c640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 msgr2=0x7fcbe01364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe4c0c640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 0x7fcbe01364e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000b3e0 tx=0x7fcbd0009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0015020 con 0x7fcbe013aff0 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: dump: 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.027 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: _finish_auth 0 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd0011a40 con 0x7fcbe013aff0 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 5 monclient: authenticate success, global_id 104291 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _renew_subs 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe01d03e0 con 0x7fcbe013aff0 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:56.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd0011e10 con 0x7fcbe013aff0 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 10 monclient: dump: 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:56.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _renew_subs 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe00738b0 con 0x7fcbe013aff0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbddffb640 4 set_mon_vals no callback set 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcbd0018020 con 0x7fcbe013aff0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbe006f840 con 0x7fcbe013aff0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 0x7fcbcc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(358..358 src has 213..358) ==== 4783+0+0 (secure 0 0 0) 0x7fcbd004fe50 con 0x7fcbe013aff0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdffff640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 0x7fcbcc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.019+0000 7fcbdffff640 10 monclient: get_auth_request con 0x7fcbcc03bb60 auth_method 0 2024-09-17T15:37:56.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.023+0000 7fcbdffff640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 0x7fcbcc03e020 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80052b0 tx=0x7fcbd8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:56.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.023+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcbd001f020 con 0x7fcbe013aff0 2024-09-17T15:37:56.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.023+0000 7fcbdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:56.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.023+0000 7fcbdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcbe01229b0 con 0x7fcbe013aff0 2024-09-17T15:37:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbdcff9640 1 -- 172.21.15.86:0/2087592652 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fcbd004e850 con 0x7fcbe013aff0 2024-09-17T15:37:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:56.258 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":96,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 msgr2=0x7fcbcc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 0x7fcbcc03e020 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80052b0 tx=0x7fcbd8009040 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 10 monclient: shutdown 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 msgr2=0x7fcbe01364e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 0x7fcbe01364e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcbd000b3e0 tx=0x7fcbd0009f90 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 shutdown_connections 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcbcc03bb60 0x7fcbcc03e020 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbe013aff0 0x7fcbe01364e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcbe00694c0 0x7fcbe0135fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 --2- 172.21.15.86:0/2087592652 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbe00689d0 0x7fcbe01cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 >> 172.21.15.86:0/2087592652 conn(0x7fcbe00a6440 msgr2=0x7fcbe009d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 shutdown_connections 2024-09-17T15:37:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:56.259+0000 7fcbe666f640 1 -- 172.21.15.86:0/2087592652 wait complete. 2024-09-17T15:37:56.634 INFO:tasks.rados.rados.0.smithi203.stdout:1576: done (1 left) 2024-09-17T15:37:56.634 INFO:tasks.rados.rados.0.smithi203.stdout:1577: done (0 left) 2024-09-17T15:37:56.634 INFO:tasks.rados.rados.0.smithi203.stdout:1578: read oid 4 snap -1 2024-09-17T15:37:56.634 INFO:tasks.rados.rados.0.smithi203.stdout:1578: expect deleted 2024-09-17T15:37:56.635 INFO:tasks.rados.rados.0.smithi203.stdout:1579: write (excl) oid 21 current snap is 172 2024-09-17T15:37:56.635 INFO:tasks.rados.rados.0.smithi203.stdout:1579: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-17T15:37:56.648 INFO:tasks.rados.rados.0.smithi203.stdout:1579: writing smithi20328086-21 from 514446 to 1251196 tid 1 2024-09-17T15:37:56.663 INFO:tasks.rados.rados.0.smithi203.stdout:1579: writing smithi20328086-21 from 1723298 to 2391167 tid 2 2024-09-17T15:37:56.678 INFO:tasks.rados.rados.0.smithi203.stdout:1579: writing smithi20328086-21 from 3073078 to 3714446 tid 3 2024-09-17T15:37:56.679 INFO:tasks.rados.rados.0.smithi203.stdout:1580: snap_create 2024-09-17T15:37:56.679 INFO:tasks.rados.rados.0.smithi203.stdout:1579: finishing write tid 1 to smithi20328086-21 2024-09-17T15:37:56.679 INFO:tasks.rados.rados.0.smithi203.stdout:1579: finishing write tid 2 to smithi20328086-21 2024-09-17T15:37:56.685 INFO:tasks.rados.rados.0.smithi203.stdout:1579: finishing write tid 3 to smithi20328086-21 2024-09-17T15:37:56.685 INFO:tasks.rados.rados.0.smithi203.stdout:1579: finishing write tid 4 to smithi20328086-21 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: finishing write tid 5 to smithi20328086-21 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: oid 21 updating version 0 to 698 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: oid 21 updating version 698 to 699 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: oid 21 version 699 is already newer than 697 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: oid 21 version 699 is already newer than 696 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 699 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:37:56.686 INFO:tasks.rados.rados.0.smithi203.stdout:1579: left oid 21 (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:37:57.527 INFO:tasks.rados.rados.0.smithi203.stdout:1578: done (2 left) 2024-09-17T15:37:57.527 INFO:tasks.rados.rados.0.smithi203.stdout:1579: done (1 left) 2024-09-17T15:37:57.527 INFO:tasks.rados.rados.0.smithi203.stdout:1580: done (0 left) 2024-09-17T15:37:57.527 INFO:tasks.rados.rados.0.smithi203.stdout:1581: write (excl) oid 47 current snap is 173 2024-09-17T15:37:57.527 INFO:tasks.rados.rados.0.smithi203.stdout:1581: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-17T15:37:57.538 INFO:tasks.rados.rados.0.smithi203.stdout:1581: writing smithi20328086-47 from 531253 to 943252 tid 1 2024-09-17T15:37:57.548 INFO:tasks.rados.rados.0.smithi203.stdout:1581: writing smithi20328086-47 from 1381780 to 1793307 tid 2 2024-09-17T15:37:57.560 INFO:tasks.rados.rados.0.smithi203.stdout:1581: writing smithi20328086-47 from 2300501 to 2887467 tid 3 2024-09-17T15:37:57.570 INFO:tasks.rados.rados.0.smithi203.stdout:1581: writing smithi20328086-47 from 3376974 to 3731253 tid 4 2024-09-17T15:37:57.570 INFO:tasks.rados.rados.0.smithi203.stdout:1582: read oid 40 snap 169 2024-09-17T15:37:57.570 INFO:tasks.rados.rados.0.smithi203.stdout:1582: expect deleted 2024-09-17T15:37:57.571 INFO:tasks.rados.rados.0.smithi203.stdout:1583: read oid 3 snap 170 2024-09-17T15:37:57.571 INFO:tasks.rados.rados.0.smithi203.stdout:1583: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:57.571 INFO:tasks.rados.rados.0.smithi203.stdout:1584: write oid 20 current snap is 173 2024-09-17T15:37:57.571 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 1 to smithi20328086-47 2024-09-17T15:37:57.571 INFO:tasks.rados.rados.0.smithi203.stdout:1584: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-17T15:37:57.583 INFO:tasks.rados.rados.0.smithi203.stdout:1584: writing smithi20328086-20 from 548060 to 1035308 tid 1 2024-09-17T15:37:57.596 INFO:tasks.rados.rados.0.smithi203.stdout:1584: writing smithi20328086-20 from 1440262 to 2111800 tid 2 2024-09-17T15:37:57.611 INFO:tasks.rados.rados.0.smithi203.stdout:1584: writing smithi20328086-20 from 2844277 to 3642515 tid 3 2024-09-17T15:37:57.612 INFO:tasks.rados.rados.0.smithi203.stdout:1585: snap_create 2024-09-17T15:37:57.612 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 2 to smithi20328086-47 2024-09-17T15:37:57.612 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 3 to smithi20328086-47 2024-09-17T15:37:57.612 INFO:tasks.rados.rados.0.smithi203.stdout:1583: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:37:57.624 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1354581 2024-09-17T15:37:57.685 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2206302 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 4 to smithi20328086-47 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 5 to smithi20328086-47 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: finishing write tid 6 to smithi20328086-47 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: oid 47 updating version 0 to 514 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: oid 47 version 514 is already newer than 512 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: oid 47 version 514 is already newer than 513 2024-09-17T15:37:57.715 INFO:tasks.rados.rados.0.smithi203.stdout:1581: oid 47 updating version 514 to 516 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1581: oid 47 version 516 is already newer than 515 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 516 (ObjNum 579 snap 173 seq_num 579) dirty exists 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1581: left oid 47 (ObjNum 579 snap 173 seq_num 579) 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: finishing write tid 1 to smithi20328086-20 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: finishing write tid 2 to smithi20328086-20 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: finishing write tid 3 to smithi20328086-20 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: finishing write tid 4 to smithi20328086-20 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: finishing write tid 5 to smithi20328086-20 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: oid 20 updating version 0 to 561 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: oid 20 updating version 561 to 562 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: oid 20 updating version 562 to 564 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:1584: oid 20 version 564 is already newer than 563 2024-09-17T15:37:57.716 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 564 (ObjNum 580 snap 173 seq_num 580) dirty exists 2024-09-17T15:37:57.717 INFO:tasks.rados.rados.0.smithi203.stdout:1584: left oid 20 (ObjNum 580 snap 173 seq_num 580) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1581: done (4 left) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1582: done (3 left) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1583: done (2 left) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1584: done (1 left) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1585: done (0 left) 2024-09-17T15:37:58.538 INFO:tasks.rados.rados.0.smithi203.stdout:1586: delete oid 46 current snap is 174 2024-09-17T15:37:58.545 INFO:tasks.rados.rados.0.smithi203.stdout:1586: done (0 left) 2024-09-17T15:37:58.545 INFO:tasks.rados.rados.0.smithi203.stdout:1587: delete oid 15 current snap is 174 2024-09-17T15:37:58.555 INFO:tasks.rados.rados.0.smithi203.stdout:1587: done (0 left) 2024-09-17T15:37:58.555 INFO:tasks.rados.rados.0.smithi203.stdout:1588: copy_from oid 30 from oid 6 current snap is 174 2024-09-17T15:37:58.555 INFO:tasks.rados.rados.0.smithi203.stdout:1589: rollback oid 48 current snap is 174 2024-09-17T15:37:58.555 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 163 2024-09-17T15:37:58.555 INFO:tasks.rados.rados.0.smithi203.stdout:1590: delete oid 4 current snap is 174 2024-09-17T15:37:58.567 INFO:tasks.rados.rados.0.smithi203.stdout:1589: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:37:58.568 INFO:tasks.rados.rados.0.smithi203.stdout:1590: done (2 left) 2024-09-17T15:37:58.568 INFO:tasks.rados.rados.0.smithi203.stdout:1591: read oid 15 snap -1 2024-09-17T15:37:58.568 INFO:tasks.rados.rados.0.smithi203.stdout:1591: expect deleted 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:1592: copy_from oid 38 from oid 28 current snap is 174 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:1589: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:1589: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 620 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:1589: done (3 left) 2024-09-17T15:37:58.569 INFO:tasks.rados.rados.0.smithi203.stdout:1593: delete oid 13 current snap is 174 2024-09-17T15:37:58.575 INFO:tasks.rados.rados.0.smithi203.stdout:1588: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:37:58.581 INFO:tasks.rados.rados.0.smithi203.stdout:1591: done (3 left) 2024-09-17T15:37:58.581 INFO:tasks.rados.rados.0.smithi203.stdout:1593: done (2 left) 2024-09-17T15:37:58.581 INFO:tasks.rados.rados.0.smithi203.stdout:1594: write (excl) oid 50 current snap is 174 2024-09-17T15:37:58.581 INFO:tasks.rados.rados.0.smithi203.stdout:1594: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-09-17T15:37:58.587 INFO:tasks.rados.rados.0.smithi203.stdout:1594: writing smithi20328086-50 from 632095 to 1095588 tid 1 2024-09-17T15:37:58.593 INFO:tasks.rados.rados.0.smithi203.stdout:1594: writing smithi20328086-50 from 1616319 to 2038853 tid 2 2024-09-17T15:37:58.600 INFO:tasks.rados.rados.0.smithi203.stdout:1594: writing smithi20328086-50 from 2748686 to 3235990 tid 3 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1595: delete oid 46 current snap is 174 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1588: finishing copy_from to smithi20328086-30 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 551 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1592: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1594: finishing write tid 1 to smithi20328086-50 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1592: finishing copy_from to smithi20328086-38 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 671 (ObjNum 567 snap 168 seq_num 567) dirty exists 2024-09-17T15:37:58.601 INFO:tasks.rados.rados.0.smithi203.stdout:1594: finishing write tid 2 to smithi20328086-50 2024-09-17T15:37:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:1588: done (3 left) 2024-09-17T15:37:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:1592: done (2 left) 2024-09-17T15:37:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:1595: done (1 left) 2024-09-17T15:37:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:1596: write (excl) oid 40 current snap is 174 2024-09-17T15:37:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:1596: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-17T15:37:58.612 INFO:tasks.rados.rados.0.smithi203.stdout:1596: writing smithi20328086-40 from 665709 to 1396053 tid 1 2024-09-17T15:37:58.620 INFO:tasks.rados.rados.0.smithi203.stdout:1596: writing smithi20328086-40 from 2133283 to 2559486 tid 2 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1596: writing smithi20328086-40 from 3203532 to 3713380 tid 3 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1597: copy_from oid 38 from oid 19 current snap is 174 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: finishing write tid 3 to smithi20328086-50 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: finishing write tid 4 to smithi20328086-50 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: finishing write tid 5 to smithi20328086-50 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: oid 50 updating version 0 to 315 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: oid 50 version 315 is already newer than 314 2024-09-17T15:37:58.627 INFO:tasks.rados.rados.0.smithi203.stdout:1594: oid 50 version 315 is already newer than 313 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1594: oid 50 version 315 is already newer than 312 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 315 (ObjNum 585 snap 174 seq_num 585) dirty exists 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1594: left oid 50 (ObjNum 585 snap 174 seq_num 585) 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1596: finishing write tid 1 to smithi20328086-40 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1596: finishing write tid 2 to smithi20328086-40 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1594: done (2 left) 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1598: read oid 1 snap -1 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1598: expect (ObjNum 556 snap 162 seq_num 556) 2024-09-17T15:37:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:1599: snap_remove snap 166 2024-09-17T15:37:59.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:37:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.407+0000 7f8728af2640 10 monclient: get_monmap_and_config 2024-09-17T15:37:59.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.407+0000 7f8728af2640 10 monclient: build_initial_monmap 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.407+0000 7f8728af2640 10 monclient: monmap: 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:37:59.411529+0000 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:37:59.411529+0000 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:37:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:37:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:37:59.410 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:37:59.410 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 Processor -- start 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 -- start start 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient: init 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-a con 0x7f8724069260 addr v2:172.21.15.86:3300/0 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8724074c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-b con 0x7f872409c6c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872409e410 con 0x7f8724074c30 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f872409d150 con 0x7f8724069260 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f872409ce80 con 0x7f872409c6c0 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8728af2640 10 monclient(hunting): authenticate will time out at 1612.377275s 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:59.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43850/0 (socket says 172.21.15.86:43850) 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 -- 172.21.15.86:0/3454510748 learned_addr learned my addr 172.21.15.86:0/3454510748 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 10 monclient(hunting): get_auth_request con 0x7f8724069260 auth_method 0 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): get_auth_request con 0x7f872409c6c0 auth_method 0 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:59.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 1 -- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 msgr2=0x7f872409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 1 -- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 msgr2=0x7f872409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 1 --2- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f872259c640 1 --2- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient(hunting): handle_auth_done global_id 104294 payload 274 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 -- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 msgr2=0x7f872409ba70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 --2- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 -- 172.21.15.86:0/3454510748 >> v1:172.21.15.203:6789/0 conn(0x7f8724074c30 legacy=0x7f872409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:37:59.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 monclient: found mon.noname-b 2024-09-17T15:37:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.411+0000 7f8722d9d640 1 -- 172.21.15.86:0/3454510748 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872409e6f0 con 0x7f872409c6c0 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8722d9d640 1 -- 172.21.15.86:0/3454510748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 msgr2=0x7f872413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8722d9d640 1 --2- 172.21.15.86:0/3454510748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f871c009e80 tx=0x7f871c01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8558792751944e98 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 1 -- 172.21.15.86:0/3454510748 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0a5520 con 0x7f872409c6c0 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: dump: 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:59.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: _finish_auth 0 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: _check_auth_tickets 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 1 -- 172.21.15.86:0/3454510748 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c0a56c0 con 0x7f872409c6c0 2024-09-17T15:37:59.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 1 -- 172.21.15.86:0/3454510748 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0a5a90 con 0x7f872409c6c0 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 5 monclient: authenticate success, global_id 104294 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 10 monclient: get_monmap_and_config success 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 4 set_mon_vals no callback set 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f872159a640 10 monclient: dump: 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 10 monclient: shutdown 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- 172.21.15.86:0/3454510748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 msgr2=0x7f872413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 --2- 172.21.15.86:0/3454510748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f871c009e80 tx=0x7f871c01c8d0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- 172.21.15.86:0/3454510748 shutdown_connections 2024-09-17T15:37:59.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 --2- 172.21.15.86:0/3454510748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f872413ba80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 --2- 172.21.15.86:0/3454510748 >> v2:172.21.15.86:3300/0 conn(0x7f8724069260 0x7f872409ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- 172.21.15.86:0/3454510748 >> 172.21.15.86:0/3454510748 conn(0x7f87240a6440 msgr2=0x7f87240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- 172.21.15.86:0/3454510748 shutdown_connections 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- 172.21.15.86:0/3454510748 wait complete. 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 10 monclient: build_initial_monmap 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 10 monclient: monmap: 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:37:59.426 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 Processor -- start 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 1 -- start start 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.415+0000 7f8728af2640 10 monclient: init 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _reopen_session rank -1 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-a con 0x7f8724069260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8724074c30 0x7f8724120830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-b con 0x7f8724074c30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:37:59.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): picked mon.noname-c con 0x7f872409c6c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f872413c380 con 0x7f8724069260 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f872413c200 con 0x7f8724074c30 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f872413c500 con 0x7f872409c6c0 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): _renew_subs 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient(hunting): authenticate will time out at 1612.384238s 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57698/0 (socket says 172.21.15.86:57698) 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43858/0 (socket says 172.21.15.86:43858) 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 -- 172.21.15.86:0/2700534981 learned_addr learned my addr 172.21.15.86:0/2700534981 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 10 monclient(hunting): get_auth_request con 0x7f8724069260 auth_method 0 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): get_auth_request con 0x7f872409c6c0 auth_method 0 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 msgr2=0x7f87241202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:37:59.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 msgr2=0x7f87241202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f872259c640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient(hunting): handle_auth_done global_id 104297 payload 824 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8724074c30 msgr2=0x7f8724120830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8724074c30 0x7f8724120830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 msgr2=0x7f87241202f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:37:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 monclient: found mon.noname-c 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 -- 172.21.15.86:0/2700534981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8724123200 con 0x7f872409c6c0 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 msgr2=0x7f8724120d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8722d9d640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f871c0ab000 tx=0x7f871c019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0bc030 con 0x7f872409c6c0 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.431 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: _finish_auth 0 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c0b8680 con 0x7f872409c6c0 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 5 monclient: authenticate success, global_id 104297 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _renew_subs 2024-09-17T15:37:59.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8724074320 con 0x7f872409c6c0 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c0a5520 con 0x7f872409c6c0 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:37:59.433 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8720d99640 4 set_mon_vals no callback set 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _renew_subs 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.419+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8724123d90 con 0x7f872409c6c0 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8728af2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8728af2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8728af2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872409ba70 con 0x7f872409c6c0 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f871c0bf070 con 0x7f872409c6c0 2024-09-17T15:37:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8712ffd640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 0x7f870403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:37:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(361..361 src has 213..361) ==== 4755+0+0 (secure 0 0 0) 0x7f871c0eaa80 con 0x7f872409c6c0 2024-09-17T15:37:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f872259c640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 0x7f870403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:37:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f872259c640 10 monclient: get_auth_request con 0x7f870403bb90 auth_method 0 2024-09-17T15:37:59.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.423+0000 7f872259c640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 0x7f870403e050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8714006e90 tx=0x7f87140091a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:37:59.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.427+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f871c0165e0 con 0x7f872409c6c0 2024-09-17T15:37:59.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.427+0000 7f8712ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:37:59.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.427+0000 7f8712ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:37:59.539 INFO:tasks.rados.rados.0.smithi203.stdout:1599: done (3 left) 2024-09-17T15:37:59.539 INFO:tasks.rados.rados.0.smithi203.stdout:1600: copy_from oid 34 from oid 39 current snap is 174 2024-09-17T15:37:59.539 INFO:tasks.rados.rados.0.smithi203.stdout:1598: expect (ObjNum 556 snap 162 seq_num 556) 2024-09-17T15:37:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8728af2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:37:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8728af2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8728af2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:37:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f872406d4f0 con 0x7f872409c6c0 2024-09-17T15:37:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8712ffd640 1 -- 172.21.15.86:0/2700534981 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f871c0c1460 con 0x7f872409c6c0 2024-09-17T15:37:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8712ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:37:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.675+0000 7f8712ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:37:59.675 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":96,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:37:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 msgr2=0x7f870403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 0x7f870403e050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8714006e90 tx=0x7f87140091a0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 10 monclient: shutdown 2024-09-17T15:37:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 msgr2=0x7f8724120d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f871c0ab000 tx=0x7f871c019670 comp rx=0 tx=0).stop 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 shutdown_connections 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f870403bb90 0x7f870403e050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f872409c6c0 0x7f8724120d70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8724074c30 0x7f8724120830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 --2- 172.21.15.86:0/2700534981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8724069260 0x7f87241202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 >> 172.21.15.86:0/2700534981 conn(0x7f87240a6440 msgr2=0x7f8724140910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 shutdown_connections 2024-09-17T15:37:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:37:59.679+0000 7f8728af2640 1 -- 172.21.15.86:0/2700534981 wait complete. 2024-09-17T15:37:59.783 INFO:tasks.rados.rados.0.smithi203.stdout:1596: finishing write tid 3 to smithi20328086-40 2024-09-17T15:37:59.783 INFO:tasks.rados.rados.0.smithi203.stdout:1596: finishing write tid 4 to smithi20328086-40 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: finishing write tid 5 to smithi20328086-40 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: oid 40 updating version 0 to 643 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: oid 40 version 643 is already newer than 642 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: oid 40 updating version 643 to 644 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: oid 40 updating version 644 to 645 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 645 (ObjNum 587 snap 174 seq_num 587) dirty exists 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: left oid 40 (ObjNum 587 snap 174 seq_num 587) 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1597: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1597: finishing copy_from to smithi20328086-38 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 542 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1600: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1596: done (3 left) 2024-09-17T15:37:59.784 INFO:tasks.rados.rados.0.smithi203.stdout:1597: done (2 left) 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1598: done (1 left) 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1601: read oid 1 snap 173 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1601: expect (ObjNum 556 snap 162 seq_num 556) 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1602: rollback oid 2 current snap is 174 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 165 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1600: finishing copy_from to smithi20328086-34 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 606 (ObjNum 562 snap 167 seq_num 562) dirty exists 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1600: done (2 left) 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1603: read oid 7 snap -1 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1603: expect (ObjNum 574 snap 170 seq_num 574) 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1604: write oid 37 current snap is 174 2024-09-17T15:37:59.785 INFO:tasks.rados.rados.0.smithi203.stdout:1604: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-17T15:37:59.791 INFO:tasks.rados.rados.0.smithi203.stdout:1604: writing smithi20328086-37 from 682516 to 1088109 tid 1 2024-09-17T15:37:59.802 INFO:tasks.rados.rados.0.smithi203.stdout:1604: writing smithi20328086-37 from 1791765 to 2361626 tid 2 2024-09-17T15:37:59.811 INFO:tasks.rados.rados.0.smithi203.stdout:1604: writing smithi20328086-37 from 2830955 to 3552075 tid 3 2024-09-17T15:37:59.811 INFO:tasks.rados.rados.0.smithi203.stdout:1605: copy_from oid 34 from oid 8 current snap is 174 2024-09-17T15:37:59.812 INFO:tasks.rados.rados.0.smithi203.stdout:1601: expect (ObjNum 556 snap 162 seq_num 556) 2024-09-17T15:38:00.036 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3324726 2024-09-17T15:38:00.043 INFO:tasks.rados.rados.0.smithi203.stdout:1602: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:38:00.043 INFO:tasks.rados.rados.0.smithi203.stdout:1602: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:38:00.043 INFO:tasks.rados.rados.0.smithi203.stdout:1602: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:38:00.044 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 465 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:38:00.044 INFO:tasks.rados.rados.0.smithi203.stdout:1603: expect (ObjNum 574 snap 170 seq_num 574) 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: finishing write tid 1 to smithi20328086-37 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: finishing write tid 2 to smithi20328086-37 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: finishing write tid 3 to smithi20328086-37 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: finishing write tid 4 to smithi20328086-37 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: finishing write tid 5 to smithi20328086-37 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: oid 37 updating version 0 to 629 2024-09-17T15:38:00.152 INFO:tasks.rados.rados.0.smithi203.stdout:1604: oid 37 updating version 629 to 631 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1604: oid 37 version 631 is already newer than 628 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1604: oid 37 version 631 is already newer than 630 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 631 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1604: left oid 37 (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1605: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1605: finishing copy_from to smithi20328086-34 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 507 (ObjNum 570 snap 168 seq_num 570) dirty exists 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1601: done (4 left) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1602: done (3 left) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1603: done (2 left) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1604: done (1 left) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1605: done (0 left) 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1606: read oid 19 snap 167 2024-09-17T15:38:00.153 INFO:tasks.rados.rados.0.smithi203.stdout:1606: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:38:00.154 INFO:tasks.rados.rados.0.smithi203.stdout:1607: copy_from oid 43 from oid 8 current snap is 174 2024-09-17T15:38:00.154 INFO:tasks.rados.rados.0.smithi203.stdout:1608: delete oid 25 current snap is 174 2024-09-17T15:38:00.155 INFO:tasks.rados.rados.0.smithi203.stdout:1606: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:38:00.239 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1383644 2024-09-17T15:38:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:1607: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:38:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:1607: finishing copy_from to smithi20328086-43 2024-09-17T15:38:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 507 (ObjNum 570 snap 168 seq_num 570) dirty exists 2024-09-17T15:38:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:1606: done (2 left) 2024-09-17T15:38:00.264 INFO:tasks.rados.rados.0.smithi203.stdout:1607: done (1 left) 2024-09-17T15:38:00.264 INFO:tasks.rados.rados.0.smithi203.stdout:1608: done (0 left) 2024-09-17T15:38:00.264 INFO:tasks.rados.rados.0.smithi203.stdout:1609: read oid 10 snap 171 2024-09-17T15:38:00.264 INFO:tasks.rados.rados.0.smithi203.stdout:1609: expect deleted 2024-09-17T15:38:00.264 INFO:tasks.rados.rados.0.smithi203.stdout:1610: delete oid 47 current snap is 174 2024-09-17T15:38:00.273 INFO:tasks.rados.rados.0.smithi203.stdout:1609: done (1 left) 2024-09-17T15:38:00.273 INFO:tasks.rados.rados.0.smithi203.stdout:1610: done (0 left) 2024-09-17T15:38:00.273 INFO:tasks.rados.rados.0.smithi203.stdout:1611: read oid 9 snap -1 2024-09-17T15:38:00.273 INFO:tasks.rados.rados.0.smithi203.stdout:1611: expect (ObjNum 454 snap 137 seq_num 454) 2024-09-17T15:38:00.273 INFO:tasks.rados.rados.0.smithi203.stdout:1612: delete oid 5 current snap is 174 2024-09-17T15:38:00.281 INFO:tasks.rados.rados.0.smithi203.stdout:1611: expect (ObjNum 454 snap 137 seq_num 454) 2024-09-17T15:38:00.459 INFO:tasks.rados.rados.0.smithi203.stdout:1611: done (1 left) 2024-09-17T15:38:00.460 INFO:tasks.rados.rados.0.smithi203.stdout:1612: done (0 left) 2024-09-17T15:38:00.460 INFO:tasks.rados.rados.0.smithi203.stdout:1613: rollback oid 30 current snap is 174 2024-09-17T15:38:00.460 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 30 to 167 2024-09-17T15:38:00.460 INFO:tasks.rados.rados.0.smithi203.stdout:1614: snap_create 2024-09-17T15:38:00.468 INFO:tasks.rados.rados.0.smithi203.stdout:1613: finishing rollback tid 0 to smithi20328086-30 2024-09-17T15:38:00.468 INFO:tasks.rados.rados.0.smithi203.stdout:1613: finishing rollback tid 1 to smithi20328086-30 2024-09-17T15:38:00.468 INFO:tasks.rados.rados.0.smithi203.stdout:1613: finishing rollback tid 2 to smithi20328086-30 2024-09-17T15:38:00.469 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 554 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:38:00.547 INFO:tasks.rados.rados.0.smithi203.stdout:1613: done (1 left) 2024-09-17T15:38:00.547 INFO:tasks.rados.rados.0.smithi203.stdout:1614: done (0 left) 2024-09-17T15:38:00.547 INFO:tasks.rados.rados.0.smithi203.stdout:1615: snap_create 2024-09-17T15:38:01.550 INFO:tasks.rados.rados.0.smithi203.stdout:1615: done (0 left) 2024-09-17T15:38:01.550 INFO:tasks.rados.rados.0.smithi203.stdout:1616: delete oid 10 current snap is 176 2024-09-17T15:38:01.554 INFO:tasks.rados.rados.0.smithi203.stdout:1616: done (0 left) 2024-09-17T15:38:01.554 INFO:tasks.rados.rados.0.smithi203.stdout:1617: snap_remove snap 165 2024-09-17T15:38:02.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: get_monmap_and_config 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: build_initial_monmap 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: monmap: 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:02.825975+0000 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:02.825975+0000 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:02.823 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:02.824 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:02.824 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:02.824 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 Processor -- start 2024-09-17T15:38:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 -- start start 2024-09-17T15:38:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: init 2024-09-17T15:38:02.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-b con 0x7fe39c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe39c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe39c13aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c074470 con 0x7fe39c13aff0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c09c8b0 con 0x7fe39c068db0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c09cbe0 con 0x7fe39c0689d0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a3509640 10 monclient(hunting): authenticate will time out at 1615.791419s 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a1aa6640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 0x7fe39c13d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48290/0 (socket says 172.21.15.86:48290) 2024-09-17T15:38:02.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/2352583603 learned_addr learned my addr 172.21.15.86:0/2352583603 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe39c0689d0 auth_method 0 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.823+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:02.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.827+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 legacy=0x7fe39c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:02.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:02.827+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.203:6789/0 conn(0x7fe39c068db0 legacy=0x7fe39c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:03.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:03.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe39c0689d0 auth_method 0 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:03.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.027+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe39c0689d0 auth_method 0 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:03.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:03.431+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:04.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.143+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 legacy=0x7fe39c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-17T15:38:04.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.143+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 legacy=0x7fe39c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:04.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.143+0000 7fe3a1aa6640 1 --1- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 0x7fe39c13d3e0 :-1 s=OPENED pgs=31 cs=1 l=1).handle_message read tag failed 2024-09-17T15:38:04.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.143+0000 7fe3a1aa6640 1 --1- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 0x7fe39c13d3e0 :-1 s=OPENED pgs=31 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:38:04.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.143+0000 7fe39bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-09-17T15:38:04.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe39c0689d0 auth_method 0 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:04.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:04.235+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:38:05.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): tick 2024-09-17T15:38:05.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 monclient(hunting): continuing hunt 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 msgr2=0x7fe39c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe39c0689d0 0x7fe39c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.203:6789/0 conn(0x7fe39c068db0 legacy=0x7fe39c0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 -- 172.21.15.86:0/2352583603 >> v1:172.21.15.86:6789/0 conn(0x7fe39c13aff0 legacy=0x7fe39c13d3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): picked mon.noname-a con 0x7fe3880036d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): picked mon.noname-c con 0x7fe388006220 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 -- 172.21.15.86:0/2352583603 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c09cbe0 con 0x7fe3880036d0 2024-09-17T15:38:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 1 -- 172.21.15.86:0/2352583603 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe388008d80 con 0x7fe388006220 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.823+0000 7fe39affd640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe3880036d0 auth_method 0 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request con 0x7fe388006220 auth_method 0 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:05.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:05.827+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe3880036d0 auth_method 0 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request con 0x7fe388006220 auth_method 0 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:06.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:06.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.031+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:06.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe3880036d0 auth_method 0 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 msgr2=0x7fe388005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request con 0x7fe388006220 auth_method 0 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:06.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:06.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:06.435+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:06.964 INFO:tasks.rados.rados.0.smithi203.stdout:1617: done (0 left) 2024-09-17T15:38:06.964 INFO:tasks.rados.rados.0.smithi203.stdout:1618: copy_from oid 45 from oid 10 current snap is 176 2024-09-17T15:38:06.964 INFO:tasks.rados.rados.0.smithi203.stdout:1619: write (excl) oid 20 current snap is 176 2024-09-17T15:38:06.964 INFO:tasks.rados.rados.0.smithi203.stdout:1619: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-17T15:38:06.984 INFO:tasks.rados.rados.0.smithi203.stdout:1619: writing smithi20328086-20 from 766551 to 1264742 tid 1 2024-09-17T15:38:07.003 INFO:tasks.rados.rados.0.smithi203.stdout:1619: writing smithi20328086-20 from 1684175 to 2405032 tid 2 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1619: writing smithi20328086-20 from 2851717 to 3545550 tid 3 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1620: rollback oid 48 current snap is 176 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 172 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1621: read oid 8 snap 164 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1621: expect deleted 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1622: rollback oid 32 current snap is 176 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 169 2024-09-17T15:38:07.020 INFO:tasks.rados.rados.0.smithi203.stdout:1623: delete oid 24 current snap is 176 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.86:3300/0 conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe3880036d0 auth_method 0 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request con 0x7fe388006220 auth_method 0 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient(hunting): handle_auth_done global_id 124098 payload 274 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 msgr2=0x7fe3880086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient: _finish_hunting 0 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 1 monclient: found mon.noname-a 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c09ce70 con 0x7fe3880036d0 2024-09-17T15:38:07.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.239+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/2352583603 >> v2:172.21.15.203:3300/0 conn(0x7fe388006220 0x7fe3880086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a12a5640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3880036d0 msgr2=0x7fe388005b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3880036d0 0x7fe388005b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3880061f0 tx=0x7fe38c01dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe0aa098df768f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 1 -- 172.21.15.86:0/2352583603 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe38c0ac070 con 0x7fe3880036d0 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: dump: 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:07.250 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:07.251 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: _finish_auth 0 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 1 -- 172.21.15.86:0/2352583603 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe38c0a6cc0 con 0x7fe3880036d0 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 1 -- 172.21.15.86:0/2352583603 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c09c2d0 con 0x7fe3880036d0 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe39bfff640 10 monclient: dump: 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 5 monclient: authenticate success, global_id 124098 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 4 set_mon_vals no callback set 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 10 monclient: shutdown 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 1 -- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3880036d0 msgr2=0x7fe388005b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3880036d0 0x7fe388005b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe3880061f0 tx=0x7fe38c01dd60 comp rx=0 tx=0).stop 2024-09-17T15:38:07.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/2352583603 reap_dead start 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 1 -- 172.21.15.86:0/2352583603 shutdown_connections 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 1 --2- 172.21.15.86:0/2352583603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3880036d0 0x7fe388005b30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.247+0000 7fe3a3509640 1 -- 172.21.15.86:0/2352583603 >> 172.21.15.86:0/2352583603 conn(0x7fe39c0a6440 msgr2=0x7fe39c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- 172.21.15.86:0/2352583603 shutdown_connections 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- 172.21.15.86:0/2352583603 wait complete. 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient: build_initial_monmap 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient: monmap: 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:07.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 Processor -- start 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- start start 2024-09-17T15:38:07.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient: init 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-c con 0x7fe3880036d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe39c1359d0 0x7fe39c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-b con 0x7fe39c1359d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 0x7fe39c11ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): picked mon.noname-a con 0x7fe39c135e30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:07.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c141bb0 con 0x7fe39c135e30 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c141a30 con 0x7fe39c1359d0 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c141d30 con 0x7fe3880036d0 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a3509640 10 monclient(hunting): authenticate will time out at 1620.218383s 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43910/0 (socket says 172.21.15.86:43910) 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 -- 172.21.15.86:0/1621079659 learned_addr learned my addr 172.21.15.86:0/1621079659 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request con 0x7fe3880036d0 auth_method 0 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:07.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 0x7fe39c11ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): get_auth_request con 0x7fe39c135e30 auth_method 0 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 msgr2=0x7fe39c11db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 msgr2=0x7fe39c11db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:07.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a0aa4640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe39c1359d0 0x7fe39c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request con 0x7fe39c1359d0 auth_method 0 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a0aa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a0aa4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient(hunting): handle_auth_done global_id 124101 payload 824 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 msgr2=0x7fe39c11db50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe39c1359d0 msgr2=0x7fe39c11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe39c1359d0 0x7fe39c11c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient: _finish_hunting 0 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 monclient: found mon.noname-a 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:07.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.251+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/1621079659 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c11e4e0 con 0x7fe39c135e30 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a1aa6640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 msgr2=0x7fe39c11ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a1aa6640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 0x7fe39c11ce00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0a0670 tx=0x7fe39400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394017070 con 0x7fe39c135e30 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: dump: 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:07.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: _finish_auth 0 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 5 monclient: authenticate success, global_id 124101 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe39400ac50 con 0x7fe39c135e30 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _renew_subs 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3940070d0 con 0x7fe39c135e30 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c11f350 con 0x7fe39c135e30 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 10 monclient: dump: 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:07.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe39b7fe640 4 set_mon_vals no callback set 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _renew_subs 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c073660 con 0x7fe39c135e30 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe39401a070 con 0x7fe39c135e30 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39c13a6f0 con 0x7fe39c135e30 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 0x7fe38803ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(365..365 src has 213..365) ==== 4799+0+0 (secure 0 0 0) 0x7fe39404f9a0 con 0x7fe39c135e30 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 0x7fe38803ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.255+0000 7fe3a12a5640 10 monclient: get_auth_request con 0x7fe38803ca50 auth_method 0 2024-09-17T15:38:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.259+0000 7fe3a12a5640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 0x7fe38803ef10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe38c01dda0 tx=0x7fe38c0a8070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.263+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe39404e390 con 0x7fe39c135e30 2024-09-17T15:38:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.263+0000 7fe399ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.263+0000 7fe399ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe39c06f800 con 0x7fe39c135e30 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe399ffb640 1 -- 172.21.15.86:0/1621079659 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fe394019cd0 con 0x7fe39c135e30 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe399ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:07.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe399ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:07.491 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":100,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 msgr2=0x7fe38803ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 0x7fe38803ef10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe38c01dda0 tx=0x7fe38c0a8070 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 10 monclient: shutdown 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.491+0000 7fe3a3509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 msgr2=0x7fe39c11ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 0x7fe39c11ce00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0a0670 tx=0x7fe39400cdb0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 shutdown_connections 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe38803ca50 0x7fe38803ef10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe39c135e30 0x7fe39c11ce00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe39c1359d0 0x7fe39c11c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 --2- 172.21.15.86:0/1621079659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3880036d0 0x7fe39c11db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:07.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 >> 172.21.15.86:0/1621079659 conn(0x7fe39c0a6440 msgr2=0x7fe39c0a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:07.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 shutdown_connections 2024-09-17T15:38:07.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:07.495+0000 7fe3a3509640 1 -- 172.21.15.86:0/1621079659 wait complete. 2024-09-17T15:38:10.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:10.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: get_monmap_and_config 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: build_initial_monmap 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: monmap: 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:10.640858+0000 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:10.640858+0000 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:10.638 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:10.639 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:10.639 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:10.639 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 Processor -- start 2024-09-17T15:38:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 -- start start 2024-09-17T15:38:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: init 2024-09-17T15:38:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbed40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-c con 0x7fbed40694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-b con 0x7fbed413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed4074470 con 0x7fbed40689d0 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbed409c900 con 0x7fbed40694c0 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbed409cb90 con 0x7fbed413aff0 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:10.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedc78f640 10 monclient(hunting): authenticate will time out at 1623.606325s 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbeda52b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbed40689d0 0x7fbed4068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42318/0 (socket says 172.21.15.86:42318) 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbeda52b640 1 -- 172.21.15.86:0/2311630359 learned_addr learned my addr 172.21.15.86:0/2311630359 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:10.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed40694c0 auth_method 0 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.639+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:10.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.643+0000 7fbeda52b640 1 -- 172.21.15.86:0/2311630359 >> v1:172.21.15.86:6789/0 conn(0x7fbed40689d0 legacy=0x7fbed4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed40694c0 auth_method 0 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:10.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:10.843+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed40694c0 auth_method 0 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:11.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:11.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:11.247+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:12.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed40694c0 auth_method 0 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:12.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:12.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:12.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:12.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.051+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 411267786 0 0) 0x7fbed4074470 con 0x7fbed40689d0 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): _negotiate 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): my global_id is 124104 2024-09-17T15:38:12.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec0003530 con 0x7fbed40689d0 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 849387854 0 0) 0x7fbec0003530 con 0x7fbed40689d0 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbed413d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbed413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 msgr2=0x7fbed406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 --2- 172.21.15.86:0/2311630359 >> v2:172.21.15.203:3300/0 conn(0x7fbed40694c0 0x7fbed406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _finish_hunting 0 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed409ce70 con 0x7fbed40689d0 2024-09-17T15:38:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _finish_auth 0 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _check_auth_tickets 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fbec40019b0 con 0x7fbed40689d0 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: dump: 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.895 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 >> v1:172.21.15.86:6789/0 conn(0x7fbed40689d0 legacy=0x7fbed4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0003ad0 0x7fbec0003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): picked mon.a con 0x7fbec0003ad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbec0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): picked mon.c con 0x7fbed413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec0004e70 0x7fbec000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): picked mon.b con 0x7fbec0004e70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbed409c900 con 0x7fbec0003ad0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbed409cb90 con 0x7fbec0004e70 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbec000bf10 con 0x7fbed413aff0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.891+0000 7fbed9529640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0003ad0 0x7fbec0003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbec0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): get_auth_request con 0x7fbec0003ad0 auth_method 0 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec0004e70 0x7fbec000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient(hunting): handle_auth_done global_id 124104 payload 293 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec0004e70 msgr2=0x7fbec000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec0004e70 0x7fbec000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0003ad0 msgr2=0x7fbec0003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbec0003ad0 0x7fbec0003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient: _finish_hunting 0 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 monclient: found mon.c 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec0003530 con 0x7fbed413aff0 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient: _finish_auth 0 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 10 monclient: _check_auth_tickets 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbed9d2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbedad2c640 10 monclient: get_auth_request con 0x7fbec0004e70 auth_method 0 2024-09-17T15:38:12.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbedad2c640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbec0004e70 0x7fbec000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:38:12.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbedc78f640 5 monclient: authenticate success, global_id 124104 2024-09-17T15:38:12.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.895+0000 7fbedc78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9d2a640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbec0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbec0004810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbed00129c0 tx=0x7fbed001d150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a927ad2e3333ff7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00b3070 con 0x7fbed413aff0 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:12.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: dump: 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed00ae410 con 0x7fbed413aff0 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:12.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 1 -- 172.21.15.86:0/2311630359 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed000f040 con 0x7fbed413aff0 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 4 set_mon_vals no callback set 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbed9529640 10 monclient: dump: 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:12.903 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 10 monclient: shutdown 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 1 -- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 msgr2=0x7fbec0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 1 --2- 172.21.15.86:0/2311630359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed413aff0 0x7fbec0004810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbed00129c0 tx=0x7fbed001d150 comp rx=0 tx=0).stop 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedad2c640 1 -- 172.21.15.86:0/2311630359 reap_dead start 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 1 -- 172.21.15.86:0/2311630359 shutdown_connections 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.899+0000 7fbedc78f640 1 -- 172.21.15.86:0/2311630359 >> 172.21.15.86:0/2311630359 conn(0x7fbed40a6440 msgr2=0x7fbed40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- 172.21.15.86:0/2311630359 shutdown_connections 2024-09-17T15:38:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- 172.21.15.86:0/2311630359 wait complete. 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient: build_initial_monmap 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient: monmap: 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:12.907 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 Processor -- start 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- start start 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient: init 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed413aff0 0x7fbed41d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-b con 0x7fbed413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-c con 0x7fbed40734e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 0x7fbed41cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): picked mon.noname-a con 0x7fbed41ccc10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbed413c060 con 0x7fbed41ccc10 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbed413bee0 con 0x7fbed413aff0 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbed413c1e0 con 0x7fbed40734e0 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedc78f640 10 monclient(hunting): authenticate will time out at 1625.870606s 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 0x7fbed41cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36808/0 (socket says 172.21.15.86:36808) 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): get_auth_request con 0x7fbed41ccc10 auth_method 0 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 -- 172.21.15.86:0/1400077965 learned_addr learned my addr 172.21.15.86:0/1400077965 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request con 0x7fbed40734e0 auth_method 0 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:12.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 msgr2=0x7fbed4073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 msgr2=0x7fbed4073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed413aff0 0x7fbed41d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbed9d2a640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): get_auth_request con 0x7fbed413aff0 auth_method 0 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbeda52b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient(hunting): handle_auth_done global_id 124107 payload 824 2024-09-17T15:38:12.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 msgr2=0x7fbed4073960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed413aff0 msgr2=0x7fbed41d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed413aff0 0x7fbed41d0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient: _finish_hunting 0 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 monclient: found mon.noname-a 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.903+0000 7fbedad2c640 1 -- 172.21.15.86:0/1400077965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed413e1a0 con 0x7fbed41ccc10 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedad2c640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 msgr2=0x7fbed41cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedad2c640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 0x7fbed41cd090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbed40698a0 tx=0x7fbecc004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01c020 con 0x7fbed41ccc10 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:12.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: dump: 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: _finish_auth 0 2024-09-17T15:38:12.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbecc018ac0 con 0x7fbed41ccc10 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc007450 con 0x7fbed41ccc10 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 5 monclient: authenticate success, global_id 124107 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _renew_subs 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbed8d28640 4 set_mon_vals no callback set 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed409d460 con 0x7fbed41ccc10 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 10 monclient: dump: 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:12.914 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _renew_subs 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed40720b0 con 0x7fbed41ccc10 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbecc01f020 con 0x7fbed41ccc10 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed4071eb0 con 0x7fbed41ccc10 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 0x7fbeb803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(365..365 src has 213..365) ==== 4799+0+0 (secure 0 0 0) 0x7fbecc050940 con 0x7fbed41ccc10 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbeda52b640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 0x7fbeb803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.907+0000 7fbeda52b640 10 monclient: get_auth_request con 0x7fbeb803bbb0 auth_method 0 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.911+0000 7fbeda52b640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 0x7fbeb803e070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbec4007fd0 tx=0x7fbec4007b70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.915+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbecc026070 con 0x7fbed41ccc10 2024-09-17T15:38:12.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.915+0000 7fbecaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:12.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:12.915+0000 7fbecaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbed406f840 con 0x7fbed41ccc10 2024-09-17T15:38:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbecaffd640 1 -- 172.21.15.86:0/1400077965 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fbecc04f330 con 0x7fbed41ccc10 2024-09-17T15:38:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbecaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbecaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:13.162 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":104,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 msgr2=0x7fbeb803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 0x7fbeb803e070 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fbec4007fd0 tx=0x7fbec4007b70 comp rx=0 tx=0).stop 2024-09-17T15:38:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 10 monclient: shutdown 2024-09-17T15:38:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 msgr2=0x7fbed41cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 0x7fbed41cd090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbed40698a0 tx=0x7fbecc004410 comp rx=0 tx=0).stop 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 shutdown_connections 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbeb803bbb0 0x7fbeb803e070 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbed41ccc10 0x7fbed41cd090 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbed40734e0 0x7fbed4073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 --2- 172.21.15.86:0/1400077965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbed413aff0 0x7fbed41d0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:13.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 >> 172.21.15.86:0/1400077965 conn(0x7fbed40a6440 msgr2=0x7fbed4132c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 shutdown_connections 2024-09-17T15:38:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:13.163+0000 7fbedc78f640 1 -- 172.21.15.86:0/1400077965 wait complete. 2024-09-17T15:38:14.168 INFO:tasks.rados.rados.0.smithi203.stdout:1618: finishing copy_from to smithi20328086-45 2024-09-17T15:38:14.168 INFO:tasks.rados.rados.0.smithi203.stdout:1618: got expected ENOENT (src dne) 2024-09-17T15:38:14.168 INFO:tasks.rados.rados.0.smithi203.stdout:1618: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:38:14.174 INFO:tasks.rados.rados.0.smithi203.stdout:1622: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:38:14.174 INFO:tasks.rados.rados.0.smithi203.stdout:1622: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:38:14.174 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 317 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:38:14.175 INFO:tasks.rados.rados.0.smithi203.stdout:1618: done (5 left) 2024-09-17T15:38:14.175 INFO:tasks.rados.rados.0.smithi203.stdout:1621: done (4 left) 2024-09-17T15:38:14.175 INFO:tasks.rados.rados.0.smithi203.stdout:1622: done (3 left) 2024-09-17T15:38:14.175 INFO:tasks.rados.rados.0.smithi203.stdout:1623: done (2 left) 2024-09-17T15:38:14.175 INFO:tasks.rados.rados.0.smithi203.stdout:1624: snap_create 2024-09-17T15:38:14.180 INFO:tasks.rados.rados.0.smithi203.stdout:1620: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:38:14.180 INFO:tasks.rados.rados.0.smithi203.stdout:1620: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:38:14.180 INFO:tasks.rados.rados.0.smithi203.stdout:1620: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:38:14.180 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 635 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:38:14.188 INFO:tasks.rados.rados.0.smithi203.stdout:1619: finishing write tid 1 to smithi20328086-20 2024-09-17T15:38:14.188 INFO:tasks.rados.rados.0.smithi203.stdout:1619: finishing write tid 2 to smithi20328086-20 2024-09-17T15:38:14.189 INFO:tasks.rados.rados.0.smithi203.stdout:1619: finishing write tid 3 to smithi20328086-20 2024-09-17T15:38:14.193 INFO:tasks.rados.rados.0.smithi203.stdout:1619: finishing write tid 4 to smithi20328086-20 2024-09-17T15:38:14.193 INFO:tasks.rados.rados.0.smithi203.stdout:1619: finishing write tid 5 to smithi20328086-20 2024-09-17T15:38:14.193 INFO:tasks.rados.rados.0.smithi203.stdout:1619: oid 20 updating version 0 to 575 2024-09-17T15:38:14.193 INFO:tasks.rados.rados.0.smithi203.stdout:1619: oid 20 version 575 is already newer than 574 2024-09-17T15:38:14.193 INFO:tasks.rados.rados.0.smithi203.stdout:1619: oid 20 version 575 is already newer than 573 2024-09-17T15:38:14.194 INFO:tasks.rados.rados.0.smithi203.stdout:1619: oid 20 version 575 is already newer than 572 2024-09-17T15:38:14.194 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 575 (ObjNum 593 snap 176 seq_num 593) dirty exists 2024-09-17T15:38:14.194 INFO:tasks.rados.rados.0.smithi203.stdout:1619: left oid 20 (ObjNum 593 snap 176 seq_num 593) 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1619: done (2 left) 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1620: done (1 left) 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1624: done (0 left) 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1625: copy_from oid 47 from oid 19 current snap is 177 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1626: write oid 18 current snap is 177 2024-09-17T15:38:15.795 INFO:tasks.rados.rados.0.smithi203.stdout:1626: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-17T15:38:15.812 INFO:tasks.rados.rados.0.smithi203.stdout:1626: writing smithi20328086-18 from 400165 to 1048854 tid 1 2024-09-17T15:38:15.827 INFO:tasks.rados.rados.0.smithi203.stdout:1626: writing smithi20328086-18 from 1684786 to 2000165 tid 2 2024-09-17T15:38:15.827 INFO:tasks.rados.rados.0.smithi203.stdout:1627: snap_create 2024-09-17T15:38:15.827 INFO:tasks.rados.rados.0.smithi203.stdout:1625: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:38:15.827 INFO:tasks.rados.rados.0.smithi203.stdout:1625: finishing copy_from to smithi20328086-47 2024-09-17T15:38:15.828 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 542 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:38:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:1626: finishing write tid 1 to smithi20328086-18 2024-09-17T15:38:15.835 INFO:tasks.rados.rados.0.smithi203.stdout:1626: finishing write tid 2 to smithi20328086-18 2024-09-17T15:38:15.836 INFO:tasks.rados.rados.0.smithi203.stdout:1626: finishing write tid 3 to smithi20328086-18 2024-09-17T15:38:15.836 INFO:tasks.rados.rados.0.smithi203.stdout:1626: finishing write tid 4 to smithi20328086-18 2024-09-17T15:38:15.836 INFO:tasks.rados.rados.0.smithi203.stdout:1626: oid 18 updating version 0 to 544 2024-09-17T15:38:15.836 INFO:tasks.rados.rados.0.smithi203.stdout:1626: oid 18 version 544 is already newer than 543 2024-09-17T15:38:15.837 INFO:tasks.rados.rados.0.smithi203.stdout:1626: oid 18 updating version 544 to 545 2024-09-17T15:38:15.837 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 545 (ObjNum 595 snap 177 seq_num 595) dirty exists 2024-09-17T15:38:15.837 INFO:tasks.rados.rados.0.smithi203.stdout:1626: left oid 18 (ObjNum 595 snap 177 seq_num 595) 2024-09-17T15:38:16.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: get_monmap_and_config 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: monmap: 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:16.317084+0000 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:16.317084+0000 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:16.314 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:16.315 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:16.315 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:16.315 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:16.315 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 Processor -- start 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 -- start start 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: init 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ea40689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1ea4068db0 0x7f1ea40691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea4068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea4139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea409d770 con 0x7f1ea40689d0 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea409ed00 con 0x7f1ea4068db0 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea409efe0 con 0x7f1ea4139f00 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea9cce640 10 monclient(hunting): authenticate will time out at 1629.282642s 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36844/0 (socket says 172.21.15.86:36844) 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 -- 172.21.15.86:0/399363547 learned_addr learned my addr 172.21.15.86:0/399363547 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): get_auth_request con 0x7f1ea4139f00 auth_method 0 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea2ffd640 1 --2- 172.21.15.86:0/399363547 >> v2:172.21.15.203:3300/0 conn(0x7f1ea4068db0 0x7f1ea40691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea2ffd640 10 monclient(hunting): get_auth_request con 0x7f1ea4068db0 auth_method 0 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_done global_id 104300 payload 274 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 -- 172.21.15.86:0/399363547 >> v2:172.21.15.203:3300/0 conn(0x7f1ea4068db0 msgr2=0x7f1ea40691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 --2- 172.21.15.86:0/399363547 >> v2:172.21.15.203:3300/0 conn(0x7f1ea4068db0 0x7f1ea40691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 -- 172.21.15.86:0/399363547 >> v1:172.21.15.203:6789/0 conn(0x7f1ea40689d0 legacy=0x7f1ea409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 monclient: found mon.noname-b 2024-09-17T15:38:16.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.315+0000 7f1ea3fff640 1 -- 172.21.15.86:0/399363547 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea4069700 con 0x7f1ea4139f00 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea3fff640 1 -- 172.21.15.86:0/399363547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 msgr2=0x7f1ea413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea3fff640 1 --2- 172.21.15.86:0/399363547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e50 tx=0x7f1e9c01c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adbb0a82ee74d243 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 1 -- 172.21.15.86:0/399363547 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0a56d0 con 0x7f1ea4139f00 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: dump: 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 1 -- 172.21.15.86:0/399363547 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9c0a5870 con 0x7f1ea4139f00 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 1 -- 172.21.15.86:0/399363547 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0a5c40 con 0x7f1ea4139f00 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea9cce640 5 monclient: authenticate success, global_id 104300 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea9cce640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea9cce640 4 set_mon_vals no callback set 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.319+0000 7f1ea27fc640 10 monclient: dump: 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.326 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: shutdown 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- 172.21.15.86:0/399363547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 msgr2=0x7f1ea413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/399363547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e50 tx=0x7f1e9c01c980 comp rx=0 tx=0).stop 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- 172.21.15.86:0/399363547 shutdown_connections 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/399363547 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea4139f00 0x7f1ea413c2f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/399363547 >> v2:172.21.15.203:3300/0 conn(0x7f1ea4068db0 0x7f1ea40691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- 172.21.15.86:0/399363547 >> 172.21.15.86:0/399363547 conn(0x7f1ea40a6440 msgr2=0x7f1ea40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- 172.21.15.86:0/399363547 shutdown_connections 2024-09-17T15:38:16.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- 172.21.15.86:0/399363547 wait complete. 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: monmap: 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 Processor -- start 2024-09-17T15:38:16.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- start start 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: init 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-c con 0x7f1ea40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4068db0 0x7f1ea41328b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-b con 0x7f1ea4068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 0x7f1ea4132df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): picked mon.noname-a con 0x7f1ea4139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea409be00 con 0x7f1ea4139f00 2024-09-17T15:38:16.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea409bc80 con 0x7f1ea4068db0 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea409bf80 con 0x7f1ea40689d0 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea9cce640 10 monclient(hunting): authenticate will time out at 1629.289908s 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36854/0 (socket says 172.21.15.86:36854) 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 0x7f1ea4132df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 -- 172.21.15.86:0/1854158574 learned_addr learned my addr 172.21.15.86:0/1854158574 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): get_auth_request con 0x7f1ea40689d0 auth_method 0 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): get_auth_request con 0x7f1ea4139f00 auth_method 0 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea2ffd640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4068db0 0x7f1ea41328b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient(hunting): handle_auth_done global_id 104303 payload 824 2024-09-17T15:38:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 -- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4068db0 msgr2=0x7f1ea41328b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4068db0 0x7f1ea41328b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 -- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 msgr2=0x7f1ea4132df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 0x7f1ea4132df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 monclient: found mon.noname-c 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea37fe640 1 -- 172.21.15.86:0/1854158574 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea4134f40 con 0x7f1ea40689d0 2024-09-17T15:38:16.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.323+0000 7f1ea3fff640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 0x7f1ea4132df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:16.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea37fe640 1 -- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 msgr2=0x7f1ea4132370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea37fe640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400c480 tx=0x7f1e9400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94012a40 con 0x7f1ea40689d0 2024-09-17T15:38:16.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: dump: 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e94017070 con 0x7f1ea40689d0 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94012e40 con 0x7f1ea40689d0 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 10 monclient: dump: 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.336 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea1ffb640 4 set_mon_vals no callback set 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 5 monclient: authenticate success, global_id 104303 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _renew_subs 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea4135220 con 0x7f1ea40689d0 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _renew_subs 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea409d460 con 0x7f1ea40689d0 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea40691c0 con 0x7f1ea40689d0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1e9401c460 con 0x7f1ea40689d0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 0x7f1e9003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(366..366 src has 213..366) ==== 4771+0+0 (secure 0 0 0) 0x7f1e9404fc70 con 0x7f1ea40689d0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea2ffd640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 0x7f1e9003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.327+0000 7f1ea2ffd640 10 monclient: get_auth_request con 0x7f1e9003bbb0 auth_method 0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.331+0000 7f1ea2ffd640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 0x7f1e9003e070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4133570 tx=0x7f1e98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.331+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1e9401b3c0 con 0x7f1ea40689d0 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.331+0000 7f1e8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.331+0000 7f1e8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1ea9cce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1ea9cce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1ea9cce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1ea41345f0 con 0x7f1ea40689d0 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1e8bfff640 1 -- 172.21.15.86:0/1854158574 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1e9401ac80 con 0x7f1ea40689d0 2024-09-17T15:38:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1e8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.563+0000 7f1e8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:16.565 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 msgr2=0x7f1e9003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 0x7f1e9003e070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4133570 tx=0x7f1e98008040 comp rx=0 tx=0).stop 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 10 monclient: shutdown 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 msgr2=0x7f1ea4132370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400c480 tx=0x7f1e9400c940 comp rx=0 tx=0).stop 2024-09-17T15:38:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 shutdown_connections 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e9003bbb0 0x7f1e9003e070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ea4139f00 0x7f1ea4132df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ea4068db0 0x7f1ea41328b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 --2- 172.21.15.86:0/1854158574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ea40689d0 0x7f1ea4132370 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 >> 172.21.15.86:0/1854158574 conn(0x7f1ea40a6440 msgr2=0x7f1ea413b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 shutdown_connections 2024-09-17T15:38:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.567+0000 7f1ea9cce640 1 -- 172.21.15.86:0/1854158574 wait complete. 2024-09-17T15:38:16.584 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:38:16.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:38:16.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: get_monmap_and_config 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: monmap: 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:16.706360+0000 2024-09-17T15:38:16.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:16.706360+0000 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:16.704 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 Processor -- start 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 -- start start 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: init 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 0x7fcf0c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-b con 0x7fcf0c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf0c0a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcf0c136160 0x7fcf0c13df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf0c136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0c0a2130 con 0x7fcf0c0a1590 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcf0c0a2410 con 0x7fcf0c136160 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c0a26f0 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.703+0000 7fcf140f6640 10 monclient(hunting): authenticate will time out at 1629.671959s 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11691640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcf0c0a1590 0x7fcf0c135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42334/0 (socket says 172.21.15.86:42334) 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11691640 1 -- 172.21.15.86:0/248320974 learned_addr learned my addr 172.21.15.86:0/248320974 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 --2- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 0x7fcf0c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): get_auth_request con 0x7fcf0c0a0c70 auth_method 0 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf12693640 1 --2- 172.21.15.86:0/248320974 >> v2:172.21.15.203:3300/0 conn(0x7fcf0c136160 0x7fcf0c13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf12693640 10 monclient(hunting): get_auth_request con 0x7fcf0c136160 auth_method 0 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf12693640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf12693640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf12693640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient(hunting): handle_auth_done global_id 104306 payload 274 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 -- 172.21.15.86:0/248320974 >> v2:172.21.15.203:3300/0 conn(0x7fcf0c136160 msgr2=0x7fcf0c13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 --2- 172.21.15.86:0/248320974 >> v2:172.21.15.203:3300/0 conn(0x7fcf0c136160 0x7fcf0c13df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 -- 172.21.15.86:0/248320974 >> v1:172.21.15.86:6789/0 conn(0x7fcf0c0a1590 legacy=0x7fcf0c135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 monclient: found mon.noname-b 2024-09-17T15:38:16.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.707+0000 7fcf11e92640 1 -- 172.21.15.86:0/248320974 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c09f9f0 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 1 -- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 msgr2=0x7fcf0c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 1 --2- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 0x7fcf0c0a1050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097c0 tx=0x7fcefc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b66eebf97f280a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 1 -- 172.21.15.86:0/248320974 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0b0070 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: dump: 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 1 -- 172.21.15.86:0/248320974 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcefc0ab600 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 5 monclient: authenticate success, global_id 104306 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 4 set_mon_vals no callback set 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 1 -- 172.21.15.86:0/248320974 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0ab9d0 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf10e90640 10 monclient: dump: 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: shutdown 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 msgr2=0x7fcf0c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 0x7fcf0c0a1050 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097c0 tx=0x7fcefc01c360 comp rx=0 tx=0).stop 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- 172.21.15.86:0/248320974 shutdown_connections 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- 172.21.15.86:0/248320974 >> v2:172.21.15.203:3300/0 conn(0x7fcf0c136160 0x7fcf0c13df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- 172.21.15.86:0/248320974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a0c70 0x7fcf0c0a1050 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- 172.21.15.86:0/248320974 >> 172.21.15.86:0/248320974 conn(0x7fcf0c0a2f40 msgr2=0x7fcf0c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- 172.21.15.86:0/248320974 shutdown_connections 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- 172.21.15.86:0/248320974 wait complete. 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: monmap: 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 Processor -- start 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- start start 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: init 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c0a0c70 0x7fcf0c09f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-b con 0x7fcf0c0a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a1590 0x7fcf0c1354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf0c0a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 0x7fcf0c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0c136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c0a00d0 con 0x7fcf0c136160 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c09ff50 con 0x7fcf0c0a0c70 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c140c90 con 0x7fcf0c0a1590 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf140f6640 10 monclient(hunting): authenticate will time out at 1629.679204s 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a1590 0x7fcf0c1354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 0x7fcf0c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 0x7fcf0c09d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59734/0 (socket says 172.21.15.86:59734) 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a1590 0x7fcf0c1354c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36866/0 (socket says 172.21.15.86:36866) 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 learned_addr learned my addr 172.21.15.86:0/4270442953 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:16.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): get_auth_request con 0x7fcf0c136160 auth_method 0 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): get_auth_request con 0x7fcf0c0a1590 auth_method 0 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11691640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c0a0c70 0x7fcf0c09f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 10 monclient(hunting): get_auth_request con 0x7fcf0c0a0c70 auth_method 0 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf11e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient(hunting): handle_auth_done global_id 124113 payload 824 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a1590 msgr2=0x7fcf0c1354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0c0a1590 0x7fcf0c1354c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c0a0c70 msgr2=0x7fcf0c09f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf0c0a0c70 0x7fcf0c09f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 monclient: found mon.noname-a 2024-09-17T15:38:16.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.711+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c09e850 con 0x7fcf0c136160 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 msgr2=0x7fcf0c09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf12693640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 0x7fcf0c09d770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007dc0 tx=0x7fcf0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014620 con 0x7fcf0c136160 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: dump: 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.726 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf04019070 con 0x7fcf0c136160 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 5 monclient: authenticate success, global_id 124113 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 monclient: _renew_subs 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c09d460 con 0x7fcf0c136160 2024-09-17T15:38:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04014a20 con 0x7fcf0c136160 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 10 monclient: dump: 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.728 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf03fff640 4 set_mon_vals no callback set 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 monclient: _renew_subs 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c121e90 con 0x7fcf0c136160 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 0x7fcf0c09dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c06fc00 con 0x7fcf0c1d0060 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 0x7fcf0c09dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient: get_auth_request con 0x7fcf0c1d0060 auth_method 0 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcf04014bc0 con 0x7fcf0c136160 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 10 monclient(hunting): handle_auth_done global_id 124116 payload 824 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11e92640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf0c0a1050 con 0x7fcf0c1d0060 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 0x7fcef003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(366..366 src has 213..366) ==== 4771+0+0 (secure 0 0 0) 0x7fcf04050760 con 0x7fcf0c136160 2024-09-17T15:38:16.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.715+0000 7fcf11691640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 0x7fcef003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11691640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 msgr2=0x7fcef003e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:38:16.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11691640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 msgr2=0x7fcef003e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:16.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11691640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 0x7fcef003e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:16.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11691640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 0x7fcef003e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11e92640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 msgr2=0x7fcf0c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf11e92640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 0x7fcf0c09dcb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ad000 tx=0x7fcefc019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0bb070 con 0x7fcf0c1d0060 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 10 monclient: dump: 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.734 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcefc0c0450 con 0x7fcf0c1d0060 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 msgr2=0x7fcf0c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 0x7fcf0c09dcb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ad000 tx=0x7fcefc019100 comp rx=0 tx=0).stop 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 msgr2=0x7fcf0c09dcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.719+0000 7fcf027fc640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c1d0060 0x7fcf0c09dcb0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ad000 tx=0x7fcefc019100 comp rx=0 tx=0).stop 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 0x7fcf0c138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c09e600 con 0x7fcf0c09e1f0 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 0x7fcf0c138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient: get_auth_request con 0x7fcf0c09e1f0 auth_method 0 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 10 monclient(hunting): handle_auth_done global_id 124119 payload 824 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 1 -- 172.21.15.86:0/4270442953 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf0c1216d0 con 0x7fcf0c09e1f0 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 msgr2=0x7fcf0c138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf11e92640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 0x7fcf0c138f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ae040 tx=0x7fcefc009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0ce020 con 0x7fcf0c09e1f0 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 10 monclient: dump: 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fcefc0b6860 con 0x7fcf0c09e1f0 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:16.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 msgr2=0x7fcf0c138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 0x7fcf0c138f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ae040 tx=0x7fcefc009190 comp rx=0 tx=0).stop 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 msgr2=0x7fcf0c138f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf027fc640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c09e1f0 0x7fcf0c138f00 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0ae040 tx=0x7fcefc009190 comp rx=0 tx=0).stop 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 27591, 2024-09-17T15:38:16.738 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.739 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.740 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:16.741 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:16.742 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.743 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.744 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.745 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:16.746 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.747 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:16.751 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 msgr2=0x7fcef003e020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcef003bb60 0x7fcef003e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 10 monclient: shutdown 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 msgr2=0x7fcf0c09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 --2- 172.21.15.86:0/4270442953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf0c136160 0x7fcf0c09d770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcf04007dc0 tx=0x7fcf0400e4b0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 reap_dead start 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf12693640 1 -- 172.21.15.86:0/4270442953 reap_dead start 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 shutdown_connections 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 >> 172.21.15.86:0/4270442953 conn(0x7fcf0c0a2f40 msgr2=0x7fcf0c0723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 shutdown_connections 2024-09-17T15:38:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.727+0000 7fcf140f6640 1 -- 172.21.15.86:0/4270442953 wait complete. 2024-09-17T15:38:16.754 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:16.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:38:16.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:38:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: get_monmap_and_config 2024-09-17T15:38:16.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: monmap: 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:16.865315+0000 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:16.865315+0000 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:16.864 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 Processor -- start 2024-09-17T15:38:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 -- start start 2024-09-17T15:38:16.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: init 2024-09-17T15:38:16.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:38:16.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eec0a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1eec136160 0x7f1eec13df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eec0a2130 con 0x7f1eec0a1590 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1eec0a2410 con 0x7f1eec136160 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec0a26f0 con 0x7f1eec0a0c70 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef44b1640 10 monclient(hunting): authenticate will time out at 1629.830451s 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36878/0 (socket says 172.21.15.86:36878) 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 -- 172.21.15.86:0/767724040 learned_addr learned my addr 172.21.15.86:0/767724040 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): get_auth_request con 0x7f1eec0a0c70 auth_method 0 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef2a4e640 1 --2- 172.21.15.86:0/767724040 >> v2:172.21.15.203:3300/0 conn(0x7f1eec136160 0x7f1eec13df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef2a4e640 10 monclient(hunting): get_auth_request con 0x7f1eec136160 auth_method 0 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef2a4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef2a4e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef2a4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient(hunting): handle_auth_done global_id 104312 payload 274 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 -- 172.21.15.86:0/767724040 >> v2:172.21.15.203:3300/0 conn(0x7f1eec136160 msgr2=0x7f1eec13df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 --2- 172.21.15.86:0/767724040 >> v2:172.21.15.203:3300/0 conn(0x7f1eec136160 0x7f1eec13df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 -- 172.21.15.86:0/767724040 >> v1:172.21.15.203:6789/0 conn(0x7f1eec0a1590 legacy=0x7f1eec135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 monclient: found mon.noname-b 2024-09-17T15:38:16.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.863+0000 7f1ef224d640 1 -- 172.21.15.86:0/767724040 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec09f9f0 con 0x7f1eec0a0c70 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef224d640 1 -- 172.21.15.86:0/767724040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 msgr2=0x7f1eec0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef224d640 1 --2- 172.21.15.86:0/767724040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009870 tx=0x7f1ee801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e15e3428d9ef8683 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 1 -- 172.21.15.86:0/767724040 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80b1070 con 0x7f1eec0a0c70 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: dump: 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.878 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 5 monclient: authenticate success, global_id 104312 2024-09-17T15:38:16.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 1 -- 172.21.15.86:0/767724040 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee80ac620 con 0x7f1eec0a0c70 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 4 set_mon_vals no callback set 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 1 -- 172.21.15.86:0/767724040 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80ac9f0 con 0x7f1eec0a0c70 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef124b640 10 monclient: dump: 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 10 monclient: shutdown 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 1 -- 172.21.15.86:0/767724040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 msgr2=0x7f1eec0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.867+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/767724040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009870 tx=0x7f1ee801c420 comp rx=0 tx=0).stop 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- 172.21.15.86:0/767724040 shutdown_connections 2024-09-17T15:38:16.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/767724040 >> v2:172.21.15.203:3300/0 conn(0x7f1eec136160 0x7f1eec13df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/767724040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec0a1050 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- 172.21.15.86:0/767724040 >> 172.21.15.86:0/767724040 conn(0x7f1eec0a2f40 msgr2=0x7f1eec0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- 172.21.15.86:0/767724040 shutdown_connections 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- 172.21.15.86:0/767724040 wait complete. 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient: build_initial_monmap 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient: monmap: 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 Processor -- start 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- start start 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient: init 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eec0a1590 0x7f1eec135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-a con 0x7f1eec0a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eec136160 0x7f1eec11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec136160 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:16.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec0a00d0 con 0x7f1eec0a1590 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec09ff50 con 0x7f1eec136160 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec140c90 con 0x7f1eec0a0c70 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef44b1640 10 monclient(hunting): authenticate will time out at 1629.837457s 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eec0a1590 0x7f1eec135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36882/0 (socket says 172.21.15.86:36882) 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 -- 172.21.15.86:0/2340460690 learned_addr learned my addr 172.21.15.86:0/2340460690 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): get_auth_request con 0x7f1eec0a0c70 auth_method 0 2024-09-17T15:38:16.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): get_auth_request con 0x7f1eec0a1590 auth_method 0 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef2a4e640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eec136160 0x7f1eec11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef2a4e640 10 monclient(hunting): get_auth_request con 0x7f1eec136160 auth_method 0 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef2a4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef2a4e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef2a4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient(hunting): handle_auth_done global_id 104315 payload 824 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eec136160 msgr2=0x7f1eec11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1eec136160 0x7f1eec11c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eec0a1590 msgr2=0x7f1eec135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eec0a1590 0x7f1eec135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient: _finish_hunting 0 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 monclient: found mon.noname-c 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef224d640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec11fd00 con 0x7f1eec0a0c70 2024-09-17T15:38:16.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.871+0000 7f1ef1a4c640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1eec0a1590 0x7f1eec135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef224d640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 msgr2=0x7f1eec13d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef224d640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ee801ccc0 tx=0x7f1ee801ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80bb020 con 0x7f1eec0a0c70 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: dump: 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.887 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee80b7420 con 0x7f1eec0a0c70 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 5 monclient: authenticate success, global_id 104315 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 monclient: _renew_subs 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec06fb90 con 0x7f1eec0a0c70 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80b1040 con 0x7f1eec0a0c70 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: dump: 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 monclient: _renew_subs 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:16.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef0a4a640 4 set_mon_vals no callback set 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec11fa20 con 0x7f1eec0a0c70 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 0x7f1eec11ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec11f6a0 con 0x7f1eec1358b0 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 0x7f1eec11ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient: get_auth_request con 0x7f1eec1358b0 auth_method 0 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1ee80be020 con 0x7f1eec0a0c70 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_done global_id 104318 payload 824 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1eec0a1050 con 0x7f1eec1358b0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ed003bbb0 0x7f1ed003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(366..366 src has 213..366) ==== 4771+0+0 (secure 0 0 0) 0x7f1ee80e9de0 con 0x7f1eec0a0c70 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef2a4e640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ed003bbb0 0x7f1ed003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef2a4e640 10 monclient: get_auth_request con 0x7f1ed003bbb0 auth_method 0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 msgr2=0x7f1eec11ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ef1a4c640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 0x7f1eec11ce00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1edc007fd0 tx=0x7f1edc00ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc018020 con 0x7f1eec1358b0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.875+0000 7f1ee2ffd640 10 monclient: dump: 2024-09-17T15:38:16.891 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1edc00dd60 con 0x7f1eec1358b0 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 msgr2=0x7f1eec11ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 0x7f1eec11ce00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1edc007fd0 tx=0x7f1edc00ff70 comp rx=0 tx=0).stop 2024-09-17T15:38:16.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 msgr2=0x7f1eec11ce00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:16.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ee2ffd640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec1358b0 0x7f1eec11ce00 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1edc007fd0 tx=0x7f1edc00ff70 comp rx=0 tx=0).stop 2024-09-17T15:38:16.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.879+0000 7f1ef2a4e640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ed003bbb0 0x7f1ed003e070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1eec071fe0 tx=0x7f1ee400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef44b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 0x7f1eec138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef44b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec11d750 con 0x7f1eec11d340 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 0x7f1eec138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient: get_auth_request con 0x7f1eec11d340 auth_method 0 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 10 monclient(hunting): handle_auth_done global_id 104321 payload 824 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.883+0000 7f1ef1a4c640 1 -- 172.21.15.86:0/2340460690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1eec1cede0 con 0x7f1eec11d340 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef1a4c640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 msgr2=0x7f1eec138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef1a4c640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 0x7f1eec138f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1edc019000 tx=0x7f1edc003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc018030 con 0x7f1eec11d340 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 10 monclient: dump: 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:16.896 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f1edc014b80 con 0x7f1eec11d340 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 msgr2=0x7f1eec138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 0x7f1eec138f00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1edc019000 tx=0x7f1edc003700 comp rx=0 tx=0).stop 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 msgr2=0x7f1eec138f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:16.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ee2ffd640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec11d340 0x7f1eec138f00 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1edc019000 tx=0x7f1edc003700 comp rx=0 tx=0).stop 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 127761, 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:16.898 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.899 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.900 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:16.901 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:16.902 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.904 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:16.905 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:16.907 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:16.908 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:16.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:16.910 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:16.911 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ed003bbb0 msgr2=0x7f1ed003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ed003bbb0 0x7f1ed003e070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1eec071fe0 tx=0x7f1ee400b040 comp rx=0 tx=0).stop 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 10 monclient: shutdown 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 msgr2=0x7f1eec13d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef2a4e640 1 -- 172.21.15.86:0/2340460690 reap_dead start 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1ee801ccc0 tx=0x7f1ee801ac60 comp rx=0 tx=0).stop 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 shutdown_connections 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 --2- 172.21.15.86:0/2340460690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1eec0a0c70 0x7f1eec13d680 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 >> 172.21.15.86:0/2340460690 conn(0x7f1eec0a2f40 msgr2=0x7f1eec072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 shutdown_connections 2024-09-17T15:38:16.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:16.887+0000 7f1ef44b1640 1 -- 172.21.15.86:0/2340460690 wait complete. 2024-09-17T15:38:16.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:16.914 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:38:16.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:38:17.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: get_monmap_and_config 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: build_initial_monmap 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: monmap: 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:17.028873+0000 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:17.028873+0000 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:17.026 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:17.027 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:17.027 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 Processor -- start 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 -- start start 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: init 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:38:17.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): picked mon.noname-a con 0x7f10400a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 0x7f1040135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): picked mon.noname-c con 0x7f10400a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 0x7f1040136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): picked mon.noname-b con 0x7f1040135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f10400a2410 con 0x7f10400a0c70 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f10400a2130 con 0x7f10400a1590 2024-09-17T15:38:17.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f10400a26f0 con 0x7f1040135f90 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f1045150640 10 monclient(hunting): authenticate will time out at 1629.994320s 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 0x7f1040136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59766/0 (socket says 172.21.15.86:59766) 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 1 -- 172.21.15.86:0/1777342788 learned_addr learned my addr 172.21.15.86:0/1777342788 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): get_auth_request con 0x7f10400a0c70 auth_method 0 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): get_auth_request con 0x7f1040135f90 auth_method 0 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103e59c640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 0x7f1040135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient(hunting): handle_auth_done global_id 104324 payload 274 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 -- 172.21.15.86:0/1777342788 >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 msgr2=0x7f1040135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 0x7f1040135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 -- 172.21.15.86:0/1777342788 >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 msgr2=0x7f10400a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient: _finish_hunting 0 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 monclient: found mon.noname-b 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103f59e640 1 -- 172.21.15.86:0/1777342788 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104009f9f0 con 0x7f1040135f90 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103e59c640 10 monclient: get_auth_request con 0x7f10400a1590 auth_method 0 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103e59c640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 0x7f1040135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:38:17.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.027+0000 7f103ed9d640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:17.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103f59e640 1 -- 172.21.15.86:0/1777342788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 msgr2=0x7f1040136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103f59e640 1 --2- 172.21.15.86:0/1777342788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 0x7f1040136420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1040074ae0 tx=0x7f103801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32c7e2a076c05b48 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 1 -- 172.21.15.86:0/1777342788 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380b2070 con 0x7f1040135f90 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: dump: 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.040 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 5 monclient: authenticate success, global_id 104324 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 1 -- 172.21.15.86:0/1777342788 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10380a5890 con 0x7f1040135f90 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 4 set_mon_vals no callback set 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 1 -- 172.21.15.86:0/1777342788 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10380a5c60 con 0x7f1040135f90 2024-09-17T15:38:17.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f103dd9b640 10 monclient: dump: 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 10 monclient: shutdown 2024-09-17T15:38:17.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.031+0000 7f1045150640 1 -- 172.21.15.86:0/1777342788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 msgr2=0x7f1040136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- 172.21.15.86:0/1777342788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 0x7f1040136420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1040074ae0 tx=0x7f103801c7f0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- 172.21.15.86:0/1777342788 shutdown_connections 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- 172.21.15.86:0/1777342788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1040135f90 0x7f1040136420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.203:3300/0 conn(0x7f10400a1590 0x7f1040135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- 172.21.15.86:0/1777342788 >> v2:172.21.15.86:3300/0 conn(0x7f10400a0c70 0x7f10400a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- 172.21.15.86:0/1777342788 >> 172.21.15.86:0/1777342788 conn(0x7f10400a2f40 msgr2=0x7f10400a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- 172.21.15.86:0/1777342788 shutdown_connections 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- 172.21.15.86:0/1777342788 wait complete. 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient: build_initial_monmap 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient: monmap: 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:17.043 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 Processor -- start 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- start start 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient: init 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 0x7f10401cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): picked mon.noname-a con 0x7f10400a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10400a1590 0x7f1040134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): picked mon.noname-c con 0x7f10400a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135f90 0x7f104011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): picked mon.noname-b con 0x7f1040135f90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f10400a00d0 con 0x7f10400a0c70 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f104009ff50 con 0x7f1040135f90 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f104013f740 con 0x7f10400a1590 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:17.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f1045150640 10 monclient(hunting): authenticate will time out at 1630.001969s 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 0x7f10401cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10400a1590 0x7f1040134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 0x7f10401cd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59774/0 (socket says 172.21.15.86:59774) 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 -- 172.21.15.86:0/3674221415 learned_addr learned my addr 172.21.15.86:0/3674221415 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): get_auth_request con 0x7f10400a0c70 auth_method 0 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): get_auth_request con 0x7f10400a1590 auth_method 0 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient(hunting): handle_auth_done global_id 124128 payload 824 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10400a1590 msgr2=0x7f1040134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103f59e640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135f90 0x7f104011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10400a1590 0x7f1040134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135f90 msgr2=0x7f104011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135f90 0x7f104011c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103ed9d640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104011fdf0 con 0x7f10400a0c70 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.035+0000 7f103e59c640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10400a1590 0x7f1040134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103ed9d640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 msgr2=0x7f10401cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103ed9d640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 0x7f10401cd9e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f103400e9d0 tx=0x7f103400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034017070 con 0x7f10400a0c70 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: dump: 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.048 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: _finish_auth 0 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: _check_auth_tickets 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 5 monclient: authenticate success, global_id 124128 2024-09-17T15:38:17.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103400cc70 con 0x7f10400a0c70 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 monclient: _renew_subs 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104009d460 con 0x7f10400a0c70 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034013670 con 0x7f10400a0c70 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 10 monclient: dump: 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.050 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103d59a640 4 set_mon_vals no callback set 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 monclient: _renew_subs 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1040120690 con 0x7f10400a0c70 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f103401a070 con 0x7f10400a0c70 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 0x7f104011ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1040120240 con 0x7f104011ee90 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f102c03bb60 0x7f102c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(366..366 src has 213..366) ==== 4771+0+0 (secure 0 0 0) 0x7f103404ff00 con 0x7f10400a0c70 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 0x7f104011ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient: get_auth_request con 0x7f104011ee90 auth_method 0 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103f59e640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f102c03bb60 0x7f102c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103f59e640 10 monclient: get_auth_request con 0x7f102c03bb60 auth_method 0 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 10 monclient(hunting): handle_auth_done global_id 114193 payload 824 2024-09-17T15:38:17.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.039+0000 7f103e59c640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1040135a50 con 0x7f104011ee90 2024-09-17T15:38:17.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f103f59e640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f102c03bb60 0x7f102c03e020 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f104006d160 tx=0x7f10380a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f103e59c640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 msgr2=0x7f104011ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f103e59c640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 0x7f104011ce00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f103000fbd0 tx=0x7f103000d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030013640 con 0x7f104011ee90 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 10 monclient: dump: 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.056 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1030013640 con 0x7f104011ee90 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 msgr2=0x7f104011ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 0x7f104011ce00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f103000fbd0 tx=0x7f103000d590 comp rx=0 tx=0).stop 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 msgr2=0x7f104011ce00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.043+0000 7f10277fe640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f104011ee90 0x7f104011ce00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f1045150640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f1045150640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 0x7f104011d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f1045150640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f104006d4f0 con 0x7f1040135230 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 0x7f104011d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient: get_auth_request con 0x7f1040135230 auth_method 0 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 10 monclient(hunting): handle_auth_done global_id 114196 payload 824 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.051+0000 7f103e59c640 1 -- 172.21.15.86:0/3674221415 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10401d8d30 con 0x7f1040135230 2024-09-17T15:38:17.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f103e59c640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 msgr2=0x7f104011d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f103e59c640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 0x7f104011d340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f104006d040 tx=0x7f10300046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103001bcd0 con 0x7f1040135230 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 10 monclient: dump: 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.059 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f1030000e50 con 0x7f1040135230 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 msgr2=0x7f104011d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 0x7f104011d340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f104006d040 tx=0x7f10300046d0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 msgr2=0x7f104011d340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f10277fe640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1040135230 0x7f104011d340 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f104006d040 tx=0x7f10300046d0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:17.060 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 79551, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:17.061 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.062 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:17.063 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:17.064 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:17.065 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:17.066 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.068 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:17.069 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:17.072 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:17.073 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:17.074 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:17.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f102c03bb60 msgr2=0x7f102c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f102c03bb60 0x7f102c03e020 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f104006d160 tx=0x7f10380a7070 comp rx=0 tx=0).stop 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 10 monclient: shutdown 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 msgr2=0x7f10401cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 --2- 172.21.15.86:0/3674221415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10400a0c70 0x7f10401cd9e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f103400e9d0 tx=0x7f103400ee90 comp rx=0 tx=0).stop 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f103f59e640 1 -- 172.21.15.86:0/3674221415 reap_dead start 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f103f59e640 1 -- 172.21.15.86:0/3674221415 reap_dead start 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 shutdown_connections 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 >> 172.21.15.86:0/3674221415 conn(0x7f10400a2f40 msgr2=0x7f10400a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 shutdown_connections 2024-09-17T15:38:17.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.055+0000 7f1045150640 1 -- 172.21.15.86:0/3674221415 wait complete. 2024-09-17T15:38:17.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:17.077 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:38:17.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:38:17.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: get_monmap_and_config 2024-09-17T15:38:17.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: build_initial_monmap 2024-09-17T15:38:17.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: monmap: 2024-09-17T15:38:17.191 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:17.194336+0000 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:17.194336+0000 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 Processor -- start 2024-09-17T15:38:17.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 -- start start 2024-09-17T15:38:17.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: init 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6d00689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 0x7fe6d006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d00694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe6d013aff0 0x7fe6d013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d0074470 con 0x7fe6d00689d0 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe6d009c8b0 con 0x7fe6d013aff0 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d009cb90 con 0x7fe6d00694c0 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d844f640 10 monclient(hunting): authenticate will time out at 1630.159809s 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d61eb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe6d00689d0 0x7fe6d0068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42350/0 (socket says 172.21.15.86:42350) 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.191+0000 7fe6d61eb640 1 -- 172.21.15.86:0/3597676485 learned_addr learned my addr 172.21.15.86:0/3597676485 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 --2- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 0x7fe6d006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): get_auth_request con 0x7fe6d00694c0 auth_method 0 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d69ec640 1 --2- 172.21.15.86:0/3597676485 >> v2:172.21.15.203:3300/0 conn(0x7fe6d013aff0 0x7fe6d013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d69ec640 10 monclient(hunting): get_auth_request con 0x7fe6d013aff0 auth_method 0 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d69ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d69ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d69ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient(hunting): handle_auth_done global_id 104330 payload 274 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 -- 172.21.15.86:0/3597676485 >> v2:172.21.15.203:3300/0 conn(0x7fe6d013aff0 msgr2=0x7fe6d013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 --2- 172.21.15.86:0/3597676485 >> v2:172.21.15.203:3300/0 conn(0x7fe6d013aff0 0x7fe6d013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 -- 172.21.15.86:0/3597676485 >> v1:172.21.15.86:6789/0 conn(0x7fe6d00689d0 legacy=0x7fe6d0068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient: _finish_hunting 0 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 monclient: found mon.noname-b 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.195+0000 7fe6d59ea640 1 -- 172.21.15.86:0/3597676485 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d009ce70 con 0x7fe6d00694c0 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 1 -- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 msgr2=0x7fe6d006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 1 --2- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 0x7fe6d006f840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0097c0 tx=0x7fe6cc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6257304b0b1446a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 1 -- 172.21.15.86:0/3597676485 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc0b0070 con 0x7fe6d00694c0 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: dump: 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.201 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: _finish_auth 0 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: _check_auth_tickets 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 1 -- 172.21.15.86:0/3597676485 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6cc0ab600 con 0x7fe6d00694c0 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 5 monclient: authenticate success, global_id 104330 2024-09-17T15:38:17.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 1 -- 172.21.15.86:0/3597676485 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d009c2d0 con 0x7fe6d00694c0 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 4 set_mon_vals no callback set 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d51e9640 10 monclient: dump: 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.203 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: shutdown 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 msgr2=0x7fe6d006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 0x7fe6d006f840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc0097c0 tx=0x7fe6cc01c360 comp rx=0 tx=0).stop 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- 172.21.15.86:0/3597676485 shutdown_connections 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- 172.21.15.86:0/3597676485 >> v2:172.21.15.203:3300/0 conn(0x7fe6d013aff0 0x7fe6d013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- 172.21.15.86:0/3597676485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d00694c0 0x7fe6d006f840 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- 172.21.15.86:0/3597676485 >> 172.21.15.86:0/3597676485 conn(0x7fe6d00a6440 msgr2=0x7fe6d00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- 172.21.15.86:0/3597676485 shutdown_connections 2024-09-17T15:38:17.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- 172.21.15.86:0/3597676485 wait complete. 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: build_initial_monmap 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: monmap: 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:17.205 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 Processor -- start 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- start start 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: init 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-a con 0x7fe6d00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6d00694c0 0x7fe6d0135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d00694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0140610 con 0x7fe6d00689d0 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0140490 con 0x7fe6d00694c0 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d0140790 con 0x7fe6d013aff0 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d844f640 10 monclient(hunting): authenticate will time out at 1630.167091s 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36948/0 (socket says 172.21.15.86:36948) 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 -- 172.21.15.86:0/2294130359 learned_addr learned my addr 172.21.15.86:0/2294130359 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): get_auth_request con 0x7fe6d013aff0 auth_method 0 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 10 monclient(hunting): get_auth_request con 0x7fe6d00689d0 auth_method 0 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 msgr2=0x7fe6d01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 msgr2=0x7fe6d01cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6d00694c0 0x7fe6d0135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d61eb640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 10 monclient(hunting): get_auth_request con 0x7fe6d00694c0 auth_method 0 2024-09-17T15:38:17.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d59ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient(hunting): handle_auth_done global_id 104333 payload 824 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6d00694c0 msgr2=0x7fe6d0135bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6d00694c0 0x7fe6d0135bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 msgr2=0x7fe6d01cfb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient: _finish_hunting 0 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 monclient: found mon.noname-c 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.199+0000 7fe6d69ec640 1 -- 172.21.15.86:0/2294130359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d009eed0 con 0x7fe6d013aff0 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d69ec640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 msgr2=0x7fe6d01360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d69ec640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007e70 tx=0x7fe6c800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8013790 con 0x7fe6d013aff0 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: dump: 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.213 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6c8019070 con 0x7fe6d013aff0 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:17.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8013b90 con 0x7fe6d013aff0 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 10 monclient: dump: 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:17.215 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 5 monclient: authenticate success, global_id 104333 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _renew_subs 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d00a3420 con 0x7fe6d013aff0 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d49e8640 4 set_mon_vals no callback set 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _renew_subs 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d009f5e0 con 0x7fe6d013aff0 2024-09-17T15:38:17.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe6c801e460 con 0x7fe6d013aff0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d006f840 con 0x7fe6d013aff0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 0x7fe6b403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(366..366 src has 213..366) ==== 4771+0+0 (secure 0 0 0) 0x7fe6c8050850 con 0x7fe6d013aff0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d61eb640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 0x7fe6b403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.203+0000 7fe6d61eb640 10 monclient: get_auth_request con 0x7fe6b403bbb0 auth_method 0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.207+0000 7fe6d61eb640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 0x7fe6b403e070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0007d50 tx=0x7fe6c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.207+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe6d00a3420 con 0x7fe6d013aff0 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.207+0000 7fe6c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:17.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.207+0000 7fe6c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:17.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.431+0000 7fe6d844f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:38:17.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.431+0000 7fe6d844f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:38:17.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.431+0000 7fe6d844f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:17.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.431+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe6d0121de0 con 0x7fe6d013aff0 2024-09-17T15:38:17.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fe6c6ffd640 1 -- 172.21.15.86:0/2294130359 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe6c804f250 con 0x7fe6d013aff0 2024-09-17T15:38:17.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fe6c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:38:17.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fe6c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:17.434 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=2781963862,config=3541143443,health=298042371,kv=3591778543,logm=2756021949}) 2024-09-17T15:38:17.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=2378722407,config=3541143443,health=3399836568,kv=215610202,logm=1356791170}) 2024-09-17T15:38:17.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=20,config=2,health=11,kv=11,logm=56} crc {auth=2781963862,config=3541143443,health=298042371,kv=3591778543,logm=2756021949}) 2024-09-17T15:38:17.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.435+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=11,kv=10,logm=59} crc {auth=4178586715,config=3541143443,health=3399836568,kv=1452539945,logm=494958551}) 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 msgr2=0x7fe6b403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 0x7fe6b403e070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0007d50 tx=0x7fe6c0009040 comp rx=0 tx=0).stop 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 10 monclient: shutdown 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 msgr2=0x7fe6d01360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007e70 tx=0x7fe6c800e4b0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 shutdown_connections 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6b403bbb0 0x7fe6b403e070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6d013aff0 0x7fe6d01360f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6d00694c0 0x7fe6d0135bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 --2- 172.21.15.86:0/2294130359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe6d00689d0 0x7fe6d01cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 >> 172.21.15.86:0/2294130359 conn(0x7fe6d00a6440 msgr2=0x7fe6d00a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 shutdown_connections 2024-09-17T15:38:17.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:17.439+0000 7fe6d844f640 1 -- 172.21.15.86:0/2294130359 wait complete. 2024-09-17T15:38:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:1625: done (2 left) 2024-09-17T15:38:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:1626: done (1 left) 2024-09-17T15:38:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:1627: done (0 left) 2024-09-17T15:38:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:1628: write oid 22 current snap is 178 2024-09-17T15:38:17.441 INFO:tasks.rados.rados.0.smithi203.stdout:1628: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-17T15:38:17.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:38:17.460 INFO:tasks.rados.rados.0.smithi203.stdout:1628: writing smithi20328086-22 from 416972 to 1140910 tid 1 2024-09-17T15:38:17.474 INFO:tasks.rados.rados.0.smithi203.stdout:1628: writing smithi20328086-22 from 1743268 to 2016972 tid 2 2024-09-17T15:38:17.474 INFO:tasks.rados.rados.0.smithi203.stdout:1629: snap_remove snap 167 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329790294}) 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143497620}) 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329790294}) 2024-09-17T15:38:17.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.603+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804509362}) 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305916107}) 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558421329}) 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305916107}) 2024-09-17T15:38:17.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.627+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074363293}) 2024-09-17T15:38:17.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918860126}) 2024-09-17T15:38:17.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3062106666}) 2024-09-17T15:38:17.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918860126}) 2024-09-17T15:38:17.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2679949774}) 2024-09-17T15:38:17.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.631+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706679570}) 2024-09-17T15:38:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1260022940}) 2024-09-17T15:38:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706679570}) 2024-09-17T15:38:17.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691100638}) 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821163620}) 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44905717}) 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821163620}) 2024-09-17T15:38:17.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.635+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869081999}) 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571832810}) 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775717548}) 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571832810}) 2024-09-17T15:38:17.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187564624}) 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754814370}) 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4181112814}) 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754814370}) 2024-09-17T15:38:17.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.639+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3451173191}) 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219039254}) 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1382201201}) 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219039254}) 2024-09-17T15:38:17.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968876179}) 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517924790}) 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488050929}) 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517924790}) 2024-09-17T15:38:17.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.643+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4268203915}) 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696127703}) 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215099408}) 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1696127703}) 2024-09-17T15:38:17.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161669362}) 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345633782}) 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127528795}) 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345633782}) 2024-09-17T15:38:17.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.647+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=319347212}) 2024-09-17T15:38:17.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795827504}) 2024-09-17T15:38:17.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818008612}) 2024-09-17T15:38:17.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1795827504}) 2024-09-17T15:38:17.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872877536}) 2024-09-17T15:38:17.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178215226}) 2024-09-17T15:38:17.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056229495}) 2024-09-17T15:38:17.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178215226}) 2024-09-17T15:38:17.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.651+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645018587}) 2024-09-17T15:38:17.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848673121}) 2024-09-17T15:38:17.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37121131}) 2024-09-17T15:38:17.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848673121}) 2024-09-17T15:38:17.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333318408}) 2024-09-17T15:38:17.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243440367}) 2024-09-17T15:38:17.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388648066}) 2024-09-17T15:38:17.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243440367}) 2024-09-17T15:38:17.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.655+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401905061}) 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686944341}) 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872301820}) 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686944341}) 2024-09-17T15:38:17.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.659+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042482290}) 2024-09-17T15:38:17.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1116374805}) 2024-09-17T15:38:17.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536479987}) 2024-09-17T15:38:17.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1116374805}) 2024-09-17T15:38:17.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162580746}) 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2017167814}) 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.663+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136348984}) 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2017167814}) 2024-09-17T15:38:17.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3000264495}) 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037669852}) 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47400853}) 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037669852}) 2024-09-17T15:38:17.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.667+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101267685}) 2024-09-17T15:38:17.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180901919}) 2024-09-17T15:38:17.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3178478747}) 2024-09-17T15:38:17.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1180901919}) 2024-09-17T15:38:17.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695268318}) 2024-09-17T15:38:17.671 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.671 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664744755}) 2024-09-17T15:38:17.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40589409}) 2024-09-17T15:38:17.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1664744755}) 2024-09-17T15:38:17.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.671+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330203655}) 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876641528}) 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763881289}) 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876641528}) 2024-09-17T15:38:17.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=298853341}) 2024-09-17T15:38:17.676 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.676 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378570734}) 2024-09-17T15:38:17.676 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805941167}) 2024-09-17T15:38:17.676 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.676 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378570734}) 2024-09-17T15:38:17.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.675+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2128495329}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568433179}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635744501}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568433179}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.679+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297400461}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474435459}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1166338955}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=474435459}) 2024-09-17T15:38:17.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405527229}) 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097830829}) 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311082407}) 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097830829}) 2024-09-17T15:38:17.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.683+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4278321654}) 2024-09-17T15:38:17.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083437217}) 2024-09-17T15:38:17.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896302164}) 2024-09-17T15:38:17.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083437217}) 2024-09-17T15:38:17.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939295241}) 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089758420}) 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3358496190}) 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089758420}) 2024-09-17T15:38:17.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.687+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326250367}) 2024-09-17T15:38:17.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=324164148}) 2024-09-17T15:38:17.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874286487}) 2024-09-17T15:38:17.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=324164148}) 2024-09-17T15:38:17.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.691+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958235613}) 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=1} crc {logm=592003247,mdsmap=2815901553,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1748228865,mon_config_key=639531838,monmap=1314516931}) 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14,monmap=3,nvmeofgw=1} crc {logm=1702223331,mdsmap=2815901553,mgr=2332442456,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=772887663,mon_config_key=639531838,monmap=4191199434,nvmeofgw=773687688}) 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=61,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=1} crc {logm=592003247,mdsmap=2815901553,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1748228865,mon_config_key=639531838,monmap=1314516931}) 2024-09-17T15:38:17.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14} crc {logm=3658336872,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3136047647,mon_config_key=639531838}) 2024-09-17T15:38:17.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=64,osdmap=24} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=294973011,osd_snap=3758000048,osdmap=1153520439}) 2024-09-17T15:38:17.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=65,osdmap=23} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=294973011,osd_snap=3418838095,osdmap=178304396}) 2024-09-17T15:38:17.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=64,osdmap=24} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=294973011,osd_snap=3758000048,osdmap=1153520439}) 2024-09-17T15:38:17.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.695+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=64,osdmap=24} crc {monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=294973011,osd_snap=1128867338,osdmap=4235828793}) 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=822223981}) 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=981602445}) 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=822223981}) 2024-09-17T15:38:17.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.699+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2778873056}) 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1517228766}) 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3660096975}) 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1517228766}) 2024-09-17T15:38:17.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:38:17.703+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=779540465}) 2024-09-17T15:38:18.453 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:38:18.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1629: done (1 left) 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1630: snap_create 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: finishing write tid 1 to smithi20328086-22 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: finishing write tid 2 to smithi20328086-22 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: finishing write tid 3 to smithi20328086-22 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: finishing write tid 4 to smithi20328086-22 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: oid 22 updating version 0 to 505 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: oid 22 updating version 505 to 507 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: oid 22 version 507 is already newer than 506 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 507 (ObjNum 596 snap 178 seq_num 596) dirty exists 2024-09-17T15:38:18.466 INFO:tasks.rados.rados.0.smithi203.stdout:1628: left oid 22 (ObjNum 596 snap 178 seq_num 596) 2024-09-17T15:38:18.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: get_monmap_and_config 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: build_initial_monmap 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: monmap: 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:18.578026+0000 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:18.578026+0000 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:18.575 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:18.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 Processor -- start 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 -- start start 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: init 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9cd00689d0 0x7f9cd009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cd0068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd0139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cd009d770 con 0x7f9cd0068db0 2024-09-17T15:38:18.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9cd009ed00 con 0x7f9cd00689d0 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd009efe0 con 0x7f9cd0139f00 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd795e640 10 monclient(hunting): authenticate will time out at 1631.543612s 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48462/0 (socket says 172.21.15.86:48462) 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 1 -- 172.21.15.86:0/562884660 learned_addr learned my addr 172.21.15.86:0/562884660 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): get_auth_request con 0x7f9cd0139f00 auth_method 0 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.575+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd56fa640 1 --2- 172.21.15.86:0/562884660 >> v2:172.21.15.203:3300/0 conn(0x7f9cd00689d0 0x7f9cd009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd56fa640 10 monclient(hunting): get_auth_request con 0x7f9cd00689d0 auth_method 0 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd56fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd56fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd56fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_done global_id 104336 payload 274 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 1 -- 172.21.15.86:0/562884660 >> v2:172.21.15.203:3300/0 conn(0x7f9cd00689d0 msgr2=0x7f9cd009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 1 --2- 172.21.15.86:0/562884660 >> v2:172.21.15.203:3300/0 conn(0x7f9cd00689d0 0x7f9cd009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 1 -- 172.21.15.86:0/562884660 >> v1:172.21.15.203:6789/0 conn(0x7f9cd0068db0 legacy=0x7f9cd00691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 10 monclient: _finish_hunting 0 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 1 monclient: found mon.noname-b 2024-09-17T15:38:18.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:18.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:18.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.579+0000 7f9cd5efb640 1 -- 172.21.15.86:0/562884660 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd00a04e0 con 0x7f9cd0139f00 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 -- 172.21.15.86:0/562884660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 msgr2=0x7f9cd013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 --2- 172.21.15.86:0/562884660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00a0770 tx=0x7f9cc401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12682e7a21e631cc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 1 -- 172.21.15.86:0/562884660 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b0070 con 0x7f9cd0139f00 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: dump: 2024-09-17T15:38:18.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: _finish_auth 0 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 1 -- 172.21.15.86:0/562884660 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc40ab600 con 0x7f9cd0139f00 2024-09-17T15:38:18.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 1 -- 172.21.15.86:0/562884660 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40ab9d0 con 0x7f9cd0139f00 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 5 monclient: authenticate success, global_id 104336 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 4 set_mon_vals no callback set 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cc3fff640 10 monclient: dump: 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.584 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: shutdown 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- 172.21.15.86:0/562884660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 msgr2=0x7f9cd013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- 172.21.15.86:0/562884660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00a0770 tx=0x7f9cc401c490 comp rx=0 tx=0).stop 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- 172.21.15.86:0/562884660 shutdown_connections 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- 172.21.15.86:0/562884660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0139f00 0x7f9cd013c2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- 172.21.15.86:0/562884660 >> v2:172.21.15.203:3300/0 conn(0x7f9cd00689d0 0x7f9cd009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- 172.21.15.86:0/562884660 >> 172.21.15.86:0/562884660 conn(0x7f9cd00a6440 msgr2=0x7f9cd00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- 172.21.15.86:0/562884660 shutdown_connections 2024-09-17T15:38:18.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- 172.21.15.86:0/562884660 wait complete. 2024-09-17T15:38:18.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: build_initial_monmap 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: monmap: 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 Processor -- start 2024-09-17T15:38:18.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- start start 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: init 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cd00689d0 0x7f9cd0134a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cd00689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:18.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cd0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cd0139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd013f490 con 0x7f9cd0139f00 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd013f310 con 0x7f9cd00689d0 2024-09-17T15:38:18.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9cd013f610 con 0x7f9cd0068db0 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd795e640 10 monclient(hunting): authenticate will time out at 1631.551173s 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58044/0 (socket says 172.21.15.86:58044) 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48470/0 (socket says 172.21.15.86:48470) 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 -- 172.21.15.86:0/739399667 learned_addr learned my addr 172.21.15.86:0/739399667 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): get_auth_request con 0x7f9cd0139f00 auth_method 0 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): get_auth_request con 0x7f9cd0068db0 auth_method 0 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient(hunting): handle_auth_done global_id 124134 payload 824 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 -- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 msgr2=0x7f9cd0134f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 -- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cd00689d0 msgr2=0x7f9cd0134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cd00689d0 0x7f9cd0134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient: _finish_hunting 0 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 monclient: found mon.noname-a 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd5efb640 1 -- 172.21.15.86:0/739399667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd0122650 con 0x7f9cd0139f00 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.583+0000 7f9cd4ef9640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd5efb640 1 -- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 msgr2=0x7f9cd00a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd5efb640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40ae040 tx=0x7f9cc4019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b0040 con 0x7f9cd0139f00 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: dump: 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc40b9070 con 0x7f9cd0139f00 2024-09-17T15:38:18.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc40b37b0 con 0x7f9cd0139f00 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 5 monclient: authenticate success, global_id 124134 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 10 monclient: dump: 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.597 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc37fe640 4 set_mon_vals no callback set 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _renew_subs 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd00a3610 con 0x7f9cd0139f00 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _renew_subs 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd0122370 con 0x7f9cd0139f00 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cd0135940 con 0x7f9cd0139f00 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9cc40b3950 con 0x7f9cd0139f00 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 0x7f9cb403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(368..368 src has 213..368) ==== 4783+0+0 (secure 0 0 0) 0x7f9cc40e9c70 con 0x7f9cd0139f00 2024-09-17T15:38:18.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd56fa640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 0x7f9cb403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.587+0000 7f9cd56fa640 10 monclient: get_auth_request con 0x7f9cb403bbe0 auth_method 0 2024-09-17T15:38:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.591+0000 7f9cd56fa640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 0x7f9cb403e0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098f0 tx=0x7f9ccc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.591+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9cd0135940 con 0x7f9cd0139f00 2024-09-17T15:38:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.591+0000 7f9cc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:18.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.591+0000 7f9cc1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:18.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cd795e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:18.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cd795e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:18.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cd795e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:18.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9cd006d4f0 con 0x7f9cd0139f00 2024-09-17T15:38:18.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cc1ffb640 1 -- 172.21.15.86:0/739399667 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9cc4016110 con 0x7f9cd0139f00 2024-09-17T15:38:18.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:18.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.807+0000 7f9cc1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:18.808 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:18.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 msgr2=0x7f9cb403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 0x7f9cb403e0a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc0098f0 tx=0x7f9ccc008040 comp rx=0 tx=0).stop 2024-09-17T15:38:18.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 10 monclient: shutdown 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 msgr2=0x7f9cd00a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40ae040 tx=0x7f9cc4019230 comp rx=0 tx=0).stop 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 shutdown_connections 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9cb403bbe0 0x7f9cb403e0a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cd0139f00 0x7f9cd00a0d00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cd0068db0 0x7f9cd0134f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 --2- 172.21.15.86:0/739399667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cd00689d0 0x7f9cd0134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 >> 172.21.15.86:0/739399667 conn(0x7f9cd00a6440 msgr2=0x7f9cd0138060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 shutdown_connections 2024-09-17T15:38:18.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.811+0000 7f9cd795e640 1 -- 172.21.15.86:0/739399667 wait complete. 2024-09-17T15:38:18.825 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:38:18.825 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:38:18.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:38:18.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:38:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: get_monmap_and_config 2024-09-17T15:38:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: build_initial_monmap 2024-09-17T15:38:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: monmap: 2024-09-17T15:38:18.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:18.944753+0000 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:18.944753+0000 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:18.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 Processor -- start 2024-09-17T15:38:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 -- start start 2024-09-17T15:38:18.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: init 2024-09-17T15:38:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-17T15:38:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b98073fe0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:18.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b98074900 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2b980a07b0 0x7f2b980a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b980a07b0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b980704f0 con 0x7f2b98074900 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2b980707d0 con 0x7f2b980a07b0 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2b98070ab0 con 0x7f2b98073fe0 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b9dcb3640 10 monclient(hunting): authenticate will time out at 1631.909925s 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48474/0 (socket says 172.21.15.86:48474) 2024-09-17T15:38:18.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 -- 172.21.15.86:0/3267137773 learned_addr learned my addr 172.21.15.86:0/3267137773 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): get_auth_request con 0x7f2b98073fe0 auth_method 0 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b97fff640 1 --2- 172.21.15.86:0/3267137773 >> v2:172.21.15.203:3300/0 conn(0x7f2b980a07b0 0x7f2b980a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b97fff640 10 monclient(hunting): get_auth_request con 0x7f2b980a07b0 auth_method 0 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient(hunting): handle_auth_done global_id 104342 payload 274 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 -- 172.21.15.86:0/3267137773 >> v2:172.21.15.203:3300/0 conn(0x7f2b980a07b0 msgr2=0x7f2b980a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 --2- 172.21.15.86:0/3267137773 >> v2:172.21.15.203:3300/0 conn(0x7f2b980a07b0 0x7f2b980a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 -- 172.21.15.86:0/3267137773 >> v1:172.21.15.203:6789/0 conn(0x7f2b98074900 legacy=0x7f2b980a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient: _finish_hunting 0 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 monclient: found mon.noname-b 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.943+0000 7f2b977fe640 1 -- 172.21.15.86:0/3267137773 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9809fe80 con 0x7f2b98073fe0 2024-09-17T15:38:18.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b977fe640 1 -- 172.21.15.86:0/3267137773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 msgr2=0x7f2b980743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b977fe640 1 --2- 172.21.15.86:0/3267137773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009e50 tx=0x7f2b8801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3447ed45eb73298 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 1 -- 172.21.15.86:0/3267137773 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b880a5530 con 0x7f2b98073fe0 2024-09-17T15:38:18.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: dump: 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:18.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 5 monclient: authenticate success, global_id 104342 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 1 -- 172.21.15.86:0/3267137773 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b880a56d0 con 0x7f2b98073fe0 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 1 -- 172.21.15.86:0/3267137773 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b880a5aa0 con 0x7f2b98073fe0 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 4 set_mon_vals no callback set 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b967fc640 10 monclient: dump: 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.954 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 10 monclient: shutdown 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/3267137773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 msgr2=0x7f2b980743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/3267137773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2b88009e50 tx=0x7f2b8801c8f0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/3267137773 shutdown_connections 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/3267137773 >> v2:172.21.15.203:3300/0 conn(0x7f2b980a07b0 0x7f2b980a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/3267137773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b980743c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/3267137773 >> 172.21.15.86:0/3267137773 conn(0x7f2b980a3630 msgr2=0x7f2b980a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/3267137773 shutdown_connections 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.947+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/3267137773 wait complete. 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient: build_initial_monmap 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient: monmap: 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:18.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 Processor -- start 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 -- start start 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient: init 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b9813f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2b98073fe0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b98074900 0x7f2b981cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2b98074900 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 0x7f2b981d0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f2b980a07b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2b981426d0 con 0x7f2b980a07b0 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98138260 con 0x7f2b98074900 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2b98142850 con 0x7f2b98073fe0 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b9dcb3640 10 monclient(hunting): authenticate will time out at 1631.917446s 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 0x7f2b981d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b9813f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b9813f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48476/0 (socket says 172.21.15.86:48476) 2024-09-17T15:38:18.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 0x7f2b981d0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58058/0 (socket says 172.21.15.86:58058) 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 -- 172.21.15.86:0/620941677 learned_addr learned my addr 172.21.15.86:0/620941677 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): get_auth_request con 0x7f2b98073fe0 auth_method 0 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): get_auth_request con 0x7f2b980a07b0 auth_method 0 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b96ffd640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b98074900 0x7f2b981cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b96ffd640 10 monclient(hunting): get_auth_request con 0x7f2b98074900 auth_method 0 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient(hunting): handle_auth_done global_id 104345 payload 824 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b98074900 msgr2=0x7f2b981cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b98074900 0x7f2b981cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 msgr2=0x7f2b981d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 0x7f2b981d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient: _finish_hunting 0 2024-09-17T15:38:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 monclient: found mon.noname-c 2024-09-17T15:38:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b977fe640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b98139110 con 0x7f2b98073fe0 2024-09-17T15:38:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.951+0000 7f2b97fff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b980a07b0 0x7f2b981d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b977fe640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 msgr2=0x7f2b9813f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b977fe640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b9813f2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b8801d190 tx=0x7f2b88019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b880bc020 con 0x7f2b98073fe0 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: dump: 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: _finish_auth 0 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b880b8880 con 0x7f2b98073fe0 2024-09-17T15:38:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 5 monclient: authenticate success, global_id 104345 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 monclient: _renew_subs 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b98135280 con 0x7f2b98073fe0 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b880a5530 con 0x7f2b98073fe0 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: dump: 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 monclient: _renew_subs 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b95ffb640 4 set_mon_vals no callback set 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9813a830 con 0x7f2b98073fe0 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 0x7f2b9802e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2b980743c0 con 0x7f2b9813f830 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 0x7f2b9802e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient: get_auth_request con 0x7f2b9813f830 auth_method 0 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2b880bf020 con 0x7f2b98073fe0 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_done global_id 124140 payload 824 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b98130bc0 con 0x7f2b9813f830 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2b7803bb90 0x7f2b7803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(368..368 src has 213..368) ==== 4783+0+0 (secure 0 0 0) 0x7f2b880eab60 con 0x7f2b98073fe0 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b97fff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2b7803bb90 0x7f2b7803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b97fff640 10 monclient: get_auth_request con 0x7f2b7803bb90 auth_method 0 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 msgr2=0x7f2b9802e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b96ffd640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 0x7f2b9802e5d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b8000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80007450 con 0x7f2b9813f830 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.955+0000 7f2b8ffff640 10 monclient: dump: 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.970 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b80007450 con 0x7f2b9813f830 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 msgr2=0x7f2b9802e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 0x7f2b9802e5d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b8000e990 comp rx=0 tx=0).stop 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 msgr2=0x7f2b9802e5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b8ffff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b9813f830 0x7f2b9802e5d0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b8000e990 comp rx=0 tx=0).stop 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.959+0000 7f2b97fff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2b7803bb90 0x7f2b7803e050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2b9813b200 tx=0x7f2b90008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b9dcb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 0x7f2b9813b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b9dcb3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2b981d0a70 con 0x7f2b981d0660 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 0x7f2b9813b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient: get_auth_request con 0x7f2b981d0660 auth_method 0 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 10 monclient(hunting): handle_auth_done global_id 124143 payload 824 2024-09-17T15:38:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.963+0000 7f2b96ffd640 1 -- 172.21.15.86:0/620941677 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b98137320 con 0x7f2b981d0660 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b96ffd640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 msgr2=0x7f2b9813b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b96ffd640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 0x7f2b9813b2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b80003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8001e960 con 0x7f2b981d0660 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 10 monclient: dump: 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f2b80028460 con 0x7f2b981d0660 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 msgr2=0x7f2b9813b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 0x7f2b9813b2d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b80003770 comp rx=0 tx=0).stop 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 msgr2=0x7f2b9813b2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.967+0000 7f2b8ffff640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2b981d0660 0x7f2b9813b2d0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2b8000e4d0 tx=0x7f2b80003770 comp rx=0 tx=0).stop 2024-09-17T15:38:18.975 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 29831, 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:18.976 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:18.977 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:18.978 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:18.979 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:18.980 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:18.981 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:18.982 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:18.983 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.984 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:18.985 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:18.986 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.987 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.988 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:18.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:18.990 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:18.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2b7803bb90 msgr2=0x7f2b7803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2b7803bb90 0x7f2b7803e050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2b9813b200 tx=0x7f2b90008040 comp rx=0 tx=0).stop 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 10 monclient: shutdown 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 msgr2=0x7f2b9813f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 --2- 172.21.15.86:0/620941677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2b98073fe0 0x7f2b9813f2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2b8801d190 tx=0x7f2b88019690 comp rx=0 tx=0).stop 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b97fff640 1 -- 172.21.15.86:0/620941677 reap_dead start 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b97fff640 1 -- 172.21.15.86:0/620941677 reap_dead start 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 shutdown_connections 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 >> 172.21.15.86:0/620941677 conn(0x7f2b980a3630 msgr2=0x7f2b98073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 shutdown_connections 2024-09-17T15:38:18.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:18.971+0000 7f2b9dcb3640 1 -- 172.21.15.86:0/620941677 wait complete. 2024-09-17T15:38:18.993 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:18.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:38:18.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:38:19.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: get_monmap_and_config 2024-09-17T15:38:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: monmap: 2024-09-17T15:38:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:19.112831+0000 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:19.112831+0000 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:19.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:19.112 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:19.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 Processor -- start 2024-09-17T15:38:19.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 -- start start 2024-09-17T15:38:19.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: init 2024-09-17T15:38:19.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T15:38:19.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff250134d90 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:19.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): picked mon.noname-b con 0x7ff2500a1130 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:19.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): picked mon.noname-a con 0x7ff25013d4a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:19.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff250135880 con 0x7ff250134d90 2024-09-17T15:38:19.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff2500a21b0 con 0x7ff25013d4a0 2024-09-17T15:38:19.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff250073b10 con 0x7ff2500a1130 2024-09-17T15:38:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff258209640 10 monclient(hunting): authenticate will time out at 1632.078311s 2024-09-17T15:38:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58082/0 (socket says 172.21.15.86:58082) 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48482/0 (socket says 172.21.15.86:48482) 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 1 -- 172.21.15.86:0/3923659607 learned_addr learned my addr 172.21.15.86:0/3923659607 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): get_auth_request con 0x7ff25013d4a0 auth_method 0 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): get_auth_request con 0x7ff2500a1130 auth_method 0 2024-09-17T15:38:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient(hunting): handle_auth_done global_id 104348 payload 274 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 -- 172.21.15.86:0/3923659607 >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 msgr2=0x7ff25013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 --2- 172.21.15.86:0/3923659607 >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 -- 172.21.15.86:0/3923659607 >> v1:172.21.15.203:6789/0 conn(0x7ff250134d90 legacy=0x7ff250135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:19.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 monclient: found mon.noname-b 2024-09-17T15:38:19.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2557a4640 1 -- 172.21.15.86:0/3923659607 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2500a1510 con 0x7ff2500a1130 2024-09-17T15:38:19.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.111+0000 7ff2567a6640 1 --2- 172.21.15.86:0/3923659607 >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:19.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff2557a4640 1 -- 172.21.15.86:0/3923659607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 msgr2=0x7ff25013cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff2557a4640 1 --2- 172.21.15.86:0/3923659607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009d90 tx=0x7ff24c01c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40d8ec75bfed695 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 1 -- 172.21.15.86:0/3923659607 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0b2070 con 0x7ff2500a1130 2024-09-17T15:38:19.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: dump: 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.143 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.143 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.143 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.143 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.143 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 1 -- 172.21.15.86:0/3923659607 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff24c0a5840 con 0x7ff2500a1130 2024-09-17T15:38:19.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 5 monclient: authenticate success, global_id 104348 2024-09-17T15:38:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 1 -- 172.21.15.86:0/3923659607 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0a5c10 con 0x7ff2500a1130 2024-09-17T15:38:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:19.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 4 set_mon_vals no callback set 2024-09-17T15:38:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff254fa3640 10 monclient: dump: 2024-09-17T15:38:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.147 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.148 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.148 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 10 monclient: shutdown 2024-09-17T15:38:19.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 1 -- 172.21.15.86:0/3923659607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 msgr2=0x7ff25013cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.115+0000 7ff258209640 1 --2- 172.21.15.86:0/3923659607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff24c009d90 tx=0x7ff24c01c800 comp rx=0 tx=0).stop 2024-09-17T15:38:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- 172.21.15.86:0/3923659607 shutdown_connections 2024-09-17T15:38:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 --2- 172.21.15.86:0/3923659607 >> v2:172.21.15.86:3300/0 conn(0x7ff25013d4a0 0x7ff25013f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 --2- 172.21.15.86:0/3923659607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2500a1130 0x7ff25013cf60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- 172.21.15.86:0/3923659607 >> 172.21.15.86:0/3923659607 conn(0x7ff2500a2f40 msgr2=0x7ff2500a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- 172.21.15.86:0/3923659607 shutdown_connections 2024-09-17T15:38:19.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- 172.21.15.86:0/3923659607 wait complete. 2024-09-17T15:38:19.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient: monmap: 2024-09-17T15:38:19.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:19.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:19.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:19.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 Processor -- start 2024-09-17T15:38:19.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- start start 2024-09-17T15:38:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient: init 2024-09-17T15:38:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:38:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500a1130 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134d90 0x7ff250133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): picked mon.noname-c con 0x7ff250134d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff25013d4a0 0x7ff25013aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): picked mon.noname-b con 0x7ff25013d4a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff250142770 con 0x7ff2500a1130 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2501425f0 con 0x7ff25013d4a0 2024-09-17T15:38:19.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff2501428f0 con 0x7ff250134d90 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff258209640 10 monclient(hunting): authenticate will time out at 1632.085554s 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134d90 0x7ff250133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58096/0 (socket says 172.21.15.86:58096) 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): get_auth_request con 0x7ff250134d90 auth_method 0 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 -- 172.21.15.86:0/567056985 learned_addr learned my addr 172.21.15.86:0/567056985 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): get_auth_request con 0x7ff2500a1130 auth_method 0 2024-09-17T15:38:19.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient(hunting): handle_auth_done global_id 124149 payload 824 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff2567a6640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff25013d4a0 0x7ff25013aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134d90 msgr2=0x7ff250133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134d90 0x7ff250133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff25013d4a0 msgr2=0x7ff25013aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff25013d4a0 0x7ff25013aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 monclient: found mon.noname-a 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.119+0000 7ff255fa5640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25011e380 con 0x7ff2500a1130 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff255fa5640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 msgr2=0x7ff2501cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff255fa5640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff24000c480 tx=0x7ff24000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240012a40 con 0x7ff2500a1130 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: dump: 2024-09-17T15:38:19.157 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff240017070 con 0x7ff2500a1130 2024-09-17T15:38:19.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240012e40 con 0x7ff2500a1130 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 10 monclient: dump: 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 5 monclient: authenticate success, global_id 124149 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 monclient: _renew_subs 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2501d0a60 con 0x7ff2500a1130 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff247fff640 4 set_mon_vals no callback set 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 monclient: _renew_subs 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2500a5930 con 0x7ff2500a1130 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 0x7ff25013afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff250135170 con 0x7ff250134210 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 0x7ff25013afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient: get_auth_request con 0x7ff250134210 auth_method 0 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff255fa5640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff24000c480 tx=0x7ff24000c940 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7ff24001c460 mgrmap(e 5) 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff24001c460 con 0x7ff2500a1130 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 10 monclient(hunting): handle_auth_done global_id 104354 payload 824 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff25011d210 con 0x7ff250134210 2024-09-17T15:38:19.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2467fc640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff23403bb60 0x7ff23403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2567a6640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff23403bb60 0x7ff23403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2567a6640 10 monclient: get_auth_request con 0x7ff23403bb60 auth_method 0 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 msgr2=0x7ff25013afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.123+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 0x7ff25013afa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0af000 tx=0x7ff24c0195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0b2040 con 0x7ff250134210 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 10 monclient: dump: 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff24c0ad440 con 0x7ff250134210 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 msgr2=0x7ff25013afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 0x7ff25013afa0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0af000 tx=0x7ff24c0195a0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 msgr2=0x7ff25013afa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2467fc640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff250134210 0x7ff25013afa0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0af000 tx=0x7ff24c0195a0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.127+0000 7ff2567a6640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff23403bb60 0x7ff23403e020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff25013c5d0 tx=0x7ff248008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff258209640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff258209640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff258209640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff25013b8c0 con 0x7ff25013b4e0 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 msgr2=0x7ff250136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 msgr2=0x7ff250136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-17T15:38:19.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.131+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:19.165 INFO:tasks.rados.rados.0.smithi203.stdout:1628: done (1 left) 2024-09-17T15:38:19.165 INFO:tasks.rados.rados.0.smithi203.stdout:1630: done (0 left) 2024-09-17T15:38:19.165 INFO:tasks.rados.rados.0.smithi203.stdout:1631: delete oid 31 current snap is 179 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1631: done (0 left) 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1632: rollback oid 9 current snap is 179 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 163 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1633: read oid 49 snap -1 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1633: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1634: read oid 19 snap -1 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1634: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1635: read oid 34 snap -1 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1635: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1636: write (excl) oid 31 current snap is 179 2024-09-17T15:38:19.166 INFO:tasks.rados.rados.0.smithi203.stdout:1636: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-17T15:38:19.176 INFO:tasks.rados.rados.0.smithi203.stdout:1636: writing smithi20328086-31 from 450586 to 925022 tid 1 2024-09-17T15:38:19.193 INFO:tasks.rados.rados.0.smithi203.stdout:1636: writing smithi20328086-31 from 1343879 to 1932085 tid 2 2024-09-17T15:38:19.193 INFO:tasks.rados.rados.0.smithi203.stdout:1637: delete oid 29 current snap is 179 2024-09-17T15:38:19.193 INFO:tasks.rados.rados.0.smithi203.stdout:1634: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:38:19.220 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1152683 2024-09-17T15:38:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient: get_auth_request con 0x7ff25013b4e0 auth_method 0 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 10 monclient(hunting): handle_auth_done global_id 104357 payload 824 2024-09-17T15:38:19.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff25009d3e0 con 0x7ff25013b4e0 2024-09-17T15:38:19.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 msgr2=0x7ff250136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.335+0000 7ff2557a4640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff24c01c630 tx=0x7ff24c0a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff24c0b2040 con 0x7ff25013b4e0 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 10 monclient: dump: 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.338 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7ff24c0bae50 con 0x7ff25013b4e0 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 msgr2=0x7ff250136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff24c01c630 tx=0x7ff24c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 msgr2=0x7ff250136a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff25013b4e0 0x7ff250136a60 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff24c01c630 tx=0x7ff24c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:19.339 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 130213, 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:38:19.340 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.341 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.342 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:19.343 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.344 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:19.345 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:19.348 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.349 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.350 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:19.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:19.352 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff23403bb60 msgr2=0x7ff23403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff23403bb60 0x7ff23403e020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff25013c5d0 tx=0x7ff248008040 comp rx=0 tx=0).stop 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 10 monclient: shutdown 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 msgr2=0x7ff2501cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff24000c480 tx=0x7ff24000c940 comp rx=0 tx=0).stop 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2567a6640 1 -- 172.21.15.86:0/567056985 reap_dead start 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff2467fc640 1 -- 172.21.15.86:0/567056985 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(368..368 src has 213..368) ==== 4783+0+0 (unknown 0 0 0) 0x7ff24004fd60 con 0x7ff2500a1130 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 shutdown_connections 2024-09-17T15:38:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 --2- 172.21.15.86:0/567056985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2500a1130 0x7ff2501cfb70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 >> 172.21.15.86:0/567056985 conn(0x7ff2500a2f40 msgr2=0x7ff2500a4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 shutdown_connections 2024-09-17T15:38:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.339+0000 7ff258209640 1 -- 172.21.15.86:0/567056985 wait complete. 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:1632: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:1632: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:1632: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 538 (ObjNum 454 snap 137 seq_num 454) dirty exists 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:1635: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:38:19.355 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:19.363 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:19.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:38:19.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:38:19.391 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:19.466 INFO:tasks.rados.rados.0.smithi203.stdout:1636: finishing write tid 1 to smithi20328086-31 2024-09-17T15:38:19.467 INFO:tasks.rados.rados.0.smithi203.stdout:1633: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: get_monmap_and_config 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: monmap: 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.489 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:19.492136+0000 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:19.492136+0000 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:19.490 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 Processor -- start 2024-09-17T15:38:19.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 -- start start 2024-09-17T15:38:19.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: init 2024-09-17T15:38:19.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-17T15:38:19.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7c80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:19.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7c80a5bd0 0x7fd7c809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c80a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:19.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd7c813d2f0 0x7fd7c813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c813d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:19.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c80a6280 con 0x7fd7c80a50e0 2024-09-17T15:38:19.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c809d2a0 con 0x7fd7c80a5bd0 2024-09-17T15:38:19.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c809c0a0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cf1bc640 10 monclient(hunting): authenticate will time out at 1632.457771s 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7ccf58640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd7c80a50e0 0x7fd7c80a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51442/0 (socket says 172.21.15.86:51442) 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7ccf58640 1 -- 172.21.15.86:0/976345867 learned_addr learned my addr 172.21.15.86:0/976345867 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:19.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7c7fff640 1 --2- 172.21.15.86:0/976345867 >> v2:172.21.15.86:3300/0 conn(0x7fd7c80a5bd0 0x7fd7c809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7c7fff640 10 monclient(hunting): get_auth_request con 0x7fd7c80a5bd0 auth_method 0 2024-09-17T15:38:19.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 --2- 172.21.15.86:0/976345867 >> v2:172.21.15.203:3300/0 conn(0x7fd7c813d2f0 0x7fd7c813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): get_auth_request con 0x7fd7c813d2f0 auth_method 0 2024-09-17T15:38:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient(hunting): handle_auth_done global_id 114214 payload 274 2024-09-17T15:38:19.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 -- 172.21.15.86:0/976345867 >> v2:172.21.15.86:3300/0 conn(0x7fd7c80a5bd0 msgr2=0x7fd7c809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:19.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 --2- 172.21.15.86:0/976345867 >> v2:172.21.15.86:3300/0 conn(0x7fd7c80a5bd0 0x7fd7c809cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 -- 172.21.15.86:0/976345867 >> v1:172.21.15.86:6789/0 conn(0x7fd7c80a50e0 legacy=0x7fd7c80a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:19.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 monclient: found mon.noname-c 2024-09-17T15:38:19.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:38:19.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.491+0000 7fd7cd759640 1 -- 172.21.15.86:0/976345867 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c809c380 con 0x7fd7c813d2f0 2024-09-17T15:38:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cd759640 1 -- 172.21.15.86:0/976345867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c813d2f0 msgr2=0x7fd7c813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cd759640 1 --2- 172.21.15.86:0/976345867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c813d2f0 0x7fd7c813f6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80a84e0 tx=0x7fd7c001c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a6227b89ab4ec42 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 1 -- 172.21.15.86:0/976345867 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00b2070 con 0x7fd7c813d2f0 2024-09-17T15:38:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: dump: 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.522 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 1 -- 172.21.15.86:0/976345867 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c00a5760 con 0x7fd7c813d2f0 2024-09-17T15:38:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 5 monclient: authenticate success, global_id 114214 2024-09-17T15:38:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 4 set_mon_vals no callback set 2024-09-17T15:38:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 1 -- 172.21.15.86:0/976345867 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00a5b30 con 0x7fd7c813d2f0 2024-09-17T15:38:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:38:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7c77fe640 10 monclient: dump: 2024-09-17T15:38:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.526 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 10 monclient: shutdown 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/976345867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c813d2f0 msgr2=0x7fd7c813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.495+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/976345867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c813d2f0 0x7fd7c813f6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd7c80a84e0 tx=0x7fd7c001c7c0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/976345867 shutdown_connections 2024-09-17T15:38:19.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/976345867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c813d2f0 0x7fd7c813f6e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/976345867 >> v2:172.21.15.86:3300/0 conn(0x7fd7c80a5bd0 0x7fd7c809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/976345867 >> 172.21.15.86:0/976345867 conn(0x7fd7c80aaae0 msgr2=0x7fd7c80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/976345867 shutdown_connections 2024-09-17T15:38:19.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/976345867 wait complete. 2024-09-17T15:38:19.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient: monmap: 2024-09-17T15:38:19.529 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.529 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:19.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:19.532 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 Processor -- start 2024-09-17T15:38:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- start start 2024-09-17T15:38:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient: init 2024-09-17T15:38:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:38:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80a50e0 0x7fd7c81cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a5bd0 0x7fd7c81d0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c80a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c8142650 con 0x7fd7c80a50e0 2024-09-17T15:38:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c80a6ad0 con 0x7fd7c80a5bd0 2024-09-17T15:38:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c81427d0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cf1bc640 10 monclient(hunting): authenticate will time out at 1632.465598s 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48506/0 (socket says 172.21.15.86:48506) 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 learned_addr learned my addr 172.21.15.86:0/1878387052 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): get_auth_request con 0x7fd7c813d2f0 auth_method 0 2024-09-17T15:38:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80a50e0 0x7fd7c81cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7c7fff640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a5bd0 0x7fd7c81d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): get_auth_request con 0x7fd7c80a50e0 auth_method 0 2024-09-17T15:38:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7c7fff640 10 monclient(hunting): get_auth_request con 0x7fd7c80a5bd0 auth_method 0 2024-09-17T15:38:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient(hunting): handle_auth_done global_id 104360 payload 824 2024-09-17T15:38:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a5bd0 msgr2=0x7fd7c81d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a5bd0 0x7fd7c81d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80a50e0 msgr2=0x7fd7c81cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c80a50e0 0x7fd7c81cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 monclient: found mon.noname-c 2024-09-17T15:38:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.499+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c813f6d0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 msgr2=0x7fd7c809faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cd759640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7c001ae70 tx=0x7fd7c0019560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00b2050 con 0x7fd7c813d2f0 2024-09-17T15:38:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: dump: 2024-09-17T15:38:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c00be070 con 0x7fd7c813d2f0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 5 monclient: authenticate success, global_id 104360 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 monclient: _renew_subs 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c80a18d0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00adaa0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 10 monclient: dump: 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 monclient: _renew_subs 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c81222f0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c6ffd640 4 set_mon_vals no callback set 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 0x7fd7c809ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c806a0e0 con 0x7fd7c80a1290 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 0x7fd7c809ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd7c00adc40 con 0x7fd7c813d2f0 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient: get_auth_request con 0x7fd7c80a1290 auth_method 0 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd7b403bbb0 0x7fd7b403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7fd7c00ea9e0 con 0x7fd7c813d2f0 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_done global_id 114220 payload 824 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7ccf58640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7c809cd60 con 0x7fd7c80a1290 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c7fff640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd7b403bbb0 0x7fd7b403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.503+0000 7fd7c7fff640 10 monclient: get_auth_request con 0x7fd7b403bbb0 auth_method 0 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7ccf58640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 msgr2=0x7fd7c809ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7ccf58640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 0x7fd7c809ffe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800eaa0 tx=0x7fd7b800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b800ee60 con 0x7fd7c80a1290 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:38:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 10 monclient: dump: 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c7fff640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd7b403bbb0 0x7fd7b403e070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8121320 tx=0x7fd7bc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7b8019070 con 0x7fd7c80a1290 2024-09-17T15:38:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 msgr2=0x7fd7c809ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 0x7fd7c809ffe0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800eaa0 tx=0x7fd7b800eb90 comp rx=0 tx=0).stop 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 msgr2=0x7fd7c809ffe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.507+0000 7fd7c57fa640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c80a1290 0x7fd7c809ffe0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800eaa0 tx=0x7fd7b800eb90 comp rx=0 tx=0).stop 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.511+0000 7fd7cf1bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.511+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 0x7fd7c80a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.511+0000 7fd7cf1bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.511+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c8074be0 con 0x7fd7c81d0770 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 0x7fd7c80a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient: get_auth_request con 0x7fd7c81d0770 auth_method 0 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 10 monclient(hunting): handle_auth_done global_id 114223 payload 824 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 1 -- 172.21.15.86:0/1878387052 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7c8138820 con 0x7fd7c81d0770 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 msgr2=0x7fd7c80a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7ccf58640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 0x7fd7c80a0520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80070c0 tx=0x7fd7b8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b801ea40 con 0x7fd7c81d0770 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.515+0000 7fd7c57fa640 10 monclient: dump: 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fd7b8014480 con 0x7fd7c81d0770 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 msgr2=0x7fd7c80a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 0x7fd7c80a0520 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80070c0 tx=0x7fd7b8004950 comp rx=0 tx=0).stop 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 msgr2=0x7fd7c80a0520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7c57fa640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7c81d0770 0x7fd7c80a0520 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd7b80070c0 tx=0x7fd7b8004950 comp rx=0 tx=0).stop 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 108, 2024-09-17T15:38:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 82015, 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:19.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:19.567 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:19.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd7b403bbb0 msgr2=0x7fd7b403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd7b403bbb0 0x7fd7b403e070 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8121320 tx=0x7fd7bc008040 comp rx=0 tx=0).stop 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 10 monclient: shutdown 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 msgr2=0x7fd7c809faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cd759640 1 -- 172.21.15.86:0/1878387052 reap_dead start 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd7c001ae70 tx=0x7fd7c0019560 comp rx=0 tx=0).stop 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 shutdown_connections 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 --2- 172.21.15.86:0/1878387052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7c813d2f0 0x7fd7c809faa0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 >> 172.21.15.86:0/1878387052 conn(0x7fd7c80aaae0 msgr2=0x7fd7c813ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 shutdown_connections 2024-09-17T15:38:19.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.519+0000 7fd7cf1bc640 1 -- 172.21.15.86:0/1878387052 wait complete. 2024-09-17T15:38:19.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:38:19.570 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:38:19.570 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:38:19.570 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:38:19.570 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:38:19.570 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:38:19.570 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:38:19.590 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:38:19.590 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:38:19.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:19.599 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 232083 2024-09-17T15:38:19.663 INFO:tasks.rados.rados.0.smithi203.stdout:1636: finishing write tid 2 to smithi20328086-31 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: finishing write tid 3 to smithi20328086-31 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: finishing write tid 4 to smithi20328086-31 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: oid 31 updating version 0 to 509 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: oid 31 updating version 509 to 511 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: oid 31 version 511 is already newer than 510 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 511 (ObjNum 598 snap 179 seq_num 598) dirty exists 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: left oid 31 (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1632: done (5 left) 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1633: done (4 left) 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1634: done (3 left) 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1635: done (2 left) 2024-09-17T15:38:19.664 INFO:tasks.rados.rados.0.smithi203.stdout:1636: done (1 left) 2024-09-17T15:38:19.665 INFO:tasks.rados.rados.0.smithi203.stdout:1637: done (0 left) 2024-09-17T15:38:19.665 INFO:tasks.rados.rados.0.smithi203.stdout:1638: read oid 10 snap -1 2024-09-17T15:38:19.665 INFO:tasks.rados.rados.0.smithi203.stdout:1638: expect deleted 2024-09-17T15:38:19.665 INFO:tasks.rados.rados.0.smithi203.stdout:1639: snap_remove snap 174 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 10 monclient: get_monmap_and_config 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 10 monclient: monmap: 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:19.706000+0000 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:19.706000+0000 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:19.704 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 1 Processor -- start 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 1 -- start start 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.703+0000 7f1972b2d640 10 monclient: init 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 0x7f196c09ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-c con 0x7f196c0a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-b con 0x7f196c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f196c137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f196c09d350 con 0x7f196c137ea0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f196c0a6530 con 0x7f196c0a50e0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f196c09c150 con 0x7f196c0a54c0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f1972b2d640 10 monclient(hunting): authenticate will time out at 1632.672346s 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48518/0 (socket says 172.21.15.86:48518) 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 -- 172.21.15.86:0/3621524494 learned_addr learned my addr 172.21.15.86:0/3621524494 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:19.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): get_auth_request con 0x7f196c0a54c0 auth_method 0 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f19710ca640 1 --1- 172.21.15.86:0/3621524494 >> v1:172.21.15.86:6789/0 conn(0x7f196c137ea0 0x7f196c138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51458/0 (socket says 172.21.15.86:51458) 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f19708c9640 1 -- 172.21.15.86:0/3621524494 >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 msgr2=0x7f196c09ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f19708c9640 1 --2- 172.21.15.86:0/3621524494 >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 0x7f196c09ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient(hunting): handle_auth_done global_id 104363 payload 274 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 -- 172.21.15.86:0/3621524494 >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 msgr2=0x7f196c09ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 --2- 172.21.15.86:0/3621524494 >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 0x7f196c09ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 -- 172.21.15.86:0/3621524494 >> v1:172.21.15.86:6789/0 conn(0x7f196c137ea0 legacy=0x7f196c138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 monclient: found mon.noname-b 2024-09-17T15:38:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.707+0000 7f196bfff640 1 -- 172.21.15.86:0/3621524494 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c09c430 con 0x7f196c0a54c0 2024-09-17T15:38:19.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196bfff640 1 -- 172.21.15.86:0/3621524494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 msgr2=0x7f196c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196bfff640 1 --2- 172.21.15.86:0/3621524494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f196c0a8280 tx=0x7f196001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69019093a770f917 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 1 -- 172.21.15.86:0/3621524494 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600b0070 con 0x7f196c0a54c0 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: dump: 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 1 -- 172.21.15.86:0/3621524494 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19600ab600 con 0x7f196c0a54c0 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 5 monclient: authenticate success, global_id 104363 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 1 -- 172.21.15.86:0/3621524494 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19600ab9d0 con 0x7f196c0a54c0 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196b7fe640 10 monclient: dump: 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 4 set_mon_vals no callback set 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: shutdown 2024-09-17T15:38:19.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- 172.21.15.86:0/3621524494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 msgr2=0x7f196c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- 172.21.15.86:0/3621524494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f196c0a8280 tx=0x7f196001c400 comp rx=0 tx=0).stop 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- 172.21.15.86:0/3621524494 shutdown_connections 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- 172.21.15.86:0/3621524494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a54c0 0x7f196c0a58d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- 172.21.15.86:0/3621524494 >> v2:172.21.15.203:3300/0 conn(0x7f196c0a50e0 0x7f196c09ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- 172.21.15.86:0/3621524494 >> 172.21.15.86:0/3621524494 conn(0x7f196c0aaae0 msgr2=0x7f196c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- 172.21.15.86:0/3621524494 shutdown_connections 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- 172.21.15.86:0/3621524494 wait complete. 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: build_initial_monmap 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: monmap: 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:19.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:19.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 Processor -- start 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- start start 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: init 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-c con 0x7f196c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:19.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 0x7f196c137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-b con 0x7f196c0a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 0x7f196c13c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): picked mon.noname-a con 0x7f196c137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f196c142cb0 con 0x7f196c137ea0 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f196c142b30 con 0x7f196c0a54c0 2024-09-17T15:38:19.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f196c142e30 con 0x7f196c0a50e0 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f1972b2d640 10 monclient(hunting): authenticate will time out at 1632.679310s 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48524/0 (socket says 172.21.15.86:48524) 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 -- 172.21.15.86:0/4074582755 learned_addr learned my addr 172.21.15.86:0/4074582755 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): get_auth_request con 0x7f196c0a50e0 auth_method 0 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 0x7f196c13c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196bfff640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 msgr2=0x7f196c137020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:19.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): get_auth_request con 0x7f196c137ea0 auth_method 0 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f196bfff640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 0x7f196c137020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient(hunting): handle_auth_done global_id 104366 payload 824 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 msgr2=0x7f196c137020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 0x7f196c137020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 msgr2=0x7f196c13c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 0x7f196c13c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient: _finish_hunting 0 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 monclient: found mon.noname-c 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19708c9640 1 -- 172.21.15.86:0/4074582755 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f196c1219b0 con 0x7f196c0a50e0 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.711+0000 7f19710ca640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 0x7f196c13c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19708c9640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 msgr2=0x7f196c1d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19708c9640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f195c00b9e0 tx=0x7f195c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c00cdb0 con 0x7f196c0a50e0 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: dump: 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.724 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: _finish_auth 0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: _check_auth_tickets 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f195c010070 con 0x7f196c0a50e0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 5 monclient: authenticate success, global_id 104366 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _renew_subs 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f196c0a18d0 con 0x7f196c0a50e0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195c0156b0 con 0x7f196c0a50e0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:19.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 10 monclient: dump: 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _renew_subs 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f196c121110 con 0x7f196c0a50e0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f196affd640 4 set_mon_vals no callback set 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f196c09ce10 con 0x7f196c0a50e0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f195c01d390 con 0x7f196c0a50e0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 0x7f195803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.715+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7f195c050210 con 0x7f196c0a50e0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.719+0000 7f196bfff640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 0x7f195803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.719+0000 7f196bfff640 10 monclient: get_auth_request con 0x7f195803bb60 auth_method 0 2024-09-17T15:38:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.719+0000 7f196bfff640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 0x7f195803e020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f196c09c7c0 tx=0x7f1960008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.723+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f195c01a060 con 0x7f196c0a50e0 2024-09-17T15:38:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.723+0000 7f19697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:19.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.723+0000 7f19697fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:19.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f196c074be0 con 0x7f196c0a50e0 2024-09-17T15:38:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f19697fa640 1 -- 172.21.15.86:0/4074582755 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f195c01cc60 con 0x7f196c0a50e0 2024-09-17T15:38:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f19697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:19.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f19697fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:19.946 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 msgr2=0x7f195803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 0x7f195803e020 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f196c09c7c0 tx=0x7f1960008040 comp rx=0 tx=0).stop 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 10 monclient: shutdown 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 msgr2=0x7f196c1d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f195c00b9e0 tx=0x7f195c00bea0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 shutdown_connections 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f195803bb60 0x7f195803e020 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f196c137ea0 0x7f196c13c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f196c0a54c0 0x7f196c137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 --2- 172.21.15.86:0/4074582755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f196c0a50e0 0x7f196c1d01f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 >> 172.21.15.86:0/4074582755 conn(0x7f196c0aaae0 msgr2=0x7f196c0a8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 shutdown_connections 2024-09-17T15:38:19.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:19.947+0000 7f1972b2d640 1 -- 172.21.15.86:0/4074582755 wait complete. 2024-09-17T15:38:22.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:23.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: get_monmap_and_config 2024-09-17T15:38:23.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: build_initial_monmap 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: monmap: 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:23.093752+0000 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:23.093752+0000 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:23.091 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:23.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 Processor -- start 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 -- start start 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: init 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6df80689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:23.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 0x7f6df80691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-c con 0x7f6df8068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6df8139f00 0x7f6df813c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-a con 0x7f6df8139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df809d770 con 0x7f6df80689d0 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6df809ed00 con 0x7f6df8139f00 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6df809efe0 con 0x7f6df8068db0 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dffc3c640 10 monclient(hunting): authenticate will time out at 1636.059658s 2024-09-17T15:38:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfd1d7640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 msgr2=0x7f6df80691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfd9d8640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6df80689d0 legacy=0x7f6df809ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfd1d7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 0x7f6df80691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6df8139f00 0x7f6df813c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6df8139f00 0x7f6df813c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58156/0 (socket says 172.21.15.86:58156) 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/3347873481 learned_addr learned my addr 172.21.15.86:0/3347873481 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 10 monclient(hunting): get_auth_request con 0x7f6df8139f00 auth_method 0 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.091+0000 7f6dfe1d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient(hunting): handle_auth_done global_id 124161 payload 274 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/3347873481 >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 msgr2=0x7f6df80691c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 1 --2- 172.21.15.86:0/3347873481 >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 0x7f6df80691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/3347873481 >> v1:172.21.15.203:6789/0 conn(0x7f6df80689d0 legacy=0x7f6df809ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient: _finish_hunting 0 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 1 monclient: found mon.noname-a 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.095+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/3347873481 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df8069700 con 0x7f6df8139f00 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/3347873481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df8139f00 msgr2=0x7f6df813c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfe1d9640 1 --2- 172.21.15.86:0/3347873481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df8139f00 0x7f6df813c2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6df80a06b0 tx=0x7f6df001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42b8b23bccc1859 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 1 -- 172.21.15.86:0/3347873481 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df00a5660 con 0x7f6df8139f00 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: dump: 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:23.099 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: _finish_auth 0 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: _check_auth_tickets 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 5 monclient: authenticate success, global_id 124161 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:23.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 1 -- 172.21.15.86:0/3347873481 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df00a5800 con 0x7f6df8139f00 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 4 set_mon_vals no callback set 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 1 -- 172.21.15.86:0/3347873481 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df00a5bd0 con 0x7f6df8139f00 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfc9d6640 10 monclient: dump: 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:23.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: shutdown 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- 172.21.15.86:0/3347873481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df8139f00 msgr2=0x7f6df813c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/3347873481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df8139f00 0x7f6df813c2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6df80a06b0 tx=0x7f6df001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- 172.21.15.86:0/3347873481 shutdown_connections 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/3347873481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df8139f00 0x7f6df813c2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/3347873481 >> v2:172.21.15.203:3300/0 conn(0x7f6df8068db0 0x7f6df80691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- 172.21.15.86:0/3347873481 >> 172.21.15.86:0/3347873481 conn(0x7f6df80a6440 msgr2=0x7f6df80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:23.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- 172.21.15.86:0/3347873481 shutdown_connections 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- 172.21.15.86:0/3347873481 wait complete. 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: build_initial_monmap 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: monmap: 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 Processor -- start 2024-09-17T15:38:23.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- start start 2024-09-17T15:38:23.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: init 2024-09-17T15:38:23.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:23.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-a con 0x7f6df80689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6df8068db0 0x7f6df81d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-c con 0x7f6df8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 0x7f6df81d0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): picked mon.noname-b con 0x7f6df8139f00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6df809be00 con 0x7f6df80689d0 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6df809bc80 con 0x7f6df8139f00 2024-09-17T15:38:23.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6df809bf80 con 0x7f6df8068db0 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dffc3c640 10 monclient(hunting): authenticate will time out at 1636.067208s 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6df8068db0 0x7f6df81d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58158/0 (socket says 172.21.15.86:58158) 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 -- 172.21.15.86:0/1033431367 learned_addr learned my addr 172.21.15.86:0/1033431367 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): get_auth_request con 0x7f6df80689d0 auth_method 0 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): get_auth_request con 0x7f6df8068db0 auth_method 0 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfe1d9640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 msgr2=0x7f6df81d0a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfe1d9640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 0x7f6df81d0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd1d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient(hunting): handle_auth_done global_id 124164 payload 824 2024-09-17T15:38:23.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6df8068db0 msgr2=0x7f6df81d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6df8068db0 0x7f6df81d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 msgr2=0x7f6df81d0a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 0x7f6df81d0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient: _finish_hunting 0 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 monclient: found mon.noname-a 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.099+0000 7f6dfd9d8640 1 -- 172.21.15.86:0/1033431367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df8140df0 con 0x7f6df80689d0 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dfd9d8640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 msgr2=0x7f6df81d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dfd9d8640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b4e0 tx=0x7f6df400b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:23.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dfd9d8640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b4e0 tx=0x7f6df400b9a0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f6df4015070 config(0 keys) 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4005070 con 0x7f6df80689d0 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: dump: 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:23.112 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df4015070 con 0x7f6df80689d0 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 5 monclient: authenticate success, global_id 124164 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _renew_subs 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df8138f10 con 0x7f6df80689d0 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6def7fe640 4 set_mon_vals no callback set 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _renew_subs 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df809d460 con 0x7f6df80689d0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df80691c0 con 0x7f6df80689d0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4005470 con 0x7f6df80689d0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 10 monclient: dump: 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:23.114 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6df4005690 con 0x7f6df80689d0 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dee7fc640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 0x7f6ddc03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dfd1d7640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 0x7f6ddc03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.103+0000 7f6dfd1d7640 10 monclient: get_auth_request con 0x7f6ddc03bb10 auth_method 0 2024-09-17T15:38:23.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.107+0000 7f6dfd1d7640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 0x7f6ddc03dfd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6de80093d0 tx=0x7f6de8009360 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:23.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.107+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7f6df404fe10 con 0x7f6df80689d0 2024-09-17T15:38:23.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.727+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6df404e7f0 con 0x7f6df80689d0 2024-09-17T15:38:23.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.727+0000 7f6dee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:23.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.727+0000 7f6dee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:23.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dffc3c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dffc3c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dffc3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6df8135a90 con 0x7f6df80689d0 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dee7fc640 1 -- 172.21.15.86:0/1033431367 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6df4051070 con 0x7f6df80689d0 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:23.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.971+0000 7f6dee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:23.972 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 msgr2=0x7f6ddc03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 0x7f6ddc03dfd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6de80093d0 tx=0x7f6de8009360 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 10 monclient: shutdown 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 msgr2=0x7f6df81d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6df400b4e0 tx=0x7f6df400b9a0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 shutdown_connections 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6ddc03bb10 0x7f6ddc03dfd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df8139f00 0x7f6df81d0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6df8068db0 0x7f6df81d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 --2- 172.21.15.86:0/1033431367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6df80689d0 0x7f6df81d0010 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:23.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 >> 172.21.15.86:0/1033431367 conn(0x7f6df80a6440 msgr2=0x7f6df813bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:23.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 shutdown_connections 2024-09-17T15:38:23.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:23.975+0000 7f6dffc3c640 1 -- 172.21.15.86:0/1033431367 wait complete. 2024-09-17T15:38:26.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:27.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 10 monclient: get_monmap_and_config 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 10 monclient: build_initial_monmap 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 10 monclient: monmap: 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:27.121891+0000 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:27.121891+0000 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:27.119 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:27.120 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 1 Processor -- start 2024-09-17T15:38:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 1 -- start start 2024-09-17T15:38:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.119+0000 7fa8288a7640 10 monclient: init 2024-09-17T15:38:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8240689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8240694c0 0x7fa82406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-a con 0x7fa8240694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-b con 0x7fa82413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa824074470 con 0x7fa8240689d0 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa82409c900 con 0x7fa8240694c0 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa82409cb90 con 0x7fa82413aff0 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa8288a7640 10 monclient(hunting): authenticate will time out at 1640.088271s 2024-09-17T15:38:27.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa82259c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fa8240689d0 legacy=0x7fa824068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48544/0 (socket says 172.21.15.86:48544) 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 -- 172.21.15.86:0/1219403234 learned_addr learned my addr 172.21.15.86:0/1219403234 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): get_auth_request con 0x7fa82413aff0 auth_method 0 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8240694c0 0x7fa82406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): get_auth_request con 0x7fa8240694c0 auth_method 0 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient(hunting): handle_auth_done global_id 104372 payload 274 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 -- 172.21.15.86:0/1219403234 >> v2:172.21.15.86:3300/0 conn(0x7fa8240694c0 msgr2=0x7fa82406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 --2- 172.21.15.86:0/1219403234 >> v2:172.21.15.86:3300/0 conn(0x7fa8240694c0 0x7fa82406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 -- 172.21.15.86:0/1219403234 >> v1:172.21.15.203:6789/0 conn(0x7fa8240689d0 legacy=0x7fa824068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 monclient: found mon.noname-b 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:27.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.123+0000 7fa822d9d640 1 -- 172.21.15.86:0/1219403234 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82409ce70 con 0x7fa82413aff0 2024-09-17T15:38:27.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa822d9d640 1 -- 172.21.15.86:0/1219403234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 msgr2=0x7fa82413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:27.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa822d9d640 1 --2- 172.21.15.86:0/1219403234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009870 tx=0x7fa81c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e4fb25d6b77fab server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:27.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 1 -- 172.21.15.86:0/1219403234 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0b1070 con 0x7fa82413aff0 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: dump: 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: _finish_auth 0 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: _check_auth_tickets 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 1 -- 172.21.15.86:0/1219403234 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81c0ac620 con 0x7fa82413aff0 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 1 -- 172.21.15.86:0/1219403234 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82409c2d0 con 0x7fa82413aff0 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 5 monclient: authenticate success, global_id 104372 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 4 set_mon_vals no callback set 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:27.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82159a640 10 monclient: dump: 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: shutdown 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:27.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- 172.21.15.86:0/1219403234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 msgr2=0x7fa82413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- 172.21.15.86:0/1219403234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa81c009870 tx=0x7fa81c01c420 comp rx=0 tx=0).stop 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- 172.21.15.86:0/1219403234 shutdown_connections 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- 172.21.15.86:0/1219403234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa82413aff0 0x7fa82413d3e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- 172.21.15.86:0/1219403234 >> v2:172.21.15.86:3300/0 conn(0x7fa8240694c0 0x7fa82406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- 172.21.15.86:0/1219403234 >> 172.21.15.86:0/1219403234 conn(0x7fa8240a6440 msgr2=0x7fa8240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- 172.21.15.86:0/1219403234 shutdown_connections 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- 172.21.15.86:0/1219403234 wait complete. 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: build_initial_monmap 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: monmap: 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:27.131 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 Processor -- start 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- start start 2024-09-17T15:38:27.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: init 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-c con 0x7fa8240689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 0x7fa8241328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-a con 0x7fa8240694c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 0x7fa824070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): picked mon.noname-b con 0x7fa82413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:27.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa824140670 con 0x7fa8240694c0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8241404f0 con 0x7fa82413aff0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa8241407f0 con 0x7fa8240689d0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa8288a7640 10 monclient(hunting): authenticate will time out at 1640.095657s 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 0x7fa8241328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48560/0 (socket says 172.21.15.86:48560) 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 1 -- 172.21.15.86:0/2530070027 learned_addr learned my addr 172.21.15.86:0/2530070027 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): get_auth_request con 0x7fa8240689d0 auth_method 0 2024-09-17T15:38:27.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): get_auth_request con 0x7fa8240694c0 auth_method 0 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa822d9d640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 msgr2=0x7fa824070bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa822d9d640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 0x7fa824070bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa821d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:27.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.127+0000 7fa82259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 10 monclient(hunting): handle_auth_done global_id 104375 payload 824 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 msgr2=0x7fa824070bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 0x7fa824070bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 msgr2=0x7fa8241328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 0x7fa8241328c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 10 monclient: _finish_hunting 0 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 monclient: found mon.noname-c 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 -- 172.21.15.86:0/2530070027 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82409d460 con 0x7fa8240689d0 2024-09-17T15:38:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa821d9b640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 0x7fa8241328c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 msgr2=0x7fa824132380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa82259c640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b7d0 tx=0x7fa81400bc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81400cbd0 con 0x7fa8240689d0 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: dump: 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: _finish_auth 0 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa814010070 con 0x7fa8240689d0 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814015470 con 0x7fa8240689d0 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 10 monclient: dump: 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa820d99640 4 set_mon_vals no callback set 2024-09-17T15:38:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 5 monclient: authenticate success, global_id 104375 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _renew_subs 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82411cd80 con 0x7fa8240689d0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _renew_subs 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82413d1e0 con 0x7fa8240689d0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82406f840 con 0x7fa8240689d0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa81401e3f0 con 0x7fa8240689d0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.131+0000 7fa812ffd640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 0x7fa80403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.135+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7fa81404fee0 con 0x7fa8240689d0 2024-09-17T15:38:27.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.135+0000 7fa821d9b640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 0x7fa80403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.135+0000 7fa821d9b640 10 monclient: get_auth_request con 0x7fa80403bb90 auth_method 0 2024-09-17T15:38:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.135+0000 7fa821d9b640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 0x7fa80403e050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa82411cf90 tx=0x7fa818009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.139+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa82409d460 con 0x7fa8240689d0 2024-09-17T15:38:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.139+0000 7fa812ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:27.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.139+0000 7fa812ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa82406d4f0 con 0x7fa8240689d0 2024-09-17T15:38:27.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa812ffd640 1 -- 172.21.15.86:0/2530070027 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fa814051030 con 0x7fa8240689d0 2024-09-17T15:38:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa812ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:27.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa812ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:27.359 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":108,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 msgr2=0x7fa80403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:27.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 0x7fa80403e050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa82411cf90 tx=0x7fa818009190 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 10 monclient: shutdown 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 msgr2=0x7fa824132380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa81400b7d0 tx=0x7fa81400bc90 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 shutdown_connections 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80403bb90 0x7fa80403e050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa82413aff0 0x7fa824070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8240694c0 0x7fa8241328c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 --2- 172.21.15.86:0/2530070027 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8240689d0 0x7fa824132380 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.359+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 >> 172.21.15.86:0/2530070027 conn(0x7fa8240a6440 msgr2=0x7fa8240a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.363+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 shutdown_connections 2024-09-17T15:38:27.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:27.363+0000 7fa8288a7640 1 -- 172.21.15.86:0/2530070027 wait complete. 2024-09-17T15:38:30.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: get_monmap_and_config 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: build_initial_monmap 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: monmap: 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:30.509522+0000 2024-09-17T15:38:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:30.509522+0000 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:30.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 Processor -- start 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 -- start start 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: init 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f591c0689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f591c068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f591c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f591c074470 con 0x7f591c068db0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f591c09c900 con 0x7f591c0689d0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f591c09cb90 con 0x7f591c13aff0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5923a6b640 10 monclient(hunting): authenticate will time out at 1643.475115s 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5921006640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f591c068db0 0x7f591c069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:35486/0 (socket says 172.21.15.86:35486) 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 learned_addr learned my addr 172.21.15.86:0/3530378080 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 10 monclient(hunting): get_auth_request con 0x7f591c13aff0 auth_method 0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 msgr2=0x7f591c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:38:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.506+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.510+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.86:6789/0 conn(0x7f591c068db0 legacy=0x7f591c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 msgr2=0x7f591c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 10 monclient(hunting): get_auth_request con 0x7f591c13aff0 auth_method 0 2024-09-17T15:38:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:30.710+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:31.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 10 monclient(hunting): get_auth_request con 0x7f591c13aff0 auth_method 0 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 msgr2=0x7f591c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:31.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.114+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 10 monclient(hunting): get_auth_request con 0x7f591c13aff0 auth_method 0 2024-09-17T15:38:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:31.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 msgr2=0x7f591c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:31.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:31.918+0000 7f5922008640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:38:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): tick 2024-09-17T15:38:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 msgr2=0x7f591c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c13aff0 0x7f591c13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 msgr2=0x7f591c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.203:3300/0 conn(0x7f591c0689d0 0x7f591c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.86:6789/0 conn(0x7f591c068db0 legacy=0x7f591c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient: _add_conns ranks=[3,1] 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5908003770 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59080062c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 -- 172.21.15.86:0/3530378080 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f591c09ce70 con 0x7f59080062c0 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 1 -- 172.21.15.86:0/3530378080 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f591c09c900 con 0x7f5908003770 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f591b7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.203:6789/0 conn(0x7f59080062c0 legacy=0x7f5908008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 10 monclient(hunting): get_auth_request con 0x7f5908003770 auth_method 0 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.510+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.203:6789/0 conn(0x7f59080062c0 legacy=0x7f5908008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 10 monclient(hunting): get_auth_request con 0x7f5908003770 auth_method 0 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:33.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:33.714+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:38:34.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.203:6789/0 conn(0x7f59080062c0 legacy=0x7f5908008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 10 monclient(hunting): get_auth_request con 0x7f5908003770 auth_method 0 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 msgr2=0x7f5908005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:38:34.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.118+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:38:34.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921006640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.203:6789/0 conn(0x7f59080062c0 legacy=0x7f5908008780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:34.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> v2:172.21.15.86:3300/0 conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): get_auth_request con 0x7f5908003770 auth_method 0 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient(hunting): handle_auth_done global_id 124173 payload 274 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> v1:172.21.15.203:6789/0 conn(0x7f59080062c0 legacy=0x7f5908008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient: _finish_hunting 0 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 1 monclient: found mon.noname-a 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:34.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.922+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f590800b0f0 con 0x7f5908003770 2024-09-17T15:38:34.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5921807640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5908003770 msgr2=0x7f5908005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:34.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5921807640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5908003770 0x7f5908005bd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5914003940 tx=0x7f591401d1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4ddbb991f32568 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 1 -- 172.21.15.86:0/3530378080 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59140b1070 con 0x7f5908003770 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: dump: 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: _finish_auth 0 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: _check_auth_tickets 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 1 -- 172.21.15.86:0/3530378080 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59140ac600 con 0x7f5908003770 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 5 monclient: authenticate success, global_id 124173 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 1 -- 172.21.15.86:0/3530378080 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c09c2d0 con 0x7f5908003770 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 4 set_mon_vals no callback set 2024-09-17T15:38:34.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5920805640 10 monclient: dump: 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:34.933 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:34.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 10 monclient: shutdown 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 1 -- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5908003770 msgr2=0x7f5908005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5923a6b640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5908003770 0x7f5908005bd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5914003940 tx=0x7f591401d1c0 comp rx=0 tx=0).stop 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.930+0000 7f5922008640 1 -- 172.21.15.86:0/3530378080 reap_dead start 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- 172.21.15.86:0/3530378080 shutdown_connections 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 --2- 172.21.15.86:0/3530378080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5908003770 0x7f5908005bd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- 172.21.15.86:0/3530378080 >> 172.21.15.86:0/3530378080 conn(0x7f591c0a6440 msgr2=0x7f591c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- 172.21.15.86:0/3530378080 shutdown_connections 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- 172.21.15.86:0/3530378080 wait complete. 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient: build_initial_monmap 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient: monmap: 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:34.935 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 Processor -- start 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- start start 2024-09-17T15:38:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient: init 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f591c135fc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 0x7f591c071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f591c121190 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c0724a0 0x7f591c072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:34.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f591c0724a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f591c140560 con 0x7f591c135fc0 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f591c1403e0 con 0x7f591c121190 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f591c1406e0 con 0x7f591c0724a0 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5923a6b640 10 monclient(hunting): authenticate will time out at 1647.903681s 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5921807640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5922008640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c0724a0 0x7f591c072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5921006640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 msgr2=0x7f591c071f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5921807640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34520/0 (socket says 172.21.15.86:34520) 2024-09-17T15:38:34.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): get_auth_request con 0x7f591c0724a0 auth_method 0 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.934+0000 7f5921006640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 0x7f591c071f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 -- 172.21.15.86:0/1258987001 learned_addr learned my addr 172.21.15.86:0/1258987001 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): get_auth_request con 0x7f591c135fc0 auth_method 0 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:34.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5922008640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient(hunting): handle_auth_done global_id 124176 payload 824 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 -- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c0724a0 msgr2=0x7f591c072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c0724a0 0x7f591c072910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 -- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 msgr2=0x7f591c071f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 0x7f591c071f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient: _finish_hunting 0 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 monclient: found mon.noname-a 2024-09-17T15:38:34.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:34.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:34.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 -- 172.21.15.86:0/1258987001 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591c1cdf80 con 0x7f591c135fc0 2024-09-17T15:38:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 -- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 msgr2=0x7f591c1363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5921807640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f591401adf0 tx=0x7f5914019f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59140ba070 con 0x7f591c135fc0 2024-09-17T15:38:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: dump: 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:34.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5914003920 con 0x7f591c135fc0 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59140b1040 con 0x7f591c135fc0 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 5 monclient: authenticate success, global_id 124176 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 10 monclient: _renew_subs 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f591c13d5a0 con 0x7f591c135fc0 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591a7fc640 10 monclient: dump: 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.945 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f591bfff640 4 set_mon_vals no callback set 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 10 monclient: _renew_subs 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.938+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f591c09d460 con 0x7f591c135fc0 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f59140bd070 con 0x7f591c135fc0 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5923a6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5923a6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:34.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5923a6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f591a7fc640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 0x7f590803ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f591c06f840 con 0x7f591c135fc0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7f59140eade0 con 0x7f591c135fc0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5921006640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 0x7f590803ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5921006640 10 monclient: get_auth_request con 0x7f590803c9b0 auth_method 0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.942+0000 7f5921006640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 0x7f590803ee70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f591c13ce40 tx=0x7f5910001af0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.946+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f591c13d5a0 con 0x7f591c135fc0 2024-09-17T15:38:34.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.946+0000 7f591a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:34.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:34.946+0000 7f591a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:35.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f591c06d4f0 con 0x7f591c135fc0 2024-09-17T15:38:35.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f591a7fc640 1 -- 172.21.15.86:0/1258987001 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f591c1cdf80 con 0x7f591c135fc0 2024-09-17T15:38:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f591a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:35.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f591a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:35.179 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":112,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 msgr2=0x7f590803ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 0x7f590803ee70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f591c13ce40 tx=0x7f5910001af0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 10 monclient: shutdown 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 msgr2=0x7f591c1363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.178+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f591401adf0 tx=0x7f5914019f60 comp rx=0 tx=0).stop 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 shutdown_connections 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f590803c9b0 0x7f590803ee70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f591c0724a0 0x7f591c072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f591c121190 0x7f591c071f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 --2- 172.21.15.86:0/1258987001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f591c135fc0 0x7f591c1363a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 >> 172.21.15.86:0/1258987001 conn(0x7f591c0a6440 msgr2=0x7f591c0a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 shutdown_connections 2024-09-17T15:38:35.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.182+0000 7f5923a6b640 1 -- 172.21.15.86:0/1258987001 wait complete. 2024-09-17T15:38:35.196 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:38:35.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:38:35.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:38:35.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: get_monmap_and_config 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: build_initial_monmap 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: monmap: 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:35.312049+0000 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:35.312049+0000 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:35.309 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:35.310 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:35.310 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 Processor -- start 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 -- start start 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: init 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3a340a0c70 0x7f3a340a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-a con 0x7f3a340a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3a340a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a34136160 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:35.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a340a2130 con 0x7f3a34136160 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a340a26f0 con 0x7f3a340a1590 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3a340a2410 con 0x7f3a340a0c70 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a39db8640 10 monclient(hunting): authenticate will time out at 1648.277511s 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a32ffd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f3a340a1590 legacy=0x7f3a34135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3a340a0c70 0x7f3a340a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a33fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3a34136160 0x7f3a3413df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:35500/0 (socket says 172.21.15.86:35500) 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a33fff640 1 -- 172.21.15.86:0/4016096570 learned_addr learned my addr 172.21.15.86:0/4016096570 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a340a0c70 auth_method 0 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient(hunting): handle_auth_done global_id 124179 payload 274 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 1 -- 172.21.15.86:0/4016096570 >> v1:172.21.15.203:6789/0 conn(0x7f3a340a1590 legacy=0x7f3a34135a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 1 -- 172.21.15.86:0/4016096570 >> v1:172.21.15.86:6789/0 conn(0x7f3a34136160 legacy=0x7f3a3413df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 1 monclient: found mon.noname-a 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:35.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.310+0000 7f3a337fe640 1 -- 172.21.15.86:0/4016096570 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3409f9f0 con 0x7f3a340a0c70 2024-09-17T15:38:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a337fe640 1 -- 172.21.15.86:0/4016096570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a0c70 msgr2=0x7f3a340a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a337fe640 1 --2- 172.21.15.86:0/4016096570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a0c70 0x7f3a340a1050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a240097c0 tx=0x7f3a2401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208d212fa0c78aeb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 1 -- 172.21.15.86:0/4016096570 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240b2070 con 0x7f3a340a0c70 2024-09-17T15:38:35.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:35.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: dump: 2024-09-17T15:38:35.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.330 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.331 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:35.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 1 -- 172.21.15.86:0/4016096570 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a240ad640 con 0x7f3a340a0c70 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 1 -- 172.21.15.86:0/4016096570 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240ada10 con 0x7f3a340a0c70 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:35.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a327fc640 10 monclient: dump: 2024-09-17T15:38:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.333 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.334 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.334 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 5 monclient: authenticate success, global_id 124179 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 4 set_mon_vals no callback set 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 10 monclient: shutdown 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:35.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 -- 172.21.15.86:0/4016096570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a0c70 msgr2=0x7f3a340a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 --2- 172.21.15.86:0/4016096570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a0c70 0x7f3a340a1050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a240097c0 tx=0x7f3a2401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 -- 172.21.15.86:0/4016096570 shutdown_connections 2024-09-17T15:38:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 --2- 172.21.15.86:0/4016096570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a0c70 0x7f3a340a1050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 -- 172.21.15.86:0/4016096570 >> 172.21.15.86:0/4016096570 conn(0x7f3a340a2f40 msgr2=0x7f3a340a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:35.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 -- 172.21.15.86:0/4016096570 shutdown_connections 2024-09-17T15:38:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.314+0000 7f3a39db8640 1 -- 172.21.15.86:0/4016096570 wait complete. 2024-09-17T15:38:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient: build_initial_monmap 2024-09-17T15:38:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient: monmap: 2024-09-17T15:38:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:35.337 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:35.338 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:35.338 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:35.338 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:35.338 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.338 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 Processor -- start 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 -- start start 2024-09-17T15:38:35.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient: init 2024-09-17T15:38:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:38:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 0x7f3a34130110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-c con 0x7f3a340a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:35.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 0x7f3a34137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-a con 0x7f3a340a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a34136160 0x7f3a341380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): picked mon.noname-b con 0x7f3a34136160 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a340a00d0 con 0x7f3a340a1590 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3409ff50 con 0x7f3a34136160 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3a34141e70 con 0x7f3a340a0c70 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:35.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a39db8640 10 monclient(hunting): authenticate will time out at 1648.285354s 2024-09-17T15:38:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 0x7f3a34137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 0x7f3a34130110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 0x7f3a34137b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34550/0 (socket says 172.21.15.86:34550) 2024-09-17T15:38:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a33fff640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a34136160 msgr2=0x7f3a341380c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:35.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 -- 172.21.15.86:0/922471519 learned_addr learned my addr 172.21.15.86:0/922471519 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:35.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request con 0x7f3a340a1590 auth_method 0 2024-09-17T15:38:35.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 0x7f3a34130110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33708/0 (socket says 172.21.15.86:33708) 2024-09-17T15:38:35.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a33fff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a34136160 0x7f3a341380c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): get_auth_request con 0x7f3a340a0c70 auth_method 0 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient(hunting): handle_auth_done global_id 124182 payload 824 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 msgr2=0x7f3a34130110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 0x7f3a34130110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a34136160 msgr2=0x7f3a341380c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a34136160 0x7f3a341380c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:38:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 monclient: found mon.noname-a 2024-09-17T15:38:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a32ffd640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3413e8e0 con 0x7f3a340a1590 2024-09-17T15:38:35.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.318+0000 7f3a337fe640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a340a0c70 0x7f3a34130110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a32ffd640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 msgr2=0x7f3a34137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a32ffd640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 0x7f3a34137b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b4e0 tx=0x7f3a2800b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28005070 con 0x7f3a340a1590 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: dump: 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.350 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.351 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: _finish_auth 0 2024-09-17T15:38:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:38:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a28015070 con 0x7f3a340a1590 2024-09-17T15:38:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:35.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28005470 con 0x7f3a340a1590 2024-09-17T15:38:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 5 monclient: authenticate success, global_id 124182 2024-09-17T15:38:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 monclient: _renew_subs 2024-09-17T15:38:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3413b9a0 con 0x7f3a340a1590 2024-09-17T15:38:35.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a31ffb640 4 set_mon_vals no callback set 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: dump: 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.354 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.356 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 monclient: _renew_subs 2024-09-17T15:38:35.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:35.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3413b6c0 con 0x7f3a340a1590 2024-09-17T15:38:35.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 0x7f3a34138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a340a1050 con 0x7f3a34130e10 2024-09-17T15:38:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 0x7f3a34138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient: get_auth_request con 0x7f3a34130e10 auth_method 0 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3a28005690 con 0x7f3a340a1590 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 10 monclient(hunting): handle_auth_done global_id 124185 payload 824 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a340008e0 con 0x7f3a34130e10 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a2003bb10 0x7f3a2003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7f3a2804fde0 con 0x7f3a340a1590 2024-09-17T15:38:35.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a33fff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a2003bb10 0x7f3a2003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a33fff640 10 monclient: get_auth_request con 0x7f3a2003bb10 auth_method 0 2024-09-17T15:38:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 msgr2=0x7f3a34138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a337fe640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 0x7f3a34138600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a3406f840 tx=0x7f3a24019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240b2040 con 0x7f3a34130e10 2024-09-17T15:38:35.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: dump: 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.363 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a240b7860 con 0x7f3a34130e10 2024-09-17T15:38:35.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:35.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:35.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 msgr2=0x7f3a34138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 0x7f3a34138600 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a3406f840 tx=0x7f3a24019190 comp rx=0 tx=0).stop 2024-09-17T15:38:35.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 msgr2=0x7f3a34138600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:35.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.322+0000 7f3a1bfff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34130e10 0x7f3a34138600 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3a3406f840 tx=0x7f3a24019190 comp rx=0 tx=0).stop 2024-09-17T15:38:35.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.326+0000 7f3a33fff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a2003bb10 0x7f3a2003dfd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3a34139cb0 tx=0x7f3a2c005130 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a39db8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:35.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a39db8640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 0x7f3a34071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a39db8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3413a1a0 con 0x7f3a34138b40 2024-09-17T15:38:35.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 0x7f3a34071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient: get_auth_request con 0x7f3a34138b40 auth_method 0 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 10 monclient(hunting): handle_auth_done global_id 124188 payload 824 2024-09-17T15:38:35.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.330+0000 7f3a337fe640 1 -- 172.21.15.86:0/922471519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a34138f50 con 0x7f3a34138b40 2024-09-17T15:38:35.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a337fe640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 msgr2=0x7f3a34071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a337fe640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 0x7f3a34071270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a2401cc90 tx=0x7f3a240a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a240b2030 con 0x7f3a34138b40 2024-09-17T15:38:35.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:35.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 10 monclient: dump: 2024-09-17T15:38:35.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f3a24000e50 con 0x7f3a34138b40 2024-09-17T15:38:35.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 msgr2=0x7f3a34071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 0x7f3a34071270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a2401cc90 tx=0x7f3a240a7000 comp rx=0 tx=0).stop 2024-09-17T15:38:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 msgr2=0x7f3a34071270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:35.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a1bfff640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a34138b40 0x7f3a34071270 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3a2401cc90 tx=0x7f3a240a7000 comp rx=0 tx=0).stop 2024-09-17T15:38:35.375 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:35.375 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:38:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 112, 2024-09-17T15:38:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 46199, 2024-09-17T15:38:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:35.376 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:35.377 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:35.380 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.381 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:35.383 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:35.384 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.385 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:38:35.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.389 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:35.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.390 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.390 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:35.390 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:35.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a2003bb10 msgr2=0x7f3a2003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3a2003bb10 0x7f3a2003dfd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3a34139cb0 tx=0x7f3a2c005130 comp rx=0 tx=0).stop 2024-09-17T15:38:35.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 10 monclient: shutdown 2024-09-17T15:38:35.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:35.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 msgr2=0x7f3a34137b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 --2- 172.21.15.86:0/922471519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a340a1590 0x7f3a34137b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a2800b4e0 tx=0x7f3a2800b9a0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a33fff640 1 -- 172.21.15.86:0/922471519 reap_dead start 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a33fff640 1 -- 172.21.15.86:0/922471519 reap_dead start 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 shutdown_connections 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 >> 172.21.15.86:0/922471519 conn(0x7f3a340a2f40 msgr2=0x7f3a34072950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 shutdown_connections 2024-09-17T15:38:35.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.334+0000 7f3a39db8640 1 -- 172.21.15.86:0/922471519 wait complete. 2024-09-17T15:38:35.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:38:35.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:38:35.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: get_monmap_and_config 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: build_initial_monmap 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: monmap: 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:35.500635+0000 2024-09-17T15:38:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:35.500635+0000 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 Processor -- start 2024-09-17T15:38:35.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 -- start start 2024-09-17T15:38:35.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: init 2024-09-17T15:38:35.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5f40a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 0x7fb5f413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f413d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f40a6280 con 0x7fb5f40a54c0 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f409d2a0 con 0x7fb5f413d2f0 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f409c0a0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f8df1640 10 monclient(hunting): authenticate will time out at 1648.466903s 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5ebfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb5f40a54c0 0x7fb5f40a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:35506/0 (socket says 172.21.15.86:35506) 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5ebfff640 1 -- 172.21.15.86:0/3690378381 learned_addr learned my addr 172.21.15.86:0/3690378381 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): get_auth_request con 0x7fb5f40a50e0 auth_method 0 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f2d9d640 1 -- 172.21.15.86:0/3690378381 >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 msgr2=0x7fb5f413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f2d9d640 1 --2- 172.21.15.86:0/3690378381 >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 0x7fb5f413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient(hunting): handle_auth_done global_id 104384 payload 274 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 -- 172.21.15.86:0/3690378381 >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 msgr2=0x7fb5f413f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 --2- 172.21.15.86:0/3690378381 >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 0x7fb5f413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 -- 172.21.15.86:0/3690378381 >> v1:172.21.15.86:6789/0 conn(0x7fb5f40a54c0 legacy=0x7fb5f40a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient: _finish_hunting 0 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 monclient: found mon.noname-b 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:35.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.498+0000 7fb5f259c640 1 -- 172.21.15.86:0/3690378381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f409c380 con 0x7fb5f40a50e0 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f259c640 1 -- 172.21.15.86:0/3690378381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 msgr2=0x7fb5f409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f259c640 1 --2- 172.21.15.86:0/3690378381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f409cd60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0097c0 tx=0x7fb5dc01c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3770c4abbb3d99e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 1 -- 172.21.15.86:0/3690378381 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0b0070 con 0x7fb5f40a50e0 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: dump: 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.505 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: _finish_auth 0 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 1 -- 172.21.15.86:0/3690378381 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0ab600 con 0x7fb5f40a50e0 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f8df1640 5 monclient: authenticate success, global_id 104384 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f8df1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:35.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 1 -- 172.21.15.86:0/3690378381 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0ab9d0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f1d9b640 10 monclient: dump: 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f8df1640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.502+0000 7fb5f8df1640 4 set_mon_vals no callback set 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: shutdown 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- 172.21.15.86:0/3690378381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 msgr2=0x7fb5f409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/3690378381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f409cd60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0097c0 tx=0x7fb5dc01c350 comp rx=0 tx=0).stop 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- 172.21.15.86:0/3690378381 shutdown_connections 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/3690378381 >> v2:172.21.15.203:3300/0 conn(0x7fb5f413d2f0 0x7fb5f413f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/3690378381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f409cd60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- 172.21.15.86:0/3690378381 >> 172.21.15.86:0/3690378381 conn(0x7fb5f40aaae0 msgr2=0x7fb5f40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- 172.21.15.86:0/3690378381 shutdown_connections 2024-09-17T15:38:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- 172.21.15.86:0/3690378381 wait complete. 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: build_initial_monmap 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: monmap: 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:35.512 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 Processor -- start 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- start start 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: init 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f40a54c0 0x7fb5f40a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-a con 0x7fb5f40a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5f413d2f0 0x7fb5f40a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f413d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f4142650 con 0x7fb5f40a54c0 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f40a6ad0 con 0x7fb5f413d2f0 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f41427d0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f8df1640 10 monclient(hunting): authenticate will time out at 1648.474131s 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f40a54c0 0x7fb5f40a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33722/0 (socket says 172.21.15.86:33722) 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 -- 172.21.15.86:0/1089643809 learned_addr learned my addr 172.21.15.86:0/1089643809 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): get_auth_request con 0x7fb5f40a50e0 auth_method 0 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): get_auth_request con 0x7fb5f40a54c0 auth_method 0 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f2d9d640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5f413d2f0 msgr2=0x7fb5f40a2d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f2d9d640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5f413d2f0 0x7fb5f40a2d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient(hunting): handle_auth_done global_id 104387 payload 824 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5f413d2f0 msgr2=0x7fb5f40a2d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5f413d2f0 0x7fb5f40a2d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f40a54c0 msgr2=0x7fb5f40a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5f40a54c0 0x7fb5f40a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient: _finish_hunting 0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 monclient: found mon.noname-c 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.506+0000 7fb5f259c640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f40a3cc0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f259c640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 msgr2=0x7fb5f40a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f259c640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc01cbf0 tx=0x7fb5dc01ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0b0050 con 0x7fb5f40a50e0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: dump: 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.518 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.518 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.518 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: _finish_auth 0 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc0bc070 con 0x7fb5f40a50e0 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 5 monclient: authenticate success, global_id 104387 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0b5c80 con 0x7fb5f40a50e0 2024-09-17T15:38:35.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 monclient: _renew_subs 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f413a5f0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f159a640 4 set_mon_vals no callback set 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 10 monclient: dump: 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 monclient: _renew_subs 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f40a18d0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 0x7fb5f41d0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f409cd60 con 0x7fb5f41cfd30 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb5dc0ab4c0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 0x7fb5f41d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient: get_auth_request con 0x7fb5f41cfd30 auth_method 0 2024-09-17T15:38:35.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5d403bb90 0x7fb5d403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(369..369 src has 213..369) ==== 4815+0+0 (secure 0 0 0) 0x7fb5dc0ea2d0 con 0x7fb5f40a50e0 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_done global_id 104390 payload 824 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5ebfff640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f40008e0 con 0x7fb5f41cfd30 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f2d9d640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5d403bb90 0x7fb5d403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.510+0000 7fb5f2d9d640 10 monclient: get_auth_request con 0x7fb5d403bb90 auth_method 0 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5ebfff640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 msgr2=0x7fb5f41d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5ebfff640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 0x7fb5f41d0110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4139580 tx=0x7fb5e400eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400c840 con 0x7fb5f41cfd30 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 10 monclient: dump: 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.527 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5e4019070 con 0x7fb5f41cfd30 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 msgr2=0x7fb5f41d0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 0x7fb5f41d0110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4139580 tx=0x7fb5e400eb30 comp rx=0 tx=0).stop 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 msgr2=0x7fb5f41d0110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5eaffd640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41cfd30 0x7fb5f41d0110 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4139580 tx=0x7fb5e400eb30 comp rx=0 tx=0).stop 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.514+0000 7fb5f2d9d640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5d403bb90 0x7fb5d403e050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40a8520 tx=0x7fb5ec009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5f8df1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 0x7fb5f409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5f8df1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f4139ee0 con 0x7fb5f41d0650 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 0x7fb5f409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient: get_auth_request con 0x7fb5f41d0650 auth_method 0 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:35.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 10 monclient(hunting): handle_auth_done global_id 104393 payload 824 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.518+0000 7fb5ebfff640 1 -- 172.21.15.86:0/1089643809 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5f413a270 con 0x7fb5f41d0650 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5ebfff640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 msgr2=0x7fb5f409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5ebfff640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 0x7fb5f409e5e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40070c0 tx=0x7fb5e4004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e401ebd0 con 0x7fb5f41d0650 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 10 monclient: dump: 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fb5e401ebd0 con 0x7fb5f41d0650 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 msgr2=0x7fb5f409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 0x7fb5f409e5e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40070c0 tx=0x7fb5e4004950 comp rx=0 tx=0).stop 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 msgr2=0x7fb5f409e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:38:35.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5eaffd640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f41d0650 0x7fb5f409e5e0 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb5e40070c0 tx=0x7fb5e4004950 comp rx=0 tx=0).stop 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 112, 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 146397, 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:38:35.532 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.534 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:38:35.535 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:38:35.536 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:38:35.537 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.538 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:38:35.540 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:38:35.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:38:35.544 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:38:35.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5d403bb90 msgr2=0x7fb5d403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5d403bb90 0x7fb5d403e050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5f40a8520 tx=0x7fb5ec009270 comp rx=0 tx=0).stop 2024-09-17T15:38:35.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 10 monclient: shutdown 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 msgr2=0x7fb5f40a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc01cbf0 tx=0x7fb5dc01ab90 comp rx=0 tx=0).stop 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f2d9d640 1 -- 172.21.15.86:0/1089643809 reap_dead start 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 shutdown_connections 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 --2- 172.21.15.86:0/1089643809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5f40a50e0 0x7fb5f40a2280 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 >> 172.21.15.86:0/1089643809 conn(0x7fb5f40aaae0 msgr2=0x7fb5f413c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 shutdown_connections 2024-09-17T15:38:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:35.522+0000 7fb5f8df1640 1 -- 172.21.15.86:0/1089643809 wait complete. 2024-09-17T15:38:35.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:38:35.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:38:36.398 INFO:tasks.rados.rados.0.smithi203.stdout:1639: done (1 left) 2024-09-17T15:38:36.398 INFO:tasks.rados.rados.0.smithi203.stdout:1640: write oid 23 current snap is 179 2024-09-17T15:38:36.398 INFO:tasks.rados.rados.0.smithi203.stdout:1640: seq_num 600 ranges {484200=624934,1860843=223357} 2024-09-17T15:38:36.416 INFO:tasks.rados.rados.0.smithi203.stdout:1640: writing smithi20328086-23 from 484200 to 1109134 tid 1 2024-09-17T15:38:36.434 INFO:tasks.rados.rados.0.smithi203.stdout:1640: writing smithi20328086-23 from 1860843 to 2084200 tid 2 2024-09-17T15:38:36.434 INFO:tasks.rados.rados.0.smithi203.stdout:1641: write oid 50 current snap is 179 2024-09-17T15:38:36.434 INFO:tasks.rados.rados.0.smithi203.stdout:1641: seq_num 601 ranges {501007=416536,1635678=465329} 2024-09-17T15:38:36.448 INFO:tasks.rados.rados.0.smithi203.stdout:1641: writing smithi20328086-50 from 501007 to 917543 tid 1 2024-09-17T15:38:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:1641: writing smithi20328086-50 from 1635678 to 2101007 tid 2 2024-09-17T15:38:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:1642: snap_create 2024-09-17T15:38:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:1640: finishing write tid 1 to smithi20328086-23 2024-09-17T15:38:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:1640: finishing write tid 2 to smithi20328086-23 2024-09-17T15:38:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:1640: finishing write tid 3 to smithi20328086-23 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1640: finishing write tid 4 to smithi20328086-23 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1640: oid 23 updating version 0 to 593 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1640: oid 23 version 593 is already newer than 592 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1640: oid 23 version 593 is already newer than 589 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 593 (ObjNum 600 snap 179 seq_num 600) dirty exists 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1640: left oid 23 (ObjNum 600 snap 179 seq_num 600) 2024-09-17T15:38:36.464 INFO:tasks.rados.rados.0.smithi203.stdout:1641: finishing write tid 1 to smithi20328086-50 2024-09-17T15:38:36.470 INFO:tasks.rados.rados.0.smithi203.stdout:1641: finishing write tid 2 to smithi20328086-50 2024-09-17T15:38:36.471 INFO:tasks.rados.rados.0.smithi203.stdout:1641: finishing write tid 3 to smithi20328086-50 2024-09-17T15:38:36.472 INFO:tasks.rados.rados.0.smithi203.stdout:1641: finishing write tid 4 to smithi20328086-50 2024-09-17T15:38:36.472 INFO:tasks.rados.rados.0.smithi203.stdout:1641: oid 50 updating version 0 to 319 2024-09-17T15:38:36.472 INFO:tasks.rados.rados.0.smithi203.stdout:1641: oid 50 updating version 319 to 320 2024-09-17T15:38:36.473 INFO:tasks.rados.rados.0.smithi203.stdout:1641: oid 50 version 320 is already newer than 318 2024-09-17T15:38:36.473 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 320 (ObjNum 601 snap 179 seq_num 601) dirty exists 2024-09-17T15:38:36.473 INFO:tasks.rados.rados.0.smithi203.stdout:1641: left oid 50 (ObjNum 601 snap 179 seq_num 601) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1638: done (3 left) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1640: done (2 left) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1641: done (1 left) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1642: done (0 left) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1643: read oid 3 snap -1 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1643: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1644: write (excl) oid 33 current snap is 180 2024-09-17T15:38:37.405 INFO:tasks.rados.rados.0.smithi203.stdout:1644: seq_num 602 ranges {517814=491785,1577807=540007} 2024-09-17T15:38:37.417 INFO:tasks.rados.rados.0.smithi203.stdout:1644: writing smithi20328086-33 from 517814 to 1009599 tid 1 2024-09-17T15:38:37.430 INFO:tasks.rados.rados.0.smithi203.stdout:1644: writing smithi20328086-33 from 1577807 to 2117814 tid 2 2024-09-17T15:38:37.430 INFO:tasks.rados.rados.0.smithi203.stdout:1645: rollback oid 42 current snap is 180 2024-09-17T15:38:37.430 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 169 2024-09-17T15:38:37.430 INFO:tasks.rados.rados.0.smithi203.stdout:1643: expect (ObjNum 496 snap 148 seq_num 496) 2024-09-17T15:38:37.500 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:37.555 INFO:tasks.rados.rados.0.smithi203.stdout:1644: finishing write tid 1 to smithi20328086-33 2024-09-17T15:38:37.555 INFO:tasks.rados.rados.0.smithi203.stdout:1645: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:38:37.555 INFO:tasks.rados.rados.0.smithi203.stdout:1644: finishing write tid 2 to smithi20328086-33 2024-09-17T15:38:37.555 INFO:tasks.rados.rados.0.smithi203.stdout:1645: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: finishing write tid 3 to smithi20328086-33 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: finishing write tid 4 to smithi20328086-33 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: oid 33 updating version 0 to 315 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: oid 33 updating version 315 to 317 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: oid 33 version 317 is already newer than 316 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 317 (ObjNum 602 snap 180 seq_num 602) dirty exists 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: left oid 33 (ObjNum 602 snap 180 seq_num 602) 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1645: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 475 (ObjNum 452 snap 137 seq_num 452) dirty exists 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1643: done (2 left) 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1644: done (1 left) 2024-09-17T15:38:37.556 INFO:tasks.rados.rados.0.smithi203.stdout:1645: done (0 left) 2024-09-17T15:38:37.557 INFO:tasks.rados.rados.0.smithi203.stdout:1646: snap_remove snap 173 2024-09-17T15:38:38.419 INFO:tasks.rados.rados.0.smithi203.stdout:1646: done (0 left) 2024-09-17T15:38:38.419 INFO:tasks.rados.rados.0.smithi203.stdout:1647: write (excl) oid 18 current snap is 180 2024-09-17T15:38:38.420 INFO:tasks.rados.rados.0.smithi203.stdout:1647: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-17T15:38:38.436 INFO:tasks.rados.rados.0.smithi203.stdout:1647: writing smithi20328086-18 from 534621 to 1101655 tid 1 2024-09-17T15:38:38.451 INFO:tasks.rados.rados.0.smithi203.stdout:1647: writing smithi20328086-18 from 1636289 to 2134621 tid 2 2024-09-17T15:38:38.451 INFO:tasks.rados.rados.0.smithi203.stdout:1648: snap_create 2024-09-17T15:38:38.457 INFO:tasks.rados.rados.0.smithi203.stdout:1647: finishing write tid 1 to smithi20328086-18 2024-09-17T15:38:38.458 INFO:tasks.rados.rados.0.smithi203.stdout:1647: finishing write tid 2 to smithi20328086-18 2024-09-17T15:38:38.458 INFO:tasks.rados.rados.0.smithi203.stdout:1647: finishing write tid 3 to smithi20328086-18 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:1647: finishing write tid 4 to smithi20328086-18 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:1647: oid 18 updating version 0 to 546 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:1647: oid 18 updating version 546 to 547 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:1647: oid 18 updating version 547 to 548 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 548 (ObjNum 603 snap 180 seq_num 603) dirty exists 2024-09-17T15:38:38.459 INFO:tasks.rados.rados.0.smithi203.stdout:1647: left oid 18 (ObjNum 603 snap 180 seq_num 603) 2024-09-17T15:38:39.155 INFO:tasks.rados.rados.0.smithi203.stdout:1647: done (1 left) 2024-09-17T15:38:39.156 INFO:tasks.rados.rados.0.smithi203.stdout:1648: done (0 left) 2024-09-17T15:38:39.156 INFO:tasks.rados.rados.0.smithi203.stdout:1649: write oid 9 current snap is 181 2024-09-17T15:38:39.156 INFO:tasks.rados.rados.0.smithi203.stdout:1649: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-17T15:38:39.173 INFO:tasks.rados.rados.0.smithi203.stdout:1649: writing smithi20328086-9 from 551428 to 1193711 tid 1 2024-09-17T15:38:39.189 INFO:tasks.rados.rados.0.smithi203.stdout:1649: writing smithi20328086-9 from 1694771 to 2151428 tid 2 2024-09-17T15:38:39.189 INFO:tasks.rados.rados.0.smithi203.stdout:1650: write oid 26 current snap is 181 2024-09-17T15:38:39.189 INFO:tasks.rados.rados.0.smithi203.stdout:1649: finishing write tid 1 to smithi20328086-9 2024-09-17T15:38:39.189 INFO:tasks.rados.rados.0.smithi203.stdout:1650: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-17T15:38:39.209 INFO:tasks.rados.rados.0.smithi203.stdout:1650: writing smithi20328086-26 from 568235 to 1285767 tid 1 2024-09-17T15:38:39.221 INFO:tasks.rados.rados.0.smithi203.stdout:1650: writing smithi20328086-26 from 1753253 to 2168235 tid 2 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1651: snap_create 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: finishing write tid 2 to smithi20328086-9 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: finishing write tid 3 to smithi20328086-9 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: finishing write tid 4 to smithi20328086-9 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: oid 9 updating version 0 to 552 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: oid 9 updating version 552 to 553 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: oid 9 updating version 553 to 554 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 554 (ObjNum 604 snap 181 seq_num 604) dirty exists 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1649: left oid 9 (ObjNum 604 snap 181 seq_num 604) 2024-09-17T15:38:39.222 INFO:tasks.rados.rados.0.smithi203.stdout:1650: finishing write tid 1 to smithi20328086-26 2024-09-17T15:38:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:1650: finishing write tid 2 to smithi20328086-26 2024-09-17T15:38:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:1650: finishing write tid 3 to smithi20328086-26 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:1650: finishing write tid 4 to smithi20328086-26 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:1650: oid 26 updating version 0 to 731 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:1650: oid 26 updating version 731 to 732 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:1650: oid 26 updating version 732 to 733 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 733 (ObjNum 605 snap 181 seq_num 605) dirty exists 2024-09-17T15:38:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:1650: left oid 26 (ObjNum 605 snap 181 seq_num 605) 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1649: done (2 left) 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1650: done (1 left) 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1651: done (0 left) 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1652: read oid 50 snap 178 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1652: expect (ObjNum 585 snap 174 seq_num 585) 2024-09-17T15:38:40.156 INFO:tasks.rados.rados.0.smithi203.stdout:1653: copy_from oid 34 from oid 33 current snap is 182 2024-09-17T15:38:40.157 INFO:tasks.rados.rados.0.smithi203.stdout:1654: rollback oid 4 current snap is 182 2024-09-17T15:38:40.157 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 177 2024-09-17T15:38:40.157 INFO:tasks.rados.rados.0.smithi203.stdout:1655: snap_remove snap 163 2024-09-17T15:38:41.158 INFO:tasks.rados.rados.0.smithi203.stdout:1655: done (3 left) 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:1656: read oid 44 snap -1 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:1656: expect (ObjNum 524 snap 157 seq_num 524) 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:1657: copy_from oid 14 from oid 11 current snap is 182 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:1654: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 0 (ObjNum 33 snap 0 seq_num 3605878208) dirty dne 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:1652: expect (ObjNum 585 snap 174 seq_num 585) 2024-09-17T15:38:41.159 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:41.298 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 627348 2024-09-17T15:38:41.337 INFO:tasks.rados.rados.0.smithi203.stdout:1653: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:38:41.337 INFO:tasks.rados.rados.0.smithi203.stdout:1653: finishing copy_from to smithi20328086-34 2024-09-17T15:38:41.337 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 317 (ObjNum 602 snap 180 seq_num 602) dirty exists 2024-09-17T15:38:41.338 INFO:tasks.rados.rados.0.smithi203.stdout:1656: expect (ObjNum 524 snap 157 seq_num 524) 2024-09-17T15:38:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1657: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1657: finishing copy_from to smithi20328086-14 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1652: done (4 left) 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1653: done (3 left) 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1654: done (2 left) 2024-09-17T15:38:41.562 INFO:tasks.rados.rados.0.smithi203.stdout:1656: done (1 left) 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1657: done (0 left) 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1658: read oid 42 snap 169 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1658: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1659: read oid 48 snap 170 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1659: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1660: copy_from oid 40 from oid 13 current snap is 182 2024-09-17T15:38:41.563 INFO:tasks.rados.rados.0.smithi203.stdout:1661: snap_create 2024-09-17T15:38:41.564 INFO:tasks.rados.rados.0.smithi203.stdout:1660: finishing copy_from to smithi20328086-40 2024-09-17T15:38:41.564 INFO:tasks.rados.rados.0.smithi203.stdout:1660: got expected ENOENT (src dne) 2024-09-17T15:38:41.564 INFO:tasks.rados.rados.0.smithi203.stdout:1660: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:38:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1659: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:41.715 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:41.766 INFO:tasks.rados.rados.0.smithi203.stdout:1658: expect (ObjNum 452 snap 137 seq_num 452) 2024-09-17T15:38:41.767 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:41.822 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1658: done (3 left) 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1659: done (2 left) 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1660: done (1 left) 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1661: done (0 left) 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1662: read oid 5 snap -1 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1662: expect deleted 2024-09-17T15:38:42.171 INFO:tasks.rados.rados.0.smithi203.stdout:1663: delete oid 15 current snap is 183 2024-09-17T15:38:42.178 INFO:tasks.rados.rados.0.smithi203.stdout:1662: done (1 left) 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1663: done (0 left) 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1664: rollback oid 3 current snap is 183 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 182 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1665: read oid 10 snap -1 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1665: expect deleted 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1666: write oid 5 current snap is 183 2024-09-17T15:38:42.179 INFO:tasks.rados.rados.0.smithi203.stdout:1666: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-17T15:38:42.190 INFO:tasks.rados.rados.0.smithi203.stdout:1666: writing smithi20328086-5 from 601849 to 1069879 tid 1 2024-09-17T15:38:42.211 INFO:tasks.rados.rados.0.smithi203.stdout:1666: writing smithi20328086-5 from 1753864 to 2201849 tid 2 2024-09-17T15:38:42.211 INFO:tasks.rados.rados.0.smithi203.stdout:1667: rollback oid 32 current snap is 183 2024-09-17T15:38:42.211 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 164 2024-09-17T15:38:42.211 INFO:tasks.rados.rados.0.smithi203.stdout:1664: finishing rollback tid 0 to smithi20328086-3 2024-09-17T15:38:42.211 INFO:tasks.rados.rados.0.smithi203.stdout:1664: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:1664: finishing rollback tid 2 to smithi20328086-3 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 483 (ObjNum 496 snap 148 seq_num 496) dirty exists 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:1666: finishing write tid 1 to smithi20328086-5 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:1664: done (3 left) 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:1665: done (2 left) 2024-09-17T15:38:42.212 INFO:tasks.rados.rados.0.smithi203.stdout:1668: rollback oid 2 current snap is 183 2024-09-17T15:38:42.213 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 182 2024-09-17T15:38:42.213 INFO:tasks.rados.rados.0.smithi203.stdout:1669: read oid 9 snap -1 2024-09-17T15:38:42.213 INFO:tasks.rados.rados.0.smithi203.stdout:1669: expect (ObjNum 604 snap 181 seq_num 604) 2024-09-17T15:38:42.213 INFO:tasks.rados.rados.0.smithi203.stdout:1670: snap_remove snap 180 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:1670: done (4 left) 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:1671: delete oid 27 current snap is 183 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:1667: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:1667: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:1669: expect (ObjNum 604 snap 181 seq_num 604) 2024-09-17T15:38:43.176 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 86147 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: finishing write tid 2 to smithi20328086-5 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: finishing write tid 3 to smithi20328086-5 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1668: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: finishing write tid 4 to smithi20328086-5 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: oid 5 updating version 0 to 320 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: oid 5 version 320 is already newer than 319 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: oid 5 version 320 is already newer than 318 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 320 (ObjNum 607 snap 183 seq_num 607) dirty exists 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1666: left oid 5 (ObjNum 607 snap 183 seq_num 607) 2024-09-17T15:38:43.296 INFO:tasks.rados.rados.0.smithi203.stdout:1667: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 323 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1668: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1668: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 468 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1666: done (4 left) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1667: done (3 left) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1668: done (2 left) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1669: done (1 left) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1671: done (0 left) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1672: rollback oid 6 current snap is 183 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 176 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1673: read oid 26 snap -1 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1673: expect (ObjNum 605 snap 181 seq_num 605) 2024-09-17T15:38:43.297 INFO:tasks.rados.rados.0.smithi203.stdout:1674: write (excl) oid 13 current snap is 183 2024-09-17T15:38:43.298 INFO:tasks.rados.rados.0.smithi203.stdout:1674: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-17T15:38:43.318 INFO:tasks.rados.rados.0.smithi203.stdout:1674: writing smithi20328086-13 from 635463 to 1370344 tid 1 2024-09-17T15:38:43.333 INFO:tasks.rados.rados.0.smithi203.stdout:1674: writing smithi20328086-13 from 1987181 to 2235463 tid 2 2024-09-17T15:38:43.333 INFO:tasks.rados.rados.0.smithi203.stdout:1675: copy_from oid 21 from oid 29 current snap is 183 2024-09-17T15:38:43.333 INFO:tasks.rados.rados.0.smithi203.stdout:1672: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:38:43.333 INFO:tasks.rados.rados.0.smithi203.stdout:1672: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:38:43.333 INFO:tasks.rados.rados.0.smithi203.stdout:1673: expect (ObjNum 605 snap 181 seq_num 605) 2024-09-17T15:38:43.381 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1591054 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1672: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 614 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1674: finishing write tid 1 to smithi20328086-13 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1672: done (3 left) 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1673: done (2 left) 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1676: copy_from oid 48 from oid 45 current snap is 183 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1674: finishing write tid 2 to smithi20328086-13 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1674: finishing write tid 3 to smithi20328086-13 2024-09-17T15:38:43.475 INFO:tasks.rados.rados.0.smithi203.stdout:1674: finishing write tid 4 to smithi20328086-13 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1674: oid 13 updating version 0 to 672 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1674: oid 13 updating version 672 to 673 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1674: oid 13 updating version 673 to 674 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 674 (ObjNum 609 snap 183 seq_num 609) dirty exists 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1674: left oid 13 (ObjNum 609 snap 183 seq_num 609) 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1675: finishing copy_from to smithi20328086-21 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1675: got expected ENOENT (src dne) 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1675: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1674: done (2 left) 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1675: done (1 left) 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1677: rollback oid 21 current snap is 183 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 181 2024-09-17T15:38:43.476 INFO:tasks.rados.rados.0.smithi203.stdout:1678: copy_from oid 13 from oid 34 current snap is 183 2024-09-17T15:38:43.477 INFO:tasks.rados.rados.0.smithi203.stdout:1679: snap_create 2024-09-17T15:38:43.477 INFO:tasks.rados.rados.0.smithi203.stdout:1676: finishing copy_from to smithi20328086-48 2024-09-17T15:38:43.477 INFO:tasks.rados.rados.0.smithi203.stdout:1676: got expected ENOENT (src dne) 2024-09-17T15:38:43.477 INFO:tasks.rados.rados.0.smithi203.stdout:1676: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:38:43.485 INFO:tasks.rados.rados.0.smithi203.stdout:1677: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:38:43.485 INFO:tasks.rados.rados.0.smithi203.stdout:1677: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:38:43.485 INFO:tasks.rados.rados.0.smithi203.stdout:1677: finishing rollback tid 2 to smithi20328086-21 2024-09-17T15:38:43.486 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:38:43.492 INFO:tasks.rados.rados.0.smithi203.stdout:1678: finishing copy_from racing read to smithi20328086-13 2024-09-17T15:38:43.496 INFO:tasks.rados.rados.0.smithi203.stdout:1678: finishing copy_from to smithi20328086-13 2024-09-17T15:38:43.497 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 317 (ObjNum 602 snap 180 seq_num 602) dirty exists 2024-09-17T15:38:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:1676: done (3 left) 2024-09-17T15:38:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1677: done (2 left) 2024-09-17T15:38:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1678: done (1 left) 2024-09-17T15:38:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1679: done (0 left) 2024-09-17T15:38:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1680: write oid 44 current snap is 184 2024-09-17T15:38:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1680: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-17T15:38:44.176 INFO:tasks.rados.rados.0.smithi203.stdout:1680: writing smithi20328086-44 from 652270 to 1062400 tid 1 2024-09-17T15:38:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1680: writing smithi20328086-44 from 1529310 to 2023177 tid 2 2024-09-17T15:38:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1681: write oid 14 current snap is 184 2024-09-17T15:38:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1681: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-17T15:38:44.212 INFO:tasks.rados.rados.0.smithi203.stdout:1681: writing smithi20328086-14 from 669077 to 1154456 tid 1 2024-09-17T15:38:44.227 INFO:tasks.rados.rados.0.smithi203.stdout:1681: writing smithi20328086-14 from 1587792 to 2269077 tid 2 2024-09-17T15:38:44.227 INFO:tasks.rados.rados.0.smithi203.stdout:1682: snap_create 2024-09-17T15:38:44.227 INFO:tasks.rados.rados.0.smithi203.stdout:1680: finishing write tid 1 to smithi20328086-44 2024-09-17T15:38:44.228 INFO:tasks.rados.rados.0.smithi203.stdout:1680: finishing write tid 2 to smithi20328086-44 2024-09-17T15:38:44.228 INFO:tasks.rados.rados.0.smithi203.stdout:1680: finishing write tid 3 to smithi20328086-44 2024-09-17T15:38:44.228 INFO:tasks.rados.rados.0.smithi203.stdout:1680: finishing write tid 4 to smithi20328086-44 2024-09-17T15:38:44.228 INFO:tasks.rados.rados.0.smithi203.stdout:1680: oid 44 updating version 0 to 653 2024-09-17T15:38:44.228 INFO:tasks.rados.rados.0.smithi203.stdout:1680: oid 44 version 653 is already newer than 651 2024-09-17T15:38:44.229 INFO:tasks.rados.rados.0.smithi203.stdout:1680: oid 44 version 653 is already newer than 652 2024-09-17T15:38:44.229 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 653 (ObjNum 610 snap 184 seq_num 610) dirty exists 2024-09-17T15:38:44.229 INFO:tasks.rados.rados.0.smithi203.stdout:1680: left oid 44 (ObjNum 610 snap 184 seq_num 610) 2024-09-17T15:38:44.229 INFO:tasks.rados.rados.0.smithi203.stdout:1681: finishing write tid 1 to smithi20328086-14 2024-09-17T15:38:44.236 INFO:tasks.rados.rados.0.smithi203.stdout:1681: finishing write tid 2 to smithi20328086-14 2024-09-17T15:38:44.236 INFO:tasks.rados.rados.0.smithi203.stdout:1681: finishing write tid 3 to smithi20328086-14 2024-09-17T15:38:44.237 INFO:tasks.rados.rados.0.smithi203.stdout:1681: finishing write tid 4 to smithi20328086-14 2024-09-17T15:38:44.237 INFO:tasks.rados.rados.0.smithi203.stdout:1681: oid 14 updating version 0 to 616 2024-09-17T15:38:44.237 INFO:tasks.rados.rados.0.smithi203.stdout:1681: oid 14 updating version 616 to 617 2024-09-17T15:38:44.238 INFO:tasks.rados.rados.0.smithi203.stdout:1681: oid 14 version 617 is already newer than 615 2024-09-17T15:38:44.238 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 617 (ObjNum 611 snap 184 seq_num 611) dirty exists 2024-09-17T15:38:44.238 INFO:tasks.rados.rados.0.smithi203.stdout:1681: left oid 14 (ObjNum 611 snap 184 seq_num 611) 2024-09-17T15:38:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:1680: done (2 left) 2024-09-17T15:38:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:1681: done (1 left) 2024-09-17T15:38:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:1682: done (0 left) 2024-09-17T15:38:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:1683: write oid 7 current snap is 185 2024-09-17T15:38:45.200 INFO:tasks.rados.rados.0.smithi203.stdout:1683: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-17T15:38:45.211 INFO:tasks.rados.rados.0.smithi203.stdout:1683: writing smithi20328086-7 from 685884 to 1246512 tid 1 2024-09-17T15:38:45.222 INFO:tasks.rados.rados.0.smithi203.stdout:1683: writing smithi20328086-7 from 2046274 to 2285884 tid 2 2024-09-17T15:38:45.223 INFO:tasks.rados.rados.0.smithi203.stdout:1684: snap_create 2024-09-17T15:38:45.229 INFO:tasks.rados.rados.0.smithi203.stdout:1683: finishing write tid 1 to smithi20328086-7 2024-09-17T15:38:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:1683: finishing write tid 2 to smithi20328086-7 2024-09-17T15:38:45.230 INFO:tasks.rados.rados.0.smithi203.stdout:1683: finishing write tid 3 to smithi20328086-7 2024-09-17T15:38:45.231 INFO:tasks.rados.rados.0.smithi203.stdout:1683: finishing write tid 4 to smithi20328086-7 2024-09-17T15:38:45.232 INFO:tasks.rados.rados.0.smithi203.stdout:1683: oid 7 updating version 0 to 469 2024-09-17T15:38:45.232 INFO:tasks.rados.rados.0.smithi203.stdout:1683: oid 7 updating version 469 to 470 2024-09-17T15:38:45.232 INFO:tasks.rados.rados.0.smithi203.stdout:1683: oid 7 updating version 470 to 471 2024-09-17T15:38:45.232 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 471 (ObjNum 612 snap 185 seq_num 612) dirty exists 2024-09-17T15:38:45.232 INFO:tasks.rados.rados.0.smithi203.stdout:1683: left oid 7 (ObjNum 612 snap 185 seq_num 612) 2024-09-17T15:38:46.205 INFO:tasks.rados.rados.0.smithi203.stdout:1683: done (1 left) 2024-09-17T15:38:46.206 INFO:tasks.rados.rados.0.smithi203.stdout:1684: done (0 left) 2024-09-17T15:38:46.206 INFO:tasks.rados.rados.0.smithi203.stdout:1685: write (excl) oid 40 current snap is 186 2024-09-17T15:38:46.206 INFO:tasks.rados.rados.0.smithi203.stdout:1685: seq_num 613 ranges {702691=635877,2104756=197935} 2024-09-17T15:38:46.221 INFO:tasks.rados.rados.0.smithi203.stdout:1685: writing smithi20328086-40 from 702691 to 1338568 tid 1 2024-09-17T15:38:46.232 INFO:tasks.rados.rados.0.smithi203.stdout:1685: writing smithi20328086-40 from 2104756 to 2302691 tid 2 2024-09-17T15:38:46.232 INFO:tasks.rados.rados.0.smithi203.stdout:1686: read oid 37 snap -1 2024-09-17T15:38:46.232 INFO:tasks.rados.rados.0.smithi203.stdout:1686: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:38:46.233 INFO:tasks.rados.rados.0.smithi203.stdout:1687: snap_remove snap 181 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1687: done (2 left) 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1688: read oid 32 snap -1 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1688: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1689: snap_create 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1685: finishing write tid 1 to smithi20328086-40 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1685: finishing write tid 2 to smithi20328086-40 2024-09-17T15:38:47.210 INFO:tasks.rados.rados.0.smithi203.stdout:1685: finishing write tid 3 to smithi20328086-40 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1685: finishing write tid 4 to smithi20328086-40 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1685: oid 40 updating version 0 to 655 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1685: oid 40 updating version 655 to 656 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1685: oid 40 updating version 656 to 657 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 657 (ObjNum 613 snap 186 seq_num 613) dirty exists 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1685: left oid 40 (ObjNum 613 snap 186 seq_num 613) 2024-09-17T15:38:47.211 INFO:tasks.rados.rados.0.smithi203.stdout:1686: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:38:47.444 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2497029 2024-09-17T15:38:47.477 INFO:tasks.rados.rados.0.smithi203.stdout:1688: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:47.478 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 136185 2024-09-17T15:38:47.568 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:48.212 INFO:tasks.rados.rados.0.smithi203.stdout:1685: done (3 left) 2024-09-17T15:38:48.212 INFO:tasks.rados.rados.0.smithi203.stdout:1686: done (2 left) 2024-09-17T15:38:48.213 INFO:tasks.rados.rados.0.smithi203.stdout:1688: done (1 left) 2024-09-17T15:38:48.213 INFO:tasks.rados.rados.0.smithi203.stdout:1689: done (0 left) 2024-09-17T15:38:48.213 INFO:tasks.rados.rados.0.smithi203.stdout:1690: rollback oid 15 current snap is 187 2024-09-17T15:38:48.213 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 178 2024-09-17T15:38:48.214 INFO:tasks.rados.rados.0.smithi203.stdout:1691: read oid 39 snap -1 2024-09-17T15:38:48.214 INFO:tasks.rados.rados.0.smithi203.stdout:1691: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:48.214 INFO:tasks.rados.rados.0.smithi203.stdout:1692: write oid 23 current snap is 187 2024-09-17T15:38:48.214 INFO:tasks.rados.rados.0.smithi203.stdout:1692: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-17T15:38:48.236 INFO:tasks.rados.rados.0.smithi203.stdout:1692: writing smithi20328086-23 from 719498 to 1430624 tid 1 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1692: writing smithi20328086-23 from 2046885 to 2319498 tid 2 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1693: read oid 12 snap 178 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1693: expect (ObjNum 544 snap 161 seq_num 544) 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1694: read oid 49 snap -1 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1694: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1695: read oid 28 snap 169 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1695: expect (ObjNum 567 snap 168 seq_num 567) 2024-09-17T15:38:48.252 INFO:tasks.rados.rados.0.smithi203.stdout:1696: snap_remove snap 176 2024-09-17T15:38:49.158 INFO:tasks.rados.rados.0.smithi203.stdout:1690: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:38:49.158 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 396 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:38:49.158 INFO:tasks.rados.rados.0.smithi203.stdout:1691: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:49.158 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:49.227 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:49.343 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3394804 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1692: finishing write tid 1 to smithi20328086-23 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1690: done (6 left) 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1691: done (5 left) 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1696: done (4 left) 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1697: write oid 26 current snap is 187 2024-09-17T15:38:49.344 INFO:tasks.rados.rados.0.smithi203.stdout:1697: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-17T15:38:49.351 INFO:tasks.rados.rados.0.smithi203.stdout:1697: writing smithi20328086-26 from 736305 to 1522680 tid 1 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1697: writing smithi20328086-26 from 2105367 to 2336305 tid 2 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1698: read oid 39 snap -1 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1698: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1699: rollback oid 40 current snap is 187 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 184 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1700: snap_create 2024-09-17T15:38:49.356 INFO:tasks.rados.rados.0.smithi203.stdout:1693: expect (ObjNum 544 snap 161 seq_num 544) 2024-09-17T15:38:49.500 INFO:tasks.rados.rados.0.smithi203.stdout:1692: finishing write tid 2 to smithi20328086-23 2024-09-17T15:38:49.500 INFO:tasks.rados.rados.0.smithi203.stdout:1694: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:49.565 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:49.649 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1731285 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: finishing write tid 3 to smithi20328086-23 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: finishing write tid 4 to smithi20328086-23 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: oid 23 updating version 0 to 625 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: oid 23 version 625 is already newer than 624 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: oid 23 version 625 is already newer than 623 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 625 (ObjNum 614 snap 187 seq_num 614) dirty exists 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1692: left oid 23 (ObjNum 614 snap 187 seq_num 614) 2024-09-17T15:38:49.682 INFO:tasks.rados.rados.0.smithi203.stdout:1695: expect (ObjNum 567 snap 168 seq_num 567) 2024-09-17T15:38:49.769 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:49.819 INFO:tasks.rados.rados.0.smithi203.stdout:1698: expect (ObjNum 562 snap 167 seq_num 562) 2024-09-17T15:38:49.950 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1218027 2024-09-17T15:38:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1699: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:38:49.996 INFO:tasks.rados.rados.0.smithi203.stdout:1699: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:38:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1699: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:38:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 669 (ObjNum 587 snap 174 seq_num 587) dirty exists 2024-09-17T15:38:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1697: finishing write tid 1 to smithi20328086-26 2024-09-17T15:38:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1697: finishing write tid 2 to smithi20328086-26 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: finishing write tid 3 to smithi20328086-26 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: finishing write tid 4 to smithi20328086-26 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: oid 26 updating version 0 to 759 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: oid 26 version 759 is already newer than 758 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: oid 26 version 759 is already newer than 757 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 759 (ObjNum 615 snap 187 seq_num 615) dirty exists 2024-09-17T15:38:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1697: left oid 26 (ObjNum 615 snap 187 seq_num 615) 2024-09-17T15:38:50.239 INFO:tasks.rados.rados.0.smithi203.stdout:1692: done (7 left) 2024-09-17T15:38:50.239 INFO:tasks.rados.rados.0.smithi203.stdout:1693: done (6 left) 2024-09-17T15:38:50.239 INFO:tasks.rados.rados.0.smithi203.stdout:1694: done (5 left) 2024-09-17T15:38:50.239 INFO:tasks.rados.rados.0.smithi203.stdout:1695: done (4 left) 2024-09-17T15:38:50.239 INFO:tasks.rados.rados.0.smithi203.stdout:1697: done (3 left) 2024-09-17T15:38:50.240 INFO:tasks.rados.rados.0.smithi203.stdout:1698: done (2 left) 2024-09-17T15:38:50.240 INFO:tasks.rados.rados.0.smithi203.stdout:1699: done (1 left) 2024-09-17T15:38:50.240 INFO:tasks.rados.rados.0.smithi203.stdout:1700: done (0 left) 2024-09-17T15:38:50.240 INFO:tasks.rados.rados.0.smithi203.stdout:1701: snap_remove snap 164 2024-09-17T15:38:51.240 INFO:tasks.rados.rados.0.smithi203.stdout:1701: done (0 left) 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1702: rollback oid 32 current snap is 188 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 187 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1703: read oid 31 snap -1 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1703: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1704: read oid 22 snap -1 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1704: expect (ObjNum 596 snap 178 seq_num 596) 2024-09-17T15:38:51.241 INFO:tasks.rados.rados.0.smithi203.stdout:1705: snap_create 2024-09-17T15:38:51.262 INFO:tasks.rados.rados.0.smithi203.stdout:1702: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:38:51.272 INFO:tasks.rados.rados.0.smithi203.stdout:1702: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:38:51.272 INFO:tasks.rados.rados.0.smithi203.stdout:1702: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:38:51.272 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 326 (ObjNum 504 snap 149 seq_num 504) dirty exists 2024-09-17T15:38:51.274 INFO:tasks.rados.rados.0.smithi203.stdout:1704: expect (ObjNum 596 snap 178 seq_num 596) 2024-09-17T15:38:51.377 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 29592 2024-09-17T15:38:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:1703: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:38:51.420 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:51.444 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1104769 2024-09-17T15:38:51.457 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:52.615 INFO:tasks.rados.rados.0.smithi203.stdout:1702: done (3 left) 2024-09-17T15:38:52.615 INFO:tasks.rados.rados.0.smithi203.stdout:1703: done (2 left) 2024-09-17T15:38:52.615 INFO:tasks.rados.rados.0.smithi203.stdout:1704: done (1 left) 2024-09-17T15:38:52.615 INFO:tasks.rados.rados.0.smithi203.stdout:1705: done (0 left) 2024-09-17T15:38:52.615 INFO:tasks.rados.rados.0.smithi203.stdout:1706: delete oid 24 current snap is 189 2024-09-17T15:38:52.622 INFO:tasks.rados.rados.0.smithi203.stdout:1706: done (0 left) 2024-09-17T15:38:52.622 INFO:tasks.rados.rados.0.smithi203.stdout:1707: read oid 27 snap -1 2024-09-17T15:38:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:1707: expect deleted 2024-09-17T15:38:52.623 INFO:tasks.rados.rados.0.smithi203.stdout:1708: snap_create 2024-09-17T15:38:53.621 INFO:tasks.rados.rados.0.smithi203.stdout:1707: done (1 left) 2024-09-17T15:38:53.622 INFO:tasks.rados.rados.0.smithi203.stdout:1708: done (0 left) 2024-09-17T15:38:53.622 INFO:tasks.rados.rados.0.smithi203.stdout:1709: snap_remove snap 175 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1709: done (0 left) 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1710: read oid 34 snap -1 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1710: expect (ObjNum 602 snap 180 seq_num 602) 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1711: read oid 48 snap -1 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1711: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1712: rollback oid 23 current snap is 190 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 179 2024-09-17T15:38:54.159 INFO:tasks.rados.rados.0.smithi203.stdout:1713: delete oid 20 current snap is 190 2024-09-17T15:38:54.174 INFO:tasks.rados.rados.0.smithi203.stdout:1711: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:54.271 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:54.322 INFO:tasks.rados.rados.0.smithi203.stdout:1712: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:38:54.323 INFO:tasks.rados.rados.0.smithi203.stdout:1710: expect (ObjNum 602 snap 180 seq_num 602) 2024-09-17T15:38:54.323 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 411384 2024-09-17T15:38:54.425 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1664633 2024-09-17T15:38:54.429 INFO:tasks.rados.rados.0.smithi203.stdout:1712: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:38:54.429 INFO:tasks.rados.rados.0.smithi203.stdout:1712: finishing rollback tid 2 to smithi20328086-23 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 643 (ObjNum 600 snap 179 seq_num 600) dirty exists 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1710: done (3 left) 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1711: done (2 left) 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1712: done (1 left) 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1713: done (0 left) 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1714: read oid 14 snap -1 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1714: expect (ObjNum 611 snap 184 seq_num 611) 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1715: rollback oid 47 current snap is 190 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 178 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1716: rollback oid 33 current snap is 190 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 171 2024-09-17T15:38:54.430 INFO:tasks.rados.rados.0.smithi203.stdout:1717: delete oid 17 current snap is 190 2024-09-17T15:38:54.436 INFO:tasks.rados.rados.0.smithi203.stdout:1714: expect (ObjNum 611 snap 184 seq_num 611) 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1715: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1716: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1716: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1715: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1715: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 551 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1716: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 323 (ObjNum 522 snap 157 seq_num 522) dirty exists 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1714: done (3 left) 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1715: done (2 left) 2024-09-17T15:38:54.578 INFO:tasks.rados.rados.0.smithi203.stdout:1716: done (1 left) 2024-09-17T15:38:54.579 INFO:tasks.rados.rados.0.smithi203.stdout:1717: done (0 left) 2024-09-17T15:38:54.579 INFO:tasks.rados.rados.0.smithi203.stdout:1718: snap_create 2024-09-17T15:38:55.173 INFO:tasks.rados.rados.0.smithi203.stdout:1718: done (0 left) 2024-09-17T15:38:55.173 INFO:tasks.rados.rados.0.smithi203.stdout:1719: snap_create 2024-09-17T15:38:55.548 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:38:55.549 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:38:55.549 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:38:55.552 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:38:55.552 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:38:55.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:55.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: get_monmap_and_config 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: build_initial_monmap 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: monmap: 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:55.681108+0000 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:55.681108+0000 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:55.678 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:55.679 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:55.679 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:55.679 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:55.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 Processor -- start 2024-09-17T15:38:55.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 -- start start 2024-09-17T15:38:55.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: init 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59700a5070 addr v1:172.21.15.203:6789/0 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-b con 0x7f59700a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f597013d280 0x7f597013f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-c con 0x7f597013d280 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59700a6f40 con 0x7f59700a5070 2024-09-17T15:38:55.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f59700a65e0 con 0x7f597013d280 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f597009c2a0 con 0x7f59700a5b60 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f5976f0f640 10 monclient(hunting): authenticate will time out at 1668.646586s 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40594/0 (socket says 172.21.15.86:40594) 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 -- 172.21.15.86:0/2024652113 learned_addr learned my addr 172.21.15.86:0/2024652113 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): get_auth_request con 0x7f59700a5b60 auth_method 0 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f59754ac640 1 --2- 172.21.15.86:0/2024652113 >> v2:172.21.15.203:3300/0 conn(0x7f597013d280 0x7f597013f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f59754ac640 10 monclient(hunting): get_auth_request con 0x7f597013d280 auth_method 0 2024-09-17T15:38:55.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f59754ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f59754ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f59754ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient(hunting): handle_auth_done global_id 104396 payload 274 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 -- 172.21.15.86:0/2024652113 >> v2:172.21.15.203:3300/0 conn(0x7f597013d280 msgr2=0x7f597013f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 --2- 172.21.15.86:0/2024652113 >> v2:172.21.15.203:3300/0 conn(0x7f597013d280 0x7f597013f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 -- 172.21.15.86:0/2024652113 >> v1:172.21.15.203:6789/0 conn(0x7f59700a5070 legacy=0x7f59700a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 monclient: found mon.noname-b 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:38:55.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.678+0000 7f596ffff640 1 -- 172.21.15.86:0/2024652113 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597009c580 con 0x7f59700a5b60 2024-09-17T15:38:55.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596ffff640 1 -- 172.21.15.86:0/2024652113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 msgr2=0x7f597009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596ffff640 1 --2- 172.21.15.86:0/2024652113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5960009870 tx=0x7f596001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d04c9ea16b61f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 1 -- 172.21.15.86:0/2024652113 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600b0070 con 0x7f59700a5b60 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: dump: 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 1 -- 172.21.15.86:0/2024652113 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59600ab600 con 0x7f59700a5b60 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 1 -- 172.21.15.86:0/2024652113 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600ab9d0 con 0x7f59700a5b60 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 5 monclient: authenticate success, global_id 104396 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 4 set_mon_vals no callback set 2024-09-17T15:38:55.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f596f7fe640 10 monclient: dump: 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 10 monclient: shutdown 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 1 -- 172.21.15.86:0/2024652113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 msgr2=0x7f597009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.682+0000 7f5976f0f640 1 --2- 172.21.15.86:0/2024652113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5960009870 tx=0x7f596001c420 comp rx=0 tx=0).stop 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- 172.21.15.86:0/2024652113 shutdown_connections 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 --2- 172.21.15.86:0/2024652113 >> v2:172.21.15.203:3300/0 conn(0x7f597013d280 0x7f597013f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 --2- 172.21.15.86:0/2024652113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59700a5b60 0x7f597009cf60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- 172.21.15.86:0/2024652113 >> 172.21.15.86:0/2024652113 conn(0x7f59700742a0 msgr2=0x7f59700746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- 172.21.15.86:0/2024652113 shutdown_connections 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- 172.21.15.86:0/2024652113 wait complete. 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient: build_initial_monmap 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient: monmap: 2024-09-17T15:38:55.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 Processor -- start 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- start start 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient: init 2024-09-17T15:38:55.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700a5070 0x7f5970137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-b con 0x7f59700a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-a con 0x7f59700a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): picked mon.noname-c con 0x7f597013d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5970142760 con 0x7f59700a5b60 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59701425e0 con 0x7f59700a5070 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:55.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59701428e0 con 0x7f597013d280 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5976f0f640 10 monclient(hunting): authenticate will time out at 1668.653578s 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:47300/0 (socket says 172.21.15.86:47300) 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 -- 172.21.15.86:0/3575579983 learned_addr learned my addr 172.21.15.86:0/3575579983 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): get_auth_request con 0x7f59700a5b60 auth_method 0 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40598/0 (socket says 172.21.15.86:40598) 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): get_auth_request con 0x7f597013d280 auth_method 0 2024-09-17T15:38:55.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5974cab640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700a5070 0x7f5970137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5974cab640 10 monclient(hunting): get_auth_request con 0x7f59700a5070 auth_method 0 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5974cab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5974cab640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:55.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f5974cab640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient(hunting): handle_auth_done global_id 124194 payload 824 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 -- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 msgr2=0x7f597013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 -- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700a5070 msgr2=0x7f5970137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700a5070 0x7f5970137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 monclient: found mon.noname-a 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f596ffff640 1 -- 172.21.15.86:0/3575579983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5970122850 con 0x7f59700a5b60 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.686+0000 7f59754ac640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596ffff640 1 -- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 msgr2=0x7f5970137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:55.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596ffff640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59600ae040 tx=0x7f59600191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600b0040 con 0x7f59700a5b60 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: dump: 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:55.699 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59600aa5b0 con 0x7f59700a5b60 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 5 monclient: authenticate success, global_id 124194 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600aa980 con 0x7f59700a5b60 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 10 monclient: dump: 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:55.700 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596effd640 4 set_mon_vals no callback set 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _renew_subs 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5970122570 con 0x7f59700a5b60 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _renew_subs 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:55.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5970138410 con 0x7f59700a5b60 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597009cf60 con 0x7f59700a5b60 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f59600aab20 con 0x7f59700a5b60 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 0x7f595c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(389..389 src has 213..389) ==== 4755+0+0 (secure 0 0 0) 0x7f59600e9d20 con 0x7f59700a5b60 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5974cab640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 0x7f595c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:55.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.690+0000 7f5974cab640 10 monclient: get_auth_request con 0x7f595c03bb60 auth_method 0 2024-09-17T15:38:55.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.694+0000 7f5974cab640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 0x7f595c03e020 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5964004e30 tx=0x7f5964009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:55.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.694+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5960016130 con 0x7f59700a5b60 2024-09-17T15:38:55.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.694+0000 7f596d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:55.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.694+0000 7f596d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.926+0000 7f5976f0f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.926+0000 7f5976f0f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.926+0000 7f5976f0f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.926+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5970069920 con 0x7f59700a5b60 2024-09-17T15:38:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f596d7fa640 1 -- 172.21.15.86:0/3575579983 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f59600aae10 con 0x7f59700a5b60 2024-09-17T15:38:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f596d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f596d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:55.929 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":112,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 msgr2=0x7f595c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 0x7f595c03e020 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5964004e30 tx=0x7f5964009040 comp rx=0 tx=0).stop 2024-09-17T15:38:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 10 monclient: shutdown 2024-09-17T15:38:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 msgr2=0x7f5970137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f59600ae040 tx=0x7f59600191c0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 shutdown_connections 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f595c03bb60 0x7f595c03e020 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f597013d280 0x7f597013aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59700a5b60 0x7f5970137ed0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 --2- 172.21.15.86:0/3575579983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700a5070 0x7f5970137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 >> 172.21.15.86:0/3575579983 conn(0x7f59700742a0 msgr2=0x7f59700a04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 shutdown_connections 2024-09-17T15:38:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:55.930+0000 7f5976f0f640 1 -- 172.21.15.86:0/3575579983 wait complete. 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1719: done (0 left) 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1720: rollback oid 33 current snap is 192 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 184 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1721: read oid 32 snap -1 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1721: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1722: read oid 17 snap -1 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1722: expect deleted 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1723: write (excl) oid 3 current snap is 192 2024-09-17T15:38:56.177 INFO:tasks.rados.rados.0.smithi203.stdout:1723: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-17T15:38:56.181 INFO:tasks.rados.rados.0.smithi203.stdout:1723: writing smithi20328086-3 from 403533 to 807257 tid 1 2024-09-17T15:38:56.190 INFO:tasks.rados.rados.0.smithi203.stdout:1723: writing smithi20328086-3 from 1539295 to 2191496 tid 2 2024-09-17T15:38:56.190 INFO:tasks.rados.rados.0.smithi203.stdout:1724: write oid 1 current snap is 192 2024-09-17T15:38:56.190 INFO:tasks.rados.rados.0.smithi203.stdout:1724: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-17T15:38:56.195 INFO:tasks.rados.rados.0.smithi203.stdout:1724: writing smithi20328086-1 from 420340 to 899313 tid 1 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1724: writing smithi20328086-1 from 1597777 to 2109989 tid 2 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1725: delete oid 36 current snap is 192 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1720: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1720: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1720: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 326 (ObjNum 602 snap 180 seq_num 602) dirty exists 2024-09-17T15:38:56.208 INFO:tasks.rados.rados.0.smithi203.stdout:1721: expect (ObjNum 504 snap 149 seq_num 504) 2024-09-17T15:38:56.284 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 711863 2024-09-17T15:38:56.325 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:56.377 INFO:tasks.rados.rados.0.smithi203.stdout:1723: finishing write tid 1 to smithi20328086-3 2024-09-17T15:38:56.377 INFO:tasks.rados.rados.0.smithi203.stdout:1723: finishing write tid 2 to smithi20328086-3 2024-09-17T15:38:56.377 INFO:tasks.rados.rados.0.smithi203.stdout:1723: finishing write tid 3 to smithi20328086-3 2024-09-17T15:38:56.377 INFO:tasks.rados.rados.0.smithi203.stdout:1723: finishing write tid 4 to smithi20328086-3 2024-09-17T15:38:56.377 INFO:tasks.rados.rados.0.smithi203.stdout:1723: oid 3 updating version 0 to 504 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1723: oid 3 version 504 is already newer than 502 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1723: oid 3 version 504 is already newer than 503 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 504 (ObjNum 619 snap 192 seq_num 619) dirty exists 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1723: left oid 3 (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: finishing write tid 1 to smithi20328086-1 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: finishing write tid 2 to smithi20328086-1 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: finishing write tid 3 to smithi20328086-1 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: finishing write tid 4 to smithi20328086-1 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: oid 1 updating version 0 to 656 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: oid 1 updating version 656 to 657 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: oid 1 version 657 is already newer than 655 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 657 (ObjNum 620 snap 192 seq_num 620) dirty exists 2024-09-17T15:38:56.378 INFO:tasks.rados.rados.0.smithi203.stdout:1724: left oid 1 (ObjNum 620 snap 192 seq_num 620) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1720: done (5 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1721: done (4 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1722: done (3 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1723: done (2 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1724: done (1 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1725: done (0 left) 2024-09-17T15:38:56.379 INFO:tasks.rados.rados.0.smithi203.stdout:1726: snap_create 2024-09-17T15:38:57.186 INFO:tasks.rados.rados.0.smithi203.stdout:1726: done (0 left) 2024-09-17T15:38:57.187 INFO:tasks.rados.rados.0.smithi203.stdout:1727: read oid 36 snap -1 2024-09-17T15:38:57.187 INFO:tasks.rados.rados.0.smithi203.stdout:1727: expect deleted 2024-09-17T15:38:57.187 INFO:tasks.rados.rados.0.smithi203.stdout:1728: read oid 7 snap -1 2024-09-17T15:38:57.187 INFO:tasks.rados.rados.0.smithi203.stdout:1728: expect (ObjNum 612 snap 185 seq_num 612) 2024-09-17T15:38:57.187 INFO:tasks.rados.rados.0.smithi203.stdout:1729: snap_create 2024-09-17T15:38:57.193 INFO:tasks.rados.rados.0.smithi203.stdout:1728: expect (ObjNum 612 snap 185 seq_num 612) 2024-09-17T15:38:57.237 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 993069 2024-09-17T15:38:57.259 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:38:58.194 INFO:tasks.rados.rados.0.smithi203.stdout:1727: done (2 left) 2024-09-17T15:38:58.194 INFO:tasks.rados.rados.0.smithi203.stdout:1728: done (1 left) 2024-09-17T15:38:58.195 INFO:tasks.rados.rados.0.smithi203.stdout:1729: done (0 left) 2024-09-17T15:38:58.195 INFO:tasks.rados.rados.0.smithi203.stdout:1730: write (excl) oid 9 current snap is 194 2024-09-17T15:38:58.195 INFO:tasks.rados.rados.0.smithi203.stdout:1730: seq_num 622 ranges {453954=629471,1598388=515881} 2024-09-17T15:38:58.212 INFO:tasks.rados.rados.0.smithi203.stdout:1730: writing smithi20328086-9 from 453954 to 1083425 tid 1 2024-09-17T15:38:58.229 INFO:tasks.rados.rados.0.smithi203.stdout:1730: writing smithi20328086-9 from 1598388 to 2114269 tid 2 2024-09-17T15:38:58.229 INFO:tasks.rados.rados.0.smithi203.stdout:1731: copy_from oid 5 from oid 43 current snap is 194 2024-09-17T15:38:58.229 INFO:tasks.rados.rados.0.smithi203.stdout:1730: finishing write tid 1 to smithi20328086-9 2024-09-17T15:38:58.230 INFO:tasks.rados.rados.0.smithi203.stdout:1732: rollback oid 23 current snap is 194 2024-09-17T15:38:58.230 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 191 2024-09-17T15:38:58.230 INFO:tasks.rados.rados.0.smithi203.stdout:1733: snap_remove snap 187 2024-09-17T15:38:58.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: get_monmap_and_config 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: build_initial_monmap 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: monmap: 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:38:59.081426+0000 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:38:59.081426+0000 2024-09-17T15:38:59.078 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:38:59.079 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 Processor -- start 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 -- start start 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: init 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-17T15:38:59.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f84500689d0 0x7f845009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-c con 0x7f84500689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8450068db0 0x7f84500691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-a con 0x7f8450068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8450134a40 addr v1:172.21.15.86:6789/0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f845009d770 con 0x7f8450134a40 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f845009ed00 con 0x7f8450068db0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f845009efe0 con 0x7f84500689d0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8457f69640 10 monclient(hunting): authenticate will time out at 1672.046934s 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8450068db0 0x7f84500691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8450068db0 0x7f84500691c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36076/0 (socket says 172.21.15.86:36076) 2024-09-17T15:38:59.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 -- 172.21.15.86:0/717595604 learned_addr learned my addr 172.21.15.86:0/717595604 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): get_auth_request con 0x7f8450068db0 auth_method 0 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455d05640 1 --2- 172.21.15.86:0/717595604 >> v2:172.21.15.203:3300/0 conn(0x7f84500689d0 0x7f845009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient(hunting): handle_auth_done global_id 124197 payload 274 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 -- 172.21.15.86:0/717595604 >> v2:172.21.15.203:3300/0 conn(0x7f84500689d0 msgr2=0x7f845009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 --2- 172.21.15.86:0/717595604 >> v2:172.21.15.203:3300/0 conn(0x7f84500689d0 0x7f845009ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 -- 172.21.15.86:0/717595604 >> v1:172.21.15.86:6789/0 conn(0x7f8450134a40 legacy=0x7f8450134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient: _finish_hunting 0 2024-09-17T15:38:59.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 monclient: found mon.noname-a 2024-09-17T15:38:59.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:59.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.078+0000 7f8455504640 1 -- 172.21.15.86:0/717595604 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84500a04e0 con 0x7f8450068db0 2024-09-17T15:38:59.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8455504640 1 -- 172.21.15.86:0/717595604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450068db0 msgr2=0x7f84500691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8455504640 1 --2- 172.21.15.86:0/717595604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450068db0 0x7f84500691c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8450069aa0 tx=0x7f844401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41dec28b87f2e8fd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 1 -- 172.21.15.86:0/717595604 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84440ab650 con 0x7f8450068db0 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: dump: 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:59.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:59.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: _finish_auth 0 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: _check_auth_tickets 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 1 -- 172.21.15.86:0/717595604 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84440b0070 con 0x7f8450068db0 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8457f69640 5 monclient: authenticate success, global_id 124197 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8457f69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8457f69640 10 monclient: get_monmap_and_config success 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8457f69640 4 set_mon_vals no callback set 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 1 -- 172.21.15.86:0/717595604 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84440aba10 con 0x7f8450068db0 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.082+0000 7f8454d03640 10 monclient: dump: 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: shutdown 2024-09-17T15:38:59.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- 172.21.15.86:0/717595604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450068db0 msgr2=0x7f84500691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- 172.21.15.86:0/717595604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450068db0 0x7f84500691c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8450069aa0 tx=0x7f844401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- 172.21.15.86:0/717595604 shutdown_connections 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- 172.21.15.86:0/717595604 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450068db0 0x7f84500691c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- 172.21.15.86:0/717595604 >> v2:172.21.15.203:3300/0 conn(0x7f84500689d0 0x7f845009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- 172.21.15.86:0/717595604 >> 172.21.15.86:0/717595604 conn(0x7f84500a6440 msgr2=0x7f84500a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- 172.21.15.86:0/717595604 shutdown_connections 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- 172.21.15.86:0/717595604 wait complete. 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: build_initial_monmap 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: monmap: 2024-09-17T15:38:59.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 Processor -- start 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- start start 2024-09-17T15:38:59.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: init 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: _reopen_session rank -1 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-c con 0x7f84500689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 0x7f845011ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-b con 0x7f8450068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 0x7f845011d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): picked mon.noname-a con 0x7f8450134a40 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f845013dac0 con 0x7f8450134a40 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f845013d940 con 0x7f8450068db0 2024-09-17T15:38:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f845013dc40 con 0x7f84500689d0 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): _renew_subs 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8457f69640 10 monclient(hunting): authenticate will time out at 1672.054277s 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59654/0 (socket says 172.21.15.86:59654) 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 0x7f845011d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 1 -- 172.21.15.86:0/118001723 learned_addr learned my addr 172.21.15.86:0/118001723 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): get_auth_request con 0x7f84500689d0 auth_method 0 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:59.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): get_auth_request con 0x7f8450134a40 auth_method 0 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455504640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 0x7f845011ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient(hunting): handle_auth_done global_id 124200 payload 824 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 -- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 msgr2=0x7f845011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:59.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 -- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 msgr2=0x7f845011ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 0x7f845011ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient: _finish_hunting 0 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 monclient: found mon.noname-a 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8456506640 1 -- 172.21.15.86:0/118001723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845013c8e0 con 0x7f8450134a40 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455504640 10 monclient: get_auth_request con 0x7f8450068db0 auth_method 0 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455504640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 0x7f845011ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:38:59.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.086+0000 7f8455d05640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8456506640 1 -- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 msgr2=0x7f845011d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8456506640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 0x7f845011d340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f844000c410 tx=0x7f844000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440017070 con 0x7f8450134a40 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: dump: 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8440005230 con 0x7f8450134a40 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 5 monclient: authenticate success, global_id 124200 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _renew_subs 2024-09-17T15:38:59.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f845011f600 con 0x7f8450134a40 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400117a0 con 0x7f8450134a40 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 10 monclient: dump: 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:38:59.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844f7fe640 4 set_mon_vals no callback set 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _renew_subs 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845011e1d0 con 0x7f8450134a40 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845009ffa0 con 0x7f8450134a40 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f844001a070 con 0x7f8450134a40 2024-09-17T15:38:59.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 0x7f843c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(392..392 src has 213..392) ==== 4755+0+0 (secure 0 0 0) 0x7f844004f130 con 0x7f8450134a40 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8455d05640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 0x7f843c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.090+0000 7f8455d05640 10 monclient: get_auth_request con 0x7f843c03bb60 auth_method 0 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.094+0000 7f8455d05640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 0x7f843c03e020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f844800d8b0 tx=0x7f8448007870 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.094+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8440004590 con 0x7f8450134a40 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.094+0000 7f844e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:38:59.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.094+0000 7f844e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1733: done (3 left) 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1734: write (excl) oid 32 current snap is 194 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1730: finishing write tid 2 to smithi20328086-9 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1730: finishing write tid 3 to smithi20328086-9 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1730: finishing write tid 4 to smithi20328086-9 2024-09-17T15:38:59.162 INFO:tasks.rados.rados.0.smithi203.stdout:1730: oid 9 updating version 0 to 581 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1730: oid 9 version 581 is already newer than 580 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1730: oid 9 version 581 is already newer than 579 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 581 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1730: left oid 9 (ObjNum 622 snap 194 seq_num 622) 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1732: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1732: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:38:59.163 INFO:tasks.rados.rados.0.smithi203.stdout:1734: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-17T15:38:59.183 INFO:tasks.rados.rados.0.smithi203.stdout:1734: writing smithi20328086-32 from 470761 to 1175481 tid 1 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:1734: writing smithi20328086-32 from 1656870 to 2316409 tid 2 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:1730: done (3 left) 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:1735: rollback oid 18 current snap is 194 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 182 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:1732: finishing rollback tid 2 to smithi20328086-23 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 661 (ObjNum 600 snap 179 seq_num 600) dirty exists 2024-09-17T15:38:59.203 INFO:tasks.rados.rados.0.smithi203.stdout:1731: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:1731: finishing copy_from to smithi20328086-5 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 507 (ObjNum 570 snap 168 seq_num 570) dirty exists 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:1734: finishing write tid 1 to smithi20328086-32 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:1731: done (3 left) 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:1732: done (2 left) 2024-09-17T15:38:59.204 INFO:tasks.rados.rados.0.smithi203.stdout:1736: delete oid 22 current snap is 194 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1736: done (2 left) 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1737: read oid 46 snap -1 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1737: expect deleted 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1738: read oid 4 snap 188 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1738: expect deleted 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1739: write oid 28 current snap is 194 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1734: finishing write tid 2 to smithi20328086-32 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1735: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1734: finishing write tid 3 to smithi20328086-32 2024-09-17T15:38:59.213 INFO:tasks.rados.rados.0.smithi203.stdout:1739: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-17T15:38:59.230 INFO:tasks.rados.rados.0.smithi203.stdout:1739: writing smithi20328086-28 from 504375 to 1075946 tid 1 2024-09-17T15:38:59.248 INFO:tasks.rados.rados.0.smithi203.stdout:1739: writing smithi20328086-28 from 1490187 to 2153395 tid 2 2024-09-17T15:38:59.248 INFO:tasks.rados.rados.0.smithi203.stdout:1740: snap_create 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1734: finishing write tid 4 to smithi20328086-32 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1734: oid 32 updating version 0 to 329 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1734: oid 32 version 329 is already newer than 328 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1734: oid 32 version 329 is already newer than 327 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 329 (ObjNum 623 snap 194 seq_num 623) dirty exists 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1734: left oid 32 (ObjNum 623 snap 194 seq_num 623) 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1735: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1735: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 554 (ObjNum 603 snap 180 seq_num 603) dirty exists 2024-09-17T15:38:59.249 INFO:tasks.rados.rados.0.smithi203.stdout:1739: finishing write tid 1 to smithi20328086-28 2024-09-17T15:38:59.255 INFO:tasks.rados.rados.0.smithi203.stdout:1739: finishing write tid 2 to smithi20328086-28 2024-09-17T15:38:59.256 INFO:tasks.rados.rados.0.smithi203.stdout:1739: finishing write tid 3 to smithi20328086-28 2024-09-17T15:38:59.256 INFO:tasks.rados.rados.0.smithi203.stdout:1739: finishing write tid 4 to smithi20328086-28 2024-09-17T15:38:59.257 INFO:tasks.rados.rados.0.smithi203.stdout:1739: oid 28 updating version 0 to 779 2024-09-17T15:38:59.257 INFO:tasks.rados.rados.0.smithi203.stdout:1739: oid 28 updating version 779 to 781 2024-09-17T15:38:59.257 INFO:tasks.rados.rados.0.smithi203.stdout:1739: oid 28 version 781 is already newer than 780 2024-09-17T15:38:59.257 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 781 (ObjNum 625 snap 194 seq_num 625) dirty exists 2024-09-17T15:38:59.257 INFO:tasks.rados.rados.0.smithi203.stdout:1739: left oid 28 (ObjNum 625 snap 194 seq_num 625) 2024-09-17T15:38:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f8457f69640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:38:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f8457f69640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f8457f69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:38:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f845011f140 con 0x7f8450134a40 2024-09-17T15:38:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f844e7fc640 1 -- 172.21.15.86:0/118001723 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f844001bdd0 con 0x7f8450134a40 2024-09-17T15:38:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f844e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:38:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.338+0000 7f844e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:38:59.341 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":112,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:38:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 msgr2=0x7f843c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 0x7f843c03e020 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f844800d8b0 tx=0x7f8448007870 comp rx=0 tx=0).stop 2024-09-17T15:38:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 10 monclient: shutdown 2024-09-17T15:38:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:38:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 msgr2=0x7f845011d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 0x7f845011d340 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f844000c410 tx=0x7f844000c8d0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 shutdown_connections 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f843c03bb60 0x7f843c03e020 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8450134a40 0x7f845011d340 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8450068db0 0x7f845011ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 --2- 172.21.15.86:0/118001723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84500689d0 0x7f845011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 >> 172.21.15.86:0/118001723 conn(0x7f84500a6440 msgr2=0x7f845009ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 shutdown_connections 2024-09-17T15:38:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:38:59.342+0000 7f8457f69640 1 -- 172.21.15.86:0/118001723 wait complete. 2024-09-17T15:39:00.205 INFO:tasks.rados.rados.0.smithi203.stdout:1734: done (5 left) 2024-09-17T15:39:00.205 INFO:tasks.rados.rados.0.smithi203.stdout:1735: done (4 left) 2024-09-17T15:39:00.205 INFO:tasks.rados.rados.0.smithi203.stdout:1737: done (3 left) 2024-09-17T15:39:00.205 INFO:tasks.rados.rados.0.smithi203.stdout:1738: done (2 left) 2024-09-17T15:39:00.205 INFO:tasks.rados.rados.0.smithi203.stdout:1739: done (1 left) 2024-09-17T15:39:00.206 INFO:tasks.rados.rados.0.smithi203.stdout:1740: done (0 left) 2024-09-17T15:39:00.206 INFO:tasks.rados.rados.0.smithi203.stdout:1741: write (excl) oid 34 current snap is 195 2024-09-17T15:39:00.206 INFO:tasks.rados.rados.0.smithi203.stdout:1741: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-17T15:39:00.220 INFO:tasks.rados.rados.0.smithi203.stdout:1741: writing smithi20328086-34 from 521182 to 1168002 tid 1 2024-09-17T15:39:00.237 INFO:tasks.rados.rados.0.smithi203.stdout:1741: writing smithi20328086-34 from 1948669 to 2471888 tid 2 2024-09-17T15:39:00.238 INFO:tasks.rados.rados.0.smithi203.stdout:1742: copy_from oid 19 from oid 23 current snap is 195 2024-09-17T15:39:00.238 INFO:tasks.rados.rados.0.smithi203.stdout:1741: finishing write tid 1 to smithi20328086-34 2024-09-17T15:39:00.238 INFO:tasks.rados.rados.0.smithi203.stdout:1743: delete oid 42 current snap is 195 2024-09-17T15:39:00.246 INFO:tasks.rados.rados.0.smithi203.stdout:1741: finishing write tid 2 to smithi20328086-34 2024-09-17T15:39:00.247 INFO:tasks.rados.rados.0.smithi203.stdout:1741: finishing write tid 3 to smithi20328086-34 2024-09-17T15:39:00.247 INFO:tasks.rados.rados.0.smithi203.stdout:1743: done (2 left) 2024-09-17T15:39:00.247 INFO:tasks.rados.rados.0.smithi203.stdout:1744: snap_create 2024-09-17T15:39:00.247 INFO:tasks.rados.rados.0.smithi203.stdout:1741: finishing write tid 4 to smithi20328086-34 2024-09-17T15:39:00.248 INFO:tasks.rados.rados.0.smithi203.stdout:1741: oid 34 updating version 0 to 608 2024-09-17T15:39:00.248 INFO:tasks.rados.rados.0.smithi203.stdout:1741: oid 34 version 608 is already newer than 607 2024-09-17T15:39:00.248 INFO:tasks.rados.rados.0.smithi203.stdout:1741: oid 34 updating version 608 to 609 2024-09-17T15:39:00.248 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 609 (ObjNum 626 snap 195 seq_num 626) dirty exists 2024-09-17T15:39:00.248 INFO:tasks.rados.rados.0.smithi203.stdout:1741: left oid 34 (ObjNum 626 snap 195 seq_num 626) 2024-09-17T15:39:00.262 INFO:tasks.rados.rados.0.smithi203.stdout:1742: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:39:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:1742: finishing copy_from to smithi20328086-19 2024-09-17T15:39:00.263 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 661 (ObjNum 600 snap 179 seq_num 600) dirty exists 2024-09-17T15:39:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:1741: done (2 left) 2024-09-17T15:39:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:1742: done (1 left) 2024-09-17T15:39:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:1744: done (0 left) 2024-09-17T15:39:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:1745: write (excl) oid 41 current snap is 196 2024-09-17T15:39:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:1745: seq_num 628 ranges {554796=797318,1949280=526888} 2024-09-17T15:39:01.233 INFO:tasks.rados.rados.0.smithi203.stdout:1745: writing smithi20328086-41 from 554796 to 1352114 tid 1 2024-09-17T15:39:01.249 INFO:tasks.rados.rados.0.smithi203.stdout:1745: writing smithi20328086-41 from 1949280 to 2476168 tid 2 2024-09-17T15:39:01.249 INFO:tasks.rados.rados.0.smithi203.stdout:1746: write oid 16 current snap is 196 2024-09-17T15:39:01.249 INFO:tasks.rados.rados.0.smithi203.stdout:1746: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-17T15:39:01.263 INFO:tasks.rados.rados.0.smithi203.stdout:1746: writing smithi20328086-16 from 571603 to 1044170 tid 1 2024-09-17T15:39:01.277 INFO:tasks.rados.rados.0.smithi203.stdout:1746: writing smithi20328086-16 from 1607762 to 2278308 tid 2 2024-09-17T15:39:01.277 INFO:tasks.rados.rados.0.smithi203.stdout:1747: copy_from oid 12 from oid 37 current snap is 196 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: finishing write tid 1 to smithi20328086-41 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: finishing write tid 2 to smithi20328086-41 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: finishing write tid 3 to smithi20328086-41 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: finishing write tid 4 to smithi20328086-41 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: oid 41 updating version 0 to 570 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: oid 41 version 570 is already newer than 568 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: oid 41 version 570 is already newer than 569 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 570 (ObjNum 628 snap 196 seq_num 628) dirty exists 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: left oid 41 (ObjNum 628 snap 196 seq_num 628) 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1746: finishing write tid 1 to smithi20328086-16 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1745: done (2 left) 2024-09-17T15:39:01.278 INFO:tasks.rados.rados.0.smithi203.stdout:1748: snap_remove snap 178 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1748: done (2 left) 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1749: delete oid 33 current snap is 196 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: finishing write tid 2 to smithi20328086-16 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: finishing write tid 3 to smithi20328086-16 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: finishing write tid 4 to smithi20328086-16 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: oid 16 updating version 0 to 677 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: oid 16 updating version 677 to 678 2024-09-17T15:39:02.226 INFO:tasks.rados.rados.0.smithi203.stdout:1746: oid 16 version 678 is already newer than 676 2024-09-17T15:39:02.227 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 678 (ObjNum 629 snap 196 seq_num 629) dirty exists 2024-09-17T15:39:02.227 INFO:tasks.rados.rados.0.smithi203.stdout:1746: left oid 16 (ObjNum 629 snap 196 seq_num 629) 2024-09-17T15:39:02.227 INFO:tasks.rados.rados.0.smithi203.stdout:1747: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:39:02.227 INFO:tasks.rados.rados.0.smithi203.stdout:1747: finishing copy_from to smithi20328086-12 2024-09-17T15:39:02.227 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 631 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:39:02.240 INFO:tasks.rados.rados.0.smithi203.stdout:1746: done (2 left) 2024-09-17T15:39:02.240 INFO:tasks.rados.rados.0.smithi203.stdout:1747: done (1 left) 2024-09-17T15:39:02.240 INFO:tasks.rados.rados.0.smithi203.stdout:1749: done (0 left) 2024-09-17T15:39:02.240 INFO:tasks.rados.rados.0.smithi203.stdout:1750: snap_remove snap 170 2024-09-17T15:39:02.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: get_monmap_and_config 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: build_initial_monmap 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: monmap: 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:02.489517+0000 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:02.489517+0000 2024-09-17T15:39:02.486 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:02.487 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 Processor -- start 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 -- start start 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: init 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 0x7f805809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-c con 0x7f80580a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80580a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f805813d2f0 0x7f805813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-a con 0x7f805813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f805809d350 con 0x7f80580a54c0 2024-09-17T15:39:02.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f805809bdc0 con 0x7f805813d2f0 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f80580a6530 con 0x7f80580a50e0 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f805e0e5640 10 monclient(hunting): authenticate will time out at 1675.455694s 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f805813d2f0 0x7f805813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f805813d2f0 0x7f805813f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36094/0 (socket says 172.21.15.86:36094) 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 1 -- 172.21.15.86:0/1053884972 learned_addr learned my addr 172.21.15.86:0/1053884972 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f804ffff640 1 --1- 172.21.15.86:0/1053884972 >> v1:172.21.15.86:6789/0 conn(0x7f80580a54c0 0x7f80580a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51272/0 (socket says 172.21.15.86:51272) 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 10 monclient(hunting): get_auth_request con 0x7f805813d2f0 auth_method 0 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.486+0000 7f8057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 1 --2- 172.21.15.86:0/1053884972 >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 0x7f805809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 10 monclient(hunting): get_auth_request con 0x7f80580a50e0 auth_method 0 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient(hunting): handle_auth_done global_id 124203 payload 274 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 1 -- 172.21.15.86:0/1053884972 >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 msgr2=0x7f805809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:02.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 1 --2- 172.21.15.86:0/1053884972 >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 0x7f805809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 1 -- 172.21.15.86:0/1053884972 >> v1:172.21.15.86:6789/0 conn(0x7f80580a54c0 legacy=0x7f80580a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient: _finish_hunting 0 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 1 monclient: found mon.noname-a 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f8057fff640 1 -- 172.21.15.86:0/1053884972 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805809c150 con 0x7f805813d2f0 2024-09-17T15:39:02.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.490+0000 7f80577fe640 1 --2- 172.21.15.86:0/1053884972 >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 0x7f805809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8057fff640 1 -- 172.21.15.86:0/1053884972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 msgr2=0x7f805813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8057fff640 1 --2- 172.21.15.86:0/1053884972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f805813f6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f80580a60a0 tx=0x7f805001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea4c21f0af7513a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 1 -- 172.21.15.86:0/1053884972 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500a56c0 con 0x7f805813d2f0 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: dump: 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.494 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:02.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 5 monclient: authenticate success, global_id 124203 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 1 -- 172.21.15.86:0/1053884972 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80500a5860 con 0x7f805813d2f0 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 1 -- 172.21.15.86:0/1053884972 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500a5c30 con 0x7f805813d2f0 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 4 set_mon_vals no callback set 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f8056ffd640 10 monclient: dump: 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: shutdown 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- 172.21.15.86:0/1053884972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 msgr2=0x7f805813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- 172.21.15.86:0/1053884972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f805813f6e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f80580a60a0 tx=0x7f805001c860 comp rx=0 tx=0).stop 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- 172.21.15.86:0/1053884972 shutdown_connections 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- 172.21.15.86:0/1053884972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f805813f6e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- 172.21.15.86:0/1053884972 >> v2:172.21.15.203:3300/0 conn(0x7f80580a50e0 0x7f805809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- 172.21.15.86:0/1053884972 >> 172.21.15.86:0/1053884972 conn(0x7f80580aaae0 msgr2=0x7f80580aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- 172.21.15.86:0/1053884972 shutdown_connections 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- 172.21.15.86:0/1053884972 wait complete. 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: build_initial_monmap 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: monmap: 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:02.503 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 Processor -- start 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- start start 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: init 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-c con 0x7f80580a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580a54c0 0x7f8058138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-b con 0x7f80580a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:02.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): picked mon.noname-a con 0x7f805813d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8058142650 con 0x7f805813d2f0 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f805806f620 con 0x7f80580a54c0 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f80581427d0 con 0x7f80580a50e0 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.494+0000 7f805e0e5640 10 monclient(hunting): authenticate will time out at 1675.463988s 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59656/0 (socket says 172.21.15.86:59656) 2024-09-17T15:39:02.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36106/0 (socket says 172.21.15.86:36106) 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 -- 172.21.15.86:0/3200789457 learned_addr learned my addr 172.21.15.86:0/3200789457 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): get_auth_request con 0x7f80580a50e0 auth_method 0 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): get_auth_request con 0x7f805813d2f0 auth_method 0 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f804ffff640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580a54c0 0x7f8058138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f804ffff640 10 monclient(hunting): get_auth_request con 0x7f80580a54c0 auth_method 0 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f804ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f804ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f804ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient(hunting): handle_auth_done global_id 104405 payload 824 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 -- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580a54c0 msgr2=0x7f8058138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580a54c0 0x7f8058138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 -- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 msgr2=0x7f80580a2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 monclient: found mon.noname-c 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 -- 172.21.15.86:0/3200789457 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8058138ec0 con 0x7f80580a50e0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8057fff640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 -- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 msgr2=0x7f80581cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80577fe640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f804000e840 tx=0x7f804000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8040017070 con 0x7f80580a50e0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:02.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: dump: 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: _finish_auth 0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f804000cae0 con 0x7f80580a50e0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 5 monclient: authenticate success, global_id 104405 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _renew_subs 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80581d0970 con 0x7f80580a50e0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804000f040 con 0x7f80580a50e0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:02.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f8054ff9640 10 monclient: dump: 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _renew_subs 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80581228d0 con 0x7f80580a50e0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f80567fc640 4 set_mon_vals no callback set 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.498+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f805813d960 con 0x7f80580a50e0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f804001a070 con 0x7f80580a50e0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f8054ff9640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 0x7f803803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(396..396 src has 213..396) ==== 4799+0+0 (secure 0 0 0) 0x7f804004fce0 con 0x7f80580a50e0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f804ffff640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 0x7f803803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f804ffff640 10 monclient: get_auth_request con 0x7f803803bb60 auth_method 0 2024-09-17T15:39:02.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.502+0000 7f804ffff640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 0x7f803803e020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f805813f940 tx=0x7f8048009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.506+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f80581d0970 con 0x7f80580a50e0 2024-09-17T15:39:02.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.506+0000 7f8054ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:02.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.506+0000 7f8054ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:02.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f805e0e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f805e0e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f805e0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f80580a7ce0 con 0x7f80580a50e0 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f8054ff9640 1 -- 172.21.15.86:0/3200789457 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8040013c60 con 0x7f80580a50e0 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f8054ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:02.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.742+0000 7f8054ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:02.743 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 msgr2=0x7f803803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 0x7f803803e020 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f805813f940 tx=0x7f8048009040 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 10 monclient: shutdown 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 msgr2=0x7f80581cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f804000e840 tx=0x7f804000ed00 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 shutdown_connections 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f803803bb60 0x7f803803e020 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805813d2f0 0x7f80580a2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580a54c0 0x7f8058138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 --2- 172.21.15.86:0/3200789457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80580a50e0 0x7f80581cfa60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 >> 172.21.15.86:0/3200789457 conn(0x7f80580aaae0 msgr2=0x7f80580a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:02.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 shutdown_connections 2024-09-17T15:39:02.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.746+0000 7f805e0e5640 1 -- 172.21.15.86:0/3200789457 wait complete. 2024-09-17T15:39:02.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:39:02.762 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:39:02.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: get_monmap_and_config 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: build_initial_monmap 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: monmap: 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:02.884362+0000 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:02.884362+0000 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:02.881 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:02.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:02.882 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:02.882 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:02.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 Processor -- start 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 -- start start 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: init 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ab409d770 0x7f2ab409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ab409d770 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2ab409e090 0x7f2ab4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2ab409e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ab4134f80 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:02.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab409ff00 con 0x7f2ab4134f80 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab409e7f0 con 0x7f2ab409d770 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab40a2110 con 0x7f2ab409e090 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2abb5dd640 10 monclient(hunting): authenticate will time out at 1675.849939s 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ab409d770 0x7f2ab409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ab409d770 0x7f2ab409db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36120/0 (socket says 172.21.15.86:36120) 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 -- 172.21.15.86:0/333336647 learned_addr learned my addr 172.21.15.86:0/333336647 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): get_auth_request con 0x7f2ab409d770 auth_method 0 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab8b78640 1 --2- 172.21.15.86:0/333336647 >> v2:172.21.15.203:3300/0 conn(0x7f2ab409e090 0x7f2ab4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab8b78640 10 monclient(hunting): get_auth_request con 0x7f2ab409e090 auth_method 0 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab8b78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab8b78640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab8b78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient(hunting): handle_auth_done global_id 124209 payload 274 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 -- 172.21.15.86:0/333336647 >> v2:172.21.15.203:3300/0 conn(0x7f2ab409e090 msgr2=0x7f2ab4134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 --2- 172.21.15.86:0/333336647 >> v2:172.21.15.203:3300/0 conn(0x7f2ab409e090 0x7f2ab4134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 -- 172.21.15.86:0/333336647 >> v1:172.21.15.203:6789/0 conn(0x7f2ab4134f80 legacy=0x7f2ab413bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient: _finish_hunting 0 2024-09-17T15:39:02.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 monclient: found mon.noname-a 2024-09-17T15:39:02.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:02.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:02.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.882+0000 7f2ab9379640 1 -- 172.21.15.86:0/333336647 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab409ec30 con 0x7f2ab409d770 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2ab9379640 1 -- 172.21.15.86:0/333336647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 msgr2=0x7f2ab409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2ab9379640 1 --2- 172.21.15.86:0/333336647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab409db50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4074de0 tx=0x7f2ab001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c66ff6e2a828bf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 1 -- 172.21.15.86:0/333336647 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00b0070 con 0x7f2ab409d770 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:02.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: dump: 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: _finish_auth 0 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:02.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 1 -- 172.21.15.86:0/333336647 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab00ab600 con 0x7f2ab409d770 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 1 -- 172.21.15.86:0/333336647 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00ab9d0 con 0x7f2ab409d770 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 5 monclient: authenticate success, global_id 124209 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 4 set_mon_vals no callback set 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2aabfff640 10 monclient: dump: 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.890 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 10 monclient: shutdown 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 -- 172.21.15.86:0/333336647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 msgr2=0x7f2ab409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/333336647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab409db50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2ab4074de0 tx=0x7f2ab001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 -- 172.21.15.86:0/333336647 shutdown_connections 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/333336647 >> v2:172.21.15.203:3300/0 conn(0x7f2ab409e090 0x7f2ab4134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/333336647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab409db50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 -- 172.21.15.86:0/333336647 >> 172.21.15.86:0/333336647 conn(0x7f2ab40a2f40 msgr2=0x7f2ab40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:02.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 -- 172.21.15.86:0/333336647 shutdown_connections 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.886+0000 7f2abb5dd640 1 -- 172.21.15.86:0/333336647 wait complete. 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient: build_initial_monmap 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient: monmap: 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:02.892 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 Processor -- start 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 -- start start 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient: init 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab41cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ab409d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 0x7f2ab413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-c con 0x7f2ab409e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4134f80 0x7f2ab4136d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): picked mon.noname-b con 0x7f2ab4134f80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab409cce0 con 0x7f2ab409d770 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab409e4a0 con 0x7f2ab4134f80 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab409ce60 con 0x7f2ab409e090 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2abb5dd640 10 monclient(hunting): authenticate will time out at 1675.857195s 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab41cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 0x7f2ab413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab41cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36136/0 (socket says 172.21.15.86:36136) 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 0x7f2ab413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59664/0 (socket says 172.21.15.86:59664) 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 1 -- 172.21.15.86:0/3769690997 learned_addr learned my addr 172.21.15.86:0/3769690997 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:02.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): get_auth_request con 0x7f2ab409d770 auth_method 0 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): get_auth_request con 0x7f2ab409e090 auth_method 0 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9b7a640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4134f80 0x7f2ab4136d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9b7a640 10 monclient(hunting): get_auth_request con 0x7f2ab4134f80 auth_method 0 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9b7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9b7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9b7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient(hunting): handle_auth_done global_id 104408 payload 824 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4134f80 msgr2=0x7f2ab4136d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ab4134f80 0x7f2ab4136d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 msgr2=0x7f2ab41cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab41cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient: _finish_hunting 0 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 monclient: found mon.noname-c 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab40a5200 con 0x7f2ab409e090 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab9379640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab409d770 0x7f2ab41cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 msgr2=0x7f2ab413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.890+0000 7f2ab8b78640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 0x7f2ab413c2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400d620 tx=0x7f2aa400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400b830 con 0x7f2ab409e090 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: dump: 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.902 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa4019070 con 0x7f2ab409e090 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400bc10 con 0x7f2ab409e090 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 5 monclient: authenticate success, global_id 104408 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 monclient: _renew_subs 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab409d460 con 0x7f2ab409e090 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aab7fe640 4 set_mon_vals no callback set 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: dump: 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 monclient: _renew_subs 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab413ab90 con 0x7f2ab409e090 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 0x7f2ab41372b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab409db50 con 0x7f2ab413c830 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 0x7f2ab41372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient: get_auth_request con 0x7f2ab413c830 auth_method 0 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2aa401e460 con 0x7f2ab409e090 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 10 monclient(hunting): handle_auth_done global_id 124215 payload 824 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ab413a350 con 0x7f2ab413c830 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a9803bbe0 0x7f2a9803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(396..396 src has 213..396) ==== 4799+0+0 (secure 0 0 0) 0x7f2aa40502d0 con 0x7f2ab409e090 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9b7a640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a9803bbe0 0x7f2a9803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9b7a640 10 monclient: get_auth_request con 0x7f2a9803bbe0 auth_method 0 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 msgr2=0x7f2ab41372b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2ab9379640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 0x7f2ab41372b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001aa90 tx=0x7f2ab001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00b0040 con 0x7f2ab413c830 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.894+0000 7f2aa9ffb640 10 monclient: dump: 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ab00b6880 con 0x7f2ab413c830 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:02.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:02.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 msgr2=0x7f2ab41372b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 0x7f2ab41372b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001aa90 tx=0x7f2ab001ac20 comp rx=0 tx=0).stop 2024-09-17T15:39:02.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 msgr2=0x7f2ab41372b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:02.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2aa9ffb640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab413c830 0x7f2ab41372b0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2ab001aa90 tx=0x7f2ab001ac20 comp rx=0 tx=0).stop 2024-09-17T15:39:02.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.898+0000 7f2ab9b7a640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a9803bbe0 0x7f2a9803e0a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2aac005550 tx=0x7f2aac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2abb5dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 0x7f2ab4072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2abb5dd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4137c00 con 0x7f2ab41377f0 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 0x7f2ab4072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient: get_auth_request con 0x7f2ab41377f0 auth_method 0 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:02.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 10 monclient(hunting): handle_auth_done global_id 124218 payload 824 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.902+0000 7f2ab9379640 1 -- 172.21.15.86:0/3769690997 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ab406d4f0 con 0x7f2ab41377f0 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2ab9379640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 msgr2=0x7f2ab4072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2ab9379640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 0x7f2ab4072160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0000c10 tx=0x7f2ab0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab00b0070 con 0x7f2ab41377f0 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 10 monclient: dump: 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:02.911 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f2ab00c1800 con 0x7f2ab41377f0 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 msgr2=0x7f2ab4072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 0x7f2ab4072160 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0000c10 tx=0x7f2ab0019180 comp rx=0 tx=0).stop 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 msgr2=0x7f2ab4072160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2aa9ffb640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ab41377f0 0x7f2ab4072160 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0000c10 tx=0x7f2ab0019180 comp rx=0 tx=0).stop 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 73770, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:02.925 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:02.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a9803bbe0 msgr2=0x7f2a9803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a9803bbe0 0x7f2a9803e0a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2aac005550 tx=0x7f2aac009040 comp rx=0 tx=0).stop 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 10 monclient: shutdown 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 msgr2=0x7f2ab413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 --2- 172.21.15.86:0/3769690997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ab409e090 0x7f2ab413c2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400d620 tx=0x7f2aa400dae0 comp rx=0 tx=0).stop 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2ab9b7a640 1 -- 172.21.15.86:0/3769690997 reap_dead start 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2ab9b7a640 1 -- 172.21.15.86:0/3769690997 reap_dead start 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 shutdown_connections 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 >> 172.21.15.86:0/3769690997 conn(0x7f2ab40a2f40 msgr2=0x7f2ab40a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 shutdown_connections 2024-09-17T15:39:02.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:02.906+0000 7f2abb5dd640 1 -- 172.21.15.86:0/3769690997 wait complete. 2024-09-17T15:39:02.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:02.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:39:02.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: get_monmap_and_config 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: monmap: 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:03.044360+0000 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:03.044360+0000 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:03.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:03.042 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:03.042 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:03.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 Processor -- start 2024-09-17T15:39:03.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 -- start start 2024-09-17T15:39:03.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: init 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4aa80a0c70 0x7f4aa80a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa80a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4aa80a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4aa8136160 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa80a2130 con 0x7f4aa80a1590 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa80a26f0 con 0x7f4aa8136160 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa80a2410 con 0x7f4aa80a0c70 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aaf3df640 10 monclient(hunting): authenticate will time out at 1676.009899s 2024-09-17T15:39:03.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aac97a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4aa80a1590 0x7f4aa8135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51280/0 (socket says 172.21.15.86:51280) 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aac97a640 1 -- 172.21.15.86:0/4118475176 learned_addr learned my addr 172.21.15.86:0/4118475176 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 1 --2- 172.21.15.86:0/4118475176 >> v2:172.21.15.86:3300/0 conn(0x7f4aa80a0c70 0x7f4aa80a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): get_auth_request con 0x7f4aa80a0c70 auth_method 0 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient(hunting): handle_auth_done global_id 124221 payload 274 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 1 -- 172.21.15.86:0/4118475176 >> v1:172.21.15.203:6789/0 conn(0x7f4aa8136160 legacy=0x7f4aa813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 1 -- 172.21.15.86:0/4118475176 >> v1:172.21.15.86:6789/0 conn(0x7f4aa80a1590 legacy=0x7f4aa8135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.042+0000 7f4aad17b640 1 -- 172.21.15.86:0/4118475176 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa809f9f0 con 0x7f4aa80a0c70 2024-09-17T15:39:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.046+0000 7f4aad17b640 1 -- 172.21.15.86:0/4118475176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80a0c70 msgr2=0x7f4aa80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 1 --2- 172.21.15.86:0/4118475176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80a0c70 0x7f4aa80a1050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa38bc795cf8afc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 1 -- 172.21.15.86:0/4118475176 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00b2070 con 0x7f4aa80a0c70 2024-09-17T15:39:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: dump: 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 1 -- 172.21.15.86:0/4118475176 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa00a5970 con 0x7f4aa80a0c70 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 1 -- 172.21.15.86:0/4118475176 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00a5d40 con 0x7f4aa80a0c70 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aa7fff640 10 monclient: dump: 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 5 monclient: authenticate success, global_id 124221 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 4 set_mon_vals no callback set 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: shutdown 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- 172.21.15.86:0/4118475176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80a0c70 msgr2=0x7f4aa80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/4118475176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80a0c70 0x7f4aa80a1050 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa001c830 comp rx=0 tx=0).stop 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- 172.21.15.86:0/4118475176 shutdown_connections 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/4118475176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa80a0c70 0x7f4aa80a1050 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- 172.21.15.86:0/4118475176 >> 172.21.15.86:0/4118475176 conn(0x7f4aa80a2f40 msgr2=0x7f4aa80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- 172.21.15.86:0/4118475176 shutdown_connections 2024-09-17T15:39:03.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- 172.21.15.86:0/4118475176 wait complete. 2024-09-17T15:39:03.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: monmap: 2024-09-17T15:39:03.058 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.058 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:03.058 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 Processor -- start 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- start start 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: init 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa80a0c70 0x7f4aa811f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa80a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:03.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80a1590 0x7f4aa811d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa80a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 0x7f4aa8135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa8136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa80a00d0 con 0x7f4aa8136160 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa809ff50 con 0x7f4aa80a0c70 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa8141e70 con 0x7f4aa80a1590 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aaf3df640 10 monclient(hunting): authenticate will time out at 1676.018494s 2024-09-17T15:39:03.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 0x7f4aa8135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80a1590 0x7f4aa811d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 0x7f4aa8135300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36190/0 (socket says 172.21.15.86:36190) 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80a1590 0x7f4aa811d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59672/0 (socket says 172.21.15.86:59672) 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 learned_addr learned my addr 172.21.15.86:0/529273791 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): get_auth_request con 0x7f4aa8136160 auth_method 0 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): get_auth_request con 0x7f4aa80a1590 auth_method 0 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aac97a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa80a0c70 0x7f4aa811f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 10 monclient(hunting): get_auth_request con 0x7f4aa80a0c70 auth_method 0 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad17b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient(hunting): handle_auth_done global_id 124224 payload 824 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80a1590 msgr2=0x7f4aa811d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa80a1590 0x7f4aa811d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa80a0c70 msgr2=0x7f4aa811f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa80a0c70 0x7f4aa811f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.050+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa8120400 con 0x7f4aa8136160 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 msgr2=0x7f4aa8135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad97c640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 0x7f4aa8135300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800c410 tx=0x7f4a9800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98017070 con 0x7f4aa8136160 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: dump: 2024-09-17T15:39:03.063 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a98005230 con 0x7f4aa8136160 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a980117a0 con 0x7f4aa8136160 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 10 monclient: dump: 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.065 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa6ffd640 4 set_mon_vals no callback set 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 5 monclient: authenticate success, global_id 124224 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 monclient: _renew_subs 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa809d460 con 0x7f4aa8136160 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 monclient: _renew_subs 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa8120ca0 con 0x7f4aa8136160 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 0x7f4aa81d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa8120920 con 0x7f4aa8135840 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 0x7f4aa81d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient: get_auth_request con 0x7f4aa8135840 auth_method 0 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4a9801a070 con 0x7f4aa8136160 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 10 monclient(hunting): handle_auth_done global_id 104414 payload 824 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aad17b640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4aa80a1050 con 0x7f4aa8135840 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a9403bbb0 0x7f4a9403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(396..396 src has 213..396) ==== 4799+0+0 (secure 0 0 0) 0x7f4a9804f140 con 0x7f4aa8136160 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aac97a640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a9403bbb0 0x7f4a9403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.054+0000 7f4aac97a640 10 monclient: get_auth_request con 0x7f4a9403bbb0 auth_method 0 2024-09-17T15:39:03.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aad17b640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 msgr2=0x7f4aa81d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aad17b640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 0x7f4aa81d0090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00af000 tx=0x7f4aa00195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00b2040 con 0x7f4aa8135840 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 10 monclient: dump: 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.068 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4aa00ad440 con 0x7f4aa8135840 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 msgr2=0x7f4aa81d0090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 0x7f4aa81d0090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00af000 tx=0x7f4aa00195d0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 msgr2=0x7f4aa81d0090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:03.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aa5ffb640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa8135840 0x7f4aa81d0090 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00af000 tx=0x7f4aa00195d0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.058+0000 7f4aac97a640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a9403bbb0 0x7f4a9403e070 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007ca0 tx=0x7f4a9c007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.062+0000 7f4aaf3df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 0x7f4aa81d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa81d0ef0 con 0x7f4aa81d05d0 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 0x7f4aa81d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient: get_auth_request con 0x7f4aa81d05d0 auth_method 0 2024-09-17T15:39:03.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 10 monclient(hunting): handle_auth_done global_id 104417 payload 824 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 1 -- 172.21.15.86:0/529273791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4aa806d4f0 con 0x7f4aa81d05d0 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 msgr2=0x7f4aa81d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad17b640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 0x7f4aa81d09b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa00a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00b2040 con 0x7f4aa81d05d0 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 10 monclient: dump: 2024-09-17T15:39:03.072 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f4aa00c37f0 con 0x7f4aa81d05d0 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:03.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 msgr2=0x7f4aa81d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 0x7f4aa81d09b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa00a7000 comp rx=0 tx=0).stop 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 msgr2=0x7f4aa81d09b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aa5ffb640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4aa81d05d0 0x7f4aa81d09b0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0009d90 tx=0x7f4aa00a7000 comp rx=0 tx=0).stop 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 173941, 2024-09-17T15:39:03.074 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.075 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:03.077 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:03.078 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:03.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.080 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:03.081 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.083 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.084 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.085 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.087 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:03.087 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a9403bbb0 msgr2=0x7f4a9403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4a9403bbb0 0x7f4a9403e070 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c007ca0 tx=0x7f4a9c007bf0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 10 monclient: shutdown 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 msgr2=0x7f4aa8135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 --2- 172.21.15.86:0/529273791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4aa8136160 0x7f4aa8135300 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800c410 tx=0x7f4a9800c8d0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 reap_dead start 2024-09-17T15:39:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aad97c640 1 -- 172.21.15.86:0/529273791 reap_dead start 2024-09-17T15:39:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 shutdown_connections 2024-09-17T15:39:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 >> 172.21.15.86:0/529273791 conn(0x7f4aa80a2f40 msgr2=0x7f4aa80721f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 shutdown_connections 2024-09-17T15:39:03.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.066+0000 7f4aaf3df640 1 -- 172.21.15.86:0/529273791 wait complete. 2024-09-17T15:39:03.089 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:03.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:39:03.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.198+0000 7f471d865640 10 monclient: get_monmap_and_config 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.198+0000 7f471d865640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.198+0000 7f471d865640 10 monclient: monmap: 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:03.203805+0000 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:03.203805+0000 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:03.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 Processor -- start 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 -- start start 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient: init 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47180a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47180a5bd0 0x7f471809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): picked mon.noname-a con 0x7f47180a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f471813d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47180a6280 con 0x7f47180a50e0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f471809c0a0 con 0x7f471813d2f0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f471809d2a0 con 0x7f47180a5bd0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f471d865640 10 monclient(hunting): authenticate will time out at 1676.169321s 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47180a5bd0 0x7f471809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f4716ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f47180a50e0 0x7f47180a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51292/0 (socket says 172.21.15.86:51292) 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f4716ffd640 1 -- 172.21.15.86:0/3029661731 learned_addr learned my addr 172.21.15.86:0/3029661731 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): get_auth_request con 0x7f47180a5bd0 auth_method 0 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient(hunting): handle_auth_done global_id 124227 payload 274 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 1 -- 172.21.15.86:0/3029661731 >> v1:172.21.15.203:6789/0 conn(0x7f471813d2f0 legacy=0x7f471813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 1 -- 172.21.15.86:0/3029661731 >> v1:172.21.15.86:6789/0 conn(0x7f47180a50e0 legacy=0x7f47180a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.202+0000 7f47167fc640 1 -- 172.21.15.86:0/3029661731 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471809c3d0 con 0x7f47180a5bd0 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f47167fc640 1 -- 172.21.15.86:0/3029661731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a5bd0 msgr2=0x7f471809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f47167fc640 1 --2- 172.21.15.86:0/3029661731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a5bd0 0x7f471809cd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f47080097c0 tx=0x7f470801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7331d60e4cf30a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 1 -- 172.21.15.86:0/3029661731 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080b0070 con 0x7f47180a5bd0 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: dump: 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 5 monclient: authenticate success, global_id 124227 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 1 -- 172.21.15.86:0/3029661731 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47080ab600 con 0x7f47180a5bd0 2024-09-17T15:39:03.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 4 set_mon_vals no callback set 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 1 -- 172.21.15.86:0/3029661731 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080ab9d0 con 0x7f47180a5bd0 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f4715ffb640 10 monclient: dump: 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.210 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 10 monclient: shutdown 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 -- 172.21.15.86:0/3029661731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a5bd0 msgr2=0x7f471809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 --2- 172.21.15.86:0/3029661731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a5bd0 0x7f471809cd60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f47080097c0 tx=0x7f470801c360 comp rx=0 tx=0).stop 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 -- 172.21.15.86:0/3029661731 shutdown_connections 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 --2- 172.21.15.86:0/3029661731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a5bd0 0x7f471809cd60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 -- 172.21.15.86:0/3029661731 >> 172.21.15.86:0/3029661731 conn(0x7f47180aaae0 msgr2=0x7f47180aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 -- 172.21.15.86:0/3029661731 shutdown_connections 2024-09-17T15:39:03.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.206+0000 7f471d865640 1 -- 172.21.15.86:0/3029661731 wait complete. 2024-09-17T15:39:03.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient: monmap: 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 Processor -- start 2024-09-17T15:39:03.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 -- start start 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient: init 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 0x7f47181cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): picked mon.noname-a con 0x7f47180a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47180a5bd0 0x7f4718138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): picked mon.noname-b con 0x7f47180a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471813d2f0 0x7f471809e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): picked mon.noname-c con 0x7f471813d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f47181439b0 con 0x7f47180a50e0 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4718143830 con 0x7f47180a5bd0 2024-09-17T15:39:03.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4718143b30 con 0x7f471813d2f0 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f471d865640 10 monclient(hunting): authenticate will time out at 1676.177177s 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 0x7f47181cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 0x7f47181cfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36206/0 (socket says 172.21.15.86:36206) 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 -- 172.21.15.86:0/2372926120 learned_addr learned my addr 172.21.15.86:0/2372926120 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): get_auth_request con 0x7f47180a50e0 auth_method 0 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471813d2f0 0x7f471809e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): get_auth_request con 0x7f471813d2f0 auth_method 0 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47180a5bd0 0x7f4718138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient(hunting): handle_auth_done global_id 124230 payload 824 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471813d2f0 msgr2=0x7f471809e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f471813d2f0 0x7f471809e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47180a5bd0 msgr2=0x7f4718138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47180a5bd0 0x7f4718138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47180a0e40 con 0x7f47180a50e0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47167fc640 10 monclient: get_auth_request con 0x7f47180a5bd0 auth_method 0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47180a5bd0 0x7f4718138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 msgr2=0x7f47181cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.210+0000 7f4716ffd640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 0x7f47181cfef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f470000c4a0 tx=0x7f470000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700017070 con 0x7f47180a50e0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: dump: 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700007070 con 0x7f47180a50e0 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 5 monclient: authenticate success, global_id 124230 2024-09-17T15:39:03.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700007460 con 0x7f47180a50e0 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 10 monclient: dump: 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.222 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 monclient: _renew_subs 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47157fa640 4 set_mon_vals no callback set 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47180a18d0 con 0x7f47180a50e0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 monclient: _renew_subs 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47180a0240 con 0x7f47180a50e0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 0x7f471809ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4718121620 con 0x7f47181219c0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f470001a070 con 0x7f47180a50e0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 0x7f471809ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient: get_auth_request con 0x7f47181219c0 auth_method 0 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46f803bb60 0x7f46f803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(396..396 src has 213..396) ==== 4799+0+0 (secure 0 0 0) 0x7f470004f430 con 0x7f47180a50e0 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47177fe640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46f803bb60 0x7f46f803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47177fe640 10 monclient: get_auth_request con 0x7f46f803bb60 auth_method 0 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 10 monclient(hunting): handle_auth_done global_id 134113 payload 824 2024-09-17T15:39:03.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.214+0000 7f47167fc640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f471809cd60 con 0x7f47181219c0 2024-09-17T15:39:03.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f47167fc640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 msgr2=0x7f471809ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 0x7f471809ee00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47180aa620 tx=0x7f470801aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f47177fe640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46f803bb60 0x7f46f803e020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f471809c980 tx=0x7f4710003d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080bb020 con 0x7f47181219c0 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 10 monclient: dump: 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.226 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f47080b1040 con 0x7f47181219c0 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 msgr2=0x7f471809ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 0x7f471809ee00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47180aa620 tx=0x7f470801aba0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 msgr2=0x7f471809ee00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.218+0000 7f470ffff640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47181219c0 0x7f471809ee00 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f47180aa620 tx=0x7f470801aba0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.226+0000 7f471d865640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.226+0000 7f471d865640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 0x7f47181392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.226+0000 7f471d865640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.226+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f471809f720 con 0x7f471809f340 2024-09-17T15:39:03.228 INFO:tasks.rados.rados.0.smithi203.stdout:1750: done (0 left) 2024-09-17T15:39:03.228 INFO:tasks.rados.rados.0.smithi203.stdout:1751: read oid 7 snap -1 2024-09-17T15:39:03.228 INFO:tasks.rados.rados.0.smithi203.stdout:1751: expect (ObjNum 612 snap 185 seq_num 612) 2024-09-17T15:39:03.228 INFO:tasks.rados.rados.0.smithi203.stdout:1752: snap_remove snap 190 2024-09-17T15:39:03.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.230+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 0x7f47181392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.230+0000 7f47167fc640 10 monclient: get_auth_request con 0x7f471809f340 auth_method 0 2024-09-17T15:39:03.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.230+0000 7f47167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.230+0000 7f47167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.230+0000 7f47167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.234+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.234+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.234+0000 7f47167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.234+0000 7f47167fc640 10 monclient(hunting): handle_auth_done global_id 134116 payload 824 2024-09-17T15:39:03.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.234+0000 7f47167fc640 1 -- 172.21.15.86:0/2372926120 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4718138ea0 con 0x7f471809f340 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f47167fc640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 msgr2=0x7f47181392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f47167fc640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 0x7f47181392d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c10 tx=0x7f4708019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47080bb070 con 0x7f471809f340 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 10 monclient: dump: 2024-09-17T15:39:03.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f47080c2220 con 0x7f471809f340 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 msgr2=0x7f47181392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 0x7f47181392d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c10 tx=0x7f4708019100 comp rx=0 tx=0).stop 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 msgr2=0x7f47181392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f470ffff640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471809f340 0x7f47181392d0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4708000c10 tx=0x7f4708019100 comp rx=0 tx=0).stop 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:03.239 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7607, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:03.240 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:03.241 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:03.242 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:03.243 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.244 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.246 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:03.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:03.249 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:03.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:03.251 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46f803bb60 msgr2=0x7f46f803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f46f803bb60 0x7f46f803e020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f471809c980 tx=0x7f4710003d50 comp rx=0 tx=0).stop 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 10 monclient: shutdown 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 msgr2=0x7f47181cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 --2- 172.21.15.86:0/2372926120 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47180a50e0 0x7f47181cfef0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f470000c4a0 tx=0x7f470000c960 comp rx=0 tx=0).stop 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f47177fe640 1 -- 172.21.15.86:0/2372926120 reap_dead start 2024-09-17T15:39:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f47177fe640 1 -- 172.21.15.86:0/2372926120 reap_dead start 2024-09-17T15:39:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 shutdown_connections 2024-09-17T15:39:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 >> 172.21.15.86:0/2372926120 conn(0x7f47180aaae0 msgr2=0x7f47180a3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 shutdown_connections 2024-09-17T15:39:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.238+0000 7f471d865640 1 -- 172.21.15.86:0/2372926120 wait complete. 2024-09-17T15:39:03.259 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:03.259 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:39:03.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:39:03.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: get_monmap_and_config 2024-09-17T15:39:03.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: monmap: 2024-09-17T15:39:03.370 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.370 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:03.373914+0000 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:03.373914+0000 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:03.371 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 Processor -- start 2024-09-17T15:39:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 -- start start 2024-09-17T15:39:03.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: init 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 0x7fd5b809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b8069260 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5b8074c30 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5b809c6c0 0x7fd5b813ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b809c6c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5b809e410 con 0x7fd5b8074c30 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b809d150 con 0x7fd5b809c6c0 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd5b809ce80 con 0x7fd5b8069260 2024-09-17T15:39:03.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5c0550640 10 monclient(hunting): authenticate will time out at 1676.339347s 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5b809c6c0 0x7fd5b813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5bdaeb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd5b8074c30 0x7fd5b809bfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51308/0 (socket says 172.21.15.86:51308) 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/3385287508 learned_addr learned my addr 172.21.15.86:0/3385287508 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): get_auth_request con 0x7fd5b809c6c0 auth_method 0 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient(hunting): handle_auth_done global_id 124233 payload 274 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5be2ec640 1 --2- 172.21.15.86:0/3385287508 >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 0x7fd5b809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 -- 172.21.15.86:0/3385287508 >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 msgr2=0x7fd5b809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 --2- 172.21.15.86:0/3385287508 >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 0x7fd5b809ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 -- 172.21.15.86:0/3385287508 >> v1:172.21.15.86:6789/0 conn(0x7fd5b8074c30 legacy=0x7fd5b809bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5beaed640 1 -- 172.21.15.86:0/3385287508 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b809e6f0 con 0x7fd5b809c6c0 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5be2ec640 10 monclient: get_auth_request con 0x7fd5b8069260 auth_method 0 2024-09-17T15:39:03.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.370+0000 7fd5be2ec640 1 --2- 172.21.15.86:0/3385287508 >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 0x7fd5b809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5beaed640 1 -- 172.21.15.86:0/3385287508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b809c6c0 msgr2=0x7fd5b813ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5beaed640 1 --2- 172.21.15.86:0/3385287508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b809c6c0 0x7fd5b813ba80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00097c0 tx=0x7fd5b001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f27f32873324d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 1 -- 172.21.15.86:0/3385287508 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00b0070 con 0x7fd5b809c6c0 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: dump: 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 1 -- 172.21.15.86:0/3385287508 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5b00ab600 con 0x7fd5b809c6c0 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 1 -- 172.21.15.86:0/3385287508 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00ab9d0 con 0x7fd5b809c6c0 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bd2ea640 10 monclient: dump: 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 5 monclient: authenticate success, global_id 124233 2024-09-17T15:39:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:03.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 4 set_mon_vals no callback set 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: shutdown 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- 172.21.15.86:0/3385287508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b809c6c0 msgr2=0x7fd5b813ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- 172.21.15.86:0/3385287508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b809c6c0 0x7fd5b813ba80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00097c0 tx=0x7fd5b001c320 comp rx=0 tx=0).stop 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- 172.21.15.86:0/3385287508 shutdown_connections 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- 172.21.15.86:0/3385287508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b809c6c0 0x7fd5b813ba80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- 172.21.15.86:0/3385287508 >> v2:172.21.15.203:3300/0 conn(0x7fd5b8069260 0x7fd5b809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- 172.21.15.86:0/3385287508 >> 172.21.15.86:0/3385287508 conn(0x7fd5b80a6440 msgr2=0x7fd5b80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- 172.21.15.86:0/3385287508 shutdown_connections 2024-09-17T15:39:03.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- 172.21.15.86:0/3385287508 wait complete. 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: build_initial_monmap 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: monmap: 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 Processor -- start 2024-09-17T15:39:03.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- start start 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: init 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5b8069260 0x7fd5b81200c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b8069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b8074c30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b809c6c0 0x7fd5b8120b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b809c6c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b813c350 con 0x7fd5b8074c30 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b813c1d0 con 0x7fd5b809c6c0 2024-09-17T15:39:03.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd5b813c4d0 con 0x7fd5b8069260 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5c0550640 10 monclient(hunting): authenticate will time out at 1676.347417s 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5b8069260 0x7fd5b81200c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36218/0 (socket says 172.21.15.86:36218) 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/4066265836 learned_addr learned my addr 172.21.15.86:0/4066265836 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): get_auth_request con 0x7fd5b8074c30 auth_method 0 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): get_auth_request con 0x7fd5b8069260 auth_method 0 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:03.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5be2ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient(hunting): handle_auth_done global_id 124236 payload 824 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5beaed640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b809c6c0 0x7fd5b8120b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5b8069260 msgr2=0x7fd5b81200c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5b8069260 0x7fd5b81200c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b809c6c0 msgr2=0x7fd5b8120b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b809c6c0 0x7fd5b8120b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient: _finish_hunting 0 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 monclient: found mon.noname-a 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.378+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/4066265836 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b8123c00 con 0x7fd5b8074c30 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5bdaeb640 1 -- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 msgr2=0x7fd5b8120600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5bdaeb640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000cc10 tx=0x7fd5a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a0012400 con 0x7fd5b8074c30 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: dump: 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5a00125e0 con 0x7fd5b8074c30 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 5 monclient: authenticate success, global_id 124236 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _renew_subs 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b8123ee0 con 0x7fd5b8074c30 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a00129d0 con 0x7fd5b8074c30 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 10 monclient: dump: 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:03.391 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b7fff640 4 set_mon_vals no callback set 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _renew_subs 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b809d460 con 0x7fd5b8074c30 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b813c960 con 0x7fd5b8074c30 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd5a0012bf0 con 0x7fd5b8074c30 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 0x7fd59803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.382+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(397..397 src has 213..397) ==== 4815+0+0 (secure 0 0 0) 0x7fd5a00505a0 con 0x7fd5b8074c30 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.386+0000 7fd5be2ec640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 0x7fd59803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.386+0000 7fd5be2ec640 10 monclient: get_auth_request con 0x7fd59803bbe0 auth_method 0 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.386+0000 7fd5be2ec640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 0x7fd59803e0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800ab60 tx=0x7fd5a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.390+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd5a001a360 con 0x7fd5b8074c30 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.390+0000 7fd5b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:03.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.390+0000 7fd5b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:03.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.622+0000 7fd5c0550640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:39:03.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.622+0000 7fd5c0550640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:39:03.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.622+0000 7fd5c0550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:03.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.622+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd5b806d4f0 con 0x7fd5b8074c30 2024-09-17T15:39:03.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5b6ffd640 1 -- 172.21.15.86:0/4066265836 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd5a001a540 con 0x7fd5b8074c30 2024-09-17T15:39:03.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:39:03.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:03.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=3446310684,config=3541143443,health=467171742,kv=3885551602,logm=3620148476}) 2024-09-17T15:39:03.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=3055210890,config=3541143443,health=467171742,kv=1440880585,logm=2850185545}) 2024-09-17T15:39:03.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=12,kv=10,logm=57} crc {auth=3446310684,config=3541143443,health=467171742,kv=3885551602,logm=3620148476}) 2024-09-17T15:39:03.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=727004554,config=3541143443,health=467171742,kv=3591778543,logm=1880303418}) 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 msgr2=0x7fd59803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 0x7fd59803e0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800ab60 tx=0x7fd5a8008040 comp rx=0 tx=0).stop 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 10 monclient: shutdown 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 msgr2=0x7fd5b8120600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000cc10 tx=0x7fd5a000a430 comp rx=0 tx=0).stop 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 shutdown_connections 2024-09-17T15:39:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd59803bbe0 0x7fd59803e0a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5b809c6c0 0x7fd5b8120b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5b8074c30 0x7fd5b8120600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 --2- 172.21.15.86:0/4066265836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5b8069260 0x7fd5b81200c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 >> 172.21.15.86:0/4066265836 conn(0x7fd5b80a6440 msgr2=0x7fd5b8139ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 shutdown_connections 2024-09-17T15:39:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:03.626+0000 7fd5c0550640 1 -- 172.21.15.86:0/4066265836 wait complete. 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527617474}) 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3352804277}) 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527617474}) 2024-09-17T15:39:03.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587114631}) 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411093695}) 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217412913}) 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411093695}) 2024-09-17T15:39:03.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.630+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613804540}) 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745206836}) 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3841056984}) 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745206836}) 2024-09-17T15:39:03.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.634+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2737197143}) 2024-09-17T15:39:03.644 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:39:03.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132362351}) 2024-09-17T15:39:03.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=867932748}) 2024-09-17T15:39:03.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3132362351}) 2024-09-17T15:39:03.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146878623}) 2024-09-17T15:39:03.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=71794023}) 2024-09-17T15:39:03.690 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984121766}) 2024-09-17T15:39:03.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=71794023}) 2024-09-17T15:39:03.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.690+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037223524}) 2024-09-17T15:39:03.692 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902493309}) 2024-09-17T15:39:03.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=603782302}) 2024-09-17T15:39:03.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902493309}) 2024-09-17T15:39:03.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.694+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581101974}) 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291285501}) 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796205685}) 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291285501}) 2024-09-17T15:39:03.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.838+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619819802}) 2024-09-17T15:39:03.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690798882}) 2024-09-17T15:39:03.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465382158}) 2024-09-17T15:39:03.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.842 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690798882}) 2024-09-17T15:39:03.842 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318430082}) 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168964090}) 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=504256331}) 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1168964090}) 2024-09-17T15:39:03.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.842+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=142581365}) 2024-09-17T15:39:03.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511829292}) 2024-09-17T15:39:03.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3996939876}) 2024-09-17T15:39:03.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511829292}) 2024-09-17T15:39:03.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.846+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638044271}) 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937741546}) 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=389069215}) 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1937741546}) 2024-09-17T15:39:03.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2001122938}) 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642740068}) 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908666902}) 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642740068}) 2024-09-17T15:39:03.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.850+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723331786}) 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881673072}) 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3905796565}) 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3881673072}) 2024-09-17T15:39:03.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126259238}) 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083822279}) 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1847010449}) 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1083822279}) 2024-09-17T15:39:03.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.854+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022989128}) 2024-09-17T15:39:03.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065187952}) 2024-09-17T15:39:03.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927504528}) 2024-09-17T15:39:03.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065187952}) 2024-09-17T15:39:03.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.858+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2797854115}) 2024-09-17T15:39:03.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702132352}) 2024-09-17T15:39:03.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949064051}) 2024-09-17T15:39:03.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702132352}) 2024-09-17T15:39:03.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267915090}) 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444721485}) 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019177246}) 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444721485}) 2024-09-17T15:39:03.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.862+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2911599293}) 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751227145}) 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603218617}) 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2751227145}) 2024-09-17T15:39:03.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:03.866+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968324296}) 2024-09-17T15:39:04.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778718241}) 2024-09-17T15:39:04.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823124432}) 2024-09-17T15:39:04.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778718241}) 2024-09-17T15:39:04.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.158+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1620908416}) 2024-09-17T15:39:04.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.174+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.174+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126867457}) 2024-09-17T15:39:04.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.174+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893952410}) 2024-09-17T15:39:04.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.174+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.178+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126867457}) 2024-09-17T15:39:04.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.178+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294427959}) 2024-09-17T15:39:04.183 INFO:tasks.rados.rados.0.smithi203.stdout:1752: done (1 left) 2024-09-17T15:39:04.183 INFO:tasks.rados.rados.0.smithi203.stdout:1753: write (excl) oid 47 current snap is 196 2024-09-17T15:39:04.183 INFO:tasks.rados.rados.0.smithi203.stdout:1753: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-17T15:39:04.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551149447}) 2024-09-17T15:39:04.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=752374899}) 2024-09-17T15:39:04.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551149447}) 2024-09-17T15:39:04.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.194+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=103495652}) 2024-09-17T15:39:04.204 INFO:tasks.rados.rados.0.smithi203.stdout:1753: writing smithi20328086-47 from 605217 to 1344635 tid 1 2024-09-17T15:39:04.221 INFO:tasks.rados.rados.0.smithi203.stdout:1753: writing smithi20328086-47 from 2124726 to 2605217 tid 2 2024-09-17T15:39:04.221 INFO:tasks.rados.rados.0.smithi203.stdout:1754: delete oid 34 current snap is 196 2024-09-17T15:39:04.221 INFO:tasks.rados.rados.0.smithi203.stdout:1751: expect (ObjNum 612 snap 185 seq_num 612) 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795133583}) 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3078616815}) 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795133583}) 2024-09-17T15:39:04.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.238+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267726437}) 2024-09-17T15:39:04.239 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1861530 2024-09-17T15:39:04.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617500436}) 2024-09-17T15:39:04.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397641315}) 2024-09-17T15:39:04.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.254 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617500436}) 2024-09-17T15:39:04.254 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.254+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547379097}) 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115482591}) 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845991879}) 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3115482591}) 2024-09-17T15:39:04.260 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.258+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832474184}) 2024-09-17T15:39:04.329 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:04.364 INFO:tasks.rados.rados.0.smithi203.stdout:1753: finishing write tid 1 to smithi20328086-47 2024-09-17T15:39:04.364 INFO:tasks.rados.rados.0.smithi203.stdout:1753: finishing write tid 2 to smithi20328086-47 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: finishing write tid 3 to smithi20328086-47 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: finishing write tid 4 to smithi20328086-47 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: oid 47 updating version 0 to 556 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: oid 47 version 556 is already newer than 555 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: oid 47 updating version 556 to 557 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 557 (ObjNum 631 snap 196 seq_num 631) dirty exists 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: left oid 47 (ObjNum 631 snap 196 seq_num 631) 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1751: done (2 left) 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1753: done (1 left) 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1754: done (0 left) 2024-09-17T15:39:04.365 INFO:tasks.rados.rados.0.smithi203.stdout:1755: snap_remove snap 185 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805264605}) 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3298379359}) 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805264605}) 2024-09-17T15:39:04.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.526+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135209482}) 2024-09-17T15:39:04.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109070235}) 2024-09-17T15:39:04.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3823098055}) 2024-09-17T15:39:04.534 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.534 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109070235}) 2024-09-17T15:39:04.534 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3247172505}) 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320379349}) 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481259563}) 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320379349}) 2024-09-17T15:39:04.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.534+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005087648}) 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201808916}) 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093936587}) 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201808916}) 2024-09-17T15:39:04.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.538+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3260524488}) 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070984474}) 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975999973}) 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070984474}) 2024-09-17T15:39:04.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966065207}) 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67319678}) 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788379957}) 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67319678}) 2024-09-17T15:39:04.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.542+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2527672922}) 2024-09-17T15:39:04.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316381993}) 2024-09-17T15:39:04.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3142324024}) 2024-09-17T15:39:04.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316381993}) 2024-09-17T15:39:04.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.546+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063533372}) 2024-09-17T15:39:04.551 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.551 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742783721}) 2024-09-17T15:39:04.551 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176289457}) 2024-09-17T15:39:04.551 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742783721}) 2024-09-17T15:39:04.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.550+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875851255}) 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860021342}) 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=953585532}) 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860021342}) 2024-09-17T15:39:04.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244674537}) 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061902943}) 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=421695188}) 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061902943}) 2024-09-17T15:39:04.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.554+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2995321188}) 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43} crc {logm=53467295,mdsmap=3164546943,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2426292082,mon_config_key=4253633104,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=790428332,osd_snap=1971403495}) 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41} crc {logm=2920631337,mdsmap=2815901553,mgr=1899709286,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3651287637,mon_config_key=1034281653,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=790428332,osd_snap=602077608}) 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43} crc {logm=53467295,mdsmap=3164546943,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2426292082,mon_config_key=4253633104,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=790428332,osd_snap=1971403495}) 2024-09-17T15:39:04.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.558+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=40} crc {logm=1386000560,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2426292082,mon_config_key=173124949,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=1318519245,osd_pg_creating=790428332,osd_snap=1355362765}) 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=2481665718,osdmap=2090457497}) 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=2391667112,osdmap=1699435711}) 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=2481665718,osdmap=2090457497}) 2024-09-17T15:39:04.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.562+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=1658503820,osdmap=3559700556}) 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2716128648}) 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4186075989}) 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2716128648}) 2024-09-17T15:39:04.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.566+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2838458866}) 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1181438739}) 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1653201277}) 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1181438739}) 2024-09-17T15:39:04.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.574+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1125314456}) 2024-09-17T15:39:04.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.590+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:39:04.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.590+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=7} crc {osdmap=2079616010}) 2024-09-17T15:39:04.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:39:04.590+0000 7fa204293640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=6} crc {osdmap=2215584348}) 2024-09-17T15:39:04.645 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:39:04.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: get_monmap_and_config 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: build_initial_monmap 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: monmap: 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:04.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:04.777858+0000 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:04.777858+0000 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:04.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 Processor -- start 2024-09-17T15:39:04.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 -- start start 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: init 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2f40689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe2f40694c0 0x7fe2f406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f40694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2f413aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409c8b0 con 0x7fe2f413aff0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f409cb90 con 0x7fe2f40689d0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f4074500 con 0x7fe2f40694c0 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2fa23f640 10 monclient(hunting): authenticate will time out at 1677.743298s 2024-09-17T15:39:04.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f3fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe2f413aff0 0x7fe2f413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51320/0 (socket says 172.21.15.86:51320) 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f3fff640 1 -- 172.21.15.86:0/373580458 learned_addr learned my addr 172.21.15.86:0/373580458 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f2ffd640 1 --2- 172.21.15.86:0/373580458 >> v2:172.21.15.203:3300/0 conn(0x7fe2f40694c0 0x7fe2f406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f2ffd640 10 monclient(hunting): get_auth_request con 0x7fe2f40694c0 auth_method 0 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.774+0000 7fe2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_done global_id 134119 payload 274 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/373580458 >> v1:172.21.15.203:6789/0 conn(0x7fe2f40689d0 legacy=0x7fe2f4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/373580458 >> v1:172.21.15.86:6789/0 conn(0x7fe2f413aff0 legacy=0x7fe2f413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:04.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:39:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 1 monclient: found mon.noname-c 2024-09-17T15:39:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:39:04.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.778+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/373580458 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f409ce70 con 0x7fe2f40694c0 2024-09-17T15:39:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/373580458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40694c0 msgr2=0x7fe2f406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f2ffd640 1 --2- 172.21.15.86:0/373580458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40694c0 0x7fe2f406f840 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4069a70 tx=0x7fe2e401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbcf409d17fcb7fd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 1 -- 172.21.15.86:0/373580458 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40b0070 con 0x7fe2f40694c0 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: dump: 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:04.782 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: _finish_auth 0 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 1 -- 172.21.15.86:0/373580458 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e40ab600 con 0x7fe2f40694c0 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 1 -- 172.21.15.86:0/373580458 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f409c2d0 con 0x7fe2f40694c0 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2f27fc640 10 monclient: dump: 2024-09-17T15:39:04.783 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 5 monclient: authenticate success, global_id 134119 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:04.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 4 set_mon_vals no callback set 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: shutdown 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- 172.21.15.86:0/373580458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40694c0 msgr2=0x7fe2f406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/373580458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40694c0 0x7fe2f406f840 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4069a70 tx=0x7fe2e401c340 comp rx=0 tx=0).stop 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- 172.21.15.86:0/373580458 shutdown_connections 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/373580458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40694c0 0x7fe2f406f840 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- 172.21.15.86:0/373580458 >> 172.21.15.86:0/373580458 conn(0x7fe2f40a6440 msgr2=0x7fe2f40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- 172.21.15.86:0/373580458 shutdown_connections 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- 172.21.15.86:0/373580458 wait complete. 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: build_initial_monmap 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: monmap: 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:04.790 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 Processor -- start 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- start start 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: init 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40689d0 0x7fe2f4071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f40694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2f413aff0 0x7fe2f40720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f413aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f409dc70 con 0x7fe2f413aff0 2024-09-17T15:39:04.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f409daf0 con 0x7fe2f40689d0 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f409ddf0 con 0x7fe2f40694c0 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.782+0000 7fe2fa23f640 10 monclient(hunting): authenticate will time out at 1677.751876s 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2f413aff0 0x7fe2f40720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59728/0 (socket says 172.21.15.86:59728) 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/141824368 learned_addr learned my addr 172.21.15.86:0/141824368 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): get_auth_request con 0x7fe2f40694c0 auth_method 0 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:04.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): get_auth_request con 0x7fe2f413aff0 auth_method 0 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f37fe640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40689d0 0x7fe2f4071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f37fe640 10 monclient(hunting): get_auth_request con 0x7fe2f40689d0 auth_method 0 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:04.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient(hunting): handle_auth_done global_id 104426 payload 824 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40689d0 msgr2=0x7fe2f4071620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40689d0 0x7fe2f4071620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2f413aff0 msgr2=0x7fe2f40720a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2f413aff0 0x7fe2f40720a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 monclient: found mon.noname-c 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:04.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/141824368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f4073460 con 0x7fe2f40694c0 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 -- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 msgr2=0x7fe2f4071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f2ffd640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40ae040 tx=0x7fe2e401ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40b0050 con 0x7fe2f40694c0 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: dump: 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:04.795 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: _finish_auth 0 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e40bc070 con 0x7fe2f40694c0 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e40b5a40 con 0x7fe2f40694c0 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 5 monclient: authenticate success, global_id 104426 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _renew_subs 2024-09-17T15:39:04.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f411d770 con 0x7fe2f40694c0 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2f1ffb640 4 set_mon_vals no callback set 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 10 monclient: dump: 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:04.797 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _renew_subs 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f411d490 con 0x7fe2f40694c0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f406f840 con 0x7fe2f40694c0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.786+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe2e40b5be0 con 0x7fe2f40694c0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.790+0000 7fe2dbfff640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 0x7fe2e003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.790+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(398..398 src has 213..398) ==== 4843+0+0 (secure 0 0 0) 0x7fe2e40ea360 con 0x7fe2f40694c0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.790+0000 7fe2f37fe640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 0x7fe2e003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:04.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.790+0000 7fe2f37fe640 10 monclient: get_auth_request con 0x7fe2e003bb90 auth_method 0 2024-09-17T15:39:04.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.790+0000 7fe2f37fe640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 0x7fe2e003e050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8004e30 tx=0x7fe2e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:04.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.794+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe2e40c5020 con 0x7fe2f40694c0 2024-09-17T15:39:04.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.794+0000 7fe2dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:04.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:04.794+0000 7fe2dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:05.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2fa23f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:05.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2fa23f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:05.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2fa23f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe2f411dc90 con 0x7fe2f40694c0 2024-09-17T15:39:05.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2dbfff640 1 -- 172.21.15.86:0/141824368 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe2e4016050 con 0x7fe2f40694c0 2024-09-17T15:39:05.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:05.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.030+0000 7fe2dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:05.030 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 msgr2=0x7fe2e003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 0x7fe2e003e050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8004e30 tx=0x7fe2e8009040 comp rx=0 tx=0).stop 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 10 monclient: shutdown 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 msgr2=0x7fe2f4071b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40ae040 tx=0x7fe2e401ab80 comp rx=0 tx=0).stop 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 shutdown_connections 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe2e003bb90 0x7fe2e003e050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2f413aff0 0x7fe2f40720a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2f40694c0 0x7fe2f4071b60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 --2- 172.21.15.86:0/141824368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe2f40689d0 0x7fe2f4071620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 >> 172.21.15.86:0/141824368 conn(0x7fe2f40a6440 msgr2=0x7fe2f4138fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 shutdown_connections 2024-09-17T15:39:05.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.034+0000 7fe2fa23f640 1 -- 172.21.15.86:0/141824368 wait complete. 2024-09-17T15:39:05.049 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:39:05.049 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:39:05.049 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:39:05.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.166+0000 7f862c3c6640 10 monclient: get_monmap_and_config 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.166+0000 7f862c3c6640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.166+0000 7f862c3c6640 10 monclient: monmap: 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.168 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:05.171658+0000 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:05.171658+0000 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:05.169 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 Processor -- start 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 -- start start 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient: init 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86240a01a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f86240a0580 0x7f862409f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f86240a0580 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86240a1380 0x7f862413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f86240a1380 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8624074e00 con 0x7f86240a01a0 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f862409e350 con 0x7f86240a1380 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f862409e630 con 0x7f86240a0580 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862c3c6640 10 monclient(hunting): authenticate will time out at 1678.137296s 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86240a1380 0x7f862413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86240a1380 0x7f862413ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36248/0 (socket says 172.21.15.86:36248) 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 -- 172.21.15.86:0/1968568657 learned_addr learned my addr 172.21.15.86:0/1968568657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): get_auth_request con 0x7f86240a1380 auth_method 0 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 1 --2- 172.21.15.86:0/1968568657 >> v2:172.21.15.203:3300/0 conn(0x7f86240a0580 0x7f862409f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): get_auth_request con 0x7f86240a0580 auth_method 0 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient(hunting): handle_auth_done global_id 124242 payload 274 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 -- 172.21.15.86:0/1968568657 >> v2:172.21.15.203:3300/0 conn(0x7f86240a0580 msgr2=0x7f862409f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 --2- 172.21.15.86:0/1968568657 >> v2:172.21.15.203:3300/0 conn(0x7f86240a0580 0x7f862409f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 -- 172.21.15.86:0/1968568657 >> v1:172.21.15.203:6789/0 conn(0x7f86240a01a0 legacy=0x7f86240a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 monclient: found mon.noname-a 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.170+0000 7f862a963640 1 -- 172.21.15.86:0/1968568657 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862409e910 con 0x7f86240a1380 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862a963640 1 -- 172.21.15.86:0/1968568657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a1380 msgr2=0x7f862413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862a963640 1 --2- 172.21.15.86:0/1968568657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a1380 0x7f862413ba80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f861c00af60 tx=0x7f861c01d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58db20d1d1579e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 1 -- 172.21.15.86:0/1968568657 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c00a2d0 con 0x7f86240a1380 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: dump: 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 5 monclient: authenticate success, global_id 124242 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 1 -- 172.21.15.86:0/1968568657 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861c0a67f0 con 0x7f86240a1380 2024-09-17T15:39:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 4 set_mon_vals no callback set 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 1 -- 172.21.15.86:0/1968568657 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f861c0a6bc0 con 0x7f86240a1380 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f8629160640 10 monclient: dump: 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 10 monclient: shutdown 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 -- 172.21.15.86:0/1968568657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a1380 msgr2=0x7f862413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1968568657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a1380 0x7f862413ba80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f861c00af60 tx=0x7f861c01d9b0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 -- 172.21.15.86:0/1968568657 shutdown_connections 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1968568657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a1380 0x7f862413ba80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1968568657 >> v2:172.21.15.203:3300/0 conn(0x7f86240a0580 0x7f862409f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 -- 172.21.15.86:0/1968568657 >> 172.21.15.86:0/1968568657 conn(0x7f86240a2f40 msgr2=0x7f86240a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 -- 172.21.15.86:0/1968568657 shutdown_connections 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 1 -- 172.21.15.86:0/1968568657 wait complete. 2024-09-17T15:39:05.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.174+0000 7f862c3c6640 10 monclient: monmap: 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 Processor -- start 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- start start 2024-09-17T15:39:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: init 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86240a01a0 0x7f86241cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f86240a01a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 0x7f8624135e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f86240a0580 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86240a1380 0x7f8624136360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f86240a1380 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f862413e390 con 0x7f86240a0580 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f862413e210 con 0x7f86240a01a0 2024-09-17T15:39:05.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f862413e510 con 0x7f86240a1380 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient(hunting): authenticate will time out at 1678.144699s 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 0x7f8624135e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 0x7f8624135e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36252/0 (socket says 172.21.15.86:36252) 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 -- 172.21.15.86:0/1869268838 learned_addr learned my addr 172.21.15.86:0/1869268838 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): get_auth_request con 0x7f86240a0580 auth_method 0 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86240a1380 0x7f8624136360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): get_auth_request con 0x7f86240a1380 auth_method 0 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a162640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86240a01a0 0x7f86241cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a162640 10 monclient(hunting): get_auth_request con 0x7f86240a01a0 auth_method 0 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a162640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862a162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient(hunting): handle_auth_done global_id 124245 payload 824 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86240a1380 msgr2=0x7f8624136360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86240a1380 0x7f8624136360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86240a01a0 msgr2=0x7f86241cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86240a01a0 0x7f86241cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 monclient: found mon.noname-a 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8624121c90 con 0x7f86240a0580 2024-09-17T15:39:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 msgr2=0x7f8624135e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8629961640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 0x7f8624135e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f861800e870 tx=0x7f861800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618017070 con 0x7f86240a0580 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: dump: 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f861800cb50 con 0x7f86240a0580 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 5 monclient: authenticate success, global_id 124245 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _renew_subs 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86241cdfe0 con 0x7f86240a0580 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8618013470 con 0x7f86240a0580 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f8616ffd640 10 monclient: dump: 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862895f640 4 set_mon_vals no callback set 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _renew_subs 2024-09-17T15:39:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.178+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86241227f0 con 0x7f86240a0580 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862c3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 0x7f862411edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f862409f010 con 0x7f862411e9d0 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 0x7f862411edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient: get_auth_request con 0x7f862411e9d0 auth_method 0 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f861801a070 con 0x7f86240a0580 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 10 monclient(hunting): handle_auth_done global_id 124248 payload 824 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86240008e0 con 0x7f862411e9d0 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f860803bb90 0x7f860803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(398..398 src has 213..398) ==== 4843+0+0 (secure 0 0 0) 0x7f8618050040 con 0x7f86240a0580 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a963640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f860803bb90 0x7f860803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a963640 10 monclient: get_auth_request con 0x7f860803bb90 auth_method 0 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 msgr2=0x7f862411edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f862a162640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 0x7f862411edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f862000e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620007450 con 0x7f862411e9d0 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 10 monclient: dump: 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8620007450 con 0x7f862411e9d0 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 msgr2=0x7f862411edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 0x7f862411edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f862000e910 comp rx=0 tx=0).stop 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 msgr2=0x7f862411edb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.182+0000 7f8616ffd640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411e9d0 0x7f862411edb0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f862000e910 comp rx=0 tx=0).stop 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.186+0000 7f862a963640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f860803bb90 0x7f860803e050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ac000 tx=0x7f861c0a8070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862c3c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:05.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 0x7f86241398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862c3c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8624121140 con 0x7f862411f2f0 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 0x7f86241398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient: get_auth_request con 0x7f862411f2f0 auth_method 0 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 10 monclient(hunting): handle_auth_done global_id 124251 payload 824 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.190+0000 7f862a162640 1 -- 172.21.15.86:0/1869268838 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8624121f50 con 0x7f862411f2f0 2024-09-17T15:39:05.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862a162640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 msgr2=0x7f86241398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862a162640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 0x7f86241398d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f8620003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8620006e60 con 0x7f862411f2f0 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 10 monclient: dump: 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f8620013420 con 0x7f862411f2f0 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 msgr2=0x7f86241398d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 0x7f86241398d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f8620003770 comp rx=0 tx=0).stop 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 msgr2=0x7f86241398d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f8616ffd640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f862411f2f0 0x7f86241398d0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f862000e450 tx=0x7f8620003770 comp rx=0 tx=0).stop 2024-09-17T15:39:05.202 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 76058, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.203 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.204 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.205 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.206 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.207 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:05.208 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.209 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.210 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.211 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.213 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.213 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.217 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:05.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:39:05.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:05.220 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f860803bb90 msgr2=0x7f860803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f860803bb90 0x7f860803e050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f861c0ac000 tx=0x7f861c0a8070 comp rx=0 tx=0).stop 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 10 monclient: shutdown 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 msgr2=0x7f8624135e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 --2- 172.21.15.86:0/1869268838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86240a0580 0x7f8624135e20 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f861800e870 tx=0x7f861800ed30 comp rx=0 tx=0).stop 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862a963640 1 -- 172.21.15.86:0/1869268838 reap_dead start 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862a963640 1 -- 172.21.15.86:0/1869268838 reap_dead start 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 shutdown_connections 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 >> 172.21.15.86:0/1869268838 conn(0x7f86240a2f40 msgr2=0x7f8624071e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 shutdown_connections 2024-09-17T15:39:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.194+0000 7f862c3c6640 1 -- 172.21.15.86:0/1869268838 wait complete. 2024-09-17T15:39:05.223 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:05.223 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:39:05.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:39:05.260 INFO:tasks.rados.rados.0.smithi203.stdout:1755: done (0 left) 2024-09-17T15:39:05.260 INFO:tasks.rados.rados.0.smithi203.stdout:1756: delete oid 50 current snap is 196 2024-09-17T15:39:05.265 INFO:tasks.rados.rados.0.smithi203.stdout:1756: done (0 left) 2024-09-17T15:39:05.265 INFO:tasks.rados.rados.0.smithi203.stdout:1757: write (excl) oid 39 current snap is 196 2024-09-17T15:39:05.266 INFO:tasks.rados.rados.0.smithi203.stdout:1757: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-17T15:39:05.289 INFO:tasks.rados.rados.0.smithi203.stdout:1757: writing smithi20328086-39 from 655638 to 1220803 tid 1 2024-09-17T15:39:05.308 INFO:tasks.rados.rados.0.smithi203.stdout:1757: writing smithi20328086-39 from 1900172 to 2321714 tid 2 2024-09-17T15:39:05.308 INFO:tasks.rados.rados.0.smithi203.stdout:1758: read oid 17 snap -1 2024-09-17T15:39:05.308 INFO:tasks.rados.rados.0.smithi203.stdout:1758: expect deleted 2024-09-17T15:39:05.308 INFO:tasks.rados.rados.0.smithi203.stdout:1759: snap_create 2024-09-17T15:39:05.308 INFO:tasks.rados.rados.0.smithi203.stdout:1757: finishing write tid 1 to smithi20328086-39 2024-09-17T15:39:05.316 INFO:tasks.rados.rados.0.smithi203.stdout:1757: finishing write tid 2 to smithi20328086-39 2024-09-17T15:39:05.316 INFO:tasks.rados.rados.0.smithi203.stdout:1757: finishing write tid 3 to smithi20328086-39 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:1757: finishing write tid 4 to smithi20328086-39 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:1757: oid 39 updating version 0 to 694 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:1757: oid 39 version 694 is already newer than 692 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:1757: oid 39 version 694 is already newer than 693 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 694 (ObjNum 634 snap 196 seq_num 634) dirty exists 2024-09-17T15:39:05.317 INFO:tasks.rados.rados.0.smithi203.stdout:1757: left oid 39 (ObjNum 634 snap 196 seq_num 634) 2024-09-17T15:39:05.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: get_monmap_and_config 2024-09-17T15:39:05.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: monmap: 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:05.348192+0000 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:05.348192+0000 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:05.345 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 Processor -- start 2024-09-17T15:39:05.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 -- start start 2024-09-17T15:39:05.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: init 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ed00a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ed00a5350 0x7f8ed00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed00a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ed013d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed00a68d0 con 0x7f8ed013d180 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed00a64a0 con 0x7f8ed00a4f70 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed009c230 con 0x7f8ed00a5350 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ed4a94640 10 monclient(hunting): authenticate will time out at 1678.313544s 2024-09-17T15:39:05.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ed00a5350 0x7f8ed00a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8eced9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8ed013d180 0x7f8ed013f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51332/0 (socket says 172.21.15.86:51332) 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ed00a5350 0x7f8ed00a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36276/0 (socket says 172.21.15.86:36276) 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8eced9d640 1 -- 172.21.15.86:0/1756695417 learned_addr learned my addr 172.21.15.86:0/1756695417 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): get_auth_request con 0x7f8ed00a5350 auth_method 0 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient(hunting): handle_auth_done global_id 124254 payload 274 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 -- 172.21.15.86:0/1756695417 >> v1:172.21.15.203:6789/0 conn(0x7f8ed00a4f70 legacy=0x7f8ed009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 -- 172.21.15.86:0/1756695417 >> v1:172.21.15.86:6789/0 conn(0x7f8ed013d180 legacy=0x7f8ed013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 monclient: found mon.noname-a 2024-09-17T15:39:05.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.346+0000 7f8ecdd9b640 1 -- 172.21.15.86:0/1756695417 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed009c510 con 0x7f8ed00a5350 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecdd9b640 1 -- 172.21.15.86:0/1756695417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00a5350 msgr2=0x7f8ed00a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecdd9b640 1 --2- 172.21.15.86:0/1756695417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00a5350 0x7f8ed00a5750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00097c0 tx=0x7f8ec001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafce0a3f40fa866 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 1 -- 172.21.15.86:0/1756695417 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00b0070 con 0x7f8ed00a5350 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: dump: 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.351 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 1 -- 172.21.15.86:0/1756695417 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec00ab600 con 0x7f8ed00a5350 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 5 monclient: authenticate success, global_id 124254 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 4 set_mon_vals no callback set 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 1 -- 172.21.15.86:0/1756695417 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00ab9d0 con 0x7f8ed00a5350 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ecd59a640 10 monclient: dump: 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 10 monclient: shutdown 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 -- 172.21.15.86:0/1756695417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00a5350 msgr2=0x7f8ed00a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/1756695417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00a5350 0x7f8ed00a5750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8ec00097c0 tx=0x7f8ec001c3f0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 -- 172.21.15.86:0/1756695417 shutdown_connections 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/1756695417 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed00a5350 0x7f8ed00a5750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 -- 172.21.15.86:0/1756695417 >> 172.21.15.86:0/1756695417 conn(0x7f8ed00a7240 msgr2=0x7f8ed00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 -- 172.21.15.86:0/1756695417 shutdown_connections 2024-09-17T15:39:05.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 -- 172.21.15.86:0/1756695417 wait complete. 2024-09-17T15:39:05.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 10 monclient: monmap: 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.350+0000 7f8ed4a94640 1 Processor -- start 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- start start 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: init 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed00a4f70 0x7f8ed00a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-c con 0x7f8ed00a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed00a5350 0x7f8ed0138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-b con 0x7f8ed00a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): picked mon.noname-a con 0x7f8ed013d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed01438d0 con 0x7f8ed013d180 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed0143750 con 0x7f8ed00a5350 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed0143a50 con 0x7f8ed00a4f70 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient(hunting): authenticate will time out at 1678.320522s 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36286/0 (socket says 172.21.15.86:36286) 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed00a4f70 0x7f8ed00a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 learned_addr learned my addr 172.21.15.86:0/2682560794 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): get_auth_request con 0x7f8ed013d180 auth_method 0 2024-09-17T15:39:05.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): get_auth_request con 0x7f8ed00a4f70 auth_method 0 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ecdd9b640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed00a5350 0x7f8ed0138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient(hunting): handle_auth_done global_id 124257 payload 824 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed00a4f70 msgr2=0x7f8ed00a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed00a4f70 0x7f8ed00a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed00a5350 msgr2=0x7f8ed0138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed00a5350 0x7f8ed0138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 monclient: found mon.noname-a 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed00a3970 con 0x7f8ed013d180 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ecdd9b640 10 monclient: get_auth_request con 0x7f8ed00a5350 auth_method 0 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ecdd9b640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ed00a5350 0x7f8ed0138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:05.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 msgr2=0x7f8ed01388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eced9d640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800caf0 tx=0x7f8ec800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800ae50 con 0x7f8ed013d180 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: dump: 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.364 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec8012400 con 0x7f8ed013d180 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 5 monclient: authenticate success, global_id 124257 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _renew_subs 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed00a1760 con 0x7f8ed013d180 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec80127d0 con 0x7f8ed013d180 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ebeffd640 10 monclient: dump: 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8eccd99640 4 set_mon_vals no callback set 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _renew_subs 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.354+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed0121750 con 0x7f8ed013d180 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ed4a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 0x7f8ed01d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed00a5750 con 0x7f8ed01d0060 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8ec80129f0 con 0x7f8ed013d180 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 0x7f8ed01d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient: get_auth_request con 0x7f8ed01d0060 auth_method 0 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8eb003bd50 0x7f8eb003e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(399..399 src has 213..399) ==== 4831+0+0 (secure 0 0 0) 0x7f8ec8050590 con 0x7f8ed013d180 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 10 monclient(hunting): handle_auth_done global_id 104435 payload 824 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ed00a5ff0 con 0x7f8ed01d0060 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ecdd9b640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8eb003bd50 0x7f8eb003e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ecdd9b640 10 monclient: get_auth_request con 0x7f8eb003bd50 auth_method 0 2024-09-17T15:39:05.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 msgr2=0x7f8ed01d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ece59c640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 0x7f8ed01d0440 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec400fa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4003600 con 0x7f8ed01d0060 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 10 monclient: dump: 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ec4018070 con 0x7f8ed01d0060 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 msgr2=0x7f8ed01d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 0x7f8ed01d0440 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec400fa20 comp rx=0 tx=0).stop 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 msgr2=0x7f8ed01d0440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ebeffd640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0060 0x7f8ed01d0440 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec400fa20 comp rx=0 tx=0).stop 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.358+0000 7f8ecdd9b640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8eb003bd50 0x7f8eb003e210 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00a6380 tx=0x7f8ec0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ed4a94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 0x7f8ed013d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ed4a94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed0121b30 con 0x7f8ed01d0980 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 0x7f8ed013d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient: get_auth_request con 0x7f8ed01d0980 auth_method 0 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 10 monclient(hunting): handle_auth_done global_id 104438 payload 824 2024-09-17T15:39:05.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.366+0000 7f8ece59c640 1 -- 172.21.15.86:0/2682560794 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ed0069920 con 0x7f8ed01d0980 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ece59c640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 msgr2=0x7f8ed013d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ece59c640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 0x7f8ed013d560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec4001030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4003ca0 con 0x7f8ed01d0980 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 10 monclient: dump: 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f8ec4003ca0 con 0x7f8ed01d0980 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 msgr2=0x7f8ed013d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 0x7f8ed013d560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec4001030 comp rx=0 tx=0).stop 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 msgr2=0x7f8ed013d560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ebeffd640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ed01d0980 0x7f8ed013d560 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400f560 tx=0x7f8ec4001030 comp rx=0 tx=0).stop 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 176245, 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:05.376 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.377 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.378 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:05.379 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:05.380 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:05.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.383 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:05.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:05.387 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:39:05.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:05.389 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8eb003bd50 msgr2=0x7f8eb003e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8eb003bd50 0x7f8eb003e210 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00a6380 tx=0x7f8ec0008040 comp rx=0 tx=0).stop 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 10 monclient: shutdown 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 msgr2=0x7f8ed01388d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800caf0 tx=0x7f8ec800a060 comp rx=0 tx=0).stop 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8eced9d640 1 -- 172.21.15.86:0/2682560794 reap_dead start 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 shutdown_connections 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 --2- 172.21.15.86:0/2682560794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ed013d180 0x7f8ed01388d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 >> 172.21.15.86:0/2682560794 conn(0x7f8ed00a7240 msgr2=0x7f8ed00a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 shutdown_connections 2024-09-17T15:39:05.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.370+0000 7f8ed4a94640 1 -- 172.21.15.86:0/2682560794 wait complete. 2024-09-17T15:39:05.392 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:05.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:39:05.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: get_monmap_and_config 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: monmap: 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:05.508850+0000 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:05.508850+0000 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:05.506 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 Processor -- start 2024-09-17T15:39:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 -- start start 2024-09-17T15:39:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: init 2024-09-17T15:39:05.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d740a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6d7413d180 0x7f6d7413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-c con 0x7f6d7413d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7409d430 con 0x7f6d740a5350 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6d7409c7f0 con 0x7f6d7413d180 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d740a68d0 con 0x7f6d740a4f70 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d7a8b0640 10 monclient(hunting): authenticate will time out at 1678.474318s 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59786/0 (socket says 172.21.15.86:59786) 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 -- 172.21.15.86:0/1611828545 learned_addr learned my addr 172.21.15.86:0/1611828545 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): get_auth_request con 0x7f6d740a4f70 auth_method 0 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d78e4d640 1 --2- 172.21.15.86:0/1611828545 >> v2:172.21.15.203:3300/0 conn(0x7f6d7413d180 0x7f6d7413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d78e4d640 10 monclient(hunting): get_auth_request con 0x7f6d7413d180 auth_method 0 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d78e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d78e4d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d78e4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 104441 payload 274 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 -- 172.21.15.86:0/1611828545 >> v2:172.21.15.203:3300/0 conn(0x7f6d7413d180 msgr2=0x7f6d7413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 --2- 172.21.15.86:0/1611828545 >> v2:172.21.15.203:3300/0 conn(0x7f6d7413d180 0x7f6d7413f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 -- 172.21.15.86:0/1611828545 >> v1:172.21.15.203:6789/0 conn(0x7f6d740a5350 legacy=0x7f6d740a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 monclient: found mon.noname-b 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.506+0000 7f6d73fff640 1 -- 172.21.15.86:0/1611828545 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d740a64a0 con 0x7f6d740a4f70 2024-09-17T15:39:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 1 -- 172.21.15.86:0/1611828545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 msgr2=0x7f6d7409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 1 --2- 172.21.15.86:0/1611828545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409e420 tx=0x7f6d6801c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c35caf8ee6debe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 1 -- 172.21.15.86:0/1611828545 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d680b0070 con 0x7f6d740a4f70 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: dump: 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 1 -- 172.21.15.86:0/1611828545 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d68004230 con 0x7f6d740a4f70 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 5 monclient: authenticate success, global_id 104441 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 1 -- 172.21.15.86:0/1611828545 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d68003620 con 0x7f6d740a4f70 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 4 set_mon_vals no callback set 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d72ffd640 10 monclient: dump: 2024-09-17T15:39:05.515 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.516 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: shutdown 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/1611828545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 msgr2=0x7f6d7409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/1611828545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409e420 tx=0x7f6d6801c3d0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/1611828545 shutdown_connections 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/1611828545 >> v2:172.21.15.203:3300/0 conn(0x7f6d7413d180 0x7f6d7413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/1611828545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4f70 0x7f6d7409cef0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/1611828545 >> 172.21.15.86:0/1611828545 conn(0x7f6d740a7240 msgr2=0x7f6d740a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/1611828545 shutdown_connections 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/1611828545 wait complete. 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: monmap: 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 Processor -- start 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- start start 2024-09-17T15:39:05.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: init 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d740a5350 0x7f6d741cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-a con 0x7f6d740a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d7413d180 0x7f6d741d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-b con 0x7f6d7413d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 0x7f6d741cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): picked mon.noname-c con 0x7f6d741d0810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741424e0 con 0x7f6d740a5350 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7406f9a0 con 0x7f6d7413d180 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d74142660 con 0x7f6d741d0810 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d7a8b0640 10 monclient(hunting): authenticate will time out at 1678.483728s 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 0x7f6d741cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 0x7f6d741cc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59798/0 (socket says 172.21.15.86:59798) 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d740a5350 0x7f6d741cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 learned_addr learned my addr 172.21.15.86:0/3809751763 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): get_auth_request con 0x7f6d741d0810 auth_method 0 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 10 monclient(hunting): get_auth_request con 0x7f6d740a5350 auth_method 0 2024-09-17T15:39:05.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.514+0000 7f6d78e4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d7413d180 0x7f6d741d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d737fe640 10 monclient(hunting): get_auth_request con 0x7f6d7413d180 auth_method 0 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 10 monclient(hunting): handle_auth_done global_id 104444 payload 824 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d7413d180 msgr2=0x7f6d741d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d7413d180 0x7f6d741d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d740a5350 msgr2=0x7f6d741cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d740a5350 0x7f6d741cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 monclient: found mon.noname-c 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7413d910 con 0x7f6d741d0810 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 msgr2=0x7f6d741cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d78e4d640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 0x7f6d741cc890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c007e40 tx=0x7f6d6c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c013790 con 0x7f6d741d0810 2024-09-17T15:39:05.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d6c019070 con 0x7f6d741d0810 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c013b90 con 0x7f6d741d0810 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 5 monclient: authenticate success, global_id 104444 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 monclient: _renew_subs 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d740a1760 con 0x7f6d741d0810 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:39:05.529 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d727fc640 4 set_mon_vals no callback set 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 monclient: _renew_subs 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d7409f710 con 0x7f6d741d0810 2024-09-17T15:39:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 0x7f6d741cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7409cef0 con 0x7f6d741ccdd0 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6d6c01e460 con 0x7f6d741d0810 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.518+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(399..399 src has 213..399) ==== 4831+0+0 (secure 0 0 0) 0x7f6d6c050850 con 0x7f6d741d0810 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 0x7f6d741cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient: get_auth_request con 0x7f6d741ccdd0 auth_method 0 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d737fe640 10 monclient: get_auth_request con 0x7f6d6003bbb0 auth_method 0 2024-09-17T15:39:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 134137 payload 824 2024-09-17T15:39:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d740a5ff0 con 0x7f6d741ccdd0 2024-09-17T15:39:05.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 msgr2=0x7f6d741cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 0x7f6d741cd1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6d74069c50 tx=0x7f6d68019170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409dcd0 tx=0x7f6d64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d680bb070 con 0x7f6d741ccdd0 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.522+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d68003a30 con 0x7f6d741ccdd0 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 msgr2=0x7f6d741cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 0x7f6d741cd1b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6d74069c50 tx=0x7f6d68019170 comp rx=0 tx=0).stop 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 msgr2=0x7f6d741cd1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.526+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d741ccdd0 0x7f6d741cd1b0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6d74069c50 tx=0x7f6d68019170 comp rx=0 tx=0).stop 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d7a8b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 0x7f6d740a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d7a8b0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d741cd6f0 con 0x7f6d740a1930 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 0x7f6d740a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient: get_auth_request con 0x7f6d740a1930 auth_method 0 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 134140 payload 824 2024-09-17T15:39:05.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.530+0000 7f6d73fff640 1 -- 172.21.15.86:0/3809751763 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d74069920 con 0x7f6d740a1930 2024-09-17T15:39:05.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d73fff640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 msgr2=0x7f6d740a1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 0x7f6d740a1d10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409d070 tx=0x7f6d680a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d680b0040 con 0x7f6d740a1930 2024-09-17T15:39:05.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f6d680c2270 con 0x7f6d740a1930 2024-09-17T15:39:05.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 msgr2=0x7f6d740a1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 0x7f6d740a1d10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409d070 tx=0x7f6d680a8000 comp rx=0 tx=0).stop 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 msgr2=0x7f6d740a1d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740a1930 0x7f6d740a1d10 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409d070 tx=0x7f6d680a8000 comp rx=0 tx=0).stop 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:39:05.539 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 114, 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9903, 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.540 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.541 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:05.542 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:05.543 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.544 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.546 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.548 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:05.549 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:05.550 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:05.551 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 msgr2=0x7f6d6003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d7409dcd0 tx=0x7f6d64009040 comp rx=0 tx=0).stop 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 10 monclient: shutdown 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 msgr2=0x7f6d741cc890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 --2- 172.21.15.86:0/3809751763 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d741d0810 0x7f6d741cc890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c007e40 tx=0x7f6d6c00e4b0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 reap_dead start 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d78e4d640 1 -- 172.21.15.86:0/3809751763 reap_dead start 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 shutdown_connections 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 >> 172.21.15.86:0/3809751763 conn(0x7f6d740a7240 msgr2=0x7f6d7413c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 shutdown_connections 2024-09-17T15:39:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.534+0000 7f6d7a8b0640 1 -- 172.21.15.86:0/3809751763 wait complete. 2024-09-17T15:39:05.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:39:05.554 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:39:05.554 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:39:05.554 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:39:05.554 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:39:05.555 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:39:05.555 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:39:05.567 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:39:05.567 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:39:05.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:05.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: get_monmap_and_config 2024-09-17T15:39:05.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: monmap: 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:05.729811+0000 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:05.729811+0000 2024-09-17T15:39:05.727 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:05.728 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:05.729 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 Processor -- start 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 -- start start 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: init 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-17T15:39:05.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 0x7fac8409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-c con 0x7fac840a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-b con 0x7fac840a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac84137e30 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac840a6f40 con 0x7fac84137e30 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fac840a65e0 con 0x7fac840a5070 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fac8409c2a0 con 0x7fac840a5450 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac89e13640 10 monclient(hunting): authenticate will time out at 1678.695420s 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59812/0 (socket says 172.21.15.86:59812) 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 -- 172.21.15.86:0/1091208218 learned_addr learned my addr 172.21.15.86:0/1091208218 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): get_auth_request con 0x7fac840a5450 auth_method 0 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac837fe640 1 --2- 172.21.15.86:0/1091208218 >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 0x7fac8409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient(hunting): handle_auth_done global_id 104447 payload 274 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 -- 172.21.15.86:0/1091208218 >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 msgr2=0x7fac8409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 --2- 172.21.15.86:0/1091208218 >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 0x7fac8409cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 -- 172.21.15.86:0/1091208218 >> v1:172.21.15.203:6789/0 conn(0x7fac84137e30 legacy=0x7fac84138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 monclient: found mon.noname-b 2024-09-17T15:39:05.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.726+0000 7fac82ffd640 1 -- 172.21.15.86:0/1091208218 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8409c580 con 0x7fac840a5450 2024-09-17T15:39:05.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.730+0000 7fac837fe640 10 monclient: get_auth_request con 0x7fac840a5070 auth_method 0 2024-09-17T15:39:05.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.730+0000 7fac837fe640 1 --2- 172.21.15.86:0/1091208218 >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 0x7fac8409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac82ffd640 1 -- 172.21.15.86:0/1091208218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 msgr2=0x7fac840a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac82ffd640 1 --2- 172.21.15.86:0/1091208218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac78009870 tx=0x7fac7801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f8fcda8f5cfb259 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 1 -- 172.21.15.86:0/1091208218 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac780b0070 con 0x7fac840a5450 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:05.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: dump: 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 1 -- 172.21.15.86:0/1091208218 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac780ab600 con 0x7fac840a5450 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 1 -- 172.21.15.86:0/1091208218 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac780ab9d0 con 0x7fac840a5450 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 5 monclient: authenticate success, global_id 104447 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 4 set_mon_vals no callback set 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac827fc640 10 monclient: dump: 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.743 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: shutdown 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- 172.21.15.86:0/1091208218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 msgr2=0x7fac840a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- 172.21.15.86:0/1091208218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fac78009870 tx=0x7fac7801c420 comp rx=0 tx=0).stop 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- 172.21.15.86:0/1091208218 shutdown_connections 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- 172.21.15.86:0/1091208218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac840a5450 0x7fac840a5860 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- 172.21.15.86:0/1091208218 >> v2:172.21.15.203:3300/0 conn(0x7fac840a5070 0x7fac8409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- 172.21.15.86:0/1091208218 >> 172.21.15.86:0/1091208218 conn(0x7fac840742a0 msgr2=0x7fac840746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- 172.21.15.86:0/1091208218 shutdown_connections 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- 172.21.15.86:0/1091208218 wait complete. 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: build_initial_monmap 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: monmap: 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:05.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 Processor -- start 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- start start 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: init 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac840a5070 0x7fac84136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-b con 0x7fac840a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:05.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 0x7fac8413ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-a con 0x7fac840a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): picked mon.noname-c con 0x7fac84137e30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fac84142c40 con 0x7fac840a5450 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac84142ac0 con 0x7fac840a5070 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fac84142dc0 con 0x7fac84137e30 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac89e13640 10 monclient(hunting): authenticate will time out at 1678.703684s 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac82ffd640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 msgr2=0x7fac8413ce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac82ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 0x7fac8413ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:05.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59824/0 (socket says 172.21.15.86:59824) 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 1 -- 172.21.15.86:0/1358385237 learned_addr learned my addr 172.21.15.86:0/1358385237 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): get_auth_request con 0x7fac84137e30 auth_method 0 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.734+0000 7fac83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 10 monclient(hunting): handle_auth_done global_id 104450 payload 824 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac837fe640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac840a5070 0x7fac84136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 -- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac840a5070 msgr2=0x7fac84136c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:05.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac840a5070 0x7fac84136c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 -- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 msgr2=0x7fac8413ce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 0x7fac8413ce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 10 monclient: _finish_hunting 0 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 monclient: found mon.noname-c 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 -- 172.21.15.86:0/1358385237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac8413e8e0 con 0x7fac84137e30 2024-09-17T15:39:05.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 -- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 msgr2=0x7fac8413d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c410 tx=0x7fac7400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74017070 con 0x7fac84137e30 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac83fff640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c410 tx=0x7fac7400c8d0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fac74003800 config(0 keys) 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: dump: 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.750 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: _finish_auth 0 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac74003800 con 0x7fac84137e30 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 5 monclient: authenticate success, global_id 104450 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac6bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac81ffb640 4 set_mon_vals no callback set 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _renew_subs 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac84121690 con 0x7fac84137e30 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _renew_subs 2024-09-17T15:39:05.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8413f440 con 0x7fac84137e30 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.738+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac8409cf60 con 0x7fac84137e30 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74003bf0 con 0x7fac84137e30 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:05.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 10 monclient: dump: 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:05.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fac7401a360 con 0x7fac84137e30 2024-09-17T15:39:05.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac6bfff640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 0x7fac7003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:05.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac837fe640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 0x7fac7003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:05.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac837fe640 10 monclient: get_auth_request con 0x7fac7003bb10 auth_method 0 2024-09-17T15:39:05.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.742+0000 7fac837fe640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 0x7fac7003dfd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac84140270 tx=0x7fac7c0084f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:05.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:05.746+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(399..399 src has 213..399) ==== 4831+0+0 (secure 0 0 0) 0x7fac7404f320 con 0x7fac84137e30 2024-09-17T15:39:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.362+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fac74003e10 con 0x7fac84137e30 2024-09-17T15:39:06.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.362+0000 7fac6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:06.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.362+0000 7fac6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac89e13640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac89e13640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac89e13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fac84140010 con 0x7fac84137e30 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac6bfff640 1 -- 172.21.15.86:0/1358385237 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fac7404f0d0 con 0x7fac84137e30 2024-09-17T15:39:06.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:06.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.598+0000 7fac6bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:06.599 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 msgr2=0x7fac7003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 0x7fac7003dfd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fac84140270 tx=0x7fac7c0084f0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 10 monclient: shutdown 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 msgr2=0x7fac8413d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c410 tx=0x7fac7400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 shutdown_connections 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fac7003bb10 0x7fac7003dfd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac84137e30 0x7fac8413d360 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac840a5450 0x7fac8413ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 --2- 172.21.15.86:0/1358385237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac840a5070 0x7fac84136c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:06.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 >> 172.21.15.86:0/1358385237 conn(0x7fac840742a0 msgr2=0x7fac840a9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:06.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 shutdown_connections 2024-09-17T15:39:06.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:06.602+0000 7fac89e13640 1 -- 172.21.15.86:0/1358385237 wait complete. 2024-09-17T15:39:09.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: get_monmap_and_config 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: build_initial_monmap 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: monmap: 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:09.749466+0000 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:09.749466+0000 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:09.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:09.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:09.747 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:09.747 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:09.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 Processor -- start 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 -- start start 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: init 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 0x7fd074068c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-a con 0x7fd07409ba70 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd07409eea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd07409c560 0x7fd0741436a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-c con 0x7fd07409c560 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd07409f2b0 con 0x7fd07409eea0 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd07409d7e0 con 0x7fd07409ba70 2024-09-17T15:39:09.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd07409e4c0 con 0x7fd07409c560 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07ad6d640 10 monclient(hunting): authenticate will time out at 1682.715016s 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd078b09640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 msgr2=0x7fd074068c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd078b09640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 0x7fd074068c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd07409c560 0x7fd0741436a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd07409c560 0x7fd0741436a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:53424/0 (socket says 172.21.15.86:53424) 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 1 -- 172.21.15.86:0/2277654827 learned_addr learned my addr 172.21.15.86:0/2277654827 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): get_auth_request con 0x7fd07409c560 auth_method 0 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.746+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:09.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd073fff640 1 -- 172.21.15.86:0/2277654827 >> v1:172.21.15.203:6789/0 conn(0x7fd07409eea0 legacy=0x7fd07409be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:09.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 10 monclient(hunting): handle_auth_done global_id 134143 payload 274 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 1 -- 172.21.15.86:0/2277654827 >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 msgr2=0x7fd074068c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 1 --2- 172.21.15.86:0/2277654827 >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 0x7fd074068c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 1 -- 172.21.15.86:0/2277654827 >> v1:172.21.15.203:6789/0 conn(0x7fd07409eea0 legacy=0x7fd07409be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 10 monclient: _finish_hunting 0 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 1 monclient: found mon.noname-c 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:39:09.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.750+0000 7fd07930a640 1 -- 172.21.15.86:0/2277654827 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07409dc20 con 0x7fd07409c560 2024-09-17T15:39:09.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd07930a640 1 -- 172.21.15.86:0/2277654827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409c560 msgr2=0x7fd0741436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:09.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd07930a640 1 --2- 172.21.15.86:0/2277654827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409c560 0x7fd0741436a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd074074d60 tx=0x7fd06c01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f75f4245c076ca1d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:09.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 1 -- 172.21.15.86:0/2277654827 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0a5830 con 0x7fd07409c560 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: dump: 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:09.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: _finish_auth 0 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: _check_auth_tickets 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 1 -- 172.21.15.86:0/2277654827 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0a59d0 con 0x7fd07409c560 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 1 -- 172.21.15.86:0/2277654827 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0a5da0 con 0x7fd07409c560 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd0737fe640 10 monclient: dump: 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd07ad6d640 5 monclient: authenticate success, global_id 134143 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd07ad6d640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:09.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.754+0000 7fd07ad6d640 4 set_mon_vals no callback set 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: shutdown 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- 172.21.15.86:0/2277654827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409c560 msgr2=0x7fd0741436a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/2277654827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409c560 0x7fd0741436a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd074074d60 tx=0x7fd06c01c950 comp rx=0 tx=0).stop 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- 172.21.15.86:0/2277654827 shutdown_connections 2024-09-17T15:39:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/2277654827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409c560 0x7fd0741436a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/2277654827 >> v2:172.21.15.86:3300/0 conn(0x7fd07409ba70 0x7fd074068c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- 172.21.15.86:0/2277654827 >> 172.21.15.86:0/2277654827 conn(0x7fd0740a6440 msgr2=0x7fd0740a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- 172.21.15.86:0/2277654827 shutdown_connections 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- 172.21.15.86:0/2277654827 wait complete. 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: build_initial_monmap 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: monmap: 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 Processor -- start 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- start start 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: init 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409ba70 0x7fd074125570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-b con 0x7fd07409ba70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 0x7fd07413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-a con 0x7fd07409c560 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): picked mon.noname-c con 0x7fd07409eea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd074146850 con 0x7fd07409c560 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741466d0 con 0x7fd07409ba70 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd0741469d0 con 0x7fd07409eea0 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07ad6d640 10 monclient(hunting): authenticate will time out at 1682.725579s 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd073fff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 msgr2=0x7fd07413c3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd073fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 0x7fd07413c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38884/0 (socket says 172.21.15.86:38884) 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 -- 172.21.15.86:0/493731311 learned_addr learned my addr 172.21.15.86:0/493731311 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:09.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): get_auth_request con 0x7fd07409eea0 auth_method 0 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd078b09640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409ba70 0x7fd074125570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd078b09640 10 monclient(hunting): get_auth_request con 0x7fd07409ba70 auth_method 0 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd078b09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd078b09640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd078b09640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient(hunting): handle_auth_done global_id 104453 payload 824 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 -- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409ba70 msgr2=0x7fd074125570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:09.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409ba70 0x7fd074125570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 -- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 msgr2=0x7fd07413c3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 0x7fd07413c3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient: _finish_hunting 0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 monclient: found mon.noname-c 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.758+0000 7fd07930a640 1 -- 172.21.15.86:0/493731311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd074126d70 con 0x7fd07409eea0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07930a640 1 -- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 msgr2=0x7fd0741416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07930a640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0af040 tx=0x7fd06c0196f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0bc020 con 0x7fd07409eea0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:09.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: dump: 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: _finish_auth 0 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: _check_auth_tickets 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0b8a90 con 0x7fd07409eea0 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 5 monclient: authenticate success, global_id 104453 2024-09-17T15:39:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _renew_subs 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07413d080 con 0x7fd07409eea0 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0b8e60 con 0x7fd07409eea0 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 10 monclient: dump: 2024-09-17T15:39:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _renew_subs 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0741cbaa0 con 0x7fd07409eea0 2024-09-17T15:39:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0727fc640 4 set_mon_vals no callback set 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd074068c30 con 0x7fd07409eea0 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd06c0bf020 con 0x7fd07409eea0 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 0x7fd06003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(399..399 src has 213..399) ==== 4831+0+0 (secure 0 0 0) 0x7fd06c0eb330 con 0x7fd07409eea0 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd078b09640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 0x7fd06003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.762+0000 7fd078b09640 10 monclient: get_auth_request con 0x7fd06003bb10 auth_method 0 2024-09-17T15:39:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.766+0000 7fd078b09640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 0x7fd06003dfd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.766+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd06c0c6030 con 0x7fd07409eea0 2024-09-17T15:39:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.766+0000 7fd0717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:09.766+0000 7fd0717fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:10.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:10.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:10.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:10.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd07406d4f0 con 0x7fd07409eea0 2024-09-17T15:39:10.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd0717fa640 1 -- 172.21.15.86:0/493731311 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd06c016660 con 0x7fd07409eea0 2024-09-17T15:39:10.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd0717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:10.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd0717fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:10.006 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:10.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 msgr2=0x7fd06003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 0x7fd06003dfd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064008040 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 10 monclient: shutdown 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 msgr2=0x7fd0741416c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0af040 tx=0x7fd06c0196f0 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 shutdown_connections 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd06003bb10 0x7fd06003dfd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd07409eea0 0x7fd0741416c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd07409c560 0x7fd07413c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 --2- 172.21.15.86:0/493731311 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd07409ba70 0x7fd074125570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 >> 172.21.15.86:0/493731311 conn(0x7fd0740a6440 msgr2=0x7fd0740a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:10.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 shutdown_connections 2024-09-17T15:39:10.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:10.006+0000 7fd07ad6d640 1 -- 172.21.15.86:0/493731311 wait complete. 2024-09-17T15:39:13.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: get_monmap_and_config 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: build_initial_monmap 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: monmap: 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:13.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:13.153855+0000 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:13.153855+0000 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:13.151 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 Processor -- start 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 -- start start 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: init 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 0x7f2fa006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa00689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fa0068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:13.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa0074470 con 0x7f2fa0068db0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa009c900 con 0x7f2fa00689d0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9effd640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f2fa0068db0 legacy=0x7f2fa0069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa009cb90 con 0x7f2fa013aff0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2fa5f7a640 10 monclient(hunting): authenticate will time out at 1686.119666s 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9f7fe640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 msgr2=0x7f2fa006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9f7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 0x7f2fa006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38888/0 (socket says 172.21.15.86:38888) 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 1 -- 172.21.15.86:0/56024854 learned_addr learned my addr 172.21.15.86:0/56024854 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:13.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): get_auth_request con 0x7f2fa013aff0 auth_method 0 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.150+0000 7f2f9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 10 monclient(hunting): handle_auth_done global_id 104456 payload 274 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 1 -- 172.21.15.86:0/56024854 >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 msgr2=0x7f2fa006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 1 --2- 172.21.15.86:0/56024854 >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 0x7f2fa006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 1 -- 172.21.15.86:0/56024854 >> v1:172.21.15.86:6789/0 conn(0x7f2fa0068db0 legacy=0x7f2fa0069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:39:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 1 monclient: found mon.noname-b 2024-09-17T15:39:13.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:13.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.154+0000 7f2f9ffff640 1 -- 172.21.15.86:0/56024854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa009ce70 con 0x7f2fa013aff0 2024-09-17T15:39:13.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9ffff640 1 -- 172.21.15.86:0/56024854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 msgr2=0x7f2fa013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9ffff640 1 --2- 172.21.15.86:0/56024854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f9801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6714f2d98455315a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 1 -- 172.21.15.86:0/56024854 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa009c2d0 con 0x7f2fa013aff0 2024-09-17T15:39:13.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: dump: 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:13.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 1 -- 172.21.15.86:0/56024854 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f980a5770 con 0x7f2fa013aff0 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 5 monclient: authenticate success, global_id 104456 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 1 -- 172.21.15.86:0/56024854 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f980a5b40 con 0x7f2fa013aff0 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 4 set_mon_vals no callback set 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:13.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9e7fc640 10 monclient: dump: 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: shutdown 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:13.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/56024854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 msgr2=0x7f2fa013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/56024854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009e50 tx=0x7f2f9801c910 comp rx=0 tx=0).stop 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/56024854 shutdown_connections 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/56024854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa013aff0 0x7f2fa013d3e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/56024854 >> v2:172.21.15.86:3300/0 conn(0x7f2fa00689d0 0x7f2fa006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/56024854 >> 172.21.15.86:0/56024854 conn(0x7f2fa00a6440 msgr2=0x7f2fa00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/56024854 shutdown_connections 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/56024854 wait complete. 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: build_initial_monmap 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: monmap: 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:13.163 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 Processor -- start 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- start start 2024-09-17T15:39:13.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: init 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa0068db0 0x7f2fa0132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa0068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 0x7f2fa0070d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa013aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa0140670 con 0x7f2fa013aff0 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa01404f0 con 0x7f2fa0068db0 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa01407f0 con 0x7f2fa00689d0 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2fa5f7a640 10 monclient(hunting): authenticate will time out at 1686.127387s 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9ffff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 msgr2=0x7f2fa0070d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 0x7f2fa0070d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38890/0 (socket says 172.21.15.86:38890) 2024-09-17T15:39:13.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 1 -- 172.21.15.86:0/647566135 learned_addr learned my addr 172.21.15.86:0/647566135 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): get_auth_request con 0x7f2fa00689d0 auth_method 0 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9effd640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa0068db0 0x7f2fa0132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9effd640 10 monclient(hunting): get_auth_request con 0x7f2fa0068db0 auth_method 0 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:13.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.158+0000 7f2f9f7fe640 10 monclient(hunting): handle_auth_done global_id 104459 payload 824 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 -- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa0068db0 msgr2=0x7f2fa0132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa0068db0 0x7f2fa0132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 -- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 msgr2=0x7f2fa0070d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 0x7f2fa0070d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 -- 172.21.15.86:0/647566135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa011d540 con 0x7f2fa00689d0 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 -- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 msgr2=0x7f2fa01cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9f7fe640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000b550 tx=0x7f2f9000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f900050f0 con 0x7f2fa00689d0 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: dump: 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:13.173 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: _finish_auth 0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: _check_auth_tickets 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f90015070 con 0x7f2fa00689d0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 5 monclient: authenticate success, global_id 104459 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _renew_subs 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa0072830 con 0x7f2fa00689d0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f900054d0 con 0x7f2fa00689d0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 10 monclient: dump: 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _renew_subs 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f9d7fa640 4 set_mon_vals no callback set 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa011cf80 con 0x7f2fa00689d0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa006f840 con 0x7f2fa00689d0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2f900056b0 con 0x7f2fa00689d0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.162+0000 7f2f87fff640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 0x7f2f8c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.166+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(399..399 src has 213..399) ==== 4831+0+0 (secure 0 0 0) 0x7f2f9004fdf0 con 0x7f2fa00689d0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.166+0000 7f2f9effd640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 0x7f2f8c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.166+0000 7f2f9effd640 10 monclient: get_auth_request con 0x7f2f8c03bb90 auth_method 0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.166+0000 7f2f9effd640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 0x7f2f8c03e050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f94006fd0 tx=0x7f2f94008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.170+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2f9000ee20 con 0x7f2fa00689d0 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.170+0000 7f2f87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:13.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.170+0000 7f2f87fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:13.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:13.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:13.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:13.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2fa011d1c0 con 0x7f2fa00689d0 2024-09-17T15:39:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2f87fff640 1 -- 172.21.15.86:0/647566135 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f2f90010dd0 con 0x7f2fa00689d0 2024-09-17T15:39:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2f87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2f87fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:13.394 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 msgr2=0x7f2f8c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 0x7f2f8c03e050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2f94006fd0 tx=0x7f2f94008040 comp rx=0 tx=0).stop 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 10 monclient: shutdown 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 msgr2=0x7f2fa01cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000b550 tx=0x7f2f9000ba10 comp rx=0 tx=0).stop 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 shutdown_connections 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f8c03bb90 0x7f2f8c03e050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa013aff0 0x7f2fa0070d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa0068db0 0x7f2fa0132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 --2- 172.21.15.86:0/647566135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa00689d0 0x7f2fa01cdbb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:13.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 >> 172.21.15.86:0/647566135 conn(0x7f2fa00a6440 msgr2=0x7f2fa0135c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:13.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 shutdown_connections 2024-09-17T15:39:13.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:13.394+0000 7f2fa5f7a640 1 -- 172.21.15.86:0/647566135 wait complete. 2024-09-17T15:39:16.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: get_monmap_and_config 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: build_initial_monmap 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: monmap: 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:16.545664+0000 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:16.545664+0000 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:16.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:16.543 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:16.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 Processor -- start 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 -- start start 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: init 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-a con 0x7f84980689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8498068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-c con 0x7f849813aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f849809c8b0 con 0x7f8498068db0 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8498074500 con 0x7f84980689d0 2024-09-17T15:39:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f849809cb90 con 0x7f849813aff0 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f84a05fe640 10 monclient(hunting): authenticate will time out at 1689.511320s 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849e39a640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 msgr2=0x7f849806f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849e39a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:53462/0 (socket says 172.21.15.86:53462) 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 learned_addr learned my addr 172.21.15.86:0/606033284 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 10 monclient(hunting): get_auth_request con 0x7f849813aff0 auth_method 0 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:39:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.542+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.546+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:16.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.546+0000 7f849db99640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.203:6789/0 conn(0x7f8498068db0 legacy=0x7f8498069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.746+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 msgr2=0x7f849806f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:16.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.746+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 10 monclient(hunting): get_auth_request con 0x7f849813aff0 auth_method 0 2024-09-17T15:39:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:16.750+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.150+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 msgr2=0x7f849806f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:17.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.150+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 10 monclient(hunting): get_auth_request con 0x7f849813aff0 auth_method 0 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:17.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:17.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.154+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:39:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.954+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 msgr2=0x7f849806f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:17.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.954+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 10 monclient(hunting): get_auth_request con 0x7f849813aff0 auth_method 0 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:17.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:17.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:17.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:17.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:17.958+0000 7f849eb9b640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:39:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): tick 2024-09-17T15:39:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:39:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 msgr2=0x7f849813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.203:3300/0 conn(0x7f849813aff0 0x7f849813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 -- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 msgr2=0x7f849806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 --2- 172.21.15.86:0/606033284 >> v2:172.21.15.86:3300/0 conn(0x7f84980689d0 0x7f849806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.203:6789/0 conn(0x7f8498068db0 legacy=0x7f8498069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient: _add_conns ranks=[2,0] 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f847c003720 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f847c0062a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 -- 172.21.15.86:0/606033284 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f849809ce70 con 0x7f847c0062a0 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 1 -- 172.21.15.86:0/606033284 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f849809cb90 con 0x7f847c003720 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f848ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849db99640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.86:6789/0 conn(0x7f847c0062a0 legacy=0x7f847c008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 10 monclient(hunting): get_auth_request con 0x7f847c003720 auth_method 0 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:19.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.546+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:19.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849db99640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.86:6789/0 conn(0x7f847c0062a0 legacy=0x7f847c008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 10 monclient(hunting): get_auth_request con 0x7f847c003720 auth_method 0 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:19.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:19.750+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849db99640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.86:6789/0 conn(0x7f847c0062a0 legacy=0x7f847c008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:20.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 10 monclient(hunting): get_auth_request con 0x7f847c003720 auth_method 0 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:20.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.154+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:39:20.544 INFO:tasks.rados.rados.0.smithi203.stdout:1757: done (2 left) 2024-09-17T15:39:20.544 INFO:tasks.rados.rados.0.smithi203.stdout:1758: done (1 left) 2024-09-17T15:39:20.544 INFO:tasks.rados.rados.0.smithi203.stdout:1759: done (0 left) 2024-09-17T15:39:20.545 INFO:tasks.rados.rados.0.smithi203.stdout:1760: snap_create 2024-09-17T15:39:20.651 INFO:tasks.rados.rados.0.smithi203.stdout:1760: done (0 left) 2024-09-17T15:39:20.651 INFO:tasks.rados.rados.0.smithi203.stdout:1761: write oid 25 current snap is 198 2024-09-17T15:39:20.651 INFO:tasks.rados.rados.0.smithi203.stdout:1761: seq_num 635 ranges {672445=640414,1842301=681553} 2024-09-17T15:39:20.671 INFO:tasks.rados.rados.0.smithi203.stdout:1761: writing smithi20328086-25 from 672445 to 1312859 tid 1 2024-09-17T15:39:20.690 INFO:tasks.rados.rados.0.smithi203.stdout:1761: writing smithi20328086-25 from 1842301 to 2523854 tid 2 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1762: read oid 18 snap 196 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1762: expect (ObjNum 603 snap 180 seq_num 603) 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1763: read oid 40 snap -1 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1763: expect (ObjNum 587 snap 174 seq_num 587) 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1764: rollback oid 14 current snap is 198 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 197 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1761: finishing write tid 1 to smithi20328086-25 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1765: read oid 21 snap -1 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1765: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:20.691 INFO:tasks.rados.rados.0.smithi203.stdout:1766: snap_create 2024-09-17T15:39:20.701 INFO:tasks.rados.rados.0.smithi203.stdout:1764: finishing rollback tid 0 to smithi20328086-14 2024-09-17T15:39:20.701 INFO:tasks.rados.rados.0.smithi203.stdout:1764: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:39:20.701 INFO:tasks.rados.rados.0.smithi203.stdout:1764: finishing rollback tid 2 to smithi20328086-14 2024-09-17T15:39:20.701 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 701 (ObjNum 611 snap 184 seq_num 611) dirty exists 2024-09-17T15:39:20.702 INFO:tasks.rados.rados.0.smithi203.stdout:1761: finishing write tid 2 to smithi20328086-25 2024-09-17T15:39:20.703 INFO:tasks.rados.rados.0.smithi203.stdout:1761: finishing write tid 3 to smithi20328086-25 2024-09-17T15:39:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1761: finishing write tid 4 to smithi20328086-25 2024-09-17T15:39:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1761: oid 25 updating version 0 to 600 2024-09-17T15:39:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1761: oid 25 version 600 is already newer than 599 2024-09-17T15:39:20.704 INFO:tasks.rados.rados.0.smithi203.stdout:1761: oid 25 version 600 is already newer than 598 2024-09-17T15:39:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 600 (ObjNum 635 snap 198 seq_num 635) dirty exists 2024-09-17T15:39:20.705 INFO:tasks.rados.rados.0.smithi203.stdout:1761: left oid 25 (ObjNum 635 snap 198 seq_num 635) 2024-09-17T15:39:20.707 INFO:tasks.rados.rados.0.smithi203.stdout:1762: expect (ObjNum 603 snap 180 seq_num 603) 2024-09-17T15:39:20.760 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1526776 2024-09-17T15:39:20.868 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1635308 2024-09-17T15:39:20.878 INFO:tasks.rados.rados.0.smithi203.stdout:1765: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:20.946 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2872165 2024-09-17T15:39:20.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:20.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849db99640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.86:6789/0 conn(0x7f847c0062a0 legacy=0x7f847c008760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): get_auth_request con 0x7f847c003720 auth_method 0 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient(hunting): handle_auth_done global_id 104462 payload 274 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> v1:172.21.15.86:6789/0 conn(0x7f847c0062a0 legacy=0x7f847c008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient: _finish_hunting 0 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 1 monclient: found mon.noname-b 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:20.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:20.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.958+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c00b110 con 0x7f847c003720 2024-09-17T15:39:20.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849e39a640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:20.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849e39a640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8488009d30 tx=0x7f848801d1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c3a06843a64d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:20.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 1 -- 172.21.15.86:0/606033284 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84880b1070 con 0x7f847c003720 2024-09-17T15:39:20.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:20.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: dump: 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: _finish_auth 0 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: _check_auth_tickets 2024-09-17T15:39:20.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 1 -- 172.21.15.86:0/606033284 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84880ac600 con 0x7f847c003720 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 1 -- 172.21.15.86:0/606033284 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849809c2d0 con 0x7f847c003720 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849d398640 10 monclient: dump: 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:20.968 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:20.969 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:20.969 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:20.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 5 monclient: authenticate success, global_id 104462 2024-09-17T15:39:20.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:20.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 4 set_mon_vals no callback set 2024-09-17T15:39:20.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 10 monclient: shutdown 2024-09-17T15:39:20.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:20.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 -- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 msgr2=0x7f847c005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 --2- 172.21.15.86:0/606033284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f847c003720 0x7f847c005b70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8488009d30 tx=0x7f848801d1e0 comp rx=0 tx=0).stop 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f849eb9b640 1 -- 172.21.15.86:0/606033284 reap_dead start 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 -- 172.21.15.86:0/606033284 shutdown_connections 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 -- 172.21.15.86:0/606033284 >> 172.21.15.86:0/606033284 conn(0x7f84980a6440 msgr2=0x7f84980a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 -- 172.21.15.86:0/606033284 shutdown_connections 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.966+0000 7f84a05fe640 1 -- 172.21.15.86:0/606033284 wait complete. 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient: build_initial_monmap 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient: monmap: 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:20.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 Processor -- start 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 -- start start 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient: init 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 0x7f8498132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-a con 0x7f84981cdad0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:20.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84981330f0 0x7f8498070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-b con 0x7f84981330f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): picked mon.noname-c con 0x7f84980711f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8498140580 con 0x7f84981cdad0 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8498140400 con 0x7f84981330f0 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8498140700 con 0x7f84980711f0 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f84a05fe640 10 monclient(hunting): authenticate will time out at 1693.938138s 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849e39a640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 msgr2=0x7f8498132bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849e39a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 0x7f8498132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:20.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38668/0 (socket says 172.21.15.86:38668) 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 -- 172.21.15.86:0/4283939725 learned_addr learned my addr 172.21.15.86:0/4283939725 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): get_auth_request con 0x7f84980711f0 auth_method 0 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84981330f0 0x7f8498070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): get_auth_request con 0x7f84981330f0 auth_method 0 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849db99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient(hunting): handle_auth_done global_id 104465 payload 824 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 -- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84981330f0 msgr2=0x7f8498070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84981330f0 0x7f8498070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 -- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 msgr2=0x7f8498132bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 0x7f8498132bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 monclient: found mon.noname-c 2024-09-17T15:39:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:20.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:20.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.970+0000 7f849eb9b640 1 -- 172.21.15.86:0/4283939725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849811d540 con 0x7f84980711f0 2024-09-17T15:39:20.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f849eb9b640 1 -- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 msgr2=0x7f8498071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:20.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f849eb9b640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fa0 tx=0x7f8490007d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849001b070 con 0x7f84980711f0 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: dump: 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:20.980 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: _finish_auth 0 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8490013c20 con 0x7f84980711f0 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 5 monclient: authenticate success, global_id 104465 2024-09-17T15:39:20.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _renew_subs 2024-09-17T15:39:20.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:20.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8498074c00 con 0x7f84980711f0 2024-09-17T15:39:20.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:20.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8490017430 con 0x7f84980711f0 2024-09-17T15:39:20.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:20.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:20.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 10 monclient: dump: 2024-09-17T15:39:20.984 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:20.984 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:20.985 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.985 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:20.985 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:20.985 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:20.985 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848ffff640 4 set_mon_vals no callback set 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _renew_subs 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:20.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f849811cca0 con 0x7f84980711f0 2024-09-17T15:39:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849809d460 con 0x7f84980711f0 2024-09-17T15:39:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f849001e070 con 0x7f84980711f0 2024-09-17T15:39:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 0x7f847c03eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(401..401 src has 213..401) ==== 4803+0+0 (secure 0 0 0) 0x7f8490050d60 con 0x7f84980711f0 2024-09-17T15:39:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f849e39a640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 0x7f847c03eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.974+0000 7f849e39a640 10 monclient: get_auth_request con 0x7f847c03ca10 auth_method 0 2024-09-17T15:39:20.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.978+0000 7f849e39a640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 0x7f847c03eed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f847c006270 tx=0x7f84880a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:20.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.982+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8490056020 con 0x7f84980711f0 2024-09-17T15:39:20.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.982+0000 7f848effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:20.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:20.982+0000 7f848effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:21.147 INFO:tasks.rados.rados.0.smithi203.stdout:1763: expect (ObjNum 587 snap 174 seq_num 587) 2024-09-17T15:39:21.147 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 93977 2024-09-17T15:39:21.188 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f84a05fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f84a05fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f84a05fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f849806d4f0 con 0x7f84980711f0 2024-09-17T15:39:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f848effd640 1 -- 172.21.15.86:0/4283939725 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f8490025020 con 0x7f84980711f0 2024-09-17T15:39:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f848effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.214+0000 7f848effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:21.216 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 msgr2=0x7f847c03eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 0x7f847c03eed0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f847c006270 tx=0x7f84880a7000 comp rx=0 tx=0).stop 2024-09-17T15:39:21.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 10 monclient: shutdown 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 msgr2=0x7f8498071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f847c009fa0 tx=0x7f8490007d80 comp rx=0 tx=0).stop 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 shutdown_connections 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f847c03ca10 0x7f847c03eed0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84980711f0 0x7f8498071650 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f84981330f0 0x7f8498070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 --2- 172.21.15.86:0/4283939725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84981cdad0 0x7f8498132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 >> 172.21.15.86:0/4283939725 conn(0x7f84980a6440 msgr2=0x7f849813cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 shutdown_connections 2024-09-17T15:39:21.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.218+0000 7f84a05fe640 1 -- 172.21.15.86:0/4283939725 wait complete. 2024-09-17T15:39:21.229 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:21.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:39:21.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:39:21.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:39:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: get_monmap_and_config 2024-09-17T15:39:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: build_initial_monmap 2024-09-17T15:39:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: monmap: 2024-09-17T15:39:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:21.361069+0000 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:21.361069+0000 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:21.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 Processor -- start 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 -- start start 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: init 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:39:21.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4440a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 0x7fa4440a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-a con 0x7fa4440a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-b con 0x7fa44413d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4440a68d0 con 0x7fa4440a4f70 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa4440a64a0 con 0x7fa4440a5350 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa44409c230 con 0x7fa44413d180 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44c282640 10 monclient(hunting): authenticate will time out at 1694.326698s 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a01e640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fa4440a4f70 legacy=0x7fa44409cef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44981d640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 msgr2=0x7fa4440a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44981d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 0x7fa4440a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38678/0 (socket says 172.21.15.86:38678) 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 -- 172.21.15.86:0/1115512580 learned_addr learned my addr 172.21.15.86:0/1115512580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): get_auth_request con 0x7fa44413d180 auth_method 0 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient(hunting): handle_auth_done global_id 104468 payload 274 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 -- 172.21.15.86:0/1115512580 >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 msgr2=0x7fa4440a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 --2- 172.21.15.86:0/1115512580 >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 0x7fa4440a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 -- 172.21.15.86:0/1115512580 >> v1:172.21.15.86:6789/0 conn(0x7fa4440a4f70 legacy=0x7fa44409cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient: _finish_hunting 0 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 monclient: found mon.noname-b 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.358+0000 7fa44a81f640 1 -- 172.21.15.86:0/1115512580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44409c510 con 0x7fa44413d180 2024-09-17T15:39:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44a81f640 1 -- 172.21.15.86:0/1115512580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 msgr2=0x7fa44413f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44a81f640 1 --2- 172.21.15.86:0/1115512580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009e50 tx=0x7fa43c01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f57a0838678645f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 1 -- 172.21.15.86:0/1115512580 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0a5770 con 0x7fa44413d180 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: dump: 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: _finish_auth 0 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: _check_auth_tickets 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 1 -- 172.21.15.86:0/1115512580 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43c0a5910 con 0x7fa44413d180 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 1 -- 172.21.15.86:0/1115512580 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0a5ce0 con 0x7fa44413d180 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44901c640 10 monclient: dump: 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44c282640 5 monclient: authenticate success, global_id 104468 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44c282640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.362+0000 7fa44c282640 4 set_mon_vals no callback set 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: shutdown 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- 172.21.15.86:0/1115512580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 msgr2=0x7fa44413f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- 172.21.15.86:0/1115512580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009e50 tx=0x7fa43c01c910 comp rx=0 tx=0).stop 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- 172.21.15.86:0/1115512580 shutdown_connections 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- 172.21.15.86:0/1115512580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44413d180 0x7fa44413f570 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- 172.21.15.86:0/1115512580 >> v2:172.21.15.86:3300/0 conn(0x7fa4440a5350 0x7fa4440a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- 172.21.15.86:0/1115512580 >> 172.21.15.86:0/1115512580 conn(0x7fa4440a7240 msgr2=0x7fa4440a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- 172.21.15.86:0/1115512580 shutdown_connections 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- 172.21.15.86:0/1115512580 wait complete. 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: build_initial_monmap 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: monmap: 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:21.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:21.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 Processor -- start 2024-09-17T15:39:21.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- start start 2024-09-17T15:39:21.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: init 2024-09-17T15:39:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:39:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 0x7fa4441cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-c con 0x7fa4440a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4440a5350 0x7fa4441d03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-b con 0x7fa4440a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44413d180 0x7fa4441396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): picked mon.noname-a con 0x7fa44413d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa4441426f0 con 0x7fa44413d180 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa444142570 con 0x7fa4440a5350 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa444142870 con 0x7fa4440a4f70 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44c282640 10 monclient(hunting): authenticate will time out at 1694.333901s 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a81f640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44413d180 msgr2=0x7fa4441396b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a81f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44413d180 0x7fa4441396b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 0x7fa4441cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 0x7fa4441cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38682/0 (socket says 172.21.15.86:38682) 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 -- 172.21.15.86:0/3581804002 learned_addr learned my addr 172.21.15.86:0/3581804002 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): get_auth_request con 0x7fa4440a4f70 auth_method 0 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient(hunting): handle_auth_done global_id 104471 payload 824 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44981d640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4440a5350 0x7fa4441d03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4440a5350 msgr2=0x7fa4441d03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4440a5350 0x7fa4441d03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44413d180 msgr2=0x7fa4441396b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa44413d180 0x7fa4441396b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient: _finish_hunting 0 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 monclient: found mon.noname-c 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.366+0000 7fa44a01e640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44413af40 con 0x7fa4440a4f70 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44a01e640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 msgr2=0x7fa4441cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44a01e640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 0x7fa4441cfea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa44000b560 tx=0x7fa44000ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440013020 con 0x7fa4440a4f70 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: dump: 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: _finish_auth 0 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440005070 con 0x7fa4440a4f70 2024-09-17T15:39:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440005460 con 0x7fa4440a4f70 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 10 monclient: dump: 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 5 monclient: authenticate success, global_id 104471 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 monclient: _renew_subs 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44413d5f0 con 0x7fa4440a4f70 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44881b640 4 set_mon_vals no callback set 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 monclient: _renew_subs 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44409f070 con 0x7fa4440a4f70 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 0x7fa444139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa4440a5750 con 0x7fa44409f240 2024-09-17T15:39:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa440016020 con 0x7fa4440a4f70 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 0x7fa444139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient: get_auth_request con 0x7fa44409f240 auth_method 0 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa42803bb60 0x7fa42803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(401..401 src has 213..401) ==== 4803+0+0 (secure 0 0 0) 0x7fa44004fe40 con 0x7fa4440a4f70 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 10 monclient(hunting): handle_auth_done global_id 104474 payload 824 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44981d640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4440a5ff0 con 0x7fa44409f240 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44a81f640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa42803bb60 0x7fa42803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.370+0000 7fa44a81f640 10 monclient: get_auth_request con 0x7fa42803bb60 auth_method 0 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa44981d640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 msgr2=0x7fa444139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa44981d640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 0x7fa444139bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa43400c450 tx=0x7fa43400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa434005070 con 0x7fa44409f240 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 10 monclient: dump: 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.384 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa434016070 con 0x7fa44409f240 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 msgr2=0x7fa444139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 0x7fa444139bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa43400c450 tx=0x7fa43400c910 comp rx=0 tx=0).stop 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 msgr2=0x7fa444139bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa43affd640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa44409f240 0x7fa444139bf0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa43400c450 tx=0x7fa43400c910 comp rx=0 tx=0).stop 2024-09-17T15:39:21.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.374+0000 7fa44a81f640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa42803bb60 0x7fa42803e020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa44406e190 tx=0x7fa43c0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44c282640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:21.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44c282640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 0x7fa44413a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44c282640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa44413baa0 con 0x7fa4441d0920 2024-09-17T15:39:21.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 0x7fa44413a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient: get_auth_request con 0x7fa4441d0920 auth_method 0 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 10 monclient(hunting): handle_auth_done global_id 104477 payload 824 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.378+0000 7fa44981d640 1 -- 172.21.15.86:0/3581804002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa444069920 con 0x7fa4441d0920 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44981d640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 msgr2=0x7fa44413a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44981d640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 0x7fa44413a130 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa434015000 tx=0x7fa4340059b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4340056c0 con 0x7fa4441d0920 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 10 monclient: dump: 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fa4340056c0 con 0x7fa4441d0920 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 msgr2=0x7fa44413a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 0x7fa44413a130 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa434015000 tx=0x7fa4340059b0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 msgr2=0x7fa44413a130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:21.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa43affd640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4441d0920 0x7fa44413a130 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa434015000 tx=0x7fa4340059b0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.391 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 118, 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 192256, 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:39:21.392 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.393 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.394 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:21.395 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.396 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:21.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.400 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.403 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.404 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:21.405 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:21.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa42803bb60 msgr2=0x7fa42803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa42803bb60 0x7fa42803e020 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa44406e190 tx=0x7fa43c0a7070 comp rx=0 tx=0).stop 2024-09-17T15:39:21.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 10 monclient: shutdown 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 msgr2=0x7fa4441cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 --2- 172.21.15.86:0/3581804002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4440a4f70 0x7fa4441cfea0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa44000b560 tx=0x7fa44000ba20 comp rx=0 tx=0).stop 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44a81f640 1 -- 172.21.15.86:0/3581804002 reap_dead start 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44a81f640 1 -- 172.21.15.86:0/3581804002 reap_dead start 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 shutdown_connections 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 >> 172.21.15.86:0/3581804002 conn(0x7fa4440a7240 msgr2=0x7fa44413c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 shutdown_connections 2024-09-17T15:39:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.382+0000 7fa44c282640 1 -- 172.21.15.86:0/3581804002 wait complete. 2024-09-17T15:39:21.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:39:21.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:39:21.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:39:21.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: get_monmap_and_config 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: build_initial_monmap 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: monmap: 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:21.524317+0000 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:21.524317+0000 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:21.521 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:21.522 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:21.522 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 Processor -- start 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 -- start start 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: init 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc18c0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:21.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc18c0a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 0x7fc18c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc18c13d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc18c0a6280 con 0x7fc18c0a50e0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc18c09d2a0 con 0x7fc18c13d2f0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc18c09c0a0 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc193e5c640 10 monclient(hunting): authenticate will time out at 1694.489797s 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc191bf8640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fc18c0a50e0 legacy=0x7fc18c0a54c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1923f9640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 msgr2=0x7fc18c13f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1923f9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 0x7fc18c13f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38722/0 (socket says 172.21.15.86:38722) 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 -- 172.21.15.86:0/4196174704 learned_addr learned my addr 172.21.15.86:0/4196174704 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): get_auth_request con 0x7fc18c0a5bd0 auth_method 0 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient(hunting): handle_auth_done global_id 104480 payload 274 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 -- 172.21.15.86:0/4196174704 >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 msgr2=0x7fc18c13f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:21.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 --2- 172.21.15.86:0/4196174704 >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 0x7fc18c13f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 -- 172.21.15.86:0/4196174704 >> v1:172.21.15.86:6789/0 conn(0x7fc18c0a50e0 legacy=0x7fc18c0a54c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient: _finish_hunting 0 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 monclient: found mon.noname-b 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.522+0000 7fc1913f7640 1 -- 172.21.15.86:0/4196174704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c09c380 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc1913f7640 1 -- 172.21.15.86:0/4196174704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 msgr2=0x7fc18c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc1913f7640 1 --2- 172.21.15.86:0/4196174704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abffadaf8dcbc0d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 1 -- 172.21.15.86:0/4196174704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800b0070 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: dump: 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: _finish_auth 0 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: _check_auth_tickets 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 1 -- 172.21.15.86:0/4196174704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1800ab600 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 5 monclient: authenticate success, global_id 104480 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 1 -- 172.21.15.86:0/4196174704 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800ab9d0 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 4 set_mon_vals no callback set 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc190bf6640 10 monclient: dump: 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 10 monclient: shutdown 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 -- 172.21.15.86:0/4196174704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 msgr2=0x7fc18c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 --2- 172.21.15.86:0/4196174704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc180009870 tx=0x7fc18001c410 comp rx=0 tx=0).stop 2024-09-17T15:39:21.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 -- 172.21.15.86:0/4196174704 shutdown_connections 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 --2- 172.21.15.86:0/4196174704 >> v2:172.21.15.86:3300/0 conn(0x7fc18c13d2f0 0x7fc18c13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 --2- 172.21.15.86:0/4196174704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a5bd0 0x7fc18c09cd60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 -- 172.21.15.86:0/4196174704 >> 172.21.15.86:0/4196174704 conn(0x7fc18c0aaae0 msgr2=0x7fc18c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 -- 172.21.15.86:0/4196174704 shutdown_connections 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.526+0000 7fc193e5c640 1 -- 172.21.15.86:0/4196174704 wait complete. 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: build_initial_monmap 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: monmap: 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:21.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 Processor -- start 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 -- start start 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: init 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:21.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-c con 0x7fc18c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc18c0a5bd0 0x7fc18c1d0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-a con 0x7fc18c0a5bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c13d2f0 0x7fc18c1d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): picked mon.noname-b con 0x7fc18c13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c142650 con 0x7fc18c0a5bd0 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c0a6b00 con 0x7fc18c13d2f0 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc18c1427d0 con 0x7fc18c0a50e0 2024-09-17T15:39:21.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient(hunting): authenticate will time out at 1694.497004s 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc1913f7640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc18c0a5bd0 msgr2=0x7fc18c1d0090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc1913f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc18c0a5bd0 0x7fc18c1d0090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38734/0 (socket says 172.21.15.86:38734) 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 -- 172.21.15.86:0/1865064543 learned_addr learned my addr 172.21.15.86:0/1865064543 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): get_auth_request con 0x7fc18c0a50e0 auth_method 0 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc1923f9640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c13d2f0 0x7fc18c1d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient(hunting): handle_auth_done global_id 104483 payload 824 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c13d2f0 msgr2=0x7fc18c1d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c13d2f0 0x7fc18c1d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc18c0a5bd0 msgr2=0x7fc18c1d0090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:21.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc18c0a5bd0 0x7fc18c1d0090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient: _finish_hunting 0 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 monclient: found mon.noname-c 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c13ae00 con 0x7fc18c0a50e0 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc1923f9640 10 monclient: get_auth_request con 0x7fc18c13d2f0 auth_method 0 2024-09-17T15:39:21.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc1923f9640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c13d2f0 0x7fc18c1d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 msgr2=0x7fc18c1cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc18400b560 tx=0x7fc18400ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184013020 con 0x7fc18c0a50e0 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc191bf8640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc18400b560 tx=0x7fc18400ba20 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fc184005460 mon_map magic: 0 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: dump: 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.543 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc184005070 con 0x7fc18c0a50e0 2024-09-17T15:39:21.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 5 monclient: authenticate success, global_id 104483 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: _renew_subs 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18c139c90 con 0x7fc18c0a50e0 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.530+0000 7fc18bfff640 4 set_mon_vals no callback set 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 10 monclient: _renew_subs 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc18c13d6d0 con 0x7fc18c0a50e0 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 0x7fc18c135250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c09cd60 con 0x7fc18c134e70 2024-09-17T15:39:21.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc184005460 con 0x7fc18c0a50e0 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 10 monclient: dump: 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.546 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc184016020 con 0x7fc18c0a50e0 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc18a7fc640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc17803bb10 0x7fc17803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 0x7fc18c135250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient: get_auth_request con 0x7fc18c134e70 auth_method 0 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1923f9640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc17803bb10 0x7fc17803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1923f9640 10 monclient: get_auth_request con 0x7fc17803bb10 auth_method 0 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 10 monclient(hunting): handle_auth_done global_id 134155 payload 824 2024-09-17T15:39:21.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.534+0000 7fc1913f7640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc18c13abe0 con 0x7fc18c134e70 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc1913f7640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 msgr2=0x7fc18c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc1913f7640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 0x7fc18c135250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1800ad000 tx=0x7fc1800191b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc1923f9640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc17803bb10 0x7fc17803dfd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0a4a00 tx=0x7fc17c007340 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800b9020 con 0x7fc18c134e70 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 10 monclient: dump: 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.548 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc1800b1040 con 0x7fc18c134e70 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 msgr2=0x7fc18c135250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 0x7fc18c135250 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1800ad000 tx=0x7fc1800191b0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 msgr2=0x7fc18c135250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:21.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.538+0000 7fc18a7fc640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c134e70 0x7fc18c135250 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc1800ad000 tx=0x7fc1800191b0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.542+0000 7fc193e5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:39:21.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.542+0000 7fc193e5c640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 0x7fc18c09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:21.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.542+0000 7fc193e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:21.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.542+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c1398e0 con 0x7fc18c135790 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 0x7fc18c09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient: get_auth_request con 0x7fc18c135790 auth_method 0 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 10 monclient(hunting): handle_auth_done global_id 134158 payload 824 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 1 -- 172.21.15.86:0/1865064543 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc18c074be0 con 0x7fc18c135790 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 msgr2=0x7fc18c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc1913f7640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 0x7fc18c09e5e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc18001c240 tx=0x7fc180009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:21.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1800b9050 con 0x7fc18c135790 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc18a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc18a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.546+0000 7fc18a7fc640 10 monclient: dump: 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:21.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7fc180000e50 con 0x7fc18c135790 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 msgr2=0x7fc18c09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 0x7fc18c09e5e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc18001c240 tx=0x7fc180009240 comp rx=0 tx=0).stop 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 msgr2=0x7fc18c09e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc18c135790 0x7fc18c09e5e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:21.554 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 118, 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 25915, 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.555 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.556 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.557 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:39:21.558 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:39:21.559 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.561 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:39:21.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:39:21.567 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:39:21.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:39:21.568 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:39:21.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:39:21.568 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:39:21.568 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:39:21.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc17803bb10 msgr2=0x7fc17803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc17803bb10 0x7fc17803dfd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc18c0a4a00 tx=0x7fc17c007340 comp rx=0 tx=0).stop 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 10 monclient: shutdown 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 msgr2=0x7fc18c1cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 --2- 172.21.15.86:0/1865064543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc18c0a50e0 0x7fc18c1cfb50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc18400b560 tx=0x7fc18400ba20 comp rx=0 tx=0).stop 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc1923f9640 1 -- 172.21.15.86:0/1865064543 reap_dead start 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc18a7fc640 1 -- 172.21.15.86:0/1865064543 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(401..401 src has 213..401) ==== 4803+0+0 (unknown 0 0 0) 0x7fc18404ff90 con 0x7fc18c0a50e0 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc1923f9640 1 -- 172.21.15.86:0/1865064543 reap_dead start 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 shutdown_connections 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 >> 172.21.15.86:0/1865064543 conn(0x7fc18c0aaae0 msgr2=0x7fc18c13f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 shutdown_connections 2024-09-17T15:39:21.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:21.550+0000 7fc193e5c640 1 -- 172.21.15.86:0/1865064543 wait complete. 2024-09-17T15:39:21.574 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:39:21.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1761: done (5 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1762: done (4 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1763: done (3 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1764: done (2 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1765: done (1 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1766: done (0 left) 2024-09-17T15:39:21.650 INFO:tasks.rados.rados.0.smithi203.stdout:1767: snap_remove snap 182 2024-09-17T15:39:22.660 INFO:tasks.rados.rados.0.smithi203.stdout:1767: done (0 left) 2024-09-17T15:39:22.660 INFO:tasks.rados.rados.0.smithi203.stdout:1768: delete oid 36 current snap is 199 2024-09-17T15:39:22.679 INFO:tasks.rados.rados.0.smithi203.stdout:1768: done (0 left) 2024-09-17T15:39:22.679 INFO:tasks.rados.rados.0.smithi203.stdout:1769: read oid 3 snap -1 2024-09-17T15:39:22.679 INFO:tasks.rados.rados.0.smithi203.stdout:1769: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:39:22.680 INFO:tasks.rados.rados.0.smithi203.stdout:1770: rollback oid 35 current snap is 199 2024-09-17T15:39:22.680 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 197 2024-09-17T15:39:22.680 INFO:tasks.rados.rados.0.smithi203.stdout:1771: rollback oid 23 current snap is 199 2024-09-17T15:39:22.680 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 177 2024-09-17T15:39:22.680 INFO:tasks.rados.rados.0.smithi203.stdout:1772: snap_create 2024-09-17T15:39:22.693 INFO:tasks.rados.rados.0.smithi203.stdout:1771: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:39:22.694 INFO:tasks.rados.rados.0.smithi203.stdout:1771: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:39:22.695 INFO:tasks.rados.rados.0.smithi203.stdout:1770: finishing rollback tid 0 to smithi20328086-35 2024-09-17T15:39:22.696 INFO:tasks.rados.rados.0.smithi203.stdout:1770: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:39:22.696 INFO:tasks.rados.rados.0.smithi203.stdout:1771: finishing rollback tid 2 to smithi20328086-23 2024-09-17T15:39:22.696 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 713 (ObjNum 530 snap 157 seq_num 530) dirty exists 2024-09-17T15:39:22.696 INFO:tasks.rados.rados.0.smithi203.stdout:1770: finishing rollback tid 2 to smithi20328086-35 2024-09-17T15:39:22.696 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 613 (ObjNum 542 snap 161 seq_num 542) dirty exists 2024-09-17T15:39:22.698 INFO:tasks.rados.rados.0.smithi203.stdout:1769: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:39:22.738 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1811125 2024-09-17T15:39:22.762 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:22.813 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1054284 2024-09-17T15:39:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:1769: done (3 left) 2024-09-17T15:39:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:1770: done (2 left) 2024-09-17T15:39:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:1771: done (1 left) 2024-09-17T15:39:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:1772: done (0 left) 2024-09-17T15:39:23.671 INFO:tasks.rados.rados.0.smithi203.stdout:1773: read oid 22 snap -1 2024-09-17T15:39:23.671 INFO:tasks.rados.rados.0.smithi203.stdout:1773: expect deleted 2024-09-17T15:39:23.671 INFO:tasks.rados.rados.0.smithi203.stdout:1774: snap_create 2024-09-17T15:39:24.678 INFO:tasks.rados.rados.0.smithi203.stdout:1773: done (1 left) 2024-09-17T15:39:24.678 INFO:tasks.rados.rados.0.smithi203.stdout:1774: done (0 left) 2024-09-17T15:39:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:1775: delete oid 14 current snap is 201 2024-09-17T15:39:24.685 INFO:tasks.rados.rados.0.smithi203.stdout:1775: done (0 left) 2024-09-17T15:39:24.685 INFO:tasks.rados.rados.0.smithi203.stdout:1776: delete oid 28 current snap is 201 2024-09-17T15:39:24.695 INFO:tasks.rados.rados.0.smithi203.stdout:1776: done (0 left) 2024-09-17T15:39:24.696 INFO:tasks.rados.rados.0.smithi203.stdout:1777: snap_create 2024-09-17T15:39:25.680 INFO:tasks.rados.rados.0.smithi203.stdout:1777: done (0 left) 2024-09-17T15:39:25.680 INFO:tasks.rados.rados.0.smithi203.stdout:1778: delete oid 18 current snap is 202 2024-09-17T15:39:25.693 INFO:tasks.rados.rados.0.smithi203.stdout:1778: done (0 left) 2024-09-17T15:39:25.693 INFO:tasks.rados.rados.0.smithi203.stdout:1779: read oid 3 snap -1 2024-09-17T15:39:25.693 INFO:tasks.rados.rados.0.smithi203.stdout:1779: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:39:25.693 INFO:tasks.rados.rados.0.smithi203.stdout:1780: write (excl) oid 23 current snap is 202 2024-09-17T15:39:25.693 INFO:tasks.rados.rados.0.smithi203.stdout:1780: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-17T15:39:25.717 INFO:tasks.rados.rados.0.smithi203.stdout:1780: writing smithi20328086-23 from 756480 to 1489492 tid 1 2024-09-17T15:39:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:1780: writing smithi20328086-23 from 2134711 to 2567260 tid 2 2024-09-17T15:39:25.736 INFO:tasks.rados.rados.0.smithi203.stdout:1781: write (excl) oid 4 current snap is 202 2024-09-17T15:39:25.736 INFO:tasks.rados.rados.0.smithi203.stdout:1781: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-17T15:39:25.751 INFO:tasks.rados.rados.0.smithi203.stdout:1781: writing smithi20328086-4 from 773287 to 1181548 tid 1 2024-09-17T15:39:25.764 INFO:tasks.rados.rados.0.smithi203.stdout:1781: writing smithi20328086-4 from 1793193 to 2369400 tid 2 2024-09-17T15:39:25.764 INFO:tasks.rados.rados.0.smithi203.stdout:1782: write (excl) oid 46 current snap is 202 2024-09-17T15:39:25.764 INFO:tasks.rados.rados.0.smithi203.stdout:1782: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-17T15:39:25.775 INFO:tasks.rados.rados.0.smithi203.stdout:1782: writing smithi20328086-46 from 790094 to 1273604 tid 1 2024-09-17T15:39:25.783 INFO:tasks.rados.rados.0.smithi203.stdout:1782: writing smithi20328086-46 from 1851675 to 2287893 tid 2 2024-09-17T15:39:25.783 INFO:tasks.rados.rados.0.smithi203.stdout:1783: delete oid 45 current snap is 202 2024-09-17T15:39:25.783 INFO:tasks.rados.rados.0.smithi203.stdout:1779: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:39:25.783 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: finishing write tid 1 to smithi20328086-23 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: finishing write tid 2 to smithi20328086-23 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: finishing write tid 3 to smithi20328086-23 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: finishing write tid 4 to smithi20328086-23 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: oid 23 updating version 0 to 719 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: oid 23 updating version 719 to 720 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:1780: oid 23 updating version 720 to 721 2024-09-17T15:39:25.946 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 721 (ObjNum 640 snap 202 seq_num 640) dirty exists 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1780: left oid 23 (ObjNum 640 snap 202 seq_num 640) 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: finishing write tid 1 to smithi20328086-4 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: finishing write tid 2 to smithi20328086-4 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: finishing write tid 3 to smithi20328086-4 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: finishing write tid 4 to smithi20328086-4 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: oid 4 updating version 0 to 365 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: oid 4 version 365 is already newer than 364 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: oid 4 updating version 365 to 366 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 366 (ObjNum 641 snap 202 seq_num 641) dirty exists 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1781: left oid 4 (ObjNum 641 snap 202 seq_num 641) 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1782: finishing write tid 1 to smithi20328086-46 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1782: finishing write tid 2 to smithi20328086-46 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1782: finishing write tid 3 to smithi20328086-46 2024-09-17T15:39:25.947 INFO:tasks.rados.rados.0.smithi203.stdout:1782: finishing write tid 4 to smithi20328086-46 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1782: oid 46 updating version 0 to 531 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1782: oid 46 version 531 is already newer than 529 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1782: oid 46 version 531 is already newer than 530 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 531 (ObjNum 642 snap 202 seq_num 642) dirty exists 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1782: left oid 46 (ObjNum 642 snap 202 seq_num 642) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1779: done (4 left) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1780: done (3 left) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1781: done (2 left) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1782: done (1 left) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1783: done (0 left) 2024-09-17T15:39:25.948 INFO:tasks.rados.rados.0.smithi203.stdout:1784: snap_remove snap 171 2024-09-17T15:39:26.691 INFO:tasks.rados.rados.0.smithi203.stdout:1784: done (0 left) 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1785: copy_from oid 1 from oid 21 current snap is 202 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1786: read oid 34 snap -1 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1786: expect deleted 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1787: read oid 27 snap -1 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1787: expect deleted 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1788: read oid 37 snap 169 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1788: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1789: read oid 47 snap -1 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1789: expect (ObjNum 631 snap 196 seq_num 631) 2024-09-17T15:39:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:1790: delete oid 38 current snap is 202 2024-09-17T15:39:26.712 INFO:tasks.rados.rados.0.smithi203.stdout:1788: expect (ObjNum 536 snap 158 seq_num 536) 2024-09-17T15:39:26.738 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2597038 2024-09-17T15:39:26.854 INFO:tasks.rados.rados.0.smithi203.stdout:1789: expect (ObjNum 631 snap 196 seq_num 631) 2024-09-17T15:39:26.891 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1785: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1785: finishing copy_from to smithi20328086-1 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1785: done (5 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1786: done (4 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1787: done (3 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1788: done (2 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1789: done (1 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1790: done (0 left) 2024-09-17T15:39:26.963 INFO:tasks.rados.rados.0.smithi203.stdout:1791: snap_create 2024-09-17T15:39:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:1791: done (0 left) 2024-09-17T15:39:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:1792: write oid 34 current snap is 203 2024-09-17T15:39:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:1792: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-17T15:39:27.707 INFO:tasks.rados.rados.0.smithi203.stdout:1792: writing smithi20328086-34 from 440515 to 1149772 tid 1 2024-09-17T15:39:27.729 INFO:tasks.rados.rados.0.smithi203.stdout:1792: writing smithi20328086-34 from 1910768 to 2494313 tid 2 2024-09-17T15:39:27.729 INFO:tasks.rados.rados.0.smithi203.stdout:1793: write (excl) oid 17 current snap is 203 2024-09-17T15:39:27.729 INFO:tasks.rados.rados.0.smithi203.stdout:1792: finishing write tid 1 to smithi20328086-34 2024-09-17T15:39:27.729 INFO:tasks.rados.rados.0.smithi203.stdout:1793: seq_num 646 ranges {457322=784506,1969250=443556} 2024-09-17T15:39:27.743 INFO:tasks.rados.rados.0.smithi203.stdout:1793: writing smithi20328086-17 from 457322 to 1241828 tid 1 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1793: writing smithi20328086-17 from 1969250 to 2412806 tid 2 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1794: copy_from oid 4 from oid 50 current snap is 203 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: finishing write tid 2 to smithi20328086-34 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: finishing write tid 3 to smithi20328086-34 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: finishing write tid 4 to smithi20328086-34 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: oid 34 updating version 0 to 615 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: oid 34 updating version 615 to 616 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: oid 34 version 616 is already newer than 614 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 616 (ObjNum 645 snap 203 seq_num 645) dirty exists 2024-09-17T15:39:27.757 INFO:tasks.rados.rados.0.smithi203.stdout:1792: left oid 34 (ObjNum 645 snap 203 seq_num 645) 2024-09-17T15:39:27.758 INFO:tasks.rados.rados.0.smithi203.stdout:1793: finishing write tid 1 to smithi20328086-17 2024-09-17T15:39:27.758 INFO:tasks.rados.rados.0.smithi203.stdout:1792: done (2 left) 2024-09-17T15:39:27.758 INFO:tasks.rados.rados.0.smithi203.stdout:1795: rollback oid 36 current snap is 203 2024-09-17T15:39:27.758 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 188 2024-09-17T15:39:27.758 INFO:tasks.rados.rados.0.smithi203.stdout:1796: snap_remove snap 193 2024-09-17T15:39:28.701 INFO:tasks.rados.rados.0.smithi203.stdout:1796: done (3 left) 2024-09-17T15:39:28.701 INFO:tasks.rados.rados.0.smithi203.stdout:1797: delete oid 23 current snap is 203 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: finishing write tid 2 to smithi20328086-17 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1795: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1795: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 828 (ObjNum 513 snap 153 seq_num 513) dirty exists 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: finishing write tid 3 to smithi20328086-17 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: finishing write tid 4 to smithi20328086-17 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: oid 17 updating version 0 to 708 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: oid 17 version 708 is already newer than 706 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: oid 17 version 708 is already newer than 707 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 708 (ObjNum 646 snap 203 seq_num 646) dirty exists 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1793: left oid 17 (ObjNum 646 snap 203 seq_num 646) 2024-09-17T15:39:28.702 INFO:tasks.rados.rados.0.smithi203.stdout:1794: finishing copy_from to smithi20328086-4 2024-09-17T15:39:28.703 INFO:tasks.rados.rados.0.smithi203.stdout:1794: got expected ENOENT (src dne) 2024-09-17T15:39:28.703 INFO:tasks.rados.rados.0.smithi203.stdout:1794: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1793: done (3 left) 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1794: done (2 left) 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1795: done (1 left) 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1797: done (0 left) 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1798: write (excl) oid 19 current snap is 203 2024-09-17T15:39:28.717 INFO:tasks.rados.rados.0.smithi203.stdout:1798: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-17T15:39:28.735 INFO:tasks.rados.rados.0.smithi203.stdout:1798: writing smithi20328086-19 from 490936 to 1142293 tid 1 2024-09-17T15:39:28.750 INFO:tasks.rados.rados.0.smithi203.stdout:1798: writing smithi20328086-19 from 1686214 to 2133439 tid 2 2024-09-17T15:39:28.762 INFO:tasks.rados.rados.0.smithi203.stdout:1798: writing smithi20328086-19 from 2861864 to 2890936 tid 3 2024-09-17T15:39:28.762 INFO:tasks.rados.rados.0.smithi203.stdout:1799: copy_from oid 37 from oid 6 current snap is 203 2024-09-17T15:39:28.762 INFO:tasks.rados.rados.0.smithi203.stdout:1798: finishing write tid 1 to smithi20328086-19 2024-09-17T15:39:28.763 INFO:tasks.rados.rados.0.smithi203.stdout:1798: finishing write tid 2 to smithi20328086-19 2024-09-17T15:39:28.763 INFO:tasks.rados.rados.0.smithi203.stdout:1800: copy_from oid 50 from oid 7 current snap is 203 2024-09-17T15:39:28.763 INFO:tasks.rados.rados.0.smithi203.stdout:1801: read oid 36 snap -1 2024-09-17T15:39:28.763 INFO:tasks.rados.rados.0.smithi203.stdout:1801: expect (ObjNum 513 snap 153 seq_num 513) 2024-09-17T15:39:28.763 INFO:tasks.rados.rados.0.smithi203.stdout:1802: snap_remove snap 186 2024-09-17T15:39:29.711 INFO:tasks.rados.rados.0.smithi203.stdout:1802: done (4 left) 2024-09-17T15:39:29.711 INFO:tasks.rados.rados.0.smithi203.stdout:1803: delete oid 20 current snap is 203 2024-09-17T15:39:29.711 INFO:tasks.rados.rados.0.smithi203.stdout:1798: finishing write tid 3 to smithi20328086-19 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: finishing write tid 4 to smithi20328086-19 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: finishing write tid 5 to smithi20328086-19 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: oid 19 updating version 0 to 714 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: oid 19 updating version 714 to 715 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: oid 19 version 715 is already newer than 712 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: oid 19 version 715 is already newer than 713 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 715 (ObjNum 648 snap 203 seq_num 648) dirty exists 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1798: left oid 19 (ObjNum 648 snap 203 seq_num 648) 2024-09-17T15:39:29.712 INFO:tasks.rados.rados.0.smithi203.stdout:1801: expect (ObjNum 513 snap 153 seq_num 513) 2024-09-17T15:39:29.778 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:29.873 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1992555 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1800: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1799: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1800: finishing copy_from to smithi20328086-50 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 471 (ObjNum 612 snap 185 seq_num 612) dirty exists 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1799: finishing copy_from to smithi20328086-37 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 614 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1798: done (4 left) 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1799: done (3 left) 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1800: done (2 left) 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1801: done (1 left) 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1803: done (0 left) 2024-09-17T15:39:29.885 INFO:tasks.rados.rados.0.smithi203.stdout:1804: rollback oid 7 current snap is 203 2024-09-17T15:39:29.886 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 191 2024-09-17T15:39:29.886 INFO:tasks.rados.rados.0.smithi203.stdout:1805: snap_create 2024-09-17T15:39:29.898 INFO:tasks.rados.rados.0.smithi203.stdout:1804: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:39:29.898 INFO:tasks.rados.rados.0.smithi203.stdout:1804: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:39:29.899 INFO:tasks.rados.rados.0.smithi203.stdout:1804: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:39:29.900 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 474 (ObjNum 612 snap 185 seq_num 612) dirty exists 2024-09-17T15:39:30.651 INFO:tasks.rados.rados.0.smithi203.stdout:1804: done (1 left) 2024-09-17T15:39:30.652 INFO:tasks.rados.rados.0.smithi203.stdout:1805: done (0 left) 2024-09-17T15:39:30.652 INFO:tasks.rados.rados.0.smithi203.stdout:1806: snap_remove snap 183 2024-09-17T15:39:31.662 INFO:tasks.rados.rados.0.smithi203.stdout:1806: done (0 left) 2024-09-17T15:39:31.662 INFO:tasks.rados.rados.0.smithi203.stdout:1807: read oid 45 snap -1 2024-09-17T15:39:31.662 INFO:tasks.rados.rados.0.smithi203.stdout:1807: expect deleted 2024-09-17T15:39:31.662 INFO:tasks.rados.rados.0.smithi203.stdout:1808: rollback oid 14 current snap is 204 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 189 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:1809: read oid 41 snap -1 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:1809: expect (ObjNum 628 snap 196 seq_num 628) 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:1810: read oid 42 snap -1 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:1810: expect deleted 2024-09-17T15:39:31.663 INFO:tasks.rados.rados.0.smithi203.stdout:1811: snap_create 2024-09-17T15:39:31.674 INFO:tasks.rados.rados.0.smithi203.stdout:1809: expect (ObjNum 628 snap 196 seq_num 628) 2024-09-17T15:39:31.778 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:31.805 INFO:tasks.rados.rados.0.smithi203.stdout:1808: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:39:31.806 INFO:tasks.rados.rados.0.smithi203.stdout:1808: finishing rollback tid 2 to smithi20328086-14 2024-09-17T15:39:31.806 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 752 (ObjNum 611 snap 184 seq_num 611) dirty exists 2024-09-17T15:39:32.680 INFO:tasks.rados.rados.0.smithi203.stdout:1807: done (4 left) 2024-09-17T15:39:32.680 INFO:tasks.rados.rados.0.smithi203.stdout:1808: done (3 left) 2024-09-17T15:39:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1809: done (2 left) 2024-09-17T15:39:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1810: done (1 left) 2024-09-17T15:39:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1811: done (0 left) 2024-09-17T15:39:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1812: write (excl) oid 42 current snap is 205 2024-09-17T15:39:32.681 INFO:tasks.rados.rados.0.smithi203.stdout:1812: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2024-09-17T15:39:32.695 INFO:tasks.rados.rados.0.smithi203.stdout:1812: writing smithi20328086-42 from 524550 to 926405 tid 1 2024-09-17T15:39:32.710 INFO:tasks.rados.rados.0.smithi203.stdout:1812: writing smithi20328086-42 from 1403178 to 1854072 tid 2 2024-09-17T15:39:32.727 INFO:tasks.rados.rados.0.smithi203.stdout:1812: writing smithi20328086-42 from 2516710 to 2924550 tid 3 2024-09-17T15:39:32.727 INFO:tasks.rados.rados.0.smithi203.stdout:1813: read oid 23 snap 179 2024-09-17T15:39:32.727 INFO:tasks.rados.rados.0.smithi203.stdout:1813: expect (ObjNum 600 snap 179 seq_num 600) 2024-09-17T15:39:32.728 INFO:tasks.rados.rados.0.smithi203.stdout:1814: rollback oid 15 current snap is 205 2024-09-17T15:39:32.728 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 191 2024-09-17T15:39:32.728 INFO:tasks.rados.rados.0.smithi203.stdout:1812: finishing write tid 1 to smithi20328086-42 2024-09-17T15:39:32.728 INFO:tasks.rados.rados.0.smithi203.stdout:1812: finishing write tid 2 to smithi20328086-42 2024-09-17T15:39:32.728 INFO:tasks.rados.rados.0.smithi203.stdout:1815: snap_remove snap 169 2024-09-17T15:39:33.682 INFO:tasks.rados.rados.0.smithi203.stdout:1815: done (3 left) 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1816: rollback oid 31 current snap is 205 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 197 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: finishing write tid 3 to smithi20328086-42 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: finishing write tid 4 to smithi20328086-42 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1814: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 396 (ObjNum 33 snap 0 seq_num 3607107088) dirty dne 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: finishing write tid 5 to smithi20328086-42 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: oid 42 updating version 0 to 549 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: oid 42 version 549 is already newer than 546 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: oid 42 version 549 is already newer than 547 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: oid 42 version 549 is already newer than 548 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 549 (ObjNum 650 snap 205 seq_num 650) dirty exists 2024-09-17T15:39:33.683 INFO:tasks.rados.rados.0.smithi203.stdout:1812: left oid 42 (ObjNum 650 snap 205 seq_num 650) 2024-09-17T15:39:33.684 INFO:tasks.rados.rados.0.smithi203.stdout:1813: expect (ObjNum 600 snap 179 seq_num 600) 2024-09-17T15:39:33.689 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 778025 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1816: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1816: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1816: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 515 (ObjNum 598 snap 179 seq_num 598) dirty exists 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1812: done (3 left) 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1813: done (2 left) 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1814: done (1 left) 2024-09-17T15:39:33.816 INFO:tasks.rados.rados.0.smithi203.stdout:1816: done (0 left) 2024-09-17T15:39:33.817 INFO:tasks.rados.rados.0.smithi203.stdout:1817: snap_create 2024-09-17T15:39:34.699 INFO:tasks.rados.rados.0.smithi203.stdout:1817: done (0 left) 2024-09-17T15:39:34.699 INFO:tasks.rados.rados.0.smithi203.stdout:1818: write oid 41 current snap is 206 2024-09-17T15:39:34.699 INFO:tasks.rados.rados.0.smithi203.stdout:1818: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-17T15:39:34.715 INFO:tasks.rados.rados.0.smithi203.stdout:1818: writing smithi20328086-41 from 541357 to 1018461 tid 1 2024-09-17T15:39:34.737 INFO:tasks.rados.rados.0.smithi203.stdout:1818: writing smithi20328086-41 from 1745307 to 2339859 tid 2 2024-09-17T15:39:34.737 INFO:tasks.rados.rados.0.smithi203.stdout:1819: snap_create 2024-09-17T15:39:34.737 INFO:tasks.rados.rados.0.smithi203.stdout:1818: finishing write tid 1 to smithi20328086-41 2024-09-17T15:39:34.745 INFO:tasks.rados.rados.0.smithi203.stdout:1818: finishing write tid 2 to smithi20328086-41 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: finishing write tid 3 to smithi20328086-41 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: finishing write tid 4 to smithi20328086-41 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: oid 41 updating version 0 to 573 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: oid 41 version 573 is already newer than 572 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: oid 41 version 573 is already newer than 571 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 573 (ObjNum 651 snap 206 seq_num 651) dirty exists 2024-09-17T15:39:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:1818: left oid 41 (ObjNum 651 snap 206 seq_num 651) 2024-09-17T15:39:35.655 INFO:tasks.rados.rados.0.smithi203.stdout:1818: done (1 left) 2024-09-17T15:39:35.655 INFO:tasks.rados.rados.0.smithi203.stdout:1819: done (0 left) 2024-09-17T15:39:35.655 INFO:tasks.rados.rados.0.smithi203.stdout:1820: read oid 4 snap 172 2024-09-17T15:39:35.655 INFO:tasks.rados.rados.0.smithi203.stdout:1820: expect deleted 2024-09-17T15:39:35.655 INFO:tasks.rados.rados.0.smithi203.stdout:1821: snap_remove snap 202 2024-09-17T15:39:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:1821: done (1 left) 2024-09-17T15:39:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:1822: delete oid 37 current snap is 207 2024-09-17T15:39:37.232 INFO:tasks.rados.rados.0.smithi203.stdout:1820: done (1 left) 2024-09-17T15:39:37.232 INFO:tasks.rados.rados.0.smithi203.stdout:1822: done (0 left) 2024-09-17T15:39:37.233 INFO:tasks.rados.rados.0.smithi203.stdout:1823: write (excl) oid 23 current snap is 207 2024-09-17T15:39:37.233 INFO:tasks.rados.rados.0.smithi203.stdout:1823: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-17T15:39:37.252 INFO:tasks.rados.rados.0.smithi203.stdout:1823: writing smithi20328086-23 from 574971 to 1202573 tid 1 2024-09-17T15:39:37.273 INFO:tasks.rados.rados.0.smithi203.stdout:1823: writing smithi20328086-23 from 1862271 to 2460492 tid 2 2024-09-17T15:39:37.273 INFO:tasks.rados.rados.0.smithi203.stdout:1824: delete oid 7 current snap is 207 2024-09-17T15:39:37.273 INFO:tasks.rados.rados.0.smithi203.stdout:1823: finishing write tid 1 to smithi20328086-23 2024-09-17T15:39:37.278 INFO:tasks.rados.rados.0.smithi203.stdout:1824: done (1 left) 2024-09-17T15:39:37.278 INFO:tasks.rados.rados.0.smithi203.stdout:1825: snap_create 2024-09-17T15:39:37.283 INFO:tasks.rados.rados.0.smithi203.stdout:1823: finishing write tid 2 to smithi20328086-23 2024-09-17T15:39:37.284 INFO:tasks.rados.rados.0.smithi203.stdout:1823: finishing write tid 3 to smithi20328086-23 2024-09-17T15:39:37.284 INFO:tasks.rados.rados.0.smithi203.stdout:1823: finishing write tid 4 to smithi20328086-23 2024-09-17T15:39:37.284 INFO:tasks.rados.rados.0.smithi203.stdout:1823: oid 23 updating version 0 to 773 2024-09-17T15:39:37.285 INFO:tasks.rados.rados.0.smithi203.stdout:1823: oid 23 updating version 773 to 775 2024-09-17T15:39:37.285 INFO:tasks.rados.rados.0.smithi203.stdout:1823: oid 23 version 775 is already newer than 774 2024-09-17T15:39:37.285 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 775 (ObjNum 653 snap 207 seq_num 653) dirty exists 2024-09-17T15:39:37.285 INFO:tasks.rados.rados.0.smithi203.stdout:1823: left oid 23 (ObjNum 653 snap 207 seq_num 653) 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1823: done (1 left) 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1825: done (0 left) 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1826: copy_from oid 25 from oid 4 current snap is 208 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1827: copy_from oid 41 from oid 34 current snap is 208 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1828: read oid 21 snap -1 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1828: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:38.234 INFO:tasks.rados.rados.0.smithi203.stdout:1829: delete oid 50 current snap is 208 2024-09-17T15:39:38.252 INFO:tasks.rados.rados.0.smithi203.stdout:1829: done (3 left) 2024-09-17T15:39:38.252 INFO:tasks.rados.rados.0.smithi203.stdout:1830: write (excl) oid 29 current snap is 208 2024-09-17T15:39:38.252 INFO:tasks.rados.rados.0.smithi203.stdout:1830: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-17T15:39:38.272 INFO:tasks.rados.rados.0.smithi203.stdout:1830: writing smithi20328086-29 from 625392 to 1195094 tid 1 2024-09-17T15:39:38.291 INFO:tasks.rados.rados.0.smithi203.stdout:1830: writing smithi20328086-29 from 1637717 to 2383265 tid 2 2024-09-17T15:39:38.301 INFO:tasks.rados.rados.0.smithi203.stdout:1830: writing smithi20328086-29 from 2848542 to 3025392 tid 3 2024-09-17T15:39:38.301 INFO:tasks.rados.rados.0.smithi203.stdout:1831: write (excl) oid 18 current snap is 208 2024-09-17T15:39:38.301 INFO:tasks.rados.rados.0.smithi203.stdout:1831: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-17T15:39:38.316 INFO:tasks.rados.rados.0.smithi203.stdout:1831: writing smithi20328086-18 from 642199 to 1287150 tid 1 2024-09-17T15:39:38.326 INFO:tasks.rados.rados.0.smithi203.stdout:1831: writing smithi20328086-18 from 1696199 to 2301758 tid 2 2024-09-17T15:39:38.332 INFO:tasks.rados.rados.0.smithi203.stdout:1831: writing smithi20328086-18 from 2992318 to 3042199 tid 3 2024-09-17T15:39:38.332 INFO:tasks.rados.rados.0.smithi203.stdout:1832: snap_remove snap 203 2024-09-17T15:39:39.237 INFO:tasks.rados.rados.0.smithi203.stdout:1832: done (5 left) 2024-09-17T15:39:39.237 INFO:tasks.rados.rados.0.smithi203.stdout:1833: snap_create 2024-09-17T15:39:39.238 INFO:tasks.rados.rados.0.smithi203.stdout:1826: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:39:39.238 INFO:tasks.rados.rados.0.smithi203.stdout:1828: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:39.382 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:39.478 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:39.573 INFO:tasks.rados.rados.0.smithi203.stdout:1826: finishing copy_from to smithi20328086-25 2024-09-17T15:39:39.573 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 366 (ObjNum 641 snap 202 seq_num 641) dirty exists 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1827: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1827: finishing copy_from to smithi20328086-41 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 616 (ObjNum 645 snap 203 seq_num 645) dirty exists 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: finishing write tid 1 to smithi20328086-29 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: finishing write tid 2 to smithi20328086-29 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: finishing write tid 3 to smithi20328086-29 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: finishing write tid 4 to smithi20328086-29 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: finishing write tid 5 to smithi20328086-29 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: oid 29 updating version 0 to 626 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: oid 29 version 626 is already newer than 623 2024-09-17T15:39:39.574 INFO:tasks.rados.rados.0.smithi203.stdout:1830: oid 29 version 626 is already newer than 624 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1830: oid 29 version 626 is already newer than 625 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 626 (ObjNum 656 snap 208 seq_num 656) dirty exists 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1830: left oid 29 (ObjNum 656 snap 208 seq_num 656) 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: finishing write tid 1 to smithi20328086-18 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: finishing write tid 2 to smithi20328086-18 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: finishing write tid 3 to smithi20328086-18 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: finishing write tid 4 to smithi20328086-18 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: finishing write tid 5 to smithi20328086-18 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: oid 18 updating version 0 to 560 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: oid 18 version 560 is already newer than 559 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: oid 18 updating version 560 to 561 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:1831: oid 18 updating version 561 to 562 2024-09-17T15:39:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 562 (ObjNum 657 snap 208 seq_num 657) dirty exists 2024-09-17T15:39:39.576 INFO:tasks.rados.rados.0.smithi203.stdout:1831: left oid 18 (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:39:40.238 INFO:tasks.rados.rados.0.smithi203.stdout:1826: done (5 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1827: done (4 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1828: done (3 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1830: done (2 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1831: done (1 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1833: done (0 left) 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1834: write (excl) oid 20 current snap is 209 2024-09-17T15:39:40.239 INFO:tasks.rados.rados.0.smithi203.stdout:1834: seq_num 658 ranges {659006=720200,2154681=749217} 2024-09-17T15:39:40.251 INFO:tasks.rados.rados.0.smithi203.stdout:1834: writing smithi20328086-20 from 659006 to 1379206 tid 1 2024-09-17T15:39:40.265 INFO:tasks.rados.rados.0.smithi203.stdout:1834: writing smithi20328086-20 from 2154681 to 2903898 tid 2 2024-09-17T15:39:40.265 INFO:tasks.rados.rados.0.smithi203.stdout:1835: write oid 25 current snap is 209 2024-09-17T15:39:40.265 INFO:tasks.rados.rados.0.smithi203.stdout:1835: seq_num 659 ranges {675813=795449,2096810=609228} 2024-09-17T15:39:40.277 INFO:tasks.rados.rados.0.smithi203.stdout:1835: writing smithi20328086-25 from 675813 to 1471262 tid 1 2024-09-17T15:39:40.287 INFO:tasks.rados.rados.0.smithi203.stdout:1835: writing smithi20328086-25 from 2096810 to 2706038 tid 2 2024-09-17T15:39:40.288 INFO:tasks.rados.rados.0.smithi203.stdout:1836: snap_remove snap 179 2024-09-17T15:39:40.655 INFO:tasks.rados.rados.0.smithi203.stdout:1836: done (2 left) 2024-09-17T15:39:40.655 INFO:tasks.rados.rados.0.smithi203.stdout:1837: delete oid 22 current snap is 209 2024-09-17T15:39:40.655 INFO:tasks.rados.rados.0.smithi203.stdout:1834: finishing write tid 1 to smithi20328086-20 2024-09-17T15:39:40.655 INFO:tasks.rados.rados.0.smithi203.stdout:1834: finishing write tid 2 to smithi20328086-20 2024-09-17T15:39:40.655 INFO:tasks.rados.rados.0.smithi203.stdout:1834: finishing write tid 3 to smithi20328086-20 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1834: finishing write tid 4 to smithi20328086-20 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1834: oid 20 updating version 0 to 776 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1834: oid 20 updating version 776 to 777 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1834: oid 20 updating version 777 to 778 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 778 (ObjNum 658 snap 209 seq_num 658) dirty exists 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1834: left oid 20 (ObjNum 658 snap 209 seq_num 658) 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: finishing write tid 1 to smithi20328086-25 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: finishing write tid 2 to smithi20328086-25 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: finishing write tid 3 to smithi20328086-25 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: finishing write tid 4 to smithi20328086-25 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: oid 25 updating version 0 to 632 2024-09-17T15:39:40.656 INFO:tasks.rados.rados.0.smithi203.stdout:1835: oid 25 version 632 is already newer than 630 2024-09-17T15:39:40.657 INFO:tasks.rados.rados.0.smithi203.stdout:1835: oid 25 version 632 is already newer than 631 2024-09-17T15:39:40.657 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 632 (ObjNum 659 snap 209 seq_num 659) dirty exists 2024-09-17T15:39:40.657 INFO:tasks.rados.rados.0.smithi203.stdout:1835: left oid 25 (ObjNum 659 snap 209 seq_num 659) 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1834: done (2 left) 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1835: done (1 left) 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1837: done (0 left) 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1838: read oid 1 snap -1 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1838: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:40.668 INFO:tasks.rados.rados.0.smithi203.stdout:1839: read oid 8 snap -1 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1839: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1840: read oid 12 snap 172 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1840: expect (ObjNum 544 snap 161 seq_num 544) 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1841: read oid 5 snap -1 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1841: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1842: copy_from oid 36 from oid 18 current snap is 209 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1843: write oid 15 current snap is 209 2024-09-17T15:39:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:1843: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-17T15:39:40.684 INFO:tasks.rados.rados.0.smithi203.stdout:1843: writing smithi20328086-15 from 709427 to 1255374 tid 1 2024-09-17T15:39:40.699 INFO:tasks.rados.rados.0.smithi203.stdout:1843: writing smithi20328086-15 from 1813774 to 2426671 tid 2 2024-09-17T15:39:40.705 INFO:tasks.rados.rados.0.smithi203.stdout:1843: writing smithi20328086-15 from 2985657 to 3109427 tid 3 2024-09-17T15:39:40.705 INFO:tasks.rados.rados.0.smithi203.stdout:1844: rollback oid 46 current snap is 209 2024-09-17T15:39:40.706 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 196 2024-09-17T15:39:40.706 INFO:tasks.rados.rados.0.smithi203.stdout:1840: expect (ObjNum 544 snap 161 seq_num 544) 2024-09-17T15:39:40.885 INFO:tasks.rados.rados.0.smithi203.stdout:1838: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:39:41.009 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1949280 2024-09-17T15:39:41.148 INFO:tasks.rados.rados.0.smithi203.stdout:1839: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:41.255 INFO:tasks.rados.rados.0.smithi203.stdout:1841: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:41.293 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:41.331 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: finishing write tid 1 to smithi20328086-15 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1842: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: finishing write tid 2 to smithi20328086-15 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: finishing write tid 3 to smithi20328086-15 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: finishing write tid 4 to smithi20328086-15 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: finishing write tid 5 to smithi20328086-15 2024-09-17T15:39:41.370 INFO:tasks.rados.rados.0.smithi203.stdout:1843: oid 15 updating version 0 to 633 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1843: oid 15 version 633 is already newer than 632 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1843: oid 15 updating version 633 to 635 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1843: oid 15 version 635 is already newer than 634 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 635 (ObjNum 661 snap 209 seq_num 661) dirty exists 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1843: left oid 15 (ObjNum 661 snap 209 seq_num 661) 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1842: finishing copy_from to smithi20328086-36 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 562 (ObjNum 657 snap 208 seq_num 657) dirty exists 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1844: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1844: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 560 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1838: done (6 left) 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1839: done (5 left) 2024-09-17T15:39:41.371 INFO:tasks.rados.rados.0.smithi203.stdout:1840: done (4 left) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1841: done (3 left) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1842: done (2 left) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1843: done (1 left) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1844: done (0 left) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1845: read oid 10 snap -1 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1845: expect deleted 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1846: read oid 31 snap -1 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1846: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1847: write oid 9 current snap is 209 2024-09-17T15:39:41.372 INFO:tasks.rados.rados.0.smithi203.stdout:1847: seq_num 662 ranges {726234=737549,1988609=756555} 2024-09-17T15:39:41.393 INFO:tasks.rados.rados.0.smithi203.stdout:1847: writing smithi20328086-9 from 726234 to 1463783 tid 1 2024-09-17T15:39:41.414 INFO:tasks.rados.rados.0.smithi203.stdout:1847: writing smithi20328086-9 from 1988609 to 2745164 tid 2 2024-09-17T15:39:41.415 INFO:tasks.rados.rados.0.smithi203.stdout:1848: rollback oid 15 current snap is 209 2024-09-17T15:39:41.415 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 192 2024-09-17T15:39:41.415 INFO:tasks.rados.rados.0.smithi203.stdout:1846: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:39:41.458 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:1847: finishing write tid 1 to smithi20328086-9 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:1848: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:1848: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 637 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:1847: finishing write tid 2 to smithi20328086-9 2024-09-17T15:39:41.485 INFO:tasks.rados.rados.0.smithi203.stdout:1847: finishing write tid 3 to smithi20328086-9 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: finishing write tid 4 to smithi20328086-9 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: oid 9 updating version 0 to 636 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: oid 9 updating version 636 to 637 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: oid 9 updating version 637 to 638 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 638 (ObjNum 662 snap 209 seq_num 662) dirty exists 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: left oid 9 (ObjNum 662 snap 209 seq_num 662) 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1845: done (3 left) 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1846: done (2 left) 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1847: done (1 left) 2024-09-17T15:39:41.486 INFO:tasks.rados.rados.0.smithi203.stdout:1848: done (0 left) 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:1849: rollback oid 49 current snap is 209 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 195 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:1850: read oid 24 snap -1 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:1850: expect deleted 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:1851: write (excl) oid 10 current snap is 209 2024-09-17T15:39:41.487 INFO:tasks.rados.rados.0.smithi203.stdout:1851: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-09-17T15:39:41.503 INFO:tasks.rados.rados.0.smithi203.stdout:1851: writing smithi20328086-10 from 743041 to 1155839 tid 1 2024-09-17T15:39:41.524 INFO:tasks.rados.rados.0.smithi203.stdout:1851: writing smithi20328086-10 from 1930738 to 2547304 tid 2 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1851: writing smithi20328086-10 from 3040503 to 3143041 tid 3 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1852: copy_from oid 37 from oid 21 current snap is 209 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1849: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1849: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1849: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 866 (ObjNum 562 snap 167 seq_num 562) dirty exists 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1851: finishing write tid 1 to smithi20328086-10 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1849: done (3 left) 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1850: done (2 left) 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1853: write oid 26 current snap is 209 2024-09-17T15:39:41.533 INFO:tasks.rados.rados.0.smithi203.stdout:1853: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-17T15:39:41.550 INFO:tasks.rados.rados.0.smithi203.stdout:1853: writing smithi20328086-26 from 759848 to 1247895 tid 1 2024-09-17T15:39:41.569 INFO:tasks.rados.rados.0.smithi203.stdout:1853: writing smithi20328086-26 from 1989220 to 2749444 tid 2 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1854: rollback oid 9 current snap is 209 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 200 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: finishing write tid 2 to smithi20328086-10 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: finishing write tid 3 to smithi20328086-10 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: finishing write tid 4 to smithi20328086-10 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: finishing write tid 5 to smithi20328086-10 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: oid 10 updating version 0 to 678 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: oid 10 updating version 678 to 679 2024-09-17T15:39:41.570 INFO:tasks.rados.rados.0.smithi203.stdout:1851: oid 10 version 679 is already newer than 677 2024-09-17T15:39:41.571 INFO:tasks.rados.rados.0.smithi203.stdout:1851: oid 10 version 679 is already newer than 676 2024-09-17T15:39:41.571 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 679 (ObjNum 663 snap 209 seq_num 663) dirty exists 2024-09-17T15:39:41.571 INFO:tasks.rados.rados.0.smithi203.stdout:1851: left oid 10 (ObjNum 663 snap 209 seq_num 663) 2024-09-17T15:39:41.571 INFO:tasks.rados.rados.0.smithi203.stdout:1852: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1852: finishing copy_from to smithi20328086-37 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1853: finishing write tid 1 to smithi20328086-26 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1851: done (3 left) 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1852: done (2 left) 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1855: write (excl) oid 39 current snap is 209 2024-09-17T15:39:41.572 INFO:tasks.rados.rados.0.smithi203.stdout:1855: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-17T15:39:41.574 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:39:41.574 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:39:41.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:39:41.578 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:39:41.578 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:39:41.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:41.580 INFO:tasks.rados.rados.0.smithi203.stdout:1855: writing smithi20328086-39 from 776655 to 1339951 tid 1 2024-09-17T15:39:41.590 INFO:tasks.rados.rados.0.smithi203.stdout:1855: writing smithi20328086-39 from 2047702 to 2667937 tid 2 2024-09-17T15:39:41.595 INFO:tasks.rados.rados.0.smithi203.stdout:1855: writing smithi20328086-39 from 3095349 to 3176655 tid 3 2024-09-17T15:39:41.595 INFO:tasks.rados.rados.0.smithi203.stdout:1856: write oid 19 current snap is 209 2024-09-17T15:39:41.595 INFO:tasks.rados.rados.0.smithi203.stdout:1854: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1854: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1853: finishing write tid 2 to smithi20328086-26 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1854: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 641 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1853: finishing write tid 3 to smithi20328086-26 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1853: finishing write tid 4 to smithi20328086-26 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1853: oid 26 updating version 0 to 870 2024-09-17T15:39:41.596 INFO:tasks.rados.rados.0.smithi203.stdout:1853: oid 26 version 870 is already newer than 868 2024-09-17T15:39:41.597 INFO:tasks.rados.rados.0.smithi203.stdout:1853: oid 26 version 870 is already newer than 869 2024-09-17T15:39:41.597 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 870 (ObjNum 664 snap 209 seq_num 664) dirty exists 2024-09-17T15:39:41.597 INFO:tasks.rados.rados.0.smithi203.stdout:1853: left oid 26 (ObjNum 664 snap 209 seq_num 664) 2024-09-17T15:39:41.597 INFO:tasks.rados.rados.0.smithi203.stdout:1855: finishing write tid 1 to smithi20328086-39 2024-09-17T15:39:41.597 INFO:tasks.rados.rados.0.smithi203.stdout:1856: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-17T15:39:41.607 INFO:tasks.rados.rados.0.smithi203.stdout:1856: writing smithi20328086-19 from 793462 to 1432007 tid 1 2024-09-17T15:39:41.619 INFO:tasks.rados.rados.0.smithi203.stdout:1856: writing smithi20328086-19 from 2106184 to 2870077 tid 2 2024-09-17T15:39:41.619 INFO:tasks.rados.rados.0.smithi203.stdout:1853: done (3 left) 2024-09-17T15:39:41.619 INFO:tasks.rados.rados.0.smithi203.stdout:1854: done (2 left) 2024-09-17T15:39:41.619 INFO:tasks.rados.rados.0.smithi203.stdout:1857: read oid 22 snap -1 2024-09-17T15:39:41.620 INFO:tasks.rados.rados.0.smithi203.stdout:1857: expect deleted 2024-09-17T15:39:41.620 INFO:tasks.rados.rados.0.smithi203.stdout:1858: copy_from oid 48 from oid 9 current snap is 209 2024-09-17T15:39:41.620 INFO:tasks.rados.rados.0.smithi203.stdout:1855: finishing write tid 2 to smithi20328086-39 2024-09-17T15:39:41.620 INFO:tasks.rados.rados.0.smithi203.stdout:1859: snap_remove snap 208 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.701+0000 7f5b48c6f640 10 monclient: get_monmap_and_config 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.701+0000 7f5b48c6f640 10 monclient: build_initial_monmap 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.701+0000 7f5b48c6f640 10 monclient: monmap: 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:41.707741+0000 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:41.707741+0000 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:41.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:41.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:41.705 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:41.705 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:41.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 Processor -- start 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 -- start start 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient: init 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b440689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5b44068db0 0x7f5b440691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f5b44068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b44139f00 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4409d770 con 0x7f5b440689d0 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b4409ed00 con 0x7f5b44139f00 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5b4409efe0 con 0x7f5b44068db0 2024-09-17T15:39:41.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b48c6f640 10 monclient(hunting): authenticate will time out at 1714.672880s 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b4259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5b440689d0 0x7f5b4409ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42370/0 (socket says 172.21.15.86:42370) 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b4259c640 1 -- 172.21.15.86:0/970538852 learned_addr learned my addr 172.21.15.86:0/970538852 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 1 --2- 172.21.15.86:0/970538852 >> v2:172.21.15.203:3300/0 conn(0x7f5b44068db0 0x7f5b440691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): get_auth_request con 0x7f5b44068db0 auth_method 0 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:41.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b4259c640 1 -- 172.21.15.86:0/970538852 >> v1:172.21.15.86:6789/0 conn(0x7f5b440689d0 legacy=0x7f5b4409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b42d9d640 1 -- 172.21.15.86:0/970538852 >> v1:172.21.15.203:6789/0 conn(0x7f5b44139f00 legacy=0x7f5b4413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient(hunting): handle_auth_done global_id 134161 payload 274 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 1 -- 172.21.15.86:0/970538852 >> v1:172.21.15.203:6789/0 conn(0x7f5b44139f00 legacy=0x7f5b4413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 1 -- 172.21.15.86:0/970538852 >> v1:172.21.15.86:6789/0 conn(0x7f5b440689d0 legacy=0x7f5b4409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:41.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 1 monclient: found mon.noname-c 2024-09-17T15:39:41.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:41.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:39:41.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.705+0000 7f5b41d9b640 1 -- 172.21.15.86:0/970538852 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b44069700 con 0x7f5b44068db0 2024-09-17T15:39:41.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 -- 172.21.15.86:0/970538852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b44068db0 msgr2=0x7f5b440691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:41.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 --2- 172.21.15.86:0/970538852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b44068db0 0x7f5b440691c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098c0 tx=0x7f5b3801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c73ac16aa7f8c522 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 1 -- 172.21.15.86:0/970538852 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380ab680 con 0x7f5b44068db0 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: dump: 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: _finish_auth 0 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: _check_auth_tickets 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:41.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 1 -- 172.21.15.86:0/970538852 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b380b0070 con 0x7f5b44068db0 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 1 -- 172.21.15.86:0/970538852 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b380aba40 con 0x7f5b44068db0 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4159a640 10 monclient: dump: 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:41.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 5 monclient: authenticate success, global_id 134161 2024-09-17T15:39:41.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 4 set_mon_vals no callback set 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: shutdown 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- 172.21.15.86:0/970538852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b44068db0 msgr2=0x7f5b440691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/970538852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b44068db0 0x7f5b440691c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b380098c0 tx=0x7f5b3801c460 comp rx=0 tx=0).stop 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- 172.21.15.86:0/970538852 shutdown_connections 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/970538852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b44068db0 0x7f5b440691c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- 172.21.15.86:0/970538852 >> 172.21.15.86:0/970538852 conn(0x7f5b440a6440 msgr2=0x7f5b440a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- 172.21.15.86:0/970538852 shutdown_connections 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- 172.21.15.86:0/970538852 wait complete. 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: build_initial_monmap 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: monmap: 2024-09-17T15:39:41.718 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:41.719 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:41.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:41.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:41.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 Processor -- start 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- start start 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: init 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b440689d0 0x7f5b440737f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-b con 0x7f5b440689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-a con 0x7f5b44068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 0x7f5b4413a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): picked mon.noname-c con 0x7f5b44139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:41.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441409d0 con 0x7f5b44068db0 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44140850 con 0x7f5b440689d0 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5b44140b50 con 0x7f5b44139f00 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b48c6f640 10 monclient(hunting): authenticate will time out at 1714.682929s 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 0x7f5b4413a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54128/0 (socket says 172.21.15.86:54128) 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 -- 172.21.15.86:0/1966415759 learned_addr learned my addr 172.21.15.86:0/1966415759 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 10 monclient(hunting): get_auth_request con 0x7f5b44068db0 auth_method 0 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:41.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): get_auth_request con 0x7f5b44139f00 auth_method 0 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 msgr2=0x7f5b4413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 msgr2=0x7f5b4413a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b41d9b640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4259c640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b440689d0 0x7f5b440737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4259c640 10 monclient(hunting): get_auth_request con 0x7f5b440689d0 auth_method 0 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient(hunting): handle_auth_done global_id 104486 payload 824 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b440689d0 msgr2=0x7f5b440737f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b440689d0 0x7f5b440737f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 msgr2=0x7f5b4413a310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 monclient: found mon.noname-c 2024-09-17T15:39:41.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.713+0000 7f5b42d9d640 1 -- 172.21.15.86:0/1966415759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b44071bf0 con 0x7f5b44139f00 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b42d9d640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 msgr2=0x7f5b4413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b42d9d640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 0x7f5b4413a850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c700 tx=0x7f5b3c00cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c017070 con 0x7f5b44139f00 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: dump: 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.730 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 5 monclient: authenticate success, global_id 104486 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c00a640 con 0x7f5b44139f00 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:41.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c00aa30 con 0x7f5b44139f00 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 10 monclient: dump: 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b40d99640 4 set_mon_vals no callback set 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _renew_subs 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4409d460 con 0x7f5b44139f00 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _renew_subs 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b440721b0 con 0x7f5b44139f00 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b440691c0 con 0x7f5b44139f00 2024-09-17T15:39:41.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5b3c01a070 con 0x7f5b44139f00 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 0x7f5b2403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.717+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(421..421 src has 213..421) ==== 4859+0+0 (secure 0 0 0) 0x7f5b3c04f450 con 0x7f5b44139f00 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.721+0000 7f5b4259c640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 0x7f5b2403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.721+0000 7f5b4259c640 10 monclient: get_auth_request con 0x7f5b2403bb60 auth_method 0 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.721+0000 7f5b4259c640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 0x7f5b2403e020 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f70 tx=0x7f5b34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.725+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5b3c010300 con 0x7f5b44139f00 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.725+0000 7f5b32ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:41.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.725+0000 7f5b32ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:41.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.957+0000 7f5b48c6f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:41.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.957+0000 7f5b48c6f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:41.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.957+0000 7f5b48c6f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:41.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.957+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5b440709b0 con 0x7f5b44139f00 2024-09-17T15:39:41.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.957+0000 7f5b32ffd640 1 -- 172.21.15.86:0/1966415759 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5b3c00adb0 con 0x7f5b44139f00 2024-09-17T15:39:41.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b32ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:41.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b32ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:41.960 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 msgr2=0x7f5b2403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 0x7f5b2403e020 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f70 tx=0x7f5b34009040 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 10 monclient: shutdown 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 msgr2=0x7f5b4413a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 0x7f5b4413a850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00c700 tx=0x7f5b3c00cbc0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 shutdown_connections 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5b2403bb60 0x7f5b2403e020 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b44139f00 0x7f5b4413a850 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b44068db0 0x7f5b4413a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 --2- 172.21.15.86:0/1966415759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b440689d0 0x7f5b440737f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:41.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 >> 172.21.15.86:0/1966415759 conn(0x7f5b440a6440 msgr2=0x7f5b4409ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:41.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 shutdown_connections 2024-09-17T15:39:41.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:41.961+0000 7f5b48c6f640 1 -- 172.21.15.86:0/1966415759 wait complete. 2024-09-17T15:39:42.130 INFO:tasks.rados.rados.0.smithi203.stdout:1859: done (4 left) 2024-09-17T15:39:42.130 INFO:tasks.rados.rados.0.smithi203.stdout:1860: snap_create 2024-09-17T15:39:42.130 INFO:tasks.rados.rados.0.smithi203.stdout:1855: finishing write tid 3 to smithi20328086-39 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: finishing write tid 4 to smithi20328086-39 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: finishing write tid 5 to smithi20328086-39 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: oid 39 updating version 0 to 759 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: oid 39 version 759 is already newer than 756 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: oid 39 version 759 is already newer than 757 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: oid 39 version 759 is already newer than 758 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 759 (ObjNum 665 snap 209 seq_num 665) dirty exists 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1855: left oid 39 (ObjNum 665 snap 209 seq_num 665) 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: finishing write tid 1 to smithi20328086-19 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: finishing write tid 2 to smithi20328086-19 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: finishing write tid 3 to smithi20328086-19 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: finishing write tid 4 to smithi20328086-19 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: oid 19 updating version 0 to 763 2024-09-17T15:39:42.131 INFO:tasks.rados.rados.0.smithi203.stdout:1856: oid 19 version 763 is already newer than 761 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:1856: oid 19 version 763 is already newer than 762 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 763 (ObjNum 666 snap 209 seq_num 666) dirty exists 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:1856: left oid 19 (ObjNum 666 snap 209 seq_num 666) 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:1858: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:1858: finishing copy_from to smithi20328086-48 2024-09-17T15:39:42.132 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 641 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-17T15:39:43.121 INFO:tasks.rados.rados.0.smithi203.stdout:1855: done (4 left) 2024-09-17T15:39:43.121 INFO:tasks.rados.rados.0.smithi203.stdout:1856: done (3 left) 2024-09-17T15:39:43.121 INFO:tasks.rados.rados.0.smithi203.stdout:1857: done (2 left) 2024-09-17T15:39:43.122 INFO:tasks.rados.rados.0.smithi203.stdout:1858: done (1 left) 2024-09-17T15:39:43.122 INFO:tasks.rados.rados.0.smithi203.stdout:1860: done (0 left) 2024-09-17T15:39:43.122 INFO:tasks.rados.rados.0.smithi203.stdout:1861: delete oid 19 current snap is 210 2024-09-17T15:39:43.133 INFO:tasks.rados.rados.0.smithi203.stdout:1861: done (0 left) 2024-09-17T15:39:43.133 INFO:tasks.rados.rados.0.smithi203.stdout:1862: read oid 31 snap -1 2024-09-17T15:39:43.133 INFO:tasks.rados.rados.0.smithi203.stdout:1862: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:39:43.133 INFO:tasks.rados.rados.0.smithi203.stdout:1863: rollback oid 10 current snap is 210 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 206 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1864: rollback oid 38 current snap is 210 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 201 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1865: read oid 27 snap -1 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1865: expect deleted 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1866: read oid 26 snap -1 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1866: expect (ObjNum 664 snap 209 seq_num 664) 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1867: copy_from oid 13 from oid 11 current snap is 210 2024-09-17T15:39:43.134 INFO:tasks.rados.rados.0.smithi203.stdout:1868: delete oid 14 current snap is 210 2024-09-17T15:39:43.141 INFO:tasks.rados.rados.0.smithi203.stdout:1865: done (6 left) 2024-09-17T15:39:43.141 INFO:tasks.rados.rados.0.smithi203.stdout:1868: done (5 left) 2024-09-17T15:39:43.141 INFO:tasks.rados.rados.0.smithi203.stdout:1869: snap_create 2024-09-17T15:39:43.144 INFO:tasks.rados.rados.0.smithi203.stdout:1862: expect (ObjNum 598 snap 179 seq_num 598) 2024-09-17T15:39:43.202 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:43.240 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 779934 2024-09-17T15:39:43.254 INFO:tasks.rados.rados.0.smithi203.stdout:1863: finishing rollback tid 0 to smithi20328086-10 2024-09-17T15:39:43.255 INFO:tasks.rados.rados.0.smithi203.stdout:1866: expect (ObjNum 664 snap 209 seq_num 664) 2024-09-17T15:39:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 722050 2024-09-17T15:39:43.368 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3008215 2024-09-17T15:39:43.443 INFO:tasks.rados.rados.0.smithi203.stdout:1863: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:39:43.443 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 681 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:1864: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:1864: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 683 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:1867: finishing copy_from racing read to smithi20328086-13 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:1867: finishing copy_from to smithi20328086-13 2024-09-17T15:39:43.444 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1862: done (5 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1863: done (4 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1864: done (3 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1866: done (2 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1867: done (1 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1869: done (0 left) 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1870: write (excl) oid 49 current snap is 211 2024-09-17T15:39:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:1870: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-09-17T15:39:44.144 INFO:tasks.rados.rados.0.smithi203.stdout:1870: writing smithi20328086-49 from 443883 to 908175 tid 1 2024-09-17T15:39:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:1870: writing smithi20328086-49 from 1365277 to 1876497 tid 2 2024-09-17T15:39:44.182 INFO:tasks.rados.rados.0.smithi203.stdout:1870: writing smithi20328086-49 from 2572335 to 3243883 tid 3 2024-09-17T15:39:44.182 INFO:tasks.rados.rados.0.smithi203.stdout:1871: snap_create 2024-09-17T15:39:44.182 INFO:tasks.rados.rados.0.smithi203.stdout:1870: finishing write tid 1 to smithi20328086-49 2024-09-17T15:39:44.182 INFO:tasks.rados.rados.0.smithi203.stdout:1870: finishing write tid 2 to smithi20328086-49 2024-09-17T15:39:44.192 INFO:tasks.rados.rados.0.smithi203.stdout:1870: finishing write tid 3 to smithi20328086-49 2024-09-17T15:39:44.193 INFO:tasks.rados.rados.0.smithi203.stdout:1870: finishing write tid 4 to smithi20328086-49 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: finishing write tid 5 to smithi20328086-49 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: oid 49 updating version 0 to 879 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: oid 49 updating version 879 to 881 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: oid 49 version 881 is already newer than 880 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: oid 49 version 881 is already newer than 878 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 881 (ObjNum 669 snap 211 seq_num 669) dirty exists 2024-09-17T15:39:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:1870: left oid 49 (ObjNum 669 snap 211 seq_num 669) 2024-09-17T15:39:44.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:45.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: get_monmap_and_config 2024-09-17T15:39:45.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: build_initial_monmap 2024-09-17T15:39:45.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: monmap: 2024-09-17T15:39:45.109 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:45.113105+0000 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:45.113105+0000 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:45.110 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 Processor -- start 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 -- start start 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: init 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35240689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3524068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f352413af10 0x7f352413d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-c con 0x7f352413af10 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352409d770 con 0x7f3524068db0 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352409ed00 con 0x7f35240689d0 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:45.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f352409efe0 con 0x7f352413af10 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352bcd5640 10 monclient(hunting): authenticate will time out at 1718.078659s 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f3529270640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3524068db0 0x7f35240691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42376/0 (socket says 172.21.15.86:42376) 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f3529270640 1 -- 172.21.15.86:0/369063328 learned_addr learned my addr 172.21.15.86:0/369063328 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 1 --2- 172.21.15.86:0/369063328 >> v2:172.21.15.203:3300/0 conn(0x7f352413af10 0x7f352413d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): get_auth_request con 0x7f352413af10 auth_method 0 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:45.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.109+0000 7f352a272640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f3529270640 1 -- 172.21.15.86:0/369063328 >> v1:172.21.15.86:6789/0 conn(0x7f3524068db0 legacy=0x7f35240691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f3529a71640 1 -- 172.21.15.86:0/369063328 >> v1:172.21.15.203:6789/0 conn(0x7f35240689d0 legacy=0x7f352409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 10 monclient(hunting): handle_auth_done global_id 134170 payload 274 2024-09-17T15:39:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 1 -- 172.21.15.86:0/369063328 >> v1:172.21.15.203:6789/0 conn(0x7f35240689d0 legacy=0x7f352409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 1 -- 172.21.15.86:0/369063328 >> v1:172.21.15.86:6789/0 conn(0x7f3524068db0 legacy=0x7f35240691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 10 monclient: _finish_hunting 0 2024-09-17T15:39:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 1 monclient: found mon.noname-c 2024-09-17T15:39:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:39:45.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.113+0000 7f352a272640 1 -- 172.21.15.86:0/369063328 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352409cde0 con 0x7f352413af10 2024-09-17T15:39:45.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352a272640 1 -- 172.21.15.86:0/369063328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f352413af10 msgr2=0x7f352413d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:45.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352a272640 1 --2- 172.21.15.86:0/369063328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f352413af10 0x7f352413d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f351c0098c0 tx=0x7f351c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c61e20ce2f824141 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:45.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 1 -- 172.21.15.86:0/369063328 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0b0070 con 0x7f352413af10 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: dump: 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: _finish_auth 0 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: _check_auth_tickets 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 1 -- 172.21.15.86:0/369063328 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351c0a97f0 con 0x7f352413af10 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 1 -- 172.21.15.86:0/369063328 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351c0a9bc0 con 0x7f352413af10 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352bcd5640 5 monclient: authenticate success, global_id 134170 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352bcd5640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352bcd5640 4 set_mon_vals no callback set 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:39:45.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f3528a6f640 10 monclient: dump: 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.117+0000 7f352bcd5640 10 monclient: shutdown 2024-09-17T15:39:45.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- 172.21.15.86:0/369063328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f352413af10 msgr2=0x7f352413d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 --2- 172.21.15.86:0/369063328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f352413af10 0x7f352413d300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f351c0098c0 tx=0x7f351c01c460 comp rx=0 tx=0).stop 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- 172.21.15.86:0/369063328 shutdown_connections 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 --2- 172.21.15.86:0/369063328 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f352413af10 0x7f352413d300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- 172.21.15.86:0/369063328 >> 172.21.15.86:0/369063328 conn(0x7f35240a6440 msgr2=0x7f35240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- 172.21.15.86:0/369063328 shutdown_connections 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- 172.21.15.86:0/369063328 wait complete. 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient: build_initial_monmap 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient: monmap: 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:45.124 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:45.125 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.125 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:45.125 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:45.125 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:45.125 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 Processor -- start 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- start start 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient: init 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:39:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-c con 0x7f35240689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 0x7f35240727a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-b con 0x7f3524068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): picked mon.noname-a con 0x7f352413af10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f352409be00 con 0x7f352413af10 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f352409bc80 con 0x7f3524068db0 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f352409bf80 con 0x7f35240689d0 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352bcd5640 10 monclient(hunting): authenticate will time out at 1718.089652s 2024-09-17T15:39:45.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37014/0 (socket says 172.21.15.86:37014) 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54138/0 (socket says 172.21.15.86:54138) 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 -- 172.21.15.86:0/612755943 learned_addr learned my addr 172.21.15.86:0/612755943 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): get_auth_request con 0x7f35240689d0 auth_method 0 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 10 monclient(hunting): get_auth_request con 0x7f352413af10 auth_method 0 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:45.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 msgr2=0x7f3524072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 msgr2=0x7f3524072ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f352a272640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529270640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 0x7f35240727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient(hunting): handle_auth_done global_id 104489 payload 824 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 msgr2=0x7f35240727a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 0x7f35240727a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 msgr2=0x7f3524072ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:45.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient: _finish_hunting 0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 monclient: found mon.noname-c 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529a71640 1 -- 172.21.15.86:0/612755943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352411d970 con 0x7f35240689d0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529270640 10 monclient: get_auth_request con 0x7f3524068db0 auth_method 0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.121+0000 7f3529270640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 0x7f35240727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f3529a71640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 msgr2=0x7f3524072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f3529a71640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f352000c7d0 tx=0x7f352000cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3520017070 con 0x7f35240689d0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:45.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 5 monclient: authenticate success, global_id 104489 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352000a840 con 0x7f35240689d0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _renew_subs 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f352409e580 con 0x7f35240689d0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352000abf0 con 0x7f35240689d0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 10 monclient: dump: 2024-09-17T15:39:45.135 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _renew_subs 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351bfff640 4 set_mon_vals no callback set 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:45.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35240743d0 con 0x7f35240689d0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f352409ffa0 con 0x7f35240689d0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f3529a71640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f352000c7d0 tx=0x7f352000cc90 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f352004f490 osd_map(424..424 src has 213..424) 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f352001a070 con 0x7f35240689d0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f351a7fc640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 0x7f350803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f3529270640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 0x7f350803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.125+0000 7f3529270640 10 monclient: get_auth_request con 0x7f350803bb60 auth_method 0 2024-09-17T15:39:45.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.129+0000 7f3529270640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 0x7f350803e020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3514005890 tx=0x7f3514009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:45.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.133+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(424..424 src has 213..424) ==== 4771+0+0 (secure 0 0 0) 0x7f352004f490 con 0x7f35240689d0 2024-09-17T15:39:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:1870: done (1 left) 2024-09-17T15:39:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:1871: done (0 left) 2024-09-17T15:39:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:1872: copy_from oid 48 from oid 20 current snap is 212 2024-09-17T15:39:45.139 INFO:tasks.rados.rados.0.smithi203.stdout:1873: copy_from oid 35 from oid 11 current snap is 212 2024-09-17T15:39:45.139 INFO:tasks.rados.rados.0.smithi203.stdout:1874: snap_create 2024-09-17T15:39:45.149 INFO:tasks.rados.rados.0.smithi203.stdout:1873: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:39:45.153 INFO:tasks.rados.rados.0.smithi203.stdout:1873: finishing copy_from to smithi20328086-35 2024-09-17T15:39:45.153 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:39:45.167 INFO:tasks.rados.rados.0.smithi203.stdout:1872: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:39:45.168 INFO:tasks.rados.rados.0.smithi203.stdout:1872: finishing copy_from to smithi20328086-48 2024-09-17T15:39:45.168 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 778 (ObjNum 658 snap 209 seq_num 658) dirty exists 2024-09-17T15:39:45.659 INFO:tasks.rados.rados.0.smithi203.stdout:1872: done (2 left) 2024-09-17T15:39:45.659 INFO:tasks.rados.rados.0.smithi203.stdout:1873: done (1 left) 2024-09-17T15:39:45.660 INFO:tasks.rados.rados.0.smithi203.stdout:1874: done (0 left) 2024-09-17T15:39:45.660 INFO:tasks.rados.rados.0.smithi203.stdout:1875: delete oid 17 current snap is 213 2024-09-17T15:39:45.669 INFO:tasks.rados.rados.0.smithi203.stdout:1875: done (0 left) 2024-09-17T15:39:45.669 INFO:tasks.rados.rados.0.smithi203.stdout:1876: delete oid 37 current snap is 213 2024-09-17T15:39:45.677 INFO:tasks.rados.rados.0.smithi203.stdout:1876: done (0 left) 2024-09-17T15:39:45.677 INFO:tasks.rados.rados.0.smithi203.stdout:1877: delete oid 38 current snap is 213 2024-09-17T15:39:45.690 INFO:tasks.rados.rados.0.smithi203.stdout:1877: done (0 left) 2024-09-17T15:39:45.690 INFO:tasks.rados.rados.0.smithi203.stdout:1878: write oid 23 current snap is 213 2024-09-17T15:39:45.690 INFO:tasks.rados.rados.0.smithi203.stdout:1878: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-17T15:39:45.706 INFO:tasks.rados.rados.0.smithi203.stdout:1878: writing smithi20328086-23 from 511111 to 992752 tid 1 2024-09-17T15:39:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:1878: writing smithi20328086-23 from 1599205 to 2117763 tid 2 2024-09-17T15:39:45.742 INFO:tasks.rados.rados.0.smithi203.stdout:1878: writing smithi20328086-23 from 2682027 to 3171974 tid 3 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1879: rollback oid 37 current snap is 213 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 209 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1878: finishing write tid 1 to smithi20328086-23 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1878: finishing write tid 2 to smithi20328086-23 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1880: read oid 33 snap -1 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1880: expect deleted 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1881: read oid 36 snap -1 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1881: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1882: read oid 5 snap -1 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1882: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1883: write oid 1 current snap is 213 2024-09-17T15:39:45.743 INFO:tasks.rados.rados.0.smithi203.stdout:1883: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-17T15:39:45.757 INFO:tasks.rados.rados.0.smithi203.stdout:1883: writing smithi20328086-1 from 527918 to 1084808 tid 1 2024-09-17T15:39:45.775 INFO:tasks.rados.rados.0.smithi203.stdout:1883: writing smithi20328086-1 from 1657687 to 2436256 tid 2 2024-09-17T15:39:45.785 INFO:tasks.rados.rados.0.smithi203.stdout:1883: writing smithi20328086-1 from 3225803 to 3327918 tid 3 2024-09-17T15:39:45.785 INFO:tasks.rados.rados.0.smithi203.stdout:1884: snap_remove snap 200 2024-09-17T15:39:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.953+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f352000ae10 con 0x7f35240689d0 2024-09-17T15:39:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.953+0000 7f351a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:45.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:45.953+0000 7f351a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:46.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f352bcd5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:46.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f352bcd5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:46.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f352bcd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:46.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f352409d410 con 0x7f35240689d0 2024-09-17T15:39:46.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f351a7fc640 1 -- 172.21.15.86:0/612755943 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f3520010dd0 con 0x7f35240689d0 2024-09-17T15:39:46.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f351a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:46.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.205+0000 7f351a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:46.207 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 msgr2=0x7f350803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 0x7f350803e020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3514005890 tx=0x7f3514009040 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 10 monclient: shutdown 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 msgr2=0x7f3524072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f352000c7d0 tx=0x7f352000cc90 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 shutdown_connections 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f350803bb60 0x7f350803e020 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f352413af10 0x7f3524072ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3524068db0 0x7f35240727a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 --2- 172.21.15.86:0/612755943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35240689d0 0x7f3524072260 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:46.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 >> 172.21.15.86:0/612755943 conn(0x7f35240a6440 msgr2=0x7f352413c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:46.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 shutdown_connections 2024-09-17T15:39:46.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:46.209+0000 7f352bcd5640 1 -- 172.21.15.86:0/612755943 wait complete. 2024-09-17T15:39:46.665 INFO:tasks.rados.rados.0.smithi203.stdout:1882: expect (ObjNum 570 snap 168 seq_num 570) 2024-09-17T15:39:46.791 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3383221 2024-09-17T15:39:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:1879: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: finishing write tid 3 to smithi20328086-23 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: finishing write tid 4 to smithi20328086-23 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: finishing write tid 5 to smithi20328086-23 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: oid 23 updating version 0 to 794 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: oid 23 updating version 794 to 795 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: oid 23 version 795 is already newer than 792 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: oid 23 version 795 is already newer than 793 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 795 (ObjNum 673 snap 213 seq_num 673) dirty exists 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1878: left oid 23 (ObjNum 673 snap 213 seq_num 673) 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1879: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 781 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:39:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:1881: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:39:46.794 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:46.861 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2287677 2024-09-17T15:39:46.870 INFO:tasks.rados.rados.0.smithi203.stdout:1883: finishing write tid 1 to smithi20328086-1 2024-09-17T15:39:46.870 INFO:tasks.rados.rados.0.smithi203.stdout:1883: finishing write tid 2 to smithi20328086-1 2024-09-17T15:39:46.870 INFO:tasks.rados.rados.0.smithi203.stdout:1883: finishing write tid 3 to smithi20328086-1 2024-09-17T15:39:46.870 INFO:tasks.rados.rados.0.smithi203.stdout:1883: finishing write tid 4 to smithi20328086-1 2024-09-17T15:39:46.870 INFO:tasks.rados.rados.0.smithi203.stdout:1883: finishing write tid 5 to smithi20328086-1 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: oid 1 updating version 0 to 800 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: oid 1 version 800 is already newer than 799 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: oid 1 version 800 is already newer than 798 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: oid 1 version 800 is already newer than 797 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 800 (ObjNum 674 snap 213 seq_num 674) dirty exists 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: left oid 1 (ObjNum 674 snap 213 seq_num 674) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1878: done (6 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1879: done (5 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1880: done (4 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1881: done (3 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1882: done (2 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1883: done (1 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1884: done (0 left) 2024-09-17T15:39:46.871 INFO:tasks.rados.rados.0.smithi203.stdout:1885: delete oid 17 current snap is 213 2024-09-17T15:39:46.874 INFO:tasks.rados.rados.0.smithi203.stdout:1885: done (0 left) 2024-09-17T15:39:46.874 INFO:tasks.rados.rados.0.smithi203.stdout:1886: write oid 45 current snap is 213 2024-09-17T15:39:46.874 INFO:tasks.rados.rados.0.smithi203.stdout:1886: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-17T15:39:46.890 INFO:tasks.rados.rados.0.smithi203.stdout:1886: writing smithi20328086-45 from 561532 to 1268920 tid 1 2024-09-17T15:39:46.909 INFO:tasks.rados.rados.0.smithi203.stdout:1886: writing smithi20328086-45 from 2058298 to 2840536 tid 2 2024-09-17T15:39:46.909 INFO:tasks.rados.rados.0.smithi203.stdout:1887: snap_remove snap 198 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1887: done (1 left) 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1888: copy_from oid 8 from oid 1 current snap is 213 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1886: finishing write tid 1 to smithi20328086-45 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1886: finishing write tid 2 to smithi20328086-45 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1886: finishing write tid 3 to smithi20328086-45 2024-09-17T15:39:47.669 INFO:tasks.rados.rados.0.smithi203.stdout:1886: finishing write tid 4 to smithi20328086-45 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1886: oid 45 updating version 0 to 368 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1886: oid 45 updating version 368 to 369 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1886: oid 45 version 369 is already newer than 367 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 369 (ObjNum 676 snap 213 seq_num 676) dirty exists 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1886: left oid 45 (ObjNum 676 snap 213 seq_num 676) 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1886: done (1 left) 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1889: copy_from oid 37 from oid 12 current snap is 213 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1890: write oid 49 current snap is 213 2024-09-17T15:39:47.670 INFO:tasks.rados.rados.0.smithi203.stdout:1890: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-17T15:39:47.686 INFO:tasks.rados.rados.0.smithi203.stdout:1890: writing smithi20328086-49 from 578339 to 1077329 tid 1 2024-09-17T15:39:47.708 INFO:tasks.rados.rados.0.smithi203.stdout:1890: writing smithi20328086-49 from 1833133 to 2359029 tid 2 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1890: writing smithi20328086-49 from 2791719 to 3378339 tid 3 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1891: copy_from oid 42 from oid 26 current snap is 213 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1888: finishing copy_from racing read to smithi20328086-8 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1890: finishing write tid 1 to smithi20328086-49 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1892: snap_create 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1889: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1888: finishing copy_from to smithi20328086-8 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 800 (ObjNum 674 snap 213 seq_num 674) dirty exists 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1889: finishing copy_from to smithi20328086-37 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 631 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:39:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:1890: finishing write tid 2 to smithi20328086-49 2024-09-17T15:39:47.729 INFO:tasks.rados.rados.0.smithi203.stdout:1890: finishing write tid 3 to smithi20328086-49 2024-09-17T15:39:47.729 INFO:tasks.rados.rados.0.smithi203.stdout:1890: finishing write tid 4 to smithi20328086-49 2024-09-17T15:39:47.730 INFO:tasks.rados.rados.0.smithi203.stdout:1890: finishing write tid 5 to smithi20328086-49 2024-09-17T15:39:47.730 INFO:tasks.rados.rados.0.smithi203.stdout:1890: oid 49 updating version 0 to 893 2024-09-17T15:39:47.730 INFO:tasks.rados.rados.0.smithi203.stdout:1890: oid 49 updating version 893 to 897 2024-09-17T15:39:47.730 INFO:tasks.rados.rados.0.smithi203.stdout:1890: oid 49 updating version 897 to 898 2024-09-17T15:39:47.730 INFO:tasks.rados.rados.0.smithi203.stdout:1890: oid 49 version 898 is already newer than 896 2024-09-17T15:39:47.731 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 898 (ObjNum 677 snap 213 seq_num 677) dirty exists 2024-09-17T15:39:47.731 INFO:tasks.rados.rados.0.smithi203.stdout:1890: left oid 49 (ObjNum 677 snap 213 seq_num 677) 2024-09-17T15:39:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:1891: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:39:47.755 INFO:tasks.rados.rados.0.smithi203.stdout:1891: finishing copy_from to smithi20328086-42 2024-09-17T15:39:47.755 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 870 (ObjNum 664 snap 209 seq_num 664) dirty exists 2024-09-17T15:39:48.672 INFO:tasks.rados.rados.0.smithi203.stdout:1888: done (4 left) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1889: done (3 left) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1890: done (2 left) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1891: done (1 left) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1892: done (0 left) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1893: read oid 1 snap -1 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1893: expect (ObjNum 674 snap 213 seq_num 674) 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1894: write (excl) oid 16 current snap is 214 2024-09-17T15:39:48.673 INFO:tasks.rados.rados.0.smithi203.stdout:1894: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-17T15:39:48.687 INFO:tasks.rados.rados.0.smithi203.stdout:1894: writing smithi20328086-16 from 595146 to 1169385 tid 1 2024-09-17T15:39:48.703 INFO:tasks.rados.rados.0.smithi203.stdout:1894: writing smithi20328086-16 from 1891615 to 2561169 tid 2 2024-09-17T15:39:48.713 INFO:tasks.rados.rados.0.smithi203.stdout:1894: writing smithi20328086-16 from 3219142 to 3395146 tid 3 2024-09-17T15:39:48.713 INFO:tasks.rados.rados.0.smithi203.stdout:1895: read oid 6 snap -1 2024-09-17T15:39:48.713 INFO:tasks.rados.rados.0.smithi203.stdout:1895: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:39:48.713 INFO:tasks.rados.rados.0.smithi203.stdout:1896: write oid 11 current snap is 214 2024-09-17T15:39:48.714 INFO:tasks.rados.rados.0.smithi203.stdout:1896: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-09-17T15:39:48.727 INFO:tasks.rados.rados.0.smithi203.stdout:1896: writing smithi20328086-11 from 611953 to 1261441 tid 1 2024-09-17T15:39:48.738 INFO:tasks.rados.rados.0.smithi203.stdout:1896: writing smithi20328086-11 from 1950097 to 2479662 tid 2 2024-09-17T15:39:48.745 INFO:tasks.rados.rados.0.smithi203.stdout:1896: writing smithi20328086-11 from 3246565 to 3411953 tid 3 2024-09-17T15:39:48.745 INFO:tasks.rados.rados.0.smithi203.stdout:1897: copy_from oid 20 from oid 17 current snap is 214 2024-09-17T15:39:48.745 INFO:tasks.rados.rados.0.smithi203.stdout:1893: expect (ObjNum 674 snap 213 seq_num 674) 2024-09-17T15:39:48.882 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1443094 2024-09-17T15:39:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:1894: finishing write tid 1 to smithi20328086-16 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:1894: finishing write tid 2 to smithi20328086-16 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:1893: done (4 left) 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:1898: snap_create 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:1895: expect (ObjNum 482 snap 145 seq_num 482) 2024-09-17T15:39:49.049 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:49.136 INFO:tasks.rados.rados.0.smithi203.stdout:1894: finishing write tid 3 to smithi20328086-16 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: finishing write tid 4 to smithi20328086-16 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: finishing write tid 5 to smithi20328086-16 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: oid 16 updating version 0 to 801 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: oid 16 version 801 is already newer than 799 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: oid 16 version 801 is already newer than 798 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: oid 16 version 801 is already newer than 800 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 801 (ObjNum 678 snap 214 seq_num 678) dirty exists 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1894: left oid 16 (ObjNum 678 snap 214 seq_num 678) 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1896: finishing write tid 1 to smithi20328086-11 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1896: finishing write tid 2 to smithi20328086-11 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1896: finishing write tid 3 to smithi20328086-11 2024-09-17T15:39:49.137 INFO:tasks.rados.rados.0.smithi203.stdout:1896: finishing write tid 4 to smithi20328086-11 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: finishing write tid 5 to smithi20328086-11 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: oid 11 updating version 0 to 734 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: oid 11 updating version 734 to 736 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: oid 11 version 736 is already newer than 735 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: oid 11 version 736 is already newer than 733 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 736 (ObjNum 679 snap 214 seq_num 679) dirty exists 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1896: left oid 11 (ObjNum 679 snap 214 seq_num 679) 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1897: finishing copy_from to smithi20328086-20 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1897: got expected ENOENT (src dne) 2024-09-17T15:39:49.138 INFO:tasks.rados.rados.0.smithi203.stdout:1897: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:39:49.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:49.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: get_monmap_and_config 2024-09-17T15:39:49.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: build_initial_monmap 2024-09-17T15:39:49.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: monmap: 2024-09-17T15:39:49.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:49.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:49.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:49.357656+0000 2024-09-17T15:39:49.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:49.357656+0000 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:49.356 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:49.357 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:49.357 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 Processor -- start 2024-09-17T15:39:49.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 -- start start 2024-09-17T15:39:49.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: init 2024-09-17T15:39:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6380689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-b con 0x7fd638068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 0x7fd63813c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-c con 0x7fd638139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63809d770 con 0x7fd6380689d0 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd63809ed00 con 0x7fd638139f00 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd63809efe0 con 0x7fd638068db0 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:49.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd63cba7640 10 monclient(hunting): authenticate will time out at 1722.323038s 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36274/0 (socket says 172.21.15.86:36274) 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 -- 172.21.15.86:0/3508194756 learned_addr learned my addr 172.21.15.86:0/3508194756 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): get_auth_request con 0x7fd638068db0 auth_method 0 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 1 --2- 172.21.15.86:0/3508194756 >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 0x7fd63813c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 10 monclient(hunting): get_auth_request con 0x7fd638139f00 auth_method 0 2024-09-17T15:39:49.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient(hunting): handle_auth_done global_id 104492 payload 274 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 -- 172.21.15.86:0/3508194756 >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 msgr2=0x7fd63813c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 --2- 172.21.15.86:0/3508194756 >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 0x7fd63813c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 -- 172.21.15.86:0/3508194756 >> v1:172.21.15.203:6789/0 conn(0x7fd6380689d0 legacy=0x7fd63809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 monclient: found mon.noname-b 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd635d9b640 1 -- 172.21.15.86:0/3508194756 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6380696d0 con 0x7fd638068db0 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.353+0000 7fd636d9d640 1 --2- 172.21.15.86:0/3508194756 >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 0x7fd63813c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.357+0000 7fd635d9b640 1 -- 172.21.15.86:0/3508194756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 msgr2=0x7fd638069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:49.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.357+0000 7fd635d9b640 1 --2- 172.21.15.86:0/3508194756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd620009870 tx=0x7fd62001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982c355f9c19b3bd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 1 -- 172.21.15.86:0/3508194756 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200b0070 con 0x7fd638068db0 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: dump: 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:49.372 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: _finish_auth 0 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: _check_auth_tickets 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 1 -- 172.21.15.86:0/3508194756 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6200ab600 con 0x7fd638068db0 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 1 -- 172.21.15.86:0/3508194756 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200ab9d0 con 0x7fd638068db0 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63559a640 10 monclient: dump: 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 5 monclient: authenticate success, global_id 104492 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 4 set_mon_vals no callback set 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: shutdown 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:49.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- 172.21.15.86:0/3508194756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 msgr2=0x7fd638069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- 172.21.15.86:0/3508194756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd620009870 tx=0x7fd62001c4b0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- 172.21.15.86:0/3508194756 shutdown_connections 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- 172.21.15.86:0/3508194756 >> v2:172.21.15.203:3300/0 conn(0x7fd638139f00 0x7fd63813c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- 172.21.15.86:0/3508194756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638069190 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- 172.21.15.86:0/3508194756 >> 172.21.15.86:0/3508194756 conn(0x7fd6380a6440 msgr2=0x7fd6380a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- 172.21.15.86:0/3508194756 shutdown_connections 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- 172.21.15.86:0/3508194756 wait complete. 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: build_initial_monmap 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: monmap: 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 Processor -- start 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- start start 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: init 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:39:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 0x7fd638073030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-b con 0x7fd6380689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-c con 0x7fd638068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): picked mon.noname-a con 0x7fd638139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd63809be00 con 0x7fd638139f00 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd63809bc80 con 0x7fd6380689d0 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd63809bf80 con 0x7fd638068db0 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63cba7640 10 monclient(hunting): authenticate will time out at 1722.330304s 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36278/0 (socket says 172.21.15.86:36278) 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 -- 172.21.15.86:0/4097151180 learned_addr learned my addr 172.21.15.86:0/4097151180 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): get_auth_request con 0x7fd638068db0 auth_method 0 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 10 monclient(hunting): get_auth_request con 0x7fd638139f00 auth_method 0 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 msgr2=0x7fd638073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 msgr2=0x7fd638073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd636d9d640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63659c640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 0x7fd638073030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient(hunting): handle_auth_done global_id 104495 payload 824 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 msgr2=0x7fd638073030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 0x7fd638073030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 msgr2=0x7fd638073ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 monclient: found mon.noname-c 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd635d9b640 1 -- 172.21.15.86:0/4097151180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63813f410 con 0x7fd638068db0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63659c640 10 monclient: get_auth_request con 0x7fd6380689d0 auth_method 0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.361+0000 7fd63659c640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 0x7fd638073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd635d9b640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 msgr2=0x7fd638073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd635d9b640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd62001ab60 tx=0x7fd62001acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200ba020 con 0x7fd638068db0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: dump: 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: _finish_auth 0 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6200abe30 con 0x7fd638068db0 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 5 monclient: authenticate success, global_id 104495 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _renew_subs 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6380a2f70 con 0x7fd638068db0 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6200b0040 con 0x7fd638068db0 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 10 monclient: dump: 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd634d99640 4 set_mon_vals no callback set 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _renew_subs 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd638120ee0 con 0x7fd638068db0 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd6200bd020 con 0x7fd638068db0 2024-09-17T15:39:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd638069190 con 0x7fd638068db0 2024-09-17T15:39:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 0x7fd61803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(429..429 src has 213..429) ==== 4787+0+0 (secure 0 0 0) 0x7fd6200ea450 con 0x7fd638068db0 2024-09-17T15:39:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63659c640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 0x7fd61803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.365+0000 7fd63659c640 10 monclient: get_auth_request con 0x7fd61803bb60 auth_method 0 2024-09-17T15:39:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.369+0000 7fd63659c640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 0x7fd61803e020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.369+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd6200c4030 con 0x7fd638068db0 2024-09-17T15:39:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.369+0000 7fd62effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.369+0000 7fd62effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:49.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:49.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:49.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:49.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd63813fa90 con 0x7fd638068db0 2024-09-17T15:39:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd62effd640 1 -- 172.21.15.86:0/4097151180 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd620016130 con 0x7fd638068db0 2024-09-17T15:39:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd62effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd62effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:49.606 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":29,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 msgr2=0x7fd61803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 0x7fd61803e020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628009040 comp rx=0 tx=0).stop 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 10 monclient: shutdown 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 msgr2=0x7fd638073570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd62001ab60 tx=0x7fd62001acf0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 shutdown_connections 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd61803bb60 0x7fd61803e020 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd638139f00 0x7fd638073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd638068db0 0x7fd638073570 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 --2- 172.21.15.86:0/4097151180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6380689d0 0x7fd638073030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.605+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 >> 172.21.15.86:0/4097151180 conn(0x7fd6380a6440 msgr2=0x7fd63813b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.609+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 shutdown_connections 2024-09-17T15:39:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:49.609+0000 7fd63cba7640 1 -- 172.21.15.86:0/4097151180 wait complete. 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1894: done (4 left) 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1895: done (3 left) 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1896: done (2 left) 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1897: done (1 left) 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1898: done (0 left) 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1899: read oid 45 snap -1 2024-09-17T15:39:49.692 INFO:tasks.rados.rados.0.smithi203.stdout:1899: expect (ObjNum 676 snap 213 seq_num 676) 2024-09-17T15:39:49.693 INFO:tasks.rados.rados.0.smithi203.stdout:1900: read oid 13 snap -1 2024-09-17T15:39:49.693 INFO:tasks.rados.rados.0.smithi203.stdout:1900: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:39:49.693 INFO:tasks.rados.rados.0.smithi203.stdout:1901: write (excl) oid 4 current snap is 215 2024-09-17T15:39:49.693 INFO:tasks.rados.rados.0.smithi203.stdout:1901: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-17T15:39:49.705 INFO:tasks.rados.rados.0.smithi203.stdout:1901: writing smithi20328086-4 from 628760 to 1353497 tid 1 2024-09-17T15:39:49.718 INFO:tasks.rados.rados.0.smithi203.stdout:1901: writing smithi20328086-4 from 1892226 to 2565449 tid 2 2024-09-17T15:39:49.727 INFO:tasks.rados.rados.0.smithi203.stdout:1901: writing smithi20328086-4 from 3157635 to 3428760 tid 3 2024-09-17T15:39:49.727 INFO:tasks.rados.rados.0.smithi203.stdout:1902: rollback oid 48 current snap is 215 2024-09-17T15:39:49.727 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 209 2024-09-17T15:39:49.727 INFO:tasks.rados.rados.0.smithi203.stdout:1899: expect (ObjNum 676 snap 213 seq_num 676) 2024-09-17T15:39:49.727 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:49.801 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 536593 2024-09-17T15:39:49.834 INFO:tasks.rados.rados.0.smithi203.stdout:1900: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:39:49.965 INFO:tasks.rados.rados.0.smithi203.stdout:1901: finishing write tid 1 to smithi20328086-4 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: finishing write tid 2 to smithi20328086-4 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1899: done (3 left) 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1900: done (2 left) 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1903: write oid 28 current snap is 215 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: finishing write tid 3 to smithi20328086-4 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: finishing write tid 4 to smithi20328086-4 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: finishing write tid 5 to smithi20328086-4 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: oid 4 updating version 0 to 372 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: oid 4 version 372 is already newer than 371 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: oid 4 version 372 is already newer than 370 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: oid 4 updating version 372 to 373 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 373 (ObjNum 680 snap 215 seq_num 680) dirty exists 2024-09-17T15:39:49.966 INFO:tasks.rados.rados.0.smithi203.stdout:1901: left oid 4 (ObjNum 680 snap 215 seq_num 680) 2024-09-17T15:39:49.967 INFO:tasks.rados.rados.0.smithi203.stdout:1902: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:39:49.967 INFO:tasks.rados.rados.0.smithi203.stdout:1902: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:39:49.967 INFO:tasks.rados.rados.0.smithi203.stdout:1902: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:39:49.967 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 805 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-17T15:39:49.967 INFO:tasks.rados.rados.0.smithi203.stdout:1903: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-17T15:39:49.973 INFO:tasks.rados.rados.0.smithi203.stdout:1903: writing smithi20328086-28 from 645567 to 1445553 tid 1 2024-09-17T15:39:49.979 INFO:tasks.rados.rados.0.smithi203.stdout:1903: writing smithi20328086-28 from 1950708 to 2483942 tid 2 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1903: writing smithi20328086-28 from 3185058 to 3445567 tid 3 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1901: done (2 left) 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1902: done (1 left) 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1904: rollback oid 36 current snap is 215 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 188 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1903: finishing write tid 1 to smithi20328086-28 2024-09-17T15:39:49.986 INFO:tasks.rados.rados.0.smithi203.stdout:1905: snap_create 2024-09-17T15:39:49.990 INFO:tasks.rados.rados.0.smithi203.stdout:1903: finishing write tid 2 to smithi20328086-28 2024-09-17T15:39:49.997 INFO:tasks.rados.rados.0.smithi203.stdout:1903: finishing write tid 3 to smithi20328086-28 2024-09-17T15:39:49.998 INFO:tasks.rados.rados.0.smithi203.stdout:1903: finishing write tid 4 to smithi20328086-28 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: finishing write tid 5 to smithi20328086-28 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: oid 28 updating version 0 to 902 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: oid 28 version 902 is already newer than 901 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: oid 28 version 902 is already newer than 900 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: oid 28 version 902 is already newer than 899 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 902 (ObjNum 681 snap 215 seq_num 681) dirty exists 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1903: left oid 28 (ObjNum 681 snap 215 seq_num 681) 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1904: finishing rollback tid 0 to smithi20328086-36 2024-09-17T15:39:49.999 INFO:tasks.rados.rados.0.smithi203.stdout:1904: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:39:50.001 INFO:tasks.rados.rados.0.smithi203.stdout:1904: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:39:50.001 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 906 (ObjNum 513 snap 153 seq_num 513) dirty exists 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1903: done (2 left) 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1904: done (1 left) 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1905: done (0 left) 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1906: read oid 44 snap -1 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1906: expect (ObjNum 610 snap 184 seq_num 610) 2024-09-17T15:39:50.660 INFO:tasks.rados.rados.0.smithi203.stdout:1907: read oid 2 snap -1 2024-09-17T15:39:50.661 INFO:tasks.rados.rados.0.smithi203.stdout:1907: expect (ObjNum 542 snap 161 seq_num 542) 2024-09-17T15:39:50.661 INFO:tasks.rados.rados.0.smithi203.stdout:1908: rollback oid 25 current snap is 216 2024-09-17T15:39:50.664 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 205 2024-09-17T15:39:50.664 INFO:tasks.rados.rados.0.smithi203.stdout:1909: snap_create 2024-09-17T15:39:50.671 INFO:tasks.rados.rados.0.smithi203.stdout:1906: expect (ObjNum 610 snap 184 seq_num 610) 2024-09-17T15:39:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1019942 2024-09-17T15:39:50.727 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:50.821 INFO:tasks.rados.rados.0.smithi203.stdout:1907: expect (ObjNum 542 snap 161 seq_num 542) 2024-09-17T15:39:50.989 INFO:tasks.rados.rados.0.smithi203.stdout:1908: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:39:50.989 INFO:tasks.rados.rados.0.smithi203.stdout:1908: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:39:50.989 INFO:tasks.rados.rados.0.smithi203.stdout:1908: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:39:50.989 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 657 (ObjNum 635 snap 198 seq_num 635) dirty exists 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1906: done (3 left) 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1907: done (2 left) 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1908: done (1 left) 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1909: done (0 left) 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1910: read oid 45 snap -1 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1910: expect (ObjNum 676 snap 213 seq_num 676) 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1911: write oid 2 current snap is 217 2024-09-17T15:39:51.661 INFO:tasks.rados.rados.0.smithi203.stdout:1911: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-17T15:39:51.679 INFO:tasks.rados.rados.0.smithi203.stdout:1911: writing smithi20328086-2 from 662374 to 1137609 tid 1 2024-09-17T15:39:51.698 INFO:tasks.rados.rados.0.smithi203.stdout:1911: writing smithi20328086-2 from 1609190 to 2286082 tid 2 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1911: writing smithi20328086-2 from 2812481 to 3436582 tid 3 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1912: read oid 33 snap -1 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1912: expect deleted 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1913: read oid 18 snap 205 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1913: expect deleted 2024-09-17T15:39:51.716 INFO:tasks.rados.rados.0.smithi203.stdout:1914: snap_create 2024-09-17T15:39:51.717 INFO:tasks.rados.rados.0.smithi203.stdout:1910: expect (ObjNum 676 snap 213 seq_num 676) 2024-09-17T15:39:51.771 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1048699 2024-09-17T15:39:51.811 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2157445 2024-09-17T15:39:51.823 INFO:tasks.rados.rados.0.smithi203.stdout:1911: finishing write tid 1 to smithi20328086-2 2024-09-17T15:39:51.823 INFO:tasks.rados.rados.0.smithi203.stdout:1911: finishing write tid 2 to smithi20328086-2 2024-09-17T15:39:51.823 INFO:tasks.rados.rados.0.smithi203.stdout:1911: finishing write tid 3 to smithi20328086-2 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: finishing write tid 4 to smithi20328086-2 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: finishing write tid 5 to smithi20328086-2 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: oid 2 updating version 0 to 479 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: oid 2 version 479 is already newer than 476 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: oid 2 version 479 is already newer than 478 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: oid 2 version 479 is already newer than 477 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 479 (ObjNum 682 snap 217 seq_num 682) dirty exists 2024-09-17T15:39:51.824 INFO:tasks.rados.rados.0.smithi203.stdout:1911: left oid 2 (ObjNum 682 snap 217 seq_num 682) 2024-09-17T15:39:52.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:52.668 INFO:tasks.rados.rados.0.smithi203.stdout:1910: done (4 left) 2024-09-17T15:39:52.668 INFO:tasks.rados.rados.0.smithi203.stdout:1911: done (3 left) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1912: done (2 left) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1913: done (1 left) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1914: done (0 left) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1915: read oid 48 snap -1 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1915: expect (ObjNum 622 snap 194 seq_num 622) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1916: read oid 38 snap 177 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1916: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:39:52.669 INFO:tasks.rados.rados.0.smithi203.stdout:1917: delete oid 50 current snap is 218 2024-09-17T15:39:52.672 INFO:tasks.rados.rados.0.smithi203.stdout:1917: done (2 left) 2024-09-17T15:39:52.672 INFO:tasks.rados.rados.0.smithi203.stdout:1918: write oid 34 current snap is 218 2024-09-17T15:39:52.672 INFO:tasks.rados.rados.0.smithi203.stdout:1918: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-17T15:39:52.688 INFO:tasks.rados.rados.0.smithi203.stdout:1918: writing smithi20328086-34 from 695988 to 1321721 tid 1 2024-09-17T15:39:52.705 INFO:tasks.rados.rados.0.smithi203.stdout:1918: writing smithi20328086-34 from 2009801 to 2690362 tid 2 2024-09-17T15:39:52.715 INFO:tasks.rados.rados.0.smithi203.stdout:1918: writing smithi20328086-34 from 3150974 to 3495988 tid 3 2024-09-17T15:39:52.715 INFO:tasks.rados.rados.0.smithi203.stdout:1919: write (excl) oid 40 current snap is 218 2024-09-17T15:39:52.715 INFO:tasks.rados.rados.0.smithi203.stdout:1919: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-17T15:39:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:1919: writing smithi20328086-40 from 712795 to 1130130 tid 1 2024-09-17T15:39:52.728 INFO:tasks.rados.rados.0.smithi203.stdout:1919: writing smithi20328086-40 from 1784636 to 2325208 tid 2 2024-09-17T15:39:52.734 INFO:tasks.rados.rados.0.smithi203.stdout:1919: writing smithi20328086-40 from 2894750 to 3352667 tid 3 2024-09-17T15:39:52.735 INFO:tasks.rados.rados.0.smithi203.stdout:1920: copy_from oid 32 from oid 9 current snap is 218 2024-09-17T15:39:52.735 INFO:tasks.rados.rados.0.smithi203.stdout:1916: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: get_monmap_and_config 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: build_initial_monmap 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: monmap: 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:52.765967+0000 2024-09-17T15:39:52.762 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:52.765967+0000 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:52.763 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 Processor -- start 2024-09-17T15:39:52.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 -- start start 2024-09-17T15:39:52.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: init 2024-09-17T15:39:52.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:52.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5680a5070 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-b con 0x7fd5680a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 0x7fd56813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-c con 0x7fd56813d280 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5680a6f40 con 0x7fd5680a5070 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd5680a65e0 con 0x7fd56813d280 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd56809c2a0 con 0x7fd5680a5b60 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd56e842640 10 monclient(hunting): authenticate will time out at 1725.731602s 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36288/0 (socket says 172.21.15.86:36288) 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 1 -- 172.21.15.86:0/3427955304 learned_addr learned my addr 172.21.15.86:0/3427955304 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): get_auth_request con 0x7fd5680a5b60 auth_method 0 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.761+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd56cddf640 1 --2- 172.21.15.86:0/3427955304 >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 0x7fd56813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 10 monclient(hunting): handle_auth_done global_id 104498 payload 274 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 1 -- 172.21.15.86:0/3427955304 >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 msgr2=0x7fd56813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 1 --2- 172.21.15.86:0/3427955304 >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 0x7fd56813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 1 -- 172.21.15.86:0/3427955304 >> v1:172.21.15.203:6789/0 conn(0x7fd5680a5070 legacy=0x7fd5680a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:39:52.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 1 monclient: found mon.noname-b 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd5677fe640 1 -- 172.21.15.86:0/3427955304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56809c580 con 0x7fd5680a5b60 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd56cddf640 10 monclient: get_auth_request con 0x7fd56813d280 auth_method 0 2024-09-17T15:39:52.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.765+0000 7fd56cddf640 1 --2- 172.21.15.86:0/3427955304 >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 0x7fd56813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 -- 172.21.15.86:0/3427955304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 msgr2=0x7fd56809cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 --2- 172.21.15.86:0/3427955304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009870 tx=0x7fd55c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c94bfd5c12a02c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 1 -- 172.21.15.86:0/3427955304 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0b0070 con 0x7fd5680a5b60 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: dump: 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: _finish_auth 0 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 1 -- 172.21.15.86:0/3427955304 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c0ab600 con 0x7fd5680a5b60 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 5 monclient: authenticate success, global_id 104498 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 1 -- 172.21.15.86:0/3427955304 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0ab9d0 con 0x7fd5680a5b60 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:52.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 4 set_mon_vals no callback set 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd566ffd640 10 monclient: dump: 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: shutdown 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- 172.21.15.86:0/3427955304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 msgr2=0x7fd56809cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- 172.21.15.86:0/3427955304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd55c009870 tx=0x7fd55c01c420 comp rx=0 tx=0).stop 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- 172.21.15.86:0/3427955304 shutdown_connections 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- 172.21.15.86:0/3427955304 >> v2:172.21.15.203:3300/0 conn(0x7fd56813d280 0x7fd56813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- 172.21.15.86:0/3427955304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd56809cf60 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- 172.21.15.86:0/3427955304 >> 172.21.15.86:0/3427955304 conn(0x7fd5680742a0 msgr2=0x7fd5680746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- 172.21.15.86:0/3427955304 shutdown_connections 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- 172.21.15.86:0/3427955304 wait complete. 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: build_initial_monmap 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: monmap: 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:52.773 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 Processor -- start 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- start start 2024-09-17T15:39:52.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: init 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5680a5070 0x7fd5681cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-b con 0x7fd5680a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-c con 0x7fd5680a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): picked mon.noname-a con 0x7fd56813d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd568142760 con 0x7fd56813d280 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd5681425e0 con 0x7fd5680a5070 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd5681428e0 con 0x7fd5680a5b60 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56e842640 10 monclient(hunting): authenticate will time out at 1725.738508s 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36296/0 (socket says 172.21.15.86:36296) 2024-09-17T15:39:52.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 -- 172.21.15.86:0/3612662580 learned_addr learned my addr 172.21.15.86:0/3612662580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): get_auth_request con 0x7fd5680a5b60 auth_method 0 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 10 monclient(hunting): get_auth_request con 0x7fd56813d280 auth_method 0 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 msgr2=0x7fd5680a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:39:52.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 msgr2=0x7fd5680a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd567fff640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5680a5070 0x7fd5681cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd56cddf640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd567fff640 10 monclient(hunting): get_auth_request con 0x7fd5680a5070 auth_method 0 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd567fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient(hunting): handle_auth_done global_id 104501 payload 824 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5680a5070 msgr2=0x7fd5681cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5680a5070 0x7fd5681cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 msgr2=0x7fd5680a2c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:39:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 monclient: found mon.noname-c 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.769+0000 7fd5677fe640 1 -- 172.21.15.86:0/3612662580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd568121630 con 0x7fd5680a5b60 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd5677fe640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 msgr2=0x7fd5681380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd5677fe640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0ae040 tx=0x7fd55c0191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0b0040 con 0x7fd5680a5b60 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:39:52.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: dump: 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: _finish_auth 0 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55c0aa5b0 con 0x7fd5680a5b60 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 5 monclient: authenticate success, global_id 104501 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _renew_subs 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd568122a60 con 0x7fd5680a5b60 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55c0aa9a0 con 0x7fd5680a5b60 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 10 monclient: dump: 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _renew_subs 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd5667fc640 4 set_mon_vals no callback set 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd568122780 con 0x7fd5680a5b60 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:52.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd56809cf60 con 0x7fd5680a5b60 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd55c0aab40 con 0x7fd5680a5b60 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 0x7fd55403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(433..433 src has 213..433) ==== 4771+0+0 (secure 0 0 0) 0x7fd55c0e9ee0 con 0x7fd5680a5b60 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd567fff640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 0x7fd55403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.773+0000 7fd567fff640 10 monclient: get_auth_request con 0x7fd55403bb90 auth_method 0 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.777+0000 7fd567fff640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 0x7fd55403e050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd5680a1120 tx=0x7fd558009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.777+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd55c016130 con 0x7fd5680a5b60 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.777+0000 7fd564ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:52.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:52.777+0000 7fd564ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:39:52.815 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2210369 2024-09-17T15:39:52.822 INFO:tasks.rados.rados.0.smithi203.stdout:1915: expect (ObjNum 622 snap 194 seq_num 622) 2024-09-17T15:39:52.869 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:52.915 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:52.961 INFO:tasks.rados.rados.0.smithi203.stdout:1918: finishing write tid 1 to smithi20328086-34 2024-09-17T15:39:52.961 INFO:tasks.rados.rados.0.smithi203.stdout:1918: finishing write tid 2 to smithi20328086-34 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: finishing write tid 3 to smithi20328086-34 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: finishing write tid 4 to smithi20328086-34 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: finishing write tid 5 to smithi20328086-34 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: oid 34 updating version 0 to 620 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: oid 34 version 620 is already newer than 618 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: oid 34 version 620 is already newer than 617 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: oid 34 version 620 is already newer than 619 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 620 (ObjNum 684 snap 218 seq_num 684) dirty exists 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1918: left oid 34 (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1919: finishing write tid 1 to smithi20328086-40 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1919: finishing write tid 2 to smithi20328086-40 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1919: finishing write tid 3 to smithi20328086-40 2024-09-17T15:39:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:1919: finishing write tid 4 to smithi20328086-40 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: finishing write tid 5 to smithi20328086-40 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: oid 40 updating version 0 to 741 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: oid 40 version 741 is already newer than 739 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: oid 40 version 741 is already newer than 738 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: oid 40 version 741 is already newer than 740 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 741 (ObjNum 685 snap 218 seq_num 685) dirty exists 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: left oid 40 (ObjNum 685 snap 218 seq_num 685) 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1920: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1920: finishing copy_from to smithi20328086-32 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 641 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1915: done (4 left) 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1916: done (3 left) 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1918: done (2 left) 2024-09-17T15:39:52.963 INFO:tasks.rados.rados.0.smithi203.stdout:1919: done (1 left) 2024-09-17T15:39:52.964 INFO:tasks.rados.rados.0.smithi203.stdout:1920: done (0 left) 2024-09-17T15:39:52.964 INFO:tasks.rados.rados.0.smithi203.stdout:1921: snap_remove snap 211 2024-09-17T15:39:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd56e842640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:39:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd56e842640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd56e842640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:39:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd568069920 con 0x7fd5680a5b60 2024-09-17T15:39:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd564ff9640 1 -- 172.21.15.86:0/3612662580 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd55c0eb030 con 0x7fd5680a5b60 2024-09-17T15:39:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd564ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:39:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.013+0000 7fd564ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:39:53.015 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":118,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":32,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 msgr2=0x7fd55403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 0x7fd55403e050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd5680a1120 tx=0x7fd558009040 comp rx=0 tx=0).stop 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 10 monclient: shutdown 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 msgr2=0x7fd5681380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:53.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0ae040 tx=0x7fd55c0191c0 comp rx=0 tx=0).stop 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 shutdown_connections 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd55403bb90 0x7fd55403e050 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd56813d280 0x7fd5680a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd5680a5b60 0x7fd5681380d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 --2- 172.21.15.86:0/3612662580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5680a5070 0x7fd5681cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 >> 172.21.15.86:0/3612662580 conn(0x7fd5680742a0 msgr2=0x7fd56813f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 shutdown_connections 2024-09-17T15:39:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:53.017+0000 7fd56e842640 1 -- 172.21.15.86:0/3612662580 wait complete. 2024-09-17T15:39:53.677 INFO:tasks.rados.rados.0.smithi203.stdout:1921: done (0 left) 2024-09-17T15:39:53.677 INFO:tasks.rados.rados.0.smithi203.stdout:1922: read oid 20 snap 172 2024-09-17T15:39:53.677 INFO:tasks.rados.rados.0.smithi203.stdout:1922: expect (ObjNum 479 snap 145 seq_num 479) 2024-09-17T15:39:53.677 INFO:tasks.rados.rados.0.smithi203.stdout:1923: delete oid 25 current snap is 218 2024-09-17T15:39:53.695 INFO:tasks.rados.rados.0.smithi203.stdout:1923: done (1 left) 2024-09-17T15:39:53.695 INFO:tasks.rados.rados.0.smithi203.stdout:1924: write (excl) oid 9 current snap is 218 2024-09-17T15:39:53.695 INFO:tasks.rados.rados.0.smithi203.stdout:1924: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-09-17T15:39:53.712 INFO:tasks.rados.rados.0.smithi203.stdout:1924: writing smithi20328086-9 from 746409 to 1314242 tid 1 2024-09-17T15:39:53.725 INFO:tasks.rados.rados.0.smithi203.stdout:1924: writing smithi20328086-9 from 1901600 to 2445841 tid 2 2024-09-17T15:39:53.739 INFO:tasks.rados.rados.0.smithi203.stdout:1924: writing smithi20328086-9 from 3065949 to 3546409 tid 3 2024-09-17T15:39:53.739 INFO:tasks.rados.rados.0.smithi203.stdout:1925: write (excl) oid 27 current snap is 218 2024-09-17T15:39:53.739 INFO:tasks.rados.rados.0.smithi203.stdout:1925: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-17T15:39:53.753 INFO:tasks.rados.rados.0.smithi203.stdout:1925: writing smithi20328086-27 from 763216 to 1406298 tid 1 2024-09-17T15:39:53.763 INFO:tasks.rados.rados.0.smithi203.stdout:1925: writing smithi20328086-27 from 1843729 to 2531628 tid 2 2024-09-17T15:39:53.772 INFO:tasks.rados.rados.0.smithi203.stdout:1925: writing smithi20328086-27 from 3260666 to 3563216 tid 3 2024-09-17T15:39:53.772 INFO:tasks.rados.rados.0.smithi203.stdout:1926: copy_from oid 17 from oid 39 current snap is 218 2024-09-17T15:39:53.772 INFO:tasks.rados.rados.0.smithi203.stdout:1922: expect (ObjNum 479 snap 145 seq_num 479) 2024-09-17T15:39:53.774 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1380824 2024-09-17T15:39:53.788 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 326225 2024-09-17T15:39:53.807 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:53.829 INFO:tasks.rados.rados.0.smithi203.stdout:1924: finishing write tid 1 to smithi20328086-9 2024-09-17T15:39:53.829 INFO:tasks.rados.rados.0.smithi203.stdout:1924: finishing write tid 2 to smithi20328086-9 2024-09-17T15:39:53.829 INFO:tasks.rados.rados.0.smithi203.stdout:1924: finishing write tid 3 to smithi20328086-9 2024-09-17T15:39:53.829 INFO:tasks.rados.rados.0.smithi203.stdout:1924: finishing write tid 4 to smithi20328086-9 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: finishing write tid 5 to smithi20328086-9 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: oid 9 updating version 0 to 666 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: oid 9 version 666 is already newer than 663 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: oid 9 version 666 is already newer than 664 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: oid 9 version 666 is already newer than 665 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 666 (ObjNum 687 snap 218 seq_num 687) dirty exists 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1924: left oid 9 (ObjNum 687 snap 218 seq_num 687) 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1925: finishing write tid 1 to smithi20328086-27 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1925: finishing write tid 2 to smithi20328086-27 2024-09-17T15:39:53.830 INFO:tasks.rados.rados.0.smithi203.stdout:1925: finishing write tid 3 to smithi20328086-27 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: finishing write tid 4 to smithi20328086-27 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: finishing write tid 5 to smithi20328086-27 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: oid 27 updating version 0 to 873 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: oid 27 version 873 is already newer than 871 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: oid 27 version 873 is already newer than 872 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: oid 27 updating version 873 to 874 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 874 (ObjNum 688 snap 218 seq_num 688) dirty exists 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: left oid 27 (ObjNum 688 snap 218 seq_num 688) 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1926: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1926: finishing copy_from to smithi20328086-17 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 759 (ObjNum 665 snap 209 seq_num 665) dirty exists 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1922: done (3 left) 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1924: done (2 left) 2024-09-17T15:39:53.831 INFO:tasks.rados.rados.0.smithi203.stdout:1925: done (1 left) 2024-09-17T15:39:53.832 INFO:tasks.rados.rados.0.smithi203.stdout:1926: done (0 left) 2024-09-17T15:39:53.832 INFO:tasks.rados.rados.0.smithi203.stdout:1927: read oid 22 snap -1 2024-09-17T15:39:53.832 INFO:tasks.rados.rados.0.smithi203.stdout:1927: expect deleted 2024-09-17T15:39:53.832 INFO:tasks.rados.rados.0.smithi203.stdout:1928: write oid 26 current snap is 218 2024-09-17T15:39:53.832 INFO:tasks.rados.rados.0.smithi203.stdout:1928: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-17T15:39:53.844 INFO:tasks.rados.rados.0.smithi203.stdout:1928: writing smithi20328086-26 from 780023 to 1498354 tid 1 2024-09-17T15:39:53.854 INFO:tasks.rados.rados.0.smithi203.stdout:1928: writing smithi20328086-26 from 1902211 to 2450121 tid 2 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1928: writing smithi20328086-26 from 3004442 to 3580023 tid 3 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1929: read oid 32 snap -1 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1929: expect (ObjNum 622 snap 194 seq_num 622) 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1930: read oid 9 snap -1 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1930: expect (ObjNum 687 snap 218 seq_num 687) 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1931: copy_from oid 25 from oid 7 current snap is 218 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1928: finishing write tid 1 to smithi20328086-26 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1928: finishing write tid 2 to smithi20328086-26 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1927: done (4 left) 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1932: copy_from oid 29 from oid 39 current snap is 218 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1933: write oid 10 current snap is 218 2024-09-17T15:39:53.866 INFO:tasks.rados.rados.0.smithi203.stdout:1933: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-17T15:39:53.881 INFO:tasks.rados.rados.0.smithi203.stdout:1933: writing smithi20328086-10 from 796830 to 1590410 tid 1 2024-09-17T15:39:53.893 INFO:tasks.rados.rados.0.smithi203.stdout:1933: writing smithi20328086-10 from 2360693 to 3052261 tid 2 2024-09-17T15:39:53.893 INFO:tasks.rados.rados.0.smithi203.stdout:1934: read oid 50 snap -1 2024-09-17T15:39:53.893 INFO:tasks.rados.rados.0.smithi203.stdout:1934: expect deleted 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1935: write (excl) oid 31 current snap is 218 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: finishing write tid 3 to smithi20328086-26 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: finishing write tid 4 to smithi20328086-26 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: finishing write tid 5 to smithi20328086-26 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: oid 26 updating version 0 to 912 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: oid 26 updating version 912 to 914 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: oid 26 version 914 is already newer than 913 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: oid 26 updating version 914 to 915 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 915 (ObjNum 689 snap 218 seq_num 689) dirty exists 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1928: left oid 26 (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:39:53.894 INFO:tasks.rados.rados.0.smithi203.stdout:1935: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-17T15:39:53.899 INFO:tasks.rados.rados.0.smithi203.stdout:1935: writing smithi20328086-31 from 413637 to 882466 tid 1 2024-09-17T15:39:53.908 INFO:tasks.rados.rados.0.smithi203.stdout:1935: writing smithi20328086-31 from 1619175 to 2054401 tid 2 2024-09-17T15:39:53.916 INFO:tasks.rados.rados.0.smithi203.stdout:1935: writing smithi20328086-31 from 2542935 to 3301190 tid 3 2024-09-17T15:39:53.917 INFO:tasks.rados.rados.0.smithi203.stdout:1928: done (7 left) 2024-09-17T15:39:53.917 INFO:tasks.rados.rados.0.smithi203.stdout:1936: write (excl) oid 28 current snap is 218 2024-09-17T15:39:53.917 INFO:tasks.rados.rados.0.smithi203.stdout:1936: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-17T15:39:53.924 INFO:tasks.rados.rados.0.smithi203.stdout:1936: writing smithi20328086-28 from 430444 to 1090875 tid 1 2024-09-17T15:39:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:1936: writing smithi20328086-28 from 1677657 to 2372894 tid 2 2024-09-17T15:39:53.940 INFO:tasks.rados.rados.0.smithi203.stdout:1936: writing smithi20328086-28 from 2970358 to 3539885 tid 3 2024-09-17T15:39:53.940 INFO:tasks.rados.rados.0.smithi203.stdout:1929: expect (ObjNum 622 snap 194 seq_num 622) 2024-09-17T15:39:53.940 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:54.013 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1626391 2024-09-17T15:39:54.073 INFO:tasks.rados.rados.0.smithi203.stdout:1930: expect (ObjNum 687 snap 218 seq_num 687) 2024-09-17T15:39:54.218 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1931: finishing copy_from to smithi20328086-25 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1931: got expected ENOENT (src dne) 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1931: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1933: finishing write tid 1 to smithi20328086-10 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1937: read oid 34 snap -1 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1937: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:39:54.290 INFO:tasks.rados.rados.0.smithi203.stdout:1929: done (8 left) 2024-09-17T15:39:54.291 INFO:tasks.rados.rados.0.smithi203.stdout:1930: done (7 left) 2024-09-17T15:39:54.291 INFO:tasks.rados.rados.0.smithi203.stdout:1931: done (6 left) 2024-09-17T15:39:54.291 INFO:tasks.rados.rados.0.smithi203.stdout:1938: snap_remove snap 204 2024-09-17T15:39:56.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:39:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 10 monclient: get_monmap_and_config 2024-09-17T15:39:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 10 monclient: build_initial_monmap 2024-09-17T15:39:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 10 monclient: monmap: 2024-09-17T15:39:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:56.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:39:56.166153+0000 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:39:56.166153+0000 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:39:56.163 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 1 Processor -- start 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 1 -- start start 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.161+0000 7f0e00faa640 10 monclient: init 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T15:39:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-b con 0x7f0dfc069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0dfc074c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0dfc09c6c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc09e410 con 0x7f0dfc09c6c0 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc09e6f0 con 0x7f0dfc074c30 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc09d150 con 0x7f0dfc069260 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0e00faa640 10 monclient(hunting): authenticate will time out at 1729.132321s 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfad9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 0x7f0dfc13ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42636/0 (socket says 172.21.15.86:42636) 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfad9d640 1 -- 172.21.15.86:0/1990090291 learned_addr learned my addr 172.21.15.86:0/1990090291 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:39:56.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0dfc069260 auth_method 0 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0dfad9d640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 legacy=0x7f0dfc13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.165+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.203:6789/0 conn(0x7f0dfc074c30 legacy=0x7f0dfc09bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0dfc069260 auth_method 0 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.665+0000 7f0dfad9d640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 legacy=0x7f0dfc13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-17T15:39:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.665+0000 7f0dfad9d640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 legacy=0x7f0dfc13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.665+0000 7f0dfad9d640 1 --1- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 0x7f0dfc13ba80 :-1 s=OPENED pgs=30 cs=1 l=1).handle_message read tag failed 2024-09-17T15:39:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.665+0000 7f0dfad9d640 1 --1- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 0x7f0dfc13ba80 :-1 s=OPENED pgs=30 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:39:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.665+0000 7f0df9d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-09-17T15:39:56.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:56.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0dfc069260 auth_method 0 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:56.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:56.773+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0dfc069260 auth_method 0 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:57.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:57.577+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): tick 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 monclient(hunting): continuing hunt 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 msgr2=0x7f0dfc09ba70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc069260 0x7f0dfc09ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.203:6789/0 conn(0x7f0dfc074c30 legacy=0x7f0dfc09bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:59.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 -- 172.21.15.86:0/1990090291 >> v1:172.21.15.86:6789/0 conn(0x7f0dfc09c6c0 legacy=0x7f0dfc13ba80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0036b0 addr v2:172.21.15.86:3300/0 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc006220 addr v2:172.21.15.203:3300/0 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 -- 172.21.15.86:0/1990090291 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc09d150 con 0x7f0ddc0036b0 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 1 -- 172.21.15.86:0/1990090291 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0ddc008d80 con 0x7f0ddc006220 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df8d99640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0ddc0036b0 auth_method 0 2024-09-17T15:39:59.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ddc006220 auth_method 0 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:59.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:59.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.165+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0ddc0036b0 auth_method 0 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:59.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ddc006220 auth_method 0 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.369+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.86:3300/0 conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0ddc0036b0 auth_method 0 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ddc006220 auth_method 0 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_done global_id 144099 payload 274 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 msgr2=0x7f0ddc0086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> v2:172.21.15.203:3300/0 conn(0x7f0ddc006220 0x7f0ddc0086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:59.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient: _finish_hunting 0 2024-09-17T15:39:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 1 monclient: found mon.noname-a 2024-09-17T15:39:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:59.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.773+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc09e9d0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0dfa59c640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:59.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0ddc005b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0061f0 tx=0x7f0dec01ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a336611a0187ccc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 1 -- 172.21.15.86:0/1990090291 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec00b040 con 0x7f0ddc0036b0 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: dump: 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.782 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: _finish_auth 0 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 1 -- 172.21.15.86:0/1990090291 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dec0a7af0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 1 -- 172.21.15.86:0/1990090291 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0af3f0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 5 monclient: authenticate success, global_id 144099 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 10 monclient: get_monmap_and_config success 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 4 set_mon_vals no callback set 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0df9d9b640 10 monclient: dump: 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:59.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:59.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 10 monclient: shutdown 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 -- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0ddc005b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0ddc005b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0061f0 tx=0x7f0dec01ef20 comp rx=0 tx=0).stop 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0dfad9d640 1 -- 172.21.15.86:0/1990090291 reap_dead start 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 -- 172.21.15.86:0/1990090291 shutdown_connections 2024-09-17T15:39:59.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 --2- 172.21.15.86:0/1990090291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0ddc005b30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 -- 172.21.15.86:0/1990090291 >> 172.21.15.86:0/1990090291 conn(0x7f0dfc0a6440 msgr2=0x7f0dfc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 -- 172.21.15.86:0/1990090291 shutdown_connections 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.781+0000 7f0e00faa640 1 -- 172.21.15.86:0/1990090291 wait complete. 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient: build_initial_monmap 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient: monmap: 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:39:59.789 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 Processor -- start 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 -- start start 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient: init 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient: _reopen_session rank -1 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0036b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0dfc133310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-b con 0x7f0dfc132e90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): picked mon.noname-c con 0x7f0dfc13b420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc13fbd0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc069680 con 0x7f0dfc132e90 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc13fd50 con 0x7f0dfc13b420 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): _renew_subs 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0e00faa640 10 monclient(hunting): authenticate will time out at 1732.753838s 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57098/0 (socket says 172.21.15.86:57098) 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42394/0 (socket says 172.21.15.86:42394) 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 learned_addr learned my addr 172.21.15.86:0/2184102089 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:39:59.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0ddc0036b0 auth_method 0 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 10 monclient(hunting): get_auth_request con 0x7f0dfc13b420 auth_method 0 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0dfc133310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request con 0x7f0dfc132e90 auth_method 0 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:39:59.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 msgr2=0x7f0dfc13b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 msgr2=0x7f0dfc13b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 msgr2=0x7f0dfc133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 msgr2=0x7f0dfc133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0dfc133310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0dfc133310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient(hunting): handle_auth_done global_id 144102 payload 824 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 msgr2=0x7f0dfc13b880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:59.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0dfc13b880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 msgr2=0x7f0dfc133310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0dfc133310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient: _finish_hunting 0 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 monclient: found mon.noname-a 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:39:59.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.785+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc11e830 con 0x7f0ddc0036b0 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0dfc1ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0b1040 tx=0x7f0dec01d760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0be030 con 0x7f0ddc0036b0 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: dump: 2024-09-17T15:39:59.796 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: _finish_auth 0 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dec0162d0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0baad0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 10 monclient: dump: 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:39:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df959a640 4 set_mon_vals no callback set 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 5 monclient: authenticate success, global_id 144102 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _renew_subs 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc0a2090 con 0x7f0ddc0036b0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _renew_subs 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc13cd50 con 0x7f0ddc0036b0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc09ba70 con 0x7f0ddc0036b0 2024-09-17T15:39:59.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0dec0c1070 con 0x7f0ddc0036b0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 0x7f0ddc03ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.789+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(434..434 src has 213..434) ==== 4815+0+0 (secure 0 0 0) 0x7f0dec0eb6e0 con 0x7f0ddc0036b0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.793+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 0x7f0ddc03ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.793+0000 7f0df1d9b640 10 monclient: get_auth_request con 0x7f0ddc03c990 auth_method 0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.793+0000 7f0df1d9b640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 0x7f0ddc03ee50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0de4007fd0 tx=0x7f0de4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.797+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0dec001a10 con 0x7f0ddc0036b0 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.797+0000 7f0df37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:39:59.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:39:59.797+0000 7f0df37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:00.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0e00faa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:00.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0e00faa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:00.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0e00faa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0dfc06d4f0 con 0x7f0ddc0036b0 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0dfc1ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0dfc1ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0b1040 tx=0x7f0dec01d760 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0dec0b1040 tx=0x7f0dec01d760 comp rx=0 tx=0).stop 2024-09-17T15:40:00.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 msgr2=0x7f0dfc1ce2a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0036b0 0x7f0dfc1ce2a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0ddc044f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): picked mon.c con 0x7f0dfc13b420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:00.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 0x7f0ddc047b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): picked mon.a con 0x7f0ddc0456e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0ddc04a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): picked mon.b con 0x7f0dfc132e90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc04b0a0 con 0x7f0ddc0456e0 2024-09-17T15:40:00.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc04b220 con 0x7f0dfc132e90 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc04b3a0 con 0x7f0dfc13b420 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0df37fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 0x7f0ddc047b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): get_auth_request con 0x7f0ddc0456e0 auth_method 0 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0ddc044f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request con 0x7f0dfc13b420 auth_method 0 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:00.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient(hunting): handle_auth_done global_id 144102 payload 293 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 msgr2=0x7f0ddc044f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0ddc044f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 msgr2=0x7f0ddc04a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0ddc04a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 monclient: found mon.a 2024-09-17T15:40:00.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f0ddc04b5a0 con 0x7f0ddc0456e0 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0dfc09e410 con 0x7f0ddc0456e0 2024-09-17T15:40:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _finish_auth 0 2024-09-17T15:40:00.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 10 monclient: _check_auth_tickets 2024-09-17T15:40:00.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.029+0000 7f0dfad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:00.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0dfad9d640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 msgr2=0x7f0ddc047b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:00.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0dfad9d640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 0x7f0ddc047b70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc09caa0 tx=0x7f0dec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:00.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4001f70 con 0x7f0ddc0456e0 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: dump: 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df4004d70 con 0x7f0ddc0456e0 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df959a640 4 set_mon_vals no callback set 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0df4001db0 con 0x7f0ddc0456e0 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 1 -- 172.21.15.86:0/2184102089 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f0df4050c60 con 0x7f0ddc0456e0 2024-09-17T15:40:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.033+0000 7f0df37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:00.047 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":122,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:00.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 msgr2=0x7f0ddc03ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:00.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 0x7f0ddc03ee50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0de4007fd0 tx=0x7f0de4009040 comp rx=0 tx=0).stop 2024-09-17T15:40:00.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 10 monclient: shutdown 2024-09-17T15:40:00.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 msgr2=0x7f0ddc047b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 0x7f0ddc047b70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc09caa0 tx=0x7f0dec008040 comp rx=0 tx=0).stop 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 shutdown_connections 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0dfc132e90 0x7f0ddc04a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ddc0456e0 0x7f0ddc047b70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dfc13b420 0x7f0ddc044f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 --2- 172.21.15.86:0/2184102089 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ddc03c990 0x7f0ddc03ee50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 >> 172.21.15.86:0/2184102089 conn(0x7f0dfc0a6440 msgr2=0x7f0dfc133b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 shutdown_connections 2024-09-17T15:40:00.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:00.037+0000 7f0e00faa640 1 -- 172.21.15.86:0/2184102089 wait complete. 2024-09-17T15:40:03.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: get_monmap_and_config 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: build_initial_monmap 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: monmap: 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:03.185778+0000 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:03.185778+0000 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:03.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:03.183 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:03.183 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:03.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 Processor -- start 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 -- start start 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: init 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f86040a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86040a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f860413d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f860409d350 con 0x7f86040a54c0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86040a6530 con 0x7f860413d2f0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f860409c150 con 0x7f86040a50e0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8609e8b640 10 monclient(hunting): authenticate will time out at 1736.151332s 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42410/0 (socket says 172.21.15.86:42410) 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f8602ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f86040a54c0 0x7f86040a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45242/0 (socket says 172.21.15.86:45242) 2024-09-17T15:40:03.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 -- 172.21.15.86:0/3293376308 learned_addr learned my addr 172.21.15.86:0/3293376308 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 10 monclient(hunting): get_auth_request con 0x7f86040a50e0 auth_method 0 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 msgr2=0x7f860409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 msgr2=0x7f860409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:03.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.181+0000 7f86037fe640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 -- 172.21.15.86:0/3293376308 >> v1:172.21.15.86:6789/0 conn(0x7f86040a54c0 legacy=0x7f86040a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447569484 0 0) 0x7f860409d350 con 0x7f86040a54c0 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): _negotiate 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:03.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): my global_id is 144105 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85f0003530 con 0x7f86040a54c0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3951785881 0 0) 0x7f85f0003530 con 0x7f86040a54c0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 msgr2=0x7f860409ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f860409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 >> v1:172.21.15.203:6789/0 conn(0x7f860413d2f0 legacy=0x7f860413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _finish_hunting 0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860409c430 con 0x7f86040a54c0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:03.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f85ec0019a0 con 0x7f86040a54c0 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: dump: 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:03.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 >> v1:172.21.15.86:6789/0 conn(0x7f86040a54c0 legacy=0x7f86040a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85f0003ab0 0x7f85f0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): picked mon.a con 0x7f85f0003ab0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f85f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): picked mon.c con 0x7f86040a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f0004f50 0x7f85f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): picked mon.b con 0x7f85f0004f50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f860409c150 con 0x7f85f0003ab0 2024-09-17T15:40:03.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85f0005650 con 0x7f85f0004f50 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f85f000bf40 con 0x7f86040a50e0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86027fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f85f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): get_auth_request con 0x7f86040a50e0 auth_method 0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85f0003ab0 0x7f85f0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): get_auth_request con 0x7f85f0003ab0 auth_method 0 2024-09-17T15:40:03.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient(hunting): handle_auth_done global_id 144105 payload 293 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f0004f50 msgr2=0x7f85f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8603fff640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f0004f50 0x7f85f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f0004f50 0x7f85f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85f0003ab0 msgr2=0x7f85f0003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85f0003ab0 0x7f85f0003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 monclient: found mon.c 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 1 -- 172.21.15.86:0/3293376308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f0003530 con 0x7f86040a50e0 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8602ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8609e8b640 5 monclient: authenticate success, global_id 144105 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8609e8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8603fff640 10 monclient: get_auth_request con 0x7f85f0004f50 auth_method 0 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f8603fff640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85f0004f50 0x7f85f000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:03.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.185+0000 7f86037fe640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85f0003ab0 0x7f85f0003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f8602ffd640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 msgr2=0x7f85f0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f8602ffd640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f85f0004810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85ec001bc0 tx=0x7f85ec01cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28e1bb9b546a40d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec00c0e0 con 0x7f86040a50e0 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:03.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: dump: 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85ec0a9020 con 0x7f86040a50e0 2024-09-17T15:40:03.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f8609e8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f8609e8b640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f8609e8b640 4 set_mon_vals no callback set 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 1 -- 172.21.15.86:0/3293376308 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85ec007310 con 0x7f86040a50e0 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.189+0000 7f86027fc640 10 monclient: dump: 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: shutdown 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 msgr2=0x7f85f0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 --2- 172.21.15.86:0/3293376308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a50e0 0x7f85f0004810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85ec001bc0 tx=0x7f85ec01cc40 comp rx=0 tx=0).stop 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 -- 172.21.15.86:0/3293376308 reap_dead start 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- 172.21.15.86:0/3293376308 shutdown_connections 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- 172.21.15.86:0/3293376308 >> 172.21.15.86:0/3293376308 conn(0x7f86040aaae0 msgr2=0x7f86040aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- 172.21.15.86:0/3293376308 shutdown_connections 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- 172.21.15.86:0/3293376308 wait complete. 2024-09-17T15:40:03.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: build_initial_monmap 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: monmap: 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 Processor -- start 2024-09-17T15:40:03.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- start start 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: init 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f86040a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-c con 0x7f86040a0b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): picked mon.noname-a con 0x7f860413d9b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f86041419a0 con 0x7f860413d9b0 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8604141820 con 0x7f86040a50e0 2024-09-17T15:40:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8604141b20 con 0x7f86040a0b60 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8609e8b640 10 monclient(hunting): authenticate will time out at 1736.162381s 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57132/0 (socket says 172.21.15.86:57132) 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 learned_addr learned my addr 172.21.15.86:0/581938889 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 10 monclient(hunting): get_auth_request con 0x7f860413d9b0 auth_method 0 2024-09-17T15:40:03.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 10 monclient(hunting): get_auth_request con 0x7f86040a0b60 auth_method 0 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 msgr2=0x7f860413de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 msgr2=0x7f860413de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 msgr2=0x7f86040a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:03.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 msgr2=0x7f86040a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f8602ffd640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 10 monclient(hunting): get_auth_request con 0x7f86040a50e0 auth_method 0 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 msgr2=0x7f86041cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:40:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 msgr2=0x7f86041cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:03.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.193+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:03.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 10 monclient(hunting): get_auth_request con 0x7f86040a0b60 auth_method 0 2024-09-17T15:40:03.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): get_auth_request con 0x7f860413d9b0 auth_method 0 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 msgr2=0x7f86040a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 msgr2=0x7f86040a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:03.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8602ffd640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86037fe640 10 monclient(hunting): get_auth_request con 0x7f86040a50e0 auth_method 0 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86037fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient(hunting): handle_auth_done global_id 144111 payload 824 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 msgr2=0x7f86040a0fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 msgr2=0x7f86041cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 monclient: found mon.noname-a 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:03.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:03.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86040a18d0 con 0x7f860413d9b0 2024-09-17T15:40:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 msgr2=0x7f860413de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8603fff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f860409c740 tx=0x7f85fc00fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc019030 con 0x7f860413d9b0 2024-09-17T15:40:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:03.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: dump: 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: _finish_auth 0 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:03.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85fc006b60 con 0x7f860413d9b0 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f400f040 con 0x7f860413d9b0 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f85fbfff640 10 monclient: dump: 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:03.402 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 5 monclient: authenticate success, global_id 144111 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f86017fa640 4 set_mon_vals no callback set 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 10 monclient: _renew_subs 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860409edb0 con 0x7f860413d9b0 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 10 monclient: _renew_subs 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.397+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860413ff20 con 0x7f860413d9b0 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f8609e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f8609e8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:03.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f8609e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:03.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86040a58d0 con 0x7f860413d9b0 2024-09-17T15:40:03.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f85fc01c070 con 0x7f860413d9b0 2024-09-17T15:40:03.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f85fbfff640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 0x7f85e403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:03.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(434..434 src has 213..434) ==== 4815+0+0 (secure 0 0 0) 0x7f85fc050150 con 0x7f860413d9b0 2024-09-17T15:40:03.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 0x7f85e403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:03.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.401+0000 7f86037fe640 10 monclient: get_auth_request con 0x7f85e403bbb0 auth_method 0 2024-09-17T15:40:03.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.405+0000 7f86037fe640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 0x7f85e403e070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007fd0 tx=0x7f85f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:03.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.409+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f85fc014450 con 0x7f860413d9b0 2024-09-17T15:40:03.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.409+0000 7f85fbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:03.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.409+0000 7f85fbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:03.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f8609e8b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:03.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f8609e8b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f8609e8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8604140500 con 0x7f860413d9b0 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f85fbfff640 1 -- 172.21.15.86:0/581938889 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f85fc0147d0 con 0x7f860413d9b0 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f85fbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.649+0000 7f85fbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:03.651 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":122,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 msgr2=0x7f85e403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 0x7f85e403e070 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f85f4007fd0 tx=0x7f85f4008040 comp rx=0 tx=0).stop 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 10 monclient: shutdown 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 msgr2=0x7f860413de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:03.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f860409c740 tx=0x7f85fc00fc80 comp rx=0 tx=0).stop 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 shutdown_connections 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f85e403bbb0 0x7f85e403e070 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f860413d9b0 0x7f860413de10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86040a0b60 0x7f86040a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 --2- 172.21.15.86:0/581938889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86040a50e0 0x7f86041cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 >> 172.21.15.86:0/581938889 conn(0x7f86040aaae0 msgr2=0x7f860413c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 shutdown_connections 2024-09-17T15:40:03.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:03.653+0000 7f8609e8b640 1 -- 172.21.15.86:0/581938889 wait complete. 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1938: done (6 left) 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1939: read oid 18 snap -1 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1939: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1940: read oid 38 snap -1 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1940: expect deleted 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1941: write oid 25 current snap is 218 2024-09-17T15:40:06.300 INFO:tasks.rados.rados.0.smithi203.stdout:1941: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-17T15:40:06.309 INFO:tasks.rados.rados.0.smithi203.stdout:1941: writing smithi20328086-25 from 447251 to 1182931 tid 1 2024-09-17T15:40:06.320 INFO:tasks.rados.rados.0.smithi203.stdout:1941: writing smithi20328086-25 from 1736139 to 2175034 tid 2 2024-09-17T15:40:06.334 INFO:tasks.rados.rados.0.smithi203.stdout:1941: writing smithi20328086-25 from 2597781 to 3378580 tid 3 2024-09-17T15:40:06.334 INFO:tasks.rados.rados.0.smithi203.stdout:1942: rollback oid 6 current snap is 218 2024-09-17T15:40:06.334 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 177 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1932: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1932: finishing copy_from to smithi20328086-29 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 759 (ObjNum 665 snap 209 seq_num 665) dirty exists 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: finishing write tid 2 to smithi20328086-10 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: finishing write tid 3 to smithi20328086-10 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: finishing write tid 4 to smithi20328086-10 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: oid 10 updating version 0 to 687 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: oid 10 version 687 is already newer than 686 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: oid 10 version 687 is already newer than 685 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 687 (ObjNum 690 snap 218 seq_num 690) dirty exists 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1933: left oid 10 (ObjNum 690 snap 218 seq_num 690) 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1935: finishing write tid 1 to smithi20328086-31 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1935: finishing write tid 2 to smithi20328086-31 2024-09-17T15:40:06.335 INFO:tasks.rados.rados.0.smithi203.stdout:1935: finishing write tid 3 to smithi20328086-31 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: finishing write tid 4 to smithi20328086-31 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: finishing write tid 5 to smithi20328086-31 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: oid 31 updating version 0 to 519 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: oid 31 version 519 is already newer than 516 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: oid 31 version 519 is already newer than 518 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: oid 31 version 519 is already newer than 517 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 519 (ObjNum 691 snap 218 seq_num 691) dirty exists 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1935: left oid 31 (ObjNum 691 snap 218 seq_num 691) 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1936: finishing write tid 1 to smithi20328086-28 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1936: finishing write tid 2 to smithi20328086-28 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1936: finishing write tid 3 to smithi20328086-28 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1932: done (9 left) 2024-09-17T15:40:06.336 INFO:tasks.rados.rados.0.smithi203.stdout:1933: done (8 left) 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1934: done (7 left) 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1935: done (6 left) 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1943: write (excl) oid 47 current snap is 218 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: finishing write tid 4 to smithi20328086-28 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: finishing write tid 5 to smithi20328086-28 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: oid 28 updating version 0 to 919 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: oid 28 version 919 is already newer than 917 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: oid 28 version 919 is already newer than 918 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: oid 28 updating version 919 to 920 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 920 (ObjNum 692 snap 218 seq_num 692) dirty exists 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1936: left oid 28 (ObjNum 692 snap 218 seq_num 692) 2024-09-17T15:40:06.337 INFO:tasks.rados.rados.0.smithi203.stdout:1943: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-17T15:40:06.339 INFO:tasks.rados.rados.0.smithi203.stdout:1943: writing smithi20328086-47 from 464058 to 874987 tid 1 2024-09-17T15:40:06.346 INFO:tasks.rados.rados.0.smithi203.stdout:1943: writing smithi20328086-47 from 1394621 to 2093527 tid 2 2024-09-17T15:40:06.353 INFO:tasks.rados.rados.0.smithi203.stdout:1943: writing smithi20328086-47 from 2625204 to 3217275 tid 3 2024-09-17T15:40:06.355 INFO:tasks.rados.rados.0.smithi203.stdout:1943: writing smithi20328086-47 from 3656107 to 3664058 tid 4 2024-09-17T15:40:06.355 INFO:tasks.rados.rados.0.smithi203.stdout:1936: done (6 left) 2024-09-17T15:40:06.356 INFO:tasks.rados.rados.0.smithi203.stdout:1944: copy_from oid 29 from oid 50 current snap is 218 2024-09-17T15:40:06.356 INFO:tasks.rados.rados.0.smithi203.stdout:1937: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:40:06.380 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2559450 2024-09-17T15:40:06.457 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1964839 2024-09-17T15:40:06.484 INFO:tasks.rados.rados.0.smithi203.stdout:1939: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: finishing write tid 1 to smithi20328086-25 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: finishing write tid 2 to smithi20328086-25 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: finishing write tid 3 to smithi20328086-25 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: finishing write tid 4 to smithi20328086-25 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1942: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: finishing write tid 5 to smithi20328086-25 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: oid 25 updating version 0 to 763 2024-09-17T15:40:06.571 INFO:tasks.rados.rados.0.smithi203.stdout:1941: oid 25 version 763 is already newer than 760 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1941: oid 25 version 763 is already newer than 762 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1941: oid 25 version 763 is already newer than 761 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 763 (ObjNum 693 snap 218 seq_num 693) dirty exists 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1941: left oid 25 (ObjNum 693 snap 218 seq_num 693) 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1942: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1937: done (6 left) 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1939: done (5 left) 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1940: done (4 left) 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1941: done (3 left) 2024-09-17T15:40:06.572 INFO:tasks.rados.rados.0.smithi203.stdout:1945: snap_remove snap 197 2024-09-17T15:40:06.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: get_monmap_and_config 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: build_initial_monmap 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: monmap: 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:06.802056+0000 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:06.802056+0000 2024-09-17T15:40:06.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:06.799 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 Processor -- start 2024-09-17T15:40:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 -- start start 2024-09-17T15:40:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: init 2024-09-17T15:40:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcf40689d0 0x7efcf406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-a con 0x7efcf40689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcf4068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-b con 0x7efcf413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf4074470 con 0x7efcf4068db0 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7efcf409c900 con 0x7efcf40689d0 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efcf409cb90 con 0x7efcf413aff0 2024-09-17T15:40:06.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcfb2db640 10 monclient(hunting): authenticate will time out at 1739.767516s 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcf40689d0 0x7efcf406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcf40689d0 0x7efcf406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57154/0 (socket says 172.21.15.86:57154) 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42450/0 (socket says 172.21.15.86:42450) 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 1 -- 172.21.15.86:0/3650193602 learned_addr learned my addr 172.21.15.86:0/3650193602 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): get_auth_request con 0x7efcf40689d0 auth_method 0 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf8876640 1 --1- 172.21.15.86:0/3650193602 >> v1:172.21.15.86:6789/0 conn(0x7efcf4068db0 0x7efcf4069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45258/0 (socket says 172.21.15.86:45258) 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): get_auth_request con 0x7efcf413aff0 auth_method 0 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.797+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 10 monclient(hunting): handle_auth_done global_id 144114 payload 274 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 1 -- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 msgr2=0x7efcf413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 1 -- 172.21.15.86:0/3650193602 >> v1:172.21.15.86:6789/0 conn(0x7efcf4068db0 legacy=0x7efcf4069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 10 monclient: _finish_hunting 0 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 1 monclient: found mon.noname-a 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9077640 1 -- 172.21.15.86:0/3650193602 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf409ce70 con 0x7efcf40689d0 2024-09-17T15:40:06.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.801+0000 7efcf9878640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:06.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 -- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 msgr2=0x7efcf406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:06.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf406f840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009870 tx=0x7efcf001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da27984c52649be0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:06.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 1 -- 172.21.15.86:0/3650193602 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00b1070 con 0x7efcf40689d0 2024-09-17T15:40:06.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: dump: 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: _finish_auth 0 2024-09-17T15:40:06.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 1 -- 172.21.15.86:0/3650193602 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf00ac620 con 0x7efcf40689d0 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 5 monclient: authenticate success, global_id 144114 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 1 -- 172.21.15.86:0/3650193602 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf409c2d0 con 0x7efcf40689d0 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 4 set_mon_vals no callback set 2024-09-17T15:40:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcebfff640 10 monclient: dump: 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:06.811 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: shutdown 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 msgr2=0x7efcf406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf406f840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009870 tx=0x7efcf001c420 comp rx=0 tx=0).stop 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- 172.21.15.86:0/3650193602 shutdown_connections 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- 172.21.15.86:0/3650193602 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf406f840 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- 172.21.15.86:0/3650193602 >> 172.21.15.86:0/3650193602 conn(0x7efcf40a6440 msgr2=0x7efcf40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- 172.21.15.86:0/3650193602 shutdown_connections 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- 172.21.15.86:0/3650193602 wait complete. 2024-09-17T15:40:06.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: build_initial_monmap 2024-09-17T15:40:06.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: monmap: 2024-09-17T15:40:06.813 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:06.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:06.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 Processor -- start 2024-09-17T15:40:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- start start 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: init 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-a con 0x7efcf40689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf4068db0 0x7efcf4135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-b con 0x7efcf4068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf4136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): picked mon.noname-c con 0x7efcf413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7efcf4140610 con 0x7efcf40689d0 2024-09-17T15:40:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efcf4140490 con 0x7efcf4068db0 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efcf4140790 con 0x7efcf413aff0 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcfb2db640 10 monclient(hunting): authenticate will time out at 1739.774910s 2024-09-17T15:40:06.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf4136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf4136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42452/0 (socket says 172.21.15.86:42452) 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57158/0 (socket says 172.21.15.86:57158) 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 1 -- 172.21.15.86:0/2351445694 learned_addr learned my addr 172.21.15.86:0/2351445694 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): get_auth_request con 0x7efcf413aff0 auth_method 0 2024-09-17T15:40:06.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): get_auth_request con 0x7efcf40689d0 auth_method 0 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient(hunting): handle_auth_done global_id 144117 payload 824 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 -- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 msgr2=0x7efcf4136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf4136330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 -- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf4068db0 msgr2=0x7efcf4135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf4068db0 0x7efcf4135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient: _finish_hunting 0 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 monclient: found mon.noname-a 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.805+0000 7efcf9077640 1 -- 172.21.15.86:0/2351445694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf41227c0 con 0x7efcf40689d0 2024-09-17T15:40:06.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcf9077640 1 -- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 msgr2=0x7efcf41cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcf9077640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf00ac000 tx=0x7efcf001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00bb030 con 0x7efcf40689d0 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: dump: 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf00ac420 con 0x7efcf40689d0 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00b1040 con 0x7efcf40689d0 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 10 monclient: dump: 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efceb7fe640 4 set_mon_vals no callback set 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 5 monclient: authenticate success, global_id 144117 2024-09-17T15:40:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _renew_subs 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf409d460 con 0x7efcf40689d0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _renew_subs 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf4120fc0 con 0x7efcf40689d0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf406f840 con 0x7efcf40689d0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7efcf00be070 con 0x7efcf40689d0 2024-09-17T15:40:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 0x7efcd803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.809+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(434..434 src has 213..434) ==== 4815+0+0 (secure 0 0 0) 0x7efcf00e9ef0 con 0x7efcf40689d0 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.813+0000 7efcf8876640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 0x7efcd803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.813+0000 7efcf8876640 10 monclient: get_auth_request con 0x7efcd803bbb0 auth_method 0 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.813+0000 7efcf8876640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 0x7efcd803e070 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcf409f950 tx=0x7efce40079f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.817+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7efcf0016130 con 0x7efcf40689d0 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.817+0000 7efce9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:06.817+0000 7efce9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:07.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efcf4121b90 con 0x7efcf40689d0 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efce9ffb640 1 -- 172.21.15.86:0/2351445694 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7efcf00ac5d0 con 0x7efcf40689d0 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efce9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efce9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:07.049 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":122,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 msgr2=0x7efcd803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 0x7efcd803e070 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7efcf409f950 tx=0x7efce40079f0 comp rx=0 tx=0).stop 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 10 monclient: shutdown 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 msgr2=0x7efcf41cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf00ac000 tx=0x7efcf001ac60 comp rx=0 tx=0).stop 2024-09-17T15:40:07.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 shutdown_connections 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efcd803bbb0 0x7efcd803e070 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcf413aff0 0x7efcf4136330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcf4068db0 0x7efcf4135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 --2- 172.21.15.86:0/2351445694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcf40689d0 0x7efcf41cfe00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 >> 172.21.15.86:0/2351445694 conn(0x7efcf40a6440 msgr2=0x7efcf40a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 shutdown_connections 2024-09-17T15:40:07.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:07.049+0000 7efcfb2db640 1 -- 172.21.15.86:0/2351445694 wait complete. 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1945: done (3 left) 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1946: copy_from oid 42 from oid 44 current snap is 218 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1942: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 824 (ObjNum 482 snap 145 seq_num 482) dirty exists 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 1 to smithi20328086-47 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1944: finishing copy_from to smithi20328086-29 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1944: got expected ENOENT (src dne) 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1944: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 2 to smithi20328086-47 2024-09-17T15:40:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 3 to smithi20328086-47 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 4 to smithi20328086-47 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 5 to smithi20328086-47 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: finishing write tid 6 to smithi20328086-47 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: oid 47 updating version 0 to 803 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: oid 47 version 803 is already newer than 802 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: oid 47 updating version 803 to 805 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: oid 47 version 805 is already newer than 804 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: oid 47 version 805 is already newer than 801 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 805 (ObjNum 694 snap 218 seq_num 694) dirty exists 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: left oid 47 (ObjNum 694 snap 218 seq_num 694) 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1942: done (3 left) 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1943: done (2 left) 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1944: done (1 left) 2024-09-17T15:40:07.253 INFO:tasks.rados.rados.0.smithi203.stdout:1947: snap_remove snap 210 2024-09-17T15:40:08.272 INFO:tasks.rados.rados.0.smithi203.stdout:1947: done (1 left) 2024-09-17T15:40:08.272 INFO:tasks.rados.rados.0.smithi203.stdout:1948: write (excl) oid 39 current snap is 218 2024-09-17T15:40:08.272 INFO:tasks.rados.rados.0.smithi203.stdout:1946: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:40:08.272 INFO:tasks.rados.rados.0.smithi203.stdout:1946: finishing copy_from to smithi20328086-42 2024-09-17T15:40:08.273 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 653 (ObjNum 610 snap 184 seq_num 610) dirty exists 2024-09-17T15:40:08.273 INFO:tasks.rados.rados.0.smithi203.stdout:1948: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-17T15:40:08.286 INFO:tasks.rados.rados.0.smithi203.stdout:1948: writing smithi20328086-39 from 480865 to 967043 tid 1 2024-09-17T15:40:08.299 INFO:tasks.rados.rados.0.smithi203.stdout:1948: writing smithi20328086-39 from 1453103 to 1895667 tid 2 2024-09-17T15:40:08.315 INFO:tasks.rados.rados.0.smithi203.stdout:1948: writing smithi20328086-39 from 2652627 to 3172323 tid 3 2024-09-17T15:40:08.315 INFO:tasks.rados.rados.0.smithi203.stdout:1946: done (1 left) 2024-09-17T15:40:08.315 INFO:tasks.rados.rados.0.smithi203.stdout:1949: write (excl) oid 2 current snap is 218 2024-09-17T15:40:08.316 INFO:tasks.rados.rados.0.smithi203.stdout:1949: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-17T15:40:08.327 INFO:tasks.rados.rados.0.smithi203.stdout:1949: writing smithi20328086-2 from 497672 to 1059099 tid 1 2024-09-17T15:40:08.341 INFO:tasks.rados.rados.0.smithi203.stdout:1949: writing smithi20328086-2 from 1795232 to 2497807 tid 2 2024-09-17T15:40:08.350 INFO:tasks.rados.rados.0.smithi203.stdout:1949: writing smithi20328086-2 from 2963697 to 3694665 tid 3 2024-09-17T15:40:08.350 INFO:tasks.rados.rados.0.smithi203.stdout:1950: write oid 17 current snap is 218 2024-09-17T15:40:08.350 INFO:tasks.rados.rados.0.smithi203.stdout:1950: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-17T15:40:08.358 INFO:tasks.rados.rados.0.smithi203.stdout:1950: writing smithi20328086-17 from 514479 to 1151155 tid 1 2024-09-17T15:40:08.366 INFO:tasks.rados.rados.0.smithi203.stdout:1950: writing smithi20328086-17 from 1853714 to 2299947 tid 2 2024-09-17T15:40:08.373 INFO:tasks.rados.rados.0.smithi203.stdout:1950: writing smithi20328086-17 from 2991120 to 3533360 tid 3 2024-09-17T15:40:08.373 INFO:tasks.rados.rados.0.smithi203.stdout:1951: write (excl) oid 47 current snap is 218 2024-09-17T15:40:08.374 INFO:tasks.rados.rados.0.smithi203.stdout:1951: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-17T15:40:08.380 INFO:tasks.rados.rados.0.smithi203.stdout:1951: writing smithi20328086-47 from 531286 to 1243211 tid 1 2024-09-17T15:40:08.388 INFO:tasks.rados.rados.0.smithi203.stdout:1951: writing smithi20328086-47 from 1912196 to 2618440 tid 2 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1951: writing smithi20328086-47 from 3018543 to 3731286 tid 3 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1952: snap_create 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: finishing write tid 1 to smithi20328086-39 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: finishing write tid 2 to smithi20328086-39 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: finishing write tid 3 to smithi20328086-39 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: finishing write tid 4 to smithi20328086-39 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: finishing write tid 5 to smithi20328086-39 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: oid 39 updating version 0 to 841 2024-09-17T15:40:08.395 INFO:tasks.rados.rados.0.smithi203.stdout:1948: oid 39 version 841 is already newer than 840 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1948: oid 39 updating version 841 to 842 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1948: oid 39 version 842 is already newer than 839 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 842 (ObjNum 695 snap 218 seq_num 695) dirty exists 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1948: left oid 39 (ObjNum 695 snap 218 seq_num 695) 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: finishing write tid 1 to smithi20328086-2 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: finishing write tid 2 to smithi20328086-2 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: finishing write tid 3 to smithi20328086-2 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: finishing write tid 4 to smithi20328086-2 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: finishing write tid 5 to smithi20328086-2 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: oid 2 updating version 0 to 481 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: oid 2 updating version 481 to 483 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: oid 2 version 483 is already newer than 482 2024-09-17T15:40:08.396 INFO:tasks.rados.rados.0.smithi203.stdout:1949: oid 2 version 483 is already newer than 480 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 483 (ObjNum 696 snap 218 seq_num 696) dirty exists 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1949: left oid 2 (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: finishing write tid 1 to smithi20328086-17 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: finishing write tid 2 to smithi20328086-17 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: finishing write tid 3 to smithi20328086-17 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: finishing write tid 4 to smithi20328086-17 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: finishing write tid 5 to smithi20328086-17 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: oid 17 updating version 0 to 766 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: oid 17 version 766 is already newer than 765 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: oid 17 updating version 766 to 768 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: oid 17 version 768 is already newer than 767 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 768 (ObjNum 697 snap 218 seq_num 697) dirty exists 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1950: left oid 17 (ObjNum 697 snap 218 seq_num 697) 2024-09-17T15:40:08.397 INFO:tasks.rados.rados.0.smithi203.stdout:1951: finishing write tid 1 to smithi20328086-47 2024-09-17T15:40:08.402 INFO:tasks.rados.rados.0.smithi203.stdout:1951: finishing write tid 2 to smithi20328086-47 2024-09-17T15:40:08.404 INFO:tasks.rados.rados.0.smithi203.stdout:1951: finishing write tid 3 to smithi20328086-47 2024-09-17T15:40:08.404 INFO:tasks.rados.rados.0.smithi203.stdout:1951: finishing write tid 4 to smithi20328086-47 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: finishing write tid 5 to smithi20328086-47 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: oid 47 updating version 0 to 808 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: oid 47 version 808 is already newer than 807 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: oid 47 version 808 is already newer than 806 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: oid 47 updating version 808 to 809 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 809 (ObjNum 698 snap 218 seq_num 698) dirty exists 2024-09-17T15:40:08.405 INFO:tasks.rados.rados.0.smithi203.stdout:1951: left oid 47 (ObjNum 698 snap 218 seq_num 698) 2024-09-17T15:40:09.271 INFO:tasks.rados.rados.0.smithi203.stdout:1948: done (4 left) 2024-09-17T15:40:09.271 INFO:tasks.rados.rados.0.smithi203.stdout:1949: done (3 left) 2024-09-17T15:40:09.271 INFO:tasks.rados.rados.0.smithi203.stdout:1950: done (2 left) 2024-09-17T15:40:09.271 INFO:tasks.rados.rados.0.smithi203.stdout:1951: done (1 left) 2024-09-17T15:40:09.271 INFO:tasks.rados.rados.0.smithi203.stdout:1952: done (0 left) 2024-09-17T15:40:09.272 INFO:tasks.rados.rados.0.smithi203.stdout:1953: copy_from oid 29 from oid 21 current snap is 219 2024-09-17T15:40:09.272 INFO:tasks.rados.rados.0.smithi203.stdout:1954: snap_remove snap 172 2024-09-17T15:40:10.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: get_monmap_and_config 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: monmap: 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:10.201215+0000 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:10.201215+0000 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:10.198 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 Processor -- start 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 -- start start 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: init 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-17T15:40:10.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5900689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): picked mon.noname-b con 0x7ff5900694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff59013aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff590074470 con 0x7ff5900689d0 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59009c8b0 con 0x7ff59013aff0 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff59009cbe0 con 0x7ff5900694c0 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff595741640 10 monclient(hunting): authenticate will time out at 1743.167209s 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff5900689d0 0x7ff590068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41604/0 (socket says 172.21.15.86:41604) 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58effd640 1 -- 172.21.15.86:0/140351970 learned_addr learned my addr 172.21.15.86:0/140351970 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:10.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): get_auth_request con 0x7ff5900694c0 auth_method 0 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_done global_id 104510 payload 274 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 1 -- 172.21.15.86:0/140351970 >> v1:172.21.15.203:6789/0 conn(0x7ff59013aff0 legacy=0x7ff59013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 1 -- 172.21.15.86:0/140351970 >> v1:172.21.15.86:6789/0 conn(0x7ff5900689d0 legacy=0x7ff590068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 1 monclient: found mon.noname-b 2024-09-17T15:40:10.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.197+0000 7ff58e7fc640 1 -- 172.21.15.86:0/140351970 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59009ce70 con 0x7ff5900694c0 2024-09-17T15:40:10.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.201+0000 7ff58e7fc640 1 -- 172.21.15.86:0/140351970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 msgr2=0x7ff59006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 --2- 172.21.15.86:0/140351970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59006f840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097f0 tx=0x7ff58401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f7853316b6fa92 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 1 -- 172.21.15.86:0/140351970 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5840b0070 con 0x7ff5900694c0 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: dump: 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 1 -- 172.21.15.86:0/140351970 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5840ab600 con 0x7ff5900694c0 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 5 monclient: authenticate success, global_id 104510 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 1 -- 172.21.15.86:0/140351970 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59009c2d0 con 0x7ff5900694c0 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58dffb640 10 monclient: dump: 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.207 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 4 set_mon_vals no callback set 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: shutdown 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- 172.21.15.86:0/140351970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 msgr2=0x7ff59006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 --2- 172.21.15.86:0/140351970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59006f840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097f0 tx=0x7ff58401c3a0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- 172.21.15.86:0/140351970 shutdown_connections 2024-09-17T15:40:10.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 --2- 172.21.15.86:0/140351970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59006f840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- 172.21.15.86:0/140351970 >> 172.21.15.86:0/140351970 conn(0x7ff5900a6440 msgr2=0x7ff5900a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- 172.21.15.86:0/140351970 shutdown_connections 2024-09-17T15:40:10.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- 172.21.15.86:0/140351970 wait complete. 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: monmap: 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:10.213 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 Processor -- start 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- start start 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: init 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 0x7ff59011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): picked mon.noname-a con 0x7ff5900689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): picked mon.noname-c con 0x7ff5900694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 0x7ff59011d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): picked mon.noname-b con 0x7ff59013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff590141ca0 con 0x7ff5900689d0 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff590141b20 con 0x7ff59013aff0 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff590141e20 con 0x7ff5900694c0 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff595741640 10 monclient(hunting): authenticate will time out at 1743.175159s 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 0x7ff59011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53688/0 (socket says 172.21.15.86:53688) 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 0x7ff59011c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48432/0 (socket says 172.21.15.86:48432) 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 -- 172.21.15.86:0/2102226953 learned_addr learned my addr 172.21.15.86:0/2102226953 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): get_auth_request con 0x7ff5900694c0 auth_method 0 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): get_auth_request con 0x7ff5900689d0 auth_method 0 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58f7fe640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 0x7ff59011d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient(hunting): handle_auth_done global_id 104513 payload 824 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 -- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 msgr2=0x7ff59011d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 0x7ff59011d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 -- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 msgr2=0x7ff59011c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 0x7ff59011c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 monclient: found mon.noname-c 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58e7fc640 1 -- 172.21.15.86:0/2102226953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59011f0e0 con 0x7ff5900694c0 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58f7fe640 10 monclient: get_auth_request con 0x7ff59013aff0 auth_method 0 2024-09-17T15:40:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.205+0000 7ff58f7fe640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 0x7ff59011d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff58e7fc640 1 -- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 msgr2=0x7ff59011ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff58e7fc640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097f0 tx=0x7ff58401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5840b0050 con 0x7ff5900694c0 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: dump: 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.218 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5840bc070 con 0x7ff5900694c0 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5840b5a80 con 0x7ff5900694c0 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 10 monclient: dump: 2024-09-17T15:40:10.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff58d7fa640 4 set_mon_vals no callback set 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 5 monclient: authenticate success, global_id 104513 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _renew_subs 2024-09-17T15:40:10.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59011f3c0 con 0x7ff5900694c0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _renew_subs 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59013e0b0 con 0x7ff5900694c0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff59006f840 con 0x7ff5900694c0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff5840b5c20 con 0x7ff5900694c0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 0x7ff57c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.209+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(438..438 src has 213..438) ==== 4803+0+0 (secure 0 0 0) 0x7ff5840eb5e0 con 0x7ff5900694c0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.213+0000 7ff58effd640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 0x7ff57c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.213+0000 7ff58effd640 10 monclient: get_auth_request con 0x7ff57c03bb90 auth_method 0 2024-09-17T15:40:10.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.213+0000 7ff58effd640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 0x7ff57c03e050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff580007ca0 tx=0x7ff580007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.217+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff5840160b0 con 0x7ff5900694c0 2024-09-17T15:40:10.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.217+0000 7ff5777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.217+0000 7ff5777fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:10.277 INFO:tasks.rados.rados.0.smithi203.stdout:1954: done (1 left) 2024-09-17T15:40:10.277 INFO:tasks.rados.rados.0.smithi203.stdout:1955: read oid 16 snap -1 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1955: expect (ObjNum 678 snap 214 seq_num 678) 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1956: rollback oid 42 current snap is 219 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 201 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1953: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1953: finishing copy_from to smithi20328086-29 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1953: done (2 left) 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1957: copy_from oid 6 from oid 10 current snap is 219 2024-09-17T15:40:10.278 INFO:tasks.rados.rados.0.smithi203.stdout:1958: snap_create 2024-09-17T15:40:10.295 INFO:tasks.rados.rados.0.smithi203.stdout:1956: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:40:10.296 INFO:tasks.rados.rados.0.smithi203.stdout:1956: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:40:10.296 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 876 (ObjNum 129 snap 0 seq_num 3605848056) dirty dne 2024-09-17T15:40:10.298 INFO:tasks.rados.rados.0.smithi203.stdout:1955: expect (ObjNum 678 snap 214 seq_num 678) 2024-09-17T15:40:10.372 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3017241 2024-09-17T15:40:10.386 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:10.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff595741640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:10.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff595741640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:10.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff595741640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff59011df90 con 0x7ff5900694c0 2024-09-17T15:40:10.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff5777fe640 1 -- 172.21.15.86:0/2102226953 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff5840e9010 con 0x7ff5900694c0 2024-09-17T15:40:10.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff5777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:10.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.461+0000 7ff5777fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:10.463 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 msgr2=0x7ff57c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 0x7ff57c03e050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff580007ca0 tx=0x7ff580007bf0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 10 monclient: shutdown 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 msgr2=0x7ff59011ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097f0 tx=0x7ff58401abe0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 shutdown_connections 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff57c03bb90 0x7ff57c03e050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff59013aff0 0x7ff59011d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5900694c0 0x7ff59011ce00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 --2- 172.21.15.86:0/2102226953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5900689d0 0x7ff59011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 >> 172.21.15.86:0/2102226953 conn(0x7ff5900a6440 msgr2=0x7ff59013ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 shutdown_connections 2024-09-17T15:40:10.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.465+0000 7ff595741640 1 -- 172.21.15.86:0/2102226953 wait complete. 2024-09-17T15:40:10.480 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:40:10.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:40:10.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:40:10.488 INFO:tasks.rados.rados.0.smithi203.stdout:1957: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:40:10.489 INFO:tasks.rados.rados.0.smithi203.stdout:1957: finishing copy_from to smithi20328086-6 2024-09-17T15:40:10.489 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 687 (ObjNum 690 snap 218 seq_num 690) dirty exists 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: get_monmap_and_config 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: monmap: 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:10.592121+0000 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:10.592121+0000 2024-09-17T15:40:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:10.589 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 Processor -- start 2024-09-17T15:40:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 -- start start 2024-09-17T15:40:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: init 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-b con 0x7fc4100a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4100a54c0 0x7fc4100a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-a con 0x7fc4100a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc410137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4100a6280 con 0x7fc410137ea0 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc41009c0a0 con 0x7fc4100a54c0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc41009c380 con 0x7fc4100a50e0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc415e88640 10 monclient(hunting): authenticate will time out at 1743.557511s 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc4100a54c0 0x7fc4100a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53704/0 (socket says 172.21.15.86:53704) 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1293870368 learned_addr learned my addr 172.21.15.86:0/1293870368 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request con 0x7fc4100a50e0 auth_method 0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): get_auth_request con 0x7fc4100a54c0 auth_method 0 2024-09-17T15:40:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient(hunting): handle_auth_done global_id 144123 payload 274 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 -- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 msgr2=0x7fc41009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 -- 172.21.15.86:0/1293870368 >> v1:172.21.15.203:6789/0 conn(0x7fc410137ea0 legacy=0x7fc410138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 monclient: found mon.noname-a 2024-09-17T15:40:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40effd640 1 -- 172.21.15.86:0/1293870368 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41009c660 con 0x7fc4100a54c0 2024-09-17T15:40:10.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.589+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:10.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40effd640 1 -- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 msgr2=0x7fc4100a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40effd640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc4100a58c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc4000097c0 tx=0x7fc40001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6ffd2f24f14fca server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 1 -- 172.21.15.86:0/1293870368 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000b0070 con 0x7fc4100a54c0 2024-09-17T15:40:10.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: dump: 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 1 -- 172.21.15.86:0/1293870368 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4000ab600 con 0x7fc4100a54c0 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 5 monclient: authenticate success, global_id 144123 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 4 set_mon_vals no callback set 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 1 -- 172.21.15.86:0/1293870368 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000ab9d0 con 0x7fc4100a54c0 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc40e7fc640 10 monclient: dump: 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.600 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 10 monclient: shutdown 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 -- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 msgr2=0x7fc4100a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc4100a58c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc4000097c0 tx=0x7fc40001c3f0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 -- 172.21.15.86:0/1293870368 shutdown_connections 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc4100a58c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 --2- 172.21.15.86:0/1293870368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc41009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 -- 172.21.15.86:0/1293870368 >> 172.21.15.86:0/1293870368 conn(0x7fc4100aaae0 msgr2=0x7fc4100aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 -- 172.21.15.86:0/1293870368 shutdown_connections 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.593+0000 7fc415e88640 1 -- 172.21.15.86:0/1293870368 wait complete. 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient: monmap: 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.602 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 Processor -- start 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 -- start start 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient: init 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc4101d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-c con 0x7fc4100a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc410137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-a con 0x7fc4100a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc410137ea0 0x7fc41013c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): picked mon.noname-b con 0x7fc410137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc410142cb0 con 0x7fc4100a54c0 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc410142b30 con 0x7fc410137ea0 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc410142e30 con 0x7fc4100a50e0 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc415e88640 10 monclient(hunting): authenticate will time out at 1743.565167s 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc4101d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc410137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc4101d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53708/0 (socket says 172.21.15.86:53708) 2024-09-17T15:40:10.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1626964275 learned_addr learned my addr 172.21.15.86:0/1626964275 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc410137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48448/0 (socket says 172.21.15.86:48448) 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): get_auth_request con 0x7fc4100a54c0 auth_method 0 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request con 0x7fc4100a50e0 auth_method 0 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40ffff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc410137ea0 0x7fc41013c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient(hunting): handle_auth_done global_id 144126 payload 824 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 msgr2=0x7fc4101d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc4101d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc410137ea0 msgr2=0x7fc41013c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc410137ea0 0x7fc41013c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 monclient: found mon.noname-a 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4101d0ff0 con 0x7fc4100a54c0 2024-09-17T15:40:10.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40ffff640 10 monclient: get_auth_request con 0x7fc410137ea0 auth_method 0 2024-09-17T15:40:10.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc4100a50e0 0x7fc4101d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:10.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40ffff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc410137ea0 0x7fc41013c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:10.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 msgr2=0x7fc410137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.597+0000 7fc40effd640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc410137000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4000ad000 tx=0x7fc400019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000b8020 con 0x7fc4100a54c0 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: dump: 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.611 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4000b4850 con 0x7fc4100a54c0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 5 monclient: authenticate success, global_id 144126 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 monclient: _renew_subs 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4101210f0 con 0x7fc4100a54c0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000abe60 con 0x7fc4100a54c0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: dump: 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 monclient: _renew_subs 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc41013dc30 con 0x7fc4100a54c0 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40dffb640 4 set_mon_vals no callback set 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 0x7fc41013c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc4100a45e0 con 0x7fc410137540 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 0x7fc41013c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient: get_auth_request con 0x7fc410137540 auth_method 0 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc4000bb020 con 0x7fc4100a54c0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_done global_id 144129 payload 824 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc41009cd60 con 0x7fc410137540 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3fc03bbb0 0x7fc3fc03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(439..439 src has 213..439) ==== 4847+0+0 (secure 0 0 0) 0x7fc4000e9af0 con 0x7fc4100a54c0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40ffff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3fc03bbb0 0x7fc3fc03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40ffff640 10 monclient: get_auth_request con 0x7fc3fc03bbb0 auth_method 0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 msgr2=0x7fc41013c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 0x7fc41013c6f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc408004980 tx=0x7fc40800e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408019020 con 0x7fc410137540 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.601+0000 7fc3f7fff640 10 monclient: dump: 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.615 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc408015870 con 0x7fc410137540 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 msgr2=0x7fc41013c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 0x7fc41013c6f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc408004980 tx=0x7fc40800e8c0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 msgr2=0x7fc41013c6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc3f7fff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc410137540 0x7fc41013c6f0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc408004980 tx=0x7fc40800e8c0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.605+0000 7fc40ffff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3fc03bbb0 0x7fc3fc03e070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc404003dc0 tx=0x7fc404009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc415e88640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc415e88640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 0x7fc4100a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc415e88640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc41013d040 con 0x7fc41013cc30 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 0x7fc4100a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient: get_auth_request con 0x7fc41013cc30 auth_method 0 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 10 monclient(hunting): handle_auth_done global_id 144132 payload 824 2024-09-17T15:40:10.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.609+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1626964275 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc410121a50 con 0x7fc41013cc30 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc40f7fe640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 msgr2=0x7fc4100a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc40f7fe640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 0x7fc4100a05f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc40801a000 tx=0x7fc408004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408019030 con 0x7fc41013cc30 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 10 monclient: dump: 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.619 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fc40801e420 con 0x7fc41013cc30 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 msgr2=0x7fc4100a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 0x7fc4100a05f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc40801a000 tx=0x7fc408004ac0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 msgr2=0x7fc4100a05f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc3f7fff640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc41013cc30 0x7fc4100a05f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28955, 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.621 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.622 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.623 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.624 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.625 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:10.626 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:10.627 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.628 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.629 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.630 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.631 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.632 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:10.633 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:10.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3fc03bbb0 msgr2=0x7fc3fc03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3fc03bbb0 0x7fc3fc03e070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc404003dc0 tx=0x7fc404009040 comp rx=0 tx=0).stop 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 10 monclient: shutdown 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 msgr2=0x7fc410137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 --2- 172.21.15.86:0/1626964275 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc4100a54c0 0x7fc410137000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc4000ad000 tx=0x7fc400019190 comp rx=0 tx=0).stop 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc40ffff640 1 -- 172.21.15.86:0/1626964275 reap_dead start 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc40ffff640 1 -- 172.21.15.86:0/1626964275 reap_dead start 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 shutdown_connections 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 >> 172.21.15.86:0/1626964275 conn(0x7fc4100aaae0 msgr2=0x7fc4100a1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 shutdown_connections 2024-09-17T15:40:10.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.613+0000 7fc415e88640 1 -- 172.21.15.86:0/1626964275 wait complete. 2024-09-17T15:40:10.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:10.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:40:10.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:40:10.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 10 monclient: get_monmap_and_config 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 10 monclient: monmap: 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:10.750862+0000 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:10.750862+0000 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:10.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:10.748 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:10.748 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:10.748 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 1 Processor -- start 2024-09-17T15:40:10.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 1 -- start start 2024-09-17T15:40:10.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.745+0000 7fc11aaf4640 10 monclient: init 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 0x7fc1140a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1140a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 0x7fc114135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1140a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc114135f90 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1140a2130 con 0x7fc114135f90 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc1140a2410 con 0x7fc1140a1590 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc1140a26f0 con 0x7fc1140a0c70 2024-09-17T15:40:10.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc11aaf4640 10 monclient(hunting): authenticate will time out at 1743.716422s 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc119091640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc114135f90 0x7fc11413cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41620/0 (socket says 172.21.15.86:41620) 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc119091640 1 -- 172.21.15.86:0/2426680658 learned_addr learned my addr 172.21.15.86:0/2426680658 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 --2- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 0x7fc1140a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): get_auth_request con 0x7fc1140a0c70 auth_method 0 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 1 --2- 172.21.15.86:0/2426680658 >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 0x7fc114135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 10 monclient(hunting): get_auth_request con 0x7fc1140a1590 auth_method 0 2024-09-17T15:40:10.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient(hunting): handle_auth_done global_id 104522 payload 274 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 -- 172.21.15.86:0/2426680658 >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 msgr2=0x7fc114135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 --2- 172.21.15.86:0/2426680658 >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 0x7fc114135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 -- 172.21.15.86:0/2426680658 >> v1:172.21.15.86:6789/0 conn(0x7fc114135f90 legacy=0x7fc11413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 monclient: found mon.noname-b 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc118890640 1 -- 172.21.15.86:0/2426680658 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11409f9f0 con 0x7fc1140a0c70 2024-09-17T15:40:10.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.749+0000 7fc10bfff640 1 --2- 172.21.15.86:0/2426680658 >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 0x7fc114135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc118890640 1 -- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 msgr2=0x7fc1140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc118890640 1 --2- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 0x7fc1140a1050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc1100097c0 tx=0x7fc11001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd9687c91502d088 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 1 -- 172.21.15.86:0/2426680658 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100b0070 con 0x7fc1140a0c70 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: dump: 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.755 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 1 -- 172.21.15.86:0/2426680658 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1100ab600 con 0x7fc1140a0c70 2024-09-17T15:40:10.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 1 -- 172.21.15.86:0/2426680658 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100ab9d0 con 0x7fc1140a0c70 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 5 monclient: authenticate success, global_id 104522 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 4 set_mon_vals no callback set 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc10b7fe640 10 monclient: dump: 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 10 monclient: shutdown 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 -- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 msgr2=0x7fc1140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 0x7fc1140a1050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc1100097c0 tx=0x7fc11001c360 comp rx=0 tx=0).stop 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 -- 172.21.15.86:0/2426680658 shutdown_connections 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/2426680658 >> v2:172.21.15.203:3300/0 conn(0x7fc1140a1590 0x7fc114135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/2426680658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1140a0c70 0x7fc1140a1050 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 -- 172.21.15.86:0/2426680658 >> 172.21.15.86:0/2426680658 conn(0x7fc1140a2f40 msgr2=0x7fc1140a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 -- 172.21.15.86:0/2426680658 shutdown_connections 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.753+0000 7fc11aaf4640 1 -- 172.21.15.86:0/2426680658 wait complete. 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient: monmap: 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 Processor -- start 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 -- start start 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient: init 2024-09-17T15:40:10.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 0x7fc11412ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1140a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1140a1590 0x7fc11412f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1140a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114135f90 0x7fc114136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): picked mon.noname-c con 0x7fc114135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc11409ce60 con 0x7fc1140a0c70 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc11409cce0 con 0x7fc1140a1590 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc11409ff50 con 0x7fc114135f90 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc11aaf4640 10 monclient(hunting): authenticate will time out at 1743.725528s 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 0x7fc11412ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114135f90 0x7fc114136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 0x7fc11412ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48478/0 (socket says 172.21.15.86:48478) 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 -- 172.21.15.86:0/1968238882 learned_addr learned my addr 172.21.15.86:0/1968238882 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): get_auth_request con 0x7fc1140a0c70 auth_method 0 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): get_auth_request con 0x7fc114135f90 auth_method 0 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc119091640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc10bfff640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1140a1590 0x7fc11412f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc10bfff640 10 monclient(hunting): get_auth_request con 0x7fc1140a1590 auth_method 0 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient(hunting): handle_auth_done global_id 144135 payload 824 2024-09-17T15:40:10.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114135f90 msgr2=0x7fc114136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114135f90 0x7fc114136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1140a1590 msgr2=0x7fc11412f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc1140a1590 0x7fc11412f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 monclient: found mon.noname-a 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.757+0000 7fc118890640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc11413a9b0 con 0x7fc1140a0c70 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc118890640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 msgr2=0x7fc11412ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc118890640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 0x7fc11412ec40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1100ae040 tx=0x7fc110019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100b0040 con 0x7fc1140a0c70 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: dump: 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1100b9070 con 0x7fc1140a0c70 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 5 monclient: authenticate success, global_id 144135 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 monclient: _renew_subs 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc11413e350 con 0x7fc1140a0c70 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1100aaa10 con 0x7fc1140a0c70 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 10 monclient: dump: 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 monclient: _renew_subs 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10a7fc640 4 set_mon_vals no callback set 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc11413a110 con 0x7fc1140a0c70 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 0x7fc114137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc11409d4b0 con 0x7fc1141391b0 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc1100ab4c0 con 0x7fc1140a0c70 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 0x7fc114137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient: get_auth_request con 0x7fc1141391b0 auth_method 0 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc10003bb60 0x7fc10003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 10 monclient(hunting): handle_auth_done global_id 104528 payload 824 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(439..439 src has 213..439) ==== 4847+0+0 (secure 0 0 0) 0x7fc1100e9df0 con 0x7fc1140a0c70 2024-09-17T15:40:10.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc10bfff640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1140a1050 con 0x7fc1141391b0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc119091640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc10003bb60 0x7fc10003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.761+0000 7fc119091640 10 monclient: get_auth_request con 0x7fc10003bb60 auth_method 0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc10bfff640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 msgr2=0x7fc114137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc10bfff640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 0x7fc114137430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc104010790 tx=0x7fc104010c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10401a030 con 0x7fc1141391b0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 10 monclient: dump: 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc10400ebe0 con 0x7fc1141391b0 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 msgr2=0x7fc114137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 0x7fc114137430 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc104010790 tx=0x7fc104010c50 comp rx=0 tx=0).stop 2024-09-17T15:40:10.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 msgr2=0x7fc114137430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc1097fa640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1141391b0 0x7fc114137430 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc104010790 tx=0x7fc104010c50 comp rx=0 tx=0).stop 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.765+0000 7fc119091640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc10003bb60 0x7fc10003e020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc10c005500 tx=0x7fc10c005490 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc11aaf4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 0x7fc114137d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc11aaf4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc1141382c0 con 0x7fc114137970 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 0x7fc114137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient: get_auth_request con 0x7fc114137970 auth_method 0 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 10 monclient(hunting): handle_auth_done global_id 104531 payload 824 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.769+0000 7fc10bfff640 1 -- 172.21.15.86:0/1968238882 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc11413a350 con 0x7fc114137970 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc10bfff640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 msgr2=0x7fc114137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc10bfff640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 0x7fc114137d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc104008010 tx=0x7fc1040046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10400f040 con 0x7fc114137970 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 10 monclient: dump: 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fc104003620 con 0x7fc114137970 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 msgr2=0x7fc114137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 0x7fc114137d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc104008010 tx=0x7fc1040046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 msgr2=0x7fc114137d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc1097fa640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc114137970 0x7fc114137d80 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc104008010 tx=0x7fc1040046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 241648, 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.781 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.782 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.783 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:10.784 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.785 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.787 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:10.788 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.789 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.790 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:10.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.792 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:10.793 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc10003bb60 msgr2=0x7fc10003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc10003bb60 0x7fc10003e020 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc10c005500 tx=0x7fc10c005490 comp rx=0 tx=0).stop 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 10 monclient: shutdown 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 msgr2=0x7fc11412ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 --2- 172.21.15.86:0/1968238882 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1140a0c70 0x7fc11412ec40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc1100ae040 tx=0x7fc110019100 comp rx=0 tx=0).stop 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc119091640 1 -- 172.21.15.86:0/1968238882 reap_dead start 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc119091640 1 -- 172.21.15.86:0/1968238882 reap_dead start 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 shutdown_connections 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 >> 172.21.15.86:0/1968238882 conn(0x7fc1140a2f40 msgr2=0x7fc1140a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 shutdown_connections 2024-09-17T15:40:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.773+0000 7fc11aaf4640 1 -- 172.21.15.86:0/1968238882 wait complete. 2024-09-17T15:40:10.796 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:10.796 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:40:10.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: get_monmap_and_config 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: monmap: 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:10.908145+0000 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:10.908145+0000 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.904 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:10.905 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:10.905 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:10.905 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:10.905 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:10.905 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 Processor -- start 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 -- start start 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: init 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca40a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0ca4137ea0 0x7f0ca4138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca4137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca40a6280 con 0x7f0ca40a54c0 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca409c0a0 con 0x7f0ca4137ea0 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca409c380 con 0x7f0ca40a50e0 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0caa5e8640 10 monclient(hunting): authenticate will time out at 1743.873531s 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53752/0 (socket says 172.21.15.86:53752) 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48486/0 (socket says 172.21.15.86:48486) 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 -- 172.21.15.86:0/1933868077 learned_addr learned my addr 172.21.15.86:0/1933868077 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): get_auth_request con 0x7f0ca40a50e0 auth_method 0 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): get_auth_request con 0x7f0ca40a54c0 auth_method 0 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca8b85640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.203:3300/0 conn(0x7f0ca4137ea0 0x7f0ca4138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca8b85640 10 monclient(hunting): get_auth_request con 0x7f0ca4137ea0 auth_method 0 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca8b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca8b85640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca8b85640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_done global_id 104534 payload 274 2024-09-17T15:40:10.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 -- 172.21.15.86:0/1933868077 >> v2:172.21.15.203:3300/0 conn(0x7f0ca4137ea0 msgr2=0x7f0ca4138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.203:3300/0 conn(0x7f0ca4137ea0 0x7f0ca4138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 -- 172.21.15.86:0/1933868077 >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 msgr2=0x7f0ca40a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 monclient: found mon.noname-b 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca3fff640 1 -- 172.21.15.86:0/1933868077 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca409c660 con 0x7f0ca40a50e0 2024-09-17T15:40:10.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.905+0000 7f0ca37fe640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:10.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca3fff640 1 -- 172.21.15.86:0/1933868077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 msgr2=0x7f0ca409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/1933868077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ca406f890 tx=0x7f0c9401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f28fc7fbb0e94c24 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 1 -- 172.21.15.86:0/1933868077 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b1070 con 0x7f0ca40a50e0 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: dump: 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.918 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 1 -- 172.21.15.86:0/1933868077 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c940ac600 con 0x7f0ca40a50e0 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 5 monclient: authenticate success, global_id 104534 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 4 set_mon_vals no callback set 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 1 -- 172.21.15.86:0/1933868077 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940ac9d0 con 0x7f0ca40a50e0 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:10.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0ca2ffd640 10 monclient: dump: 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 10 monclient: shutdown 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- 172.21.15.86:0/1933868077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 msgr2=0x7f0ca409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/1933868077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ca406f890 tx=0x7f0c9401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- 172.21.15.86:0/1933868077 shutdown_connections 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.203:3300/0 conn(0x7f0ca4137ea0 0x7f0ca4138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/1933868077 >> v2:172.21.15.86:3300/0 conn(0x7f0ca40a54c0 0x7f0ca40a58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/1933868077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca40a50e0 0x7f0ca409cd60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- 172.21.15.86:0/1933868077 >> 172.21.15.86:0/1933868077 conn(0x7f0ca40aaae0 msgr2=0x7f0ca40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- 172.21.15.86:0/1933868077 shutdown_connections 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- 172.21.15.86:0/1933868077 wait complete. 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 10 monclient: build_initial_monmap 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 10 monclient: monmap: 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:10.921 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 Processor -- start 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 1 -- start start 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.909+0000 7f0caa5e8640 10 monclient: init 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca40a50e0 0x7f0ca41cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-b con 0x7f0ca40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:10.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-a con 0x7f0ca40a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca4137ea0 0x7f0ca413c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): picked mon.noname-c con 0x7f0ca4137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca40a6ae0 con 0x7f0ca40a54c0 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca40a6960 con 0x7f0ca40a50e0 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca4141b20 con 0x7f0ca4137ea0 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): authenticate will time out at 1743.880386s 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca4137ea0 0x7f0ca413c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48494/0 (socket says 172.21.15.86:48494) 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 -- 172.21.15.86:0/2801547980 learned_addr learned my addr 172.21.15.86:0/2801547980 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca4137ea0 0x7f0ca413c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53754/0 (socket says 172.21.15.86:53754) 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): get_auth_request con 0x7f0ca40a54c0 auth_method 0 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): get_auth_request con 0x7f0ca4137ea0 auth_method 0 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca8b85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient(hunting): handle_auth_done global_id 144141 payload 824 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca4137ea0 msgr2=0x7f0ca413c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ca4137ea0 0x7f0ca413c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca40a50e0 msgr2=0x7f0ca41cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca40a50e0 0x7f0ca41cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca40a50e0 0x7f0ca41cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 monclient: found mon.noname-a 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:10.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca41d0c30 con 0x7f0ca40a54c0 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 msgr2=0x7f0ca4137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca37fe640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800ebe0 tx=0x7f0c9800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98017070 con 0x7f0ca40a54c0 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: dump: 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9800ce40 con 0x7f0ca40a54c0 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 5 monclient: authenticate success, global_id 144141 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _renew_subs 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca40a18d0 con 0x7f0ca40a54c0 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98013690 con 0x7f0ca40a54c0 2024-09-17T15:40:10.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca0ff9640 10 monclient: dump: 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _renew_subs 2024-09-17T15:40:10.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca413dc50 con 0x7f0ca40a54c0 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0ca1ffb640 4 set_mon_vals no callback set 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 0x7f0ca413c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.913+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca406a0e0 con 0x7f0ca4137590 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0c9801a070 con 0x7f0ca40a54c0 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 0x7f0ca413c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient: get_auth_request con 0x7f0ca4137590 auth_method 0 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0c9003bb60 0x7f0c9003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(439..439 src has 213..439) ==== 4847+0+0 (secure 0 0 0) 0x7f0c9804ff20 con 0x7f0ca40a54c0 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca8b85640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0c9003bb60 0x7f0c9003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca8b85640 10 monclient: get_auth_request con 0x7f0c9003bb60 auth_method 0 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_done global_id 134191 payload 824 2024-09-17T15:40:10.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ca40a58c0 con 0x7f0ca4137590 2024-09-17T15:40:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 msgr2=0x7f0ca413c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 0x7f0ca413c860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ae000 tx=0x7f0c9401ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940b1040 con 0x7f0ca4137590 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca0ff9640 10 monclient: dump: 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.917+0000 7f0ca8b85640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0c9003bb60 0x7f0c9003e020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4120f80 tx=0x7f0c9c009820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c94009490 con 0x7f0ca4137590 2024-09-17T15:40:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:10.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 msgr2=0x7f0ca413c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 0x7f0ca413c860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ae000 tx=0x7f0c9401ac20 comp rx=0 tx=0).stop 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 msgr2=0x7f0ca413c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.921+0000 7f0ca0ff9640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca4137590 0x7f0ca413c860 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0c940ae000 tx=0x7f0c9401ac20 comp rx=0 tx=0).stop 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0caa5e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 0x7f0ca40a2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0caa5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca413d1b0 con 0x7f0ca413cda0 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 0x7f0ca40a2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient: get_auth_request con 0x7f0ca413cda0 auth_method 0 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:10.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 10 monclient(hunting): handle_auth_done global_id 134194 payload 824 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.925+0000 7f0ca3fff640 1 -- 172.21.15.86:0/2801547980 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ca4122770 con 0x7f0ca413cda0 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca3fff640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 msgr2=0x7f0ca40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca3fff640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 0x7f0ca40a2b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401aa90 tx=0x7f0c94019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c940c5070 con 0x7f0ca413cda0 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 10 monclient: dump: 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.937 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f0c940c1c10 con 0x7f0ca413cda0 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 msgr2=0x7f0ca40a2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 0x7f0ca40a2b80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401aa90 tx=0x7f0c94019180 comp rx=0 tx=0).stop 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 msgr2=0x7f0ca40a2b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:10.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca0ff9640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ca413cda0 0x7f0ca40a2b80 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c9401aa90 tx=0x7f0c94019180 comp rx=0 tx=0).stop 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 75299, 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:10.939 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.940 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.941 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:10.942 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:10.943 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.944 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.945 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.946 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.947 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.949 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.950 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:10.951 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:10.952 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:10.952 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:10.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:10.952 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:10.952 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0c9003bb60 msgr2=0x7f0c9003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0c9003bb60 0x7f0c9003e020 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4120f80 tx=0x7f0c9c009820 comp rx=0 tx=0).stop 2024-09-17T15:40:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 10 monclient: shutdown 2024-09-17T15:40:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:10.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 msgr2=0x7f0ca4137050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800ebe0 tx=0x7f0c9800c520 comp rx=0 tx=0).stop 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0ca8b85640 1 -- 172.21.15.86:0/2801547980 reap_dead start 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 shutdown_connections 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 --2- 172.21.15.86:0/2801547980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ca40a54c0 0x7f0ca4137050 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 >> 172.21.15.86:0/2801547980 conn(0x7f0ca40aaae0 msgr2=0x7f0ca40a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 shutdown_connections 2024-09-17T15:40:10.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:10.929+0000 7f0caa5e8640 1 -- 172.21.15.86:0/2801547980 wait complete. 2024-09-17T15:40:10.954 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:10.954 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:40:10.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:40:11.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.061+0000 7f0d4337c640 10 monclient: get_monmap_and_config 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.061+0000 7f0d4337c640 10 monclient: build_initial_monmap 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.061+0000 7f0d4337c640 10 monclient: monmap: 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:11.067647+0000 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:11.067647+0000 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:11.064 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:11.065 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:11.065 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:11.065 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 Processor -- start 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 -- start start 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient: init 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0d3c0689d0 0x7f0d3c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c0689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d3c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c13aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3c09c8b0 con 0x7f0d3c068db0 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0d3c074500 con 0x7f0d3c13aff0 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0d3c09cb90 con 0x7f0d3c0689d0 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d4337c640 10 monclient(hunting): authenticate will time out at 1744.032993s 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48498/0 (socket says 172.21.15.86:48498) 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 -- 172.21.15.86:0/669887481 learned_addr learned my addr 172.21.15.86:0/669887481 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): get_auth_request con 0x7f0d3c13aff0 auth_method 0 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41118640 1 --2- 172.21.15.86:0/669887481 >> v2:172.21.15.203:3300/0 conn(0x7f0d3c0689d0 0x7f0d3c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41118640 10 monclient(hunting): get_auth_request con 0x7f0d3c0689d0 auth_method 0 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41118640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41118640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient(hunting): handle_auth_done global_id 144144 payload 274 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 -- 172.21.15.86:0/669887481 >> v2:172.21.15.203:3300/0 conn(0x7f0d3c0689d0 msgr2=0x7f0d3c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 --2- 172.21.15.86:0/669887481 >> v2:172.21.15.203:3300/0 conn(0x7f0d3c0689d0 0x7f0d3c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 -- 172.21.15.86:0/669887481 >> v1:172.21.15.203:6789/0 conn(0x7f0d3c068db0 legacy=0x7f0d3c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient: _finish_hunting 0 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 monclient: found mon.noname-a 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.065+0000 7f0d41919640 1 -- 172.21.15.86:0/669887481 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c09ce70 con 0x7f0d3c13aff0 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d41919640 1 -- 172.21.15.86:0/669887481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c13aff0 msgr2=0x7f0d3c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d41919640 1 --2- 172.21.15.86:0/669887481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c069940 tx=0x7f0d3401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2638c63f19dc35b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 1 -- 172.21.15.86:0/669887481 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c09c2d0 con 0x7f0d3c13aff0 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: dump: 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:11.078 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: _finish_auth 0 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 1 -- 172.21.15.86:0/669887481 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d340a5730 con 0x7f0d3c13aff0 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 5 monclient: authenticate success, global_id 144144 2024-09-17T15:40:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 4 set_mon_vals no callback set 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 1 -- 172.21.15.86:0/669887481 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d340a5b00 con 0x7f0d3c13aff0 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d33fff640 10 monclient: dump: 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:11.080 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 10 monclient: shutdown 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- 172.21.15.86:0/669887481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c13aff0 msgr2=0x7f0d3c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:11.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 --2- 172.21.15.86:0/669887481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c069940 tx=0x7f0d3401c890 comp rx=0 tx=0).stop 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- 172.21.15.86:0/669887481 shutdown_connections 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 --2- 172.21.15.86:0/669887481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c13aff0 0x7f0d3c13d3e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 --2- 172.21.15.86:0/669887481 >> v2:172.21.15.203:3300/0 conn(0x7f0d3c0689d0 0x7f0d3c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- 172.21.15.86:0/669887481 >> 172.21.15.86:0/669887481 conn(0x7f0d3c0a6440 msgr2=0x7f0d3c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- 172.21.15.86:0/669887481 shutdown_connections 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- 172.21.15.86:0/669887481 wait complete. 2024-09-17T15:40:11.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 10 monclient: build_initial_monmap 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 10 monclient: monmap: 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 Processor -- start 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 1 -- start start 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.069+0000 7f0d4337c640 10 monclient: init 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c0689d0 0x7f0d3c1cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d3c068db0 0x7f0d3c132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c1405f0 con 0x7f0d3c0689d0 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3c140470 con 0x7f0d3c068db0 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0d3c140770 con 0x7f0d3c13aff0 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient(hunting): authenticate will time out at 1744.040196s 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c0689d0 0x7f0d3c1cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53766/0 (socket says 172.21.15.86:53766) 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 -- 172.21.15.86:0/771743304 learned_addr learned my addr 172.21.15.86:0/771743304 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): get_auth_request con 0x7f0d3c13aff0 auth_method 0 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): get_auth_request con 0x7f0d3c0689d0 auth_method 0 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41118640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d40917640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d3c068db0 0x7f0d3c132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d40917640 10 monclient(hunting): get_auth_request con 0x7f0d3c068db0 auth_method 0 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d40917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:11.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d40917640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d40917640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient(hunting): handle_auth_done global_id 104540 payload 824 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 -- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d3c068db0 msgr2=0x7f0d3c132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d3c068db0 0x7f0d3c132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 -- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c0689d0 msgr2=0x7f0d3c1cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c0689d0 0x7f0d3c1cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient: _finish_hunting 0 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 monclient: found mon.noname-c 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 -- 172.21.15.86:0/771743304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3c11d4d0 con 0x7f0d3c13aff0 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 -- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 msgr2=0x7f0d3c070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:11.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d41919640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d3401af40 tx=0x7f0d34019630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d340bc020 con 0x7f0d3c13aff0 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: dump: 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:11.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:11.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d340b8880 con 0x7f0d3c13aff0 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 5 monclient: authenticate success, global_id 104540 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _renew_subs 2024-09-17T15:40:11.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d3c09d460 con 0x7f0d3c13aff0 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d340b8c50 con 0x7f0d3c13aff0 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d31ffb640 10 monclient: dump: 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d337fe640 4 set_mon_vals no callback set 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _renew_subs 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.073+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3c1ce870 con 0x7f0d3c13aff0 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0d340bf020 con 0x7f0d3c13aff0 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d4337c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d4337c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:11.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d4337c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3c06f840 con 0x7f0d3c13aff0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d31ffb640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 0x7f0d2003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(439..439 src has 213..439) ==== 4847+0+0 (secure 0 0 0) 0x7f0d340eaf80 con 0x7f0d3c13aff0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d41118640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 0x7f0d2003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d41118640 10 monclient: get_auth_request con 0x7f0d2003bb60 auth_method 0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.077+0000 7f0d41118640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 0x7f0d2003e020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0d38007a60 tx=0x7f0d38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:11.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.081+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0d340c6030 con 0x7f0d3c13aff0 2024-09-17T15:40:11.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.081+0000 7f0d31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:11.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.081+0000 7f0d31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:11.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.301+0000 7f0d4337c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:40:11.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.301+0000 7f0d4337c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:40:11.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.301+0000 7f0d4337c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:11.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.301+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0d3c06d4f0 con 0x7f0d3c13aff0 2024-09-17T15:40:11.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=1,health=12,kv=11,logm=54} crc {auth=451444947,config=1228700967,health=3234334190,kv=3591778543,logm=911204396}) 2024-09-17T15:40:11.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=21,config=2,health=12,kv=10,logm=55} crc {auth=1576054076,config=3541143443,health=3234334190,kv=1466466540,logm=1363940028}) 2024-09-17T15:40:11.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=1,health=12,kv=11,logm=54} crc {auth=451444947,config=1228700967,health=3234334190,kv=3591778543,logm=911204396}) 2024-09-17T15:40:11.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,health=12,kv=11,logm=56} crc {auth=2852773879,health=3234334190,kv=3591778543,logm=3841406486}) 2024-09-17T15:40:11.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d31ffb640 1 -- 172.21.15.86:0/771743304 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0d340165a0 con 0x7f0d3c13aff0 2024-09-17T15:40:11.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:40:11.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1955: done (3 left) 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1956: done (2 left) 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1957: done (1 left) 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1958: done (0 left) 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1959: write oid 47 current snap is 220 2024-09-17T15:40:11.312 INFO:tasks.rados.rados.0.smithi203.stdout:1959: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 msgr2=0x7f0d2003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 0x7f0d2003e020 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0d38007a60 tx=0x7f0d38008040 comp rx=0 tx=0).stop 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 10 monclient: shutdown 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 msgr2=0x7f0d3c070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0d3401af40 tx=0x7f0d34019630 comp rx=0 tx=0).stop 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 shutdown_connections 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d2003bb60 0x7f0d2003e020 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0d3c13aff0 0x7f0d3c070e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d3c068db0 0x7f0d3c132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 --2- 172.21.15.86:0/771743304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0d3c0689d0 0x7f0d3c1cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 >> 172.21.15.86:0/771743304 conn(0x7f0d3c0a6440 msgr2=0x7f0d3c13bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 shutdown_connections 2024-09-17T15:40:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:11.309+0000 7f0d4337c640 1 -- 172.21.15.86:0/771743304 wait complete. 2024-09-17T15:40:11.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872493774}) 2024-09-17T15:40:11.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792207329}) 2024-09-17T15:40:11.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=872493774}) 2024-09-17T15:40:11.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.313+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3303509486}) 2024-09-17T15:40:11.321 INFO:tasks.rados.rados.0.smithi203.stdout:1959: writing smithi20328086-47 from 548093 to 1335267 tid 1 2024-09-17T15:40:11.328 INFO:tasks.rados.rados.0.smithi203.stdout:1959: writing smithi20328086-47 from 1970678 to 2420580 tid 2 2024-09-17T15:40:11.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:40:11.337 INFO:tasks.rados.rados.0.smithi203.stdout:1959: writing smithi20328086-47 from 3045966 to 3727103 tid 3 2024-09-17T15:40:11.337 INFO:tasks.rados.rados.0.smithi203.stdout:1960: snap_remove snap 188 2024-09-17T15:40:11.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642376564}) 2024-09-17T15:40:11.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1116074566}) 2024-09-17T15:40:11.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642376564}) 2024-09-17T15:40:11.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855177288}) 2024-09-17T15:40:11.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356483457}) 2024-09-17T15:40:11.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2714078340}) 2024-09-17T15:40:11.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.381+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356483457}) 2024-09-17T15:40:11.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1043150741}) 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637869216}) 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244829608}) 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637869216}) 2024-09-17T15:40:11.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.385+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096940125}) 2024-09-17T15:40:11.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526211337}) 2024-09-17T15:40:11.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2752987684}) 2024-09-17T15:40:11.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526211337}) 2024-09-17T15:40:11.389 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.389+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553135309}) 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638806665}) 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895172728}) 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638806665}) 2024-09-17T15:40:11.611 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2349704637}) 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456923237}) 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4119473449}) 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456923237}) 2024-09-17T15:40:11.613 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74888398}) 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192838857}) 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039002178}) 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192838857}) 2024-09-17T15:40:11.616 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.613+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866745717}) 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690081109}) 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627809666}) 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1690081109}) 2024-09-17T15:40:11.617 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373718011}) 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903509654}) 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=92774840}) 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903509654}) 2024-09-17T15:40:11.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.617+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315071048}) 2024-09-17T15:40:11.621 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.621 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291562577}) 2024-09-17T15:40:11.621 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484429816}) 2024-09-17T15:40:11.621 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.621 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291562577}) 2024-09-17T15:40:11.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460706842}) 2024-09-17T15:40:11.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1896781221}) 2024-09-17T15:40:11.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=480024680}) 2024-09-17T15:40:11.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1896781221}) 2024-09-17T15:40:11.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.621+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814643128}) 2024-09-17T15:40:11.625 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.625 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767155200}) 2024-09-17T15:40:11.625 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059212406}) 2024-09-17T15:40:11.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1767155200}) 2024-09-17T15:40:11.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256956725}) 2024-09-17T15:40:11.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436101318}) 2024-09-17T15:40:11.627 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2910636875}) 2024-09-17T15:40:11.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436101318}) 2024-09-17T15:40:11.628 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.625+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030597521}) 2024-09-17T15:40:11.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881387774}) 2024-09-17T15:40:11.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404050053}) 2024-09-17T15:40:11.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881387774}) 2024-09-17T15:40:11.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386393147}) 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258738873}) 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2832333654}) 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.629+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258738873}) 2024-09-17T15:40:11.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3553181789}) 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815949481}) 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920133981}) 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=815949481}) 2024-09-17T15:40:11.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.633+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=100906183}) 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787483017}) 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3552776948}) 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787483017}) 2024-09-17T15:40:11.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051477568}) 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999637694}) 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2606024074}) 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999637694}) 2024-09-17T15:40:11.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.637+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465750107}) 2024-09-17T15:40:11.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913679967}) 2024-09-17T15:40:11.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960872347}) 2024-09-17T15:40:11.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913679967}) 2024-09-17T15:40:11.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533473439}) 2024-09-17T15:40:11.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133528211}) 2024-09-17T15:40:11.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819784508}) 2024-09-17T15:40:11.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1133528211}) 2024-09-17T15:40:11.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.641+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=478791665}) 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844435256}) 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818136770}) 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844435256}) 2024-09-17T15:40:11.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600997132}) 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822927315}) 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1179368063}) 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822927315}) 2024-09-17T15:40:11.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.645+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817919805}) 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317755640}) 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427620441}) 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317755640}) 2024-09-17T15:40:11.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1283666795}) 2024-09-17T15:40:11.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259399155}) 2024-09-17T15:40:11.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101392918}) 2024-09-17T15:40:11.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259399155}) 2024-09-17T15:40:11.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.649+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106357166}) 2024-09-17T15:40:11.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928475919}) 2024-09-17T15:40:11.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2604083584}) 2024-09-17T15:40:11.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928475919}) 2024-09-17T15:40:11.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271143719}) 2024-09-17T15:40:11.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384660027}) 2024-09-17T15:40:11.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548097885}) 2024-09-17T15:40:11.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384660027}) 2024-09-17T15:40:11.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.653+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407374936}) 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440694535}) 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1597911752}) 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440694535}) 2024-09-17T15:40:11.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866595315}) 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.657+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160662452}) 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993466354}) 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160662452}) 2024-09-17T15:40:11.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3488519306}) 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951835970}) 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490023241}) 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951835970}) 2024-09-17T15:40:11.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.661+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266885847}) 2024-09-17T15:40:11.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714967614}) 2024-09-17T15:40:11.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2581065987}) 2024-09-17T15:40:11.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1714967614}) 2024-09-17T15:40:11.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2493886972}) 2024-09-17T15:40:11.666 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.666 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916384397}) 2024-09-17T15:40:11.666 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3703470122}) 2024-09-17T15:40:11.666 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.666 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916384397}) 2024-09-17T15:40:11.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.665+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465210650}) 2024-09-17T15:40:11.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410027143}) 2024-09-17T15:40:11.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3870799895}) 2024-09-17T15:40:11.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.671 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410027143}) 2024-09-17T15:40:11.671 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.669+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767286889}) 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673714474}) 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842938871}) 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673714474}) 2024-09-17T15:40:11.678 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.677+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2389005666}) 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238205987}) 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3354306217}) 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238205987}) 2024-09-17T15:40:11.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.685+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3106148426}) 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1960: done (1 left) 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1961: write oid 36 current snap is 220 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1959: finishing write tid 1 to smithi20328086-47 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1959: finishing write tid 2 to smithi20328086-47 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1959: finishing write tid 3 to smithi20328086-47 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1959: finishing write tid 4 to smithi20328086-47 2024-09-17T15:40:11.688 INFO:tasks.rados.rados.0.smithi203.stdout:1961: seq_num 700 ranges {564900=578776,1629160=709913,3073389=492409} 2024-09-17T15:40:11.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629291781}) 2024-09-17T15:40:11.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741225234}) 2024-09-17T15:40:11.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629291781}) 2024-09-17T15:40:11.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.693+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696588907}) 2024-09-17T15:40:11.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128770958}) 2024-09-17T15:40:11.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2219948047}) 2024-09-17T15:40:11.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128770958}) 2024-09-17T15:40:11.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.697+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=4145253445,mdsmap=1467291461}) 2024-09-17T15:40:11.702 INFO:tasks.rados.rados.0.smithi203.stdout:1961: writing smithi20328086-36 from 564900 to 1143676 tid 1 2024-09-17T15:40:11.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {logm=1981301052,mdsmap=2815901553,mgr=333540546,mgr_metadata=2984965231,mgrstat=3322172481,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=2907016926,osd_snap=2400773625}) 2024-09-17T15:40:11.707 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6,mon_config_key=12,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=57} crc {logm=1981301052,mdsmap=2815901553,mgr=2700465641,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3313419438,mon_config_key=1691377683,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2907016926,osd_snap=4293598693}) 2024-09-17T15:40:11.707 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.707 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=8,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {logm=1981301052,mdsmap=2815901553,mgr=333540546,mgr_metadata=2984965231,mgrstat=3322172481,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=4043941930,osd_pg_creating=2907016926,osd_snap=2400773625}) 2024-09-17T15:40:11.707 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.705+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {logm=1981301052,mdsmap=2815901553,mgr=991077062,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3002693232,mon_config_key=578014828,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2907016926,osd_snap=1225839451}) 2024-09-17T15:40:11.713 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=3785079060,osdmap=453536208}) 2024-09-17T15:40:11.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=804051714,osdmap=3876273012}) 2024-09-17T15:40:11.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=3785079060,osdmap=453536208}) 2024-09-17T15:40:11.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.713+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=495857306,osdmap=1793369378}) 2024-09-17T15:40:11.717 INFO:tasks.rados.rados.0.smithi203.stdout:1961: writing smithi20328086-36 from 1629160 to 2339073 tid 2 2024-09-17T15:40:11.731 INFO:tasks.rados.rados.0.smithi203.stdout:1961: writing smithi20328086-36 from 3073389 to 3565798 tid 3 2024-09-17T15:40:11.731 INFO:tasks.rados.rados.0.smithi203.stdout:1962: rollback oid 7 current snap is 220 2024-09-17T15:40:11.731 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 201 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: finishing write tid 5 to smithi20328086-47 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: oid 47 updating version 0 to 813 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: oid 47 version 813 is already newer than 811 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: oid 47 version 813 is already newer than 812 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: oid 47 version 813 is already newer than 810 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 813 (ObjNum 699 snap 220 seq_num 699) dirty exists 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: left oid 47 (ObjNum 699 snap 220 seq_num 699) 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1961: finishing write tid 1 to smithi20328086-36 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1959: done (2 left) 2024-09-17T15:40:11.732 INFO:tasks.rados.rados.0.smithi203.stdout:1963: snap_create 2024-09-17T15:40:11.736 INFO:tasks.rados.rados.0.smithi203.stdout:1961: finishing write tid 2 to smithi20328086-36 2024-09-17T15:40:11.736 INFO:tasks.rados.rados.0.smithi203.stdout:1962: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:40:11.737 INFO:tasks.rados.rados.0.smithi203.stdout:1961: finishing write tid 3 to smithi20328086-36 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: finishing write tid 4 to smithi20328086-36 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1962: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 488 (ObjNum 612 snap 185 seq_num 612) dirty exists 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: finishing write tid 5 to smithi20328086-36 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: oid 36 updating version 0 to 961 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: oid 36 version 961 is already newer than 960 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: oid 36 version 961 is already newer than 959 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: oid 36 version 961 is already newer than 954 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 961 (ObjNum 700 snap 220 seq_num 700) dirty exists 2024-09-17T15:40:11.738 INFO:tasks.rados.rados.0.smithi203.stdout:1961: left oid 36 (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1359362339}) 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2293670684}) 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1359362339}) 2024-09-17T15:40:11.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:11.833+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1786533246}) 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2207386136}) 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=614257146}) 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2207386136}) 2024-09-17T15:40:12.009 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.009+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4145781737}) 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1062504557}) 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=74} crc {osdmap=2305564918}) 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=74} crc {osdmap=1062504557}) 2024-09-17T15:40:12.018 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:12.017+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=73} crc {osdmap=3466798739}) 2024-09-17T15:40:12.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:40:12.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:12.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: get_monmap_and_config 2024-09-17T15:40:12.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: build_initial_monmap 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: monmap: 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:12.461455+0000 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:12.461455+0000 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:12.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 Processor -- start 2024-09-17T15:40:12.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 -- start start 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: init 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44500689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44500694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f445006f840 0x7f445013e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): picked mon.noname-c con 0x7f445006f840 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4450074470 con 0x7f44500694c0 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f445009c8b0 con 0x7f44500689d0 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f445009cb90 con 0x7f445006f840 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f445768b640 10 monclient(hunting): authenticate will time out at 1745.427122s 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4454c26640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f44500694c0 0x7f445013aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41632/0 (socket says 172.21.15.86:41632) 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4454c26640 1 -- 172.21.15.86:0/4165264441 learned_addr learned my addr 172.21.15.86:0/4165264441 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 1 --2- 172.21.15.86:0/4165264441 >> v2:172.21.15.203:3300/0 conn(0x7f445006f840 0x7f445013e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): get_auth_request con 0x7f445006f840 auth_method 0 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient(hunting): handle_auth_done global_id 134203 payload 274 2024-09-17T15:40:12.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 1 -- 172.21.15.86:0/4165264441 >> v1:172.21.15.203:6789/0 conn(0x7f44500689d0 legacy=0x7f4450068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 1 -- 172.21.15.86:0/4165264441 >> v1:172.21.15.86:6789/0 conn(0x7f44500694c0 legacy=0x7f445013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient: _finish_hunting 0 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 1 monclient: found mon.noname-c 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:40:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.457+0000 7f4455c28640 1 -- 172.21.15.86:0/4165264441 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445009ce70 con 0x7f445006f840 2024-09-17T15:40:12.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f4455c28640 1 -- 172.21.15.86:0/4165264441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f445006f840 msgr2=0x7f445013e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f4455c28640 1 --2- 172.21.15.86:0/4165264441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f445006f840 0x7f445013e720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44400097c0 tx=0x7f444001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=175101cf31580e0d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 1 -- 172.21.15.86:0/4165264441 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44400b0070 con 0x7f445006f840 2024-09-17T15:40:12.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:12.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: dump: 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: _finish_auth 0 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:12.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 1 -- 172.21.15.86:0/4165264441 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44400ab600 con 0x7f445006f840 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 1 -- 172.21.15.86:0/4165264441 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445009c2d0 con 0x7f445006f840 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f444ffff640 10 monclient: dump: 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.468 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.468 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 5 monclient: authenticate success, global_id 134203 2024-09-17T15:40:12.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:12.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 4 set_mon_vals no callback set 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: shutdown 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- 172.21.15.86:0/4165264441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f445006f840 msgr2=0x7f445013e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 --2- 172.21.15.86:0/4165264441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f445006f840 0x7f445013e720 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44400097c0 tx=0x7f444001c360 comp rx=0 tx=0).stop 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- 172.21.15.86:0/4165264441 shutdown_connections 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 --2- 172.21.15.86:0/4165264441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f445006f840 0x7f445013e720 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- 172.21.15.86:0/4165264441 >> 172.21.15.86:0/4165264441 conn(0x7f44500a6440 msgr2=0x7f44500a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- 172.21.15.86:0/4165264441 shutdown_connections 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- 172.21.15.86:0/4165264441 wait complete. 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: build_initial_monmap 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: monmap: 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:12.474 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 Processor -- start 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 -- start start 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: init 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.465+0000 7f445768b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): picked mon.noname-c con 0x7f44500689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44500694c0 0x7f4450132480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): picked mon.noname-b con 0x7f44500694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 0x7f44501329c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): picked mon.noname-a con 0x7f445006f840 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f445009dbc0 con 0x7f445006f840 2024-09-17T15:40:12.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f445009da40 con 0x7f44500694c0 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f445009dd40 con 0x7f44500689d0 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient(hunting): authenticate will time out at 1745.436091s 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53782/0 (socket says 172.21.15.86:53782) 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 0x7f44501329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 -- 172.21.15.86:0/4180933087 learned_addr learned my addr 172.21.15.86:0/4180933087 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): get_auth_request con 0x7f44500689d0 auth_method 0 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): get_auth_request con 0x7f445006f840 auth_method 0 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4454c26640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44500694c0 0x7f4450132480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4454c26640 10 monclient(hunting): get_auth_request con 0x7f44500694c0 auth_method 0 2024-09-17T15:40:12.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4454c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4454c26640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4454c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient(hunting): handle_auth_done global_id 104543 payload 824 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 -- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44500694c0 msgr2=0x7f4450132480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44500694c0 0x7f4450132480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 -- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 msgr2=0x7f44501329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 0x7f44501329c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient: _finish_hunting 0 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 monclient: found mon.noname-c 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 -- 172.21.15.86:0/4180933087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4450135fc0 con 0x7f44500689d0 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455c28640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 0x7f44501329c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 -- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 msgr2=0x7f44501366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f4455427640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7f0 tx=0x7f444800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448016020 con 0x7f44500689d0 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: dump: 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.479 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444800ac60 con 0x7f44500689d0 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448012660 con 0x7f44500689d0 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444dffb640 10 monclient: dump: 2024-09-17T15:40:12.480 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 5 monclient: authenticate success, global_id 104543 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient: _renew_subs 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f444f7fe640 4 set_mon_vals no callback set 2024-09-17T15:40:12.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4450073940 con 0x7f44500689d0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient: _renew_subs 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.469+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f445009d460 con 0x7f44500689d0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f445768b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f445768b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f445768b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4450068db0 con 0x7f44500689d0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4448019020 con 0x7f44500689d0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f444dffb640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 0x7f443c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(441..441 src has 213..441) ==== 4875+0+0 (secure 0 0 0) 0x7f444804ff50 con 0x7f44500689d0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f4454c26640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 0x7f443c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f4454c26640 10 monclient: get_auth_request con 0x7f443c03bb60 auth_method 0 2024-09-17T15:40:12.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.473+0000 7f4454c26640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 0x7f443c03e020 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4444007fd0 tx=0x7f4444009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.477+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4448020030 con 0x7f44500689d0 2024-09-17T15:40:12.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.477+0000 7f444dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:12.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.477+0000 7f444dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1961: done (2 left) 2024-09-17T15:40:12.695 INFO:tasks.rados.rados.0.smithi203.stdout:1962: done (1 left) 2024-09-17T15:40:12.696 INFO:tasks.rados.rados.0.smithi203.stdout:1963: done (0 left) 2024-09-17T15:40:12.696 INFO:tasks.rados.rados.0.smithi203.stdout:1964: rollback oid 17 current snap is 221 2024-09-17T15:40:12.696 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 209 2024-09-17T15:40:12.696 INFO:tasks.rados.rados.0.smithi203.stdout:1965: snap_remove snap 199 2024-09-17T15:40:12.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.713+0000 7f445768b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:12.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.713+0000 7f445768b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:12.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4450130cf0 con 0x7f44500689d0 2024-09-17T15:40:12.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f444dffb640 1 -- 172.21.15.86:0/4180933087 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f444800ae10 con 0x7f44500689d0 2024-09-17T15:40:12.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f444dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:12.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f444dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:12.717 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 msgr2=0x7f443c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 0x7f443c03e020 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4444007fd0 tx=0x7f4444009040 comp rx=0 tx=0).stop 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 10 monclient: shutdown 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 msgr2=0x7f44501366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7f0 tx=0x7f444800ccb0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 shutdown_connections 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f443c03bb60 0x7f443c03e020 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f445006f840 0x7f44501329c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44500694c0 0x7f4450132480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 --2- 172.21.15.86:0/4180933087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44500689d0 0x7f44501366c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 >> 172.21.15.86:0/4180933087 conn(0x7f44500a6440 msgr2=0x7f445012fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:12.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 shutdown_connections 2024-09-17T15:40:12.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.717+0000 7f445768b640 1 -- 172.21.15.86:0/4180933087 wait complete. 2024-09-17T15:40:12.735 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:40:12.735 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:40:12.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:40:12.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:40:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: get_monmap_and_config 2024-09-17T15:40:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: build_initial_monmap 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: monmap: 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:12.852572+0000 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:12.852572+0000 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:12.849 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 Processor -- start 2024-09-17T15:40:12.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 -- start start 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: init 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2334134d90 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 0x7f233413cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f23340a1130 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f233413d4a0 0x7f233413f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f233413d4a0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2334135880 con 0x7f2334134d90 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f23340a21b0 con 0x7f233413d4a0 2024-09-17T15:40:12.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2334073b10 con 0x7f23340a1130 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f233b8c8640 10 monclient(hunting): authenticate will time out at 1745.817992s 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 0x7f233413cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339664640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2334134d90 0x7f2334135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41634/0 (socket says 172.21.15.86:41634) 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339664640 1 -- 172.21.15.86:0/543239860 learned_addr learned my addr 172.21.15.86:0/543239860 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): get_auth_request con 0x7f23340a1130 auth_method 0 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339e65640 1 --2- 172.21.15.86:0/543239860 >> v2:172.21.15.203:3300/0 conn(0x7f233413d4a0 0x7f233413f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339e65640 10 monclient(hunting): get_auth_request con 0x7f233413d4a0 auth_method 0 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339e65640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2339e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient(hunting): handle_auth_done global_id 104546 payload 274 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 -- 172.21.15.86:0/543239860 >> v2:172.21.15.203:3300/0 conn(0x7f233413d4a0 msgr2=0x7f233413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 --2- 172.21.15.86:0/543239860 >> v2:172.21.15.203:3300/0 conn(0x7f233413d4a0 0x7f233413f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 -- 172.21.15.86:0/543239860 >> v1:172.21.15.86:6789/0 conn(0x7f2334134d90 legacy=0x7f2334135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient: _finish_hunting 0 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 monclient: found mon.noname-b 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.849+0000 7f2338e63640 1 -- 172.21.15.86:0/543239860 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23340a1510 con 0x7f23340a1130 2024-09-17T15:40:12.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f2338e63640 1 -- 172.21.15.86:0/543239860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 msgr2=0x7f233413cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f2338e63640 1 --2- 172.21.15.86:0/543239860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 0x7f233413cf60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23300097c0 tx=0x7f233001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c22516ef1991ea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 1 -- 172.21.15.86:0/543239860 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23300b0070 con 0x7f23340a1130 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: dump: 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: _finish_auth 0 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 5 monclient: authenticate success, global_id 104546 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 1 -- 172.21.15.86:0/543239860 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23300ab600 con 0x7f23340a1130 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 4 set_mon_vals no callback set 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 1 -- 172.21.15.86:0/543239860 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23300ab9d0 con 0x7f23340a1130 2024-09-17T15:40:12.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f232bfff640 10 monclient: dump: 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 10 monclient: shutdown 2024-09-17T15:40:12.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 -- 172.21.15.86:0/543239860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 msgr2=0x7f233413cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 --2- 172.21.15.86:0/543239860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 0x7f233413cf60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f23300097c0 tx=0x7f233001c330 comp rx=0 tx=0).stop 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 -- 172.21.15.86:0/543239860 shutdown_connections 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 --2- 172.21.15.86:0/543239860 >> v2:172.21.15.203:3300/0 conn(0x7f233413d4a0 0x7f233413f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 --2- 172.21.15.86:0/543239860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23340a1130 0x7f233413cf60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 -- 172.21.15.86:0/543239860 >> 172.21.15.86:0/543239860 conn(0x7f23340a2f40 msgr2=0x7f23340a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 -- 172.21.15.86:0/543239860 shutdown_connections 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.853+0000 7f233b8c8640 1 -- 172.21.15.86:0/543239860 wait complete. 2024-09-17T15:40:12.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient: build_initial_monmap 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient: monmap: 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.864 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 Processor -- start 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 -- start start 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient: init 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23340a1130 0x7f23341cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f23340a1130 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 0x7f2334133f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2334134d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f233413d4a0 0x7f233413acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f233413d4a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2334142770 con 0x7f2334134d90 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23341425f0 con 0x7f23340a1130 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f23341428f0 con 0x7f233413d4a0 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f233b8c8640 10 monclient(hunting): authenticate will time out at 1745.825008s 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 0x7f2334133f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f233413d4a0 0x7f233413acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 0x7f2334133f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48528/0 (socket says 172.21.15.86:48528) 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 -- 172.21.15.86:0/1383058603 learned_addr learned my addr 172.21.15.86:0/1383058603 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): get_auth_request con 0x7f2334134d90 auth_method 0 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): get_auth_request con 0x7f233413d4a0 auth_method 0 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339664640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23340a1130 0x7f23341cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339664640 10 monclient(hunting): get_auth_request con 0x7f23340a1130 auth_method 0 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339664640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339664640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient(hunting): handle_auth_done global_id 144153 payload 824 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f233413d4a0 msgr2=0x7f233413acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f233413d4a0 0x7f233413acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23340a1130 msgr2=0x7f23341cfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23340a1130 0x7f23341cfdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient: _finish_hunting 0 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 monclient: found mon.noname-a 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233411e2e0 con 0x7f2334134d90 2024-09-17T15:40:12.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2339e65640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f233413d4a0 0x7f233413acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 msgr2=0x7f2334133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2338e63640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 0x7f2334133f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f233001cbd0 tx=0x7f233001ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.857+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23300b0050 con 0x7f2334134d90 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:12.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: dump: 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 5 monclient: authenticate success, global_id 144153 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23300bc070 con 0x7f2334134d90 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 monclient: _renew_subs 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f233411e5c0 con 0x7f2334134d90 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23300b5c80 con 0x7f2334134d90 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: dump: 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.875 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 monclient: _renew_subs 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f232b7fe640 4 set_mon_vals no callback set 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23340a58b0 con 0x7f2334134d90 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 0x7f233402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f233411e000 con 0x7f2334134450 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 0x7f233402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient: get_auth_request con 0x7f2334134450 auth_method 0 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f23300ab4c0 con 0x7f2334134d90 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 10 monclient(hunting): handle_auth_done global_id 144156 payload 824 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2334135170 con 0x7f2334134450 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f231803bbb0 0x7f231803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7f23300ea760 con 0x7f2334134d90 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339e65640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f231803bbb0 0x7f231803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339e65640 10 monclient: get_auth_request con 0x7f231803bbb0 auth_method 0 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 msgr2=0x7f233402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2339664640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 0x7f233402d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f232400e500 tx=0x7f232400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2324004e40 con 0x7f2334134450 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: dump: 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.878 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2324018070 con 0x7f2334134450 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 msgr2=0x7f233402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 0x7f233402d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f232400e500 tx=0x7f232400e9c0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 msgr2=0x7f233402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.861+0000 7f2329ffb640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134450 0x7f233402d450 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f232400e500 tx=0x7f232400e9c0 comp rx=0 tx=0).stop 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.865+0000 7f2339e65640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f231803bbb0 0x7f231803e070 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f23340a1d80 tx=0x7f232c009770 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f233b8c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f233b8c8640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 0x7f233402dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f233b8c8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2334137160 con 0x7f233402d990 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 0x7f233402dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient: get_auth_request con 0x7f233402d990 auth_method 0 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:12.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 10 monclient(hunting): handle_auth_done global_id 144159 payload 824 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.869+0000 7f2339664640 1 -- 172.21.15.86:0/1383058603 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23341d0c70 con 0x7f233402d990 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2339664640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 msgr2=0x7f233402dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2339664640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 0x7f233402dd70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f232400e4d0 tx=0x7f2324004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23240064e0 con 0x7f233402d990 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 10 monclient: dump: 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f23240064e0 con 0x7f233402d990 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 msgr2=0x7f233402dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 0x7f233402dd70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f232400e4d0 tx=0x7f2324004950 comp rx=0 tx=0).stop 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 msgr2=0x7f233402dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:12.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2329ffb640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f233402d990 0x7f233402dd70 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f232400e4d0 tx=0x7f2324004950 comp rx=0 tx=0).stop 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 31215, 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:12.883 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:12.884 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:12.885 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:12.886 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:12.887 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:12.888 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:12.888 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:12.888 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:12.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:12.890 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:12.893 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:12.894 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:12.895 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f231803bbb0 msgr2=0x7f231803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f231803bbb0 0x7f231803e070 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f23340a1d80 tx=0x7f232c009770 comp rx=0 tx=0).stop 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 10 monclient: shutdown 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 msgr2=0x7f2334133f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 --2- 172.21.15.86:0/1383058603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2334134d90 0x7f2334133f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f233001cbd0 tx=0x7f233001ab70 comp rx=0 tx=0).stop 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2339e65640 1 -- 172.21.15.86:0/1383058603 reap_dead start 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f2339e65640 1 -- 172.21.15.86:0/1383058603 reap_dead start 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 shutdown_connections 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 >> 172.21.15.86:0/1383058603 conn(0x7f23340a2f40 msgr2=0x7f233409ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 shutdown_connections 2024-09-17T15:40:12.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:12.873+0000 7f233b8c8640 1 -- 172.21.15.86:0/1383058603 wait complete. 2024-09-17T15:40:12.898 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:12.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:40:12.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:40:13.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: get_monmap_and_config 2024-09-17T15:40:13.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: monmap: 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:13.016525+0000 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:13.016525+0000 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:13.013 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 Processor -- start 2024-09-17T15:40:13.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 -- start start 2024-09-17T15:40:13.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: init 2024-09-17T15:40:13.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce00a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1ce00a5350 0x7f1ce00a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce00a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 0x7f1ce01381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce0137d30 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce009d430 con 0x7f1ce00a5350 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce00a68d0 con 0x7f1ce0137d30 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce00a64a0 con 0x7f1ce00a4f70 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce7e5c640 10 monclient(hunting): authenticate will time out at 1745.981958s 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1ce00a5350 0x7f1ce00a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1ce00a5350 0x7f1ce00a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48550/0 (socket says 172.21.15.86:48550) 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 -- 172.21.15.86:0/2796419219 learned_addr learned my addr 172.21.15.86:0/2796419219 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): get_auth_request con 0x7f1ce00a5350 auth_method 0 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request con 0x7f1ce00a4f70 auth_method 0 2024-09-17T15:40:13.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce63f9640 1 --2- 172.21.15.86:0/2796419219 >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 0x7f1ce01381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_done global_id 144162 payload 274 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 -- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 msgr2=0x7f1ce009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce009cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 -- 172.21.15.86:0/2796419219 >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 msgr2=0x7f1ce01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/2796419219 >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 0x7f1ce01381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce53f7640 1 -- 172.21.15.86:0/2796419219 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce009c230 con 0x7f1ce00a5350 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce63f9640 10 monclient: get_auth_request con 0x7f1ce0137d30 auth_method 0 2024-09-17T15:40:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce63f9640 1 --2- 172.21.15.86:0/2796419219 >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 0x7f1ce01381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:13.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.013+0000 7f1ce5bf8640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce009cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce53f7640 1 -- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 msgr2=0x7f1ce00a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce00a5750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce009e420 tx=0x7f1cd001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc98552d37a25a6b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 1 -- 172.21.15.86:0/2796419219 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00b2070 con 0x7f1ce00a5350 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: dump: 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.022 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 1 -- 172.21.15.86:0/2796419219 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd00ad680 con 0x7f1ce00a5350 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 1 -- 172.21.15.86:0/2796419219 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00ada50 con 0x7f1ce00a5350 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 5 monclient: authenticate success, global_id 144162 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 4 set_mon_vals no callback set 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce4bf6640 10 monclient: dump: 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 10 monclient: shutdown 2024-09-17T15:40:13.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 msgr2=0x7f1ce00a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce00a5750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1ce009e420 tx=0x7f1cd001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/2796419219 shutdown_connections 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/2796419219 >> v2:172.21.15.203:3300/0 conn(0x7f1ce0137d30 0x7f1ce01381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce00a5750 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/2796419219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce009cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/2796419219 >> 172.21.15.86:0/2796419219 conn(0x7f1ce00a7240 msgr2=0x7f1ce00a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/2796419219 shutdown_connections 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.021+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/2796419219 wait complete. 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient: monmap: 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:13.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 Processor -- start 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 -- start start 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient: init 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:13.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce0135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce00a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce00a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ce0137d30 0x7f1ce01d0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce0137d30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce006f980 con 0x7f1ce00a5350 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce006f800 con 0x7f1ce0137d30 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce01419b0 con 0x7f1ce00a4f70 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce7e5c640 10 monclient(hunting): authenticate will time out at 1745.993296s 2024-09-17T15:40:13.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce0135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48556/0 (socket says 172.21.15.86:48556) 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce0135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53816/0 (socket says 172.21.15.86:53816) 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 -- 172.21.15.86:0/4123245071 learned_addr learned my addr 172.21.15.86:0/4123245071 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): get_auth_request con 0x7f1ce00a5350 auth_method 0 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request con 0x7f1ce00a4f70 auth_method 0 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce63f9640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ce0137d30 0x7f1ce01d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce63f9640 10 monclient(hunting): get_auth_request con 0x7f1ce0137d30 auth_method 0 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce63f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce63f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce63f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient(hunting): handle_auth_done global_id 144165 payload 824 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 msgr2=0x7f1ce0135450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce00a4f70 0x7f1ce0135450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ce0137d30 msgr2=0x7f1ce01d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ce0137d30 0x7f1ce01d0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.025+0000 7f1ce53f7640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce013c290 con 0x7f1ce00a5350 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce53f7640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 msgr2=0x7f1ce01d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce53f7640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001cc80 tx=0x7f1cd0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00b2040 con 0x7f1ce00a5350 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: dump: 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.034 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd00bb070 con 0x7f1ce00a5350 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 5 monclient: authenticate success, global_id 144165 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 monclient: _renew_subs 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce013dd50 con 0x7f1ce00a5350 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00b6440 con 0x7f1ce00a5350 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 10 monclient: dump: 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.036 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.037 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd7fff640 4 set_mon_vals no callback set 2024-09-17T15:40:13.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 monclient: _renew_subs 2024-09-17T15:40:13.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce0135990 con 0x7f1ce00a5350 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1cd00b65e0 con 0x7f1ce00a5350 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 0x7f1ce01cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce00a1760 con 0x7f1ce01ccc60 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 0x7f1ce01cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient: get_auth_request con 0x7f1ce01ccc60 auth_method 0 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1cc403bbb0 0x7f1cc403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7f1cd00e9640 con 0x7f1ce00a5350 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_done global_id 104558 payload 824 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce5bf8640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ce006ef30 con 0x7f1ce01ccc60 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce63f9640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1cc403bbb0 0x7f1cc403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.029+0000 7f1ce63f9640 10 monclient: get_auth_request con 0x7f1cc403bbb0 auth_method 0 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1ce5bf8640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 msgr2=0x7f1ce01cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1ce5bf8640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 0x7f1ce01cd040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0119d0 tx=0x7f1cdc011e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc014440 con 0x7f1ce01ccc60 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 10 monclient: dump: 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1cdc014440 con 0x7f1ce01ccc60 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 msgr2=0x7f1ce01cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 0x7f1ce01cd040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0119d0 tx=0x7f1cdc011e90 comp rx=0 tx=0).stop 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 msgr2=0x7f1ce01cd040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1cd67fc640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01ccc60 0x7f1ce01cd040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.033+0000 7f1ce63f9640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1cc403bbb0 0x7f1cc403e070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1ce013e100 tx=0x7f1cd8007500 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce7e5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 0x7f1ce009f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce7e5c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce0135c20 con 0x7f1ce01cd580 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 0x7f1ce009f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient: get_auth_request con 0x7f1ce01cd580 auth_method 0 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 10 monclient(hunting): handle_auth_done global_id 104561 payload 824 2024-09-17T15:40:13.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.037+0000 7f1ce5bf8640 1 -- 172.21.15.86:0/4123245071 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ce00a4990 con 0x7f1ce01cd580 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce5bf8640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 msgr2=0x7f1ce009f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce5bf8640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 0x7f1ce009f670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0119d0 tx=0x7f1cdc0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc022b90 con 0x7f1ce01cd580 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 10 monclient: dump: 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.045 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f1cdc022b90 con 0x7f1ce01cd580 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 msgr2=0x7f1ce009f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 0x7f1ce009f670 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0119d0 tx=0x7f1cdc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 msgr2=0x7f1ce009f670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1cd67fc640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ce01cd580 0x7f1ce009f670 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc0119d0 tx=0x7f1cdc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.046 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 243916, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:13.047 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.048 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:13.050 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.051 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.053 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.054 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.055 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.056 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.057 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.059 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:13.059 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1cc403bbb0 msgr2=0x7f1cc403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1cc403bbb0 0x7f1cc403e070 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1ce013e100 tx=0x7f1cd8007500 comp rx=0 tx=0).stop 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 10 monclient: shutdown 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 msgr2=0x7f1ce01d01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001cc80 tx=0x7f1cd0019180 comp rx=0 tx=0).stop 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce63f9640 1 -- 172.21.15.86:0/4123245071 reap_dead start 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 shutdown_connections 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 --2- 172.21.15.86:0/4123245071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ce00a5350 0x7f1ce01d01e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 >> 172.21.15.86:0/4123245071 conn(0x7f1ce00a7240 msgr2=0x7f1ce00a9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 shutdown_connections 2024-09-17T15:40:13.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.041+0000 7f1ce7e5c640 1 -- 172.21.15.86:0/4123245071 wait complete. 2024-09-17T15:40:13.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:13.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:40:13.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: get_monmap_and_config 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: monmap: 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:13.177813+0000 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:13.177813+0000 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:13.174 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:13.175 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:13.175 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 Processor -- start 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 -- start start 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: init 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0c70 0x7fcc500a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-b con 0x7fcc500a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc500a1590 0x7fcc50135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-a con 0x7fcc500a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc50135f90 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:13.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc500a2130 con 0x7fcc50135f90 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcc500a2410 con 0x7fcc500a1590 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcc500a26f0 con 0x7fcc500a0c70 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc54add640 10 monclient(hunting): authenticate will time out at 1746.143351s 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc500a1590 0x7fcc50135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc500a1590 0x7fcc50135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48568/0 (socket says 172.21.15.86:48568) 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/3986702605 learned_addr learned my addr 172.21.15.86:0/3986702605 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): get_auth_request con 0x7fcc500a1590 auth_method 0 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0c70 0x7fcc500a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): get_auth_request con 0x7fcc500a0c70 auth_method 0 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_done global_id 144168 payload 274 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0c70 msgr2=0x7fcc500a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0c70 0x7fcc500a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/3986702605 >> v1:172.21.15.203:6789/0 conn(0x7fcc50135f90 legacy=0x7fcc5013cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:13.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.173+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/3986702605 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5009f9f0 con 0x7fcc500a1590 2024-09-17T15:40:13.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 msgr2=0x7fcc50135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc50135a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6181fefb350fd1d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 1 -- 172.21.15.86:0/3986702605 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440b1070 con 0x7fcc500a1590 2024-09-17T15:40:13.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: dump: 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 1 -- 172.21.15.86:0/3986702605 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc440ac600 con 0x7fcc500a1590 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 1 -- 172.21.15.86:0/3986702605 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440ac9d0 con 0x7fcc500a1590 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 5 monclient: authenticate success, global_id 144168 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 4 set_mon_vals no callback set 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4d59a640 10 monclient: dump: 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.185 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: shutdown 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 msgr2=0x7fcc50135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc50135a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4401c420 comp rx=0 tx=0).stop 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- 172.21.15.86:0/3986702605 shutdown_connections 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc50135a50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- 172.21.15.86:0/3986702605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc500a0c70 0x7fcc500a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- 172.21.15.86:0/3986702605 >> 172.21.15.86:0/3986702605 conn(0x7fcc500a2f40 msgr2=0x7fcc500a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- 172.21.15.86:0/3986702605 shutdown_connections 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- 172.21.15.86:0/3986702605 wait complete. 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: monmap: 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:13.187 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 Processor -- start 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- start start 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: init 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc500a0c70 0x7fcc501d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-b con 0x7fcc500a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc501d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-a con 0x7fcc500a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc50135f90 0x7fcc501d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): picked mon.noname-c con 0x7fcc50135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5009ce60 con 0x7fcc500a1590 2024-09-17T15:40:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5009cce0 con 0x7fcc500a0c70 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5009ff50 con 0x7fcc50135f90 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc54add640 10 monclient(hunting): authenticate will time out at 1746.150447s 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc501d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc50135f90 0x7fcc501d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc501d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48584/0 (socket says 172.21.15.86:48584) 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/836034453 learned_addr learned my addr 172.21.15.86:0/836034453 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): get_auth_request con 0x7fcc500a1590 auth_method 0 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): get_auth_request con 0x7fcc50135f90 auth_method 0 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc500a0c70 0x7fcc501d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient(hunting): handle_auth_done global_id 144171 payload 824 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc50135f90 msgr2=0x7fcc501d0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc50135f90 0x7fcc501d0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc500a0c70 msgr2=0x7fcc501d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc500a0c70 0x7fcc501d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5009f230 con 0x7fcc500a1590 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4e59c640 10 monclient: get_auth_request con 0x7fcc500a0c70 auth_method 0 2024-09-17T15:40:13.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.181+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc500a0c70 0x7fcc501d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4dd9b640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 msgr2=0x7fcc501d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4dd9b640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc501d0690 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc4401c250 tx=0x7fcc4401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440bc070 con 0x7fcc500a1590 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: dump: 2024-09-17T15:40:13.194 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 5 monclient: authenticate success, global_id 144171 2024-09-17T15:40:13.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 monclient: _renew_subs 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc440b7430 con 0x7fcc500a1590 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5006f990 con 0x7fcc500a1590 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc440b1040 con 0x7fcc500a1590 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 10 monclient: dump: 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.196 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 monclient: _renew_subs 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4cd99640 4 set_mon_vals no callback set 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5013c6e0 con 0x7fcc500a1590 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 0x7fcc5012edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc500a1050 con 0x7fcc5012e9e0 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcc440b63d0 con 0x7fcc500a1590 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 0x7fcc5012edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcc3003bbb0 0x7fcc3003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient: get_auth_request con 0x7fcc5012e9e0 auth_method 0 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7fcc440e9640 con 0x7fcc500a1590 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4ed9d640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcc3003bbb0 0x7fcc3003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4ed9d640 10 monclient: get_auth_request con 0x7fcc3003bbb0 auth_method 0 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_done global_id 134218 payload 824 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.185+0000 7fcc4e59c640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc500008e0 con 0x7fcc5012e9e0 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc4e59c640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 msgr2=0x7fcc5012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 0x7fcc5012edc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc40010820 tx=0x7fcc40010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc4ed9d640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcc3003bbb0 0x7fcc3003e070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcc5013d470 tx=0x7fcc48003ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000f040 con 0x7fcc5012e9e0 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 10 monclient: dump: 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.199 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcc40004020 con 0x7fcc5012e9e0 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 msgr2=0x7fcc5012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 0x7fcc5012edc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc40010820 tx=0x7fcc40010ce0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 msgr2=0x7fcc5012edc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:13.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.189+0000 7fcc3effd640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012e9e0 0x7fcc5012edc0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc40010820 tx=0x7fcc40010ce0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc54add640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:13.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc54add640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 0x7fcc50071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc54add640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5006d4f0 con 0x7fcc5012f300 2024-09-17T15:40:13.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 0x7fcc50071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient: get_auth_request con 0x7fcc5012f300 auth_method 0 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 10 monclient(hunting): handle_auth_done global_id 134221 payload 824 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 1 -- 172.21.15.86:0/836034453 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcc5012f6e0 con 0x7fcc5012f300 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 msgr2=0x7fcc50071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.197+0000 7fcc4e59c640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 0x7fcc50071200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc40010820 tx=0x7fcc400046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40003d80 con 0x7fcc5012f300 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 10 monclient: dump: 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fcc40003d80 con 0x7fcc5012f300 2024-09-17T15:40:13.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 msgr2=0x7fcc50071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 0x7fcc50071200 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc40010820 tx=0x7fcc400046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 msgr2=0x7fcc50071200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc3effd640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcc5012f300 0x7fcc50071200 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:13.205 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 126, 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 77567, 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 6, 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.206 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.207 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:13.208 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:13.209 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.210 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:13.211 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.212 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:13.213 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:13.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:13.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:13.218 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:13.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:13.218 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:13.218 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:13.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcc3003bbb0 msgr2=0x7fcc3003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcc3003bbb0 0x7fcc3003e070 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcc5013d470 tx=0x7fcc48003ca0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 10 monclient: shutdown 2024-09-17T15:40:13.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 msgr2=0x7fcc501d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 --2- 172.21.15.86:0/836034453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc500a1590 0x7fcc501d0690 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcc4401c250 tx=0x7fcc4401ac60 comp rx=0 tx=0).stop 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc4ed9d640 1 -- 172.21.15.86:0/836034453 reap_dead start 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc4ed9d640 1 -- 172.21.15.86:0/836034453 reap_dead start 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 shutdown_connections 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 >> 172.21.15.86:0/836034453 conn(0x7fcc500a2f40 msgr2=0x7fcc500a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 shutdown_connections 2024-09-17T15:40:13.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.201+0000 7fcc54add640 1 -- 172.21.15.86:0/836034453 wait complete. 2024-09-17T15:40:13.220 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:13.221 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:40:13.221 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:40:13.221 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:40:13.221 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:40:13.221 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:40:13.221 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:40:13.240 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:40:13.240 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:40:13.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: get_monmap_and_config 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: monmap: 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:13.357089+0000 2024-09-17T15:40:13.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:13.357089+0000 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:13.354 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 Processor -- start 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 -- start start 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: init 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 0x7fc21006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2100689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc210068db0 0x7fc210069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-a con 0x7fc210068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc210135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc21009c8b0 con 0x7fc210135ae0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc21009cb90 con 0x7fc210068db0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc21009ce70 con 0x7fc2100689d0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc2148d1640 10 monclient(hunting): authenticate will time out at 1746.322569s 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc210068db0 0x7fc210069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20ed9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc210135ae0 0x7fc210135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41644/0 (socket says 172.21.15.86:41644) 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20ed9d640 1 -- 172.21.15.86:0/3134220911 learned_addr learned my addr 172.21.15.86:0/3134220911 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20e59c640 1 -- 172.21.15.86:0/3134220911 >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 msgr2=0x7fc21006f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:13.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): get_auth_request con 0x7fc210068db0 auth_method 0 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20e59c640 1 --2- 172.21.15.86:0/3134220911 >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 0x7fc21006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient(hunting): handle_auth_done global_id 144174 payload 274 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 -- 172.21.15.86:0/3134220911 >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 msgr2=0x7fc21006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 --2- 172.21.15.86:0/3134220911 >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 0x7fc21006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 -- 172.21.15.86:0/3134220911 >> v1:172.21.15.86:6789/0 conn(0x7fc210135ae0 legacy=0x7fc210135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.353+0000 7fc20dd9b640 1 -- 172.21.15.86:0/3134220911 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc210069740 con 0x7fc210068db0 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20dd9b640 1 -- 172.21.15.86:0/3134220911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210068db0 msgr2=0x7fc210069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20dd9b640 1 --2- 172.21.15.86:0/3134220911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210068db0 0x7fc210069200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc210069a70 tx=0x7fc20001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36634ab987f14135 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 1 -- 172.21.15.86:0/3134220911 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2000b2070 con 0x7fc210068db0 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: dump: 2024-09-17T15:40:13.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 1 -- 172.21.15.86:0/3134220911 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2000a5800 con 0x7fc210068db0 2024-09-17T15:40:13.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc2148d1640 5 monclient: authenticate success, global_id 144174 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc2148d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc2148d1640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 1 -- 172.21.15.86:0/3134220911 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2000a5bd0 con 0x7fc210068db0 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc2148d1640 4 set_mon_vals no callback set 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.357+0000 7fc20d59a640 10 monclient: dump: 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.363 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: shutdown 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- 172.21.15.86:0/3134220911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210068db0 msgr2=0x7fc210069200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- 172.21.15.86:0/3134220911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210068db0 0x7fc210069200 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc210069a70 tx=0x7fc20001c8a0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- 172.21.15.86:0/3134220911 shutdown_connections 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- 172.21.15.86:0/3134220911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210068db0 0x7fc210069200 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- 172.21.15.86:0/3134220911 >> v2:172.21.15.203:3300/0 conn(0x7fc2100689d0 0x7fc21006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- 172.21.15.86:0/3134220911 >> 172.21.15.86:0/3134220911 conn(0x7fc2100a6440 msgr2=0x7fc2100a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- 172.21.15.86:0/3134220911 shutdown_connections 2024-09-17T15:40:13.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- 172.21.15.86:0/3134220911 wait complete. 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: build_initial_monmap 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: monmap: 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 Processor -- start 2024-09-17T15:40:13.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- start start 2024-09-17T15:40:13.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: init 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2100689d0 0x7fc21013c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2100689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 0x7fc210134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-b con 0x7fc210068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): picked mon.noname-a con 0x7fc210135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc210140a40 con 0x7fc210135ae0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc21009c2d0 con 0x7fc210068db0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc210140bc0 con 0x7fc2100689d0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc2148d1640 10 monclient(hunting): authenticate will time out at 1746.330074s 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48604/0 (socket says 172.21.15.86:48604) 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20dd9b640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 msgr2=0x7fc210134f10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 -- 172.21.15.86:0/2666579803 learned_addr learned my addr 172.21.15.86:0/2666579803 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): get_auth_request con 0x7fc210135ae0 auth_method 0 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20dd9b640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 0x7fc210134f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2100689d0 0x7fc21013c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): get_auth_request con 0x7fc2100689d0 auth_method 0 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient(hunting): handle_auth_done global_id 144177 payload 824 2024-09-17T15:40:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 -- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2100689d0 msgr2=0x7fc21013c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2100689d0 0x7fc21013c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 -- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 msgr2=0x7fc210134f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 0x7fc210134f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.361+0000 7fc20ed9d640 1 -- 172.21.15.86:0/2666579803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc21013cad0 con 0x7fc210135ae0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc20ed9d640 1 -- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 msgr2=0x7fc210135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc20ed9d640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c3f0 tx=0x7fc20800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc208017070 con 0x7fc210135ae0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:13.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: dump: 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:13.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2080070d0 con 0x7fc210135ae0 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 5 monclient: authenticate success, global_id 144177 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _renew_subs 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2101218e0 con 0x7fc210135ae0 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2080074c0 con 0x7fc210135ae0 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 10 monclient: dump: 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc207fff640 4 set_mon_vals no callback set 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _renew_subs 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc210120cd0 con 0x7fc210135ae0 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc21006f840 con 0x7fc210135ae0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc20801a070 con 0x7fc210135ae0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 0x7fc1f003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7fc20804f380 con 0x7fc210135ae0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc20e59c640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 0x7fc1f003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.365+0000 7fc20e59c640 10 monclient: get_auth_request con 0x7fc1f003bb60 auth_method 0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.369+0000 7fc20e59c640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 0x7fc1f003e020 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80093d0 tx=0x7fc1f8009360 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.369+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc20801bbb0 con 0x7fc210135ae0 2024-09-17T15:40:13.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.369+0000 7fc206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:13.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.369+0000 7fc206ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:13.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc2148d1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:13.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc2148d1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:13.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc2148d1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:13.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc21006d4f0 con 0x7fc210135ae0 2024-09-17T15:40:13.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc206ffd640 1 -- 172.21.15.86:0/2666579803 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc20804f080 con 0x7fc210135ae0 2024-09-17T15:40:13.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc206ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:13.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.597+0000 7fc206ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:13.600 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 msgr2=0x7fc1f003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 0x7fc1f003e020 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc1f80093d0 tx=0x7fc1f8009360 comp rx=0 tx=0).stop 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 10 monclient: shutdown 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 msgr2=0x7fc210135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc20800c3f0 tx=0x7fc20800c8b0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 shutdown_connections 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc1f003bb60 0x7fc1f003e020 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc210135ae0 0x7fc210135450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc210068db0 0x7fc210134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 --2- 172.21.15.86:0/2666579803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2100689d0 0x7fc21013c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 >> 172.21.15.86:0/2666579803 conn(0x7fc2100a6440 msgr2=0x7fc210070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 shutdown_connections 2024-09-17T15:40:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:13.601+0000 7fc2148d1640 1 -- 172.21.15.86:0/2666579803 wait complete. 2024-09-17T15:40:16.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: get_monmap_and_config 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: build_initial_monmap 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: monmap: 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:16.750082+0000 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:16.750082+0000 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:16.746 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:16.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 Processor -- start 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 -- start start 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: init 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-b con 0x7f1f840a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f840a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 0x7f1f8413f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8413d280 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f840a6f40 con 0x7f1f840a5450 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:16.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1f840a65e0 con 0x7f1f8413d280 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8409c2a0 con 0x7f1f840a5070 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f88afe640 10 monclient(hunting): authenticate will time out at 1749.715501s 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:53866/0 (socket says 172.21.15.86:53866) 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f81d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1f840a5450 0x7f1f840a5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41650/0 (socket says 172.21.15.86:41650) 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 1 -- 172.21.15.86:0/2502063289 learned_addr learned my addr 172.21.15.86:0/2502063289 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): get_auth_request con 0x7f1f840a5070 auth_method 0 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 1 --2- 172.21.15.86:0/2502063289 >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 0x7f1f8413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request con 0x7f1f8413d280 auth_method 0 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.745+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 10 monclient(hunting): handle_auth_done global_id 104573 payload 274 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 1 -- 172.21.15.86:0/2502063289 >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 msgr2=0x7f1f8413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 1 --2- 172.21.15.86:0/2502063289 >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 0x7f1f8413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 1 -- 172.21.15.86:0/2502063289 >> v1:172.21.15.86:6789/0 conn(0x7f1f840a5450 legacy=0x7f1f840a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:16.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 10 monclient: _finish_hunting 0 2024-09-17T15:40:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 1 monclient: found mon.noname-b 2024-09-17T15:40:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f8259c640 1 -- 172.21.15.86:0/2502063289 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8409c580 con 0x7f1f840a5070 2024-09-17T15:40:16.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.749+0000 7f1f82d9d640 1 --2- 172.21.15.86:0/2502063289 >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 0x7f1f8413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8259c640 1 -- 172.21.15.86:0/2502063289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 msgr2=0x7f1f8409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8259c640 1 --2- 172.21.15.86:0/2502063289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009870 tx=0x7f1f7801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19d0ac21be6783f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:16.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 1 -- 172.21.15.86:0/2502063289 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f780b0070 con 0x7f1f840a5070 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: dump: 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.757 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: _finish_auth 0 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: _check_auth_tickets 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 1 -- 172.21.15.86:0/2502063289 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f780ab600 con 0x7f1f840a5070 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 5 monclient: authenticate success, global_id 104573 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 4 set_mon_vals no callback set 2024-09-17T15:40:16.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 1 -- 172.21.15.86:0/2502063289 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f780ab9d0 con 0x7f1f840a5070 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8159a640 10 monclient: dump: 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: shutdown 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- 172.21.15.86:0/2502063289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 msgr2=0x7f1f8409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2502063289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009870 tx=0x7f1f7801c4b0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- 172.21.15.86:0/2502063289 shutdown_connections 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2502063289 >> v2:172.21.15.86:3300/0 conn(0x7f1f8413d280 0x7f1f8413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2502063289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5070 0x7f1f8409cf60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- 172.21.15.86:0/2502063289 >> 172.21.15.86:0/2502063289 conn(0x7f1f840742a0 msgr2=0x7f1f840746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- 172.21.15.86:0/2502063289 shutdown_connections 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- 172.21.15.86:0/2502063289 wait complete. 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: build_initial_monmap 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: monmap: 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:16.760 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:16.761 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 Processor -- start 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- start start 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: init 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 0x7f1f84137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-b con 0x7f1f840a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-c con 0x7f1f840a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8413d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1f84142760 con 0x7f1f8413d280 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f841425e0 con 0x7f1f840a5070 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1f841428e0 con 0x7f1f840a5450 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f88afe640 10 monclient(hunting): authenticate will time out at 1749.723238s 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f81d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:16.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8259c640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 msgr2=0x7f1f84137ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f81d9b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 msgr2=0x7f1f840a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f81d9b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 msgr2=0x7f1f840a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f81d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f8259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 0x7f1f84137ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f81d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48620/0 (socket says 172.21.15.86:48620) 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 -- 172.21.15.86:0/2846599085 learned_addr learned my addr 172.21.15.86:0/2846599085 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request con 0x7f1f8413d280 auth_method 0 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:16.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient(hunting): handle_auth_done global_id 144183 payload 824 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 -- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 msgr2=0x7f1f840a2a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 -- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 msgr2=0x7f1f84137ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 0x7f1f84137ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 monclient: found mon.noname-a 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:16.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.753+0000 7f1f82d9d640 1 -- 172.21.15.86:0/2846599085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f840a1860 con 0x7f1f8413d280 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f82d9d640 1 -- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 msgr2=0x7f1f840a2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f82d9d640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00efc0 tx=0x7f1f7c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c018030 con 0x7f1f8413d280 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: dump: 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:16.768 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f7c0039a0 con 0x7f1f8413d280 2024-09-17T15:40:16.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c003d70 con 0x7f1f8413d280 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 5 monclient: authenticate success, global_id 144183 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _renew_subs 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f84122660 con 0x7f1f8413d280 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f80d99640 4 set_mon_vals no callback set 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 10 monclient: dump: 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:16.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _renew_subs 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8413d8d0 con 0x7f1f8413d280 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:16.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f8409cf60 con 0x7f1f8413d280 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1f7c01b070 con 0x7f1f8413d280 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 0x7f1f6403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7f1f7c050d90 con 0x7f1f8413d280 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f8259c640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 0x7f1f6403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.757+0000 7f1f8259c640 10 monclient: get_auth_request con 0x7f1f6403bb60 auth_method 0 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.761+0000 7f1f8259c640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 0x7f1f6403e020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1f8406e260 tx=0x7f1f78008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.761+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1f7c022070 con 0x7f1f8413d280 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.761+0000 7f1f72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:16.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.761+0000 7f1f72ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:16.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f88afe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:16.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f88afe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:16.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f88afe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:16.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1f840008e0 con 0x7f1f8413d280 2024-09-17T15:40:16.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f72ffd640 1 -- 172.21.15.86:0/2846599085 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1f840a1860 con 0x7f1f8413d280 2024-09-17T15:40:16.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:16.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.977+0000 7f1f72ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:16.979 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 msgr2=0x7f1f6403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 0x7f1f6403e020 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1f8406e260 tx=0x7f1f78008040 comp rx=0 tx=0).stop 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 10 monclient: shutdown 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 msgr2=0x7f1f840a2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00efc0 tx=0x7f1f7c009f90 comp rx=0 tx=0).stop 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 shutdown_connections 2024-09-17T15:40:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1f6403bb60 0x7f1f6403e020 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f8413d280 0x7f1f840a2fa0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f840a5450 0x7f1f840a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 --2- 172.21.15.86:0/2846599085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f840a5070 0x7f1f84137ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 >> 172.21.15.86:0/2846599085 conn(0x7f1f840742a0 msgr2=0x7f1f840a0410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 shutdown_connections 2024-09-17T15:40:16.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:16.981+0000 7f1f88afe640 1 -- 172.21.15.86:0/2846599085 wait complete. 2024-09-17T15:40:19.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:20.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 10 monclient: get_monmap_and_config 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 10 monclient: build_initial_monmap 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 10 monclient: monmap: 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:20.130696+0000 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:20.130696+0000 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:20.127 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:20.128 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:20.128 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:20.128 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 1 Processor -- start 2024-09-17T15:40:20.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 1 -- start start 2024-09-17T15:40:20.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.125+0000 7f15e9825640 10 monclient: init 2024-09-17T15:40:20.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15e40689d0 0x7f15e406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-a con 0x7f15e40689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15e4068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 0x7f15e413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-c con 0x7f15e413aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15e409c8b0 con 0x7f15e4068db0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f15e4074500 con 0x7f15e40689d0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f15e409cb90 con 0x7f15e413aff0 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:20.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e9825640 10 monclient(hunting): authenticate will time out at 1753.096256s 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15e40689d0 0x7f15e406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e27fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f15e4068db0 0x7f15e4069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49086/0 (socket says 172.21.15.86:49086) 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e37fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 msgr2=0x7f15e413d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e27fc640 1 -- 172.21.15.86:0/1335066289 learned_addr learned my addr 172.21.15.86:0/1335066289 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e37fe640 1 --2- 172.21.15.86:0/1335066289 >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 0x7f15e413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e1ffb640 1 -- 172.21.15.86:0/1335066289 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): get_auth_request con 0x7f15e40689d0 auth_method 0 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:20.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_done global_id 144186 payload 274 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 -- 172.21.15.86:0/1335066289 >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 msgr2=0x7f15e413d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 --2- 172.21.15.86:0/1335066289 >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 0x7f15e413d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 -- 172.21.15.86:0/1335066289 >> v1:172.21.15.86:6789/0 conn(0x7f15e4068db0 legacy=0x7f15e4069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.129+0000 7f15e2ffd640 1 -- 172.21.15.86:0/1335066289 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e409ce70 con 0x7f15e40689d0 2024-09-17T15:40:20.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e2ffd640 1 -- 172.21.15.86:0/1335066289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e40689d0 msgr2=0x7f15e406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:20.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e2ffd640 1 --2- 172.21.15.86:0/1335066289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e40689d0 0x7f15e406f840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15e4069a70 tx=0x7f15dc01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2ff5af2698a06e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:20.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 1 -- 172.21.15.86:0/1335066289 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0b0070 con 0x7f15e40689d0 2024-09-17T15:40:20.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:20.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:20.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: dump: 2024-09-17T15:40:20.135 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:20.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 5 monclient: authenticate success, global_id 144186 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 1 -- 172.21.15.86:0/1335066289 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15dc0ab600 con 0x7f15e40689d0 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 1 -- 172.21.15.86:0/1335066289 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e409c2d0 con 0x7f15e40689d0 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 4 set_mon_vals no callback set 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e1ffb640 10 monclient: dump: 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.137 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 10 monclient: shutdown 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 -- 172.21.15.86:0/1335066289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e40689d0 msgr2=0x7f15e406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 --2- 172.21.15.86:0/1335066289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e40689d0 0x7f15e406f840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f15e4069a70 tx=0x7f15dc01c320 comp rx=0 tx=0).stop 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 -- 172.21.15.86:0/1335066289 shutdown_connections 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 --2- 172.21.15.86:0/1335066289 >> v2:172.21.15.203:3300/0 conn(0x7f15e413aff0 0x7f15e413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 --2- 172.21.15.86:0/1335066289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e40689d0 0x7f15e406f840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 -- 172.21.15.86:0/1335066289 >> 172.21.15.86:0/1335066289 conn(0x7f15e40a6440 msgr2=0x7f15e40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 -- 172.21.15.86:0/1335066289 shutdown_connections 2024-09-17T15:40:20.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.133+0000 7f15e9825640 1 -- 172.21.15.86:0/1335066289 wait complete. 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient: build_initial_monmap 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient: monmap: 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:20.141 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 Processor -- start 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 -- start start 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient: init 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-c con 0x7f15e40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:20.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-a con 0x7f15e4068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 0x7f15e41364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): picked mon.noname-b con 0x7f15e413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41405f0 con 0x7f15e4068db0 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4140470 con 0x7f15e413aff0 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f15e4140770 con 0x7f15e40689d0 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e9825640 10 monclient(hunting): authenticate will time out at 1753.105430s 2024-09-17T15:40:20.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39586/0 (socket says 172.21.15.86:39586) 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42816/0 (socket says 172.21.15.86:42816) 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 1 -- 172.21.15.86:0/1398279104 learned_addr learned my addr 172.21.15.86:0/1398279104 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): get_auth_request con 0x7f15e40689d0 auth_method 0 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e37fe640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 msgr2=0x7f15e41364a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): get_auth_request con 0x7f15e4068db0 auth_method 0 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:20.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e37fe640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 0x7f15e41364a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient(hunting): handle_auth_done global_id 144189 payload 824 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 msgr2=0x7f15e41cfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 msgr2=0x7f15e41364a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 0x7f15e41364a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 monclient: found mon.noname-a 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e27fc640 1 -- 172.21.15.86:0/1398279104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e4122630 con 0x7f15e4068db0 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.137+0000 7f15e2ffd640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e27fc640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 msgr2=0x7f15e4135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e27fc640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15cc00caf0 tx=0x7f15cc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:20.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc012400 con 0x7f15e4068db0 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: dump: 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: _finish_auth 0 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15cc0125a0 con 0x7f15e4068db0 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15cc012990 con 0x7f15e4068db0 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 5 monclient: authenticate success, global_id 144189 2024-09-17T15:40:20.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _renew_subs 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e413d080 con 0x7f15e4068db0 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e17fa640 4 set_mon_vals no callback set 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 10 monclient: dump: 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _renew_subs 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e4122350 con 0x7f15e4068db0 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15e406f840 con 0x7f15e4068db0 2024-09-17T15:40:20.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.141+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f15cc012bf0 con 0x7f15e4068db0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.145+0000 7f15dbfff640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 0x7f15c403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.145+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7f15cc050550 con 0x7f15e4068db0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.145+0000 7f15e2ffd640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 0x7f15c403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.145+0000 7f15e2ffd640 10 monclient: get_auth_request con 0x7f15c403bb90 auth_method 0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.145+0000 7f15e2ffd640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 0x7f15c403e050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f15dc01c360 tx=0x7f15dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.149+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f15cc010d10 con 0x7f15e4068db0 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.149+0000 7f15dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:20.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.149+0000 7f15dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:20.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15e9825640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15e9825640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15e9825640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f15e406d4f0 con 0x7f15e4068db0 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15dbfff640 1 -- 172.21.15.86:0/1398279104 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f15cc019300 con 0x7f15e4068db0 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:20.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.381+0000 7f15dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:20.384 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 msgr2=0x7f15c403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 0x7f15c403e050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f15dc01c360 tx=0x7f15dc008040 comp rx=0 tx=0).stop 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 10 monclient: shutdown 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 msgr2=0x7f15e4135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f15cc00caf0 tx=0x7f15cc00a060 comp rx=0 tx=0).stop 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 shutdown_connections 2024-09-17T15:40:20.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15c403bb90 0x7f15c403e050 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15e413aff0 0x7f15e41364a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15e4068db0 0x7f15e4135f60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 --2- 172.21.15.86:0/1398279104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15e40689d0 0x7f15e41cfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 >> 172.21.15.86:0/1398279104 conn(0x7f15e40a6440 msgr2=0x7f15e40a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 shutdown_connections 2024-09-17T15:40:20.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:20.385+0000 7f15e9825640 1 -- 172.21.15.86:0/1398279104 wait complete. 2024-09-17T15:40:23.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: get_monmap_and_config 2024-09-17T15:40:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: build_initial_monmap 2024-09-17T15:40:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: monmap: 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:23.533627+0000 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:23.533627+0000 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:23.530 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 Processor -- start 2024-09-17T15:40:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 -- start start 2024-09-17T15:40:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: init 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-b con 0x7fa9680689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa968068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa96813aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa968074470 con 0x7fa96813aff0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa96809c8b0 con 0x7fa968068db0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa96809cbe0 con 0x7fa9680689d0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.529+0000 7fa96e1ca640 10 monclient(hunting): authenticate will time out at 1756.499902s 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa967fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa96813aff0 0x7fa96813d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49092/0 (socket says 172.21.15.86:49092) 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa967fff640 1 -- 172.21.15.86:0/2674225492 learned_addr learned my addr 172.21.15.86:0/2674225492 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.203:6789/0 conn(0x7fa968068db0 legacy=0x7fa9680691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa9680689d0 auth_method 0 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:23.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:23.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:23.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:23.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:23.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:23.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.533+0000 7fa967fff640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.86:6789/0 conn(0x7fa96813aff0 legacy=0x7fa96813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:23.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.203:6789/0 conn(0x7fa968068db0 legacy=0x7fa9680691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:23.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa9680689d0 auth_method 0 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:23.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:40:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa9680689d0 auth_method 0 2024-09-17T15:40:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:40:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.141+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.203:6789/0 conn(0x7fa968068db0 legacy=0x7fa9680691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:24.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.203:6789/0 conn(0x7fa968068db0 legacy=0x7fa9680691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:24.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa9680689d0 auth_method 0 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:24.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:24.945+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:40:26.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.529+0000 7fa965ffb640 10 monclient(hunting): tick 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.529+0000 7fa965ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.529+0000 7fa965ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.529+0000 7fa965ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.529+0000 7fa965ffb640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 msgr2=0x7fa96806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9680689d0 0x7fa96806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.203:6789/0 conn(0x7fa968068db0 legacy=0x7fa9680691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 -- 172.21.15.86:0/2674225492 >> v1:172.21.15.86:6789/0 conn(0x7fa96813aff0 legacy=0x7fa96813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fa948003760 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:26.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fa9480062d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 -- 172.21.15.86:0/2674225492 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa96809cbe0 con 0x7fa948003760 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 1 -- 172.21.15.86:0/2674225492 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa948008e30 con 0x7fa9480062d0 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa965ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 msgr2=0x7fa948008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa95ffff640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa948003760 auth_method 0 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:26.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:26.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:26.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:26.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.533+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:26.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 msgr2=0x7fa948008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa95ffff640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa948003760 auth_method 0 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:26.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:26.737+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:40:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 msgr2=0x7fa948008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:27.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa95ffff640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa948003760 auth_method 0 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 msgr2=0x7fa948005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:27.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:27.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.141+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:40:27.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa95ffff640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 msgr2=0x7fa948008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:27.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa95ffff640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.86:3300/0 conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa948003760 auth_method 0 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient(hunting): handle_auth_done global_id 144192 payload 274 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 msgr2=0x7fa948008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> v2:172.21.15.203:3300/0 conn(0x7fa9480062d0 0x7fa948008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:40:27.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 1 monclient: found mon.noname-a 2024-09-17T15:40:27.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:27.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:27.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.945+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96809ce70 con 0x7fa948003760 2024-09-17T15:40:27.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa9677fe640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa948003760 msgr2=0x7fa948005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:27.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa9677fe640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa948003760 0x7fa948005be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9480062a0 tx=0x7fa95801dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3ac22bc45b741 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 1 -- 172.21.15.86:0/2674225492 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9580ac070 con 0x7fa948003760 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: dump: 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:27.954 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 1 -- 172.21.15.86:0/2674225492 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9580a6da0 con 0x7fa948003760 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 1 -- 172.21.15.86:0/2674225492 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96809c2d0 con 0x7fa948003760 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa966ffd640 10 monclient: dump: 2024-09-17T15:40:27.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 5 monclient: authenticate success, global_id 144192 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:27.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 4 set_mon_vals no callback set 2024-09-17T15:40:27.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 10 monclient: shutdown 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 -- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa948003760 msgr2=0x7fa948005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa948003760 0x7fa948005be0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa9480062a0 tx=0x7fa95801dd60 comp rx=0 tx=0).stop 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa967fff640 1 -- 172.21.15.86:0/2674225492 reap_dead start 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 -- 172.21.15.86:0/2674225492 shutdown_connections 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/2674225492 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa948003760 0x7fa948005be0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 -- 172.21.15.86:0/2674225492 >> 172.21.15.86:0/2674225492 conn(0x7fa9680a6440 msgr2=0x7fa9680a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 -- 172.21.15.86:0/2674225492 shutdown_connections 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.953+0000 7fa96e1ca640 1 -- 172.21.15.86:0/2674225492 wait complete. 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient: build_initial_monmap 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient: monmap: 2024-09-17T15:40:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 Processor -- start 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 -- start start 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient: init 2024-09-17T15:40:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-c con 0x7fa948003760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa968135c30 0x7fa9681360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-a con 0x7fa968135c30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 0x7fa9681d0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): picked mon.noname-b con 0x7fa9681d0080 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa968141be0 con 0x7fa968135c30 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa968141a60 con 0x7fa9681d0080 2024-09-17T15:40:27.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa968141d60 con 0x7fa948003760 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa96e1ca640 10 monclient(hunting): authenticate will time out at 1760.926095s 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42868/0 (socket says 172.21.15.86:42868) 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 -- 172.21.15.86:0/462984584 learned_addr learned my addr 172.21.15.86:0/462984584 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): get_auth_request con 0x7fa948003760 auth_method 0 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa967fff640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 msgr2=0x7fa9681d0530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:27.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa967fff640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 0x7fa9681d0530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa968135c30 0x7fa9681360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): get_auth_request con 0x7fa968135c30 auth_method 0 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa95ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient(hunting): handle_auth_done global_id 104579 payload 824 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 msgr2=0x7fa9681d0530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:27.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 0x7fa9681d0530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa968135c30 msgr2=0x7fa9681360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa968135c30 0x7fa9681360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 monclient: found mon.noname-c 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.957+0000 7fa9677fe640 1 -- 172.21.15.86:0/462984584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96809d460 con 0x7fa948003760 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa9677fe640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 msgr2=0x7fa968070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa9677fe640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa958008870 tx=0x7fa958008a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9580ac040 con 0x7fa948003760 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:27.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: dump: 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: _finish_auth 0 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:40:27.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa958003a30 con 0x7fa948003760 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95800f040 con 0x7fa948003760 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 10 monclient: dump: 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:27.969 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 5 monclient: authenticate success, global_id 104579 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _renew_subs 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa9667fc640 4 set_mon_vals no callback set 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa968072500 con 0x7fa948003760 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _renew_subs 2024-09-17T15:40:27.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681d0a70 con 0x7fa948003760 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96806f840 con 0x7fa948003760 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa9580a6c60 con 0x7fa948003760 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 0x7fa94803ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7fa9580eb9b0 con 0x7fa948003760 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa95ffff640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 0x7fa94803ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.961+0000 7fa95ffff640 10 monclient: get_auth_request con 0x7fa94803c940 auth_method 0 2024-09-17T15:40:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.965+0000 7fa95ffff640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 0x7fa94803ee00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9500096d0 tx=0x7fa950009660 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:27.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.969+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa9580b6070 con 0x7fa948003760 2024-09-17T15:40:27.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.969+0000 7fa964ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:27.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:27.969+0000 7fa964ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:28.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa96e1ca640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:28.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa96e1ca640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:28.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa96e1ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa968071f30 con 0x7fa948003760 2024-09-17T15:40:28.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa964ff9640 1 -- 172.21.15.86:0/462984584 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa95800a8d0 con 0x7fa948003760 2024-09-17T15:40:28.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa964ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:28.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.197+0000 7fa964ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:28.198 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 msgr2=0x7fa94803ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 0x7fa94803ee00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa9500096d0 tx=0x7fa950009660 comp rx=0 tx=0).stop 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 10 monclient: shutdown 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 msgr2=0x7fa968070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa958008870 tx=0x7fa958008a00 comp rx=0 tx=0).stop 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 shutdown_connections 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa94803c940 0x7fa94803ee00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9681d0080 0x7fa9681d0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa968135c30 0x7fa9681360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 --2- 172.21.15.86:0/462984584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa948003760 0x7fa968070ef0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 >> 172.21.15.86:0/462984584 conn(0x7fa9680a6440 msgr2=0x7fa9680a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 shutdown_connections 2024-09-17T15:40:28.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.201+0000 7fa96e1ca640 1 -- 172.21.15.86:0/462984584 wait complete. 2024-09-17T15:40:28.216 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:40:28.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:40:28.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:40:28.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.321+0000 7f67e60d7640 10 monclient: get_monmap_and_config 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.321+0000 7f67e60d7640 10 monclient: build_initial_monmap 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.321+0000 7f67e60d7640 10 monclient: monmap: 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:28.327852+0000 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:28.327852+0000 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:28.324 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:28.325 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:28.325 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 Processor -- start 2024-09-17T15:40:28.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 -- start start 2024-09-17T15:40:28.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient: init 2024-09-17T15:40:28.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:28.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67e006dee0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f67e006f8c0 0x7f67e00a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-c con 0x7f67e006f8c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67e013d420 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e00a4950 con 0x7f67e013d420 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67e006ec00 con 0x7f67e006dee0 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f67e006e2c0 con 0x7f67e006f8c0 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67e60d7640 10 monclient(hunting): authenticate will time out at 1761.293301s 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67dffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f67e013d420 0x7f67e013f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45032/0 (socket says 172.21.15.86:45032) 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 learned_addr learned my addr 172.21.15.86:0/1531047820 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:28.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67df7fe640 1 -- 172.21.15.86:0/1531047820 >> v1:172.21.15.203:6789/0 conn(0x7f67e006dee0 legacy=0x7f67e00a2e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:28.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67d6ffd640 1 -- 172.21.15.86:0/1531047820 >> v2:172.21.15.203:3300/0 conn(0x7f67e006f8c0 msgr2=0x7f67e00a3580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:28.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67d6ffd640 1 --2- 172.21.15.86:0/1531047820 >> v2:172.21.15.203:3300/0 conn(0x7f67e006f8c0 0x7f67e00a3580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:28.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 >> v1:172.21.15.86:6789/0 conn(0x7f67e013d420 legacy=0x7f67e013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684482159 0 0) 0x7f67e00a4950 con 0x7f67e013d420 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 10 monclient(hunting): _negotiate 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 10 monclient(hunting): my global_id is 144198 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.325+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67cc003530 con 0x7f67e013d420 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 960811624 0 0) 0x7f67cc003530 con 0x7f67e013d420 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 >> v2:172.21.15.203:3300/0 conn(0x7f67e006f8c0 msgr2=0x7f67e00a3580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 --2- 172.21.15.86:0/1531047820 >> v2:172.21.15.203:3300/0 conn(0x7f67e006f8c0 0x7f67e00a3580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 >> v1:172.21.15.203:6789/0 conn(0x7f67e006dee0 legacy=0x7f67e00a2e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e006eee0 con 0x7f67e013d420 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _finish_auth 0 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _check_auth_tickets 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f67d8001d30 con 0x7f67e013d420 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: dump: 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 >> v1:172.21.15.86:6789/0 conn(0x7f67e013d420 legacy=0x7f67e013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67cc003a30 0x7f67cc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): picked mon.b con 0x7f67cc003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:28.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67cc004810 0x7f67cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): picked mon.c con 0x7f67cc004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67cc00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): picked mon.a con 0x7f67cc0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67e006e2c0 con 0x7f67cc0099c0 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc00c3a0 con 0x7f67cc003a30 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f67cc00c5a0 con 0x7f67cc004810 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67deffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67cc00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67df7fe640 1 -- 172.21.15.86:0/1531047820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67cc003a30 msgr2=0x7f67cc003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67cc004810 0x7f67cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): get_auth_request con 0x7f67cc0099c0 auth_method 0 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67df7fe640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67cc003a30 0x7f67cc003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): get_auth_request con 0x7f67cc004810 auth_method 0 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient(hunting): handle_auth_done global_id 144198 payload 293 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67cc004810 msgr2=0x7f67cc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67cc004810 0x7f67cc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67cc003a30 msgr2=0x7f67cc003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67cc003a30 0x7f67cc003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 monclient: found mon.a 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:28.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc003530 con 0x7f67cc0099c0 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient: _finish_auth 0 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67e60d7640 5 monclient: authenticate success, global_id 144198 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67e60d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:28.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.329+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 reap_dead start 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 -- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 msgr2=0x7f67cc00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67cc00bdb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67d8001f50 tx=0x7f67d801d3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81f02e8c7741694b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d80a9070 con 0x7f67cc0099c0 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: dump: 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.344 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e00a3cd0 con 0x7f67cc0099c0 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 1 -- 172.21.15.86:0/1531047820 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d800a690 con 0x7f67cc0099c0 2024-09-17T15:40:28.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 4 set_mon_vals no callback set 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67deffd640 10 monclient: dump: 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.346 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: shutdown 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 msgr2=0x7f67cc00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67cc00bdb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67d8001f50 tx=0x7f67d801d3e0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- 172.21.15.86:0/1531047820 shutdown_connections 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 --2- 172.21.15.86:0/1531047820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67cc00bdb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- 172.21.15.86:0/1531047820 >> 172.21.15.86:0/1531047820 conn(0x7f67e009bc70 msgr2=0x7f67e00a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- 172.21.15.86:0/1531047820 shutdown_connections 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- 172.21.15.86:0/1531047820 wait complete. 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: build_initial_monmap 2024-09-17T15:40:28.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: monmap: 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 Processor -- start 2024-09-17T15:40:28.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- start start 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: init 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67e0139f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-a con 0x7f67cc0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67e009db30 0x7f67e009df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-b con 0x7f67e009db30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): picked mon.noname-c con 0x7f67e009e4d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67e00a3200 con 0x7f67cc0099c0 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f67e00a3080 con 0x7f67e009db30 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f67e00a3380 con 0x7f67e009e4d0 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67e60d7640 10 monclient(hunting): authenticate will time out at 1761.303270s 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67e0139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67d6ffd640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67e009db30 msgr2=0x7f67e009df90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:35442/0 (socket says 172.21.15.86:35442) 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 learned_addr learned my addr 172.21.15.86:0/129465930 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67e0139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39802/0 (socket says 172.21.15.86:39802) 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): get_auth_request con 0x7f67e009e4d0 auth_method 0 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67d6ffd640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67e009db30 0x7f67e009df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): get_auth_request con 0x7f67cc0099c0 auth_method 0 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient(hunting): handle_auth_done global_id 104582 payload 824 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67e009db30 msgr2=0x7f67e009df90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67e009db30 0x7f67e009df90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 msgr2=0x7f67e0139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67cc0099c0 0x7f67e0139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 monclient: found mon.noname-c 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.333+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e013b740 con 0x7f67e009e4d0 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 msgr2=0x7f67e01d6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dffff640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f67d8001470 tx=0x7f67cc005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d80a9070 con 0x7f67e009e4d0 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: dump: 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.353 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: _finish_auth 0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 5 monclient: authenticate success, global_id 104582 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d80072e0 con 0x7f67e009e4d0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 monclient: _renew_subs 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e009e950 con 0x7f67e009e4d0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d80b2070 con 0x7f67e009e4d0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:28.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 10 monclient: dump: 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67de7fc640 4 set_mon_vals no callback set 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 monclient: _renew_subs 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e00a6370 con 0x7f67e009e4d0 2024-09-17T15:40:28.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 0x7f67e009fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67e013b0e0 con 0x7f67e009f660 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 0x7f67e009fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient: get_auth_request con 0x7f67e009f660 auth_method 0 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f67d80076d0 con 0x7f67e009e4d0 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 10 monclient(hunting): handle_auth_done global_id 144204 payload 824 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67df7fe640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67e00a3cd0 con 0x7f67e009f660 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f67c003bb60 0x7f67c003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7f67d80e9990 con 0x7f67e009e4d0 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67d6ffd640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f67c003bb60 0x7f67c003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.337+0000 7f67d6ffd640 10 monclient: get_auth_request con 0x7f67c003bb60 auth_method 0 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67df7fe640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 msgr2=0x7f67e009fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67df7fe640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 0x7f67e009fa40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f67d0003fd0 tx=0x7f67d0009a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0016070 con 0x7f67e009f660 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 10 monclient: dump: 2024-09-17T15:40:28.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f67d000e970 con 0x7f67e009f660 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:28.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 msgr2=0x7f67e009fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 0x7f67e009fa40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f67d0003fd0 tx=0x7f67d0009a40 comp rx=0 tx=0).stop 2024-09-17T15:40:28.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 msgr2=0x7f67e009fa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:28.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67dcff9640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009f660 0x7f67e009fa40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.341+0000 7f67d6ffd640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f67c003bb60 0x7f67c003e020 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f67c8003910 tx=0x7f67c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 0x7f67e00a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f67e013a840 con 0x7f67e009ff80 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 0x7f67e00a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient: get_auth_request con 0x7f67e009ff80 auth_method 0 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 10 monclient(hunting): handle_auth_done global_id 144207 payload 824 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 1 -- 172.21.15.86:0/129465930 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f67e00a3710 con 0x7f67e009ff80 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 msgr2=0x7f67e00a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67df7fe640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 0x7f67e00a0390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67d0018000 tx=0x7f67d00059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0016070 con 0x7f67e009ff80 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 10 monclient: dump: 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f67d00067b0 con 0x7f67e009ff80 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 msgr2=0x7f67e00a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 0x7f67e00a0390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67d0018000 tx=0x7f67d00059e0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 msgr2=0x7f67e00a0390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:28.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dcff9640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f67e009ff80 0x7f67e00a0390 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f67d0018000 tx=0x7f67d00059e0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 130, 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 46691, 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:28.364 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.365 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.366 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.368 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:28.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:28.370 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.371 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.373 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:28.374 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.375 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:28.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.377 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:28.378 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f67c003bb60 msgr2=0x7f67c003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f67c003bb60 0x7f67c003e020 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f67c8003910 tx=0x7f67c8008040 comp rx=0 tx=0).stop 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 10 monclient: shutdown 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 msgr2=0x7f67e01d6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67dffff640 1 -- 172.21.15.86:0/129465930 reap_dead start 2024-09-17T15:40:28.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f67d8001470 tx=0x7f67cc005840 comp rx=0 tx=0).stop 2024-09-17T15:40:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 shutdown_connections 2024-09-17T15:40:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 --2- 172.21.15.86:0/129465930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f67e009e4d0 0x7f67e01d6510 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 >> 172.21.15.86:0/129465930 conn(0x7f67e009bc70 msgr2=0x7f67e0072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 shutdown_connections 2024-09-17T15:40:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.349+0000 7f67e60d7640 1 -- 172.21.15.86:0/129465930 wait complete. 2024-09-17T15:40:28.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:40:28.380 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:40:28.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: get_monmap_and_config 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: build_initial_monmap 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: monmap: 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:28.500392+0000 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:28.500392+0000 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:28.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 Processor -- start 2024-09-17T15:40:28.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 -- start start 2024-09-17T15:40:28.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: init 2024-09-17T15:40:28.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:28.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdde409d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 0x7fdde4139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-b con 0x7fdde409e260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdde413a440 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde409ff00 con 0x7fdde413a440 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdde409ec30 con 0x7fdde409d770 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdde409e7f0 con 0x7fdde409e260 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fddeb905640 10 monclient(hunting): authenticate will time out at 1761.465842s 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde9ea2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdde413a440 0x7fdde413c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45048/0 (socket says 172.21.15.86:45048) 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde9ea2640 1 -- 172.21.15.86:0/3502256170 learned_addr learned my addr 172.21.15.86:0/3502256170 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:28.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde96a1640 1 -- 172.21.15.86:0/3502256170 >> v1:172.21.15.203:6789/0 conn(0x7fdde409d770 legacy=0x7fdde409db50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 0x7fdde4139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): get_auth_request con 0x7fdde409e260 auth_method 0 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_done global_id 104585 payload 274 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 1 -- 172.21.15.86:0/3502256170 >> v1:172.21.15.203:6789/0 conn(0x7fdde409d770 legacy=0x7fdde409db50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 1 -- 172.21.15.86:0/3502256170 >> v1:172.21.15.86:6789/0 conn(0x7fdde413a440 legacy=0x7fdde413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient: _finish_hunting 0 2024-09-17T15:40:28.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 1 monclient: found mon.noname-b 2024-09-17T15:40:28.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:28.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.497+0000 7fdde8ea0640 1 -- 172.21.15.86:0/3502256170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde409ef10 con 0x7fdde409e260 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fdde8ea0640 1 -- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 msgr2=0x7fdde4139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 0x7fdde4139f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdde00097f0 tx=0x7fdde001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e683a48bff4a55b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 1 -- 172.21.15.86:0/3502256170 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde00b0070 con 0x7fdde409e260 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: dump: 2024-09-17T15:40:28.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: _finish_auth 0 2024-09-17T15:40:28.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 1 -- 172.21.15.86:0/3502256170 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde00ab600 con 0x7fdde409e260 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 1 -- 172.21.15.86:0/3502256170 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde00ab9d0 con 0x7fdde409e260 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 5 monclient: authenticate success, global_id 104585 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 4 set_mon_vals no callback set 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddd7fff640 10 monclient: dump: 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.508 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 10 monclient: shutdown 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 -- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 msgr2=0x7fdde4139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 --2- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 0x7fdde4139f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdde00097f0 tx=0x7fdde001c3a0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 -- 172.21.15.86:0/3502256170 shutdown_connections 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 --2- 172.21.15.86:0/3502256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409e260 0x7fdde4139f00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 -- 172.21.15.86:0/3502256170 >> 172.21.15.86:0/3502256170 conn(0x7fdde40a2f40 msgr2=0x7fdde40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 -- 172.21.15.86:0/3502256170 shutdown_connections 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 1 -- 172.21.15.86:0/3502256170 wait complete. 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 10 monclient: build_initial_monmap 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.501+0000 7fddeb905640 10 monclient: monmap: 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.510 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 Processor -- start 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- start start 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: init 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 0x7fdde41cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-c con 0x7fdde409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde409e260 0x7fdde41344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-b con 0x7fdde409e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 0x7fdde41349f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): picked mon.noname-a con 0x7fdde413a440 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdde41408f0 con 0x7fdde413a440 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4140770 con 0x7fdde409e260 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdde4140a70 con 0x7fdde409d770 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient(hunting): authenticate will time out at 1761.472623s 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 0x7fdde41349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 0x7fdde41cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 0x7fdde41349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39836/0 (socket says 172.21.15.86:39836) 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 1 -- 172.21.15.86:0/2499496073 learned_addr learned my addr 172.21.15.86:0/2499496073 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:28.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): get_auth_request con 0x7fdde413a440 auth_method 0 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde8ea0640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde409e260 msgr2=0x7fdde41344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): get_auth_request con 0x7fdde409d770 auth_method 0 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde409e260 0x7fdde41344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient(hunting): handle_auth_done global_id 104588 payload 824 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde409e260 msgr2=0x7fdde41344b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdde409e260 0x7fdde41344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 msgr2=0x7fdde41349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 0x7fdde41349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient: _finish_hunting 0 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 monclient: found mon.noname-c 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde4120b80 con 0x7fdde409d770 2024-09-17T15:40:28.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde9ea2640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdde413a440 0x7fdde41349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 msgr2=0x7fdde41cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fdde96a1640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 0x7fdde41cfdb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00bb30 tx=0x7fdddc00c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc015470 con 0x7fdde409d770 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: dump: 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.518 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 5 monclient: authenticate success, global_id 104588 2024-09-17T15:40:28.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdddc010070 con 0x7fdde409d770 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _renew_subs 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde411fbc0 con 0x7fdde409d770 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdddc015870 con 0x7fdde409d770 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd5ffb640 10 monclient: dump: 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddd77fe640 4 set_mon_vals no callback set 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _renew_subs 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.505+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde41205c0 con 0x7fdde409d770 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddeb905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddeb905640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 0x7fdde411d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdde4120800 con 0x7fdde411f6c0 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 0x7fdde411d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient: get_auth_request con 0x7fdde411f6c0 auth_method 0 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_done global_id 104591 payload 824 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdde409db50 con 0x7fdde411f6c0 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdddc01d390 con 0x7fdde409d770 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fddc803bb60 0x7fddc803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(442..442 src has 213..442) ==== 4755+0+0 (secure 0 0 0) 0x7fdddc0503c0 con 0x7fdde409d770 2024-09-17T15:40:28.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde9ea2640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fddc803bb60 0x7fddc803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde9ea2640 10 monclient: get_auth_request con 0x7fddc803bb60 auth_method 0 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 msgr2=0x7fdde411d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 0x7fdde411d800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ad000 tx=0x7fdde001abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde00bb020 con 0x7fdde411f6c0 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 10 monclient: dump: 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.523 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdde00b6470 con 0x7fdde411f6c0 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 msgr2=0x7fdde411d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 0x7fdde411d800 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ad000 tx=0x7fdde001abe0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 msgr2=0x7fdde411d800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fddd5ffb640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde411f6c0 0x7fdde411d800 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ad000 tx=0x7fdde001abe0 comp rx=0 tx=0).stop 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.509+0000 7fdde9ea2640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fddc803bb60 0x7fddc803e020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007d30 tx=0x7fddd8007c80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fddeb905640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fddeb905640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 0x7fdde402d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fddeb905640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdde4071e00 con 0x7fdde402d450 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 0x7fdde402d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient: get_auth_request con 0x7fdde402d450 auth_method 0 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:28.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:28.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:28.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:28.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 10 monclient(hunting): handle_auth_done global_id 104594 payload 824 2024-09-17T15:40:28.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 1 -- 172.21.15.86:0/2499496073 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdde406d4f0 con 0x7fdde402d450 2024-09-17T15:40:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 msgr2=0x7fdde402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.517+0000 7fdde8ea0640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 0x7fdde402d860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009740 tx=0x7fdde0019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde00bb070 con 0x7fdde402d450 2024-09-17T15:40:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 10 monclient: dump: 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:28.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fdde00c2020 con 0x7fdde402d450 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 msgr2=0x7fdde402d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 0x7fdde402d860 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009740 tx=0x7fdde0019140 comp rx=0 tx=0).stop 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 msgr2=0x7fdde402d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddd5ffb640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde402d450 0x7fdde402d860 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009740 tx=0x7fdde0019140 comp rx=0 tx=0).stop 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:28.529 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 130, 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 259396, 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.530 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.531 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:28.533 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:28.534 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.535 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.537 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:28.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.539 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:28.541 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.542 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:28.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:40:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:28.544 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:28.544 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:28.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fddc803bb60 msgr2=0x7fddc803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fddc803bb60 0x7fddc803e020 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007d30 tx=0x7fddd8007c80 comp rx=0 tx=0).stop 2024-09-17T15:40:28.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 10 monclient: shutdown 2024-09-17T15:40:28.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:28.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 msgr2=0x7fdde41cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 --2- 172.21.15.86:0/2499496073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdde409d770 0x7fdde41cfdb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdddc00bb30 tx=0x7fdddc00c040 comp rx=0 tx=0).stop 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fdde9ea2640 1 -- 172.21.15.86:0/2499496073 reap_dead start 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fdde9ea2640 1 -- 172.21.15.86:0/2499496073 reap_dead start 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 shutdown_connections 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 >> 172.21.15.86:0/2499496073 conn(0x7fdde40a2f40 msgr2=0x7fdde413d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 shutdown_connections 2024-09-17T15:40:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:28.521+0000 7fddeb905640 1 -- 172.21.15.86:0/2499496073 wait complete. 2024-09-17T15:40:28.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:40:28.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:40:28.719 INFO:tasks.rados.rados.0.smithi203.stdout:1965: done (1 left) 2024-09-17T15:40:28.719 INFO:tasks.rados.rados.0.smithi203.stdout:1966: delete oid 6 current snap is 221 2024-09-17T15:40:28.720 INFO:tasks.rados.rados.0.smithi203.stdout:1964: finishing rollback tid 0 to smithi20328086-17 2024-09-17T15:40:28.720 INFO:tasks.rados.rados.0.smithi203.stdout:1964: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:40:28.720 INFO:tasks.rados.rados.0.smithi203.stdout:1964: finishing rollback tid 2 to smithi20328086-17 2024-09-17T15:40:28.720 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 784 (ObjNum 646 snap 203 seq_num 646) dirty exists 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:1964: done (1 left) 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:1966: done (0 left) 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:1967: rollback oid 31 current snap is 221 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 194 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:1968: read oid 38 snap -1 2024-09-17T15:40:28.730 INFO:tasks.rados.rados.0.smithi203.stdout:1968: expect deleted 2024-09-17T15:40:28.731 INFO:tasks.rados.rados.0.smithi203.stdout:1969: write oid 9 current snap is 221 2024-09-17T15:40:28.731 INFO:tasks.rados.rados.0.smithi203.stdout:1969: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2024-09-17T15:40:28.751 INFO:tasks.rados.rados.0.smithi203.stdout:1969: writing smithi20328086-9 from 598514 to 1327788 tid 1 2024-09-17T15:40:28.766 INFO:tasks.rados.rados.0.smithi203.stdout:1969: writing smithi20328086-9 from 1746124 to 2343353 tid 2 2024-09-17T15:40:28.783 INFO:tasks.rados.rados.0.smithi203.stdout:1969: writing smithi20328086-9 from 3128235 to 3643188 tid 3 2024-09-17T15:40:28.783 INFO:tasks.rados.rados.0.smithi203.stdout:1970: read oid 35 snap -1 2024-09-17T15:40:28.783 INFO:tasks.rados.rados.0.smithi203.stdout:1970: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:40:28.784 INFO:tasks.rados.rados.0.smithi203.stdout:1971: write (excl) oid 27 current snap is 221 2024-09-17T15:40:28.784 INFO:tasks.rados.rados.0.smithi203.stdout:1971: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-09-17T15:40:28.795 INFO:tasks.rados.rados.0.smithi203.stdout:1971: writing smithi20328086-27 from 615321 to 1019844 tid 1 2024-09-17T15:40:28.810 INFO:tasks.rados.rados.0.smithi203.stdout:1971: writing smithi20328086-27 from 1804606 to 2261846 tid 2 2024-09-17T15:40:28.822 INFO:tasks.rados.rados.0.smithi203.stdout:1971: writing smithi20328086-27 from 2755658 to 3481883 tid 3 2024-09-17T15:40:28.822 INFO:tasks.rados.rados.0.smithi203.stdout:1972: delete oid 7 current snap is 221 2024-09-17T15:40:28.822 INFO:tasks.rados.rados.0.smithi203.stdout:1967: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1967: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1967: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 522 (ObjNum 598 snap 179 seq_num 598) dirty exists 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: finishing write tid 1 to smithi20328086-9 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: finishing write tid 2 to smithi20328086-9 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: finishing write tid 3 to smithi20328086-9 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: finishing write tid 4 to smithi20328086-9 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: finishing write tid 5 to smithi20328086-9 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: oid 9 updating version 0 to 765 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: oid 9 updating version 765 to 766 2024-09-17T15:40:28.823 INFO:tasks.rados.rados.0.smithi203.stdout:1969: oid 9 version 766 is already newer than 764 2024-09-17T15:40:28.824 INFO:tasks.rados.rados.0.smithi203.stdout:1969: oid 9 updating version 766 to 767 2024-09-17T15:40:28.824 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 767 (ObjNum 702 snap 221 seq_num 702) dirty exists 2024-09-17T15:40:28.824 INFO:tasks.rados.rados.0.smithi203.stdout:1969: left oid 9 (ObjNum 702 snap 221 seq_num 702) 2024-09-17T15:40:28.824 INFO:tasks.rados.rados.0.smithi203.stdout:1970: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:40:28.963 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1815711 2024-09-17T15:40:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:1971: finishing write tid 1 to smithi20328086-27 2024-09-17T15:40:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:1971: finishing write tid 2 to smithi20328086-27 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: finishing write tid 3 to smithi20328086-27 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: finishing write tid 4 to smithi20328086-27 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: finishing write tid 5 to smithi20328086-27 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: oid 27 updating version 0 to 879 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: oid 27 updating version 879 to 880 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: oid 27 version 880 is already newer than 877 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: oid 27 version 880 is already newer than 878 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 880 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1971: left oid 27 (ObjNum 703 snap 221 seq_num 703) 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1967: done (5 left) 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1968: done (4 left) 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1969: done (3 left) 2024-09-17T15:40:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:1970: done (2 left) 2024-09-17T15:40:28.987 INFO:tasks.rados.rados.0.smithi203.stdout:1971: done (1 left) 2024-09-17T15:40:28.987 INFO:tasks.rados.rados.0.smithi203.stdout:1972: done (0 left) 2024-09-17T15:40:28.987 INFO:tasks.rados.rados.0.smithi203.stdout:1973: read oid 40 snap 217 2024-09-17T15:40:28.987 INFO:tasks.rados.rados.0.smithi203.stdout:1973: expect (ObjNum 587 snap 174 seq_num 587) 2024-09-17T15:40:28.987 INFO:tasks.rados.rados.0.smithi203.stdout:1974: read oid 41 snap -1 2024-09-17T15:40:28.988 INFO:tasks.rados.rados.0.smithi203.stdout:1974: expect (ObjNum 645 snap 203 seq_num 645) 2024-09-17T15:40:28.988 INFO:tasks.rados.rados.0.smithi203.stdout:1975: rollback oid 11 current snap is 221 2024-09-17T15:40:28.988 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 209 2024-09-17T15:40:28.988 INFO:tasks.rados.rados.0.smithi203.stdout:1976: write (excl) oid 12 current snap is 221 2024-09-17T15:40:28.988 INFO:tasks.rados.rados.0.smithi203.stdout:1976: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-09-17T15:40:28.997 INFO:tasks.rados.rados.0.smithi203.stdout:1976: writing smithi20328086-12 from 648935 to 1203956 tid 1 2024-09-17T15:40:29.009 INFO:tasks.rados.rados.0.smithi203.stdout:1976: writing smithi20328086-12 from 1805217 to 2266126 tid 2 2024-09-17T15:40:29.018 INFO:tasks.rados.rados.0.smithi203.stdout:1976: writing smithi20328086-12 from 2694151 to 3159273 tid 3 2024-09-17T15:40:29.027 INFO:tasks.rados.rados.0.smithi203.stdout:1976: writing smithi20328086-12 from 3817442 to 3848935 tid 4 2024-09-17T15:40:29.027 INFO:tasks.rados.rados.0.smithi203.stdout:1977: write oid 35 current snap is 221 2024-09-17T15:40:29.027 INFO:tasks.rados.rados.0.smithi203.stdout:1977: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2024-09-17T15:40:29.039 INFO:tasks.rados.rados.0.smithi203.stdout:1977: writing smithi20328086-35 from 665742 to 1296012 tid 1 2024-09-17T15:40:29.049 INFO:tasks.rados.rados.0.smithi203.stdout:1977: writing smithi20328086-35 from 1863699 to 2468266 tid 2 2024-09-17T15:40:29.059 INFO:tasks.rados.rados.0.smithi203.stdout:1977: writing smithi20328086-35 from 3121574 to 3797968 tid 3 2024-09-17T15:40:29.059 INFO:tasks.rados.rados.0.smithi203.stdout:1978: write (excl) oid 43 current snap is 221 2024-09-17T15:40:29.059 INFO:tasks.rados.rados.0.smithi203.stdout:1978: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-17T15:40:29.068 INFO:tasks.rados.rados.0.smithi203.stdout:1978: writing smithi20328086-43 from 682549 to 1388068 tid 1 2024-09-17T15:40:29.075 INFO:tasks.rados.rados.0.smithi203.stdout:1978: writing smithi20328086-43 from 1922181 to 2386759 tid 2 2024-09-17T15:40:29.083 INFO:tasks.rados.rados.0.smithi203.stdout:1978: writing smithi20328086-43 from 3148997 to 3636663 tid 3 2024-09-17T15:40:29.084 INFO:tasks.rados.rados.0.smithi203.stdout:1979: read oid 37 snap 216 2024-09-17T15:40:29.084 INFO:tasks.rados.rados.0.smithi203.stdout:1979: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:40:29.084 INFO:tasks.rados.rados.0.smithi203.stdout:1980: rollback oid 15 current snap is 221 2024-09-17T15:40:29.084 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 220 2024-09-17T15:40:29.084 INFO:tasks.rados.rados.0.smithi203.stdout:1974: expect (ObjNum 645 snap 203 seq_num 645) 2024-09-17T15:40:29.094 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2564893 2024-09-17T15:40:29.141 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2077470 2024-09-17T15:40:29.149 INFO:tasks.rados.rados.0.smithi203.stdout:1973: expect (ObjNum 587 snap 174 seq_num 587) 2024-09-17T15:40:29.182 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:29.241 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3860348 2024-09-17T15:40:29.245 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 1 to smithi20328086-12 2024-09-17T15:40:29.245 INFO:tasks.rados.rados.0.smithi203.stdout:1973: done (7 left) 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1974: done (6 left) 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1981: rollback oid 47 current snap is 221 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 194 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1975: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1975: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1975: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 792 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 2 to smithi20328086-12 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 3 to smithi20328086-12 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 4 to smithi20328086-12 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 5 to smithi20328086-12 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: finishing write tid 6 to smithi20328086-12 2024-09-17T15:40:29.246 INFO:tasks.rados.rados.0.smithi203.stdout:1976: oid 12 updating version 0 to 641 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1976: oid 12 version 641 is already newer than 639 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1976: oid 12 updating version 641 to 642 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1976: oid 12 version 642 is already newer than 640 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1976: oid 12 version 642 is already newer than 638 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 642 (ObjNum 705 snap 221 seq_num 705) dirty exists 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1976: left oid 12 (ObjNum 705 snap 221 seq_num 705) 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: finishing write tid 1 to smithi20328086-35 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: finishing write tid 2 to smithi20328086-35 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: finishing write tid 3 to smithi20328086-35 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: finishing write tid 4 to smithi20328086-35 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: finishing write tid 5 to smithi20328086-35 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: oid 35 updating version 0 to 621 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: oid 35 updating version 621 to 622 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: oid 35 updating version 622 to 624 2024-09-17T15:40:29.247 INFO:tasks.rados.rados.0.smithi203.stdout:1977: oid 35 version 624 is already newer than 623 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 624 (ObjNum 706 snap 221 seq_num 706) dirty exists 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1977: left oid 35 (ObjNum 706 snap 221 seq_num 706) 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: finishing write tid 1 to smithi20328086-43 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: finishing write tid 2 to smithi20328086-43 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1980: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 0 (ObjNum 33 snap 0 seq_num 3586005048) dirty dne 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: finishing write tid 3 to smithi20328086-43 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: finishing write tid 4 to smithi20328086-43 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: finishing write tid 5 to smithi20328086-43 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: oid 43 updating version 0 to 617 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: oid 43 updating version 617 to 619 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: oid 43 version 619 is already newer than 618 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:1978: oid 43 updating version 619 to 620 2024-09-17T15:40:29.248 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 620 (ObjNum 707 snap 221 seq_num 707) dirty exists 2024-09-17T15:40:29.249 INFO:tasks.rados.rados.0.smithi203.stdout:1978: left oid 43 (ObjNum 707 snap 221 seq_num 707) 2024-09-17T15:40:29.249 INFO:tasks.rados.rados.0.smithi203.stdout:1979: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:40:29.249 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 303848 2024-09-17T15:40:29.424 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2956180 2024-09-17T15:40:29.439 INFO:tasks.rados.rados.0.smithi203.stdout:1981: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1981: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1981: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 816 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1975: done (6 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1976: done (5 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1977: done (4 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1978: done (3 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1979: done (2 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1980: done (1 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1981: done (0 left) 2024-09-17T15:40:29.440 INFO:tasks.rados.rados.0.smithi203.stdout:1982: snap_create 2024-09-17T15:40:29.722 INFO:tasks.rados.rados.0.smithi203.stdout:1982: done (0 left) 2024-09-17T15:40:29.722 INFO:tasks.rados.rados.0.smithi203.stdout:1983: snap_remove snap 196 2024-09-17T15:40:30.723 INFO:tasks.rados.rados.0.smithi203.stdout:1983: done (0 left) 2024-09-17T15:40:30.723 INFO:tasks.rados.rados.0.smithi203.stdout:1984: snap_remove snap 206 2024-09-17T15:40:31.681 INFO:tasks.rados.rados.0.smithi203.stdout:1984: done (0 left) 2024-09-17T15:40:31.681 INFO:tasks.rados.rados.0.smithi203.stdout:1985: copy_from oid 43 from oid 13 current snap is 222 2024-09-17T15:40:31.681 INFO:tasks.rados.rados.0.smithi203.stdout:1986: snap_create 2024-09-17T15:40:31.722 INFO:tasks.rados.rados.0.smithi203.stdout:1985: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:40:31.726 INFO:tasks.rados.rados.0.smithi203.stdout:1985: finishing copy_from to smithi20328086-43 2024-09-17T15:40:31.726 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:40:32.692 INFO:tasks.rados.rados.0.smithi203.stdout:1985: done (1 left) 2024-09-17T15:40:32.692 INFO:tasks.rados.rados.0.smithi203.stdout:1986: done (0 left) 2024-09-17T15:40:32.693 INFO:tasks.rados.rados.0.smithi203.stdout:1987: snap_remove snap 192 2024-09-17T15:40:33.691 INFO:tasks.rados.rados.0.smithi203.stdout:1987: done (0 left) 2024-09-17T15:40:33.692 INFO:tasks.rados.rados.0.smithi203.stdout:1988: read oid 39 snap -1 2024-09-17T15:40:33.692 INFO:tasks.rados.rados.0.smithi203.stdout:1988: expect (ObjNum 695 snap 218 seq_num 695) 2024-09-17T15:40:33.692 INFO:tasks.rados.rados.0.smithi203.stdout:1989: snap_remove snap 217 2024-09-17T15:40:35.579 INFO:tasks.rados.rados.0.smithi203.stdout:1989: done (1 left) 2024-09-17T15:40:35.579 INFO:tasks.rados.rados.0.smithi203.stdout:1990: copy_from oid 25 from oid 13 current snap is 223 2024-09-17T15:40:35.579 INFO:tasks.rados.rados.0.smithi203.stdout:1988: expect (ObjNum 695 snap 218 seq_num 695) 2024-09-17T15:40:35.579 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:35.805 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:35.895 INFO:tasks.rados.rados.0.smithi203.stdout:1990: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:1990: finishing copy_from to smithi20328086-25 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 611 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:1988: done (1 left) 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:1990: done (0 left) 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:1991: write oid 4 current snap is 223 2024-09-17T15:40:35.896 INFO:tasks.rados.rados.0.smithi203.stdout:1991: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-17T15:40:35.915 INFO:tasks.rados.rados.0.smithi203.stdout:1991: writing smithi20328086-4 from 699356 to 1196477 tid 1 2024-09-17T15:40:35.938 INFO:tasks.rados.rados.0.smithi203.stdout:1991: writing smithi20328086-4 from 1980663 to 2588899 tid 2 2024-09-17T15:40:35.953 INFO:tasks.rados.rados.0.smithi203.stdout:1991: writing smithi20328086-4 from 3176420 to 3591711 tid 3 2024-09-17T15:40:35.953 INFO:tasks.rados.rados.0.smithi203.stdout:1992: write oid 22 current snap is 223 2024-09-17T15:40:35.953 INFO:tasks.rados.rados.0.smithi203.stdout:1991: finishing write tid 1 to smithi20328086-4 2024-09-17T15:40:35.953 INFO:tasks.rados.rados.0.smithi203.stdout:1991: finishing write tid 2 to smithi20328086-4 2024-09-17T15:40:35.953 INFO:tasks.rados.rados.0.smithi203.stdout:1992: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-17T15:40:35.968 INFO:tasks.rados.rados.0.smithi203.stdout:1992: writing smithi20328086-22 from 716163 to 1288533 tid 1 2024-09-17T15:40:35.984 INFO:tasks.rados.rados.0.smithi203.stdout:1992: writing smithi20328086-22 from 2039145 to 2507392 tid 2 2024-09-17T15:40:35.997 INFO:tasks.rados.rados.0.smithi203.stdout:1992: writing smithi20328086-22 from 3203843 to 3830406 tid 3 2024-09-17T15:40:35.997 INFO:tasks.rados.rados.0.smithi203.stdout:1993: read oid 47 snap -1 2024-09-17T15:40:35.997 INFO:tasks.rados.rados.0.smithi203.stdout:1993: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1994: write (excl) oid 49 current snap is 223 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: finishing write tid 3 to smithi20328086-4 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: finishing write tid 4 to smithi20328086-4 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: finishing write tid 5 to smithi20328086-4 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: oid 4 updating version 0 to 377 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: oid 4 version 377 is already newer than 376 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: oid 4 version 377 is already newer than 375 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: oid 4 version 377 is already newer than 374 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 377 (ObjNum 708 snap 223 seq_num 708) dirty exists 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1991: left oid 4 (ObjNum 708 snap 223 seq_num 708) 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1992: finishing write tid 1 to smithi20328086-22 2024-09-17T15:40:35.998 INFO:tasks.rados.rados.0.smithi203.stdout:1994: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-17T15:40:36.010 INFO:tasks.rados.rados.0.smithi203.stdout:1994: writing smithi20328086-49 from 732970 to 1380589 tid 1 2024-09-17T15:40:36.023 INFO:tasks.rados.rados.0.smithi203.stdout:1994: writing smithi20328086-49 from 2097627 to 2709532 tid 2 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1994: writing smithi20328086-49 from 3231266 to 3669101 tid 3 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1991: done (3 left) 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1995: rollback oid 37 current snap is 223 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 218 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: finishing write tid 2 to smithi20328086-22 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: finishing write tid 3 to smithi20328086-22 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: finishing write tid 4 to smithi20328086-22 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: finishing write tid 5 to smithi20328086-22 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: oid 22 updating version 0 to 525 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: oid 22 version 525 is already newer than 524 2024-09-17T15:40:36.032 INFO:tasks.rados.rados.0.smithi203.stdout:1992: oid 22 updating version 525 to 526 2024-09-17T15:40:36.033 INFO:tasks.rados.rados.0.smithi203.stdout:1992: oid 22 version 526 is already newer than 523 2024-09-17T15:40:36.033 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 526 (ObjNum 709 snap 223 seq_num 709) dirty exists 2024-09-17T15:40:36.033 INFO:tasks.rados.rados.0.smithi203.stdout:1992: left oid 22 (ObjNum 709 snap 223 seq_num 709) 2024-09-17T15:40:36.033 INFO:tasks.rados.rados.0.smithi203.stdout:1993: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:40:36.101 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2348474 2024-09-17T15:40:36.108 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 69937 2024-09-17T15:40:36.146 INFO:tasks.rados.rados.0.smithi203.stdout:1994: finishing write tid 1 to smithi20328086-49 2024-09-17T15:40:36.146 INFO:tasks.rados.rados.0.smithi203.stdout:1994: finishing write tid 2 to smithi20328086-49 2024-09-17T15:40:36.146 INFO:tasks.rados.rados.0.smithi203.stdout:1994: finishing write tid 3 to smithi20328086-49 2024-09-17T15:40:36.146 INFO:tasks.rados.rados.0.smithi203.stdout:1995: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: finishing write tid 4 to smithi20328086-49 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1995: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1995: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 902 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: finishing write tid 5 to smithi20328086-49 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: oid 49 updating version 0 to 996 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: oid 49 version 996 is already newer than 994 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: oid 49 version 996 is already newer than 993 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: oid 49 version 996 is already newer than 995 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 996 (ObjNum 710 snap 223 seq_num 710) dirty exists 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1994: left oid 49 (ObjNum 710 snap 223 seq_num 710) 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1992: done (3 left) 2024-09-17T15:40:36.147 INFO:tasks.rados.rados.0.smithi203.stdout:1993: done (2 left) 2024-09-17T15:40:36.148 INFO:tasks.rados.rados.0.smithi203.stdout:1994: done (1 left) 2024-09-17T15:40:36.148 INFO:tasks.rados.rados.0.smithi203.stdout:1995: done (0 left) 2024-09-17T15:40:36.148 INFO:tasks.rados.rados.0.smithi203.stdout:1996: write oid 19 current snap is 223 2024-09-17T15:40:36.148 INFO:tasks.rados.rados.0.smithi203.stdout:1996: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-17T15:40:36.158 INFO:tasks.rados.rados.0.smithi203.stdout:1996: writing smithi20328086-19 from 749777 to 1472645 tid 1 2024-09-17T15:40:36.167 INFO:tasks.rados.rados.0.smithi203.stdout:1996: writing smithi20328086-19 from 2156109 to 2628025 tid 2 2024-09-17T15:40:36.178 INFO:tasks.rados.rados.0.smithi203.stdout:1996: writing smithi20328086-19 from 3258689 to 3907796 tid 3 2024-09-17T15:40:36.178 INFO:tasks.rados.rados.0.smithi203.stdout:1997: write oid 17 current snap is 223 2024-09-17T15:40:36.178 INFO:tasks.rados.rados.0.smithi203.stdout:1996: finishing write tid 1 to smithi20328086-19 2024-09-17T15:40:36.178 INFO:tasks.rados.rados.0.smithi203.stdout:1996: finishing write tid 2 to smithi20328086-19 2024-09-17T15:40:36.179 INFO:tasks.rados.rados.0.smithi203.stdout:1997: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-17T15:40:36.193 INFO:tasks.rados.rados.0.smithi203.stdout:1997: writing smithi20328086-17 from 766584 to 1564701 tid 1 2024-09-17T15:40:36.204 INFO:tasks.rados.rados.0.smithi203.stdout:1997: writing smithi20328086-17 from 2098238 to 2713812 tid 2 2024-09-17T15:40:36.212 INFO:tasks.rados.rados.0.smithi203.stdout:1997: writing smithi20328086-17 from 3169759 to 3630138 tid 3 2024-09-17T15:40:36.212 INFO:tasks.rados.rados.0.smithi203.stdout:1998: snap_remove snap 216 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: finishing write tid 3 to smithi20328086-19 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: finishing write tid 4 to smithi20328086-19 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: finishing write tid 5 to smithi20328086-19 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: oid 19 updating version 0 to 904 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: oid 19 updating version 904 to 905 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: oid 19 version 905 is already newer than 903 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: oid 19 updating version 905 to 906 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 906 (ObjNum 711 snap 223 seq_num 711) dirty exists 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1996: left oid 19 (ObjNum 711 snap 223 seq_num 711) 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1997: finishing write tid 1 to smithi20328086-17 2024-09-17T15:40:36.681 INFO:tasks.rados.rados.0.smithi203.stdout:1997: finishing write tid 2 to smithi20328086-17 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: finishing write tid 3 to smithi20328086-17 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: finishing write tid 4 to smithi20328086-17 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: finishing write tid 5 to smithi20328086-17 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: oid 17 updating version 0 to 812 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: oid 17 updating version 812 to 815 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: oid 17 version 815 is already newer than 813 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: oid 17 version 815 is already newer than 814 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 815 (ObjNum 712 snap 223 seq_num 712) dirty exists 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: left oid 17 (ObjNum 712 snap 223 seq_num 712) 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1996: done (2 left) 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1997: done (1 left) 2024-09-17T15:40:36.682 INFO:tasks.rados.rados.0.smithi203.stdout:1998: done (0 left) 2024-09-17T15:40:36.683 INFO:tasks.rados.rados.0.smithi203.stdout:1999: snap_remove snap 205 2024-09-17T15:40:37.685 INFO:tasks.rados.rados.0.smithi203.stdout:1999: done (0 left) 2024-09-17T15:40:37.686 INFO:tasks.rados.rados.0.smithi203.stdout:2000: snap_remove snap 189 2024-09-17T15:40:38.696 INFO:tasks.rados.rados.0.smithi203.stdout:2000: done (0 left) 2024-09-17T15:40:38.696 INFO:tasks.rados.rados.0.smithi203.stdout:2001: delete oid 19 current snap is 223 2024-09-17T15:40:38.713 INFO:tasks.rados.rados.0.smithi203.stdout:2001: done (0 left) 2024-09-17T15:40:38.713 INFO:tasks.rados.rados.0.smithi203.stdout:2002: rollback oid 12 current snap is 223 2024-09-17T15:40:38.713 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 218 2024-09-17T15:40:38.713 INFO:tasks.rados.rados.0.smithi203.stdout:2003: snap_remove snap 219 2024-09-17T15:40:39.701 INFO:tasks.rados.rados.0.smithi203.stdout:2003: done (1 left) 2024-09-17T15:40:39.701 INFO:tasks.rados.rados.0.smithi203.stdout:2004: snap_create 2024-09-17T15:40:39.701 INFO:tasks.rados.rados.0.smithi203.stdout:2002: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:40:39.701 INFO:tasks.rados.rados.0.smithi203.stdout:2002: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:40:39.701 INFO:tasks.rados.rados.0.smithi203.stdout:2002: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:40:39.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 645 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:40:40.710 INFO:tasks.rados.rados.0.smithi203.stdout:2002: done (1 left) 2024-09-17T15:40:40.710 INFO:tasks.rados.rados.0.smithi203.stdout:2004: done (0 left) 2024-09-17T15:40:40.710 INFO:tasks.rados.rados.0.smithi203.stdout:2005: write oid 39 current snap is 224 2024-09-17T15:40:40.710 INFO:tasks.rados.rados.0.smithi203.stdout:2005: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-17T15:40:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:2005: writing smithi20328086-39 from 400198 to 948813 tid 1 2024-09-17T15:40:40.744 INFO:tasks.rados.rados.0.smithi203.stdout:2005: writing smithi20328086-39 from 1415202 to 2000198 tid 2 2024-09-17T15:40:40.744 INFO:tasks.rados.rados.0.smithi203.stdout:2006: snap_remove snap 213 2024-09-17T15:40:41.684 INFO:tasks.rados.rados.0.smithi203.stdout:2006: done (1 left) 2024-09-17T15:40:41.684 INFO:tasks.rados.rados.0.smithi203.stdout:2007: read oid 19 snap -1 2024-09-17T15:40:41.684 INFO:tasks.rados.rados.0.smithi203.stdout:2007: expect deleted 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2008: rollback oid 45 current snap is 224 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 207 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: finishing write tid 1 to smithi20328086-39 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: finishing write tid 2 to smithi20328086-39 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: finishing write tid 3 to smithi20328086-39 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: finishing write tid 4 to smithi20328086-39 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: oid 39 updating version 0 to 933 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: oid 39 updating version 933 to 935 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: oid 39 version 935 is already newer than 934 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 935 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: left oid 39 (ObjNum 714 snap 224 seq_num 714) 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2005: done (2 left) 2024-09-17T15:40:41.685 INFO:tasks.rados.rados.0.smithi203.stdout:2009: read oid 28 snap 218 2024-09-17T15:40:41.686 INFO:tasks.rados.rados.0.smithi203.stdout:2009: expect (ObjNum 692 snap 218 seq_num 692) 2024-09-17T15:40:41.686 INFO:tasks.rados.rados.0.smithi203.stdout:2010: snap_remove snap 221 2024-09-17T15:40:42.695 INFO:tasks.rados.rados.0.smithi203.stdout:2010: done (3 left) 2024-09-17T15:40:42.695 INFO:tasks.rados.rados.0.smithi203.stdout:2011: write (excl) oid 7 current snap is 224 2024-09-17T15:40:42.695 INFO:tasks.rados.rados.0.smithi203.stdout:2011: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-17T15:40:42.712 INFO:tasks.rados.rados.0.smithi203.stdout:2011: writing smithi20328086-7 from 417005 to 1157222 tid 1 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2011: writing smithi20328086-7 from 1590037 to 2017005 tid 2 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2012: read oid 3 snap 195 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2012: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2013: snap_create 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2008: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2008: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 379 (ObjNum 289 snap 0 seq_num 3606675248) dirty dne 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:2009: expect (ObjNum 692 snap 218 seq_num 692) 2024-09-17T15:40:42.728 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: finishing write tid 1 to smithi20328086-7 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: finishing write tid 2 to smithi20328086-7 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: finishing write tid 3 to smithi20328086-7 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: finishing write tid 4 to smithi20328086-7 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: oid 7 updating version 0 to 523 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: oid 7 version 523 is already newer than 522 2024-09-17T15:40:42.935 INFO:tasks.rados.rados.0.smithi203.stdout:2011: oid 7 version 523 is already newer than 521 2024-09-17T15:40:42.936 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 523 (ObjNum 715 snap 224 seq_num 715) dirty exists 2024-09-17T15:40:42.936 INFO:tasks.rados.rados.0.smithi203.stdout:2011: left oid 7 (ObjNum 715 snap 224 seq_num 715) 2024-09-17T15:40:42.936 INFO:tasks.rados.rados.0.smithi203.stdout:2012: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:40:42.936 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:42.979 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 892780 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2007: done (5 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2008: done (4 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2009: done (3 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2011: done (2 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2012: done (1 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2013: done (0 left) 2024-09-17T15:40:43.727 INFO:tasks.rados.rados.0.smithi203.stdout:2014: delete oid 7 current snap is 225 2024-09-17T15:40:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:2014: done (0 left) 2024-09-17T15:40:43.733 INFO:tasks.rados.rados.0.smithi203.stdout:2015: snap_create 2024-09-17T15:40:44.732 INFO:tasks.rados.rados.0.smithi203.stdout:2015: done (0 left) 2024-09-17T15:40:44.732 INFO:tasks.rados.rados.0.smithi203.stdout:2016: write (excl) oid 32 current snap is 226 2024-09-17T15:40:44.732 INFO:tasks.rados.rados.0.smithi203.stdout:2016: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-17T15:40:44.747 INFO:tasks.rados.rados.0.smithi203.stdout:2016: writing smithi20328086-32 from 450619 to 941334 tid 1 2024-09-17T15:40:44.765 INFO:tasks.rados.rados.0.smithi203.stdout:2016: writing smithi20328086-32 from 1590648 to 2050619 tid 2 2024-09-17T15:40:44.765 INFO:tasks.rados.rados.0.smithi203.stdout:2017: snap_create 2024-09-17T15:40:44.765 INFO:tasks.rados.rados.0.smithi203.stdout:2016: finishing write tid 1 to smithi20328086-32 2024-09-17T15:40:44.775 INFO:tasks.rados.rados.0.smithi203.stdout:2016: finishing write tid 2 to smithi20328086-32 2024-09-17T15:40:44.775 INFO:tasks.rados.rados.0.smithi203.stdout:2016: finishing write tid 3 to smithi20328086-32 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:2016: finishing write tid 4 to smithi20328086-32 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:2016: oid 32 updating version 0 to 643 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:2016: oid 32 updating version 643 to 644 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:2016: oid 32 version 644 is already newer than 642 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 644 (ObjNum 717 snap 226 seq_num 717) dirty exists 2024-09-17T15:40:44.776 INFO:tasks.rados.rados.0.smithi203.stdout:2016: left oid 32 (ObjNum 717 snap 226 seq_num 717) 2024-09-17T15:40:45.739 INFO:tasks.rados.rados.0.smithi203.stdout:2016: done (1 left) 2024-09-17T15:40:45.739 INFO:tasks.rados.rados.0.smithi203.stdout:2017: done (0 left) 2024-09-17T15:40:45.740 INFO:tasks.rados.rados.0.smithi203.stdout:2018: read oid 27 snap -1 2024-09-17T15:40:45.740 INFO:tasks.rados.rados.0.smithi203.stdout:2018: expect (ObjNum 703 snap 221 seq_num 703) 2024-09-17T15:40:45.740 INFO:tasks.rados.rados.0.smithi203.stdout:2019: write (excl) oid 13 current snap is 227 2024-09-17T15:40:45.740 INFO:tasks.rados.rados.0.smithi203.stdout:2019: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-17T15:40:45.756 INFO:tasks.rados.rados.0.smithi203.stdout:2019: writing smithi20328086-13 from 467426 to 1033390 tid 1 2024-09-17T15:40:45.773 INFO:tasks.rados.rados.0.smithi203.stdout:2019: writing smithi20328086-13 from 1649130 to 2067426 tid 2 2024-09-17T15:40:45.773 INFO:tasks.rados.rados.0.smithi203.stdout:2018: expect (ObjNum 703 snap 221 seq_num 703) 2024-09-17T15:40:45.781 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 858776 2024-09-17T15:40:45.851 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1112141 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: finishing write tid 1 to smithi20328086-13 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: finishing write tid 2 to smithi20328086-13 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: finishing write tid 3 to smithi20328086-13 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: finishing write tid 4 to smithi20328086-13 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: oid 13 updating version 0 to 688 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: oid 13 updating version 688 to 689 2024-09-17T15:40:45.947 INFO:tasks.rados.rados.0.smithi203.stdout:2019: oid 13 updating version 689 to 690 2024-09-17T15:40:45.948 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 690 (ObjNum 718 snap 227 seq_num 718) dirty exists 2024-09-17T15:40:45.948 INFO:tasks.rados.rados.0.smithi203.stdout:2019: left oid 13 (ObjNum 718 snap 227 seq_num 718) 2024-09-17T15:40:45.948 INFO:tasks.rados.rados.0.smithi203.stdout:2018: done (1 left) 2024-09-17T15:40:45.948 INFO:tasks.rados.rados.0.smithi203.stdout:2019: done (0 left) 2024-09-17T15:40:45.948 INFO:tasks.rados.rados.0.smithi203.stdout:2020: snap_remove snap 222 2024-09-17T15:40:46.686 INFO:tasks.rados.rados.0.smithi203.stdout:2020: done (0 left) 2024-09-17T15:40:46.687 INFO:tasks.rados.rados.0.smithi203.stdout:2021: copy_from oid 42 from oid 41 current snap is 227 2024-09-17T15:40:46.687 INFO:tasks.rados.rados.0.smithi203.stdout:2022: snap_remove snap 194 2024-09-17T15:40:47.687 INFO:tasks.rados.rados.0.smithi203.stdout:2022: done (1 left) 2024-09-17T15:40:47.687 INFO:tasks.rados.rados.0.smithi203.stdout:2023: read oid 19 snap 191 2024-09-17T15:40:47.688 INFO:tasks.rados.rados.0.smithi203.stdout:2023: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:40:47.688 INFO:tasks.rados.rados.0.smithi203.stdout:2024: write (excl) oid 33 current snap is 227 2024-09-17T15:40:47.688 INFO:tasks.rados.rados.0.smithi203.stdout:2024: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-17T15:40:47.694 INFO:tasks.rados.rados.0.smithi203.stdout:2024: writing smithi20328086-33 from 484233 to 1125446 tid 1 2024-09-17T15:40:47.700 INFO:tasks.rados.rados.0.smithi203.stdout:2024: writing smithi20328086-33 from 1707612 to 2084233 tid 2 2024-09-17T15:40:47.701 INFO:tasks.rados.rados.0.smithi203.stdout:2025: write oid 28 current snap is 227 2024-09-17T15:40:47.701 INFO:tasks.rados.rados.0.smithi203.stdout:2025: seq_num 720 ranges {501040=716462,1766094=334946} 2024-09-17T15:40:47.708 INFO:tasks.rados.rados.0.smithi203.stdout:2025: writing smithi20328086-28 from 501040 to 1217502 tid 1 2024-09-17T15:40:47.715 INFO:tasks.rados.rados.0.smithi203.stdout:2025: writing smithi20328086-28 from 1766094 to 2101040 tid 2 2024-09-17T15:40:47.715 INFO:tasks.rados.rados.0.smithi203.stdout:2026: read oid 6 snap -1 2024-09-17T15:40:47.715 INFO:tasks.rados.rados.0.smithi203.stdout:2026: expect deleted 2024-09-17T15:40:47.715 INFO:tasks.rados.rados.0.smithi203.stdout:2027: write oid 47 current snap is 227 2024-09-17T15:40:47.715 INFO:tasks.rados.rados.0.smithi203.stdout:2027: seq_num 721 ranges {517847=791711,2108223=9624} 2024-09-17T15:40:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:2027: writing smithi20328086-47 from 517847 to 1309558 tid 1 2024-09-17T15:40:47.727 INFO:tasks.rados.rados.0.smithi203.stdout:2027: writing smithi20328086-47 from 2108223 to 2117847 tid 2 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:2028: delete oid 5 current snap is 227 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:2021: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:2021: finishing copy_from to smithi20328086-42 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 616 (ObjNum 645 snap 203 seq_num 645) dirty exists 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:2023: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:40:47.728 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:47.813 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:47.853 INFO:tasks.rados.rados.0.smithi203.stdout:2024: finishing write tid 1 to smithi20328086-33 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: finishing write tid 2 to smithi20328086-33 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: finishing write tid 3 to smithi20328086-33 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: finishing write tid 4 to smithi20328086-33 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: oid 33 updating version 0 to 509 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: oid 33 updating version 509 to 510 2024-09-17T15:40:47.854 INFO:tasks.rados.rados.0.smithi203.stdout:2024: oid 33 updating version 510 to 511 2024-09-17T15:40:47.855 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 511 (ObjNum 719 snap 227 seq_num 719) dirty exists 2024-09-17T15:40:47.855 INFO:tasks.rados.rados.0.smithi203.stdout:2024: left oid 33 (ObjNum 719 snap 227 seq_num 719) 2024-09-17T15:40:47.855 INFO:tasks.rados.rados.0.smithi203.stdout:2025: finishing write tid 1 to smithi20328086-28 2024-09-17T15:40:47.855 INFO:tasks.rados.rados.0.smithi203.stdout:2025: finishing write tid 2 to smithi20328086-28 2024-09-17T15:40:47.855 INFO:tasks.rados.rados.0.smithi203.stdout:2025: finishing write tid 3 to smithi20328086-28 2024-09-17T15:40:47.856 INFO:tasks.rados.rados.0.smithi203.stdout:2025: finishing write tid 4 to smithi20328086-28 2024-09-17T15:40:47.856 INFO:tasks.rados.rados.0.smithi203.stdout:2025: oid 28 updating version 0 to 1041 2024-09-17T15:40:47.856 INFO:tasks.rados.rados.0.smithi203.stdout:2025: oid 28 version 1041 is already newer than 1040 2024-09-17T15:40:47.856 INFO:tasks.rados.rados.0.smithi203.stdout:2025: oid 28 updating version 1041 to 1042 2024-09-17T15:40:47.856 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1042 (ObjNum 720 snap 227 seq_num 720) dirty exists 2024-09-17T15:40:47.857 INFO:tasks.rados.rados.0.smithi203.stdout:2025: left oid 28 (ObjNum 720 snap 227 seq_num 720) 2024-09-17T15:40:47.857 INFO:tasks.rados.rados.0.smithi203.stdout:2027: finishing write tid 1 to smithi20328086-47 2024-09-17T15:40:47.857 INFO:tasks.rados.rados.0.smithi203.stdout:2027: finishing write tid 2 to smithi20328086-47 2024-09-17T15:40:47.857 INFO:tasks.rados.rados.0.smithi203.stdout:2027: finishing write tid 3 to smithi20328086-47 2024-09-17T15:40:47.858 INFO:tasks.rados.rados.0.smithi203.stdout:2027: finishing write tid 4 to smithi20328086-47 2024-09-17T15:40:47.858 INFO:tasks.rados.rados.0.smithi203.stdout:2027: oid 47 updating version 0 to 819 2024-09-17T15:40:47.858 INFO:tasks.rados.rados.0.smithi203.stdout:2027: oid 47 version 819 is already newer than 818 2024-09-17T15:40:47.858 INFO:tasks.rados.rados.0.smithi203.stdout:2027: oid 47 version 819 is already newer than 817 2024-09-17T15:40:47.858 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 819 (ObjNum 721 snap 227 seq_num 721) dirty exists 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2027: left oid 47 (ObjNum 721 snap 227 seq_num 721) 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2021: done (6 left) 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2023: done (5 left) 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2024: done (4 left) 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2025: done (3 left) 2024-09-17T15:40:47.859 INFO:tasks.rados.rados.0.smithi203.stdout:2026: done (2 left) 2024-09-17T15:40:47.860 INFO:tasks.rados.rados.0.smithi203.stdout:2027: done (1 left) 2024-09-17T15:40:47.860 INFO:tasks.rados.rados.0.smithi203.stdout:2028: done (0 left) 2024-09-17T15:40:47.860 INFO:tasks.rados.rados.0.smithi203.stdout:2029: snap_remove snap 177 2024-09-17T15:40:48.547 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:40:48.548 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:40:48.548 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:40:48.551 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:40:48.552 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:40:48.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 10 monclient: get_monmap_and_config 2024-09-17T15:40:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 10 monclient: build_initial_monmap 2024-09-17T15:40:48.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 10 monclient: monmap: 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:48.686559+0000 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:48.686559+0000 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:48.683 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:48.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 1 Processor -- start 2024-09-17T15:40:48.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 1 -- start start 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.680+0000 7f34e63be640 10 monclient: init 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34e009a490 0x7f34e009a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-a con 0x7f34e009a490 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-c con 0x7f34e00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:48.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34e009ef90 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e00a0310 con 0x7f34e009ef90 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f34e00a00f0 con 0x7f34e009a490 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f34e00a05f0 con 0x7f34e00689d0 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34e63be640 10 monclient(hunting): authenticate will time out at 1781.652623s 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34e009a490 0x7f34e009a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34e009a490 0x7f34e009a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55500/0 (socket says 172.21.15.86:55500) 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 -- 172.21.15.86:0/1711647043 learned_addr learned my addr 172.21.15.86:0/1711647043 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:48.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): get_auth_request con 0x7f34e009a490 auth_method 0 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 1 --2- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 10 monclient(hunting): get_auth_request con 0x7f34e00689d0 auth_method 0 2024-09-17T15:40:48.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 1 -- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 msgr2=0x7f34e0134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient(hunting): handle_auth_done global_id 144213 payload 274 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 1 -- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 msgr2=0x7f34e0134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 1 --2- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 -- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 msgr2=0x7f34e0134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34d7fff640 1 --2- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 --2- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 -- 172.21.15.86:0/1711647043 >> v1:172.21.15.203:6789/0 conn(0x7f34e009ef90 legacy=0x7f34e0134e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:48.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:48.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 monclient: found mon.noname-a 2024-09-17T15:40:48.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:48.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.684+0000 7f34dffff640 1 -- 172.21.15.86:0/1711647043 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e009eaf0 con 0x7f34e009a490 2024-09-17T15:40:48.693 INFO:tasks.rados.rados.0.smithi203.stdout:2029: done (0 left) 2024-09-17T15:40:48.694 INFO:tasks.rados.rados.0.smithi203.stdout:2030: snap_remove snap 220 2024-09-17T15:40:48.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34dffff640 1 -- 172.21.15.86:0/1711647043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 msgr2=0x7f34e009a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34dffff640 1 --2- 172.21.15.86:0/1711647043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e009a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f34e006fcd0 tx=0x7f34d001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfce8de7e456331f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 1 -- 172.21.15.86:0/1711647043 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d00b0070 con 0x7f34e009a490 2024-09-17T15:40:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: dump: 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:48.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: _finish_auth 0 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 5 monclient: authenticate success, global_id 144213 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 1 -- 172.21.15.86:0/1711647043 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e009afc0 con 0x7f34e009a490 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 1 -- 172.21.15.86:0/1711647043 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d00ab820 con 0x7f34e009a490 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34df7fe640 10 monclient: dump: 2024-09-17T15:40:48.701 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 4 set_mon_vals no callback set 2024-09-17T15:40:48.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 10 monclient: shutdown 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 -- 172.21.15.86:0/1711647043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 msgr2=0x7f34e009a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 --2- 172.21.15.86:0/1711647043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e009a870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f34e006fcd0 tx=0x7f34d001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 -- 172.21.15.86:0/1711647043 shutdown_connections 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 --2- 172.21.15.86:0/1711647043 >> v2:172.21.15.203:3300/0 conn(0x7f34e00689d0 0x7f34e0134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 --2- 172.21.15.86:0/1711647043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e009a870 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 -- 172.21.15.86:0/1711647043 >> 172.21.15.86:0/1711647043 conn(0x7f34e00a73f0 msgr2=0x7f34e00a77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 -- 172.21.15.86:0/1711647043 shutdown_connections 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 1 -- 172.21.15.86:0/1711647043 wait complete. 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 10 monclient: build_initial_monmap 2024-09-17T15:40:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.688+0000 7f34e63be640 10 monclient: monmap: 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:48.704 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 Processor -- start 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 -- start start 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient: init 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 0x7f34e01d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-c con 0x7f34e00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-a con 0x7f34e009a490 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34e009ef90 0x7f34e01317e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): picked mon.noname-b con 0x7f34e009ef90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f34e006d500 con 0x7f34e009a490 2024-09-17T15:40:48.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f34e006d380 con 0x7f34e009ef90 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f34e006d680 con 0x7f34e00689d0 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34e63be640 10 monclient(hunting): authenticate will time out at 1781.661144s 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55504/0 (socket says 172.21.15.86:55504) 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 -- 172.21.15.86:0/4029879686 learned_addr learned my addr 172.21.15.86:0/4029879686 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): get_auth_request con 0x7f34e009a490 auth_method 0 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:48.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34dffff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 0x7f34e01d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient(hunting): handle_auth_done global_id 144216 payload 824 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 -- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 msgr2=0x7f34e01d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 0x7f34e01d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 -- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34e009ef90 msgr2=0x7f34e01317e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34e009ef90 0x7f34e01317e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 monclient: found mon.noname-a 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:48.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34d7fff640 1 -- 172.21.15.86:0/4029879686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e0135ad0 con 0x7f34e009a490 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34dffff640 10 monclient: get_auth_request con 0x7f34e00689d0 auth_method 0 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.692+0000 7f34dffff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 0x7f34e01d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34d7fff640 1 -- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 msgr2=0x7f34e01d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34d7fff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34c800eb40 tx=0x7f34c800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c8019480 con 0x7f34e009a490 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: dump: 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c8019620 con 0x7f34e009a490 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34c80199d0 con 0x7f34e009a490 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:48.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34dd7fa640 10 monclient: dump: 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34deffd640 4 set_mon_vals no callback set 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 5 monclient: authenticate success, global_id 144216 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _renew_subs 2024-09-17T15:40:48.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e0137e30 con 0x7f34e009a490 2024-09-17T15:40:48.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _renew_subs 2024-09-17T15:40:48.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e009e5e0 con 0x7f34e009a490 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.696+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e009a870 con 0x7f34e009a490 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.700+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f34c8019bb0 con 0x7f34e009a490 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.700+0000 7f34dd7fa640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 0x7f34c003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.700+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(462..462 src has 213..462) ==== 4831+0+0 (secure 0 0 0) 0x7f34c8050ee0 con 0x7f34e009a490 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.700+0000 7f34dffff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 0x7f34c003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:48.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.700+0000 7f34dffff640 10 monclient: get_auth_request con 0x7f34c003bbb0 auth_method 0 2024-09-17T15:40:48.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.704+0000 7f34dffff640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 0x7f34c003e070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f34e00a19e0 tx=0x7f34d0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:48.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.704+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f34c8010d40 con 0x7f34e009a490 2024-09-17T15:40:48.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.704+0000 7f34dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:48.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.704+0000 7f34dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:48.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34e63be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:48.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34e63be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:48.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34e63be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:48.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f34e00a71d0 con 0x7f34e009a490 2024-09-17T15:40:48.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34dd7fa640 1 -- 172.21.15.86:0/4029879686 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f34c8016d40 con 0x7f34e009a490 2024-09-17T15:40:48.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:48.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.948+0000 7f34dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:48.950 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:48.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 msgr2=0x7f34c003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:48.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 0x7f34c003e070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f34e00a19e0 tx=0x7f34d0008040 comp rx=0 tx=0).stop 2024-09-17T15:40:48.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 10 monclient: shutdown 2024-09-17T15:40:48.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:48.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 msgr2=0x7f34e01d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f34c800eb40 tx=0x7f34c800cf70 comp rx=0 tx=0).stop 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 shutdown_connections 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f34c003bbb0 0x7f34c003e070 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34e009ef90 0x7f34e01317e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e009a490 0x7f34e01d11f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 --2- 172.21.15.86:0/4029879686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e00689d0 0x7f34e01d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 >> 172.21.15.86:0/4029879686 conn(0x7f34e00a73f0 msgr2=0x7f34e0133d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 shutdown_connections 2024-09-17T15:40:48.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:48.952+0000 7f34e63be640 1 -- 172.21.15.86:0/4029879686 wait complete. 2024-09-17T15:40:50.325 INFO:tasks.rados.rados.0.smithi203.stdout:2030: done (0 left) 2024-09-17T15:40:50.326 INFO:tasks.rados.rados.0.smithi203.stdout:2031: rollback oid 47 current snap is 227 2024-09-17T15:40:50.326 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 191 2024-09-17T15:40:50.326 INFO:tasks.rados.rados.0.smithi203.stdout:2032: read oid 19 snap 212 2024-09-17T15:40:50.326 INFO:tasks.rados.rados.0.smithi203.stdout:2032: expect deleted 2024-09-17T15:40:50.326 INFO:tasks.rados.rados.0.smithi203.stdout:2033: snap_remove snap 184 2024-09-17T15:40:51.332 INFO:tasks.rados.rados.0.smithi203.stdout:2033: done (2 left) 2024-09-17T15:40:51.332 INFO:tasks.rados.rados.0.smithi203.stdout:2034: read oid 30 snap -1 2024-09-17T15:40:51.332 INFO:tasks.rados.rados.0.smithi203.stdout:2034: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:40:51.332 INFO:tasks.rados.rados.0.smithi203.stdout:2035: copy_from oid 44 from oid 39 current snap is 227 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2031: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2031: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2031: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 822 (ObjNum 506 snap 151 seq_num 506) dirty exists 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2031: done (3 left) 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2032: done (2 left) 2024-09-17T15:40:51.333 INFO:tasks.rados.rados.0.smithi203.stdout:2036: snap_remove snap 225 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2036: done (2 left) 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2037: read oid 2 snap 224 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2037: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2038: read oid 13 snap -1 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2038: expect (ObjNum 718 snap 227 seq_num 718) 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2039: delete oid 48 current snap is 227 2024-09-17T15:40:51.688 INFO:tasks.rados.rados.0.smithi203.stdout:2034: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:40:51.728 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2035: finishing copy_from racing read to smithi20328086-44 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2035: finishing copy_from to smithi20328086-44 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 935 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2034: done (4 left) 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2035: done (3 left) 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2039: done (2 left) 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2040: write oid 20 current snap is 227 2024-09-17T15:40:51.799 INFO:tasks.rados.rados.0.smithi203.stdout:2038: expect (ObjNum 718 snap 227 seq_num 718) 2024-09-17T15:40:51.885 INFO:tasks.rados.rados.0.smithi203.stdout:2037: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:40:51.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:51.970 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: get_monmap_and_config 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: build_initial_monmap 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: monmap: 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:52.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:52.101611+0000 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:52.101611+0000 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:52.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 Processor -- start 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 -- start start 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: init 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94bc0689d0 0x7f94bc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-a con 0x7f94bc0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94bc068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:52.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc13aff0 0x7f94bc13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-b con 0x7f94bc13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94bc074470 con 0x7f94bc068db0 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f94bc09c900 con 0x7f94bc0689d0 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc09cb90 con 0x7f94bc13aff0 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c3895640 10 monclient(hunting): authenticate will time out at 1785.067068s 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94bc0689d0 0x7f94bc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94bc0689d0 0x7f94bc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55508/0 (socket says 172.21.15.86:55508) 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 -- 172.21.15.86:0/1326941654 learned_addr learned my addr 172.21.15.86:0/1326941654 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): get_auth_request con 0x7f94bc0689d0 auth_method 0 2024-09-17T15:40:52.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1e32640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc13aff0 0x7f94bc13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1e32640 10 monclient(hunting): get_auth_request con 0x7f94bc13aff0 auth_method 0 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1e32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1e32640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1e32640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient(hunting): handle_auth_done global_id 144219 payload 274 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 -- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc13aff0 msgr2=0x7f94bc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc13aff0 0x7f94bc13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 -- 172.21.15.86:0/1326941654 >> v1:172.21.15.203:6789/0 conn(0x7f94bc068db0 legacy=0x7f94bc069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:52.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient: _finish_hunting 0 2024-09-17T15:40:52.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 monclient: found mon.noname-a 2024-09-17T15:40:52.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:52.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.096+0000 7f94c1631640 1 -- 172.21.15.86:0/1326941654 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc09ce70 con 0x7f94bc0689d0 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.100+0000 7f94c1631640 1 -- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc0689d0 msgr2=0x7f94bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.100+0000 7f94c1631640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc0689d0 0x7f94bc06f840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfc7c826ab47c3c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 1 -- 172.21.15.86:0/1326941654 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac0b1070 con 0x7f94bc0689d0 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: dump: 2024-09-17T15:40:52.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: _finish_auth 0 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 1 -- 172.21.15.86:0/1326941654 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94ac0ac620 con 0x7f94bc0689d0 2024-09-17T15:40:52.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 5 monclient: authenticate success, global_id 144219 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 1 -- 172.21.15.86:0/1326941654 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc09c2d0 con 0x7f94bc0689d0 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 4 set_mon_vals no callback set 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94b3fff640 10 monclient: dump: 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.107 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: shutdown 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc0689d0 msgr2=0x7f94bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc0689d0 0x7f94bc06f840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009870 tx=0x7f94ac01c420 comp rx=0 tx=0).stop 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- 172.21.15.86:0/1326941654 shutdown_connections 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc13aff0 0x7f94bc13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- 172.21.15.86:0/1326941654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc0689d0 0x7f94bc06f840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- 172.21.15.86:0/1326941654 >> 172.21.15.86:0/1326941654 conn(0x7f94bc0a6440 msgr2=0x7f94bc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:52.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- 172.21.15.86:0/1326941654 shutdown_connections 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- 172.21.15.86:0/1326941654 wait complete. 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: build_initial_monmap 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: monmap: 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 Processor -- start 2024-09-17T15:40:52.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- start start 2024-09-17T15:40:52.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: init 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-c con 0x7f94bc0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc068db0 0x7f94bc135d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-b con 0x7f94bc068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 0x7f94bc1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): picked mon.noname-a con 0x7f94bc13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc140610 con 0x7f94bc13aff0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc140490 con 0x7f94bc068db0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94bc140790 con 0x7f94bc0689d0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c3895640 10 monclient(hunting): authenticate will time out at 1785.074576s 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 0x7f94bc1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57440/0 (socket says 172.21.15.86:57440) 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): get_auth_request con 0x7f94bc13aff0 auth_method 0 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 1 -- 172.21.15.86:0/3062753627 learned_addr learned my addr 172.21.15.86:0/3062753627 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): get_auth_request con 0x7f94bc0689d0 auth_method 0 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c0e30640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc068db0 0x7f94bc135d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c0e30640 10 monclient(hunting): get_auth_request con 0x7f94bc068db0 auth_method 0 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c0e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c0e30640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c0e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient(hunting): handle_auth_done global_id 144222 payload 824 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 -- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 msgr2=0x7f94bc1cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 -- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc068db0 msgr2=0x7f94bc135d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc068db0 0x7f94bc135d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient: _finish_hunting 0 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 monclient: found mon.noname-a 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1e32640 1 -- 172.21.15.86:0/3062753627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc13d390 con 0x7f94bc13aff0 2024-09-17T15:40:52.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.104+0000 7f94c1631640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c1e32640 1 -- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 msgr2=0x7f94bc1362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c1e32640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 0x7f94bc1362c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94b4007d90 tx=0x7f94b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b40039a0 con 0x7f94bc13aff0 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: dump: 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b4003b40 con 0x7f94bc13aff0 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4004020 con 0x7f94bc13aff0 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 10 monclient: dump: 2024-09-17T15:40:52.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b37fe640 4 set_mon_vals no callback set 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 5 monclient: authenticate success, global_id 144222 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _renew_subs 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc122730 con 0x7f94bc13aff0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _renew_subs 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc073690 con 0x7f94bc13aff0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc06f840 con 0x7f94bc13aff0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f94b401b020 con 0x7f94bc13aff0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(465..465 src has 213..465) ==== 4891+0+0 (secure 0 0 0) 0x7f94b4050140 con 0x7f94bc13aff0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c1631640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.108+0000 7f94c1631640 10 monclient: get_auth_request con 0x7f94a003bbb0 auth_method 0 2024-09-17T15:40:52.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.112+0000 7f94c1631640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0af040 tx=0x7f94ac0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:52.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.112+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f94bc13d390 con 0x7f94bc13aff0 2024-09-17T15:40:52.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.112+0000 7f94b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:52.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.112+0000 7f94b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:52.129 INFO:tasks.rados.rados.0.smithi203.stdout:2040: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-17T15:40:52.136 INFO:tasks.rados.rados.0.smithi203.stdout:2040: writing smithi20328086-20 from 568268 to 1302079 tid 1 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2040: writing smithi20328086-20 from 2000022 to 2168268 tid 2 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2037: done (2 left) 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2038: done (1 left) 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2041: read oid 29 snap -1 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2041: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2042: write oid 33 current snap is 227 2024-09-17T15:40:52.142 INFO:tasks.rados.rados.0.smithi203.stdout:2042: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-17T15:40:52.147 INFO:tasks.rados.rados.0.smithi203.stdout:2042: writing smithi20328086-33 from 585075 to 994135 tid 1 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2042: writing smithi20328086-33 from 1542151 to 2185075 tid 2 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2043: copy_from oid 32 from oid 44 current snap is 227 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: finishing write tid 1 to smithi20328086-20 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: finishing write tid 2 to smithi20328086-20 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: finishing write tid 3 to smithi20328086-20 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2044: snap_create 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: finishing write tid 4 to smithi20328086-20 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: oid 20 updating version 0 to 949 2024-09-17T15:40:52.155 INFO:tasks.rados.rados.0.smithi203.stdout:2040: oid 20 version 949 is already newer than 948 2024-09-17T15:40:52.156 INFO:tasks.rados.rados.0.smithi203.stdout:2040: oid 20 updating version 949 to 950 2024-09-17T15:40:52.156 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 950 (ObjNum 724 snap 227 seq_num 724) dirty exists 2024-09-17T15:40:52.156 INFO:tasks.rados.rados.0.smithi203.stdout:2040: left oid 20 (ObjNum 724 snap 227 seq_num 724) 2024-09-17T15:40:52.156 INFO:tasks.rados.rados.0.smithi203.stdout:2041: expect (ObjNum 578 snap 172 seq_num 578) 2024-09-17T15:40:52.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94c3895640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:52.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94c3895640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:52.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94c3895640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:40:52.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f94bc1224b0 con 0x7f94bc13aff0 2024-09-17T15:40:52.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94b1ffb640 1 -- 172.21.15.86:0/3062753627 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f94b4013c60 con 0x7f94bc13aff0 2024-09-17T15:40:52.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:52.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.356+0000 7f94b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:52.357 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:52.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 msgr2=0x7f94a003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:52.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f94ac0af040 tx=0x7f94ac0a7000 comp rx=0 tx=0).stop 2024-09-17T15:40:52.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 10 monclient: shutdown 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 msgr2=0x7f94bc1362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 0x7f94bc1362c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94b4007d90 tx=0x7f94b400e4b0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 shutdown_connections 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94a003bbb0 0x7f94a003e070 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94bc13aff0 0x7f94bc1362c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc068db0 0x7f94bc135d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 --2- 172.21.15.86:0/3062753627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94bc0689d0 0x7f94bc1cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 >> 172.21.15.86:0/3062753627 conn(0x7f94bc0a6440 msgr2=0x7f94bc0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 shutdown_connections 2024-09-17T15:40:52.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:52.360+0000 7f94c3895640 1 -- 172.21.15.86:0/3062753627 wait complete. 2024-09-17T15:40:52.365 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3446881 2024-09-17T15:40:52.459 INFO:tasks.rados.rados.0.smithi203.stdout:2042: finishing write tid 1 to smithi20328086-33 2024-09-17T15:40:52.459 INFO:tasks.rados.rados.0.smithi203.stdout:2042: finishing write tid 2 to smithi20328086-33 2024-09-17T15:40:52.459 INFO:tasks.rados.rados.0.smithi203.stdout:2042: finishing write tid 3 to smithi20328086-33 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:2042: finishing write tid 4 to smithi20328086-33 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:2042: oid 33 updating version 0 to 515 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:2042: oid 33 version 515 is already newer than 514 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:2042: oid 33 version 515 is already newer than 513 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 515 (ObjNum 725 snap 227 seq_num 725) dirty exists 2024-09-17T15:40:52.460 INFO:tasks.rados.rados.0.smithi203.stdout:2042: left oid 33 (ObjNum 725 snap 227 seq_num 725) 2024-09-17T15:40:52.461 INFO:tasks.rados.rados.0.smithi203.stdout:2043: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:40:52.461 INFO:tasks.rados.rados.0.smithi203.stdout:2043: finishing copy_from to smithi20328086-32 2024-09-17T15:40:52.461 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 935 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2040: done (4 left) 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2041: done (3 left) 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2042: done (2 left) 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2043: done (1 left) 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2044: done (0 left) 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2045: write (excl) oid 11 current snap is 228 2024-09-17T15:40:52.692 INFO:tasks.rados.rados.0.smithi203.stdout:2045: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-17T15:40:52.702 INFO:tasks.rados.rados.0.smithi203.stdout:2045: writing smithi20328086-11 from 601882 to 1086191 tid 1 2024-09-17T15:40:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2045: writing smithi20328086-11 from 1600633 to 2125537 tid 2 2024-09-17T15:40:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2046: write (excl) oid 31 current snap is 228 2024-09-17T15:40:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2046: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-17T15:40:52.725 INFO:tasks.rados.rados.0.smithi203.stdout:2046: writing smithi20328086-31 from 618689 to 1178247 tid 1 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2046: writing smithi20328086-31 from 1659115 to 2218689 tid 2 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2047: snap_create 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2045: finishing write tid 1 to smithi20328086-11 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2045: finishing write tid 2 to smithi20328086-11 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2045: finishing write tid 3 to smithi20328086-11 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2045: finishing write tid 4 to smithi20328086-11 2024-09-17T15:40:52.736 INFO:tasks.rados.rados.0.smithi203.stdout:2045: oid 11 updating version 0 to 936 2024-09-17T15:40:52.737 INFO:tasks.rados.rados.0.smithi203.stdout:2045: oid 11 updating version 936 to 937 2024-09-17T15:40:52.737 INFO:tasks.rados.rados.0.smithi203.stdout:2045: oid 11 updating version 937 to 938 2024-09-17T15:40:52.737 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 938 (ObjNum 726 snap 228 seq_num 726) dirty exists 2024-09-17T15:40:52.737 INFO:tasks.rados.rados.0.smithi203.stdout:2045: left oid 11 (ObjNum 726 snap 228 seq_num 726) 2024-09-17T15:40:52.742 INFO:tasks.rados.rados.0.smithi203.stdout:2046: finishing write tid 1 to smithi20328086-31 2024-09-17T15:40:52.743 INFO:tasks.rados.rados.0.smithi203.stdout:2046: finishing write tid 2 to smithi20328086-31 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: finishing write tid 3 to smithi20328086-31 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: finishing write tid 4 to smithi20328086-31 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: oid 31 updating version 0 to 529 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: oid 31 version 529 is already newer than 528 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: oid 31 version 529 is already newer than 527 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 529 (ObjNum 727 snap 228 seq_num 727) dirty exists 2024-09-17T15:40:52.744 INFO:tasks.rados.rados.0.smithi203.stdout:2046: left oid 31 (ObjNum 727 snap 228 seq_num 727) 2024-09-17T15:40:53.702 INFO:tasks.rados.rados.0.smithi203.stdout:2045: done (2 left) 2024-09-17T15:40:53.702 INFO:tasks.rados.rados.0.smithi203.stdout:2046: done (1 left) 2024-09-17T15:40:53.702 INFO:tasks.rados.rados.0.smithi203.stdout:2047: done (0 left) 2024-09-17T15:40:53.702 INFO:tasks.rados.rados.0.smithi203.stdout:2048: delete oid 20 current snap is 229 2024-09-17T15:40:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2048: done (0 left) 2024-09-17T15:40:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2049: read oid 2 snap -1 2024-09-17T15:40:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2049: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:40:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2050: copy_from oid 31 from oid 5 current snap is 229 2024-09-17T15:40:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2051: snap_remove snap 191 2024-09-17T15:40:54.930 INFO:tasks.rados.rados.0.smithi203.stdout:2051: done (2 left) 2024-09-17T15:40:54.931 INFO:tasks.rados.rados.0.smithi203.stdout:2052: read oid 3 snap -1 2024-09-17T15:40:54.931 INFO:tasks.rados.rados.0.smithi203.stdout:2052: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:40:54.931 INFO:tasks.rados.rados.0.smithi203.stdout:2053: delete oid 22 current snap is 229 2024-09-17T15:40:54.931 INFO:tasks.rados.rados.0.smithi203.stdout:2049: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:40:55.140 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:55.227 INFO:tasks.rados.rados.0.smithi203.stdout:2050: finishing copy_from to smithi20328086-31 2024-09-17T15:40:55.227 INFO:tasks.rados.rados.0.smithi203.stdout:2050: got expected ENOENT (src dne) 2024-09-17T15:40:55.227 INFO:tasks.rados.rados.0.smithi203.stdout:2050: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:40:55.227 INFO:tasks.rados.rados.0.smithi203.stdout:2052: expect (ObjNum 619 snap 192 seq_num 619) 2024-09-17T15:40:55.227 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:55.266 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 516203 2024-09-17T15:40:55.311 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1818256 2024-09-17T15:40:55.324 INFO:tasks.rados.rados.0.smithi203.stdout:2049: done (3 left) 2024-09-17T15:40:55.324 INFO:tasks.rados.rados.0.smithi203.stdout:2050: done (2 left) 2024-09-17T15:40:55.325 INFO:tasks.rados.rados.0.smithi203.stdout:2052: done (1 left) 2024-09-17T15:40:55.325 INFO:tasks.rados.rados.0.smithi203.stdout:2053: done (0 left) 2024-09-17T15:40:55.325 INFO:tasks.rados.rados.0.smithi203.stdout:2054: snap_remove snap 195 2024-09-17T15:40:55.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: get_monmap_and_config 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: build_initial_monmap 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: monmap: 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:55.513224+0000 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:55.513224+0000 2024-09-17T15:40:55.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:55.510 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 Processor -- start 2024-09-17T15:40:55.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 -- start start 2024-09-17T15:40:55.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: init 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa2640a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 0x7fa26409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-b con 0x7fa2640a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa26413d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2640a6f40 con 0x7fa2640a5070 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2640a65e0 con 0x7fa26413d280 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa26409c2a0 con 0x7fa2640a5b60 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa268ac6640 10 monclient(hunting): authenticate will time out at 1788.479510s 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 0x7fa26409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa26259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa2640a5070 0x7fa2640a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36630/0 (socket says 172.21.15.86:36630) 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa26259c640 1 -- 172.21.15.86:0/3628721956 learned_addr learned my addr 172.21.15.86:0/3628721956 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): get_auth_request con 0x7fa2640a5b60 auth_method 0 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.508+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 10 monclient(hunting): handle_auth_done global_id 104603 payload 274 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 1 -- 172.21.15.86:0/3628721956 >> v1:172.21.15.203:6789/0 conn(0x7fa26413d280 legacy=0x7fa26413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 1 -- 172.21.15.86:0/3628721956 >> v1:172.21.15.86:6789/0 conn(0x7fa2640a5070 legacy=0x7fa2640a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 1 monclient: found mon.noname-b 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.512+0000 7fa259d9b640 1 -- 172.21.15.86:0/3628721956 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26409c580 con 0x7fa2640a5b60 2024-09-17T15:40:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 1 -- 172.21.15.86:0/3628721956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 msgr2=0x7fa26409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 1 --2- 172.21.15.86:0/3628721956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 0x7fa26409cf60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0097f0 tx=0x7fa24c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb79305854c8a449 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 1 -- 172.21.15.86:0/3628721956 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0b0070 con 0x7fa2640a5b60 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: dump: 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.518 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: _finish_auth 0 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 1 -- 172.21.15.86:0/3628721956 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa24c0ab600 con 0x7fa2640a5b60 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 1 -- 172.21.15.86:0/3628721956 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24c0ab9d0 con 0x7fa2640a5b60 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 5 monclient: authenticate success, global_id 104603 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 4 set_mon_vals no callback set 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:55.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa261d9b640 10 monclient: dump: 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: shutdown 2024-09-17T15:40:55.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- 172.21.15.86:0/3628721956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 msgr2=0x7fa26409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 --2- 172.21.15.86:0/3628721956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 0x7fa26409cf60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0097f0 tx=0x7fa24c01c3a0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- 172.21.15.86:0/3628721956 shutdown_connections 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 --2- 172.21.15.86:0/3628721956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa2640a5b60 0x7fa26409cf60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- 172.21.15.86:0/3628721956 >> 172.21.15.86:0/3628721956 conn(0x7fa2640742a0 msgr2=0x7fa2640746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- 172.21.15.86:0/3628721956 shutdown_connections 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- 172.21.15.86:0/3628721956 wait complete. 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: build_initial_monmap 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: monmap: 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:55.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 Processor -- start 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- start start 2024-09-17T15:40:55.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: init 2024-09-17T15:40:55.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:55.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:40:55.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2640a5070 0x7fa264138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-b con 0x7fa2640a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-a con 0x7fa2640a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): picked mon.noname-c con 0x7fa26413d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa264143940 con 0x7fa2640a5b60 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641437c0 con 0x7fa2640a5070 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa264143ac0 con 0x7fa26413d280 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa268ac6640 10 monclient(hunting): authenticate will time out at 1788.486767s 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57448/0 (socket says 172.21.15.86:57448) 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 -- 172.21.15.86:0/1497118006 learned_addr learned my addr 172.21.15.86:0/1497118006 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55534/0 (socket says 172.21.15.86:55534) 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): get_auth_request con 0x7fa26413d280 auth_method 0 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): get_auth_request con 0x7fa2640a5b60 auth_method 0 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa26259c640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2640a5070 0x7fa264138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa26259c640 10 monclient(hunting): get_auth_request con 0x7fa2640a5070 auth_method 0 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa26259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa26259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa26259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient(hunting): handle_auth_done global_id 104606 payload 824 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 -- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2640a5070 msgr2=0x7fa264138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2640a5070 0x7fa264138030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 -- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 msgr2=0x7fa26409e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 monclient: found mon.noname-c 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa262d9d640 1 -- 172.21.15.86:0/1497118006 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2641219b0 con 0x7fa26413d280 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.516+0000 7fa259d9b640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa262d9d640 1 -- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 msgr2=0x7fa26409e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa262d9d640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c780 tx=0x7fa25c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c00ac60 con 0x7fa26413d280 2024-09-17T15:40:55.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: dump: 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25c00ae00 con 0x7fa26413d280 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 5 monclient: authenticate success, global_id 104606 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _renew_subs 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26409fcc0 con 0x7fa26413d280 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c004bb0 con 0x7fa26413d280 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 10 monclient: dump: 2024-09-17T15:40:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa26159a640 4 set_mon_vals no callback set 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _renew_subs 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa264120cc0 con 0x7fa26413d280 2024-09-17T15:40:55.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa26409cfe0 con 0x7fa26413d280 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa25c006b60 con 0x7fa26413d280 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 0x7fa24403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(468..468 src has 213..468) ==== 4831+0+0 (secure 0 0 0) 0x7fa25c050220 con 0x7fa26413d280 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa26259c640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 0x7fa24403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.520+0000 7fa26259c640 10 monclient: get_auth_request con 0x7fa24403bb60 auth_method 0 2024-09-17T15:40:55.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.524+0000 7fa26259c640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 0x7fa24403e020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa254007fd0 tx=0x7fa254009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.524+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa25c020070 con 0x7fa26413d280 2024-09-17T15:40:55.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.524+0000 7fa25b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:55.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.524+0000 7fa25b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:55.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.760+0000 7fa268ac6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:40:55.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.760+0000 7fa268ac6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:55.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.760+0000 7fa268ac6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.760+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa264121630 con 0x7fa26413d280 2024-09-17T15:40:55.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa25b7fe640 1 -- 172.21.15.86:0/1497118006 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fa25c004e10 con 0x7fa26413d280 2024-09-17T15:40:55.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa25b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:40:55.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa25b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:55.764 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 msgr2=0x7fa24403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 0x7fa24403e020 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa254007fd0 tx=0x7fa254009040 comp rx=0 tx=0).stop 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 10 monclient: shutdown 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 msgr2=0x7fa26409e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00c780 tx=0x7fa25c00cc40 comp rx=0 tx=0).stop 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 shutdown_connections 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa24403bb60 0x7fa24403e020 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa26413d280 0x7fa26409e7c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2640a5b60 0x7fa26409e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 --2- 172.21.15.86:0/1497118006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2640a5070 0x7fa264138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 >> 172.21.15.86:0/1497118006 conn(0x7fa2640742a0 msgr2=0x7fa2640a40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:55.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 shutdown_connections 2024-09-17T15:40:55.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.764+0000 7fa268ac6640 1 -- 172.21.15.86:0/1497118006 wait complete. 2024-09-17T15:40:55.783 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:40:55.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:40:55.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:40:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: get_monmap_and_config 2024-09-17T15:40:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: build_initial_monmap 2024-09-17T15:40:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: monmap: 2024-09-17T15:40:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:55.908699+0000 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:55.908699+0000 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:55.905 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 Processor -- start 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 -- start start 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: init 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbd140a4f70 0x7fbd1409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-c con 0x7fbd140a4f70 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd140a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1413d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd1409d430 con 0x7fbd140a5350 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbd1409c7f0 con 0x7fbd140a4f70 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbd140a68d0 con 0x7fbd1413d180 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1bb51640 10 monclient(hunting): authenticate will time out at 1788.874154s 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57460/0 (socket says 172.21.15.86:57460) 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/19711034 learned_addr learned my addr 172.21.15.86:0/19711034 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:55.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): get_auth_request con 0x7fbd1413d180 auth_method 0 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd198ed640 1 --2- 172.21.15.86:0/19711034 >> v2:172.21.15.203:3300/0 conn(0x7fbd140a4f70 0x7fbd1409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd198ed640 10 monclient(hunting): get_auth_request con 0x7fbd140a4f70 auth_method 0 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd198ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd198ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd198ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient(hunting): handle_auth_done global_id 104609 payload 274 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/19711034 >> v2:172.21.15.203:3300/0 conn(0x7fbd140a4f70 msgr2=0x7fbd1409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 --2- 172.21.15.86:0/19711034 >> v2:172.21.15.203:3300/0 conn(0x7fbd140a4f70 0x7fbd1409cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/19711034 >> v1:172.21.15.203:6789/0 conn(0x7fbd140a5350 legacy=0x7fbd140a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient: _finish_hunting 0 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 monclient: found mon.noname-b 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.904+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/19711034 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd140a64a0 con 0x7fbd1413d180 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/19711034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 msgr2=0x7fbd1413f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1a0ee640 1 --2- 172.21.15.86:0/19711034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd1406e190 tx=0x7fbd0c01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=552906a81c454283 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 1 -- 172.21.15.86:0/19711034 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0a56c0 con 0x7fbd1413d180 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: dump: 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: _finish_auth 0 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: _check_auth_tickets 2024-09-17T15:40:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 1 -- 172.21.15.86:0/19711034 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c0a5860 con 0x7fbd1413d180 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 5 monclient: authenticate success, global_id 104609 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 1 -- 172.21.15.86:0/19711034 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0a5c30 con 0x7fbd1413d180 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 4 set_mon_vals no callback set 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd188eb640 10 monclient: dump: 2024-09-17T15:40:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 10 monclient: shutdown 2024-09-17T15:40:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 1 -- 172.21.15.86:0/19711034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 msgr2=0x7fbd1413f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/19711034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd1406e190 tx=0x7fbd0c01c880 comp rx=0 tx=0).stop 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.908+0000 7fbd1bb51640 1 -- 172.21.15.86:0/19711034 shutdown_connections 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/19711034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd1413d180 0x7fbd1413f570 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/19711034 >> v2:172.21.15.203:3300/0 conn(0x7fbd140a4f70 0x7fbd1409cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- 172.21.15.86:0/19711034 >> 172.21.15.86:0/19711034 conn(0x7fbd140a7240 msgr2=0x7fbd140a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- 172.21.15.86:0/19711034 shutdown_connections 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- 172.21.15.86:0/19711034 wait complete. 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient: build_initial_monmap 2024-09-17T15:40:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient: monmap: 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 Processor -- start 2024-09-17T15:40:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- start start 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient: init 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 0x7fbd141cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-c con 0x7fbd140a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd140a5350 0x7fbd141d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-a con 0x7fbd140a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd1413d180 0x7fbd141d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1413d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbd141424e0 con 0x7fbd140a5350 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1406f9a0 con 0x7fbd1413d180 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbd14142660 con 0x7fbd140a4f70 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1bb51640 10 monclient(hunting): authenticate will time out at 1788.881745s 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 0x7fbd141cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 0x7fbd141cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57470/0 (socket says 172.21.15.86:57470) 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 -- 172.21.15.86:0/1375063929 learned_addr learned my addr 172.21.15.86:0/1375063929 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): get_auth_request con 0x7fbd140a4f70 auth_method 0 2024-09-17T15:40:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd140a5350 0x7fbd141d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): get_auth_request con 0x7fbd140a5350 auth_method 0 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1a0ee640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd1413d180 0x7fbd141d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1a0ee640 10 monclient(hunting): get_auth_request con 0x7fbd1413d180 auth_method 0 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1a0ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1a0ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd1a0ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient(hunting): handle_auth_done global_id 104612 payload 824 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd1413d180 msgr2=0x7fbd141d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd1413d180 0x7fbd141d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd140a5350 msgr2=0x7fbd141d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd140a5350 0x7fbd141d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient: _finish_hunting 0 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 monclient: found mon.noname-c 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd198ed640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd1413de80 con 0x7fbd140a4f70 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.912+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd140a5350 0x7fbd141d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.916+0000 7fbd198ed640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 msgr2=0x7fbd141cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.916+0000 7fbd198ed640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 0x7fbd141cfaf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000d800 tx=0x7fbd1000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.927 INFO:tasks.rados.rados.0.smithi203.stdout:2054: done (0 left) 2024-09-17T15:40:55.927 INFO:tasks.rados.rados.0.smithi203.stdout:2055: snap_remove snap 209 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000b8d0 con 0x7fbd140a4f70 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: dump: 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd10019070 con 0x7fbd140a4f70 2024-09-17T15:40:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 5 monclient: authenticate success, global_id 104612 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 monclient: _renew_subs 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd1413b2d0 con 0x7fbd140a4f70 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1000bc70 con 0x7fbd140a4f70 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0a7fc640 10 monclient: dump: 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd0bfff640 4 set_mon_vals no callback set 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 monclient: _renew_subs 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd14139160 con 0x7fbd140a4f70 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 0x7fbd141350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.924+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1409cef0 con 0x7fbd14134d00 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbd1001e460 con 0x7fbd140a4f70 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 0x7fbd141350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient: get_auth_request con 0x7fbd14134d00 auth_method 0 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 10 monclient(hunting): handle_auth_done global_id 144231 payload 824 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd1413d950 con 0x7fbd14134d00 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbcf803bb60 0x7fbcf803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(469..469 src has 213..469) ==== 4847+0+0 (secure 0 0 0) 0x7fbd10054170 con 0x7fbd140a4f70 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd1a0ee640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbcf803bb60 0x7fbcf803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd1a0ee640 10 monclient: get_auth_request con 0x7fbcf803bb60 auth_method 0 2024-09-17T15:40:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 msgr2=0x7fbd141350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 0x7fbd141350e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400fae0 tx=0x7fbd04004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0400ddc0 con 0x7fbd14134d00 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 10 monclient: dump: 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbd0400ddc0 con 0x7fbd14134d00 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 msgr2=0x7fbd141350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 0x7fbd141350e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400fae0 tx=0x7fbd04004510 comp rx=0 tx=0).stop 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 msgr2=0x7fbd141350e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.928+0000 7fbd0a7fc640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14134d00 0x7fbd141350e0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbd0400fae0 tx=0x7fbd04004510 comp rx=0 tx=0).stop 2024-09-17T15:40:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.932+0000 7fbd1a0ee640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbcf803bb60 0x7fbcf803e020 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbd1409d9c0 tx=0x7fbd0c0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd1bb51640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:55.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 0x7fbd1409df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:55.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd1bb51640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:55.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbd140a5ff0 con 0x7fbd14135620 2024-09-17T15:40:55.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 0x7fbd1409df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient: get_auth_request con 0x7fbd14135620 auth_method 0 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 10 monclient(hunting): handle_auth_done global_id 144234 payload 824 2024-09-17T15:40:55.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 1 -- 172.21.15.86:0/1375063929 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd14069920 con 0x7fbd14135620 2024-09-17T15:40:55.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 msgr2=0x7fbd1409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:55.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd190ec640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 0x7fbd1409df40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd04018000 tx=0x7fbd04003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd040047f0 con 0x7fbd14135620 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 10 monclient: dump: 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.943 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fbd040047f0 con 0x7fbd14135620 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 msgr2=0x7fbd1409df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 0x7fbd1409df40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd04018000 tx=0x7fbd04003770 comp rx=0 tx=0).stop 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 msgr2=0x7fbd1409df40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.940+0000 7fbd0a7fc640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd14135620 0x7fbd1409df40 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbd04018000 tx=0x7fbd04003770 comp rx=0 tx=0).stop 2024-09-17T15:40:55.945 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 74282, 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.946 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:55.947 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:55.948 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:55.949 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:55.950 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:55.951 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:55.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.953 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:55.954 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:55.956 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:55.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.959 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.960 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:55.961 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbcf803bb60 msgr2=0x7fbcf803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbcf803bb60 0x7fbcf803e020 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fbd1409d9c0 tx=0x7fbd0c0a7070 comp rx=0 tx=0).stop 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 10 monclient: shutdown 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 msgr2=0x7fbd141cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 --2- 172.21.15.86:0/1375063929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd140a4f70 0x7fbd141cfaf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbd1000d800 tx=0x7fbd1000dcc0 comp rx=0 tx=0).stop 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/1375063929 reap_dead start 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1a0ee640 1 -- 172.21.15.86:0/1375063929 reap_dead start 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 shutdown_connections 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 >> 172.21.15.86:0/1375063929 conn(0x7fbd140a7240 msgr2=0x7fbd1413c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 shutdown_connections 2024-09-17T15:40:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:55.944+0000 7fbd1bb51640 1 -- 172.21.15.86:0/1375063929 wait complete. 2024-09-17T15:40:55.964 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:55.964 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:40:55.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:40:56.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: get_monmap_and_config 2024-09-17T15:40:56.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: build_initial_monmap 2024-09-17T15:40:56.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: monmap: 2024-09-17T15:40:56.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:56.080646+0000 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:56.080646+0000 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:56.077 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 Processor -- start 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 -- start start 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: init 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f796009d770 0x7f796009db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-c con 0x7f796009d770 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f796009e090 addr v1:172.21.15.86:6789/0 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7960135150 0x7f796013cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-a con 0x7f7960135150 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:56.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f796009ff00 con 0x7f796009e090 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f796009e7f0 con 0x7f7960135150 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f79600a2110 con 0x7f796009d770 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7964df5640 10 monclient(hunting): authenticate will time out at 1789.046215s 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7960135150 0x7f796013cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7955d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f796009e090 0x7f7960134a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36640/0 (socket says 172.21.15.86:36640) 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7960135150 0x7f796013cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55570/0 (socket says 172.21.15.86:55570) 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f7955d9b640 1 -- 172.21.15.86:0/521107587 learned_addr learned my addr 172.21.15.86:0/521107587 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): get_auth_request con 0x7f7960135150 auth_method 0 2024-09-17T15:40:56.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795e59c640 1 --2- 172.21.15.86:0/521107587 >> v2:172.21.15.203:3300/0 conn(0x7f796009d770 0x7f796009db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient(hunting): handle_auth_done global_id 144237 payload 274 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 -- 172.21.15.86:0/521107587 >> v2:172.21.15.203:3300/0 conn(0x7f796009d770 msgr2=0x7f796009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 --2- 172.21.15.86:0/521107587 >> v2:172.21.15.203:3300/0 conn(0x7f796009d770 0x7f796009db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 -- 172.21.15.86:0/521107587 >> v1:172.21.15.86:6789/0 conn(0x7f796009e090 legacy=0x7f7960134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:40:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:40:56.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.076+0000 7f795ed9d640 1 -- 172.21.15.86:0/521107587 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796009ec30 con 0x7f7960135150 2024-09-17T15:40:56.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795ed9d640 1 -- 172.21.15.86:0/521107587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7960135150 msgr2=0x7f796013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795ed9d640 1 --2- 172.21.15.86:0/521107587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7960135150 0x7f796013cf20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f796006d160 tx=0x7f795801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a932cdf5e82aa33 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 1 -- 172.21.15.86:0/521107587 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580b2070 con 0x7f7960135150 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: dump: 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 1 -- 172.21.15.86:0/521107587 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79580a5800 con 0x7f7960135150 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f7964df5640 5 monclient: authenticate success, global_id 144237 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f7964df5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f7964df5640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:56.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f7964df5640 4 set_mon_vals no callback set 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 1 -- 172.21.15.86:0/521107587 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580a5bd0 con 0x7f7960135150 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.080+0000 7f795dd9b640 10 monclient: dump: 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: shutdown 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- 172.21.15.86:0/521107587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7960135150 msgr2=0x7f796013cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- 172.21.15.86:0/521107587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7960135150 0x7f796013cf20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f796006d160 tx=0x7f795801c7f0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- 172.21.15.86:0/521107587 shutdown_connections 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- 172.21.15.86:0/521107587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7960135150 0x7f796013cf20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- 172.21.15.86:0/521107587 >> v2:172.21.15.203:3300/0 conn(0x7f796009d770 0x7f796009db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- 172.21.15.86:0/521107587 >> 172.21.15.86:0/521107587 conn(0x7f79600a2f40 msgr2=0x7f79600a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:56.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- 172.21.15.86:0/521107587 shutdown_connections 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- 172.21.15.86:0/521107587 wait complete. 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: build_initial_monmap 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: monmap: 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:56.094 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 Processor -- start 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- start start 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: init 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:40:56.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796009d770 0x7f79601ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-a con 0x7f796009d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:56.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f796009e090 0x7f7960133080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-b con 0x7f796009e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:56.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 0x7f79601d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): picked mon.noname-c con 0x7f7960135150 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f796009cce0 con 0x7f796009d770 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79600a0750 con 0x7f796009e090 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f796009ce60 con 0x7f7960135150 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:56.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7964df5640 10 monclient(hunting): authenticate will time out at 1789.053567s 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 0x7f79601d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 0x7f79601d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57472/0 (socket says 172.21.15.86:57472) 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 learned_addr learned my addr 172.21.15.86:0/3569548759 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): get_auth_request con 0x7f7960135150 auth_method 0 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796009d770 0x7f79601ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): get_auth_request con 0x7f796009d770 auth_method 0 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7955d9b640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f796009e090 0x7f7960133080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7955d9b640 10 monclient(hunting): get_auth_request con 0x7f796009e090 auth_method 0 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7955d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7955d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f7955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient(hunting): handle_auth_done global_id 104615 payload 824 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f796009e090 msgr2=0x7f7960133080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f796009e090 0x7f7960133080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796009d770 msgr2=0x7f79601ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796009d770 0x7f79601ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 monclient: found mon.noname-c 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79600a5600 con 0x7f7960135150 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.084+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f796009d770 0x7f79601ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 msgr2=0x7f79601d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795ed9d640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 0x7f79601d00f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f79580b0040 tx=0x7f7958019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580b2050 con 0x7f7960135150 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: dump: 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: _finish_auth 0 2024-09-17T15:40:56.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: _check_auth_tickets 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79580be070 con 0x7f7960135150 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79580ada60 con 0x7f7960135150 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 10 monclient: dump: 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.102 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 5 monclient: authenticate success, global_id 104615 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 monclient: _renew_subs 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795d59a640 4 set_mon_vals no callback set 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796011e6e0 con 0x7f7960135150 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 monclient: _renew_subs 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7960120a40 con 0x7f7960135150 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 0x7f79601d0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f796009db50 con 0x7f796011f240 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 0x7f79601d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient: get_auth_request con 0x7f796011f240 auth_method 0 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f79580adcc0 con 0x7f7960135150 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 10 monclient(hunting): handle_auth_done global_id 104618 payload 824 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79600008e0 con 0x7f796011f240 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f794003bbe0 0x7f794003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(469..469 src has 213..469) ==== 4847+0+0 (secure 0 0 0) 0x7f79580eafd0 con 0x7f7960135150 2024-09-17T15:40:56.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7955d9b640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f794003bbe0 0x7f794003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f7955d9b640 10 monclient: get_auth_request con 0x7f794003bbe0 auth_method 0 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 msgr2=0x7f79601d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.088+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 0x7f79601d0630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f795000fbc0 tx=0x7f795000d590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950004220 con 0x7f796011f240 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 10 monclient: dump: 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7950004220 con 0x7f796011f240 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 msgr2=0x7f79601d0630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 0x7f79601d0630 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f795000fbc0 tx=0x7f795000d590 comp rx=0 tx=0).stop 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 msgr2=0x7f79601d0630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f79577fe640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f796011f240 0x7f79601d0630 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.092+0000 7f7955d9b640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f794003bbe0 0x7f794003e0a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7948007fd0 tx=0x7f7948009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f7964df5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f7964df5640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 0x7f79601d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f7964df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f79601db4a0 con 0x7f79601335c0 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 0x7f79601d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient: get_auth_request con 0x7f79601335c0 auth_method 0 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 10 monclient(hunting): handle_auth_done global_id 104621 payload 824 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.096+0000 7f795e59c640 1 -- 172.21.15.86:0/3569548759 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79601db650 con 0x7f79601335c0 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f795e59c640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 msgr2=0x7f79601d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f795e59c640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 0x7f79601d0b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f795000fbc0 tx=0x7f79500046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950006040 con 0x7f79601335c0 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 10 monclient: dump: 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f7950006000 con 0x7f79601335c0 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:56.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 msgr2=0x7f79601d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 0x7f79601d0b70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f795000fbc0 tx=0x7f79500046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 msgr2=0x7f79601d0b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f79577fe640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79601335c0 0x7f79601d0b70 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f795000fbc0 tx=0x7f79500046d0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:56.112 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 286975, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:56.113 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:56.115 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:56.116 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:56.117 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:56.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:56.119 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:56.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.123 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:56.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:56.125 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:56.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f794003bbe0 msgr2=0x7f794003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f794003bbe0 0x7f794003e0a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7948007fd0 tx=0x7f7948009040 comp rx=0 tx=0).stop 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 10 monclient: shutdown 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 msgr2=0x7f79601d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 --2- 172.21.15.86:0/3569548759 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7960135150 0x7f79601d00f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f79580b0040 tx=0x7f7958019590 comp rx=0 tx=0).stop 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 reap_dead start 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f795ed9d640 1 -- 172.21.15.86:0/3569548759 reap_dead start 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 shutdown_connections 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 >> 172.21.15.86:0/3569548759 conn(0x7f79600a2f40 msgr2=0x7f7960133d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 shutdown_connections 2024-09-17T15:40:56.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.100+0000 7f7964df5640 1 -- 172.21.15.86:0/3569548759 wait complete. 2024-09-17T15:40:56.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:56.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:40:56.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:40:56.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: get_monmap_and_config 2024-09-17T15:40:56.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: build_initial_monmap 2024-09-17T15:40:56.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: monmap: 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:56.248150+0000 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:56.248150+0000 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:56.245 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:56.246 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 Processor -- start 2024-09-17T15:40:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 -- start start 2024-09-17T15:40:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: init 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 0x7fab1c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-a con 0x7fab1c0a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-b con 0x7fab1c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fab1c137ea0 0x7fab1c1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-c con 0x7fab1c137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c0a6280 con 0x7fab1c0a50e0 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fab1c09c0a0 con 0x7fab1c137ea0 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c09c380 con 0x7fab1c0a54c0 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab23ea9640 10 monclient(hunting): authenticate will time out at 1789.213664s 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 0x7fab1c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57494/0 (socket says 172.21.15.86:57494) 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 -- 172.21.15.86:0/2711495826 learned_addr learned my addr 172.21.15.86:0/2711495826 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): get_auth_request con 0x7fab1c0a54c0 auth_method 0 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): get_auth_request con 0x7fab1c0a50e0 auth_method 0 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab22446640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.203:3300/0 conn(0x7fab1c137ea0 0x7fab1c1382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab22446640 10 monclient(hunting): get_auth_request con 0x7fab1c137ea0 auth_method 0 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab22446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab22446640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab22446640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient(hunting): handle_auth_done global_id 104624 payload 274 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 -- 172.21.15.86:0/2711495826 >> v2:172.21.15.203:3300/0 conn(0x7fab1c137ea0 msgr2=0x7fab1c1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.203:3300/0 conn(0x7fab1c137ea0 0x7fab1c1382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 -- 172.21.15.86:0/2711495826 >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 msgr2=0x7fab1c09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 0x7fab1c09cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient: _finish_hunting 0 2024-09-17T15:40:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 monclient: found mon.noname-b 2024-09-17T15:40:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21444640 1 -- 172.21.15.86:0/2711495826 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c09c660 con 0x7fab1c0a54c0 2024-09-17T15:40:56.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.244+0000 7fab21c45640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 0x7fab1c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab21444640 1 -- 172.21.15.86:0/2711495826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 msgr2=0x7fab1c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab21444640 1 --2- 172.21.15.86:0/2711495826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab1c06f8c0 tx=0x7fab1001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8314a711f578d334 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 1 -- 172.21.15.86:0/2711495826 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100a56f0 con 0x7fab1c0a54c0 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: dump: 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: _finish_auth 0 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: _check_auth_tickets 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 5 monclient: authenticate success, global_id 104624 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 1 -- 172.21.15.86:0/2711495826 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab100a5890 con 0x7fab1c0a54c0 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 1 -- 172.21.15.86:0/2711495826 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100a5c60 con 0x7fab1c0a54c0 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab20c43640 10 monclient: dump: 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.259 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 4 set_mon_vals no callback set 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 10 monclient: shutdown 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 -- 172.21.15.86:0/2711495826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 msgr2=0x7fab1c0a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2711495826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab1c06f8c0 tx=0x7fab1001c8b0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 -- 172.21.15.86:0/2711495826 shutdown_connections 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.203:3300/0 conn(0x7fab1c137ea0 0x7fab1c1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2711495826 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a54c0 0x7fab1c0a58f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2711495826 >> v2:172.21.15.86:3300/0 conn(0x7fab1c0a50e0 0x7fab1c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 -- 172.21.15.86:0/2711495826 >> 172.21.15.86:0/2711495826 conn(0x7fab1c0aaae0 msgr2=0x7fab1c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 -- 172.21.15.86:0/2711495826 shutdown_connections 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.248+0000 7fab23ea9640 1 -- 172.21.15.86:0/2711495826 wait complete. 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient: build_initial_monmap 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient: monmap: 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:56.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 Processor -- start 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 -- start start 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient: init 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 0x7fab1c13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-c con 0x7fab1c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:56.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1c0a54c0 0x7fab1c136fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-a con 0x7fab1c0a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137ea0 0x7fab1c1cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): picked mon.noname-b con 0x7fab1c137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0a6ae0 con 0x7fab1c0a54c0 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c0a6960 con 0x7fab1c137ea0 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c141b20 con 0x7fab1c0a50e0 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab23ea9640 10 monclient(hunting): authenticate will time out at 1789.221104s 2024-09-17T15:40:56.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 0x7fab1c13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 0x7fab1c13d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57508/0 (socket says 172.21.15.86:57508) 2024-09-17T15:40:56.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 -- 172.21.15.86:0/2031477656 learned_addr learned my addr 172.21.15.86:0/2031477656 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:56.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): get_auth_request con 0x7fab1c0a50e0 auth_method 0 2024-09-17T15:40:56.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1c0a54c0 0x7fab1c136fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): get_auth_request con 0x7fab1c0a54c0 auth_method 0 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab22446640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137ea0 0x7fab1c1cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab22446640 10 monclient(hunting): get_auth_request con 0x7fab1c137ea0 auth_method 0 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab22446640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab22446640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab22446640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient(hunting): handle_auth_done global_id 104627 payload 824 2024-09-17T15:40:56.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137ea0 msgr2=0x7fab1c1cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137ea0 0x7fab1c1cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1c0a54c0 msgr2=0x7fab1c136fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1c0a54c0 0x7fab1c136fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient: _finish_hunting 0 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 monclient: found mon.noname-c 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c13ebb0 con 0x7fab1c0a50e0 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab1c0a54c0 0x7fab1c136fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 msgr2=0x7fab1c13d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.252+0000 7fab21c45640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 0x7fab1c13d5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e980 tx=0x7fab1400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14017070 con 0x7fab1c0a50e0 2024-09-17T15:40:56.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: dump: 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1400cc10 con 0x7fab1c0a50e0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 5 monclient: authenticate success, global_id 104627 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 monclient: _renew_subs 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1c0a45e0 con 0x7fab1c0a50e0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14013470 con 0x7fab1c0a50e0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:56.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 10 monclient: dump: 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 monclient: _renew_subs 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1bfff640 4 set_mon_vals no callback set 2024-09-17T15:40:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1c13db10 con 0x7fab1c0a50e0 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c06a0e0 con 0x7fab1c137510 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fab1401a070 con 0x7fab1c0a50e0 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c137510 auth_method 0 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab0803bbb0 0x7fab0803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:56.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(469..469 src has 213..469) ==== 4847+0+0 (secure 0 0 0) 0x7fab1404fe60 con 0x7fab1c0a50e0 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab22446640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab0803bbb0 0x7fab0803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.256+0000 7fab22446640 10 monclient: get_auth_request con 0x7fab0803bbb0 auth_method 0 2024-09-17T15:40:56.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.260+0000 7fab22446640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab0803bbb0 0x7fab0803e070 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00d8d0 tx=0x7fab0c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:56.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c137510 auth_method 0 2024-09-17T15:40:56.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:56.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.460+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:40:56.694 INFO:tasks.rados.rados.0.smithi203.stdout:2055: done (0 left) 2024-09-17T15:40:56.695 INFO:tasks.rados.rados.0.smithi203.stdout:2056: snap_remove snap 201 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c137510 auth_method 0 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:56.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:56.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:56.864+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c137510 auth_method 0 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:40:57.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:40:57.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:57.668+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:40:58.030 INFO:tasks.rados.rados.0.smithi203.stdout:2056: done (0 left) 2024-09-17T15:40:58.030 INFO:tasks.rados.rados.0.smithi203.stdout:2057: delete oid 27 current snap is 229 2024-09-17T15:40:58.050 INFO:tasks.rados.rados.0.smithi203.stdout:2057: done (0 left) 2024-09-17T15:40:58.050 INFO:tasks.rados.rados.0.smithi203.stdout:2058: write (excl) oid 42 current snap is 229 2024-09-17T15:40:58.050 INFO:tasks.rados.rados.0.smithi203.stdout:2058: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-17T15:40:58.073 INFO:tasks.rados.rados.0.smithi203.stdout:2058: writing smithi20328086-42 from 685917 to 1262824 tid 1 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:2058: writing smithi20328086-42 from 1893043 to 2285917 tid 2 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:2059: rollback oid 7 current snap is 229 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 228 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:2058: finishing write tid 1 to smithi20328086-42 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:2060: write (excl) oid 24 current snap is 229 2024-09-17T15:40:58.091 INFO:tasks.rados.rados.0.smithi203.stdout:2060: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-17T15:40:58.108 INFO:tasks.rados.rados.0.smithi203.stdout:2060: writing smithi20328086-24 from 702724 to 1354880 tid 1 2024-09-17T15:40:58.118 INFO:tasks.rados.rados.0.smithi203.stdout:2060: writing smithi20328086-24 from 1951525 to 2302724 tid 2 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2061: snap_create 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2059: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 525 (ObjNum 129 snap 0 seq_num 3605826032) dirty dne 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: finishing write tid 2 to smithi20328086-42 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: finishing write tid 3 to smithi20328086-42 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: finishing write tid 4 to smithi20328086-42 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: oid 42 updating version 0 to 883 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: oid 42 updating version 883 to 884 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: oid 42 version 884 is already newer than 882 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 884 (ObjNum 731 snap 229 seq_num 731) dirty exists 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2058: left oid 42 (ObjNum 731 snap 229 seq_num 731) 2024-09-17T15:40:58.119 INFO:tasks.rados.rados.0.smithi203.stdout:2060: finishing write tid 1 to smithi20328086-24 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: finishing write tid 2 to smithi20328086-24 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: finishing write tid 3 to smithi20328086-24 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: finishing write tid 4 to smithi20328086-24 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: oid 24 updating version 0 to 773 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: oid 24 updating version 773 to 774 2024-09-17T15:40:58.124 INFO:tasks.rados.rados.0.smithi203.stdout:2060: oid 24 version 774 is already newer than 772 2024-09-17T15:40:58.125 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 774 (ObjNum 732 snap 229 seq_num 732) dirty exists 2024-09-17T15:40:58.125 INFO:tasks.rados.rados.0.smithi203.stdout:2060: left oid 24 (ObjNum 732 snap 229 seq_num 732) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2058: done (3 left) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2059: done (2 left) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2060: done (1 left) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2061: done (0 left) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2062: copy_from oid 13 from oid 29 current snap is 230 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2063: read oid 18 snap 226 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2063: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2064: rollback oid 34 current snap is 230 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 218 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2065: read oid 28 snap -1 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2065: expect (ObjNum 720 snap 227 seq_num 720) 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2066: write oid 42 current snap is 230 2024-09-17T15:40:59.034 INFO:tasks.rados.rados.0.smithi203.stdout:2066: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-17T15:40:59.049 INFO:tasks.rados.rados.0.smithi203.stdout:2066: writing smithi20328086-42 from 719531 to 1446936 tid 1 2024-09-17T15:40:59.058 INFO:tasks.rados.rados.0.smithi203.stdout:2066: writing smithi20328086-42 from 1893654 to 2319531 tid 2 2024-09-17T15:40:59.058 INFO:tasks.rados.rados.0.smithi203.stdout:2067: write oid 44 current snap is 230 2024-09-17T15:40:59.058 INFO:tasks.rados.rados.0.smithi203.stdout:2067: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-17T15:40:59.069 INFO:tasks.rados.rados.0.smithi203.stdout:2067: writing smithi20328086-44 from 736338 to 1138992 tid 1 2024-09-17T15:40:59.078 INFO:tasks.rados.rados.0.smithi203.stdout:2067: writing smithi20328086-44 from 1552136 to 2091716 tid 2 2024-09-17T15:40:59.079 INFO:tasks.rados.rados.0.smithi203.stdout:2068: write (excl) oid 49 current snap is 230 2024-09-17T15:40:59.079 INFO:tasks.rados.rados.0.smithi203.stdout:2068: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-17T15:40:59.090 INFO:tasks.rados.rados.0.smithi203.stdout:2068: writing smithi20328086-49 from 753145 to 1231048 tid 1 2024-09-17T15:40:59.100 INFO:tasks.rados.rados.0.smithi203.stdout:2068: writing smithi20328086-49 from 2010618 to 2353145 tid 2 2024-09-17T15:40:59.100 INFO:tasks.rados.rados.0.smithi203.stdout:2069: delete oid 12 current snap is 230 2024-09-17T15:40:59.100 INFO:tasks.rados.rados.0.smithi203.stdout:2063: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:40:59.224 INFO:tasks.rados.rados.0.smithi203.stdout:2065: expect (ObjNum 720 snap 227 seq_num 720) 2024-09-17T15:40:59.224 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:59.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.252+0000 7fab197fa640 10 monclient: tick 2024-09-17T15:40:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.252+0000 7fab197fa640 10 monclient: _check_auth_tickets 2024-09-17T15:40:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.252+0000 7fab197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:59.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.252+0000 7fab197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:59.262 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:40:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c137510 auth_method 0 2024-09-17T15:40:59.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 10 monclient(hunting): handle_auth_done global_id 154099 payload 824 2024-09-17T15:40:59.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.272+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab1c122540 con 0x7fab1c137510 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab100ae000 tx=0x7fab10019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100c1440 con 0x7fab1c137510 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 10 monclient: dump: 2024-09-17T15:40:59.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fab100c1440 con 0x7fab1c137510 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab100ae000 tx=0x7fab10019650 comp rx=0 tx=0).stop 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 msgr2=0x7fab1c1d02b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:59.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.276+0000 7fab1a7fc640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c137510 0x7fab1c1d02b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab23ea9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:40:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 0x7fab1c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab23ea9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab1c1d0c00 con 0x7fab1c1d07f0 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 0x7fab1c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient: get_auth_request con 0x7fab1c1d07f0 auth_method 0 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 10 monclient(hunting): handle_auth_done global_id 154102 payload 824 2024-09-17T15:40:59.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.284+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab1c0a58f0 con 0x7fab1c1d07f0 2024-09-17T15:40:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab21444640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 msgr2=0x7fab1c1392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:59.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab21444640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 0x7fab1c1392d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab1001d150 tx=0x7fab100b8040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab100ca970 con 0x7fab1c1d07f0 2024-09-17T15:40:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 10 monclient: dump: 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.289 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fab100ca970 con 0x7fab1c1d07f0 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 msgr2=0x7fab1c1392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 0x7fab1c1392d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab1001d150 tx=0x7fab100b8040 comp rx=0 tx=0).stop 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 msgr2=0x7fab1c1392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:40:59.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab1a7fc640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab1c1d07f0 0x7fab1c1392d0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab1001d150 tx=0x7fab100b8040 comp rx=0 tx=0).stop 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10651, 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:40:59.291 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:59.292 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:59.293 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:40:59.294 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:40:59.295 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:40:59.296 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:40:59.298 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:40:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:40:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:40:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:40:59.303 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:40:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab0803bbb0 msgr2=0x7fab0803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab0803bbb0 0x7fab0803e070 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fab0c00d8d0 tx=0x7fab0c008040 comp rx=0 tx=0).stop 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 10 monclient: shutdown 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 msgr2=0x7fab1c13d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 --2- 172.21.15.86:0/2031477656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab1c0a50e0 0x7fab1c13d5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e980 tx=0x7fab1400ee40 comp rx=0 tx=0).stop 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab22446640 1 -- 172.21.15.86:0/2031477656 reap_dead start 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab22446640 1 -- 172.21.15.86:0/2031477656 reap_dead start 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 shutdown_connections 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 >> 172.21.15.86:0/2031477656 conn(0x7fab1c0aaae0 msgr2=0x7fab1c0a3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 shutdown_connections 2024-09-17T15:40:59.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.288+0000 7fab23ea9640 1 -- 172.21.15.86:0/2031477656 wait complete. 2024-09-17T15:40:59.313 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:40:59.313 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:40:59.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:40:59.313 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1516978 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2064: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2064: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2064: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 627 (ObjNum 684 snap 218 seq_num 684) dirty exists 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2066: finishing write tid 1 to smithi20328086-42 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2066: finishing write tid 2 to smithi20328086-42 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2066: finishing write tid 3 to smithi20328086-42 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2062: finishing copy_from racing read to smithi20328086-13 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2066: finishing write tid 4 to smithi20328086-42 2024-09-17T15:40:59.332 INFO:tasks.rados.rados.0.smithi203.stdout:2066: oid 42 updating version 0 to 886 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2066: oid 42 updating version 886 to 887 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2066: oid 42 version 887 is already newer than 885 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 887 (ObjNum 733 snap 230 seq_num 733) dirty exists 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2066: left oid 42 (ObjNum 733 snap 230 seq_num 733) 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2062: finishing copy_from to smithi20328086-13 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: finishing write tid 1 to smithi20328086-44 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: finishing write tid 2 to smithi20328086-44 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: finishing write tid 3 to smithi20328086-44 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: finishing write tid 4 to smithi20328086-44 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: oid 44 updating version 0 to 941 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: oid 44 version 941 is already newer than 940 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:2067: oid 44 version 941 is already newer than 939 2024-09-17T15:40:59.333 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 941 (ObjNum 734 snap 230 seq_num 734) dirty exists 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2067: left oid 44 (ObjNum 734 snap 230 seq_num 734) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2068: finishing write tid 1 to smithi20328086-49 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2068: finishing write tid 2 to smithi20328086-49 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2068: finishing write tid 3 to smithi20328086-49 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2062: done (7 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2063: done (6 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2064: done (5 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2065: done (4 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2066: done (3 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2067: done (2 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2069: done (1 left) 2024-09-17T15:40:59.334 INFO:tasks.rados.rados.0.smithi203.stdout:2070: snap_remove snap 218 2024-09-17T15:40:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 10 monclient: get_monmap_and_config 2024-09-17T15:40:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 10 monclient: build_initial_monmap 2024-09-17T15:40:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 10 monclient: monmap: 2024-09-17T15:40:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:59.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:40:59.438905+0000 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:40:59.438905+0000 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:40:59.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 1 Processor -- start 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 1 -- start start 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.432+0000 7f1601dc3640 10 monclient: init 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-17T15:40:59.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-b con 0x7f15fc0a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15fc0a5450 addr v1:172.21.15.203:6789/0 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f15fc13d280 0x7f15fc13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-c con 0x7f15fc13d280 addr v2:172.21.15.203:3300/0 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15fc0a6f40 con 0x7f15fc0a5450 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc0a65e0 con 0x7f15fc13d280 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc09c2a0 con 0x7f15fc0a5070 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f1601dc3640 10 monclient(hunting): authenticate will time out at 1792.404314s 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54822/0 (socket says 172.21.15.86:54822) 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 -- 172.21.15.86:0/457675427 learned_addr learned my addr 172.21.15.86:0/457675427 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): get_auth_request con 0x7f15fc0a5070 auth_method 0 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 1 --2- 172.21.15.86:0/457675427 >> v2:172.21.15.203:3300/0 conn(0x7f15fc13d280 0x7f15fc13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): get_auth_request con 0x7f15fc13d280 auth_method 0 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient(hunting): handle_auth_done global_id 104630 payload 274 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 -- 172.21.15.86:0/457675427 >> v2:172.21.15.203:3300/0 conn(0x7f15fc13d280 msgr2=0x7f15fc13f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 --2- 172.21.15.86:0/457675427 >> v2:172.21.15.203:3300/0 conn(0x7f15fc13d280 0x7f15fc13f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 -- 172.21.15.86:0/457675427 >> v1:172.21.15.203:6789/0 conn(0x7f15fc0a5450 legacy=0x7f15fc0a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 monclient: found mon.noname-b 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.436+0000 7f15fb7fe640 1 -- 172.21.15.86:0/457675427 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc09c580 con 0x7f15fc0a5070 2024-09-17T15:40:59.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fb7fe640 1 -- 172.21.15.86:0/457675427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 msgr2=0x7f15fc09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:59.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fb7fe640 1 --2- 172.21.15.86:0/457675427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009870 tx=0x7f15f401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=694b93781c50bcf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:59.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 1 -- 172.21.15.86:0/457675427 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f40b0070 con 0x7f15fc0a5070 2024-09-17T15:40:59.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: dump: 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: _finish_auth 0 2024-09-17T15:40:59.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 1 -- 172.21.15.86:0/457675427 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f40ab600 con 0x7f15fc0a5070 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 1 -- 172.21.15.86:0/457675427 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f40ab9d0 con 0x7f15fc0a5070 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f15fa7fc640 10 monclient: dump: 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 5 monclient: authenticate success, global_id 104630 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 10 monclient: get_monmap_and_config success 2024-09-17T15:40:59.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 4 set_mon_vals no callback set 2024-09-17T15:40:59.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 10 monclient: shutdown 2024-09-17T15:40:59.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:59.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 -- 172.21.15.86:0/457675427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 msgr2=0x7f15fc09cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 --2- 172.21.15.86:0/457675427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f15f4009870 tx=0x7f15f401c4b0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 -- 172.21.15.86:0/457675427 shutdown_connections 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 --2- 172.21.15.86:0/457675427 >> v2:172.21.15.203:3300/0 conn(0x7f15fc13d280 0x7f15fc13f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 --2- 172.21.15.86:0/457675427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5070 0x7f15fc09cf60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 -- 172.21.15.86:0/457675427 >> 172.21.15.86:0/457675427 conn(0x7f15fc0742a0 msgr2=0x7f15fc0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 -- 172.21.15.86:0/457675427 shutdown_connections 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.440+0000 7f1601dc3640 1 -- 172.21.15.86:0/457675427 wait complete. 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient: build_initial_monmap 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient: monmap: 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:40:59.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 Processor -- start 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 -- start start 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient: init 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient: _reopen_session rank -1 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc0a5070 0x7f15fc121180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-b con 0x7f15fc0a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:40:59.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-c con 0x7f15fc0a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 0x7f15fc0a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): picked mon.noname-a con 0x7f15fc13d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc142760 con 0x7f15fc13d280 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc1425e0 con 0x7f15fc0a5070 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc1428e0 con 0x7f15fc0a5450 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): _renew_subs 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f1601dc3640 10 monclient(hunting): authenticate will time out at 1792.413769s 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 0x7f15fc0a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54836/0 (socket says 172.21.15.86:54836) 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 0x7f15fc0a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60420/0 (socket says 172.21.15.86:60420) 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 -- 172.21.15.86:0/2880758458 learned_addr learned my addr 172.21.15.86:0/2880758458 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): get_auth_request con 0x7f15fc0a5450 auth_method 0 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): get_auth_request con 0x7f15fc13d280 auth_method 0 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fb7fe640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc0a5070 0x7f15fc121180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fb7fe640 10 monclient(hunting): get_auth_request con 0x7f15fc0a5070 auth_method 0 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient(hunting): handle_auth_done global_id 104633 payload 824 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 -- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc0a5070 msgr2=0x7f15fc121180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc0a5070 0x7f15fc121180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 -- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 msgr2=0x7f15fc0a2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 0x7f15fc0a2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient: _finish_hunting 0 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 monclient: found mon.noname-c 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.444+0000 7f15faffd640 1 -- 172.21.15.86:0/2880758458 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc1223c0 con 0x7f15fc0a5450 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15faffd640 1 -- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 msgr2=0x7f15fc137ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15faffd640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c450 tx=0x7f15e400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4012a40 con 0x7f15fc0a5450 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: dump: 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: _finish_auth 0 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e4017070 con 0x7f15fc0a5450 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 5 monclient: authenticate success, global_id 104633 2024-09-17T15:40:59.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _renew_subs 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc1226a0 con 0x7f15fc0a5450 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4012e40 con 0x7f15fc0a5450 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 10 monclient: dump: 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:40:59.456 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f97fa640 4 set_mon_vals no callback set 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _renew_subs 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc138530 con 0x7f15fc0a5450 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc09cf60 con 0x7f15fc0a5450 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f15e401c460 con 0x7f15fc0a5450 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 0x7f15dc03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(472..472 src has 213..472) ==== 4787+0+0 (secure 0 0 0) 0x7f15e404fcf0 con 0x7f15fc0a5450 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15fb7fe640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 0x7f15dc03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:40:59.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.448+0000 7f15fb7fe640 10 monclient: get_auth_request con 0x7f15dc03bbb0 auth_method 0 2024-09-17T15:40:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.452+0000 7f15fb7fe640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 0x7f15dc03e070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f15fc06e260 tx=0x7f15f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:40:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.452+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f15e401c930 con 0x7f15fc0a5450 2024-09-17T15:40:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.452+0000 7f15f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:40:59.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.452+0000 7f15f3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:40:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.688+0000 7f1601dc3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:40:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.688+0000 7f1601dc3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:40:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.688+0000 7f1601dc3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:40:59.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.688+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f15fc0a4520 con 0x7f15fc0a5450 2024-09-17T15:40:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f15f3fff640 1 -- 172.21.15.86:0/2880758458 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f15e4010cf0 con 0x7f15fc0a5450 2024-09-17T15:40:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f15f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:40:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f15f3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=9,kv=11,logm=54} crc {auth=2078537242,config=3541143443,health=4111534708,kv=2657738229,logm=2773089118}) 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=10,logm=54} crc {auth=3104190445,config=3541143444,health=3862454582,kv=215610202,logm=135870842}) 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=9,kv=11,logm=54} crc {auth=2078537242,config=3541143443,health=4111534708,kv=2657738229,logm=2773089118}) 2024-09-17T15:40:59.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=10,kv=11,logm=53} crc {auth=2078537242,config=3541143443,health=1473262794,kv=3591778543,logm=1019426696}) 2024-09-17T15:40:59.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 msgr2=0x7f15dc03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 0x7f15dc03e070 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f15fc06e260 tx=0x7f15f4008040 comp rx=0 tx=0).stop 2024-09-17T15:40:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 10 monclient: shutdown 2024-09-17T15:40:59.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021021548}) 2024-09-17T15:40:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:40:59.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 msgr2=0x7f15fc137ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:40:59.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c450 tx=0x7f15e400c910 comp rx=0 tx=0).stop 2024-09-17T15:40:59.697 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229546437}) 2024-09-17T15:40:59.697 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.697 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021021548}) 2024-09-17T15:40:59.697 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2037572121}) 2024-09-17T15:40:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 shutdown_connections 2024-09-17T15:40:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15dc03bbb0 0x7f15dc03e070 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15fc13d280 0x7f15fc0a2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15fc0a5450 0x7f15fc137ff0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 --2- 172.21.15.86:0/2880758458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15fc0a5070 0x7f15fc121180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:40:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 >> 172.21.15.86:0/2880758458 conn(0x7f15fc0742a0 msgr2=0x7f15fc0a02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:40:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 shutdown_connections 2024-09-17T15:40:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f1601dc3640 1 -- 172.21.15.86:0/2880758458 wait complete. 2024-09-17T15:40:59.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190848106}) 2024-09-17T15:40:59.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824866029}) 2024-09-17T15:40:59.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2190848106}) 2024-09-17T15:40:59.700 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1084015820}) 2024-09-17T15:40:59.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4799508}) 2024-09-17T15:40:59.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=330012346}) 2024-09-17T15:40:59.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4799508}) 2024-09-17T15:40:59.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915732885}) 2024-09-17T15:40:59.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085462758}) 2024-09-17T15:40:59.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808732637}) 2024-09-17T15:40:59.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085462758}) 2024-09-17T15:40:59.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684821534}) 2024-09-17T15:40:59.705 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291131821}) 2024-09-17T15:40:59.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777446758}) 2024-09-17T15:40:59.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291131821}) 2024-09-17T15:40:59.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136075019}) 2024-09-17T15:40:59.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760869526}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031500013}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3760869526}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967390226}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637945325}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181455345}) 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.709 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637945325}) 2024-09-17T15:40:59.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3528567333}) 2024-09-17T15:40:59.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396704325}) 2024-09-17T15:40:59.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552148046}) 2024-09-17T15:40:59.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396704325}) 2024-09-17T15:40:59.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6989824}) 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100040658}) 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190893531}) 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100040658}) 2024-09-17T15:40:59.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2843743245}) 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89277520}) 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561735266}) 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89277520}) 2024-09-17T15:40:59.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900603963}) 2024-09-17T15:40:59.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3958443591}) 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=324951752}) 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3958443591}) 2024-09-17T15:40:59.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994010085}) 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43921708}) 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184926615}) 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43921708}) 2024-09-17T15:40:59.718 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284421167}) 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666144776}) 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366103619}) 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666144776}) 2024-09-17T15:40:59.720 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276386641}) 2024-09-17T15:40:59.721 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.721 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321219321}) 2024-09-17T15:40:59.721 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570630757}) 2024-09-17T15:40:59.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321219321}) 2024-09-17T15:40:59.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265304682}) 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256526007}) 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830898757}) 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256526007}) 2024-09-17T15:40:59.724 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1088628599}) 2024-09-17T15:40:59.725 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.725 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333981725}) 2024-09-17T15:40:59.725 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2569948588}) 2024-09-17T15:40:59.725 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.725 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333981725}) 2024-09-17T15:40:59.726 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948727631}) 2024-09-17T15:40:59.727 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.727 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578357865}) 2024-09-17T15:40:59.727 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714293772}) 2024-09-17T15:40:59.727 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.728 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578357865}) 2024-09-17T15:40:59.728 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.724+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696027463}) 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305439120}) 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700742785}) 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1305439120}) 2024-09-17T15:40:59.729 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265965088}) 2024-09-17T15:40:59.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.730 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031694519}) 2024-09-17T15:40:59.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39293300}) 2024-09-17T15:40:59.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031694519}) 2024-09-17T15:40:59.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2148691220}) 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642910557}) 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2181858689}) 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642910557}) 2024-09-17T15:40:59.732 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=565400280}) 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535486740}) 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3882215852}) 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535486740}) 2024-09-17T15:40:59.734 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.732+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455072565}) 2024-09-17T15:40:59.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426953864}) 2024-09-17T15:40:59.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905317267}) 2024-09-17T15:40:59.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426953864}) 2024-09-17T15:40:59.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1526857919}) 2024-09-17T15:40:59.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846329050}) 2024-09-17T15:40:59.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921780902}) 2024-09-17T15:40:59.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846329050}) 2024-09-17T15:40:59.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.736+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=899742026}) 2024-09-17T15:40:59.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932585533}) 2024-09-17T15:40:59.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070996069}) 2024-09-17T15:40:59.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.741 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932585533}) 2024-09-17T15:40:59.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458214314}) 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579700002}) 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024144974}) 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579700002}) 2024-09-17T15:40:59.743 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.740+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373256810}) 2024-09-17T15:40:59.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316832682}) 2024-09-17T15:40:59.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627813769}) 2024-09-17T15:40:59.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316832682}) 2024-09-17T15:40:59.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890976818}) 2024-09-17T15:40:59.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674730358}) 2024-09-17T15:40:59.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3453543117}) 2024-09-17T15:40:59.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674730358}) 2024-09-17T15:40:59.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.744+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753817231}) 2024-09-17T15:40:59.748 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611925139}) 2024-09-17T15:40:59.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106391663}) 2024-09-17T15:40:59.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611925139}) 2024-09-17T15:40:59.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261782985}) 2024-09-17T15:40:59.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859376839}) 2024-09-17T15:40:59.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969868298}) 2024-09-17T15:40:59.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859376839}) 2024-09-17T15:40:59.751 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.748+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4175072949}) 2024-09-17T15:40:59.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868663875}) 2024-09-17T15:40:59.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1446262990}) 2024-09-17T15:40:59.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868663875}) 2024-09-17T15:40:59.753 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1482398990}) 2024-09-17T15:40:59.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828702664}) 2024-09-17T15:40:59.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2482683523}) 2024-09-17T15:40:59.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828702664}) 2024-09-17T15:40:59.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.752+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911442641}) 2024-09-17T15:40:59.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202062144}) 2024-09-17T15:40:59.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3721934954}) 2024-09-17T15:40:59.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=202062144}) 2024-09-17T15:40:59.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643671724}) 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451069484}) 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604561691}) 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1451069484}) 2024-09-17T15:40:59.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.756+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164454506}) 2024-09-17T15:40:59.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858987540}) 2024-09-17T15:40:59.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840857463}) 2024-09-17T15:40:59.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2858987540}) 2024-09-17T15:40:59.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189858949}) 2024-09-17T15:40:59.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34468621}) 2024-09-17T15:40:59.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101570659}) 2024-09-17T15:40:59.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34468621}) 2024-09-17T15:40:59.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.760+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626711266}) 2024-09-17T15:40:59.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764346911}) 2024-09-17T15:40:59.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=202086230}) 2024-09-17T15:40:59.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.765 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764346911}) 2024-09-17T15:40:59.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611779423}) 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706215227}) 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457599423}) 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3706215227}) 2024-09-17T15:40:59.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.764+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9402062}) 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114757247}) 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606127102}) 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114757247}) 2024-09-17T15:40:59.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684309831}) 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235894250}) 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585376146}) 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1235894250}) 2024-09-17T15:40:59.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.768+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291254098}) 2024-09-17T15:40:59.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920805051}) 2024-09-17T15:40:59.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2994529965}) 2024-09-17T15:40:59.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920805051}) 2024-09-17T15:40:59.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294065191}) 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=7} crc {logm=319235179,mdsmap=3164546943,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4248096279,mon_config_key=3890275977}) 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10} crc {logm=514741538,mdsmap=2815901553,mgr=1375715255,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4275842610,mon_config_key=3416188778}) 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=7} crc {logm=319235179,mdsmap=3164546943,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4248096279,mon_config_key=3890275977}) 2024-09-17T15:40:59.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.772+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9} crc {logm=3804136006,mdsmap=2815901553,mgr=2224701651,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4275842610,mon_config_key=2160652126}) 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=3033931621,monmap=2119989493,nvmeofgw=598337686,osd_metadata=3311363537,osd_pg_creating=149932639,osd_snap=2654517443}) 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=7,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=81} crc {mon_config_key=3484601819,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=149932639,osd_snap=3423456774}) 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=3033931621,monmap=2119989493,nvmeofgw=598337686,osd_metadata=3311363537,osd_pg_creating=149932639,osd_snap=2654517443}) 2024-09-17T15:40:59.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=7,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=80} crc {mon_config_key=3484601819,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=149932639,osd_snap=2855721879}) 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3628811849,osdmap=2738784528}) 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=965579819,osdmap=3116380356}) 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3628811849,osdmap=2738784528}) 2024-09-17T15:40:59.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.776+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=965579819,osdmap=2946305217}) 2024-09-17T15:40:59.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1712768011}) 2024-09-17T15:40:59.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3150206463}) 2024-09-17T15:40:59.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1712768011}) 2024-09-17T15:40:59.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.780+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2792215974}) 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2907938031}) 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=710628616}) 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2907938031}) 2024-09-17T15:40:59.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.784+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2003554454}) 2024-09-17T15:40:59.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3674756490}) 2024-09-17T15:40:59.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=98} crc {osdmap=3384704584}) 2024-09-17T15:40:59.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:40:59.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3674756490}) 2024-09-17T15:40:59.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:40:59.796+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=97} crc {osdmap=1170539922}) 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2070: done (1 left) 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2071: read oid 48 snap -1 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2071: expect deleted 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2072: read oid 34 snap -1 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2072: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:41:00.085 INFO:tasks.rados.rados.0.smithi203.stdout:2073: copy_from oid 26 from oid 46 current snap is 230 2024-09-17T15:41:00.086 INFO:tasks.rados.rados.0.smithi203.stdout:2074: write (excl) oid 17 current snap is 230 2024-09-17T15:41:00.086 INFO:tasks.rados.rados.0.smithi203.stdout:2074: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-17T15:41:00.093 INFO:tasks.rados.rados.0.smithi203.stdout:2074: writing smithi20328086-17 from 786759 to 1415160 tid 1 2024-09-17T15:41:00.099 INFO:tasks.rados.rados.0.smithi203.stdout:2074: writing smithi20328086-17 from 2011229 to 2386759 tid 2 2024-09-17T15:41:00.099 INFO:tasks.rados.rados.0.smithi203.stdout:2075: snap_remove snap 215 2024-09-17T15:41:00.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:41:00.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: get_monmap_and_config 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: build_initial_monmap 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: monmap: 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:00.849465+0000 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:00.849465+0000 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:00.846 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 Processor -- start 2024-09-17T15:41:00.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 -- start start 2024-09-17T15:41:00.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: init 2024-09-17T15:41:00.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c200689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c200694c0 0x7f7c2006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c200694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7c2013aff0 0x7f7c2013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c2013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c2009c8b0 con 0x7f7c200689d0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7c20074500 con 0x7f7c200694c0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7c2009cb90 con 0x7f7c2013aff0 2024-09-17T15:41:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c2821c640 10 monclient(hunting): authenticate will time out at 1793.815048s 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c200694c0 0x7f7c2006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c25fb8640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7c200689d0 0x7f7c20068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56806/0 (socket says 172.21.15.86:56806) 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c200694c0 0x7f7c2006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60424/0 (socket says 172.21.15.86:60424) 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c25fb8640 1 -- 172.21.15.86:0/2170512504 learned_addr learned my addr 172.21.15.86:0/2170512504 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): get_auth_request con 0x7f7c200694c0 auth_method 0 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c267b9640 1 --2- 172.21.15.86:0/2170512504 >> v2:172.21.15.203:3300/0 conn(0x7f7c2013aff0 0x7f7c2013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c267b9640 10 monclient(hunting): get_auth_request con 0x7f7c2013aff0 auth_method 0 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c267b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c267b9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c267b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient(hunting): handle_auth_done global_id 144252 payload 274 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 -- 172.21.15.86:0/2170512504 >> v2:172.21.15.203:3300/0 conn(0x7f7c2013aff0 msgr2=0x7f7c2013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 --2- 172.21.15.86:0/2170512504 >> v2:172.21.15.203:3300/0 conn(0x7f7c2013aff0 0x7f7c2013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 -- 172.21.15.86:0/2170512504 >> v1:172.21.15.86:6789/0 conn(0x7f7c200689d0 legacy=0x7f7c20068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient: _finish_hunting 0 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 monclient: found mon.noname-a 2024-09-17T15:41:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.844+0000 7f7c257b7640 1 -- 172.21.15.86:0/2170512504 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2009ce70 con 0x7f7c200694c0 2024-09-17T15:41:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 1 -- 172.21.15.86:0/2170512504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200694c0 msgr2=0x7f7c2006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 1 --2- 172.21.15.86:0/2170512504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200694c0 0x7f7c2006f840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c2006fdb0 tx=0x7f7c1801c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a52485c570373c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 1 -- 172.21.15.86:0/2170512504 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c180b2070 con 0x7f7c200694c0 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: dump: 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:00.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: _finish_auth 0 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: _check_auth_tickets 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 1 -- 172.21.15.86:0/2170512504 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c180a5870 con 0x7f7c200694c0 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 1 -- 172.21.15.86:0/2170512504 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2009c2d0 con 0x7f7c200694c0 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c24fb6640 10 monclient: dump: 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 5 monclient: authenticate success, global_id 144252 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 4 set_mon_vals no callback set 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: shutdown 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- 172.21.15.86:0/2170512504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200694c0 msgr2=0x7f7c2006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- 172.21.15.86:0/2170512504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200694c0 0x7f7c2006f840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c2006fdb0 tx=0x7f7c1801c7d0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- 172.21.15.86:0/2170512504 shutdown_connections 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- 172.21.15.86:0/2170512504 >> v2:172.21.15.203:3300/0 conn(0x7f7c2013aff0 0x7f7c2013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- 172.21.15.86:0/2170512504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200694c0 0x7f7c2006f840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- 172.21.15.86:0/2170512504 >> 172.21.15.86:0/2170512504 conn(0x7f7c200a6440 msgr2=0x7f7c200a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- 172.21.15.86:0/2170512504 shutdown_connections 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- 172.21.15.86:0/2170512504 wait complete. 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: build_initial_monmap 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: monmap: 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 Processor -- start 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- start start 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: init 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-a con 0x7f7c200689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c200694c0 0x7f7c2011ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-c con 0x7f7c200694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 0x7f7c20135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): picked mon.noname-b con 0x7f7c2013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7c201405c0 con 0x7f7c200689d0 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c20140440 con 0x7f7c2013aff0 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7c20140740 con 0x7f7c200694c0 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c2821c640 10 monclient(hunting): authenticate will time out at 1793.822943s 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c200694c0 0x7f7c2011ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60440/0 (socket says 172.21.15.86:60440) 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 -- 172.21.15.86:0/815263003 learned_addr learned my addr 172.21.15.86:0/815263003 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): get_auth_request con 0x7f7c200689d0 auth_method 0 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): get_auth_request con 0x7f7c200694c0 auth_method 0 2024-09-17T15:41:00.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c257b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c267b9640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 0x7f7c20135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient(hunting): handle_auth_done global_id 144255 payload 824 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 -- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c200694c0 msgr2=0x7f7c2011ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:00.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c200694c0 0x7f7c2011ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 -- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 msgr2=0x7f7c20135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 0x7f7c20135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient: _finish_hunting 0 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 monclient: found mon.noname-a 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c25fb8640 1 -- 172.21.15.86:0/815263003 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2013b3d0 con 0x7f7c200689d0 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c267b9640 10 monclient: get_auth_request con 0x7f7c2013aff0 auth_method 0 2024-09-17T15:41:00.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.852+0000 7f7c267b9640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 0x7f7c20135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c25fb8640 1 -- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 msgr2=0x7f7c2011cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c25fb8640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c470 tx=0x7f7c1000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c10017070 con 0x7f7c200689d0 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: dump: 2024-09-17T15:41:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: _finish_auth 0 2024-09-17T15:41:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: _check_auth_tickets 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c100070d0 con 0x7f7c200689d0 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c100074c0 con 0x7f7c200689d0 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 10 monclient: dump: 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c177fe640 4 set_mon_vals no callback set 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 5 monclient: authenticate success, global_id 144255 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _renew_subs 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2009d460 con 0x7f7c200689d0 2024-09-17T15:41:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _renew_subs 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c20136820 con 0x7f7c200689d0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c2006f840 con 0x7f7c200689d0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7c1001a070 con 0x7f7c200689d0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 0x7f7c0403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(473..473 src has 213..473) ==== 4831+0+0 (secure 0 0 0) 0x7f7c1004f5c0 con 0x7f7c200689d0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.856+0000 7f7c257b7640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 0x7f7c0403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.860+0000 7f7c257b7640 10 monclient: get_auth_request con 0x7f7c0403bb60 auth_method 0 2024-09-17T15:41:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.860+0000 7f7c257b7640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 0x7f7c0403e020 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7c20069a70 tx=0x7f7c180a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.864+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7c1001bc50 con 0x7f7c200689d0 2024-09-17T15:41:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.864+0000 7f7c167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:00.864+0000 7f7c167fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:01.060 INFO:tasks.rados.rados.0.smithi203.stdout:2068: finishing write tid 4 to smithi20328086-49 2024-09-17T15:41:01.060 INFO:tasks.rados.rados.0.smithi203.stdout:2068: oid 49 updating version 0 to 1093 2024-09-17T15:41:01.060 INFO:tasks.rados.rados.0.smithi203.stdout:2068: oid 49 version 1093 is already newer than 1092 2024-09-17T15:41:01.060 INFO:tasks.rados.rados.0.smithi203.stdout:2068: oid 49 updating version 1093 to 1094 2024-09-17T15:41:01.060 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1094 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:2068: left oid 49 (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:2073: finishing copy_from to smithi20328086-26 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:2073: got expected ENOENT (src dne) 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:2073: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:2072: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:41:01.061 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 268385 2024-09-17T15:41:01.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c2821c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:01.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c2821c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:01.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c2821c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7c2006d4f0 con 0x7f7c200689d0 2024-09-17T15:41:01.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c167fc640 1 -- 172.21.15.86:0/815263003 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f7c10010300 con 0x7f7c200689d0 2024-09-17T15:41:01.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:01.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.108+0000 7f7c167fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:01.112 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:01.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 msgr2=0x7f7c0403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 0x7f7c0403e020 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7c20069a70 tx=0x7f7c180a7070 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 10 monclient: shutdown 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 msgr2=0x7f7c2011cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000c470 tx=0x7f7c1000c930 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 shutdown_connections 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c0403bb60 0x7f7c0403e020 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c2013aff0 0x7f7c20135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c200694c0 0x7f7c2011ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 --2- 172.21.15.86:0/815263003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c200689d0 0x7f7c2011cda0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 >> 172.21.15.86:0/815263003 conn(0x7f7c200a6440 msgr2=0x7f7c200a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 shutdown_connections 2024-09-17T15:41:01.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.112+0000 7f7c2821c640 1 -- 172.21.15.86:0/815263003 wait complete. 2024-09-17T15:41:01.130 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:41:01.130 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:41:01.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:41:01.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2074: finishing write tid 1 to smithi20328086-17 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2074: finishing write tid 2 to smithi20328086-17 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2074: finishing write tid 3 to smithi20328086-17 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2068: done (5 left) 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2071: done (4 left) 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2072: done (3 left) 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2073: done (2 left) 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2075: done (1 left) 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2076: rollback oid 27 current snap is 230 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 228 2024-09-17T15:41:01.214 INFO:tasks.rados.rados.0.smithi203.stdout:2074: finishing write tid 4 to smithi20328086-17 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2074: oid 17 updating version 0 to 944 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2074: oid 17 version 944 is already newer than 943 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2074: oid 17 version 944 is already newer than 942 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 944 (ObjNum 737 snap 230 seq_num 737) dirty exists 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2074: left oid 17 (ObjNum 737 snap 230 seq_num 737) 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2074: done (1 left) 2024-09-17T15:41:01.215 INFO:tasks.rados.rados.0.smithi203.stdout:2077: snap_create 2024-09-17T15:41:01.220 INFO:tasks.rados.rados.0.smithi203.stdout:2076: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:41:01.221 INFO:tasks.rados.rados.0.smithi203.stdout:2076: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:41:01.221 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 889 (ObjNum 703 snap 221 seq_num 703) dirty exists 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: get_monmap_and_config 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: monmap: 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:01.258370+0000 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:01.258370+0000 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.254 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:01.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:01.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:01.255 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:01.255 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:01.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 Processor -- start 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 -- start start 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: init 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-b con 0x7fbc680a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc680a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc68136160 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc680a2130 con 0x7fbc68136160 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc680a26f0 con 0x7fbc680a1590 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbc680a2410 con 0x7fbc680a0c70 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.252+0000 7fbc6cb43640 10 monclient(hunting): authenticate will time out at 1794.223889s 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54846/0 (socket says 172.21.15.86:54846) 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 -- 172.21.15.86:0/1697411913 learned_addr learned my addr 172.21.15.86:0/1697411913 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): get_auth_request con 0x7fbc680a0c70 auth_method 0 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient(hunting): handle_auth_done global_id 104639 payload 274 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 -- 172.21.15.86:0/1697411913 >> v1:172.21.15.203:6789/0 conn(0x7fbc680a1590 legacy=0x7fbc68135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 -- 172.21.15.86:0/1697411913 >> v1:172.21.15.86:6789/0 conn(0x7fbc68136160 legacy=0x7fbc6813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 monclient: found mon.noname-b 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:01.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.256+0000 7fbc6659c640 1 -- 172.21.15.86:0/1697411913 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6809f9f0 con 0x7fbc680a0c70 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 1 -- 172.21.15.86:0/1697411913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 msgr2=0x7fbc680a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1697411913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbc600098a0 tx=0x7fbc6001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba6cbc2f87e6cb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 1 -- 172.21.15.86:0/1697411913 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc600b2070 con 0x7fbc680a0c70 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: dump: 2024-09-17T15:41:01.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 1 -- 172.21.15.86:0/1697411913 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc600ad680 con 0x7fbc680a0c70 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 5 monclient: authenticate success, global_id 104639 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 1 -- 172.21.15.86:0/1697411913 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc600ada50 con 0x7fbc680a0c70 2024-09-17T15:41:01.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 4 set_mon_vals no callback set 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6559a640 10 monclient: dump: 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.264 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: shutdown 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1697411913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 msgr2=0x7fbc680a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1697411913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbc600098a0 tx=0x7fbc6001c440 comp rx=0 tx=0).stop 2024-09-17T15:41:01.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1697411913 shutdown_connections 2024-09-17T15:41:01.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1697411913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc680a0c70 0x7fbc680a1050 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1697411913 >> 172.21.15.86:0/1697411913 conn(0x7fbc680a2f40 msgr2=0x7fbc680a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1697411913 shutdown_connections 2024-09-17T15:41:01.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1697411913 wait complete. 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: monmap: 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:01.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 Processor -- start 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- start start 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: init 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc680a1590 0x7fbc68137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-b con 0x7fbc680a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 0x7fbc68138440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-a con 0x7fbc68136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc68138980 0x7fbc681ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): picked mon.noname-c con 0x7fbc68138980 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbc680a00d0 con 0x7fbc68136160 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6809ff50 con 0x7fbc680a1590 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbc68141e70 con 0x7fbc68138980 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6cb43640 10 monclient(hunting): authenticate will time out at 1794.231068s 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 0x7fbc68138440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc68138980 0x7fbc681ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 0x7fbc68138440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60454/0 (socket says 172.21.15.86:60454) 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 -- 172.21.15.86:0/1807275422 learned_addr learned my addr 172.21.15.86:0/1807275422 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): get_auth_request con 0x7fbc68136160 auth_method 0 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): get_auth_request con 0x7fbc68138980 auth_method 0 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc680a1590 0x7fbc68137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 10 monclient(hunting): get_auth_request con 0x7fbc680a1590 auth_method 0 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient(hunting): handle_auth_done global_id 144258 payload 824 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc68138980 msgr2=0x7fbc681ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc68138980 0x7fbc681ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc680a1590 msgr2=0x7fbc68137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbc680a1590 0x7fbc68137f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 monclient: found mon.noname-a 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc65d9b640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc6813b760 con 0x7fbc68136160 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.260+0000 7fbc66d9d640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc68138980 0x7fbc681ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc65d9b640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 msgr2=0x7fbc68138440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc65d9b640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 0x7fbc68138440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c7f0 tx=0x7fbc5000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50016020 con 0x7fbc68136160 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: dump: 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc5000ac60 con 0x7fbc68136160 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 5 monclient: authenticate success, global_id 144258 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 monclient: _renew_subs 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc6813a520 con 0x7fbc68136160 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc50012660 con 0x7fbc68136160 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 10 monclient: dump: 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.280 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5ffff640 4 set_mon_vals no callback set 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 monclient: _renew_subs 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc681ce810 con 0x7fbc68136160 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 0x7fbc68131d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6813ec60 con 0x7fbc681cead0 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 0x7fbc68131d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient: get_auth_request con 0x7fbc681cead0 auth_method 0 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbc50019020 con 0x7fbc68136160 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 10 monclient(hunting): handle_auth_done global_id 144261 payload 824 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc6659c640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc680a1050 con 0x7fbc681cead0 2024-09-17T15:41:01.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc4803bbb0 0x7fbc4803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(474..474 src has 213..474) ==== 4847+0+0 (secure 0 0 0) 0x7fbc5004ff80 con 0x7fbc68136160 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc66d9d640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc4803bbb0 0x7fbc4803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.264+0000 7fbc66d9d640 10 monclient: get_auth_request con 0x7fbc4803bbb0 auth_method 0 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc6659c640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 msgr2=0x7fbc68131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 0x7fbc68131d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbc680a40b0 tx=0x7fbc600191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc600bd070 con 0x7fbc681cead0 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 10 monclient: dump: 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc600ad970 con 0x7fbc681cead0 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 msgr2=0x7fbc68131d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 0x7fbc68131d90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbc680a40b0 tx=0x7fbc600191e0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 msgr2=0x7fbc68131d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc5effd640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681cead0 0x7fbc68131d90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.268+0000 7fbc66d9d640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc4803bbb0 0x7fbc4803e070 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc580042b0 tx=0x7fbc58007d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.272+0000 7fbc6cb43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:01.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.272+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 0x7fbc681326e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.272+0000 7fbc6cb43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.272+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbc6806fa90 con 0x7fbc681322d0 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 0x7fbc681326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient: get_auth_request con 0x7fbc681322d0 auth_method 0 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 10 monclient(hunting): handle_auth_done global_id 144264 payload 824 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 1 -- 172.21.15.86:0/1807275422 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc681399c0 con 0x7fbc681322d0 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 msgr2=0x7fbc681326e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6659c640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 0x7fbc681326e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbc6001c480 tx=0x7fbc60008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc600b2040 con 0x7fbc681322d0 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 10 monclient: dump: 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.288 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fbc60003b00 con 0x7fbc681322d0 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 msgr2=0x7fbc681326e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 0x7fbc681326e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbc6001c480 tx=0x7fbc60008040 comp rx=0 tx=0).stop 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 msgr2=0x7fbc681326e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc5effd640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc681322d0 0x7fbc681326e0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbc6001c480 tx=0x7fbc60008040 comp rx=0 tx=0).stop 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:01.289 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 79618, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:01.290 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:01.291 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:01.293 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.294 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.296 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.299 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.301 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:01.302 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:01.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc4803bbb0 msgr2=0x7fbc4803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbc4803bbb0 0x7fbc4803e070 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbc580042b0 tx=0x7fbc58007d10 comp rx=0 tx=0).stop 2024-09-17T15:41:01.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 10 monclient: shutdown 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 msgr2=0x7fbc68138440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 --2- 172.21.15.86:0/1807275422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc68136160 0x7fbc68138440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000c7f0 tx=0x7fbc5000ccb0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc66d9d640 1 -- 172.21.15.86:0/1807275422 reap_dead start 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc66d9d640 1 -- 172.21.15.86:0/1807275422 reap_dead start 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 shutdown_connections 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 >> 172.21.15.86:0/1807275422 conn(0x7fbc680a2f40 msgr2=0x7fbc680a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 shutdown_connections 2024-09-17T15:41:01.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.276+0000 7fbc6cb43640 1 -- 172.21.15.86:0/1807275422 wait complete. 2024-09-17T15:41:01.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:01.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:41:01.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:41:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 10 monclient: get_monmap_and_config 2024-09-17T15:41:01.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 10 monclient: monmap: 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:01.422928+0000 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:01.422928+0000 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:01.419 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:01.420 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 1 Processor -- start 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 1 -- start start 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.416+0000 7f7178bf9640 10 monclient: init 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71740a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-b con 0x7f71740a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 0x7f717413d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-a con 0x7f717413b450 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71740a2130 con 0x7f71740a0c70 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f71740a2410 con 0x7f717413b450 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f71740a26f0 con 0x7f71740a1760 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7178bf9640 10 monclient(hunting): authenticate will time out at 1794.388394s 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 0x7f717413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54868/0 (socket says 172.21.15.86:54868) 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 -- 172.21.15.86:0/3723394684 learned_addr learned my addr 172.21.15.86:0/3723394684 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): get_auth_request con 0x7f71740a1760 auth_method 0 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): get_auth_request con 0x7f717413b450 auth_method 0 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient(hunting): handle_auth_done global_id 104645 payload 274 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 -- 172.21.15.86:0/3723394684 >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 msgr2=0x7f717413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 --2- 172.21.15.86:0/3723394684 >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 0x7f717413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 -- 172.21.15.86:0/3723394684 >> v1:172.21.15.203:6789/0 conn(0x7f71740a0c70 legacy=0x7f71740a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 monclient: found mon.noname-b 2024-09-17T15:41:01.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:01.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7171d9b640 1 -- 172.21.15.86:0/3723394684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717409f9f0 con 0x7f71740a1760 2024-09-17T15:41:01.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.420+0000 7f7172d9d640 1 --2- 172.21.15.86:0/3723394684 >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 0x7f717413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7171d9b640 1 -- 172.21.15.86:0/3723394684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 msgr2=0x7f717413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7171d9b640 1 --2- 172.21.15.86:0/3723394684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7164009870 tx=0x7f716401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46ede66c3de7994 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 1 -- 172.21.15.86:0/3723394684 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640b1070 con 0x7f71740a1760 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: dump: 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.433 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 5 monclient: authenticate success, global_id 104645 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 1 -- 172.21.15.86:0/3723394684 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71640ac620 con 0x7f71740a1760 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 1 -- 172.21.15.86:0/3723394684 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640ac9f0 con 0x7f71740a1760 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717159a640 10 monclient: dump: 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 4 set_mon_vals no callback set 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: shutdown 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- 172.21.15.86:0/3723394684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 msgr2=0x7f717413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3723394684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7164009870 tx=0x7f716401c420 comp rx=0 tx=0).stop 2024-09-17T15:41:01.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- 172.21.15.86:0/3723394684 shutdown_connections 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3723394684 >> v2:172.21.15.86:3300/0 conn(0x7f717413b450 0x7f717413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3723394684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a1760 0x7f717413af10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- 172.21.15.86:0/3723394684 >> 172.21.15.86:0/3723394684 conn(0x7f71740a2f40 msgr2=0x7f71740a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- 172.21.15.86:0/3723394684 shutdown_connections 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- 172.21.15.86:0/3723394684 wait complete. 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: monmap: 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.437 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 Processor -- start 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- start start 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: init 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a0c70 0x7f71741cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-c con 0x7f71740a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71740a1760 0x7f71741d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-b con 0x7f71740a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:01.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 0x7f71741d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): picked mon.noname-a con 0x7f717413b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7174140720 con 0x7f717413b450 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f71741405a0 con 0x7f71740a1760 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f71741408a0 con 0x7f71740a0c70 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7178bf9640 10 monclient(hunting): authenticate will time out at 1794.395724s 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 0x7f71741d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a0c70 0x7f71741cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a0c70 0x7f71741cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54874/0 (socket says 172.21.15.86:54874) 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 0x7f71741d07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60474/0 (socket says 172.21.15.86:60474) 2024-09-17T15:41:01.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 1 -- 172.21.15.86:0/3379831486 learned_addr learned my addr 172.21.15.86:0/3379831486 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): get_auth_request con 0x7f71740a0c70 auth_method 0 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): get_auth_request con 0x7f717413b450 auth_method 0 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f7172d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.424+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 10 monclient(hunting): handle_auth_done global_id 144270 payload 824 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a0c70 msgr2=0x7f71741cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71740a0c70 0x7f71741cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71740a1760 msgr2=0x7f71741d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7171d9b640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71740a1760 0x7f71741d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71740a1760 0x7f71741d0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 monclient: found mon.noname-a 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7174135b70 con 0x7f717413b450 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 msgr2=0x7f71741d07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7172d9d640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 0x7f71741d07b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f716c009d90 tx=0x7f716c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c018030 con 0x7f717413b450 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: dump: 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716c0039a0 con 0x7f717413b450 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c003d70 con 0x7f717413b450 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 5 monclient: authenticate success, global_id 144270 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 monclient: _renew_subs 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717413ba80 con 0x7f717413b450 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 10 monclient: dump: 2024-09-17T15:41:01.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 monclient: _renew_subs 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7174135890 con 0x7f717413b450 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7170d99640 4 set_mon_vals no callback set 2024-09-17T15:41:01.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 0x7f7158004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7158007cc0 con 0x7f7158002890 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 0x7f7158004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient: get_auth_request con 0x7f7158002890 auth_method 0 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f716c01b070 con 0x7f717413b450 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 10 monclient(hunting): handle_auth_done global_id 104651 payload 824 2024-09-17T15:41:01.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.428+0000 7f717259c640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7158007ec0 con 0x7f7158002890 2024-09-17T15:41:01.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f715403bbb0 0x7f715403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(474..474 src has 213..474) ==== 4847+0+0 (secure 0 0 0) 0x7f716c050880 con 0x7f717413b450 2024-09-17T15:41:01.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7171d9b640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f715403bbb0 0x7f715403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7171d9b640 10 monclient: get_auth_request con 0x7f715403bbb0 auth_method 0 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f717259c640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 msgr2=0x7f7158004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f717259c640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 0x7f7158004d30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f716800f7a0 tx=0x7f716800fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168018030 con 0x7f7158002890 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 10 monclient: dump: 2024-09-17T15:41:01.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f716800dbc0 con 0x7f7158002890 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 msgr2=0x7f7158004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 0x7f7158004d30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f716800f7a0 tx=0x7f716800fc60 comp rx=0 tx=0).stop 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 msgr2=0x7f7158004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7162ffd640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158002890 0x7f7158004d30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.432+0000 7f7171d9b640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f715403bbb0 0x7f715403e070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f717406d160 tx=0x7f7164008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7178bf9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 0x7f7158007170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7178bf9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f715800a200 con 0x7f7158006ce0 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 0x7f7158007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient: get_auth_request con 0x7f7158006ce0 auth_method 0 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 10 monclient(hunting): handle_auth_done global_id 104654 payload 824 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 1 -- 172.21.15.86:0/3379831486 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71580076b0 con 0x7f7158006ce0 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 msgr2=0x7f7158007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f717259c640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 0x7f7158007170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7168018000 tx=0x7f7168007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168004000 con 0x7f7158006ce0 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 10 monclient: dump: 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f7168004000 con 0x7f7158006ce0 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 msgr2=0x7f7158007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 0x7f7158007170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7168018000 tx=0x7f7168007f90 comp rx=0 tx=0).stop 2024-09-17T15:41:01.452 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 292315, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:01.453 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.454 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.455 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:01.456 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.457 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.458 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.459 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.462 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:01.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:01.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:01.465 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 msgr2=0x7f7158007170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.440+0000 7f7162ffd640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7158006ce0 0x7f7158007170 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7168018000 tx=0x7f7168007f90 comp rx=0 tx=0).stop 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f715403bbb0 msgr2=0x7f715403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f715403bbb0 0x7f715403e070 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f717406d160 tx=0x7f7164008040 comp rx=0 tx=0).stop 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 10 monclient: shutdown 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 msgr2=0x7f71741d07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 --2- 172.21.15.86:0/3379831486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f717413b450 0x7f71741d07b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f716c009d90 tx=0x7f716c00c490 comp rx=0 tx=0).stop 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 reap_dead start 2024-09-17T15:41:01.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7172d9d640 1 -- 172.21.15.86:0/3379831486 reap_dead start 2024-09-17T15:41:01.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 shutdown_connections 2024-09-17T15:41:01.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 >> 172.21.15.86:0/3379831486 conn(0x7f71740a2f40 msgr2=0x7f71740a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 shutdown_connections 2024-09-17T15:41:01.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.444+0000 7f7178bf9640 1 -- 172.21.15.86:0/3379831486 wait complete. 2024-09-17T15:41:01.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:01.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:41:01.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: get_monmap_and_config 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: monmap: 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:01.588250+0000 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:01.588250+0000 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:01.584 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:01.585 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:01.585 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:01.585 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:01.585 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 Processor -- start 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 -- start start 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: init 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71040a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71040a1760 0x7f710413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): picked mon.noname-c con 0x7f71040a1760 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f710413b450 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71040a2130 con 0x7f71040a0c70 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71040a26f0 con 0x7f710413b450 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f71040a2410 con 0x7f71040a1760 2024-09-17T15:41:01.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f710b489640 10 monclient(hunting): authenticate will time out at 1794.553705s 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7109225640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f71040a0c70 0x7f71040a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56826/0 (socket says 172.21.15.86:56826) 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7109225640 1 -- 172.21.15.86:0/2735826974 learned_addr learned my addr 172.21.15.86:0/2735826974 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 1 --2- 172.21.15.86:0/2735826974 >> v2:172.21.15.203:3300/0 conn(0x7f71040a1760 0x7f710413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): get_auth_request con 0x7f71040a1760 auth_method 0 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient(hunting): handle_auth_done global_id 154117 payload 274 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 1 -- 172.21.15.86:0/2735826974 >> v1:172.21.15.203:6789/0 conn(0x7f710413b450 legacy=0x7f710413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 1 -- 172.21.15.86:0/2735826974 >> v1:172.21.15.86:6789/0 conn(0x7f71040a0c70 legacy=0x7f71040a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 1 monclient: found mon.noname-c 2024-09-17T15:41:01.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:41:01.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.584+0000 7f7108a24640 1 -- 172.21.15.86:0/2735826974 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710409f9f0 con 0x7f71040a1760 2024-09-17T15:41:01.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.588+0000 7f7108a24640 1 -- 172.21.15.86:0/2735826974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a1760 msgr2=0x7f710413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.588+0000 7f7108a24640 1 --2- 172.21.15.86:0/2735826974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a1760 0x7f710413af10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7100009d90 tx=0x7f710001c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc8d1a7ff574b6b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 1 -- 172.21.15.86:0/2735826974 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000b2070 con 0x7f71040a1760 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: dump: 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 5 monclient: authenticate success, global_id 154117 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 1 -- 172.21.15.86:0/2735826974 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71000a5970 con 0x7f71040a1760 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 4 set_mon_vals no callback set 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 1 -- 172.21.15.86:0/2735826974 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71000a5d40 con 0x7f71040a1760 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f70f7fff640 10 monclient: dump: 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.594 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: shutdown 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- 172.21.15.86:0/2735826974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a1760 msgr2=0x7f710413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 --2- 172.21.15.86:0/2735826974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a1760 0x7f710413af10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7100009d90 tx=0x7f710001c830 comp rx=0 tx=0).stop 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- 172.21.15.86:0/2735826974 shutdown_connections 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 --2- 172.21.15.86:0/2735826974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a1760 0x7f710413af10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- 172.21.15.86:0/2735826974 >> 172.21.15.86:0/2735826974 conn(0x7f71040a2f40 msgr2=0x7f71040a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- 172.21.15.86:0/2735826974 shutdown_connections 2024-09-17T15:41:01.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- 172.21.15.86:0/2735826974 wait complete. 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: monmap: 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:01.601 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 Processor -- start 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- start start 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: init 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a0c70 0x7f7104072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): picked mon.noname-b con 0x7f71040a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 0x7f710409d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): picked mon.noname-c con 0x7f71040a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 0x7f710409dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): picked mon.noname-a con 0x7f710413b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7104141900 con 0x7f710413b450 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7104141780 con 0x7f71040a0c70 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7104141a80 con 0x7f71040a1760 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f710b489640 10 monclient(hunting): authenticate will time out at 1794.562255s 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 0x7f710409dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 0x7f710409d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 0x7f710409dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60484/0 (socket says 172.21.15.86:60484) 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 0x7f710409d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54890/0 (socket says 172.21.15.86:54890) 2024-09-17T15:41:01.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 learned_addr learned my addr 172.21.15.86:0/245436379 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): get_auth_request con 0x7f710413b450 auth_method 0 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): get_auth_request con 0x7f71040a1760 auth_method 0 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a0c70 0x7f7104072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient(hunting): handle_auth_done global_id 144273 payload 824 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 msgr2=0x7f710409d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 0x7f710409d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a0c70 msgr2=0x7f7104072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a0c70 0x7f7104072bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 monclient: found mon.noname-a 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f710409e830 con 0x7f710413b450 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109225640 10 monclient: get_auth_request con 0x7f71040a0c70 auth_method 0 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7108a24640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71040a1760 0x7f710409d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.592+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71040a0c70 0x7f7104072bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 msgr2=0x7f710409dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109a26640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 0x7f710409dcb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00c410 tx=0x7f70fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc017070 con 0x7f710413b450 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: dump: 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70fc005230 con 0x7f710413b450 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 5 monclient: authenticate success, global_id 144273 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 monclient: _renew_subs 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f710406fec0 con 0x7f710413b450 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc0117a0 con 0x7f710413b450 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 10 monclient: dump: 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.608 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f77fe640 4 set_mon_vals no callback set 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 monclient: _renew_subs 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71040701a0 con 0x7f710413b450 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 0x7f710409e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f710402e9b0 con 0x7f710402e5d0 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f70fc01a070 con 0x7f710413b450 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f70e803bb60 0x7f70e803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(474..474 src has 213..474) ==== 4847+0+0 (secure 0 0 0) 0x7f70fc04f150 con 0x7f710413b450 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 0x7f710409e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient: get_auth_request con 0x7f710402e5d0 auth_method 0 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7108a24640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f70e803bb60 0x7f70e803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7108a24640 10 monclient: get_auth_request con 0x7f70e803bb60 auth_method 0 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 10 monclient(hunting): handle_auth_done global_id 154120 payload 824 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.596+0000 7f7109225640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71040a1050 con 0x7f710402e5d0 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f7109225640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 msgr2=0x7f710409e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 0x7f710409e1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f800d980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f7108a24640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f70e803bb60 0x7f70e803e020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f710406d160 tx=0x7f71000a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8007070 con 0x7f710402e5d0 2024-09-17T15:41:01.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:01.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 10 monclient: dump: 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f70f8017070 con 0x7f710402e5d0 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 msgr2=0x7f710409e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 0x7f710409e1f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f800d980 comp rx=0 tx=0).stop 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 msgr2=0x7f710409e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.600+0000 7f70f5ffb640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f710402e5d0 0x7f710409e1f0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f800d980 comp rx=0 tx=0).stop 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f710b489640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f710b489640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 0x7f7104064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f710b489640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f710413a610 con 0x7f7104063e00 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 0x7f7104064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient: get_auth_request con 0x7f7104063e00 auth_method 0 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 10 monclient(hunting): handle_auth_done global_id 154123 payload 824 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.608+0000 7f7109225640 1 -- 172.21.15.86:0/245436379 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f710406d4f0 con 0x7f7104063e00 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f7109225640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 msgr2=0x7f7104064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f7109225640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 0x7f7104064210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f8007710 con 0x7f7104063e00 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 10 monclient: dump: 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f70f8007710 con 0x7f7104063e00 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 msgr2=0x7f7104064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 0x7f7104064210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f8004950 comp rx=0 tx=0).stop 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 msgr2=0x7f7104064210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:01.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f70f5ffb640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7104063e00 0x7f7104064210 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f70f800d4c0 tx=0x7f70f8004950 comp rx=0 tx=0).stop 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 132, 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12975, 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:01.618 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.619 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.620 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.621 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:01.622 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:01.623 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.624 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.625 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.626 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:01.627 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.628 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:01.628 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:01.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:01.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:01.631 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f70e803bb60 msgr2=0x7f70e803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f70e803bb60 0x7f70e803e020 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f710406d160 tx=0x7f71000a7070 comp rx=0 tx=0).stop 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 10 monclient: shutdown 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 msgr2=0x7f710409dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 --2- 172.21.15.86:0/245436379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f710413b450 0x7f710409dcb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00c410 tx=0x7f70fc00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 reap_dead start 2024-09-17T15:41:01.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f7109a26640 1 -- 172.21.15.86:0/245436379 reap_dead start 2024-09-17T15:41:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 shutdown_connections 2024-09-17T15:41:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 >> 172.21.15.86:0/245436379 conn(0x7f71040a2f40 msgr2=0x7f710413db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 shutdown_connections 2024-09-17T15:41:01.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.612+0000 7f710b489640 1 -- 172.21.15.86:0/245436379 wait complete. 2024-09-17T15:41:01.634 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:01.634 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:41:01.634 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:41:01.635 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:41:01.635 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:41:01.635 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:41:01.635 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:41:01.649 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:41:01.650 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:41:01.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: get_monmap_and_config 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: monmap: 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:01.809405+0000 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:01.809405+0000 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 Processor -- start 2024-09-17T15:41:01.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 -- start start 2024-09-17T15:41:01.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: init 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55dc0a50e0 0x7f55dc09ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-a con 0x7f55dc0a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55dc0a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f55dc13d2f0 0x7f55dc13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-c con 0x7f55dc13d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55dc09d350 con 0x7f55dc0a54c0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f55dc09bdc0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f55dc0a6530 con 0x7f55dc13d2f0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e4650640 10 monclient(hunting): authenticate will time out at 1794.775765s 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55dc0a50e0 0x7f55dc09ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55dc0a50e0 0x7f55dc09ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60506/0 (socket says 172.21.15.86:60506) 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 1 -- 172.21.15.86:0/3127612658 learned_addr learned my addr 172.21.15.86:0/3127612658 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 10 monclient(hunting): get_auth_request con 0x7f55dc0a50e0 auth_method 0 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.804+0000 7f55e23ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e2bed640 1 --2- 172.21.15.86:0/3127612658 >> v2:172.21.15.203:3300/0 conn(0x7f55dc13d2f0 0x7f55dc13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient(hunting): handle_auth_done global_id 144276 payload 274 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 1 -- 172.21.15.86:0/3127612658 >> v2:172.21.15.203:3300/0 conn(0x7f55dc13d2f0 msgr2=0x7f55dc13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 1 --2- 172.21.15.86:0/3127612658 >> v2:172.21.15.203:3300/0 conn(0x7f55dc13d2f0 0x7f55dc13f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 1 -- 172.21.15.86:0/3127612658 >> v1:172.21.15.86:6789/0 conn(0x7f55dc0a54c0 legacy=0x7f55dc0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 1 monclient: found mon.noname-a 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.808+0000 7f55e23ec640 1 -- 172.21.15.86:0/3127612658 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc09c150 con 0x7f55dc0a50e0 2024-09-17T15:41:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 -- 172.21.15.86:0/3127612658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 msgr2=0x7f55dc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 --2- 172.21.15.86:0/3127612658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc09ce10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0a8280 tx=0x7f55d801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f891af90e6f7c405 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 1 -- 172.21.15.86:0/3127612658 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80b0070 con 0x7f55dc0a50e0 2024-09-17T15:41:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: dump: 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 1 -- 172.21.15.86:0/3127612658 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d80ab600 con 0x7f55dc0a50e0 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 5 monclient: authenticate success, global_id 144276 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 1 -- 172.21.15.86:0/3127612658 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80ab9d0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 4 set_mon_vals no callback set 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e13ea640 10 monclient: dump: 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.815 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: shutdown 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- 172.21.15.86:0/3127612658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 msgr2=0x7f55dc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- 172.21.15.86:0/3127612658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc09ce10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0a8280 tx=0x7f55d801c390 comp rx=0 tx=0).stop 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- 172.21.15.86:0/3127612658 shutdown_connections 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- 172.21.15.86:0/3127612658 >> v2:172.21.15.203:3300/0 conn(0x7f55dc13d2f0 0x7f55dc13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- 172.21.15.86:0/3127612658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc09ce10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- 172.21.15.86:0/3127612658 >> 172.21.15.86:0/3127612658 conn(0x7f55dc0aaae0 msgr2=0x7f55dc0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:01.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- 172.21.15.86:0/3127612658 shutdown_connections 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- 172.21.15.86:0/3127612658 wait complete. 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: build_initial_monmap 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: monmap: 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:01.821 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 Processor -- start 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- start start 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: init 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-a con 0x7f55dc0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 0x7f55dc1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-c con 0x7f55dc0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55dc13d2f0 0x7f55dc0a2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): picked mon.noname-b con 0x7f55dc13d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:01.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc142650 con 0x7f55dc0a50e0 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55dc06f620 con 0x7f55dc13d2f0 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f55dc1427d0 con 0x7f55dc0a54c0 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e4650640 10 monclient(hunting): authenticate will time out at 1794.782856s 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e1beb640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 msgr2=0x7f55dc1382f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e1beb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 0x7f55dc1382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60522/0 (socket says 172.21.15.86:60522) 2024-09-17T15:41:01.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 -- 172.21.15.86:0/224437148 learned_addr learned my addr 172.21.15.86:0/224437148 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): get_auth_request con 0x7f55dc0a50e0 auth_method 0 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient(hunting): handle_auth_done global_id 144279 payload 824 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 msgr2=0x7f55dc1382f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 0x7f55dc1382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55dc13d2f0 msgr2=0x7f55dc0a2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:01.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55dc13d2f0 0x7f55dc0a2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient: _finish_hunting 0 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 monclient: found mon.noname-a 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.812+0000 7f55e23ec640 1 -- 172.21.15.86:0/224437148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55dc0a18d0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e23ec640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 msgr2=0x7f55dc1cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e23ec640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55d80ae040 tx=0x7f55d801abd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:01.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80ba020 con 0x7f55dc0a50e0 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: dump: 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d80b6850 con 0x7f55dc0a50e0 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 5 monclient: authenticate success, global_id 144279 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _renew_subs 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55dc0a48c0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80b0040 con 0x7f55dc0a50e0 2024-09-17T15:41:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 10 monclient: dump: 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:01.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e0be9640 4 set_mon_vals no callback set 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _renew_subs 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55dc121770 con 0x7f55dc0a50e0 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55dc09ce10 con 0x7f55dc0a50e0 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f55d80bd020 con 0x7f55dc0a50e0 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:01.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(474..474 src has 213..474) ==== 4847+0+0 (secure 0 0 0) 0x7f55d80ea4f0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e1beb640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:01.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.816+0000 7f55e1beb640 10 monclient: get_auth_request con 0x7f55c003bb10 auth_method 0 2024-09-17T15:41:01.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55e1beb640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 msgr2=0x7f55c003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55e1beb640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 msgr2=0x7f55c003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55e1beb640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f55dc13f950 tx=0x7f55cc007a20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55e1beb640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f55d80160d0 con 0x7f55dc0a50e0 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:01.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:01.820+0000 7f55d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:02.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.024+0000 7f55e1beb640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:02.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.024+0000 7f55e1beb640 10 monclient: get_auth_request con 0x7f55c003bb10 auth_method 0 2024-09-17T15:41:02.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.028+0000 7f55e1beb640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f55dc13f950 tx=0x7f55cc007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:02.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f55dc074be0 con 0x7f55dc0a50e0 2024-09-17T15:41:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55d2ffd640 1 -- 172.21.15.86:0/224437148 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f55d80be3b0 con 0x7f55dc0a50e0 2024-09-17T15:41:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:02.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:02.061 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:02.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 msgr2=0x7f55c003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:02.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f55dc13f950 tx=0x7f55cc007000 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 10 monclient: shutdown 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 msgr2=0x7f55dc1cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f55d80ae040 tx=0x7f55d801abd0 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 shutdown_connections 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f55c003bb10 0x7f55c003dfd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55dc13d2f0 0x7f55dc0a2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55dc0a54c0 0x7f55dc1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 --2- 172.21.15.86:0/224437148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55dc0a50e0 0x7f55dc1cfb20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:02.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 >> 172.21.15.86:0/224437148 conn(0x7f55dc0aaae0 msgr2=0x7f55dc0a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:02.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.060+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 shutdown_connections 2024-09-17T15:41:02.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:02.064+0000 7f55e4650640 1 -- 172.21.15.86:0/224437148 wait complete. 2024-09-17T15:41:05.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:05.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 10 monclient: get_monmap_and_config 2024-09-17T15:41:05.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 10 monclient: build_initial_monmap 2024-09-17T15:41:05.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 10 monclient: monmap: 2024-09-17T15:41:05.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:05.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:05.214821+0000 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:05.214821+0000 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:05.211 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 1 Processor -- start 2024-09-17T15:41:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 1 -- start start 2024-09-17T15:41:05.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.208+0000 7fbaabb83640 10 monclient: init 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbaa40689d0 0x7fbaa409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa40689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbaa4068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbaa4139f00 0x7fbaa413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa4139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa409d770 con 0x7fbaa4068db0 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa409ed00 con 0x7fbaa40689d0 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa409efe0 con 0x7fbaa4139f00 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:05.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaabb83640 10 monclient(hunting): authenticate will time out at 1798.180318s 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbaa40689d0 0x7fbaa409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa911e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbaa4068db0 0x7fbaa40691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56842/0 (socket says 172.21.15.86:56842) 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa911e640 1 -- 172.21.15.86:0/3700597269 learned_addr learned my addr 172.21.15.86:0/3700597269 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): get_auth_request con 0x7fbaa40689d0 auth_method 0 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaaa120640 1 --2- 172.21.15.86:0/3700597269 >> v2:172.21.15.203:3300/0 conn(0x7fbaa4139f00 0x7fbaa413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient(hunting): handle_auth_done global_id 144282 payload 274 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 -- 172.21.15.86:0/3700597269 >> v2:172.21.15.203:3300/0 conn(0x7fbaa4139f00 msgr2=0x7fbaa413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:05.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 --2- 172.21.15.86:0/3700597269 >> v2:172.21.15.203:3300/0 conn(0x7fbaa4139f00 0x7fbaa413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 -- 172.21.15.86:0/3700597269 >> v1:172.21.15.86:6789/0 conn(0x7fbaa4068db0 legacy=0x7fbaa40691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient: _finish_hunting 0 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 monclient: found mon.noname-a 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.212+0000 7fbaa991f640 1 -- 172.21.15.86:0/3700597269 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa40a04e0 con 0x7fbaa40689d0 2024-09-17T15:41:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa991f640 1 -- 172.21.15.86:0/3700597269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 msgr2=0x7fbaa409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:05.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa991f640 1 --2- 172.21.15.86:0/3700597269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa409ffa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbaa409f2c0 tx=0x7fbaa001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea7f3dbce5672b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:05.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 1 -- 172.21.15.86:0/3700597269 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00b0070 con 0x7fbaa40689d0 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: dump: 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: _finish_auth 0 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: _check_auth_tickets 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 1 -- 172.21.15.86:0/3700597269 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa00ab600 con 0x7fbaa40689d0 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 1 -- 172.21.15.86:0/3700597269 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00ab9d0 con 0x7fbaa40689d0 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 5 monclient: authenticate success, global_id 144282 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 4 set_mon_vals no callback set 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:05.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaa891d640 10 monclient: dump: 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 10 monclient: shutdown 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:05.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 -- 172.21.15.86:0/3700597269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 msgr2=0x7fbaa409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3700597269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa409ffa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbaa409f2c0 tx=0x7fbaa001c340 comp rx=0 tx=0).stop 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 -- 172.21.15.86:0/3700597269 shutdown_connections 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3700597269 >> v2:172.21.15.203:3300/0 conn(0x7fbaa4139f00 0x7fbaa413c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3700597269 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa409ffa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 -- 172.21.15.86:0/3700597269 >> 172.21.15.86:0/3700597269 conn(0x7fbaa40a6440 msgr2=0x7fbaa40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 -- 172.21.15.86:0/3700597269 shutdown_connections 2024-09-17T15:41:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 1 -- 172.21.15.86:0/3700597269 wait complete. 2024-09-17T15:41:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 10 monclient: build_initial_monmap 2024-09-17T15:41:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.216+0000 7fbaabb83640 10 monclient: monmap: 2024-09-17T15:41:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:05.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 Processor -- start 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- start start 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: init 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:05.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa40689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 0x7fbaa4134fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 0x7fbaa413a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa4139f00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa413f3e0 con 0x7fbaa40689d0 2024-09-17T15:41:05.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa413f260 con 0x7fbaa4139f00 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa413f560 con 0x7fbaa4068db0 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient(hunting): authenticate will time out at 1798.188842s 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa911e640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 msgr2=0x7fbaa4134fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa911e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 0x7fbaa4134fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:05.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60538/0 (socket says 172.21.15.86:60538) 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 -- 172.21.15.86:0/3055279132 learned_addr learned my addr 172.21.15.86:0/3055279132 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): get_auth_request con 0x7fbaa40689d0 auth_method 0 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaaa120640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 0x7fbaa413a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient(hunting): handle_auth_done global_id 144285 payload 824 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 -- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 msgr2=0x7fbaa4134fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:05.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 0x7fbaa4134fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 -- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 msgr2=0x7fbaa413a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 0x7fbaa413a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient: _finish_hunting 0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 monclient: found mon.noname-a 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 -- 172.21.15.86:0/3055279132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa413bcb0 con 0x7fbaa40689d0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaaa120640 10 monclient: get_auth_request con 0x7fbaa4139f00 auth_method 0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaaa120640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 0x7fbaa413a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 -- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 msgr2=0x7fbaa41cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaa991f640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0019f70 tx=0x7fbaa00190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00b9070 con 0x7fbaa40689d0 2024-09-17T15:41:05.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: dump: 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa00b5ac0 con 0x7fbaa40689d0 2024-09-17T15:41:05.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 5 monclient: authenticate success, global_id 144285 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _renew_subs 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa409d460 con 0x7fbaa40689d0 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa00b0040 con 0x7fbaa40689d0 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9a7fc640 10 monclient: dump: 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:05.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fba9bfff640 4 set_mon_vals no callback set 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _renew_subs 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.220+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa4122200 con 0x7fbaa40689d0 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaabb83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaabb83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaabb83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa409ffa0 con 0x7fbaa40689d0 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbaa00bc070 con 0x7fbaa40689d0 2024-09-17T15:41:05.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fba9a7fc640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 0x7fba8803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(474..474 src has 213..474) ==== 4847+0+0 (secure 0 0 0) 0x7fbaa00ea1c0 con 0x7fbaa40689d0 2024-09-17T15:41:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaa911e640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 0x7fba8803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.224+0000 7fbaa911e640 10 monclient: get_auth_request con 0x7fba8803bb60 auth_method 0 2024-09-17T15:41:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.228+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:41:05.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.228+0000 7fbaa911e640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 0x7fba8803e020 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40a1d70 tx=0x7fba94007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:05.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.228+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fbaa00c3070 con 0x7fbaa40689d0 2024-09-17T15:41:05.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.228+0000 7fba9a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:05.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.228+0000 7fba9a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fbaabb83640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fbaabb83640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fbaabb83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:05.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbaa4120f80 con 0x7fbaa40689d0 2024-09-17T15:41:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fba9a7fc640 1 -- 172.21.15.86:0/3055279132 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fbaa0016050 con 0x7fbaa40689d0 2024-09-17T15:41:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fba9a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:05.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.456+0000 7fba9a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:05.459 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 msgr2=0x7fba8803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 0x7fba8803e020 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40a1d70 tx=0x7fba94007b30 comp rx=0 tx=0).stop 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 10 monclient: shutdown 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 msgr2=0x7fbaa41cf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbaa0019f70 tx=0x7fbaa00190e0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 shutdown_connections 2024-09-17T15:41:05.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fba8803bb60 0x7fba8803e020 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbaa4139f00 0x7fbaa413a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbaa4068db0 0x7fbaa4134fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 --2- 172.21.15.86:0/3055279132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbaa40689d0 0x7fbaa41cf9b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 >> 172.21.15.86:0/3055279132 conn(0x7fbaa40a6440 msgr2=0x7fbaa41380a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 shutdown_connections 2024-09-17T15:41:05.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:05.460+0000 7fbaabb83640 1 -- 172.21.15.86:0/3055279132 wait complete. 2024-09-17T15:41:08.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: get_monmap_and_config 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: build_initial_monmap 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: monmap: 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:08.613464+0000 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:08.613464+0000 2024-09-17T15:41:08.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:08.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 Processor -- start 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 -- start start 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: init 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5e80689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5e80694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e806f840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e8074470 con 0x7fb5e80694c0 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e809c8b0 con 0x7fb5e80689d0 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e809cb90 con 0x7fb5e806f840 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5ece1f640 10 monclient(hunting): authenticate will time out at 1801.578942s 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5e6d9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 msgr2=0x7fb5e813e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5e6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5e5d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb5e80694c0 0x7fb5e813aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47260/0 (socket says 172.21.15.86:47260) 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.608+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 learned_addr learned my addr 172.21.15.86:0/2831378811 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.612+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.86:6789/0 conn(0x7fb5e80694c0 legacy=0x7fb5e813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.612+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.203:6789/0 conn(0x7fb5e80689d0 legacy=0x7fb5e8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:08.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.812+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 msgr2=0x7fb5e813e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:08.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:08.812+0000 7fb5e6d9d640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:41:09.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:09.216+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 msgr2=0x7fb5e813e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:09.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:09.216+0000 7fb5e6d9d640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:41:10.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:10.020+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 msgr2=0x7fb5e813e720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:10.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:10.020+0000 7fb5e6d9d640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:41:11.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): tick 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 msgr2=0x7fb5e813e720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e806f840 0x7fb5e813e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.203:6789/0 conn(0x7fb5e80689d0 legacy=0x7fb5e8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.86:6789/0 conn(0x7fb5e80694c0 legacy=0x7fb5e813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb5c8003820 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:11.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb5c8006390 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb5e809cb90 con 0x7fb5c8003820 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb5c8008ed0 con 0x7fb5c8006390 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5d7fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5c8003820 auth_method 0 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:11.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006390 auth_method 0 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:11.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5c8003820 auth_method 0 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:11.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006390 auth_method 0 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:11.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:11.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:41:12.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5c8003820 auth_method 0 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006390 auth_method 0 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:12.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:12.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:41:13.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:13.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006390 auth_method 0 2024-09-17T15:41:13.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:13.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:13.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5c8003820 auth_method 0 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:13.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:13.024+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): tick 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 msgr2=0x7fb5c8008830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.203:3300/0 conn(0x7fb5c8006390 0x7fb5c8008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 msgr2=0x7fb5c8005ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8003820 0x7fb5c8005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 reap_dead start 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5e80694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb5c8006820 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5c8004140 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5e809ce70 con 0x7fb5e80694c0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5c8005e20 con 0x7fb5c8004140 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 1 -- 172.21.15.86:0/2831378811 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb5e809cb90 con 0x7fb5c8006820 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.608+0000 7fb5d7fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006820 auth_method 0 2024-09-17T15:41:14.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:14.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.86:6789/0 conn(0x7fb5e80694c0 legacy=0x7fb5c800b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:41:14.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.612+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.203:6789/0 conn(0x7fb5c8004140 legacy=0x7fb5c80045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006820 auth_method 0 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:14.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:14.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:14.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:14.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:14.816+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006820 auth_method 0 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:15.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:15.220+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:41:16.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5c8006820 auth_method 0 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.024+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3394334399 0 0) 0x7fb5c8005e20 con 0x7fb5c8004140 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 10 monclient(hunting): _negotiate 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 10 monclient(hunting): my global_id is 154126 2024-09-17T15:41:16.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.112+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5d0003530 con 0x7fb5c8004140 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703559350 0 0) 0x7fb5e809ce70 con 0x7fb5e80694c0 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 10 monclient(hunting): _negotiate 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 10 monclient(hunting): my global_id is 144288 2024-09-17T15:41:16.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.116+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5c8005e20 con 0x7fb5e80694c0 2024-09-17T15:41:16.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 104404603 0 0) 0x7fb5d0003530 con 0x7fb5c8004140 2024-09-17T15:41:16.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 msgr2=0x7fb5c8003c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:16.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 --2- 172.21.15.86:0/2831378811 >> v2:172.21.15.86:3300/0 conn(0x7fb5c8006820 0x7fb5c8003c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.86:6789/0 conn(0x7fb5e80694c0 legacy=0x7fb5c800b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5c8010ae0 con 0x7fb5c8004140 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fb5e00043f0 con 0x7fb5c8004140 2024-09-17T15:41:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: dump: 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.130 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.131 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:41:16.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 >> v1:172.21.15.203:6789/0 conn(0x7fb5c8004140 legacy=0x7fb5c80045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:16.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 0x7fb5d0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): picked mon.a con 0x7fb5d0003bc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:16.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d0004810 0x7fb5d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): picked mon.c con 0x7fb5d0004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0005390 0x7fb5d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): picked mon.b con 0x7fb5d0005390 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e809cb90 con 0x7fb5d0003bc0 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d000c2b0 con 0x7fb5d0005390 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5d00108c0 con 0x7fb5d0004810 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.120+0000 7fb5e559a640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d0004810 msgr2=0x7fb5d0004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d0004810 0x7fb5d0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 0x7fb5d0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5d0003bc0 auth_method 0 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0005390 0x7fb5d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): get_auth_request con 0x7fb5d0005390 auth_method 0 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient(hunting): handle_auth_done global_id 154126 payload 293 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d0004810 msgr2=0x7fb5d0004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:16.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5d0004810 0x7fb5d0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0005390 msgr2=0x7fb5d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0005390 0x7fb5d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 monclient: found mon.a 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e6d9d640 1 -- 172.21.15.86:0/2831378811 reap_dead start 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d0003530 con 0x7fb5d0003bc0 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5e659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5ece1f640 5 monclient: authenticate success, global_id 154126 2024-09-17T15:41:16.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.124+0000 7fb5ece1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:16.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e659c640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 msgr2=0x7fb5d0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 0x7fb5d0003fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8007e80 tx=0x7fb5d801d480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7113293d7da6209 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80b1070 con 0x7fb5d0003bc0 2024-09-17T15:41:16.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: dump: 2024-09-17T15:41:16.144 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.145 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5d8005b50 con 0x7fb5d0003bc0 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 1 -- 172.21.15.86:0/2831378811 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8001da0 con 0x7fb5d0003bc0 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 4 set_mon_vals no callback set 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5e559a640 10 monclient: dump: 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.148 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.149 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 10 monclient: shutdown 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 msgr2=0x7fb5d0003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 0x7fb5d0003fc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8007e80 tx=0x7fb5d801d480 comp rx=0 tx=0).stop 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2831378811 shutdown_connections 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2831378811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5d0003bc0 0x7fb5d0003fc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2831378811 >> 172.21.15.86:0/2831378811 conn(0x7fb5e80a6440 msgr2=0x7fb5e80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2831378811 shutdown_connections 2024-09-17T15:41:16.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.128+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2831378811 wait complete. 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient: build_initial_monmap 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient: monmap: 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 Processor -- start 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 -- start start 2024-09-17T15:41:16.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient: init 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0003bc0 0x7fb5e81cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-b con 0x7fb5d0003bc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e81cc580 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:16.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 0x7fb5e8123930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e81cca00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:16.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81362f0 con 0x7fb5e81cc580 2024-09-17T15:41:16.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e809c2d0 con 0x7fb5d0003bc0 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e8136470 con 0x7fb5e81cca00 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5ece1f640 10 monclient(hunting): authenticate will time out at 1809.101934s 2024-09-17T15:41:16.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e6d9d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 msgr2=0x7fb5e8123930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 0x7fb5e8123930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55356/0 (socket says 172.21.15.86:55356) 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2498319302 learned_addr learned my addr 172.21.15.86:0/2498319302 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request con 0x7fb5e81cc580 auth_method 0 2024-09-17T15:41:16.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0003bc0 0x7fb5e81cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e659c640 10 monclient(hunting): get_auth_request con 0x7fb5d0003bc0 auth_method 0 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient(hunting): handle_auth_done global_id 144291 payload 824 2024-09-17T15:41:16.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 msgr2=0x7fb5e8123930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:16.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 0x7fb5e8123930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0003bc0 msgr2=0x7fb5e81cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0003bc0 0x7fb5e81cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 monclient: found mon.noname-a 2024-09-17T15:41:16.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.132+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2498319302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e809d460 con 0x7fb5e81cc580 2024-09-17T15:41:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5e5d9b640 1 -- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 msgr2=0x7fb5e81233f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5e5d9b640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0073c0 tx=0x7fb5dc007880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc001d10 con 0x7fb5e81cc580 2024-09-17T15:41:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:16.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: dump: 2024-09-17T15:41:16.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.163 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.164 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc004c10 con 0x7fb5e81cc580 2024-09-17T15:41:16.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 5 monclient: authenticate success, global_id 144291 2024-09-17T15:41:16.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _renew_subs 2024-09-17T15:41:16.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e8124eb0 con 0x7fb5e81cc580 2024-09-17T15:41:16.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc004de0 con 0x7fb5e81cc580 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 10 monclient: dump: 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.167 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5e4d99640 4 set_mon_vals no callback set 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _renew_subs 2024-09-17T15:41:16.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e81357c0 con 0x7fb5e81cc580 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e8136800 con 0x7fb5e81cc580 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb5dc005050 con 0x7fb5e81cc580 2024-09-17T15:41:16.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 0x7fb5c803f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(475..475 src has 213..475) ==== 4831+0+0 (secure 0 0 0) 0x7fb5dc051770 con 0x7fb5e81cc580 2024-09-17T15:41:16.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 0x7fb5c803f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.136+0000 7fb5e659c640 10 monclient: get_auth_request con 0x7fb5c803cf00 auth_method 0 2024-09-17T15:41:16.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.140+0000 7fb5e659c640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 0x7fb5c803f3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8008fd0 tx=0x7fb5d80018b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.140+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb5dc0173d0 con 0x7fb5e81cc580 2024-09-17T15:41:16.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.140+0000 7fb5d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.140+0000 7fb5d6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5ece1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5ece1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5ece1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb5e806d4f0 con 0x7fb5e81cc580 2024-09-17T15:41:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5d6ffd640 1 -- 172.21.15.86:0/2498319302 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fb5dc019970 con 0x7fb5e81cc580 2024-09-17T15:41:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:16.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.372+0000 7fb5d6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:16.375 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":136,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:16.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 msgr2=0x7fb5c803f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 0x7fb5c803f3c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8008fd0 tx=0x7fb5d80018b0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 10 monclient: shutdown 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 msgr2=0x7fb5e81233f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0073c0 tx=0x7fb5dc007880 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 shutdown_connections 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb5c803cf00 0x7fb5c803f3c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e81cca00 0x7fb5e8123930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e81cc580 0x7fb5e81233f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 --2- 172.21.15.86:0/2498319302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0003bc0 0x7fb5e81cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 >> 172.21.15.86:0/2498319302 conn(0x7fb5e80a6440 msgr2=0x7fb5e8070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 shutdown_connections 2024-09-17T15:41:16.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.376+0000 7fb5ece1f640 1 -- 172.21.15.86:0/2498319302 wait complete. 2024-09-17T15:41:16.392 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:41:16.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:41:16.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.496+0000 7fae8e734640 10 monclient: get_monmap_and_config 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.496+0000 7fae8e734640 10 monclient: build_initial_monmap 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.496+0000 7fae8e734640 10 monclient: monmap: 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:16.503744+0000 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:16.503744+0000 2024-09-17T15:41:16.499 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:16.500 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 Processor -- start 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 -- start start 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient: init 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae880a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fae880a5350 0x7fae880a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-c con 0x7fae880a5350 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:16.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae8813d180 0x7fae8813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-a con 0x7fae8813d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae880a68d0 con 0x7fae880a4f70 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fae880a64a0 con 0x7fae8813d180 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fae8809c230 con 0x7fae880a5350 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8e734640 10 monclient(hunting): authenticate will time out at 1809.469260s 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae8813d180 0x7fae8813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae87fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fae880a4f70 0x7fae8809cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47290/0 (socket says 172.21.15.86:47290) 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae87fff640 1 -- 172.21.15.86:0/726920046 learned_addr learned my addr 172.21.15.86:0/726920046 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): get_auth_request con 0x7fae8813d180 auth_method 0 2024-09-17T15:41:16.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae877fe640 1 --2- 172.21.15.86:0/726920046 >> v2:172.21.15.203:3300/0 conn(0x7fae880a5350 0x7fae880a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae877fe640 10 monclient(hunting): get_auth_request con 0x7fae880a5350 auth_method 0 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient(hunting): handle_auth_done global_id 144294 payload 274 2024-09-17T15:41:16.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 -- 172.21.15.86:0/726920046 >> v2:172.21.15.203:3300/0 conn(0x7fae880a5350 msgr2=0x7fae880a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 --2- 172.21.15.86:0/726920046 >> v2:172.21.15.203:3300/0 conn(0x7fae880a5350 0x7fae880a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 -- 172.21.15.86:0/726920046 >> v1:172.21.15.86:6789/0 conn(0x7fae880a4f70 legacy=0x7fae8809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 monclient: found mon.noname-a 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.500+0000 7fae8ccd1640 1 -- 172.21.15.86:0/726920046 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8809c510 con 0x7fae8813d180 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8ccd1640 1 -- 172.21.15.86:0/726920046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8813d180 msgr2=0x7fae8813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8ccd1640 1 --2- 172.21.15.86:0/726920046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8813d180 0x7fae8813f570 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae8809d9c0 tx=0x7fae8001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef336a2a74343748 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 1 -- 172.21.15.86:0/726920046 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800b2070 con 0x7fae8813d180 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: dump: 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 1 -- 172.21.15.86:0/726920046 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae800a5890 con 0x7fae8813d180 2024-09-17T15:41:16.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 5 monclient: authenticate success, global_id 144294 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 1 -- 172.21.15.86:0/726920046 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800a5c60 con 0x7fae8813d180 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 4 set_mon_vals no callback set 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae86ffd640 10 monclient: dump: 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 10 monclient: shutdown 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 -- 172.21.15.86:0/726920046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8813d180 msgr2=0x7fae8813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 --2- 172.21.15.86:0/726920046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8813d180 0x7fae8813f570 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae8809d9c0 tx=0x7fae8001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 -- 172.21.15.86:0/726920046 shutdown_connections 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 --2- 172.21.15.86:0/726920046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8813d180 0x7fae8813f570 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 --2- 172.21.15.86:0/726920046 >> v2:172.21.15.203:3300/0 conn(0x7fae880a5350 0x7fae880a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 -- 172.21.15.86:0/726920046 >> 172.21.15.86:0/726920046 conn(0x7fae880a7240 msgr2=0x7fae880a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 -- 172.21.15.86:0/726920046 shutdown_connections 2024-09-17T15:41:16.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 -- 172.21.15.86:0/726920046 wait complete. 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 10 monclient: build_initial_monmap 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 10 monclient: monmap: 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.504+0000 7fae8e734640 1 Processor -- start 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- start start 2024-09-17T15:41:16.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: init 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae880a4f70 0x7fae88132120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-b con 0x7fae880a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 0x7fae88132660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-a con 0x7fae880a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8813d180 0x7fae880a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): picked mon.noname-c con 0x7fae8813d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae881426a0 con 0x7fae880a5350 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fae88142520 con 0x7fae880a4f70 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fae88142820 con 0x7fae8813d180 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient(hunting): authenticate will time out at 1809.476603s 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8ccd1640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8813d180 msgr2=0x7fae880a2f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8ccd1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8813d180 0x7fae880a2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 0x7fae88132660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 0x7fae88132660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55374/0 (socket says 172.21.15.86:55374) 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 -- 172.21.15.86:0/1150282409 learned_addr learned my addr 172.21.15.86:0/1150282409 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): get_auth_request con 0x7fae880a5350 auth_method 0 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae87fff640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae880a4f70 0x7fae88132120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae87fff640 10 monclient(hunting): get_auth_request con 0x7fae880a4f70 auth_method 0 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient(hunting): handle_auth_done global_id 144297 payload 824 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8813d180 msgr2=0x7fae880a2f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8813d180 0x7fae880a2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae880a4f70 msgr2=0x7fae88132120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fae880a4f70 0x7fae88132120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 monclient: found mon.noname-a 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8813dc60 con 0x7fae880a5350 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 msgr2=0x7fae88132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae877fe640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 0x7fae88132660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00d7d0 tx=0x7fae7c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c004240 con 0x7fae880a5350 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: dump: 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae7c004420 con 0x7fae880a5350 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 5 monclient: authenticate success, global_id 144297 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _renew_subs 2024-09-17T15:41:16.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae8813d710 con 0x7fae880a5350 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae7c00b900 con 0x7fae880a5350 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae84ff9640 10 monclient: dump: 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _renew_subs 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae867fc640 4 set_mon_vals no callback set 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.508+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae8809eb50 con 0x7fae880a5350 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8e734640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8e734640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 0x7fae880a3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae880a5750 con 0x7fae880a0f30 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 0x7fae880a3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient: get_auth_request con 0x7fae880a0f30 auth_method 0 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fae7c00baa0 con 0x7fae880a5350 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 10 monclient(hunting): handle_auth_done global_id 144300 payload 824 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae880a5ff0 con 0x7fae880a0f30 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae7403bb10 0x7fae7403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(475..475 src has 213..475) ==== 4831+0+0 (secure 0 0 0) 0x7fae7c050340 con 0x7fae880a5350 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8ccd1640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae7403bb10 0x7fae7403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae8ccd1640 10 monclient: get_auth_request con 0x7fae7403bb10 auth_method 0 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 msgr2=0x7fae880a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae87fff640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 0x7fae880a3440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae7800e500 tx=0x7fae7800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae78004e40 con 0x7fae880a0f30 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 10 monclient: dump: 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fae78018070 con 0x7fae880a0f30 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 msgr2=0x7fae880a3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 0x7fae880a3440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae7800e500 tx=0x7fae7800e9c0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 msgr2=0x7fae880a3440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.512+0000 7fae84ff9640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a0f30 0x7fae880a3440 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fae7800e500 tx=0x7fae7800e9c0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.516+0000 7fae8ccd1640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae7403bb10 0x7fae7403dfd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae8806e190 tx=0x7fae800a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae8e734640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae8e734640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 0x7fae88136240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae8e734640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fae880a3d90 con 0x7fae880a3980 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 0x7fae88136240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient: get_auth_request con 0x7fae880a3980 auth_method 0 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 10 monclient(hunting): handle_auth_done global_id 144303 payload 824 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 1 -- 172.21.15.86:0/1150282409 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae8809efd0 con 0x7fae880a3980 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 msgr2=0x7fae88136240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae87fff640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 0x7fae88136240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae78017000 tx=0x7fae78004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae780064e0 con 0x7fae880a3980 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.520+0000 7fae84ff9640 10 monclient: dump: 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7fae780064e0 con 0x7fae880a3980 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 msgr2=0x7fae88136240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 0x7fae88136240 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae78017000 tx=0x7fae78004950 comp rx=0 tx=0).stop 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 msgr2=0x7fae88136240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:16.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae84ff9640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a3980 0x7fae88136240 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fae78017000 tx=0x7fae78004950 comp rx=0 tx=0).stop 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 136, 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 94862, 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:41:16.531 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.533 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:16.534 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.535 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.536 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.537 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.539 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:41:16.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.543 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.544 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:16.544 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae7403bb10 msgr2=0x7fae7403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae7403bb10 0x7fae7403dfd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fae8806e190 tx=0x7fae800a7070 comp rx=0 tx=0).stop 2024-09-17T15:41:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 10 monclient: shutdown 2024-09-17T15:41:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 msgr2=0x7fae88132660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 --2- 172.21.15.86:0/1150282409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae880a5350 0x7fae88132660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fae7c00d7d0 tx=0x7fae7c00dc90 comp rx=0 tx=0).stop 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8ccd1640 1 -- 172.21.15.86:0/1150282409 reap_dead start 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8ccd1640 1 -- 172.21.15.86:0/1150282409 reap_dead start 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 shutdown_connections 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 >> 172.21.15.86:0/1150282409 conn(0x7fae880a7240 msgr2=0x7fae8813c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 shutdown_connections 2024-09-17T15:41:16.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.524+0000 7fae8e734640 1 -- 172.21.15.86:0/1150282409 wait complete. 2024-09-17T15:41:16.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:41:16.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:41:16.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:41:16.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: get_monmap_and_config 2024-09-17T15:41:16.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: build_initial_monmap 2024-09-17T15:41:16.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: monmap: 2024-09-17T15:41:16.653 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:16.653 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:16.657225+0000 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:16.657225+0000 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:16.654 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 Processor -- start 2024-09-17T15:41:16.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 -- start start 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: init 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7100a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe7100a5350 0x7fe7100a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-a con 0x7fe7100a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe71013d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7100a68d0 con 0x7fe71013d180 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7100a64a0 con 0x7fe7100a4f70 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe71009c230 con 0x7fe7100a5350 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe717cde640 10 monclient(hunting): authenticate will time out at 1809.622351s 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe7100a5350 0x7fe7100a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe71627b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe71013d180 0x7fe71013f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47296/0 (socket says 172.21.15.86:47296) 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe71627b640 1 -- 172.21.15.86:0/2391965201 learned_addr learned my addr 172.21.15.86:0/2391965201 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): get_auth_request con 0x7fe7100a5350 auth_method 0 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient(hunting): handle_auth_done global_id 144306 payload 274 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 1 -- 172.21.15.86:0/2391965201 >> v1:172.21.15.203:6789/0 conn(0x7fe7100a4f70 legacy=0x7fe71009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 1 -- 172.21.15.86:0/2391965201 >> v1:172.21.15.86:6789/0 conn(0x7fe71013d180 legacy=0x7fe71013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 1 monclient: found mon.noname-a 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.652+0000 7fe715279640 1 -- 172.21.15.86:0/2391965201 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71009c510 con 0x7fe7100a5350 2024-09-17T15:41:16.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe715279640 1 -- 172.21.15.86:0/2391965201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a5350 msgr2=0x7fe7100a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe715279640 1 --2- 172.21.15.86:0/2391965201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a5350 0x7fe7100a5750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7040097c0 tx=0x7fe70401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960ffd9d0f690bda server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 1 -- 172.21.15.86:0/2391965201 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7040b0070 con 0x7fe7100a5350 2024-09-17T15:41:16.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: dump: 2024-09-17T15:41:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 1 -- 172.21.15.86:0/2391965201 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7040ab600 con 0x7fe7100a5350 2024-09-17T15:41:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 5 monclient: authenticate success, global_id 144306 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 4 set_mon_vals no callback set 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 1 -- 172.21.15.86:0/2391965201 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7040ab9d0 con 0x7fe7100a5350 2024-09-17T15:41:16.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe714a78640 10 monclient: dump: 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.679 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 10 monclient: shutdown 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 1 -- 172.21.15.86:0/2391965201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a5350 msgr2=0x7fe7100a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.656+0000 7fe717cde640 1 --2- 172.21.15.86:0/2391965201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a5350 0x7fe7100a5750 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe7040097c0 tx=0x7fe70401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- 172.21.15.86:0/2391965201 shutdown_connections 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 --2- 172.21.15.86:0/2391965201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a5350 0x7fe7100a5750 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- 172.21.15.86:0/2391965201 >> 172.21.15.86:0/2391965201 conn(0x7fe7100a7240 msgr2=0x7fe7100a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- 172.21.15.86:0/2391965201 shutdown_connections 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- 172.21.15.86:0/2391965201 wait complete. 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient: build_initial_monmap 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient: monmap: 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 Processor -- start 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- start start 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient: init 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 0x7fe7101cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-a con 0x7fe7100a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7100a5350 0x7fe7101380b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-b con 0x7fe7100a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:16.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71013d180 0x7fe71013d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): picked mon.noname-c con 0x7fe71013d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe710143920 con 0x7fe7100a4f70 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7101437a0 con 0x7fe7100a5350 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe710143aa0 con 0x7fe71013d180 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe717cde640 10 monclient(hunting): authenticate will time out at 1809.629372s 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe71627b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71013d180 msgr2=0x7fe71013d980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe71627b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71013d180 0x7fe71013d980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:16.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 0x7fe7101cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 0x7fe7101cfe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55394/0 (socket says 172.21.15.86:55394) 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 -- 172.21.15.86:0/3573569016 learned_addr learned my addr 172.21.15.86:0/3573569016 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): get_auth_request con 0x7fe7100a4f70 auth_method 0 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient(hunting): handle_auth_done global_id 144309 payload 824 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71013d180 msgr2=0x7fe71013d980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe71013d180 0x7fe71013d980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7100a5350 msgr2=0x7fe7101380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7100a5350 0x7fe7101380b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient: _finish_hunting 0 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 monclient: found mon.noname-a 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71013f280 con 0x7fe7100a4f70 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 msgr2=0x7fe7101cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.660+0000 7fe715a7a640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 0x7fe7101cfe80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c7a0 tx=0x7fe70000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700017070 con 0x7fe7100a4f70 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: dump: 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe700005070 con 0x7fe7100a4f70 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 5 monclient: authenticate success, global_id 144309 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 monclient: _renew_subs 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe710121a30 con 0x7fe7100a4f70 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe700005460 con 0x7fe7100a4f70 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 10 monclient: dump: 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70ffff640 4 set_mon_vals no callback set 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 monclient: _renew_subs 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe710121190 con 0x7fe7100a4f70 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 0x7fe71013e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe71013e7e0 con 0x7fe71013dec0 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 0x7fe71013e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient: get_auth_request con 0x7fe71013dec0 auth_method 0 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe70001a070 con 0x7fe7100a4f70 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6fc03bb60 0x7fe6fc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(475..475 src has 213..475) ==== 4831+0+0 (secure 0 0 0) 0x7fe70004f720 con 0x7fe7100a4f70 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe71627b640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6fc03bb60 0x7fe6fc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe71627b640 10 monclient: get_auth_request con 0x7fe6fc03bb60 auth_method 0 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 10 monclient(hunting): handle_auth_done global_id 154138 payload 824 2024-09-17T15:41:16.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.664+0000 7fe715279640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe710121850 con 0x7fe71013dec0 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe715279640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 msgr2=0x7fe71013e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe715279640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 0x7fe71013e2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7100a6380 tx=0x7fe7040abaa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe71627b640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6fc03bb60 0x7fe6fc03e020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe708004820 tx=0x7fe708007ab0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7040b9070 con 0x7fe71013dec0 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 10 monclient: dump: 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.691 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe7040b5480 con 0x7fe71013dec0 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 msgr2=0x7fe71013e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 0x7fe71013e2a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7100a6380 tx=0x7fe7040abaa0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 msgr2=0x7fe71013e2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:16.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.668+0000 7fe70e7fc640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71013dec0 0x7fe71013e2a0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe7100a6380 tx=0x7fe7040abaa0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe717cde640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe717cde640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 0x7fe7100a0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe717cde640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7100a5750 con 0x7fe71009fd10 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 0x7fe7100a0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient: get_auth_request con 0x7fe71009fd10 auth_method 0 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 10 monclient(hunting): handle_auth_done global_id 154141 payload 824 2024-09-17T15:41:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 1 -- 172.21.15.86:0/3573569016 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7100a5ff0 con 0x7fe71009fd10 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 msgr2=0x7fe7100a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.676+0000 7fe715279640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 0x7fe7100a0120 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7040097c0 tx=0x7fe7040a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7040b9040 con 0x7fe71009fd10 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 10 monclient: dump: 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:16.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7fe7040c7430 con 0x7fe71009fd10 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 msgr2=0x7fe7100a0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 0x7fe7100a0120 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7040097c0 tx=0x7fe7040a8000 comp rx=0 tx=0).stop 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 msgr2=0x7fe7100a0120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:16.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe70e7fc640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe71009fd10 0x7fe7100a0120 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 136, 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28043, 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.699 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.700 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.701 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:16.703 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:16.704 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:16.705 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:16.706 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:16.706 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:16.706 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:16.706 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:16.706 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.707 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:16.710 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.711 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:16.712 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.714 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:16.714 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:16.714 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:16.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:16.716 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.717 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:16.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:16.719 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6fc03bb60 msgr2=0x7fe6fc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe6fc03bb60 0x7fe6fc03e020 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe708004820 tx=0x7fe708007ab0 comp rx=0 tx=0).stop 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 10 monclient: shutdown 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 msgr2=0x7fe7101cfe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 --2- 172.21.15.86:0/3573569016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7100a4f70 0x7fe7101cfe80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe70000c7a0 tx=0x7fe70000cc60 comp rx=0 tx=0).stop 2024-09-17T15:41:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe71627b640 1 -- 172.21.15.86:0/3573569016 reap_dead start 2024-09-17T15:41:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe71627b640 1 -- 172.21.15.86:0/3573569016 reap_dead start 2024-09-17T15:41:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 shutdown_connections 2024-09-17T15:41:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 >> 172.21.15.86:0/3573569016 conn(0x7fe7100a7240 msgr2=0x7fe71009edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 shutdown_connections 2024-09-17T15:41:16.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:16.680+0000 7fe717cde640 1 -- 172.21.15.86:0/3573569016 wait complete. 2024-09-17T15:41:16.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:41:16.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:41:17.104 INFO:tasks.rados.rados.0.smithi203.stdout:2076: done (1 left) 2024-09-17T15:41:17.104 INFO:tasks.rados.rados.0.smithi203.stdout:2077: done (0 left) 2024-09-17T15:41:17.105 INFO:tasks.rados.rados.0.smithi203.stdout:2078: read oid 6 snap -1 2024-09-17T15:41:17.105 INFO:tasks.rados.rados.0.smithi203.stdout:2078: expect deleted 2024-09-17T15:41:17.105 INFO:tasks.rados.rados.0.smithi203.stdout:2079: write (excl) oid 33 current snap is 231 2024-09-17T15:41:17.105 INFO:tasks.rados.rados.0.smithi203.stdout:2079: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-17T15:41:17.118 INFO:tasks.rados.rados.0.smithi203.stdout:2079: writing smithi20328086-33 from 403566 to 823569 tid 1 2024-09-17T15:41:17.136 INFO:tasks.rados.rados.0.smithi203.stdout:2079: writing smithi20328086-33 from 1386064 to 1932982 tid 2 2024-09-17T15:41:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:2080: copy_from oid 41 from oid 20 current snap is 231 2024-09-17T15:41:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:2079: finishing write tid 1 to smithi20328086-33 2024-09-17T15:41:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:2078: done (2 left) 2024-09-17T15:41:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:2081: write (excl) oid 14 current snap is 231 2024-09-17T15:41:17.137 INFO:tasks.rados.rados.0.smithi203.stdout:2081: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-17T15:41:17.151 INFO:tasks.rados.rados.0.smithi203.stdout:2081: writing smithi20328086-14 from 420373 to 915625 tid 1 2024-09-17T15:41:17.166 INFO:tasks.rados.rados.0.smithi203.stdout:2081: writing smithi20328086-14 from 1444546 to 2135122 tid 2 2024-09-17T15:41:17.166 INFO:tasks.rados.rados.0.smithi203.stdout:2082: snap_remove snap 227 2024-09-17T15:41:18.110 INFO:tasks.rados.rados.0.smithi203.stdout:2082: done (3 left) 2024-09-17T15:41:18.110 INFO:tasks.rados.rados.0.smithi203.stdout:2083: write oid 50 current snap is 231 2024-09-17T15:41:18.111 INFO:tasks.rados.rados.0.smithi203.stdout:2080: finishing copy_from to smithi20328086-41 2024-09-17T15:41:18.111 INFO:tasks.rados.rados.0.smithi203.stdout:2080: got expected ENOENT (src dne) 2024-09-17T15:41:18.111 INFO:tasks.rados.rados.0.smithi203.stdout:2080: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:41:18.111 INFO:tasks.rados.rados.0.smithi203.stdout:2079: finishing write tid 2 to smithi20328086-33 2024-09-17T15:41:18.111 INFO:tasks.rados.rados.0.smithi203.stdout:2083: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-17T15:41:18.121 INFO:tasks.rados.rados.0.smithi203.stdout:2083: writing smithi20328086-50 from 437180 to 1007681 tid 1 2024-09-17T15:41:18.133 INFO:tasks.rados.rados.0.smithi203.stdout:2083: writing smithi20328086-50 from 1503028 to 2053615 tid 2 2024-09-17T15:41:18.133 INFO:tasks.rados.rados.0.smithi203.stdout:2080: done (3 left) 2024-09-17T15:41:18.134 INFO:tasks.rados.rados.0.smithi203.stdout:2084: snap_remove snap 224 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2084: done (3 left) 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2085: read oid 25 snap -1 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2085: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2079: finishing write tid 3 to smithi20328086-33 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2079: finishing write tid 4 to smithi20328086-33 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2079: oid 33 updating version 0 to 518 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2079: oid 33 version 518 is already newer than 517 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:2079: oid 33 version 518 is already newer than 516 2024-09-17T15:41:19.122 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 518 (ObjNum 738 snap 231 seq_num 738) dirty exists 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2079: left oid 33 (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: finishing write tid 1 to smithi20328086-14 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: finishing write tid 2 to smithi20328086-14 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: finishing write tid 3 to smithi20328086-14 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: finishing write tid 4 to smithi20328086-14 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: oid 14 updating version 0 to 994 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: oid 14 version 994 is already newer than 993 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: oid 14 updating version 994 to 995 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 995 (ObjNum 739 snap 231 seq_num 739) dirty exists 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2081: left oid 14 (ObjNum 739 snap 231 seq_num 739) 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2083: finishing write tid 1 to smithi20328086-50 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2083: finishing write tid 2 to smithi20328086-50 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2083: finishing write tid 3 to smithi20328086-50 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2083: finishing write tid 4 to smithi20328086-50 2024-09-17T15:41:19.123 INFO:tasks.rados.rados.0.smithi203.stdout:2083: oid 50 updating version 0 to 936 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2083: oid 50 updating version 936 to 938 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2083: oid 50 version 938 is already newer than 937 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 938 (ObjNum 740 snap 231 seq_num 740) dirty exists 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2083: left oid 50 (ObjNum 740 snap 231 seq_num 740) 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2079: done (3 left) 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2081: done (2 left) 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2083: done (1 left) 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2086: write (excl) oid 23 current snap is 231 2024-09-17T15:41:19.124 INFO:tasks.rados.rados.0.smithi203.stdout:2086: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-17T15:41:19.135 INFO:tasks.rados.rados.0.smithi203.stdout:2086: writing smithi20328086-23 from 453987 to 1099737 tid 1 2024-09-17T15:41:19.154 INFO:tasks.rados.rados.0.smithi203.stdout:2086: writing smithi20328086-23 from 1845157 to 2453987 tid 2 2024-09-17T15:41:19.154 INFO:tasks.rados.rados.0.smithi203.stdout:2087: write (excl) oid 46 current snap is 231 2024-09-17T15:41:19.154 INFO:tasks.rados.rados.0.smithi203.stdout:2087: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-17T15:41:19.168 INFO:tasks.rados.rados.0.smithi203.stdout:2087: writing smithi20328086-46 from 470794 to 1191793 tid 1 2024-09-17T15:41:19.182 INFO:tasks.rados.rados.0.smithi203.stdout:2087: writing smithi20328086-46 from 1903639 to 2457895 tid 2 2024-09-17T15:41:19.182 INFO:tasks.rados.rados.0.smithi203.stdout:2088: read oid 26 snap -1 2024-09-17T15:41:19.183 INFO:tasks.rados.rados.0.smithi203.stdout:2088: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:19.183 INFO:tasks.rados.rados.0.smithi203.stdout:2089: copy_from oid 50 from oid 49 current snap is 231 2024-09-17T15:41:19.183 INFO:tasks.rados.rados.0.smithi203.stdout:2085: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:41:19.259 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 757992 2024-09-17T15:41:19.297 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:19.343 INFO:tasks.rados.rados.0.smithi203.stdout:2086: finishing write tid 1 to smithi20328086-23 2024-09-17T15:41:19.343 INFO:tasks.rados.rados.0.smithi203.stdout:2086: finishing write tid 2 to smithi20328086-23 2024-09-17T15:41:19.343 INFO:tasks.rados.rados.0.smithi203.stdout:2086: finishing write tid 3 to smithi20328086-23 2024-09-17T15:41:19.343 INFO:tasks.rados.rados.0.smithi203.stdout:2086: finishing write tid 4 to smithi20328086-23 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2086: oid 23 updating version 0 to 1000 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2086: oid 23 updating version 1000 to 1001 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2086: oid 23 version 1001 is already newer than 999 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1001 (ObjNum 741 snap 231 seq_num 741) dirty exists 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2086: left oid 23 (ObjNum 741 snap 231 seq_num 741) 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: finishing write tid 1 to smithi20328086-46 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: finishing write tid 2 to smithi20328086-46 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: finishing write tid 3 to smithi20328086-46 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: finishing write tid 4 to smithi20328086-46 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: oid 46 updating version 0 to 891 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: oid 46 version 891 is already newer than 890 2024-09-17T15:41:19.344 INFO:tasks.rados.rados.0.smithi203.stdout:2087: oid 46 updating version 891 to 892 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 892 (ObjNum 742 snap 231 seq_num 742) dirty exists 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:2087: left oid 46 (ObjNum 742 snap 231 seq_num 742) 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:2089: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:2089: finishing copy_from to smithi20328086-50 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1094 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:2088: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:19.345 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:19.515 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 220167 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2085: done (4 left) 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2086: done (3 left) 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2087: done (2 left) 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2088: done (1 left) 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2089: done (0 left) 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2090: write (excl) oid 10 current snap is 231 2024-09-17T15:41:19.600 INFO:tasks.rados.rados.0.smithi203.stdout:2090: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-17T15:41:19.615 INFO:tasks.rados.rados.0.smithi203.stdout:2090: writing smithi20328086-10 from 487601 to 1283849 tid 1 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2090: writing smithi20328086-10 from 1962121 to 2487601 tid 2 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2091: read oid 49 snap 229 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2091: expect (ObjNum 710 snap 223 seq_num 710) 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2092: read oid 7 snap -1 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2092: expect deleted 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2093: copy_from oid 3 from oid 2 current snap is 231 2024-09-17T15:41:19.631 INFO:tasks.rados.rados.0.smithi203.stdout:2094: snap_remove snap 230 2024-09-17T15:41:20.135 INFO:tasks.rados.rados.0.smithi203.stdout:2094: done (4 left) 2024-09-17T15:41:20.135 INFO:tasks.rados.rados.0.smithi203.stdout:2095: snap_remove snap 228 2024-09-17T15:41:21.142 INFO:tasks.rados.rados.0.smithi203.stdout:2095: done (4 left) 2024-09-17T15:41:21.142 INFO:tasks.rados.rados.0.smithi203.stdout:2096: write oid 41 current snap is 231 2024-09-17T15:41:21.142 INFO:tasks.rados.rados.0.smithi203.stdout:2096: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-17T15:41:21.155 INFO:tasks.rados.rados.0.smithi203.stdout:2096: writing smithi20328086-41 from 504408 to 975905 tid 1 2024-09-17T15:41:21.172 INFO:tasks.rados.rados.0.smithi203.stdout:2096: writing smithi20328086-41 from 1620603 to 2178528 tid 2 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2097: rollback oid 29 current snap is 231 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 29 to 229 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: finishing write tid 1 to smithi20328086-10 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: finishing write tid 2 to smithi20328086-10 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: finishing write tid 3 to smithi20328086-10 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: finishing write tid 4 to smithi20328086-10 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: oid 10 updating version 0 to 751 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: oid 10 version 751 is already newer than 750 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: oid 10 updating version 751 to 752 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 752 (ObjNum 743 snap 231 seq_num 743) dirty exists 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2090: left oid 10 (ObjNum 743 snap 231 seq_num 743) 2024-09-17T15:41:21.173 INFO:tasks.rados.rados.0.smithi203.stdout:2091: expect (ObjNum 710 snap 223 seq_num 710) 2024-09-17T15:41:21.363 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:21.437 INFO:tasks.rados.rados.0.smithi203.stdout:2093: finishing copy_from racing read to smithi20328086-3 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2093: finishing copy_from to smithi20328086-3 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 483 (ObjNum 696 snap 218 seq_num 696) dirty exists 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: finishing write tid 1 to smithi20328086-41 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: finishing write tid 2 to smithi20328086-41 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: finishing write tid 3 to smithi20328086-41 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: finishing write tid 4 to smithi20328086-41 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: oid 41 updating version 0 to 622 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: oid 41 version 622 is already newer than 621 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: oid 41 updating version 622 to 623 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 623 (ObjNum 744 snap 231 seq_num 744) dirty exists 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2096: left oid 41 (ObjNum 744 snap 231 seq_num 744) 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2097: finishing rollback tid 0 to smithi20328086-29 2024-09-17T15:41:21.438 INFO:tasks.rados.rados.0.smithi203.stdout:2097: finishing rollback tid 1 to smithi20328086-29 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2097: finishing rollback tid 2 to smithi20328086-29 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 790 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2090: done (5 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2091: done (4 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2092: done (3 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2093: done (2 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2096: done (1 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2097: done (0 left) 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2098: rollback oid 45 current snap is 231 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 226 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2099: write (excl) oid 4 current snap is 231 2024-09-17T15:41:21.439 INFO:tasks.rados.rados.0.smithi203.stdout:2099: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-17T15:41:21.453 INFO:tasks.rados.rados.0.smithi203.stdout:2099: writing smithi20328086-4 from 521215 to 1067961 tid 1 2024-09-17T15:41:21.470 INFO:tasks.rados.rados.0.smithi203.stdout:2099: writing smithi20328086-4 from 1562732 to 2264315 tid 2 2024-09-17T15:41:21.470 INFO:tasks.rados.rados.0.smithi203.stdout:2100: snap_remove snap 214 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2100: done (2 left) 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2101: rollback oid 14 current snap is 231 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 212 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2098: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 379 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: finishing write tid 1 to smithi20328086-4 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: finishing write tid 2 to smithi20328086-4 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: finishing write tid 3 to smithi20328086-4 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: finishing write tid 4 to smithi20328086-4 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: oid 4 updating version 0 to 404 2024-09-17T15:41:21.697 INFO:tasks.rados.rados.0.smithi203.stdout:2099: oid 4 updating version 404 to 405 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2099: oid 4 version 405 is already newer than 403 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 405 (ObjNum 745 snap 231 seq_num 745) dirty exists 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2099: left oid 4 (ObjNum 745 snap 231 seq_num 745) 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2098: done (2 left) 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2099: done (1 left) 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2102: read oid 23 snap 226 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2102: expect (ObjNum 673 snap 213 seq_num 673) 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2103: write (excl) oid 9 current snap is 231 2024-09-17T15:41:21.698 INFO:tasks.rados.rados.0.smithi203.stdout:2103: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-17T15:41:21.718 INFO:tasks.rados.rados.0.smithi203.stdout:2103: writing smithi20328086-9 from 538022 to 1276370 tid 1 2024-09-17T15:41:21.735 INFO:tasks.rados.rados.0.smithi203.stdout:2103: writing smithi20328086-9 from 1737567 to 2299161 tid 2 2024-09-17T15:41:21.736 INFO:tasks.rados.rados.0.smithi203.stdout:2102: expect (ObjNum 673 snap 213 seq_num 673) 2024-09-17T15:41:21.736 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:21.826 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:21.877 INFO:tasks.rados.rados.0.smithi203.stdout:2101: finishing rollback tid 0 to smithi20328086-14 2024-09-17T15:41:21.877 INFO:tasks.rados.rados.0.smithi203.stdout:2101: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:41:21.877 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1011 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:41:21.877 INFO:tasks.rados.rados.0.smithi203.stdout:2101: done (2 left) 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2102: done (1 left) 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2104: copy_from oid 36 from oid 38 current snap is 231 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: finishing write tid 1 to smithi20328086-9 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: finishing write tid 2 to smithi20328086-9 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: finishing write tid 3 to smithi20328086-9 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: finishing write tid 4 to smithi20328086-9 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: oid 9 updating version 0 to 796 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: oid 9 updating version 796 to 800 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: oid 9 version 800 is already newer than 799 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 800 (ObjNum 746 snap 231 seq_num 746) dirty exists 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: left oid 9 (ObjNum 746 snap 231 seq_num 746) 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2103: done (1 left) 2024-09-17T15:41:21.878 INFO:tasks.rados.rados.0.smithi203.stdout:2105: delete oid 19 current snap is 231 2024-09-17T15:41:21.879 INFO:tasks.rados.rados.0.smithi203.stdout:2104: finishing copy_from to smithi20328086-36 2024-09-17T15:41:21.879 INFO:tasks.rados.rados.0.smithi203.stdout:2104: got expected ENOENT (src dne) 2024-09-17T15:41:21.880 INFO:tasks.rados.rados.0.smithi203.stdout:2104: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:41:21.881 INFO:tasks.rados.rados.0.smithi203.stdout:2104: done (1 left) 2024-09-17T15:41:21.881 INFO:tasks.rados.rados.0.smithi203.stdout:2105: done (0 left) 2024-09-17T15:41:21.881 INFO:tasks.rados.rados.0.smithi203.stdout:2106: read oid 18 snap -1 2024-09-17T15:41:21.881 INFO:tasks.rados.rados.0.smithi203.stdout:2106: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:41:21.881 INFO:tasks.rados.rados.0.smithi203.stdout:2107: snap_remove snap 226 2024-09-17T15:41:22.715 INFO:tasks.rados.rados.0.smithi203.stdout:2107: done (1 left) 2024-09-17T15:41:22.715 INFO:tasks.rados.rados.0.smithi203.stdout:2108: copy_from oid 46 from oid 32 current snap is 231 2024-09-17T15:41:22.715 INFO:tasks.rados.rados.0.smithi203.stdout:2106: expect (ObjNum 657 snap 208 seq_num 657) 2024-09-17T15:41:22.826 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 727651 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:2108: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:2108: finishing copy_from to smithi20328086-46 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 935 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:2106: done (1 left) 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:2108: done (0 left) 2024-09-17T15:41:22.850 INFO:tasks.rados.rados.0.smithi203.stdout:2109: read oid 3 snap -1 2024-09-17T15:41:22.851 INFO:tasks.rados.rados.0.smithi203.stdout:2109: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:41:22.851 INFO:tasks.rados.rados.0.smithi203.stdout:2110: delete oid 28 current snap is 231 2024-09-17T15:41:22.863 INFO:tasks.rados.rados.0.smithi203.stdout:2110: done (1 left) 2024-09-17T15:41:22.863 INFO:tasks.rados.rados.0.smithi203.stdout:2111: write oid 37 current snap is 231 2024-09-17T15:41:22.863 INFO:tasks.rados.rados.0.smithi203.stdout:2111: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-17T15:41:22.882 INFO:tasks.rados.rados.0.smithi203.stdout:2111: writing smithi20328086-37 from 588443 to 1152538 tid 1 2024-09-17T15:41:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2111: writing smithi20328086-37 from 1796660 to 2505581 tid 2 2024-09-17T15:41:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2112: write (excl) oid 25 current snap is 231 2024-09-17T15:41:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2112: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-17T15:41:22.921 INFO:tasks.rados.rados.0.smithi203.stdout:2112: writing smithi20328086-25 from 605250 to 1244594 tid 1 2024-09-17T15:41:22.933 INFO:tasks.rados.rados.0.smithi203.stdout:2112: writing smithi20328086-25 from 1855142 to 2307721 tid 2 2024-09-17T15:41:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2113: snap_create 2024-09-17T15:41:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2109: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:41:23.038 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:23.192 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3526308 2024-09-17T15:41:23.198 INFO:tasks.rados.rados.0.smithi203.stdout:2111: finishing write tid 1 to smithi20328086-37 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: finishing write tid 2 to smithi20328086-37 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: finishing write tid 3 to smithi20328086-37 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: finishing write tid 4 to smithi20328086-37 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: oid 37 updating version 0 to 1043 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: oid 37 updating version 1043 to 1044 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: oid 37 version 1044 is already newer than 1042 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1044 (ObjNum 749 snap 231 seq_num 749) dirty exists 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2111: left oid 37 (ObjNum 749 snap 231 seq_num 749) 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2112: finishing write tid 1 to smithi20328086-25 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2112: finishing write tid 2 to smithi20328086-25 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2112: finishing write tid 3 to smithi20328086-25 2024-09-17T15:41:23.199 INFO:tasks.rados.rados.0.smithi203.stdout:2112: finishing write tid 4 to smithi20328086-25 2024-09-17T15:41:23.200 INFO:tasks.rados.rados.0.smithi203.stdout:2112: oid 25 updating version 0 to 808 2024-09-17T15:41:23.200 INFO:tasks.rados.rados.0.smithi203.stdout:2112: oid 25 version 808 is already newer than 807 2024-09-17T15:41:23.200 INFO:tasks.rados.rados.0.smithi203.stdout:2112: oid 25 version 808 is already newer than 806 2024-09-17T15:41:23.200 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 808 (ObjNum 750 snap 231 seq_num 750) dirty exists 2024-09-17T15:41:23.200 INFO:tasks.rados.rados.0.smithi203.stdout:2112: left oid 25 (ObjNum 750 snap 231 seq_num 750) 2024-09-17T15:41:23.723 INFO:tasks.rados.rados.0.smithi203.stdout:2109: done (3 left) 2024-09-17T15:41:23.723 INFO:tasks.rados.rados.0.smithi203.stdout:2111: done (2 left) 2024-09-17T15:41:23.723 INFO:tasks.rados.rados.0.smithi203.stdout:2112: done (1 left) 2024-09-17T15:41:23.724 INFO:tasks.rados.rados.0.smithi203.stdout:2113: done (0 left) 2024-09-17T15:41:23.724 INFO:tasks.rados.rados.0.smithi203.stdout:2114: rollback oid 12 current snap is 232 2024-09-17T15:41:23.724 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 229 2024-09-17T15:41:23.724 INFO:tasks.rados.rados.0.smithi203.stdout:2115: read oid 41 snap -1 2024-09-17T15:41:23.724 INFO:tasks.rados.rados.0.smithi203.stdout:2115: expect (ObjNum 744 snap 231 seq_num 744) 2024-09-17T15:41:23.725 INFO:tasks.rados.rados.0.smithi203.stdout:2116: write oid 29 current snap is 232 2024-09-17T15:41:23.725 INFO:tasks.rados.rados.0.smithi203.stdout:2116: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-17T15:41:23.736 INFO:tasks.rados.rados.0.smithi203.stdout:2116: writing smithi20328086-29 from 622057 to 1336650 tid 1 2024-09-17T15:41:23.750 INFO:tasks.rados.rados.0.smithi203.stdout:2116: writing smithi20328086-29 from 1913624 to 2622057 tid 2 2024-09-17T15:41:23.751 INFO:tasks.rados.rados.0.smithi203.stdout:2117: write oid 37 current snap is 232 2024-09-17T15:41:23.751 INFO:tasks.rados.rados.0.smithi203.stdout:2117: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-17T15:41:23.764 INFO:tasks.rados.rados.0.smithi203.stdout:2117: writing smithi20328086-37 from 638864 to 1428706 tid 1 2024-09-17T15:41:23.775 INFO:tasks.rados.rados.0.smithi203.stdout:2117: writing smithi20328086-37 from 1972106 to 2428354 tid 2 2024-09-17T15:41:23.775 INFO:tasks.rados.rados.0.smithi203.stdout:2118: read oid 35 snap -1 2024-09-17T15:41:23.776 INFO:tasks.rados.rados.0.smithi203.stdout:2118: expect (ObjNum 706 snap 221 seq_num 706) 2024-09-17T15:41:23.776 INFO:tasks.rados.rados.0.smithi203.stdout:2119: delete oid 19 current snap is 232 2024-09-17T15:41:23.776 INFO:tasks.rados.rados.0.smithi203.stdout:2115: expect (ObjNum 744 snap 231 seq_num 744) 2024-09-17T15:41:23.833 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2392120 2024-09-17T15:41:23.862 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2448998 2024-09-17T15:41:23.867 INFO:tasks.rados.rados.0.smithi203.stdout:2114: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:41:23.867 INFO:tasks.rados.rados.0.smithi203.stdout:2114: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:41:23.867 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 648 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:41:23.867 INFO:tasks.rados.rados.0.smithi203.stdout:2116: finishing write tid 1 to smithi20328086-29 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: finishing write tid 2 to smithi20328086-29 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: finishing write tid 3 to smithi20328086-29 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: finishing write tid 4 to smithi20328086-29 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: oid 29 updating version 0 to 812 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: oid 29 version 812 is already newer than 810 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: oid 29 version 812 is already newer than 811 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 812 (ObjNum 751 snap 232 seq_num 751) dirty exists 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2116: left oid 29 (ObjNum 751 snap 232 seq_num 751) 2024-09-17T15:41:23.868 INFO:tasks.rados.rados.0.smithi203.stdout:2117: finishing write tid 1 to smithi20328086-37 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: finishing write tid 2 to smithi20328086-37 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: finishing write tid 3 to smithi20328086-37 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: finishing write tid 4 to smithi20328086-37 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: oid 37 updating version 0 to 1048 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: oid 37 version 1048 is already newer than 1046 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: oid 37 version 1048 is already newer than 1047 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1048 (ObjNum 752 snap 232 seq_num 752) dirty exists 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2117: left oid 37 (ObjNum 752 snap 232 seq_num 752) 2024-09-17T15:41:23.869 INFO:tasks.rados.rados.0.smithi203.stdout:2118: expect (ObjNum 706 snap 221 seq_num 706) 2024-09-17T15:41:23.939 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:24.011 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2114: done (5 left) 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2115: done (4 left) 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2116: done (3 left) 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2117: done (2 left) 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2118: done (1 left) 2024-09-17T15:41:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2119: done (0 left) 2024-09-17T15:41:24.083 INFO:tasks.rados.rados.0.smithi203.stdout:2120: write (excl) oid 41 current snap is 232 2024-09-17T15:41:24.083 INFO:tasks.rados.rados.0.smithi203.stdout:2120: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-17T15:41:24.100 INFO:tasks.rados.rados.0.smithi203.stdout:2120: writing smithi20328086-41 from 672478 to 1329171 tid 1 2024-09-17T15:41:24.116 INFO:tasks.rados.rados.0.smithi203.stdout:2120: writing smithi20328086-41 from 2089070 to 2548987 tid 2 2024-09-17T15:41:24.116 INFO:tasks.rados.rados.0.smithi203.stdout:2121: read oid 5 snap -1 2024-09-17T15:41:24.116 INFO:tasks.rados.rados.0.smithi203.stdout:2121: expect deleted 2024-09-17T15:41:24.116 INFO:tasks.rados.rados.0.smithi203.stdout:2122: snap_remove snap 212 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2122: done (2 left) 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2123: snap_create 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: finishing write tid 1 to smithi20328086-41 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: finishing write tid 2 to smithi20328086-41 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: finishing write tid 3 to smithi20328086-41 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: finishing write tid 4 to smithi20328086-41 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: oid 41 updating version 0 to 625 2024-09-17T15:41:24.729 INFO:tasks.rados.rados.0.smithi203.stdout:2120: oid 41 updating version 625 to 626 2024-09-17T15:41:24.730 INFO:tasks.rados.rados.0.smithi203.stdout:2120: oid 41 version 626 is already newer than 624 2024-09-17T15:41:24.730 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 626 (ObjNum 754 snap 232 seq_num 754) dirty exists 2024-09-17T15:41:24.730 INFO:tasks.rados.rados.0.smithi203.stdout:2120: left oid 41 (ObjNum 754 snap 232 seq_num 754) 2024-09-17T15:41:25.737 INFO:tasks.rados.rados.0.smithi203.stdout:2120: done (2 left) 2024-09-17T15:41:25.737 INFO:tasks.rados.rados.0.smithi203.stdout:2121: done (1 left) 2024-09-17T15:41:25.737 INFO:tasks.rados.rados.0.smithi203.stdout:2123: done (0 left) 2024-09-17T15:41:25.737 INFO:tasks.rados.rados.0.smithi203.stdout:2124: read oid 22 snap -1 2024-09-17T15:41:25.738 INFO:tasks.rados.rados.0.smithi203.stdout:2124: expect deleted 2024-09-17T15:41:25.738 INFO:tasks.rados.rados.0.smithi203.stdout:2125: write (excl) oid 48 current snap is 233 2024-09-17T15:41:25.738 INFO:tasks.rados.rados.0.smithi203.stdout:2125: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-17T15:41:25.757 INFO:tasks.rados.rados.0.smithi203.stdout:2125: writing smithi20328086-48 from 689285 to 1421227 tid 1 2024-09-17T15:41:25.776 INFO:tasks.rados.rados.0.smithi203.stdout:2125: writing smithi20328086-48 from 2147552 to 2689285 tid 2 2024-09-17T15:41:25.776 INFO:tasks.rados.rados.0.smithi203.stdout:2126: snap_create 2024-09-17T15:41:25.776 INFO:tasks.rados.rados.0.smithi203.stdout:2125: finishing write tid 1 to smithi20328086-48 2024-09-17T15:41:25.783 INFO:tasks.rados.rados.0.smithi203.stdout:2125: finishing write tid 2 to smithi20328086-48 2024-09-17T15:41:25.784 INFO:tasks.rados.rados.0.smithi203.stdout:2125: finishing write tid 3 to smithi20328086-48 2024-09-17T15:41:25.784 INFO:tasks.rados.rados.0.smithi203.stdout:2125: finishing write tid 4 to smithi20328086-48 2024-09-17T15:41:25.784 INFO:tasks.rados.rados.0.smithi203.stdout:2125: oid 48 updating version 0 to 1059 2024-09-17T15:41:25.784 INFO:tasks.rados.rados.0.smithi203.stdout:2125: oid 48 version 1059 is already newer than 1058 2024-09-17T15:41:25.785 INFO:tasks.rados.rados.0.smithi203.stdout:2125: oid 48 updating version 1059 to 1060 2024-09-17T15:41:25.785 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1060 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:41:25.785 INFO:tasks.rados.rados.0.smithi203.stdout:2125: left oid 48 (ObjNum 755 snap 233 seq_num 755) 2024-09-17T15:41:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2124: done (2 left) 2024-09-17T15:41:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2125: done (1 left) 2024-09-17T15:41:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2126: done (0 left) 2024-09-17T15:41:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2127: delete oid 35 current snap is 234 2024-09-17T15:41:26.717 INFO:tasks.rados.rados.0.smithi203.stdout:2127: done (0 left) 2024-09-17T15:41:26.717 INFO:tasks.rados.rados.0.smithi203.stdout:2128: snap_remove snap 231 2024-09-17T15:41:27.702 INFO:tasks.rados.rados.0.smithi203.stdout:2128: done (0 left) 2024-09-17T15:41:27.702 INFO:tasks.rados.rados.0.smithi203.stdout:2129: delete oid 18 current snap is 234 2024-09-17T15:41:27.711 INFO:tasks.rados.rados.0.smithi203.stdout:2129: done (0 left) 2024-09-17T15:41:27.711 INFO:tasks.rados.rados.0.smithi203.stdout:2130: write (excl) oid 2 current snap is 234 2024-09-17T15:41:27.711 INFO:tasks.rados.rados.0.smithi203.stdout:2130: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-17T15:41:27.732 INFO:tasks.rados.rados.0.smithi203.stdout:2130: writing smithi20328086-2 from 739706 to 1297395 tid 1 2024-09-17T15:41:27.750 INFO:tasks.rados.rados.0.smithi203.stdout:2130: writing smithi20328086-2 from 1806645 to 2273900 tid 2 2024-09-17T15:41:27.750 INFO:tasks.rados.rados.0.smithi203.stdout:2131: delete oid 1 current snap is 234 2024-09-17T15:41:27.751 INFO:tasks.rados.rados.0.smithi203.stdout:2130: finishing write tid 1 to smithi20328086-2 2024-09-17T15:41:27.760 INFO:tasks.rados.rados.0.smithi203.stdout:2130: finishing write tid 2 to smithi20328086-2 2024-09-17T15:41:27.763 INFO:tasks.rados.rados.0.smithi203.stdout:2130: finishing write tid 3 to smithi20328086-2 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2130: finishing write tid 4 to smithi20328086-2 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2130: oid 2 updating version 0 to 564 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2130: oid 2 updating version 564 to 565 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2130: oid 2 updating version 565 to 566 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 566 (ObjNum 758 snap 234 seq_num 758) dirty exists 2024-09-17T15:41:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2130: left oid 2 (ObjNum 758 snap 234 seq_num 758) 2024-09-17T15:41:27.766 INFO:tasks.rados.rados.0.smithi203.stdout:2130: done (1 left) 2024-09-17T15:41:27.766 INFO:tasks.rados.rados.0.smithi203.stdout:2131: done (0 left) 2024-09-17T15:41:27.766 INFO:tasks.rados.rados.0.smithi203.stdout:2132: snap_create 2024-09-17T15:41:28.715 INFO:tasks.rados.rados.0.smithi203.stdout:2132: done (0 left) 2024-09-17T15:41:28.715 INFO:tasks.rados.rados.0.smithi203.stdout:2133: read oid 26 snap -1 2024-09-17T15:41:28.715 INFO:tasks.rados.rados.0.smithi203.stdout:2133: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:28.715 INFO:tasks.rados.rados.0.smithi203.stdout:2134: copy_from oid 11 from oid 15 current snap is 235 2024-09-17T15:41:28.715 INFO:tasks.rados.rados.0.smithi203.stdout:2135: rollback oid 34 current snap is 235 2024-09-17T15:41:28.716 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 234 2024-09-17T15:41:28.716 INFO:tasks.rados.rados.0.smithi203.stdout:2136: snap_create 2024-09-17T15:41:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2134: finishing copy_from to smithi20328086-11 2024-09-17T15:41:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2134: got expected ENOENT (src dne) 2024-09-17T15:41:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2134: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:41:28.726 INFO:tasks.rados.rados.0.smithi203.stdout:2135: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:41:28.728 INFO:tasks.rados.rados.0.smithi203.stdout:2135: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:41:28.729 INFO:tasks.rados.rados.0.smithi203.stdout:2135: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:41:28.729 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 631 (ObjNum 684 snap 218 seq_num 684) dirty exists 2024-09-17T15:41:28.733 INFO:tasks.rados.rados.0.smithi203.stdout:2133: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:28.759 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 736014 2024-09-17T15:41:29.717 INFO:tasks.rados.rados.0.smithi203.stdout:2133: done (3 left) 2024-09-17T15:41:29.717 INFO:tasks.rados.rados.0.smithi203.stdout:2134: done (2 left) 2024-09-17T15:41:29.717 INFO:tasks.rados.rados.0.smithi203.stdout:2135: done (1 left) 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2136: done (0 left) 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2137: read oid 36 snap -1 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2137: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2138: rollback oid 19 current snap is 236 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 235 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2139: read oid 31 snap -1 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2139: expect (ObjNum 727 snap 228 seq_num 727) 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2140: read oid 43 snap -1 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2140: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2141: copy_from oid 24 from oid 16 current snap is 236 2024-09-17T15:41:29.718 INFO:tasks.rados.rados.0.smithi203.stdout:2142: delete oid 3 current snap is 236 2024-09-17T15:41:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2138: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:41:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 921 (ObjNum 129 snap 0 seq_num 3606002936) dirty dne 2024-09-17T15:41:29.743 INFO:tasks.rados.rados.0.smithi203.stdout:2138: done (5 left) 2024-09-17T15:41:29.744 INFO:tasks.rados.rados.0.smithi203.stdout:2142: done (4 left) 2024-09-17T15:41:29.744 INFO:tasks.rados.rados.0.smithi203.stdout:2143: write oid 2 current snap is 236 2024-09-17T15:41:29.744 INFO:tasks.rados.rados.0.smithi203.stdout:2143: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-17T15:41:29.753 INFO:tasks.rados.rados.0.smithi203.stdout:2143: writing smithi20328086-2 from 790127 to 1289916 tid 1 2024-09-17T15:41:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2143: writing smithi20328086-2 from 1698444 to 2429379 tid 2 2024-09-17T15:41:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2144: read oid 4 snap 232 2024-09-17T15:41:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2144: expect (ObjNum 745 snap 231 seq_num 745) 2024-09-17T15:41:29.762 INFO:tasks.rados.rados.0.smithi203.stdout:2145: write oid 11 current snap is 236 2024-09-17T15:41:29.762 INFO:tasks.rados.rados.0.smithi203.stdout:2145: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-17T15:41:29.771 INFO:tasks.rados.rados.0.smithi203.stdout:2145: writing smithi20328086-11 from 406934 to 981972 tid 1 2024-09-17T15:41:29.784 INFO:tasks.rados.rados.0.smithi203.stdout:2145: writing smithi20328086-11 from 1640573 to 2115166 tid 2 2024-09-17T15:41:29.790 INFO:tasks.rados.rados.0.smithi203.stdout:2145: writing smithi20328086-11 from 2642791 to 2806934 tid 3 2024-09-17T15:41:29.791 INFO:tasks.rados.rados.0.smithi203.stdout:2146: rollback oid 32 current snap is 236 2024-09-17T15:41:29.791 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 233 2024-09-17T15:41:29.791 INFO:tasks.rados.rados.0.smithi203.stdout:2140: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:41:29.791 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:29.856 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:29.907 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 702829 2024-09-17T15:41:29.945 INFO:tasks.rados.rados.0.smithi203.stdout:2137: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:41:29.952 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 834769 2024-09-17T15:41:30.085 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2746840 2024-09-17T15:41:30.194 INFO:tasks.rados.rados.0.smithi203.stdout:2139: expect (ObjNum 727 snap 228 seq_num 727) 2024-09-17T15:41:30.194 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 130403 2024-09-17T15:41:30.278 INFO:tasks.rados.rados.0.smithi203.stdout:2141: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:41:30.278 INFO:tasks.rados.rados.0.smithi203.stdout:2141: finishing copy_from to smithi20328086-24 2024-09-17T15:41:30.278 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 801 (ObjNum 678 snap 214 seq_num 678) dirty exists 2024-09-17T15:41:30.278 INFO:tasks.rados.rados.0.smithi203.stdout:2143: finishing write tid 1 to smithi20328086-2 2024-09-17T15:41:30.278 INFO:tasks.rados.rados.0.smithi203.stdout:2144: expect (ObjNum 745 snap 231 seq_num 745) 2024-09-17T15:41:30.414 INFO:tasks.rados.rados.0.smithi203.stdout:2143: finishing write tid 2 to smithi20328086-2 2024-09-17T15:41:30.414 INFO:tasks.rados.rados.0.smithi203.stdout:2143: finishing write tid 3 to smithi20328086-2 2024-09-17T15:41:30.414 INFO:tasks.rados.rados.0.smithi203.stdout:2137: done (7 left) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2139: done (6 left) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2140: done (5 left) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2141: done (4 left) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2144: done (3 left) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2147: read oid 12 snap -1 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2147: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2148: write (excl) oid 19 current snap is 236 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2143: finishing write tid 4 to smithi20328086-2 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2143: oid 2 updating version 0 to 569 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2143: oid 2 updating version 569 to 570 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2143: oid 2 version 570 is already newer than 568 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 570 (ObjNum 761 snap 236 seq_num 761) dirty exists 2024-09-17T15:41:30.415 INFO:tasks.rados.rados.0.smithi203.stdout:2143: left oid 2 (ObjNum 761 snap 236 seq_num 761) 2024-09-17T15:41:30.416 INFO:tasks.rados.rados.0.smithi203.stdout:2145: finishing write tid 1 to smithi20328086-11 2024-09-17T15:41:30.416 INFO:tasks.rados.rados.0.smithi203.stdout:2145: finishing write tid 2 to smithi20328086-11 2024-09-17T15:41:30.416 INFO:tasks.rados.rados.0.smithi203.stdout:2145: finishing write tid 3 to smithi20328086-11 2024-09-17T15:41:30.416 INFO:tasks.rados.rados.0.smithi203.stdout:2148: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-17T15:41:30.420 INFO:tasks.rados.rados.0.smithi203.stdout:2148: writing smithi20328086-19 from 423741 to 1074028 tid 1 2024-09-17T15:41:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:2148: writing smithi20328086-19 from 1699055 to 2317306 tid 2 2024-09-17T15:41:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:2143: done (4 left) 2024-09-17T15:41:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:2149: rollback oid 10 current snap is 236 2024-09-17T15:41:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 233 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: finishing write tid 4 to smithi20328086-11 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: finishing write tid 5 to smithi20328086-11 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: oid 11 updating version 0 to 959 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: oid 11 version 959 is already newer than 957 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: oid 11 version 959 is already newer than 958 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: oid 11 updating version 959 to 960 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 960 (ObjNum 762 snap 236 seq_num 762) dirty exists 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: left oid 11 (ObjNum 762 snap 236 seq_num 762) 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2146: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2146: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2146: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1097 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2145: done (4 left) 2024-09-17T15:41:30.429 INFO:tasks.rados.rados.0.smithi203.stdout:2146: done (3 left) 2024-09-17T15:41:30.430 INFO:tasks.rados.rados.0.smithi203.stdout:2150: write (excl) oid 25 current snap is 236 2024-09-17T15:41:30.430 INFO:tasks.rados.rados.0.smithi203.stdout:2150: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-17T15:41:30.434 INFO:tasks.rados.rados.0.smithi203.stdout:2150: writing smithi20328086-25 from 440548 to 1166084 tid 1 2024-09-17T15:41:30.440 INFO:tasks.rados.rados.0.smithi203.stdout:2150: writing smithi20328086-25 from 1757537 to 2235799 tid 2 2024-09-17T15:41:30.444 INFO:tasks.rados.rados.0.smithi203.stdout:2150: writing smithi20328086-25 from 2813990 to 2840548 tid 3 2024-09-17T15:41:30.444 INFO:tasks.rados.rados.0.smithi203.stdout:2151: rollback oid 38 current snap is 236 2024-09-17T15:41:30.444 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 233 2024-09-17T15:41:30.444 INFO:tasks.rados.rados.0.smithi203.stdout:2147: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:41:30.459 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1309644 2024-09-17T15:41:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1794053 2024-09-17T15:41:30.572 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: finishing write tid 1 to smithi20328086-19 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: finishing write tid 2 to smithi20328086-19 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: finishing write tid 3 to smithi20328086-19 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: finishing write tid 4 to smithi20328086-19 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: oid 19 updating version 0 to 1065 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: oid 19 version 1065 is already newer than 1064 2024-09-17T15:41:30.642 INFO:tasks.rados.rados.0.smithi203.stdout:2148: oid 19 updating version 1065 to 1066 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1066 (ObjNum 763 snap 236 seq_num 763) dirty exists 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2148: left oid 19 (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2149: finishing rollback tid 0 to smithi20328086-10 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2149: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2149: finishing rollback tid 2 to smithi20328086-10 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 755 (ObjNum 743 snap 231 seq_num 743) dirty exists 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2151: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 0 (ObjNum 33 snap 0 seq_num 3607109056) dirty dne 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2150: finishing write tid 1 to smithi20328086-25 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2150: finishing write tid 2 to smithi20328086-25 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2147: done (4 left) 2024-09-17T15:41:30.643 INFO:tasks.rados.rados.0.smithi203.stdout:2148: done (3 left) 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2149: done (2 left) 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2151: done (1 left) 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2152: write oid 20 current snap is 236 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2150: finishing write tid 3 to smithi20328086-25 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2150: finishing write tid 4 to smithi20328086-25 2024-09-17T15:41:30.644 INFO:tasks.rados.rados.0.smithi203.stdout:2152: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-17T15:41:30.646 INFO:tasks.rados.rados.0.smithi203.stdout:2152: writing smithi20328086-20 from 457355 to 858140 tid 1 2024-09-17T15:41:30.652 INFO:tasks.rados.rados.0.smithi203.stdout:2152: writing smithi20328086-20 from 1416019 to 2037939 tid 2 2024-09-17T15:41:30.657 INFO:tasks.rados.rados.0.smithi203.stdout:2152: writing smithi20328086-20 from 2725060 to 2857355 tid 3 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2153: read oid 18 snap -1 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2153: expect deleted 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2154: snap_create 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: finishing write tid 5 to smithi20328086-25 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: oid 25 updating version 0 to 827 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: oid 25 version 827 is already newer than 825 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: oid 25 version 827 is already newer than 826 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: oid 25 version 827 is already newer than 824 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 827 (ObjNum 764 snap 236 seq_num 764) dirty exists 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2150: left oid 25 (ObjNum 764 snap 236 seq_num 764) 2024-09-17T15:41:30.658 INFO:tasks.rados.rados.0.smithi203.stdout:2152: finishing write tid 1 to smithi20328086-20 2024-09-17T15:41:30.662 INFO:tasks.rados.rados.0.smithi203.stdout:2152: finishing write tid 2 to smithi20328086-20 2024-09-17T15:41:30.662 INFO:tasks.rados.rados.0.smithi203.stdout:2152: finishing write tid 3 to smithi20328086-20 2024-09-17T15:41:30.662 INFO:tasks.rados.rados.0.smithi203.stdout:2152: finishing write tid 4 to smithi20328086-20 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: finishing write tid 5 to smithi20328086-20 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: oid 20 updating version 0 to 1035 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: oid 20 version 1035 is already newer than 1034 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: oid 20 version 1035 is already newer than 1033 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: oid 20 version 1035 is already newer than 1032 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1035 (ObjNum 765 snap 236 seq_num 765) dirty exists 2024-09-17T15:41:30.663 INFO:tasks.rados.rados.0.smithi203.stdout:2152: left oid 20 (ObjNum 765 snap 236 seq_num 765) 2024-09-17T15:41:30.737 INFO:tasks.rados.rados.0.smithi203.stdout:2150: done (3 left) 2024-09-17T15:41:30.738 INFO:tasks.rados.rados.0.smithi203.stdout:2152: done (2 left) 2024-09-17T15:41:30.738 INFO:tasks.rados.rados.0.smithi203.stdout:2153: done (1 left) 2024-09-17T15:41:30.738 INFO:tasks.rados.rados.0.smithi203.stdout:2154: done (0 left) 2024-09-17T15:41:30.738 INFO:tasks.rados.rados.0.smithi203.stdout:2155: snap_remove snap 234 2024-09-17T15:41:31.703 INFO:tasks.rados.rados.0.smithi203.stdout:2155: done (0 left) 2024-09-17T15:41:31.703 INFO:tasks.rados.rados.0.smithi203.stdout:2156: copy_from oid 3 from oid 40 current snap is 237 2024-09-17T15:41:31.703 INFO:tasks.rados.rados.0.smithi203.stdout:2157: snap_create 2024-09-17T15:41:31.748 INFO:tasks.rados.rados.0.smithi203.stdout:2156: finishing copy_from racing read to smithi20328086-3 2024-09-17T15:41:31.755 INFO:tasks.rados.rados.0.smithi203.stdout:2156: finishing copy_from to smithi20328086-3 2024-09-17T15:41:31.756 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 741 (ObjNum 685 snap 218 seq_num 685) dirty exists 2024-09-17T15:41:32.710 INFO:tasks.rados.rados.0.smithi203.stdout:2156: done (1 left) 2024-09-17T15:41:32.710 INFO:tasks.rados.rados.0.smithi203.stdout:2157: done (0 left) 2024-09-17T15:41:32.710 INFO:tasks.rados.rados.0.smithi203.stdout:2158: snap_remove snap 223 2024-09-17T15:41:33.715 INFO:tasks.rados.rados.0.smithi203.stdout:2158: done (0 left) 2024-09-17T15:41:33.716 INFO:tasks.rados.rados.0.smithi203.stdout:2159: write oid 31 current snap is 238 2024-09-17T15:41:33.716 INFO:tasks.rados.rados.0.smithi203.stdout:2159: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-17T15:41:33.733 INFO:tasks.rados.rados.0.smithi203.stdout:2159: writing smithi20328086-31 from 474162 to 950196 tid 1 2024-09-17T15:41:33.750 INFO:tasks.rados.rados.0.smithi203.stdout:2159: writing smithi20328086-31 from 1358148 to 1840079 tid 2 2024-09-17T15:41:33.768 INFO:tasks.rados.rados.0.smithi203.stdout:2159: writing smithi20328086-31 from 2352483 to 2817786 tid 3 2024-09-17T15:41:33.769 INFO:tasks.rados.rados.0.smithi203.stdout:2160: snap_remove snap 229 2024-09-17T15:41:34.726 INFO:tasks.rados.rados.0.smithi203.stdout:2160: done (1 left) 2024-09-17T15:41:34.726 INFO:tasks.rados.rados.0.smithi203.stdout:2161: delete oid 11 current snap is 238 2024-09-17T15:41:34.726 INFO:tasks.rados.rados.0.smithi203.stdout:2159: finishing write tid 1 to smithi20328086-31 2024-09-17T15:41:34.726 INFO:tasks.rados.rados.0.smithi203.stdout:2159: finishing write tid 2 to smithi20328086-31 2024-09-17T15:41:34.726 INFO:tasks.rados.rados.0.smithi203.stdout:2159: finishing write tid 3 to smithi20328086-31 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: finishing write tid 4 to smithi20328086-31 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: finishing write tid 5 to smithi20328086-31 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: oid 31 updating version 0 to 533 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: oid 31 version 533 is already newer than 532 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: oid 31 version 533 is already newer than 531 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: oid 31 updating version 533 to 534 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 534 (ObjNum 766 snap 238 seq_num 766) dirty exists 2024-09-17T15:41:34.727 INFO:tasks.rados.rados.0.smithi203.stdout:2159: left oid 31 (ObjNum 766 snap 238 seq_num 766) 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2159: done (1 left) 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2161: done (0 left) 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2162: read oid 33 snap -1 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2162: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2163: read oid 22 snap -1 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2163: expect deleted 2024-09-17T15:41:34.748 INFO:tasks.rados.rados.0.smithi203.stdout:2164: delete oid 10 current snap is 238 2024-09-17T15:41:34.758 INFO:tasks.rados.rados.0.smithi203.stdout:2163: done (2 left) 2024-09-17T15:41:34.758 INFO:tasks.rados.rados.0.smithi203.stdout:2164: done (1 left) 2024-09-17T15:41:34.758 INFO:tasks.rados.rados.0.smithi203.stdout:2165: snap_create 2024-09-17T15:41:34.759 INFO:tasks.rados.rados.0.smithi203.stdout:2162: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:41:34.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 821315 2024-09-17T15:41:34.855 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 999743 2024-09-17T15:41:35.723 INFO:tasks.rados.rados.0.smithi203.stdout:2162: done (1 left) 2024-09-17T15:41:35.723 INFO:tasks.rados.rados.0.smithi203.stdout:2165: done (0 left) 2024-09-17T15:41:35.723 INFO:tasks.rados.rados.0.smithi203.stdout:2166: read oid 41 snap -1 2024-09-17T15:41:35.723 INFO:tasks.rados.rados.0.smithi203.stdout:2166: expect (ObjNum 754 snap 232 seq_num 754) 2024-09-17T15:41:35.723 INFO:tasks.rados.rados.0.smithi203.stdout:2167: snap_create 2024-09-17T15:41:35.731 INFO:tasks.rados.rados.0.smithi203.stdout:2166: expect (ObjNum 754 snap 232 seq_num 754) 2024-09-17T15:41:35.740 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1476713 2024-09-17T15:41:35.827 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1909455 2024-09-17T15:41:36.723 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:41:36.724 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:41:36.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:41:36.727 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:41:36.727 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:41:36.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2166: done (1 left) 2024-09-17T15:41:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2167: done (0 left) 2024-09-17T15:41:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2168: write oid 16 current snap is 240 2024-09-17T15:41:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2168: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-17T15:41:36.743 INFO:tasks.rados.rados.0.smithi203.stdout:2168: writing smithi20328086-16 from 524583 to 942717 tid 1 2024-09-17T15:41:36.767 INFO:tasks.rados.rados.0.smithi203.stdout:2168: writing smithi20328086-16 from 1649947 to 2279205 tid 2 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2168: writing smithi20328086-16 from 2834752 to 2924583 tid 3 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2169: rollback oid 4 current snap is 240 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 207 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2168: finishing write tid 1 to smithi20328086-16 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2168: finishing write tid 2 to smithi20328086-16 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2170: rollback oid 46 current snap is 240 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 239 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2171: copy_from oid 45 from oid 48 current snap is 240 2024-09-17T15:41:36.778 INFO:tasks.rados.rados.0.smithi203.stdout:2172: read oid 32 snap 239 2024-09-17T15:41:36.779 INFO:tasks.rados.rados.0.smithi203.stdout:2172: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-17T15:41:36.779 INFO:tasks.rados.rados.0.smithi203.stdout:2173: snap_create 2024-09-17T15:41:36.784 INFO:tasks.rados.rados.0.smithi203.stdout:2168: finishing write tid 3 to smithi20328086-16 2024-09-17T15:41:36.784 INFO:tasks.rados.rados.0.smithi203.stdout:2170: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:41:36.784 INFO:tasks.rados.rados.0.smithi203.stdout:2168: finishing write tid 4 to smithi20328086-16 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: finishing write tid 5 to smithi20328086-16 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: oid 16 updating version 0 to 1076 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: oid 16 updating version 1076 to 1077 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: oid 16 updating version 1077 to 1079 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: oid 16 version 1079 is already newer than 1078 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1079 (ObjNum 769 snap 240 seq_num 769) dirty exists 2024-09-17T15:41:36.785 INFO:tasks.rados.rados.0.smithi203.stdout:2168: left oid 16 (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:41:36.786 INFO:tasks.rados.rados.0.smithi203.stdout:2170: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:41:36.786 INFO:tasks.rados.rados.0.smithi203.stdout:2170: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:41:36.786 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 939 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:41:36.791 INFO:tasks.rados.rados.0.smithi203.stdout:2169: finishing rollback tid 0 to smithi20328086-4 2024-09-17T15:41:36.792 INFO:tasks.rados.rados.0.smithi203.stdout:2169: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:41:36.792 INFO:tasks.rados.rados.0.smithi203.stdout:2169: finishing rollback tid 2 to smithi20328086-4 2024-09-17T15:41:36.792 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 423 (ObjNum 641 snap 202 seq_num 641) dirty exists 2024-09-17T15:41:36.794 INFO:tasks.rados.rados.0.smithi203.stdout:2172: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-17T15:41:36.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: get_monmap_and_config 2024-09-17T15:41:36.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: build_initial_monmap 2024-09-17T15:41:36.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: monmap: 2024-09-17T15:41:36.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:36.857752+0000 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:36.857752+0000 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 Processor -- start 2024-09-17T15:41:36.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 -- start start 2024-09-17T15:41:36.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: init 2024-09-17T15:41:36.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-b con 0x7f40b40a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f40b40a5450 0x7f40b40a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-c con 0x7f40b40a5450 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40b4137e30 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:36.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40b40a6f40 con 0x7f40b4137e30 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f40b40a65e0 con 0x7f40b40a5450 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f40b409c2a0 con 0x7f40b40a5070 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b8ad0640 10 monclient(hunting): authenticate will time out at 1829.822894s 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:45228/0 (socket says 172.21.15.86:45228) 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b2d9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f40b4137e30 0x7f40b4138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58832/0 (socket says 172.21.15.86:58832) 2024-09-17T15:41:36.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 -- 172.21.15.86:0/1138418336 learned_addr learned my addr 172.21.15.86:0/1138418336 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:36.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 10 monclient(hunting): get_auth_request con 0x7f40b40a5070 auth_method 0 2024-09-17T15:41:36.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:36.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:36.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 -- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 msgr2=0x7f40b409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 -- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 msgr2=0x7f40b409cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b259c640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/1138418336 >> v2:172.21.15.203:3300/0 conn(0x7f40b40a5450 0x7f40b40a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): get_auth_request con 0x7f40b40a5450 auth_method 0 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_done global_id 154144 payload 274 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 -- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 msgr2=0x7f40b409cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 -- 172.21.15.86:0/1138418336 >> v1:172.21.15.86:6789/0 conn(0x7f40b4137e30 legacy=0x7f40b4138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:36.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 monclient: found mon.noname-c 2024-09-17T15:41:36.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:36.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:41:36.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.852+0000 7f40b1d9b640 1 -- 172.21.15.86:0/1138418336 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b409c580 con 0x7f40b40a5450 2024-09-17T15:41:36.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 -- 172.21.15.86:0/1138418336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5450 msgr2=0x7f40b40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5450 0x7f40b40a5830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40a80097c0 tx=0x7f40a801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c759ee44a7d111 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 1 -- 172.21.15.86:0/1138418336 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80b0070 con 0x7f40b40a5450 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: dump: 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:36.869 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: _finish_auth 0 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 1 -- 172.21.15.86:0/1138418336 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a80ab600 con 0x7f40b40a5450 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 1 -- 172.21.15.86:0/1138418336 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80ab9d0 con 0x7f40b40a5450 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 5 monclient: authenticate success, global_id 154144 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 4 set_mon_vals no callback set 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b159a640 10 monclient: dump: 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:36.871 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: shutdown 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- 172.21.15.86:0/1138418336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5450 msgr2=0x7f40b40a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:36.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5450 0x7f40b40a5830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40a80097c0 tx=0x7f40a801c360 comp rx=0 tx=0).stop 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- 172.21.15.86:0/1138418336 shutdown_connections 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5450 0x7f40b40a5830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/1138418336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b40a5070 0x7f40b409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- 172.21.15.86:0/1138418336 >> 172.21.15.86:0/1138418336 conn(0x7f40b40742a0 msgr2=0x7f40b40746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- 172.21.15.86:0/1138418336 shutdown_connections 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- 172.21.15.86:0/1138418336 wait complete. 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: build_initial_monmap 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: monmap: 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:36.873 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 Processor -- start 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- start start 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: init 2024-09-17T15:41:36.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5070 0x7f40b41cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-b con 0x7f40b40a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-a con 0x7f40b40a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): picked mon.noname-c con 0x7f40b4137e30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:36.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f40b4142ac0 con 0x7f40b40a5450 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40b40a6a70 con 0x7f40b40a5070 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f40b4142c40 con 0x7f40b4137e30 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b8ad0640 10 monclient(hunting): authenticate will time out at 1829.830428s 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:36962/0 (socket says 172.21.15.86:36962) 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 10 monclient(hunting): get_auth_request con 0x7f40b4137e30 auth_method 0 2024-09-17T15:41:36.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 -- 172.21.15.86:0/2217022748 learned_addr learned my addr 172.21.15.86:0/2217022748 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): get_auth_request con 0x7f40b40a5450 auth_method 0 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 msgr2=0x7f40b413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 msgr2=0x7f40b413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:36.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5070 0x7f40b41cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): get_auth_request con 0x7f40b40a5070 auth_method 0 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b2d9d640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient(hunting): handle_auth_done global_id 144312 payload 824 2024-09-17T15:41:36.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 msgr2=0x7f40b413d0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5070 msgr2=0x7f40b41cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5070 0x7f40b41cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 monclient: found mon.noname-a 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.860+0000 7f40b1d9b640 1 -- 172.21.15.86:0/2217022748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b41d0e90 con 0x7f40b40a5450 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b1d9b640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 msgr2=0x7f40b4136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b1d9b640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ae040 tx=0x7f40a8019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80b0040 con 0x7f40b40a5450 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:36.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: dump: 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:36.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40a80b9070 con 0x7f40b40a5450 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 5 monclient: authenticate success, global_id 144312 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _renew_subs 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40b40a1d00 con 0x7f40b40a5450 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a80aaa10 con 0x7f40b40a5450 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 10 monclient: dump: 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:36.881 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _renew_subs 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b0d99640 4 set_mon_vals no callback set 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40b4121030 con 0x7f40b40a5450 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:36.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40b4068a10 con 0x7f40b40a5450 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f40a80ab4c0 con 0x7f40b40a5450 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 0x7f409403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(496..496 src has 438..496) ==== 4771+0+0 (secure 0 0 0) 0x7f40a80e9d80 con 0x7f40b40a5450 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b259c640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 0x7f409403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.864+0000 7f40b259c640 10 monclient: get_auth_request con 0x7f409403bb60 auth_method 0 2024-09-17T15:41:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.868+0000 7f40b259c640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 0x7f409403e020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f40b409dcd0 tx=0x7f40a4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:36.884 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:36.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.868+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f40a8016070 con 0x7f40b40a5450 2024-09-17T15:41:36.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.868+0000 7f40a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:36.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:36.868+0000 7f40a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:36.922 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:36.961 INFO:tasks.rados.rados.0.smithi203.stdout:2171: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:41:36.961 INFO:tasks.rados.rados.0.smithi203.stdout:2171: finishing copy_from to smithi20328086-45 2024-09-17T15:41:36.961 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1060 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:41:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40b8ad0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40b8ad0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40b8ad0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f40b409cf60 con 0x7f40b40a5450 2024-09-17T15:41:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40a2ffd640 1 -- 172.21.15.86:0/2217022748 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f40a80b3370 con 0x7f40b40a5450 2024-09-17T15:41:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.104+0000 7f40a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:37.106 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":136,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 msgr2=0x7f409403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 0x7f409403e020 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f40b409dcd0 tx=0x7f40a4008040 comp rx=0 tx=0).stop 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 10 monclient: shutdown 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 msgr2=0x7f40b4136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f40a80ae040 tx=0x7f40a8019100 comp rx=0 tx=0).stop 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 shutdown_connections 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f409403bb60 0x7f409403e020 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40b4137e30 0x7f40b413d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40b40a5450 0x7f40b4136f40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 --2- 172.21.15.86:0/2217022748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40b40a5070 0x7f40b41cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 >> 172.21.15.86:0/2217022748 conn(0x7f40b40742a0 msgr2=0x7f40b40a9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 shutdown_connections 2024-09-17T15:41:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:37.108+0000 7f40b8ad0640 1 -- 172.21.15.86:0/2217022748 wait complete. 2024-09-17T15:41:37.751 INFO:tasks.rados.rados.0.smithi203.stdout:2168: done (5 left) 2024-09-17T15:41:37.751 INFO:tasks.rados.rados.0.smithi203.stdout:2169: done (4 left) 2024-09-17T15:41:37.751 INFO:tasks.rados.rados.0.smithi203.stdout:2170: done (3 left) 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2171: done (2 left) 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2172: done (1 left) 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2173: done (0 left) 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2174: read oid 34 snap 232 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2174: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:41:37.752 INFO:tasks.rados.rados.0.smithi203.stdout:2175: snap_create 2024-09-17T15:41:37.767 INFO:tasks.rados.rados.0.smithi203.stdout:2174: expect (ObjNum 684 snap 218 seq_num 684) 2024-09-17T15:41:37.776 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1063205 2024-09-17T15:41:37.836 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2077298 2024-09-17T15:41:38.759 INFO:tasks.rados.rados.0.smithi203.stdout:2174: done (1 left) 2024-09-17T15:41:38.759 INFO:tasks.rados.rados.0.smithi203.stdout:2175: done (0 left) 2024-09-17T15:41:38.759 INFO:tasks.rados.rados.0.smithi203.stdout:2176: read oid 28 snap -1 2024-09-17T15:41:38.759 INFO:tasks.rados.rados.0.smithi203.stdout:2176: expect deleted 2024-09-17T15:41:38.759 INFO:tasks.rados.rados.0.smithi203.stdout:2177: snap_remove snap 207 2024-09-17T15:41:39.767 INFO:tasks.rados.rados.0.smithi203.stdout:2177: done (1 left) 2024-09-17T15:41:39.768 INFO:tasks.rados.rados.0.smithi203.stdout:2178: delete oid 2 current snap is 242 2024-09-17T15:41:39.798 INFO:tasks.rados.rados.0.smithi203.stdout:2176: done (1 left) 2024-09-17T15:41:39.798 INFO:tasks.rados.rados.0.smithi203.stdout:2178: done (0 left) 2024-09-17T15:41:39.798 INFO:tasks.rados.rados.0.smithi203.stdout:2179: copy_from oid 46 from oid 38 current snap is 242 2024-09-17T15:41:39.798 INFO:tasks.rados.rados.0.smithi203.stdout:2180: snap_remove snap 238 2024-09-17T15:41:40.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: get_monmap_and_config 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: build_initial_monmap 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: monmap: 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:40.261811+0000 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:40.261811+0000 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:40.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:40.258 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 Processor -- start 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 -- start start 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: init 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:41:40.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a200a5070 0x7f6a2009cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a200a5070 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 0x7f6a200a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a200a5450 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a20137e30 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a200a6f40 con 0x7f6a20137e30 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6a200a65e0 con 0x7f6a200a5070 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6a2009c2a0 con 0x7f6a200a5450 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a270e6640 10 monclient(hunting): authenticate will time out at 1833.227504s 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a200a5070 0x7f6a2009cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a200a5070 0x7f6a2009cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59602/0 (socket says 172.21.15.86:59602) 2024-09-17T15:41:40.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 1 -- 172.21.15.86:0/1402374280 learned_addr learned my addr 172.21.15.86:0/1402374280 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): get_auth_request con 0x7f6a200a5070 auth_method 0 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a1ffff640 1 --2- 172.21.15.86:0/1402374280 >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 0x7f6a200a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 10 monclient(hunting): handle_auth_done global_id 144315 payload 274 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 1 -- 172.21.15.86:0/1402374280 >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 msgr2=0x7f6a200a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.256+0000 7f6a24e82640 1 --2- 172.21.15.86:0/1402374280 >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 0x7f6a200a5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 1 -- 172.21.15.86:0/1402374280 >> v1:172.21.15.203:6789/0 conn(0x7f6a20137e30 legacy=0x7f6a20138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 10 monclient: _finish_hunting 0 2024-09-17T15:41:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 1 monclient: found mon.noname-a 2024-09-17T15:41:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a24e82640 1 -- 172.21.15.86:0/1402374280 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2009c580 con 0x7f6a200a5070 2024-09-17T15:41:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a1ffff640 10 monclient: get_auth_request con 0x7f6a200a5450 auth_method 0 2024-09-17T15:41:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.260+0000 7f6a1ffff640 1 --2- 172.21.15.86:0/1402374280 >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 0x7f6a200a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 -- 172.21.15.86:0/1402374280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5070 msgr2=0x7f6a2009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 --2- 172.21.15.86:0/1402374280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5070 0x7f6a2009cf60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6a200a17e0 tx=0x7f6a1001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b255ce3b76a89 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 1 -- 172.21.15.86:0/1402374280 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100b1070 con 0x7f6a200a5070 2024-09-17T15:41:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: dump: 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:41:40.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 5 monclient: authenticate success, global_id 144315 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 1 -- 172.21.15.86:0/1402374280 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a100ac620 con 0x7f6a200a5070 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 1 -- 172.21.15.86:0/1402374280 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a100ac9f0 con 0x7f6a200a5070 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1f7fe640 10 monclient: dump: 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:40.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 4 set_mon_vals no callback set 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: shutdown 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- 172.21.15.86:0/1402374280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5070 msgr2=0x7f6a2009cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- 172.21.15.86:0/1402374280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5070 0x7f6a2009cf60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6a200a17e0 tx=0x7f6a1001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- 172.21.15.86:0/1402374280 shutdown_connections 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- 172.21.15.86:0/1402374280 >> v2:172.21.15.203:3300/0 conn(0x7f6a200a5450 0x7f6a200a5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- 172.21.15.86:0/1402374280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5070 0x7f6a2009cf60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- 172.21.15.86:0/1402374280 >> 172.21.15.86:0/1402374280 conn(0x7f6a200742a0 msgr2=0x7f6a200746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:40.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- 172.21.15.86:0/1402374280 shutdown_connections 2024-09-17T15:41:40.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- 172.21.15.86:0/1402374280 wait complete. 2024-09-17T15:41:40.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: build_initial_monmap 2024-09-17T15:41:40.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: monmap: 2024-09-17T15:41:40.271 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:40.271 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 Processor -- start 2024-09-17T15:41:40.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- start start 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: init 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-c con 0x7f6a200a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-a con 0x7f6a200a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a20137e30 0x7f6a2013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): picked mon.noname-b con 0x7f6a20137e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6a20142ac0 con 0x7f6a200a5450 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:40.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a200a6a70 con 0x7f6a20137e30 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6a20142c40 con 0x7f6a200a5070 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a270e6640 10 monclient(hunting): authenticate will time out at 1833.235276s 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59606/0 (socket says 172.21.15.86:59606) 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 -- 172.21.15.86:0/2736271121 learned_addr learned my addr 172.21.15.86:0/2736271121 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): get_auth_request con 0x7f6a200a5450 auth_method 0 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 10 monclient(hunting): get_auth_request con 0x7f6a200a5070 auth_method 0 2024-09-17T15:41:40.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 msgr2=0x7f6a201cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 msgr2=0x7f6a201cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a24e82640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient(hunting): handle_auth_done global_id 144318 payload 824 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 msgr2=0x7f6a201cfec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a20137e30 msgr2=0x7f6a2013d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:40.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a25683640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a20137e30 0x7f6a2013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a20137e30 0x7f6a2013d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 monclient: found mon.noname-a 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.264+0000 7f6a1ffff640 1 -- 172.21.15.86:0/2736271121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a20121340 con 0x7f6a200a5450 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1ffff640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 msgr2=0x7f6a20136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1ffff640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c7c0 tx=0x7f6a1400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14007c40 con 0x7f6a200a5450 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: dump: 2024-09-17T15:41:40.276 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a14007de0 con 0x7f6a200a5450 2024-09-17T15:41:40.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14011630 con 0x7f6a200a5450 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 10 monclient: dump: 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:40.278 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 5 monclient: authenticate success, global_id 144318 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _renew_subs 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1effd640 4 set_mon_vals no callback set 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a20121620 con 0x7f6a200a5450 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _renew_subs 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a201d0d40 con 0x7f6a200a5450 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2009cf60 con 0x7f6a200a5450 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6a140117d0 con 0x7f6a200a5450 2024-09-17T15:41:40.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 0x7f6a0c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(499..499 src has 438..499) ==== 4815+0+0 (secure 0 0 0) 0x7f6a14050280 con 0x7f6a200a5450 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a24e82640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 0x7f6a0c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.268+0000 7f6a24e82640 10 monclient: get_auth_request con 0x7f6a0c03bbb0 auth_method 0 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.272+0000 7f6a24e82640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 0x7f6a0c03e070 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6a200a8470 tx=0x7f6a10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.272+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6a14015020 con 0x7f6a200a5450 2024-09-17T15:41:40.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.272+0000 7f6a1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:40.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.272+0000 7f6a1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:40.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a270e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:40.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a270e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a270e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6a20137c10 con 0x7f6a200a5450 2024-09-17T15:41:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a1d7fa640 1 -- 172.21.15.86:0/2736271121 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f6a20137c10 con 0x7f6a200a5450 2024-09-17T15:41:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:40.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.508+0000 7f6a1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:40.513 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":136,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:40.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 msgr2=0x7f6a0c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:40.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 0x7f6a0c03e070 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6a200a8470 tx=0x7f6a10008040 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 10 monclient: shutdown 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 msgr2=0x7f6a20136f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6a1400c7c0 tx=0x7f6a1400cc80 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 shutdown_connections 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6a0c03bbb0 0x7f6a0c03e070 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a20137e30 0x7f6a2013d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a200a5450 0x7f6a20136f70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 --2- 172.21.15.86:0/2736271121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a200a5070 0x7f6a201cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 >> 172.21.15.86:0/2736271121 conn(0x7f6a200742a0 msgr2=0x7f6a200a9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 shutdown_connections 2024-09-17T15:41:40.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:40.512+0000 7f6a270e6640 1 -- 172.21.15.86:0/2736271121 wait complete. 2024-09-17T15:41:40.779 INFO:tasks.rados.rados.0.smithi203.stdout:2180: done (1 left) 2024-09-17T15:41:40.779 INFO:tasks.rados.rados.0.smithi203.stdout:2181: delete oid 12 current snap is 242 2024-09-17T15:41:40.779 INFO:tasks.rados.rados.0.smithi203.stdout:2179: finishing copy_from to smithi20328086-46 2024-09-17T15:41:40.779 INFO:tasks.rados.rados.0.smithi203.stdout:2179: got expected ENOENT (src dne) 2024-09-17T15:41:40.779 INFO:tasks.rados.rados.0.smithi203.stdout:2179: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:41:40.788 INFO:tasks.rados.rados.0.smithi203.stdout:2179: done (1 left) 2024-09-17T15:41:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:2181: done (0 left) 2024-09-17T15:41:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:2182: rollback oid 44 current snap is 242 2024-09-17T15:41:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 233 2024-09-17T15:41:40.790 INFO:tasks.rados.rados.0.smithi203.stdout:2183: rollback oid 28 current snap is 242 2024-09-17T15:41:40.790 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 28 to 235 2024-09-17T15:41:40.790 INFO:tasks.rados.rados.0.smithi203.stdout:2184: write oid 38 current snap is 242 2024-09-17T15:41:40.790 INFO:tasks.rados.rados.0.smithi203.stdout:2184: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-17T15:41:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:2184: writing smithi20328086-38 from 575004 to 1218885 tid 1 2024-09-17T15:41:40.825 INFO:tasks.rados.rados.0.smithi203.stdout:2184: writing smithi20328086-38 from 1709040 to 2201978 tid 2 2024-09-17T15:41:40.838 INFO:tasks.rados.rados.0.smithi203.stdout:2184: writing smithi20328086-38 from 2917021 to 2975004 tid 3 2024-09-17T15:41:40.838 INFO:tasks.rados.rados.0.smithi203.stdout:2185: write oid 40 current snap is 242 2024-09-17T15:41:40.838 INFO:tasks.rados.rados.0.smithi203.stdout:2185: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-09-17T15:41:40.853 INFO:tasks.rados.rados.0.smithi203.stdout:2185: writing smithi20328086-40 from 591811 to 1310941 tid 1 2024-09-17T15:41:40.865 INFO:tasks.rados.rados.0.smithi203.stdout:2185: writing smithi20328086-40 from 1767522 to 2404118 tid 2 2024-09-17T15:41:40.871 INFO:tasks.rados.rados.0.smithi203.stdout:2185: writing smithi20328086-40 from 2828091 to 2991811 tid 3 2024-09-17T15:41:40.871 INFO:tasks.rados.rados.0.smithi203.stdout:2186: write oid 42 current snap is 242 2024-09-17T15:41:40.871 INFO:tasks.rados.rados.0.smithi203.stdout:2183: finishing rollback tid 1 to smithi20328086-28 2024-09-17T15:41:40.871 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 0 (ObjNum 129 snap 0 seq_num 3605939184) dirty dne 2024-09-17T15:41:40.871 INFO:tasks.rados.rados.0.smithi203.stdout:2186: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-17T15:41:40.882 INFO:tasks.rados.rados.0.smithi203.stdout:2186: writing smithi20328086-42 from 608618 to 1402997 tid 1 2024-09-17T15:41:40.895 INFO:tasks.rados.rados.0.smithi203.stdout:2186: writing smithi20328086-42 from 2109651 to 2889905 tid 2 2024-09-17T15:41:40.895 INFO:tasks.rados.rados.0.smithi203.stdout:2183: done (4 left) 2024-09-17T15:41:40.895 INFO:tasks.rados.rados.0.smithi203.stdout:2187: rollback oid 12 current snap is 242 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 236 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2182: finishing rollback tid 0 to smithi20328086-44 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2182: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2182: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 988 (ObjNum 734 snap 230 seq_num 734) dirty exists 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2184: finishing write tid 1 to smithi20328086-38 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2184: finishing write tid 2 to smithi20328086-38 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2182: done (4 left) 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2188: copy_from oid 46 from oid 30 current snap is 242 2024-09-17T15:41:40.896 INFO:tasks.rados.rados.0.smithi203.stdout:2184: finishing write tid 3 to smithi20328086-38 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: finishing write tid 4 to smithi20328086-38 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: finishing write tid 5 to smithi20328086-38 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: oid 38 updating version 0 to 757 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: oid 38 updating version 757 to 758 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: oid 38 updating version 758 to 759 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: oid 38 updating version 759 to 760 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 760 (ObjNum 772 snap 242 seq_num 772) dirty exists 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: left oid 38 (ObjNum 772 snap 242 seq_num 772) 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2185: finishing write tid 1 to smithi20328086-40 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2184: done (4 left) 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2189: copy_from oid 13 from oid 50 current snap is 242 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2185: finishing write tid 2 to smithi20328086-40 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2185: finishing write tid 3 to smithi20328086-40 2024-09-17T15:41:40.897 INFO:tasks.rados.rados.0.smithi203.stdout:2185: finishing write tid 4 to smithi20328086-40 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: finishing write tid 5 to smithi20328086-40 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: oid 40 updating version 0 to 991 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: oid 40 updating version 991 to 992 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: oid 40 version 992 is already newer than 990 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: oid 40 updating version 992 to 993 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 993 (ObjNum 773 snap 242 seq_num 773) dirty exists 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: left oid 40 (ObjNum 773 snap 242 seq_num 773) 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2185: done (4 left) 2024-09-17T15:41:40.898 INFO:tasks.rados.rados.0.smithi203.stdout:2190: snap_remove snap 237 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2190: done (4 left) 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2191: copy_from oid 47 from oid 21 current snap is 242 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2186: finishing write tid 1 to smithi20328086-42 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2187: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2187: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 651 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2186: finishing write tid 2 to smithi20328086-42 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2186: finishing write tid 3 to smithi20328086-42 2024-09-17T15:41:41.705 INFO:tasks.rados.rados.0.smithi203.stdout:2186: finishing write tid 4 to smithi20328086-42 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2186: oid 42 updating version 0 to 940 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2186: oid 42 updating version 940 to 941 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2186: oid 42 updating version 941 to 942 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 942 (ObjNum 774 snap 242 seq_num 774) dirty exists 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2186: left oid 42 (ObjNum 774 snap 242 seq_num 774) 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2189: finishing copy_from racing read to smithi20328086-13 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2189: finishing copy_from to smithi20328086-13 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1094 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2188: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2188: finishing copy_from to smithi20328086-46 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 554 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2186: done (4 left) 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2187: done (3 left) 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2188: done (2 left) 2024-09-17T15:41:41.706 INFO:tasks.rados.rados.0.smithi203.stdout:2189: done (1 left) 2024-09-17T15:41:41.707 INFO:tasks.rados.rados.0.smithi203.stdout:2192: write (excl) oid 11 current snap is 242 2024-09-17T15:41:41.707 INFO:tasks.rados.rados.0.smithi203.stdout:2192: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-17T15:41:41.727 INFO:tasks.rados.rados.0.smithi203.stdout:2192: writing smithi20328086-11 from 625425 to 1095053 tid 1 2024-09-17T15:41:41.737 INFO:tasks.rados.rados.0.smithi203.stdout:2192: writing smithi20328086-11 from 1768133 to 2408398 tid 2 2024-09-17T15:41:41.737 INFO:tasks.rados.rados.0.smithi203.stdout:2193: snap_remove snap 236 2024-09-17T15:41:42.790 INFO:tasks.rados.rados.0.smithi203.stdout:2193: done (2 left) 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2194: write (excl) oid 22 current snap is 242 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: finishing write tid 1 to smithi20328086-11 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: finishing write tid 2 to smithi20328086-11 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: finishing write tid 3 to smithi20328086-11 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: finishing write tid 4 to smithi20328086-11 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: oid 11 updating version 0 to 1000 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: oid 11 updating version 1000 to 1001 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: oid 11 updating version 1001 to 1002 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1002 (ObjNum 775 snap 242 seq_num 775) dirty exists 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2192: left oid 11 (ObjNum 775 snap 242 seq_num 775) 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2191: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2191: finishing copy_from to smithi20328086-47 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 749 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:41:42.791 INFO:tasks.rados.rados.0.smithi203.stdout:2194: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-17T15:41:42.804 INFO:tasks.rados.rados.0.smithi203.stdout:2194: writing smithi20328086-22 from 642232 to 1303462 tid 1 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2194: writing smithi20328086-22 from 1942968 to 2726891 tid 2 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2191: done (2 left) 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2192: done (1 left) 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2195: read oid 50 snap -1 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2195: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2196: read oid 26 snap -1 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2196: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2197: write oid 4 current snap is 242 2024-09-17T15:41:42.819 INFO:tasks.rados.rados.0.smithi203.stdout:2194: finishing write tid 1 to smithi20328086-22 2024-09-17T15:41:42.820 INFO:tasks.rados.rados.0.smithi203.stdout:2197: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-17T15:41:42.832 INFO:tasks.rados.rados.0.smithi203.stdout:2197: writing smithi20328086-4 from 659039 to 1395518 tid 1 2024-09-17T15:41:42.845 INFO:tasks.rados.rados.0.smithi203.stdout:2197: writing smithi20328086-4 from 2001450 to 2645384 tid 2 2024-09-17T15:41:42.845 INFO:tasks.rados.rados.0.smithi203.stdout:2198: snap_create 2024-09-17T15:41:42.845 INFO:tasks.rados.rados.0.smithi203.stdout:2195: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:41:42.845 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:42.930 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:2194: finishing write tid 2 to smithi20328086-22 2024-09-17T15:41:43.029 INFO:tasks.rados.rados.0.smithi203.stdout:2196: expect (ObjNum 689 snap 218 seq_num 689) 2024-09-17T15:41:43.169 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2640185 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: finishing write tid 3 to smithi20328086-22 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: finishing write tid 4 to smithi20328086-22 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: oid 22 updating version 0 to 537 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: oid 22 version 537 is already newer than 535 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: oid 22 version 537 is already newer than 536 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 537 (ObjNum 776 snap 242 seq_num 776) dirty exists 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2194: left oid 22 (ObjNum 776 snap 242 seq_num 776) 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2197: finishing write tid 1 to smithi20328086-4 2024-09-17T15:41:43.262 INFO:tasks.rados.rados.0.smithi203.stdout:2197: finishing write tid 2 to smithi20328086-4 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: finishing write tid 3 to smithi20328086-4 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: finishing write tid 4 to smithi20328086-4 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: oid 4 updating version 0 to 1063 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: oid 4 version 1063 is already newer than 1061 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: oid 4 version 1063 is already newer than 1062 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1063 (ObjNum 777 snap 242 seq_num 777) dirty exists 2024-09-17T15:41:43.263 INFO:tasks.rados.rados.0.smithi203.stdout:2197: left oid 4 (ObjNum 777 snap 242 seq_num 777) 2024-09-17T15:41:43.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:43.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: get_monmap_and_config 2024-09-17T15:41:43.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: build_initial_monmap 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: monmap: 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:43.665252+0000 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:43.665252+0000 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:43.661 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 Processor -- start 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 -- start start 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: init 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea940689d0 0x7fea9406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): picked mon.noname-a con 0x7fea940689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea94068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fea9413aff0 0x7fea9413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): picked mon.noname-c con 0x7fea9413aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea9409c8b0 con 0x7fea94068db0 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:43.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fea94074500 con 0x7fea940689d0 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fea9409cb90 con 0x7fea9413aff0 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea99146640 10 monclient(hunting): authenticate will time out at 1836.630834s 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fea940689d0 0x7fea9406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea8a59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fea94068db0 0x7fea94069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48592/0 (socket says 172.21.15.86:48592) 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea8a59c640 1 -- 172.21.15.86:0/436998248 learned_addr learned my addr 172.21.15.86:0/436998248 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): get_auth_request con 0x7fea940689d0 auth_method 0 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea9359e640 1 --2- 172.21.15.86:0/436998248 >> v2:172.21.15.203:3300/0 conn(0x7fea9413aff0 0x7fea9413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:43.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea9359e640 10 monclient(hunting): get_auth_request con 0x7fea9413aff0 auth_method 0 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea9359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea9359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea9359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient(hunting): handle_auth_done global_id 144321 payload 274 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 -- 172.21.15.86:0/436998248 >> v2:172.21.15.203:3300/0 conn(0x7fea9413aff0 msgr2=0x7fea9413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 --2- 172.21.15.86:0/436998248 >> v2:172.21.15.203:3300/0 conn(0x7fea9413aff0 0x7fea9413d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 -- 172.21.15.86:0/436998248 >> v1:172.21.15.86:6789/0 conn(0x7fea94068db0 legacy=0x7fea94069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 monclient: found mon.noname-a 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:43.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.660+0000 7fea92d9d640 1 -- 172.21.15.86:0/436998248 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9409ce70 con 0x7fea940689d0 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.664+0000 7fea92d9d640 1 -- 172.21.15.86:0/436998248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea940689d0 msgr2=0x7fea9406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.664+0000 7fea92d9d640 1 --2- 172.21.15.86:0/436998248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea940689d0 0x7fea9406f840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea94069a70 tx=0x7fea8401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0dd5c1a662cff4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 1 -- 172.21.15.86:0/436998248 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea840b0070 con 0x7fea940689d0 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: dump: 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:43.669 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: _finish_auth 0 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: _check_auth_tickets 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 1 -- 172.21.15.86:0/436998248 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea840ab600 con 0x7fea940689d0 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 1 -- 172.21.15.86:0/436998248 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9409c2d0 con 0x7fea940689d0 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 5 monclient: authenticate success, global_id 144321 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 4 set_mon_vals no callback set 2024-09-17T15:41:43.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9259c640 10 monclient: dump: 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:43.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: shutdown 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- 172.21.15.86:0/436998248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea940689d0 msgr2=0x7fea9406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- 172.21.15.86:0/436998248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea940689d0 0x7fea9406f840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea94069a70 tx=0x7fea8401c300 comp rx=0 tx=0).stop 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- 172.21.15.86:0/436998248 shutdown_connections 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- 172.21.15.86:0/436998248 >> v2:172.21.15.203:3300/0 conn(0x7fea9413aff0 0x7fea9413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- 172.21.15.86:0/436998248 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea940689d0 0x7fea9406f840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- 172.21.15.86:0/436998248 >> 172.21.15.86:0/436998248 conn(0x7fea940a6440 msgr2=0x7fea940a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- 172.21.15.86:0/436998248 shutdown_connections 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- 172.21.15.86:0/436998248 wait complete. 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: build_initial_monmap 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: monmap: 2024-09-17T15:41:43.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:43.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 Processor -- start 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- start start 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: init 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea940689d0 0x7fea941cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): picked mon.noname-b con 0x7fea940689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): picked mon.noname-c con 0x7fea94068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): picked mon.noname-a con 0x7fea9413aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea941405c0 con 0x7fea9413aff0 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea94140440 con 0x7fea940689d0 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:43.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fea94140740 con 0x7fea94068db0 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea99146640 10 monclient(hunting): authenticate will time out at 1836.638428s 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59634/0 (socket says 172.21.15.86:59634) 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:33938/0 (socket says 172.21.15.86:33938) 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 -- 172.21.15.86:0/2482138580 learned_addr learned my addr 172.21.15.86:0/2482138580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): get_auth_request con 0x7fea9413aff0 auth_method 0 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 10 monclient(hunting): get_auth_request con 0x7fea94068db0 auth_method 0 2024-09-17T15:41:43.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 msgr2=0x7fea94135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 msgr2=0x7fea94135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea8a59c640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea92d9d640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea940689d0 0x7fea941cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea92d9d640 10 monclient(hunting): get_auth_request con 0x7fea940689d0 auth_method 0 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient(hunting): handle_auth_done global_id 144324 payload 824 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 msgr2=0x7fea94135ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea940689d0 msgr2=0x7fea941cfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea940689d0 0x7fea941cfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient: _finish_hunting 0 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 monclient: found mon.noname-a 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.668+0000 7fea9359e640 1 -- 172.21.15.86:0/2482138580 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea94122750 con 0x7fea9413aff0 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea9359e640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 msgr2=0x7fea941361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea9359e640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea8c007e40 tx=0x7fea8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c013790 con 0x7fea9413aff0 2024-09-17T15:41:43.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: dump: 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:43.683 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: _finish_auth 0 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c019070 con 0x7fea9413aff0 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c013b90 con 0x7fea9413aff0 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 10 monclient: dump: 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:43.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea91d9b640 4 set_mon_vals no callback set 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 5 monclient: authenticate success, global_id 144324 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _renew_subs 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea940a3850 con 0x7fea9413aff0 2024-09-17T15:41:43.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _renew_subs 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea941210d0 con 0x7fea9413aff0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea9406f840 con 0x7fea9413aff0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fea8c01e460 con 0x7fea9413aff0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 0x7fea7403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(502..502 src has 438..502) ==== 4783+0+0 (secure 0 0 0) 0x7fea8c050810 con 0x7fea9413aff0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea92d9d640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 0x7fea7403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.672+0000 7fea92d9d640 10 monclient: get_auth_request con 0x7fea7403bbe0 auth_method 0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.676+0000 7fea92d9d640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 0x7fea7403e0a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fea9406fdb0 tx=0x7fea840a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.676+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fea8c013d30 con 0x7fea9413aff0 2024-09-17T15:41:43.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.676+0000 7fea8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:43.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.676+0000 7fea8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.912+0000 7fea99146640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.912+0000 7fea99146640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.912+0000 7fea99146640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:43.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.912+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea94073620 con 0x7fea9413aff0 2024-09-17T15:41:43.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea8bfff640 1 -- 172.21.15.86:0/2482138580 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fea8c012cd0 con 0x7fea9413aff0 2024-09-17T15:41:43.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:43.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:43.916 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 msgr2=0x7fea7403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 0x7fea7403e0a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fea9406fdb0 tx=0x7fea840a6000 comp rx=0 tx=0).stop 2024-09-17T15:41:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 10 monclient: shutdown 2024-09-17T15:41:43.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 msgr2=0x7fea941361e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea8c007e40 tx=0x7fea8c00e4b0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 shutdown_connections 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fea7403bbe0 0x7fea7403e0a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea9413aff0 0x7fea941361e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea94068db0 0x7fea94135ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 --2- 172.21.15.86:0/2482138580 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea940689d0 0x7fea941cfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 >> 172.21.15.86:0/2482138580 conn(0x7fea940a6440 msgr2=0x7fea940a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 shutdown_connections 2024-09-17T15:41:43.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:43.916+0000 7fea99146640 1 -- 172.21.15.86:0/2482138580 wait complete. 2024-09-17T15:41:43.934 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:41:43.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:41:43.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:41:44.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: get_monmap_and_config 2024-09-17T15:41:44.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: build_initial_monmap 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: monmap: 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:44.056169+0000 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:44.056169+0000 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:44.052 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:44.053 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 Processor -- start 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 -- start start 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: init 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe204135fc0 0x7fe2041363a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-a con 0x7fe204135fc0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2040704c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2040a2c50 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe204073a80 con 0x7fe2040a2c50 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe204073d60 con 0x7fe2040704c0 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe2041368e0 con 0x7fe204135fc0 2024-09-17T15:41:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20bab7640 10 monclient(hunting): authenticate will time out at 1837.021663s 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe204135fc0 0x7fe2041363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20a054640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe2040a2c50 0x7fe20413dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48608/0 (socket says 172.21.15.86:48608) 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe20a054640 1 -- 172.21.15.86:0/2096657832 learned_addr learned my addr 172.21.15.86:0/2096657832 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): get_auth_request con 0x7fe204135fc0 auth_method 0 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient(hunting): handle_auth_done global_id 144327 payload 274 2024-09-17T15:41:44.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 1 -- 172.21.15.86:0/2096657832 >> v1:172.21.15.203:6789/0 conn(0x7fe2040704c0 legacy=0x7fe2040a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 1 -- 172.21.15.86:0/2096657832 >> v1:172.21.15.86:6789/0 conn(0x7fe2040a2c50 legacy=0x7fe20413dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient: _finish_hunting 0 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 1 monclient: found mon.noname-a 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.052+0000 7fe209853640 1 -- 172.21.15.86:0/2096657832 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe204074040 con 0x7fe204135fc0 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe209853640 1 -- 172.21.15.86:0/2096657832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 msgr2=0x7fe2041363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe209853640 1 --2- 172.21.15.86:0/2096657832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe2041363a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40097c0 tx=0x7fe1f401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db2e6e761e0fe99 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 1 -- 172.21.15.86:0/2096657832 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40b0070 con 0x7fe204135fc0 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: dump: 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: _finish_auth 0 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: _check_auth_tickets 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 1 -- 172.21.15.86:0/2096657832 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f40ab600 con 0x7fe204135fc0 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe20bab7640 5 monclient: authenticate success, global_id 144327 2024-09-17T15:41:44.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 1 -- 172.21.15.86:0/2096657832 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40ab9d0 con 0x7fe204135fc0 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.056+0000 7fe208851640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe208851640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe208851640 10 monclient: dump: 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:44.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 4 set_mon_vals no callback set 2024-09-17T15:41:44.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: shutdown 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- 172.21.15.86:0/2096657832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 msgr2=0x7fe2041363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 --2- 172.21.15.86:0/2096657832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe2041363a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40097c0 tx=0x7fe1f401c360 comp rx=0 tx=0).stop 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- 172.21.15.86:0/2096657832 shutdown_connections 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 --2- 172.21.15.86:0/2096657832 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe2041363a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- 172.21.15.86:0/2096657832 >> 172.21.15.86:0/2096657832 conn(0x7fe2040a3630 msgr2=0x7fe2040a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- 172.21.15.86:0/2096657832 shutdown_connections 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- 172.21.15.86:0/2096657832 wait complete. 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: build_initial_monmap 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: monmap: 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:44.068 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 Processor -- start 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- start start 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: init 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:41:44.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 0x7fe20412f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-c con 0x7fe2040a2c50 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe20412fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-a con 0x7fe204135fc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20413aea0 0x7fe204136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): picked mon.noname-b con 0x7fe20413aea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe204141db0 con 0x7fe204135fc0 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe204141c30 con 0x7fe20413aea0 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe204141f30 con 0x7fe2040a2c50 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20bab7640 10 monclient(hunting): authenticate will time out at 1837.030277s 2024-09-17T15:41:44.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe20412fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 0x7fe20412f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe20412fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59650/0 (socket says 172.21.15.86:59650) 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 -- 172.21.15.86:0/1906307349 learned_addr learned my addr 172.21.15.86:0/1906307349 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): get_auth_request con 0x7fe204135fc0 auth_method 0 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 10 monclient(hunting): get_auth_request con 0x7fe2040a2c50 auth_method 0 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 msgr2=0x7fe20412f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 msgr2=0x7fe20412f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 0x7fe20412f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 0x7fe20412f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20a054640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20413aea0 0x7fe204136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20a054640 10 monclient(hunting): get_auth_request con 0x7fe20413aea0 auth_method 0 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20a054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20a054640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe20a054640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient(hunting): handle_auth_done global_id 144330 payload 824 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 msgr2=0x7fe20412f590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:44.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe2040a2c50 0x7fe20412f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20413aea0 msgr2=0x7fe204136a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe20413aea0 0x7fe204136a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient: _finish_hunting 0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 monclient: found mon.noname-a 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.060+0000 7fe209052640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20413ee30 con 0x7fe204135fc0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209052640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 msgr2=0x7fe20412fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209052640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe20412fad0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe20000c480 tx=0x7fe20000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200012a40 con 0x7fe204135fc0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: dump: 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: _finish_auth 0 2024-09-17T15:41:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe200017070 con 0x7fe204135fc0 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200012e40 con 0x7fe204135fc0 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 10 monclient: dump: 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fbfff640 4 set_mon_vals no callback set 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 5 monclient: authenticate success, global_id 144330 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 monclient: _renew_subs 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe204139c20 con 0x7fe204135fc0 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 monclient: _renew_subs 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe204130010 con 0x7fe204135fc0 2024-09-17T15:41:44.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 0x7fe204136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe204136420 con 0x7fe2040704c0 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 0x7fe204136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient: get_auth_request con 0x7fe2040704c0 auth_method 0 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe20001c460 con 0x7fe204135fc0 2024-09-17T15:41:44.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 10 monclient(hunting): handle_auth_done global_id 144333 payload 824 2024-09-17T15:41:44.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe204070e50 con 0x7fe2040704c0 2024-09-17T15:41:44.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe1e803bbe0 0x7fe1e803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(502..502 src has 438..502) ==== 4783+0+0 (secure 0 0 0) 0x7fe20004fda0 con 0x7fe204135fc0 2024-09-17T15:41:44.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20a054640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe1e803bbe0 0x7fe1e803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.064+0000 7fe20a054640 10 monclient: get_auth_request con 0x7fe1e803bbe0 auth_method 0 2024-09-17T15:41:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 msgr2=0x7fe204136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 0x7fe204136fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40ad000 tx=0x7fe1f4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40bb070 con 0x7fe2040704c0 2024-09-17T15:41:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 10 monclient: dump: 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe1f40b7460 con 0x7fe2040704c0 2024-09-17T15:41:44.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 msgr2=0x7fe204136fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 0x7fe204136fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40ad000 tx=0x7fe1f4019100 comp rx=0 tx=0).stop 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 msgr2=0x7fe204136fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe1fa7fc640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040704c0 0x7fe204136fa0 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40ad000 tx=0x7fe1f4019100 comp rx=0 tx=0).stop 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.068+0000 7fe20a054640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe1e803bbe0 0x7fe1e803e0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc007af0 tx=0x7fe1fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe20bab7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe20bab7640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 0x7fe2041378c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe20bab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe204137e00 con 0x7fe2041374e0 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 0x7fe2041378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient: get_auth_request con 0x7fe2041374e0 auth_method 0 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.072+0000 7fe209853640 10 monclient(hunting): handle_auth_done global_id 144336 payload 824 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe20406dd50 con 0x7fe2041374e0 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe209853640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 msgr2=0x7fe2041378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe209853640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 0x7fe2041378c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1f401c3a0 tx=0x7fe1f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40cc020 con 0x7fe2041374e0 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 10 monclient: dump: 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7fe1f40b7610 con 0x7fe2041374e0 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 msgr2=0x7fe2041378c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 0x7fe2041378c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1f401c3a0 tx=0x7fe1f4008040 comp rx=0 tx=0).stop 2024-09-17T15:41:44.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 msgr2=0x7fe2041378c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 122418, 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:44.085 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:44.086 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:44.087 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:44.088 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:44.089 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:44.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:44.091 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.095 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:44.096 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.097 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:41:44.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:44.099 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:44.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe1fa7fc640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041374e0 0x7fe2041378c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe1e803bbe0 msgr2=0x7fe1e803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe1e803bbe0 0x7fe1e803e0a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc007af0 tx=0x7fe1fc009040 comp rx=0 tx=0).stop 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 10 monclient: shutdown 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 msgr2=0x7fe20412fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 --2- 172.21.15.86:0/1906307349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe204135fc0 0x7fe20412fad0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe20000c480 tx=0x7fe20000c940 comp rx=0 tx=0).stop 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20a054640 1 -- 172.21.15.86:0/1906307349 reap_dead start 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20a054640 1 -- 172.21.15.86:0/1906307349 reap_dead start 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 shutdown_connections 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 >> 172.21.15.86:0/1906307349 conn(0x7fe2040a3630 msgr2=0x7fe2041323c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 shutdown_connections 2024-09-17T15:41:44.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.076+0000 7fe20bab7640 1 -- 172.21.15.86:0/1906307349 wait complete. 2024-09-17T15:41:44.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:44.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:41:44.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2194: done (4 left) 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2195: done (3 left) 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2196: done (2 left) 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2197: done (1 left) 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2198: done (0 left) 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2199: write oid 34 current snap is 243 2024-09-17T15:41:44.130 INFO:tasks.rados.rados.0.smithi203.stdout:2199: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-17T15:41:44.140 INFO:tasks.rados.rados.0.smithi203.stdout:2199: writing smithi20328086-34 from 675846 to 1087574 tid 1 2024-09-17T15:41:44.153 INFO:tasks.rados.rados.0.smithi203.stdout:2199: writing smithi20328086-34 from 1543579 to 2331171 tid 2 2024-09-17T15:41:44.160 INFO:tasks.rados.rados.0.smithi203.stdout:2199: writing smithi20328086-34 from 2848853 to 3075846 tid 3 2024-09-17T15:41:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:2200: write (excl) oid 47 current snap is 243 2024-09-17T15:41:44.161 INFO:tasks.rados.rados.0.smithi203.stdout:2200: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-17T15:41:44.172 INFO:tasks.rados.rados.0.smithi203.stdout:2200: writing smithi20328086-47 from 692653 to 1179630 tid 1 2024-09-17T15:41:44.182 INFO:tasks.rados.rados.0.smithi203.stdout:2200: writing smithi20328086-47 from 1602061 to 2249664 tid 2 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2200: writing smithi20328086-47 from 2992629 to 3092653 tid 3 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2201: delete oid 11 current snap is 243 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2199: finishing write tid 1 to smithi20328086-34 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2199: finishing write tid 2 to smithi20328086-34 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2199: finishing write tid 3 to smithi20328086-34 2024-09-17T15:41:44.190 INFO:tasks.rados.rados.0.smithi203.stdout:2199: finishing write tid 4 to smithi20328086-34 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: finishing write tid 5 to smithi20328086-34 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: oid 34 updating version 0 to 634 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: oid 34 updating version 634 to 635 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: oid 34 version 635 is already newer than 633 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: oid 34 version 635 is already newer than 632 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 635 (ObjNum 778 snap 243 seq_num 778) dirty exists 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2199: left oid 34 (ObjNum 778 snap 243 seq_num 778) 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2200: finishing write tid 1 to smithi20328086-47 2024-09-17T15:41:44.191 INFO:tasks.rados.rados.0.smithi203.stdout:2200: finishing write tid 2 to smithi20328086-47 2024-09-17T15:41:44.193 INFO:tasks.rados.rados.0.smithi203.stdout:2200: finishing write tid 3 to smithi20328086-47 2024-09-17T15:41:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:2199: done (2 left) 2024-09-17T15:41:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:2201: done (1 left) 2024-09-17T15:41:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:2202: copy_from oid 15 from oid 1 current snap is 243 2024-09-17T15:41:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:2203: copy_from oid 2 from oid 16 current snap is 243 2024-09-17T15:41:44.194 INFO:tasks.rados.rados.0.smithi203.stdout:2204: snap_remove snap 240 2024-09-17T15:41:44.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: get_monmap_and_config 2024-09-17T15:41:44.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: build_initial_monmap 2024-09-17T15:41:44.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: monmap: 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:44.225369+0000 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:44.225369+0000 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:44.221 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:44.222 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 Processor -- start 2024-09-17T15:41:44.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 -- start start 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: init 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ed80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-b con 0x7f1ed80a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1ed813d2f0 0x7f1ed813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed80a6280 con 0x7f1ed80a50e0 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed809d2a0 con 0x7f1ed813d2f0 2024-09-17T15:41:44.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed809c0a0 con 0x7f1ed80a5bd0 2024-09-17T15:41:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edfcc3640 10 monclient(hunting): authenticate will time out at 1837.190700s 2024-09-17T15:41:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edda5f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1ed80a50e0 0x7f1ed80a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48620/0 (socket says 172.21.15.86:48620) 2024-09-17T15:41:44.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1ed813d2f0 0x7f1ed813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edda5f640 1 -- 172.21.15.86:0/1296200786 learned_addr learned my addr 172.21.15.86:0/1296200786 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 10 monclient(hunting): get_auth_request con 0x7f1ed80a5bd0 auth_method 0 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): get_auth_request con 0x7f1ed813d2f0 auth_method 0 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 1 -- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 1 -- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1edd25e640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient(hunting): handle_auth_done global_id 144339 payload 274 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 -- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed809cd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 -- 172.21.15.86:0/1296200786 >> v1:172.21.15.86:6789/0 conn(0x7f1ed80a50e0 legacy=0x7f1ed80a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient: _finish_hunting 0 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 monclient: found mon.noname-a 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.220+0000 7f1ede260640 1 -- 172.21.15.86:0/1296200786 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed809c380 con 0x7f1ed813d2f0 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1ede260640 1 -- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed813d2f0 msgr2=0x7f1ed813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1ede260640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed813d2f0 0x7f1ed813f6e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009d90 tx=0x7f1ed001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bb256efa42ad07 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 1 -- 172.21.15.86:0/1296200786 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00b2070 con 0x7f1ed813d2f0 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: dump: 2024-09-17T15:41:44.235 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: _finish_auth 0 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: _check_auth_tickets 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:44.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 1 -- 172.21.15.86:0/1296200786 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed00a5760 con 0x7f1ed813d2f0 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 1 -- 172.21.15.86:0/1296200786 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00a5b30 con 0x7f1ed813d2f0 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 5 monclient: authenticate success, global_id 144339 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 4 set_mon_vals no callback set 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edca5d640 10 monclient: dump: 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.237 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 10 monclient: shutdown 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 1 -- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed813d2f0 msgr2=0x7f1ed813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed813d2f0 0x7f1ed813f6e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009d90 tx=0x7f1ed001c800 comp rx=0 tx=0).stop 2024-09-17T15:41:44.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 1 -- 172.21.15.86:0/1296200786 shutdown_connections 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed813d2f0 0x7f1ed813f6e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.224+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/1296200786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- 172.21.15.86:0/1296200786 >> 172.21.15.86:0/1296200786 conn(0x7f1ed80aaae0 msgr2=0x7f1ed80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- 172.21.15.86:0/1296200786 shutdown_connections 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- 172.21.15.86:0/1296200786 wait complete. 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient: build_initial_monmap 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient: monmap: 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:44.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 Processor -- start 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- start start 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient: init 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-a con 0x7f1ed80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed813b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-c con 0x7f1ed80a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed813d2f0 0x7f1ed813b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): picked mon.noname-b con 0x7f1ed813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed8142650 con 0x7f1ed80a50e0 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed80a6b00 con 0x7f1ed813d2f0 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed81427d0 con 0x7f1ed80a5bd0 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:44.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edfcc3640 10 monclient(hunting): authenticate will time out at 1837.197526s 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed813b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59696/0 (socket says 172.21.15.86:59696) 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 -- 172.21.15.86:0/4201574960 learned_addr learned my addr 172.21.15.86:0/4201574960 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): get_auth_request con 0x7f1ed80a50e0 auth_method 0 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 10 monclient(hunting): get_auth_request con 0x7f1ed80a5bd0 auth_method 0 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed813b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed813b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed813b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed813b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient(hunting): handle_auth_done global_id 144342 payload 824 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 msgr2=0x7f1ed813b290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed80a5bd0 0x7f1ed813b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed813d2f0 msgr2=0x7f1ed813b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ed813d2f0 0x7f1ed813b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:44.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient: _finish_hunting 0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 monclient: found mon.noname-a 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.228+0000 7f1edda5f640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed8122780 con 0x7f1ed80a50e0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edda5f640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 msgr2=0x7f1ed8137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edda5f640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c470 tx=0x7f1ed400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4017070 con 0x7f1ed80a50e0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: dump: 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: _finish_auth 0 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:44.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed40070d0 con 0x7f1ed80a50e0 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 5 monclient: authenticate success, global_id 144342 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 monclient: _renew_subs 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed813d810 con 0x7f1ed80a50e0 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40074c0 con 0x7f1ed80a50e0 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 10 monclient: dump: 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ecf7fe640 4 set_mon_vals no callback set 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 monclient: _renew_subs 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed8121530 con 0x7f1ed80a50e0 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:44.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed806a0e0 con 0x7f1ed81384a0 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed81384a0 auth_method 0 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1ed401a070 con 0x7f1ed80a50e0 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ebc03bbb0 0x7f1ebc03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(503..503 src has 438..503) ==== 4755+0+0 (secure 0 0 0) 0x7f1ed404f4e0 con 0x7f1ed80a50e0 2024-09-17T15:41:44.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ede260640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ebc03bbb0 0x7f1ebc03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.232+0000 7f1ede260640 10 monclient: get_auth_request con 0x7f1ebc03bbb0 auth_method 0 2024-09-17T15:41:44.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.236+0000 7f1ede260640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ebc03bbb0 0x7f1ebc03e070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80aa620 tx=0x7f1ed0004ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:44.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed81384a0 auth_method 0 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.436+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed81384a0 auth_method 0 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:44.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:44.840+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:41:45.142 INFO:tasks.rados.rados.0.smithi203.stdout:2204: done (3 left) 2024-09-17T15:41:45.142 INFO:tasks.rados.rados.0.smithi203.stdout:2205: write oid 18 current snap is 243 2024-09-17T15:41:45.142 INFO:tasks.rados.rados.0.smithi203.stdout:2200: finishing write tid 4 to smithi20328086-47 2024-09-17T15:41:45.142 INFO:tasks.rados.rados.0.smithi203.stdout:2200: finishing write tid 5 to smithi20328086-47 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2200: oid 47 updating version 0 to 826 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2200: oid 47 version 826 is already newer than 825 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2200: oid 47 updating version 826 to 827 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2200: oid 47 version 827 is already newer than 824 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 827 (ObjNum 779 snap 243 seq_num 779) dirty exists 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2200: left oid 47 (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2202: finishing copy_from to smithi20328086-15 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2202: got expected ENOENT (src dne) 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2202: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2203: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2203: finishing copy_from to smithi20328086-2 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1079 (ObjNum 769 snap 240 seq_num 769) dirty exists 2024-09-17T15:41:45.143 INFO:tasks.rados.rados.0.smithi203.stdout:2205: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-17T15:41:45.155 INFO:tasks.rados.rados.0.smithi203.stdout:2205: writing smithi20328086-18 from 726267 to 1363742 tid 1 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2205: writing smithi20328086-18 from 2119025 to 2770297 tid 2 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2200: done (3 left) 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2202: done (2 left) 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2203: done (1 left) 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2206: write oid 24 current snap is 243 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2205: finishing write tid 1 to smithi20328086-18 2024-09-17T15:41:45.169 INFO:tasks.rados.rados.0.smithi203.stdout:2206: seq_num 782 ranges {743074=712724,2061154=794930} 2024-09-17T15:41:45.183 INFO:tasks.rados.rados.0.smithi203.stdout:2206: writing smithi20328086-24 from 743074 to 1455798 tid 1 2024-09-17T15:41:45.198 INFO:tasks.rados.rados.0.smithi203.stdout:2206: writing smithi20328086-24 from 2061154 to 2856084 tid 2 2024-09-17T15:41:45.198 INFO:tasks.rados.rados.0.smithi203.stdout:2207: snap_create 2024-09-17T15:41:45.198 INFO:tasks.rados.rados.0.smithi203.stdout:2205: finishing write tid 2 to smithi20328086-18 2024-09-17T15:41:45.198 INFO:tasks.rados.rados.0.smithi203.stdout:2205: finishing write tid 3 to smithi20328086-18 2024-09-17T15:41:45.198 INFO:tasks.rados.rados.0.smithi203.stdout:2205: finishing write tid 4 to smithi20328086-18 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:2205: oid 18 updating version 0 to 829 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:2205: oid 18 version 829 is already newer than 828 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:2205: oid 18 updating version 829 to 830 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 830 (ObjNum 781 snap 243 seq_num 781) dirty exists 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:2205: left oid 18 (ObjNum 781 snap 243 seq_num 781) 2024-09-17T15:41:45.199 INFO:tasks.rados.rados.0.smithi203.stdout:2206: finishing write tid 1 to smithi20328086-24 2024-09-17T15:41:45.208 INFO:tasks.rados.rados.0.smithi203.stdout:2206: finishing write tid 2 to smithi20328086-24 2024-09-17T15:41:45.208 INFO:tasks.rados.rados.0.smithi203.stdout:2206: finishing write tid 3 to smithi20328086-24 2024-09-17T15:41:45.209 INFO:tasks.rados.rados.0.smithi203.stdout:2206: finishing write tid 4 to smithi20328086-24 2024-09-17T15:41:45.209 INFO:tasks.rados.rados.0.smithi203.stdout:2206: oid 24 updating version 0 to 851 2024-09-17T15:41:45.209 INFO:tasks.rados.rados.0.smithi203.stdout:2206: oid 24 updating version 851 to 852 2024-09-17T15:41:45.209 INFO:tasks.rados.rados.0.smithi203.stdout:2206: oid 24 updating version 852 to 853 2024-09-17T15:41:45.210 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 853 (ObjNum 782 snap 243 seq_num 782) dirty exists 2024-09-17T15:41:45.210 INFO:tasks.rados.rados.0.smithi203.stdout:2206: left oid 24 (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed81384a0 auth_method 0 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:45.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:41:45.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:45.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:45.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:45.644+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:41:46.926 INFO:tasks.rados.rados.0.smithi203.stdout:2205: done (2 left) 2024-09-17T15:41:46.926 INFO:tasks.rados.rados.0.smithi203.stdout:2206: done (1 left) 2024-09-17T15:41:46.926 INFO:tasks.rados.rados.0.smithi203.stdout:2207: done (0 left) 2024-09-17T15:41:46.926 INFO:tasks.rados.rados.0.smithi203.stdout:2208: delete oid 23 current snap is 244 2024-09-17T15:41:46.937 INFO:tasks.rados.rados.0.smithi203.stdout:2208: done (0 left) 2024-09-17T15:41:46.937 INFO:tasks.rados.rados.0.smithi203.stdout:2209: read oid 1 snap -1 2024-09-17T15:41:46.938 INFO:tasks.rados.rados.0.smithi203.stdout:2209: expect deleted 2024-09-17T15:41:46.938 INFO:tasks.rados.rados.0.smithi203.stdout:2210: delete oid 19 current snap is 244 2024-09-17T15:41:46.944 INFO:tasks.rados.rados.0.smithi203.stdout:2209: done (1 left) 2024-09-17T15:41:46.945 INFO:tasks.rados.rados.0.smithi203.stdout:2210: done (0 left) 2024-09-17T15:41:46.945 INFO:tasks.rados.rados.0.smithi203.stdout:2211: snap_create 2024-09-17T15:41:47.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.232+0000 7f1ecd7fa640 10 monclient: tick 2024-09-17T15:41:47.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.232+0000 7f1ecd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.232+0000 7f1ecd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.232+0000 7f1ecd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed81384a0 auth_method 0 2024-09-17T15:41:47.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 10 monclient(hunting): handle_auth_done global_id 164099 payload 824 2024-09-17T15:41:47.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ed809cd60 con 0x7f1ed81384a0 2024-09-17T15:41:47.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec8018e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec800f040 con 0x7f1ed81384a0 2024-09-17T15:41:47.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.248+0000 7f1ece7fc640 10 monclient: dump: 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.251 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ec8022400 con 0x7f1ed81384a0 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec8018e20 comp rx=0 tx=0).stop 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 msgr2=0x7f1ed813bd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:47.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.252+0000 7f1ece7fc640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed81384a0 0x7f1ed813bd10 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec8018e20 comp rx=0 tx=0).stop 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edfcc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 0x7f1ed809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edfcc3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed813c630 con 0x7f1ed813c250 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 0x7f1ed809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient: get_auth_request con 0x7f1ed813c250 auth_method 0 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.256+0000 7f1edd25e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edd25e640 10 monclient(hunting): handle_auth_done global_id 164102 payload 824 2024-09-17T15:41:47.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ed8074be0 con 0x7f1ed813c250 2024-09-17T15:41:47.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edd25e640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 msgr2=0x7f1ed809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edd25e640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 0x7f1ed809e5e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec80046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec8028a30 con 0x7f1ed813c250 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 10 monclient: dump: 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f1ec8028a30 con 0x7f1ed813c250 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:47.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 msgr2=0x7f1ed809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 0x7f1ed809e5e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec80046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 msgr2=0x7f1ed809e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ece7fc640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1ed813c250 0x7f1ed809e5e0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1ed81223d0 tx=0x7f1ec80046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:47.264 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10447, 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:47.267 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:47.268 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.269 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:47.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.276 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:47.277 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ebc03bbb0 msgr2=0x7f1ebc03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1ebc03bbb0 0x7f1ebc03e070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1ed80aa620 tx=0x7f1ed0004ac0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 10 monclient: shutdown 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 msgr2=0x7f1ed8137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400c470 tx=0x7f1ed400c930 comp rx=0 tx=0).stop 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.260+0000 7f1ede260640 1 -- 172.21.15.86:0/4201574960 reap_dead start 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.264+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 shutdown_connections 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.264+0000 7f1edfcc3640 1 --2- 172.21.15.86:0/4201574960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1ed80a50e0 0x7f1ed8137f60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.264+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 >> 172.21.15.86:0/4201574960 conn(0x7f1ed80aaae0 msgr2=0x7f1ed80a3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.264+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 shutdown_connections 2024-09-17T15:41:47.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.264+0000 7f1edfcc3640 1 -- 172.21.15.86:0/4201574960 wait complete. 2024-09-17T15:41:47.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:47.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:41:47.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:41:47.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: get_monmap_and_config 2024-09-17T15:41:47.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: build_initial_monmap 2024-09-17T15:41:47.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: monmap: 2024-09-17T15:41:47.407 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:47.412217+0000 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:47.412217+0000 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:47.408 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 Processor -- start 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 -- start start 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: init 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc0a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7bc0a5350 0x7fc7bc0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-a con 0x7fc7bc0a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc7bc137d30 0x7fc7bc1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-c con 0x7fc7bc137d30 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc7bc09d430 con 0x7fc7bc0a5350 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc7bc0a68d0 con 0x7fc7bc137d30 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc0a64a0 con 0x7fc7bc0a4f70 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c334c640 10 monclient(hunting): authenticate will time out at 1840.377705s 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7bc0a5350 0x7fc7bc0a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7bc0a5350 0x7fc7bc0a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59714/0 (socket says 172.21.15.86:59714) 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 -- 172.21.15.86:0/4125150509 learned_addr learned my addr 172.21.15.86:0/4125150509 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a5350 auth_method 0 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a4f70 auth_method 0 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c18e9640 1 --2- 172.21.15.86:0/4125150509 >> v2:172.21.15.203:3300/0 conn(0x7fc7bc137d30 0x7fc7bc1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c18e9640 10 monclient(hunting): get_auth_request con 0x7fc7bc137d30 auth_method 0 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c18e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c18e9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c18e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_done global_id 144345 payload 274 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 -- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 msgr2=0x7fc7bc09cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc09cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 -- 172.21.15.86:0/4125150509 >> v2:172.21.15.203:3300/0 conn(0x7fc7bc137d30 msgr2=0x7fc7bc1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/4125150509 >> v2:172.21.15.203:3300/0 conn(0x7fc7bc137d30 0x7fc7bc1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient: _finish_hunting 0 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 monclient: found mon.noname-a 2024-09-17T15:41:47.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c08e7640 1 -- 172.21.15.86:0/4125150509 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc09c230 con 0x7fc7bc0a5350 2024-09-17T15:41:47.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.408+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc09cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:47.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c08e7640 1 -- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 msgr2=0x7fc7bc0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc0a5750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc09e420 tx=0x7fc7b801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacce2a677381824 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 1 -- 172.21.15.86:0/4125150509 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80b2070 con 0x7fc7bc0a5350 2024-09-17T15:41:47.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: dump: 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: _finish_auth 0 2024-09-17T15:41:47.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 1 -- 172.21.15.86:0/4125150509 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b80ad680 con 0x7fc7bc0a5350 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 1 -- 172.21.15.86:0/4125150509 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80ada50 con 0x7fc7bc0a5350 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 5 monclient: authenticate success, global_id 144345 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 4 set_mon_vals no callback set 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7b3fff640 10 monclient: dump: 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.420 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 10 monclient: shutdown 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 -- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 msgr2=0x7fc7bc0a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc0a5750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc09e420 tx=0x7fc7b801c3e0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 -- 172.21.15.86:0/4125150509 shutdown_connections 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 --2- 172.21.15.86:0/4125150509 >> v2:172.21.15.203:3300/0 conn(0x7fc7bc137d30 0x7fc7bc1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc0a5750 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 --2- 172.21.15.86:0/4125150509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc09cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 -- 172.21.15.86:0/4125150509 >> 172.21.15.86:0/4125150509 conn(0x7fc7bc0a7240 msgr2=0x7fc7bc0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 -- 172.21.15.86:0/4125150509 shutdown_connections 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.412+0000 7fc7c334c640 1 -- 172.21.15.86:0/4125150509 wait complete. 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient: build_initial_monmap 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient: monmap: 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:47.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 Processor -- start 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 -- start start 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient: init 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc1cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-c con 0x7fc7bc0a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:47.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-a con 0x7fc7bc0a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc137d30 0x7fc7bc13a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc137d30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc06f980 con 0x7fc7bc0a5350 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc06f800 con 0x7fc7bc137d30 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc1419b0 con 0x7fc7bc0a4f70 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c334c640 10 monclient(hunting): authenticate will time out at 1840.384946s 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc135120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59726/0 (socket says 172.21.15.86:59726) 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 -- 172.21.15.86:0/3994240230 learned_addr learned my addr 172.21.15.86:0/3994240230 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a5350 auth_method 0 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc1cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a4f70 auth_method 0 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c18e9640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc137d30 0x7fc7bc13a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient(hunting): handle_auth_done global_id 144348 payload 824 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 msgr2=0x7fc7bc1cffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc1cffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc137d30 msgr2=0x7fc7bc13a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc137d30 0x7fc7bc13a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient: _finish_hunting 0 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 monclient: found mon.noname-a 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc13d770 con 0x7fc7bc0a5350 2024-09-17T15:41:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c18e9640 10 monclient: get_auth_request con 0x7fc7bc137d30 auth_method 0 2024-09-17T15:41:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c18e9640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc137d30 0x7fc7bc13a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7bc0a4f70 0x7fc7bc1cffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 msgr2=0x7fc7bc135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.416+0000 7fc7c08e7640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc135120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc7b801cc80 tx=0x7fc7b801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80bd070 con 0x7fc7bc0a5350 2024-09-17T15:41:47.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:47.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b80add00 con 0x7fc7bc0a5350 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b80b2040 con 0x7fc7bc0a5350 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 5 monclient: authenticate success, global_id 144348 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 monclient: _renew_subs 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc13dcf0 con 0x7fc7bc0a5350 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b37fe640 4 set_mon_vals no callback set 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.431 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 monclient: _renew_subs 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc06efa0 con 0x7fc7bc0a5350 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 0x7fc7bc13a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc1d0ea0 con 0x7fc7bc13cab0 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc7b80c0070 con 0x7fc7bc0a5350 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 0x7fc7bc13a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc7a003bbb0 0x7fc7a003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient: get_auth_request con 0x7fc7bc13cab0 auth_method 0 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(505..505 src has 438..505) ==== 4771+0+0 (secure 0 0 0) 0x7fc7b80e9630 con 0x7fc7bc0a5350 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c18e9640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc7a003bbb0 0x7fc7a003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c18e9640 10 monclient: get_auth_request con 0x7fc7a003bbb0 auth_method 0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_done global_id 154162 payload 824 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.420+0000 7fc7c10e8640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc7bc13df30 con 0x7fc7bc13cab0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7c10e8640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 msgr2=0x7fc7bc13a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 0x7fc7bc13a9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7c18e9640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc7a003bbb0 0x7fc7a003e070 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009ef0 tx=0x7fc7b400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac019070 con 0x7fc7bc13cab0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc7ac00fe70 con 0x7fc7bc13cab0 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 msgr2=0x7fc7bc13a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 0x7fc7bc13a9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac00f610 comp rx=0 tx=0).stop 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 msgr2=0x7fc7bc13a9d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:47.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.424+0000 7fc7b1ffb640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc13cab0 0x7fc7bc13a9d0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac00f610 comp rx=0 tx=0).stop 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c334c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c334c640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 0x7fc7bc13af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c334c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc0a5750 con 0x7fc7bc135660 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 0x7fc7bc13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient: get_auth_request con 0x7fc7bc135660 auth_method 0 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 10 monclient(hunting): handle_auth_done global_id 154165 payload 824 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.432+0000 7fc7c10e8640 1 -- 172.21.15.86:0/3994240230 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc7bc0a5ff0 con 0x7fc7bc135660 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c10e8640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 msgr2=0x7fc7bc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c10e8640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 0x7fc7bc13af10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac019040 con 0x7fc7bc135660 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fc7ac01e870 con 0x7fc7bc135660 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 msgr2=0x7fc7bc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 0x7fc7bc13af10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac0046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 msgr2=0x7fc7bc13af10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7b1ffb640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7bc135660 0x7fc7bc13af10 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac011cf0 tx=0x7fc7ac0046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:47.440 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 58806, 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.441 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.442 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:47.443 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:47.444 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.445 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.446 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.447 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.448 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:47.449 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:47.451 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:47.452 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:47.453 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:47.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc7a003bbb0 msgr2=0x7fc7a003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc7a003bbb0 0x7fc7a003e070 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009ef0 tx=0x7fc7b400b040 comp rx=0 tx=0).stop 2024-09-17T15:41:47.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 10 monclient: shutdown 2024-09-17T15:41:47.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:47.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 msgr2=0x7fc7bc135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 --2- 172.21.15.86:0/3994240230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7bc0a5350 0x7fc7bc135120 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc7b801cc80 tx=0x7fc7b801ac20 comp rx=0 tx=0).stop 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c18e9640 1 -- 172.21.15.86:0/3994240230 reap_dead start 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c18e9640 1 -- 172.21.15.86:0/3994240230 reap_dead start 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 shutdown_connections 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 >> 172.21.15.86:0/3994240230 conn(0x7fc7bc0a7240 msgr2=0x7fc7bc0a9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 shutdown_connections 2024-09-17T15:41:47.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.436+0000 7fc7c334c640 1 -- 172.21.15.86:0/3994240230 wait complete. 2024-09-17T15:41:47.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:47.455 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:41:47.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 10 monclient: get_monmap_and_config 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 10 monclient: build_initial_monmap 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 10 monclient: monmap: 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:47.575065+0000 2024-09-17T15:41:47.570 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:47.575065+0000 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:47.571 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 1 Processor -- start 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 1 -- start start 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.568+0000 7f73fa670640 10 monclient: init 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73f40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f73f40694c0 0x7f73f406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-c con 0x7f73f40694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73f413aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f409c8b0 con 0x7f73f40689d0 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f409cb90 con 0x7f73f413aff0 2024-09-17T15:41:47.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f73f4074500 con 0x7f73f40694c0 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 10 monclient(hunting): authenticate will time out at 1840.540587s 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f3fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f73f40689d0 0x7f73f4068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48622/0 (socket says 172.21.15.86:48622) 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f3fff640 1 -- 172.21.15.86:0/1836560667 learned_addr learned my addr 172.21.15.86:0/1836560667 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 --2- 172.21.15.86:0/1836560667 >> v2:172.21.15.203:3300/0 conn(0x7f73f40694c0 0x7f73f406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): get_auth_request con 0x7f73f40694c0 auth_method 0 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f3fff640 1 -- 172.21.15.86:0/1836560667 >> v1:172.21.15.86:6789/0 conn(0x7f73f40689d0 legacy=0x7f73f4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228760829 0 0) 0x7f73f409c8b0 con 0x7f73f40689d0 2024-09-17T15:41:47.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): my global_id is 144351 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e0003530 con 0x7f73f40689d0 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3664290314 0 0) 0x7f73e0003530 con 0x7f73f40689d0 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 >> v2:172.21.15.203:3300/0 conn(0x7f73f40694c0 msgr2=0x7f73f406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 --2- 172.21.15.86:0/1836560667 >> v2:172.21.15.203:3300/0 conn(0x7f73f40694c0 0x7f73f406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 >> v1:172.21.15.203:6789/0 conn(0x7f73f413aff0 legacy=0x7f73f413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:41:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f409ce70 con 0x7f73f40689d0 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f73e40019a0 con 0x7f73f40689d0 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: dump: 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.579 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 >> v1:172.21.15.86:6789/0 conn(0x7f73f40689d0 legacy=0x7f73f4068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73e0003ad0 0x7f73e0003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): picked mon.b con 0x7f73e0003ad0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 0x7f73e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): picked mon.a con 0x7f73e0004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e00053d0 0x7f73e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): picked mon.c con 0x7f73e00053d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73e000c260 con 0x7f73e0004810 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73e000c410 con 0x7f73e0003ad0 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73e000c610 con 0x7f73e00053d0 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 0x7f73e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): get_auth_request con 0x7f73e0004810 auth_method 0 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e00053d0 0x7f73e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): get_auth_request con 0x7f73e00053d0 auth_method 0 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:41:47.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient(hunting): handle_auth_done global_id 144351 payload 293 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 -- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e00053d0 msgr2=0x7f73e000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e00053d0 0x7f73e000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 -- 172.21.15.86:0/1836560667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73e0003ad0 msgr2=0x7f73e0003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73e0003ad0 0x7f73e0003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 monclient: found mon.a 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 1 -- 172.21.15.86:0/1836560667 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e0003530 con 0x7f73e0004810 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient: _finish_auth 0 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73f8c0d640 1 -- 172.21.15.86:0/1836560667 reap_dead start 2024-09-17T15:41:47.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 5 monclient: authenticate success, global_id 144351 2024-09-17T15:41:47.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.572+0000 7f73fa670640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.576+0000 7f73f37fe640 1 -- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 msgr2=0x7f73e0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 0x7f73e0004c90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ae0 tx=0x7f73dc01ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b93a5ac86768de5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0b2070 con 0x7f73e0004810 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: dump: 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73dc0a5540 con 0x7f73e0004810 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 1 -- 172.21.15.86:0/1836560667 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0096b0 con 0x7f73e0004810 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 4 set_mon_vals no callback set 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:47.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f2ffd640 10 monclient: dump: 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: shutdown 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 msgr2=0x7f73e0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 0x7f73e0004c90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009ae0 tx=0x7f73dc01ca00 comp rx=0 tx=0).stop 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- 172.21.15.86:0/1836560667 shutdown_connections 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 --2- 172.21.15.86:0/1836560667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73e0004810 0x7f73e0004c90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- 172.21.15.86:0/1836560667 >> 172.21.15.86:0/1836560667 conn(0x7f73f40a6440 msgr2=0x7f73f40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- 172.21.15.86:0/1836560667 shutdown_connections 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- 172.21.15.86:0/1836560667 wait complete. 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: build_initial_monmap 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: monmap: 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:47.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 Processor -- start 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- start start 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: init 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-c con 0x7f73e0004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:47.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f41d0cd0 0x7f73f409d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-b con 0x7f73f41d0cd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): picked mon.noname-a con 0x7f73f409dce0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73f41362f0 con 0x7f73f409dce0 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73f409c2d0 con 0x7f73f41d0cd0 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73f4136470 con 0x7f73e0004810 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73fa670640 10 monclient(hunting): authenticate will time out at 1840.550419s 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59740/0 (socket says 172.21.15.86:59740) 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:34076/0 (socket says 172.21.15.86:34076) 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 -- 172.21.15.86:0/79459585 learned_addr learned my addr 172.21.15.86:0/79459585 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): get_auth_request con 0x7f73f409dce0 auth_method 0 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): get_auth_request con 0x7f73e0004810 auth_method 0 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f41d0cd0 0x7f73f409d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 10 monclient(hunting): get_auth_request con 0x7f73f41d0cd0 auth_method 0 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient(hunting): handle_auth_done global_id 144354 payload 824 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 -- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 msgr2=0x7f73f41d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 -- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f41d0cd0 msgr2=0x7f73f409d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f41d0cd0 0x7f73f409d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient: _finish_hunting 0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 monclient: found mon.noname-a 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f8c0d640 1 -- 172.21.15.86:0/79459585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f4126d90 con 0x7f73f409dce0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.580+0000 7f73f3fff640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f8c0d640 1 -- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 msgr2=0x7f73f409e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f8c0d640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f73f40a4140 tx=0x7f73ec00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec017020 con 0x7f73f409dce0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: dump: 2024-09-17T15:41:47.597 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: _finish_auth 0 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ec00daa0 con 0x7f73f409dce0 2024-09-17T15:41:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ec00de70 con 0x7f73f409dce0 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 5 monclient: authenticate success, global_id 144354 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _renew_subs 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f4127070 con 0x7f73f409dce0 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f27fc640 4 set_mon_vals no callback set 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 10 monclient: dump: 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.599 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _renew_subs 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f40a25f0 con 0x7f73f409dce0 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f41368a0 con 0x7f73f409dce0 2024-09-17T15:41:47.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f73ec01a020 con 0x7f73f409dce0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 0x7f73d403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(505..505 src has 438..505) ==== 4771+0+0 (secure 0 0 0) 0x7f73ec04ff20 con 0x7f73f409dce0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f3fff640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 0x7f73d403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.584+0000 7f73f3fff640 10 monclient: get_auth_request con 0x7f73d403bbb0 auth_method 0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.588+0000 7f73f3fff640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 0x7f73d403e070 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f73e4007fd0 tx=0x7f73e4001920 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.588+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f73ec021020 con 0x7f73f409dce0 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.588+0000 7f73f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:47.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.588+0000 7f73f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:47.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73fa670640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:41:47.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73fa670640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:41:47.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73fa670640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:47.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f73f411dfd0 con 0x7f73f409dce0 2024-09-17T15:41:47.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73f0ff9640 1 -- 172.21.15.86:0/79459585 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f73ec01cda0 con 0x7f73f409dce0 2024-09-17T15:41:47.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:41:47.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.820+0000 7f73f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1089084344,config=3541143443,health=123281230,kv=3885551602,logm=3408599856}) 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=3471013289,config=3541143443,health=416335068,kv=3591778543,logm=3329210340}) 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1089084344,config=3541143443,health=123281230,kv=3885551602,logm=3408599856}) 2024-09-17T15:41:47.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=1943113799,config=3541143443,health=2858219974,kv=2732825231,logm=158135755}) 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 msgr2=0x7f73d403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 0x7f73d403e070 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f73e4007fd0 tx=0x7f73e4001920 comp rx=0 tx=0).stop 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 10 monclient: shutdown 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 msgr2=0x7f73f409e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f73f40a4140 tx=0x7f73ec00fc90 comp rx=0 tx=0).stop 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 shutdown_connections 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f73d403bbb0 0x7f73d403e070 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73f409dce0 0x7f73f409e170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73f41d0cd0 0x7f73f409d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 --2- 172.21.15.86:0/79459585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73e0004810 0x7f73f41d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:47.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 >> 172.21.15.86:0/79459585 conn(0x7f73f40a6440 msgr2=0x7f73f4073690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:47.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 shutdown_connections 2024-09-17T15:41:47.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f73fa670640 1 -- 172.21.15.86:0/79459585 wait complete. 2024-09-17T15:41:47.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277431332}) 2024-09-17T15:41:47.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1436658957}) 2024-09-17T15:41:47.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2277431332}) 2024-09-17T15:41:47.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.824+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778276131}) 2024-09-17T15:41:47.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:41:47.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187890018}) 2024-09-17T15:41:47.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693313418}) 2024-09-17T15:41:47.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187890018}) 2024-09-17T15:41:47.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.920+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117834164}) 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066149371}) 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937068303}) 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066149371}) 2024-09-17T15:41:47.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.928+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2887471930}) 2024-09-17T15:41:47.940 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.940 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203040692}) 2024-09-17T15:41:47.940 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2272298850}) 2024-09-17T15:41:47.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:47.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203040692}) 2024-09-17T15:41:47.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:47.940+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290871133}) 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2211: done (0 left) 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2212: copy_from oid 9 from oid 5 current snap is 245 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2213: rollback oid 14 current snap is 245 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 242 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2214: read oid 19 snap -1 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2214: expect deleted 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2215: write oid 36 current snap is 245 2024-09-17T15:41:47.944 INFO:tasks.rados.rados.0.smithi203.stdout:2215: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-17T15:41:47.968 INFO:tasks.rados.rados.0.smithi203.stdout:2215: writing smithi20328086-36 from 793495 to 1448319 tid 1 2024-09-17T15:41:47.987 INFO:tasks.rados.rados.0.smithi203.stdout:2215: writing smithi20328086-36 from 1952953 to 2611563 tid 2 2024-09-17T15:41:47.996 INFO:tasks.rados.rados.0.smithi203.stdout:2215: writing smithi20328086-36 from 3157167 to 3193495 tid 3 2024-09-17T15:41:47.996 INFO:tasks.rados.rados.0.smithi203.stdout:2216: write (excl) oid 21 current snap is 245 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:2212: finishing copy_from to smithi20328086-9 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:2212: got expected ENOENT (src dne) 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:2212: finishing copy_from racing read to smithi20328086-9 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:2213: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 0 (ObjNum 33 snap 0 seq_num 3605768656) dirty dne 2024-09-17T15:41:47.997 INFO:tasks.rados.rados.0.smithi203.stdout:2215: finishing write tid 1 to smithi20328086-36 2024-09-17T15:41:47.998 INFO:tasks.rados.rados.0.smithi203.stdout:2215: finishing write tid 2 to smithi20328086-36 2024-09-17T15:41:47.998 INFO:tasks.rados.rados.0.smithi203.stdout:2216: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-17T15:41:48.013 INFO:tasks.rados.rados.0.smithi203.stdout:2216: writing smithi20328086-21 from 410302 to 1140375 tid 1 2024-09-17T15:41:48.028 INFO:tasks.rados.rados.0.smithi203.stdout:2216: writing smithi20328086-21 from 1895082 to 2297350 tid 2 2024-09-17T15:41:48.039 INFO:tasks.rados.rados.0.smithi203.stdout:2216: writing smithi20328086-21 from 2951884 to 3210302 tid 3 2024-09-17T15:41:48.039 INFO:tasks.rados.rados.0.smithi203.stdout:2212: done (4 left) 2024-09-17T15:41:48.039 INFO:tasks.rados.rados.0.smithi203.stdout:2213: done (3 left) 2024-09-17T15:41:48.039 INFO:tasks.rados.rados.0.smithi203.stdout:2214: done (2 left) 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2217: read oid 19 snap 242 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2217: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2218: snap_create 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: finishing write tid 3 to smithi20328086-36 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: finishing write tid 4 to smithi20328086-36 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: finishing write tid 5 to smithi20328086-36 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: oid 36 updating version 0 to 1151 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: oid 36 updating version 1151 to 1153 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: oid 36 version 1153 is already newer than 1152 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: oid 36 version 1153 is already newer than 1150 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1153 (ObjNum 785 snap 245 seq_num 785) dirty exists 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2215: left oid 36 (ObjNum 785 snap 245 seq_num 785) 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2216: finishing write tid 1 to smithi20328086-21 2024-09-17T15:41:48.040 INFO:tasks.rados.rados.0.smithi203.stdout:2216: finishing write tid 2 to smithi20328086-21 2024-09-17T15:41:48.045 INFO:tasks.rados.rados.0.smithi203.stdout:2216: finishing write tid 3 to smithi20328086-21 2024-09-17T15:41:48.046 INFO:tasks.rados.rados.0.smithi203.stdout:2216: finishing write tid 4 to smithi20328086-21 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: finishing write tid 5 to smithi20328086-21 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: oid 21 updating version 0 to 1155 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: oid 21 updating version 1155 to 1158 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: oid 21 version 1158 is already newer than 1156 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: oid 21 version 1158 is already newer than 1157 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1158 (ObjNum 786 snap 245 seq_num 786) dirty exists 2024-09-17T15:41:48.047 INFO:tasks.rados.rados.0.smithi203.stdout:2216: left oid 21 (ObjNum 786 snap 245 seq_num 786) 2024-09-17T15:41:48.050 INFO:tasks.rados.rados.0.smithi203.stdout:2217: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:41:48.112 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714767051}) 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1086795083}) 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714767051}) 2024-09-17T15:41:48.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.148+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94385095}) 2024-09-17T15:41:48.157 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1262359 2024-09-17T15:41:48.619 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2362400506}) 2024-09-17T15:41:48.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=582396956}) 2024-09-17T15:41:48.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2362400506}) 2024-09-17T15:41:48.620 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4270740625}) 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863432071}) 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149679930}) 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863432071}) 2024-09-17T15:41:48.622 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.620+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222822570}) 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759949077}) 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4003800914}) 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759949077}) 2024-09-17T15:41:48.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2605540810}) 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151335037}) 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796806690}) 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151335037}) 2024-09-17T15:41:48.626 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.624+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=808950751}) 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984481463}) 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2183001324}) 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=984481463}) 2024-09-17T15:41:48.629 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879038040}) 2024-09-17T15:41:48.630 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353006189}) 2024-09-17T15:41:48.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3408852360}) 2024-09-17T15:41:48.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353006189}) 2024-09-17T15:41:48.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.628+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770594391}) 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465697421}) 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782050999}) 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465697421}) 2024-09-17T15:41:48.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969642588}) 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245484637}) 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3511146910}) 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245484637}) 2024-09-17T15:41:48.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.632+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798571442}) 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653627514}) 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4141610308}) 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653627514}) 2024-09-17T15:41:48.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.636+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1695618703}) 2024-09-17T15:41:48.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130865542}) 2024-09-17T15:41:48.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170402046}) 2024-09-17T15:41:48.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130865542}) 2024-09-17T15:41:48.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225517219}) 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563936576}) 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3578955486}) 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563936576}) 2024-09-17T15:41:48.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.640+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252524478}) 2024-09-17T15:41:48.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426527011}) 2024-09-17T15:41:48.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=861050334}) 2024-09-17T15:41:48.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426527011}) 2024-09-17T15:41:48.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1383334294}) 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532806118}) 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2325982217}) 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1532806118}) 2024-09-17T15:41:48.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.644+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298206431}) 2024-09-17T15:41:48.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346742381}) 2024-09-17T15:41:48.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808757638}) 2024-09-17T15:41:48.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346742381}) 2024-09-17T15:41:48.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.648+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865645931}) 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736892511}) 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992543277}) 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736892511}) 2024-09-17T15:41:48.652 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950024585}) 2024-09-17T15:41:48.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1681961282}) 2024-09-17T15:41:48.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054424087}) 2024-09-17T15:41:48.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1681961282}) 2024-09-17T15:41:48.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.652+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447078739}) 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191292218}) 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205011272}) 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1191292218}) 2024-09-17T15:41:48.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.656+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=924276755}) 2024-09-17T15:41:48.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885390572}) 2024-09-17T15:41:48.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656313840}) 2024-09-17T15:41:48.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885390572}) 2024-09-17T15:41:48.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2618127699}) 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850969661}) 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588140423}) 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850969661}) 2024-09-17T15:41:48.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.660+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2937642030}) 2024-09-17T15:41:48.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005106351}) 2024-09-17T15:41:48.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2872080906}) 2024-09-17T15:41:48.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005106351}) 2024-09-17T15:41:48.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959361071}) 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51346917}) 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960160253}) 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.664+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51346917}) 2024-09-17T15:41:48.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145906785}) 2024-09-17T15:41:48.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187951088}) 2024-09-17T15:41:48.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=815013229}) 2024-09-17T15:41:48.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187951088}) 2024-09-17T15:41:48.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.668+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768245987}) 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216178215}) 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1870457808}) 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216178215}) 2024-09-17T15:41:48.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907248899}) 2024-09-17T15:41:48.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3309765179}) 2024-09-17T15:41:48.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3597022553}) 2024-09-17T15:41:48.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3309765179}) 2024-09-17T15:41:48.675 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.672+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903637644}) 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440733328}) 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=440054354}) 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1440733328}) 2024-09-17T15:41:48.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714186018}) 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.676+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319315608}) 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211565604}) 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3319315608}) 2024-09-17T15:41:48.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1621411852}) 2024-09-17T15:41:48.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195942914}) 2024-09-17T15:41:48.682 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2535569265}) 2024-09-17T15:41:48.682 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.682 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195942914}) 2024-09-17T15:41:48.682 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.680+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4163391455}) 2024-09-17T15:41:48.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763418114}) 2024-09-17T15:41:48.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627562444}) 2024-09-17T15:41:48.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763418114}) 2024-09-17T15:41:48.685 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2431086263}) 2024-09-17T15:41:48.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379183626}) 2024-09-17T15:41:48.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426415232}) 2024-09-17T15:41:48.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379183626}) 2024-09-17T15:41:48.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.684+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840043979}) 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476396823}) 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968327588}) 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476396823}) 2024-09-17T15:41:48.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.688+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908263270}) 2024-09-17T15:41:48.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481054391}) 2024-09-17T15:41:48.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=327669989}) 2024-09-17T15:41:48.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481054391}) 2024-09-17T15:41:48.692 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660447599}) 2024-09-17T15:41:48.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344111430}) 2024-09-17T15:41:48.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962812011}) 2024-09-17T15:41:48.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344111430}) 2024-09-17T15:41:48.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.692+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316740673}) 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276294983}) 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363725708}) 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276294983}) 2024-09-17T15:41:48.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2334512171}) 2024-09-17T15:41:48.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036818564}) 2024-09-17T15:41:48.698 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615139452}) 2024-09-17T15:41:48.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036818564}) 2024-09-17T15:41:48.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.696+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246759681}) 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071240037}) 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1200159617}) 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071240037}) 2024-09-17T15:41:48.701 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.700+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051600690}) 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128701320}) 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2242890661}) 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128701320}) 2024-09-17T15:41:48.703 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829768473}) 2024-09-17T15:41:48.705 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711051540}) 2024-09-17T15:41:48.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517751613}) 2024-09-17T15:41:48.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711051540}) 2024-09-17T15:41:48.706 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.704+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3754675004}) 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320231945}) 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2202206554}) 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320231945}) 2024-09-17T15:41:48.708 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857531863}) 2024-09-17T15:41:48.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407380481}) 2024-09-17T15:41:48.710 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3644052005}) 2024-09-17T15:41:48.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407380481}) 2024-09-17T15:41:48.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.708+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353829646}) 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=3345503837,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1347727015}) 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2655924053,mdsmap=2815901553,mgr=642532334,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1347727015}) 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=3345503837,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1347727015}) 2024-09-17T15:41:48.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.712+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11} crc {logm=3011248444,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4094336767}) 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mgrstat=2282370,mon_config_key=639531838,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4199293138,osd_snap=1968779867}) 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=75} crc {mon_config_key=2837151273,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_snap=653014410}) 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mgrstat=2282370,mon_config_key=639531838,monmap=2119989493,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4199293138,osd_snap=1968779867}) 2024-09-17T15:41:48.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mgrstat=2282370,mon_config_key=639531838,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4199293138,osd_snap=434142342}) 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2447863094,osdmap=655691452}) 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=2193935332,osdmap=794914551}) 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.716+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2447863094,osdmap=655691452}) 2024-09-17T15:41:48.719 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.720+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=20,osdmap=80} crc {osd_snap=2447863094,osdmap=1641855863}) 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=984524439}) 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=36} crc {osdmap=927327143}) 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=984524439}) 2024-09-17T15:41:48.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:41:48.728+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=2861722404}) 2024-09-17T15:41:48.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:41:48.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:2215: done (3 left) 2024-09-17T15:41:48.944 INFO:tasks.rados.rados.0.smithi203.stdout:2216: done (2 left) 2024-09-17T15:41:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:2217: done (1 left) 2024-09-17T15:41:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:2218: done (0 left) 2024-09-17T15:41:48.945 INFO:tasks.rados.rados.0.smithi203.stdout:2219: snap_remove snap 235 2024-09-17T15:41:48.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: get_monmap_and_config 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: build_initial_monmap 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: monmap: 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:48.977669+0000 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:48.977669+0000 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.973 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:48.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:48.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:48.974 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:48.974 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:48.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 Processor -- start 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 -- start start 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: init 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f500a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-b con 0x7f0f500a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0f5013d2f0 0x7f0f5013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-c con 0x7f0f5013d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f500a6530 con 0x7f0f500a50e0 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0f5009c150 con 0x7f0f5013d2f0 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0f5009c430 con 0x7f0f500a54c0 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f5495c640 10 monclient(hunting): authenticate will time out at 1841.943212s 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52842/0 (socket says 172.21.15.86:52842) 2024-09-17T15:41:48.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1958433611 learned_addr learned my addr 172.21.15.86:0/1958433611 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): get_auth_request con 0x7f0f500a54c0 auth_method 0 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4ed9d640 1 --2- 172.21.15.86:0/1958433611 >> v2:172.21.15.203:3300/0 conn(0x7f0f5013d2f0 0x7f0f5013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_done global_id 164114 payload 274 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1958433611 >> v2:172.21.15.203:3300/0 conn(0x7f0f5013d2f0 msgr2=0x7f0f5013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 --2- 172.21.15.86:0/1958433611 >> v2:172.21.15.203:3300/0 conn(0x7f0f5013d2f0 0x7f0f5013f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:48.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1958433611 >> v1:172.21.15.203:6789/0 conn(0x7f0f500a50e0 legacy=0x7f0f5009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 monclient: found mon.noname-b 2024-09-17T15:41:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:48.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.972+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1958433611 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f500a6940 con 0x7f0f500a54c0 2024-09-17T15:41:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.976+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1958433611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 msgr2=0x7f0f500a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.976+0000 7f0f4dd9b640 1 --2- 172.21.15.86:0/1958433611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009870 tx=0x7f0f4401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e54beee42335773e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 1 -- 172.21.15.86:0/1958433611 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440b1070 con 0x7f0f500a54c0 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: dump: 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: _finish_auth 0 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 1 -- 172.21.15.86:0/1958433611 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f440ac620 con 0x7f0f500a54c0 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 5 monclient: authenticate success, global_id 164114 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 1 -- 172.21.15.86:0/1958433611 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440ac9f0 con 0x7f0f500a54c0 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 4 set_mon_vals no callback set 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:48.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4d59a640 10 monclient: dump: 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: shutdown 2024-09-17T15:41:48.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- 172.21.15.86:0/1958433611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 msgr2=0x7f0f500a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1958433611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009870 tx=0x7f0f4401c420 comp rx=0 tx=0).stop 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- 172.21.15.86:0/1958433611 shutdown_connections 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1958433611 >> v2:172.21.15.203:3300/0 conn(0x7f0f5013d2f0 0x7f0f5013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1958433611 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f500a58d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- 172.21.15.86:0/1958433611 >> 172.21.15.86:0/1958433611 conn(0x7f0f500aaae0 msgr2=0x7f0f500aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- 172.21.15.86:0/1958433611 shutdown_connections 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- 172.21.15.86:0/1958433611 wait complete. 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: build_initial_monmap 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: monmap: 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 Processor -- start 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- start start 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: init 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:41:48.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f500a50e0 0x7f0f501cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-b con 0x7f0f500a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-c con 0x7f0f500a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5013d2f0 0x7f0f500a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0f501427d0 con 0x7f0f5013d2f0 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f50142650 con 0x7f0f500a50e0 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:48.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0f50142950 con 0x7f0f500a54c0 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f5495c640 10 monclient(hunting): authenticate will time out at 1841.950394s 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5013d2f0 0x7f0f500a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52846/0 (socket says 172.21.15.86:52846) 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1843052514 learned_addr learned my addr 172.21.15.86:0/1843052514 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): get_auth_request con 0x7f0f500a54c0 auth_method 0 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:48.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): get_auth_request con 0x7f0f5013d2f0 auth_method 0 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4e59c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f500a50e0 0x7f0f501cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:48.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4e59c640 10 monclient(hunting): get_auth_request con 0x7f0f500a50e0 auth_method 0 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient(hunting): handle_auth_done global_id 164117 payload 824 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f500a50e0 msgr2=0x7f0f501cfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f500a50e0 0x7f0f501cfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5013d2f0 msgr2=0x7f0f500a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5013d2f0 0x7f0f500a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:41:48.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 monclient: found mon.noname-c 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.980+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1843052514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f500a4740 con 0x7f0f500a54c0 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f4dd9b640 1 -- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 msgr2=0x7f0f50138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f4dd9b640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0f4401c250 tx=0x7f0f4401ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440bb020 con 0x7f0f500a54c0 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: dump: 2024-09-17T15:41:48.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: _finish_auth 0 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:48.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 5 monclient: authenticate success, global_id 164117 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f440b7420 con 0x7f0f500a54c0 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _renew_subs 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f440b1040 con 0x7f0f500a54c0 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f501228d0 con 0x7f0f500a54c0 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f4cd99640 4 set_mon_vals no callback set 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 10 monclient: dump: 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:48.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _renew_subs 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f5013d8b0 con 0x7f0f500a54c0 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:48.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f500a58d0 con 0x7f0f500a54c0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0f440be020 con 0x7f0f500a54c0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 0x7f0f3003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7f0f440ea230 con 0x7f0f500a54c0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f4e59c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 0x7f0f3003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.984+0000 7f0f4e59c640 10 monclient: get_auth_request con 0x7f0f3003bbb0 auth_method 0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.988+0000 7f0f4e59c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 0x7f0f3003e070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0f400016c0 tx=0x7f0f40009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.988+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0f440c5030 con 0x7f0f500a54c0 2024-09-17T15:41:49.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.988+0000 7f0f3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:48.988+0000 7f0f3effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:49.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:49.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:49.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f501213b0 con 0x7f0f500a54c0 2024-09-17T15:41:49.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f3effd640 1 -- 172.21.15.86:0/1843052514 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0f44016130 con 0x7f0f500a54c0 2024-09-17T15:41:49.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:49.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f3effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:49.225 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 msgr2=0x7f0f3003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 0x7f0f3003e070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0f400016c0 tx=0x7f0f40009040 comp rx=0 tx=0).stop 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 10 monclient: shutdown 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 msgr2=0x7f0f50138200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0f4401c250 tx=0x7f0f4401ac60 comp rx=0 tx=0).stop 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 shutdown_connections 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f3003bbb0 0x7f0f3003e070 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0f400016c0 tx=0x7f0f40009040 comp rx=0 tx=0).stop 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5013d2f0 0x7f0f500a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f500a54c0 0x7f0f50138200 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 --2- 172.21.15.86:0/1843052514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f500a50e0 0x7f0f501cfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.224+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 >> 172.21.15.86:0/1843052514 conn(0x7f0f500aaae0 msgr2=0x7f0f500a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.228+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 shutdown_connections 2024-09-17T15:41:49.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.228+0000 7f0f5495c640 1 -- 172.21.15.86:0/1843052514 wait complete. 2024-09-17T15:41:49.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:41:49.243 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:41:49.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:41:49.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:41:49.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: get_monmap_and_config 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: monmap: 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:49.364519+0000 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:49.364519+0000 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:49.361 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:49.361 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:49.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 Processor -- start 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 -- start start 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: init 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-17T15:41:49.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9809d770 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2e9809e090 0x7f2e98134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9809e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 0x7f2e98135410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-b con 0x7f2e98134f80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2e9809e7f0 con 0x7f2e9809d770 2024-09-17T15:41:49.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2e980a2110 con 0x7f2e9809e090 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9809ff00 con 0x7f2e98134f80 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9e34a640 10 monclient(hunting): authenticate will time out at 1842.330021s 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59112/0 (socket says 172.21.15.86:59112) 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 1 -- 172.21.15.86:0/4259129399 learned_addr learned my addr 172.21.15.86:0/4259129399 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): get_auth_request con 0x7f2e9809d770 auth_method 0 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 0x7f2e98135410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): get_auth_request con 0x7f2e98134f80 auth_method 0 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e977fe640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.203:3300/0 conn(0x7f2e9809e090 0x7f2e98134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e977fe640 10 monclient(hunting): get_auth_request con 0x7f2e9809e090 auth_method 0 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient(hunting): handle_auth_done global_id 164120 payload 274 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/4259129399 >> v2:172.21.15.203:3300/0 conn(0x7f2e9809e090 msgr2=0x7f2e98134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.203:3300/0 conn(0x7f2e9809e090 0x7f2e98134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/4259129399 >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 msgr2=0x7f2e9809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 monclient: found mon.noname-b 2024-09-17T15:41:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/4259129399 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9809ec30 con 0x7f2e98134f80 2024-09-17T15:41:49.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.360+0000 7f2e97fff640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 msgr2=0x7f2e98135410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 0x7f2e98135410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e98074de0 tx=0x7f2e9001c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4612c2e95b2ae0d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 1 -- 172.21.15.86:0/4259129399 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900a6700 con 0x7f2e98134f80 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: dump: 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 1 -- 172.21.15.86:0/4259129399 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e900a68a0 con 0x7f2e98134f80 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 1 -- 172.21.15.86:0/4259129399 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900a6c70 con 0x7f2e98134f80 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 5 monclient: authenticate success, global_id 164120 2024-09-17T15:41:49.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 4 set_mon_vals no callback set 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e96ffd640 10 monclient: dump: 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 10 monclient: shutdown 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 -- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 msgr2=0x7f2e98135410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 0x7f2e98135410 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e98074de0 tx=0x7f2e9001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 -- 172.21.15.86:0/4259129399 shutdown_connections 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/4259129399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e98134f80 0x7f2e98135410 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.203:3300/0 conn(0x7f2e9809e090 0x7f2e98134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/4259129399 >> v2:172.21.15.86:3300/0 conn(0x7f2e9809d770 0x7f2e9809db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 -- 172.21.15.86:0/4259129399 >> 172.21.15.86:0/4259129399 conn(0x7f2e980a2f40 msgr2=0x7f2e980a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 -- 172.21.15.86:0/4259129399 shutdown_connections 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.364+0000 7f2e9e34a640 1 -- 172.21.15.86:0/4259129399 wait complete. 2024-09-17T15:41:49.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient: monmap: 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 Processor -- start 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 -- start start 2024-09-17T15:41:49.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient: init 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9809d770 0x7f2e98120cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9809d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 0x7f2e98121210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9809e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98134f80 0x7f2e98121750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): picked mon.noname-b con 0x7f2e98134f80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9809ce60 con 0x7f2e9809d770 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9809cce0 con 0x7f2e98134f80 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e980a1480 con 0x7f2e9809e090 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9e34a640 10 monclient(hunting): authenticate will time out at 1842.337162s 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 0x7f2e98121210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9809d770 0x7f2e98120cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 0x7f2e98121210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52870/0 (socket says 172.21.15.86:52870) 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 -- 172.21.15.86:0/1192097517 learned_addr learned my addr 172.21.15.86:0/1192097517 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): get_auth_request con 0x7f2e9809e090 auth_method 0 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): get_auth_request con 0x7f2e9809d770 auth_method 0 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98134f80 0x7f2e98121750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9c8e7640 10 monclient(hunting): get_auth_request con 0x7f2e98134f80 auth_method 0 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9c8e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9c8e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e9c8e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient(hunting): handle_auth_done global_id 164123 payload 824 2024-09-17T15:41:49.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98134f80 msgr2=0x7f2e98121750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e98134f80 0x7f2e98121750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9809d770 msgr2=0x7f2e98120cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9809d770 0x7f2e98120cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 monclient: found mon.noname-c 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e981237a0 con 0x7f2e9809e090 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e97fff640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9809d770 0x7f2e98120cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 msgr2=0x7f2e98121210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.368+0000 7f2e977fe640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 0x7f2e98121210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00d800 tx=0x7f2e8c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00b8d0 con 0x7f2e9809e090 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: dump: 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e8c019070 con 0x7f2e9809e090 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8c00bcf0 con 0x7f2e9809e090 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: dump: 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.386 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 5 monclient: authenticate success, global_id 164123 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 monclient: _renew_subs 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e967fc640 4 set_mon_vals no callback set 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e98073fd0 con 0x7f2e9809e090 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 monclient: _renew_subs 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e98122280 con 0x7f2e9809e090 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 0x7f2e9802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e980a1890 con 0x7f2e981340f0 2024-09-17T15:41:49.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 0x7f2e9802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient: get_auth_request con 0x7f2e981340f0 auth_method 0 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2e8c01e460 con 0x7f2e9809e090 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 10 monclient(hunting): handle_auth_done global_id 144366 payload 824 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e981375f0 con 0x7f2e981340f0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e8403bb90 0x7f2e8403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7f2e8c050360 con 0x7f2e9809e090 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e8403bb90 0x7f2e8403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e9c8e7640 10 monclient: get_auth_request con 0x7f2e8403bb90 auth_method 0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 msgr2=0x7f2e9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e97fff640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 0x7f2e9802d450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011f50 tx=0x7f2e8800f860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8801e070 con 0x7f2e981340f0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: dump: 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e8800e840 con 0x7f2e981340f0 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 msgr2=0x7f2e9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 0x7f2e9802d450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011f50 tx=0x7f2e8800f860 comp rx=0 tx=0).stop 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 msgr2=0x7f2e9802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.372+0000 7f2e94ff9640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981340f0 0x7f2e9802d450 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e88011f50 tx=0x7f2e8800f860 comp rx=0 tx=0).stop 2024-09-17T15:41:49.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.376+0000 7f2e9c8e7640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e8403bb90 0x7f2e8403e050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e9806d160 tx=0x7f2e900a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e9e34a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 0x7f2e9802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e9e34a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e98123520 con 0x7f2e981344d0 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 0x7f2e9802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient: get_auth_request con 0x7f2e981344d0 auth_method 0 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 10 monclient(hunting): handle_auth_done global_id 144369 payload 824 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.380+0000 7f2e97fff640 1 -- 172.21.15.86:0/1192097517 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e9806d4f0 con 0x7f2e981344d0 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e97fff640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 msgr2=0x7f2e9802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e97fff640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 0x7f2e9802d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e8801b000 tx=0x7f2e880046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8801e040 con 0x7f2e981344d0 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 10 monclient: dump: 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f2e88021cc0 con 0x7f2e981344d0 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:49.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 msgr2=0x7f2e9802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 0x7f2e9802d990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e8801b000 tx=0x7f2e880046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 msgr2=0x7f2e9802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e94ff9640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e981344d0 0x7f2e9802d990 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2e8801b000 tx=0x7f2e880046d0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:49.396 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 127726, 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.397 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.398 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:49.399 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:49.400 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.401 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.403 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:49.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.406 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.410 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:49.411 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e8403bb90 msgr2=0x7f2e8403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e8403bb90 0x7f2e8403e050 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2e9806d160 tx=0x7f2e900a7000 comp rx=0 tx=0).stop 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 10 monclient: shutdown 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 msgr2=0x7f2e98121210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 --2- 172.21.15.86:0/1192097517 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9809e090 0x7f2e98121210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c00d800 tx=0x7f2e8c00dcc0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/1192097517 reap_dead start 2024-09-17T15:41:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9c8e7640 1 -- 172.21.15.86:0/1192097517 reap_dead start 2024-09-17T15:41:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 shutdown_connections 2024-09-17T15:41:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 >> 172.21.15.86:0/1192097517 conn(0x7f2e980a2f40 msgr2=0x7f2e98072e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 shutdown_connections 2024-09-17T15:41:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.384+0000 7f2e9e34a640 1 -- 172.21.15.86:0/1192097517 wait complete. 2024-09-17T15:41:49.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:49.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:41:49.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 10 monclient: get_monmap_and_config 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 10 monclient: monmap: 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.526 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:49.531146+0000 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:49.531146+0000 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:49.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 1 Processor -- start 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 1 -- start start 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.524+0000 7fba0b606640 10 monclient: init 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:41:49.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba040a4f70 0x7fba0409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-a con 0x7fba040a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba040a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 0x7fba0413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-c con 0x7fba0413d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba0409d430 con 0x7fba040a5350 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fba0409c7f0 con 0x7fba040a4f70 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fba040a68d0 con 0x7fba0413d180 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba0b606640 10 monclient(hunting): authenticate will time out at 1842.496716s 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba040a4f70 0x7fba0409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba040a4f70 0x7fba0409cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59142/0 (socket says 172.21.15.86:59142) 2024-09-17T15:41:49.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 -- 172.21.15.86:0/3566661347 learned_addr learned my addr 172.21.15.86:0/3566661347 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): get_auth_request con 0x7fba040a4f70 auth_method 0 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba09ba3640 1 --2- 172.21.15.86:0/3566661347 >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 0x7fba0413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient(hunting): handle_auth_done global_id 144372 payload 274 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 -- 172.21.15.86:0/3566661347 >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 msgr2=0x7fba0413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 --2- 172.21.15.86:0/3566661347 >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 0x7fba0413f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 -- 172.21.15.86:0/3566661347 >> v1:172.21.15.203:6789/0 conn(0x7fba040a5350 legacy=0x7fba040a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 monclient: found mon.noname-a 2024-09-17T15:41:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba093a2640 1 -- 172.21.15.86:0/3566661347 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba040a64a0 con 0x7fba040a4f70 2024-09-17T15:41:49.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba09ba3640 10 monclient: get_auth_request con 0x7fba0413d180 auth_method 0 2024-09-17T15:41:49.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.528+0000 7fba09ba3640 1 --2- 172.21.15.86:0/3566661347 >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 0x7fba0413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:41:49.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 1 -- 172.21.15.86:0/3566661347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba040a4f70 msgr2=0x7fba0409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 1 --2- 172.21.15.86:0/3566661347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba040a4f70 0x7fba0409cef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba0409c970 tx=0x7fb9f401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfb47142c0267c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 1 -- 172.21.15.86:0/3566661347 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40b0070 con 0x7fba040a4f70 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: dump: 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.534 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 1 -- 172.21.15.86:0/3566661347 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f40ab600 con 0x7fba040a4f70 2024-09-17T15:41:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 5 monclient: authenticate success, global_id 144372 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 1 -- 172.21.15.86:0/3566661347 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40ab9d0 con 0x7fba040a4f70 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 4 set_mon_vals no callback set 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fb9fbfff640 10 monclient: dump: 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: shutdown 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- 172.21.15.86:0/3566661347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba040a4f70 msgr2=0x7fba0409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- 172.21.15.86:0/3566661347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba040a4f70 0x7fba0409cef0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fba0409c970 tx=0x7fb9f401c3b0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- 172.21.15.86:0/3566661347 shutdown_connections 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- 172.21.15.86:0/3566661347 >> v2:172.21.15.203:3300/0 conn(0x7fba0413d180 0x7fba0413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- 172.21.15.86:0/3566661347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba040a4f70 0x7fba0409cef0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- 172.21.15.86:0/3566661347 >> 172.21.15.86:0/3566661347 conn(0x7fba040a7240 msgr2=0x7fba040a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- 172.21.15.86:0/3566661347 shutdown_connections 2024-09-17T15:41:49.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- 172.21.15.86:0/3566661347 wait complete. 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: monmap: 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 Processor -- start 2024-09-17T15:41:49.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- start start 2024-09-17T15:41:49.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: init 2024-09-17T15:41:49.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 0x7fba040a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-c con 0x7fba040a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba040a5350 0x7fba040a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-b con 0x7fba040a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 0x7fba040a2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): picked mon.noname-a con 0x7fba0413d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fba04142450 con 0x7fba0413d180 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba041422d0 con 0x7fba040a5350 2024-09-17T15:41:49.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fba041425d0 con 0x7fba040a4f70 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba0b606640 10 monclient(hunting): authenticate will time out at 1842.503653s 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 0x7fba040a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 0x7fba040a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 0x7fba040a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59146/0 (socket says 172.21.15.86:59146) 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 1 -- 172.21.15.86:0/1512801270 learned_addr learned my addr 172.21.15.86:0/1512801270 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 0x7fba040a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52874/0 (socket says 172.21.15.86:52874) 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): get_auth_request con 0x7fba0413d180 auth_method 0 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): get_auth_request con 0x7fba040a4f70 auth_method 0 2024-09-17T15:41:49.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba093a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba09ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.532+0000 7fba08ba1640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba040a5350 0x7fba040a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): get_auth_request con 0x7fba040a5350 auth_method 0 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 10 monclient(hunting): handle_auth_done global_id 164126 payload 824 2024-09-17T15:41:49.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba040a5350 msgr2=0x7fba040a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba040a5350 0x7fba040a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 msgr2=0x7fba040a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 0x7fba040a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 monclient: found mon.noname-c 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0413de60 con 0x7fba040a4f70 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba09ba3640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba0413d180 0x7fba040a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 msgr2=0x7fba040a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba093a2640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 0x7fba040a1d90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40ad000 tx=0x7fb9f4019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40b9020 con 0x7fba040a4f70 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: dump: 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.548 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f40b1040 con 0x7fba040a4f70 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 5 monclient: authenticate success, global_id 164126 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 monclient: _renew_subs 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba0413d910 con 0x7fba040a4f70 2024-09-17T15:41:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40b0050 con 0x7fba040a4f70 2024-09-17T15:41:49.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 10 monclient: dump: 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9fb7fe640 4 set_mon_vals no callback set 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 monclient: _renew_subs 2024-09-17T15:41:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba040a42c0 con 0x7fba040a4f70 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 0x7fba041cff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fba0409cef0 con 0x7fba041cfba0 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 0x7fba041cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient: get_auth_request con 0x7fba041cfba0 auth_method 0 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 10 monclient(hunting): handle_auth_done global_id 164129 payload 824 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb9f40bc020 con 0x7fba040a4f70 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fba08ba1640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba04139fb0 con 0x7fba041cfba0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.536+0000 7fb9f9ffb640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb9e803bbb0 0x7fb9e803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7fb9f40e9de0 con 0x7fba040a4f70 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fba09ba3640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb9e803bbb0 0x7fb9e803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fba09ba3640 10 monclient: get_auth_request con 0x7fb9e803bbb0 auth_method 0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fba08ba1640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 msgr2=0x7fba041cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fba08ba1640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 0x7fba041cff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba0413eb10 tx=0x7fba0000e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00007020 con 0x7fba041cfba0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 10 monclient: dump: 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fba00007020 con 0x7fba041cfba0 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 msgr2=0x7fba041cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 0x7fba041cff80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba0413eb10 tx=0x7fba0000e8b0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 msgr2=0x7fba041cff80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fb9f9ffb640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041cfba0 0x7fba041cff80 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fba0413eb10 tx=0x7fba0000e8b0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.540+0000 7fba09ba3640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb9e803bbb0 0x7fb9e803e070 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fba0413c0b0 tx=0x7fb9fc007810 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba0b606640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba0b606640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 0x7fba0409e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba0b606640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fba04069920 con 0x7fba041d04c0 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 0x7fba0409e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient: get_auth_request con 0x7fba041d04c0 auth_method 0 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 10 monclient(hunting): handle_auth_done global_id 164132 payload 824 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 1 -- 172.21.15.86:0/1512801270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba041395b0 con 0x7fba041d04c0 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 msgr2=0x7fba0409e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fba08ba1640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 0x7fba0409e470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba0000efd0 tx=0x7fba00001030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00006bc0 con 0x7fba041d04c0 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 10 monclient: dump: 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.559 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fba00006bc0 con 0x7fba041d04c0 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 msgr2=0x7fba0409e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 0x7fba0409e470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba0000efd0 tx=0x7fba00001030 comp rx=0 tx=0).stop 2024-09-17T15:41:49.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 msgr2=0x7fba0409e470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.548+0000 7fb9f9ffb640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba041d04c0 0x7fba0409e470 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fba0000efd0 tx=0x7fba00001030 comp rx=0 tx=0).stop 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12735, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:49.562 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.563 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.564 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:49.565 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.566 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.567 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.568 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:49.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.573 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:49.574 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb9e803bbb0 msgr2=0x7fb9e803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb9e803bbb0 0x7fb9e803e070 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fba0413c0b0 tx=0x7fb9fc007810 comp rx=0 tx=0).stop 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 10 monclient: shutdown 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 msgr2=0x7fba040a1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 --2- 172.21.15.86:0/1512801270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba040a4f70 0x7fba040a1d90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9f40ad000 tx=0x7fb9f4019150 comp rx=0 tx=0).stop 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba09ba3640 1 -- 172.21.15.86:0/1512801270 reap_dead start 2024-09-17T15:41:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba09ba3640 1 -- 172.21.15.86:0/1512801270 reap_dead start 2024-09-17T15:41:49.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 shutdown_connections 2024-09-17T15:41:49.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 >> 172.21.15.86:0/1512801270 conn(0x7fba040a7240 msgr2=0x7fba0413c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 shutdown_connections 2024-09-17T15:41:49.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.552+0000 7fba0b606640 1 -- 172.21.15.86:0/1512801270 wait complete. 2024-09-17T15:41:49.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:49.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:41:49.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:41:49.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: get_monmap_and_config 2024-09-17T15:41:49.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: monmap: 2024-09-17T15:41:49.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:49.688283+0000 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:49.688283+0000 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:49.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 Processor -- start 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 -- start start 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: init 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff80a4970 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2ff809ae20 0x7f2ff80a34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff809ae20 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ff813d530 0x7f2ff813f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff813d530 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff809b980 con 0x7f2ff80a4970 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff806d1d0 con 0x7f2ff813d530 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff80a4d50 con 0x7f2ff809ae20 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffec5d640 10 monclient(hunting): authenticate will time out at 1842.653702s 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffc9f9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2ff80a4970 0x7f2ff80a2d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34978/0 (socket says 172.21.15.86:34978) 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ff813d530 0x7f2ff813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3834781707 learned_addr learned my addr 172.21.15.86:0/3834781707 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): get_auth_request con 0x7f2ff813d530 auth_method 0 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ff7fff640 1 --2- 172.21.15.86:0/3834781707 >> v2:172.21.15.203:3300/0 conn(0x7f2ff809ae20 0x7f2ff80a34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request con 0x7f2ff809ae20 auth_method 0 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient(hunting): handle_auth_done global_id 144378 payload 274 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 -- 172.21.15.86:0/3834781707 >> v2:172.21.15.203:3300/0 conn(0x7f2ff809ae20 msgr2=0x7f2ff80a34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 --2- 172.21.15.86:0/3834781707 >> v2:172.21.15.203:3300/0 conn(0x7f2ff809ae20 0x7f2ff80a34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 -- 172.21.15.86:0/3834781707 >> v1:172.21.15.86:6789/0 conn(0x7f2ff80a4970 legacy=0x7f2ff80a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 monclient: found mon.noname-a 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.684+0000 7f2ffd1fa640 1 -- 172.21.15.86:0/3834781707 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff809bf40 con 0x7f2ff813d530 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffd1fa640 1 -- 172.21.15.86:0/3834781707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff813d530 msgr2=0x7f2ff813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffd1fa640 1 --2- 172.21.15.86:0/3834781707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff813d530 0x7f2ff813f920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80a1a40 tx=0x7f2ff001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c18d0f5d21aa795 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 1 -- 172.21.15.86:0/3834781707 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00ac070 con 0x7f2ff813d530 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: dump: 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 1 -- 172.21.15.86:0/3834781707 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ff00a6680 con 0x7f2ff813d530 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 5 monclient: authenticate success, global_id 144378 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 1 -- 172.21.15.86:0/3834781707 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00a6a50 con 0x7f2ff813d530 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 4 set_mon_vals no callback set 2024-09-17T15:41:49.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ff77fe640 10 monclient: dump: 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 10 monclient: shutdown 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3834781707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff813d530 msgr2=0x7f2ff813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3834781707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff813d530 0x7f2ff813f920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80a1a40 tx=0x7f2ff001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3834781707 shutdown_connections 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3834781707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff813d530 0x7f2ff813f920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3834781707 >> v2:172.21.15.203:3300/0 conn(0x7f2ff809ae20 0x7f2ff80a34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3834781707 >> 172.21.15.86:0/3834781707 conn(0x7f2ff80a3e80 msgr2=0x7f2ff80a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3834781707 shutdown_connections 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3834781707 wait complete. 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.688+0000 7f2ffec5d640 10 monclient: monmap: 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.696 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 Processor -- start 2024-09-17T15:41:49.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- start start 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: init 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 0x7f2ff80996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff809ae20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 0x7f2ff8091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff80a4970 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813d530 0x7f2ff8091950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff813d530 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8142890 con 0x7f2ff809ae20 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff809bc60 con 0x7f2ff813d530 2024-09-17T15:41:49.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff8142a10 con 0x7f2ff80a4970 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient(hunting): authenticate will time out at 1842.660736s 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 0x7f2ff80996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 0x7f2ff80996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59154/0 (socket says 172.21.15.86:59154) 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 0x7f2ff8091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3261621440 learned_addr learned my addr 172.21.15.86:0/3261621440 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): get_auth_request con 0x7f2ff809ae20 auth_method 0 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request con 0x7f2ff80a4970 auth_method 0 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffd1fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813d530 0x7f2ff8091950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffd1fa640 10 monclient(hunting): get_auth_request con 0x7f2ff813d530 auth_method 0 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffd1fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffd1fa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffd1fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient(hunting): handle_auth_done global_id 164135 payload 824 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813d530 msgr2=0x7f2ff8091950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813d530 0x7f2ff8091950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 msgr2=0x7f2ff80996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 0x7f2ff80996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 monclient: found mon.noname-c 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff8094680 con 0x7f2ff80a4970 2024-09-17T15:41:49.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffc9f9640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ff809ae20 0x7f2ff80996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:49.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 msgr2=0x7f2ff8091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff7fff640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 0x7f2ff8091410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00d5f0 tx=0x7f2fec00dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec012800 con 0x7f2ff80a4970 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: dump: 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fec019070 con 0x7f2ff80a4970 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 5 monclient: authenticate success, global_id 164135 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _renew_subs 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff809e570 con 0x7f2ff80a4970 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec012bc0 con 0x7f2ff80a4970 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff57fa640 10 monclient: dump: 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _renew_subs 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff8094f20 con 0x7f2ff80a4970 2024-09-17T15:41:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.692+0000 7f2ff6ffd640 4 set_mon_vals no callback set 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffec5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 0x7f2ff8091e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff80a2d90 con 0x7f2ff813ca40 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2fec01e460 con 0x7f2ff80a4970 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 0x7f2ff8091e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient: get_auth_request con 0x7f2ff813ca40 auth_method 0 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2fe403bb60 0x7f2fe403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7f2fec050410 con 0x7f2ff80a4970 2024-09-17T15:41:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffd1fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2fe403bb60 0x7f2fe403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_done global_id 154192 payload 824 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ff80008e0 con 0x7f2ff813ca40 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffd1fa640 10 monclient: get_auth_request con 0x7f2fe403bb60 auth_method 0 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 msgr2=0x7f2ff8091e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ffc9f9640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 0x7f2ff8091e90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80072a0 tx=0x7f2fe8007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800fde0 con 0x7f2ff813ca40 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.696+0000 7f2ff57fa640 10 monclient: dump: 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ffd1fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2fe403bb60 0x7f2fe403e020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8070220 tx=0x7f2ff00a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2fe800fde0 con 0x7f2ff813ca40 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 msgr2=0x7f2ff8091e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 0x7f2ff8091e90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80072a0 tx=0x7f2fe8007760 comp rx=0 tx=0).stop 2024-09-17T15:41:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 msgr2=0x7f2ff8091e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.700+0000 7f2ff57fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff813ca40 0x7f2ff8091e90 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2fe80072a0 tx=0x7f2fe8007760 comp rx=0 tx=0).stop 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffec5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 0x7f2ff8071010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffec5d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8093840 con 0x7f2ff8099bf0 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 0x7f2ff8071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient: get_auth_request con 0x7f2ff8099bf0 auth_method 0 2024-09-17T15:41:49.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 10 monclient(hunting): handle_auth_done global_id 154195 payload 824 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.704+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3261621440 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ff81dab80 con 0x7f2ff8099bf0 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffc9f9640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 msgr2=0x7f2ff8071010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffc9f9640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 0x7f2ff8071010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8017000 tx=0x7f2fe8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe801ee40 con 0x7f2ff8099bf0 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:41:49.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 10 monclient: dump: 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f2fe8006240 con 0x7f2ff8099bf0 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:49.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 msgr2=0x7f2ff8071010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 0x7f2ff8071010 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8017000 tx=0x7f2fe8004950 comp rx=0 tx=0).stop 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 msgr2=0x7f2ff8071010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ff57fa640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ff8099bf0 0x7f2ff8071010 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8017000 tx=0x7f2fe8004950 comp rx=0 tx=0).stop 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 138, 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 61078, 2024-09-17T15:41:49.715 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.716 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.717 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:41:49.718 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:41:49.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:41:49.720 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.721 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.722 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.723 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.724 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:41:49.725 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:41:49.726 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:41:49.727 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:41:49.728 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:41:49.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2fe403bb60 msgr2=0x7f2fe403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2fe403bb60 0x7f2fe403e020 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8070220 tx=0x7f2ff00a7000 comp rx=0 tx=0).stop 2024-09-17T15:41:49.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 10 monclient: shutdown 2024-09-17T15:41:49.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 msgr2=0x7f2ff8091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 0x7f2ff8091410 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2fec00d5f0 tx=0x7f2fec00dab0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffd1fa640 1 -- 172.21.15.86:0/3261621440 reap_dead start 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 shutdown_connections 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 --2- 172.21.15.86:0/3261621440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ff80a4970 0x7f2ff8091410 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 >> 172.21.15.86:0/3261621440 conn(0x7f2ff80a3e80 msgr2=0x7f2ff813e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 shutdown_connections 2024-09-17T15:41:49.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.708+0000 7f2ffec5d640 1 -- 172.21.15.86:0/3261621440 wait complete. 2024-09-17T15:41:49.730 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:41:49.731 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:41:49.731 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:41:49.731 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:41:49.731 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:41:49.731 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:41:49.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:41:49.757 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:41:49.757 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:41:49.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: get_monmap_and_config 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: monmap: 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:49.873752+0000 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:49.873752+0000 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.869 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:49.870 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 Processor -- start 2024-09-17T15:41:49.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 -- start start 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: init 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 0x7fe9f806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f80689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9f8135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9f809c8b0 con 0x7fe9f8135ae0 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe9f8074500 con 0x7fe9f80689d0 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f809cbe0 con 0x7fe9f8068db0 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9ff022640 10 monclient(hunting): authenticate will time out at 1842.839239s 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 0x7fe9f806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 0x7fe9f806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59168/0 (socket says 172.21.15.86:59168) 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 1 -- 172.21.15.86:0/596564580 learned_addr learned my addr 172.21.15.86:0/596564580 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52902/0 (socket says 172.21.15.86:52902) 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 10 monclient(hunting): get_auth_request con 0x7fe9f80689d0 auth_method 0 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9fcdbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request con 0x7fe9f8068db0 auth_method 0 2024-09-17T15:41:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_done global_id 164138 payload 274 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 -- 172.21.15.86:0/596564580 >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 msgr2=0x7fe9f806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/596564580 >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 0x7fe9f806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 -- 172.21.15.86:0/596564580 >> v1:172.21.15.86:6789/0 conn(0x7fe9f8135ae0 legacy=0x7fe9f8135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 monclient: found mon.noname-b 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:49.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.868+0000 7fe9f7fff640 1 -- 172.21.15.86:0/596564580 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f809ce70 con 0x7fe9f8068db0 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.872+0000 7fe9f7fff640 1 -- 172.21.15.86:0/596564580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 msgr2=0x7fe9f80691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.872+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/596564580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80097c0 tx=0x7fe9e801c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50fcd038c7a545b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 1 -- 172.21.15.86:0/596564580 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80b0070 con 0x7fe9f8068db0 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:49.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: dump: 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 1 -- 172.21.15.86:0/596564580 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e80ab600 con 0x7fe9f8068db0 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 1 -- 172.21.15.86:0/596564580 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80ab9d0 con 0x7fe9f8068db0 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 5 monclient: authenticate success, global_id 164138 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 4 set_mon_vals no callback set 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f77fe640 10 monclient: dump: 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: shutdown 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- 172.21.15.86:0/596564580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 msgr2=0x7fe9f80691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- 172.21.15.86:0/596564580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe9e80097c0 tx=0x7fe9e801c300 comp rx=0 tx=0).stop 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- 172.21.15.86:0/596564580 shutdown_connections 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- 172.21.15.86:0/596564580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f80691d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- 172.21.15.86:0/596564580 >> v2:172.21.15.86:3300/0 conn(0x7fe9f80689d0 0x7fe9f806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- 172.21.15.86:0/596564580 >> 172.21.15.86:0/596564580 conn(0x7fe9f80a6440 msgr2=0x7fe9f80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:49.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- 172.21.15.86:0/596564580 shutdown_connections 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- 172.21.15.86:0/596564580 wait complete. 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: build_initial_monmap 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: monmap: 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:49.884 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 Processor -- start 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- start start 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: init 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9f811dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f8134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f8135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f81409b0 con 0x7fe9f8135ae0 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9f8140830 con 0x7fe9f80689d0 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe9f8140b30 con 0x7fe9f8068db0 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9ff022640 10 monclient(hunting): authenticate will time out at 1842.846182s 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f8134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59180/0 (socket says 172.21.15.86:59180) 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 learned_addr learned my addr 172.21.15.86:0/1891067189 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): get_auth_request con 0x7fe9f8135ae0 auth_method 0 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f8134ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52916/0 (socket says 172.21.15.86:52916) 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request con 0x7fe9f8068db0 auth_method 0 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fcdbe640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 msgr2=0x7fe9f811dcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fcdbe640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9f811dcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient(hunting): handle_auth_done global_id 144387 payload 824 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 msgr2=0x7fe9f8134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f8134ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 msgr2=0x7fe9f811dcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9f811dcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 monclient: found mon.noname-a 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9f811f2f0 con 0x7fe9f8135ae0 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.876+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9f8134ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 msgr2=0x7fe9f81351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000c780 tx=0x7fe9f000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f000ac60 con 0x7fe9f8135ae0 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: dump: 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.890 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 5 monclient: authenticate success, global_id 144387 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _renew_subs 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9f8120db0 con 0x7fe9f8135ae0 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _renew_subs 2024-09-17T15:41:49.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9f811e230 con 0x7fe9f8135ae0 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9f000ae00 con 0x7fe9f8135ae0 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0004b70 con 0x7fe9f8135ae0 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f67fc640 4 set_mon_vals no callback set 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: dump: 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.893 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.893 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe9f0004d10 con 0x7fe9f8135ae0 2024-09-17T15:41:49.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d8005090 con 0x7fe9f8135ae0 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 msgr2=0x7fe9f81351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 msgr2=0x7fe9f81351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000c780 tx=0x7fe9f000cc40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fd5bf640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000c780 tx=0x7fe9f000cc40 comp rx=0 tx=0).stop 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 0x7fe9e403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 msgr2=0x7fe9f81351e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9f8135ae0 0x7fe9f81351e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:41:49.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 0x7fe9e4040ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): picked mon.a con 0x7fe9e403e6e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9e4043450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): picked mon.c con 0x7fe9f8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9e4045e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): picked mon.b con 0x7fe9f80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4046870 con 0x7fe9e403e6e0 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e40469f0 con 0x7fe9f80689d0 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:49.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e4046b70 con 0x7fe9f8068db0 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f57fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 0x7fe9e4040ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request con 0x7fe9e403e6e0 auth_method 0 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fcdbe640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 msgr2=0x7fe9e4045e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fcdbe640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9e4045e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fcdbe640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 0x7fe9e403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:49.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient(hunting): handle_auth_done global_id 144387 payload 293 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 msgr2=0x7fe9e4043450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9e4043450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 msgr2=0x7fe9e4045e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9e4045e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 monclient: found mon.a 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fe9e4046cf0 con 0x7fe9e403e6e0 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9f809ce70 con 0x7fe9e403e6e0 2024-09-17T15:41:49.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _finish_auth 0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.880+0000 7fe9fcdbe640 10 monclient: get_auth_request con 0x7fe9e403bb10 auth_method 0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f7fff640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 msgr2=0x7fe9e4040ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f7fff640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 0x7fe9e4040ad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8000c10 tx=0x7fe9f0003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9fcdbe640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 0x7fe9e403dfd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8072fb0 tx=0x7fe9ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80bb090 con 0x7fe9e403e6e0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 10 monclient: dump: 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:49.901 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9e80abd60 con 0x7fe9e403e6e0 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.884+0000 7fe9f6ffd640 4 set_mon_vals no callback set 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.888+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe9e80b6470 con 0x7fe9e403e6e0 2024-09-17T15:41:49.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.888+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 4 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7fe9e80e87d0 con 0x7fe9e403e6e0 2024-09-17T15:41:49.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.892+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe9e8016070 con 0x7fe9e403e6e0 2024-09-17T15:41:49.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.892+0000 7fe9f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:49.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:49.892+0000 7fe9f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9ff022640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9ff022640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9ff022640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe9d8005380 con 0x7fe9e403e6e0 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9f57fa640 1 -- 172.21.15.86:0/1891067189 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe9e80e6220 con 0x7fe9e403e6e0 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:50.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.120+0000 7fe9f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:50.122 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 msgr2=0x7fe9e403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 0x7fe9e403dfd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8072fb0 tx=0x7fe9ec008040 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 10 monclient: shutdown 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 msgr2=0x7fe9e4040ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 0x7fe9e4040ad0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe9e8000c10 tx=0x7fe9f0003df0 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 shutdown_connections 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9f80689d0 0x7fe9e4045e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9f8068db0 0x7fe9e4043450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9e403e6e0 0x7fe9e4040ad0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 --2- 172.21.15.86:0/1891067189 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9e403bb10 0x7fe9e403dfd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:50.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 >> 172.21.15.86:0/1891067189 conn(0x7fe9f80a6440 msgr2=0x7fe9f80a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:50.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 shutdown_connections 2024-09-17T15:41:50.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:50.124+0000 7fe9ff022640 1 -- 172.21.15.86:0/1891067189 wait complete. 2024-09-17T15:41:53.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:53.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.268+0000 7f57bae18640 10 monclient: get_monmap_and_config 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.268+0000 7f57bae18640 10 monclient: build_initial_monmap 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.268+0000 7f57bae18640 10 monclient: monmap: 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:53.274499+0000 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:53.274499+0000 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:53.270 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:53.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.268+0000 7f57bae18640 1 Processor -- start 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 -- start start 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient: init 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57b4138020 addr v1:172.21.15.86:6789/0 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 0x7f57b40a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-c con 0x7f57b406fb00 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 0x7f57b413f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-b con 0x7f57b4138b10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57b40a4bc0 con 0x7f57b4138020 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f57b406f120 con 0x7f57b406fb00 2024-09-17T15:41:53.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:53.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b406de80 con 0x7f57b4138b10 2024-09-17T15:41:53.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:53.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57bae18640 10 monclient(hunting): authenticate will time out at 1846.240764s 2024-09-17T15:41:53.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b8bb4640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f57b4138020 0x7f57b4138400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34998/0 (socket says 172.21.15.86:34998) 2024-09-17T15:41:53.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b8bb4640 1 -- 172.21.15.86:0/1596910315 learned_addr learned my addr 172.21.15.86:0/1596910315 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b3fff640 1 -- 172.21.15.86:0/1596910315 >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 msgr2=0x7f57b40a3e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b3fff640 1 --2- 172.21.15.86:0/1596910315 >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 0x7f57b40a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 --2- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 0x7f57b413f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): get_auth_request con 0x7f57b4138b10 auth_method 0 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient(hunting): handle_auth_done global_id 164144 payload 274 2024-09-17T15:41:53.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 -- 172.21.15.86:0/1596910315 >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 msgr2=0x7f57b40a3e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 --2- 172.21.15.86:0/1596910315 >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 0x7f57b40a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 -- 172.21.15.86:0/1596910315 >> v1:172.21.15.86:6789/0 conn(0x7f57b4138020 legacy=0x7f57b4138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient: _finish_hunting 0 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 monclient: found mon.noname-b 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.272+0000 7f57b93b5640 1 -- 172.21.15.86:0/1596910315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b40a2300 con 0x7f57b4138b10 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b93b5640 1 -- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 msgr2=0x7f57b413f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b93b5640 1 --2- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 0x7f57b413f480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009d90 tx=0x7f57ac01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d29626e6355258fb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 1 -- 172.21.15.86:0/1596910315 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0b2070 con 0x7f57b4138b10 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: dump: 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: _finish_auth 0 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 1 -- 172.21.15.86:0/1596910315 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ac0a5970 con 0x7f57b4138b10 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 1 -- 172.21.15.86:0/1596910315 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0a5d40 con 0x7f57b4138b10 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:53.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57b37fe640 10 monclient: dump: 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 5 monclient: authenticate success, global_id 164144 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 4 set_mon_vals no callback set 2024-09-17T15:41:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: shutdown 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 msgr2=0x7f57b413f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 0x7f57b413f480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57ac009d90 tx=0x7f57ac01c850 comp rx=0 tx=0).stop 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- 172.21.15.86:0/1596910315 shutdown_connections 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- 172.21.15.86:0/1596910315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b4138b10 0x7f57b413f480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- 172.21.15.86:0/1596910315 >> v2:172.21.15.203:3300/0 conn(0x7f57b406fb00 0x7f57b40a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- 172.21.15.86:0/1596910315 >> 172.21.15.86:0/1596910315 conn(0x7f57b40a8470 msgr2=0x7f57b40a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- 172.21.15.86:0/1596910315 shutdown_connections 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- 172.21.15.86:0/1596910315 wait complete. 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: build_initial_monmap 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: monmap: 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:53.284 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 Processor -- start 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- start start 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: init 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:41:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-c con 0x7f57b406fb00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 0x7f57b41cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-b con 0x7f57b4138020 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b4138b10 0x7f57b41d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): picked mon.noname-a con 0x7f57b4138b10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f57b41425a0 con 0x7f57b4138b10 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f57b4142420 con 0x7f57b4138020 2024-09-17T15:41:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f57b4142720 con 0x7f57b406fb00 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.276+0000 7f57bae18640 10 monclient(hunting): authenticate will time out at 1846.247928s 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52944/0 (socket says 172.21.15.86:52944) 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 -- 172.21.15.86:0/3277261445 learned_addr learned my addr 172.21.15.86:0/3277261445 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): get_auth_request con 0x7f57b406fb00 auth_method 0 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b3fff640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 msgr2=0x7f57b41cffe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b3fff640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 0x7f57b41cffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b4138b10 0x7f57b41d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): get_auth_request con 0x7f57b4138b10 auth_method 0 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b93b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient(hunting): handle_auth_done global_id 164147 payload 824 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 msgr2=0x7f57b41cffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 0x7f57b41cffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b4138b10 msgr2=0x7f57b41d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b4138b10 0x7f57b41d0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient: _finish_hunting 0 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 monclient: found mon.noname-c 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 -- 172.21.15.86:0/3277261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b413dae0 con 0x7f57b406fb00 2024-09-17T15:41:53.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 msgr2=0x7f57b41cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:53.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b8bb4640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c4a0 tx=0x7f579c00c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:53.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c017070 con 0x7f57b406fb00 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: dump: 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:53.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: _finish_auth 0 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f579c007070 con 0x7f57b406fb00 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 5 monclient: authenticate success, global_id 164147 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _renew_subs 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b413cc50 con 0x7f57b406fb00 2024-09-17T15:41:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f579c007460 con 0x7f57b406fb00 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b17fa640 10 monclient: dump: 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:53.294 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _renew_subs 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57b2ffd640 4 set_mon_vals no callback set 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b40a7280 con 0x7f57b406fb00 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.280+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b4138400 con 0x7f57b406fb00 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f579c01a070 con 0x7f57b406fb00 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b17fa640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 0x7f579403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:53.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7f579c04f440 con 0x7f57b406fb00 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b3fff640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 0x7f579403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b3fff640 10 monclient: get_auth_request con 0x7f579403bb10 auth_method 0 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.284+0000 7f57b3fff640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 0x7f579403dfd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f57b41d0b60 tx=0x7f57a4006d90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.288+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f579c007680 con 0x7f57b406fb00 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.288+0000 7f57b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.288+0000 7f57b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57bae18640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57bae18640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57bae18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:41:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f57b40699d0 con 0x7f57b406fb00 2024-09-17T15:41:53.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57b17fa640 1 -- 172.21.15.86:0/3277261445 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f579c01bc00 con 0x7f57b406fb00 2024-09-17T15:41:53.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:53.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.508+0000 7f57b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:53.510 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:53.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 msgr2=0x7f579403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:53.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 0x7f579403dfd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f57b41d0b60 tx=0x7f57a4006d90 comp rx=0 tx=0).stop 2024-09-17T15:41:53.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 10 monclient: shutdown 2024-09-17T15:41:53.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 msgr2=0x7f57b41cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f579c00c4a0 tx=0x7f579c00c960 comp rx=0 tx=0).stop 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 shutdown_connections 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f579403bb10 0x7f579403dfd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57b4138b10 0x7f57b41d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57b4138020 0x7f57b41cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 --2- 172.21.15.86:0/3277261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57b406fb00 0x7f57b41cfaa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 >> 172.21.15.86:0/3277261445 conn(0x7f57b40a8470 msgr2=0x7f57b409fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 shutdown_connections 2024-09-17T15:41:53.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:53.512+0000 7f57bae18640 1 -- 172.21.15.86:0/3277261445 wait complete. 2024-09-17T15:41:56.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:41:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 10 monclient: get_monmap_and_config 2024-09-17T15:41:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 10 monclient: build_initial_monmap 2024-09-17T15:41:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 10 monclient: monmap: 2024-09-17T15:41:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:41:56.657976+0000 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:41:56.657976+0000 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:41:56.654 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 1 Processor -- start 2024-09-17T15:41:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 1 -- start start 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 10 monclient: init 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.652+0000 7ff76e6d2640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 0x7ff76809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-c con 0x7ff7680a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-b con 0x7ff7680a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff768137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff76809d350 con 0x7ff768137ea0 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff7680a6530 con 0x7ff7680a50e0 2024-09-17T15:41:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff76809c150 con 0x7ff7680a54c0 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76e6d2640 10 monclient(hunting): authenticate will time out at 1849.624122s 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52954/0 (socket says 172.21.15.86:52954) 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff76cc6f640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7ff768137ea0 legacy=0x7ff768138330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 -- 172.21.15.86:0/1717999137 learned_addr learned my addr 172.21.15.86:0/1717999137 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): get_auth_request con 0x7ff7680a54c0 auth_method 0 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff767fff640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 msgr2=0x7ff76809ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:56.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff767fff640 1 --2- 172.21.15.86:0/1717999137 >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 0x7ff76809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient(hunting): handle_auth_done global_id 164150 payload 274 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 -- 172.21.15.86:0/1717999137 >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 msgr2=0x7ff76809ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 --2- 172.21.15.86:0/1717999137 >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 0x7ff76809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 -- 172.21.15.86:0/1717999137 >> v1:172.21.15.203:6789/0 conn(0x7ff768137ea0 legacy=0x7ff768138330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient: _finish_hunting 0 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 monclient: found mon.noname-b 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:41:56.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.656+0000 7ff7677fe640 1 -- 172.21.15.86:0/1717999137 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76809c430 con 0x7ff7680a54c0 2024-09-17T15:41:56.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff7677fe640 1 -- 172.21.15.86:0/1717999137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 msgr2=0x7ff7680a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:56.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff7677fe640 1 --2- 172.21.15.86:0/1717999137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7680a8280 tx=0x7ff75c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c88fb57f836e1392 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:56.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 1 -- 172.21.15.86:0/1717999137 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c0b0070 con 0x7ff7680a54c0 2024-09-17T15:41:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:41:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: dump: 2024-09-17T15:41:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: _finish_auth 0 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 5 monclient: authenticate success, global_id 164150 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 1 -- 172.21.15.86:0/1717999137 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c0ab600 con 0x7ff7680a54c0 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: get_monmap_and_config success 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 4 set_mon_vals no callback set 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 1 -- 172.21.15.86:0/1717999137 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75c0ab9d0 con 0x7ff7680a54c0 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff766ffd640 10 monclient: dump: 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.664 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: shutdown 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- 172.21.15.86:0/1717999137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 msgr2=0x7ff7680a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/1717999137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff7680a8280 tx=0x7ff75c01c420 comp rx=0 tx=0).stop 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- 172.21.15.86:0/1717999137 shutdown_connections 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/1717999137 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7680a54c0 0x7ff7680a58d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/1717999137 >> v2:172.21.15.203:3300/0 conn(0x7ff7680a50e0 0x7ff76809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- 172.21.15.86:0/1717999137 >> 172.21.15.86:0/1717999137 conn(0x7ff7680aaae0 msgr2=0x7ff7680aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- 172.21.15.86:0/1717999137 shutdown_connections 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- 172.21.15.86:0/1717999137 wait complete. 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: build_initial_monmap 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: monmap: 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:41:56.666 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:41:56.667 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 Processor -- start 2024-09-17T15:41:56.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- start start 2024-09-17T15:41:56.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: init 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: _reopen_session rank -1 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-a con 0x7ff7680a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 0x7ff768136da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-b con 0x7ff7680a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): picked mon.noname-c con 0x7ff768137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:56.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff768142cb0 con 0x7ff7680a50e0 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff768142b30 con 0x7ff7680a54c0 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff768142e30 con 0x7ff768137ea0 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): _renew_subs 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76e6d2640 10 monclient(hunting): authenticate will time out at 1849.631260s 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:59204/0 (socket says 172.21.15.86:59204) 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52966/0 (socket says 172.21.15.86:52966) 2024-09-17T15:41:56.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 -- 172.21.15.86:0/4101784175 learned_addr learned my addr 172.21.15.86:0/4101784175 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): get_auth_request con 0x7ff7680a50e0 auth_method 0 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): get_auth_request con 0x7ff768137ea0 auth_method 0 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff7677fe640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 msgr2=0x7ff768136da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff7677fe640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 0x7ff768136da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:56.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient(hunting): handle_auth_done global_id 144393 payload 824 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 msgr2=0x7ff76813cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 msgr2=0x7ff768136da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 0x7ff768136da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient: _finish_hunting 0 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 monclient: found mon.noname-a 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:41:56.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff767fff640 1 -- 172.21.15.86:0/4101784175 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7680a18d0 con 0x7ff7680a50e0 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.660+0000 7ff76cc6f640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff767fff640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 msgr2=0x7ff768136860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff767fff640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff75800ba10 tx=0x7ff75800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75800cde0 con 0x7ff7680a50e0 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: dump: 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:56.675 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: _finish_auth 0 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff758010070 con 0x7ff7680a50e0 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7580156b0 con 0x7ff7680a50e0 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 5 monclient: authenticate success, global_id 144393 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _renew_subs 2024-09-17T15:41:56.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7681211f0 con 0x7ff7680a50e0 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff7667fc640 4 set_mon_vals no callback set 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 10 monclient: dump: 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:41:56.677 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _renew_subs 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7680a7b10 con 0x7ff7680a50e0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76809ce10 con 0x7ff7680a50e0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff75801e3f0 con 0x7ff7680a50e0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 0x7ff75403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(507..507 src has 438..507) ==== 4771+0+0 (secure 0 0 0) 0x7ff758050200 con 0x7ff7680a50e0 2024-09-17T15:41:56.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.664+0000 7ff7677fe640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 0x7ff75403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:41:56.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.668+0000 7ff7677fe640 10 monclient: get_auth_request con 0x7ff75403bb40 auth_method 0 2024-09-17T15:41:56.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.668+0000 7ff7677fe640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 0x7ff75403e000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0ab000 tx=0x7ff75c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:41:56.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.668+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff7680a18d0 con 0x7ff7680a50e0 2024-09-17T15:41:56.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.668+0000 7ff764ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:41:56.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.668+0000 7ff764ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:41:56.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff76e6d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:41:56.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff76e6d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:56.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff76e6d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:41:56.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff768074be0 con 0x7ff7680a50e0 2024-09-17T15:41:56.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff764ff9640 1 -- 172.21.15.86:0/4101784175 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7ff75801e8c0 con 0x7ff7680a50e0 2024-09-17T15:41:56.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff764ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:41:56.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.892+0000 7ff764ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:41:56.895 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:41:56.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 msgr2=0x7ff75403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:56.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 0x7ff75403e000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0ab000 tx=0x7ff75c008040 comp rx=0 tx=0).stop 2024-09-17T15:41:56.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 10 monclient: shutdown 2024-09-17T15:41:56.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 msgr2=0x7ff768136860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff75800ba10 tx=0x7ff75800bed0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 shutdown_connections 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff75403bb40 0x7ff75403e000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff768137ea0 0x7ff76813cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7680a54c0 0x7ff768136da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 --2- 172.21.15.86:0/4101784175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7680a50e0 0x7ff768136860 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 >> 172.21.15.86:0/4101784175 conn(0x7ff7680aaae0 msgr2=0x7ff76809fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 shutdown_connections 2024-09-17T15:41:56.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:41:56.896+0000 7ff76e6d2640 1 -- 172.21.15.86:0/4101784175 wait complete. 2024-09-17T15:41:59.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: get_monmap_and_config 2024-09-17T15:42:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: build_initial_monmap 2024-09-17T15:42:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: monmap: 2024-09-17T15:42:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:00.044 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:00.049249+0000 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:00.049249+0000 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:00.045 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 Processor -- start 2024-09-17T15:42:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 -- start start 2024-09-17T15:42:00.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: init 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-c con 0x7f78b40689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78b4068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78b413aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b409c8b0 con 0x7f78b413aff0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78b409cb90 con 0x7f78b4068db0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f78b4074500 con 0x7f78b40689d0 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78bbb01640 10 monclient(hunting): authenticate will time out at 1853.014725s 2024-09-17T15:42:00.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78b989d640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 msgr2=0x7f78b406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:00.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78b909c640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f78b4068db0 legacy=0x7f78b4069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:42:00.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78b989d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:00.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78ba09e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f78b413aff0 0x7f78b413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42092/0 (socket says 172.21.15.86:42092) 2024-09-17T15:42:00.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.044+0000 7f78ba09e640 1 -- 172.21.15.86:0/2601764393 learned_addr learned my addr 172.21.15.86:0/2601764393 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:00.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.048+0000 7f78ba09e640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.86:6789/0 conn(0x7f78b413aff0 legacy=0x7f78b413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.248+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.203:6789/0 conn(0x7f78b4068db0 legacy=0x7f78b4069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:42:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.248+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 msgr2=0x7f78b406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.248+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.656+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.203:6789/0 conn(0x7f78b4068db0 legacy=0x7f78b4069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:42:00.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.656+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 msgr2=0x7f78b406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:00.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:00.656+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:01.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:01.464+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.203:6789/0 conn(0x7f78b4068db0 legacy=0x7f78b4069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:42:01.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:01.464+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 msgr2=0x7f78b406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:01.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:01.464+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): tick 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 msgr2=0x7f78b406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.203:3300/0 conn(0x7f78b40689d0 0x7f78b406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.203:6789/0 conn(0x7f78b4068db0 legacy=0x7f78b4069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 -- 172.21.15.86:0/2601764393 >> v1:172.21.15.86:6789/0 conn(0x7f78b413aff0 legacy=0x7f78b413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient: _add_conns ranks=[3,2] 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f7898003750 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f7898006260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 -- 172.21.15.86:0/2601764393 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7898008db0 con 0x7f7898003750 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:03.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 1 -- 172.21.15.86:0/2601764393 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7898008f60 con 0x7f7898006260 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78ab7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 10 monclient(hunting): get_auth_request con 0x7f7898006260 auth_method 0 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 10 monclient(hunting): get_auth_request con 0x7f7898003750 auth_method 0 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.047+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 10 monclient(hunting): get_auth_request con 0x7f7898006260 auth_method 0 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 10 monclient(hunting): get_auth_request con 0x7f7898003750 auth_method 0 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:03.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.255+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 10 monclient(hunting): get_auth_request con 0x7f7898006260 auth_method 0 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 10 monclient(hunting): get_auth_request con 0x7f7898003750 auth_method 0 2024-09-17T15:42:03.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:03.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:03.663+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:04.036 INFO:tasks.rados.rados.0.smithi203.stdout:2219: done (0 left) 2024-09-17T15:42:04.037 INFO:tasks.rados.rados.0.smithi203.stdout:2220: rollback oid 7 current snap is 246 2024-09-17T15:42:04.037 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 239 2024-09-17T15:42:04.037 INFO:tasks.rados.rados.0.smithi203.stdout:2221: write (excl) oid 8 current snap is 246 2024-09-17T15:42:04.037 INFO:tasks.rados.rados.0.smithi203.stdout:2221: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-17T15:42:04.047 INFO:tasks.rados.rados.0.smithi203.stdout:2221: writing smithi20328086-8 from 427109 to 832431 tid 1 2024-09-17T15:42:04.064 INFO:tasks.rados.rados.0.smithi203.stdout:2221: writing smithi20328086-8 from 1553564 to 2099490 tid 2 2024-09-17T15:42:04.081 INFO:tasks.rados.rados.0.smithi203.stdout:2221: writing smithi20328086-8 from 2579307 to 3227109 tid 3 2024-09-17T15:42:04.081 INFO:tasks.rados.rados.0.smithi203.stdout:2222: copy_from oid 12 from oid 23 current snap is 246 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:2220: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:2221: finishing write tid 1 to smithi20328086-8 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:2220: done (2 left) 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:2223: read oid 50 snap -1 2024-09-17T15:42:04.082 INFO:tasks.rados.rados.0.smithi203.stdout:2223: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2224: read oid 16 snap -1 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2224: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2225: read oid 5 snap 241 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2225: expect deleted 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2226: read oid 46 snap -1 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2226: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:42:04.083 INFO:tasks.rados.rados.0.smithi203.stdout:2227: copy_from oid 30 from oid 32 current snap is 246 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:2228: rollback oid 49 current snap is 246 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 245 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:2229: delete oid 10 current snap is 246 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:2221: finishing write tid 2 to smithi20328086-8 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:2222: finishing copy_from to smithi20328086-12 2024-09-17T15:42:04.084 INFO:tasks.rados.rados.0.smithi203.stdout:2222: got expected ENOENT (src dne) 2024-09-17T15:42:04.085 INFO:tasks.rados.rados.0.smithi203.stdout:2222: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:42:04.088 INFO:tasks.rados.rados.0.smithi203.stdout:2223: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:42:04.124 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2001839 2024-09-17T15:42:04.145 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 589844 2024-09-17T15:42:04.192 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 489035 2024-09-17T15:42:04.237 INFO:tasks.rados.rados.0.smithi203.stdout:2224: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:04.245 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1406080 2024-09-17T15:42:04.288 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.346 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 379675 2024-09-17T15:42:04.402 INFO:tasks.rados.rados.0.smithi203.stdout:2226: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:42:04.402 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): get_auth_request con 0x7f7898006260 auth_method 0 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): get_auth_request con 0x7f7898003750 auth_method 0 2024-09-17T15:42:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient(hunting): handle_auth_done global_id 164156 payload 274 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 msgr2=0x7f7898005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient: _finish_hunting 0 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 1 monclient: found mon.noname-b 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b409ce70 con 0x7f7898006260 2024-09-17T15:42:04.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.467+0000 7f78b989d640 1 --2- 172.21.15.86:0/2601764393 >> v2:172.21.15.86:3300/0 conn(0x7f7898003750 0x7f7898005bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:04.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b909c640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b909c640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f78b00038c0 tx=0x7f78b001d6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73676f5d4a423 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 1 -- 172.21.15.86:0/2601764393 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b409c2d0 con 0x7f7898006260 2024-09-17T15:42:04.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: dump: 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.475 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: _finish_auth 0 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: _check_auth_tickets 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 1 -- 172.21.15.86:0/2601764393 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b00a76c0 con 0x7f7898006260 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 1 -- 172.21.15.86:0/2601764393 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b00a7a90 con 0x7f7898006260 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 5 monclient: authenticate success, global_id 164156 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 4 set_mon_vals no callback set 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:04.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78b889b640 10 monclient: dump: 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 10 monclient: shutdown 2024-09-17T15:42:04.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 -- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f7898008700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 --2- 172.21.15.86:0/2601764393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f7898008700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f78b00038c0 tx=0x7f78b001d6c0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78ba09e640 1 -- 172.21.15.86:0/2601764393 reap_dead start 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 -- 172.21.15.86:0/2601764393 shutdown_connections 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 -- 172.21.15.86:0/2601764393 >> 172.21.15.86:0/2601764393 conn(0x7f78b40a6440 msgr2=0x7f78b40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 -- 172.21.15.86:0/2601764393 shutdown_connections 2024-09-17T15:42:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.475+0000 7f78bbb01640 1 -- 172.21.15.86:0/2601764393 wait complete. 2024-09-17T15:42:04.479 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient: build_initial_monmap 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient: monmap: 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:04.486 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 Processor -- start 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 -- start start 2024-09-17T15:42:04.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient: init 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-c con 0x7f7898006260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 0x7f78b411d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-b con 0x7f78b411ce00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:04.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 0x7f78b41ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): picked mon.noname-a con 0x7f78b41cdee0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4141bd0 con 0x7f78b41cdee0 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f78b4141a50 con 0x7f78b411ce00 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f78b4141d50 con 0x7f7898006260 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78bbb01640 10 monclient(hunting): authenticate will time out at 1857.446617s 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b909c640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 msgr2=0x7f78b411d260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 0x7f78b41ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b909c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 0x7f78b411d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52478/0 (socket says 172.21.15.86:52478) 2024-09-17T15:42:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 1 -- 172.21.15.86:0/293499837 learned_addr learned my addr 172.21.15.86:0/293499837 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): get_auth_request con 0x7f78b41cdee0 auth_method 0 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): get_auth_request con 0x7f7898006260 auth_method 0 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient(hunting): handle_auth_done global_id 144399 payload 824 2024-09-17T15:42:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 -- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 msgr2=0x7f78b411c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 -- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 msgr2=0x7f78b411d260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 0x7f78b411d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient: _finish_hunting 0 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 monclient: found mon.noname-a 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78ba09e640 1 -- 172.21.15.86:0/293499837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78b413de30 con 0x7f78b41cdee0 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.479+0000 7f78b989d640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78ba09e640 1 -- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 msgr2=0x7f78b41ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78ba09e640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 0x7f78b41ce340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a0660 tx=0x7f78ac00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac017070 con 0x7f78b41cdee0 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: dump: 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ac00acf0 con 0x7f78b41cdee0 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 5 monclient: authenticate success, global_id 144399 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _renew_subs 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b40a3810 con 0x7f78b41cdee0 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac00aea0 con 0x7f78b41cdee0 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 10 monclient: dump: 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.496 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _renew_subs 2024-09-17T15:42:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b411ec80 con 0x7f78b41cdee0 2024-09-17T15:42:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78abfff640 4 set_mon_vals no callback set 2024-09-17T15:42:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:04.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78b411def0 con 0x7f78b41cdee0 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f78ac01a070 con 0x7f78b41cdee0 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.483+0000 7f78aa7fc640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 0x7f789803f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.487+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(508..508 src has 438..508) ==== 4799+0+0 (secure 0 0 0) 0x7f78ac04f900 con 0x7f78b41cdee0 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.487+0000 7f78b989d640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 0x7f789803f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.487+0000 7f78b989d640 10 monclient: get_auth_request con 0x7f789803ccd0 auth_method 0 2024-09-17T15:42:04.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.487+0000 7f78b989d640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 0x7f789803f190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009670 tx=0x7f78a4003dc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.491+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f78ac04e2f0 con 0x7f78b41cdee0 2024-09-17T15:42:04.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.491+0000 7f78aa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:04.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.491+0000 7f78aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2221: finishing write tid 3 to smithi20328086-8 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2221: finishing write tid 4 to smithi20328086-8 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2228: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2221: finishing write tid 5 to smithi20328086-8 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2221: oid 8 updating version 0 to 832 2024-09-17T15:42:04.510 INFO:tasks.rados.rados.0.smithi203.stdout:2221: oid 8 version 832 is already newer than 831 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:2221: oid 8 updating version 832 to 834 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:2221: oid 8 version 834 is already newer than 833 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 834 (ObjNum 787 snap 246 seq_num 787) dirty exists 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:2221: left oid 8 (ObjNum 787 snap 246 seq_num 787) 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:2228: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:2228: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:42:04.511 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1166 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2227: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2227: finishing copy_from to smithi20328086-30 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1097 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2221: done (8 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2222: done (7 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2223: done (6 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2224: done (5 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2225: done (4 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2226: done (3 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2227: done (2 left) 2024-09-17T15:42:04.512 INFO:tasks.rados.rados.0.smithi203.stdout:2228: done (1 left) 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2229: done (0 left) 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2230: rollback oid 34 current snap is 246 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 245 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2231: read oid 32 snap -1 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2231: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2232: read oid 33 snap 245 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2232: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2233: read oid 48 snap -1 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2233: expect (ObjNum 755 snap 233 seq_num 755) 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2234: rollback oid 19 current snap is 246 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 243 2024-09-17T15:42:04.513 INFO:tasks.rados.rados.0.smithi203.stdout:2235: rollback oid 16 current snap is 246 2024-09-17T15:42:04.514 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 232 2024-09-17T15:42:04.514 INFO:tasks.rados.rados.0.smithi203.stdout:2236: write (excl) oid 36 current snap is 246 2024-09-17T15:42:04.514 INFO:tasks.rados.rados.0.smithi203.stdout:2236: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-09-17T15:42:04.531 INFO:tasks.rados.rados.0.smithi203.stdout:2236: writing smithi20328086-36 from 460723 to 1016543 tid 1 2024-09-17T15:42:04.552 INFO:tasks.rados.rados.0.smithi203.stdout:2236: writing smithi20328086-36 from 1670528 to 2220123 tid 2 2024-09-17T15:42:04.567 INFO:tasks.rados.rados.0.smithi203.stdout:2236: writing smithi20328086-36 from 2634153 to 3260723 tid 3 2024-09-17T15:42:04.567 INFO:tasks.rados.rados.0.smithi203.stdout:2237: read oid 20 snap -1 2024-09-17T15:42:04.567 INFO:tasks.rados.rados.0.smithi203.stdout:2237: expect (ObjNum 765 snap 236 seq_num 765) 2024-09-17T15:42:04.567 INFO:tasks.rados.rados.0.smithi203.stdout:2238: copy_from oid 10 from oid 4 current snap is 246 2024-09-17T15:42:04.567 INFO:tasks.rados.rados.0.smithi203.stdout:2232: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:42:04.702 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:04.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:04.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78b413a6f0 con 0x7f78b41cdee0 2024-09-17T15:42:04.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78aa7fc640 1 -- 172.21.15.86:0/293499837 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f78ac019cd0 con 0x7f78b41cdee0 2024-09-17T15:42:04.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78aa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:04.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78aa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:04.737 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 msgr2=0x7f789803f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 0x7f789803f190 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009670 tx=0x7f78a4003dc0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 10 monclient: shutdown 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 msgr2=0x7f78b41ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.739+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 0x7f78b41ce340 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f78b40a0660 tx=0x7f78ac00cde0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 shutdown_connections 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f789803ccd0 0x7f789803f190 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b41cdee0 0x7f78b41ce340 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78b411ce00 0x7f78b411d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 --2- 172.21.15.86:0/293499837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7898006260 0x7f78b411c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 >> 172.21.15.86:0/293499837 conn(0x7f78b40a6440 msgr2=0x7f78b40a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 shutdown_connections 2024-09-17T15:42:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.743+0000 7f78bbb01640 1 -- 172.21.15.86:0/293499837 wait complete. 2024-09-17T15:42:04.745 INFO:tasks.rados.rados.0.smithi203.stdout:2231: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-17T15:42:04.745 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.756 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:42:04.756 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:42:04.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:42:04.798 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1199533 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:2230: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:2230: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:2230: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 638 (ObjNum 778 snap 243 seq_num 778) dirty exists 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:2233: expect (ObjNum 755 snap 233 seq_num 755) 2024-09-17T15:42:04.854 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: get_monmap_and_config 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: build_initial_monmap 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: monmap: 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:04.880835+0000 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:04.880835+0000 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:04.877 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:04.877 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:04.877 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:04.877 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 Processor -- start 2024-09-17T15:42:04.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 -- start start 2024-09-17T15:42:04.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: init 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05dc0a0c70 0x7f05dc0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-a con 0x7f05dc0a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-b con 0x7f05dc0a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 0x7f05dc136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-c con 0x7f05dc135f90 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc0a2410 con 0x7f05dc0a0c70 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc0a2130 con 0x7f05dc135f90 2024-09-17T15:42:04.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f05dc0a26f0 con 0x7f05dc0a1590 2024-09-17T15:42:04.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e454c640 10 monclient(hunting): authenticate will time out at 1857.846313s 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05dc0a0c70 0x7f05dc0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e2ae9640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 msgr2=0x7f05dc136420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e2ae9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 0x7f05dc136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05dc0a0c70 0x7f05dc0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39166/0 (socket says 172.21.15.86:39166) 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 -- 172.21.15.86:0/2988001076 learned_addr learned my addr 172.21.15.86:0/2988001076 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): get_auth_request con 0x7f05dc0a0c70 auth_method 0 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52492/0 (socket says 172.21.15.86:52492) 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): get_auth_request con 0x7f05dc0a1590 auth_method 0 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient(hunting): handle_auth_done global_id 144402 payload 274 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 -- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 msgr2=0x7f05dc135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 -- 172.21.15.86:0/2988001076 >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 msgr2=0x7f05dc136420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2988001076 >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 0x7f05dc136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient: _finish_hunting 0 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 monclient: found mon.noname-a 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e22e8640 1 -- 172.21.15.86:0/2988001076 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc09f9f0 con 0x7f05dc0a0c70 2024-09-17T15:42:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.879+0000 7f05e1ae7640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e22e8640 1 -- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc0a0c70 msgr2=0x7f05dc0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc0a0c70 0x7f05dc0a1050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a40b0 tx=0x7f05cc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1043ace15507b005 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 1 -- 172.21.15.86:0/2988001076 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0b1070 con 0x7f05dc0a0c70 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: dump: 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: _finish_auth 0 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: _check_auth_tickets 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 1 -- 172.21.15.86:0/2988001076 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05cc0a8050 con 0x7f05dc0a0c70 2024-09-17T15:42:04.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 1 -- 172.21.15.86:0/2988001076 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0ac860 con 0x7f05dc0a0c70 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e12e6640 10 monclient: dump: 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e454c640 5 monclient: authenticate success, global_id 144402 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e454c640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.883+0000 7f05e454c640 4 set_mon_vals no callback set 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: shutdown 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc0a0c70 msgr2=0x7f05dc0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc0a0c70 0x7f05dc0a1050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0a40b0 tx=0x7f05cc01c400 comp rx=0 tx=0).stop 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- 172.21.15.86:0/2988001076 shutdown_connections 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- 172.21.15.86:0/2988001076 >> v2:172.21.15.203:3300/0 conn(0x7f05dc135f90 0x7f05dc136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a1590 0x7f05dc135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- 172.21.15.86:0/2988001076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc0a0c70 0x7f05dc0a1050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- 172.21.15.86:0/2988001076 >> 172.21.15.86:0/2988001076 conn(0x7f05dc0a2f40 msgr2=0x7f05dc0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- 172.21.15.86:0/2988001076 shutdown_connections 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- 172.21.15.86:0/2988001076 wait complete. 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: build_initial_monmap 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: monmap: 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:04.897 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 Processor -- start 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- start start 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: init 2024-09-17T15:42:04.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 0x7f05dc1cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-c con 0x7f05dc0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05dc0a1590 0x7f05dc134ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-b con 0x7f05dc0a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 0x7f05dc11cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): picked mon.noname-a con 0x7f05dc135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc0a00d0 con 0x7f05dc135f90 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc09ff50 con 0x7f05dc0a1590 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f05dc13f740 con 0x7f05dc0a0c70 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e454c640 10 monclient(hunting): authenticate will time out at 1857.854231s 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e1ae7640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05dc0a1590 msgr2=0x7f05dc134ff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e1ae7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05dc0a1590 0x7f05dc134ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 0x7f05dc1cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 0x7f05dc11cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 0x7f05dc11cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39172/0 (socket says 172.21.15.86:39172) 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 1 -- 172.21.15.86:0/2516016452 learned_addr learned my addr 172.21.15.86:0/2516016452 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:04.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): get_auth_request con 0x7f05dc0a0c70 auth_method 0 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): get_auth_request con 0x7f05dc135f90 auth_method 0 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient(hunting): handle_auth_done global_id 164165 payload 824 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05dc0a1590 msgr2=0x7f05dc134ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05dc0a1590 0x7f05dc134ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 msgr2=0x7f05dc11cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 0x7f05dc11cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient: _finish_hunting 0 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 monclient: found mon.noname-c 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:04.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e22e8640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc1209b0 con 0x7f05dc0a0c70 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.887+0000 7f05e2ae9640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135f90 0x7f05dc11cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e22e8640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 msgr2=0x7f05dc1cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e22e8640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 0x7f05dc1cdc20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0ac000 tx=0x7f05cc0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0b1040 con 0x7f05dc0a0c70 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: dump: 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05cc0b5420 con 0x7f05dc0a0c70 2024-09-17T15:42:04.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0b5810 con 0x7f05dc0a0c70 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 10 monclient: dump: 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.905 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e0ae5640 4 set_mon_vals no callback set 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 5 monclient: authenticate success, global_id 164165 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 monclient: _renew_subs 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05dc11e650 con 0x7f05dc0a0c70 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 monclient: _renew_subs 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05dc11f1b0 con 0x7f05dc0a0c70 2024-09-17T15:42:04.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 0x7f05dc11d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc073570 con 0x7f05dc135530 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 0x7f05dc11d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient: get_auth_request con 0x7f05dc135530 auth_method 0 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_done global_id 144408 payload 824 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e1ae7640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05dc1204d0 con 0x7f05dc135530 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f05cc0bc060 con 0x7f05dc0a0c70 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05c003bb10 0x7f05c003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(508..508 src has 438..508) ==== 4799+0+0 (secure 0 0 0) 0x7f05cc0e8e00 con 0x7f05dc0a0c70 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e2ae9640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05c003bb10 0x7f05c003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.891+0000 7f05e2ae9640 10 monclient: get_auth_request con 0x7f05c003bb10 auth_method 0 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05e1ae7640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 msgr2=0x7f05dc11d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05e1ae7640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 0x7f05dc11d320 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05dc120bc0 tx=0x7f05d800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d800cb40 con 0x7f05dc135530 2024-09-17T15:42:04.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 10 monclient: dump: 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.909 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05d800cb40 con 0x7f05dc135530 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 msgr2=0x7f05dc11d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 0x7f05dc11d320 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05dc120bc0 tx=0x7f05d800ec00 comp rx=0 tx=0).stop 2024-09-17T15:42:04.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 msgr2=0x7f05dc11d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:04.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05d2ffd640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc135530 0x7f05dc11d320 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f05dc120bc0 tx=0x7f05d800ec00 comp rx=0 tx=0).stop 2024-09-17T15:42:04.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.895+0000 7f05e2ae9640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05c003bb10 0x7f05c003dfd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f05d400b7f0 tx=0x7f05d400d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.912 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 594361 2024-09-17T15:42:04.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e454c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:04.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e454c640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 0x7f05dc137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e454c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc11dc70 con 0x7f05dc11d860 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 0x7f05dc137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient: get_auth_request con 0x7f05dc11d860 auth_method 0 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 10 monclient(hunting): handle_auth_done global_id 144411 payload 824 2024-09-17T15:42:04.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.899+0000 7f05e1ae7640 1 -- 172.21.15.86:0/2516016452 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05dc135a50 con 0x7f05dc11d860 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e1ae7640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 msgr2=0x7f05dc137f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e1ae7640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 0x7f05dc137f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05d8018000 tx=0x7f05d8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8006250 con 0x7f05dc11d860 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 10 monclient: dump: 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:04.915 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f05d8006250 con 0x7f05dc11d860 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 msgr2=0x7f05dc137f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 0x7f05dc137f00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05d8018000 tx=0x7f05d8004740 comp rx=0 tx=0).stop 2024-09-17T15:42:04.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 msgr2=0x7f05dc137f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05d2ffd640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05dc11d860 0x7f05dc137f00 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f05d8018000 tx=0x7f05d8004740 comp rx=0 tx=0).stop 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:42:04.917 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 142, 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 143246, 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:04.918 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.919 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:04.920 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:04.921 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:04.922 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:04.923 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.924 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:04.925 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:04.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:04.927 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.928 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:04.929 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:04.930 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:04.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:04.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:04.933 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:04.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:04.933 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05c003bb10 msgr2=0x7f05c003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f05c003bb10 0x7f05c003dfd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f05d400b7f0 tx=0x7f05d400d040 comp rx=0 tx=0).stop 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 10 monclient: shutdown 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 msgr2=0x7f05dc1cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 --2- 172.21.15.86:0/2516016452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05dc0a0c70 0x7f05dc1cdc20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f05cc0ac000 tx=0x7f05cc0191a0 comp rx=0 tx=0).stop 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e2ae9640 1 -- 172.21.15.86:0/2516016452 reap_dead start 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e2ae9640 1 -- 172.21.15.86:0/2516016452 reap_dead start 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 shutdown_connections 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 >> 172.21.15.86:0/2516016452 conn(0x7f05dc0a2f40 msgr2=0x7f05dc06fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 shutdown_connections 2024-09-17T15:42:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:04.903+0000 7f05e454c640 1 -- 172.21.15.86:0/2516016452 wait complete. 2024-09-17T15:42:04.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:42:04.936 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:42:04.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2234: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2234: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1099 (ObjNum 763 snap 236 seq_num 763) dirty exists 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2235: finishing rollback tid 0 to smithi20328086-16 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2235: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2235: finishing rollback tid 2 to smithi20328086-16 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1103 (ObjNum 678 snap 214 seq_num 678) dirty exists 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2236: finishing write tid 1 to smithi20328086-36 2024-09-17T15:42:04.937 INFO:tasks.rados.rados.0.smithi203.stdout:2236: finishing write tid 2 to smithi20328086-36 2024-09-17T15:42:04.938 INFO:tasks.rados.rados.0.smithi203.stdout:2237: expect (ObjNum 765 snap 236 seq_num 765) 2024-09-17T15:42:04.962 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1098061 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: finishing write tid 3 to smithi20328086-36 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: finishing write tid 4 to smithi20328086-36 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: finishing write tid 5 to smithi20328086-36 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: oid 36 updating version 0 to 1169 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: oid 36 updating version 1169 to 1170 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: oid 36 updating version 1170 to 1171 2024-09-17T15:42:05.007 INFO:tasks.rados.rados.0.smithi203.stdout:2236: oid 36 version 1171 is already newer than 1168 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1171 (ObjNum 789 snap 246 seq_num 789) dirty exists 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2236: left oid 36 (ObjNum 789 snap 246 seq_num 789) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2238: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2238: finishing copy_from to smithi20328086-10 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1063 (ObjNum 777 snap 242 seq_num 777) dirty exists 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2230: done (8 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2231: done (7 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2232: done (6 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2233: done (5 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2234: done (4 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2235: done (3 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2236: done (2 left) 2024-09-17T15:42:05.008 INFO:tasks.rados.rados.0.smithi203.stdout:2237: done (1 left) 2024-09-17T15:42:05.009 INFO:tasks.rados.rados.0.smithi203.stdout:2238: done (0 left) 2024-09-17T15:42:05.009 INFO:tasks.rados.rados.0.smithi203.stdout:2239: read oid 2 snap 233 2024-09-17T15:42:05.009 INFO:tasks.rados.rados.0.smithi203.stdout:2239: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:42:05.009 INFO:tasks.rados.rados.0.smithi203.stdout:2240: write oid 30 current snap is 246 2024-09-17T15:42:05.009 INFO:tasks.rados.rados.0.smithi203.stdout:2240: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-17T15:42:05.025 INFO:tasks.rados.rados.0.smithi203.stdout:2240: writing smithi20328086-30 from 477530 to 1108599 tid 1 2024-09-17T15:42:05.040 INFO:tasks.rados.rados.0.smithi203.stdout:2240: writing smithi20328086-30 from 1612657 to 2022263 tid 2 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: get_monmap_and_config 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: build_initial_monmap 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: monmap: 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:05.060118+0000 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:05.060118+0000 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:05.055 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:05.056 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 Processor -- start 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 -- start start 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: init 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f83400a50e0 0x7f834009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-c con 0x7f83400a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83400a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f834013d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83400a6280 con 0x7f83400a54c0 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834009c0a0 con 0x7f834013d2f0 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f834009d2a0 con 0x7f83400a50e0 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f8346c40640 10 monclient(hunting): authenticate will time out at 1858.025575s 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f83449dc640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f83400a50e0 msgr2=0x7f834009cd60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f83451dd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f834013d2f0 legacy=0x7f834013f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f83449dc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f83400a50e0 0x7f834009cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f833ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f83400a54c0 0x7f83400a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42098/0 (socket says 172.21.15.86:42098) 2024-09-17T15:42:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.059+0000 7f833ffff640 1 -- 172.21.15.86:0/4148716599 learned_addr learned my addr 172.21.15.86:0/4148716599 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:05.060 INFO:tasks.rados.rados.0.smithi203.stdout:2240: writing smithi20328086-30 from 2545223 to 3062819 tid 3 2024-09-17T15:42:05.060 INFO:tasks.rados.rados.0.smithi203.stdout:2241: write oid 28 current snap is 246 2024-09-17T15:42:05.060 INFO:tasks.rados.rados.0.smithi203.stdout:2241: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833ffff640 1 -- 172.21.15.86:0/4148716599 >> v1:172.21.15.86:6789/0 conn(0x7f83400a54c0 legacy=0x7f83400a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3260348741 0 0) 0x7f83400a6280 con 0x7f83400a54c0 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): my global_id is 144414 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8328003530 con 0x7f83400a54c0 2024-09-17T15:42:05.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1373158707 0 0) 0x7f8328003530 con 0x7f83400a54c0 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 >> v2:172.21.15.203:3300/0 conn(0x7f83400a50e0 msgr2=0x7f834009cd60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 --2- 172.21.15.86:0/4148716599 >> v2:172.21.15.203:3300/0 conn(0x7f83400a50e0 0x7f834009cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 >> v1:172.21.15.203:6789/0 conn(0x7f834013d2f0 legacy=0x7f834013f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:05.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834009c3d0 con 0x7f83400a54c0 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f83340019b0 con 0x7f83400a54c0 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: dump: 2024-09-17T15:42:05.065 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:05.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 >> v1:172.21.15.86:6789/0 conn(0x7f83400a54c0 legacy=0x7f83400a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8328003a80 0x7f8328003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): picked mon.c con 0x7f8328003a80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8328004810 0x7f8328004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): picked mon.b con 0x7f8328004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 0x7f832800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): picked mon.a con 0x7f83280099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f832800c3a0 con 0x7f83280099c0 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f832800c550 con 0x7f8328004810 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f832800c750 con 0x7f8328003a80 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83449dc640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8328003a80 0x7f8328003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83449dc640 10 monclient(hunting): get_auth_request con 0x7f8328003a80 auth_method 0 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83449dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83449dc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83449dc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:05.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 0x7f832800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833ffff640 1 -- 172.21.15.86:0/4148716599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8328004810 msgr2=0x7f8328004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:05.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f833ffff640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8328004810 0x7f8328004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:05.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): get_auth_request con 0x7f83280099c0 auth_method 0 2024-09-17T15:42:05.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:42:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient(hunting): handle_auth_done global_id 144414 payload 293 2024-09-17T15:42:05.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 -- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8328003a80 msgr2=0x7f8328003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:05.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8328003a80 0x7f8328003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 -- 172.21.15.86:0/4148716599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8328004810 msgr2=0x7f8328004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:05.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8328004810 0x7f8328004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient: _finish_hunting 0 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 monclient: found mon.a 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 -- 172.21.15.86:0/4148716599 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8328003530 con 0x7f83280099c0 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient: _finish_auth 0 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f8346c40640 5 monclient: authenticate success, global_id 144414 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f8346c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:05.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.063+0000 7f83451dd640 1 -- 172.21.15.86:0/4148716599 reap_dead start 2024-09-17T15:42:05.073 INFO:tasks.rados.rados.0.smithi203.stdout:2241: writing smithi20328086-28 from 494337 to 917008 tid 1 2024-09-17T15:42:05.080 INFO:tasks.rados.rados.0.smithi203.stdout:2241: writing smithi20328086-28 from 1387492 to 1940756 tid 2 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f83451dd640 1 -- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 msgr2=0x7f832800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f83451dd640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 0x7f832800bdb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f83380038c0 tx=0x7f833801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c0d335b5d4773c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338009470 con 0x7f83280099c0 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: dump: 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83380038f0 con 0x7f83280099c0 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 1 -- 172.21.15.86:0/4148716599 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338003ce0 con 0x7f83280099c0 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 4 set_mon_vals no callback set 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f833f7fe640 10 monclient: dump: 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.085 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 10 monclient: shutdown 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 msgr2=0x7f832800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 0x7f832800bdb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f83380038c0 tx=0x7f833801c340 comp rx=0 tx=0).stop 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- 172.21.15.86:0/4148716599 shutdown_connections 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 --2- 172.21.15.86:0/4148716599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83280099c0 0x7f832800bdb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- 172.21.15.86:0/4148716599 >> 172.21.15.86:0/4148716599 conn(0x7f83400aaae0 msgr2=0x7f83400aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- 172.21.15.86:0/4148716599 shutdown_connections 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- 172.21.15.86:0/4148716599 wait complete. 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 10 monclient: build_initial_monmap 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 10 monclient: monmap: 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:05.087 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 Processor -- start 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 1 -- start start 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.067+0000 7f8346c40640 10 monclient: init 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83280099c0 0x7f83401cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-b con 0x7f83280099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-c con 0x7f83401382d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 0x7f834013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): picked mon.noname-a con 0x7f834013da70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f834009d0f0 con 0x7f834013da70 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f834009cf70 con 0x7f83280099c0 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83400a58d0 con 0x7f83401382d0 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient(hunting): authenticate will time out at 1858.036099s 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83449dc640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83280099c0 msgr2=0x7f83401cf580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83449dc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83280099c0 0x7f83401cf580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 0x7f834013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52514/0 (socket says 172.21.15.86:52514) 2024-09-17T15:42:05.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 0x7f834013dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39196/0 (socket says 172.21.15.86:39196) 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 -- 172.21.15.86:0/3089015408 learned_addr learned my addr 172.21.15.86:0/3089015408 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): get_auth_request con 0x7f83401382d0 auth_method 0 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): get_auth_request con 0x7f834013da70 auth_method 0 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:05.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient(hunting): handle_auth_done global_id 164168 payload 824 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83280099c0 msgr2=0x7f83401cf580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83280099c0 0x7f83401cf580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 msgr2=0x7f834013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 0x7f834013dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 monclient: found mon.noname-c 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834013f330 con 0x7f83401382d0 2024-09-17T15:42:05.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f83451dd640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f834013da70 0x7f834013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 msgr2=0x7f8340138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833ffff640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8334001c90 tx=0x7f833400cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334007730 con 0x7f83401382d0 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: dump: 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833400ce80 con 0x7f83401382d0 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 5 monclient: authenticate success, global_id 164168 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _renew_subs 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834013ff30 con 0x7f83401382d0 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:05.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334004a30 con 0x7f83401382d0 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833d7fa640 10 monclient: dump: 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.094 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f833effd640 4 set_mon_vals no callback set 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _renew_subs 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.071+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8340120d50 con 0x7f83401382d0 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f8346c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f8346c40640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 0x7f83401392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83400a5d00 con 0x7f834013e400 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 0x7f83401392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient: get_auth_request con 0x7f834013e400 auth_method 0 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 10 monclient(hunting): handle_auth_done global_id 164171 payload 824 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8340121020 con 0x7f834013e400 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8334011070 con 0x7f83401382d0 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f832c03bb60 0x7f832c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(508..508 src has 438..508) ==== 4799+0+0 (secure 0 0 0) 0x7f833404fb00 con 0x7f83401382d0 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83451dd640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f832c03bb60 0x7f832c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83451dd640 10 monclient: get_auth_request con 0x7f832c03bb60 auth_method 0 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 msgr2=0x7f83401392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f83449dc640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 0x7f83401392d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8330003cc0 tx=0x7f833000b6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833000e810 con 0x7f834013e400 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 10 monclient: dump: 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f833000e810 con 0x7f834013e400 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 msgr2=0x7f83401392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 0x7f83401392d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8330003cc0 tx=0x7f833000b6e0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 msgr2=0x7f83401392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.075+0000 7f833d7fa640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f834013e400 0x7f83401392d0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8330003cc0 tx=0x7f833000b6e0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.079+0000 7f83451dd640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f832c03bb60 0x7f832c03e020 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f834006f800 tx=0x7f83380059c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f8346c40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f8346c40640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 0x7f8340139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:05.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f8346c40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8340121410 con 0x7f8340139810 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 0x7f8340139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient: get_auth_request con 0x7f8340139810 auth_method 0 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 10 monclient(hunting): handle_auth_done global_id 164174 payload 824 2024-09-17T15:42:05.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.083+0000 7f83449dc640 1 -- 172.21.15.86:0/3089015408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8340074be0 con 0x7f8340139810 2024-09-17T15:42:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f83449dc640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 msgr2=0x7f8340139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f83449dc640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 0x7f8340139c20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8330017000 tx=0x7f8330005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8330027440 con 0x7f8340139810 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 10 monclient: dump: 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:05.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 142, 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28275, 2024-09-17T15:42:05.102 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:05.103 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:05.104 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:05.105 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:05.106 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:05.107 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:05.109 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:05.110 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:05.111 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.112 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:05.113 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:05.114 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:05.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:05.115 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:05.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:05.115 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:05.115 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:05.115 INFO:tasks.rados.rados.0.smithi203.stdout:2241: writing smithi20328086-28 from 2688999 to 3134220 tid 3 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:2242: rollback oid 1 current snap is 246 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 245 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:2243: rollback oid 38 current snap is 246 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 243 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:2239: expect (ObjNum 696 snap 218 seq_num 696) 2024-09-17T15:42:05.116 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f8330027440 con 0x7f8340139810 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 msgr2=0x7f8340139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 0x7f8340139c20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8330017000 tx=0x7f8330005e10 comp rx=0 tx=0).stop 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 msgr2=0x7f8340139c20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f833d7fa640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8340139810 0x7f8340139c20 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8330017000 tx=0x7f8330005e10 comp rx=0 tx=0).stop 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f832c03bb60 msgr2=0x7f832c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f832c03bb60 0x7f832c03e020 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f834006f800 tx=0x7f83380059c0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 10 monclient: shutdown 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 msgr2=0x7f8340138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8334001c90 tx=0x7f833400cb70 comp rx=0 tx=0).stop 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f83451dd640 1 -- 172.21.15.86:0/3089015408 reap_dead start 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 shutdown_connections 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 --2- 172.21.15.86:0/3089015408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83401382d0 0x7f8340138750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 >> 172.21.15.86:0/3089015408 conn(0x7f83400aaae0 msgr2=0x7f834013c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 shutdown_connections 2024-09-17T15:42:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:05.087+0000 7f8346c40640 1 -- 172.21.15.86:0/3089015408 wait complete. 2024-09-17T15:42:05.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:42:05.120 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:42:05.274 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 82547 2024-09-17T15:42:05.360 INFO:tasks.rados.rados.0.smithi203.stdout:2240: finishing write tid 1 to smithi20328086-30 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: finishing write tid 2 to smithi20328086-30 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: finishing write tid 3 to smithi20328086-30 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: finishing write tid 4 to smithi20328086-30 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: finishing write tid 5 to smithi20328086-30 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: oid 30 updating version 0 to 1099 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: oid 30 updating version 1099 to 1100 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: oid 30 version 1100 is already newer than 1098 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: oid 30 updating version 1100 to 1101 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1101 (ObjNum 790 snap 246 seq_num 790) dirty exists 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2240: left oid 30 (ObjNum 790 snap 246 seq_num 790) 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2241: finishing write tid 1 to smithi20328086-28 2024-09-17T15:42:05.361 INFO:tasks.rados.rados.0.smithi203.stdout:2241: finishing write tid 2 to smithi20328086-28 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2242: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1031 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2243: finishing rollback tid 0 to smithi20328086-38 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2243: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2243: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1097 (ObjNum 772 snap 242 seq_num 772) dirty exists 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: finishing write tid 3 to smithi20328086-28 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: finishing write tid 4 to smithi20328086-28 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: finishing write tid 5 to smithi20328086-28 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: oid 28 updating version 0 to 1173 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: oid 28 updating version 1173 to 1174 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: oid 28 updating version 1174 to 1175 2024-09-17T15:42:05.362 INFO:tasks.rados.rados.0.smithi203.stdout:2241: oid 28 version 1175 is already newer than 1172 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1175 (ObjNum 791 snap 246 seq_num 791) dirty exists 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2241: left oid 28 (ObjNum 791 snap 246 seq_num 791) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2239: done (4 left) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2240: done (3 left) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2241: done (2 left) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2242: done (1 left) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2243: done (0 left) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2244: read oid 16 snap 244 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2244: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:05.363 INFO:tasks.rados.rados.0.smithi203.stdout:2245: snap_remove snap 233 2024-09-17T15:42:06.014 INFO:tasks.rados.rados.0.smithi203.stdout:2245: done (1 left) 2024-09-17T15:42:06.014 INFO:tasks.rados.rados.0.smithi203.stdout:2246: rollback oid 30 current snap is 246 2024-09-17T15:42:06.014 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 30 to 239 2024-09-17T15:42:06.014 INFO:tasks.rados.rados.0.smithi203.stdout:2247: write (excl) oid 12 current snap is 246 2024-09-17T15:42:06.014 INFO:tasks.rados.rados.0.smithi203.stdout:2247: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-17T15:42:06.020 INFO:tasks.rados.rados.0.smithi203.stdout:2247: writing smithi20328086-12 from 511144 to 1009064 tid 1 2024-09-17T15:42:06.025 INFO:tasks.rados.rados.0.smithi203.stdout:2247: writing smithi20328086-12 from 1445974 to 1859249 tid 2 2024-09-17T15:42:06.032 INFO:tasks.rados.rados.0.smithi203.stdout:2247: writing smithi20328086-12 from 2316422 to 2972915 tid 3 2024-09-17T15:42:06.032 INFO:tasks.rados.rados.0.smithi203.stdout:2248: snap_remove snap 243 2024-09-17T15:42:06.710 INFO:tasks.rados.rados.0.smithi203.stdout:2244: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:06.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:06.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2664293 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2246: finishing rollback tid 0 to smithi20328086-30 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2246: finishing rollback tid 1 to smithi20328086-30 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2246: finishing rollback tid 2 to smithi20328086-30 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1104 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2247: finishing write tid 1 to smithi20328086-12 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2247: finishing write tid 2 to smithi20328086-12 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2247: finishing write tid 3 to smithi20328086-12 2024-09-17T15:42:06.872 INFO:tasks.rados.rados.0.smithi203.stdout:2247: finishing write tid 4 to smithi20328086-12 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: finishing write tid 5 to smithi20328086-12 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: oid 12 updating version 0 to 1106 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: oid 12 updating version 1106 to 1108 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: oid 12 version 1108 is already newer than 1107 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: oid 12 version 1108 is already newer than 1105 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: left oid 12 (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2244: done (3 left) 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2246: done (2 left) 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2247: done (1 left) 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2248: done (0 left) 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2249: copy_from oid 46 from oid 35 current snap is 246 2024-09-17T15:42:06.873 INFO:tasks.rados.rados.0.smithi203.stdout:2250: read oid 44 snap -1 2024-09-17T15:42:06.874 INFO:tasks.rados.rados.0.smithi203.stdout:2250: expect (ObjNum 734 snap 230 seq_num 734) 2024-09-17T15:42:06.874 INFO:tasks.rados.rados.0.smithi203.stdout:2251: write oid 14 current snap is 246 2024-09-17T15:42:06.874 INFO:tasks.rados.rados.0.smithi203.stdout:2251: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-09-17T15:42:06.879 INFO:tasks.rados.rados.0.smithi203.stdout:2251: writing smithi20328086-14 from 527951 to 1101120 tid 1 2024-09-17T15:42:06.891 INFO:tasks.rados.rados.0.smithi203.stdout:2251: writing smithi20328086-14 from 1504456 to 2061389 tid 2 2024-09-17T15:42:06.909 INFO:tasks.rados.rados.0.smithi203.stdout:2251: writing smithi20328086-14 from 2743845 to 3211610 tid 3 2024-09-17T15:42:06.909 INFO:tasks.rados.rados.0.smithi203.stdout:2252: write oid 32 current snap is 246 2024-09-17T15:42:06.909 INFO:tasks.rados.rados.0.smithi203.stdout:2249: finishing copy_from to smithi20328086-46 2024-09-17T15:42:06.910 INFO:tasks.rados.rados.0.smithi203.stdout:2249: got expected ENOENT (src dne) 2024-09-17T15:42:06.910 INFO:tasks.rados.rados.0.smithi203.stdout:2249: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:42:06.910 INFO:tasks.rados.rados.0.smithi203.stdout:2250: expect (ObjNum 734 snap 230 seq_num 734) 2024-09-17T15:42:07.021 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:07.070 INFO:tasks.rados.rados.0.smithi203.stdout:2251: finishing write tid 1 to smithi20328086-14 2024-09-17T15:42:07.070 INFO:tasks.rados.rados.0.smithi203.stdout:2251: finishing write tid 2 to smithi20328086-14 2024-09-17T15:42:07.070 INFO:tasks.rados.rados.0.smithi203.stdout:2252: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-17T15:42:07.076 INFO:tasks.rados.rados.0.smithi203.stdout:2252: writing smithi20328086-32 from 544758 to 1193176 tid 1 2024-09-17T15:42:07.083 INFO:tasks.rados.rados.0.smithi203.stdout:2252: writing smithi20328086-32 from 1846585 to 2263529 tid 2 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2252: writing smithi20328086-32 from 2771268 to 3344758 tid 3 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2249: done (3 left) 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2250: done (2 left) 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2253: delete oid 10 current snap is 246 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: finishing write tid 3 to smithi20328086-14 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: finishing write tid 4 to smithi20328086-14 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: finishing write tid 5 to smithi20328086-14 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: oid 14 updating version 0 to 1068 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: oid 14 version 1068 is already newer than 1066 2024-09-17T15:42:07.089 INFO:tasks.rados.rados.0.smithi203.stdout:2251: oid 14 version 1068 is already newer than 1067 2024-09-17T15:42:07.090 INFO:tasks.rados.rados.0.smithi203.stdout:2251: oid 14 version 1068 is already newer than 1065 2024-09-17T15:42:07.090 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1068 (ObjNum 793 snap 246 seq_num 793) dirty exists 2024-09-17T15:42:07.090 INFO:tasks.rados.rados.0.smithi203.stdout:2251: left oid 14 (ObjNum 793 snap 246 seq_num 793) 2024-09-17T15:42:07.090 INFO:tasks.rados.rados.0.smithi203.stdout:2252: finishing write tid 1 to smithi20328086-32 2024-09-17T15:42:07.090 INFO:tasks.rados.rados.0.smithi203.stdout:2252: finishing write tid 2 to smithi20328086-32 2024-09-17T15:42:07.093 INFO:tasks.rados.rados.0.smithi203.stdout:2251: done (2 left) 2024-09-17T15:42:07.093 INFO:tasks.rados.rados.0.smithi203.stdout:2253: done (1 left) 2024-09-17T15:42:07.093 INFO:tasks.rados.rados.0.smithi203.stdout:2254: snap_create 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: finishing write tid 3 to smithi20328086-32 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: finishing write tid 4 to smithi20328086-32 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: finishing write tid 5 to smithi20328086-32 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: oid 32 updating version 0 to 1100 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: oid 32 version 1100 is already newer than 1099 2024-09-17T15:42:07.096 INFO:tasks.rados.rados.0.smithi203.stdout:2252: oid 32 updating version 1100 to 1101 2024-09-17T15:42:07.097 INFO:tasks.rados.rados.0.smithi203.stdout:2252: oid 32 version 1101 is already newer than 1098 2024-09-17T15:42:07.097 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1101 (ObjNum 794 snap 246 seq_num 794) dirty exists 2024-09-17T15:42:07.097 INFO:tasks.rados.rados.0.smithi203.stdout:2252: left oid 32 (ObjNum 794 snap 246 seq_num 794) 2024-09-17T15:42:08.023 INFO:tasks.rados.rados.0.smithi203.stdout:2252: done (1 left) 2024-09-17T15:42:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2254: done (0 left) 2024-09-17T15:42:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2255: delete oid 20 current snap is 247 2024-09-17T15:42:08.034 INFO:tasks.rados.rados.0.smithi203.stdout:2255: done (0 left) 2024-09-17T15:42:08.034 INFO:tasks.rados.rados.0.smithi203.stdout:2256: snap_create 2024-09-17T15:42:09.027 INFO:tasks.rados.rados.0.smithi203.stdout:2256: done (0 left) 2024-09-17T15:42:09.028 INFO:tasks.rados.rados.0.smithi203.stdout:2257: snap_remove snap 241 2024-09-17T15:42:10.030 INFO:tasks.rados.rados.0.smithi203.stdout:2257: done (0 left) 2024-09-17T15:42:10.030 INFO:tasks.rados.rados.0.smithi203.stdout:2258: write (excl) oid 19 current snap is 248 2024-09-17T15:42:10.030 INFO:tasks.rados.rados.0.smithi203.stdout:2258: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-17T15:42:10.049 INFO:tasks.rados.rados.0.smithi203.stdout:2258: writing smithi20328086-19 from 595179 to 1069344 tid 1 2024-09-17T15:42:10.069 INFO:tasks.rados.rados.0.smithi203.stdout:2258: writing smithi20328086-19 from 1622031 to 2186302 tid 2 2024-09-17T15:42:10.086 INFO:tasks.rados.rados.0.smithi203.stdout:2258: writing smithi20328086-19 from 2737184 to 3366390 tid 3 2024-09-17T15:42:10.086 INFO:tasks.rados.rados.0.smithi203.stdout:2259: delete oid 4 current snap is 248 2024-09-17T15:42:10.086 INFO:tasks.rados.rados.0.smithi203.stdout:2258: finishing write tid 1 to smithi20328086-19 2024-09-17T15:42:10.086 INFO:tasks.rados.rados.0.smithi203.stdout:2258: finishing write tid 2 to smithi20328086-19 2024-09-17T15:42:10.095 INFO:tasks.rados.rados.0.smithi203.stdout:2259: done (1 left) 2024-09-17T15:42:10.095 INFO:tasks.rados.rados.0.smithi203.stdout:2260: delete oid 29 current snap is 248 2024-09-17T15:42:10.097 INFO:tasks.rados.rados.0.smithi203.stdout:2258: finishing write tid 3 to smithi20328086-19 2024-09-17T15:42:10.097 INFO:tasks.rados.rados.0.smithi203.stdout:2258: finishing write tid 4 to smithi20328086-19 2024-09-17T15:42:10.098 INFO:tasks.rados.rados.0.smithi203.stdout:2258: finishing write tid 5 to smithi20328086-19 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:2258: oid 19 updating version 0 to 1116 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:2258: oid 19 version 1116 is already newer than 1115 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:2258: oid 19 updating version 1116 to 1117 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:2258: oid 19 updating version 1117 to 1118 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1118 (ObjNum 797 snap 248 seq_num 797) dirty exists 2024-09-17T15:42:10.099 INFO:tasks.rados.rados.0.smithi203.stdout:2258: left oid 19 (ObjNum 797 snap 248 seq_num 797) 2024-09-17T15:42:10.101 INFO:tasks.rados.rados.0.smithi203.stdout:2258: done (1 left) 2024-09-17T15:42:10.101 INFO:tasks.rados.rados.0.smithi203.stdout:2260: done (0 left) 2024-09-17T15:42:10.102 INFO:tasks.rados.rados.0.smithi203.stdout:2261: write (excl) oid 15 current snap is 248 2024-09-17T15:42:10.102 INFO:tasks.rados.rados.0.smithi203.stdout:2261: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-17T15:42:10.115 INFO:tasks.rados.rados.0.smithi203.stdout:2261: writing smithi20328086-15 from 645600 to 1061865 tid 1 2024-09-17T15:42:10.132 INFO:tasks.rados.rados.0.smithi203.stdout:2261: writing smithi20328086-15 from 1797477 to 2509075 tid 2 2024-09-17T15:42:10.142 INFO:tasks.rados.rados.0.smithi203.stdout:2261: writing smithi20328086-15 from 3219453 to 3445600 tid 3 2024-09-17T15:42:10.142 INFO:tasks.rados.rados.0.smithi203.stdout:2262: read oid 16 snap 244 2024-09-17T15:42:10.142 INFO:tasks.rados.rados.0.smithi203.stdout:2262: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:10.142 INFO:tasks.rados.rados.0.smithi203.stdout:2263: read oid 47 snap 239 2024-09-17T15:42:10.142 INFO:tasks.rados.rados.0.smithi203.stdout:2263: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2264: read oid 25 snap 232 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2264: expect (ObjNum 750 snap 231 seq_num 750) 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2265: write oid 6 current snap is 248 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2261: finishing write tid 1 to smithi20328086-15 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2261: finishing write tid 2 to smithi20328086-15 2024-09-17T15:42:10.143 INFO:tasks.rados.rados.0.smithi203.stdout:2265: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-09-17T15:42:10.155 INFO:tasks.rados.rados.0.smithi203.stdout:2265: writing smithi20328086-6 from 662407 to 1153921 tid 1 2024-09-17T15:42:10.168 INFO:tasks.rados.rados.0.smithi203.stdout:2265: writing smithi20328086-6 from 1855959 to 2427568 tid 2 2024-09-17T15:42:10.177 INFO:tasks.rados.rados.0.smithi203.stdout:2265: writing smithi20328086-6 from 2846876 to 3462407 tid 3 2024-09-17T15:42:10.177 INFO:tasks.rados.rados.0.smithi203.stdout:2266: write oid 43 current snap is 248 2024-09-17T15:42:10.177 INFO:tasks.rados.rados.0.smithi203.stdout:2266: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-17T15:42:10.186 INFO:tasks.rados.rados.0.smithi203.stdout:2266: writing smithi20328086-43 from 679214 to 1245977 tid 1 2024-09-17T15:42:10.195 INFO:tasks.rados.rados.0.smithi203.stdout:2266: writing smithi20328086-43 from 1798088 to 2513355 tid 2 2024-09-17T15:42:10.201 INFO:tasks.rados.rados.0.smithi203.stdout:2266: writing smithi20328086-43 from 3157946 to 3479214 tid 3 2024-09-17T15:42:10.201 INFO:tasks.rados.rados.0.smithi203.stdout:2267: copy_from oid 7 from oid 1 current snap is 248 2024-09-17T15:42:10.202 INFO:tasks.rados.rados.0.smithi203.stdout:2261: finishing write tid 3 to smithi20328086-15 2024-09-17T15:42:10.202 INFO:tasks.rados.rados.0.smithi203.stdout:2261: finishing write tid 4 to smithi20328086-15 2024-09-17T15:42:10.202 INFO:tasks.rados.rados.0.smithi203.stdout:2264: expect (ObjNum 750 snap 231 seq_num 750) 2024-09-17T15:42:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:2262: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:10.513 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 336693 2024-09-17T15:42:10.569 INFO:tasks.rados.rados.0.smithi203.stdout:2261: finishing write tid 5 to smithi20328086-15 2024-09-17T15:42:10.569 INFO:tasks.rados.rados.0.smithi203.stdout:2261: oid 15 updating version 0 to 1109 2024-09-17T15:42:10.569 INFO:tasks.rados.rados.0.smithi203.stdout:2261: oid 15 updating version 1109 to 1111 2024-09-17T15:42:10.569 INFO:tasks.rados.rados.0.smithi203.stdout:2261: oid 15 updating version 1111 to 1112 2024-09-17T15:42:10.570 INFO:tasks.rados.rados.0.smithi203.stdout:2261: oid 15 version 1112 is already newer than 1110 2024-09-17T15:42:10.570 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1112 (ObjNum 800 snap 248 seq_num 800) dirty exists 2024-09-17T15:42:10.570 INFO:tasks.rados.rados.0.smithi203.stdout:2261: left oid 15 (ObjNum 800 snap 248 seq_num 800) 2024-09-17T15:42:10.570 INFO:tasks.rados.rados.0.smithi203.stdout:2263: expect (ObjNum 506 snap 151 seq_num 506) 2024-09-17T15:42:10.584 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1701945 2024-09-17T15:42:10.599 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 123800 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: finishing write tid 1 to smithi20328086-6 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: finishing write tid 2 to smithi20328086-6 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: finishing write tid 3 to smithi20328086-6 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: finishing write tid 4 to smithi20328086-6 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: finishing write tid 5 to smithi20328086-6 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: oid 6 updating version 0 to 1075 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: oid 6 updating version 1075 to 1078 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: oid 6 version 1078 is already newer than 1077 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:2265: oid 6 version 1078 is already newer than 1076 2024-09-17T15:42:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1078 (ObjNum 801 snap 248 seq_num 801) dirty exists 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2265: left oid 6 (ObjNum 801 snap 248 seq_num 801) 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2267: finishing copy_from to smithi20328086-7 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2267: got expected ENOENT (src dne) 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2267: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: finishing write tid 1 to smithi20328086-43 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: finishing write tid 2 to smithi20328086-43 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: finishing write tid 3 to smithi20328086-43 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: finishing write tid 4 to smithi20328086-43 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: finishing write tid 5 to smithi20328086-43 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: oid 43 updating version 0 to 628 2024-09-17T15:42:10.678 INFO:tasks.rados.rados.0.smithi203.stdout:2266: oid 43 updating version 628 to 629 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2266: oid 43 version 629 is already newer than 627 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2266: oid 43 updating version 629 to 630 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 630 (ObjNum 802 snap 248 seq_num 802) dirty exists 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2266: left oid 43 (ObjNum 802 snap 248 seq_num 802) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2261: done (6 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2262: done (5 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2263: done (4 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2264: done (3 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2265: done (2 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2266: done (1 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2267: done (0 left) 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2268: write (excl) oid 15 current snap is 248 2024-09-17T15:42:10.679 INFO:tasks.rados.rados.0.smithi203.stdout:2268: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-17T15:42:10.688 INFO:tasks.rados.rados.0.smithi203.stdout:2268: writing smithi20328086-15 from 696021 to 1338033 tid 1 2024-09-17T15:42:10.700 INFO:tasks.rados.rados.0.smithi203.stdout:2268: writing smithi20328086-15 from 1856570 to 2431848 tid 2 2024-09-17T15:42:10.712 INFO:tasks.rados.rados.0.smithi203.stdout:2268: writing smithi20328086-15 from 3185369 to 3496021 tid 3 2024-09-17T15:42:10.712 INFO:tasks.rados.rados.0.smithi203.stdout:2269: write oid 44 current snap is 248 2024-09-17T15:42:10.712 INFO:tasks.rados.rados.0.smithi203.stdout:2268: finishing write tid 1 to smithi20328086-15 2024-09-17T15:42:10.712 INFO:tasks.rados.rados.0.smithi203.stdout:2268: finishing write tid 2 to smithi20328086-15 2024-09-17T15:42:10.712 INFO:tasks.rados.rados.0.smithi203.stdout:2269: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-17T15:42:10.727 INFO:tasks.rados.rados.0.smithi203.stdout:2269: writing smithi20328086-44 from 712828 to 1430089 tid 1 2024-09-17T15:42:10.740 INFO:tasks.rados.rados.0.smithi203.stdout:2269: writing smithi20328086-44 from 1915052 to 2633988 tid 2 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2269: writing smithi20328086-44 from 3212792 to 3512828 tid 3 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2270: copy_from oid 32 from oid 21 current snap is 248 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: finishing write tid 3 to smithi20328086-15 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: finishing write tid 4 to smithi20328086-15 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: finishing write tid 5 to smithi20328086-15 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: oid 15 updating version 0 to 1114 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: oid 15 version 1114 is already newer than 1113 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: oid 15 updating version 1114 to 1115 2024-09-17T15:42:10.749 INFO:tasks.rados.rados.0.smithi203.stdout:2268: oid 15 updating version 1115 to 1116 2024-09-17T15:42:10.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1116 (ObjNum 803 snap 248 seq_num 803) dirty exists 2024-09-17T15:42:10.750 INFO:tasks.rados.rados.0.smithi203.stdout:2268: left oid 15 (ObjNum 803 snap 248 seq_num 803) 2024-09-17T15:42:10.750 INFO:tasks.rados.rados.0.smithi203.stdout:2269: finishing write tid 1 to smithi20328086-44 2024-09-17T15:42:10.750 INFO:tasks.rados.rados.0.smithi203.stdout:2268: done (2 left) 2024-09-17T15:42:10.750 INFO:tasks.rados.rados.0.smithi203.stdout:2271: snap_remove snap 232 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: finishing write tid 2 to smithi20328086-44 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: finishing write tid 3 to smithi20328086-44 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: finishing write tid 4 to smithi20328086-44 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: finishing write tid 5 to smithi20328086-44 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: oid 44 updating version 0 to 1033 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: oid 44 updating version 1033 to 1035 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: oid 44 version 1035 is already newer than 1034 2024-09-17T15:42:11.057 INFO:tasks.rados.rados.0.smithi203.stdout:2269: oid 44 version 1035 is already newer than 1032 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1035 (ObjNum 804 snap 248 seq_num 804) dirty exists 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2269: left oid 44 (ObjNum 804 snap 248 seq_num 804) 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2270: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2270: finishing copy_from to smithi20328086-32 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1158 (ObjNum 786 snap 245 seq_num 786) dirty exists 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2269: done (2 left) 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2270: done (1 left) 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2271: done (0 left) 2024-09-17T15:42:11.058 INFO:tasks.rados.rados.0.smithi203.stdout:2272: delete oid 6 current snap is 248 2024-09-17T15:42:11.073 INFO:tasks.rados.rados.0.smithi203.stdout:2272: done (0 left) 2024-09-17T15:42:11.073 INFO:tasks.rados.rados.0.smithi203.stdout:2273: snap_remove snap 245 2024-09-17T15:42:11.715 INFO:tasks.rados.rados.0.smithi203.stdout:2273: done (0 left) 2024-09-17T15:42:11.715 INFO:tasks.rados.rados.0.smithi203.stdout:2274: copy_from oid 9 from oid 4 current snap is 248 2024-09-17T15:42:11.716 INFO:tasks.rados.rados.0.smithi203.stdout:2275: snap_create 2024-09-17T15:42:11.724 INFO:tasks.rados.rados.0.smithi203.stdout:2274: finishing copy_from to smithi20328086-9 2024-09-17T15:42:11.724 INFO:tasks.rados.rados.0.smithi203.stdout:2274: got expected ENOENT (src dne) 2024-09-17T15:42:11.724 INFO:tasks.rados.rados.0.smithi203.stdout:2274: finishing copy_from racing read to smithi20328086-9 2024-09-17T15:42:12.718 INFO:tasks.rados.rados.0.smithi203.stdout:2274: done (1 left) 2024-09-17T15:42:12.719 INFO:tasks.rados.rados.0.smithi203.stdout:2275: done (0 left) 2024-09-17T15:42:12.719 INFO:tasks.rados.rados.0.smithi203.stdout:2276: write (excl) oid 1 current snap is 249 2024-09-17T15:42:12.719 INFO:tasks.rados.rados.0.smithi203.stdout:2276: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-17T15:42:12.734 INFO:tasks.rados.rados.0.smithi203.stdout:2276: writing smithi20328086-1 from 746442 to 1330554 tid 1 2024-09-17T15:42:12.749 INFO:tasks.rados.rados.0.smithi203.stdout:2276: writing smithi20328086-1 from 1748369 to 2470974 tid 2 2024-09-17T15:42:12.763 INFO:tasks.rados.rados.0.smithi203.stdout:2276: writing smithi20328086-1 from 2983991 to 3546442 tid 3 2024-09-17T15:42:12.763 INFO:tasks.rados.rados.0.smithi203.stdout:2277: write (excl) oid 46 current snap is 249 2024-09-17T15:42:12.764 INFO:tasks.rados.rados.0.smithi203.stdout:2277: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-17T15:42:12.779 INFO:tasks.rados.rados.0.smithi203.stdout:2277: writing smithi20328086-46 from 763249 to 1422610 tid 1 2024-09-17T15:42:12.790 INFO:tasks.rados.rados.0.smithi203.stdout:2277: writing smithi20328086-46 from 2090498 to 2673114 tid 2 2024-09-17T15:42:12.797 INFO:tasks.rados.rados.0.smithi203.stdout:2277: writing smithi20328086-46 from 3295061 to 3563249 tid 3 2024-09-17T15:42:12.797 INFO:tasks.rados.rados.0.smithi203.stdout:2278: snap_create 2024-09-17T15:42:12.797 INFO:tasks.rados.rados.0.smithi203.stdout:2276: finishing write tid 1 to smithi20328086-1 2024-09-17T15:42:12.797 INFO:tasks.rados.rados.0.smithi203.stdout:2276: finishing write tid 2 to smithi20328086-1 2024-09-17T15:42:12.797 INFO:tasks.rados.rados.0.smithi203.stdout:2276: finishing write tid 3 to smithi20328086-1 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: finishing write tid 4 to smithi20328086-1 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: finishing write tid 5 to smithi20328086-1 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: oid 1 updating version 0 to 1087 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: oid 1 updating version 1087 to 1089 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: oid 1 version 1089 is already newer than 1086 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: oid 1 version 1089 is already newer than 1088 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1089 (ObjNum 806 snap 249 seq_num 806) dirty exists 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2276: left oid 1 (ObjNum 806 snap 249 seq_num 806) 2024-09-17T15:42:12.798 INFO:tasks.rados.rados.0.smithi203.stdout:2277: finishing write tid 1 to smithi20328086-46 2024-09-17T15:42:12.802 INFO:tasks.rados.rados.0.smithi203.stdout:2277: finishing write tid 2 to smithi20328086-46 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: finishing write tid 3 to smithi20328086-46 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: finishing write tid 4 to smithi20328086-46 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: finishing write tid 5 to smithi20328086-46 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: oid 46 updating version 0 to 945 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: oid 46 updating version 945 to 946 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: oid 46 version 946 is already newer than 943 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: oid 46 version 946 is already newer than 944 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 946 (ObjNum 807 snap 249 seq_num 807) dirty exists 2024-09-17T15:42:12.803 INFO:tasks.rados.rados.0.smithi203.stdout:2277: left oid 46 (ObjNum 807 snap 249 seq_num 807) 2024-09-17T15:42:13.725 INFO:tasks.rados.rados.0.smithi203.stdout:2276: done (2 left) 2024-09-17T15:42:13.725 INFO:tasks.rados.rados.0.smithi203.stdout:2277: done (1 left) 2024-09-17T15:42:13.725 INFO:tasks.rados.rados.0.smithi203.stdout:2278: done (0 left) 2024-09-17T15:42:13.725 INFO:tasks.rados.rados.0.smithi203.stdout:2279: write oid 30 current snap is 250 2024-09-17T15:42:13.725 INFO:tasks.rados.rados.0.smithi203.stdout:2279: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-17T15:42:13.736 INFO:tasks.rados.rados.0.smithi203.stdout:2279: writing smithi20328086-30 from 780056 to 1514666 tid 1 2024-09-17T15:42:13.746 INFO:tasks.rados.rados.0.smithi203.stdout:2279: writing smithi20328086-30 from 2148980 to 2875254 tid 2 2024-09-17T15:42:13.751 INFO:tasks.rados.rados.0.smithi203.stdout:2279: writing smithi20328086-30 from 3322484 to 3580056 tid 3 2024-09-17T15:42:13.751 INFO:tasks.rados.rados.0.smithi203.stdout:2280: copy_from oid 22 from oid 41 current snap is 250 2024-09-17T15:42:13.751 INFO:tasks.rados.rados.0.smithi203.stdout:2281: read oid 24 snap -1 2024-09-17T15:42:13.751 INFO:tasks.rados.rados.0.smithi203.stdout:2281: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:13.752 INFO:tasks.rados.rados.0.smithi203.stdout:2282: delete oid 45 current snap is 250 2024-09-17T15:42:13.752 INFO:tasks.rados.rados.0.smithi203.stdout:2279: finishing write tid 1 to smithi20328086-30 2024-09-17T15:42:13.757 INFO:tasks.rados.rados.0.smithi203.stdout:2279: finishing write tid 2 to smithi20328086-30 2024-09-17T15:42:13.757 INFO:tasks.rados.rados.0.smithi203.stdout:2279: finishing write tid 3 to smithi20328086-30 2024-09-17T15:42:13.760 INFO:tasks.rados.rados.0.smithi203.stdout:2279: finishing write tid 4 to smithi20328086-30 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: finishing write tid 5 to smithi20328086-30 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: oid 30 updating version 0 to 1118 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: oid 30 updating version 1118 to 1119 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: oid 30 updating version 1119 to 1120 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: oid 30 version 1120 is already newer than 1117 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1120 (ObjNum 808 snap 250 seq_num 808) dirty exists 2024-09-17T15:42:13.761 INFO:tasks.rados.rados.0.smithi203.stdout:2279: left oid 30 (ObjNum 808 snap 250 seq_num 808) 2024-09-17T15:42:13.764 INFO:tasks.rados.rados.0.smithi203.stdout:2279: done (3 left) 2024-09-17T15:42:13.764 INFO:tasks.rados.rados.0.smithi203.stdout:2282: done (2 left) 2024-09-17T15:42:13.764 INFO:tasks.rados.rados.0.smithi203.stdout:2283: copy_from oid 43 from oid 11 current snap is 250 2024-09-17T15:42:13.764 INFO:tasks.rados.rados.0.smithi203.stdout:2284: snap_create 2024-09-17T15:42:13.766 INFO:tasks.rados.rados.0.smithi203.stdout:2281: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:13.766 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:13.851 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:13.925 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1430188 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:2283: finishing copy_from to smithi20328086-43 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:2283: got expected ENOENT (src dne) 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:2283: finishing copy_from racing read to smithi20328086-43 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:2280: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:2280: finishing copy_from to smithi20328086-22 2024-09-17T15:42:13.961 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 626 (ObjNum 754 snap 232 seq_num 754) dirty exists 2024-09-17T15:42:14.734 INFO:tasks.rados.rados.0.smithi203.stdout:2280: done (3 left) 2024-09-17T15:42:14.734 INFO:tasks.rados.rados.0.smithi203.stdout:2281: done (2 left) 2024-09-17T15:42:14.734 INFO:tasks.rados.rados.0.smithi203.stdout:2283: done (1 left) 2024-09-17T15:42:14.734 INFO:tasks.rados.rados.0.smithi203.stdout:2284: done (0 left) 2024-09-17T15:42:14.735 INFO:tasks.rados.rados.0.smithi203.stdout:2285: read oid 28 snap -1 2024-09-17T15:42:14.735 INFO:tasks.rados.rados.0.smithi203.stdout:2285: expect (ObjNum 791 snap 246 seq_num 791) 2024-09-17T15:42:14.735 INFO:tasks.rados.rados.0.smithi203.stdout:2286: write oid 20 current snap is 251 2024-09-17T15:42:14.735 INFO:tasks.rados.rados.0.smithi203.stdout:2286: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-17T15:42:14.743 INFO:tasks.rados.rados.0.smithi203.stdout:2286: writing smithi20328086-20 from 413670 to 898778 tid 1 2024-09-17T15:42:14.755 INFO:tasks.rados.rados.0.smithi203.stdout:2286: writing smithi20328086-20 from 1465944 to 2195887 tid 2 2024-09-17T15:42:14.767 INFO:tasks.rados.rados.0.smithi203.stdout:2286: writing smithi20328086-20 from 2977330 to 3502131 tid 3 2024-09-17T15:42:14.767 INFO:tasks.rados.rados.0.smithi203.stdout:2287: snap_create 2024-09-17T15:42:14.767 INFO:tasks.rados.rados.0.smithi203.stdout:2286: finishing write tid 1 to smithi20328086-20 2024-09-17T15:42:14.767 INFO:tasks.rados.rados.0.smithi203.stdout:2285: expect (ObjNum 791 snap 246 seq_num 791) 2024-09-17T15:42:14.768 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:14.815 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 194766 2024-09-17T15:42:14.889 INFO:tasks.rados.rados.0.smithi203.stdout:2286: finishing write tid 2 to smithi20328086-20 2024-09-17T15:42:14.889 INFO:tasks.rados.rados.0.smithi203.stdout:2286: finishing write tid 3 to smithi20328086-20 2024-09-17T15:42:14.889 INFO:tasks.rados.rados.0.smithi203.stdout:2286: finishing write tid 4 to smithi20328086-20 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: finishing write tid 5 to smithi20328086-20 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: oid 20 updating version 0 to 1092 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: oid 20 version 1092 is already newer than 1091 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: oid 20 version 1092 is already newer than 1090 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: oid 20 updating version 1092 to 1093 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1093 (ObjNum 810 snap 251 seq_num 810) dirty exists 2024-09-17T15:42:14.890 INFO:tasks.rados.rados.0.smithi203.stdout:2286: left oid 20 (ObjNum 810 snap 251 seq_num 810) 2024-09-17T15:42:15.735 INFO:tasks.rados.rados.0.smithi203.stdout:2285: done (2 left) 2024-09-17T15:42:15.735 INFO:tasks.rados.rados.0.smithi203.stdout:2286: done (1 left) 2024-09-17T15:42:15.736 INFO:tasks.rados.rados.0.smithi203.stdout:2287: done (0 left) 2024-09-17T15:42:15.736 INFO:tasks.rados.rados.0.smithi203.stdout:2288: write (excl) oid 45 current snap is 252 2024-09-17T15:42:15.736 INFO:tasks.rados.rados.0.smithi203.stdout:2288: seq_num 811 ranges {430477=560357,1408073=473601,2488400=736073} 2024-09-17T15:42:15.745 INFO:tasks.rados.rados.0.smithi203.stdout:2288: writing smithi20328086-45 from 430477 to 990834 tid 1 2024-09-17T15:42:15.754 INFO:tasks.rados.rados.0.smithi203.stdout:2288: writing smithi20328086-45 from 1408073 to 1881674 tid 2 2024-09-17T15:42:15.768 INFO:tasks.rados.rados.0.smithi203.stdout:2288: writing smithi20328086-45 from 2488400 to 3224473 tid 3 2024-09-17T15:42:15.768 INFO:tasks.rados.rados.0.smithi203.stdout:2289: read oid 31 snap -1 2024-09-17T15:42:15.768 INFO:tasks.rados.rados.0.smithi203.stdout:2289: expect (ObjNum 766 snap 238 seq_num 766) 2024-09-17T15:42:15.768 INFO:tasks.rados.rados.0.smithi203.stdout:2290: snap_remove snap 250 2024-09-17T15:42:16.751 INFO:tasks.rados.rados.0.smithi203.stdout:2290: done (2 left) 2024-09-17T15:42:16.751 INFO:tasks.rados.rados.0.smithi203.stdout:2291: read oid 30 snap 247 2024-09-17T15:42:16.751 INFO:tasks.rados.rados.0.smithi203.stdout:2291: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:42:16.751 INFO:tasks.rados.rados.0.smithi203.stdout:2292: read oid 5 snap 251 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2292: expect deleted 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2293: copy_from oid 17 from oid 14 current snap is 252 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: finishing write tid 1 to smithi20328086-45 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: finishing write tid 2 to smithi20328086-45 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: finishing write tid 3 to smithi20328086-45 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: finishing write tid 4 to smithi20328086-45 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: finishing write tid 5 to smithi20328086-45 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: oid 45 updating version 0 to 1069 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: oid 45 version 1069 is already newer than 1068 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: oid 45 version 1069 is already newer than 1067 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: oid 45 version 1069 is already newer than 1066 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1069 (ObjNum 811 snap 252 seq_num 811) dirty exists 2024-09-17T15:42:16.752 INFO:tasks.rados.rados.0.smithi203.stdout:2288: left oid 45 (ObjNum 811 snap 252 seq_num 811) 2024-09-17T15:42:16.753 INFO:tasks.rados.rados.0.smithi203.stdout:2289: expect (ObjNum 766 snap 238 seq_num 766) 2024-09-17T15:42:16.767 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1125122 2024-09-17T15:42:16.814 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:16.918 INFO:tasks.rados.rados.0.smithi203.stdout:2288: done (4 left) 2024-09-17T15:42:16.918 INFO:tasks.rados.rados.0.smithi203.stdout:2289: done (3 left) 2024-09-17T15:42:16.918 INFO:tasks.rados.rados.0.smithi203.stdout:2292: done (2 left) 2024-09-17T15:42:16.918 INFO:tasks.rados.rados.0.smithi203.stdout:2294: snap_remove snap 251 2024-09-17T15:42:17.754 INFO:tasks.rados.rados.0.smithi203.stdout:2294: done (2 left) 2024-09-17T15:42:17.754 INFO:tasks.rados.rados.0.smithi203.stdout:2295: write (excl) oid 36 current snap is 252 2024-09-17T15:42:17.754 INFO:tasks.rados.rados.0.smithi203.stdout:2295: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-17T15:42:17.769 INFO:tasks.rados.rados.0.smithi203.stdout:2295: writing smithi20328086-36 from 447284 to 1082890 tid 1 2024-09-17T15:42:17.791 INFO:tasks.rados.rados.0.smithi203.stdout:2295: writing smithi20328086-36 from 1866555 to 2600167 tid 2 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2295: writing smithi20328086-36 from 3315823 to 3647284 tid 3 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2296: read oid 44 snap 244 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2296: expect (ObjNum 734 snap 230 seq_num 734) 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2297: rollback oid 1 current snap is 252 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 246 2024-09-17T15:42:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2291: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:42:17.968 INFO:tasks.rados.rados.0.smithi203.stdout:2293: finishing copy_from racing read to smithi20328086-17 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2293: finishing copy_from to smithi20328086-17 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1068 (ObjNum 793 snap 246 seq_num 793) dirty exists 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: finishing write tid 1 to smithi20328086-36 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: finishing write tid 2 to smithi20328086-36 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: finishing write tid 3 to smithi20328086-36 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: finishing write tid 4 to smithi20328086-36 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: finishing write tid 5 to smithi20328086-36 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: oid 36 updating version 0 to 1207 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: oid 36 updating version 1207 to 1208 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: oid 36 version 1208 is already newer than 1206 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: oid 36 version 1208 is already newer than 1205 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1208 (ObjNum 812 snap 252 seq_num 812) dirty exists 2024-09-17T15:42:17.969 INFO:tasks.rados.rados.0.smithi203.stdout:2295: left oid 36 (ObjNum 812 snap 252 seq_num 812) 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2291: done (4 left) 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2293: done (3 left) 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2295: done (2 left) 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2298: read oid 23 snap -1 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2298: expect deleted 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2299: rollback oid 35 current snap is 252 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 248 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2297: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:42:17.970 INFO:tasks.rados.rados.0.smithi203.stdout:2296: expect (ObjNum 734 snap 230 seq_num 734) 2024-09-17T15:42:18.025 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1280977 2024-09-17T15:42:18.064 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 972372 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:2297: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1096 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:2299: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 0 (ObjNum 33 snap 0 seq_num 3607061936) dirty dne 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:2296: done (3 left) 2024-09-17T15:42:18.101 INFO:tasks.rados.rados.0.smithi203.stdout:2297: done (2 left) 2024-09-17T15:42:18.102 INFO:tasks.rados.rados.0.smithi203.stdout:2298: done (1 left) 2024-09-17T15:42:18.102 INFO:tasks.rados.rados.0.smithi203.stdout:2299: done (0 left) 2024-09-17T15:42:18.102 INFO:tasks.rados.rados.0.smithi203.stdout:2300: snap_remove snap 246 2024-09-17T15:42:18.769 INFO:tasks.rados.rados.0.smithi203.stdout:2300: done (0 left) 2024-09-17T15:42:18.769 INFO:tasks.rados.rados.0.smithi203.stdout:2301: read oid 6 snap -1 2024-09-17T15:42:18.770 INFO:tasks.rados.rados.0.smithi203.stdout:2301: expect deleted 2024-09-17T15:42:18.770 INFO:tasks.rados.rados.0.smithi203.stdout:2302: rollback oid 31 current snap is 252 2024-09-17T15:42:18.770 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 247 2024-09-17T15:42:18.770 INFO:tasks.rados.rados.0.smithi203.stdout:2303: delete oid 13 current snap is 252 2024-09-17T15:42:18.796 INFO:tasks.rados.rados.0.smithi203.stdout:2301: done (2 left) 2024-09-17T15:42:18.797 INFO:tasks.rados.rados.0.smithi203.stdout:2303: done (1 left) 2024-09-17T15:42:18.797 INFO:tasks.rados.rados.0.smithi203.stdout:2304: read oid 15 snap 242 2024-09-17T15:42:18.797 INFO:tasks.rados.rados.0.smithi203.stdout:2304: expect deleted 2024-09-17T15:42:18.797 INFO:tasks.rados.rados.0.smithi203.stdout:2305: write oid 30 current snap is 252 2024-09-17T15:42:18.797 INFO:tasks.rados.rados.0.smithi203.stdout:2305: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-17T15:42:18.802 INFO:tasks.rados.rados.0.smithi203.stdout:2305: writing smithi20328086-30 from 480898 to 983355 tid 1 2024-09-17T15:42:18.810 INFO:tasks.rados.rados.0.smithi203.stdout:2305: writing smithi20328086-30 from 1699872 to 2437153 tid 2 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2305: writing smithi20328086-30 from 3087022 to 3680898 tid 3 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2306: read oid 12 snap 239 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2306: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2307: rollback oid 19 current snap is 252 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 242 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2302: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2302: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2302: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 629 (ObjNum 766 snap 238 seq_num 766) dirty exists 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2305: finishing write tid 1 to smithi20328086-30 2024-09-17T15:42:18.818 INFO:tasks.rados.rados.0.smithi203.stdout:2302: done (4 left) 2024-09-17T15:42:18.819 INFO:tasks.rados.rados.0.smithi203.stdout:2304: done (3 left) 2024-09-17T15:42:18.819 INFO:tasks.rados.rados.0.smithi203.stdout:2308: write (excl) oid 7 current snap is 252 2024-09-17T15:42:18.819 INFO:tasks.rados.rados.0.smithi203.stdout:2308: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-17T15:42:18.824 INFO:tasks.rados.rados.0.smithi203.stdout:2308: writing smithi20328086-7 from 497705 to 1075411 tid 1 2024-09-17T15:42:18.830 INFO:tasks.rados.rados.0.smithi203.stdout:2308: writing smithi20328086-7 from 1642001 to 2122940 tid 2 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2308: writing smithi20328086-7 from 2598092 to 3095606 tid 3 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2309: snap_create 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2305: finishing write tid 2 to smithi20328086-30 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2305: finishing write tid 3 to smithi20328086-30 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2307: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:2306: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:42:18.837 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:19.073 INFO:tasks.rados.rados.0.smithi203.stdout:2308: finishing write tid 1 to smithi20328086-7 2024-09-17T15:42:19.073 INFO:tasks.rados.rados.0.smithi203.stdout:2305: finishing write tid 4 to smithi20328086-30 2024-09-17T15:42:19.073 INFO:tasks.rados.rados.0.smithi203.stdout:2305: finishing write tid 5 to smithi20328086-30 2024-09-17T15:42:19.073 INFO:tasks.rados.rados.0.smithi203.stdout:2305: oid 30 updating version 0 to 1121 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:2305: oid 30 updating version 1121 to 1124 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:2305: oid 30 version 1124 is already newer than 1122 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:2305: oid 30 version 1124 is already newer than 1123 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1124 (ObjNum 814 snap 252 seq_num 814) dirty exists 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:2305: left oid 30 (ObjNum 814 snap 252 seq_num 814) 2024-09-17T15:42:19.074 INFO:tasks.rados.rados.0.smithi203.stdout:2307: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2307: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1128 (ObjNum 763 snap 236 seq_num 763) dirty exists 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: finishing write tid 2 to smithi20328086-7 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: finishing write tid 3 to smithi20328086-7 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: finishing write tid 4 to smithi20328086-7 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: finishing write tid 5 to smithi20328086-7 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: oid 7 updating version 0 to 1083 2024-09-17T15:42:19.075 INFO:tasks.rados.rados.0.smithi203.stdout:2308: oid 7 version 1083 is already newer than 1080 2024-09-17T15:42:19.076 INFO:tasks.rados.rados.0.smithi203.stdout:2308: oid 7 version 1083 is already newer than 1082 2024-09-17T15:42:19.076 INFO:tasks.rados.rados.0.smithi203.stdout:2308: oid 7 version 1083 is already newer than 1081 2024-09-17T15:42:19.076 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1083 (ObjNum 815 snap 252 seq_num 815) dirty exists 2024-09-17T15:42:19.076 INFO:tasks.rados.rados.0.smithi203.stdout:2308: left oid 7 (ObjNum 815 snap 252 seq_num 815) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2305: done (4 left) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2306: done (3 left) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2307: done (2 left) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2308: done (1 left) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2309: done (0 left) 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2310: read oid 5 snap 239 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2310: expect deleted 2024-09-17T15:42:19.771 INFO:tasks.rados.rados.0.smithi203.stdout:2311: snap_remove snap 242 2024-09-17T15:42:20.784 INFO:tasks.rados.rados.0.smithi203.stdout:2311: done (1 left) 2024-09-17T15:42:20.784 INFO:tasks.rados.rados.0.smithi203.stdout:2312: snap_remove snap 248 2024-09-17T15:42:21.713 INFO:tasks.rados.rados.0.smithi203.stdout:2312: done (1 left) 2024-09-17T15:42:21.713 INFO:tasks.rados.rados.0.smithi203.stdout:2313: copy_from oid 40 from oid 32 current snap is 253 2024-09-17T15:42:21.713 INFO:tasks.rados.rados.0.smithi203.stdout:2310: done (1 left) 2024-09-17T15:42:21.713 INFO:tasks.rados.rados.0.smithi203.stdout:2314: snap_create 2024-09-17T15:42:21.755 INFO:tasks.rados.rados.0.smithi203.stdout:2313: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:42:21.760 INFO:tasks.rados.rados.0.smithi203.stdout:2313: finishing copy_from to smithi20328086-40 2024-09-17T15:42:21.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1158 (ObjNum 786 snap 245 seq_num 786) dirty exists 2024-09-17T15:42:22.713 INFO:tasks.rados.rados.0.smithi203.stdout:2313: done (1 left) 2024-09-17T15:42:22.713 INFO:tasks.rados.rados.0.smithi203.stdout:2314: done (0 left) 2024-09-17T15:42:22.713 INFO:tasks.rados.rados.0.smithi203.stdout:2315: write oid 40 current snap is 254 2024-09-17T15:42:22.714 INFO:tasks.rados.rados.0.smithi203.stdout:2315: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-09-17T15:42:22.732 INFO:tasks.rados.rados.0.smithi203.stdout:2315: writing smithi20328086-40 from 514512 to 1167467 tid 1 2024-09-17T15:42:22.753 INFO:tasks.rados.rados.0.smithi203.stdout:2315: writing smithi20328086-40 from 1700483 to 2441433 tid 2 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2315: writing smithi20328086-40 from 3025515 to 3714512 tid 3 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2316: copy_from oid 10 from oid 12 current snap is 254 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2315: finishing write tid 1 to smithi20328086-40 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2315: finishing write tid 2 to smithi20328086-40 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2317: write (excl) oid 27 current snap is 254 2024-09-17T15:42:22.772 INFO:tasks.rados.rados.0.smithi203.stdout:2317: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2024-09-17T15:42:22.785 INFO:tasks.rados.rados.0.smithi203.stdout:2317: writing smithi20328086-27 from 531319 to 1259523 tid 1 2024-09-17T15:42:22.794 INFO:tasks.rados.rados.0.smithi203.stdout:2317: writing smithi20328086-27 from 1758965 to 2243573 tid 2 2024-09-17T15:42:22.803 INFO:tasks.rados.rados.0.smithi203.stdout:2317: writing smithi20328086-27 from 2652938 to 3172996 tid 3 2024-09-17T15:42:22.803 INFO:tasks.rados.rados.0.smithi203.stdout:2318: write (excl) oid 17 current snap is 254 2024-09-17T15:42:22.803 INFO:tasks.rados.rados.0.smithi203.stdout:2318: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-17T15:42:22.811 INFO:tasks.rados.rados.0.smithi203.stdout:2318: writing smithi20328086-17 from 548126 to 951579 tid 1 2024-09-17T15:42:22.817 INFO:tasks.rados.rados.0.smithi203.stdout:2318: writing smithi20328086-17 from 1417447 to 2162066 tid 2 2024-09-17T15:42:22.824 INFO:tasks.rados.rados.0.smithi203.stdout:2318: writing smithi20328086-17 from 2680361 to 3128044 tid 3 2024-09-17T15:42:22.829 INFO:tasks.rados.rados.0.smithi203.stdout:2318: writing smithi20328086-17 from 3680450 to 3748126 tid 4 2024-09-17T15:42:22.829 INFO:tasks.rados.rados.0.smithi203.stdout:2319: copy_from oid 42 from oid 20 current snap is 254 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: finishing write tid 3 to smithi20328086-40 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: finishing write tid 4 to smithi20328086-40 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: finishing write tid 5 to smithi20328086-40 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: oid 40 updating version 0 to 1161 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: oid 40 version 1161 is already newer than 1160 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: oid 40 version 1161 is already newer than 1159 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: oid 40 updating version 1161 to 1162 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1162 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: left oid 40 (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2317: finishing write tid 1 to smithi20328086-27 2024-09-17T15:42:22.830 INFO:tasks.rados.rados.0.smithi203.stdout:2315: done (4 left) 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2320: rollback oid 31 current snap is 254 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 249 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2317: finishing write tid 2 to smithi20328086-27 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2316: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 1 to smithi20328086-17 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2317: finishing write tid 3 to smithi20328086-27 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2317: finishing write tid 4 to smithi20328086-27 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2316: finishing copy_from to smithi20328086-10 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:42:22.831 INFO:tasks.rados.rados.0.smithi203.stdout:2317: finishing write tid 5 to smithi20328086-27 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: oid 27 updating version 0 to 948 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: oid 27 updating version 948 to 949 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: oid 27 version 949 is already newer than 947 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: oid 27 updating version 949 to 950 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 950 (ObjNum 817 snap 254 seq_num 817) dirty exists 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: left oid 27 (ObjNum 817 snap 254 seq_num 817) 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 2 to smithi20328086-17 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2316: done (4 left) 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2317: done (3 left) 2024-09-17T15:42:22.832 INFO:tasks.rados.rados.0.smithi203.stdout:2321: write oid 25 current snap is 254 2024-09-17T15:42:22.833 INFO:tasks.rados.rados.0.smithi203.stdout:2321: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-09-17T15:42:22.835 INFO:tasks.rados.rados.0.smithi203.stdout:2321: writing smithi20328086-25 from 564933 to 1043635 tid 1 2024-09-17T15:42:22.844 INFO:tasks.rados.rados.0.smithi203.stdout:2321: writing smithi20328086-25 from 1759576 to 2247853 tid 2 2024-09-17T15:42:22.854 INFO:tasks.rados.rados.0.smithi203.stdout:2321: writing smithi20328086-25 from 2991431 to 3650386 tid 3 2024-09-17T15:42:22.855 INFO:tasks.rados.rados.0.smithi203.stdout:2322: write oid 5 current snap is 254 2024-09-17T15:42:22.855 INFO:tasks.rados.rados.0.smithi203.stdout:2322: seq_num 820 ranges {581740=553951,1818058=748288,3018854=470227} 2024-09-17T15:42:22.862 INFO:tasks.rados.rados.0.smithi203.stdout:2322: writing smithi20328086-5 from 581740 to 1135691 tid 1 2024-09-17T15:42:22.873 INFO:tasks.rados.rados.0.smithi203.stdout:2322: writing smithi20328086-5 from 1818058 to 2566346 tid 2 2024-09-17T15:42:22.879 INFO:tasks.rados.rados.0.smithi203.stdout:2322: writing smithi20328086-5 from 3018854 to 3489081 tid 3 2024-09-17T15:42:22.879 INFO:tasks.rados.rados.0.smithi203.stdout:2323: write oid 38 current snap is 254 2024-09-17T15:42:22.879 INFO:tasks.rados.rados.0.smithi203.stdout:2323: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-17T15:42:22.886 INFO:tasks.rados.rados.0.smithi203.stdout:2323: writing smithi20328086-38 from 598547 to 1227747 tid 1 2024-09-17T15:42:22.893 INFO:tasks.rados.rados.0.smithi203.stdout:2323: writing smithi20328086-38 from 1876540 to 2368486 tid 2 2024-09-17T15:42:22.901 INFO:tasks.rados.rados.0.smithi203.stdout:2323: writing smithi20328086-38 from 3046277 to 3727776 tid 3 2024-09-17T15:42:22.901 INFO:tasks.rados.rados.0.smithi203.stdout:2324: delete oid 41 current snap is 254 2024-09-17T15:42:22.901 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 3 to smithi20328086-17 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 4 to smithi20328086-17 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 5 to smithi20328086-17 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: finishing write tid 6 to smithi20328086-17 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: oid 17 updating version 0 to 1166 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: oid 17 version 1166 is already newer than 1163 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: oid 17 version 1166 is already newer than 1165 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: oid 17 updating version 1166 to 1167 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: oid 17 version 1167 is already newer than 1164 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1167 (ObjNum 818 snap 254 seq_num 818) dirty exists 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2318: left oid 17 (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2320: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2320: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:42:22.902 INFO:tasks.rados.rados.0.smithi203.stdout:2320: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 632 (ObjNum 766 snap 238 seq_num 766) dirty exists 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2319: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: finishing write tid 1 to smithi20328086-25 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: finishing write tid 2 to smithi20328086-25 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2319: finishing copy_from to smithi20328086-42 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1093 (ObjNum 810 snap 251 seq_num 810) dirty exists 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: finishing write tid 3 to smithi20328086-25 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: finishing write tid 4 to smithi20328086-25 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: finishing write tid 5 to smithi20328086-25 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: oid 25 updating version 0 to 881 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: oid 25 updating version 881 to 884 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: oid 25 version 884 is already newer than 883 2024-09-17T15:42:22.903 INFO:tasks.rados.rados.0.smithi203.stdout:2321: oid 25 version 884 is already newer than 882 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 884 (ObjNum 819 snap 254 seq_num 819) dirty exists 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2321: left oid 25 (ObjNum 819 snap 254 seq_num 819) 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: finishing write tid 1 to smithi20328086-5 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: finishing write tid 2 to smithi20328086-5 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: finishing write tid 3 to smithi20328086-5 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: finishing write tid 4 to smithi20328086-5 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: finishing write tid 5 to smithi20328086-5 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: oid 5 updating version 0 to 522 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: oid 5 version 522 is already newer than 520 2024-09-17T15:42:22.904 INFO:tasks.rados.rados.0.smithi203.stdout:2322: oid 5 version 522 is already newer than 519 2024-09-17T15:42:22.905 INFO:tasks.rados.rados.0.smithi203.stdout:2322: oid 5 version 522 is already newer than 521 2024-09-17T15:42:22.905 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 522 (ObjNum 820 snap 254 seq_num 820) dirty exists 2024-09-17T15:42:22.905 INFO:tasks.rados.rados.0.smithi203.stdout:2322: left oid 5 (ObjNum 820 snap 254 seq_num 820) 2024-09-17T15:42:22.905 INFO:tasks.rados.rados.0.smithi203.stdout:2323: finishing write tid 1 to smithi20328086-38 2024-09-17T15:42:22.905 INFO:tasks.rados.rados.0.smithi203.stdout:2323: finishing write tid 2 to smithi20328086-38 2024-09-17T15:42:22.907 INFO:tasks.rados.rados.0.smithi203.stdout:2323: finishing write tid 3 to smithi20328086-38 2024-09-17T15:42:22.907 INFO:tasks.rados.rados.0.smithi203.stdout:2323: finishing write tid 4 to smithi20328086-38 2024-09-17T15:42:22.907 INFO:tasks.rados.rados.0.smithi203.stdout:2323: finishing write tid 5 to smithi20328086-38 2024-09-17T15:42:22.907 INFO:tasks.rados.rados.0.smithi203.stdout:2323: oid 38 updating version 0 to 1112 2024-09-17T15:42:22.907 INFO:tasks.rados.rados.0.smithi203.stdout:2323: oid 38 version 1112 is already newer than 1110 2024-09-17T15:42:22.908 INFO:tasks.rados.rados.0.smithi203.stdout:2323: oid 38 version 1112 is already newer than 1111 2024-09-17T15:42:22.908 INFO:tasks.rados.rados.0.smithi203.stdout:2323: oid 38 version 1112 is already newer than 1109 2024-09-17T15:42:22.908 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1112 (ObjNum 821 snap 254 seq_num 821) dirty exists 2024-09-17T15:42:22.908 INFO:tasks.rados.rados.0.smithi203.stdout:2323: left oid 38 (ObjNum 821 snap 254 seq_num 821) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2318: done (6 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2319: done (5 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2320: done (4 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2321: done (3 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2322: done (2 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2323: done (1 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2324: done (0 left) 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2325: write (excl) oid 27 current snap is 254 2024-09-17T15:42:22.909 INFO:tasks.rados.rados.0.smithi203.stdout:2325: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-17T15:42:22.915 INFO:tasks.rados.rados.0.smithi203.stdout:2325: writing smithi20328086-27 from 632161 to 1128212 tid 1 2024-09-17T15:42:22.920 INFO:tasks.rados.rados.0.smithi203.stdout:2325: writing smithi20328086-27 from 1593504 to 2089119 tid 2 2024-09-17T15:42:22.926 INFO:tasks.rados.rados.0.smithi203.stdout:2325: writing smithi20328086-27 from 2701123 to 3121519 tid 3 2024-09-17T15:42:22.930 INFO:tasks.rados.rados.0.smithi203.stdout:2325: writing smithi20328086-27 from 3611645 to 3832161 tid 4 2024-09-17T15:42:22.931 INFO:tasks.rados.rados.0.smithi203.stdout:2326: rollback oid 23 current snap is 254 2024-09-17T15:42:22.931 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 253 2024-09-17T15:42:22.931 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 1 to smithi20328086-27 2024-09-17T15:42:22.931 INFO:tasks.rados.rados.0.smithi203.stdout:2327: delete oid 29 current snap is 254 2024-09-17T15:42:22.931 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 2 to smithi20328086-27 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2326: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1055 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2326: done (2 left) 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2327: done (1 left) 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2328: write (excl) oid 3 current snap is 254 2024-09-17T15:42:22.934 INFO:tasks.rados.rados.0.smithi203.stdout:2328: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-17T15:42:22.942 INFO:tasks.rados.rados.0.smithi203.stdout:2328: writing smithi20328086-3 from 665775 to 1312324 tid 1 2024-09-17T15:42:22.949 INFO:tasks.rados.rados.0.smithi203.stdout:2328: writing smithi20328086-3 from 2110468 to 2609752 tid 2 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2328: writing smithi20328086-3 from 3155969 to 3598909 tid 3 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2329: read oid 45 snap -1 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2329: expect (ObjNum 811 snap 252 seq_num 811) 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2330: read oid 18 snap -1 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2330: expect (ObjNum 781 snap 243 seq_num 781) 2024-09-17T15:42:22.955 INFO:tasks.rados.rados.0.smithi203.stdout:2331: write oid 16 current snap is 254 2024-09-17T15:42:22.956 INFO:tasks.rados.rados.0.smithi203.stdout:2331: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-09-17T15:42:22.963 INFO:tasks.rados.rados.0.smithi203.stdout:2331: writing smithi20328086-16 from 682582 to 1404380 tid 1 2024-09-17T15:42:22.973 INFO:tasks.rados.rados.0.smithi203.stdout:2331: writing smithi20328086-16 from 2168950 to 2811892 tid 2 2024-09-17T15:42:22.979 INFO:tasks.rados.rados.0.smithi203.stdout:2331: writing smithi20328086-16 from 3583392 to 3882582 tid 3 2024-09-17T15:42:22.979 INFO:tasks.rados.rados.0.smithi203.stdout:2332: write oid 22 current snap is 254 2024-09-17T15:42:22.979 INFO:tasks.rados.rados.0.smithi203.stdout:2332: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-17T15:42:22.987 INFO:tasks.rados.rados.0.smithi203.stdout:2332: writing smithi20328086-22 from 699389 to 1496436 tid 1 2024-09-17T15:42:22.994 INFO:tasks.rados.rados.0.smithi203.stdout:2332: writing smithi20328086-22 from 2111079 to 2614032 tid 2 2024-09-17T15:42:23.000 INFO:tasks.rados.rados.0.smithi203.stdout:2332: writing smithi20328086-22 from 3094462 to 3676299 tid 3 2024-09-17T15:42:23.000 INFO:tasks.rados.rados.0.smithi203.stdout:2333: write oid 7 current snap is 254 2024-09-17T15:42:23.000 INFO:tasks.rados.rados.0.smithi203.stdout:2333: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-09-17T15:42:23.006 INFO:tasks.rados.rados.0.smithi203.stdout:2333: writing smithi20328086-7 from 716196 to 1188492 tid 1 2024-09-17T15:42:23.013 INFO:tasks.rados.rados.0.smithi203.stdout:2333: writing smithi20328086-7 from 1769561 to 2416172 tid 2 2024-09-17T15:42:23.022 INFO:tasks.rados.rados.0.smithi203.stdout:2333: writing smithi20328086-7 from 3121885 to 3914994 tid 3 2024-09-17T15:42:23.022 INFO:tasks.rados.rados.0.smithi203.stdout:2334: write (excl) oid 39 current snap is 254 2024-09-17T15:42:23.022 INFO:tasks.rados.rados.0.smithi203.stdout:2334: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-17T15:42:23.030 INFO:tasks.rados.rados.0.smithi203.stdout:2334: writing smithi20328086-39 from 733003 to 1396901 tid 1 2024-09-17T15:42:23.036 INFO:tasks.rados.rados.0.smithi203.stdout:2334: writing smithi20328086-39 from 1944396 to 2451018 tid 2 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:2334: writing smithi20328086-39 from 2865661 to 3470042 tid 3 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:2335: rollback oid 47 current snap is 254 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 252 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 3 to smithi20328086-27 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:2336: write oid 37 current snap is 254 2024-09-17T15:42:23.043 INFO:tasks.rados.rados.0.smithi203.stdout:2336: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-17T15:42:23.051 INFO:tasks.rados.rados.0.smithi203.stdout:2336: writing smithi20328086-37 from 749810 to 1488957 tid 1 2024-09-17T15:42:23.057 INFO:tasks.rados.rados.0.smithi203.stdout:2336: writing smithi20328086-37 from 2002878 to 2653158 tid 2 2024-09-17T15:42:23.064 INFO:tasks.rados.rados.0.smithi203.stdout:2336: writing smithi20328086-37 from 3293084 to 3708737 tid 3 2024-09-17T15:42:23.064 INFO:tasks.rados.rados.0.smithi203.stdout:2337: rollback oid 50 current snap is 254 2024-09-17T15:42:23.064 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 239 2024-09-17T15:42:23.064 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 4 to smithi20328086-27 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 5 to smithi20328086-27 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: finishing write tid 6 to smithi20328086-27 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: oid 27 updating version 0 to 1098 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: oid 27 version 1098 is already newer than 1095 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: oid 27 version 1098 is already newer than 1094 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: oid 27 version 1098 is already newer than 1097 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: oid 27 version 1098 is already newer than 1096 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1098 (ObjNum 823 snap 254 seq_num 823) dirty exists 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2325: left oid 27 (ObjNum 823 snap 254 seq_num 823) 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2328: finishing write tid 1 to smithi20328086-3 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2328: finishing write tid 2 to smithi20328086-3 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:2330: expect (ObjNum 781 snap 243 seq_num 781) 2024-09-17T15:42:23.065 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:23.205 INFO:tasks.rados.rados.0.smithi203.stdout:2328: finishing write tid 3 to smithi20328086-3 2024-09-17T15:42:23.205 INFO:tasks.rados.rados.0.smithi203.stdout:2328: finishing write tid 4 to smithi20328086-3 2024-09-17T15:42:23.205 INFO:tasks.rados.rados.0.smithi203.stdout:2328: finishing write tid 5 to smithi20328086-3 2024-09-17T15:42:23.205 INFO:tasks.rados.rados.0.smithi203.stdout:2328: oid 3 updating version 0 to 1101 2024-09-17T15:42:23.205 INFO:tasks.rados.rados.0.smithi203.stdout:2328: oid 3 version 1101 is already newer than 1099 2024-09-17T15:42:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:2328: oid 3 version 1101 is already newer than 1100 2024-09-17T15:42:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:2328: oid 3 updating version 1101 to 1102 2024-09-17T15:42:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1102 (ObjNum 825 snap 254 seq_num 825) dirty exists 2024-09-17T15:42:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:2328: left oid 3 (ObjNum 825 snap 254 seq_num 825) 2024-09-17T15:42:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:2329: expect (ObjNum 811 snap 252 seq_num 811) 2024-09-17T15:42:23.278 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:23.317 INFO:tasks.rados.rados.0.smithi203.stdout:2331: finishing write tid 1 to smithi20328086-16 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: finishing write tid 2 to smithi20328086-16 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: finishing write tid 3 to smithi20328086-16 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: finishing write tid 4 to smithi20328086-16 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: finishing write tid 5 to smithi20328086-16 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: oid 16 updating version 0 to 1139 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: oid 16 version 1139 is already newer than 1136 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: oid 16 version 1139 is already newer than 1137 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: oid 16 version 1139 is already newer than 1138 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1139 (ObjNum 826 snap 254 seq_num 826) dirty exists 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2331: left oid 16 (ObjNum 826 snap 254 seq_num 826) 2024-09-17T15:42:23.318 INFO:tasks.rados.rados.0.smithi203.stdout:2332: finishing write tid 1 to smithi20328086-22 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: finishing write tid 2 to smithi20328086-22 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: finishing write tid 3 to smithi20328086-22 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: finishing write tid 4 to smithi20328086-22 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: finishing write tid 5 to smithi20328086-22 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: oid 22 updating version 0 to 634 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: oid 22 updating version 634 to 636 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: oid 22 version 636 is already newer than 635 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: oid 22 version 636 is already newer than 633 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 636 (ObjNum 827 snap 254 seq_num 827) dirty exists 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2332: left oid 22 (ObjNum 827 snap 254 seq_num 827) 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2333: finishing write tid 1 to smithi20328086-7 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2333: finishing write tid 2 to smithi20328086-7 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2333: finishing write tid 3 to smithi20328086-7 2024-09-17T15:42:23.319 INFO:tasks.rados.rados.0.smithi203.stdout:2333: finishing write tid 4 to smithi20328086-7 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: finishing write tid 5 to smithi20328086-7 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: oid 7 updating version 0 to 1086 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: oid 7 version 1086 is already newer than 1085 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: oid 7 updating version 1086 to 1087 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: oid 7 version 1087 is already newer than 1084 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1087 (ObjNum 828 snap 254 seq_num 828) dirty exists 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2333: left oid 7 (ObjNum 828 snap 254 seq_num 828) 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: finishing write tid 1 to smithi20328086-39 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: finishing write tid 2 to smithi20328086-39 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: finishing write tid 3 to smithi20328086-39 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: finishing write tid 4 to smithi20328086-39 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: finishing write tid 5 to smithi20328086-39 2024-09-17T15:42:23.320 INFO:tasks.rados.rados.0.smithi203.stdout:2334: oid 39 updating version 0 to 1141 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2334: oid 39 updating version 1141 to 1144 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2334: oid 39 version 1144 is already newer than 1142 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2334: oid 39 version 1144 is already newer than 1143 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1144 (ObjNum 829 snap 254 seq_num 829) dirty exists 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2334: left oid 39 (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2335: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2335: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2335: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 837 (ObjNum 779 snap 243 seq_num 779) dirty exists 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2336: finishing write tid 1 to smithi20328086-37 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2336: finishing write tid 2 to smithi20328086-37 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2336: finishing write tid 3 to smithi20328086-37 2024-09-17T15:42:23.321 INFO:tasks.rados.rados.0.smithi203.stdout:2336: finishing write tid 4 to smithi20328086-37 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: finishing write tid 5 to smithi20328086-37 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: oid 37 updating version 0 to 1148 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: oid 37 version 1148 is already newer than 1147 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: oid 37 updating version 1148 to 1149 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: oid 37 version 1149 is already newer than 1146 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1149 (ObjNum 830 snap 254 seq_num 830) dirty exists 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2336: left oid 37 (ObjNum 830 snap 254 seq_num 830) 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2337: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2337: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2337: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1161 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2325: done (10 left) 2024-09-17T15:42:23.322 INFO:tasks.rados.rados.0.smithi203.stdout:2328: done (9 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2329: done (8 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2330: done (7 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2331: done (6 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2332: done (5 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2333: done (4 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2334: done (3 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2335: done (2 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2336: done (1 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2337: done (0 left) 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2338: read oid 29 snap -1 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2338: expect deleted 2024-09-17T15:42:23.323 INFO:tasks.rados.rados.0.smithi203.stdout:2339: rollback oid 21 current snap is 254 2024-09-17T15:42:23.324 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 239 2024-09-17T15:42:23.324 INFO:tasks.rados.rados.0.smithi203.stdout:2340: read oid 25 snap -1 2024-09-17T15:42:23.324 INFO:tasks.rados.rados.0.smithi203.stdout:2340: expect (ObjNum 819 snap 254 seq_num 819) 2024-09-17T15:42:23.324 INFO:tasks.rados.rados.0.smithi203.stdout:2341: delete oid 4 current snap is 254 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2338: done (3 left) 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2341: done (2 left) 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2342: copy_from oid 11 from oid 12 current snap is 254 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2343: read oid 10 snap -1 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2343: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2344: write oid 9 current snap is 254 2024-09-17T15:42:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:2344: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-17T15:42:23.334 INFO:tasks.rados.rados.0.smithi203.stdout:2344: writing smithi20328086-9 from 783424 to 1273069 tid 1 2024-09-17T15:42:23.349 INFO:tasks.rados.rados.0.smithi203.stdout:2344: writing smithi20328086-9 from 2003489 to 2657438 tid 2 2024-09-17T15:42:23.360 INFO:tasks.rados.rados.0.smithi203.stdout:2344: writing smithi20328086-9 from 3231577 to 3786127 tid 3 2024-09-17T15:42:23.360 INFO:tasks.rados.rados.0.smithi203.stdout:2345: snap_remove snap 252 2024-09-17T15:42:23.802 INFO:tasks.rados.rados.0.smithi203.stdout:2345: done (5 left) 2024-09-17T15:42:23.802 INFO:tasks.rados.rados.0.smithi203.stdout:2346: read oid 24 snap -1 2024-09-17T15:42:23.802 INFO:tasks.rados.rados.0.smithi203.stdout:2346: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:23.803 INFO:tasks.rados.rados.0.smithi203.stdout:2347: read oid 32 snap -1 2024-09-17T15:42:23.803 INFO:tasks.rados.rados.0.smithi203.stdout:2347: expect (ObjNum 786 snap 245 seq_num 786) 2024-09-17T15:42:23.803 INFO:tasks.rados.rados.0.smithi203.stdout:2348: snap_create 2024-09-17T15:42:23.803 INFO:tasks.rados.rados.0.smithi203.stdout:2339: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:42:23.803 INFO:tasks.rados.rados.0.smithi203.stdout:2340: expect (ObjNum 819 snap 254 seq_num 819) 2024-09-17T15:42:23.977 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2733778 2024-09-17T15:42:24.099 INFO:tasks.rados.rados.0.smithi203.stdout:2339: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:42:24.099 INFO:tasks.rados.rados.0.smithi203.stdout:2339: finishing rollback tid 2 to smithi20328086-21 2024-09-17T15:42:24.099 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1222 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:42:24.099 INFO:tasks.rados.rados.0.smithi203.stdout:2343: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:24.099 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:24.205 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2342: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2342: finishing copy_from to smithi20328086-11 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2344: finishing write tid 1 to smithi20328086-9 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2344: finishing write tid 2 to smithi20328086-9 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2344: finishing write tid 3 to smithi20328086-9 2024-09-17T15:42:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2344: finishing write tid 4 to smithi20328086-9 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: finishing write tid 5 to smithi20328086-9 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: oid 9 updating version 0 to 888 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: oid 9 version 888 is already newer than 887 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: oid 9 updating version 888 to 889 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: oid 9 updating version 889 to 890 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 890 (ObjNum 832 snap 254 seq_num 832) dirty exists 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2344: left oid 9 (ObjNum 832 snap 254 seq_num 832) 2024-09-17T15:42:24.263 INFO:tasks.rados.rados.0.smithi203.stdout:2346: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:24.323 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 619316 2024-09-17T15:42:24.370 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 154721 2024-09-17T15:42:24.426 INFO:tasks.rados.rados.0.smithi203.stdout:2347: expect (ObjNum 786 snap 245 seq_num 786) 2024-09-17T15:42:24.427 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:24.521 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:24.663 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2023795 2024-09-17T15:42:24.816 INFO:tasks.rados.rados.0.smithi203.stdout:2339: done (7 left) 2024-09-17T15:42:24.816 INFO:tasks.rados.rados.0.smithi203.stdout:2340: done (6 left) 2024-09-17T15:42:24.816 INFO:tasks.rados.rados.0.smithi203.stdout:2342: done (5 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2343: done (4 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2344: done (3 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2346: done (2 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2347: done (1 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2348: done (0 left) 2024-09-17T15:42:24.817 INFO:tasks.rados.rados.0.smithi203.stdout:2349: snap_remove snap 253 2024-09-17T15:42:25.120 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:42:25.120 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:42:25.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:42:25.123 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:42:25.124 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:42:25.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:25.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 10 monclient: get_monmap_and_config 2024-09-17T15:42:25.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 10 monclient: build_initial_monmap 2024-09-17T15:42:25.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 10 monclient: monmap: 2024-09-17T15:42:25.253 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:25.253 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:25.258477+0000 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:25.258477+0000 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:25.254 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 1 Processor -- start 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 1 -- start start 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 10 monclient: init 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.255+0000 7f4865e27640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-b con 0x7f48600689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4860068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 0x7f486013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-a con 0x7f4860139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f486009d770 con 0x7f4860068db0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f486009ed00 con 0x7f4860139f00 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f486009efe0 con 0x7f48600689d0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f4865e27640 10 monclient(hunting): authenticate will time out at 1878.224125s 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 0x7f486013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54002/0 (socket says 172.21.15.86:54002) 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 -- 172.21.15.86:0/1020739722 learned_addr learned my addr 172.21.15.86:0/1020739722 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): get_auth_request con 0x7f48600689d0 auth_method 0 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:25.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): get_auth_request con 0x7f4860139f00 auth_method 0 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient(hunting): handle_auth_done global_id 164177 payload 274 2024-09-17T15:42:25.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 -- 172.21.15.86:0/1020739722 >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 msgr2=0x7f486013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 --2- 172.21.15.86:0/1020739722 >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 0x7f486013c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 -- 172.21.15.86:0/1020739722 >> v1:172.21.15.203:6789/0 conn(0x7f4860068db0 legacy=0x7f4860069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 monclient: found mon.noname-b 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485f7fe640 1 -- 172.21.15.86:0/1020739722 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48600a04e0 con 0x7f48600689d0 2024-09-17T15:42:25.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.259+0000 7f485ffff640 1 --2- 172.21.15.86:0/1020739722 >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 0x7f486013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485f7fe640 1 -- 172.21.15.86:0/1020739722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 msgr2=0x7f486009ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485f7fe640 1 --2- 172.21.15.86:0/1020739722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48600a0770 tx=0x7f485801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75ed1b636b8f9678 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 1 -- 172.21.15.86:0/1020739722 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48580b0070 con 0x7f48600689d0 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: dump: 2024-09-17T15:42:25.262 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 1 -- 172.21.15.86:0/1020739722 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48580ab600 con 0x7f48600689d0 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 5 monclient: authenticate success, global_id 164177 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 4 set_mon_vals no callback set 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 1 -- 172.21.15.86:0/1020739722 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48580ab9d0 con 0x7f48600689d0 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f485e7fc640 10 monclient: dump: 2024-09-17T15:42:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 10 monclient: shutdown 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 -- 172.21.15.86:0/1020739722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 msgr2=0x7f486009ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 --2- 172.21.15.86:0/1020739722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48600a0770 tx=0x7f485801c490 comp rx=0 tx=0).stop 2024-09-17T15:42:25.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 -- 172.21.15.86:0/1020739722 shutdown_connections 2024-09-17T15:42:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 --2- 172.21.15.86:0/1020739722 >> v2:172.21.15.86:3300/0 conn(0x7f4860139f00 0x7f486013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 --2- 172.21.15.86:0/1020739722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48600689d0 0x7f486009ffa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 -- 172.21.15.86:0/1020739722 >> 172.21.15.86:0/1020739722 conn(0x7f48600a6440 msgr2=0x7f48600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 -- 172.21.15.86:0/1020739722 shutdown_connections 2024-09-17T15:42:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.263+0000 7f4865e27640 1 -- 172.21.15.86:0/1020739722 wait complete. 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient: build_initial_monmap 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient: monmap: 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:25.271 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 Processor -- start 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 -- start start 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient: init 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 0x7f486011f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-b con 0x7f48600689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4860068db0 0x7f4860134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-a con 0x7f4860068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): picked mon.noname-c con 0x7f4860139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f486013f3e0 con 0x7f4860068db0 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f486013f260 con 0x7f48600689d0 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f486013f560 con 0x7f4860139f00 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:25.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4865e27640 10 monclient(hunting): authenticate will time out at 1878.233057s 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54016/0 (socket says 172.21.15.86:54016) 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 -- 172.21.15.86:0/3690731430 learned_addr learned my addr 172.21.15.86:0/3690731430 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4860068db0 0x7f4860134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): get_auth_request con 0x7f4860139f00 auth_method 0 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): get_auth_request con 0x7f4860068db0 auth_method 0 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:25.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485f7fe640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 0x7f486011f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient(hunting): handle_auth_done global_id 164180 payload 824 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 -- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 msgr2=0x7f486011f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 0x7f486011f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 -- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4860068db0 msgr2=0x7f4860134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4860068db0 0x7f4860134f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 monclient: found mon.noname-c 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 -- 172.21.15.86:0/3690731430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4860139bf0 con 0x7f4860139f00 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485f7fe640 10 monclient: get_auth_request con 0x7f48600689d0 auth_method 0 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485f7fe640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 0x7f486011f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 -- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 msgr2=0x7f48601cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f485ffff640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f485000e920 tx=0x7f485000ede0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850017070 con 0x7f4860139f00 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:25.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: dump: 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: _finish_auth 0 2024-09-17T15:42:25.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f485000cc10 con 0x7f4860139f00 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.267+0000 7f4847fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4850013470 con 0x7f4860139f00 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 5 monclient: authenticate success, global_id 164180 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _renew_subs 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f486009d460 con 0x7f4860139f00 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f485dffb640 4 set_mon_vals no callback set 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 10 monclient: dump: 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:25.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _renew_subs 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48601208d0 con 0x7f4860139f00 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f486009ffa0 con 0x7f4860139f00 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f485001a070 con 0x7f4860139f00 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 0x7f484c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(528..528 src has 438..528) ==== 4787+0+0 (secure 0 0 0) 0x7f485004ff90 con 0x7f4860139f00 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f485f7fe640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 0x7f484c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.271+0000 7f485f7fe640 10 monclient: get_auth_request con 0x7f484c03bbb0 auth_method 0 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.275+0000 7f485f7fe640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 0x7f484c03e070 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f48580ab000 tx=0x7f48580a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.275+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f485004e980 con 0x7f4860139f00 2024-09-17T15:42:25.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.275+0000 7f4847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:25.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.275+0000 7f4847fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:25.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4865e27640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:25.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4865e27640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4865e27640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4860120030 con 0x7f4860139f00 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4847fff640 1 -- 172.21.15.86:0/3690731430 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4850009340 con 0x7f4860139f00 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.507+0000 7f4847fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:25.506 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 msgr2=0x7f484c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 0x7f484c03e070 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f48580ab000 tx=0x7f48580a6000 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 10 monclient: shutdown 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 msgr2=0x7f48601cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f485000e920 tx=0x7f485000ede0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 shutdown_connections 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f484c03bbb0 0x7f484c03e070 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4860139f00 0x7f48601cd8c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4860068db0 0x7f4860134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 --2- 172.21.15.86:0/3690731430 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48600689d0 0x7f486011f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 >> 172.21.15.86:0/3690731430 conn(0x7f48600a6440 msgr2=0x7f4860137fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 shutdown_connections 2024-09-17T15:42:25.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:25.511+0000 7f4865e27640 1 -- 172.21.15.86:0/3690731430 wait complete. 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:2349: done (0 left) 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:2350: rollback oid 45 current snap is 255 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 247 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:2351: read oid 49 snap -1 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:2351: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:42:25.825 INFO:tasks.rados.rados.0.smithi203.stdout:2352: snap_remove snap 249 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2352: done (2 left) 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2353: snap_create 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2350: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2350: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2350: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1072 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:2351: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-17T15:42:26.714 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:26.854 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1942870 2024-09-17T15:42:26.864 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2350: done (2 left) 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2351: done (1 left) 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2353: done (0 left) 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2354: rollback oid 23 current snap is 256 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 255 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2355: write oid 4 current snap is 256 2024-09-17T15:42:27.715 INFO:tasks.rados.rados.0.smithi203.stdout:2355: seq_num 833 ranges {400231=564894,1661971=338260} 2024-09-17T15:42:27.730 INFO:tasks.rados.rados.0.smithi203.stdout:2355: writing smithi20328086-4 from 400231 to 965125 tid 1 2024-09-17T15:42:27.748 INFO:tasks.rados.rados.0.smithi203.stdout:2355: writing smithi20328086-4 from 1661971 to 2000231 tid 2 2024-09-17T15:42:27.748 INFO:tasks.rados.rados.0.smithi203.stdout:2356: write oid 3 current snap is 256 2024-09-17T15:42:27.748 INFO:tasks.rados.rados.0.smithi203.stdout:2356: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-17T15:42:27.764 INFO:tasks.rados.rados.0.smithi203.stdout:2356: writing smithi20328086-3 from 417038 to 1057181 tid 1 2024-09-17T15:42:27.777 INFO:tasks.rados.rados.0.smithi203.stdout:2356: writing smithi20328086-3 from 1720453 to 2017038 tid 2 2024-09-17T15:42:27.777 INFO:tasks.rados.rados.0.smithi203.stdout:2357: read oid 17 snap -1 2024-09-17T15:42:27.777 INFO:tasks.rados.rados.0.smithi203.stdout:2357: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:42:27.777 INFO:tasks.rados.rados.0.smithi203.stdout:2358: read oid 41 snap -1 2024-09-17T15:42:27.778 INFO:tasks.rados.rados.0.smithi203.stdout:2358: expect deleted 2024-09-17T15:42:27.778 INFO:tasks.rados.rados.0.smithi203.stdout:2359: snap_remove snap 247 2024-09-17T15:42:28.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: get_monmap_and_config 2024-09-17T15:42:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: build_initial_monmap 2024-09-17T15:42:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: monmap: 2024-09-17T15:42:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:28.656 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:28.661481+0000 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:28.661481+0000 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:28.657 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 Processor -- start 2024-09-17T15:42:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 -- start start 2024-09-17T15:42:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: init 2024-09-17T15:42:28.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f799c0a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 0x7f799c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-b con 0x7f799c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f799c13d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c0a6530 con 0x7f799c13d2f0 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799c09c150 con 0x7f799c0a50e0 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f799c09c430 con 0x7f799c0a54c0 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a2b01640 10 monclient(hunting): authenticate will time out at 1881.627775s 2024-09-17T15:42:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a109e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f799c13d2f0 0x7f799c13f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54976/0 (socket says 172.21.15.86:54976) 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f79a109e640 1 -- 172.21.15.86:0/1298653085 learned_addr learned my addr 172.21.15.86:0/1298653085 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.659+0000 7f799bfff640 1 --2- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 0x7f799c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): get_auth_request con 0x7f799c0a54c0 auth_method 0 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient(hunting): handle_auth_done global_id 164183 payload 274 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 1 -- 172.21.15.86:0/1298653085 >> v1:172.21.15.203:6789/0 conn(0x7f799c0a50e0 legacy=0x7f799c09ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 1 -- 172.21.15.86:0/1298653085 >> v1:172.21.15.86:6789/0 conn(0x7f799c13d2f0 legacy=0x7f799c13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 1 monclient: found mon.noname-b 2024-09-17T15:42:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.663+0000 7f799bfff640 1 -- 172.21.15.86:0/1298653085 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c0a6940 con 0x7f799c0a54c0 2024-09-17T15:42:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799bfff640 1 -- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 msgr2=0x7f799c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799bfff640 1 --2- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 0x7f799c0a58d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f79880097f0 tx=0x7f798801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d3a2888f5b2bd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 1 -- 172.21.15.86:0/1298653085 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79880b0070 con 0x7f799c0a54c0 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: dump: 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 5 monclient: authenticate success, global_id 164183 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 1 -- 172.21.15.86:0/1298653085 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79880ab600 con 0x7f799c0a54c0 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 4 set_mon_vals no callback set 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 1 -- 172.21.15.86:0/1298653085 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79880ab9d0 con 0x7f799c0a54c0 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f799b7fe640 10 monclient: dump: 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: shutdown 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 msgr2=0x7f799c0a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 --2- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 0x7f799c0a58d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f79880097f0 tx=0x7f798801c3a0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- 172.21.15.86:0/1298653085 shutdown_connections 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 --2- 172.21.15.86:0/1298653085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a54c0 0x7f799c0a58d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- 172.21.15.86:0/1298653085 >> 172.21.15.86:0/1298653085 conn(0x7f799c0aaae0 msgr2=0x7f799c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- 172.21.15.86:0/1298653085 shutdown_connections 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- 172.21.15.86:0/1298653085 wait complete. 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: build_initial_monmap 2024-09-17T15:42:28.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: monmap: 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:28.671 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 Processor -- start 2024-09-17T15:42:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- start start 2024-09-17T15:42:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: init 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-c con 0x7f799c0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f799c0a54c0 0x7f799c138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-b con 0x7f799c0a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 0x7f799c0a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): picked mon.noname-a con 0x7f799c13d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f799c143a10 con 0x7f799c13d2f0 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f799c143890 con 0x7f799c0a54c0 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f799c143b90 con 0x7f799c0a50e0 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.667+0000 7f79a2b01640 10 monclient(hunting): authenticate will time out at 1881.635970s 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 0x7f799c0a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 0x7f799c0a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35568/0 (socket says 172.21.15.86:35568) 2024-09-17T15:42:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 1 -- 172.21.15.86:0/830445891 learned_addr learned my addr 172.21.15.86:0/830445891 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): get_auth_request con 0x7f799c13d2f0 auth_method 0 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37682/0 (socket says 172.21.15.86:37682) 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): get_auth_request con 0x7f799c0a50e0 auth_method 0 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a109e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799bfff640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f799c0a54c0 0x7f799c138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799bfff640 10 monclient(hunting): get_auth_request con 0x7f799c0a54c0 auth_method 0 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient(hunting): handle_auth_done global_id 164186 payload 824 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 -- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f799c0a54c0 msgr2=0x7f799c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f799c0a54c0 0x7f799c138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 -- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 msgr2=0x7f799c0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 0x7f799c0a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient: _finish_hunting 0 2024-09-17T15:42:28.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 monclient: found mon.noname-c 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 -- 172.21.15.86:0/830445891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c121a90 con 0x7f799c0a50e0 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 -- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 msgr2=0x7f799c1cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a089d640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f799000c410 tx=0x7f799000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990017070 con 0x7f799c0a50e0 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: dump: 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.679 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: _finish_auth 0 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: _check_auth_tickets 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79900069e0 con 0x7f799c0a50e0 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7990006dd0 con 0x7f799c0a50e0 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79997fa640 10 monclient: dump: 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f799affd640 4 set_mon_vals no callback set 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 5 monclient: authenticate success, global_id 164186 2024-09-17T15:42:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 10 monclient: _renew_subs 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c0a18d0 con 0x7f799c0a50e0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 10 monclient: _renew_subs 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.671+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c0a4240 con 0x7f799c0a50e0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79a2b01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79a2b01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79a2b01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f799001a070 con 0x7f799c0a50e0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c0a58d0 con 0x7f799c0a50e0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79997fa640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 0x7f798003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(531..531 src has 438..531) ==== 4771+0+0 (secure 0 0 0) 0x7f799004f2e0 con 0x7f799c0a50e0 2024-09-17T15:42:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f799bfff640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 0x7f798003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.675+0000 7f799bfff640 10 monclient: get_auth_request con 0x7f798003bb60 auth_method 0 2024-09-17T15:42:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.679+0000 7f799bfff640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 0x7f798003e020 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f799c09cd60 tx=0x7f79880a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.679+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f799004dce0 con 0x7f799c0a50e0 2024-09-17T15:42:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.679+0000 7f79997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.679+0000 7f79997fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:28.721 INFO:tasks.rados.rados.0.smithi203.stdout:2359: done (5 left) 2024-09-17T15:42:28.721 INFO:tasks.rados.rados.0.smithi203.stdout:2360: read oid 29 snap -1 2024-09-17T15:42:28.721 INFO:tasks.rados.rados.0.smithi203.stdout:2360: expect deleted 2024-09-17T15:42:28.722 INFO:tasks.rados.rados.0.smithi203.stdout:2361: write (excl) oid 18 current snap is 256 2024-09-17T15:42:28.722 INFO:tasks.rados.rados.0.smithi203.stdout:2361: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-17T15:42:28.735 INFO:tasks.rados.rados.0.smithi203.stdout:2361: writing smithi20328086-18 from 433845 to 1149237 tid 1 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2361: writing smithi20328086-18 from 1662582 to 2033845 tid 2 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2362: copy_from oid 26 from oid 12 current snap is 256 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2354: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1055 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: finishing write tid 1 to smithi20328086-4 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: finishing write tid 2 to smithi20328086-4 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: finishing write tid 3 to smithi20328086-4 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: finishing write tid 4 to smithi20328086-4 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: oid 4 updating version 0 to 1073 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: oid 4 updating version 1073 to 1075 2024-09-17T15:42:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2355: oid 4 version 1075 is already newer than 1074 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1075 (ObjNum 833 snap 256 seq_num 833) dirty exists 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2355: left oid 4 (ObjNum 833 snap 256 seq_num 833) 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: finishing write tid 1 to smithi20328086-3 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: finishing write tid 2 to smithi20328086-3 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: finishing write tid 3 to smithi20328086-3 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: finishing write tid 4 to smithi20328086-3 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: oid 3 updating version 0 to 1103 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: oid 3 updating version 1103 to 1105 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: oid 3 version 1105 is already newer than 1104 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1105 (ObjNum 834 snap 256 seq_num 834) dirty exists 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2356: left oid 3 (ObjNum 834 snap 256 seq_num 834) 2024-09-17T15:42:28.747 INFO:tasks.rados.rados.0.smithi203.stdout:2357: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:42:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 872413 2024-09-17T15:42:28.834 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:28.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79a2b01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:28.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79a2b01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:28.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79a2b01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:28.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f799c0a36e0 con 0x7f799c0a50e0 2024-09-17T15:42:28.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79997fa640 1 -- 172.21.15.86:0/830445891 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f799004f080 con 0x7f799c0a50e0 2024-09-17T15:42:28.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:28.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.931+0000 7f79997fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:28.929 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 msgr2=0x7f798003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 0x7f798003e020 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f799c09cd60 tx=0x7f79880a6000 comp rx=0 tx=0).stop 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 10 monclient: shutdown 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 msgr2=0x7f799c1cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f799000c410 tx=0x7f799000c8d0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 shutdown_connections 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f798003bb60 0x7f798003e020 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f799c13d2f0 0x7f799c0a1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f799c0a54c0 0x7f799c138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 --2- 172.21.15.86:0/830445891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f799c0a50e0 0x7f799c1cfe20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:28.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 >> 172.21.15.86:0/830445891 conn(0x7f799c0aaae0 msgr2=0x7f799c0a0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:28.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 shutdown_connections 2024-09-17T15:42:28.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:28.935+0000 7f79a2b01640 1 -- 172.21.15.86:0/830445891 wait complete. 2024-09-17T15:42:28.949 INFO:tasks.rados.rados.0.smithi203.stdout:2361: finishing write tid 1 to smithi20328086-18 2024-09-17T15:42:28.949 INFO:tasks.rados.rados.0.smithi203.stdout:2361: finishing write tid 2 to smithi20328086-18 2024-09-17T15:42:28.949 INFO:tasks.rados.rados.0.smithi203.stdout:2361: finishing write tid 3 to smithi20328086-18 2024-09-17T15:42:28.949 INFO:tasks.rados.rados.0.smithi203.stdout:2361: finishing write tid 4 to smithi20328086-18 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2361: oid 18 updating version 0 to 839 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2361: oid 18 version 839 is already newer than 838 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2361: oid 18 updating version 839 to 840 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 840 (ObjNum 835 snap 256 seq_num 835) dirty exists 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2361: left oid 18 (ObjNum 835 snap 256 seq_num 835) 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2362: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2362: finishing copy_from to smithi20328086-26 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2354: done (7 left) 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2355: done (6 left) 2024-09-17T15:42:28.950 INFO:tasks.rados.rados.0.smithi203.stdout:2356: done (5 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2357: done (4 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2358: done (3 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2360: done (2 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2361: done (1 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2362: done (0 left) 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2363: write (excl) oid 50 current snap is 256 2024-09-17T15:42:28.951 INFO:tasks.rados.rados.0.smithi203.stdout:2363: seq_num 836 ranges {450652=790641,1721064=329588} 2024-09-17T15:42:28.968 INFO:tasks.rados.rados.0.smithi203.stdout:2363: writing smithi20328086-50 from 450652 to 1241293 tid 1 2024-09-17T15:42:28.980 INFO:tasks.rados.rados.0.smithi203.stdout:2363: writing smithi20328086-50 from 1721064 to 2050652 tid 2 2024-09-17T15:42:28.980 INFO:tasks.rados.rados.0.smithi203.stdout:2364: delete oid 11 current snap is 256 2024-09-17T15:42:28.981 INFO:tasks.rados.rados.0.smithi203.stdout:2363: finishing write tid 1 to smithi20328086-50 2024-09-17T15:42:28.984 INFO:tasks.rados.rados.0.smithi203.stdout:2363: finishing write tid 2 to smithi20328086-50 2024-09-17T15:42:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:2363: finishing write tid 3 to smithi20328086-50 2024-09-17T15:42:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:2363: finishing write tid 4 to smithi20328086-50 2024-09-17T15:42:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:2363: oid 50 updating version 0 to 1162 2024-09-17T15:42:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:2363: oid 50 updating version 1162 to 1164 2024-09-17T15:42:28.985 INFO:tasks.rados.rados.0.smithi203.stdout:2363: oid 50 version 1164 is already newer than 1163 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1164 (ObjNum 836 snap 256 seq_num 836) dirty exists 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:2363: left oid 50 (ObjNum 836 snap 256 seq_num 836) 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:2363: done (1 left) 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:2364: done (0 left) 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:2365: write (excl) oid 15 current snap is 256 2024-09-17T15:42:28.986 INFO:tasks.rados.rados.0.smithi203.stdout:2365: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-17T15:42:29.003 INFO:tasks.rados.rados.0.smithi203.stdout:2365: writing smithi20328086-15 from 484266 to 1141758 tid 1 2024-09-17T15:42:29.018 INFO:tasks.rados.rados.0.smithi203.stdout:2365: writing smithi20328086-15 from 1554381 to 2084266 tid 2 2024-09-17T15:42:29.018 INFO:tasks.rados.rados.0.smithi203.stdout:2366: write (excl) oid 33 current snap is 256 2024-09-17T15:42:29.018 INFO:tasks.rados.rados.0.smithi203.stdout:2366: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-17T15:42:29.033 INFO:tasks.rados.rados.0.smithi203.stdout:2366: writing smithi20328086-33 from 501073 to 1233814 tid 1 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2366: writing smithi20328086-33 from 1896510 to 2101073 tid 2 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2367: rollback oid 35 current snap is 256 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: finishing write tid 1 to smithi20328086-15 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: finishing write tid 2 to smithi20328086-15 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: finishing write tid 3 to smithi20328086-15 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: finishing write tid 4 to smithi20328086-15 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: oid 15 updating version 0 to 1125 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: oid 15 updating version 1125 to 1126 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: oid 15 updating version 1126 to 1127 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1127 (ObjNum 838 snap 256 seq_num 838) dirty exists 2024-09-17T15:42:29.042 INFO:tasks.rados.rados.0.smithi203.stdout:2365: left oid 15 (ObjNum 838 snap 256 seq_num 838) 2024-09-17T15:42:29.043 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 244 2024-09-17T15:42:29.043 INFO:tasks.rados.rados.0.smithi203.stdout:2365: done (2 left) 2024-09-17T15:42:29.043 INFO:tasks.rados.rados.0.smithi203.stdout:2368: write (excl) oid 41 current snap is 256 2024-09-17T15:42:29.043 INFO:tasks.rados.rados.0.smithi203.stdout:2368: seq_num 840 ranges {517880=407990,1554992=562888} 2024-09-17T15:42:29.049 INFO:tasks.rados.rados.0.smithi203.stdout:2368: writing smithi20328086-41 from 517880 to 925870 tid 1 2024-09-17T15:42:29.061 INFO:tasks.rados.rados.0.smithi203.stdout:2368: writing smithi20328086-41 from 1554992 to 2117880 tid 2 2024-09-17T15:42:29.062 INFO:tasks.rados.rados.0.smithi203.stdout:2369: write (excl) oid 50 current snap is 256 2024-09-17T15:42:29.062 INFO:tasks.rados.rados.0.smithi203.stdout:2369: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-17T15:42:29.065 INFO:tasks.rados.rados.0.smithi203.stdout:2369: writing smithi20328086-50 from 534687 to 1017926 tid 1 2024-09-17T15:42:29.072 INFO:tasks.rados.rados.0.smithi203.stdout:2369: writing smithi20328086-50 from 1613474 to 2025757 tid 2 2024-09-17T15:42:29.072 INFO:tasks.rados.rados.0.smithi203.stdout:2370: write (excl) oid 5 current snap is 256 2024-09-17T15:42:29.072 INFO:tasks.rados.rados.0.smithi203.stdout:2370: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-17T15:42:29.078 INFO:tasks.rados.rados.0.smithi203.stdout:2370: writing smithi20328086-5 from 551494 to 1109982 tid 1 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2370: writing smithi20328086-5 from 1671956 to 2151494 tid 2 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2367: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 0 (ObjNum 33 snap 0 seq_num 3606672864) dirty dne 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2366: finishing write tid 1 to smithi20328086-33 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2371: read oid 26 snap -1 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2371: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2367: done (5 left) 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2372: rollback oid 12 current snap is 256 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 239 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2366: finishing write tid 2 to smithi20328086-33 2024-09-17T15:42:29.086 INFO:tasks.rados.rados.0.smithi203.stdout:2366: finishing write tid 3 to smithi20328086-33 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2366: finishing write tid 4 to smithi20328086-33 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2366: oid 33 updating version 0 to 525 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2366: oid 33 version 525 is already newer than 524 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2366: oid 33 version 525 is already newer than 523 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 525 (ObjNum 839 snap 256 seq_num 839) dirty exists 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2366: left oid 33 (ObjNum 839 snap 256 seq_num 839) 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: finishing write tid 1 to smithi20328086-41 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: finishing write tid 2 to smithi20328086-41 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: finishing write tid 3 to smithi20328086-41 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: finishing write tid 4 to smithi20328086-41 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: oid 41 updating version 0 to 632 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: oid 41 updating version 632 to 633 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:2368: oid 41 updating version 633 to 634 2024-09-17T15:42:29.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 634 (ObjNum 840 snap 256 seq_num 840) dirty exists 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2368: left oid 41 (ObjNum 840 snap 256 seq_num 840) 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: finishing write tid 1 to smithi20328086-50 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: finishing write tid 2 to smithi20328086-50 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: finishing write tid 3 to smithi20328086-50 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: finishing write tid 4 to smithi20328086-50 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: oid 50 updating version 0 to 1166 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: oid 50 version 1166 is already newer than 1165 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: oid 50 updating version 1166 to 1167 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1167 (ObjNum 841 snap 256 seq_num 841) dirty exists 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: left oid 50 (ObjNum 841 snap 256 seq_num 841) 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2366: done (5 left) 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2368: done (4 left) 2024-09-17T15:42:29.088 INFO:tasks.rados.rados.0.smithi203.stdout:2369: done (3 left) 2024-09-17T15:42:29.089 INFO:tasks.rados.rados.0.smithi203.stdout:2373: read oid 37 snap -1 2024-09-17T15:42:29.089 INFO:tasks.rados.rados.0.smithi203.stdout:2373: expect (ObjNum 830 snap 254 seq_num 830) 2024-09-17T15:42:29.089 INFO:tasks.rados.rados.0.smithi203.stdout:2374: copy_from oid 15 from oid 25 current snap is 256 2024-09-17T15:42:29.089 INFO:tasks.rados.rados.0.smithi203.stdout:2375: snap_create 2024-09-17T15:42:29.095 INFO:tasks.rados.rados.0.smithi203.stdout:2372: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:42:29.097 INFO:tasks.rados.rados.0.smithi203.stdout:2370: finishing write tid 1 to smithi20328086-5 2024-09-17T15:42:29.097 INFO:tasks.rados.rados.0.smithi203.stdout:2370: finishing write tid 2 to smithi20328086-5 2024-09-17T15:42:29.097 INFO:tasks.rados.rados.0.smithi203.stdout:2370: finishing write tid 3 to smithi20328086-5 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:2370: finishing write tid 4 to smithi20328086-5 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:2370: oid 5 updating version 0 to 526 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:2370: oid 5 updating version 526 to 528 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:2370: oid 5 version 528 is already newer than 527 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 528 (ObjNum 842 snap 256 seq_num 842) dirty exists 2024-09-17T15:42:29.098 INFO:tasks.rados.rados.0.smithi203.stdout:2370: left oid 5 (ObjNum 842 snap 256 seq_num 842) 2024-09-17T15:42:29.100 INFO:tasks.rados.rados.0.smithi203.stdout:2371: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:29.211 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:29.267 INFO:tasks.rados.rados.0.smithi203.stdout:2373: expect (ObjNum 830 snap 254 seq_num 830) 2024-09-17T15:42:29.316 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3132236 2024-09-17T15:42:29.454 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2272090 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:2372: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:2372: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1130 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:2374: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:2374: finishing copy_from to smithi20328086-15 2024-09-17T15:42:29.482 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 884 (ObjNum 819 snap 254 seq_num 819) dirty exists 2024-09-17T15:42:29.732 INFO:tasks.rados.rados.0.smithi203.stdout:2370: done (5 left) 2024-09-17T15:42:29.732 INFO:tasks.rados.rados.0.smithi203.stdout:2371: done (4 left) 2024-09-17T15:42:29.732 INFO:tasks.rados.rados.0.smithi203.stdout:2372: done (3 left) 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2373: done (2 left) 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2374: done (1 left) 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2375: done (0 left) 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2376: rollback oid 49 current snap is 257 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 244 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2377: rollback oid 36 current snap is 257 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 239 2024-09-17T15:42:29.733 INFO:tasks.rados.rados.0.smithi203.stdout:2378: delete oid 6 current snap is 257 2024-09-17T15:42:29.741 INFO:tasks.rados.rados.0.smithi203.stdout:2378: done (2 left) 2024-09-17T15:42:29.742 INFO:tasks.rados.rados.0.smithi203.stdout:2379: write (excl) oid 50 current snap is 257 2024-09-17T15:42:29.742 INFO:tasks.rados.rados.0.smithi203.stdout:2379: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-17T15:42:29.750 INFO:tasks.rados.rados.0.smithi203.stdout:2379: writing smithi20328086-50 from 585108 to 1010447 tid 1 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2379: writing smithi20328086-50 from 1788920 to 2185108 tid 2 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2380: snap_create 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2376: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2376: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2376: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1240 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-17T15:42:29.761 INFO:tasks.rados.rados.0.smithi203.stdout:2379: finishing write tid 1 to smithi20328086-50 2024-09-17T15:42:29.764 INFO:tasks.rados.rados.0.smithi203.stdout:2377: finishing rollback tid 0 to smithi20328086-36 2024-09-17T15:42:29.764 INFO:tasks.rados.rados.0.smithi203.stdout:2377: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:42:29.764 INFO:tasks.rados.rados.0.smithi203.stdout:2377: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:42:29.764 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1244 (ObjNum 700 snap 220 seq_num 700) dirty exists 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: finishing write tid 2 to smithi20328086-50 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: finishing write tid 3 to smithi20328086-50 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: finishing write tid 4 to smithi20328086-50 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: oid 50 updating version 0 to 1169 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: oid 50 updating version 1169 to 1170 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2379: oid 50 version 1170 is already newer than 1168 2024-09-17T15:42:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1170 (ObjNum 844 snap 257 seq_num 844) dirty exists 2024-09-17T15:42:29.769 INFO:tasks.rados.rados.0.smithi203.stdout:2379: left oid 50 (ObjNum 844 snap 257 seq_num 844) 2024-09-17T15:42:31.257 INFO:tasks.rados.rados.0.smithi203.stdout:2376: done (3 left) 2024-09-17T15:42:31.257 INFO:tasks.rados.rados.0.smithi203.stdout:2377: done (2 left) 2024-09-17T15:42:31.257 INFO:tasks.rados.rados.0.smithi203.stdout:2379: done (1 left) 2024-09-17T15:42:31.257 INFO:tasks.rados.rados.0.smithi203.stdout:2380: done (0 left) 2024-09-17T15:42:31.257 INFO:tasks.rados.rados.0.smithi203.stdout:2381: snap_create 2024-09-17T15:42:31.719 INFO:tasks.rados.rados.0.smithi203.stdout:2381: done (0 left) 2024-09-17T15:42:31.719 INFO:tasks.rados.rados.0.smithi203.stdout:2382: snap_create 2024-09-17T15:42:31.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: get_monmap_and_config 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: monmap: 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:32.081993+0000 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:32.081993+0000 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:32.078 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:32.078 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:32.078 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:32.078 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 Processor -- start 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 -- start start 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: init 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:42:32.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0cbc0689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0cbc0694c0 0x7f0cbc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc0694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0cbc13aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cbc09c8b0 con 0x7f0cbc13aff0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cbc09cb90 con 0x7f0cbc0689d0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0cbc074500 con 0x7f0cbc0694c0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cc15ea640 10 monclient(hunting): authenticate will time out at 1885.047422s 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0cbc0694c0 0x7f0cbc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0cbc0694c0 0x7f0cbc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35582/0 (socket says 172.21.15.86:35582) 2024-09-17T15:42:32.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 1 -- 172.21.15.86:0/569320699 learned_addr learned my addr 172.21.15.86:0/569320699 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): get_auth_request con 0x7f0cbc0694c0 auth_method 0 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.079+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cbb7fe640 1 -- 172.21.15.86:0/569320699 >> v1:172.21.15.86:6789/0 conn(0x7f0cbc13aff0 legacy=0x7f0cbc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_done global_id 144429 payload 274 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 1 -- 172.21.15.86:0/569320699 >> v1:172.21.15.203:6789/0 conn(0x7f0cbc0689d0 legacy=0x7f0cbc068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 1 -- 172.21.15.86:0/569320699 >> v1:172.21.15.86:6789/0 conn(0x7f0cbc13aff0 legacy=0x7f0cbc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 1 monclient: found mon.noname-a 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:32.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.083+0000 7f0cba7fc640 1 -- 172.21.15.86:0/569320699 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc09ce70 con 0x7f0cbc0694c0 2024-09-17T15:42:32.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cba7fc640 1 -- 172.21.15.86:0/569320699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0694c0 msgr2=0x7f0cbc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cba7fc640 1 --2- 172.21.15.86:0/569320699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0694c0 0x7f0cbc06f840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00098f0 tx=0x7f0cb001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d2532df15d9eca server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 1 -- 172.21.15.86:0/569320699 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc09c2d0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: dump: 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.086 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 1 -- 172.21.15.86:0/569320699 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb00af070 con 0x7f0cbc0694c0 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 5 monclient: authenticate success, global_id 144429 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 1 -- 172.21.15.86:0/569320699 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00aa920 con 0x7f0cbc0694c0 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 4 set_mon_vals no callback set 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cb9ffb640 10 monclient: dump: 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.089 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 10 monclient: shutdown 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 -- 172.21.15.86:0/569320699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0694c0 msgr2=0x7f0cbc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/569320699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0694c0 0x7f0cbc06f840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00098f0 tx=0x7f0cb001c4a0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 -- 172.21.15.86:0/569320699 shutdown_connections 2024-09-17T15:42:32.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/569320699 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0694c0 0x7f0cbc06f840 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 -- 172.21.15.86:0/569320699 >> 172.21.15.86:0/569320699 conn(0x7f0cbc0a6440 msgr2=0x7f0cbc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 -- 172.21.15.86:0/569320699 shutdown_connections 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.087+0000 7f0cc15ea640 1 -- 172.21.15.86:0/569320699 wait complete. 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient: monmap: 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:32.096 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 Processor -- start 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 -- start start 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient: init 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc0694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cbc13aff0 0x7f0cbc1d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc13aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc141c00 con 0x7f0cbc0689d0 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0cbc141a80 con 0x7f0cbc13aff0 2024-09-17T15:42:32.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0cbc141d80 con 0x7f0cbc0694c0 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cc15ea640 10 monclient(hunting): authenticate will time out at 1885.057514s 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35598/0 (socket says 172.21.15.86:35598) 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37688/0 (socket says 172.21.15.86:37688) 2024-09-17T15:42:32.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 1 -- 172.21.15.86:0/1431436584 learned_addr learned my addr 172.21.15.86:0/1431436584 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): get_auth_request con 0x7f0cbc0689d0 auth_method 0 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): get_auth_request con 0x7f0cbc0694c0 auth_method 0 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbb7fe640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cbc13aff0 0x7f0cbc1d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbb7fe640 10 monclient(hunting): get_auth_request con 0x7f0cbc13aff0 auth_method 0 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient(hunting): handle_auth_done global_id 164189 payload 824 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 -- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cbc13aff0 msgr2=0x7f0cbc1d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cbc13aff0 0x7f0cbc1d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 -- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 msgr2=0x7f0cbc072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 monclient: found mon.noname-c 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cba7fc640 1 -- 172.21.15.86:0/1431436584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cbc073990 con 0x7f0cbc0694c0 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.091+0000 7f0cbaffd640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cba7fc640 1 -- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 msgr2=0x7f0cbc135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cba7fc640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00ae000 tx=0x7f0cb001ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00bd3f0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: dump: 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb00bd5d0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00bd9c0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 5 monclient: authenticate success, global_id 164189 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _renew_subs 2024-09-17T15:42:32.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cbc09d460 con 0x7f0cbc0694c0 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cb97fa640 4 set_mon_vals no callback set 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 10 monclient: dump: 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.105 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _renew_subs 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cbc120dd0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cbc06f840 con 0x7f0cbc0694c0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0cb00bdb60 con 0x7f0cbc0694c0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 0x7f0ca803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(535..535 src has 438..535) ==== 4783+0+0 (secure 0 0 0) 0x7f0cb00ea780 con 0x7f0cbc0694c0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cbaffd640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 0x7f0ca803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.095+0000 7f0cbaffd640 10 monclient: get_auth_request con 0x7f0ca803bb60 auth_method 0 2024-09-17T15:42:32.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.099+0000 7f0cbaffd640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 0x7f0ca803e020 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc120d80 tx=0x7f0cac003e70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.099+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0cbc09d460 con 0x7f0cbc0694c0 2024-09-17T15:42:32.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.099+0000 7f0ca37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.099+0000 7f0ca37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:32.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.335+0000 7f0cc15ea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:32.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:32.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0cbc06d4f0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0ca37fe640 1 -- 172.21.15.86:0/1431436584 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0cb00161b0 con 0x7f0cbc0694c0 2024-09-17T15:42:32.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0ca37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:32.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0ca37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:32.336 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:32.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 msgr2=0x7f0ca803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 0x7f0ca803e020 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc120d80 tx=0x7f0cac003e70 comp rx=0 tx=0).stop 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 10 monclient: shutdown 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 msgr2=0x7f0cbc135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0cb00ae000 tx=0x7f0cb001ace0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 shutdown_connections 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ca803bb60 0x7f0ca803e020 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0cbc13aff0 0x7f0cbc1d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc0694c0 0x7f0cbc135fb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 --2- 172.21.15.86:0/1431436584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0689d0 0x7f0cbc072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 >> 172.21.15.86:0/1431436584 conn(0x7f0cbc0a6440 msgr2=0x7f0cbc13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 shutdown_connections 2024-09-17T15:42:32.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.339+0000 7f0cc15ea640 1 -- 172.21.15.86:0/1431436584 wait complete. 2024-09-17T15:42:32.353 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:42:32.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:42:32.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:42:32.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.479+0000 7ff493df8640 10 monclient: get_monmap_and_config 2024-09-17T15:42:32.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.479+0000 7ff493df8640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.479+0000 7ff493df8640 10 monclient: monmap: 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:32.482541+0000 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:32.482541+0000 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:32.478 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.479+0000 7ff493df8640 1 Processor -- start 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 -- start start 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient: init 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff48c0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 0x7ff48c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-a con 0x7ff48c0a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48c0a6280 con 0x7ff48c0a50e0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff48c09d2a0 con 0x7ff48c0a5bd0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c09c0a0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff493df8640 10 monclient(hunting): authenticate will time out at 1885.448724s 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 0x7ff48c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 0x7ff48c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35614/0 (socket says 172.21.15.86:35614) 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 1 -- 172.21.15.86:0/4147793507 learned_addr learned my addr 172.21.15.86:0/4147793507 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): get_auth_request con 0x7ff48c0a5bd0 auth_method 0 2024-09-17T15:42:32.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): get_auth_request con 0x7ff48c13d2f0 auth_method 0 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff491393640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient(hunting): handle_auth_done global_id 164192 payload 274 2024-09-17T15:42:32.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 -- 172.21.15.86:0/4147793507 >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 msgr2=0x7ff48c09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 --2- 172.21.15.86:0/4147793507 >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 0x7ff48c09cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 -- 172.21.15.86:0/4147793507 >> v1:172.21.15.86:6789/0 conn(0x7ff48c0a50e0 legacy=0x7ff48c0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 monclient: found mon.noname-b 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.483+0000 7ff492395640 1 -- 172.21.15.86:0/4147793507 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c09c380 con 0x7ff48c13d2f0 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff492395640 1 -- 172.21.15.86:0/4147793507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 msgr2=0x7ff48c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff492395640 1 --2- 172.21.15.86:0/4147793507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c13f6e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff484009d90 tx=0x7ff48401c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0ba1fb2b40939bf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 1 -- 172.21.15.86:0/4147793507 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b2070 con 0x7ff48c13d2f0 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:32.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: dump: 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.491 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 5 monclient: authenticate success, global_id 164192 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 1 -- 172.21.15.86:0/4147793507 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4840a5740 con 0x7ff48c13d2f0 2024-09-17T15:42:32.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 1 -- 172.21.15.86:0/4147793507 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840a5b10 con 0x7ff48c13d2f0 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 4 set_mon_vals no callback set 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff490b92640 10 monclient: dump: 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.493 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 10 monclient: shutdown 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- 172.21.15.86:0/4147793507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 msgr2=0x7ff48c13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 --2- 172.21.15.86:0/4147793507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c13f6e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff484009d90 tx=0x7ff48401c7a0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- 172.21.15.86:0/4147793507 shutdown_connections 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 --2- 172.21.15.86:0/4147793507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c13f6e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 --2- 172.21.15.86:0/4147793507 >> v2:172.21.15.86:3300/0 conn(0x7ff48c0a5bd0 0x7ff48c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- 172.21.15.86:0/4147793507 >> 172.21.15.86:0/4147793507 conn(0x7ff48c0aaae0 msgr2=0x7ff48c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- 172.21.15.86:0/4147793507 shutdown_connections 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- 172.21.15.86:0/4147793507 wait complete. 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 10 monclient: monmap: 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:32.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 Processor -- start 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 1 -- start start 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.487+0000 7ff493df8640 10 monclient: init 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 0x7ff48c121410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-a con 0x7ff48c0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c0a5bd0 0x7ff48c09fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c0a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:32.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c142650 con 0x7ff48c0a50e0 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0a6b00 con 0x7ff48c0a5bd0 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c1427d0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient(hunting): authenticate will time out at 1885.456387s 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 0x7ff48c121410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c138280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37706/0 (socket says 172.21.15.86:37706) 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 0x7ff48c121410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35616/0 (socket says 172.21.15.86:35616) 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 learned_addr learned my addr 172.21.15.86:0/1978094380 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): get_auth_request con 0x7ff48c13d2f0 auth_method 0 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): get_auth_request con 0x7ff48c0a50e0 auth_method 0 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491393640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c0a5bd0 0x7ff48c09fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491393640 10 monclient(hunting): get_auth_request con 0x7ff48c0a5bd0 auth_method 0 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491393640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491393640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491393640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient(hunting): handle_auth_done global_id 164195 payload 824 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c0a5bd0 msgr2=0x7ff48c09fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff48c0a5bd0 0x7ff48c09fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 msgr2=0x7ff48c121410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 0x7ff48c121410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 monclient: found mon.noname-c 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c0a12d0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff491b94640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c0a50e0 0x7ff48c121410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 msgr2=0x7ff48c138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff492395640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c138280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff484009d90 tx=0x7ff4840195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840b2050 con 0x7ff48c13d2f0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:32.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: dump: 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4840be070 con 0x7ff48c13d2f0 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 5 monclient: authenticate success, global_id 164195 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _renew_subs 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c122c10 con 0x7ff48c13d2f0 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840ada80 con 0x7ff48c13d2f0 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff4827fc640 10 monclient: dump: 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.502 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff483fff640 4 set_mon_vals no callback set 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _renew_subs 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.491+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c0a0bd0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff493df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff493df8640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 0x7ff48c1cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c06a0e0 con 0x7ff48c1387c0 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff4840adca0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 0x7ff48c1cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient: get_auth_request con 0x7ff48c1387c0 auth_method 0 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff47003bbb0 0x7ff47003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(535..535 src has 438..535) ==== 4783+0+0 (secure 0 0 0) 0x7ff4840ea8f0 con 0x7ff48c13d2f0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 10 monclient(hunting): handle_auth_done global_id 144441 payload 824 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff48c0a18d0 con 0x7ff48c1387c0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491393640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff47003bbb0 0x7ff47003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491393640 10 monclient: get_auth_request con 0x7ff47003bbb0 auth_method 0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 msgr2=0x7ff48c1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff491b94640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 0x7ff48c1cfba0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff488007fd0 tx=0x7ff48800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488018070 con 0x7ff48c1387c0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 10 monclient: dump: 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff48800ddc0 con 0x7ff48c1387c0 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 msgr2=0x7ff48c1cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 0x7ff48c1cfba0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff488007fd0 tx=0x7ff48800ff70 comp rx=0 tx=0).stop 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 msgr2=0x7ff48c1cfba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:32.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.495+0000 7ff4827fc640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1387c0 0x7ff48c1cfba0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff488007fd0 tx=0x7ff48800ff70 comp rx=0 tx=0).stop 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.499+0000 7ff491393640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff47003bbb0 0x7ff47003e070 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff48c121f50 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff493df8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff493df8640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 0x7ff48c1d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff493df8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c1d0a30 con 0x7ff48c1d00e0 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 0x7ff48c1d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient: get_auth_request con 0x7ff48c1d00e0 auth_method 0 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 10 monclient(hunting): handle_auth_done global_id 144444 payload 824 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.503+0000 7ff491b94640 1 -- 172.21.15.86:0/1978094380 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff48c09cd60 con 0x7ff48c1d00e0 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff491b94640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 msgr2=0x7ff48c1d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff491b94640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 0x7ff48c1d04f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff488017000 tx=0x7ff488003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff488006e00 con 0x7ff48c1d00e0 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 10 monclient: dump: 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7ff488006e00 con 0x7ff48c1d00e0 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 msgr2=0x7ff48c1d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 0x7ff48c1d04f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff488017000 tx=0x7ff488003770 comp rx=0 tx=0).stop 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 msgr2=0x7ff48c1d04f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff4827fc640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff48c1d00e0 0x7ff48c1d04f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 170849, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:42:32.514 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.515 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.516 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.517 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:32.518 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.519 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:32.521 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:32.523 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.524 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.525 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.526 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:32.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.528 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:32.529 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff47003bbb0 msgr2=0x7ff47003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff47003bbb0 0x7ff47003e070 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff48c121f50 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 10 monclient: shutdown 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 msgr2=0x7ff48c138280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 --2- 172.21.15.86:0/1978094380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff48c13d2f0 0x7ff48c138280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff484009d90 tx=0x7ff4840195a0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 reap_dead start 2024-09-17T15:42:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff492395640 1 -- 172.21.15.86:0/1978094380 reap_dead start 2024-09-17T15:42:32.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 shutdown_connections 2024-09-17T15:42:32.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 >> 172.21.15.86:0/1978094380 conn(0x7ff48c0aaae0 msgr2=0x7ff48c13dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 shutdown_connections 2024-09-17T15:42:32.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.507+0000 7ff493df8640 1 -- 172.21.15.86:0/1978094380 wait complete. 2024-09-17T15:42:32.531 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:32.531 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:42:32.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: get_monmap_and_config 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: monmap: 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:32.652230+0000 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:32.652230+0000 2024-09-17T15:42:32.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:32.648 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 Processor -- start 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 -- start start 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: init 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f0c0a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 0x7f3f0c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0c0a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c13b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0c0a2130 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3f0c0a2410 con 0x7f3f0c0a1760 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c0a26f0 con 0x7f3f0c13b450 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f13749640 10 monclient(hunting): authenticate will time out at 1885.617697s 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 0x7f3f0c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 0x7f3f0c13af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35626/0 (socket says 172.21.15.86:35626) 2024-09-17T15:42:32.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 1 -- 172.21.15.86:0/861563568 learned_addr learned my addr 172.21.15.86:0/861563568 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): get_auth_request con 0x7f3f0c0a1760 auth_method 0 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): get_auth_request con 0x7f3f0c13b450 auth_method 0 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_done global_id 164198 payload 274 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 -- 172.21.15.86:0/861563568 >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 msgr2=0x7f3f0c13af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 --2- 172.21.15.86:0/861563568 >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 0x7f3f0c13af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 -- 172.21.15.86:0/861563568 >> v1:172.21.15.203:6789/0 conn(0x7f3f0c0a0c70 legacy=0x7f3f0c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 monclient: found mon.noname-b 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.651+0000 7f3f11ce6640 1 -- 172.21.15.86:0/861563568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c09f9f0 con 0x7f3f0c13b450 2024-09-17T15:42:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f11ce6640 1 -- 172.21.15.86:0/861563568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 msgr2=0x7f3f0c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f11ce6640 1 --2- 172.21.15.86:0/861563568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c13d840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009d90 tx=0x7f3f0801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ce8ecedeca9687 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 1 -- 172.21.15.86:0/861563568 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f080b2070 con 0x7f3f0c13b450 2024-09-17T15:42:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: dump: 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 1 -- 172.21.15.86:0/861563568 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f080a5890 con 0x7f3f0c13b450 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 5 monclient: authenticate success, global_id 164198 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 1 -- 172.21.15.86:0/861563568 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f080a5c60 con 0x7f3f0c13b450 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 4 set_mon_vals no callback set 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f03fff640 10 monclient: dump: 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 10 monclient: shutdown 2024-09-17T15:42:32.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 -- 172.21.15.86:0/861563568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 msgr2=0x7f3f0c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 --2- 172.21.15.86:0/861563568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c13d840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3f08009d90 tx=0x7f3f0801c830 comp rx=0 tx=0).stop 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 -- 172.21.15.86:0/861563568 shutdown_connections 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 --2- 172.21.15.86:0/861563568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c13d840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 --2- 172.21.15.86:0/861563568 >> v2:172.21.15.86:3300/0 conn(0x7f3f0c0a1760 0x7f3f0c13af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 -- 172.21.15.86:0/861563568 >> 172.21.15.86:0/861563568 conn(0x7f3f0c0a2f40 msgr2=0x7f3f0c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 -- 172.21.15.86:0/861563568 shutdown_connections 2024-09-17T15:42:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 1 -- 172.21.15.86:0/861563568 wait complete. 2024-09-17T15:42:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.655+0000 7f3f13749640 10 monclient: monmap: 2024-09-17T15:42:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 Processor -- start 2024-09-17T15:42:32.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 -- start start 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: init 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-a con 0x7f3f0c0a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f0c0a1760 0x7f3f0c1ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-b con 0x7f3f0c0a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c137e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): picked mon.noname-c con 0x7f3f0c13b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c140720 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1405a0 con 0x7f3f0c0a1760 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c1408a0 con 0x7f3f0c13b450 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient(hunting): authenticate will time out at 1885.624902s 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35638/0 (socket says 172.21.15.86:35638) 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 -- 172.21.15.86:0/1791192926 learned_addr learned my addr 172.21.15.86:0/1791192926 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): get_auth_request con 0x7f3f0c0a0c70 auth_method 0 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): get_auth_request con 0x7f3f0c13b450 auth_method 0 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f11ce6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f0c0a1760 0x7f3f0c1ce360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient(hunting): handle_auth_done global_id 144450 payload 824 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 msgr2=0x7f3f0c137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c13b450 0x7f3f0c137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f0c0a1760 msgr2=0x7f3f0c1ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f0c0a1760 0x7f3f0c1ce360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 monclient: found mon.noname-a 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c13b8e0 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f10ce4640 10 monclient: get_auth_request con 0x7f3f0c0a1760 auth_method 0 2024-09-17T15:42:32.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3f0c0a1760 0x7f3f0c1ce360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:32.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 msgr2=0x7f3f0c1cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f114e5640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c450 tx=0x7f3f0400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04012a40 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: dump: 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f04017070 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 5 monclient: authenticate success, global_id 144450 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04012e40 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f01ffb640 10 monclient: dump: 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.674 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _renew_subs 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f037fe640 4 set_mon_vals no callback set 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0c09d460 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _renew_subs 2024-09-17T15:42:32.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.659+0000 7f3f13749640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0c11d620 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f13749640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f13749640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 0x7f3f0c1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c0a1050 con 0x7f3f0c1ce8a0 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 0x7f3f0c1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient: get_auth_request con 0x7f3f0c1ce8a0 auth_method 0 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3f0401c460 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_done global_id 164204 payload 824 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f0c0008e0 con 0x7f3f0c1ce8a0 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3ef003bb60 0x7f3ef003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(535..535 src has 438..535) ==== 4783+0+0 (secure 0 0 0) 0x7f3f0404fd00 con 0x7f3f0c0a0c70 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f11ce6640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3ef003bb60 0x7f3ef003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f11ce6640 10 monclient: get_auth_request con 0x7f3ef003bb60 auth_method 0 2024-09-17T15:42:32.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 msgr2=0x7f3f0c1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 0x7f3f0c1383b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3efc010810 tx=0x7f3efc010cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc00f040 con 0x7f3f0c1ce8a0 2024-09-17T15:42:32.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 10 monclient: dump: 2024-09-17T15:42:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3efc004020 con 0x7f3f0c1ce8a0 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:32.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 msgr2=0x7f3f0c1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 0x7f3f0c1383b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3efc010810 tx=0x7f3efc010cd0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 msgr2=0x7f3f0c1383b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.663+0000 7f3f01ffb640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1ce8a0 0x7f3f0c1383b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.667+0000 7f3f11ce6640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3ef003bb60 0x7f3ef003e020 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c06d160 tx=0x7f3f080a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f13749640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f13749640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 0x7f3f0c06fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f13749640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c138d00 con 0x7f3f0c1388f0 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 0x7f3f0c06fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient: get_auth_request con 0x7f3f0c1388f0 auth_method 0 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 10 monclient(hunting): handle_auth_done global_id 164207 payload 824 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.671+0000 7f3f10ce4640 1 -- 172.21.15.86:0/1791192926 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f0c11d090 con 0x7f3f0c1388f0 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f10ce4640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 msgr2=0x7f3f0c06fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f10ce4640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 0x7f3f0c06fa00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3efc008010 tx=0x7f3efc0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc027470 con 0x7f3f0c1388f0 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 10 monclient: dump: 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.682 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f3efc01c400 con 0x7f3f0c1388f0 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:32.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 msgr2=0x7f3f0c06fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 0x7f3f0c06fa00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3efc008010 tx=0x7f3efc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 msgr2=0x7f3f0c06fa00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f01ffb640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3f0c1388f0 0x7f3f0c06fa00 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3efc008010 tx=0x7f3efc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:32.684 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 55863, 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.685 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.686 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.687 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:32.688 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:32.690 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:32.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:32.692 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.694 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:32.697 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.698 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:32.699 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:32.700 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3ef003bb60 msgr2=0x7f3ef003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3ef003bb60 0x7f3ef003e020 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c06d160 tx=0x7f3f080a7070 comp rx=0 tx=0).stop 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 10 monclient: shutdown 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 msgr2=0x7f3f0c1cdc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3f0400c450 tx=0x7f3f0400c910 comp rx=0 tx=0).stop 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f11ce6640 1 -- 172.21.15.86:0/1791192926 reap_dead start 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 shutdown_connections 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 --2- 172.21.15.86:0/1791192926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3f0c0a0c70 0x7f3f0c1cdc30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 >> 172.21.15.86:0/1791192926 conn(0x7f3f0c0a2f40 msgr2=0x7f3f0c0a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 shutdown_connections 2024-09-17T15:42:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.675+0000 7f3f13749640 1 -- 172.21.15.86:0/1791192926 wait complete. 2024-09-17T15:42:32.704 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:32.704 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:42:32.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:42:32.715 INFO:tasks.rados.rados.0.smithi203.stdout:2382: done (0 left) 2024-09-17T15:42:32.715 INFO:tasks.rados.rados.0.smithi203.stdout:2383: write oid 30 current snap is 260 2024-09-17T15:42:32.715 INFO:tasks.rados.rados.0.smithi203.stdout:2383: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-17T15:42:32.730 INFO:tasks.rados.rados.0.smithi203.stdout:2383: writing smithi20328086-30 from 601915 to 1102503 tid 1 2024-09-17T15:42:32.745 INFO:tasks.rados.rados.0.smithi203.stdout:2383: writing smithi20328086-30 from 1847402 to 2201915 tid 2 2024-09-17T15:42:32.745 INFO:tasks.rados.rados.0.smithi203.stdout:2384: read oid 39 snap -1 2024-09-17T15:42:32.745 INFO:tasks.rados.rados.0.smithi203.stdout:2384: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:42:32.745 INFO:tasks.rados.rados.0.smithi203.stdout:2385: copy_from oid 31 from oid 19 current snap is 260 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2386: read oid 5 snap -1 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2386: expect (ObjNum 842 snap 256 seq_num 842) 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2387: copy_from oid 25 from oid 26 current snap is 260 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2383: finishing write tid 1 to smithi20328086-30 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2388: read oid 18 snap -1 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2388: expect (ObjNum 835 snap 256 seq_num 835) 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2389: write oid 3 current snap is 260 2024-09-17T15:42:32.746 INFO:tasks.rados.rados.0.smithi203.stdout:2389: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-17T15:42:32.762 INFO:tasks.rados.rados.0.smithi203.stdout:2389: writing smithi20328086-3 from 618722 to 1194559 tid 1 2024-09-17T15:42:32.774 INFO:tasks.rados.rados.0.smithi203.stdout:2389: writing smithi20328086-3 from 1905884 to 2218722 tid 2 2024-09-17T15:42:32.774 INFO:tasks.rados.rados.0.smithi203.stdout:2390: snap_remove snap 259 2024-09-17T15:42:32.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: get_monmap_and_config 2024-09-17T15:42:32.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: monmap: 2024-09-17T15:42:32.837 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:32.829018+0000 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:32.829018+0000 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 Processor -- start 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 -- start start 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: init 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c0a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 0x7f5f5c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c0a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5f5c0a2410 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5f5c0a2130 con 0x7f5f5c0a1590 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5c0a26f0 con 0x7f5f5c135f90 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f61b06640 10 monclient(hunting): authenticate will time out at 1885.794403s 2024-09-17T15:42:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35646/0 (socket says 172.21.15.86:35646) 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/103089717 learned_addr learned my addr 172.21.15.86:0/103089717 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f5c0a0c70 auth_method 0 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request con 0x7f5f5c135f90 auth_method 0 2024-09-17T15:42:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_done global_id 144453 payload 274 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 msgr2=0x7f5f5c136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5affd640 1 --2- 172.21.15.86:0/103089717 >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 0x7f5f5c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/103089717 >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 msgr2=0x7f5f5c135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/103089717 >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 0x7f5f5c135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:32.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/103089717 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5c09f9f0 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5affd640 10 monclient: get_auth_request con 0x7f5f5c0a1590 auth_method 0 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.827+0000 7f5f5affd640 1 --2- 172.21.15.86:0/103089717 >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 0x7f5f5c135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 msgr2=0x7f5f5c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c074ae0 tx=0x7f5f4c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bb2a4aeec172c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 1 -- 172.21.15.86:0/103089717 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0a5750 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: dump: 2024-09-17T15:42:32.845 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.846 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 5 monclient: authenticate success, global_id 144453 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 1 -- 172.21.15.86:0/103089717 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4c0a58f0 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 4 set_mon_vals no callback set 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 1 -- 172.21.15.86:0/103089717 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0a5cc0 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f5a7fc640 10 monclient: dump: 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 10 monclient: shutdown 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 -- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 msgr2=0x7f5f5c0a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 --2- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c074ae0 tx=0x7f5f4c01c8b0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 -- 172.21.15.86:0/103089717 shutdown_connections 2024-09-17T15:42:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 --2- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 --2- 172.21.15.86:0/103089717 >> v2:172.21.15.203:3300/0 conn(0x7f5f5c0a1590 0x7f5f5c135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 --2- 172.21.15.86:0/103089717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c0a1050 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 -- 172.21.15.86:0/103089717 >> 172.21.15.86:0/103089717 conn(0x7f5f5c0a2f40 msgr2=0x7f5f5c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 -- 172.21.15.86:0/103089717 shutdown_connections 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.831+0000 7f5f61b06640 1 -- 172.21.15.86:0/103089717 wait complete. 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient: build_initial_monmap 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient: monmap: 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 Processor -- start 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 -- start start 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient: init 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c1cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c0a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c0a1590 0x7f5f5c134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c0a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c0a00d0 con 0x7f5f5c0a0c70 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c09ff50 con 0x7f5f5c0a1590 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5c13f740 con 0x7f5f5c135f90 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f61b06640 10 monclient(hunting): authenticate will time out at 1885.801374s 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c1cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c1cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35648/0 (socket says 172.21.15.86:35648) 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37784/0 (socket says 172.21.15.86:37784) 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 learned_addr learned my addr 172.21.15.86:0/1149094696 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5f5c0a0c70 auth_method 0 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request con 0x7f5f5c135f90 auth_method 0 2024-09-17T15:42:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5affd640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c0a1590 0x7f5f5c134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient(hunting): handle_auth_done global_id 164213 payload 824 2024-09-17T15:42:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c0a1590 msgr2=0x7f5f5c134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c0a1590 0x7f5f5c134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 msgr2=0x7f5f5c1cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c1cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 monclient: found mon.noname-c 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5bfff640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5c1208b0 con 0x7f5f5c135f90 2024-09-17T15:42:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f5c0a0c70 0x7f5f5c1cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5affd640 10 monclient: get_auth_request con 0x7f5f5c0a1590 auth_method 0 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.835+0000 7f5f5affd640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c0a1590 0x7f5f5c134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5bfff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 msgr2=0x7f5f5c11cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5bfff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009ab0 tx=0x7f5f54009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5400bca0 con 0x7f5f5c135f90 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: dump: 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: _finish_auth 0 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5400be40 con 0x7f5f5c135f90 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 5 monclient: authenticate success, global_id 164213 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 monclient: _renew_subs 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5c09d460 con 0x7f5f5c135f90 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f54004240 con 0x7f5f5c135f90 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 10 monclient: dump: 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 monclient: _renew_subs 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f5c1205d0 con 0x7f5f5c135f90 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f59ffb640 4 set_mon_vals no callback set 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c09f780 con 0x7f5f5c11f0b0 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5f54003620 con 0x7f5f5c135f90 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f4803bbb0 0x7f5f4803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c11f0b0 auth_method 0 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(536..536 src has 438..536) ==== 4755+0+0 (secure 0 0 0) 0x7f5f54053970 con 0x7f5f5c135f90 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:42:32.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:42:32.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:32.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:32.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5affd640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f4803bbb0 0x7f5f4803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:32.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.839+0000 7f5f5affd640 10 monclient: get_auth_request con 0x7f5f4803bbb0 auth_method 0 2024-09-17T15:42:32.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:32.843+0000 7f5f5affd640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f4803bbb0 0x7f5f4803e070 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5f50008930 tx=0x7f5f50008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c11f0b0 auth_method 0 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.043+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:33.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:33.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c11f0b0 auth_method 0 2024-09-17T15:42:33.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:33.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:33.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:33.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:42:33.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:42:33.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:33.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:33.447+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:33.735 INFO:tasks.rados.rados.0.smithi203.stdout:2390: done (7 left) 2024-09-17T15:42:33.735 INFO:tasks.rados.rados.0.smithi203.stdout:2391: read oid 22 snap -1 2024-09-17T15:42:33.735 INFO:tasks.rados.rados.0.smithi203.stdout:2391: expect (ObjNum 827 snap 254 seq_num 827) 2024-09-17T15:42:33.736 INFO:tasks.rados.rados.0.smithi203.stdout:2392: read oid 7 snap -1 2024-09-17T15:42:33.736 INFO:tasks.rados.rados.0.smithi203.stdout:2392: expect (ObjNum 828 snap 254 seq_num 828) 2024-09-17T15:42:33.736 INFO:tasks.rados.rados.0.smithi203.stdout:2393: snap_create 2024-09-17T15:42:33.736 INFO:tasks.rados.rados.0.smithi203.stdout:2388: expect (ObjNum 835 snap 256 seq_num 835) 2024-09-17T15:42:33.762 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1489935 2024-09-17T15:42:33.793 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: finishing write tid 2 to smithi20328086-30 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: finishing write tid 3 to smithi20328086-30 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: finishing write tid 4 to smithi20328086-30 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: oid 30 updating version 0 to 1133 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: oid 30 version 1133 is already newer than 1132 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:2383: oid 30 version 1133 is already newer than 1131 2024-09-17T15:42:33.865 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1133 (ObjNum 845 snap 260 seq_num 845) dirty exists 2024-09-17T15:42:33.866 INFO:tasks.rados.rados.0.smithi203.stdout:2383: left oid 30 (ObjNum 845 snap 260 seq_num 845) 2024-09-17T15:42:33.866 INFO:tasks.rados.rados.0.smithi203.stdout:2385: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:42:33.866 INFO:tasks.rados.rados.0.smithi203.stdout:2386: expect (ObjNum 842 snap 256 seq_num 842) 2024-09-17T15:42:33.866 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:33.893 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:33.951 INFO:tasks.rados.rados.0.smithi203.stdout:2385: finishing copy_from to smithi20328086-31 2024-09-17T15:42:33.951 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1128 (ObjNum 763 snap 236 seq_num 763) dirty exists 2024-09-17T15:42:33.952 INFO:tasks.rados.rados.0.smithi203.stdout:2384: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:42:34.037 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1465825 2024-09-17T15:42:34.076 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2389: finishing write tid 1 to smithi20328086-3 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2387: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2387: finishing copy_from to smithi20328086-25 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2389: finishing write tid 2 to smithi20328086-3 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2389: finishing write tid 3 to smithi20328086-3 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2389: finishing write tid 4 to smithi20328086-3 2024-09-17T15:42:34.142 INFO:tasks.rados.rados.0.smithi203.stdout:2389: oid 3 updating version 0 to 1108 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:2389: oid 3 version 1108 is already newer than 1106 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:2389: oid 3 version 1108 is already newer than 1107 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1108 (ObjNum 846 snap 260 seq_num 846) dirty exists 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:2389: left oid 3 (ObjNum 846 snap 260 seq_num 846) 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:2392: expect (ObjNum 828 snap 254 seq_num 828) 2024-09-17T15:42:34.143 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:34.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c11f0b0 auth_method 0 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:34.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:34.251+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:42:34.307 INFO:tasks.rados.rados.0.smithi203.stdout:2391: expect (ObjNum 827 snap 254 seq_num 827) 2024-09-17T15:42:34.307 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:34.448 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:2383: done (9 left) 2024-09-17T15:42:34.746 INFO:tasks.rados.rados.0.smithi203.stdout:2384: done (8 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2385: done (7 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2386: done (6 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2387: done (5 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2388: done (4 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2389: done (3 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2391: done (2 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2392: done (1 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2393: done (0 left) 2024-09-17T15:42:34.747 INFO:tasks.rados.rados.0.smithi203.stdout:2394: delete oid 1 current snap is 261 2024-09-17T15:42:34.753 INFO:tasks.rados.rados.0.smithi203.stdout:2394: done (0 left) 2024-09-17T15:42:34.753 INFO:tasks.rados.rados.0.smithi203.stdout:2395: read oid 31 snap -1 2024-09-17T15:42:34.753 INFO:tasks.rados.rados.0.smithi203.stdout:2395: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:42:34.753 INFO:tasks.rados.rados.0.smithi203.stdout:2396: snap_remove snap 257 2024-09-17T15:42:35.754 INFO:tasks.rados.rados.0.smithi203.stdout:2396: done (1 left) 2024-09-17T15:42:35.754 INFO:tasks.rados.rados.0.smithi203.stdout:2397: delete oid 24 current snap is 261 2024-09-17T15:42:35.756 INFO:tasks.rados.rados.0.smithi203.stdout:2395: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:42:35.787 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 284657 2024-09-17T15:42:35.813 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:35.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.835+0000 7f5f42ffd640 10 monclient: tick 2024-09-17T15:42:35.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.835+0000 7f5f42ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:35.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.835+0000 7f5f42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:35.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.835+0000 7f5f42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2395: done (1 left) 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2397: done (0 left) 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2398: read oid 2 snap -1 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2398: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2399: read oid 27 snap -1 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2399: expect (ObjNum 823 snap 254 seq_num 823) 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2400: copy_from oid 20 from oid 14 current snap is 261 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2401: write (excl) oid 30 current snap is 261 2024-09-17T15:42:35.842 INFO:tasks.rados.rados.0.smithi203.stdout:2401: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-17T15:42:35.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:35.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c11f0b0 auth_method 0 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_done global_id 174100 payload 824 2024-09-17T15:42:35.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.855+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f5c135a50 con 0x7f5f5c11f0b0 2024-09-17T15:42:35.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:35.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c01d150 tx=0x7f5f4c019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:35.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0c1640 con 0x7f5f5c11f0b0 2024-09-17T15:42:35.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:35.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:35.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 10 monclient: dump: 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:35.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5f4c0c1640 con 0x7f5f5c11f0b0 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c01d150 tx=0x7f5f4c019650 comp rx=0 tx=0).stop 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 msgr2=0x7f5f5c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:35.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.859+0000 7f5f43fff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c11f0b0 0x7f5f5c02d450 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:35.859 INFO:tasks.rados.rados.0.smithi203.stdout:2401: writing smithi20328086-30 from 669143 to 1070727 tid 1 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f61b06640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f61b06640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 0x7f5f5c02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f61b06640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5c0726d0 con 0x7f5f5c120030 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 0x7f5f5c02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient: get_auth_request con 0x7f5f5c120030 auth_method 0 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:35.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:35.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:35.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 10 monclient(hunting): handle_auth_done global_id 174103 payload 824 2024-09-17T15:42:35.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.867+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f5c1366d0 con 0x7f5f5c120030 2024-09-17T15:42:35.868 INFO:tasks.rados.rados.0.smithi203.stdout:2401: writing smithi20328086-30 from 1564977 to 1991936 tid 2 2024-09-17T15:42:35.868 INFO:tasks.rados.rados.0.smithi203.stdout:2402: delete oid 16 current snap is 261 2024-09-17T15:42:35.868 INFO:tasks.rados.rados.0.smithi203.stdout:2398: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:42:35.868 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:35.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f5b7fe640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 msgr2=0x7f5f5c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:35.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f5b7fe640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 0x7f5f5c02d990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009850 tx=0x7f5f4c0b8040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:35.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c0cd070 con 0x7f5f5c120030 2024-09-17T15:42:35.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:35.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:35.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 10 monclient: dump: 2024-09-17T15:42:35.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f5f4c0c9b60 con 0x7f5f5c120030 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 msgr2=0x7f5f5c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:35.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 0x7f5f5c02d990 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009850 tx=0x7f5f4c0b8040 comp rx=0 tx=0).stop 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 msgr2=0x7f5f5c02d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.871+0000 7f5f43fff640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f5c120030 0x7f5f5c02d990 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009850 tx=0x7f5f4c0b8040 comp rx=0 tx=0).stop 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:35.872 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10663, 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:35.873 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:35.874 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:35.875 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:35.876 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:35.877 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:35.878 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:35.879 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:35.880 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:35.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:35.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.883 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:35.884 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:35.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f4803bbb0 msgr2=0x7f5f4803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f4803bbb0 0x7f5f4803e070 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5f50008930 tx=0x7f5f50008040 comp rx=0 tx=0).stop 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 10 monclient: shutdown 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 msgr2=0x7f5f5c11cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009ab0 tx=0x7f5f54009f70 comp rx=0 tx=0).stop 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f5bfff640 1 -- 172.21.15.86:0/1149094696 reap_dead start 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 shutdown_connections 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 --2- 172.21.15.86:0/1149094696 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f5c135f90 0x7f5f5c11cce0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:35.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 >> 172.21.15.86:0/1149094696 conn(0x7f5f5c0a2f40 msgr2=0x7f5f5c074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:35.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 shutdown_connections 2024-09-17T15:42:35.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:35.875+0000 7f5f61b06640 1 -- 172.21.15.86:0/1149094696 wait complete. 2024-09-17T15:42:35.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:35.894 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:42:35.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:42:35.980 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.015+0000 7f6a08f07640 10 monclient: get_monmap_and_config 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.015+0000 7f6a08f07640 10 monclient: build_initial_monmap 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.015+0000 7f6a08f07640 10 monclient: monmap: 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:36.019214+0000 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:36.019214+0000 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.014 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:36.015 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:36.015 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:36.015 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:36.015 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:36.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 Processor -- start 2024-09-17T15:42:36.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 -- start start 2024-09-17T15:42:36.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient: init 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a040a50e0 0x7f6a0409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-a con 0x7f6a040a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a040a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a0413d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0409d350 con 0x7f6a0413d2f0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a040a6530 con 0x7f6a040a54c0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6a0409bdc0 con 0x7f6a040a50e0 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a08f07640 10 monclient(hunting): authenticate will time out at 1888.984864s 2024-09-17T15:42:36.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a040a50e0 0x7f6a0409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a040a50e0 0x7f6a0409ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35652/0 (socket says 172.21.15.86:35652) 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 -- 172.21.15.86:0/1532544105 learned_addr learned my addr 172.21.15.86:0/1532544105 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): get_auth_request con 0x7f6a040a50e0 auth_method 0 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a02d9d640 1 --1- 172.21.15.86:0/1532544105 >> v1:172.21.15.86:6789/0 conn(0x7f6a0413d2f0 0x7f6a0413f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:55002/0 (socket says 172.21.15.86:55002) 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:36.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a02d9d640 1 -- 172.21.15.86:0/1532544105 >> v1:172.21.15.86:6789/0 conn(0x7f6a0413d2f0 legacy=0x7f6a0413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient(hunting): handle_auth_done global_id 144459 payload 274 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 -- 172.21.15.86:0/1532544105 >> v1:172.21.15.203:6789/0 conn(0x7f6a040a54c0 legacy=0x7f6a040a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a01d9b640 1 -- 172.21.15.86:0/1532544105 >> v1:172.21.15.203:6789/0 conn(0x7f6a040a54c0 legacy=0x7f6a040a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 -- 172.21.15.86:0/1532544105 >> v1:172.21.15.86:6789/0 conn(0x7f6a0413d2f0 legacy=0x7f6a0413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient: _finish_hunting 0 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 monclient: found mon.noname-a 2024-09-17T15:42:36.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:36.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.019+0000 7f6a0259c640 1 -- 172.21.15.86:0/1532544105 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0409c150 con 0x7f6a040a50e0 2024-09-17T15:42:36.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0259c640 1 -- 172.21.15.86:0/1532544105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a040a50e0 msgr2=0x7f6a0409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:36.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0259c640 1 --2- 172.21.15.86:0/1532544105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a040a50e0 0x7f6a0409ce10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009970 tx=0x7f69ec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d2426f52c698b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:36.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 1 -- 172.21.15.86:0/1532544105 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0ab430 con 0x7f6a040a50e0 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: dump: 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:36.023 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: _finish_auth 0 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: _check_auth_tickets 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 1 -- 172.21.15.86:0/1532544105 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ec0b0070 con 0x7f6a040a50e0 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a08f07640 5 monclient: authenticate success, global_id 144459 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a08f07640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 1 -- 172.21.15.86:0/1532544105 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ec0ab7f0 con 0x7f6a040a50e0 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a08f07640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:36.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a08f07640 4 set_mon_vals no callback set 2024-09-17T15:42:36.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.023+0000 7f6a0159a640 10 monclient: dump: 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:36.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.028 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:36.028 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:36.028 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:36.028 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: shutdown 2024-09-17T15:42:36.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- 172.21.15.86:0/1532544105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a040a50e0 msgr2=0x7f6a0409ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 --2- 172.21.15.86:0/1532544105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a040a50e0 0x7f6a0409ce10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f69ec009970 tx=0x7f69ec01c520 comp rx=0 tx=0).stop 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- 172.21.15.86:0/1532544105 shutdown_connections 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 --2- 172.21.15.86:0/1532544105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a040a50e0 0x7f6a0409ce10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- 172.21.15.86:0/1532544105 >> 172.21.15.86:0/1532544105 conn(0x7f6a040aaae0 msgr2=0x7f6a040aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- 172.21.15.86:0/1532544105 shutdown_connections 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- 172.21.15.86:0/1532544105 wait complete. 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: build_initial_monmap 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: monmap: 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:36.034 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 Processor -- start 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- start start 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: init 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a040a54c0 0x7f6a041cfe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:36.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-b con 0x7f6a040a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a0413a320 0x7f6a0413a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0413a320 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6a04143a60 con 0x7f6a0413d2f0 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6a041438e0 con 0x7f6a040a54c0 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6a04143be0 con 0x7f6a0413a320 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a08f07640 10 monclient(hunting): authenticate will time out at 1888.994284s 2024-09-17T15:42:36.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35656/0 (socket says 172.21.15.86:35656) 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 -- 172.21.15.86:0/3705296610 learned_addr learned my addr 172.21.15.86:0/3705296610 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): get_auth_request con 0x7f6a0413d2f0 auth_method 0 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a0413a320 0x7f6a0413a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): get_auth_request con 0x7f6a0413a320 auth_method 0 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:36.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a0259c640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a040a54c0 0x7f6a041cfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a0259c640 10 monclient(hunting): get_auth_request con 0x7f6a040a54c0 auth_method 0 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a0259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient(hunting): handle_auth_done global_id 144465 payload 824 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 -- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a0413a320 msgr2=0x7f6a0413a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a0413a320 0x7f6a0413a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 -- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a040a54c0 msgr2=0x7f6a041cfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a040a54c0 0x7f6a041cfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 monclient: found mon.noname-a 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.027+0000 7f6a01d9b640 1 -- 172.21.15.86:0/3705296610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0413bcf0 con 0x7f6a0413d2f0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a01d9b640 1 -- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 msgr2=0x7f6a04138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a01d9b640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c870 tx=0x7f69f400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4017070 con 0x7f6a0413d2f0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: dump: 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: _finish_auth 0 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69f400a840 con 0x7f6a0413d2f0 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 5 monclient: authenticate success, global_id 144465 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _renew_subs 2024-09-17T15:42:36.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a041d0d70 con 0x7f6a0413d2f0 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f400ac30 con 0x7f6a0413d2f0 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 10 monclient: dump: 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:36.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a00d99640 4 set_mon_vals no callback set 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _renew_subs 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a040a18d0 con 0x7f6a0413d2f0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f69f401a070 con 0x7f6a0413d2f0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a0409ce10 con 0x7f6a0413d2f0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 0x7f69e403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(539..539 src has 438..539) ==== 4815+0+0 (secure 0 0 0) 0x7f69f404f5b0 con 0x7f6a0413d2f0 2024-09-17T15:42:36.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a0259c640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 0x7f69e403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:36.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.031+0000 7f6a0259c640 10 monclient: get_auth_request con 0x7f69e403bbe0 auth_method 0 2024-09-17T15:42:36.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.035+0000 7f6a0259c640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 0x7f69e403e0a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0ac000 tx=0x7f69ec0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:36.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.035+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f69f404df90 con 0x7f6a0413d2f0 2024-09-17T15:42:36.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.035+0000 7f69faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:36.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.035+0000 7f69faffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:36.043 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:36.097 INFO:tasks.rados.rados.0.smithi203.stdout:2399: expect (ObjNum 823 snap 254 seq_num 823) 2024-09-17T15:42:36.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.271+0000 7f6a08f07640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:42:36.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.271+0000 7f6a08f07640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:42:36.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.271+0000 7f6a08f07640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:36.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.271+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6a04121b50 con 0x7f6a0413d2f0 2024-09-17T15:42:36.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f69faffd640 1 -- 172.21.15.86:0/3705296610 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f69f404f270 con 0x7f6a0413d2f0 2024-09-17T15:42:36.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f69faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:42:36.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f69faffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:36.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=2227569759,config=3541143443,health=3727880303,kv=3772292693,logm=1261916139}) 2024-09-17T15:42:36.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=1853243865,config=3541143443,health=3727880303,kv=1466466540,logm=2214138075}) 2024-09-17T15:42:36.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=2227569759,config=3541143443,health=3727880303,kv=3772292693,logm=1261916139}) 2024-09-17T15:42:36.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.275+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=12,kv=11,logm=53} crc {auth=2665117112,config=3541143443,health=3727880303,kv=3591778543,logm=3606539502}) 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 msgr2=0x7f69e403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 0x7f69e403e0a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f69ec0ac000 tx=0x7f69ec0a7000 comp rx=0 tx=0).stop 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 10 monclient: shutdown 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 msgr2=0x7f6a04138110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f69f400c870 tx=0x7f69f400cd30 comp rx=0 tx=0).stop 2024-09-17T15:42:36.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 shutdown_connections 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69e403bbe0 0x7f69e403e0a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a0413a320 0x7f6a0413a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a0413d2f0 0x7f6a04138110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 --2- 172.21.15.86:0/3705296610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6a040a54c0 0x7f6a041cfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 >> 172.21.15.86:0/3705296610 conn(0x7f6a040aaae0 msgr2=0x7f6a040a0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 shutdown_connections 2024-09-17T15:42:36.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f6a08f07640 1 -- 172.21.15.86:0/3705296610 wait complete. 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682260559}) 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506683504}) 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682260559}) 2024-09-17T15:42:36.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.279+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957800431}) 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3429789356}) 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409447882}) 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3429789356}) 2024-09-17T15:42:36.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.283+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=862313436}) 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104319997}) 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=926929848}) 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104319997}) 2024-09-17T15:42:36.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.287+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737136540}) 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917159161}) 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878997585}) 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917159161}) 2024-09-17T15:42:36.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681221590}) 2024-09-17T15:42:36.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.291+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762327123}) 2024-09-17T15:42:36.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979010201}) 2024-09-17T15:42:36.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762327123}) 2024-09-17T15:42:36.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2383004780}) 2024-09-17T15:42:36.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:42:36.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342173002}) 2024-09-17T15:42:36.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551988587}) 2024-09-17T15:42:36.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3342173002}) 2024-09-17T15:42:36.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.295+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2882243124}) 2024-09-17T15:42:36.296 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012620055}) 2024-09-17T15:42:36.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=362947235}) 2024-09-17T15:42:36.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012620055}) 2024-09-17T15:42:36.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.299+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835193151}) 2024-09-17T15:42:36.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014579984}) 2024-09-17T15:42:36.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186742898}) 2024-09-17T15:42:36.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014579984}) 2024-09-17T15:42:36.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012075079}) 2024-09-17T15:42:36.302 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.302 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368957961}) 2024-09-17T15:42:36.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=22014062}) 2024-09-17T15:42:36.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368957961}) 2024-09-17T15:42:36.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.303+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255144996}) 2024-09-17T15:42:36.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1665459781}) 2024-09-17T15:42:36.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3434799535}) 2024-09-17T15:42:36.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1665459781}) 2024-09-17T15:42:36.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.307+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430124194}) 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011110445}) 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129898484}) 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1011110445}) 2024-09-17T15:42:36.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.311+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645474866}) 2024-09-17T15:42:36.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352603889}) 2024-09-17T15:42:36.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105864186}) 2024-09-17T15:42:36.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352603889}) 2024-09-17T15:42:36.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166330822}) 2024-09-17T15:42:36.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883011638}) 2024-09-17T15:42:36.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2060616077}) 2024-09-17T15:42:36.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883011638}) 2024-09-17T15:42:36.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.315+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935182979}) 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174413799}) 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=197510103}) 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174413799}) 2024-09-17T15:42:36.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730189635}) 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125097713}) 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997439510}) 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125097713}) 2024-09-17T15:42:36.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4076885265}) 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090851213}) 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654469213}) 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090851213}) 2024-09-17T15:42:36.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.319+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2733155529}) 2024-09-17T15:42:36.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126334256}) 2024-09-17T15:42:36.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483616117}) 2024-09-17T15:42:36.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126334256}) 2024-09-17T15:42:36.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156616339}) 2024-09-17T15:42:36.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101813804}) 2024-09-17T15:42:36.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1706079554}) 2024-09-17T15:42:36.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101813804}) 2024-09-17T15:42:36.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347600959}) 2024-09-17T15:42:36.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.321 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684094201}) 2024-09-17T15:42:36.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2672653696}) 2024-09-17T15:42:36.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684094201}) 2024-09-17T15:42:36.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771296902}) 2024-09-17T15:42:36.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023612794}) 2024-09-17T15:42:36.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3610572659}) 2024-09-17T15:42:36.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023612794}) 2024-09-17T15:42:36.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.323+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541281997}) 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638728562}) 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199171955}) 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638728562}) 2024-09-17T15:42:36.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.327+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355123088}) 2024-09-17T15:42:36.379 INFO:tasks.rados.rados.0.smithi203.stdout:2400: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2400: finishing copy_from to smithi20328086-20 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1068 (ObjNum 793 snap 246 seq_num 793) dirty exists 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: finishing write tid 1 to smithi20328086-30 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: finishing write tid 2 to smithi20328086-30 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: finishing write tid 3 to smithi20328086-30 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: finishing write tid 4 to smithi20328086-30 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: oid 30 updating version 0 to 1134 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: oid 30 updating version 1134 to 1135 2024-09-17T15:42:36.380 INFO:tasks.rados.rados.0.smithi203.stdout:2401: oid 30 updating version 1135 to 1136 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1136 (ObjNum 849 snap 261 seq_num 849) dirty exists 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2401: left oid 30 (ObjNum 849 snap 261 seq_num 849) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2398: done (4 left) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2399: done (3 left) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2400: done (2 left) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2401: done (1 left) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2402: done (0 left) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2403: read oid 26 snap -1 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2403: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2404: copy_from oid 15 from oid 1 current snap is 261 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2405: copy_from oid 49 from oid 46 current snap is 261 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2406: copy_from oid 47 from oid 35 current snap is 261 2024-09-17T15:42:36.381 INFO:tasks.rados.rados.0.smithi203.stdout:2407: write oid 41 current snap is 261 2024-09-17T15:42:36.382 INFO:tasks.rados.rados.0.smithi203.stdout:2407: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-17T15:42:36.388 INFO:tasks.rados.rados.0.smithi203.stdout:2407: writing smithi20328086-41 from 702757 to 1254839 tid 1 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2407: writing smithi20328086-41 from 1681941 to 2112569 tid 2 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2408: read oid 7 snap -1 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2408: expect (ObjNum 828 snap 254 seq_num 828) 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2409: copy_from oid 9 from oid 4 current snap is 261 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2406: finishing copy_from to smithi20328086-47 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2406: got expected ENOENT (src dne) 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2404: finishing copy_from to smithi20328086-15 2024-09-17T15:42:36.404 INFO:tasks.rados.rados.0.smithi203.stdout:2404: got expected ENOENT (src dne) 2024-09-17T15:42:36.405 INFO:tasks.rados.rados.0.smithi203.stdout:2406: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:42:36.405 INFO:tasks.rados.rados.0.smithi203.stdout:2404: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:42:36.405 INFO:tasks.rados.rados.0.smithi203.stdout:2403: expect (ObjNum 792 snap 246 seq_num 792) 2024-09-17T15:42:36.405 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:36.566 INFO:tasks.rados.rados.0.smithi203.stdout:2407: finishing write tid 1 to smithi20328086-41 2024-09-17T15:42:36.566 INFO:tasks.rados.rados.0.smithi203.stdout:2407: finishing write tid 2 to smithi20328086-41 2024-09-17T15:42:36.566 INFO:tasks.rados.rados.0.smithi203.stdout:2407: finishing write tid 3 to smithi20328086-41 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2407: finishing write tid 4 to smithi20328086-41 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2407: oid 41 updating version 0 to 637 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2407: oid 41 version 637 is already newer than 636 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2407: oid 41 version 637 is already newer than 635 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 637 (ObjNum 851 snap 261 seq_num 851) dirty exists 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2407: left oid 41 (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:42:36.567 INFO:tasks.rados.rados.0.smithi203.stdout:2408: expect (ObjNum 828 snap 254 seq_num 828) 2024-09-17T15:42:36.579 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1707166 2024-09-17T15:42:36.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589295736}) 2024-09-17T15:42:36.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2187868406}) 2024-09-17T15:42:36.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589295736}) 2024-09-17T15:42:36.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827540445}) 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118513751}) 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728949704}) 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118513751}) 2024-09-17T15:42:36.634 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.635+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=974279630}) 2024-09-17T15:42:36.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1897026353}) 2024-09-17T15:42:36.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3251562291}) 2024-09-17T15:42:36.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1897026353}) 2024-09-17T15:42:36.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.639+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007027627}) 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056824594}) 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2773021841}) 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056824594}) 2024-09-17T15:42:36.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2595419432}) 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399436666}) 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297352795}) 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399436666}) 2024-09-17T15:42:36.641 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.643+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2714872281}) 2024-09-17T15:42:36.643 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588728418}) 2024-09-17T15:42:36.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328165110}) 2024-09-17T15:42:36.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588728418}) 2024-09-17T15:42:36.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1537555666}) 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037671916}) 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826426207}) 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037671916}) 2024-09-17T15:42:36.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.647+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355609993}) 2024-09-17T15:42:36.648 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.648 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995600822}) 2024-09-17T15:42:36.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223577913}) 2024-09-17T15:42:36.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=995600822}) 2024-09-17T15:42:36.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1955624894}) 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394038402}) 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674371850}) 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394038402}) 2024-09-17T15:42:36.650 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.651+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092166493}) 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231522665}) 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=674145388}) 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231522665}) 2024-09-17T15:42:36.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.655+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495639505}) 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097159406}) 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230137693}) 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097159406}) 2024-09-17T15:42:36.655 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3179052457}) 2024-09-17T15:42:36.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739286086}) 2024-09-17T15:42:36.657 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452957476}) 2024-09-17T15:42:36.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=739286086}) 2024-09-17T15:42:36.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.659+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927685308}) 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492704178}) 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636280546}) 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492704178}) 2024-09-17T15:42:36.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2045329770}) 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982502217}) 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107777648}) 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982502217}) 2024-09-17T15:42:36.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.663+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227958257}) 2024-09-17T15:42:36.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455292162}) 2024-09-17T15:42:36.664 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=510344645}) 2024-09-17T15:42:36.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455292162}) 2024-09-17T15:42:36.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.667+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392278829}) 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23584160}) 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290030851}) 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23584160}) 2024-09-17T15:42:36.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.671+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425634923}) 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455780781}) 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=584600508}) 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455780781}) 2024-09-17T15:42:36.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851778962}) 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450837049}) 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454608769}) 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450837049}) 2024-09-17T15:42:36.681 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.683+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1729661560}) 2024-09-17T15:42:36.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3216431826}) 2024-09-17T15:42:36.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=66023093}) 2024-09-17T15:42:36.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3216431826}) 2024-09-17T15:42:36.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.687+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1889017917}) 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763563434}) 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188678164}) 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763563434}) 2024-09-17T15:42:36.711 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.715+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220121078}) 2024-09-17T15:42:36.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162756943}) 2024-09-17T15:42:36.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=41990644}) 2024-09-17T15:42:36.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.722 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162756943}) 2024-09-17T15:42:36.723 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.723+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009223963}) 2024-09-17T15:42:36.726 INFO:tasks.rados.rados.0.smithi203.stdout:2405: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2405: finishing copy_from to smithi20328086-49 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 946 (ObjNum 807 snap 249 seq_num 807) dirty exists 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2409: finishing copy_from racing read to smithi20328086-9 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2409: finishing copy_from to smithi20328086-9 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1075 (ObjNum 833 snap 256 seq_num 833) dirty exists 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2403: done (6 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2404: done (5 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2405: done (4 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2406: done (3 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2407: done (2 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2408: done (1 left) 2024-09-17T15:42:36.727 INFO:tasks.rados.rados.0.smithi203.stdout:2409: done (0 left) 2024-09-17T15:42:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2410: rollback oid 7 current snap is 261 2024-09-17T15:42:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 239 2024-09-17T15:42:36.728 INFO:tasks.rados.rados.0.smithi203.stdout:2411: snap_create 2024-09-17T15:42:36.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486143808}) 2024-09-17T15:42:36.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467006001}) 2024-09-17T15:42:36.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486143808}) 2024-09-17T15:42:36.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.735+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1083563067}) 2024-09-17T15:42:36.739 INFO:tasks.rados.rados.0.smithi203.stdout:2410: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:42:36.739 INFO:tasks.rados.rados.0.smithi203.stdout:2410: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:42:36.739 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1089 (ObjNum 33 snap 0 seq_num 3605967584) dirty dne 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524546685}) 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2245482291}) 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2524546685}) 2024-09-17T15:42:36.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.755+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496672889}) 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599526820}) 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017266219}) 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1599526820}) 2024-09-17T15:42:36.762 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.763+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007167918}) 2024-09-17T15:42:36.788 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463688122}) 2024-09-17T15:42:36.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279479013}) 2024-09-17T15:42:36.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463688122}) 2024-09-17T15:42:36.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.791+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1343732413}) 2024-09-17T15:42:36.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1591971695}) 2024-09-17T15:42:36.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=388806541}) 2024-09-17T15:42:36.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1591971695}) 2024-09-17T15:42:36.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293119244}) 2024-09-17T15:42:36.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074237909}) 2024-09-17T15:42:36.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.847+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483752844}) 2024-09-17T15:42:36.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074237909}) 2024-09-17T15:42:36.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024174841}) 2024-09-17T15:42:36.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882280819}) 2024-09-17T15:42:36.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1411650312}) 2024-09-17T15:42:36.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882280819}) 2024-09-17T15:42:36.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626680958}) 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159946760}) 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312297581}) 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159946760}) 2024-09-17T15:42:36.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.851+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2026691274}) 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143979258}) 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663686149}) 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143979258}) 2024-09-17T15:42:36.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.855+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568741836}) 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=480002475,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3318502136,mon_config_key=2443226221,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1251818990,osd_pg_creating=2013667972,osd_snap=1868106700}) 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=4020434961,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3318502136,mon_config_key=578014829,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1259201597,osd_pg_creating=2013667972,osd_snap=1818495881}) 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=480002475,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3318502136,mon_config_key=2443226221,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1251818990,osd_pg_creating=2013667972,osd_snap=1868106700}) 2024-09-17T15:42:36.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.859+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=41,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=838517726,mdsmap=2815901553,mgr=2224701651,mgr_command_descs=1770323759,mgrstat=2835906094,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2013667972,osd_snap=1656265895}) 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=91,osdmap=9} crc {osd_snap=1108480624,osdmap=1680966676}) 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=91,osdmap=9} crc {osd_snap=1488583968,osdmap=1709294548}) 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=91,osdmap=9} crc {osd_snap=1108480624,osdmap=1680966676}) 2024-09-17T15:42:36.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.863+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=91,osdmap=9} crc {osd_snap=3250819487,osdmap=1680966676}) 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2879772910}) 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3621354391}) 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2879772910}) 2024-09-17T15:42:36.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.867+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=450255393}) 2024-09-17T15:42:36.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=266446859}) 2024-09-17T15:42:36.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=113916503}) 2024-09-17T15:42:36.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:42:36.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=266446859}) 2024-09-17T15:42:36.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:42:36.875+0000 7f97f0082640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=57} crc {osdmap=2176407483}) 2024-09-17T15:42:37.293 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:42:37.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: get_monmap_and_config 2024-09-17T15:42:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: build_initial_monmap 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: monmap: 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:37.425650+0000 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:37.425650+0000 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:37.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 Processor -- start 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 -- start start 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: init 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f85940689d0 0x7f859406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f85940689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8594068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f859413aff0 0x7f859413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f859413aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f859409c8b0 con 0x7f8594068db0 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8594074500 con 0x7f859413aff0 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f859409cb90 con 0x7f85940689d0 2024-09-17T15:42:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8598bbb640 10 monclient(hunting): authenticate will time out at 1890.391106s 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f859413aff0 0x7f859413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f859413aff0 0x7f859413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35678/0 (socket says 172.21.15.86:35678) 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 -- 172.21.15.86:0/1772607928 learned_addr learned my addr 172.21.15.86:0/1772607928 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): get_auth_request con 0x7f859413aff0 auth_method 0 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 1 --2- 172.21.15.86:0/1772607928 >> v2:172.21.15.203:3300/0 conn(0x7f85940689d0 0x7f859406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): get_auth_request con 0x7f85940689d0 auth_method 0 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient(hunting): handle_auth_done global_id 144468 payload 274 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 -- 172.21.15.86:0/1772607928 >> v2:172.21.15.203:3300/0 conn(0x7f85940689d0 msgr2=0x7f859406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 --2- 172.21.15.86:0/1772607928 >> v2:172.21.15.203:3300/0 conn(0x7f85940689d0 0x7f859406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 -- 172.21.15.86:0/1772607928 >> v1:172.21.15.203:6789/0 conn(0x7f8594068db0 legacy=0x7f8594069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 monclient: found mon.noname-a 2024-09-17T15:42:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.423+0000 7f8592d9d640 1 -- 172.21.15.86:0/1772607928 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f859409ce70 con 0x7f859413aff0 2024-09-17T15:42:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 1 -- 172.21.15.86:0/1772607928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f859413aff0 msgr2=0x7f859413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 1 --2- 172.21.15.86:0/1772607928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f859413aff0 0x7f859413d3e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8594069940 tx=0x7f858c01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e5c7fccdc239b86 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 1 -- 172.21.15.86:0/1772607928 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859409c2d0 con 0x7f859413aff0 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: dump: 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: _finish_auth 0 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 1 -- 172.21.15.86:0/1772607928 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858c0a5730 con 0x7f859413aff0 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 5 monclient: authenticate success, global_id 144468 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 1 -- 172.21.15.86:0/1772607928 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858c0a5b00 con 0x7f859413aff0 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 4 set_mon_vals no callback set 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8591d9b640 10 monclient: dump: 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: shutdown 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- 172.21.15.86:0/1772607928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f859413aff0 msgr2=0x7f859413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- 172.21.15.86:0/1772607928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f859413aff0 0x7f859413d3e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8594069940 tx=0x7f858c01c890 comp rx=0 tx=0).stop 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- 172.21.15.86:0/1772607928 shutdown_connections 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- 172.21.15.86:0/1772607928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f859413aff0 0x7f859413d3e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- 172.21.15.86:0/1772607928 >> v2:172.21.15.203:3300/0 conn(0x7f85940689d0 0x7f859406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- 172.21.15.86:0/1772607928 >> 172.21.15.86:0/1772607928 conn(0x7f85940a6440 msgr2=0x7f85940a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- 172.21.15.86:0/1772607928 shutdown_connections 2024-09-17T15:42:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- 172.21.15.86:0/1772607928 wait complete. 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: build_initial_monmap 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: monmap: 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 Processor -- start 2024-09-17T15:42:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- start start 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: init 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f85940689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594068db0 0x7f859409d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f8594068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f859413aff0 0x7f859409dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f859413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f85941405f0 con 0x7f85940689d0 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8594140470 con 0x7f8594068db0 2024-09-17T15:42:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8594140770 con 0x7f859413aff0 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8598bbb640 10 monclient(hunting): authenticate will time out at 1890.399298s 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35684/0 (socket says 172.21.15.86:35684) 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 -- 172.21.15.86:0/607098204 learned_addr learned my addr 172.21.15.86:0/607098204 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): get_auth_request con 0x7f85940689d0 auth_method 0 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f859413aff0 0x7f859409dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): get_auth_request con 0x7f859413aff0 auth_method 0 2024-09-17T15:42:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f8592d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f858bfff640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594068db0 0x7f859409d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f858bfff640 10 monclient(hunting): get_auth_request con 0x7f8594068db0 auth_method 0 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f858bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f858bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f858bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient(hunting): handle_auth_done global_id 144471 payload 824 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 -- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f859413aff0 msgr2=0x7f859409dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f859413aff0 0x7f859409dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 -- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594068db0 msgr2=0x7f859409d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594068db0 0x7f859409d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient: _finish_hunting 0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 monclient: found mon.noname-a 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.431+0000 7f859259c640 1 -- 172.21.15.86:0/607098204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8594072790 con 0x7f85940689d0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f859259c640 1 -- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 msgr2=0x7f8594133230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f859259c640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f858400e930 tx=0x7f858400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8584017070 con 0x7f85940689d0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: dump: 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: _finish_auth 0 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:37.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f858400cc10 con 0x7f85940689d0 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8584013470 con 0x7f85940689d0 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 5 monclient: authenticate success, global_id 144471 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _renew_subs 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8594072a70 con 0x7f85940689d0 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 10 monclient: dump: 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f859159a640 4 set_mon_vals no callback set 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _renew_subs 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85940724b0 con 0x7f85940689d0 2024-09-17T15:42:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f859406f840 con 0x7f85940689d0 2024-09-17T15:42:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f858401a070 con 0x7f85940689d0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 0x7f857403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(540..540 src has 438..540) ==== 4799+0+0 (secure 0 0 0) 0x7f8584050070 con 0x7f85940689d0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.435+0000 7f858bfff640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 0x7f857403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.439+0000 7f858bfff640 10 monclient: get_auth_request con 0x7f857403bbb0 auth_method 0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.439+0000 7f858bfff640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 0x7f857403e070 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f859409f310 tx=0x7f857c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.443+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8584023d00 con 0x7f85940689d0 2024-09-17T15:42:37.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.443+0000 7f858affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:37.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.443+0000 7f858affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f8598bbb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f8598bbb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f8598bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:37.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f859406d4f0 con 0x7f85940689d0 2024-09-17T15:42:37.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f858affd640 1 -- 172.21.15.86:0/607098204 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f85840123d0 con 0x7f85940689d0 2024-09-17T15:42:37.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f858affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:37.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.675+0000 7f858affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:37.674 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 msgr2=0x7f857403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 0x7f857403e070 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f859409f310 tx=0x7f857c009040 comp rx=0 tx=0).stop 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 10 monclient: shutdown 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 msgr2=0x7f8594133230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f858400e930 tx=0x7f858400edf0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 shutdown_connections 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f857403bbb0 0x7f857403e070 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f859413aff0 0x7f859409dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8594068db0 0x7f859409d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 --2- 172.21.15.86:0/607098204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85940689d0 0x7f8594133230 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 >> 172.21.15.86:0/607098204 conn(0x7f85940a6440 msgr2=0x7f859413bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 shutdown_connections 2024-09-17T15:42:37.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.679+0000 7f8598bbb640 1 -- 172.21.15.86:0/607098204 wait complete. 2024-09-17T15:42:37.693 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:42:37.693 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:42:37.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:42:37.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:42:37.731 INFO:tasks.rados.rados.0.smithi203.stdout:2410: done (1 left) 2024-09-17T15:42:37.731 INFO:tasks.rados.rados.0.smithi203.stdout:2411: done (0 left) 2024-09-17T15:42:37.731 INFO:tasks.rados.rados.0.smithi203.stdout:2412: write oid 42 current snap is 262 2024-09-17T15:42:37.731 INFO:tasks.rados.rados.0.smithi203.stdout:2412: seq_num 852 ranges {719564=743684,2140423=179141} 2024-09-17T15:42:37.739 INFO:tasks.rados.rados.0.smithi203.stdout:2412: writing smithi20328086-42 from 719564 to 1463248 tid 1 2024-09-17T15:42:37.744 INFO:tasks.rados.rados.0.smithi203.stdout:2412: writing smithi20328086-42 from 2140423 to 2319564 tid 2 2024-09-17T15:42:37.744 INFO:tasks.rados.rados.0.smithi203.stdout:2413: copy_from oid 22 from oid 36 current snap is 262 2024-09-17T15:42:37.744 INFO:tasks.rados.rados.0.smithi203.stdout:2414: write (excl) oid 4 current snap is 262 2024-09-17T15:42:37.744 INFO:tasks.rados.rados.0.smithi203.stdout:2414: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-17T15:42:37.750 INFO:tasks.rados.rados.0.smithi203.stdout:2414: writing smithi20328086-4 from 736371 to 1155304 tid 1 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2414: writing smithi20328086-4 from 1798905 to 2233202 tid 2 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2415: delete oid 15 current snap is 262 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2412: finishing write tid 1 to smithi20328086-42 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2412: finishing write tid 2 to smithi20328086-42 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2412: finishing write tid 3 to smithi20328086-42 2024-09-17T15:42:37.757 INFO:tasks.rados.rados.0.smithi203.stdout:2412: finishing write tid 4 to smithi20328086-42 2024-09-17T15:42:37.758 INFO:tasks.rados.rados.0.smithi203.stdout:2412: oid 42 updating version 0 to 1109 2024-09-17T15:42:37.758 INFO:tasks.rados.rados.0.smithi203.stdout:2412: oid 42 updating version 1109 to 1110 2024-09-17T15:42:37.758 INFO:tasks.rados.rados.0.smithi203.stdout:2412: oid 42 updating version 1110 to 1111 2024-09-17T15:42:37.758 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1111 (ObjNum 852 snap 262 seq_num 852) dirty exists 2024-09-17T15:42:37.758 INFO:tasks.rados.rados.0.smithi203.stdout:2412: left oid 42 (ObjNum 852 snap 262 seq_num 852) 2024-09-17T15:42:37.764 INFO:tasks.rados.rados.0.smithi203.stdout:2412: done (3 left) 2024-09-17T15:42:37.764 INFO:tasks.rados.rados.0.smithi203.stdout:2415: done (2 left) 2024-09-17T15:42:37.765 INFO:tasks.rados.rados.0.smithi203.stdout:2416: rollback oid 25 current snap is 262 2024-09-17T15:42:37.765 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 239 2024-09-17T15:42:37.765 INFO:tasks.rados.rados.0.smithi203.stdout:2417: snap_create 2024-09-17T15:42:37.765 INFO:tasks.rados.rados.0.smithi203.stdout:2414: finishing write tid 1 to smithi20328086-4 2024-09-17T15:42:37.766 INFO:tasks.rados.rados.0.smithi203.stdout:2413: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:42:37.766 INFO:tasks.rados.rados.0.smithi203.stdout:2414: finishing write tid 2 to smithi20328086-4 2024-09-17T15:42:37.769 INFO:tasks.rados.rados.0.smithi203.stdout:2414: finishing write tid 3 to smithi20328086-4 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2414: finishing write tid 4 to smithi20328086-4 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2414: oid 4 updating version 0 to 1077 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2414: oid 4 version 1077 is already newer than 1076 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2414: oid 4 updating version 1077 to 1078 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1078 (ObjNum 853 snap 262 seq_num 853) dirty exists 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2414: left oid 4 (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:42:37.770 INFO:tasks.rados.rados.0.smithi203.stdout:2413: finishing copy_from to smithi20328086-22 2024-09-17T15:42:37.771 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1244 (ObjNum 700 snap 220 seq_num 700) dirty exists 2024-09-17T15:42:37.771 INFO:tasks.rados.rados.0.smithi203.stdout:2416: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:42:37.771 INFO:tasks.rados.rados.0.smithi203.stdout:2416: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:42:37.772 INFO:tasks.rados.rados.0.smithi203.stdout:2416: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:42:37.772 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1112 (ObjNum 764 snap 236 seq_num 764) dirty exists 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: get_monmap_and_config 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: build_initial_monmap 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: monmap: 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:37.828156+0000 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:37.828156+0000 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:37.823 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:37.824 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 Processor -- start 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 -- start start 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: init 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 0x7feac409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-a con 0x7feac40a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac40a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-b con 0x7feac40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7feac4137ea0 0x7feac41382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-c con 0x7feac4137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7feac40a6280 con 0x7feac40a50e0 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7feac409c0a0 con 0x7feac4137ea0 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feac409c380 con 0x7feac40a54c0 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feacbc15640 10 monclient(hunting): authenticate will time out at 1890.793565s 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 0x7feac409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 0x7feac409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35690/0 (socket says 172.21.15.86:35690) 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 1 -- 172.21.15.86:0/1994219800 learned_addr learned my addr 172.21.15.86:0/1994219800 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): get_auth_request con 0x7feac40a50e0 auth_method 0 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 --2- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac40a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): get_auth_request con 0x7feac40a54c0 auth_method 0 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feaca1b2640 1 --2- 172.21.15.86:0/1994219800 >> v2:172.21.15.203:3300/0 conn(0x7feac4137ea0 0x7feac41382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feaca1b2640 10 monclient(hunting): get_auth_request con 0x7feac4137ea0 auth_method 0 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feaca1b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feaca1b2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feaca1b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient(hunting): handle_auth_done global_id 164222 payload 274 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 -- 172.21.15.86:0/1994219800 >> v2:172.21.15.203:3300/0 conn(0x7feac4137ea0 msgr2=0x7feac41382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 --2- 172.21.15.86:0/1994219800 >> v2:172.21.15.203:3300/0 conn(0x7feac4137ea0 0x7feac41382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 -- 172.21.15.86:0/1994219800 >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 msgr2=0x7feac409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 --2- 172.21.15.86:0/1994219800 >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 0x7feac409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient: _finish_hunting 0 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 monclient: found mon.noname-b 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.827+0000 7feac91b0640 1 -- 172.21.15.86:0/1994219800 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac409c660 con 0x7feac40a54c0 2024-09-17T15:42:37.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac91b0640 1 -- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 msgr2=0x7feac40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac91b0640 1 --2- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac40a58f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feac406f8c0 tx=0x7feab801c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b8381341f2870b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 1 -- 172.21.15.86:0/1994219800 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab80b0070 con 0x7feac40a54c0 2024-09-17T15:42:37.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: dump: 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: _finish_auth 0 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: _check_auth_tickets 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 1 -- 172.21.15.86:0/1994219800 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feab80ab600 con 0x7feac40a54c0 2024-09-17T15:42:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 5 monclient: authenticate success, global_id 164222 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 1 -- 172.21.15.86:0/1994219800 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab80ab9d0 con 0x7feac40a54c0 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 4 set_mon_vals no callback set 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feac89af640 10 monclient: dump: 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.842 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 10 monclient: shutdown 2024-09-17T15:42:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 -- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 msgr2=0x7feac40a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 --2- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac40a58f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7feac406f8c0 tx=0x7feab801c340 comp rx=0 tx=0).stop 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 -- 172.21.15.86:0/1994219800 shutdown_connections 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 --2- 172.21.15.86:0/1994219800 >> v2:172.21.15.203:3300/0 conn(0x7feac4137ea0 0x7feac41382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 --2- 172.21.15.86:0/1994219800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac40a58f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 --2- 172.21.15.86:0/1994219800 >> v2:172.21.15.86:3300/0 conn(0x7feac40a50e0 0x7feac409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 -- 172.21.15.86:0/1994219800 >> 172.21.15.86:0/1994219800 conn(0x7feac40aaae0 msgr2=0x7feac40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 -- 172.21.15.86:0/1994219800 shutdown_connections 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 1 -- 172.21.15.86:0/1994219800 wait complete. 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 10 monclient: build_initial_monmap 2024-09-17T15:42:37.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.831+0000 7feacbc15640 10 monclient: monmap: 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 Processor -- start 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- start start 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: init 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac40a50e0 0x7feac4136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-b con 0x7feac40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac413d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-c con 0x7feac40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137ea0 0x7feac413d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): picked mon.noname-a con 0x7feac4137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feac40a6ae0 con 0x7feac4137ea0 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feac40a6960 con 0x7feac40a50e0 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feac4141b20 con 0x7feac40a54c0 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient(hunting): authenticate will time out at 1890.800772s 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac413d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137ea0 0x7feac413d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac413d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37816/0 (socket says 172.21.15.86:37816) 2024-09-17T15:42:37.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 -- 172.21.15.86:0/3096768483 learned_addr learned my addr 172.21.15.86:0/3096768483 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): get_auth_request con 0x7feac40a54c0 auth_method 0 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): get_auth_request con 0x7feac4137ea0 auth_method 0 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac99b1640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac40a50e0 0x7feac4136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac99b1640 10 monclient(hunting): get_auth_request con 0x7feac40a50e0 auth_method 0 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac99b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac99b1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac99b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient(hunting): handle_auth_done global_id 164225 payload 824 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac40a50e0 msgr2=0x7feac4136ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feac40a50e0 0x7feac4136ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137ea0 msgr2=0x7feac413d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137ea0 0x7feac413d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient: _finish_hunting 0 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 monclient: found mon.noname-c 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac40a18d0 con 0x7feac40a54c0 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feaca1b2640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137ea0 0x7feac413d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 msgr2=0x7feac413d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac91b0640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac413d0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feab8019f70 tx=0x7feab80190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab80b8020 con 0x7feac40a54c0 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: dump: 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.851 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feab80b4850 con 0x7feac40a54c0 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 5 monclient: authenticate success, global_id 164225 2024-09-17T15:42:37.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _renew_subs 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feac413eaa0 con 0x7feac40a54c0 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feab80b0040 con 0x7feac40a54c0 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac27fc640 10 monclient: dump: 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feac3fff640 4 set_mon_vals no callback set 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _renew_subs 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.835+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feac4122890 con 0x7feac40a54c0 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feacbc15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feacbc15640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 0x7feac413db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feac406a0e0 con 0x7feac4137410 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 0x7feac413db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient: get_auth_request con 0x7feac4137410 auth_method 0 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7feab80bb020 con 0x7feac40a54c0 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feab003bbe0 0x7feab003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 10 monclient(hunting): handle_auth_done global_id 144480 payload 824 2024-09-17T15:42:37.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7feab80e9c90 con 0x7feac40a54c0 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feac40a7a20 con 0x7feac4137410 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feaca1b2640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feab003bbe0 0x7feab003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feaca1b2640 10 monclient: get_auth_request con 0x7feab003bbe0 auth_method 0 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 msgr2=0x7feac413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac99b1640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 0x7feac413db50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc015640 con 0x7feac4137410 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 10 monclient: dump: 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.856 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feabc015640 con 0x7feac4137410 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 msgr2=0x7feac413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 0x7feac413db50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc010590 comp rx=0 tx=0).stop 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 msgr2=0x7feac413db50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:37.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.839+0000 7feac27fc640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac4137410 0x7feac413db50 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc010590 comp rx=0 tx=0).stop 2024-09-17T15:42:37.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.843+0000 7feaca1b2640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feab003bbe0 0x7feab003e0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7feab4006aa0 tx=0x7feab4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feacbc15640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:37.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feacbc15640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 0x7feac41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feacbc15640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feac40a45e0 con 0x7feac413e090 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 0x7feac41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient: get_auth_request con 0x7feac413e090 auth_method 0 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 10 monclient(hunting): handle_auth_done global_id 144483 payload 824 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.847+0000 7feac99b1640 1 -- 172.21.15.86:0/3096768483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feac40a58f0 con 0x7feac413e090 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac99b1640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 msgr2=0x7feac41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac99b1640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 0x7feac41392d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feabc020030 con 0x7feac413e090 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 10 monclient: dump: 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7feabc016810 con 0x7feac413e090 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 msgr2=0x7feac41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 0x7feac41392d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc004a50 comp rx=0 tx=0).stop 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 msgr2=0x7feac41392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feac27fc640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feac413e090 0x7feac41392d0 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feabc007d30 tx=0x7feabc004a50 comp rx=0 tx=0).stop 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:37.862 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 176193, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:37.863 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:37.864 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:37.866 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:37.867 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.868 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:37.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:37.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:37.872 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:37.874 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:37.875 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:37.876 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feab003bbe0 msgr2=0x7feab003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7feab003bbe0 0x7feab003e0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7feab4006aa0 tx=0x7feab4008040 comp rx=0 tx=0).stop 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 10 monclient: shutdown 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 msgr2=0x7feac413d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 --2- 172.21.15.86:0/3096768483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feac40a54c0 0x7feac413d0d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7feab8019f70 tx=0x7feab80190e0 comp rx=0 tx=0).stop 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feaca1b2640 1 -- 172.21.15.86:0/3096768483 reap_dead start 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feaca1b2640 1 -- 172.21.15.86:0/3096768483 reap_dead start 2024-09-17T15:42:37.878 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 shutdown_connections 2024-09-17T15:42:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 >> 172.21.15.86:0/3096768483 conn(0x7feac40aaae0 msgr2=0x7feac40a34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 shutdown_connections 2024-09-17T15:42:37.879 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.851+0000 7feacbc15640 1 -- 172.21.15.86:0/3096768483 wait complete. 2024-09-17T15:42:37.879 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:37.879 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:42:37.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:42:37.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: get_monmap_and_config 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: build_initial_monmap 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: monmap: 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:38.000179+0000 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:38.000179+0000 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:37.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:37.996 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:37.996 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:37.996 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:37.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 Processor -- start 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 -- start start 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: init 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5c640a50e0 0x7f5c6409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f5c640a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c640a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-b con 0x7f5c640a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c64137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c640a6280 con 0x7f5c64137ea0 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6409c0a0 con 0x7f5c640a50e0 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6409c380 con 0x7f5c640a54c0 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c68ccf640 10 monclient(hunting): authenticate will time out at 1890.965682s 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c62d9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5c64137ea0 0x7f5c641382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:55012/0 (socket says 172.21.15.86:55012) 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c62d9d640 1 -- 172.21.15.86:0/3798383183 learned_addr learned my addr 172.21.15.86:0/3798383183 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c640a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): get_auth_request con 0x7f5c640a54c0 auth_method 0 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:37.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c6259c640 1 --2- 172.21.15.86:0/3798383183 >> v2:172.21.15.203:3300/0 conn(0x7f5c640a50e0 0x7f5c6409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c6259c640 10 monclient(hunting): get_auth_request con 0x7f5c640a50e0 auth_method 0 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_done global_id 164228 payload 274 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 -- 172.21.15.86:0/3798383183 >> v2:172.21.15.203:3300/0 conn(0x7f5c640a50e0 msgr2=0x7f5c6409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/3798383183 >> v2:172.21.15.203:3300/0 conn(0x7f5c640a50e0 0x7f5c6409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 -- 172.21.15.86:0/3798383183 >> v1:172.21.15.86:6789/0 conn(0x7f5c64137ea0 legacy=0x7f5c641382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:38.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 monclient: found mon.noname-b 2024-09-17T15:42:38.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:37.999+0000 7f5c61d9b640 1 -- 172.21.15.86:0/3798383183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6409c660 con 0x7f5c640a54c0 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c61d9b640 1 -- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 msgr2=0x7f5c640a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c640a58f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0097c0 tx=0x7f5c4c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9d73ea71d4b42dc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 1 -- 172.21.15.86:0/3798383183 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0b0070 con 0x7f5c640a54c0 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: dump: 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 1 -- 172.21.15.86:0/3798383183 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c4c0ab600 con 0x7f5c640a54c0 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 1 -- 172.21.15.86:0/3798383183 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0ab9d0 con 0x7f5c640a54c0 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 5 monclient: authenticate success, global_id 164228 2024-09-17T15:42:38.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 4 set_mon_vals no callback set 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c6159a640 10 monclient: dump: 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.007 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 10 monclient: shutdown 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 -- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 msgr2=0x7f5c640a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c640a58f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c0097c0 tx=0x7f5c4c01c3f0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 -- 172.21.15.86:0/3798383183 shutdown_connections 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/3798383183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c640a58f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/3798383183 >> v2:172.21.15.203:3300/0 conn(0x7f5c640a50e0 0x7f5c6409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 -- 172.21.15.86:0/3798383183 >> 172.21.15.86:0/3798383183 conn(0x7f5c640aaae0 msgr2=0x7f5c640aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 -- 172.21.15.86:0/3798383183 shutdown_connections 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 1 -- 172.21.15.86:0/3798383183 wait complete. 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 10 monclient: build_initial_monmap 2024-09-17T15:42:38.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.003+0000 7f5c68ccf640 10 monclient: monmap: 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 Processor -- start 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- start start 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: init 2024-09-17T15:42:38.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 0x7f5c641d03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-a con 0x7f5c640a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-c con 0x7f5c640a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c64137ea0 0x7f5c6413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): picked mon.noname-b con 0x7f5c64137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64142cb0 con 0x7f5c640a50e0 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c64142b30 con 0x7f5c64137ea0 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5c64142e30 con 0x7f5c640a54c0 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient(hunting): authenticate will time out at 1890.972841s 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 0x7f5c641d03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 0x7f5c641d03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35738/0 (socket says 172.21.15.86:35738) 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37826/0 (socket says 172.21.15.86:37826) 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 1 -- 172.21.15.86:0/1072136190 learned_addr learned my addr 172.21.15.86:0/1072136190 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): get_auth_request con 0x7f5c640a50e0 auth_method 0 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): get_auth_request con 0x7f5c640a54c0 auth_method 0 2024-09-17T15:42:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient(hunting): handle_auth_done global_id 164231 payload 824 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c64137ea0 msgr2=0x7f5c6413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c62d9d640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c64137ea0 0x7f5c6413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c64137ea0 0x7f5c6413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 msgr2=0x7f5c641d03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 0x7f5c641d03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 monclient: found mon.noname-c 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6413eea0 con 0x7f5c640a54c0 2024-09-17T15:42:38.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c6259c640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c640a50e0 0x7f5c641d03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 msgr2=0x7f5c641372c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c6409c550 tx=0x7f5c4c019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0b0040 con 0x7f5c640a54c0 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: dump: 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.018 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 5 monclient: authenticate success, global_id 164231 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c4c0b9070 con 0x7f5c640a54c0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _renew_subs 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c640a18d0 con 0x7f5c640a54c0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c0aaa30 con 0x7f5c640a54c0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c5affd640 10 monclient: dump: 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _renew_subs 2024-09-17T15:42:38.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c60d99640 4 set_mon_vals no callback set 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.007+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c641213b0 con 0x7f5c640a54c0 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c68ccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 0x7f5c6413dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6406a0e0 con 0x7f5c6413d920 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 0x7f5c6413dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient: get_auth_request con 0x7f5c6413d920 auth_method 0 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c61d9b640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c6409c550 tx=0x7f5c4c019190 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f5c4c0e9c60 osd_map(541..541 src has 438..541) 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5c4c0ab4c0 con 0x7f5c640a54c0 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 10 monclient(hunting): handle_auth_done global_id 164234 payload 824 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c6413e240 con 0x7f5c6413d920 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5c4803bbe0 0x7f5c4803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c62d9d640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5c4803bbe0 0x7f5c4803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c62d9d640 10 monclient: get_auth_request con 0x7f5c4803bbe0 auth_method 0 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 msgr2=0x7f5c6413dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c6259c640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 0x7f5c6413dd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c007590 tx=0x7f5c5c007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c0158e0 con 0x7f5c6413d920 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 10 monclient: dump: 2024-09-17T15:42:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c5c01b070 con 0x7f5c6413d920 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 msgr2=0x7f5c6413dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 0x7f5c6413dd00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c007590 tx=0x7f5c5c007a50 comp rx=0 tx=0).stop 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 msgr2=0x7f5c6413dd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.011+0000 7f5c5affd640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c6413d920 0x7f5c6413dd00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.015+0000 7f5c62d9d640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5c4803bbe0 0x7f5c4803e0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c54004fe0 tx=0x7f5c54009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c68ccf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 0x7f5c641396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c68ccf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6409cd60 con 0x7f5c641392d0 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 0x7f5c641396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient: get_auth_request con 0x7f5c641392d0 auth_method 0 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 10 monclient(hunting): handle_auth_done global_id 164237 payload 824 2024-09-17T15:42:38.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.019+0000 7f5c6259c640 1 -- 172.21.15.86:0/1072136190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c64074be0 con 0x7f5c641392d0 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c6259c640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 msgr2=0x7f5c641396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c6259c640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 0x7f5c641396b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c01a000 tx=0x7f5c5c004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c016890 con 0x7f5c641392d0 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 10 monclient: dump: 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.026 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f5c5c016890 con 0x7f5c641392d0 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 msgr2=0x7f5c641396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 0x7f5c641396b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c01a000 tx=0x7f5c5c004a50 comp rx=0 tx=0).stop 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 msgr2=0x7f5c641396b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:38.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c641392d0 0x7f5c641396b0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c01a000 tx=0x7f5c5c004a50 comp rx=0 tx=0).stop 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 61211, 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:38.028 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.029 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.030 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:38.031 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.032 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:38.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.034 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.035 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:38.036 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:38.037 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.039 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.041 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:38.041 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5c4803bbe0 msgr2=0x7f5c4803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5c4803bbe0 0x7f5c4803e0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5c54004fe0 tx=0x7f5c54009040 comp rx=0 tx=0).stop 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 10 monclient: shutdown 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 msgr2=0x7f5c641372c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 --2- 172.21.15.86:0/1072136190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c640a54c0 0x7f5c641372c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5c6409c550 tx=0x7f5c4c019190 comp rx=0 tx=0).stop 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c62d9d640 1 -- 172.21.15.86:0/1072136190 reap_dead start 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c5affd640 1 -- 172.21.15.86:0/1072136190 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (unknown 0 0 0) 0x7f5c4c0e9c60 con 0x7f5c640a54c0 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c62d9d640 1 -- 172.21.15.86:0/1072136190 reap_dead start 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 shutdown_connections 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 >> 172.21.15.86:0/1072136190 conn(0x7f5c640aaae0 msgr2=0x7f5c640a31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 shutdown_connections 2024-09-17T15:42:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.023+0000 7f5c68ccf640 1 -- 172.21.15.86:0/1072136190 wait complete. 2024-09-17T15:42:38.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:38.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:42:38.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:42:38.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: get_monmap_and_config 2024-09-17T15:42:38.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: build_initial_monmap 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: monmap: 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:38.160960+0000 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:38.160960+0000 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:38.156 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 Processor -- start 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 -- start start 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: init 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 0x7f29300a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-b con 0x7f29300a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29300a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 0x7f293013df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-a con 0x7f2930136160 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29300a2130 con 0x7f29300a1590 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f29300a2410 con 0x7f2930136160 2024-09-17T15:42:38.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f29300a26f0 con 0x7f29300a0c70 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2936e43640 10 monclient(hunting): authenticate will time out at 1891.126394s 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 0x7f293013df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 0x7f29300a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 0x7f293013df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58498/0 (socket says 172.21.15.86:58498) 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 1 -- 172.21.15.86:0/1224626645 learned_addr learned my addr 172.21.15.86:0/1224626645 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): get_auth_request con 0x7f2930136160 auth_method 0 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): get_auth_request con 0x7f29300a0c70 auth_method 0 2024-09-17T15:42:38.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient(hunting): handle_auth_done global_id 164240 payload 274 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 -- 172.21.15.86:0/1224626645 >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 msgr2=0x7f293013df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 --2- 172.21.15.86:0/1224626645 >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 0x7f293013df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 -- 172.21.15.86:0/1224626645 >> v1:172.21.15.86:6789/0 conn(0x7f29300a1590 legacy=0x7f2930135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 monclient: found mon.noname-b 2024-09-17T15:42:38.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.159+0000 7f2934bdf640 1 -- 172.21.15.86:0/1224626645 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293009f9f0 con 0x7f29300a0c70 2024-09-17T15:42:38.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2934bdf640 1 -- 172.21.15.86:0/1224626645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 msgr2=0x7f29300a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2934bdf640 1 --2- 172.21.15.86:0/1224626645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 0x7f29300a1050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29240097c0 tx=0x7f292401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ee08dab358bedd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 1 -- 172.21.15.86:0/1224626645 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240b0070 con 0x7f29300a0c70 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: dump: 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.164 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 1 -- 172.21.15.86:0/1224626645 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29240ab600 con 0x7f29300a0c70 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 5 monclient: authenticate success, global_id 164240 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 1 -- 172.21.15.86:0/1224626645 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240ab9d0 con 0x7f29300a0c70 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 4 set_mon_vals no callback set 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f292f7fe640 10 monclient: dump: 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: shutdown 2024-09-17T15:42:38.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- 172.21.15.86:0/1224626645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 msgr2=0x7f29300a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- 172.21.15.86:0/1224626645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 0x7f29300a1050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f29240097c0 tx=0x7f292401c360 comp rx=0 tx=0).stop 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- 172.21.15.86:0/1224626645 shutdown_connections 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- 172.21.15.86:0/1224626645 >> v2:172.21.15.86:3300/0 conn(0x7f2930136160 0x7f293013df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- 172.21.15.86:0/1224626645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29300a0c70 0x7f29300a1050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- 172.21.15.86:0/1224626645 >> 172.21.15.86:0/1224626645 conn(0x7f29300a2f40 msgr2=0x7f29300a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- 172.21.15.86:0/1224626645 shutdown_connections 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- 172.21.15.86:0/1224626645 wait complete. 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: build_initial_monmap 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: monmap: 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 Processor -- start 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- start start 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: init 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 0x7f293009f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-a con 0x7f29300a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:38.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29300a1590 0x7f2930135530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-b con 0x7f29300a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930136160 0x7f293009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): picked mon.noname-c con 0x7f2930136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f29300a00d0 con 0x7f29300a0c70 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f293009ff50 con 0x7f29300a1590 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2930140c90 con 0x7f2930136160 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2936e43640 10 monclient(hunting): authenticate will time out at 1891.135880s 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.167+0000 7f2934bdf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 0x7f293009f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930136160 0x7f293009d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 0x7f293009f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58512/0 (socket says 172.21.15.86:58512) 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 -- 172.21.15.86:0/4049769701 learned_addr learned my addr 172.21.15.86:0/4049769701 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): get_auth_request con 0x7f29300a0c70 auth_method 0 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): get_auth_request con 0x7f2930136160 auth_method 0 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292ffff640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29300a1590 0x7f2930135530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292ffff640 10 monclient(hunting): get_auth_request con 0x7f29300a1590 auth_method 0 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient(hunting): handle_auth_done global_id 144492 payload 824 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930136160 msgr2=0x7f293009d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930136160 0x7f293009d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29300a1590 msgr2=0x7f2930135530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29300a1590 0x7f2930135530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 monclient: found mon.noname-a 2024-09-17T15:42:38.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293006fb90 con 0x7f29300a0c70 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f29353e0640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2930136160 0x7f293009d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 msgr2=0x7f293009f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2934bdf640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 0x7f293009f3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f29240ae040 tx=0x7f292401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240b0050 con 0x7f29300a0c70 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: dump: 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.175 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29240bc070 con 0x7f29300a0c70 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240b5a80 con 0x7f29300a0c70 2024-09-17T15:42:38.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 10 monclient: dump: 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 5 monclient: authenticate success, global_id 144492 2024-09-17T15:42:38.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 monclient: _renew_subs 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292effd640 4 set_mon_vals no callback set 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2930121f00 con 0x7f29300a0c70 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 monclient: _renew_subs 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:42:38.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f293009e850 con 0x7f29300a0c70 2024-09-17T15:42:38.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 0x7f293009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29300a1050 con 0x7f29301d00d0 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.171+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f29240b5c60 con 0x7f29300a0c70 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 0x7f293009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient: get_auth_request con 0x7f29301d00d0 auth_method 0 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f291c03bb60 0x7f291c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7f29240ea700 con 0x7f29300a0c70 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f29353e0640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f291c03bb60 0x7f291c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f29353e0640 10 monclient: get_auth_request con 0x7f291c03bb60 auth_method 0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 10 monclient(hunting): handle_auth_done global_id 174130 payload 824 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29300008e0 con 0x7f29301d00d0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 msgr2=0x7f293009dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292ffff640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 0x7f293009dcb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f292000e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920007450 con 0x7f29301d00d0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f292d7fa640 10 monclient: dump: 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.175+0000 7f29353e0640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f291c03bb60 0x7f291c03e020 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f293009f920 tx=0x7f292800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2920007450 con 0x7f29301d00d0 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 msgr2=0x7f293009dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 0x7f293009dcb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f292000e910 comp rx=0 tx=0).stop 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 msgr2=0x7f293009dcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:38.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.179+0000 7f292d7fa640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29301d00d0 0x7f293009dcb0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f292000e910 comp rx=0 tx=0).stop 2024-09-17T15:42:38.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f2936e43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:38.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f2936e43640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 0x7f293009e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f2936e43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2930121b60 con 0x7f2930135a70 2024-09-17T15:42:38.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 0x7f293009e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient: get_auth_request con 0x7f2930135a70 auth_method 0 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 10 monclient(hunting): handle_auth_done global_id 174133 payload 824 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.183+0000 7f292ffff640 1 -- 172.21.15.86:0/4049769701 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2930121d10 con 0x7f2930135a70 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292ffff640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 msgr2=0x7f293009e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292ffff640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 0x7f293009e1f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f2920003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292001e960 con 0x7f2930135a70 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 10 monclient: dump: 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.187 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f2920028420 con 0x7f2930135a70 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 msgr2=0x7f293009e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 0x7f293009e1f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f2920003770 comp rx=0 tx=0).stop 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 msgr2=0x7f293009e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f292d7fa640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2930135a70 0x7f293009e1f0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f292000e450 tx=0x7f2920003770 comp rx=0 tx=0).stop 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:38.188 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 144, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12979, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:38.189 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:38.190 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:38.192 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:38.193 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.194 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.195 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.196 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:38.197 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:38.198 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:38.199 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:38.200 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:38.201 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:38.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f291c03bb60 msgr2=0x7f291c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f291c03bb60 0x7f291c03e020 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f293009f920 tx=0x7f292800b040 comp rx=0 tx=0).stop 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 10 monclient: shutdown 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 msgr2=0x7f293009f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 --2- 172.21.15.86:0/4049769701 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29300a0c70 0x7f293009f3b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f29240ae040 tx=0x7f292401aba0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f29353e0640 1 -- 172.21.15.86:0/4049769701 reap_dead start 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f29353e0640 1 -- 172.21.15.86:0/4049769701 reap_dead start 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 shutdown_connections 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 >> 172.21.15.86:0/4049769701 conn(0x7f29300a2f40 msgr2=0x7f2930072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 shutdown_connections 2024-09-17T15:42:38.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.187+0000 7f2936e43640 1 -- 172.21.15.86:0/4049769701 wait complete. 2024-09-17T15:42:38.204 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:42:38.204 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:42:38.204 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:42:38.204 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:42:38.204 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:42:38.204 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:42:38.205 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:42:38.218 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:42:38.219 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:42:38.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:38.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: get_monmap_and_config 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: build_initial_monmap 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: monmap: 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:38.385323+0000 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:38.385323+0000 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 Processor -- start 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 -- start start 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: init 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 0x7f345c09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f345c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f345c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f345c139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345c09d770 con 0x7f345c068db0 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f345c09ed00 con 0x7f345c0689d0 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f345c09efe0 con 0x7f345c139f00 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f3461c9f640 10 monclient(hunting): authenticate will time out at 1891.351346s 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345b7fe640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 msgr2=0x7f345c09ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 0x7f345c09ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39770/0 (socket says 172.21.15.86:39770) 2024-09-17T15:42:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 -- 172.21.15.86:0/2517663135 learned_addr learned my addr 172.21.15.86:0/2517663135 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): get_auth_request con 0x7f345c139f00 auth_method 0 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient(hunting): handle_auth_done global_id 164246 payload 274 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 -- 172.21.15.86:0/2517663135 >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 msgr2=0x7f345c09ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 --2- 172.21.15.86:0/2517663135 >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 0x7f345c09ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 -- 172.21.15.86:0/2517663135 >> v1:172.21.15.203:6789/0 conn(0x7f345c068db0 legacy=0x7f345c0691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 monclient: found mon.noname-b 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.383+0000 7f345bfff640 1 -- 172.21.15.86:0/2517663135 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345c0a04e0 con 0x7f345c139f00 2024-09-17T15:42:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345bfff640 1 -- 172.21.15.86:0/2517663135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 msgr2=0x7f345c13c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345bfff640 1 --2- 172.21.15.86:0/2517663135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f345c0a0770 tx=0x7f345401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4771d271e5d2f2b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 1 -- 172.21.15.86:0/2517663135 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540a56d0 con 0x7f345c139f00 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: dump: 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 1 -- 172.21.15.86:0/2517663135 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34540a5870 con 0x7f345c139f00 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 1 -- 172.21.15.86:0/2517663135 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34540a5c40 con 0x7f345c139f00 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 5 monclient: authenticate success, global_id 164246 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 4 set_mon_vals no callback set 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345a7fc640 10 monclient: dump: 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.396 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: shutdown 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- 172.21.15.86:0/2517663135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 msgr2=0x7f345c13c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- 172.21.15.86:0/2517663135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f345c0a0770 tx=0x7f345401c980 comp rx=0 tx=0).stop 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- 172.21.15.86:0/2517663135 shutdown_connections 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- 172.21.15.86:0/2517663135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c139f00 0x7f345c13c2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- 172.21.15.86:0/2517663135 >> v2:172.21.15.86:3300/0 conn(0x7f345c0689d0 0x7f345c09ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- 172.21.15.86:0/2517663135 >> 172.21.15.86:0/2517663135 conn(0x7f345c0a6440 msgr2=0x7f345c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- 172.21.15.86:0/2517663135 shutdown_connections 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- 172.21.15.86:0/2517663135 wait complete. 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: build_initial_monmap 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: monmap: 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 Processor -- start 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- start start 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: init 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-c con 0x7f345c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f345c068db0 0x7f345c070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-b con 0x7f345c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 0x7f345c071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): picked mon.noname-a con 0x7f345c139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f345c13f3e0 con 0x7f345c139f00 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f345c13f260 con 0x7f345c068db0 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f345c13f560 con 0x7f345c0689d0 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f3461c9f640 10 monclient(hunting): authenticate will time out at 1891.358508s 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345bfff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 msgr2=0x7f345c071510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 0x7f345c071510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39774/0 (socket says 172.21.15.86:39774) 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 -- 172.21.15.86:0/3745596352 learned_addr learned my addr 172.21.15.86:0/3745596352 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): get_auth_request con 0x7f345c0689d0 auth_method 0 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient(hunting): handle_auth_done global_id 164249 payload 824 2024-09-17T15:42:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 -- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f345c068db0 msgr2=0x7f345c070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f345c068db0 0x7f345c070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 -- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 msgr2=0x7f345c071510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 0x7f345c071510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.391+0000 7f345b7fe640 1 -- 172.21.15.86:0/3745596352 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345c072270 con 0x7f345c0689d0 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f345b7fe640 1 -- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 msgr2=0x7f345c072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f345b7fe640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f344c00d9a0 tx=0x7f344c00de60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c0039c0 con 0x7f345c0689d0 2024-09-17T15:42:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: dump: 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: _finish_auth 0 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344c012070 con 0x7f345c0689d0 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c003dc0 con 0x7f345c0689d0 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 10 monclient: dump: 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 5 monclient: authenticate success, global_id 164249 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _renew_subs 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345c132840 con 0x7f345c0689d0 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3459ffb640 4 set_mon_vals no callback set 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _renew_subs 2024-09-17T15:42:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f345c071a50 con 0x7f345c0689d0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f345c09ffa0 con 0x7f345c0689d0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f344c005070 con 0x7f345c0689d0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 0x7f344803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7f344c0520d0 con 0x7f345c0689d0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f345affd640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 0x7f344803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.395+0000 7f345affd640 10 monclient: get_auth_request con 0x7f344803bb60 auth_method 0 2024-09-17T15:42:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.399+0000 7f345affd640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 0x7f344803e020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3450003cf0 tx=0x7f34500079c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.399+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f344c0203f0 con 0x7f345c0689d0 2024-09-17T15:42:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.399+0000 7f3443fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.399+0000 7f3443fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:38.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3461c9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:38.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3461c9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:38.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3461c9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:38.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f345c06d4f0 con 0x7f345c0689d0 2024-09-17T15:42:38.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3443fff640 1 -- 172.21.15.86:0/3745596352 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f344c01edb0 con 0x7f345c0689d0 2024-09-17T15:42:38.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3443fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:38.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.635+0000 7f3443fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:38.633 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:38.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 msgr2=0x7f344803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 0x7f344803e020 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3450003cf0 tx=0x7f34500079c0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 10 monclient: shutdown 2024-09-17T15:42:38.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:38.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 msgr2=0x7f345c072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f344c00d9a0 tx=0x7f344c00de60 comp rx=0 tx=0).stop 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 shutdown_connections 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f344803bb60 0x7f344803e020 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f345c139f00 0x7f345c071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f345c068db0 0x7f345c070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 --2- 172.21.15.86:0/3745596352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f345c0689d0 0x7f345c072dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 >> 172.21.15.86:0/3745596352 conn(0x7f345c0a6440 msgr2=0x7f345c134cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 shutdown_connections 2024-09-17T15:42:38.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:38.639+0000 7f3461c9f640 1 -- 172.21.15.86:0/3745596352 wait complete. 2024-09-17T15:42:41.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:41.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: get_monmap_and_config 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: build_initial_monmap 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: monmap: 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:41.781925+0000 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:41.781925+0000 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:41.777 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:41.778 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 Processor -- start 2024-09-17T15:42:41.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 -- start start 2024-09-17T15:42:41.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: init 2024-09-17T15:42:41.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f49e00689d0 0x7f49e009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-c con 0x7f49e00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-b con 0x7f49e0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49e0134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49e009d770 con 0x7f49e0134a40 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f49e009ed00 con 0x7f49e00689d0 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f49e009efe0 con 0x7f49e0068db0 2024-09-17T15:42:41.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49e5928640 10 monclient(hunting): authenticate will time out at 1894.747429s 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39784/0 (socket says 172.21.15.86:39784) 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 1 -- 172.21.15.86:0/1002732003 learned_addr learned my addr 172.21.15.86:0/1002732003 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): get_auth_request con 0x7f49e0068db0 auth_method 0 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49deffd640 1 --2- 172.21.15.86:0/1002732003 >> v2:172.21.15.203:3300/0 conn(0x7f49e00689d0 0x7f49e009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49deffd640 10 monclient(hunting): get_auth_request con 0x7f49e00689d0 auth_method 0 2024-09-17T15:42:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49deffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.779+0000 7f49de7fc640 10 monclient(hunting): handle_auth_done global_id 164252 payload 274 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 1 -- 172.21.15.86:0/1002732003 >> v2:172.21.15.203:3300/0 conn(0x7f49e00689d0 msgr2=0x7f49e009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 1 --2- 172.21.15.86:0/1002732003 >> v2:172.21.15.203:3300/0 conn(0x7f49e00689d0 0x7f49e009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 1 -- 172.21.15.86:0/1002732003 >> v1:172.21.15.203:6789/0 conn(0x7f49e0134a40 legacy=0x7f49e0134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 1 monclient: found mon.noname-b 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:41.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.783+0000 7f49de7fc640 1 -- 172.21.15.86:0/1002732003 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e00a04e0 con 0x7f49e0068db0 2024-09-17T15:42:41.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 -- 172.21.15.86:0/1002732003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 msgr2=0x7f49e0069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:41.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 --2- 172.21.15.86:0/1002732003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49d000bb10 tx=0x7f49d001eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=971774ab2d7de29a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:41.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 1 -- 172.21.15.86:0/1002732003 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00b4070 con 0x7f49e0068db0 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: dump: 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:41.788 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 1 -- 172.21.15.86:0/1002732003 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d00a7af0 con 0x7f49e0068db0 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 1 -- 172.21.15.86:0/1002732003 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00af440 con 0x7f49e0068db0 2024-09-17T15:42:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:41.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:41.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49ddffb640 10 monclient: dump: 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 5 monclient: authenticate success, global_id 164252 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:41.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 4 set_mon_vals no callback set 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: shutdown 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- 172.21.15.86:0/1002732003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 msgr2=0x7f49e0069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- 172.21.15.86:0/1002732003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49d000bb10 tx=0x7f49d001eb80 comp rx=0 tx=0).stop 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- 172.21.15.86:0/1002732003 shutdown_connections 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- 172.21.15.86:0/1002732003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e0069190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- 172.21.15.86:0/1002732003 >> v2:172.21.15.203:3300/0 conn(0x7f49e00689d0 0x7f49e009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- 172.21.15.86:0/1002732003 >> 172.21.15.86:0/1002732003 conn(0x7f49e00a6440 msgr2=0x7f49e00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- 172.21.15.86:0/1002732003 shutdown_connections 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- 172.21.15.86:0/1002732003 wait complete. 2024-09-17T15:42:41.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: build_initial_monmap 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: monmap: 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:41.794 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.795 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:41.795 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:41.795 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:41.795 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 Processor -- start 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- start start 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: init 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49e00689d0 0x7f49e0133920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-b con 0x7f49e00689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-c con 0x7f49e0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 0x7f49e013b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): picked mon.noname-a con 0x7f49e0134a40 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f49e009c650 con 0x7f49e0134a40 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49e009c4d0 con 0x7f49e00689d0 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f49e009c7d0 con 0x7f49e0068db0 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49e5928640 10 monclient(hunting): authenticate will time out at 1894.754937s 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49df7fe640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 msgr2=0x7f49e013b1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49df7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 0x7f49e013b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:41.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39800/0 (socket says 172.21.15.86:39800) 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 -- 172.21.15.86:0/862511307 learned_addr learned my addr 172.21.15.86:0/862511307 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): get_auth_request con 0x7f49e0068db0 auth_method 0 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient(hunting): handle_auth_done global_id 164255 payload 824 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 -- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49e00689d0 msgr2=0x7f49e0133920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:41.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49e00689d0 0x7f49e0133920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 -- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 msgr2=0x7f49e013b1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 0x7f49e013b1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 monclient: found mon.noname-c 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:41.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.787+0000 7f49de7fc640 1 -- 172.21.15.86:0/862511307 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e011db10 con 0x7f49e0068db0 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49de7fc640 1 -- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 msgr2=0x7f49e013ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49de7fc640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b2c0 tx=0x7f49d001b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49de7fc640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b2c0 tx=0x7f49d001b920 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f49d00b4040 mon_map magic: 0 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00b4040 con 0x7f49e0068db0 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: dump: 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.802 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: _finish_auth 0 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 5 monclient: authenticate success, global_id 164255 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49d00bf070 con 0x7f49e0068db0 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:41.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49dcff9640 4 set_mon_vals no callback set 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _renew_subs 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e011ddf0 con 0x7f49e0068db0 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _renew_subs 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e00a3880 con 0x7f49e0068db0 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49e009ffa0 con 0x7f49e0068db0 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00b8a40 con 0x7f49e0068db0 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 10 monclient: dump: 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:41.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f49d00b8c20 con 0x7f49e0068db0 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49c7fff640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 0x7f49cc03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49deffd640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 0x7f49cc03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.791+0000 7f49deffd640 10 monclient: get_auth_request con 0x7f49cc03bb40 auth_method 0 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.795+0000 7f49deffd640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 0x7f49cc03e000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49d400b850 tx=0x7f49d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:41.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:41.795+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7f49d00ed590 con 0x7f49e0068db0 2024-09-17T15:42:42.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.415+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f49d0018800 con 0x7f49e0068db0 2024-09-17T15:42:42.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.415+0000 7f49c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:42.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.415+0000 7f49c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49e5928640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:42.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49e5928640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:42.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49e5928640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:42.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f49e011cbb0 con 0x7f49e0068db0 2024-09-17T15:42:42.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49c7fff640 1 -- 172.21.15.86:0/862511307 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f49d00ebfa0 con 0x7f49e0068db0 2024-09-17T15:42:42.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:42.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.655+0000 7f49c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:42.656 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:42.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 msgr2=0x7f49cc03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:42.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 0x7f49cc03e000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49d400b850 tx=0x7f49d4008040 comp rx=0 tx=0).stop 2024-09-17T15:42:42.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 10 monclient: shutdown 2024-09-17T15:42:42.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:42.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 msgr2=0x7f49e013ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:42.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f49d000b2c0 tx=0x7f49d001b920 comp rx=0 tx=0).stop 2024-09-17T15:42:42.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 shutdown_connections 2024-09-17T15:42:42.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f49cc03bb40 0x7f49cc03e000 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:42.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49e0134a40 0x7f49e013b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:42.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49e0068db0 0x7f49e013ac90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:42.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 --2- 172.21.15.86:0/862511307 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49e00689d0 0x7f49e0133920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:42.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 >> 172.21.15.86:0/862511307 conn(0x7f49e00a6440 msgr2=0x7f49e0074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:42.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 shutdown_connections 2024-09-17T15:42:42.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:42.659+0000 7f49e5928640 1 -- 172.21.15.86:0/862511307 wait complete. 2024-09-17T15:42:45.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: get_monmap_and_config 2024-09-17T15:42:45.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: build_initial_monmap 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: monmap: 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:45.805701+0000 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:45.805701+0000 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:45.801 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:45.802 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:45.802 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 Processor -- start 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 -- start start 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: init 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-b con 0x7f7ca80689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 0x7f7ca8069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-a con 0x7f7ca8068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ca8134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:45.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ca809d770 con 0x7f7ca8134a40 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7ca809ed00 con 0x7f7ca8068db0 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca809efe0 con 0x7f7ca80689d0 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.803+0000 7f7cae881640 10 monclient(hunting): authenticate will time out at 1898.771939s 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca77fe640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 msgr2=0x7f7ca8069190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 0x7f7ca8069190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:45.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39802/0 (socket says 172.21.15.86:39802) 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 -- 172.21.15.86:0/3726851120 learned_addr learned my addr 172.21.15.86:0/3726851120 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): get_auth_request con 0x7f7ca80689d0 auth_method 0 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_done global_id 164258 payload 274 2024-09-17T15:42:45.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 -- 172.21.15.86:0/3726851120 >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 msgr2=0x7f7ca8069190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/3726851120 >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 0x7f7ca8069190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 -- 172.21.15.86:0/3726851120 >> v1:172.21.15.203:6789/0 conn(0x7f7ca8134a40 legacy=0x7f7ca8134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 monclient: found mon.noname-b 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:42:45.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.807+0000 7f7ca7fff640 1 -- 172.21.15.86:0/3726851120 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca80a04e0 con 0x7f7ca80689d0 2024-09-17T15:42:45.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 -- 172.21.15.86:0/3726851120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 msgr2=0x7f7ca809ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:45.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/3726851120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800bb10 tx=0x7f7c9801eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86c4e1fcb48fde45 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 1 -- 172.21.15.86:0/3726851120 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c980b4070 con 0x7f7ca80689d0 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: dump: 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 1 -- 172.21.15.86:0/3726851120 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c980a7bf0 con 0x7f7ca80689d0 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 5 monclient: authenticate success, global_id 164258 2024-09-17T15:42:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 4 set_mon_vals no callback set 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 1 -- 172.21.15.86:0/3726851120 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c980af440 con 0x7f7ca80689d0 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca6ffd640 10 monclient: dump: 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: shutdown 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:45.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- 172.21.15.86:0/3726851120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 msgr2=0x7f7ca809ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- 172.21.15.86:0/3726851120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800bb10 tx=0x7f7c9801eba0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- 172.21.15.86:0/3726851120 shutdown_connections 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- 172.21.15.86:0/3726851120 >> v2:172.21.15.86:3300/0 conn(0x7f7ca8068db0 0x7f7ca8069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- 172.21.15.86:0/3726851120 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca809ffa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- 172.21.15.86:0/3726851120 >> 172.21.15.86:0/3726851120 conn(0x7f7ca80a6440 msgr2=0x7f7ca80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- 172.21.15.86:0/3726851120 shutdown_connections 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- 172.21.15.86:0/3726851120 wait complete. 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: build_initial_monmap 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: monmap: 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:45.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 Processor -- start 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- start start 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: init 2024-09-17T15:42:45.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-c con 0x7f7ca80689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 0x7f7ca8133d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-b con 0x7f7ca8068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 0x7f7ca813abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): picked mon.noname-a con 0x7f7ca8134a40 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca809c650 con 0x7f7ca8134a40 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca809c4d0 con 0x7f7ca8068db0 2024-09-17T15:42:45.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca809c7d0 con 0x7f7ca80689d0 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cae881640 10 monclient(hunting): authenticate will time out at 1898.779583s 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cace1e640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 msgr2=0x7f7ca813abd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7cace1e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 0x7f7ca813abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:39810/0 (socket says 172.21.15.86:39810) 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 -- 172.21.15.86:0/2171714114 learned_addr learned my addr 172.21.15.86:0/2171714114 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): get_auth_request con 0x7f7ca80689d0 auth_method 0 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:45.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 10 monclient(hunting): handle_auth_done global_id 164261 payload 824 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca77fe640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 0x7f7ca8133d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.811+0000 7f7ca7fff640 1 -- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 msgr2=0x7f7ca8133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 0x7f7ca8133d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 -- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 msgr2=0x7f7ca813abd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 0x7f7ca813abd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:45.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 monclient: found mon.noname-c 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 -- 172.21.15.86:0/2171714114 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca809d460 con 0x7f7ca80689d0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca77fe640 10 monclient: get_auth_request con 0x7f7ca8068db0 auth_method 0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca77fe640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 0x7f7ca8133d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 -- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 msgr2=0x7f7ca8133800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b2c0 tx=0x7f7c9801b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca7fff640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b2c0 tx=0x7f7c9801b940 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f7c980ba470 config(0 keys) 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c980bf020 con 0x7f7ca80689d0 2024-09-17T15:42:45.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: dump: 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:45.825 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c980ba470 con 0x7f7ca80689d0 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 5 monclient: authenticate success, global_id 164261 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _renew_subs 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca811e0b0 con 0x7f7ca80689d0 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca67fc640 4 set_mon_vals no callback set 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _renew_subs 2024-09-17T15:42:45.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca811cef0 con 0x7f7ca80689d0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c980b4040 con 0x7f7ca80689d0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca809ffa0 con 0x7f7ca80689d0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 10 monclient: dump: 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7c980c2020 con 0x7f7ca80689d0 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.815+0000 7f7ca4ff9640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 0x7f7c9403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:45.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.819+0000 7f7ca77fe640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 0x7f7c9403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:45.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.819+0000 7f7ca77fe640 10 monclient: get_auth_request con 0x7f7c9403bb40 auth_method 0 2024-09-17T15:42:45.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.819+0000 7f7ca77fe640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 0x7f7c9403e000 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7ca811d380 tx=0x7f7c9c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:45.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:45.823+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7f7c980ed3c0 con 0x7f7ca80689d0 2024-09-17T15:42:46.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.439+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7c980c9030 con 0x7f7ca80689d0 2024-09-17T15:42:46.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.439+0000 7f7ca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:46.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.439+0000 7f7ca4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:46.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.667+0000 7f7cae881640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:46.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.667+0000 7f7cae881640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:46.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.667+0000 7f7cae881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:46.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.667+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7ca806d4f0 con 0x7f7ca80689d0 2024-09-17T15:42:46.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.671+0000 7f7ca4ff9640 1 -- 172.21.15.86:0/2171714114 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f7ca809d460 con 0x7f7ca80689d0 2024-09-17T15:42:46.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.671+0000 7f7ca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:46.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.671+0000 7f7ca4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:46.668 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 msgr2=0x7f7c9403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 0x7f7c9403e000 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7ca811d380 tx=0x7f7c9c008040 comp rx=0 tx=0).stop 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 10 monclient: shutdown 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 msgr2=0x7f7ca8133800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800b2c0 tx=0x7f7c9801b940 comp rx=0 tx=0).stop 2024-09-17T15:42:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 shutdown_connections 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7c9403bb40 0x7f7c9403e000 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ca8134a40 0x7f7ca813abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8068db0 0x7f7ca8133d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 --2- 172.21.15.86:0/2171714114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ca80689d0 0x7f7ca8133800 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 >> 172.21.15.86:0/2171714114 conn(0x7f7ca80a6440 msgr2=0x7f7ca8074170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 shutdown_connections 2024-09-17T15:42:46.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:46.675+0000 7f7cae881640 1 -- 172.21.15.86:0/2171714114 wait complete. 2024-09-17T15:42:49.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: get_monmap_and_config 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: build_initial_monmap 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: monmap: 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:49.821395+0000 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:49.821395+0000 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:49.816 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:49.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:49.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 Processor -- start 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 -- start start 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: init 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-c con 0x7ff8680689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff868068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff86813aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86809c8b0 con 0x7ff86813aff0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86809cb90 con 0x7ff868068db0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff868074500 con 0x7ff8680689d0 2024-09-17T15:42:49.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86cacf640 10 monclient(hunting): authenticate will time out at 1902.786880s 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff866d9d640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7ff86813aff0 legacy=0x7ff86813d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff865d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff868068db0 0x7ff868069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:52098/0 (socket says 172.21.15.86:52098) 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 learned_addr learned my addr 172.21.15.86:0/807116448 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 10 monclient(hunting): get_auth_request con 0x7ff8680689d0 auth_method 0 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:49.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.819+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:49.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:49.823+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.203:6789/0 conn(0x7ff868068db0 legacy=0x7ff868069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:50.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff866d9d640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.86:6789/0 conn(0x7ff86813aff0 legacy=0x7ff86813d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:42:50.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 10 monclient(hunting): get_auth_request con 0x7ff8680689d0 auth_method 0 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:50.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.023+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:50.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff866d9d640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.86:6789/0 conn(0x7ff86813aff0 legacy=0x7ff86813d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 10 monclient(hunting): get_auth_request con 0x7ff8680689d0 auth_method 0 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:50.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:50.427+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff866d9d640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.86:6789/0 conn(0x7ff86813aff0 legacy=0x7ff86813d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 10 monclient(hunting): get_auth_request con 0x7ff8680689d0 auth_method 0 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:51.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:51.231+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:42:52.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): tick 2024-09-17T15:42:52.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:42:52.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:42:52.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:42:52.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 msgr2=0x7ff86806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff8680689d0 0x7ff86806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.203:6789/0 conn(0x7ff868068db0 legacy=0x7ff868069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 >> v1:172.21.15.86:6789/0 conn(0x7ff86813aff0 legacy=0x7ff86813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient: _add_conns ranks=[3,2] 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): picked mon.noname-a con 0x7ff848003740 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): picked mon.noname-b con 0x7ff848006230 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff848008da0 con 0x7ff848003740 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff848008f50 con 0x7ff848006230 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff85ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:52.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 msgr2=0x7ff848005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006230 auth_method 0 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:52.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:52.819+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 msgr2=0x7ff848005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:53.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006230 auth_method 0 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.023+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 msgr2=0x7ff848005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006230 auth_method 0 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:42:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:53.427+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006230 auth_method 0 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 msgr2=0x7ff848005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:42:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:42:54.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:42:54.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:42:54.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:54.231+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): tick 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 msgr2=0x7ff8480086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848006230 0x7ff8480086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 msgr2=0x7ff848005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003740 0x7ff848005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:42:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff866d9d640 1 -- 172.21.15.86:0/807116448 reap_dead start 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 0x7ff8480066e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): picked mon.noname-a con 0x7ff848003b90 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff848006c20 0x7ff848007000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): picked mon.noname-c con 0x7ff848006c20 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 0x7ff8480079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): picked mon.noname-b con 0x7ff848007540 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff848008da0 con 0x7ff848003b90 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff848008f50 con 0x7ff848006c20 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 1 -- 172.21.15.86:0/807116448 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff848008120 con 0x7ff848007540 2024-09-17T15:42:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff85ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff86659c640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 msgr2=0x7ff8480066e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.819+0000 7ff86659c640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 0x7ff8480066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 0x7ff8480079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.203:3300/0 conn(0x7ff848006c20 0x7ff848007000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): get_auth_request con 0x7ff848007540 auth_method 0 2024-09-17T15:42:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006c20 auth_method 0 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient(hunting): handle_auth_done global_id 174139 payload 274 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 msgr2=0x7ff8480079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 0x7ff8480079b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 msgr2=0x7ff8480066e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 0x7ff8480066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 monclient: found mon.noname-c 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86809ce70 con 0x7ff848006c20 2024-09-17T15:42:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.823+0000 7ff866d9d640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 0x7ff8480079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:42:55.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff865d9b640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 msgr2=0x7ff848007000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:55.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff865d9b640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff848007000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff848008cf0 tx=0x7ff85001d170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11cca6e45e8f7f76 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 1 -- 172.21.15.86:0/807116448 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8500ab070 con 0x7ff848006c20 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: dump: 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: _finish_auth 0 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: _check_auth_tickets 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 1 -- 172.21.15.86:0/807116448 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8500af400 con 0x7ff848006c20 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 1 -- 172.21.15.86:0/807116448 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8500af7f0 con 0x7ff848006c20 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86559a640 10 monclient: dump: 2024-09-17T15:42:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 5 monclient: authenticate success, global_id 174139 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:55.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 4 set_mon_vals no callback set 2024-09-17T15:42:55.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 10 monclient: shutdown 2024-09-17T15:42:55.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:55.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 -- 172.21.15.86:0/807116448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 msgr2=0x7ff848007000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:55.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff848007000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff848008cf0 tx=0x7ff85001d170 comp rx=0 tx=0).stop 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 -- 172.21.15.86:0/807116448 shutdown_connections 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848007540 0x7ff8480079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 --2- 172.21.15.86:0/807116448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff848007000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 --2- 172.21.15.86:0/807116448 >> v2:172.21.15.86:3300/0 conn(0x7ff848003b90 0x7ff8480066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 -- 172.21.15.86:0/807116448 >> 172.21.15.86:0/807116448 conn(0x7ff8680a6440 msgr2=0x7ff8680a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 -- 172.21.15.86:0/807116448 shutdown_connections 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.831+0000 7ff86cacf640 1 -- 172.21.15.86:0/807116448 wait complete. 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient: build_initial_monmap 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient: monmap: 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:55.836 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 Processor -- start 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 -- start start 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient: init 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-c con 0x7ff848003b90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff8681d0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-b con 0x7ff848006c20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 0x7ff868123660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): picked mon.noname-a con 0x7ff848007540 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff8681362f0 con 0x7ff848007540 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff86809c2d0 con 0x7ff848006c20 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff868136470 con 0x7ff848003b90 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86cacf640 10 monclient(hunting): authenticate will time out at 1908.801295s 2024-09-17T15:42:55.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff866d9d640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 msgr2=0x7ff868123660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff866d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 0x7ff868123660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58984/0 (socket says 172.21.15.86:58984) 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 -- 172.21.15.86:0/1744060407 learned_addr learned my addr 172.21.15.86:0/1744060407 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): get_auth_request con 0x7ff848003b90 auth_method 0 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff8681d0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:55.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): get_auth_request con 0x7ff848006c20 auth_method 0 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff865d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient(hunting): handle_auth_done global_id 164267 payload 824 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 -- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 msgr2=0x7ff8681d0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff8681d0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 -- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 msgr2=0x7ff868123660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 0x7ff868123660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient: _finish_hunting 0 2024-09-17T15:42:55.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 monclient: found mon.noname-c 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.835+0000 7ff86659c640 1 -- 172.21.15.86:0/1744060407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff868125090 con 0x7ff848003b90 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86659c640 1 -- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 msgr2=0x7ff8681d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86659c640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff8600139c0 tx=0x7ff860013e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86001a070 con 0x7ff848003b90 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: dump: 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.841 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: _finish_auth 0 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86000ed20 con 0x7ff848003b90 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 5 monclient: authenticate success, global_id 164267 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _renew_subs 2024-09-17T15:42:55.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff868127dd0 con 0x7ff848003b90 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86000f040 con 0x7ff848003b90 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 10 monclient: dump: 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _renew_subs 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff864d99640 4 set_mon_vals no callback set 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8681d0ec0 con 0x7ff848003b90 2024-09-17T15:42:55.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff86001d070 con 0x7ff848003b90 2024-09-17T15:42:55.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:55.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:55.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8681368a0 con 0x7ff848003b90 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 0x7ff848040080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7ff860050540 con 0x7ff848003b90 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff865d9b640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 0x7ff848040080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.839+0000 7ff865d9b640 10 monclient: get_auth_request con 0x7ff84803dbc0 auth_method 0 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.843+0000 7ff865d9b640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 0x7ff848040080 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8500ad000 tx=0x7ff850010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.847+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff86004ef50 con 0x7ff848003b90 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.847+0000 7ff85effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:55.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:55.847+0000 7ff85effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:56.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.071+0000 7ff86cacf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:42:56.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.071+0000 7ff86cacf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:56.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.071+0000 7ff86cacf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.071+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff86806d4f0 con 0x7ff848003b90 2024-09-17T15:42:56.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff85effd640 1 -- 172.21.15.86:0/1744060407 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7ff86001edc0 con 0x7ff848003b90 2024-09-17T15:42:56.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff85effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:42:56.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff85effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:56.071 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:42:56.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 msgr2=0x7ff848040080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 0x7ff848040080 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8500ad000 tx=0x7ff850010040 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 10 monclient: shutdown 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 msgr2=0x7ff8681d0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff8600139c0 tx=0x7ff860013e80 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 shutdown_connections 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84803dbc0 0x7ff848040080 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff848007540 0x7ff868123660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff848006c20 0x7ff8681d0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 --2- 172.21.15.86:0/1744060407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff848003b90 0x7ff8681d0440 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 >> 172.21.15.86:0/1744060407 conn(0x7ff8680a6440 msgr2=0x7ff868073780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:56.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 shutdown_connections 2024-09-17T15:42:56.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.075+0000 7ff86cacf640 1 -- 172.21.15.86:0/1744060407 wait complete. 2024-09-17T15:42:56.089 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:42:56.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:42:56.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: get_monmap_and_config 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: build_initial_monmap 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: monmap: 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:56.201045+0000 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:56.201045+0000 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:56.196 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:56.197 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:56.197 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:56.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 Processor -- start 2024-09-17T15:42:56.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 -- start start 2024-09-17T15:42:56.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: init 2024-09-17T15:42:56.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0700a01a0 0x7fb07009f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0700a01a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0700a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb0700a1380 addr v1:172.21.15.86:6789/0 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb070074e00 con 0x7fb0700a1380 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07009e350 con 0x7fb0700a0580 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fb07009e630 con 0x7fb0700a01a0 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb0764d8640 10 monclient(hunting): authenticate will time out at 1909.167046s 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb074a75640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fb0700a1380 legacy=0x7fb07013ba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:42:56.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb06ffff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fb0700a01a0 msgr2=0x7fb07009f010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb06ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0700a01a0 0x7fb07009f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb06f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fb0700a0580 0x7fb0700a0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:52110/0 (socket says 172.21.15.86:52110) 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.199+0000 7fb06f7fe640 1 -- 172.21.15.86:0/2981850028 learned_addr learned my addr 172.21.15.86:0/2981850028 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 1 -- 172.21.15.86:0/2981850028 >> v1:172.21.15.203:6789/0 conn(0x7fb0700a0580 legacy=0x7fb0700a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031151171 0 0) 0x7fb07009e350 con 0x7fb0700a0580 2024-09-17T15:42:56.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): _negotiate 2024-09-17T15:42:56.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): my global_id is 174145 2024-09-17T15:42:56.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb058003530 con 0x7fb0700a0580 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3367160476 0 0) 0x7fb058003530 con 0x7fb0700a0580 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 >> v2:172.21.15.86:3300/0 conn(0x7fb0700a01a0 msgr2=0x7fb07009f010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 --2- 172.21.15.86:0/2981850028 >> v2:172.21.15.86:3300/0 conn(0x7fb0700a01a0 0x7fb07009f010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 >> v1:172.21.15.86:6789/0 conn(0x7fb0700a1380 legacy=0x7fb07013ba80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _finish_hunting 0 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07009e910 con 0x7fb0700a0580 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _finish_auth 0 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _check_auth_tickets 2024-09-17T15:42:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fb0640019b0 con 0x7fb0700a0580 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb0764d8640 5 monclient: authenticate success, global_id 174145 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb0764d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: dump: 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.205 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 >> v1:172.21.15.203:6789/0 conn(0x7fb0700a0580 legacy=0x7fb0700a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058003a30 0x7fb058003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): picked mon.a con 0x7fb058003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb058004810 0x7fb058004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): picked mon.b con 0x7fb058004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 0x7fb05800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): picked mon.c con 0x7fb0580099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:56.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb07009e630 con 0x7fb058003a30 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb05800c3a0 con 0x7fb058004810 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb05800e7b0 con 0x7fb0580099c0 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06effd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb0764d8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06ffff640 1 -- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058003a30 msgr2=0x7fb058003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06ffff640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058003a30 0x7fb058003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 0x7fb05800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): get_auth_request con 0x7fb0580099c0 auth_method 0 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb058004810 0x7fb058004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request con 0x7fb058004810 auth_method 0 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb06f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient(hunting): handle_auth_done global_id 174145 payload 293 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 -- 172.21.15.86:0/2981850028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb058004810 msgr2=0x7fb058004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb058004810 0x7fb058004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 -- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058003a30 msgr2=0x7fb058003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb058003a30 0x7fb058003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient: _finish_hunting 0 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 monclient: found mon.c 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 1 -- 172.21.15.86:0/2981850028 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058003530 con 0x7fb0580099c0 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient: _finish_auth 0 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 10 monclient: _check_auth_tickets 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.203+0000 7fb074a75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:56.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb074a75640 1 -- 172.21.15.86:0/2981850028 reap_dead start 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb074a75640 1 -- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 msgr2=0x7fb05800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb074a75640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 0x7fb05800bdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb068009e30 tx=0x7fb06801caf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed2fdd7b0e2fbbfc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068009a30 con 0x7fb0580099c0 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: dump: 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb0764d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0680a9070 con 0x7fb0580099c0 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb0764d8640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 1 -- 172.21.15.86:0/2981850028 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0680a5bf0 con 0x7fb0580099c0 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb0764d8640 4 set_mon_vals no callback set 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.207+0000 7fb06effd640 10 monclient: dump: 2024-09-17T15:42:56.218 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: shutdown 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 msgr2=0x7fb05800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 0x7fb05800bdb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb068009e30 tx=0x7fb06801caf0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.219 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- 172.21.15.86:0/2981850028 shutdown_connections 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 --2- 172.21.15.86:0/2981850028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0580099c0 0x7fb05800bdb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- 172.21.15.86:0/2981850028 >> 172.21.15.86:0/2981850028 conn(0x7fb0700a2f40 msgr2=0x7fb0700a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- 172.21.15.86:0/2981850028 shutdown_connections 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- 172.21.15.86:0/2981850028 wait complete. 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: build_initial_monmap 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: monmap: 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:56.220 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 Processor -- start 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- start start 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: init 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0580099c0 0x7fb07013c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-b con 0x7fb0580099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 0x7fb0701cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-c con 0x7fb07013ccb0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0701cb9f0 0x7fb0701cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): picked mon.noname-a con 0x7fb0701cb9f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb07009f3a0 con 0x7fb0701cb9f0 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb07009f220 con 0x7fb0580099c0 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb0700a0c70 con 0x7fb07013ccb0 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb0764d8640 10 monclient(hunting): authenticate will time out at 1909.177903s 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb074a75640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0701cb9f0 msgr2=0x7fb0701cbe50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb074a75640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0701cb9f0 0x7fb0701cbe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:56.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 0x7fb0701cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 0x7fb0701cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59004/0 (socket says 172.21.15.86:59004) 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 -- 172.21.15.86:0/165238097 learned_addr learned my addr 172.21.15.86:0/165238097 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request con 0x7fb07013ccb0 auth_method 0 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0580099c0 0x7fb07013c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient(hunting): handle_auth_done global_id 164270 payload 824 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0580099c0 msgr2=0x7fb07013c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0580099c0 0x7fb07013c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0701cb9f0 msgr2=0x7fb0701cbe50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0701cb9f0 0x7fb0701cbe50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06f7fe640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0701c89d0 con 0x7fb07013ccb0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06ffff640 10 monclient: get_auth_request con 0x7fb0580099c0 auth_method 0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.211+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb0580099c0 0x7fb07013c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06f7fe640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 msgr2=0x7fb0701cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06f7fe640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 0x7fb0701cb4b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb064005e90 tx=0x7fb064007420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06400e8d0 con 0x7fb07013ccb0 2024-09-17T15:42:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: dump: 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: _finish_auth 0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 5 monclient: authenticate success, global_id 164270 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb064018070 con 0x7fb07013ccb0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 monclient: _renew_subs 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb070125e10 con 0x7fb07013ccb0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06400ecd0 con 0x7fb07013ccb0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 10 monclient: dump: 2024-09-17T15:42:56.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 monclient: _renew_subs 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06e7fc640 4 set_mon_vals no callback set 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07009d460 con 0x7fb07013ccb0 2024-09-17T15:42:56.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb06401d460 con 0x7fb07013ccb0 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 0x7fb07011e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb0700a1000 con 0x7fb070125850 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 0x7fb07011e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient: get_auth_request con 0x7fb070125850 auth_method 0 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb05c03bb60 0x7fb05c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7fb06404fed0 con 0x7fb07013ccb0 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 10 monclient(hunting): handle_auth_done global_id 164273 payload 824 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb06ffff640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb070123d90 con 0x7fb070125850 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb074a75640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb05c03bb60 0x7fb05c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.215+0000 7fb074a75640 10 monclient: get_auth_request con 0x7fb05c03bb60 auth_method 0 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06ffff640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 msgr2=0x7fb07011e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 0x7fb07011e8d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb06000c6f0 tx=0x7fb06000cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060003dd0 con 0x7fb070125850 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 10 monclient: dump: 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb060016070 con 0x7fb070125850 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 msgr2=0x7fb07011e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 0x7fb07011e8d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb06000c6f0 tx=0x7fb06000cbb0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 msgr2=0x7fb07011e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:56.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb06cff9640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb070125850 0x7fb07011e8d0 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb06000c6f0 tx=0x7fb06000cbb0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.219+0000 7fb074a75640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb05c03bb60 0x7fb05c03e020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb0700a2c90 tx=0x7fb068005e10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb0764d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:56.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb0764d8640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 0x7fb07011f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb0764d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb070124ca0 con 0x7fb07011ee10 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 0x7fb07011f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient: get_auth_request con 0x7fb07011ee10 auth_method 0 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 10 monclient(hunting): handle_auth_done global_id 164276 payload 824 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.223+0000 7fb06ffff640 1 -- 172.21.15.86:0/165238097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb07006d4f0 con 0x7fb07011ee10 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06ffff640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 msgr2=0x7fb07011f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06ffff640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 0x7fb07011f220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb060015000 tx=0x7fb0600059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060005690 con 0x7fb07011ee10 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 10 monclient: dump: 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7fb060005690 con 0x7fb07011ee10 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 msgr2=0x7fb07011f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 0x7fb07011f220 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb060015000 tx=0x7fb0600059e0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 msgr2=0x7fb07011f220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb06cff9640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07011ee10 0x7fb07011f220 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb060015000 tx=0x7fb0600059e0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:56.235 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 146, 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 79415, 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:56.236 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:56.237 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.238 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.238 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:56.239 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:56.240 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:56.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.242 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:56.243 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.245 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.246 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:56.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.249 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:56.249 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb05c03bb60 msgr2=0x7fb05c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb05c03bb60 0x7fb05c03e020 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb0700a2c90 tx=0x7fb068005e10 comp rx=0 tx=0).stop 2024-09-17T15:42:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 10 monclient: shutdown 2024-09-17T15:42:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:56.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 msgr2=0x7fb0701cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 --2- 172.21.15.86:0/165238097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb07013ccb0 0x7fb0701cb4b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb064005e90 tx=0x7fb064007420 comp rx=0 tx=0).stop 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb074a75640 1 -- 172.21.15.86:0/165238097 reap_dead start 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb074a75640 1 -- 172.21.15.86:0/165238097 reap_dead start 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 shutdown_connections 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 >> 172.21.15.86:0/165238097 conn(0x7fb0700a2f40 msgr2=0x7fb0700a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 shutdown_connections 2024-09-17T15:42:56.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.227+0000 7fb0764d8640 1 -- 172.21.15.86:0/165238097 wait complete. 2024-09-17T15:42:56.251 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:42:56.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:42:56.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:42:56.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.367+0000 7f5acb4f1640 10 monclient: get_monmap_and_config 2024-09-17T15:42:56.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.367+0000 7f5acb4f1640 10 monclient: build_initial_monmap 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.367+0000 7f5acb4f1640 10 monclient: monmap: 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:42:56.371889+0000 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:42:56.371889+0000 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:42:56.367 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 Processor -- start 2024-09-17T15:42:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 -- start start 2024-09-17T15:42:56.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient: init 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5ac40a50e0 0x7f5ac409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac40a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 0x7f5ac40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac40a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ac4137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac40a6280 con 0x7f5ac4137ea0 2024-09-17T15:42:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac409c0a0 con 0x7f5ac40a54c0 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac409c380 con 0x7f5ac40a50e0 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5acb4f1640 10 monclient(hunting): authenticate will time out at 1909.337467s 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac8a8c640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 msgr2=0x7f5ac40a58d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:56.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac8a8c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 0x7f5ac40a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5ac40a50e0 0x7f5ac409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac9a8e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5ac4137ea0 0x7f5ac4138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:52120/0 (socket says 172.21.15.86:52120) 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/1887230945 learned_addr learned my addr 172.21.15.86:0/1887230945 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): get_auth_request con 0x7f5ac40a50e0 auth_method 0 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient(hunting): handle_auth_done global_id 174148 payload 274 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 -- 172.21.15.86:0/1887230945 >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 msgr2=0x7f5ac40a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 --2- 172.21.15.86:0/1887230945 >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 0x7f5ac40a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 -- 172.21.15.86:0/1887230945 >> v1:172.21.15.203:6789/0 conn(0x7f5ac4137ea0 legacy=0x7f5ac4138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient: _finish_hunting 0 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 monclient: found mon.noname-c 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:42:56.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.371+0000 7f5ac928d640 1 -- 172.21.15.86:0/1887230945 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac409c660 con 0x7f5ac40a50e0 2024-09-17T15:42:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5ac928d640 1 -- 172.21.15.86:0/1887230945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a50e0 msgr2=0x7f5ac409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5ac928d640 1 --2- 172.21.15.86:0/1887230945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a50e0 0x7f5ac409cd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40aa620 tx=0x7f5ac001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=178ec3a75d639ebc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5abbfff640 1 -- 172.21.15.86:0/1887230945 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00b0070 con 0x7f5ac40a50e0 2024-09-17T15:42:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5abbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.375+0000 7f5abbfff640 10 monclient: dump: 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: _finish_auth 0 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 1 -- 172.21.15.86:0/1887230945 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ac00ab600 con 0x7f5ac40a50e0 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 1 -- 172.21.15.86:0/1887230945 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00ab9d0 con 0x7f5ac40a50e0 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5abbfff640 10 monclient: dump: 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 5 monclient: authenticate success, global_id 174148 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: get_monmap_and_config success 2024-09-17T15:42:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 4 set_mon_vals no callback set 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: shutdown 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- 172.21.15.86:0/1887230945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a50e0 msgr2=0x7f5ac409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/1887230945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a50e0 0x7f5ac409cd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5ac40aa620 tx=0x7f5ac001c3f0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- 172.21.15.86:0/1887230945 shutdown_connections 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/1887230945 >> v2:172.21.15.86:3300/0 conn(0x7f5ac40a54c0 0x7f5ac40a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/1887230945 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a50e0 0x7f5ac409cd60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- 172.21.15.86:0/1887230945 >> 172.21.15.86:0/1887230945 conn(0x7f5ac40aaae0 msgr2=0x7f5ac40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- 172.21.15.86:0/1887230945 shutdown_connections 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- 172.21.15.86:0/1887230945 wait complete. 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: build_initial_monmap 2024-09-17T15:42:56.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: monmap: 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 Processor -- start 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- start start 2024-09-17T15:42:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: init 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: _reopen_session rank -1 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac40a50e0 0x7f5ac41d0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac40a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a54c0 0x7f5ac4137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac40a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 0x7f5ac413d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac4137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac4142cb0 con 0x7f5ac40a50e0 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac4142b30 con 0x7f5ac40a54c0 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac4142e30 con 0x7f5ac4137ea0 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): _renew_subs 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5acb4f1640 10 monclient(hunting): authenticate will time out at 1909.346627s 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac928d640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac40a50e0 msgr2=0x7f5ac41d0040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac928d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac40a50e0 0x7f5ac41d0040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 0x7f5ac413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 0x7f5ac413d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59034/0 (socket says 172.21.15.86:59034) 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 learned_addr learned my addr 172.21.15.86:0/2070012704 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): get_auth_request con 0x7f5ac4137ea0 auth_method 0 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac8a8c640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a54c0 0x7f5ac4137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac8a8c640 10 monclient(hunting): get_auth_request con 0x7f5ac40a54c0 auth_method 0 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac8a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac8a8c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac8a8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient(hunting): handle_auth_done global_id 164279 payload 824 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a54c0 msgr2=0x7f5ac4137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac40a54c0 0x7f5ac4137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac40a50e0 msgr2=0x7f5ac41d0040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac40a50e0 0x7f5ac41d0040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient: _finish_hunting 0 2024-09-17T15:42:56.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 monclient: found mon.noname-c 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.379+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac413f7e0 con 0x7f5ac4137ea0 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 msgr2=0x7f5ac413d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac9a8e640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 0x7f5ac413d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00cb20 tx=0x7f5abc00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc00ae50 con 0x7f5ac4137ea0 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: dump: 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 5 monclient: authenticate success, global_id 164279 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 monclient: _renew_subs 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac413ec80 con 0x7f5ac4137ea0 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5abc012400 con 0x7f5ac4137ea0 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0127d0 con 0x7f5ac4137ea0 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 10 monclient: dump: 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5abb7fe640 4 set_mon_vals no callback set 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 monclient: _renew_subs 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac41d0ec0 con 0x7f5ac4137ea0 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 0x7f5ac413d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac41215d0 con 0x7f5ac41375b0 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5abc012a30 con 0x7f5ac4137ea0 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 0x7f5ac413d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient: get_auth_request con 0x7f5ac41375b0 auth_method 0 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5aa803bcb0 0x7f5aa803e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(541..541 src has 438..541) ==== 4755+0+0 (secure 0 0 0) 0x7f5abc0506c0 con 0x7f5ac4137ea0 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac8a8c640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5aa803bcb0 0x7f5aa803e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac8a8c640 10 monclient: get_auth_request con 0x7f5aa803bcb0 auth_method 0 2024-09-17T15:42:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 10 monclient(hunting): handle_auth_done global_id 174154 payload 824 2024-09-17T15:42:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.383+0000 7f5ac928d640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5ac409cd60 con 0x7f5ac41375b0 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ac928d640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 msgr2=0x7f5ac413d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ac928d640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 0x7f5ac413d870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00ad000 tx=0x7f5ac001ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ac8a8c640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5aa803bcb0 0x7f5aa803e170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4006fd0 tx=0x7f5ab4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00bb020 con 0x7f5ac41375b0 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 10 monclient: dump: 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ac00b6470 con 0x7f5ac41375b0 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 msgr2=0x7f5ac413d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 0x7f5ac413d870 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00ad000 tx=0x7f5ac001ac30 comp rx=0 tx=0).stop 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 msgr2=0x7f5ac413d870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.387+0000 7f5ab9ffb640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac41375b0 0x7f5ac413d870 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00ad000 tx=0x7f5ac001ac30 comp rx=0 tx=0).stop 2024-09-17T15:42:56.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5acb4f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 0x7f5ac41392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5acb4f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac413e1c0 con 0x7f5ac413ddb0 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 0x7f5ac41392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient: get_auth_request con 0x7f5ac413ddb0 auth_method 0 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 10 monclient(hunting): handle_auth_done global_id 174157 payload 824 2024-09-17T15:42:56.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.395+0000 7f5ac928d640 1 -- 172.21.15.86:0/2070012704 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5ac40a18d0 con 0x7f5ac413ddb0 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ac928d640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 msgr2=0x7f5ac41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ac928d640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 0x7f5ac41392d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5ac001aaa0 tx=0x7f5ac0019190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00bb070 con 0x7f5ac413ddb0 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 10 monclient: dump: 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:42:56.400 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f5ac00c2220 con 0x7f5ac413ddb0 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 msgr2=0x7f5ac41392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 0x7f5ac41392d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5ac001aaa0 tx=0x7f5ac0019190 comp rx=0 tx=0).stop 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 msgr2=0x7f5ac41392d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ab9ffb640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ac413ddb0 0x7f5ac41392d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:42:56.401 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 146, 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 31187, 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:42:56.402 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:42:56.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.404 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:42:56.405 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:42:56.406 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:42:56.408 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:42:56.411 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.412 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:56.413 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.414 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:42:56.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:42:56.416 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:42:56.416 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5aa803bcb0 msgr2=0x7f5aa803e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5aa803bcb0 0x7f5aa803e170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4006fd0 tx=0x7f5ab4008040 comp rx=0 tx=0).stop 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 10 monclient: shutdown 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 msgr2=0x7f5ac413d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:42:56.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 --2- 172.21.15.86:0/2070012704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac4137ea0 0x7f5ac413d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00cb20 tx=0x7f5abc00a060 comp rx=0 tx=0).stop 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 reap_dead start 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5ac9a8e640 1 -- 172.21.15.86:0/2070012704 reap_dead start 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 shutdown_connections 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 >> 172.21.15.86:0/2070012704 conn(0x7f5ac40aaae0 msgr2=0x7f5ac40a8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 shutdown_connections 2024-09-17T15:42:56.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:42:56.399+0000 7f5acb4f1640 1 -- 172.21.15.86:0/2070012704 wait complete. 2024-09-17T15:42:56.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:42:56.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:42:56.877 INFO:tasks.rados.rados.0.smithi203.stdout:2413: done (3 left) 2024-09-17T15:42:56.877 INFO:tasks.rados.rados.0.smithi203.stdout:2414: done (2 left) 2024-09-17T15:42:56.878 INFO:tasks.rados.rados.0.smithi203.stdout:2416: done (1 left) 2024-09-17T15:42:56.878 INFO:tasks.rados.rados.0.smithi203.stdout:2417: done (0 left) 2024-09-17T15:42:56.878 INFO:tasks.rados.rados.0.smithi203.stdout:2418: write oid 44 current snap is 263 2024-09-17T15:42:56.878 INFO:tasks.rados.rados.0.smithi203.stdout:2418: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-17T15:42:56.891 INFO:tasks.rados.rados.0.smithi203.stdout:2418: writing smithi20328086-44 from 769985 to 1339416 tid 1 2024-09-17T15:42:56.902 INFO:tasks.rados.rados.0.smithi203.stdout:2418: writing smithi20328086-44 from 1915869 to 2353835 tid 2 2024-09-17T15:42:56.902 INFO:tasks.rados.rados.0.smithi203.stdout:2419: copy_from oid 3 from oid 2 current snap is 263 2024-09-17T15:42:56.902 INFO:tasks.rados.rados.0.smithi203.stdout:2420: delete oid 45 current snap is 263 2024-09-17T15:42:56.910 INFO:tasks.rados.rados.0.smithi203.stdout:2418: finishing write tid 1 to smithi20328086-44 2024-09-17T15:42:56.910 INFO:tasks.rados.rados.0.smithi203.stdout:2420: done (2 left) 2024-09-17T15:42:56.911 INFO:tasks.rados.rados.0.smithi203.stdout:2421: write (excl) oid 19 current snap is 263 2024-09-17T15:42:56.911 INFO:tasks.rados.rados.0.smithi203.stdout:2421: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-17T15:42:56.924 INFO:tasks.rados.rados.0.smithi203.stdout:2421: writing smithi20328086-19 from 403599 to 1123528 tid 1 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2421: writing smithi20328086-19 from 1916480 to 2358115 tid 2 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2422: read oid 4 snap -1 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2422: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2423: read oid 16 snap -1 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2423: expect deleted 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2424: write oid 11 current snap is 263 2024-09-17T15:42:56.939 INFO:tasks.rados.rados.0.smithi203.stdout:2424: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-17T15:42:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:2424: writing smithi20328086-11 from 420406 to 1215584 tid 1 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2424: writing smithi20328086-11 from 1974962 to 2420406 tid 2 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2425: read oid 24 snap 255 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2425: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2426: delete oid 22 current snap is 263 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2418: finishing write tid 2 to smithi20328086-44 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2418: finishing write tid 3 to smithi20328086-44 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2418: finishing write tid 4 to smithi20328086-44 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2418: oid 44 updating version 0 to 1171 2024-09-17T15:42:56.964 INFO:tasks.rados.rados.0.smithi203.stdout:2418: oid 44 version 1171 is already newer than 1170 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2418: oid 44 version 1171 is already newer than 1169 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1171 (ObjNum 855 snap 263 seq_num 855) dirty exists 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2418: left oid 44 (ObjNum 855 snap 263 seq_num 855) 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2419: finishing copy_from racing read to smithi20328086-3 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2419: finishing copy_from to smithi20328086-3 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1079 (ObjNum 769 snap 240 seq_num 769) dirty exists 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2421: finishing write tid 1 to smithi20328086-19 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2421: finishing write tid 2 to smithi20328086-19 2024-09-17T15:42:56.965 INFO:tasks.rados.rados.0.smithi203.stdout:2422: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:42:57.063 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1743816 2024-09-17T15:42:57.078 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: finishing write tid 3 to smithi20328086-19 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: finishing write tid 4 to smithi20328086-19 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: oid 19 updating version 0 to 1183 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: oid 19 version 1183 is already newer than 1181 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: oid 19 version 1183 is already newer than 1182 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1183 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2421: left oid 19 (ObjNum 857 snap 263 seq_num 857) 2024-09-17T15:42:57.115 INFO:tasks.rados.rados.0.smithi203.stdout:2424: finishing write tid 1 to smithi20328086-11 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: finishing write tid 2 to smithi20328086-11 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: finishing write tid 3 to smithi20328086-11 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: finishing write tid 4 to smithi20328086-11 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: oid 11 updating version 0 to 1172 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: oid 11 updating version 1172 to 1174 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: oid 11 version 1174 is already newer than 1173 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1174 (ObjNum 858 snap 263 seq_num 858) dirty exists 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2424: left oid 11 (ObjNum 858 snap 263 seq_num 858) 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:2425: expect (ObjNum 782 snap 243 seq_num 782) 2024-09-17T15:42:57.116 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:57.172 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:57.230 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2418: done (7 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2419: done (6 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2421: done (5 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2422: done (4 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2423: done (3 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2424: done (2 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2425: done (1 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2426: done (0 left) 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2427: write (excl) oid 37 current snap is 263 2024-09-17T15:42:57.289 INFO:tasks.rados.rados.0.smithi203.stdout:2427: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-17T15:42:57.301 INFO:tasks.rados.rados.0.smithi203.stdout:2427: writing smithi20328086-37 from 454020 to 1116049 tid 1 2024-09-17T15:42:57.315 INFO:tasks.rados.rados.0.smithi203.stdout:2427: writing smithi20328086-37 from 1691926 to 2280888 tid 2 2024-09-17T15:42:57.315 INFO:tasks.rados.rados.0.smithi203.stdout:2428: rollback oid 21 current snap is 263 2024-09-17T15:42:57.316 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 256 2024-09-17T15:42:57.316 INFO:tasks.rados.rados.0.smithi203.stdout:2427: finishing write tid 1 to smithi20328086-37 2024-09-17T15:42:57.316 INFO:tasks.rados.rados.0.smithi203.stdout:2429: delete oid 23 current snap is 263 2024-09-17T15:42:57.321 INFO:tasks.rados.rados.0.smithi203.stdout:2429: done (2 left) 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2430: copy_from oid 5 from oid 7 current snap is 263 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2431: read oid 18 snap -1 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2431: expect (ObjNum 835 snap 256 seq_num 835) 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2432: read oid 34 snap -1 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2432: expect (ObjNum 778 snap 243 seq_num 778) 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2433: read oid 46 snap -1 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2433: expect (ObjNum 807 snap 249 seq_num 807) 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2434: copy_from oid 35 from oid 41 current snap is 263 2024-09-17T15:42:57.322 INFO:tasks.rados.rados.0.smithi203.stdout:2435: snap_remove snap 255 2024-09-17T15:42:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2435: done (7 left) 2024-09-17T15:42:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2436: rollback oid 19 current snap is 263 2024-09-17T15:42:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 258 2024-09-17T15:42:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2430: finishing copy_from to smithi20328086-5 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2430: got expected ENOENT (src dne) 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2430: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: finishing write tid 2 to smithi20328086-37 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: finishing write tid 3 to smithi20328086-37 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: finishing write tid 4 to smithi20328086-37 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: oid 37 updating version 0 to 1186 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: oid 37 version 1186 is already newer than 1185 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: oid 37 updating version 1186 to 1187 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1187 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2427: left oid 37 (ObjNum 860 snap 263 seq_num 860) 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2431: expect (ObjNum 835 snap 256 seq_num 835) 2024-09-17T15:42:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:58.766 INFO:tasks.rados.rados.0.smithi203.stdout:2432: expect (ObjNum 778 snap 243 seq_num 778) 2024-09-17T15:42:58.826 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:42:58.938 INFO:tasks.rados.rados.0.smithi203.stdout:2433: expect (ObjNum 807 snap 249 seq_num 807) 2024-09-17T15:42:58.951 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1699296 2024-09-17T15:42:59.027 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 142494 2024-09-17T15:42:59.074 INFO:tasks.rados.rados.0.smithi203.stdout:2428: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:42:59.074 INFO:tasks.rados.rados.0.smithi203.stdout:2428: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2428: finishing rollback tid 2 to smithi20328086-21 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1250 (ObjNum 578 snap 172 seq_num 578) dirty exists 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2434: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2434: finishing copy_from to smithi20328086-35 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 637 (ObjNum 851 snap 261 seq_num 851) dirty exists 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2436: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2436: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2436: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1190 (ObjNum 763 snap 236 seq_num 763) dirty exists 2024-09-17T15:42:59.075 INFO:tasks.rados.rados.0.smithi203.stdout:2427: done (7 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2428: done (6 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2430: done (5 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2431: done (4 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2432: done (3 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2433: done (2 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2434: done (1 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2436: done (0 left) 2024-09-17T15:42:59.076 INFO:tasks.rados.rados.0.smithi203.stdout:2437: snap_create 2024-09-17T15:42:59.778 INFO:tasks.rados.rados.0.smithi203.stdout:2437: done (0 left) 2024-09-17T15:42:59.778 INFO:tasks.rados.rados.0.smithi203.stdout:2438: snap_remove snap 258 2024-09-17T15:43:00.705 INFO:tasks.rados.rados.0.smithi203.stdout:2438: done (0 left) 2024-09-17T15:43:00.705 INFO:tasks.rados.rados.0.smithi203.stdout:2439: snap_create 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2439: done (0 left) 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2440: copy_from oid 36 from oid 22 current snap is 265 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2441: read oid 33 snap 239 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2441: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2442: read oid 41 snap 262 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2442: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:01.681 INFO:tasks.rados.rados.0.smithi203.stdout:2443: delete oid 7 current snap is 265 2024-09-17T15:43:01.685 INFO:tasks.rados.rados.0.smithi203.stdout:2440: finishing copy_from to smithi20328086-36 2024-09-17T15:43:01.686 INFO:tasks.rados.rados.0.smithi203.stdout:2440: got expected ENOENT (src dne) 2024-09-17T15:43:01.686 INFO:tasks.rados.rados.0.smithi203.stdout:2440: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:43:01.692 INFO:tasks.rados.rados.0.smithi203.stdout:2441: expect (ObjNum 738 snap 231 seq_num 738) 2024-09-17T15:43:01.741 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 937381 2024-09-17T15:43:01.814 INFO:tasks.rados.rados.0.smithi203.stdout:2442: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:01.871 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2440: done (3 left) 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2441: done (2 left) 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2442: done (1 left) 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2443: done (0 left) 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2444: rollback oid 46 current snap is 265 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 244 2024-09-17T15:43:01.907 INFO:tasks.rados.rados.0.smithi203.stdout:2445: snap_remove snap 262 2024-09-17T15:43:02.817 INFO:tasks.rados.rados.0.smithi203.stdout:2445: done (1 left) 2024-09-17T15:43:02.817 INFO:tasks.rados.rados.0.smithi203.stdout:2446: write (excl) oid 29 current snap is 265 2024-09-17T15:43:02.817 INFO:tasks.rados.rados.0.smithi203.stdout:2446: seq_num 863 ranges {504441=487776,1467372=736289} 2024-09-17T15:43:02.833 INFO:tasks.rados.rados.0.smithi203.stdout:2446: writing smithi20328086-29 from 504441 to 992217 tid 1 2024-09-17T15:43:02.854 INFO:tasks.rados.rados.0.smithi203.stdout:2446: writing smithi20328086-29 from 1467372 to 2203661 tid 2 2024-09-17T15:43:02.854 INFO:tasks.rados.rados.0.smithi203.stdout:2447: write oid 14 current snap is 265 2024-09-17T15:43:02.854 INFO:tasks.rados.rados.0.smithi203.stdout:2447: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-17T15:43:02.869 INFO:tasks.rados.rados.0.smithi203.stdout:2447: writing smithi20328086-14 from 521248 to 1084273 tid 1 2024-09-17T15:43:02.884 INFO:tasks.rados.rados.0.smithi203.stdout:2447: writing smithi20328086-14 from 1809501 to 2405801 tid 2 2024-09-17T15:43:02.884 INFO:tasks.rados.rados.0.smithi203.stdout:2448: copy_from oid 27 from oid 37 current snap is 265 2024-09-17T15:43:02.884 INFO:tasks.rados.rados.0.smithi203.stdout:2444: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2444: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2444: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1114 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: finishing write tid 1 to smithi20328086-29 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: finishing write tid 2 to smithi20328086-29 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: finishing write tid 3 to smithi20328086-29 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: finishing write tid 4 to smithi20328086-29 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: oid 29 updating version 0 to 1115 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: oid 29 version 1115 is already newer than 1113 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:2446: oid 29 version 1115 is already newer than 1114 2024-09-17T15:43:02.885 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1115 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-17T15:43:02.886 INFO:tasks.rados.rados.0.smithi203.stdout:2446: left oid 29 (ObjNum 863 snap 265 seq_num 863) 2024-09-17T15:43:02.886 INFO:tasks.rados.rados.0.smithi203.stdout:2447: finishing write tid 1 to smithi20328086-14 2024-09-17T15:43:02.886 INFO:tasks.rados.rados.0.smithi203.stdout:2444: done (3 left) 2024-09-17T15:43:02.886 INFO:tasks.rados.rados.0.smithi203.stdout:2446: done (2 left) 2024-09-17T15:43:02.886 INFO:tasks.rados.rados.0.smithi203.stdout:2449: delete oid 43 current snap is 265 2024-09-17T15:43:02.892 INFO:tasks.rados.rados.0.smithi203.stdout:2449: done (2 left) 2024-09-17T15:43:02.892 INFO:tasks.rados.rados.0.smithi203.stdout:2450: copy_from oid 30 from oid 36 current snap is 265 2024-09-17T15:43:02.892 INFO:tasks.rados.rados.0.smithi203.stdout:2451: copy_from oid 42 from oid 17 current snap is 265 2024-09-17T15:43:02.892 INFO:tasks.rados.rados.0.smithi203.stdout:2452: snap_create 2024-09-17T15:43:02.893 INFO:tasks.rados.rados.0.smithi203.stdout:2447: finishing write tid 2 to smithi20328086-14 2024-09-17T15:43:02.893 INFO:tasks.rados.rados.0.smithi203.stdout:2447: finishing write tid 3 to smithi20328086-14 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:2447: finishing write tid 4 to smithi20328086-14 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:2447: oid 14 updating version 0 to 1119 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:2447: oid 14 version 1119 is already newer than 1117 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:2447: oid 14 version 1119 is already newer than 1118 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1119 (ObjNum 864 snap 265 seq_num 864) dirty exists 2024-09-17T15:43:02.894 INFO:tasks.rados.rados.0.smithi203.stdout:2447: left oid 14 (ObjNum 864 snap 265 seq_num 864) 2024-09-17T15:43:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:2450: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:43:02.923 INFO:tasks.rados.rados.0.smithi203.stdout:2450: finishing copy_from to smithi20328086-30 2024-09-17T15:43:02.923 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1244 (ObjNum 700 snap 220 seq_num 700) dirty exists 2024-09-17T15:43:02.928 INFO:tasks.rados.rados.0.smithi203.stdout:2451: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:43:02.929 INFO:tasks.rados.rados.0.smithi203.stdout:2448: finishing copy_from racing read to smithi20328086-27 2024-09-17T15:43:02.929 INFO:tasks.rados.rados.0.smithi203.stdout:2448: finishing copy_from to smithi20328086-27 2024-09-17T15:43:02.930 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1187 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-17T15:43:02.937 INFO:tasks.rados.rados.0.smithi203.stdout:2451: finishing copy_from to smithi20328086-42 2024-09-17T15:43:02.937 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1167 (ObjNum 818 snap 254 seq_num 818) dirty exists 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2447: done (4 left) 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2448: done (3 left) 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2450: done (2 left) 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2451: done (1 left) 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2452: done (0 left) 2024-09-17T15:43:03.836 INFO:tasks.rados.rados.0.smithi203.stdout:2453: snap_create 2024-09-17T15:43:04.846 INFO:tasks.rados.rados.0.smithi203.stdout:2453: done (0 left) 2024-09-17T15:43:04.846 INFO:tasks.rados.rados.0.smithi203.stdout:2454: snap_create 2024-09-17T15:43:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:2454: done (0 left) 2024-09-17T15:43:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:2455: write oid 36 current snap is 268 2024-09-17T15:43:05.228 INFO:tasks.rados.rados.0.smithi203.stdout:2455: seq_num 866 ranges {554862=713523,1926465=599969} 2024-09-17T15:43:05.248 INFO:tasks.rados.rados.0.smithi203.stdout:2455: writing smithi20328086-36 from 554862 to 1268385 tid 1 2024-09-17T15:43:05.266 INFO:tasks.rados.rados.0.smithi203.stdout:2455: writing smithi20328086-36 from 1926465 to 2526434 tid 2 2024-09-17T15:43:05.267 INFO:tasks.rados.rados.0.smithi203.stdout:2456: read oid 19 snap -1 2024-09-17T15:43:05.267 INFO:tasks.rados.rados.0.smithi203.stdout:2456: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:43:05.267 INFO:tasks.rados.rados.0.smithi203.stdout:2457: copy_from oid 5 from oid 4 current snap is 268 2024-09-17T15:43:05.267 INFO:tasks.rados.rados.0.smithi203.stdout:2458: delete oid 21 current snap is 268 2024-09-17T15:43:05.275 INFO:tasks.rados.rados.0.smithi203.stdout:2456: expect (ObjNum 763 snap 236 seq_num 763) 2024-09-17T15:43:05.275 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:05.383 INFO:tasks.rados.rados.0.smithi203.stdout:2455: finishing write tid 1 to smithi20328086-36 2024-09-17T15:43:05.383 INFO:tasks.rados.rados.0.smithi203.stdout:2455: finishing write tid 2 to smithi20328086-36 2024-09-17T15:43:05.383 INFO:tasks.rados.rados.0.smithi203.stdout:2457: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:43:05.383 INFO:tasks.rados.rados.0.smithi203.stdout:2455: finishing write tid 3 to smithi20328086-36 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: finishing write tid 4 to smithi20328086-36 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: oid 36 updating version 0 to 1268 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: oid 36 version 1268 is already newer than 1267 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: oid 36 version 1268 is already newer than 1266 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1268 (ObjNum 866 snap 268 seq_num 866) dirty exists 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: left oid 36 (ObjNum 866 snap 268 seq_num 866) 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2457: finishing copy_from to smithi20328086-5 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1078 (ObjNum 853 snap 262 seq_num 853) dirty exists 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2455: done (3 left) 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2456: done (2 left) 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2457: done (1 left) 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2458: done (0 left) 2024-09-17T15:43:05.384 INFO:tasks.rados.rados.0.smithi203.stdout:2459: snap_create 2024-09-17T15:43:06.237 INFO:tasks.rados.rados.0.smithi203.stdout:2459: done (0 left) 2024-09-17T15:43:06.238 INFO:tasks.rados.rados.0.smithi203.stdout:2460: read oid 41 snap -1 2024-09-17T15:43:06.238 INFO:tasks.rados.rados.0.smithi203.stdout:2460: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:06.238 INFO:tasks.rados.rados.0.smithi203.stdout:2461: write (excl) oid 20 current snap is 269 2024-09-17T15:43:06.238 INFO:tasks.rados.rados.0.smithi203.stdout:2461: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-17T15:43:06.256 INFO:tasks.rados.rados.0.smithi203.stdout:2461: writing smithi20328086-20 from 588476 to 1168850 tid 1 2024-09-17T15:43:06.275 INFO:tasks.rados.rados.0.smithi203.stdout:2461: writing smithi20328086-20 from 1643429 to 2247067 tid 2 2024-09-17T15:43:06.275 INFO:tasks.rados.rados.0.smithi203.stdout:2462: snap_remove snap 263 2024-09-17T15:43:07.241 INFO:tasks.rados.rados.0.smithi203.stdout:2462: done (2 left) 2024-09-17T15:43:07.241 INFO:tasks.rados.rados.0.smithi203.stdout:2463: delete oid 49 current snap is 269 2024-09-17T15:43:07.241 INFO:tasks.rados.rados.0.smithi203.stdout:2460: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:07.252 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1495639 2024-09-17T15:43:07.316 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2260120 2024-09-17T15:43:07.330 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1060993 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: finishing write tid 1 to smithi20328086-20 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: finishing write tid 2 to smithi20328086-20 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2460: done (2 left) 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2463: done (1 left) 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2464: write oid 17 current snap is 269 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: finishing write tid 3 to smithi20328086-20 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: finishing write tid 4 to smithi20328086-20 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: oid 20 updating version 0 to 1122 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: oid 20 updating version 1122 to 1123 2024-09-17T15:43:07.352 INFO:tasks.rados.rados.0.smithi203.stdout:2461: oid 20 version 1123 is already newer than 1121 2024-09-17T15:43:07.353 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1123 (ObjNum 868 snap 269 seq_num 868) dirty exists 2024-09-17T15:43:07.353 INFO:tasks.rados.rados.0.smithi203.stdout:2461: left oid 20 (ObjNum 868 snap 269 seq_num 868) 2024-09-17T15:43:07.353 INFO:tasks.rados.rados.0.smithi203.stdout:2464: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-17T15:43:07.359 INFO:tasks.rados.rados.0.smithi203.stdout:2464: writing smithi20328086-17 from 622090 to 1352962 tid 1 2024-09-17T15:43:07.365 INFO:tasks.rados.rados.0.smithi203.stdout:2464: writing smithi20328086-17 from 1760393 to 2367700 tid 2 2024-09-17T15:43:07.365 INFO:tasks.rados.rados.0.smithi203.stdout:2461: done (1 left) 2024-09-17T15:43:07.365 INFO:tasks.rados.rados.0.smithi203.stdout:2465: read oid 37 snap 261 2024-09-17T15:43:07.365 INFO:tasks.rados.rados.0.smithi203.stdout:2465: expect (ObjNum 830 snap 254 seq_num 830) 2024-09-17T15:43:07.366 INFO:tasks.rados.rados.0.smithi203.stdout:2466: write (excl) oid 15 current snap is 269 2024-09-17T15:43:07.366 INFO:tasks.rados.rados.0.smithi203.stdout:2466: seq_num 871 ranges {638897=406121,1818875=750965} 2024-09-17T15:43:07.376 INFO:tasks.rados.rados.0.smithi203.stdout:2466: writing smithi20328086-15 from 638897 to 1045018 tid 1 2024-09-17T15:43:07.393 INFO:tasks.rados.rados.0.smithi203.stdout:2466: writing smithi20328086-15 from 1818875 to 2569840 tid 2 2024-09-17T15:43:07.394 INFO:tasks.rados.rados.0.smithi203.stdout:2467: write (excl) oid 43 current snap is 269 2024-09-17T15:43:07.394 INFO:tasks.rados.rados.0.smithi203.stdout:2467: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-17T15:43:07.404 INFO:tasks.rados.rados.0.smithi203.stdout:2467: writing smithi20328086-43 from 655704 to 1137074 tid 1 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2467: writing smithi20328086-43 from 1761004 to 2255627 tid 2 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2468: read oid 42 snap -1 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2468: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2469: copy_from oid 19 from oid 39 current snap is 269 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2464: finishing write tid 1 to smithi20328086-17 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2464: finishing write tid 2 to smithi20328086-17 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2464: finishing write tid 3 to smithi20328086-17 2024-09-17T15:43:07.414 INFO:tasks.rados.rados.0.smithi203.stdout:2464: finishing write tid 4 to smithi20328086-17 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2464: oid 17 updating version 0 to 1177 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2464: oid 17 version 1177 is already newer than 1176 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2464: oid 17 version 1177 is already newer than 1175 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1177 (ObjNum 870 snap 269 seq_num 870) dirty exists 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2464: left oid 17 (ObjNum 870 snap 269 seq_num 870) 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2464: done (5 left) 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2470: read oid 35 snap -1 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2470: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2471: write (excl) oid 27 current snap is 269 2024-09-17T15:43:07.415 INFO:tasks.rados.rados.0.smithi203.stdout:2471: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-17T15:43:07.424 INFO:tasks.rados.rados.0.smithi203.stdout:2471: writing smithi20328086-27 from 672511 to 1229130 tid 1 2024-09-17T15:43:07.436 INFO:tasks.rados.rados.0.smithi203.stdout:2471: writing smithi20328086-27 from 1819486 to 2574120 tid 2 2024-09-17T15:43:07.436 INFO:tasks.rados.rados.0.smithi203.stdout:2472: copy_from oid 22 from oid 7 current snap is 269 2024-09-17T15:43:07.436 INFO:tasks.rados.rados.0.smithi203.stdout:2465: expect (ObjNum 830 snap 254 seq_num 830) 2024-09-17T15:43:07.510 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3792098 2024-09-17T15:43:07.588 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:07.670 INFO:tasks.rados.rados.0.smithi203.stdout:2466: finishing write tid 1 to smithi20328086-15 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2465: done (7 left) 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2473: copy_from oid 31 from oid 12 current snap is 269 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: finishing write tid 2 to smithi20328086-15 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: finishing write tid 3 to smithi20328086-15 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: finishing write tid 4 to smithi20328086-15 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: oid 15 updating version 0 to 1247 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: oid 15 version 1247 is already newer than 1246 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: oid 15 version 1247 is already newer than 1245 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1247 (ObjNum 871 snap 269 seq_num 871) dirty exists 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2466: left oid 15 (ObjNum 871 snap 269 seq_num 871) 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2467: finishing write tid 1 to smithi20328086-43 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2467: finishing write tid 2 to smithi20328086-43 2024-09-17T15:43:07.671 INFO:tasks.rados.rados.0.smithi203.stdout:2467: finishing write tid 3 to smithi20328086-43 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2467: finishing write tid 4 to smithi20328086-43 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2467: oid 43 updating version 0 to 640 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2467: oid 43 updating version 640 to 641 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2467: oid 43 version 641 is already newer than 639 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 641 (ObjNum 872 snap 269 seq_num 872) dirty exists 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2467: left oid 43 (ObjNum 872 snap 269 seq_num 872) 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:2470: expect (ObjNum 851 snap 261 seq_num 851) 2024-09-17T15:43:07.672 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:07.748 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1324908 2024-09-17T15:43:07.770 INFO:tasks.rados.rados.0.smithi203.stdout:2468: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:07.943 INFO:tasks.rados.rados.0.smithi203.stdout:2472: finishing copy_from to smithi20328086-22 2024-09-17T15:43:07.943 INFO:tasks.rados.rados.0.smithi203.stdout:2472: got expected ENOENT (src dne) 2024-09-17T15:43:07.943 INFO:tasks.rados.rados.0.smithi203.stdout:2472: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:43:07.943 INFO:tasks.rados.rados.0.smithi203.stdout:2469: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:43:07.943 INFO:tasks.rados.rados.0.smithi203.stdout:2471: finishing write tid 1 to smithi20328086-27 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2471: finishing write tid 2 to smithi20328086-27 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2466: done (7 left) 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2467: done (6 left) 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2468: done (5 left) 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2470: done (4 left) 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2472: done (3 left) 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2474: write (excl) oid 14 current snap is 269 2024-09-17T15:43:07.944 INFO:tasks.rados.rados.0.smithi203.stdout:2474: seq_num 874 ranges {689318=631868,1877968=498292} 2024-09-17T15:43:07.950 INFO:tasks.rados.rados.0.smithi203.stdout:2474: writing smithi20328086-14 from 689318 to 1321186 tid 1 2024-09-17T15:43:07.956 INFO:tasks.rados.rados.0.smithi203.stdout:2474: writing smithi20328086-14 from 1877968 to 2376260 tid 2 2024-09-17T15:43:07.956 INFO:tasks.rados.rados.0.smithi203.stdout:2475: snap_remove snap 239 2024-09-17T15:43:08.246 INFO:tasks.rados.rados.0.smithi203.stdout:2475: done (4 left) 2024-09-17T15:43:08.246 INFO:tasks.rados.rados.0.smithi203.stdout:2476: rollback oid 17 current snap is 269 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 264 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: finishing write tid 3 to smithi20328086-27 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: finishing write tid 4 to smithi20328086-27 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: oid 27 updating version 0 to 1188 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: oid 27 updating version 1188 to 1189 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: oid 27 updating version 1189 to 1190 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1190 (ObjNum 873 snap 269 seq_num 873) dirty exists 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2471: left oid 27 (ObjNum 873 snap 269 seq_num 873) 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2469: finishing copy_from to smithi20328086-19 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1144 (ObjNum 829 snap 254 seq_num 829) dirty exists 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2473: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2473: finishing copy_from to smithi20328086-31 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1130 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:43:08.247 INFO:tasks.rados.rados.0.smithi203.stdout:2474: finishing write tid 1 to smithi20328086-14 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: finishing write tid 2 to smithi20328086-14 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: finishing write tid 3 to smithi20328086-14 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: finishing write tid 4 to smithi20328086-14 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: oid 14 updating version 0 to 1130 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: oid 14 updating version 1130 to 1131 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: oid 14 version 1131 is already newer than 1129 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1131 (ObjNum 874 snap 269 seq_num 874) dirty exists 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: left oid 14 (ObjNum 874 snap 269 seq_num 874) 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2469: done (4 left) 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2471: done (3 left) 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2473: done (2 left) 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2474: done (1 left) 2024-09-17T15:43:08.248 INFO:tasks.rados.rados.0.smithi203.stdout:2477: read oid 30 snap -1 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:2477: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:2478: rollback oid 43 current snap is 269 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 244 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:2479: read oid 2 snap 254 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:2479: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:43:08.249 INFO:tasks.rados.rados.0.smithi203.stdout:2480: delete oid 7 current snap is 269 2024-09-17T15:43:08.260 INFO:tasks.rados.rados.0.smithi203.stdout:2480: done (4 left) 2024-09-17T15:43:08.260 INFO:tasks.rados.rados.0.smithi203.stdout:2481: delete oid 4 current snap is 269 2024-09-17T15:43:08.260 INFO:tasks.rados.rados.0.smithi203.stdout:2479: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:43:08.318 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 286701 2024-09-17T15:43:08.417 INFO:tasks.rados.rados.0.smithi203.stdout:2479: done (4 left) 2024-09-17T15:43:08.417 INFO:tasks.rados.rados.0.smithi203.stdout:2481: done (3 left) 2024-09-17T15:43:08.417 INFO:tasks.rados.rados.0.smithi203.stdout:2482: delete oid 28 current snap is 269 2024-09-17T15:43:08.417 INFO:tasks.rados.rados.0.smithi203.stdout:2477: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:08.591 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2476: finishing rollback tid 0 to smithi20328086-17 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2478: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2478: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2478: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 644 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2476: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2476: finishing rollback tid 2 to smithi20328086-17 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1180 (ObjNum 818 snap 254 seq_num 818) dirty exists 2024-09-17T15:43:08.674 INFO:tasks.rados.rados.0.smithi203.stdout:2476: done (3 left) 2024-09-17T15:43:08.675 INFO:tasks.rados.rados.0.smithi203.stdout:2477: done (2 left) 2024-09-17T15:43:08.675 INFO:tasks.rados.rados.0.smithi203.stdout:2478: done (1 left) 2024-09-17T15:43:08.675 INFO:tasks.rados.rados.0.smithi203.stdout:2482: done (0 left) 2024-09-17T15:43:08.675 INFO:tasks.rados.rados.0.smithi203.stdout:2483: delete oid 41 current snap is 269 2024-09-17T15:43:08.681 INFO:tasks.rados.rados.0.smithi203.stdout:2483: done (0 left) 2024-09-17T15:43:08.681 INFO:tasks.rados.rados.0.smithi203.stdout:2484: write (excl) oid 12 current snap is 269 2024-09-17T15:43:08.682 INFO:tasks.rados.rados.0.smithi203.stdout:2484: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-17T15:43:08.689 INFO:tasks.rados.rados.0.smithi203.stdout:2484: writing smithi20328086-12 from 773353 to 1497819 tid 1 2024-09-17T15:43:08.696 INFO:tasks.rados.rados.0.smithi203.stdout:2484: writing smithi20328086-12 from 2170378 to 2773353 tid 2 2024-09-17T15:43:08.696 INFO:tasks.rados.rados.0.smithi203.stdout:2485: snap_remove snap 256 2024-09-17T15:43:09.260 INFO:tasks.rados.rados.0.smithi203.stdout:2484: finishing write tid 1 to smithi20328086-12 2024-09-17T15:43:09.260 INFO:tasks.rados.rados.0.smithi203.stdout:2484: finishing write tid 2 to smithi20328086-12 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: finishing write tid 3 to smithi20328086-12 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: finishing write tid 4 to smithi20328086-12 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: oid 12 updating version 0 to 1250 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: oid 12 version 1250 is already newer than 1248 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: oid 12 version 1250 is already newer than 1249 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1250 (ObjNum 879 snap 269 seq_num 879) dirty exists 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: left oid 12 (ObjNum 879 snap 269 seq_num 879) 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2484: done (1 left) 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2485: done (0 left) 2024-09-17T15:43:09.261 INFO:tasks.rados.rados.0.smithi203.stdout:2486: delete oid 12 current snap is 269 2024-09-17T15:43:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:2486: done (0 left) 2024-09-17T15:43:09.274 INFO:tasks.rados.rados.0.smithi203.stdout:2487: snap_create 2024-09-17T15:43:10.230 INFO:tasks.rados.rados.0.smithi203.stdout:2487: done (0 left) 2024-09-17T15:43:10.230 INFO:tasks.rados.rados.0.smithi203.stdout:2488: write oid 7 current snap is 270 2024-09-17T15:43:10.230 INFO:tasks.rados.rados.0.smithi203.stdout:2488: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-17T15:43:10.243 INFO:tasks.rados.rados.0.smithi203.stdout:2488: writing smithi20328086-7 from 406967 to 881931 tid 1 2024-09-17T15:43:10.264 INFO:tasks.rados.rados.0.smithi203.stdout:2488: writing smithi20328086-7 from 1370989 to 2140299 tid 2 2024-09-17T15:43:10.273 INFO:tasks.rados.rados.0.smithi203.stdout:2488: writing smithi20328086-7 from 2677186 to 2806967 tid 3 2024-09-17T15:43:10.273 INFO:tasks.rados.rados.0.smithi203.stdout:2489: write (excl) oid 28 current snap is 270 2024-09-17T15:43:10.273 INFO:tasks.rados.rados.0.smithi203.stdout:2489: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-17T15:43:10.287 INFO:tasks.rados.rados.0.smithi203.stdout:2489: writing smithi20328086-28 from 423774 to 1090340 tid 1 2024-09-17T15:43:10.297 INFO:tasks.rados.rados.0.smithi203.stdout:2489: writing smithi20328086-28 from 1545824 to 2058792 tid 2 2024-09-17T15:43:10.304 INFO:tasks.rados.rados.0.smithi203.stdout:2489: writing smithi20328086-28 from 2704609 to 2823774 tid 3 2024-09-17T15:43:10.304 INFO:tasks.rados.rados.0.smithi203.stdout:2490: snap_remove snap 267 2024-09-17T15:43:11.270 INFO:tasks.rados.rados.0.smithi203.stdout:2490: done (2 left) 2024-09-17T15:43:11.270 INFO:tasks.rados.rados.0.smithi203.stdout:2491: snap_create 2024-09-17T15:43:11.270 INFO:tasks.rados.rados.0.smithi203.stdout:2488: finishing write tid 1 to smithi20328086-7 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: finishing write tid 2 to smithi20328086-7 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: finishing write tid 3 to smithi20328086-7 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: finishing write tid 4 to smithi20328086-7 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: finishing write tid 5 to smithi20328086-7 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: oid 7 updating version 0 to 1090 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: oid 7 updating version 1090 to 1092 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: oid 7 updating version 1092 to 1093 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: oid 7 version 1093 is already newer than 1091 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1093 (ObjNum 881 snap 270 seq_num 881) dirty exists 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2488: left oid 7 (ObjNum 881 snap 270 seq_num 881) 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2489: finishing write tid 1 to smithi20328086-28 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2489: finishing write tid 2 to smithi20328086-28 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2489: finishing write tid 3 to smithi20328086-28 2024-09-17T15:43:11.271 INFO:tasks.rados.rados.0.smithi203.stdout:2489: finishing write tid 4 to smithi20328086-28 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: finishing write tid 5 to smithi20328086-28 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: oid 28 updating version 0 to 1295 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: oid 28 updating version 1295 to 1296 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: oid 28 updating version 1296 to 1297 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: oid 28 updating version 1297 to 1298 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1298 (ObjNum 882 snap 270 seq_num 882) dirty exists 2024-09-17T15:43:11.272 INFO:tasks.rados.rados.0.smithi203.stdout:2489: left oid 28 (ObjNum 882 snap 270 seq_num 882) 2024-09-17T15:43:12.280 INFO:tasks.rados.rados.0.smithi203.stdout:2488: done (2 left) 2024-09-17T15:43:12.280 INFO:tasks.rados.rados.0.smithi203.stdout:2489: done (1 left) 2024-09-17T15:43:12.280 INFO:tasks.rados.rados.0.smithi203.stdout:2491: done (0 left) 2024-09-17T15:43:12.281 INFO:tasks.rados.rados.0.smithi203.stdout:2492: write oid 26 current snap is 271 2024-09-17T15:43:12.281 INFO:tasks.rados.rados.0.smithi203.stdout:2492: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-17T15:43:12.295 INFO:tasks.rados.rados.0.smithi203.stdout:2492: writing smithi20328086-26 from 440581 to 1182396 tid 1 2024-09-17T15:43:12.308 INFO:tasks.rados.rados.0.smithi203.stdout:2492: writing smithi20328086-26 from 1604306 to 2260932 tid 2 2024-09-17T15:43:12.315 INFO:tasks.rados.rados.0.smithi203.stdout:2492: writing smithi20328086-26 from 2732032 to 2840581 tid 3 2024-09-17T15:43:12.315 INFO:tasks.rados.rados.0.smithi203.stdout:2493: read oid 22 snap -1 2024-09-17T15:43:12.316 INFO:tasks.rados.rados.0.smithi203.stdout:2493: expect deleted 2024-09-17T15:43:12.316 INFO:tasks.rados.rados.0.smithi203.stdout:2494: delete oid 14 current snap is 271 2024-09-17T15:43:12.316 INFO:tasks.rados.rados.0.smithi203.stdout:2492: finishing write tid 1 to smithi20328086-26 2024-09-17T15:43:12.316 INFO:tasks.rados.rados.0.smithi203.stdout:2492: finishing write tid 2 to smithi20328086-26 2024-09-17T15:43:12.323 INFO:tasks.rados.rados.0.smithi203.stdout:2492: finishing write tid 3 to smithi20328086-26 2024-09-17T15:43:12.324 INFO:tasks.rados.rados.0.smithi203.stdout:2493: done (2 left) 2024-09-17T15:43:12.324 INFO:tasks.rados.rados.0.smithi203.stdout:2494: done (1 left) 2024-09-17T15:43:12.324 INFO:tasks.rados.rados.0.smithi203.stdout:2495: delete oid 11 current snap is 271 2024-09-17T15:43:12.325 INFO:tasks.rados.rados.0.smithi203.stdout:2492: finishing write tid 4 to smithi20328086-26 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: finishing write tid 5 to smithi20328086-26 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: oid 26 updating version 0 to 1308 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: oid 26 updating version 1308 to 1310 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: oid 26 version 1310 is already newer than 1309 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: oid 26 updating version 1310 to 1311 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1311 (ObjNum 883 snap 271 seq_num 883) dirty exists 2024-09-17T15:43:12.326 INFO:tasks.rados.rados.0.smithi203.stdout:2492: left oid 26 (ObjNum 883 snap 271 seq_num 883) 2024-09-17T15:43:12.333 INFO:tasks.rados.rados.0.smithi203.stdout:2492: done (1 left) 2024-09-17T15:43:12.334 INFO:tasks.rados.rados.0.smithi203.stdout:2495: done (0 left) 2024-09-17T15:43:12.334 INFO:tasks.rados.rados.0.smithi203.stdout:2496: write oid 26 current snap is 271 2024-09-17T15:43:12.334 INFO:tasks.rados.rados.0.smithi203.stdout:2496: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-17T15:43:12.348 INFO:tasks.rados.rados.0.smithi203.stdout:2496: writing smithi20328086-26 from 491002 to 1058564 tid 1 2024-09-17T15:43:12.364 INFO:tasks.rados.rados.0.smithi203.stdout:2496: writing smithi20328086-26 from 1663399 to 2183705 tid 2 2024-09-17T15:43:12.372 INFO:tasks.rados.rados.0.smithi203.stdout:2496: writing smithi20328086-26 from 2814301 to 2891002 tid 3 2024-09-17T15:43:12.372 INFO:tasks.rados.rados.0.smithi203.stdout:2497: write (excl) oid 1 current snap is 271 2024-09-17T15:43:12.372 INFO:tasks.rados.rados.0.smithi203.stdout:2496: finishing write tid 1 to smithi20328086-26 2024-09-17T15:43:12.372 INFO:tasks.rados.rados.0.smithi203.stdout:2496: finishing write tid 2 to smithi20328086-26 2024-09-17T15:43:12.372 INFO:tasks.rados.rados.0.smithi203.stdout:2497: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-17T15:43:12.383 INFO:tasks.rados.rados.0.smithi203.stdout:2497: writing smithi20328086-1 from 507809 to 1150620 tid 1 2024-09-17T15:43:12.397 INFO:tasks.rados.rados.0.smithi203.stdout:2497: writing smithi20328086-1 from 1721881 to 2385845 tid 2 2024-09-17T15:43:12.397 INFO:tasks.rados.rados.0.smithi203.stdout:2498: rollback oid 32 current snap is 271 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 260 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: finishing write tid 3 to smithi20328086-26 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: finishing write tid 4 to smithi20328086-26 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: finishing write tid 5 to smithi20328086-26 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: oid 26 updating version 0 to 1314 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: oid 26 version 1314 is already newer than 1313 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: oid 26 version 1314 is already newer than 1312 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: oid 26 updating version 1314 to 1315 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1315 (ObjNum 886 snap 271 seq_num 886) dirty exists 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: left oid 26 (ObjNum 886 snap 271 seq_num 886) 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2497: finishing write tid 1 to smithi20328086-1 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2496: done (2 left) 2024-09-17T15:43:12.398 INFO:tasks.rados.rados.0.smithi203.stdout:2499: read oid 5 snap -1 2024-09-17T15:43:12.399 INFO:tasks.rados.rados.0.smithi203.stdout:2499: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:43:12.399 INFO:tasks.rados.rados.0.smithi203.stdout:2500: snap_remove snap 269 2024-09-17T15:43:13.285 INFO:tasks.rados.rados.0.smithi203.stdout:2500: done (3 left) 2024-09-17T15:43:13.285 INFO:tasks.rados.rados.0.smithi203.stdout:2501: write (excl) oid 36 current snap is 271 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: finishing write tid 2 to smithi20328086-1 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: finishing write tid 3 to smithi20328086-1 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: finishing write tid 4 to smithi20328086-1 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: oid 1 updating version 0 to 1148 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: oid 1 version 1148 is already newer than 1147 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: oid 1 version 1148 is already newer than 1146 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1148 (ObjNum 887 snap 271 seq_num 887) dirty exists 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2497: left oid 1 (ObjNum 887 snap 271 seq_num 887) 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2498: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:43:13.286 INFO:tasks.rados.rados.0.smithi203.stdout:2501: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-17T15:43:13.298 INFO:tasks.rados.rados.0.smithi203.stdout:2501: writing smithi20328086-36 from 524616 to 1242676 tid 1 2024-09-17T15:43:13.308 INFO:tasks.rados.rados.0.smithi203.stdout:2501: writing smithi20328086-36 from 1664010 to 2187985 tid 2 2024-09-17T15:43:13.316 INFO:tasks.rados.rados.0.smithi203.stdout:2501: writing smithi20328086-36 from 2752794 to 2924616 tid 3 2024-09-17T15:43:13.316 INFO:tasks.rados.rados.0.smithi203.stdout:2497: done (3 left) 2024-09-17T15:43:13.316 INFO:tasks.rados.rados.0.smithi203.stdout:2502: write oid 24 current snap is 271 2024-09-17T15:43:13.316 INFO:tasks.rados.rados.0.smithi203.stdout:2502: seq_num 889 ranges {541423=793309,2122492=667633} 2024-09-17T15:43:13.330 INFO:tasks.rados.rados.0.smithi203.stdout:2502: writing smithi20328086-24 from 541423 to 1334732 tid 1 2024-09-17T15:43:13.343 INFO:tasks.rados.rados.0.smithi203.stdout:2502: writing smithi20328086-24 from 2122492 to 2790125 tid 2 2024-09-17T15:43:13.344 INFO:tasks.rados.rados.0.smithi203.stdout:2503: write oid 34 current snap is 271 2024-09-17T15:43:13.344 INFO:tasks.rados.rados.0.smithi203.stdout:2503: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-17T15:43:13.353 INFO:tasks.rados.rados.0.smithi203.stdout:2503: writing smithi20328086-34 from 558230 to 1143141 tid 1 2024-09-17T15:43:13.362 INFO:tasks.rados.rados.0.smithi203.stdout:2503: writing smithi20328086-34 from 1897327 to 2424971 tid 2 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:2503: writing smithi20328086-34 from 2923993 to 2958230 tid 3 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:2504: copy_from oid 7 from oid 40 current snap is 271 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:2498: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:2498: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1173 (ObjNum 786 snap 245 seq_num 786) dirty exists 2024-09-17T15:43:13.366 INFO:tasks.rados.rados.0.smithi203.stdout:2499: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:43:13.367 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 25567 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: finishing write tid 1 to smithi20328086-36 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: finishing write tid 2 to smithi20328086-36 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: finishing write tid 3 to smithi20328086-36 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: finishing write tid 4 to smithi20328086-36 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: finishing write tid 5 to smithi20328086-36 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: oid 36 updating version 0 to 1319 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: oid 36 updating version 1319 to 1322 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: oid 36 version 1322 is already newer than 1321 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:2501: oid 36 version 1322 is already newer than 1320 2024-09-17T15:43:13.526 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1322 (ObjNum 888 snap 271 seq_num 888) dirty exists 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2501: left oid 36 (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: finishing write tid 1 to smithi20328086-24 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: finishing write tid 2 to smithi20328086-24 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: finishing write tid 3 to smithi20328086-24 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: finishing write tid 4 to smithi20328086-24 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: oid 24 updating version 0 to 1118 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: oid 24 version 1118 is already newer than 1116 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: oid 24 version 1118 is already newer than 1117 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1118 (ObjNum 889 snap 271 seq_num 889) dirty exists 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2502: left oid 24 (ObjNum 889 snap 271 seq_num 889) 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2503: finishing write tid 1 to smithi20328086-34 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2498: done (5 left) 2024-09-17T15:43:13.527 INFO:tasks.rados.rados.0.smithi203.stdout:2499: done (4 left) 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2501: done (3 left) 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2502: done (2 left) 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2505: read oid 24 snap -1 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2505: expect (ObjNum 889 snap 271 seq_num 889) 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2506: read oid 39 snap -1 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2506: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2507: write oid 18 current snap is 271 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: finishing write tid 2 to smithi20328086-34 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: finishing write tid 3 to smithi20328086-34 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: finishing write tid 4 to smithi20328086-34 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: finishing write tid 5 to smithi20328086-34 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: oid 34 updating version 0 to 639 2024-09-17T15:43:13.528 INFO:tasks.rados.rados.0.smithi203.stdout:2503: oid 34 updating version 639 to 640 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2503: oid 34 updating version 640 to 642 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2503: oid 34 version 642 is already newer than 641 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 642 (ObjNum 890 snap 271 seq_num 890) dirty exists 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2503: left oid 34 (ObjNum 890 snap 271 seq_num 890) 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2504: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2504: finishing copy_from to smithi20328086-7 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1162 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:43:13.529 INFO:tasks.rados.rados.0.smithi203.stdout:2507: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-17T15:43:13.537 INFO:tasks.rados.rados.0.smithi203.stdout:2507: writing smithi20328086-18 from 575037 to 1235197 tid 1 2024-09-17T15:43:13.550 INFO:tasks.rados.rados.0.smithi203.stdout:2507: writing smithi20328086-18 from 1955809 to 2627111 tid 2 2024-09-17T15:43:13.550 INFO:tasks.rados.rados.0.smithi203.stdout:2503: done (4 left) 2024-09-17T15:43:13.550 INFO:tasks.rados.rados.0.smithi203.stdout:2504: done (3 left) 2024-09-17T15:43:13.550 INFO:tasks.rados.rados.0.smithi203.stdout:2508: snap_create 2024-09-17T15:43:13.550 INFO:tasks.rados.rados.0.smithi203.stdout:2506: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:43:13.767 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3745960 2024-09-17T15:43:13.774 INFO:tasks.rados.rados.0.smithi203.stdout:2505: expect (ObjNum 889 snap 271 seq_num 889) 2024-09-17T15:43:13.774 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:13.845 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2360382 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: finishing write tid 1 to smithi20328086-18 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: finishing write tid 2 to smithi20328086-18 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: finishing write tid 3 to smithi20328086-18 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: finishing write tid 4 to smithi20328086-18 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: oid 18 updating version 0 to 841 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: oid 18 updating version 841 to 843 2024-09-17T15:43:13.850 INFO:tasks.rados.rados.0.smithi203.stdout:2507: oid 18 version 843 is already newer than 842 2024-09-17T15:43:13.851 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 843 (ObjNum 891 snap 271 seq_num 891) dirty exists 2024-09-17T15:43:13.851 INFO:tasks.rados.rados.0.smithi203.stdout:2507: left oid 18 (ObjNum 891 snap 271 seq_num 891) 2024-09-17T15:43:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:2505: done (3 left) 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2506: done (2 left) 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2507: done (1 left) 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2508: done (0 left) 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2509: read oid 20 snap -1 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2509: expect (ObjNum 868 snap 269 seq_num 868) 2024-09-17T15:43:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:2510: snap_remove snap 260 2024-09-17T15:43:15.235 INFO:tasks.rados.rados.0.smithi203.stdout:2510: done (1 left) 2024-09-17T15:43:15.235 INFO:tasks.rados.rados.0.smithi203.stdout:2511: write oid 34 current snap is 272 2024-09-17T15:43:15.235 INFO:tasks.rados.rados.0.smithi203.stdout:2511: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-17T15:43:15.243 INFO:tasks.rados.rados.0.smithi203.stdout:2511: writing smithi20328086-34 from 591844 to 1327253 tid 1 2024-09-17T15:43:15.250 INFO:tasks.rados.rados.0.smithi203.stdout:2511: writing smithi20328086-34 from 1897938 to 2429251 tid 2 2024-09-17T15:43:15.253 INFO:tasks.rados.rados.0.smithi203.stdout:2511: writing smithi20328086-34 from 2862486 to 2991844 tid 3 2024-09-17T15:43:15.253 INFO:tasks.rados.rados.0.smithi203.stdout:2512: copy_from oid 40 from oid 12 current snap is 272 2024-09-17T15:43:15.253 INFO:tasks.rados.rados.0.smithi203.stdout:2509: expect (ObjNum 868 snap 269 seq_num 868) 2024-09-17T15:43:15.253 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 327315 2024-09-17T15:43:15.360 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2218981 2024-09-17T15:43:15.365 INFO:tasks.rados.rados.0.smithi203.stdout:2512: finishing copy_from to smithi20328086-40 2024-09-17T15:43:15.365 INFO:tasks.rados.rados.0.smithi203.stdout:2512: got expected ENOENT (src dne) 2024-09-17T15:43:15.365 INFO:tasks.rados.rados.0.smithi203.stdout:2512: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: finishing write tid 1 to smithi20328086-34 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: finishing write tid 2 to smithi20328086-34 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2509: done (2 left) 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2512: done (1 left) 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2513: rollback oid 6 current snap is 272 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 264 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: finishing write tid 3 to smithi20328086-34 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: finishing write tid 4 to smithi20328086-34 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: finishing write tid 5 to smithi20328086-34 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: oid 34 updating version 0 to 645 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: oid 34 updating version 645 to 646 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: oid 34 version 646 is already newer than 643 2024-09-17T15:43:15.366 INFO:tasks.rados.rados.0.smithi203.stdout:2511: oid 34 version 646 is already newer than 644 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 646 (ObjNum 892 snap 272 seq_num 892) dirty exists 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2511: left oid 34 (ObjNum 892 snap 272 seq_num 892) 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2511: done (1 left) 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2514: read oid 30 snap -1 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2514: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2515: copy_from oid 29 from oid 50 current snap is 272 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2516: write oid 4 current snap is 272 2024-09-17T15:43:15.367 INFO:tasks.rados.rados.0.smithi203.stdout:2516: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-17T15:43:15.372 INFO:tasks.rados.rados.0.smithi203.stdout:2516: writing smithi20328086-4 from 608651 to 1019309 tid 1 2024-09-17T15:43:15.381 INFO:tasks.rados.rados.0.smithi203.stdout:2516: writing smithi20328086-4 from 1556420 to 2231391 tid 2 2024-09-17T15:43:15.386 INFO:tasks.rados.rados.0.smithi203.stdout:2516: writing smithi20328086-4 from 2773556 to 3008651 tid 3 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:2517: rollback oid 7 current snap is 272 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 271 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:2513: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:2514: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:15.387 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:15.548 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3298933 2024-09-17T15:43:15.646 INFO:tasks.rados.rados.0.smithi203.stdout:2516: finishing write tid 1 to smithi20328086-4 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2513: done (4 left) 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2514: done (3 left) 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2518: delete oid 28 current snap is 272 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2515: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2515: finishing copy_from to smithi20328086-29 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1170 (ObjNum 844 snap 257 seq_num 844) dirty exists 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2516: finishing write tid 2 to smithi20328086-4 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2516: finishing write tid 3 to smithi20328086-4 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2516: finishing write tid 4 to smithi20328086-4 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2516: finishing write tid 5 to smithi20328086-4 2024-09-17T15:43:15.647 INFO:tasks.rados.rados.0.smithi203.stdout:2516: oid 4 updating version 0 to 1082 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2516: oid 4 version 1082 is already newer than 1081 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2516: oid 4 updating version 1082 to 1084 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2516: oid 4 version 1084 is already newer than 1083 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1084 (ObjNum 893 snap 272 seq_num 893) dirty exists 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2516: left oid 4 (ObjNum 893 snap 272 seq_num 893) 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2517: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2517: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:2517: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:43:15.648 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1165 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:43:15.652 INFO:tasks.rados.rados.0.smithi203.stdout:2515: done (3 left) 2024-09-17T15:43:15.652 INFO:tasks.rados.rados.0.smithi203.stdout:2516: done (2 left) 2024-09-17T15:43:15.652 INFO:tasks.rados.rados.0.smithi203.stdout:2517: done (1 left) 2024-09-17T15:43:15.652 INFO:tasks.rados.rados.0.smithi203.stdout:2518: done (0 left) 2024-09-17T15:43:15.652 INFO:tasks.rados.rados.0.smithi203.stdout:2519: delete oid 1 current snap is 272 2024-09-17T15:43:15.656 INFO:tasks.rados.rados.0.smithi203.stdout:2519: done (0 left) 2024-09-17T15:43:15.657 INFO:tasks.rados.rados.0.smithi203.stdout:2520: read oid 21 snap -1 2024-09-17T15:43:15.657 INFO:tasks.rados.rados.0.smithi203.stdout:2520: expect deleted 2024-09-17T15:43:15.657 INFO:tasks.rados.rados.0.smithi203.stdout:2521: write oid 33 current snap is 272 2024-09-17T15:43:15.657 INFO:tasks.rados.rados.0.smithi203.stdout:2521: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-17T15:43:15.664 INFO:tasks.rados.rados.0.smithi203.stdout:2521: writing smithi20328086-33 from 659072 to 1295477 tid 1 2024-09-17T15:43:15.669 INFO:tasks.rados.rados.0.smithi203.stdout:2521: writing smithi20328086-33 from 1731866 to 2154164 tid 2 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2521: writing smithi20328086-33 from 2855825 to 3059072 tid 3 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2522: copy_from oid 34 from oid 35 current snap is 272 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2521: finishing write tid 1 to smithi20328086-33 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2520: done (2 left) 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2523: write oid 21 current snap is 272 2024-09-17T15:43:15.675 INFO:tasks.rados.rados.0.smithi203.stdout:2523: seq_num 897 ranges {675879=711654,2073995=682309} 2024-09-17T15:43:15.682 INFO:tasks.rados.rados.0.smithi203.stdout:2523: writing smithi20328086-21 from 675879 to 1387533 tid 1 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2523: writing smithi20328086-21 from 2073995 to 2756304 tid 2 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2524: copy_from oid 45 from oid 40 current snap is 272 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: finishing write tid 2 to smithi20328086-33 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: finishing write tid 3 to smithi20328086-33 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: finishing write tid 4 to smithi20328086-33 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: finishing write tid 5 to smithi20328086-33 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: oid 33 updating version 0 to 1080 2024-09-17T15:43:15.692 INFO:tasks.rados.rados.0.smithi203.stdout:2521: oid 33 updating version 1080 to 1082 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2521: oid 33 version 1082 is already newer than 1081 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2521: oid 33 version 1082 is already newer than 1079 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1082 (ObjNum 896 snap 272 seq_num 896) dirty exists 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2521: left oid 33 (ObjNum 896 snap 272 seq_num 896) 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2522: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2522: finishing copy_from to smithi20328086-34 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 637 (ObjNum 851 snap 261 seq_num 851) dirty exists 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2521: done (3 left) 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2522: done (2 left) 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:2525: rollback oid 48 current snap is 272 2024-09-17T15:43:15.693 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 264 2024-09-17T15:43:15.694 INFO:tasks.rados.rados.0.smithi203.stdout:2526: copy_from oid 42 from oid 11 current snap is 272 2024-09-17T15:43:15.694 INFO:tasks.rados.rados.0.smithi203.stdout:2527: snap_create 2024-09-17T15:43:15.694 INFO:tasks.rados.rados.0.smithi203.stdout:2523: finishing write tid 1 to smithi20328086-21 2024-09-17T15:43:15.697 INFO:tasks.rados.rados.0.smithi203.stdout:2526: finishing copy_from to smithi20328086-42 2024-09-17T15:43:15.697 INFO:tasks.rados.rados.0.smithi203.stdout:2526: got expected ENOENT (src dne) 2024-09-17T15:43:15.697 INFO:tasks.rados.rados.0.smithi203.stdout:2525: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:43:15.697 INFO:tasks.rados.rados.0.smithi203.stdout:2525: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:43:15.697 INFO:tasks.rados.rados.0.smithi203.stdout:2526: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:43:15.698 INFO:tasks.rados.rados.0.smithi203.stdout:2525: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:43:15.698 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1234 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:43:15.700 INFO:tasks.rados.rados.0.smithi203.stdout:2523: finishing write tid 2 to smithi20328086-21 2024-09-17T15:43:15.701 INFO:tasks.rados.rados.0.smithi203.stdout:2523: finishing write tid 3 to smithi20328086-21 2024-09-17T15:43:15.701 INFO:tasks.rados.rados.0.smithi203.stdout:2523: finishing write tid 4 to smithi20328086-21 2024-09-17T15:43:15.701 INFO:tasks.rados.rados.0.smithi203.stdout:2523: oid 21 updating version 0 to 1337 2024-09-17T15:43:15.702 INFO:tasks.rados.rados.0.smithi203.stdout:2523: oid 21 version 1337 is already newer than 1335 2024-09-17T15:43:15.702 INFO:tasks.rados.rados.0.smithi203.stdout:2523: oid 21 version 1337 is already newer than 1336 2024-09-17T15:43:15.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1337 (ObjNum 897 snap 272 seq_num 897) dirty exists 2024-09-17T15:43:15.702 INFO:tasks.rados.rados.0.smithi203.stdout:2523: left oid 21 (ObjNum 897 snap 272 seq_num 897) 2024-09-17T15:43:15.706 INFO:tasks.rados.rados.0.smithi203.stdout:2524: finishing copy_from racing read to smithi20328086-45 2024-09-17T15:43:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:2524: finishing copy_from to smithi20328086-45 2024-09-17T15:43:15.712 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1162 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:43:16.306 INFO:tasks.rados.rados.0.smithi203.stdout:2523: done (4 left) 2024-09-17T15:43:16.306 INFO:tasks.rados.rados.0.smithi203.stdout:2524: done (3 left) 2024-09-17T15:43:16.306 INFO:tasks.rados.rados.0.smithi203.stdout:2525: done (2 left) 2024-09-17T15:43:16.307 INFO:tasks.rados.rados.0.smithi203.stdout:2526: done (1 left) 2024-09-17T15:43:16.307 INFO:tasks.rados.rados.0.smithi203.stdout:2527: done (0 left) 2024-09-17T15:43:16.307 INFO:tasks.rados.rados.0.smithi203.stdout:2528: delete oid 4 current snap is 273 2024-09-17T15:43:16.312 INFO:tasks.rados.rados.0.smithi203.stdout:2528: done (0 left) 2024-09-17T15:43:16.312 INFO:tasks.rados.rados.0.smithi203.stdout:2529: read oid 46 snap -1 2024-09-17T15:43:16.313 INFO:tasks.rados.rados.0.smithi203.stdout:2529: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:43:16.313 INFO:tasks.rados.rados.0.smithi203.stdout:2530: write oid 13 current snap is 273 2024-09-17T15:43:16.313 INFO:tasks.rados.rados.0.smithi203.stdout:2530: seq_num 899 ranges {709493=578505,1907312=685978} 2024-09-17T15:43:16.328 INFO:tasks.rados.rados.0.smithi203.stdout:2530: writing smithi20328086-13 from 709493 to 1287998 tid 1 2024-09-17T15:43:16.342 INFO:tasks.rados.rados.0.smithi203.stdout:2530: writing smithi20328086-13 from 1907312 to 2593290 tid 2 2024-09-17T15:43:16.342 INFO:tasks.rados.rados.0.smithi203.stdout:2531: read oid 17 snap -1 2024-09-17T15:43:16.342 INFO:tasks.rados.rados.0.smithi203.stdout:2531: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:16.343 INFO:tasks.rados.rados.0.smithi203.stdout:2532: delete oid 33 current snap is 273 2024-09-17T15:43:16.343 INFO:tasks.rados.rados.0.smithi203.stdout:2529: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:43:16.419 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:43:16.419 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:43:16.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:43:16.423 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:43:16.423 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:43:16.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:16.426 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 401876 2024-09-17T15:43:16.474 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1780736 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: finishing write tid 1 to smithi20328086-13 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: finishing write tid 2 to smithi20328086-13 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: finishing write tid 3 to smithi20328086-13 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: finishing write tid 4 to smithi20328086-13 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: oid 13 updating version 0 to 1114 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: oid 13 version 1114 is already newer than 1113 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: oid 13 updating version 1114 to 1115 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1115 (ObjNum 899 snap 273 seq_num 899) dirty exists 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2530: left oid 13 (ObjNum 899 snap 273 seq_num 899) 2024-09-17T15:43:16.491 INFO:tasks.rados.rados.0.smithi203.stdout:2531: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:16.544 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 10 monclient: get_monmap_and_config 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 10 monclient: build_initial_monmap 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 10 monclient: monmap: 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:16.554953+0000 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:16.554953+0000 2024-09-17T15:43:16.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:16.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 1 Processor -- start 2024-09-17T15:43:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 1 -- start start 2024-09-17T15:43:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.551+0000 7f7b66f91640 10 monclient: init 2024-09-17T15:43:16.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7b600a50e0 0x7f7b6009ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-c con 0x7f7b600a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-a con 0x7f7b600a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 0x7f7b60138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-b con 0x7f7b60137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7b6009bdc0 con 0x7f7b600a54c0 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7b6009d350 con 0x7f7b600a50e0 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7b600a6530 con 0x7f7b60137ea0 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b66f91640 10 monclient(hunting): authenticate will time out at 1929.520181s 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 0x7f7b60138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43474/0 (socket says 172.21.15.86:43474) 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 -- 172.21.15.86:0/2368955406 learned_addr learned my addr 172.21.15.86:0/2368955406 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 10 monclient(hunting): get_auth_request con 0x7f7b600a54c0 auth_method 0 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): get_auth_request con 0x7f7b60137ea0 auth_method 0 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:16.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 -- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 msgr2=0x7f7b600a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 -- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 msgr2=0x7f7b600a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b64d2d640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.203:3300/0 conn(0x7f7b600a50e0 0x7f7b6009ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b64d2d640 10 monclient(hunting): get_auth_request con 0x7f7b600a50e0 auth_method 0 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b64d2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b64d2d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b64d2d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:16.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient(hunting): handle_auth_done global_id 164282 payload 274 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 -- 172.21.15.86:0/2368955406 >> v2:172.21.15.203:3300/0 conn(0x7f7b600a50e0 msgr2=0x7f7b6009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.203:3300/0 conn(0x7f7b600a50e0 0x7f7b6009ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 -- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 msgr2=0x7f7b600a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient: _finish_hunting 0 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 monclient: found mon.noname-b 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.555+0000 7f7b6552e640 1 -- 172.21.15.86:0/2368955406 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6009c150 con 0x7f7b60137ea0 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 -- 172.21.15.86:0/2368955406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 msgr2=0x7f7b60138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 --2- 172.21.15.86:0/2368955406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 0x7f7b60138330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b600a8280 tx=0x7f7b5801c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56f2ae4bd158a5c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 1 -- 172.21.15.86:0/2368955406 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580b2070 con 0x7f7b60137ea0 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: dump: 2024-09-17T15:43:16.561 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 1 -- 172.21.15.86:0/2368955406 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b580a5840 con 0x7f7b60137ea0 2024-09-17T15:43:16.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 5 monclient: authenticate success, global_id 164282 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 1 -- 172.21.15.86:0/2368955406 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580a5c10 con 0x7f7b60137ea0 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 4 set_mon_vals no callback set 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5f7fe640 10 monclient: dump: 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.563 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: shutdown 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- 172.21.15.86:0/2368955406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 msgr2=0x7f7b60138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2368955406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 0x7f7b60138330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b600a8280 tx=0x7f7b5801c7c0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- 172.21.15.86:0/2368955406 shutdown_connections 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2368955406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b60137ea0 0x7f7b60138330 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.86:3300/0 conn(0x7f7b600a54c0 0x7f7b600a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2368955406 >> v2:172.21.15.203:3300/0 conn(0x7f7b600a50e0 0x7f7b6009ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- 172.21.15.86:0/2368955406 >> 172.21.15.86:0/2368955406 conn(0x7f7b600aaae0 msgr2=0x7f7b600aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- 172.21.15.86:0/2368955406 shutdown_connections 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- 172.21.15.86:0/2368955406 wait complete. 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: build_initial_monmap 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: monmap: 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 Processor -- start 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- start start 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: init 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-c con 0x7f7b600a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 0x7f7b6013c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-b con 0x7f7b600a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): picked mon.noname-a con 0x7f7b60137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6006f660 con 0x7f7b60137ea0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6006f4e0 con 0x7f7b600a54c0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7b60141b20 con 0x7f7b600a50e0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b66f91640 10 monclient(hunting): authenticate will time out at 1929.527220s 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54244/0 (socket says 172.21.15.86:54244) 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 -- 172.21.15.86:0/2754499577 learned_addr learned my addr 172.21.15.86:0/2754499577 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): get_auth_request con 0x7f7b600a50e0 auth_method 0 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:16.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 10 monclient(hunting): get_auth_request con 0x7f7b60137ea0 auth_method 0 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 msgr2=0x7f7b6013c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 msgr2=0x7f7b6013c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b6552e640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 0x7f7b6013c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient(hunting): handle_auth_done global_id 164285 payload 824 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 msgr2=0x7f7b6013c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 0x7f7b6013c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 msgr2=0x7f7b6013c680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient: _finish_hunting 0 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 monclient: found mon.noname-c 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b64d2d640 1 -- 172.21.15.86:0/2754499577 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b600a18d0 con 0x7f7b600a50e0 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5ffff640 10 monclient: get_auth_request con 0x7f7b600a54c0 auth_method 0 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.559+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 0x7f7b6013c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b64d2d640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 msgr2=0x7f7b60136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b64d2d640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400d800 tx=0x7f7b5400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:16.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5400b8d0 con 0x7f7b600a50e0 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: dump: 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b54019070 con 0x7f7b600a50e0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 5 monclient: authenticate success, global_id 164285 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _renew_subs 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b6013dac0 con 0x7f7b600a50e0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5400bc70 con 0x7f7b600a50e0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 10 monclient: dump: 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:16.571 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _renew_subs 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5effd640 4 set_mon_vals no callback set 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b60122890 con 0x7f7b600a50e0 2024-09-17T15:43:16.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:16.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:16.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b60122160 con 0x7f7b600a50e0 2024-09-17T15:43:16.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7b5401e460 con 0x7f7b600a50e0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 0x7f7b4403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(561..561 src has 438..561) ==== 4771+0+0 (secure 0 0 0) 0x7f7b54050400 con 0x7f7b600a50e0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 0x7f7b4403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.563+0000 7f7b5ffff640 10 monclient: get_auth_request con 0x7f7b4403bbe0 auth_method 0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.567+0000 7f7b5ffff640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 0x7f7b4403e0a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009100 tx=0x7f7b4c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.567+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7b60122160 con 0x7f7b600a50e0 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.567+0000 7f7b5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:16.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.567+0000 7f7b5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:16.601 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:16.659 INFO:tasks.rados.rados.0.smithi203.stdout:2529: done (3 left) 2024-09-17T15:43:16.659 INFO:tasks.rados.rados.0.smithi203.stdout:2530: done (2 left) 2024-09-17T15:43:16.659 INFO:tasks.rados.rados.0.smithi203.stdout:2531: done (1 left) 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2532: done (0 left) 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2533: rollback oid 45 current snap is 273 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 244 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2534: read oid 17 snap -1 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2534: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2535: read oid 15 snap -1 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2535: expect (ObjNum 871 snap 269 seq_num 871) 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2536: read oid 28 snap -1 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2536: expect deleted 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2537: write oid 16 current snap is 273 2024-09-17T15:43:16.660 INFO:tasks.rados.rados.0.smithi203.stdout:2537: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-17T15:43:16.685 INFO:tasks.rados.rados.0.smithi203.stdout:2537: writing smithi20328086-16 from 743107 to 1472110 tid 1 2024-09-17T15:43:16.702 INFO:tasks.rados.rados.0.smithi203.stdout:2537: writing smithi20328086-16 from 1907923 to 2597570 tid 2 2024-09-17T15:43:16.702 INFO:tasks.rados.rados.0.smithi203.stdout:2538: write oid 20 current snap is 273 2024-09-17T15:43:16.703 INFO:tasks.rados.rados.0.smithi203.stdout:2538: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-09-17T15:43:16.718 INFO:tasks.rados.rados.0.smithi203.stdout:2538: writing smithi20328086-20 from 759914 to 1164166 tid 1 2024-09-17T15:43:16.727 INFO:tasks.rados.rados.0.smithi203.stdout:2538: writing smithi20328086-20 from 1566405 to 1999710 tid 2 2024-09-17T15:43:16.738 INFO:tasks.rados.rados.0.smithi203.stdout:2538: writing smithi20328086-20 from 2504010 to 3159914 tid 3 2024-09-17T15:43:16.739 INFO:tasks.rados.rados.0.smithi203.stdout:2539: read oid 2 snap 265 2024-09-17T15:43:16.739 INFO:tasks.rados.rados.0.smithi203.stdout:2539: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:43:16.739 INFO:tasks.rados.rados.0.smithi203.stdout:2540: delete oid 26 current snap is 273 2024-09-17T15:43:16.739 INFO:tasks.rados.rados.0.smithi203.stdout:2534: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:16.788 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2870705 2024-09-17T15:43:16.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b66f91640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:16.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b66f91640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:16.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b66f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:16.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7b600a58d0 con 0x7f7b600a50e0 2024-09-17T15:43:16.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b5d7fa640 1 -- 172.21.15.86:0/2754499577 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f7b5400be10 con 0x7f7b600a50e0 2024-09-17T15:43:16.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:16.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.807+0000 7f7b5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:16.805 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:16.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 msgr2=0x7f7b4403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 0x7f7b4403e0a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c009100 tx=0x7f7b4c008040 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 10 monclient: shutdown 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 msgr2=0x7f7b60136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400d800 tx=0x7f7b5400dcc0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 shutdown_connections 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b4403bbe0 0x7f7b4403e0a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b60137ea0 0x7f7b6013c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b600a54c0 0x7f7b6013c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 --2- 172.21.15.86:0/2754499577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b600a50e0 0x7f7b60136ed0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 >> 172.21.15.86:0/2754499577 conn(0x7f7b600aaae0 msgr2=0x7f7b6013e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:16.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 shutdown_connections 2024-09-17T15:43:16.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:16.811+0000 7f7b66f91640 1 -- 172.21.15.86:0/2754499577 wait complete. 2024-09-17T15:43:16.825 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:16.898 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2510669 2024-09-17T15:43:16.930 INFO:tasks.rados.rados.0.smithi203.stdout:2533: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:43:16.930 INFO:tasks.rados.rados.0.smithi203.stdout:2533: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:43:16.930 INFO:tasks.rados.rados.0.smithi203.stdout:2533: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:43:16.930 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1166 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:43:16.930 INFO:tasks.rados.rados.0.smithi203.stdout:2535: expect (ObjNum 871 snap 269 seq_num 871) 2024-09-17T15:43:16.969 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: finishing write tid 1 to smithi20328086-16 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: finishing write tid 2 to smithi20328086-16 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: finishing write tid 3 to smithi20328086-16 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: finishing write tid 4 to smithi20328086-16 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: oid 16 updating version 0 to 1237 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: oid 16 version 1237 is already newer than 1235 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: oid 16 version 1237 is already newer than 1236 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1237 (ObjNum 901 snap 273 seq_num 901) dirty exists 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2537: left oid 16 (ObjNum 901 snap 273 seq_num 901) 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2538: finishing write tid 1 to smithi20328086-20 2024-09-17T15:43:16.998 INFO:tasks.rados.rados.0.smithi203.stdout:2538: finishing write tid 2 to smithi20328086-20 2024-09-17T15:43:16.999 INFO:tasks.rados.rados.0.smithi203.stdout:2539: expect (ObjNum 769 snap 240 seq_num 769) 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: finishing write tid 3 to smithi20328086-20 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: finishing write tid 4 to smithi20328086-20 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: finishing write tid 5 to smithi20328086-20 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: oid 20 updating version 0 to 1162 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: oid 20 version 1162 is already newer than 1160 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: oid 20 version 1162 is already newer than 1161 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: oid 20 version 1162 is already newer than 1159 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1162 (ObjNum 902 snap 273 seq_num 902) dirty exists 2024-09-17T15:43:17.167 INFO:tasks.rados.rados.0.smithi203.stdout:2538: left oid 20 (ObjNum 902 snap 273 seq_num 902) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2533: done (7 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2534: done (6 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2535: done (5 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2536: done (4 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2537: done (3 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2538: done (2 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2539: done (1 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2540: done (0 left) 2024-09-17T15:43:17.168 INFO:tasks.rados.rados.0.smithi203.stdout:2541: snap_remove snap 265 2024-09-17T15:43:17.308 INFO:tasks.rados.rados.0.smithi203.stdout:2541: done (0 left) 2024-09-17T15:43:17.308 INFO:tasks.rados.rados.0.smithi203.stdout:2542: snap_remove snap 266 2024-09-17T15:43:18.311 INFO:tasks.rados.rados.0.smithi203.stdout:2542: done (0 left) 2024-09-17T15:43:18.311 INFO:tasks.rados.rados.0.smithi203.stdout:2543: delete oid 16 current snap is 273 2024-09-17T15:43:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:2543: done (0 left) 2024-09-17T15:43:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:2544: copy_from oid 19 from oid 15 current snap is 273 2024-09-17T15:43:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:2545: snap_create 2024-09-17T15:43:18.352 INFO:tasks.rados.rados.0.smithi203.stdout:2544: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:43:18.354 INFO:tasks.rados.rados.0.smithi203.stdout:2544: finishing copy_from to smithi20328086-19 2024-09-17T15:43:18.354 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1247 (ObjNum 871 snap 269 seq_num 871) dirty exists 2024-09-17T15:43:19.339 INFO:tasks.rados.rados.0.smithi203.stdout:2544: done (1 left) 2024-09-17T15:43:19.339 INFO:tasks.rados.rados.0.smithi203.stdout:2545: done (0 left) 2024-09-17T15:43:19.339 INFO:tasks.rados.rados.0.smithi203.stdout:2546: snap_remove snap 244 2024-09-17T15:43:19.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:19.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: get_monmap_and_config 2024-09-17T15:43:19.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: build_initial_monmap 2024-09-17T15:43:19.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: monmap: 2024-09-17T15:43:19.952 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:19.952 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:19.957814+0000 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:19.957814+0000 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:19.953 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 Processor -- start 2024-09-17T15:43:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 -- start start 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: init 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 0x7f286406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-b con 0x7f28640689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2864068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f286413aff0 0x7f286413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-c con 0x7f286413aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2864074470 con 0x7f2864068db0 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f286409c8b0 con 0x7f286413aff0 2024-09-17T15:43:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f286409cb90 con 0x7f28640689d0 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f286a47e640 10 monclient(hunting): authenticate will time out at 1932.923277s 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 0x7f286406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f28637fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2864068db0 0x7f28640691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50672/0 (socket says 172.21.15.86:50672) 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f28637fe640 1 -- 172.21.15.86:0/3166911199 learned_addr learned my addr 172.21.15.86:0/3166911199 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): get_auth_request con 0x7f28640689d0 auth_method 0 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:19.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient(hunting): handle_auth_done global_id 164288 payload 274 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2868a1b640 1 --2- 172.21.15.86:0/3166911199 >> v2:172.21.15.203:3300/0 conn(0x7f286413aff0 0x7f286413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 -- 172.21.15.86:0/3166911199 >> v2:172.21.15.203:3300/0 conn(0x7f286413aff0 msgr2=0x7f286413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 --2- 172.21.15.86:0/3166911199 >> v2:172.21.15.203:3300/0 conn(0x7f286413aff0 0x7f286413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 -- 172.21.15.86:0/3166911199 >> v1:172.21.15.86:6789/0 conn(0x7f2864068db0 legacy=0x7f28640691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient: _finish_hunting 0 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 monclient: found mon.noname-b 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:19.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.955+0000 7f2863fff640 1 -- 172.21.15.86:0/3166911199 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286409ce70 con 0x7f28640689d0 2024-09-17T15:43:19.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 -- 172.21.15.86:0/3166911199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 msgr2=0x7f286406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:19.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 --2- 172.21.15.86:0/3166911199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 0x7f286406f840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097c0 tx=0x7f285c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa1fa71dffae7b5f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:19.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 1 -- 172.21.15.86:0/3166911199 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c0b0070 con 0x7f28640689d0 2024-09-17T15:43:19.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:19.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: dump: 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:19.961 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 1 -- 172.21.15.86:0/3166911199 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c0ab600 con 0x7f28640689d0 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 5 monclient: authenticate success, global_id 164288 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 1 -- 172.21.15.86:0/3166911199 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286409c2d0 con 0x7f28640689d0 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 4 set_mon_vals no callback set 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2862ffd640 10 monclient: dump: 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: shutdown 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- 172.21.15.86:0/3166911199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 msgr2=0x7f286406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- 172.21.15.86:0/3166911199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 0x7f286406f840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097c0 tx=0x7f285c01c360 comp rx=0 tx=0).stop 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- 172.21.15.86:0/3166911199 shutdown_connections 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- 172.21.15.86:0/3166911199 >> v2:172.21.15.203:3300/0 conn(0x7f286413aff0 0x7f286413d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- 172.21.15.86:0/3166911199 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28640689d0 0x7f286406f840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- 172.21.15.86:0/3166911199 >> 172.21.15.86:0/3166911199 conn(0x7f28640a6440 msgr2=0x7f28640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- 172.21.15.86:0/3166911199 shutdown_connections 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- 172.21.15.86:0/3166911199 wait complete. 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: build_initial_monmap 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: monmap: 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:19.964 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 Processor -- start 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- start start 2024-09-17T15:43:19.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: init 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-a con 0x7f28640689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 0x7f2864135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-c con 0x7f2864068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f286413aff0 0x7f28641363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): picked mon.noname-b con 0x7f286413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:19.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2864140610 con 0x7f28640689d0 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2864140490 con 0x7f286413aff0 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2864140790 con 0x7f2864068db0 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f286a47e640 10 monclient(hunting): authenticate will time out at 1932.930923s 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 0x7f2864135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33134/0 (socket says 172.21.15.86:33134) 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 -- 172.21.15.86:0/654938450 learned_addr learned my addr 172.21.15.86:0/654938450 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:19.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 10 monclient(hunting): get_auth_request con 0x7f28640689d0 auth_method 0 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): get_auth_request con 0x7f2864068db0 auth_method 0 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 msgr2=0x7f28641cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 msgr2=0x7f28641cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:19.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2863fff640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2868a1b640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f286413aff0 0x7f28641363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2868a1b640 10 monclient(hunting): get_auth_request con 0x7f286413aff0 auth_method 0 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2868a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2868a1b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f2868a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient(hunting): handle_auth_done global_id 164291 payload 824 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f286413aff0 msgr2=0x7f28641363a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f286413aff0 0x7f28641363a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 msgr2=0x7f28641cfd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:19.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 monclient: found mon.noname-c 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.963+0000 7f28637fe640 1 -- 172.21.15.86:0/654938450 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2864122830 con 0x7f2864068db0 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f28637fe640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 msgr2=0x7f2864135e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f28637fe640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 0x7f2864135e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c410 tx=0x7f284c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c017070 con 0x7f2864068db0 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: dump: 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:19.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: _finish_auth 0 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284c0070d0 con 0x7f2864068db0 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c0074c0 con 0x7f2864068db0 2024-09-17T15:43:19.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 10 monclient: dump: 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:19.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 5 monclient: authenticate success, global_id 164291 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f28627fc640 4 set_mon_vals no callback set 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _renew_subs 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286413d2f0 con 0x7f2864068db0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _renew_subs 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2864122550 con 0x7f2864068db0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f284c01a070 con 0x7f2864068db0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:19.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286406f840 con 0x7f2864068db0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 0x7f284403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(564..564 src has 438..564) ==== 4771+0+0 (secure 0 0 0) 0x7f284c04f3b0 con 0x7f2864068db0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2863fff640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 0x7f284403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.967+0000 7f2863fff640 10 monclient: get_auth_request con 0x7f284403bbb0 auth_method 0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.971+0000 7f2863fff640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 0x7f284403e070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f285c01c190 tx=0x7f285c0a4f90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.971+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f284c04ddb0 con 0x7f2864068db0 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.971+0000 7f2860ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:19.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:19.971+0000 7f2860ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f286a47e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f286a47e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f286a47e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f286406d4f0 con 0x7f2864068db0 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f2860ff9640 1 -- 172.21.15.86:0/654938450 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f284c04f090 con 0x7f2864068db0 2024-09-17T15:43:20.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f2860ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:20.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.203+0000 7f2860ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:20.201 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:20.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 msgr2=0x7f284403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:20.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 0x7f284403e070 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f285c01c190 tx=0x7f285c0a4f90 comp rx=0 tx=0).stop 2024-09-17T15:43:20.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 10 monclient: shutdown 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 msgr2=0x7f2864135e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 0x7f2864135e60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f284c00c410 tx=0x7f284c00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 shutdown_connections 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f284403bbb0 0x7f284403e070 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f286413aff0 0x7f28641363a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2864068db0 0x7f2864135e60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 --2- 172.21.15.86:0/654938450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28640689d0 0x7f28641cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:20.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 >> 172.21.15.86:0/654938450 conn(0x7f28640a6440 msgr2=0x7f286409d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:20.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 shutdown_connections 2024-09-17T15:43:20.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:20.207+0000 7f286a47e640 1 -- 172.21.15.86:0/654938450 wait complete. 2024-09-17T15:43:20.234 INFO:tasks.rados.rados.0.smithi203.stdout:2546: done (0 left) 2024-09-17T15:43:20.234 INFO:tasks.rados.rados.0.smithi203.stdout:2547: write oid 49 current snap is 274 2024-09-17T15:43:20.234 INFO:tasks.rados.rados.0.smithi203.stdout:2547: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2024-09-17T15:43:20.257 INFO:tasks.rados.rados.0.smithi203.stdout:2547: writing smithi20328086-49 from 410335 to 1156687 tid 1 2024-09-17T15:43:20.281 INFO:tasks.rados.rados.0.smithi203.stdout:2547: writing smithi20328086-49 from 1741851 to 2438836 tid 2 2024-09-17T15:43:20.292 INFO:tasks.rados.rados.0.smithi203.stdout:2547: writing smithi20328086-49 from 3102632 to 3210335 tid 3 2024-09-17T15:43:20.292 INFO:tasks.rados.rados.0.smithi203.stdout:2548: snap_create 2024-09-17T15:43:20.292 INFO:tasks.rados.rados.0.smithi203.stdout:2547: finishing write tid 1 to smithi20328086-49 2024-09-17T15:43:20.292 INFO:tasks.rados.rados.0.smithi203.stdout:2547: finishing write tid 2 to smithi20328086-49 2024-09-17T15:43:20.300 INFO:tasks.rados.rados.0.smithi203.stdout:2547: finishing write tid 3 to smithi20328086-49 2024-09-17T15:43:20.301 INFO:tasks.rados.rados.0.smithi203.stdout:2547: finishing write tid 4 to smithi20328086-49 2024-09-17T15:43:20.301 INFO:tasks.rados.rados.0.smithi203.stdout:2547: finishing write tid 5 to smithi20328086-49 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:2547: oid 49 updating version 0 to 1369 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:2547: oid 49 updating version 1369 to 1370 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:2547: oid 49 version 1370 is already newer than 1364 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:2547: oid 49 updating version 1370 to 1371 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1371 (ObjNum 905 snap 274 seq_num 905) dirty exists 2024-09-17T15:43:20.302 INFO:tasks.rados.rados.0.smithi203.stdout:2547: left oid 49 (ObjNum 905 snap 274 seq_num 905) 2024-09-17T15:43:21.235 INFO:tasks.rados.rados.0.smithi203.stdout:2547: done (1 left) 2024-09-17T15:43:21.235 INFO:tasks.rados.rados.0.smithi203.stdout:2548: done (0 left) 2024-09-17T15:43:21.235 INFO:tasks.rados.rados.0.smithi203.stdout:2549: read oid 27 snap 261 2024-09-17T15:43:21.235 INFO:tasks.rados.rados.0.smithi203.stdout:2549: expect (ObjNum 823 snap 254 seq_num 823) 2024-09-17T15:43:21.235 INFO:tasks.rados.rados.0.smithi203.stdout:2550: rollback oid 18 current snap is 275 2024-09-17T15:43:21.236 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 270 2024-09-17T15:43:21.236 INFO:tasks.rados.rados.0.smithi203.stdout:2551: snap_remove snap 274 2024-09-17T15:43:22.243 INFO:tasks.rados.rados.0.smithi203.stdout:2551: done (2 left) 2024-09-17T15:43:22.243 INFO:tasks.rados.rados.0.smithi203.stdout:2552: snap_remove snap 273 2024-09-17T15:43:23.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:23.247 INFO:tasks.rados.rados.0.smithi203.stdout:2552: done (2 left) 2024-09-17T15:43:23.247 INFO:tasks.rados.rados.0.smithi203.stdout:2553: write (excl) oid 28 current snap is 275 2024-09-17T15:43:23.248 INFO:tasks.rados.rados.0.smithi203.stdout:2553: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-09-17T15:43:23.261 INFO:tasks.rados.rados.0.smithi203.stdout:2553: writing smithi20328086-28 from 427142 to 848743 tid 1 2024-09-17T15:43:23.278 INFO:tasks.rados.rados.0.smithi203.stdout:2553: writing smithi20328086-28 from 1400333 to 1840976 tid 2 2024-09-17T15:43:23.298 INFO:tasks.rados.rados.0.smithi203.stdout:2553: writing smithi20328086-28 from 2613702 to 3064028 tid 3 2024-09-17T15:43:23.299 INFO:tasks.rados.rados.0.smithi203.stdout:2554: snap_create 2024-09-17T15:43:23.299 INFO:tasks.rados.rados.0.smithi203.stdout:2550: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:43:23.299 INFO:tasks.rados.rados.0.smithi203.stdout:2550: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:43:23.299 INFO:tasks.rados.rados.0.smithi203.stdout:2549: expect (ObjNum 823 snap 254 seq_num 823) 2024-09-17T15:43:23.299 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: get_monmap_and_config 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: build_initial_monmap 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: monmap: 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:23.357668+0000 2024-09-17T15:43:23.352 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:23.357668+0000 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:23.353 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 Processor -- start 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 -- start start 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: init 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-b con 0x7ff8580689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff858068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-a con 0x7ff85813aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff858074470 con 0x7ff858068db0 2024-09-17T15:43:23.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff85809c8b0 con 0x7ff85813aff0 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff85809cb90 con 0x7ff8580689d0 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff85db92640 10 monclient(hunting): authenticate will time out at 1936.323211s 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff856ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff858068db0 0x7ff8580691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50686/0 (socket says 172.21.15.86:50686) 2024-09-17T15:43:23.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59654/0 (socket says 172.21.15.86:59654) 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff856ffd640 1 -- 172.21.15.86:0/2605580986 learned_addr learned my addr 172.21.15.86:0/2605580986 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): get_auth_request con 0x7ff8580689d0 auth_method 0 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 10 monclient(hunting): get_auth_request con 0x7ff85813aff0 auth_method 0 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 1 -- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 msgr2=0x7ff85813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:43:23.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 1 -- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 msgr2=0x7ff85813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 1 --2- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff857fff640 1 --2- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient(hunting): handle_auth_done global_id 164294 payload 274 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 -- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 msgr2=0x7ff85813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 --2- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 -- 172.21.15.86:0/2605580986 >> v1:172.21.15.86:6789/0 conn(0x7ff858068db0 legacy=0x7ff8580691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 monclient: found mon.noname-b 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.355+0000 7ff8577fe640 1 -- 172.21.15.86:0/2605580986 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85809ce70 con 0x7ff8580689d0 2024-09-17T15:43:23.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8577fe640 1 -- 172.21.15.86:0/2605580986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 msgr2=0x7ff85806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:23.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8577fe640 1 --2- 172.21.15.86:0/2605580986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0097c0 tx=0x7ff84c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=735c1941798d9002 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:23.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 1 -- 172.21.15.86:0/2605580986 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0b0070 con 0x7ff8580689d0 2024-09-17T15:43:23.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: dump: 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 1 -- 172.21.15.86:0/2605580986 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84c0ab600 con 0x7ff8580689d0 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 1 -- 172.21.15.86:0/2605580986 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85809c2d0 con 0x7ff8580689d0 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff8567fc640 10 monclient: dump: 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 5 monclient: authenticate success, global_id 164294 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 4 set_mon_vals no callback set 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: shutdown 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- 172.21.15.86:0/2605580986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 msgr2=0x7ff85806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:23.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 --2- 172.21.15.86:0/2605580986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0097c0 tx=0x7ff84c01c360 comp rx=0 tx=0).stop 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- 172.21.15.86:0/2605580986 shutdown_connections 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 --2- 172.21.15.86:0/2605580986 >> v2:172.21.15.86:3300/0 conn(0x7ff85813aff0 0x7ff85813d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 --2- 172.21.15.86:0/2605580986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff85806f840 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- 172.21.15.86:0/2605580986 >> 172.21.15.86:0/2605580986 conn(0x7ff8580a6440 msgr2=0x7ff8580a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- 172.21.15.86:0/2605580986 shutdown_connections 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- 172.21.15.86:0/2605580986 wait complete. 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: build_initial_monmap 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: monmap: 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:23.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 Processor -- start 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 -- start start 2024-09-17T15:43:23.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: init 2024-09-17T15:43:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:43:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-c con 0x7ff8580689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.363+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-a con 0x7ff858068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff85813aff0 0x7ff858070da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): picked mon.noname-b con 0x7ff85813aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff858140610 con 0x7ff858068db0 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff858140490 con 0x7ff85813aff0 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff858140790 con 0x7ff8580689d0 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient(hunting): authenticate will time out at 1936.332051s 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59658/0 (socket says 172.21.15.86:59658) 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 -- 172.21.15.86:0/354262642 learned_addr learned my addr 172.21.15.86:0/354262642 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): get_auth_request con 0x7ff8580689d0 auth_method 0 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 10 monclient(hunting): get_auth_request con 0x7ff858068db0 auth_method 0 2024-09-17T15:43:23.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 msgr2=0x7ff858132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 msgr2=0x7ff858132c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff856ffd640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient(hunting): handle_auth_done global_id 164297 payload 824 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff857fff640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff85813aff0 0x7ff858070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff85813aff0 msgr2=0x7ff858070da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:23.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff85813aff0 0x7ff858070da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 msgr2=0x7ff858132c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 monclient: found mon.noname-c 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 -- 172.21.15.86:0/354262642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85809d460 con 0x7ff8580689d0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 msgr2=0x7ff8581cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff8577fe640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01cc00 tx=0x7ff84c01aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0b0050 con 0x7ff8580689d0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:23.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: dump: 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: _finish_auth 0 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84c0bc070 con 0x7ff8580689d0 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0b5c80 con 0x7ff8580689d0 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff83ffff640 10 monclient: dump: 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 5 monclient: authenticate success, global_id 164297 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _renew_subs 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff858072860 con 0x7ff8580689d0 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff855ffb640 4 set_mon_vals no callback set 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _renew_subs 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85811cdf0 con 0x7ff8580689d0 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.367+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85806f840 con 0x7ff8580689d0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff84c0ab4c0 con 0x7ff8580689d0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff83ffff640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 0x7ff84403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(568..568 src has 438..568) ==== 4831+0+0 (secure 0 0 0) 0x7ff84c0ea840 con 0x7ff8580689d0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff856ffd640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 0x7ff84403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff856ffd640 10 monclient: get_auth_request con 0x7ff84403bbb0 auth_method 0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.371+0000 7ff856ffd640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 0x7ff84403e070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff848007d50 tx=0x7ff848009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.375+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff84c016070 con 0x7ff8580689d0 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.375+0000 7ff83ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.375+0000 7ff83ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:23.512 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3488538 2024-09-17T15:43:23.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff85db92640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:23.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff85db92640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:23.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff85db92640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:23.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff85806d4f0 con 0x7ff8580689d0 2024-09-17T15:43:23.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff83ffff640 1 -- 172.21.15.86:0/354262642 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ff84c0c1c10 con 0x7ff8580689d0 2024-09-17T15:43:23.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff83ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:23.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.607+0000 7ff83ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:23.604 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 msgr2=0x7ff84403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 0x7ff84403e070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff848007d50 tx=0x7ff848009040 comp rx=0 tx=0).stop 2024-09-17T15:43:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 10 monclient: shutdown 2024-09-17T15:43:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:23.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 msgr2=0x7ff8581cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff84c01cc00 tx=0x7ff84c01aba0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 shutdown_connections 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff84403bbb0 0x7ff84403e070 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff85813aff0 0x7ff858070da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff858068db0 0x7ff858132c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 --2- 172.21.15.86:0/354262642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8580689d0 0x7ff8581cdc00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 >> 172.21.15.86:0/354262642 conn(0x7ff8580a6440 msgr2=0x7ff858135ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 shutdown_connections 2024-09-17T15:43:23.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:23.611+0000 7ff85db92640 1 -- 172.21.15.86:0/354262642 wait complete. 2024-09-17T15:43:23.610 INFO:tasks.rados.rados.0.smithi203.stdout:2550: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:43:23.610 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 846 (ObjNum 835 snap 256 seq_num 835) dirty exists 2024-09-17T15:43:23.610 INFO:tasks.rados.rados.0.smithi203.stdout:2553: finishing write tid 1 to smithi20328086-28 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: finishing write tid 2 to smithi20328086-28 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: finishing write tid 3 to smithi20328086-28 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: finishing write tid 4 to smithi20328086-28 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: finishing write tid 5 to smithi20328086-28 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: oid 28 updating version 0 to 1375 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: oid 28 version 1375 is already newer than 1373 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: oid 28 version 1375 is already newer than 1374 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: oid 28 version 1375 is already newer than 1372 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1375 (ObjNum 906 snap 275 seq_num 906) dirty exists 2024-09-17T15:43:23.611 INFO:tasks.rados.rados.0.smithi203.stdout:2553: left oid 28 (ObjNum 906 snap 275 seq_num 906) 2024-09-17T15:43:24.261 INFO:tasks.rados.rados.0.smithi203.stdout:2549: done (3 left) 2024-09-17T15:43:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2550: done (2 left) 2024-09-17T15:43:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2553: done (1 left) 2024-09-17T15:43:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2554: done (0 left) 2024-09-17T15:43:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2555: write (excl) oid 49 current snap is 276 2024-09-17T15:43:24.262 INFO:tasks.rados.rados.0.smithi203.stdout:2555: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-17T15:43:24.274 INFO:tasks.rados.rados.0.smithi203.stdout:2555: writing smithi20328086-49 from 443949 to 940799 tid 1 2024-09-17T15:43:24.290 INFO:tasks.rados.rados.0.smithi203.stdout:2555: writing smithi20328086-49 from 1458815 to 2043116 tid 2 2024-09-17T15:43:24.306 INFO:tasks.rados.rados.0.smithi203.stdout:2555: writing smithi20328086-49 from 2641125 to 3243949 tid 3 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2556: read oid 17 snap -1 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2556: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2557: write oid 34 current snap is 276 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2555: finishing write tid 1 to smithi20328086-49 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2555: finishing write tid 2 to smithi20328086-49 2024-09-17T15:43:24.307 INFO:tasks.rados.rados.0.smithi203.stdout:2557: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-17T15:43:24.323 INFO:tasks.rados.rados.0.smithi203.stdout:2557: writing smithi20328086-34 from 460756 to 1032855 tid 1 2024-09-17T15:43:24.335 INFO:tasks.rados.rados.0.smithi203.stdout:2557: writing smithi20328086-34 from 1517297 to 1961609 tid 2 2024-09-17T15:43:24.347 INFO:tasks.rados.rados.0.smithi203.stdout:2557: writing smithi20328086-34 from 2668548 to 3141418 tid 3 2024-09-17T15:43:24.347 INFO:tasks.rados.rados.0.smithi203.stdout:2558: write (excl) oid 37 current snap is 276 2024-09-17T15:43:24.347 INFO:tasks.rados.rados.0.smithi203.stdout:2558: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-17T15:43:24.356 INFO:tasks.rados.rados.0.smithi203.stdout:2558: writing smithi20328086-37 from 477563 to 1124911 tid 1 2024-09-17T15:43:24.371 INFO:tasks.rados.rados.0.smithi203.stdout:2558: writing smithi20328086-37 from 1859426 to 2447396 tid 2 2024-09-17T15:43:24.373 INFO:tasks.rados.rados.0.smithi203.stdout:2558: writing smithi20328086-37 from 2979618 to 3277563 tid 3 2024-09-17T15:43:24.373 INFO:tasks.rados.rados.0.smithi203.stdout:2559: rollback oid 3 current snap is 276 2024-09-17T15:43:24.373 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 261 2024-09-17T15:43:24.373 INFO:tasks.rados.rados.0.smithi203.stdout:2555: finishing write tid 3 to smithi20328086-49 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: finishing write tid 4 to smithi20328086-49 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: finishing write tid 5 to smithi20328086-49 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: oid 49 updating version 0 to 1380 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: oid 49 version 1380 is already newer than 1378 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: oid 49 version 1380 is already newer than 1379 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: oid 49 version 1380 is already newer than 1377 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1380 (ObjNum 907 snap 276 seq_num 907) dirty exists 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2555: left oid 49 (ObjNum 907 snap 276 seq_num 907) 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:2556: expect (ObjNum 818 snap 254 seq_num 818) 2024-09-17T15:43:24.374 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:24.600 INFO:tasks.rados.rados.0.smithi203.stdout:2557: finishing write tid 1 to smithi20328086-34 2024-09-17T15:43:24.600 INFO:tasks.rados.rados.0.smithi203.stdout:2557: finishing write tid 2 to smithi20328086-34 2024-09-17T15:43:24.600 INFO:tasks.rados.rados.0.smithi203.stdout:2557: finishing write tid 3 to smithi20328086-34 2024-09-17T15:43:24.600 INFO:tasks.rados.rados.0.smithi203.stdout:2557: finishing write tid 4 to smithi20328086-34 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: finishing write tid 5 to smithi20328086-34 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: oid 34 updating version 0 to 648 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: oid 34 version 648 is already newer than 647 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: oid 34 updating version 648 to 649 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: oid 34 updating version 649 to 650 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 650 (ObjNum 908 snap 276 seq_num 908) dirty exists 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2557: left oid 34 (ObjNum 908 snap 276 seq_num 908) 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: finishing write tid 1 to smithi20328086-37 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: finishing write tid 2 to smithi20328086-37 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: finishing write tid 3 to smithi20328086-37 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: finishing write tid 4 to smithi20328086-37 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: finishing write tid 5 to smithi20328086-37 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: oid 37 updating version 0 to 1258 2024-09-17T15:43:24.601 INFO:tasks.rados.rados.0.smithi203.stdout:2558: oid 37 updating version 1258 to 1260 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2558: oid 37 updating version 1260 to 1261 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2558: oid 37 version 1261 is already newer than 1259 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1261 (ObjNum 909 snap 276 seq_num 909) dirty exists 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2558: left oid 37 (ObjNum 909 snap 276 seq_num 909) 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2559: finishing rollback tid 0 to smithi20328086-3 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2559: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2559: finishing rollback tid 2 to smithi20328086-3 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1193 (ObjNum 846 snap 260 seq_num 846) dirty exists 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2555: done (4 left) 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2556: done (3 left) 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2557: done (2 left) 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2558: done (1 left) 2024-09-17T15:43:24.602 INFO:tasks.rados.rados.0.smithi203.stdout:2559: done (0 left) 2024-09-17T15:43:24.603 INFO:tasks.rados.rados.0.smithi203.stdout:2560: read oid 47 snap 254 2024-09-17T15:43:24.603 INFO:tasks.rados.rados.0.smithi203.stdout:2560: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:43:24.603 INFO:tasks.rados.rados.0.smithi203.stdout:2561: snap_remove snap 272 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:2561: done (1 left) 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:2562: read oid 24 snap -1 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:2562: expect (ObjNum 889 snap 271 seq_num 889) 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:2563: delete oid 15 current snap is 276 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:2560: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:43:25.237 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:25.541 INFO:tasks.rados.rados.0.smithi203.stdout:2562: expect (ObjNum 889 snap 271 seq_num 889) 2024-09-17T15:43:25.624 INFO:tasks.rados.rados.0.smithi203.stdout:2560: done (2 left) 2024-09-17T15:43:25.624 INFO:tasks.rados.rados.0.smithi203.stdout:2562: done (1 left) 2024-09-17T15:43:25.624 INFO:tasks.rados.rados.0.smithi203.stdout:2563: done (0 left) 2024-09-17T15:43:25.624 INFO:tasks.rados.rados.0.smithi203.stdout:2564: delete oid 35 current snap is 276 2024-09-17T15:43:25.631 INFO:tasks.rados.rados.0.smithi203.stdout:2564: done (0 left) 2024-09-17T15:43:25.631 INFO:tasks.rados.rados.0.smithi203.stdout:2565: write oid 50 current snap is 276 2024-09-17T15:43:25.631 INFO:tasks.rados.rados.0.smithi203.stdout:2565: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-09-17T15:43:25.647 INFO:tasks.rados.rados.0.smithi203.stdout:2565: writing smithi20328086-50 from 527984 to 1001079 tid 1 2024-09-17T15:43:25.665 INFO:tasks.rados.rados.0.smithi203.stdout:2565: writing smithi20328086-50 from 1634872 to 2086522 tid 2 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:2565: writing smithi20328086-50 from 2661887 to 3296198 tid 3 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:2566: rollback oid 17 current snap is 276 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 264 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:2565: finishing write tid 1 to smithi20328086-50 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:2565: finishing write tid 2 to smithi20328086-50 2024-09-17T15:43:25.684 INFO:tasks.rados.rados.0.smithi203.stdout:2567: write (excl) oid 5 current snap is 276 2024-09-17T15:43:25.685 INFO:tasks.rados.rados.0.smithi203.stdout:2567: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-17T15:43:25.691 INFO:tasks.rados.rados.0.smithi203.stdout:2567: writing smithi20328086-5 from 544791 to 1209488 tid 1 2024-09-17T15:43:25.701 INFO:tasks.rados.rados.0.smithi203.stdout:2567: writing smithi20328086-5 from 1693354 to 2288662 tid 2 2024-09-17T15:43:25.709 INFO:tasks.rados.rados.0.smithi203.stdout:2567: writing smithi20328086-5 from 2689310 to 3134893 tid 3 2024-09-17T15:43:25.709 INFO:tasks.rados.rados.0.smithi203.stdout:2568: rollback oid 12 current snap is 276 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 264 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: finishing write tid 3 to smithi20328086-50 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: finishing write tid 4 to smithi20328086-50 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2569: read oid 38 snap -1 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2569: expect (ObjNum 821 snap 254 seq_num 821) 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2570: rollback oid 13 current snap is 276 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 254 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2571: write (excl) oid 37 current snap is 276 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: finishing write tid 5 to smithi20328086-50 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: oid 50 updating version 0 to 1175 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: oid 50 updating version 1175 to 1177 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: oid 50 version 1177 is already newer than 1176 2024-09-17T15:43:25.710 INFO:tasks.rados.rados.0.smithi203.stdout:2565: oid 50 version 1177 is already newer than 1174 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1177 (ObjNum 912 snap 276 seq_num 912) dirty exists 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2565: left oid 50 (ObjNum 912 snap 276 seq_num 912) 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2566: finishing rollback tid 0 to smithi20328086-17 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2566: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2566: finishing rollback tid 2 to smithi20328086-17 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1184 (ObjNum 818 snap 254 seq_num 818) dirty exists 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2567: finishing write tid 1 to smithi20328086-5 2024-09-17T15:43:25.711 INFO:tasks.rados.rados.0.smithi203.stdout:2571: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-09-17T15:43:25.719 INFO:tasks.rados.rados.0.smithi203.stdout:2571: writing smithi20328086-37 from 561598 to 1301544 tid 1 2024-09-17T15:43:25.726 INFO:tasks.rados.rados.0.smithi203.stdout:2571: writing smithi20328086-37 from 1751836 to 2207155 tid 2 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2571: writing smithi20328086-37 from 2716733 to 3361598 tid 3 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2565: done (6 left) 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2566: done (5 left) 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2572: delete oid 44 current snap is 276 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2567: finishing write tid 2 to smithi20328086-5 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2568: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2568: finishing rollback tid 2 to smithi20328086-12 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1254 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:43:25.734 INFO:tasks.rados.rados.0.smithi203.stdout:2567: finishing write tid 3 to smithi20328086-5 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: finishing write tid 4 to smithi20328086-5 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: finishing write tid 5 to smithi20328086-5 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: oid 5 updating version 0 to 1086 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: oid 5 version 1086 is already newer than 1085 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: oid 5 updating version 1086 to 1087 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: oid 5 version 1087 is already newer than 1084 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1087 (ObjNum 913 snap 276 seq_num 913) dirty exists 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2567: left oid 5 (ObjNum 913 snap 276 seq_num 913) 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:2569: expect (ObjNum 821 snap 254 seq_num 821) 2024-09-17T15:43:25.735 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 490672 2024-09-17T15:43:25.877 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1821979 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2570: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2570: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1117 (ObjNum 129 snap 0 seq_num 3606750664) dirty dne 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2571: finishing write tid 1 to smithi20328086-37 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2571: finishing write tid 2 to smithi20328086-37 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2567: done (5 left) 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2568: done (4 left) 2024-09-17T15:43:25.914 INFO:tasks.rados.rados.0.smithi203.stdout:2569: done (3 left) 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2570: done (2 left) 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2572: done (1 left) 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2573: copy_from oid 42 from oid 21 current snap is 276 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: finishing write tid 3 to smithi20328086-37 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: finishing write tid 4 to smithi20328086-37 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: finishing write tid 5 to smithi20328086-37 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: oid 37 updating version 0 to 1267 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: oid 37 updating version 1267 to 1269 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: oid 37 version 1269 is already newer than 1268 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: oid 37 version 1269 is already newer than 1266 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1269 (ObjNum 914 snap 276 seq_num 914) dirty exists 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: left oid 37 (ObjNum 914 snap 276 seq_num 914) 2024-09-17T15:43:25.915 INFO:tasks.rados.rados.0.smithi203.stdout:2571: done (1 left) 2024-09-17T15:43:25.916 INFO:tasks.rados.rados.0.smithi203.stdout:2574: delete oid 22 current snap is 276 2024-09-17T15:43:25.918 INFO:tasks.rados.rados.0.smithi203.stdout:2574: done (1 left) 2024-09-17T15:43:25.918 INFO:tasks.rados.rados.0.smithi203.stdout:2575: write (excl) oid 15 current snap is 276 2024-09-17T15:43:25.918 INFO:tasks.rados.rados.0.smithi203.stdout:2575: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-17T15:43:25.925 INFO:tasks.rados.rados.0.smithi203.stdout:2575: writing smithi20328086-15 from 612019 to 1177712 tid 1 2024-09-17T15:43:25.931 INFO:tasks.rados.rados.0.smithi203.stdout:2575: writing smithi20328086-15 from 1810929 to 2413575 tid 2 2024-09-17T15:43:25.940 INFO:tasks.rados.rados.0.smithi203.stdout:2575: writing smithi20328086-15 from 3082649 to 3412019 tid 3 2024-09-17T15:43:25.940 INFO:tasks.rados.rados.0.smithi203.stdout:2576: read oid 30 snap -1 2024-09-17T15:43:25.940 INFO:tasks.rados.rados.0.smithi203.stdout:2576: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:25.941 INFO:tasks.rados.rados.0.smithi203.stdout:2577: snap_remove snap 264 2024-09-17T15:43:26.274 INFO:tasks.rados.rados.0.smithi203.stdout:2577: done (3 left) 2024-09-17T15:43:26.274 INFO:tasks.rados.rados.0.smithi203.stdout:2578: snap_remove snap 271 2024-09-17T15:43:26.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:26.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: get_monmap_and_config 2024-09-17T15:43:26.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: build_initial_monmap 2024-09-17T15:43:26.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: monmap: 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:26.757305+0000 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:26.757305+0000 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:26.752 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 Processor -- start 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 -- start start 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: init 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99100a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f99100a54c0 0x7f99100a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-c con 0x7f99100a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:26.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f991013d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99100a6530 con 0x7f991013d2f0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f991009c150 con 0x7f99100a50e0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f991009c430 con 0x7f99100a54c0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f99177c1640 10 monclient(hunting): authenticate will time out at 1939.723623s 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f9915d5e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f991013d2f0 0x7f991013f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50698/0 (socket says 172.21.15.86:50698) 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.754+0000 7f9915d5e640 1 -- 172.21.15.86:0/1932963791 learned_addr learned my addr 172.21.15.86:0/1932963791 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1932963791 >> v2:172.21.15.203:3300/0 conn(0x7f99100a54c0 0x7f99100a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): get_auth_request con 0x7f99100a54c0 auth_method 0 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:26.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient(hunting): handle_auth_done global_id 174166 payload 274 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 1 -- 172.21.15.86:0/1932963791 >> v1:172.21.15.203:6789/0 conn(0x7f99100a50e0 legacy=0x7f991009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 1 -- 172.21.15.86:0/1932963791 >> v1:172.21.15.86:6789/0 conn(0x7f991013d2f0 legacy=0x7f991013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient: _finish_hunting 0 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 1 monclient: found mon.noname-c 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:43:26.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.758+0000 7f9914d5c640 1 -- 172.21.15.86:0/1932963791 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99100a6940 con 0x7f99100a54c0 2024-09-17T15:43:26.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9914d5c640 1 -- 172.21.15.86:0/1932963791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99100a54c0 msgr2=0x7f99100a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:26.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1932963791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99100a54c0 0x7f99100a58d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99000097c0 tx=0x7f990001c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9a50d83dfa3e3e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 1 -- 172.21.15.86:0/1932963791 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000b0070 con 0x7f99100a54c0 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: dump: 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:26.760 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: _finish_auth 0 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 1 -- 172.21.15.86:0/1932963791 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99000ab600 con 0x7f99100a54c0 2024-09-17T15:43:26.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 1 -- 172.21.15.86:0/1932963791 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99000ab9d0 con 0x7f99100a54c0 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f9907fff640 10 monclient: dump: 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:26.762 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 5 monclient: authenticate success, global_id 174166 2024-09-17T15:43:26.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:26.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 4 set_mon_vals no callback set 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 10 monclient: shutdown 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 -- 172.21.15.86:0/1932963791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99100a54c0 msgr2=0x7f99100a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 --2- 172.21.15.86:0/1932963791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99100a54c0 0x7f99100a58d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99000097c0 tx=0x7f990001c3f0 comp rx=0 tx=0).stop 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 -- 172.21.15.86:0/1932963791 shutdown_connections 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 --2- 172.21.15.86:0/1932963791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99100a54c0 0x7f99100a58d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 -- 172.21.15.86:0/1932963791 >> 172.21.15.86:0/1932963791 conn(0x7f99100aaae0 msgr2=0x7f99100aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 -- 172.21.15.86:0/1932963791 shutdown_connections 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 1 -- 172.21.15.86:0/1932963791 wait complete. 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 10 monclient: build_initial_monmap 2024-09-17T15:43:26.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.762+0000 7f99177c1640 10 monclient: monmap: 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 Processor -- start 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 -- start start 2024-09-17T15:43:26.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient: init 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 0x7f99101cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-c con 0x7f99100a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-a con 0x7f99100a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 0x7f99100a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): picked mon.noname-b con 0x7f991013d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9910143a50 con 0x7f99100a54c0 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:26.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f99101438d0 con 0x7f991013d2f0 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9910143bd0 con 0x7f99100a50e0 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient(hunting): authenticate will time out at 1939.732759s 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 0x7f99101cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:33186/0 (socket says 172.21.15.86:33186) 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 -- 172.21.15.86:0/1146401183 learned_addr learned my addr 172.21.15.86:0/1146401183 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 10 monclient(hunting): get_auth_request con 0x7f99100a54c0 auth_method 0 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:26.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): get_auth_request con 0x7f99100a50e0 auth_method 0 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 msgr2=0x7f9910138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 msgr2=0x7f9910138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 0x7f99100a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 10 monclient(hunting): get_auth_request con 0x7f991013d2f0 auth_method 0 2024-09-17T15:43:26.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient(hunting): handle_auth_done global_id 164300 payload 824 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 msgr2=0x7f99100a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 0x7f99100a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 msgr2=0x7f9910138110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient: _finish_hunting 0 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 monclient: found mon.noname-c 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 -- 172.21.15.86:0/1146401183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99100a3630 con 0x7f99100a50e0 2024-09-17T15:43:26.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9915d5e640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 0x7f99100a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 msgr2=0x7f99101cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f991555d640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 0x7f99101cfe20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f990c00c730 tx=0x7f990c00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c016020 con 0x7f99100a50e0 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 10 monclient: dump: 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.774 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990c00ac60 con 0x7f99100a50e0 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 5 monclient: authenticate success, global_id 164300 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient: _renew_subs 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.766+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99101211f0 con 0x7f99100a50e0 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f990c012660 con 0x7f99100a50e0 2024-09-17T15:43:26.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 10 monclient: dump: 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99077fe640 4 set_mon_vals no callback set 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 10 monclient: _renew_subs 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99100a18d0 con 0x7f99100a50e0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99101213c0 con 0x7f99100a50e0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f990c019020 con 0x7f99100a50e0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 0x7f98f403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(571..571 src has 438..571) ==== 4799+0+0 (secure 0 0 0) 0x7f990c050250 con 0x7f99100a50e0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 0x7f98f403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:26.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.770+0000 7f9914d5c640 10 monclient: get_auth_request con 0x7f98f403bb60 auth_method 0 2024-09-17T15:43:26.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.774+0000 7f9914d5c640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 0x7f98f403e020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99000097c0 tx=0x7f99000a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:26.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.774+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f990c020020 con 0x7f99100a50e0 2024-09-17T15:43:26.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.774+0000 7f9905ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:26.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:26.774+0000 7f9905ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f99177c1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f99177c1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:27.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f99177c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f99100a58d0 con 0x7f99100a50e0 2024-09-17T15:43:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f9905ffb640 1 -- 172.21.15.86:0/1146401183 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f990c00ae10 con 0x7f99100a50e0 2024-09-17T15:43:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f9905ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:27.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.010+0000 7f9905ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:27.009 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":146,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 msgr2=0x7f98f403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 0x7f98f403e020 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99000097c0 tx=0x7f99000a6000 comp rx=0 tx=0).stop 2024-09-17T15:43:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 10 monclient: shutdown 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 msgr2=0x7f99101cfe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 0x7f99101cfe20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f990c00c730 tx=0x7f990c00cbf0 comp rx=0 tx=0).stop 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 shutdown_connections 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f98f403bb60 0x7f98f403e020 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f991013d2f0 0x7f99100a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99100a54c0 0x7f9910138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 --2- 172.21.15.86:0/1146401183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99100a50e0 0x7f99101cfe20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 >> 172.21.15.86:0/1146401183 conn(0x7f99100aaae0 msgr2=0x7f99100a0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 shutdown_connections 2024-09-17T15:43:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:27.014+0000 7f99177c1640 1 -- 172.21.15.86:0/1146401183 wait complete. 2024-09-17T15:43:27.287 INFO:tasks.rados.rados.0.smithi203.stdout:2578: done (3 left) 2024-09-17T15:43:27.287 INFO:tasks.rados.rados.0.smithi203.stdout:2579: read oid 20 snap -1 2024-09-17T15:43:27.287 INFO:tasks.rados.rados.0.smithi203.stdout:2579: expect (ObjNum 902 snap 273 seq_num 902) 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2580: delete oid 12 current snap is 276 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2575: finishing write tid 1 to smithi20328086-15 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2575: finishing write tid 2 to smithi20328086-15 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2573: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2573: finishing copy_from to smithi20328086-42 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1337 (ObjNum 897 snap 272 seq_num 897) dirty exists 2024-09-17T15:43:27.288 INFO:tasks.rados.rados.0.smithi203.stdout:2576: expect (ObjNum 700 snap 220 seq_num 700) 2024-09-17T15:43:27.377 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: finishing write tid 3 to smithi20328086-15 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: finishing write tid 4 to smithi20328086-15 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: finishing write tid 5 to smithi20328086-15 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: oid 15 updating version 0 to 1257 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: oid 15 version 1257 is already newer than 1255 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: oid 15 version 1257 is already newer than 1256 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:2575: oid 15 updating version 1257 to 1258 2024-09-17T15:43:27.553 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1258 (ObjNum 917 snap 276 seq_num 917) dirty exists 2024-09-17T15:43:27.554 INFO:tasks.rados.rados.0.smithi203.stdout:2575: left oid 15 (ObjNum 917 snap 276 seq_num 917) 2024-09-17T15:43:27.554 INFO:tasks.rados.rados.0.smithi203.stdout:2579: expect (ObjNum 902 snap 273 seq_num 902) 2024-09-17T15:43:27.554 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 232508 2024-09-17T15:43:27.603 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:27.695 INFO:tasks.rados.rados.0.smithi203.stdout:2573: done (4 left) 2024-09-17T15:43:27.695 INFO:tasks.rados.rados.0.smithi203.stdout:2575: done (3 left) 2024-09-17T15:43:27.695 INFO:tasks.rados.rados.0.smithi203.stdout:2576: done (2 left) 2024-09-17T15:43:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:2579: done (1 left) 2024-09-17T15:43:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:2580: done (0 left) 2024-09-17T15:43:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:2581: snap_create 2024-09-17T15:43:28.299 INFO:tasks.rados.rados.0.smithi203.stdout:2581: done (0 left) 2024-09-17T15:43:28.299 INFO:tasks.rados.rados.0.smithi203.stdout:2582: write (excl) oid 22 current snap is 277 2024-09-17T15:43:28.300 INFO:tasks.rados.rados.0.smithi203.stdout:2582: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-17T15:43:28.321 INFO:tasks.rados.rados.0.smithi203.stdout:2582: writing smithi20328086-22 from 645633 to 1361824 tid 1 2024-09-17T15:43:28.341 INFO:tasks.rados.rados.0.smithi203.stdout:2582: writing smithi20328086-22 from 1927893 to 2534208 tid 2 2024-09-17T15:43:28.354 INFO:tasks.rados.rados.0.smithi203.stdout:2582: writing smithi20328086-22 from 3137495 to 3445633 tid 3 2024-09-17T15:43:28.354 INFO:tasks.rados.rados.0.smithi203.stdout:2583: rollback oid 40 current snap is 277 2024-09-17T15:43:28.355 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 276 2024-09-17T15:43:28.355 INFO:tasks.rados.rados.0.smithi203.stdout:2582: finishing write tid 1 to smithi20328086-22 2024-09-17T15:43:28.355 INFO:tasks.rados.rados.0.smithi203.stdout:2582: finishing write tid 2 to smithi20328086-22 2024-09-17T15:43:28.355 INFO:tasks.rados.rados.0.smithi203.stdout:2584: snap_remove snap 254 2024-09-17T15:43:29.307 INFO:tasks.rados.rados.0.smithi203.stdout:2584: done (2 left) 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2585: delete oid 20 current snap is 277 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: finishing write tid 3 to smithi20328086-22 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: finishing write tid 4 to smithi20328086-22 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: finishing write tid 5 to smithi20328086-22 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: oid 22 updating version 0 to 1246 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: oid 22 updating version 1246 to 1248 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: oid 22 updating version 1248 to 1249 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: oid 22 version 1249 is already newer than 1247 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1249 (ObjNum 919 snap 277 seq_num 919) dirty exists 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2582: left oid 22 (ObjNum 919 snap 277 seq_num 919) 2024-09-17T15:43:29.308 INFO:tasks.rados.rados.0.smithi203.stdout:2583: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:43:29.309 INFO:tasks.rados.rados.0.smithi203.stdout:2583: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:43:29.309 INFO:tasks.rados.rados.0.smithi203.stdout:2583: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:43:29.309 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1188 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:43:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:2582: done (2 left) 2024-09-17T15:43:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:2583: done (1 left) 2024-09-17T15:43:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:2585: done (0 left) 2024-09-17T15:43:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:2586: write (excl) oid 17 current snap is 277 2024-09-17T15:43:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:2586: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-17T15:43:29.355 INFO:tasks.rados.rados.0.smithi203.stdout:2586: writing smithi20328086-17 from 679247 to 1262289 tid 1 2024-09-17T15:43:29.374 INFO:tasks.rados.rados.0.smithi203.stdout:2586: writing smithi20328086-17 from 2044857 to 2654841 tid 2 2024-09-17T15:43:29.384 INFO:tasks.rados.rados.0.smithi203.stdout:2586: writing smithi20328086-17 from 3192341 to 3479247 tid 3 2024-09-17T15:43:29.384 INFO:tasks.rados.rados.0.smithi203.stdout:2587: write (excl) oid 22 current snap is 277 2024-09-17T15:43:29.384 INFO:tasks.rados.rados.0.smithi203.stdout:2586: finishing write tid 1 to smithi20328086-17 2024-09-17T15:43:29.385 INFO:tasks.rados.rados.0.smithi203.stdout:2586: finishing write tid 2 to smithi20328086-17 2024-09-17T15:43:29.385 INFO:tasks.rados.rados.0.smithi203.stdout:2587: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-17T15:43:29.399 INFO:tasks.rados.rados.0.smithi203.stdout:2587: writing smithi20328086-22 from 696054 to 1354345 tid 1 2024-09-17T15:43:29.416 INFO:tasks.rados.rados.0.smithi203.stdout:2587: writing smithi20328086-22 from 2103339 to 2856981 tid 2 2024-09-17T15:43:29.416 INFO:tasks.rados.rados.0.smithi203.stdout:2588: snap_remove snap 275 2024-09-17T15:43:29.531 INFO:tasks.rados.rados.0.smithi203.stdout:2588: done (2 left) 2024-09-17T15:43:29.531 INFO:tasks.rados.rados.0.smithi203.stdout:2589: copy_from oid 19 from oid 26 current snap is 277 2024-09-17T15:43:29.531 INFO:tasks.rados.rados.0.smithi203.stdout:2586: finishing write tid 3 to smithi20328086-17 2024-09-17T15:43:29.531 INFO:tasks.rados.rados.0.smithi203.stdout:2586: finishing write tid 4 to smithi20328086-17 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: finishing write tid 5 to smithi20328086-17 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: oid 17 updating version 0 to 1191 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: oid 17 updating version 1191 to 1192 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: oid 17 version 1192 is already newer than 1190 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: oid 17 version 1192 is already newer than 1189 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1192 (ObjNum 921 snap 277 seq_num 921) dirty exists 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2586: left oid 17 (ObjNum 921 snap 277 seq_num 921) 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: finishing write tid 1 to smithi20328086-22 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: finishing write tid 2 to smithi20328086-22 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: finishing write tid 3 to smithi20328086-22 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: finishing write tid 4 to smithi20328086-22 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: oid 22 updating version 0 to 1251 2024-09-17T15:43:29.532 INFO:tasks.rados.rados.0.smithi203.stdout:2587: oid 22 updating version 1251 to 1252 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2587: oid 22 version 1252 is already newer than 1250 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1252 (ObjNum 922 snap 277 seq_num 922) dirty exists 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2587: left oid 22 (ObjNum 922 snap 277 seq_num 922) 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2586: done (2 left) 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2587: done (1 left) 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2590: copy_from oid 30 from oid 10 current snap is 277 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2591: read oid 38 snap 276 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2591: expect (ObjNum 821 snap 254 seq_num 821) 2024-09-17T15:43:29.533 INFO:tasks.rados.rados.0.smithi203.stdout:2592: snap_create 2024-09-17T15:43:29.538 INFO:tasks.rados.rados.0.smithi203.stdout:2589: finishing copy_from to smithi20328086-19 2024-09-17T15:43:29.538 INFO:tasks.rados.rados.0.smithi203.stdout:2589: got expected ENOENT (src dne) 2024-09-17T15:43:29.538 INFO:tasks.rados.rados.0.smithi203.stdout:2589: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:43:29.548 INFO:tasks.rados.rados.0.smithi203.stdout:2591: expect (ObjNum 821 snap 254 seq_num 821) 2024-09-17T15:43:29.767 INFO:tasks.rados.rados.0.smithi203.stdout:2590: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:43:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:2590: finishing copy_from to smithi20328086-30 2024-09-17T15:43:29.768 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1108 (ObjNum 792 snap 246 seq_num 792) dirty exists 2024-09-17T15:43:30.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:30.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: get_monmap_and_config 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: monmap: 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:30.165507+0000 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:30.165507+0000 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:30.160 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:30.161 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:30.161 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:30.161 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 Processor -- start 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 -- start start 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: init 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a340689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a34068db0 0x7f0a340691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-a con 0x7f0a34068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-b con 0x7f0a34139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a3409d770 con 0x7f0a340689d0 2024-09-17T15:43:30.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0a3409ed00 con 0x7f0a34068db0 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3409efe0 con 0x7f0a34139f00 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a399bc640 10 monclient(hunting): authenticate will time out at 1943.130976s 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a34068db0 0x7f0a340691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36404/0 (socket says 172.21.15.86:36404) 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 1 -- 172.21.15.86:0/4015717986 learned_addr learned my addr 172.21.15.86:0/4015717986 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): get_auth_request con 0x7f0a34139f00 auth_method 0 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): get_auth_request con 0x7f0a34068db0 auth_method 0 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient(hunting): handle_auth_done global_id 184098 payload 274 2024-09-17T15:43:30.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 -- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 msgr2=0x7f0a3413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 -- 172.21.15.86:0/4015717986 >> v1:172.21.15.203:6789/0 conn(0x7f0a340689d0 legacy=0x7f0a3409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a327fc640 1 -- 172.21.15.86:0/4015717986 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a34069700 con 0x7f0a34068db0 2024-09-17T15:43:30.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.162+0000 7f0a337fe640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:30.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a327fc640 1 -- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34068db0 msgr2=0x7f0a340691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a327fc640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34068db0 0x7f0a340691c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097c0 tx=0x7f0a2801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529427242edc17b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 1 -- 172.21.15.86:0/4015717986 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280b0070 con 0x7f0a34068db0 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 10 monclient: dump: 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.173 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.166+0000 7f0a31ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 1 -- 172.21.15.86:0/4015717986 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a280ab600 con 0x7f0a34068db0 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 5 monclient: authenticate success, global_id 184098 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:30.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 1 -- 172.21.15.86:0/4015717986 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280ab9d0 con 0x7f0a34068db0 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 4 set_mon_vals no callback set 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a31ffb640 10 monclient: dump: 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.175 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: shutdown 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34068db0 msgr2=0x7f0a340691c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34068db0 0x7f0a340691c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a280097c0 tx=0x7f0a2801c3f0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- 172.21.15.86:0/4015717986 shutdown_connections 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34139f00 0x7f0a3413c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- 172.21.15.86:0/4015717986 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34068db0 0x7f0a340691c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- 172.21.15.86:0/4015717986 >> 172.21.15.86:0/4015717986 conn(0x7f0a340a6440 msgr2=0x7f0a340a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- 172.21.15.86:0/4015717986 shutdown_connections 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- 172.21.15.86:0/4015717986 wait complete. 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: monmap: 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 Processor -- start 2024-09-17T15:43:30.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- start start 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: init 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 0x7f0a34134e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-b con 0x7f0a340689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34068db0 0x7f0a34135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-c con 0x7f0a34068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): picked mon.noname-a con 0x7f0a34139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3409c610 con 0x7f0a34139f00 2024-09-17T15:43:30.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a3409c490 con 0x7f0a340689d0 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3409c790 con 0x7f0a34068db0 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a399bc640 10 monclient(hunting): authenticate will time out at 1943.138146s 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34068db0 0x7f0a34135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55336/0 (socket says 172.21.15.86:55336) 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 -- 172.21.15.86:0/2838630007 learned_addr learned my addr 172.21.15.86:0/2838630007 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): get_auth_request con 0x7f0a34139f00 auth_method 0 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): get_auth_request con 0x7f0a34068db0 auth_method 0 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a32ffd640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 0x7f0a34134e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient(hunting): handle_auth_done global_id 184101 payload 824 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 -- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34068db0 msgr2=0x7f0a34135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34068db0 0x7f0a34135380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 -- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 msgr2=0x7f0a34134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 0x7f0a34134e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a337fe640 1 -- 172.21.15.86:0/2838630007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3409d460 con 0x7f0a34139f00 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a32ffd640 10 monclient: get_auth_request con 0x7f0a340689d0 auth_method 0 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.170+0000 7f0a32ffd640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 0x7f0a34134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a337fe640 1 -- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 msgr2=0x7f0a3411ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a337fe640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400ecc0 tx=0x7f0a2400d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24010850 con 0x7f0a34139f00 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: dump: 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24010a10 con 0x7f0a34139f00 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 5 monclient: authenticate success, global_id 184101 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _renew_subs 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a340a0c70 con 0x7f0a34139f00 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24010e00 con 0x7f0a34139f00 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 10 monclient: dump: 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.183 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a317fa640 4 set_mon_vals no callback set 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _renew_subs 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a34122820 con 0x7f0a34139f00 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0a24018400 con 0x7f0a34139f00 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a340691c0 con 0x7f0a34139f00 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 0x7f0a2003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(575..575 src has 438..575) ==== 4815+0+0 (secure 0 0 0) 0x7f0a24050920 con 0x7f0a34139f00 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a32ffd640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 0x7f0a2003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.174+0000 7f0a32ffd640 10 monclient: get_auth_request con 0x7f0a2003bbe0 auth_method 0 2024-09-17T15:43:30.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.178+0000 7f0a32ffd640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 0x7f0a2003e0a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c007fd0 tx=0x7f0a2c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.178+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0a3409d460 con 0x7f0a34139f00 2024-09-17T15:43:30.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.178+0000 7f0a1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.178+0000 7f0a1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:30.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a399bc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:30.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a399bc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:30.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a399bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0a3406d4f0 con 0x7f0a34139f00 2024-09-17T15:43:30.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a1bfff640 1 -- 172.21.15.86:0/2838630007 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0a24014030 con 0x7f0a34139f00 2024-09-17T15:43:30.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:30.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.414+0000 7f0a1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:30.413 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 msgr2=0x7f0a2003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 0x7f0a2003e0a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c007fd0 tx=0x7f0a2c009040 comp rx=0 tx=0).stop 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 10 monclient: shutdown 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 msgr2=0x7f0a3411ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0a2400ecc0 tx=0x7f0a2400d720 comp rx=0 tx=0).stop 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 shutdown_connections 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a2003bbe0 0x7f0a2003e0a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a34139f00 0x7f0a3411ea00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a34068db0 0x7f0a34135380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 --2- 172.21.15.86:0/2838630007 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a340689d0 0x7f0a34134e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 >> 172.21.15.86:0/2838630007 conn(0x7f0a340a6440 msgr2=0x7f0a341c35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 shutdown_connections 2024-09-17T15:43:30.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.418+0000 7f0a399bc640 1 -- 172.21.15.86:0/2838630007 wait complete. 2024-09-17T15:43:30.432 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:43:30.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:43:30.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2589: done (3 left) 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2590: done (2 left) 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2591: done (1 left) 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2592: done (0 left) 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2593: read oid 11 snap -1 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2593: expect deleted 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2594: read oid 41 snap -1 2024-09-17T15:43:30.542 INFO:tasks.rados.rados.0.smithi203.stdout:2594: expect deleted 2024-09-17T15:43:30.543 INFO:tasks.rados.rados.0.smithi203.stdout:2595: read oid 13 snap -1 2024-09-17T15:43:30.543 INFO:tasks.rados.rados.0.smithi203.stdout:2595: expect deleted 2024-09-17T15:43:30.543 INFO:tasks.rados.rados.0.smithi203.stdout:2596: write oid 32 current snap is 278 2024-09-17T15:43:30.543 INFO:tasks.rados.rados.0.smithi203.stdout:2596: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-17T15:43:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: get_monmap_and_config 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: monmap: 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:30.556716+0000 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:30.556716+0000 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:30.552 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 Processor -- start 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 -- start start 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: init 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40dc0a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f40dc0a1760 0x7f40dc13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-a con 0x7f40dc0a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f40dc13b450 0x7f40dc13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-c con 0x7f40dc13b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40dc0a2130 con 0x7f40dc0a0c70 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f40dc0a2410 con 0x7f40dc0a1760 2024-09-17T15:43:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f40dc0a26f0 con 0x7f40dc13b450 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e3e22640 10 monclient(hunting): authenticate will time out at 1943.522261s 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f40dc0a1760 0x7f40dc13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f40dc0a1760 0x7f40dc13af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55350/0 (socket says 172.21.15.86:55350) 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 -- 172.21.15.86:0/240590071 learned_addr learned my addr 172.21.15.86:0/240590071 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): get_auth_request con 0x7f40dc0a1760 auth_method 0 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e23bf640 1 --2- 172.21.15.86:0/240590071 >> v2:172.21.15.203:3300/0 conn(0x7f40dc13b450 0x7f40dc13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e23bf640 10 monclient(hunting): get_auth_request con 0x7f40dc13b450 auth_method 0 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e23bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e23bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e23bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient(hunting): handle_auth_done global_id 184104 payload 274 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 -- 172.21.15.86:0/240590071 >> v2:172.21.15.203:3300/0 conn(0x7f40dc13b450 msgr2=0x7f40dc13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 --2- 172.21.15.86:0/240590071 >> v2:172.21.15.203:3300/0 conn(0x7f40dc13b450 0x7f40dc13d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 -- 172.21.15.86:0/240590071 >> v1:172.21.15.203:6789/0 conn(0x7f40dc0a0c70 legacy=0x7f40dc0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.554+0000 7f40e13bd640 1 -- 172.21.15.86:0/240590071 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc09f9f0 con 0x7f40dc0a1760 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e13bd640 1 -- 172.21.15.86:0/240590071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a1760 msgr2=0x7f40dc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e13bd640 1 --2- 172.21.15.86:0/240590071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a1760 0x7f40dc13af10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f40dc06f840 tx=0x7f40cc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1321e067e1e3a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 1 -- 172.21.15.86:0/240590071 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0b1070 con 0x7f40dc0a1760 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: dump: 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.558 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 1 -- 172.21.15.86:0/240590071 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40cc0ac620 con 0x7f40dc0a1760 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 1 -- 172.21.15.86:0/240590071 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0ac9f0 con 0x7f40dc0a1760 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e3e22640 5 monclient: authenticate success, global_id 184104 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e3e22640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e3e22640 4 set_mon_vals no callback set 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.558+0000 7f40e0bbc640 10 monclient: dump: 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.561 INFO:tasks.rados.rados.0.smithi203.stdout:2596: writing smithi20328086-32 from 712861 to 1446401 tid 1 2024-09-17T15:43:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: shutdown 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- 172.21.15.86:0/240590071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a1760 msgr2=0x7f40dc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- 172.21.15.86:0/240590071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a1760 0x7f40dc13af10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f40dc06f840 tx=0x7f40cc01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- 172.21.15.86:0/240590071 shutdown_connections 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- 172.21.15.86:0/240590071 >> v2:172.21.15.203:3300/0 conn(0x7f40dc13b450 0x7f40dc13d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- 172.21.15.86:0/240590071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a1760 0x7f40dc13af10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- 172.21.15.86:0/240590071 >> 172.21.15.86:0/240590071 conn(0x7f40dc0a2f40 msgr2=0x7f40dc0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- 172.21.15.86:0/240590071 shutdown_connections 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- 172.21.15.86:0/240590071 wait complete. 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: monmap: 2024-09-17T15:43:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 Processor -- start 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- start start 2024-09-17T15:43:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: init 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 0x7f40dc1cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-a con 0x7f40dc0a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40dc0a1760 0x7f40dc1d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-b con 0x7f40dc0a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40dc13b450 0x7f40dc1d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): picked mon.noname-c con 0x7f40dc13b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc140720 con 0x7f40dc0a0c70 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc1405a0 con 0x7f40dc0a1760 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f40dc1408a0 con 0x7f40dc13b450 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e3e22640 10 monclient(hunting): authenticate will time out at 1943.530148s 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 0x7f40dc1cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 0x7f40dc1cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55352/0 (socket says 172.21.15.86:55352) 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 -- 172.21.15.86:0/2746340016 learned_addr learned my addr 172.21.15.86:0/2746340016 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): get_auth_request con 0x7f40dc0a0c70 auth_method 0 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40dc13b450 0x7f40dc1d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): get_auth_request con 0x7f40dc13b450 auth_method 0 2024-09-17T15:43:30.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40dc0a1760 0x7f40dc1d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 10 monclient(hunting): get_auth_request con 0x7f40dc0a1760 auth_method 0 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e23bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient(hunting): handle_auth_done global_id 184107 payload 824 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40dc13b450 msgr2=0x7f40dc1d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40dc13b450 0x7f40dc1d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40dc0a1760 msgr2=0x7f40dc1d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40dc0a1760 0x7f40dc1d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e1bbe640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40dc13ba30 con 0x7f40dc0a0c70 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.562+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40dc0a1760 0x7f40dc1d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e1bbe640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 msgr2=0x7f40dc1cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e1bbe640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 0x7f40dc1cfd30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f40d800c450 tx=0x7f40d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8012a40 con 0x7f40dc0a0c70 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: dump: 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d8017070 con 0x7f40dc0a0c70 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40d8012e40 con 0x7f40dc0a0c70 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 10 monclient: dump: 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d3fff640 4 set_mon_vals no callback set 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 5 monclient: authenticate success, global_id 184107 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 monclient: _renew_subs 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40dc09d460 con 0x7f40dc0a0c70 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 monclient: _renew_subs 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40dc134470 con 0x7f40dc0a0c70 2024-09-17T15:43:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f40d801c460 con 0x7f40dc0a0c70 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 0x7f40dc12e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc134700 con 0x7f40dc1363d0 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 0x7f40dc12e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f40c003bbb0 0x7f40c003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient: get_auth_request con 0x7f40dc1363d0 auth_method 0 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(576..576 src has 438..576) ==== 4787+0+0 (secure 0 0 0) 0x7f40d804fc60 con 0x7f40dc0a0c70 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 10 monclient(hunting): handle_auth_done global_id 184110 payload 824 2024-09-17T15:43:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e13bd640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40dc0a1050 con 0x7f40dc1363d0 2024-09-17T15:43:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e23bf640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f40c003bbb0 0x7f40c003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.566+0000 7f40e23bf640 10 monclient: get_auth_request con 0x7f40c003bbb0 auth_method 0 2024-09-17T15:43:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:2596: writing smithi20328086-32 from 2161821 to 2775474 tid 2 2024-09-17T15:43:30.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40e13bd640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 msgr2=0x7f40dc12e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 0x7f40dc12e9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0ae000 tx=0x7f40cc01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0bb070 con 0x7f40dc1363d0 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 10 monclient: dump: 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.580 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40cc0b6460 con 0x7f40dc1363d0 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 msgr2=0x7f40dc12e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 0x7f40dc12e9e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0ae000 tx=0x7f40cc01ac20 comp rx=0 tx=0).stop 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 msgr2=0x7f40dc12e9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40d27fc640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc1363d0 0x7f40dc12e9e0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f40cc0ae000 tx=0x7f40cc01ac20 comp rx=0 tx=0).stop 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.570+0000 7f40e23bf640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f40c003bbb0 0x7f40c003e070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f40d4003ab0 tx=0x7f40d4009770 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e3e22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e3e22640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 0x7f40dc12f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e3e22640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f40dc12f840 con 0x7f40dc12ef20 2024-09-17T15:43:30.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 0x7f40dc12f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient: get_auth_request con 0x7f40dc12ef20 auth_method 0 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 10 monclient(hunting): handle_auth_done global_id 184113 payload 824 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 1 -- 172.21.15.86:0/2746340016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40dc06d4f0 con 0x7f40dc12ef20 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 msgr2=0x7f40dc12f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40e13bd640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 0x7f40dc12f300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01aa90 tx=0x7f40cc019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40cc0bb040 con 0x7f40dc12ef20 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.578+0000 7f40d27fc640 10 monclient: dump: 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f40cc0b6810 con 0x7f40dc12ef20 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 msgr2=0x7f40dc12f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 0x7f40dc12f300 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01aa90 tx=0x7f40cc019180 comp rx=0 tx=0).stop 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 msgr2=0x7f40dc12f300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40d27fc640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc12ef20 0x7f40dc12f300 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40cc01aa90 tx=0x7f40cc019180 comp rx=0 tx=0).stop 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:30.584 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 14071, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:30.585 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:30.586 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:30.587 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:30.588 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.589 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.592 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.594 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.595 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:30.596 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:30.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f40c003bbb0 msgr2=0x7f40c003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f40c003bbb0 0x7f40c003e070 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f40d4003ab0 tx=0x7f40d4009770 comp rx=0 tx=0).stop 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 10 monclient: shutdown 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 msgr2=0x7f40dc1cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 --2- 172.21.15.86:0/2746340016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f40dc0a0c70 0x7f40dc1cfd30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f40d800c450 tx=0x7f40d800c910 comp rx=0 tx=0).stop 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e23bf640 1 -- 172.21.15.86:0/2746340016 reap_dead start 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e23bf640 1 -- 172.21.15.86:0/2746340016 reap_dead start 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 shutdown_connections 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 >> 172.21.15.86:0/2746340016 conn(0x7f40dc0a2f40 msgr2=0x7f40dc0a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 shutdown_connections 2024-09-17T15:43:30.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.582+0000 7f40e3e22640 1 -- 172.21.15.86:0/2746340016 wait complete. 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2596: writing smithi20328086-32 from 3247187 to 3512861 tid 3 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2597: read oid 47 snap -1 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2597: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2598: copy_from oid 21 from oid 26 current snap is 278 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2593: done (5 left) 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2594: done (4 left) 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2595: done (3 left) 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2599: rollback oid 6 current snap is 278 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 277 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:2600: rollback oid 50 current snap is 278 2024-09-17T15:43:30.600 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 277 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2596: finishing write tid 1 to smithi20328086-32 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2596: finishing write tid 2 to smithi20328086-32 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2601: rollback oid 22 current snap is 278 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 22 to 277 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2602: snap_create 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2599: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2598: finishing copy_from to smithi20328086-21 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2598: got expected ENOENT (src dne) 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2598: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2601: finishing rollback tid 0 to smithi20328086-22 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2601: finishing rollback tid 1 to smithi20328086-22 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:2601: finishing rollback tid 2 to smithi20328086-22 2024-09-17T15:43:30.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1255 (ObjNum 922 snap 277 seq_num 922) dirty exists 2024-09-17T15:43:30.602 INFO:tasks.rados.rados.0.smithi203.stdout:2597: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:43:30.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:30.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:43:30.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: get_monmap_and_config 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: monmap: 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:30.724663+0000 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:30.724663+0000 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:30.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:30.720 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:30.720 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:30.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 Processor -- start 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 -- start start 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: init 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 0x7f3b780a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b780a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b780a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b78136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b780a2130 con 0x7f3b780a1590 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3b780a2410 con 0x7f3b780a0c70 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b780a26f0 con 0x7f3b78136160 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b7d9e9640 10 monclient(hunting): authenticate will time out at 1943.690210s 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 0x7f3b780a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 0x7f3b780a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55376/0 (socket says 172.21.15.86:55376) 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36430/0 (socket says 172.21.15.86:36430) 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2522148780 learned_addr learned my addr 172.21.15.86:0/2522148780 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request con 0x7f3b780a0c70 auth_method 0 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): get_auth_request con 0x7f3b78136160 auth_method 0 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient(hunting): handle_auth_done global_id 164312 payload 274 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 -- 172.21.15.86:0/2522148780 >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 msgr2=0x7f3b780a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 --2- 172.21.15.86:0/2522148780 >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 0x7f3b780a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 -- 172.21.15.86:0/2522148780 >> v1:172.21.15.86:6789/0 conn(0x7f3b780a1590 legacy=0x7f3b78135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 monclient: found mon.noname-b 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:30.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.722+0000 7f3b777fe640 1 -- 172.21.15.86:0/2522148780 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7809f9f0 con 0x7f3b78136160 2024-09-17T15:43:30.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b777fe640 1 -- 172.21.15.86:0/2522148780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 msgr2=0x7f3b7813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b777fe640 1 --2- 172.21.15.86:0/2522148780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009d90 tx=0x7f3b6801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb0c6be56cf4966 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 1 -- 172.21.15.86:0/2522148780 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b680b2070 con 0x7f3b78136160 2024-09-17T15:43:30.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: dump: 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 5 monclient: authenticate success, global_id 164312 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 1 -- 172.21.15.86:0/2522148780 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b680a5890 con 0x7f3b78136160 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 1 -- 172.21.15.86:0/2522148780 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b680a5c60 con 0x7f3b78136160 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 4 set_mon_vals no callback set 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b75ffb640 10 monclient: dump: 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.731 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 10 monclient: shutdown 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2522148780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 msgr2=0x7f3b7813df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2522148780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3b68009d90 tx=0x7f3b6801c830 comp rx=0 tx=0).stop 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2522148780 shutdown_connections 2024-09-17T15:43:30.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2522148780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78136160 0x7f3b7813df30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2522148780 >> v2:172.21.15.86:3300/0 conn(0x7f3b780a0c70 0x7f3b780a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2522148780 >> 172.21.15.86:0/2522148780 conn(0x7f3b780a2f40 msgr2=0x7f3b780a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2522148780 shutdown_connections 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.726+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2522148780 wait complete. 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient: monmap: 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:30.733 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 Processor -- start 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 -- start start 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient: init 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 0x7f3b7811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-a con 0x7f3b780a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b780a1590 0x7f3b7811ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-c con 0x7f3b780a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b78136160 0x7f3b7811d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): picked mon.noname-b con 0x7f3b78136160 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:30.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3b780a00d0 con 0x7f3b780a0c70 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7809ff50 con 0x7f3b78136160 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b78140c90 con 0x7f3b780a1590 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b7d9e9640 10 monclient(hunting): authenticate will time out at 1943.697546s 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 0x7f3b7811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 0x7f3b7811c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55384/0 (socket says 172.21.15.86:55384) 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2200095564 learned_addr learned my addr 172.21.15.86:0/2200095564 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request con 0x7f3b780a0c70 auth_method 0 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b780a1590 0x7f3b7811ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): get_auth_request con 0x7f3b780a1590 auth_method 0 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b777fe640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b78136160 0x7f3b7811d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b777fe640 10 monclient(hunting): get_auth_request con 0x7f3b78136160 auth_method 0 2024-09-17T15:43:30.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient(hunting): handle_auth_done global_id 184119 payload 824 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b780a1590 msgr2=0x7f3b7811ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b780a1590 0x7f3b7811ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b78136160 msgr2=0x7f3b7811d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b78136160 0x7f3b7811d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.730+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7806fb90 con 0x7f3b780a0c70 2024-09-17T15:43:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b76ffd640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 msgr2=0x7f3b7811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b76ffd640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 0x7f3b7811c8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e7e0 tx=0x7f3b7000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b70017070 con 0x7f3b780a0c70 2024-09-17T15:43:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: dump: 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7000cae0 con 0x7f3b780a0c70 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 5 monclient: authenticate success, global_id 184119 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 monclient: _renew_subs 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7811da00 con 0x7f3b780a0c70 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000f040 con 0x7f3b780a0c70 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 10 monclient: dump: 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 monclient: _renew_subs 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b757fa640 4 set_mon_vals no callback set 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7811f4c0 con 0x7f3b780a0c70 2024-09-17T15:43:30.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 0x7f3b78134f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7811e620 con 0x7f3b78134b20 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 0x7f3b78134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient: get_auth_request con 0x7f3b78134b20 auth_method 0 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3b7001a070 con 0x7f3b780a0c70 2024-09-17T15:43:30.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 10 monclient(hunting): handle_auth_done global_id 164318 payload 824 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b767fc640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b780a1050 con 0x7f3b78134b20 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b6403bb60 0x7f3b6403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(576..576 src has 438..576) ==== 4787+0+0 (secure 0 0 0) 0x7f3b7004fd10 con 0x7f3b780a0c70 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b777fe640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b6403bb60 0x7f3b6403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.734+0000 7f3b777fe640 10 monclient: get_auth_request con 0x7f3b6403bb60 auth_method 0 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b767fc640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 msgr2=0x7f3b78134f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b767fc640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 0x7f3b78134f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c00fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c003b80 con 0x7f3b78134b20 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 10 monclient: dump: 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.750 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3b6c019070 con 0x7f3b78134b20 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 msgr2=0x7f3b78134f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 0x7f3b78134f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c00fc00 comp rx=0 tx=0).stop 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 msgr2=0x7f3b78134f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b5ffff640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78134b20 0x7f3b78134f00 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c00fc00 comp rx=0 tx=0).stop 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.738+0000 7f3b777fe640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b6403bb60 0x7f3b6403e020 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3b78074ae0 tx=0x7f3b680a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b7d9e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 0x7f3b78138f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b7d9e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7806d4f0 con 0x7f3b78135440 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 0x7f3b78138f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient: get_auth_request con 0x7f3b78135440 auth_method 0 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 10 monclient(hunting): handle_auth_done global_id 164321 payload 824 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.742+0000 7f3b767fc640 1 -- 172.21.15.86:0/2200095564 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b7811e9a0 con 0x7f3b78135440 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b767fc640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 msgr2=0x7f3b78138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b767fc640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 0x7f3b78138f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c006e00 con 0x7f3b78135440 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 10 monclient: dump: 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f3b6c006e00 con 0x7f3b78135440 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 msgr2=0x7f3b78138f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 0x7f3b78138f00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c003770 comp rx=0 tx=0).stop 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 msgr2=0x7f3b78138f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b5ffff640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b78135440 0x7f3b78138f00 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c007fd0 tx=0x7f3b6c003770 comp rx=0 tx=0).stop 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 113934, 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:30.755 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.756 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.757 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:30.758 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:30.759 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.760 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:30.763 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:30.764 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.766 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.767 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:30.768 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:30.770 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1394566 2024-09-17T15:43:30.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b6403bb60 msgr2=0x7f3b6403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b6403bb60 0x7f3b6403e020 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3b78074ae0 tx=0x7f3b680a7070 comp rx=0 tx=0).stop 2024-09-17T15:43:30.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 10 monclient: shutdown 2024-09-17T15:43:30.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 msgr2=0x7f3b7811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 --2- 172.21.15.86:0/2200095564 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b780a0c70 0x7f3b7811c8c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000e7e0 tx=0x7f3b7000eca0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b777fe640 1 -- 172.21.15.86:0/2200095564 reap_dead start 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b777fe640 1 -- 172.21.15.86:0/2200095564 reap_dead start 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 shutdown_connections 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 >> 172.21.15.86:0/2200095564 conn(0x7f3b780a2f40 msgr2=0x7f3b78071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 shutdown_connections 2024-09-17T15:43:30.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.746+0000 7f3b7d9e9640 1 -- 172.21.15.86:0/2200095564 wait complete. 2024-09-17T15:43:30.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:30.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:43:30.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.886+0000 7f75eb74e640 10 monclient: get_monmap_and_config 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.886+0000 7f75eb74e640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.886+0000 7f75eb74e640 10 monclient: monmap: 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:30.891969+0000 2024-09-17T15:43:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:30.891969+0000 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:30.887 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 Processor -- start 2024-09-17T15:43:30.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 -- start start 2024-09-17T15:43:30.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient: init 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75e40a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75e40a1760 0x7f75e413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-a con 0x7f75e40a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 0x7f75e413d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-b con 0x7f75e413b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e40a2130 con 0x7f75e40a0c70 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f75e40a2410 con 0x7f75e40a1760 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f75e40a26f0 con 0x7f75e413b450 2024-09-17T15:43:30.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75eb74e640 10 monclient(hunting): authenticate will time out at 1943.857417s 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75e40a1760 0x7f75e413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 0x7f75e413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 0x7f75e413d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36454/0 (socket says 172.21.15.86:36454) 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75e40a1760 0x7f75e413af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55390/0 (socket says 172.21.15.86:55390) 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 1 -- 172.21.15.86:0/2130971689 learned_addr learned my addr 172.21.15.86:0/2130971689 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): get_auth_request con 0x7f75e413b450 auth_method 0 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request con 0x7f75e40a1760 auth_method 0 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_done global_id 184122 payload 274 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 msgr2=0x7f75e413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 0x7f75e413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2130971689 >> v1:172.21.15.203:6789/0 conn(0x7f75e40a0c70 legacy=0x7f75e40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 monclient: found mon.noname-a 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:30.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.890+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2130971689 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e409f9f0 con 0x7f75e40a1760 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e40a1760 msgr2=0x7f75e413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e40a1760 0x7f75e413af10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75d80097c0 tx=0x7f75d801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa63f5b787ae789 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 1 -- 172.21.15.86:0/2130971689 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d80b0070 con 0x7f75e40a1760 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:30.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: dump: 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 1 -- 172.21.15.86:0/2130971689 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75d80ab600 con 0x7f75e40a1760 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 1 -- 172.21.15.86:0/2130971689 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d80ab9d0 con 0x7f75e40a1760 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 5 monclient: authenticate success, global_id 184122 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 4 set_mon_vals no callback set 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75e3fff640 10 monclient: dump: 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.898 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 10 monclient: shutdown 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 -- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e40a1760 msgr2=0x7f75e413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e40a1760 0x7f75e413af10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f75d80097c0 tx=0x7f75d801c360 comp rx=0 tx=0).stop 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 -- 172.21.15.86:0/2130971689 shutdown_connections 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e413b450 0x7f75e413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2130971689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e40a1760 0x7f75e413af10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 -- 172.21.15.86:0/2130971689 >> 172.21.15.86:0/2130971689 conn(0x7f75e40a2f40 msgr2=0x7f75e40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 -- 172.21.15.86:0/2130971689 shutdown_connections 2024-09-17T15:43:30.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 1 -- 172.21.15.86:0/2130971689 wait complete. 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 10 monclient: build_initial_monmap 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.894+0000 7f75eb74e640 10 monclient: monmap: 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 Processor -- start 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- start start 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: init 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 0x7f75e411ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-c con 0x7f75e40a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e40a1760 0x7f75e4135d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-b con 0x7f75e40a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e413b450 0x7f75e41cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): picked mon.noname-a con 0x7f75e413b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f75e4140720 con 0x7f75e413b450 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75e41405a0 con 0x7f75e40a1760 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f75e41408a0 con 0x7f75e40a0c70 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient(hunting): authenticate will time out at 1943.864647s 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 0x7f75e411ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 0x7f75e411ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36466/0 (socket says 172.21.15.86:36466) 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 -- 172.21.15.86:0/2885532117 learned_addr learned my addr 172.21.15.86:0/2885532117 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): get_auth_request con 0x7f75e40a0c70 auth_method 0 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e413b450 0x7f75e41cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): get_auth_request con 0x7f75e413b450 auth_method 0 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e40a1760 0x7f75e4135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request con 0x7f75e40a1760 auth_method 0 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e8ce9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e8ce9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient(hunting): handle_auth_done global_id 164327 payload 824 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e40a1760 msgr2=0x7f75e4135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e40a1760 0x7f75e4135d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e413b450 msgr2=0x7f75e41cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e413b450 0x7f75e41cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient: _finish_hunting 0 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 monclient: found mon.noname-c 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e41203b0 con 0x7f75e40a0c70 2024-09-17T15:43:30.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e9ceb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75e413b450 0x7f75e41cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 msgr2=0x7f75e411ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e94ea640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 0x7f75e411ee90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00c450 tx=0x7f75dc00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc012a40 con 0x7f75e40a0c70 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: dump: 2024-09-17T15:43:30.908 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75dc017070 con 0x7f75e40a0c70 2024-09-17T15:43:30.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 5 monclient: authenticate success, global_id 164327 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _renew_subs 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e4120690 con 0x7f75e40a0c70 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc012e40 con 0x7f75e40a0c70 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e1ffb640 10 monclient: dump: 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.910 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _renew_subs 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75e37fe640 4 set_mon_vals no callback set 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.898+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e411fdf0 con 0x7f75e40a0c70 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75eb74e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 0x7f75e411cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75e40a1050 con 0x7f75e4136240 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f75dc01c460 con 0x7f75e40a0c70 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 0x7f75e411cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e1ffb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f75d003bb60 0x7f75d003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient: get_auth_request con 0x7f75e4136240 auth_method 0 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(576..576 src has 438..576) ==== 4787+0+0 (secure 0 0 0) 0x7f75dc04fcf0 con 0x7f75e40a0c70 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e9ceb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f75d003bb60 0x7f75d003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e9ceb640 10 monclient: get_auth_request con 0x7f75d003bb60 auth_method 0 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_done global_id 174184 payload 824 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75e4120030 con 0x7f75e4136240 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e9ceb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f75d003bb60 0x7f75d003e020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f75d4003b50 tx=0x7f75d400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 msgr2=0x7f75e411cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.902+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 0x7f75e411cd10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75d80ad000 tx=0x7f75d801aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d80b0040 con 0x7f75e4136240 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:30.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 10 monclient: dump: 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f75d80b6670 con 0x7f75e4136240 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 msgr2=0x7f75e411cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 0x7f75e411cd10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75d80ad000 tx=0x7f75d801aba0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 msgr2=0x7f75e411cd10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.906+0000 7f75e1ffb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e4136240 0x7f75e411cd10 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f75d80ad000 tx=0x7f75d801aba0 comp rx=0 tx=0).stop 2024-09-17T15:43:30.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75eb74e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:30.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 0x7f75e41ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:30.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75eb74e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f75e41ce7f0 con 0x7f75e41cded0 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 0x7f75e41ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient: get_auth_request con 0x7f75e41cded0 auth_method 0 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 10 monclient(hunting): handle_auth_done global_id 174187 payload 824 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.910+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2885532117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f75e41d8ce0 con 0x7f75e41cded0 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e8ce9640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 msgr2=0x7f75e41ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:30.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e8ce9640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 0x7f75e41ce2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75d8000c10 tx=0x7f75d8019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d80b0050 con 0x7f75e41cded0 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 10 monclient: dump: 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:30.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f75d80c5070 con 0x7f75e41cded0 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 msgr2=0x7f75e41ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 0x7f75e41ce2b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75d8000c10 tx=0x7f75d8019100 comp rx=0 tx=0).stop 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 msgr2=0x7f75e41ce2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e1ffb640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75e41cded0 0x7f75e41ce2b0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75d8000c10 tx=0x7f75d8019100 comp rx=0 tx=0).stop 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:30.919 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 65706, 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.920 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.921 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:30.923 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:30.924 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.928 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:30.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.930 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.931 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:30.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:30.933 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:30.933 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: finishing write tid 3 to smithi20328086-32 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: finishing write tid 4 to smithi20328086-32 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: finishing write tid 5 to smithi20328086-32 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: oid 32 updating version 0 to 1180 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: oid 32 version 1180 is already newer than 1178 2024-09-17T15:43:30.934 INFO:tasks.rados.rados.0.smithi203.stdout:2596: oid 32 version 1180 is already newer than 1179 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:2596: oid 32 updating version 1180 to 1181 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1181 (ObjNum 923 snap 278 seq_num 923) dirty exists 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:2596: left oid 32 (ObjNum 923 snap 278 seq_num 923) 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:2600: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:2600: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:2600: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:43:30.935 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1184 (ObjNum 912 snap 276 seq_num 912) dirty exists 2024-09-17T15:43:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f75d003bb60 msgr2=0x7f75d003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f75d003bb60 0x7f75d003e020 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f75d4003b50 tx=0x7f75d400b040 comp rx=0 tx=0).stop 2024-09-17T15:43:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 10 monclient: shutdown 2024-09-17T15:43:30.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 msgr2=0x7f75e411ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 --2- 172.21.15.86:0/2885532117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75e40a0c70 0x7f75e411ee90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00c450 tx=0x7f75dc00c910 comp rx=0 tx=0).stop 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e9ceb640 1 -- 172.21.15.86:0/2885532117 reap_dead start 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75e9ceb640 1 -- 172.21.15.86:0/2885532117 reap_dead start 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 shutdown_connections 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 >> 172.21.15.86:0/2885532117 conn(0x7f75e40a2f40 msgr2=0x7f75e40a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 shutdown_connections 2024-09-17T15:43:30.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:30.914+0000 7f75eb74e640 1 -- 172.21.15.86:0/2885532117 wait complete. 2024-09-17T15:43:30.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:30.936 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:43:30.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:43:31.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 10 monclient: get_monmap_and_config 2024-09-17T15:43:31.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 10 monclient: build_initial_monmap 2024-09-17T15:43:31.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 10 monclient: monmap: 2024-09-17T15:43:31.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:31.050 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:31.050 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:31.054680+0000 2024-09-17T15:43:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:31.054680+0000 2024-09-17T15:43:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:31.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 1 Processor -- start 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 1 -- start start 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.050+0000 7ff6105a9640 10 monclient: init 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-b con 0x7ff6080689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 0x7ff608069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff608068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff608134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff60809d770 con 0x7ff608134a40 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:31.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff60809ed00 con 0x7ff608068db0 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff60809efe0 con 0x7ff6080689d0 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff6105a9640 10 monclient(hunting): authenticate will time out at 1944.020650s 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36478/0 (socket says 172.21.15.86:36478) 2024-09-17T15:43:31.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 -- 172.21.15.86:0/3055347932 learned_addr learned my addr 172.21.15.86:0/3055347932 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): get_auth_request con 0x7ff6080689d0 auth_method 0 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60db44640 1 --2- 172.21.15.86:0/3055347932 >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 0x7ff608069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient(hunting): handle_auth_done global_id 164330 payload 274 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 -- 172.21.15.86:0/3055347932 >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 msgr2=0x7ff608069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 --2- 172.21.15.86:0/3055347932 >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 0x7ff608069190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 -- 172.21.15.86:0/3055347932 >> v1:172.21.15.203:6789/0 conn(0x7ff608134a40 legacy=0x7ff608134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient: _finish_hunting 0 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 monclient: found mon.noname-b 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60e345640 1 -- 172.21.15.86:0/3055347932 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6080a04e0 con 0x7ff6080689d0 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60db44640 10 monclient: get_auth_request con 0x7ff608068db0 auth_method 0 2024-09-17T15:43:31.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.054+0000 7ff60db44640 1 --2- 172.21.15.86:0/3055347932 >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 0x7ff608069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60e345640 1 -- 172.21.15.86:0/3055347932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 msgr2=0x7ff60809ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60e345640 1 --2- 172.21.15.86:0/3055347932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009870 tx=0x7ff5f801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b720f817ce7ec7f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 1 -- 172.21.15.86:0/3055347932 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80ab650 con 0x7ff6080689d0 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:31.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: dump: 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:31.076 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:31.077 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: _finish_auth 0 2024-09-17T15:43:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: _check_auth_tickets 2024-09-17T15:43:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 1 -- 172.21.15.86:0/3055347932 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f80b0070 con 0x7ff6080689d0 2024-09-17T15:43:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 5 monclient: authenticate success, global_id 164330 2024-09-17T15:43:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 1 -- 172.21.15.86:0/3055347932 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80aba10 con 0x7ff6080689d0 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 4 set_mon_vals no callback set 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:31.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60d343640 10 monclient: dump: 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:31.080 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: shutdown 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- 172.21.15.86:0/3055347932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 msgr2=0x7ff60809ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3055347932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff5f8009870 tx=0x7ff5f801c4b0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- 172.21.15.86:0/3055347932 shutdown_connections 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3055347932 >> v2:172.21.15.203:3300/0 conn(0x7ff608068db0 0x7ff608069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3055347932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6080689d0 0x7ff60809ffa0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- 172.21.15.86:0/3055347932 >> 172.21.15.86:0/3055347932 conn(0x7ff6080a6440 msgr2=0x7ff6080a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:31.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- 172.21.15.86:0/3055347932 shutdown_connections 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- 172.21.15.86:0/3055347932 wait complete. 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: build_initial_monmap 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: monmap: 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:31.082 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 Processor -- start 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- start start 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: init 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6080689d0 0x7ff60813bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-b con 0x7ff6080689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:31.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-c con 0x7ff608068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff608134a40 0x7ff608134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): picked mon.noname-a con 0x7ff608134a40 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff60813fad0 con 0x7ff608134a40 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff60813f950 con 0x7ff6080689d0 2024-09-17T15:43:31.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff60813fc50 con 0x7ff608068db0 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff6105a9640 10 monclient(hunting): authenticate will time out at 1944.027747s 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36486/0 (socket says 172.21.15.86:36486) 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 1 -- 172.21.15.86:0/3589341561 learned_addr learned my addr 172.21.15.86:0/3589341561 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 10 monclient(hunting): get_auth_request con 0x7ff608068db0 auth_method 0 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:31.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60db44640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60eb46640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff608134a40 0x7ff608134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60eb46640 10 monclient(hunting): get_auth_request con 0x7ff608134a40 auth_method 0 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60eb46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60eb46640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.058+0000 7ff60eb46640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:31.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60eb46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60eb46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60eb46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60e345640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6080689d0 0x7ff60813bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60e345640 10 monclient(hunting): get_auth_request con 0x7ff6080689d0 auth_method 0 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60e345640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60e345640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60e345640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient(hunting): handle_auth_done global_id 164333 payload 824 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 -- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6080689d0 msgr2=0x7ff60813bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6080689d0 0x7ff60813bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 -- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff608134a40 msgr2=0x7ff608134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff608134a40 0x7ff608134110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient: _finish_hunting 0 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 monclient: found mon.noname-c 2024-09-17T15:43:31.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 -- 172.21.15.86:0/3589341561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6080a2a60 con 0x7ff608068db0 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 -- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 msgr2=0x7ff608133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60db44640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c7f0 tx=0x7ff60400ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604007c40 con 0x7ff608068db0 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: dump: 2024-09-17T15:43:31.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:31.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 5 monclient: authenticate success, global_id 164333 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff604007de0 con 0x7ff608068db0 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _renew_subs 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60811f670 con 0x7ff608068db0 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:31.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604011630 con 0x7ff608068db0 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 10 monclient: dump: 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:31.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _renew_subs 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff60cb42640 4 set_mon_vals no callback set 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6080a3350 con 0x7ff608068db0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff60809d460 con 0x7ff608068db0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.062+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff6040117d0 con 0x7ff608068db0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.066+0000 7ff5feffd640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 0x7ff5ec03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.066+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(576..576 src has 438..576) ==== 4787+0+0 (secure 0 0 0) 0x7ff604050200 con 0x7ff608068db0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.066+0000 7ff60e345640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 0x7ff5ec03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.066+0000 7ff60e345640 10 monclient: get_auth_request con 0x7ff5ec03bbb0 auth_method 0 2024-09-17T15:43:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.066+0000 7ff60e345640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 0x7ff5ec03e070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80ac000 tx=0x7ff5f80a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.070+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff604015020 con 0x7ff608068db0 2024-09-17T15:43:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.070+0000 7ff5feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:31.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.070+0000 7ff5feffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:31.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.298+0000 7ff6105a9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:43:31.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.298+0000 7ff6105a9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:43:31.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.298+0000 7ff6105a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:31.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.298+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff60813c930 con 0x7ff608068db0 2024-09-17T15:43:31.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff5feffd640 1 -- 172.21.15.86:0/3589341561 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff604021350 con 0x7ff608068db0 2024-09-17T15:43:31.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff5feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:43:31.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff5feffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1251566579,config=3541143443,health=86150175,kv=3885551602,logm=2745493027}) 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=1464875573,config=3541143443,health=3188305645,kv=1452539945,logm=2622309833}) 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1251566579,config=3541143443,health=86150175,kv=3885551602,logm=2745493027}) 2024-09-17T15:43:31.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=2651570637,config=3541143443,health=86150175,kv=3885551602,logm=3104463607}) 2024-09-17T15:43:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 msgr2=0x7ff5ec03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 0x7ff5ec03e070 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff5f80ac000 tx=0x7ff5f80a7000 comp rx=0 tx=0).stop 2024-09-17T15:43:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 10 monclient: shutdown 2024-09-17T15:43:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:31.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 msgr2=0x7ff608133bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff60400c7f0 tx=0x7ff60400ccb0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 shutdown_connections 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5ec03bbb0 0x7ff5ec03e070 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff608134a40 0x7ff608134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff608068db0 0x7ff608133bd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 --2- 172.21.15.86:0/3589341561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6080689d0 0x7ff60813bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:31.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 >> 172.21.15.86:0/3589341561 conn(0x7ff6080a6440 msgr2=0x7ff6080a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:31.304 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992286097}) 2024-09-17T15:43:31.304 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729668303}) 2024-09-17T15:43:31.304 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.302+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.302+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 shutdown_connections 2024-09-17T15:43:31.305 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=992286097}) 2024-09-17T15:43:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:31.306+0000 7ff6105a9640 1 -- 172.21.15.86:0/3589341561 wait complete. 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=7664104}) 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470604491}) 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810487612}) 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470604491}) 2024-09-17T15:43:31.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.306+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810462588}) 2024-09-17T15:43:31.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284927571}) 2024-09-17T15:43:31.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631463634}) 2024-09-17T15:43:31.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284927571}) 2024-09-17T15:43:31.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1581941317}) 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755037585}) 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2114476361}) 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755037585}) 2024-09-17T15:43:31.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.310+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283914153}) 2024-09-17T15:43:31.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101897254}) 2024-09-17T15:43:31.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084266216}) 2024-09-17T15:43:31.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3101897254}) 2024-09-17T15:43:31.312 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.314+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901577009}) 2024-09-17T15:43:31.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114933543}) 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2447721156}) 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1114933543}) 2024-09-17T15:43:31.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.514+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359007210}) 2024-09-17T15:43:31.518 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588104937}) 2024-09-17T15:43:31.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854331506}) 2024-09-17T15:43:31.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1588104937}) 2024-09-17T15:43:31.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.522+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574654977}) 2024-09-17T15:43:31.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819247030}) 2024-09-17T15:43:31.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745960231}) 2024-09-17T15:43:31.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2819247030}) 2024-09-17T15:43:31.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.530+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546832937}) 2024-09-17T15:43:31.528 INFO:tasks.rados.rados.0.smithi203.stdout:2596: done (6 left) 2024-09-17T15:43:31.528 INFO:tasks.rados.rados.0.smithi203.stdout:2597: done (5 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2598: done (4 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2599: done (3 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2600: done (2 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2601: done (1 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2602: done (0 left) 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2603: write oid 41 current snap is 279 2024-09-17T15:43:31.529 INFO:tasks.rados.rados.0.smithi203.stdout:2603: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534085998}) 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314552913}) 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534085998}) 2024-09-17T15:43:31.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1332851667}) 2024-09-17T15:43:31.542 INFO:tasks.rados.rados.0.smithi203.stdout:2603: writing smithi20328086-41 from 729668 to 1138457 tid 1 2024-09-17T15:43:31.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448151068}) 2024-09-17T15:43:31.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3485729234}) 2024-09-17T15:43:31.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:31.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448151068}) 2024-09-17T15:43:31.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:31.546+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554513095}) 2024-09-17T15:43:31.555 INFO:tasks.rados.rados.0.smithi203.stdout:2603: writing smithi20328086-41 from 1820303 to 2577614 tid 2 2024-09-17T15:43:31.561 INFO:tasks.rados.rados.0.smithi203.stdout:2603: writing smithi20328086-41 from 3274610 to 3529668 tid 3 2024-09-17T15:43:31.561 INFO:tasks.rados.rados.0.smithi203.stdout:2604: rollback oid 37 current snap is 279 2024-09-17T15:43:31.561 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 277 2024-09-17T15:43:31.561 INFO:tasks.rados.rados.0.smithi203.stdout:2603: finishing write tid 1 to smithi20328086-41 2024-09-17T15:43:31.562 INFO:tasks.rados.rados.0.smithi203.stdout:2605: rollback oid 15 current snap is 279 2024-09-17T15:43:31.562 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 268 2024-09-17T15:43:31.562 INFO:tasks.rados.rados.0.smithi203.stdout:2606: copy_from oid 8 from oid 43 current snap is 279 2024-09-17T15:43:31.562 INFO:tasks.rados.rados.0.smithi203.stdout:2607: snap_create 2024-09-17T15:43:31.565 INFO:tasks.rados.rados.0.smithi203.stdout:2603: finishing write tid 2 to smithi20328086-41 2024-09-17T15:43:31.568 INFO:tasks.rados.rados.0.smithi203.stdout:2603: finishing write tid 3 to smithi20328086-41 2024-09-17T15:43:31.570 INFO:tasks.rados.rados.0.smithi203.stdout:2605: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:43:31.570 INFO:tasks.rados.rados.0.smithi203.stdout:2605: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:43:31.571 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1261 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:43:31.573 INFO:tasks.rados.rados.0.smithi203.stdout:2603: finishing write tid 4 to smithi20328086-41 2024-09-17T15:43:31.573 INFO:tasks.rados.rados.0.smithi203.stdout:2603: finishing write tid 5 to smithi20328086-41 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:2603: oid 41 updating version 0 to 647 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:2603: oid 41 updating version 647 to 648 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:2603: oid 41 updating version 648 to 649 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:2603: oid 41 version 649 is already newer than 646 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 649 (ObjNum 924 snap 279 seq_num 924) dirty exists 2024-09-17T15:43:31.574 INFO:tasks.rados.rados.0.smithi203.stdout:2603: left oid 41 (ObjNum 924 snap 279 seq_num 924) 2024-09-17T15:43:31.577 INFO:tasks.rados.rados.0.smithi203.stdout:2604: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:43:31.579 INFO:tasks.rados.rados.0.smithi203.stdout:2604: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:43:31.579 INFO:tasks.rados.rados.0.smithi203.stdout:2604: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:43:31.579 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1295 (ObjNum 914 snap 276 seq_num 914) dirty exists 2024-09-17T15:43:31.594 INFO:tasks.rados.rados.0.smithi203.stdout:2606: finishing copy_from racing read to smithi20328086-8 2024-09-17T15:43:31.601 INFO:tasks.rados.rados.0.smithi203.stdout:2606: finishing copy_from to smithi20328086-8 2024-09-17T15:43:31.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 644 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446951336}) 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=389037271}) 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446951336}) 2024-09-17T15:43:32.084 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.086+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193665153}) 2024-09-17T15:43:32.086 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.086 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949919656}) 2024-09-17T15:43:32.087 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=940946921}) 2024-09-17T15:43:32.087 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.087 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949919656}) 2024-09-17T15:43:32.087 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=833731786}) 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204973348}) 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=934714516}) 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2204973348}) 2024-09-17T15:43:32.089 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.090+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948146260}) 2024-09-17T15:43:32.091 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.092 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933860212}) 2024-09-17T15:43:32.092 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041466849}) 2024-09-17T15:43:32.092 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.092 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933860212}) 2024-09-17T15:43:32.092 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.094+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728409546}) 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269256035}) 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3021927078}) 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269256035}) 2024-09-17T15:43:32.096 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856141915}) 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980568476}) 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705916216}) 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1980568476}) 2024-09-17T15:43:32.098 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.098+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379457901}) 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061667738}) 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53718957}) 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061667738}) 2024-09-17T15:43:32.101 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.102+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075787095}) 2024-09-17T15:43:32.104 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.104 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178418232}) 2024-09-17T15:43:32.104 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1794434588}) 2024-09-17T15:43:32.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178418232}) 2024-09-17T15:43:32.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.106+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456948568}) 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485732131}) 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2702414416}) 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485732131}) 2024-09-17T15:43:32.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.110+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202472942}) 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930913686}) 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3045567216}) 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930913686}) 2024-09-17T15:43:32.111 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.114+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639396507}) 2024-09-17T15:43:32.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519432893}) 2024-09-17T15:43:32.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1998433365}) 2024-09-17T15:43:32.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=519432893}) 2024-09-17T15:43:32.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1054422449}) 2024-09-17T15:43:32.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640777128}) 2024-09-17T15:43:32.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681001575}) 2024-09-17T15:43:32.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=640777128}) 2024-09-17T15:43:32.117 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.118+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260828469}) 2024-09-17T15:43:32.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704565087}) 2024-09-17T15:43:32.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258059969}) 2024-09-17T15:43:32.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704565087}) 2024-09-17T15:43:32.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317165018}) 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787947151}) 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3778302379}) 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787947151}) 2024-09-17T15:43:32.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.122+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2502256146}) 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318452709}) 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423534392}) 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1318452709}) 2024-09-17T15:43:32.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215488357}) 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521443837}) 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456310520}) 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521443837}) 2024-09-17T15:43:32.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.126+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1482506362}) 2024-09-17T15:43:32.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604227455}) 2024-09-17T15:43:32.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367876188}) 2024-09-17T15:43:32.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.127 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604227455}) 2024-09-17T15:43:32.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572673307}) 2024-09-17T15:43:32.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172839364}) 2024-09-17T15:43:32.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921921021}) 2024-09-17T15:43:32.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172839364}) 2024-09-17T15:43:32.130 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.130+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663601105}) 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360226341}) 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908529451}) 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360226341}) 2024-09-17T15:43:32.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288351523}) 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622008464}) 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1219032017}) 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622008464}) 2024-09-17T15:43:32.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.134+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282083849}) 2024-09-17T15:43:32.136 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951056759}) 2024-09-17T15:43:32.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1913160066}) 2024-09-17T15:43:32.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951056759}) 2024-09-17T15:43:32.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2276663346}) 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613986424}) 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.138+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2127363599}) 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613986424}) 2024-09-17T15:43:32.138 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984463632}) 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689252483}) 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034133591}) 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689252483}) 2024-09-17T15:43:32.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.142+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320869838}) 2024-09-17T15:43:32.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004342794}) 2024-09-17T15:43:32.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1530656050}) 2024-09-17T15:43:32.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.143 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004342794}) 2024-09-17T15:43:32.143 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1535248280}) 2024-09-17T15:43:32.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810489489}) 2024-09-17T15:43:32.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398166894}) 2024-09-17T15:43:32.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810489489}) 2024-09-17T15:43:32.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.146+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2818198344}) 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970680626}) 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699488351}) 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970680626}) 2024-09-17T15:43:32.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346403066}) 2024-09-17T15:43:32.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138061357}) 2024-09-17T15:43:32.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679524653}) 2024-09-17T15:43:32.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138061357}) 2024-09-17T15:43:32.150 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.150+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=64549202}) 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844195286}) 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472506706}) 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844195286}) 2024-09-17T15:43:32.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1659299992}) 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101209190}) 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2515125705}) 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101209190}) 2024-09-17T15:43:32.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.154+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333882365}) 2024-09-17T15:43:32.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559407125}) 2024-09-17T15:43:32.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315574982}) 2024-09-17T15:43:32.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559407125}) 2024-09-17T15:43:32.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590946762}) 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624118043}) 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1217744401}) 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624118043}) 2024-09-17T15:43:32.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.158+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965990182}) 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2392129939}) 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917309701}) 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2392129939}) 2024-09-17T15:43:32.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278259423}) 2024-09-17T15:43:32.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868278293}) 2024-09-17T15:43:32.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3242177556}) 2024-09-17T15:43:32.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868278293}) 2024-09-17T15:43:32.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.162+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990257838}) 2024-09-17T15:43:32.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710740298}) 2024-09-17T15:43:32.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304091495}) 2024-09-17T15:43:32.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710740298}) 2024-09-17T15:43:32.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723445824}) 2024-09-17T15:43:32.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967432483}) 2024-09-17T15:43:32.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3595143721}) 2024-09-17T15:43:32.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967432483}) 2024-09-17T15:43:32.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.166+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189130881}) 2024-09-17T15:43:32.167 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449399445}) 2024-09-17T15:43:32.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505366890}) 2024-09-17T15:43:32.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449399445}) 2024-09-17T15:43:32.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821160230}) 2024-09-17T15:43:32.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367964200}) 2024-09-17T15:43:32.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498995194}) 2024-09-17T15:43:32.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367964200}) 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.170+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1142362526}) 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4202195206}) 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413302353}) 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4202195206}) 2024-09-17T15:43:32.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434238411}) 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521182644}) 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904346702}) 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.174+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521182644}) 2024-09-17T15:43:32.174 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3826862132}) 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020224911}) 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463908571}) 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020224911}) 2024-09-17T15:43:32.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2775457698}) 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230545417}) 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.178+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3600281761}) 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230545417}) 2024-09-17T15:43:32.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401950273}) 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436251877}) 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1977259307}) 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436251877}) 2024-09-17T15:43:32.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.182+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945365847}) 2024-09-17T15:43:32.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197853301}) 2024-09-17T15:43:32.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056855529}) 2024-09-17T15:43:32.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197853301}) 2024-09-17T15:43:32.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081665758}) 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763258232}) 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343391267}) 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763258232}) 2024-09-17T15:43:32.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2822219850}) 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=37} crc {logm=844584573,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4126480798,mon_config_key=1928020585,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=3311363537,osd_pg_creating=2184513372,osd_snap=4255126787}) 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42} crc {logm=3099048241,mdsmap=2815901553,mgr=991077062,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2001903635,mon_config_key=3346977109,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2184513372,osd_snap=204087103}) 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=12,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=37} crc {logm=844584573,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4126480798,mon_config_key=1928020585,monmap=4191199434,nvmeofgw=1596038039,osd_metadata=3311363537,osd_pg_creating=2184513372,osd_snap=4255126787}) 2024-09-17T15:43:32.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=15,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=1736775388,mdsmap=2815901553,mgr=1842696677,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2278854801,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2184513372,osd_snap=1262282946}) 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=71,osdmap=29} crc {osd_snap=2377499494,osdmap=4277703175}) 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=3333900083,osdmap=4188011061}) 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=71,osdmap=29} crc {osd_snap=2377499494,osdmap=4277703175}) 2024-09-17T15:43:32.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.190+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=72,osdmap=28} crc {osd_snap=1238731138,osdmap=3963089037}) 2024-09-17T15:43:32.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=892938302}) 2024-09-17T15:43:32.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=802282672}) 2024-09-17T15:43:32.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=892938302}) 2024-09-17T15:43:32.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.194+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1099173328}) 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=3770220899}) 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=92} crc {osdmap=3284205567}) 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=3770220899}) 2024-09-17T15:43:32.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:43:32.206+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=89} crc {osdmap=681022892}) 2024-09-17T15:43:32.321 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:43:32.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.446+0000 7f41c021a640 10 monclient: get_monmap_and_config 2024-09-17T15:43:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.446+0000 7f41c021a640 10 monclient: build_initial_monmap 2024-09-17T15:43:32.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.446+0000 7f41c021a640 10 monclient: monmap: 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:32.451316+0000 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:32.451316+0000 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:32.446 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:32.447 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:32.447 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:32.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 Processor -- start 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 -- start start 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient: init 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41b80a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41b80a5b60 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f41b809cf60 0x7f41b81409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-c con 0x7f41b809cf60 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b80a6f40 con 0x7f41b80a5070 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b80a65e0 con 0x7f41b80a5b60 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f41b809d390 con 0x7f41b809cf60 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41c021a640 10 monclient(hunting): authenticate will time out at 1945.416770s 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41bdfb6640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f41b80a5070 0x7f41b80a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:33498/0 (socket says 172.21.15.86:33498) 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41bdfb6640 1 -- 172.21.15.86:0/24626864 learned_addr learned my addr 172.21.15.86:0/24626864 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 1 --2- 172.21.15.86:0/24626864 >> v2:172.21.15.203:3300/0 conn(0x7f41b809cf60 0x7f41b81409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): get_auth_request con 0x7f41b809cf60 auth_method 0 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient(hunting): handle_auth_done global_id 174193 payload 274 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 1 -- 172.21.15.86:0/24626864 >> v1:172.21.15.203:6789/0 conn(0x7f41b80a5b60 legacy=0x7f41b813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 1 -- 172.21.15.86:0/24626864 >> v1:172.21.15.86:6789/0 conn(0x7f41b80a5070 legacy=0x7f41b80a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient: _finish_hunting 0 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 1 monclient: found mon.noname-c 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:43:32.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.450+0000 7f41be7b7640 1 -- 172.21.15.86:0/24626864 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b809c2a0 con 0x7f41b809cf60 2024-09-17T15:43:32.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 1 -- 172.21.15.86:0/24626864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 msgr2=0x7f41b81409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 1 --2- 172.21.15.86:0/24626864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b81409b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009d90 tx=0x7f41b001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce5f3f81650b041c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 1 -- 172.21.15.86:0/24626864 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b00b2070 con 0x7f41b809cf60 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: dump: 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.455 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: _finish_auth 0 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: _check_auth_tickets 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 1 -- 172.21.15.86:0/24626864 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b00a5940 con 0x7f41b809cf60 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 1 -- 172.21.15.86:0/24626864 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b00a5d10 con 0x7f41b809cf60 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bcfb4640 10 monclient: dump: 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 5 monclient: authenticate success, global_id 174193 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 4 set_mon_vals no callback set 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: shutdown 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- 172.21.15.86:0/24626864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 msgr2=0x7f41b81409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 --2- 172.21.15.86:0/24626864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b81409b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009d90 tx=0x7f41b001c800 comp rx=0 tx=0).stop 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- 172.21.15.86:0/24626864 shutdown_connections 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 --2- 172.21.15.86:0/24626864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b81409b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- 172.21.15.86:0/24626864 >> 172.21.15.86:0/24626864 conn(0x7f41b80742a0 msgr2=0x7f41b80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- 172.21.15.86:0/24626864 shutdown_connections 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- 172.21.15.86:0/24626864 wait complete. 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: build_initial_monmap 2024-09-17T15:43:32.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: monmap: 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 Processor -- start 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- start start 2024-09-17T15:43:32.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: init 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b8137720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-b con 0x7f41b809cf60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-a con 0x7f41b80a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 0x7f41b813d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): picked mon.noname-c con 0x7f41b80a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f41b8143890 con 0x7f41b80a5070 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f41b8143710 con 0x7f41b809cf60 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f41b8143a10 con 0x7f41b80a5b60 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41c021a640 10 monclient(hunting): authenticate will time out at 1945.427562s 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 0x7f41b813d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55406/0 (socket says 172.21.15.86:55406) 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 1 -- 172.21.15.86:0/2091093601 learned_addr learned my addr 172.21.15.86:0/2091093601 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): get_auth_request con 0x7f41b80a5070 auth_method 0 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): get_auth_request con 0x7f41b80a5b60 auth_method 0 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bd7b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bdfb6640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b8137720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41be7b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bdfb6640 10 monclient(hunting): get_auth_request con 0x7f41b809cf60 auth_method 0 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bdfb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bdfb6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.458+0000 7f41bdfb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bdfb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bdfb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bdfb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 10 monclient(hunting): handle_auth_done global_id 184131 payload 824 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 -- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 msgr2=0x7f41b813d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 0x7f41b813d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 -- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 msgr2=0x7f41b8137720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b8137720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 10 monclient: _finish_hunting 0 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 monclient: found mon.noname-a 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 -- 172.21.15.86:0/2091093601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b813f7e0 con 0x7f41b80a5070 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41be7b7640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 0x7f41b813d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 -- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 msgr2=0x7f41b8137c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41bd7b5640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41a800c450 tx=0x7f41a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8012a40 con 0x7f41b80a5070 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: dump: 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.469 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:32.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 5 monclient: authenticate success, global_id 184131 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8017070 con 0x7f41b80a5070 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8012e40 con 0x7f41b80a5070 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41af7fe640 4 set_mon_vals no callback set 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 10 monclient: dump: 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.471 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _renew_subs 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b80a4440 con 0x7f41b80a5070 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _renew_subs 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:32.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b8121700 con 0x7f41b80a5070 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41b80a1860 con 0x7f41b80a5070 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.462+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f41a801c460 con 0x7f41b80a5070 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.466+0000 7f41ae7fc640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 0x7f419c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.466+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(577..577 src has 438..577) ==== 4815+0+0 (secure 0 0 0) 0x7f41a804fd00 con 0x7f41b80a5070 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.466+0000 7f41bdfb6640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 0x7f419c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.466+0000 7f41bdfb6640 10 monclient: get_auth_request con 0x7f419c03bb90 auth_method 0 2024-09-17T15:43:32.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.466+0000 7f41bdfb6640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 0x7f419c03e050 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f41b4007ca0 tx=0x7f41b4007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.470+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f41b80a4440 con 0x7f41b80a5070 2024-09-17T15:43:32.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.470+0000 7f41ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:32.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.470+0000 7f41ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2603: done (4 left) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2604: done (3 left) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2605: done (2 left) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2606: done (1 left) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2607: done (0 left) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2608: read oid 27 snap -1 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2608: expect (ObjNum 873 snap 269 seq_num 873) 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2609: write oid 32 current snap is 280 2024-09-17T15:43:32.568 INFO:tasks.rados.rados.0.smithi203.stdout:2609: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-17T15:43:32.585 INFO:tasks.rados.rados.0.smithi203.stdout:2609: writing smithi20328086-32 from 746475 to 1230513 tid 1 2024-09-17T15:43:32.599 INFO:tasks.rados.rados.0.smithi203.stdout:2609: writing smithi20328086-32 from 1762432 to 2379754 tid 2 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:2609: writing smithi20328086-32 from 2785680 to 3315586 tid 3 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:2610: read oid 5 snap 268 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:2610: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:2611: rollback oid 24 current snap is 280 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 268 2024-09-17T15:43:32.611 INFO:tasks.rados.rados.0.smithi203.stdout:2608: expect (ObjNum 873 snap 269 seq_num 873) 2024-09-17T15:43:32.666 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1822397 2024-09-17T15:43:32.703 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1455860 2024-09-17T15:43:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41c021a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41c021a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:32.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41c021a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f41b80a54d0 con 0x7f41b80a5070 2024-09-17T15:43:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41ae7fc640 1 -- 172.21.15.86:0/2091093601 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f41a8010cf0 con 0x7f41b80a5070 2024-09-17T15:43:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:32.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.706+0000 7f41ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:32.704 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 msgr2=0x7f419c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 0x7f419c03e050 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f41b4007ca0 tx=0x7f41b4007bf0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 10 monclient: shutdown 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 msgr2=0x7f41b8137c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41a800c450 tx=0x7f41a800c910 comp rx=0 tx=0).stop 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 shutdown_connections 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f419c03bb90 0x7f419c03e050 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41b80a5b60 0x7f41b813d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41b80a5070 0x7f41b8137c60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 --2- 172.21.15.86:0/2091093601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41b809cf60 0x7f41b8137720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 >> 172.21.15.86:0/2091093601 conn(0x7f41b80742a0 msgr2=0x7f41b80a2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 shutdown_connections 2024-09-17T15:43:32.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.710+0000 7f41c021a640 1 -- 172.21.15.86:0/2091093601 wait complete. 2024-09-17T15:43:32.723 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:43:32.723 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:43:32.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:43:32.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:43:32.779 INFO:tasks.rados.rados.0.smithi203.stdout:2609: finishing write tid 1 to smithi20328086-32 2024-09-17T15:43:32.779 INFO:tasks.rados.rados.0.smithi203.stdout:2609: finishing write tid 2 to smithi20328086-32 2024-09-17T15:43:32.779 INFO:tasks.rados.rados.0.smithi203.stdout:2609: finishing write tid 3 to smithi20328086-32 2024-09-17T15:43:32.779 INFO:tasks.rados.rados.0.smithi203.stdout:2609: finishing write tid 4 to smithi20328086-32 2024-09-17T15:43:32.779 INFO:tasks.rados.rados.0.smithi203.stdout:2609: finishing write tid 5 to smithi20328086-32 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2609: oid 32 updating version 0 to 1187 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2609: oid 32 version 1187 is already newer than 1186 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2609: oid 32 updating version 1187 to 1188 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2609: oid 32 version 1188 is already newer than 1185 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1188 (ObjNum 925 snap 280 seq_num 925) dirty exists 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2609: left oid 32 (ObjNum 925 snap 280 seq_num 925) 2024-09-17T15:43:32.780 INFO:tasks.rados.rados.0.smithi203.stdout:2610: expect (ObjNum 853 snap 262 seq_num 853) 2024-09-17T15:43:32.800 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2127324 2024-09-17T15:43:32.814 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1052799 2024-09-17T15:43:32.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: get_monmap_and_config 2024-09-17T15:43:32.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: build_initial_monmap 2024-09-17T15:43:32.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: monmap: 2024-09-17T15:43:32.835 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:32.835 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:32.841217+0000 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:32.841217+0000 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:32.836 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 Processor -- start 2024-09-17T15:43:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 -- start start 2024-09-17T15:43:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: init 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d5809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-b con 0x7f8d580a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d580a5350 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8d5813d180 0x7f8d5813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-c con 0x7f8d5813d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d5809d430 con 0x7f8d580a5350 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8d5809c7f0 con 0x7f8d5813d180 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8d580a68d0 con 0x7f8d580a4f70 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5fb67640 10 monclient(hunting): authenticate will time out at 1945.806700s 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d102640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8d580a5350 0x7f8d580a5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:33500/0 (socket says 172.21.15.86:33500) 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d102640 1 -- 172.21.15.86:0/1826839719 learned_addr learned my addr 172.21.15.86:0/1826839719 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 --2- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d5809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): get_auth_request con 0x7f8d580a4f70 auth_method 0 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5e104640 1 --2- 172.21.15.86:0/1826839719 >> v2:172.21.15.203:3300/0 conn(0x7f8d5813d180 0x7f8d5813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5e104640 10 monclient(hunting): get_auth_request con 0x7f8d5813d180 auth_method 0 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5e104640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5e104640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5e104640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient(hunting): handle_auth_done global_id 164339 payload 274 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 -- 172.21.15.86:0/1826839719 >> v2:172.21.15.203:3300/0 conn(0x7f8d5813d180 msgr2=0x7f8d5813f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 --2- 172.21.15.86:0/1826839719 >> v2:172.21.15.203:3300/0 conn(0x7f8d5813d180 0x7f8d5813f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 -- 172.21.15.86:0/1826839719 >> v1:172.21.15.86:6789/0 conn(0x7f8d580a5350 legacy=0x7f8d580a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:32.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient: _finish_hunting 0 2024-09-17T15:43:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 monclient: found mon.noname-b 2024-09-17T15:43:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:32.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.838+0000 7f8d5d903640 1 -- 172.21.15.86:0/1826839719 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d580a64a0 con 0x7f8d580a4f70 2024-09-17T15:43:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 -- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 msgr2=0x7f8d5809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 --2- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d5809cef0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d5809e420 tx=0x7f8d4801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5593219506b85272 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 1 -- 172.21.15.86:0/1826839719 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480b0070 con 0x7f8d580a4f70 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: dump: 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.847 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.848 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: _finish_auth 0 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: _check_auth_tickets 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 5 monclient: authenticate success, global_id 164339 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 1 -- 172.21.15.86:0/1826839719 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d480ab600 con 0x7f8d580a4f70 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 1 -- 172.21.15.86:0/1826839719 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480ab9d0 con 0x7f8d580a4f70 2024-09-17T15:43:32.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 4 set_mon_vals no callback set 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5c901640 10 monclient: dump: 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.850 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: shutdown 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 msgr2=0x7f8d5809cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d5809cef0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d5809e420 tx=0x7f8d4801c330 comp rx=0 tx=0).stop 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- 172.21.15.86:0/1826839719 shutdown_connections 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/1826839719 >> v2:172.21.15.203:3300/0 conn(0x7f8d5813d180 0x7f8d5813f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/1826839719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d5809cef0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- 172.21.15.86:0/1826839719 >> 172.21.15.86:0/1826839719 conn(0x7f8d580a7240 msgr2=0x7f8d580a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- 172.21.15.86:0/1826839719 shutdown_connections 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- 172.21.15.86:0/1826839719 wait complete. 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: build_initial_monmap 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: monmap: 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:32.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 Processor -- start 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- start start 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: init 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:32.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-c con 0x7f8d580a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 0x7f8d58132760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-a con 0x7f8d580a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d5813d180 0x7f8d581cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): picked mon.noname-b con 0x7f8d5813d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8d581424e0 con 0x7f8d580a5350 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5806f9a0 con 0x7f8d5813d180 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8d58142660 con 0x7f8d580a4f70 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5fb67640 10 monclient(hunting): authenticate will time out at 1945.814491s 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 0x7f8d58132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 0x7f8d58132760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55420/0 (socket says 172.21.15.86:55420) 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36524/0 (socket says 172.21.15.86:36524) 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 1 -- 172.21.15.86:0/347936207 learned_addr learned my addr 172.21.15.86:0/347936207 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): get_auth_request con 0x7f8d580a5350 auth_method 0 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): get_auth_request con 0x7f8d580a4f70 auth_method 0 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5e104640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d5813d180 0x7f8d581cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5e104640 10 monclient(hunting): get_auth_request con 0x7f8d5813d180 auth_method 0 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5e104640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5e104640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5e104640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient(hunting): handle_auth_done global_id 164342 payload 824 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d5813d180 msgr2=0x7f8d581cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d5813d180 0x7f8d581cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 msgr2=0x7f8d58132760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 0x7f8d58132760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient: _finish_hunting 0 2024-09-17T15:43:32.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 monclient: found mon.noname-c 2024-09-17T15:43:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d903640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d580a07b0 con 0x7f8d580a4f70 2024-09-17T15:43:32.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.846+0000 7f8d5d102640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a5350 0x7f8d58132760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:32.859 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1508610 2024-09-17T15:43:32.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d903640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 msgr2=0x7f8d58132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d903640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8d5809c970 tx=0x7f8d480190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480b0040 con 0x7f8d580a4f70 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: dump: 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.864 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d480b9070 con 0x7f8d580a4f70 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d480aa7d0 con 0x7f8d580a4f70 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:32.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 10 monclient: dump: 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 5 monclient: authenticate success, global_id 164342 2024-09-17T15:43:32.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 monclient: _renew_subs 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4f7fe640 4 set_mon_vals no callback set 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5813dcd0 con 0x7f8d580a4f70 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 monclient: _renew_subs 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d580a1760 con 0x7f8d580a4f70 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 0x7f8d581d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8d5809cef0 con 0x7f8d580a0f30 2024-09-17T15:43:32.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 0x7f8d581d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient: get_auth_request con 0x7f8d580a0f30 auth_method 0 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8d480aa970 con 0x7f8d580a4f70 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 10 monclient(hunting): handle_auth_done global_id 184137 payload 824 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5d102640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d580a5ff0 con 0x7f8d580a0f30 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8d3c03bbb0 0x7f8d3c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7f8d480e9c30 con 0x7f8d580a4f70 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5e104640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8d3c03bbb0 0x7f8d3c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.850+0000 7f8d5e104640 10 monclient: get_auth_request con 0x7f8d3c03bbb0 auth_method 0 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d5d102640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 msgr2=0x7f8d581d0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d5d102640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 0x7f8d581d0080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54007770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400fd30 con 0x7f8d580a0f30 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 10 monclient: dump: 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.870 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d5400fd30 con 0x7f8d580a0f30 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:32.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 msgr2=0x7f8d581d0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 0x7f8d581d0080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54007770 comp rx=0 tx=0).stop 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 msgr2=0x7f8d581d0080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d4e7fc640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d580a0f30 0x7f8d581d0080 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54007770 comp rx=0 tx=0).stop 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.854+0000 7f8d5e104640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8d3c03bbb0 0x7f8d3c03e070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a0ca0 tx=0x7f8d5000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5fb67640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 0x7f8d581361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5fb67640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8d581d09d0 con 0x7f8d581d05c0 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 0x7f8d581361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient: get_auth_request con 0x7f8d581d05c0 auth_method 0 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:32.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 10 monclient(hunting): handle_auth_done global_id 184140 payload 824 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.858+0000 7f8d5d102640 1 -- 172.21.15.86:0/347936207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d5809efb0 con 0x7f8d581d05c0 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5d102640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 msgr2=0x7f8d581361b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5d102640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 0x7f8d581361b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d54006030 con 0x7f8d581d05c0 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:32.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 10 monclient: dump: 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:32.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f8d54006030 con 0x7f8d581d05c0 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 msgr2=0x7f8d581361b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 0x7f8d581361b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54004950 comp rx=0 tx=0).stop 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 msgr2=0x7f8d581361b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d4e7fc640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8d581d05c0 0x7f8d581361b0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a45c0 tx=0x7f8d54004950 comp rx=0 tx=0).stop 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:32.875 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 16355, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:43:32.876 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:32.877 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:32.878 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:32.879 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:32.880 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:32.881 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:32.882 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.883 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:32.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.886 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.887 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.888 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:32.889 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:32.891 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:32.892 INFO:tasks.rados.rados.0.smithi203.stdout:2611: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:43:32.892 INFO:tasks.rados.rados.0.smithi203.stdout:2611: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:43:32.892 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1172 (ObjNum 417 snap 0 seq_num 3606611872) dirty dne 2024-09-17T15:43:32.892 INFO:tasks.rados.rados.0.smithi203.stdout:2608: done (3 left) 2024-09-17T15:43:32.893 INFO:tasks.rados.rados.0.smithi203.stdout:2609: done (2 left) 2024-09-17T15:43:32.893 INFO:tasks.rados.rados.0.smithi203.stdout:2610: done (1 left) 2024-09-17T15:43:32.893 INFO:tasks.rados.rados.0.smithi203.stdout:2611: done (0 left) 2024-09-17T15:43:32.893 INFO:tasks.rados.rados.0.smithi203.stdout:2612: snap_create 2024-09-17T15:43:32.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8d3c03bbb0 msgr2=0x7f8d3c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8d3c03bbb0 0x7f8d3c03e070 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8d580a0ca0 tx=0x7f8d5000b040 comp rx=0 tx=0).stop 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 10 monclient: shutdown 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 msgr2=0x7f8d58132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8d5809c970 tx=0x7f8d480190d0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5e104640 1 -- 172.21.15.86:0/347936207 reap_dead start 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 shutdown_connections 2024-09-17T15:43:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 --2- 172.21.15.86:0/347936207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8d580a4f70 0x7f8d58132220 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 >> 172.21.15.86:0/347936207 conn(0x7f8d580a7240 msgr2=0x7f8d5813f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 shutdown_connections 2024-09-17T15:43:32.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:32.862+0000 7f8d5fb67640 1 -- 172.21.15.86:0/347936207 wait complete. 2024-09-17T15:43:32.895 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:32.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:43:32.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:43:33.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: get_monmap_and_config 2024-09-17T15:43:33.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: monmap: 2024-09-17T15:43:33.010 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.010 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:33.016167+0000 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:33.016167+0000 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:33.011 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 Processor -- start 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 -- start start 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: init 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3040a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3040a5bd0 0x7fd30409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-a con 0x7fd3040a5bd0 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd30413d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3040a6280 con 0x7fd3040a50e0 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd30409c0a0 con 0x7fd30413d2f0 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd30409d2a0 con 0x7fd3040a5bd0 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3096c6640 10 monclient(hunting): authenticate will time out at 1945.981741s 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3040a5bd0 0x7fd30409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3040a5bd0 0x7fd30409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55438/0 (socket says 172.21.15.86:55438) 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 1 -- 172.21.15.86:0/3150747761 learned_addr learned my addr 172.21.15.86:0/3150747761 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 10 monclient(hunting): get_auth_request con 0x7fd3040a5bd0 auth_method 0 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.014+0000 7fd3027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd302ffd640 1 -- 172.21.15.86:0/3150747761 >> v1:172.21.15.86:6789/0 conn(0x7fd3040a50e0 legacy=0x7fd3040a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient(hunting): handle_auth_done global_id 184143 payload 274 2024-09-17T15:43:33.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 1 -- 172.21.15.86:0/3150747761 >> v1:172.21.15.203:6789/0 conn(0x7fd30413d2f0 legacy=0x7fd30413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 1 -- 172.21.15.86:0/3150747761 >> v1:172.21.15.86:6789/0 conn(0x7fd3040a50e0 legacy=0x7fd3040a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 1 monclient: found mon.noname-a 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.018+0000 7fd3027fc640 1 -- 172.21.15.86:0/3150747761 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd30409c3d0 con 0x7fd3040a5bd0 2024-09-17T15:43:33.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3027fc640 1 -- 172.21.15.86:0/3150747761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3040a5bd0 msgr2=0x7fd30409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3027fc640 1 --2- 172.21.15.86:0/3150747761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3040a5bd0 0x7fd30409cd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0098f0 tx=0x7fd2ec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449b7d9f663f911 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 1 -- 172.21.15.86:0/3150747761 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0b1070 con 0x7fd3040a5bd0 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: dump: 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 1 -- 172.21.15.86:0/3150747761 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ec0ac410 con 0x7fd3040a5bd0 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 5 monclient: authenticate success, global_id 184143 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 4 set_mon_vals no callback set 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 1 -- 172.21.15.86:0/3150747761 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec0ac7e0 con 0x7fd3040a5bd0 2024-09-17T15:43:33.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd301ffb640 10 monclient: dump: 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.024 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 10 monclient: shutdown 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- 172.21.15.86:0/3150747761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3040a5bd0 msgr2=0x7fd30409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3150747761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3040a5bd0 0x7fd30409cd60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0098f0 tx=0x7fd2ec01c4a0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- 172.21.15.86:0/3150747761 shutdown_connections 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3150747761 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3040a5bd0 0x7fd30409cd60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- 172.21.15.86:0/3150747761 >> 172.21.15.86:0/3150747761 conn(0x7fd3040aaae0 msgr2=0x7fd3040aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- 172.21.15.86:0/3150747761 shutdown_connections 2024-09-17T15:43:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- 172.21.15.86:0/3150747761 wait complete. 2024-09-17T15:43:33.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 10 monclient: monmap: 2024-09-17T15:43:33.030 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:33.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 Processor -- start 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 1 -- start start 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.022+0000 7fd3096c6640 10 monclient: init 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3040a50e0 0x7fd304138180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-b con 0x7fd3040a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3040a5bd0 0x7fd30413a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-c con 0x7fd3040a5bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:33.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 0x7fd30413a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): picked mon.noname-a con 0x7fd30413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd3041439b0 con 0x7fd30413d2f0 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd304143830 con 0x7fd3040a50e0 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd304143b30 con 0x7fd3040a5bd0 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient(hunting): authenticate will time out at 1945.992252s 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3040a5bd0 0x7fd30413a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 0x7fd30413a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 0x7fd30413a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55444/0 (socket says 172.21.15.86:55444) 2024-09-17T15:43:33.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3040a5bd0 0x7fd30413a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36526/0 (socket says 172.21.15.86:36526) 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 learned_addr learned my addr 172.21.15.86:0/3900790856 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): get_auth_request con 0x7fd30413d2f0 auth_method 0 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): get_auth_request con 0x7fd3040a5bd0 auth_method 0 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd302ffd640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3040a50e0 0x7fd304138180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd302ffd640 10 monclient(hunting): get_auth_request con 0x7fd3040a50e0 auth_method 0 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient(hunting): handle_auth_done global_id 184146 payload 824 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3040a5bd0 msgr2=0x7fd30413a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3040a5bd0 0x7fd30413a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3040a50e0 msgr2=0x7fd304138180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3040a50e0 0x7fd304138180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 monclient: found mon.noname-a 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd304140990 con 0x7fd30413d2f0 2024-09-17T15:43:33.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 msgr2=0x7fd30413a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3037fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 0x7fd30413a820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c3c0 tx=0x7fd2f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40119a0 con 0x7fd30413d2f0 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: dump: 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.037 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2f4017070 con 0x7fd30413d2f0 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2f4011d80 con 0x7fd30413d2f0 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:33.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd2fb7fe640 10 monclient: dump: 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 5 monclient: authenticate success, global_id 184146 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _renew_subs 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd300ff9640 4 set_mon_vals no callback set 2024-09-17T15:43:33.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3040a18d0 con 0x7fd30413d2f0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _renew_subs 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd304121b00 con 0x7fd30413d2f0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.026+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 0x7fd30409e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd30413b140 con 0x7fd30413ad60 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 0x7fd30409e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient: get_auth_request con 0x7fd30413ad60 auth_method 0 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd2f401c460 con 0x7fd30413d2f0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 10 monclient(hunting): handle_auth_done global_id 164348 payload 824 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd30409cd60 con 0x7fd30413ad60 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2e403bbb0 0x7fd2e403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7fd2f404fa30 con 0x7fd30413d2f0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd3027fc640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2e403bbb0 0x7fd2e403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd3027fc640 10 monclient: get_auth_request con 0x7fd2e403bbb0 auth_method 0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 msgr2=0x7fd30409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd302ffd640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 0x7fd30409e5e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc00ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00c640 con 0x7fd30413ad60 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 10 monclient: dump: 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd2fc018070 con 0x7fd30413ad60 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:33.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 msgr2=0x7fd30409e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 0x7fd30409e5e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc00ec60 comp rx=0 tx=0).stop 2024-09-17T15:43:33.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 msgr2=0x7fd30409e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:33.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd2fb7fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30413ad60 0x7fd30409e5e0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc00ec60 comp rx=0 tx=0).stop 2024-09-17T15:43:33.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.030+0000 7fd3027fc640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2e403bbb0 0x7fd2e403e070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0ac000 tx=0x7fd2ec0a6040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd3096c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 0x7fd30409ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd3096c6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd30413c8b0 con 0x7fd30409eb20 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 0x7fd30409ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient: get_auth_request con 0x7fd30409eb20 auth_method 0 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 10 monclient(hunting): handle_auth_done global_id 164351 payload 824 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 1 -- 172.21.15.86:0/3900790856 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd304121620 con 0x7fd30409eb20 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 msgr2=0x7fd30409ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd302ffd640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 0x7fd30409ef30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc001260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc00ca50 con 0x7fd30409eb20 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd2fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.038+0000 7fd2fb7fe640 10 monclient: dump: 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.046 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fd2fc00ca50 con 0x7fd30409eb20 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 msgr2=0x7fd30409ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 0x7fd30409ef30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc001260 comp rx=0 tx=0).stop 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 msgr2=0x7fd30409ef30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd2fb7fe640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd30409eb20 0x7fd30409ef30 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd30413cb30 tx=0x7fd2fc001260 comp rx=0 tx=0).stop 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:33.048 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 116226, 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.049 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.050 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.051 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:33.052 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.053 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:33.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.055 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:33.056 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.058 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.059 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.060 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:33.061 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2e403bbb0 msgr2=0x7fd2e403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2e403bbb0 0x7fd2e403e070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0ac000 tx=0x7fd2ec0a6040 comp rx=0 tx=0).stop 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 10 monclient: shutdown 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 msgr2=0x7fd30413a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 --2- 172.21.15.86:0/3900790856 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd30413d2f0 0x7fd30413a820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400c3c0 tx=0x7fd2f400c880 comp rx=0 tx=0).stop 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 reap_dead start 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3037fe640 1 -- 172.21.15.86:0/3900790856 reap_dead start 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 shutdown_connections 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 >> 172.21.15.86:0/3900790856 conn(0x7fd3040aaae0 msgr2=0x7fd3040a3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 shutdown_connections 2024-09-17T15:43:33.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.042+0000 7fd3096c6640 1 -- 172.21.15.86:0/3900790856 wait complete. 2024-09-17T15:43:33.064 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:33.064 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:43:33.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:43:33.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: get_monmap_and_config 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: monmap: 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:33.185564+0000 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:33.185564+0000 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:33.180 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:33.181 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:33.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 Processor -- start 2024-09-17T15:43:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 -- start start 2024-09-17T15:43:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: init 2024-09-17T15:43:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f871c09d770 0x7f871c09db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-a con 0x7f871c09d770 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f871c09e090 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f871c135150 0x7f871c13cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-c con 0x7f871c135150 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f871c09ff00 con 0x7f871c09e090 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f871c09e7f0 con 0x7f871c09d770 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f871c0a2110 con 0x7f871c135150 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f8720bda640 10 monclient(hunting): authenticate will time out at 1946.151202s 2024-09-17T15:43:33.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f871c09d770 0x7f871c09db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f871c09d770 0x7f871c09db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55458/0 (socket says 172.21.15.86:55458) 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 -- 172.21.15.86:0/2898529376 learned_addr learned my addr 172.21.15.86:0/2898529376 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): get_auth_request con 0x7f871c09d770 auth_method 0 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871ad9d640 1 --2- 172.21.15.86:0/2898529376 >> v2:172.21.15.203:3300/0 conn(0x7f871c135150 0x7f871c13cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient(hunting): handle_auth_done global_id 184149 payload 274 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 -- 172.21.15.86:0/2898529376 >> v2:172.21.15.203:3300/0 conn(0x7f871c135150 msgr2=0x7f871c13cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 --2- 172.21.15.86:0/2898529376 >> v2:172.21.15.203:3300/0 conn(0x7f871c135150 0x7f871c13cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 -- 172.21.15.86:0/2898529376 >> v1:172.21.15.203:6789/0 conn(0x7f871c09e090 legacy=0x7f871c134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 monclient: found mon.noname-a 2024-09-17T15:43:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.182+0000 7f871a59c640 1 -- 172.21.15.86:0/2898529376 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c09ec30 con 0x7f871c09d770 2024-09-17T15:43:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 1 -- 172.21.15.86:0/2898529376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c09d770 msgr2=0x7f871c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 1 --2- 172.21.15.86:0/2898529376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c09d770 0x7f871c09db50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f871c09d290 tx=0x7f871401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d035f419cb8ae7e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 1 -- 172.21.15.86:0/2898529376 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140a5530 con 0x7f871c09d770 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: dump: 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 1 -- 172.21.15.86:0/2898529376 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87140a56d0 con 0x7f871c09d770 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 5 monclient: authenticate success, global_id 184149 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 4 set_mon_vals no callback set 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 1 -- 172.21.15.86:0/2898529376 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140a5aa0 con 0x7f871c09d770 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871959a640 10 monclient: dump: 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: shutdown 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- 172.21.15.86:0/2898529376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c09d770 msgr2=0x7f871c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- 172.21.15.86:0/2898529376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c09d770 0x7f871c09db50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f871c09d290 tx=0x7f871401c8b0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- 172.21.15.86:0/2898529376 shutdown_connections 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- 172.21.15.86:0/2898529376 >> v2:172.21.15.203:3300/0 conn(0x7f871c135150 0x7f871c13cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- 172.21.15.86:0/2898529376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c09d770 0x7f871c09db50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- 172.21.15.86:0/2898529376 >> 172.21.15.86:0/2898529376 conn(0x7f871c0a2f40 msgr2=0x7f871c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- 172.21.15.86:0/2898529376 shutdown_connections 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- 172.21.15.86:0/2898529376 wait complete. 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: monmap: 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.192 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:33.193 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:33.193 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 Processor -- start 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- start start 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: init 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c09d770 0x7f871c1d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-b con 0x7f871c09d770 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 0x7f871c133f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-c con 0x7f871c09e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): picked mon.noname-a con 0x7f871c135150 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f871c09cce0 con 0x7f871c135150 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f871c0a0750 con 0x7f871c09d770 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f871c09ce60 con 0x7f871c09e090 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8720bda640 10 monclient(hunting): authenticate will time out at 1946.158771s 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 0x7f871c133f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55464/0 (socket says 172.21.15.86:55464) 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 learned_addr learned my addr 172.21.15.86:0/309246741 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 0x7f871c133f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36562/0 (socket says 172.21.15.86:36562) 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): get_auth_request con 0x7f871c135150 auth_method 0 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): get_auth_request con 0x7f871c09e090 auth_method 0 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c09d770 0x7f871c1d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 10 monclient(hunting): get_auth_request con 0x7f871c09d770 auth_method 0 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient(hunting): handle_auth_done global_id 184152 payload 824 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 msgr2=0x7f871c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 0x7f871c133f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c09d770 msgr2=0x7f871c1d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c09d770 0x7f871c1d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 monclient: found mon.noname-a 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c1210a0 con 0x7f871c135150 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.190+0000 7f8719d9b640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871c09e090 0x7f871c133f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:33.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 msgr2=0x7f871c1344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871ad9d640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c5f0 tx=0x7f870c00cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c015070 con 0x7f871c135150 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c004220 con 0x7f871c135150 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 5 monclient: authenticate success, global_id 184152 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 monclient: _renew_subs 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c0a56c0 con 0x7f871c135150 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c011810 con 0x7f871c135150 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8718d99640 4 set_mon_vals no callback set 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 monclient: _renew_subs 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c121940 con 0x7f871c135150 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 0x7f871c11db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f871c09db50 con 0x7f871c120140 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f870c0119b0 con 0x7f871c135150 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 0x7f871c11db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient: get_auth_request con 0x7f871c120140 auth_method 0 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f86fc03bbe0 0x7f86fc03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7f870c04fde0 con 0x7f871c135150 2024-09-17T15:43:33.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8719d9b640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f86fc03bbe0 0x7f86fc03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f8719d9b640 10 monclient: get_auth_request con 0x7f86fc03bbe0 auth_method 0 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 10 monclient(hunting): handle_auth_done global_id 174211 payload 824 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.194+0000 7f871a59c640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f871c0008e0 con 0x7f871c120140 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f871a59c640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 msgr2=0x7f871c11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f871a59c640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 0x7f871c11db20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87140ae000 tx=0x7f8714019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8719d9b640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f86fc03bbe0 0x7f86fc03e0a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f871c120a80 tx=0x7f8704007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87140a5e60 con 0x7f871c120140 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.210 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87140b6830 con 0x7f871c120140 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 msgr2=0x7f871c11db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 0x7f871c11db20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87140ae000 tx=0x7f8714019650 comp rx=0 tx=0).stop 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 msgr2=0x7f871c11db20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:33.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.198+0000 7f8712ffd640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c120140 0x7f871c11db20 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87140ae000 tx=0x7f8714019650 comp rx=0 tx=0).stop 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f8720bda640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f8720bda640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 0x7f871c11e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f8720bda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f871c11e9b0 con 0x7f871c11e060 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 0x7f871c11e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient: get_auth_request con 0x7f871c11e060 auth_method 0 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 10 monclient(hunting): handle_auth_done global_id 174214 payload 824 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.206+0000 7f871a59c640 1 -- 172.21.15.86:0/309246741 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f871c1db470 con 0x7f871c11e060 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f871a59c640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 msgr2=0x7f871c11e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f871a59c640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 0x7f871c11e470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8714000f90 tx=0x7f87140a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714004030 con 0x7f871c11e060 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 10 monclient: dump: 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f87140c5070 con 0x7f871c11e060 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 msgr2=0x7f871c11e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 0x7f871c11e470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8714000f90 tx=0x7f87140a7000 comp rx=0 tx=0).stop 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 msgr2=0x7f871c11e470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8712ffd640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f871c11e060 0x7f871c11e470 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8714000f90 tx=0x7f87140a7000 comp rx=0 tx=0).stop 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 150, 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 67998, 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:33.216 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.217 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.218 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:33.219 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:33.220 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:33.221 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.225 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:33.226 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:33.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:43:33.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:33.229 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:33.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f86fc03bbe0 msgr2=0x7f86fc03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f86fc03bbe0 0x7f86fc03e0a0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f871c120a80 tx=0x7f8704007e40 comp rx=0 tx=0).stop 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 10 monclient: shutdown 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 msgr2=0x7f871c1344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f871ad9d640 1 -- 172.21.15.86:0/309246741 reap_dead start 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f870c00c5f0 tx=0x7f870c00cab0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 shutdown_connections 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 --2- 172.21.15.86:0/309246741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f871c135150 0x7f871c1344c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 >> 172.21.15.86:0/309246741 conn(0x7f871c0a2f40 msgr2=0x7f871c0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 shutdown_connections 2024-09-17T15:43:33.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.210+0000 7f8720bda640 1 -- 172.21.15.86:0/309246741 wait complete. 2024-09-17T15:43:33.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:43:33.232 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:43:33.232 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:43:33.232 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:43:33.232 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:43:33.232 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:43:33.232 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:43:33.252 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:43:33.252 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:43:33.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.362+0000 7f69dcb1d640 10 monclient: get_monmap_and_config 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.362+0000 7f69dcb1d640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.362+0000 7f69dcb1d640 10 monclient: monmap: 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:33.367923+0000 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:33.367923+0000 2024-09-17T15:43:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:33.363 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 Processor -- start 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 -- start start 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient: init 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 0x7f69d8073e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f69d80a2ec0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f69d80a32a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:33.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69d8135d80 0x7f69d8136210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f69d8135d80 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f69d80a49f0 con 0x7f69d8135d80 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f69d8073190 con 0x7f69d80a2ec0 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69d8073470 con 0x7f69d80a32a0 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69dcb1d640 10 monclient(hunting): authenticate will time out at 1946.333842s 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36574/0 (socket says 172.21.15.86:36574) 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 -- 172.21.15.86:0/2835478956 learned_addr learned my addr 172.21.15.86:0/2835478956 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): get_auth_request con 0x7f69d80a32a0 auth_method 0 2024-09-17T15:43:33.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69d8135d80 0x7f69d8136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d659c640 1 -- 172.21.15.86:0/2835478956 >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 msgr2=0x7f69d8073e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d659c640 1 --2- 172.21.15.86:0/2835478956 >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 0x7f69d8073e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): get_auth_request con 0x7f69d8135d80 auth_method 0 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient(hunting): handle_auth_done global_id 164357 payload 274 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 -- 172.21.15.86:0/2835478956 >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 msgr2=0x7f69d8073e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/2835478956 >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 0x7f69d8073e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 -- 172.21.15.86:0/2835478956 >> v2:172.21.15.86:3300/0 conn(0x7f69d8135d80 msgr2=0x7f69d8136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/2835478956 >> v2:172.21.15.86:3300/0 conn(0x7f69d8135d80 0x7f69d8136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 monclient: found mon.noname-b 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:33.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.366+0000 7f69d5d9b640 1 -- 172.21.15.86:0/2835478956 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d8073750 con 0x7f69d80a32a0 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d5d9b640 1 -- 172.21.15.86:0/2835478956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 msgr2=0x7f69d80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/2835478956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a91b0 tx=0x7f69c801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbed4b7740417d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 1 -- 172.21.15.86:0/2835478956 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c80b1070 con 0x7f69d80a32a0 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: dump: 2024-09-17T15:43:33.370 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 5 monclient: authenticate success, global_id 164357 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 1 -- 172.21.15.86:0/2835478956 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c80ac620 con 0x7f69d80a32a0 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 1 -- 172.21.15.86:0/2835478956 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c80ac9f0 con 0x7f69d80a32a0 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69d559a640 10 monclient: dump: 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.372 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 4 set_mon_vals no callback set 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 10 monclient: shutdown 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 -- 172.21.15.86:0/2835478956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 msgr2=0x7f69d80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/2835478956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a91b0 tx=0x7f69c801c400 comp rx=0 tx=0).stop 2024-09-17T15:43:33.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 -- 172.21.15.86:0/2835478956 shutdown_connections 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/2835478956 >> v2:172.21.15.86:3300/0 conn(0x7f69d8135d80 0x7f69d8136210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/2835478956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80a32a0 0x7f69d80a36d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/2835478956 >> v2:172.21.15.203:3300/0 conn(0x7f69d80a2ec0 0x7f69d8073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 -- 172.21.15.86:0/2835478956 >> 172.21.15.86:0/2835478956 conn(0x7f69d809be70 msgr2=0x7f69d80a85c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 -- 172.21.15.86:0/2835478956 shutdown_connections 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.370+0000 7f69dcb1d640 1 -- 172.21.15.86:0/2835478956 wait complete. 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient: build_initial_monmap 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient: monmap: 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:33.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 Processor -- start 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 -- start start 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient: init 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f69d80a2ec0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 0x7f69d8135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f69d80a32a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d8135d80 0x7f69d8135760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f69d8135d80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f69d80726f0 con 0x7f69d80a2ec0 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8072570 con 0x7f69d80a32a0 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f69d813fa00 con 0x7f69d8135d80 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69dcb1d640 10 monclient(hunting): authenticate will time out at 1946.341399s 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55484/0 (socket says 172.21.15.86:55484) 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 -- 172.21.15.86:0/125311207 learned_addr learned my addr 172.21.15.86:0/125311207 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): get_auth_request con 0x7f69d80a2ec0 auth_method 0 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d8135d80 0x7f69d8135760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d5d9b640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 msgr2=0x7f69d8135220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 0x7f69d8135220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): get_auth_request con 0x7f69d8135d80 auth_method 0 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:33.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient(hunting): handle_auth_done global_id 184158 payload 824 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d8135d80 msgr2=0x7f69d8135760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d8135d80 0x7f69d8135760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 msgr2=0x7f69d8135220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 0x7f69d8135220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient: _finish_hunting 0 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 monclient: found mon.noname-a 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.374+0000 7f69d659c640 1 -- 172.21.15.86:0/125311207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d80a5b20 con 0x7f69d80a2ec0 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69d659c640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 msgr2=0x7f69d81cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69d659c640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b530 tx=0x7f69c000b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:43:33.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0013020 con 0x7f69d80a2ec0 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: dump: 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 5 monclient: authenticate success, global_id 184158 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c0005070 con 0x7f69d80a2ec0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _renew_subs 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d806fa70 con 0x7f69d80a2ec0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0005460 con 0x7f69d80a2ec0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 10 monclient: dump: 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:33.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69d4d99640 4 set_mon_vals no callback set 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _renew_subs 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d806f790 con 0x7f69d80a2ec0 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d80a36d0 con 0x7f69d80a2ec0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f69c0016020 con 0x7f69d80a2ec0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 0x7f69b803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7f69c004fb60 con 0x7f69d80a2ec0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 0x7f69b803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.378+0000 7f69d5d9b640 10 monclient: get_auth_request con 0x7f69b803bb60 auth_method 0 2024-09-17T15:43:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.382+0000 7f69d5d9b640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 0x7f69b803e020 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a4fd0 tx=0x7f69c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:33.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.386+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f69c001d020 con 0x7f69d80a2ec0 2024-09-17T15:43:33.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.386+0000 7f69ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:33.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.386+0000 7f69ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69dcb1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69dcb1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69dcb1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:33.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f69d806d6e0 con 0x7f69d80a2ec0 2024-09-17T15:43:33.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69ceffd640 1 -- 172.21.15.86:0/125311207 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f69d80a5b20 con 0x7f69d80a2ec0 2024-09-17T15:43:33.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:33.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.602+0000 7f69ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:33.601 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 msgr2=0x7f69b803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 0x7f69b803e020 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a4fd0 tx=0x7f69c8008040 comp rx=0 tx=0).stop 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 10 monclient: shutdown 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 msgr2=0x7f69d81cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69c000b530 tx=0x7f69c000b9f0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 shutdown_connections 2024-09-17T15:43:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f69b803bb60 0x7f69b803e020 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d8135d80 0x7f69d8135760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69d80a32a0 0x7f69d8135220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 --2- 172.21.15.86:0/125311207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80a2ec0 0x7f69d81cff50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 >> 172.21.15.86:0/125311207 conn(0x7f69d809be70 msgr2=0x7f69d809fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 shutdown_connections 2024-09-17T15:43:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:33.606+0000 7f69dcb1d640 1 -- 172.21.15.86:0/125311207 wait complete. 2024-09-17T15:43:36.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:36.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: get_monmap_and_config 2024-09-17T15:43:36.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: build_initial_monmap 2024-09-17T15:43:36.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: monmap: 2024-09-17T15:43:36.748 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:36.748 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:36.753951+0000 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:36.753951+0000 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:36.749 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 1 Processor -- start 2024-09-17T15:43:36.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 1 -- start start 2024-09-17T15:43:36.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: init 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff440a5070 0x7eff4409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-c con 0x7eff440a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff440a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff4413d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff440a6f40 con 0x7eff440a5450 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff440a65e0 con 0x7eff4413d280 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.750+0000 7eff49fa1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7eff4409c2a0 con 0x7eff440a5070 2024-09-17T15:43:36.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 10 monclient(hunting): authenticate will time out at 1949.720023s 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff437fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7eff440a5070 msgr2=0x7eff4409cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff437fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff440a5070 0x7eff4409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7eff440a5450 0x7eff440a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:33520/0 (socket says 172.21.15.86:33520) 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 1 -- 172.21.15.86:0/785483833 learned_addr learned my addr 172.21.15.86:0/785483833 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:36.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 >> v1:172.21.15.203:6789/0 conn(0x7eff4413d280 legacy=0x7eff4413f670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 1 -- 172.21.15.86:0/785483833 >> v1:172.21.15.86:6789/0 conn(0x7eff440a5450 legacy=0x7eff440a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346348495 0 0) 0x7eff440a6f40 con 0x7eff440a5450 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): my global_id is 184161 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff30003530 con 0x7eff440a5450 2024-09-17T15:43:36.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2805907960 0 0) 0x7eff30003530 con 0x7eff440a5450 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 >> v2:172.21.15.203:3300/0 conn(0x7eff440a5070 msgr2=0x7eff4409cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 --2- 172.21.15.86:0/785483833 >> v2:172.21.15.203:3300/0 conn(0x7eff440a5070 0x7eff4409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 >> v1:172.21.15.203:6789/0 conn(0x7eff4413d280 legacy=0x7eff4413f670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4409c580 con 0x7eff440a5450 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 5 monclient: authenticate success, global_id 184161 2024-09-17T15:43:36.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7eff2c0019b0 con 0x7eff440a5450 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: dump: 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:36.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 >> v1:172.21.15.86:6789/0 conn(0x7eff440a5450 legacy=0x7eff440a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff30003a30 0x7eff30003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): picked mon.b con 0x7eff30003a30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff30004810 0x7eff30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): picked mon.c con 0x7eff30004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 0x7eff3000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): picked mon.a con 0x7eff300099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7eff4409c2a0 con 0x7eff300099c0 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff3000c3a0 con 0x7eff30003a30 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff3000c5a0 con 0x7eff30004810 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff42ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff49fa1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 0x7eff3000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff30004810 0x7eff30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff437fe640 1 -- 172.21.15.86:0/785483833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff30003a30 msgr2=0x7eff30003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): get_auth_request con 0x7eff30004810 auth_method 0 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:36.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): get_auth_request con 0x7eff300099c0 auth_method 0 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff437fe640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff30003a30 0x7eff30003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient(hunting): handle_auth_done global_id 184161 payload 293 2024-09-17T15:43:36.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff30004810 msgr2=0x7eff30004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff30004810 0x7eff30004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff30003a30 msgr2=0x7eff30003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff30003a30 0x7eff30003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient: _finish_hunting 0 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 monclient: found mon.a 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff30003530 con 0x7eff300099c0 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient: _finish_auth 0 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.754+0000 7eff3bfff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff30004810 0x7eff30004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:36.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 reap_dead start 2024-09-17T15:43:36.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff43fff640 1 -- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 msgr2=0x7eff3000bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:36.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff43fff640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 0x7eff3000bdb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009e30 tx=0x7eff3c01caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e5f5f4e3562f54 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00c0e0 con 0x7eff300099c0 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: dump: 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.768 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c0a9070 con 0x7eff300099c0 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 1 -- 172.21.15.86:0/785483833 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff3c009a30 con 0x7eff300099c0 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff49fa1640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff49fa1640 4 set_mon_vals no callback set 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:36.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.758+0000 7eff42ffd640 10 monclient: dump: 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: shutdown 2024-09-17T15:43:36.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 msgr2=0x7eff3000bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 0x7eff3000bdb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff3c009e30 tx=0x7eff3c01caf0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- 172.21.15.86:0/785483833 shutdown_connections 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 --2- 172.21.15.86:0/785483833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff300099c0 0x7eff3000bdb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- 172.21.15.86:0/785483833 >> 172.21.15.86:0/785483833 conn(0x7eff440742a0 msgr2=0x7eff440746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- 172.21.15.86:0/785483833 shutdown_connections 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- 172.21.15.86:0/785483833 wait complete. 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: build_initial_monmap 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: monmap: 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:36.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 Processor -- start 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- start start 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: init 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:36.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 0x7eff440a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-b con 0x7eff300099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-c con 0x7eff4409eb30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 0x7eff441cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): picked mon.noname-a con 0x7eff441cf720 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7eff4409d2f0 con 0x7eff441cf720 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff4409d170 con 0x7eff300099c0 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:36.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7eff440a6a70 con 0x7eff4409eb30 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff49fa1640 10 monclient(hunting): authenticate will time out at 1949.729893s 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 0x7eff441cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36596/0 (socket says 172.21.15.86:36596) 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 -- 172.21.15.86:0/2484426963 learned_addr learned my addr 172.21.15.86:0/2484426963 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): get_auth_request con 0x7eff4409eb30 auth_method 0 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:36.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff437fe640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 msgr2=0x7eff440a0b50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): get_auth_request con 0x7eff441cf720 auth_method 0 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff437fe640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 0x7eff440a0b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:36.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient(hunting): handle_auth_done global_id 164363 payload 824 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 msgr2=0x7eff440a0b50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 0x7eff440a0b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 msgr2=0x7eff441cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 0x7eff441cfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 monclient: found mon.noname-c 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff3bfff640 1 -- 172.21.15.86:0/2484426963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff44121cc0 con 0x7eff4409eb30 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.762+0000 7eff43fff640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 0x7eff441cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff3bfff640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 msgr2=0x7eff44138340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:36.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff3bfff640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001b20 tx=0x7eff2c0077a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c007e30 con 0x7eff4409eb30 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: dump: 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:36.777 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: _finish_auth 0 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff2c018070 con 0x7eff4409eb30 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 5 monclient: authenticate success, global_id 164363 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _renew_subs 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff440a4300 con 0x7eff4409eb30 2024-09-17T15:43:36.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00fd40 con 0x7eff4409eb30 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 10 monclient: dump: 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:36.779 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _renew_subs 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff427fc640 4 set_mon_vals no callback set 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff441219e0 con 0x7eff4409eb30 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff440a5b90 con 0x7eff4409eb30 2024-09-17T15:43:36.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7eff2c01d460 con 0x7eff4409eb30 2024-09-17T15:43:36.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 0x7eff2403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:36.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7eff2c04fd80 con 0x7eff4409eb30 2024-09-17T15:43:36.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff437fe640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 0x7eff2403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:36.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.766+0000 7eff437fe640 10 monclient: get_auth_request con 0x7eff2403bb10 auth_method 0 2024-09-17T15:43:36.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.770+0000 7eff437fe640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 0x7eff2403dfd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7eff34006fd0 tx=0x7eff34006da0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:36.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.770+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7eff2c04e790 con 0x7eff4409eb30 2024-09-17T15:43:36.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.770+0000 7eff40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:36.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.770+0000 7eff40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:36.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff49fa1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:36.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff49fa1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:36.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff49fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:36.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff4409fa20 con 0x7eff4409eb30 2024-09-17T15:43:36.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff40ff9640 1 -- 172.21.15.86:0/2484426963 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7eff2c012d40 con 0x7eff4409eb30 2024-09-17T15:43:36.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:36.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:36.998+0000 7eff40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:36.998 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:37.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 msgr2=0x7eff2403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:37.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 0x7eff2403dfd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7eff34006fd0 tx=0x7eff34006da0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 10 monclient: shutdown 2024-09-17T15:43:37.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 msgr2=0x7eff44138340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001b20 tx=0x7eff2c0077a0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 shutdown_connections 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7eff2403bb10 0x7eff2403dfd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff441cf720 0x7eff441cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff4409eb30 0x7eff44138340 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 --2- 172.21.15.86:0/2484426963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff300099c0 0x7eff440a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 >> 172.21.15.86:0/2484426963 conn(0x7eff440742a0 msgr2=0x7eff440a2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:37.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 shutdown_connections 2024-09-17T15:43:37.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:37.002+0000 7eff49fa1640 1 -- 172.21.15.86:0/2484426963 wait complete. 2024-09-17T15:43:40.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: get_monmap_and_config 2024-09-17T15:43:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: build_initial_monmap 2024-09-17T15:43:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: monmap: 2024-09-17T15:43:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:40.144 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:40.149988+0000 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:40.149988+0000 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:40.145 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 Processor -- start 2024-09-17T15:43:40.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 -- start start 2024-09-17T15:43:40.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: init 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4a00689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4a0068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a013d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a013af10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a009d770 con 0x7fa4a0068db0 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a009ed00 con 0x7fa4a00689d0 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a009efe0 con 0x7fa4a013af10 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa4a5a37640 10 monclient(hunting): authenticate will time out at 1953.115824s 2024-09-17T15:43:40.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49effd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fa4a00689d0 legacy=0x7fa4a009ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49e7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa4a0068db0 0x7fa4a00691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49102/0 (socket says 172.21.15.86:49102) 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a013d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49e7fc640 1 -- 172.21.15.86:0/2600423284 learned_addr learned my addr 172.21.15.86:0/2600423284 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49f7fe640 10 monclient(hunting): get_auth_request con 0x7fa4a013af10 auth_method 0 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.146+0000 7fa49f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_done global_id 164366 payload 274 2024-09-17T15:43:40.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 1 -- 172.21.15.86:0/2600423284 >> v1:172.21.15.203:6789/0 conn(0x7fa4a00689d0 legacy=0x7fa4a009ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 1 -- 172.21.15.86:0/2600423284 >> v1:172.21.15.86:6789/0 conn(0x7fa4a0068db0 legacy=0x7fa4a00691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 1 monclient: found mon.noname-b 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:40.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.150+0000 7fa49f7fe640 1 -- 172.21.15.86:0/2600423284 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a009cde0 con 0x7fa4a013af10 2024-09-17T15:43:40.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 1 -- 172.21.15.86:0/2600423284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 msgr2=0x7fa4a013d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:40.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 1 --2- 172.21.15.86:0/2600423284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a013d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097f0 tx=0x7fa49801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b9664da6f3a006 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:40.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 1 -- 172.21.15.86:0/2600423284 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4980b0070 con 0x7fa4a013af10 2024-09-17T15:43:40.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: dump: 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:40.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: _finish_auth 0 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 5 monclient: authenticate success, global_id 164366 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 1 -- 172.21.15.86:0/2600423284 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4980a97f0 con 0x7fa4a013af10 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 4 set_mon_vals no callback set 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 1 -- 172.21.15.86:0/2600423284 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4980a9bc0 con 0x7fa4a013af10 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49dffb640 10 monclient: dump: 2024-09-17T15:43:40.153 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: shutdown 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- 172.21.15.86:0/2600423284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 msgr2=0x7fa4a013d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:40.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/2600423284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a013d300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa4980097f0 tx=0x7fa49801c430 comp rx=0 tx=0).stop 2024-09-17T15:43:40.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- 172.21.15.86:0/2600423284 shutdown_connections 2024-09-17T15:43:40.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/2600423284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a013d300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- 172.21.15.86:0/2600423284 >> 172.21.15.86:0/2600423284 conn(0x7fa4a00a6440 msgr2=0x7fa4a00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:40.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- 172.21.15.86:0/2600423284 shutdown_connections 2024-09-17T15:43:40.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- 172.21.15.86:0/2600423284 wait complete. 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: build_initial_monmap 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: monmap: 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:40.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 Processor -- start 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- start start 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: init 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 0x7fa4a01351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a0068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a01cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a013af10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a0140800 con 0x7fa4a00689d0 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a0140680 con 0x7fa4a0068db0 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a0140980 con 0x7fa4a013af10 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa4a5a37640 10 monclient(hunting): authenticate will time out at 1953.122977s 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:40.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49228/0 (socket says 172.21.15.86:49228) 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a01cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 -- 172.21.15.86:0/3846762498 learned_addr learned my addr 172.21.15.86:0/3846762498 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): get_auth_request con 0x7fa4a00689d0 auth_method 0 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): get_auth_request con 0x7fa4a013af10 auth_method 0 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49e7fc640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 msgr2=0x7fa4a01351a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49e7fc640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 0x7fa4a01351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:40.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient(hunting): handle_auth_done global_id 184167 payload 824 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 msgr2=0x7fa4a01cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a01cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 msgr2=0x7fa4a01351a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 0x7fa4a01351a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient: _finish_hunting 0 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 monclient: found mon.noname-a 2024-09-17T15:43:40.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.154+0000 7fa49effd640 1 -- 172.21.15.86:0/3846762498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a0136f20 con 0x7fa4a00689d0 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa49effd640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 msgr2=0x7fa4a0134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa49effd640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa49000b530 tx=0x7fa49000b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490013020 con 0x7fa4a00689d0 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: dump: 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.164 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: _finish_auth 0 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: _check_auth_tickets 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa490005070 con 0x7fa4a00689d0 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490005460 con 0x7fa4a00689d0 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:40.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 10 monclient: dump: 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 5 monclient: authenticate success, global_id 184167 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _renew_subs 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa49d7fa640 4 set_mon_vals no callback set 2024-09-17T15:43:40.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a01363a0 con 0x7fa4a00689d0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _renew_subs 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a0121430 con 0x7fa4a00689d0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a013ce70 con 0x7fa4a00689d0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa490016020 con 0x7fa4a00689d0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 0x7fa48003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(578..578 src has 438..578) ==== 4755+0+0 (secure 0 0 0) 0x7fa49004fb60 con 0x7fa4a00689d0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa49e7fc640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 0x7fa48003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:40.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.158+0000 7fa49e7fc640 10 monclient: get_auth_request con 0x7fa48003bb10 auth_method 0 2024-09-17T15:43:40.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.162+0000 7fa49e7fc640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 0x7fa48003dfd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0137720 tx=0x7fa488005110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:40.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.162+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa49001d020 con 0x7fa4a00689d0 2024-09-17T15:43:40.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.162+0000 7fa497fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:40.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.162+0000 7fa497fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.386+0000 7fa4a5a37640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.386+0000 7fa4a5a37640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.386+0000 7fa4a5a37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.386+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4a006d4f0 con 0x7fa4a00689d0 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa497fff640 1 -- 172.21.15.86:0/3846762498 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fa4a01363a0 con 0x7fa4a00689d0 2024-09-17T15:43:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa497fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:40.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa497fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:40.387 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 msgr2=0x7fa48003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 0x7fa48003dfd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0137720 tx=0x7fa488005110 comp rx=0 tx=0).stop 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 10 monclient: shutdown 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 msgr2=0x7fa4a0134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa49000b530 tx=0x7fa49000b9f0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 shutdown_connections 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa48003bb10 0x7fa48003dfd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4a013af10 0x7fa4a01cfb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a0068db0 0x7fa4a01351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 --2- 172.21.15.86:0/3846762498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4a00689d0 0x7fa4a0134c60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:40.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 >> 172.21.15.86:0/3846762498 conn(0x7fa4a00a6440 msgr2=0x7fa4a00a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:40.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 shutdown_connections 2024-09-17T15:43:40.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:40.390+0000 7fa4a5a37640 1 -- 172.21.15.86:0/3846762498 wait complete. 2024-09-17T15:43:43.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: get_monmap_and_config 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: build_initial_monmap 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: monmap: 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:43.541079+0000 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:43.541079+0000 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:43.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 Processor -- start 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 -- start start 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: init 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-a con 0x7f5f540a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f540a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:43.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5413d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f5409d350 con 0x7f5f540a54c0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5f5409bdc0 con 0x7f5f540a50e0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5f540a6530 con 0x7f5f5413d2f0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f5a7be640 10 monclient(hunting): authenticate will time out at 1956.506726s 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f58d5b640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 msgr2=0x7f5f5413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f58d5b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f537fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5f540a54c0 0x7f5f540a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49104/0 (socket says 172.21.15.86:49104) 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 learned_addr learned my addr 172.21.15.86:0/2809996847 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 10 monclient(hunting): get_auth_request con 0x7f5f540a50e0 auth_method 0 2024-09-17T15:43:43.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.538+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:43.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.542+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.86:6789/0 conn(0x7f5f540a54c0 legacy=0x7f5f540a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:43.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f58d5b640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 msgr2=0x7f5f5413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:43.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f58d5b640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:43:43.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:43.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 10 monclient(hunting): get_auth_request con 0x7f5f540a50e0 auth_method 0 2024-09-17T15:43:43.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:43.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:43.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:43.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:43:43.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:43.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:43.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:43.742+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 10 monclient(hunting): get_auth_request con 0x7f5f540a50e0 auth_method 0 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f58d5b640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 msgr2=0x7f5f5413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f58d5b640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:43:44.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.146+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f58d5b640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 msgr2=0x7f5f5413f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f58d5b640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 10 monclient(hunting): get_auth_request con 0x7f5f540a50e0 auth_method 0 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:44.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:44.950+0000 7f5f53fff640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): tick 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 msgr2=0x7f5f5413f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.203:3300/0 conn(0x7f5f5413d2f0 0x7f5f5413f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 -- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 msgr2=0x7f5f5409ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 --2- 172.21.15.86:0/2809996847 >> v2:172.21.15.86:3300/0 conn(0x7f5f540a50e0 0x7f5f5409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:46.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.86:6789/0 conn(0x7f5f540a54c0 legacy=0x7f5f540a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient: _add_conns ranks=[1,2] 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f34003740 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f34007410 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 -- 172.21.15.86:0/2809996847 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f5409c150 con 0x7f5f34003740 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 1 -- 172.21.15.86:0/2809996847 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f5409bdc0 con 0x7f5f34007410 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f51ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.203:6789/0 conn(0x7f5f34003740 legacy=0x7f5f34005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 10 monclient(hunting): get_auth_request con 0x7f5f34007410 auth_method 0 2024-09-17T15:43:46.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:46.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.538+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.203:6789/0 conn(0x7f5f34003740 legacy=0x7f5f34005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 10 monclient(hunting): get_auth_request con 0x7f5f34007410 auth_method 0 2024-09-17T15:43:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:46.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:46.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:46.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:43:46.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:46.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:46.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:46.742+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f53fff640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.203:6789/0 conn(0x7f5f34003740 legacy=0x7f5f34005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 10 monclient(hunting): get_auth_request con 0x7f5f34007410 auth_method 0 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:43:47.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:43:47.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:43:47.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.146+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): get_auth_request con 0x7f5f34007410 auth_method 0 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:47.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient(hunting): handle_auth_done global_id 164372 payload 274 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> v1:172.21.15.203:6789/0 conn(0x7f5f34003740 legacy=0x7f5f34005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient: _finish_hunting 0 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 1 monclient: found mon.noname-b 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:47.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.950+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f3400b110 con 0x7f5f34007410 2024-09-17T15:43:47.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f537fe640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:47.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f537fe640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f44002220 tx=0x7f5f4401d760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b22cad3e60ca3b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:47.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 1 -- 172.21.15.86:0/2809996847 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440ac720 con 0x7f5f34007410 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: dump: 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:47.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: _finish_auth 0 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 1 -- 172.21.15.86:0/2809996847 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f440b1070 con 0x7f5f34007410 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 1 -- 172.21.15.86:0/2809996847 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440acae0 con 0x7f5f34007410 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 5 monclient: authenticate success, global_id 164372 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 4 set_mon_vals no callback set 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:47.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f52ffd640 10 monclient: dump: 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:47.958 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:47.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 10 monclient: shutdown 2024-09-17T15:43:47.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 -- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 msgr2=0x7f5f340098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/2809996847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34007410 0x7f5f340098d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5f44002220 tx=0x7f5f4401d760 comp rx=0 tx=0).stop 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f58d5b640 1 -- 172.21.15.86:0/2809996847 reap_dead start 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 -- 172.21.15.86:0/2809996847 shutdown_connections 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 -- 172.21.15.86:0/2809996847 >> 172.21.15.86:0/2809996847 conn(0x7f5f540aaae0 msgr2=0x7f5f540aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 -- 172.21.15.86:0/2809996847 shutdown_connections 2024-09-17T15:43:47.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.958+0000 7f5f5a7be640 1 -- 172.21.15.86:0/2809996847 wait complete. 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient: build_initial_monmap 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient: monmap: 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:47.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 Processor -- start 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 -- start start 2024-09-17T15:43:47.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient: init 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-c con 0x7f5f541cfb20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 0x7f5f54134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-b con 0x7f5f541d0440 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 0x7f5f54135820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): picked mon.noname-a con 0x7f5f541353b0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5f541425c0 con 0x7f5f541353b0 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:47.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f54142440 con 0x7f5f541d0440 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f54142740 con 0x7f5f541cfb20 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f5a7be640 10 monclient(hunting): authenticate will time out at 1960.930400s 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f537fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 msgr2=0x7f5f54134e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 0x7f5f54135820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58458/0 (socket says 172.21.15.86:58458) 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f537fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 0x7f5f54134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 -- 172.21.15.86:0/1835620354 learned_addr learned my addr 172.21.15.86:0/1835620354 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): get_auth_request con 0x7f5f541353b0 auth_method 0 2024-09-17T15:43:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): get_auth_request con 0x7f5f541cfb20 auth_method 0 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:47.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient(hunting): handle_auth_done global_id 164375 payload 824 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 -- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 msgr2=0x7f5f54134e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 0x7f5f54134e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 -- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 msgr2=0x7f5f54135820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 0x7f5f54135820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient: _finish_hunting 0 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 monclient: found mon.noname-c 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f53fff640 1 -- 172.21.15.86:0/1835620354 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f5413ad60 con 0x7f5f541cfb20 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.962+0000 7f5f58d5b640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 0x7f5f54135820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f53fff640 1 -- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 msgr2=0x7f5f541cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f53fff640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0139b0 tx=0x7f5f4c013db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:47.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c00f040 con 0x7f5f541cfb20 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: dump: 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:47.971 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: _finish_auth 0 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f4c00ec60 con 0x7f5f541cfb20 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 5 monclient: authenticate success, global_id 164375 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _renew_subs 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f5413b040 con 0x7f5f541cfb20 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c016460 con 0x7f5f541cfb20 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:47.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 10 monclient: dump: 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f527fc640 4 set_mon_vals no callback set 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _renew_subs 2024-09-17T15:43:47.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f5413d6d0 con 0x7f5f541cfb20 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5f4c016600 con 0x7f5f541cfb20 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f5409ce10 con 0x7f5f541cfb20 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 0x7f5f3403eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(579..579 src has 438..579) ==== 4755+0+0 (secure 0 0 0) 0x7f5f4c0506d0 con 0x7f5f541cfb20 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.966+0000 7f5f537fe640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 0x7f5f3403eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:47.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.970+0000 7f5f537fe640 10 monclient: get_auth_request con 0x7f5f3403cb30 auth_method 0 2024-09-17T15:43:47.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.970+0000 7f5f537fe640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 0x7f5f3403eff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5f440ad000 tx=0x7f5f44010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:47.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.974+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5f4c01a020 con 0x7f5f541cfb20 2024-09-17T15:43:47.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.974+0000 7f5f50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:47.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:47.974+0000 7f5f50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:48.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f5a7be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:43:48.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f5a7be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:48.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f5a7be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:43:48.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5f54139b20 con 0x7f5f541cfb20 2024-09-17T15:43:48.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f50ff9640 1 -- 172.21.15.86:0/1835620354 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5f4c04f0e0 con 0x7f5f541cfb20 2024-09-17T15:43:48.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:43:48.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.210+0000 7f5f50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:48.210 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:43:48.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 msgr2=0x7f5f3403eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 0x7f5f3403eff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5f440ad000 tx=0x7f5f44010040 comp rx=0 tx=0).stop 2024-09-17T15:43:48.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 10 monclient: shutdown 2024-09-17T15:43:48.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:48.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 msgr2=0x7f5f541cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0139b0 tx=0x7f5f4c013db0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 shutdown_connections 2024-09-17T15:43:48.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f3403cb30 0x7f5f3403eff0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f541353b0 0x7f5f54135820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f541d0440 0x7f5f54134e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 --2- 172.21.15.86:0/1835620354 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f541cfb20 0x7f5f541cff00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 >> 172.21.15.86:0/1835620354 conn(0x7f5f540aaae0 msgr2=0x7f5f5413c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:48.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 shutdown_connections 2024-09-17T15:43:48.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.214+0000 7f5f5a7be640 1 -- 172.21.15.86:0/1835620354 wait complete. 2024-09-17T15:43:48.227 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:43:48.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:43:48.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: get_monmap_and_config 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: build_initial_monmap 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: monmap: 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:48.348763+0000 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:48.348763+0000 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:48.343 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:48.344 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 Processor -- start 2024-09-17T15:43:48.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 -- start start 2024-09-17T15:43:48.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: init 2024-09-17T15:43:48.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-b con 0x7fbf4409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-a con 0x7fbf4409e090 addr v2:172.21.15.86:3300/0 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf44134f80 addr v1:172.21.15.203:6789/0 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf4409ff00 con 0x7fbf44134f80 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbf4409e7f0 con 0x7fbf4409e090 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbf440a2110 con 0x7fbf4409d770 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4bb58640 10 monclient(hunting): authenticate will time out at 1961.314608s 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf4a0f5640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fbf44134f80 legacy=0x7fbf4413bf10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:37646/0 (socket says 172.21.15.86:37646) 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 -- 172.21.15.86:0/3819888571 learned_addr learned my addr 172.21.15.86:0/3819888571 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60912/0 (socket says 172.21.15.86:60912) 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): get_auth_request con 0x7fbf4409d770 auth_method 0 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): get_auth_request con 0x7fbf4409e090 auth_method 0 2024-09-17T15:43:48.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient(hunting): handle_auth_done global_id 164378 payload 274 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 -- 172.21.15.86:0/3819888571 >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 msgr2=0x7fbf44134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 --2- 172.21.15.86:0/3819888571 >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 -- 172.21.15.86:0/3819888571 >> v1:172.21.15.203:6789/0 conn(0x7fbf44134f80 legacy=0x7fbf4413bf10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:48.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient: _finish_hunting 0 2024-09-17T15:43:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 monclient: found mon.noname-b 2024-09-17T15:43:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf498f4640 1 -- 172.21.15.86:0/3819888571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4409ec30 con 0x7fbf4409d770 2024-09-17T15:43:48.351 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.346+0000 7fbf490f3640 1 --2- 172.21.15.86:0/3819888571 >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf498f4640 1 -- 172.21.15.86:0/3819888571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 msgr2=0x7fbf4409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf498f4640 1 --2- 172.21.15.86:0/3819888571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009870 tx=0x7fbf4001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a0a1a5bf26a7805 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 1 -- 172.21.15.86:0/3819888571 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400b0070 con 0x7fbf4409d770 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: dump: 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: _finish_auth 0 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: _check_auth_tickets 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 1 -- 172.21.15.86:0/3819888571 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf400ab600 con 0x7fbf4409d770 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf4bb58640 5 monclient: authenticate success, global_id 164378 2024-09-17T15:43:48.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf4bb58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf4bb58640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf4bb58640 4 set_mon_vals no callback set 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 1 -- 172.21.15.86:0/3819888571 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400ab9d0 con 0x7fbf4409d770 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.350+0000 7fbf488f2640 10 monclient: dump: 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: shutdown 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- 172.21.15.86:0/3819888571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 msgr2=0x7fbf4409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/3819888571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf40009870 tx=0x7fbf4001c420 comp rx=0 tx=0).stop 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- 172.21.15.86:0/3819888571 shutdown_connections 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/3819888571 >> v2:172.21.15.86:3300/0 conn(0x7fbf4409e090 0x7fbf44134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/3819888571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409d770 0x7fbf4409db50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- 172.21.15.86:0/3819888571 >> 172.21.15.86:0/3819888571 conn(0x7fbf440a2f40 msgr2=0x7fbf440a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- 172.21.15.86:0/3819888571 shutdown_connections 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- 172.21.15.86:0/3819888571 wait complete. 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: build_initial_monmap 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: monmap: 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.363 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 Processor -- start 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- start start 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: init 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 0x7fbf441cbf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-a con 0x7fbf4409d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409e090 0x7fbf4413c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-c con 0x7fbf4409e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf44134f80 0x7fbf4413c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): picked mon.noname-b con 0x7fbf44134f80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4409cce0 con 0x7fbf4409d770 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbf44135770 con 0x7fbf44134f80 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4409ce60 con 0x7fbf4409e090 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:48.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4bb58640 10 monclient(hunting): authenticate will time out at 1961.322003s 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 0x7fbf441cbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 0x7fbf441cbf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60920/0 (socket says 172.21.15.86:60920) 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 -- 172.21.15.86:0/751393596 learned_addr learned my addr 172.21.15.86:0/751393596 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409e090 0x7fbf4413c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): get_auth_request con 0x7fbf4409d770 auth_method 0 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4a0f5640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf44134f80 msgr2=0x7fbf4413c560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): get_auth_request con 0x7fbf4409e090 auth_method 0 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf4a0f5640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf44134f80 0x7fbf4413c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient(hunting): handle_auth_done global_id 184176 payload 824 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409e090 msgr2=0x7fbf4413c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf4409e090 0x7fbf4413c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf44134f80 msgr2=0x7fbf4413c560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf44134f80 0x7fbf4413c560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient: _finish_hunting 0 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 monclient: found mon.noname-a 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.354+0000 7fbf498f4640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf440a5700 con 0x7fbf4409d770 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf498f4640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 msgr2=0x7fbf441cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf498f4640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 0x7fbf441cbf10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf400ae040 tx=0x7fbf4001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400ba020 con 0x7fbf4409d770 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: dump: 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf400b6630 con 0x7fbf4409d770 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf400b0040 con 0x7fbf4409d770 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 10 monclient: dump: 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 5 monclient: authenticate success, global_id 184176 2024-09-17T15:43:48.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 monclient: _renew_subs 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3bfff640 4 set_mon_vals no callback set 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4409d460 con 0x7fbf4409d770 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 monclient: _renew_subs 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf44139240 con 0x7fbf4409d770 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 0x7fbf441362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbf440a51f0 con 0x7fbf44135ee0 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 0x7fbf441362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient: get_auth_request con 0x7fbf44135ee0 auth_method 0 2024-09-17T15:43:48.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbf400bd020 con 0x7fbf4409d770 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 10 monclient(hunting): handle_auth_done global_id 184179 payload 824 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbf2803bb10 0x7fbf2803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf490f3640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf4409db50 con 0x7fbf44135ee0 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(579..579 src has 438..579) ==== 4755+0+0 (secure 0 0 0) 0x7fbf400ea0d0 con 0x7fbf4409d770 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4a0f5640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbf2803bb10 0x7fbf2803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.358+0000 7fbf4a0f5640 10 monclient: get_auth_request con 0x7fbf2803bb10 auth_method 0 2024-09-17T15:43:48.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf490f3640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 msgr2=0x7fbf441362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf490f3640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 0x7fbf441362c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf44073620 tx=0x7fbf34011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3400fe20 con 0x7fbf44135ee0 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 10 monclient: dump: 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbf3400fe20 con 0x7fbf44135ee0 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 msgr2=0x7fbf441362c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 0x7fbf441362c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf44073620 tx=0x7fbf34011e30 comp rx=0 tx=0).stop 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 msgr2=0x7fbf441362c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf3a7fc640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44135ee0 0x7fbf441362c0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbf44073620 tx=0x7fbf34011e30 comp rx=0 tx=0).stop 2024-09-17T15:43:48.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.362+0000 7fbf4a0f5640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbf2803bb10 0x7fbf2803dfd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c003cc0 tx=0x7fbf3c0091f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf4bb58640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:48.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 0x7fbf44130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf4bb58640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbf44139b40 con 0x7fbf44136800 2024-09-17T15:43:48.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 0x7fbf44130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient: get_auth_request con 0x7fbf44136800 auth_method 0 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 10 monclient(hunting): handle_auth_done global_id 184182 payload 824 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.366+0000 7fbf490f3640 1 -- 172.21.15.86:0/751393596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf4406d4f0 con 0x7fbf44136800 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf490f3640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 msgr2=0x7fbf44130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf490f3640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 0x7fbf44130c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c000 tx=0x7fbf340046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf34022770 con 0x7fbf44136800 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 10 monclient: dump: 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7fbf3402a3e0 con 0x7fbf44136800 2024-09-17T15:43:48.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 msgr2=0x7fbf44130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 0x7fbf44130c10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c000 tx=0x7fbf340046d0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 msgr2=0x7fbf44130c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf3a7fc640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf44136800 0x7fbf44130c10 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c000 tx=0x7fbf340046d0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:48.381 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 154, 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 31863, 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:48.382 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.383 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.384 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:48.385 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.386 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:48.387 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.388 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.391 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:48.392 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.393 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.394 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:48.395 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbf2803bb10 msgr2=0x7fbf2803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbf2803bb10 0x7fbf2803dfd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c003cc0 tx=0x7fbf3c0091f0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 10 monclient: shutdown 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 msgr2=0x7fbf441cbf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 --2- 172.21.15.86:0/751393596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf4409d770 0x7fbf441cbf10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbf400ae040 tx=0x7fbf4001ac60 comp rx=0 tx=0).stop 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4a0f5640 1 -- 172.21.15.86:0/751393596 reap_dead start 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4a0f5640 1 -- 172.21.15.86:0/751393596 reap_dead start 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 shutdown_connections 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 >> 172.21.15.86:0/751393596 conn(0x7fbf440a2f40 msgr2=0x7fbf440a4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 shutdown_connections 2024-09-17T15:43:48.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.370+0000 7fbf4bb58640 1 -- 172.21.15.86:0/751393596 wait complete. 2024-09-17T15:43:48.398 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:43:48.398 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:43:48.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:43:48.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 10 monclient: get_monmap_and_config 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 10 monclient: build_initial_monmap 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 10 monclient: monmap: 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:43:48.518705+0000 2024-09-17T15:43:48.513 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:43:48.518705+0000 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:43:48.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 1 Processor -- start 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 1 -- start start 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.514+0000 7fa17fdcf640 10 monclient: init 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa17809d770 addr v1:172.21.15.86:6789/0 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 0x7fa178139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa17809e260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 0x7fa17813c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa17813a440 addr v2:172.21.15.203:3300/0 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa17809ff00 con 0x7fa17809d770 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa17809e7f0 con 0x7fa17813a440 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa1780a2110 con 0x7fa17809e260 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17fdcf640 10 monclient(hunting): authenticate will time out at 1961.484236s 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17db6b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa17809d770 0x7fa17809db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54440/0 (socket says 172.21.15.86:54440) 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17db6b640 1 -- 172.21.15.86:0/2328995872 learned_addr learned my addr 172.21.15.86:0/2328995872 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:43:48.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17e36c640 1 -- 172.21.15.86:0/2328995872 >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 msgr2=0x7fa17813c830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 --2- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 0x7fa178139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17e36c640 1 --2- 172.21.15.86:0/2328995872 >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 0x7fa17813c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): get_auth_request con 0x7fa17809e260 auth_method 0 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient(hunting): handle_auth_done global_id 164384 payload 274 2024-09-17T15:43:48.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 -- 172.21.15.86:0/2328995872 >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 msgr2=0x7fa17813c830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 --2- 172.21.15.86:0/2328995872 >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 0x7fa17813c830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 -- 172.21.15.86:0/2328995872 >> v1:172.21.15.86:6789/0 conn(0x7fa17809d770 legacy=0x7fa17809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient: _finish_hunting 0 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 monclient: found mon.noname-b 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:43:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.518+0000 7fa17d36a640 1 -- 172.21.15.86:0/2328995872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17809ec30 con 0x7fa17809e260 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 -- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 msgr2=0x7fa178139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 --2- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 0x7fa178139f00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa168009d90 tx=0x7fa16801c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372721e8f7aac439 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 1 -- 172.21.15.86:0/2328995872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1680b2070 con 0x7fa17809e260 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: dump: 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: _finish_auth 0 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: _check_auth_tickets 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 1 -- 172.21.15.86:0/2328995872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1680a5820 con 0x7fa17809e260 2024-09-17T15:43:48.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 5 monclient: authenticate success, global_id 164384 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: get_monmap_and_config success 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 4 set_mon_vals no callback set 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 1 -- 172.21.15.86:0/2328995872 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1680a5bf0 con 0x7fa17809e260 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17cb69640 10 monclient: dump: 2024-09-17T15:43:48.532 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.533 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: shutdown 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 msgr2=0x7fa178139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 0x7fa178139f00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa168009d90 tx=0x7fa16801c850 comp rx=0 tx=0).stop 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- 172.21.15.86:0/2328995872 shutdown_connections 2024-09-17T15:43:48.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/2328995872 >> v2:172.21.15.203:3300/0 conn(0x7fa17813a440 0x7fa17813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/2328995872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17809e260 0x7fa178139f00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- 172.21.15.86:0/2328995872 >> 172.21.15.86:0/2328995872 conn(0x7fa1780a2f40 msgr2=0x7fa1780a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- 172.21.15.86:0/2328995872 shutdown_connections 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- 172.21.15.86:0/2328995872 wait complete. 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: build_initial_monmap 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: monmap: 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:43:48.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 Processor -- start 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- start start 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: init 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: _reopen_session rank -1 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17809d770 0x7fa1781cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa17809d770 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 0x7fa1781335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-a con 0x7fa17809e260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17813a440 0x7fa178133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa17813a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa17813f590 con 0x7fa17809e260 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa1780a0740 con 0x7fa17809d770 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa17813f710 con 0x7fa17813a440 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): _renew_subs 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17fdcf640 10 monclient(hunting): authenticate will time out at 1961.491214s 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 0x7fa1781335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 0x7fa1781335a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60946/0 (socket says 172.21.15.86:60946) 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 -- 172.21.15.86:0/4266290661 learned_addr learned my addr 172.21.15.86:0/4266290661 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): get_auth_request con 0x7fa17809e260 auth_method 0 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17813a440 0x7fa178133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17db6b640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17809d770 msgr2=0x7fa1781cfa90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): get_auth_request con 0x7fa17813a440 auth_method 0 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17db6b640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17809d770 0x7fa1781cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17e36c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient(hunting): handle_auth_done global_id 184185 payload 824 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17813a440 msgr2=0x7fa178133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17813a440 0x7fa178133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17809d770 msgr2=0x7fa1781cfa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa17809d770 0x7fa1781cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient: _finish_hunting 0 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 monclient: found mon.noname-a 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:43:48.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.522+0000 7fa17d36a640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa178120ca0 con 0x7fa17809e260 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17d36a640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 msgr2=0x7fa1781335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17d36a640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 0x7fa1781335a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa16801c680 tx=0x7fa1680195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1680bd020 con 0x7fa17809e260 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: dump: 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: _finish_auth 0 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: _check_auth_tickets 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 5 monclient: authenticate success, global_id 184185 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 monclient: _renew_subs 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1680b3040 con 0x7fa17809e260 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa178120f80 con 0x7fa17809e260 2024-09-17T15:43:48.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1680b2070 con 0x7fa17809e260 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 10 monclient: dump: 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.544 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 monclient: _renew_subs 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1777fe640 4 set_mon_vals no callback set 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa17811fc30 con 0x7fa17809e260 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 0x7fa17811d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa17809d4f0 con 0x7fa1781206e0 2024-09-17T15:43:48.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 0x7fa17811d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient: get_auth_request con 0x7fa1781206e0 auth_method 0 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa1680c0020 con 0x7fa17809e260 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 10 monclient(hunting): handle_auth_done global_id 164390 payload 824 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16403bb40 0x7fa16403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17db6b640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa17811ec80 con 0x7fa1781206e0 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(579..579 src has 438..579) ==== 4755+0+0 (secure 0 0 0) 0x7fa1680eaae0 con 0x7fa17809e260 2024-09-17T15:43:48.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17e36c640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16403bb40 0x7fa16403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.526+0000 7fa17e36c640 10 monclient: get_auth_request con 0x7fa16403bb40 auth_method 0 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa17db6b640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 msgr2=0x7fa17811d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa17db6b640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 0x7fa17811d160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa16c004880 tx=0x7fa16c00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c017070 con 0x7fa1781206e0 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 10 monclient: dump: 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa16c0070d0 con 0x7fa1781206e0 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 msgr2=0x7fa17811d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 0x7fa17811d160 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa16c004880 tx=0x7fa16c00c8e0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 msgr2=0x7fa17811d160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:48.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa1767fc640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1781206e0 0x7fa17811d160 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.530+0000 7fa17e36c640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16403bb40 0x7fa16403e000 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa17811f510 tx=0x7fa170003800 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.534+0000 7fa17fdcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.534+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 0x7fa17811dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.534+0000 7fa17fdcf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.534+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa17811dff0 con 0x7fa17811d6a0 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 0x7fa17811dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient: get_auth_request con 0x7fa17811d6a0 auth_method 0 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:43:48.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 10 monclient(hunting): handle_auth_done global_id 164393 payload 824 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 1 -- 172.21.15.86:0/4266290661 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa17806d4f0 con 0x7fa17811d6a0 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 msgr2=0x7fa17811dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17db6b640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 0x7fa17811dab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa16c014000 tx=0x7fa16c004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c017030 con 0x7fa17811d6a0 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 10 monclient: dump: 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:43:48.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fa16c022410 con 0x7fa17811d6a0 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 msgr2=0x7fa17811dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 0x7fa17811dab0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa16c014000 tx=0x7fa16c004a20 comp rx=0 tx=0).stop 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 msgr2=0x7fa17811dab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:43:48.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa1767fc640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa17811d6a0 0x7fa17811dab0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa16c014000 tx=0x7fa16c004a20 comp rx=0 tx=0).stop 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 154, 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 131726, 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:43:48.554 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.555 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.556 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:43:48.557 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:43:48.558 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:43:48.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.561 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:43:48.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:43:48.564 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.565 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:43:48.567 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:43:48.568 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:43:48.569 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16403bb40 msgr2=0x7fa16403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16403bb40 0x7fa16403e000 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa17811f510 tx=0x7fa170003800 comp rx=0 tx=0).stop 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 10 monclient: shutdown 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 msgr2=0x7fa1781335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:43:48.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 --2- 172.21.15.86:0/4266290661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa17809e260 0x7fa1781335a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa16801c680 tx=0x7fa1680195f0 comp rx=0 tx=0).stop 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17e36c640 1 -- 172.21.15.86:0/4266290661 reap_dead start 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17e36c640 1 -- 172.21.15.86:0/4266290661 reap_dead start 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 shutdown_connections 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 >> 172.21.15.86:0/4266290661 conn(0x7fa1780a2f40 msgr2=0x7fa1780a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 shutdown_connections 2024-09-17T15:43:48.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:43:48.538+0000 7fa17fdcf640 1 -- 172.21.15.86:0/4266290661 wait complete. 2024-09-17T15:43:48.572 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:43:48.573 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:43:48.605 INFO:tasks.rados.rados.0.smithi203.stdout:2612: done (0 left) 2024-09-17T15:43:48.605 INFO:tasks.rados.rados.0.smithi203.stdout:2613: read oid 26 snap -1 2024-09-17T15:43:48.605 INFO:tasks.rados.rados.0.smithi203.stdout:2613: expect deleted 2024-09-17T15:43:48.605 INFO:tasks.rados.rados.0.smithi203.stdout:2614: write oid 50 current snap is 281 2024-09-17T15:43:48.605 INFO:tasks.rados.rados.0.smithi203.stdout:2614: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-17T15:43:48.626 INFO:tasks.rados.rados.0.smithi203.stdout:2614: writing smithi20328086-50 from 763282 to 1322569 tid 1 2024-09-17T15:43:48.647 INFO:tasks.rados.rados.0.smithi203.stdout:2614: writing smithi20328086-50 from 1820914 to 2581894 tid 2 2024-09-17T15:43:48.661 INFO:tasks.rados.rados.0.smithi203.stdout:2614: writing smithi20328086-50 from 3213103 to 3563282 tid 3 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2615: read oid 12 snap -1 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2615: expect deleted 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2616: read oid 36 snap -1 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2616: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2617: snap_create 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2614: finishing write tid 1 to smithi20328086-50 2024-09-17T15:43:48.662 INFO:tasks.rados.rados.0.smithi203.stdout:2614: finishing write tid 2 to smithi20328086-50 2024-09-17T15:43:48.668 INFO:tasks.rados.rados.0.smithi203.stdout:2614: finishing write tid 3 to smithi20328086-50 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: finishing write tid 4 to smithi20328086-50 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: finishing write tid 5 to smithi20328086-50 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: oid 50 updating version 0 to 1191 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: oid 50 updating version 1191 to 1192 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: oid 50 version 1192 is already newer than 1190 2024-09-17T15:43:48.671 INFO:tasks.rados.rados.0.smithi203.stdout:2614: oid 50 version 1192 is already newer than 1189 2024-09-17T15:43:48.672 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1192 (ObjNum 926 snap 281 seq_num 926) dirty exists 2024-09-17T15:43:48.672 INFO:tasks.rados.rados.0.smithi203.stdout:2614: left oid 50 (ObjNum 926 snap 281 seq_num 926) 2024-09-17T15:43:48.676 INFO:tasks.rados.rados.0.smithi203.stdout:2616: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:48.677 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:48.901 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 117430 2024-09-17T15:43:49.618 INFO:tasks.rados.rados.0.smithi203.stdout:2613: done (4 left) 2024-09-17T15:43:49.618 INFO:tasks.rados.rados.0.smithi203.stdout:2614: done (3 left) 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2615: done (2 left) 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2616: done (1 left) 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2617: done (0 left) 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2618: read oid 31 snap -1 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2618: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:43:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2619: delete oid 19 current snap is 282 2024-09-17T15:43:49.630 INFO:tasks.rados.rados.0.smithi203.stdout:2619: done (1 left) 2024-09-17T15:43:49.630 INFO:tasks.rados.rados.0.smithi203.stdout:2620: snap_create 2024-09-17T15:43:49.633 INFO:tasks.rados.rados.0.smithi203.stdout:2618: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:43:49.678 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2930385 2024-09-17T15:43:50.626 INFO:tasks.rados.rados.0.smithi203.stdout:2618: done (1 left) 2024-09-17T15:43:50.626 INFO:tasks.rados.rados.0.smithi203.stdout:2620: done (0 left) 2024-09-17T15:43:50.626 INFO:tasks.rados.rados.0.smithi203.stdout:2621: read oid 36 snap -1 2024-09-17T15:43:50.626 INFO:tasks.rados.rados.0.smithi203.stdout:2621: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:50.626 INFO:tasks.rados.rados.0.smithi203.stdout:2622: write (excl) oid 48 current snap is 283 2024-09-17T15:43:50.627 INFO:tasks.rados.rados.0.smithi203.stdout:2622: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2024-09-17T15:43:50.646 INFO:tasks.rados.rados.0.smithi203.stdout:2622: writing smithi20328086-48 from 796896 to 1223034 tid 1 2024-09-17T15:43:50.667 INFO:tasks.rados.rados.0.smithi203.stdout:2622: writing smithi20328086-48 from 1654231 to 2418880 tid 2 2024-09-17T15:43:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:2622: writing smithi20328086-48 from 2984302 to 3464377 tid 3 2024-09-17T15:43:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:2623: read oid 39 snap 261 2024-09-17T15:43:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:2623: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:43:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:2624: snap_create 2024-09-17T15:43:50.683 INFO:tasks.rados.rados.0.smithi203.stdout:2621: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:50.940 INFO:tasks.rados.rados.0.smithi203.stdout:2622: finishing write tid 1 to smithi20328086-48 2024-09-17T15:43:50.941 INFO:tasks.rados.rados.0.smithi203.stdout:2622: finishing write tid 2 to smithi20328086-48 2024-09-17T15:43:50.941 INFO:tasks.rados.rados.0.smithi203.stdout:2623: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:43:51.110 INFO:tasks.rados.rados.0.smithi203.stdout:2622: finishing write tid 3 to smithi20328086-48 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: finishing write tid 4 to smithi20328086-48 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: finishing write tid 5 to smithi20328086-48 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: oid 48 updating version 0 to 1300 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: oid 48 updating version 1300 to 1301 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: oid 48 version 1301 is already newer than 1299 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: oid 48 updating version 1301 to 1302 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1302 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:43:51.111 INFO:tasks.rados.rados.0.smithi203.stdout:2622: left oid 48 (ObjNum 928 snap 283 seq_num 928) 2024-09-17T15:43:51.522 INFO:tasks.rados.rados.0.smithi203.stdout:2621: done (3 left) 2024-09-17T15:43:51.522 INFO:tasks.rados.rados.0.smithi203.stdout:2622: done (2 left) 2024-09-17T15:43:51.522 INFO:tasks.rados.rados.0.smithi203.stdout:2623: done (1 left) 2024-09-17T15:43:51.522 INFO:tasks.rados.rados.0.smithi203.stdout:2624: done (0 left) 2024-09-17T15:43:51.522 INFO:tasks.rados.rados.0.smithi203.stdout:2625: snap_remove snap 276 2024-09-17T15:43:52.531 INFO:tasks.rados.rados.0.smithi203.stdout:2625: done (0 left) 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:2626: copy_from oid 39 from oid 13 current snap is 284 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:2627: read oid 5 snap -1 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:2627: expect (ObjNum 913 snap 276 seq_num 913) 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:2628: rollback oid 27 current snap is 284 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 279 2024-09-17T15:43:52.532 INFO:tasks.rados.rados.0.smithi203.stdout:2629: delete oid 25 current snap is 284 2024-09-17T15:43:52.538 INFO:tasks.rados.rados.0.smithi203.stdout:2626: finishing copy_from to smithi20328086-39 2024-09-17T15:43:52.538 INFO:tasks.rados.rados.0.smithi203.stdout:2626: got expected ENOENT (src dne) 2024-09-17T15:43:52.539 INFO:tasks.rados.rados.0.smithi203.stdout:2626: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:43:52.545 INFO:tasks.rados.rados.0.smithi203.stdout:2627: expect (ObjNum 913 snap 276 seq_num 913) 2024-09-17T15:43:52.597 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:52.659 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2628: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2628: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2628: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1340 (ObjNum 873 snap 269 seq_num 873) dirty exists 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2626: done (3 left) 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2627: done (2 left) 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2628: done (1 left) 2024-09-17T15:43:52.720 INFO:tasks.rados.rados.0.smithi203.stdout:2629: done (0 left) 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2630: read oid 16 snap 261 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2630: expect deleted 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2631: copy_from oid 20 from oid 31 current snap is 284 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2632: read oid 11 snap -1 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2632: expect deleted 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2633: rollback oid 13 current snap is 284 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 268 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2634: write (excl) oid 33 current snap is 284 2024-09-17T15:43:52.721 INFO:tasks.rados.rados.0.smithi203.stdout:2634: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-17T15:43:52.738 INFO:tasks.rados.rados.0.smithi203.stdout:2634: writing smithi20328086-33 from 430510 to 1007146 tid 1 2024-09-17T15:43:52.756 INFO:tasks.rados.rados.0.smithi203.stdout:2634: writing smithi20328086-33 from 1654842 to 2423160 tid 2 2024-09-17T15:43:52.767 INFO:tasks.rados.rados.0.smithi203.stdout:2634: writing smithi20328086-33 from 2922795 to 3425414 tid 3 2024-09-17T15:43:52.767 INFO:tasks.rados.rados.0.smithi203.stdout:2635: read oid 36 snap -1 2024-09-17T15:43:52.767 INFO:tasks.rados.rados.0.smithi203.stdout:2635: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2636: snap_create 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2633: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2634: finishing write tid 1 to smithi20328086-33 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2631: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2631: finishing copy_from to smithi20328086-20 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1130 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:43:52.768 INFO:tasks.rados.rados.0.smithi203.stdout:2634: finishing write tid 2 to smithi20328086-33 2024-09-17T15:43:52.770 INFO:tasks.rados.rados.0.smithi203.stdout:2634: finishing write tid 3 to smithi20328086-33 2024-09-17T15:43:52.770 INFO:tasks.rados.rados.0.smithi203.stdout:2634: finishing write tid 4 to smithi20328086-33 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: finishing write tid 5 to smithi20328086-33 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: oid 33 updating version 0 to 1091 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: oid 33 version 1091 is already newer than 1089 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: oid 33 version 1091 is already newer than 1090 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: oid 33 version 1091 is already newer than 1088 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1091 (ObjNum 930 snap 284 seq_num 930) dirty exists 2024-09-17T15:43:52.771 INFO:tasks.rados.rados.0.smithi203.stdout:2634: left oid 33 (ObjNum 930 snap 284 seq_num 930) 2024-09-17T15:43:52.778 INFO:tasks.rados.rados.0.smithi203.stdout:2635: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:43:52.778 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2630: done (6 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2631: done (5 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2632: done (4 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2633: done (3 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2634: done (2 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2635: done (1 left) 2024-09-17T15:43:53.637 INFO:tasks.rados.rados.0.smithi203.stdout:2636: done (0 left) 2024-09-17T15:43:53.638 INFO:tasks.rados.rados.0.smithi203.stdout:2637: snap_create 2024-09-17T15:43:54.645 INFO:tasks.rados.rados.0.smithi203.stdout:2637: done (0 left) 2024-09-17T15:43:54.645 INFO:tasks.rados.rados.0.smithi203.stdout:2638: snap_remove snap 282 2024-09-17T15:43:55.647 INFO:tasks.rados.rados.0.smithi203.stdout:2638: done (0 left) 2024-09-17T15:43:55.647 INFO:tasks.rados.rados.0.smithi203.stdout:2639: rollback oid 36 current snap is 286 2024-09-17T15:43:55.647 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 283 2024-09-17T15:43:55.647 INFO:tasks.rados.rados.0.smithi203.stdout:2640: write (excl) oid 27 current snap is 286 2024-09-17T15:43:55.648 INFO:tasks.rados.rados.0.smithi203.stdout:2640: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2024-09-17T15:43:55.665 INFO:tasks.rados.rados.0.smithi203.stdout:2640: writing smithi20328086-27 from 447317 to 1099202 tid 1 2024-09-17T15:43:55.676 INFO:tasks.rados.rados.0.smithi203.stdout:2640: writing smithi20328086-27 from 1713324 to 2341653 tid 2 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2640: writing smithi20328086-27 from 3066571 to 3647317 tid 3 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2641: delete oid 31 current snap is 286 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2639: finishing rollback tid 0 to smithi20328086-36 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2639: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2639: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1412 (ObjNum 888 snap 271 seq_num 888) dirty exists 2024-09-17T15:43:55.684 INFO:tasks.rados.rados.0.smithi203.stdout:2640: finishing write tid 1 to smithi20328086-27 2024-09-17T15:43:55.692 INFO:tasks.rados.rados.0.smithi203.stdout:2640: finishing write tid 2 to smithi20328086-27 2024-09-17T15:43:55.692 INFO:tasks.rados.rados.0.smithi203.stdout:2640: finishing write tid 3 to smithi20328086-27 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2639: done (2 left) 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2641: done (1 left) 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2642: snap_create 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2640: finishing write tid 4 to smithi20328086-27 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2640: finishing write tid 5 to smithi20328086-27 2024-09-17T15:43:55.693 INFO:tasks.rados.rados.0.smithi203.stdout:2640: oid 27 updating version 0 to 1343 2024-09-17T15:43:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:2640: oid 27 version 1343 is already newer than 1342 2024-09-17T15:43:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:2640: oid 27 version 1343 is already newer than 1341 2024-09-17T15:43:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:2640: oid 27 updating version 1343 to 1344 2024-09-17T15:43:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1344 (ObjNum 931 snap 286 seq_num 931) dirty exists 2024-09-17T15:43:55.694 INFO:tasks.rados.rados.0.smithi203.stdout:2640: left oid 27 (ObjNum 931 snap 286 seq_num 931) 2024-09-17T15:43:56.663 INFO:tasks.rados.rados.0.smithi203.stdout:2640: done (1 left) 2024-09-17T15:43:56.664 INFO:tasks.rados.rados.0.smithi203.stdout:2642: done (0 left) 2024-09-17T15:43:56.664 INFO:tasks.rados.rados.0.smithi203.stdout:2643: copy_from oid 31 from oid 46 current snap is 287 2024-09-17T15:43:56.664 INFO:tasks.rados.rados.0.smithi203.stdout:2644: read oid 14 snap 270 2024-09-17T15:43:56.664 INFO:tasks.rados.rados.0.smithi203.stdout:2644: expect (ObjNum 874 snap 269 seq_num 874) 2024-09-17T15:43:56.664 INFO:tasks.rados.rados.0.smithi203.stdout:2645: delete oid 6 current snap is 287 2024-09-17T15:43:56.679 INFO:tasks.rados.rados.0.smithi203.stdout:2644: expect (ObjNum 874 snap 269 seq_num 874) 2024-09-17T15:43:56.679 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:56.808 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2643: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2643: finishing copy_from to smithi20328086-31 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1114 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2643: done (2 left) 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2644: done (1 left) 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2645: done (0 left) 2024-09-17T15:43:56.859 INFO:tasks.rados.rados.0.smithi203.stdout:2646: snap_remove snap 277 2024-09-17T15:43:57.671 INFO:tasks.rados.rados.0.smithi203.stdout:2646: done (0 left) 2024-09-17T15:43:57.671 INFO:tasks.rados.rados.0.smithi203.stdout:2647: write (excl) oid 11 current snap is 287 2024-09-17T15:43:57.671 INFO:tasks.rados.rados.0.smithi203.stdout:2647: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-17T15:43:57.688 INFO:tasks.rados.rados.0.smithi203.stdout:2647: writing smithi20328086-11 from 497738 to 975370 tid 1 2024-09-17T15:43:57.714 INFO:tasks.rados.rados.0.smithi203.stdout:2647: writing smithi20328086-11 from 1772417 to 2548073 tid 2 2024-09-17T15:43:57.730 INFO:tasks.rados.rados.0.smithi203.stdout:2647: writing smithi20328086-11 from 3316134 to 3697738 tid 3 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2648: read oid 12 snap -1 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2648: expect deleted 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2649: read oid 41 snap 284 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2649: expect (ObjNum 924 snap 279 seq_num 924) 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2650: copy_from oid 26 from oid 31 current snap is 287 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2647: finishing write tid 1 to smithi20328086-11 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2647: finishing write tid 2 to smithi20328086-11 2024-09-17T15:43:57.731 INFO:tasks.rados.rados.0.smithi203.stdout:2651: delete oid 23 current snap is 287 2024-09-17T15:43:57.735 INFO:tasks.rados.rados.0.smithi203.stdout:2648: done (4 left) 2024-09-17T15:43:57.736 INFO:tasks.rados.rados.0.smithi203.stdout:2651: done (3 left) 2024-09-17T15:43:57.736 INFO:tasks.rados.rados.0.smithi203.stdout:2652: write oid 6 current snap is 287 2024-09-17T15:43:57.736 INFO:tasks.rados.rados.0.smithi203.stdout:2652: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-17T15:43:57.746 INFO:tasks.rados.rados.0.smithi203.stdout:2652: writing smithi20328086-6 from 531352 to 1275835 tid 1 2024-09-17T15:43:57.758 INFO:tasks.rados.rados.0.smithi203.stdout:2652: writing smithi20328086-6 from 2005734 to 2785059 tid 2 2024-09-17T15:43:57.765 INFO:tasks.rados.rados.0.smithi203.stdout:2652: writing smithi20328086-6 from 3487333 to 3731352 tid 3 2024-09-17T15:43:57.765 INFO:tasks.rados.rados.0.smithi203.stdout:2653: write oid 23 current snap is 287 2024-09-17T15:43:57.765 INFO:tasks.rados.rados.0.smithi203.stdout:2653: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-17T15:43:57.773 INFO:tasks.rados.rados.0.smithi203.stdout:2653: writing smithi20328086-23 from 548159 to 967891 tid 1 2024-09-17T15:43:57.782 INFO:tasks.rados.rados.0.smithi203.stdout:2653: writing smithi20328086-23 from 1664216 to 2187199 tid 2 2024-09-17T15:43:57.790 INFO:tasks.rados.rados.0.smithi203.stdout:2653: writing smithi20328086-23 from 2714756 to 3328985 tid 3 2024-09-17T15:43:57.790 INFO:tasks.rados.rados.0.smithi203.stdout:2654: delete oid 10 current snap is 287 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: finishing write tid 3 to smithi20328086-11 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: finishing write tid 4 to smithi20328086-11 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: finishing write tid 5 to smithi20328086-11 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: oid 11 updating version 0 to 1196 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: oid 11 version 1196 is already newer than 1193 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: oid 11 version 1196 is already newer than 1195 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: oid 11 version 1196 is already newer than 1194 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1196 (ObjNum 934 snap 287 seq_num 934) dirty exists 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2647: left oid 11 (ObjNum 934 snap 287 seq_num 934) 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:2649: expect (ObjNum 924 snap 279 seq_num 924) 2024-09-17T15:43:57.791 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:43:57.888 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 644959 2024-09-17T15:43:57.922 INFO:tasks.rados.rados.0.smithi203.stdout:2650: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:43:57.922 INFO:tasks.rados.rados.0.smithi203.stdout:2652: finishing write tid 1 to smithi20328086-6 2024-09-17T15:43:57.922 INFO:tasks.rados.rados.0.smithi203.stdout:2650: finishing copy_from to smithi20328086-26 2024-09-17T15:43:57.922 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1114 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: finishing write tid 2 to smithi20328086-6 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: finishing write tid 3 to smithi20328086-6 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: finishing write tid 4 to smithi20328086-6 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: finishing write tid 5 to smithi20328086-6 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: oid 6 updating version 0 to 1199 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: oid 6 version 1199 is already newer than 1198 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: oid 6 updating version 1199 to 1200 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: oid 6 version 1200 is already newer than 1197 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1200 (ObjNum 936 snap 287 seq_num 936) dirty exists 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2652: left oid 6 (ObjNum 936 snap 287 seq_num 936) 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2647: done (5 left) 2024-09-17T15:43:57.923 INFO:tasks.rados.rados.0.smithi203.stdout:2649: done (4 left) 2024-09-17T15:43:57.924 INFO:tasks.rados.rados.0.smithi203.stdout:2650: done (3 left) 2024-09-17T15:43:57.924 INFO:tasks.rados.rados.0.smithi203.stdout:2652: done (2 left) 2024-09-17T15:43:57.924 INFO:tasks.rados.rados.0.smithi203.stdout:2654: done (1 left) 2024-09-17T15:43:57.924 INFO:tasks.rados.rados.0.smithi203.stdout:2655: snap_remove snap 261 2024-09-17T15:43:58.680 INFO:tasks.rados.rados.0.smithi203.stdout:2655: done (1 left) 2024-09-17T15:43:58.680 INFO:tasks.rados.rados.0.smithi203.stdout:2656: copy_from oid 1 from oid 48 current snap is 287 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: finishing write tid 1 to smithi20328086-23 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: finishing write tid 2 to smithi20328086-23 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: finishing write tid 3 to smithi20328086-23 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: finishing write tid 4 to smithi20328086-23 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: finishing write tid 5 to smithi20328086-23 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: oid 23 updating version 0 to 1204 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: oid 23 version 1204 is already newer than 1203 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: oid 23 version 1204 is already newer than 1202 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: oid 23 version 1204 is already newer than 1201 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1204 (ObjNum 937 snap 287 seq_num 937) dirty exists 2024-09-17T15:43:58.681 INFO:tasks.rados.rados.0.smithi203.stdout:2653: left oid 23 (ObjNum 937 snap 287 seq_num 937) 2024-09-17T15:43:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2653: done (1 left) 2024-09-17T15:43:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2657: read oid 49 snap -1 2024-09-17T15:43:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2657: expect (ObjNum 907 snap 276 seq_num 907) 2024-09-17T15:43:58.682 INFO:tasks.rados.rados.0.smithi203.stdout:2658: snap_remove snap 280 2024-09-17T15:43:59.681 INFO:tasks.rados.rados.0.smithi203.stdout:2658: done (2 left) 2024-09-17T15:43:59.681 INFO:tasks.rados.rados.0.smithi203.stdout:2659: read oid 19 snap 279 2024-09-17T15:43:59.681 INFO:tasks.rados.rados.0.smithi203.stdout:2659: expect (ObjNum 871 snap 269 seq_num 871) 2024-09-17T15:43:59.681 INFO:tasks.rados.rados.0.smithi203.stdout:2660: rollback oid 36 current snap is 287 2024-09-17T15:43:59.681 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 281 2024-09-17T15:43:59.682 INFO:tasks.rados.rados.0.smithi203.stdout:2657: expect (ObjNum 907 snap 276 seq_num 907) 2024-09-17T15:43:59.835 INFO:tasks.rados.rados.0.smithi203.stdout:2656: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:43:59.835 INFO:tasks.rados.rados.0.smithi203.stdout:2656: finishing copy_from to smithi20328086-1 2024-09-17T15:43:59.836 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1302 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:43:59.836 INFO:tasks.rados.rados.0.smithi203.stdout:2659: expect (ObjNum 871 snap 269 seq_num 871) 2024-09-17T15:43:59.893 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1190008 2024-09-17T15:43:59.911 INFO:tasks.rados.rados.0.smithi203.stdout:2660: finishing rollback tid 0 to smithi20328086-36 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2660: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2660: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1429 (ObjNum 888 snap 271 seq_num 888) dirty exists 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2656: done (3 left) 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2657: done (2 left) 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2659: done (1 left) 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2660: done (0 left) 2024-09-17T15:43:59.912 INFO:tasks.rados.rados.0.smithi203.stdout:2661: delete oid 25 current snap is 287 2024-09-17T15:43:59.914 INFO:tasks.rados.rados.0.smithi203.stdout:2661: done (0 left) 2024-09-17T15:43:59.915 INFO:tasks.rados.rados.0.smithi203.stdout:2662: write (excl) oid 9 current snap is 287 2024-09-17T15:43:59.915 INFO:tasks.rados.rados.0.smithi203.stdout:2662: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-17T15:43:59.935 INFO:tasks.rados.rados.0.smithi203.stdout:2662: writing smithi20328086-9 from 598580 to 1244059 tid 1 2024-09-17T15:43:59.956 INFO:tasks.rados.rados.0.smithi203.stdout:2662: writing smithi20328086-9 from 1723309 to 2509972 tid 2 2024-09-17T15:43:59.971 INFO:tasks.rados.rados.0.smithi203.stdout:2662: writing smithi20328086-9 from 3080672 to 3528717 tid 3 2024-09-17T15:43:59.971 INFO:tasks.rados.rados.0.smithi203.stdout:2663: write (excl) oid 18 current snap is 287 2024-09-17T15:43:59.971 INFO:tasks.rados.rados.0.smithi203.stdout:2663: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-17T15:43:59.986 INFO:tasks.rados.rados.0.smithi203.stdout:2663: writing smithi20328086-18 from 615387 to 1336115 tid 1 2024-09-17T15:43:59.995 INFO:tasks.rados.rados.0.smithi203.stdout:2663: writing smithi20328086-18 from 1781791 to 2312112 tid 2 2024-09-17T15:44:00.007 INFO:tasks.rados.rados.0.smithi203.stdout:2663: writing smithi20328086-18 from 3108095 to 3815387 tid 3 2024-09-17T15:44:00.008 INFO:tasks.rados.rados.0.smithi203.stdout:2664: read oid 14 snap -1 2024-09-17T15:44:00.008 INFO:tasks.rados.rados.0.smithi203.stdout:2664: expect deleted 2024-09-17T15:44:00.008 INFO:tasks.rados.rados.0.smithi203.stdout:2665: read oid 47 snap -1 2024-09-17T15:44:00.008 INFO:tasks.rados.rados.0.smithi203.stdout:2665: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:44:00.008 INFO:tasks.rados.rados.0.smithi203.stdout:2666: snap_remove snap 286 2024-09-17T15:44:00.697 INFO:tasks.rados.rados.0.smithi203.stdout:2666: done (4 left) 2024-09-17T15:44:00.697 INFO:tasks.rados.rados.0.smithi203.stdout:2667: write (excl) oid 12 current snap is 287 2024-09-17T15:44:00.697 INFO:tasks.rados.rados.0.smithi203.stdout:2662: finishing write tid 1 to smithi20328086-9 2024-09-17T15:44:00.697 INFO:tasks.rados.rados.0.smithi203.stdout:2662: finishing write tid 2 to smithi20328086-9 2024-09-17T15:44:00.697 INFO:tasks.rados.rados.0.smithi203.stdout:2667: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-17T15:44:00.709 INFO:tasks.rados.rados.0.smithi203.stdout:2667: writing smithi20328086-12 from 632194 to 1428171 tid 1 2024-09-17T15:44:00.723 INFO:tasks.rados.rados.0.smithi203.stdout:2667: writing smithi20328086-12 from 2123920 to 2914252 tid 2 2024-09-17T15:44:00.732 INFO:tasks.rados.rados.0.smithi203.stdout:2667: writing smithi20328086-12 from 3419165 to 3832194 tid 3 2024-09-17T15:44:00.732 INFO:tasks.rados.rados.0.smithi203.stdout:2668: rollback oid 6 current snap is 287 2024-09-17T15:44:00.732 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 270 2024-09-17T15:44:00.732 INFO:tasks.rados.rados.0.smithi203.stdout:2662: finishing write tid 3 to smithi20328086-9 2024-09-17T15:44:00.732 INFO:tasks.rados.rados.0.smithi203.stdout:2662: finishing write tid 4 to smithi20328086-9 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: finishing write tid 5 to smithi20328086-9 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: oid 9 updating version 0 to 1177 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: oid 9 version 1177 is already newer than 1175 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: oid 9 version 1177 is already newer than 1174 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: oid 9 version 1177 is already newer than 1176 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1177 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2662: left oid 9 (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: finishing write tid 1 to smithi20328086-18 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: finishing write tid 2 to smithi20328086-18 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: finishing write tid 3 to smithi20328086-18 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: finishing write tid 4 to smithi20328086-18 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: finishing write tid 5 to smithi20328086-18 2024-09-17T15:44:00.733 INFO:tasks.rados.rados.0.smithi203.stdout:2663: oid 18 updating version 0 to 850 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:2663: oid 18 version 850 is already newer than 847 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:2663: oid 18 version 850 is already newer than 848 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:2663: oid 18 version 850 is already newer than 849 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 850 (ObjNum 941 snap 287 seq_num 941) dirty exists 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:2663: left oid 18 (ObjNum 941 snap 287 seq_num 941) 2024-09-17T15:44:00.734 INFO:tasks.rados.rados.0.smithi203.stdout:2665: expect (ObjNum 779 snap 243 seq_num 779) 2024-09-17T15:44:00.841 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2581926 2024-09-17T15:44:00.855 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:01.016 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2756604 2024-09-17T15:44:01.028 INFO:tasks.rados.rados.0.smithi203.stdout:2667: finishing write tid 1 to smithi20328086-12 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: finishing write tid 2 to smithi20328086-12 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2668: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2668: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1304 (ObjNum 129 snap 0 seq_num 3606721400) dirty dne 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: finishing write tid 3 to smithi20328086-12 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: finishing write tid 4 to smithi20328086-12 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: finishing write tid 5 to smithi20328086-12 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: oid 12 updating version 0 to 1265 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: oid 12 version 1265 is already newer than 1263 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: oid 12 version 1265 is already newer than 1262 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:2667: oid 12 version 1265 is already newer than 1264 2024-09-17T15:44:01.029 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1265 (ObjNum 942 snap 287 seq_num 942) dirty exists 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2667: left oid 12 (ObjNum 942 snap 287 seq_num 942) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2662: done (5 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2663: done (4 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2664: done (3 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2665: done (2 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2667: done (1 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2668: done (0 left) 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2669: read oid 24 snap -1 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2669: expect deleted 2024-09-17T15:44:01.030 INFO:tasks.rados.rados.0.smithi203.stdout:2670: snap_remove snap 268 2024-09-17T15:44:01.523 INFO:tasks.rados.rados.0.smithi203.stdout:2670: done (1 left) 2024-09-17T15:44:01.523 INFO:tasks.rados.rados.0.smithi203.stdout:2671: copy_from oid 20 from oid 25 current snap is 287 2024-09-17T15:44:01.523 INFO:tasks.rados.rados.0.smithi203.stdout:2669: done (1 left) 2024-09-17T15:44:01.524 INFO:tasks.rados.rados.0.smithi203.stdout:2672: snap_remove snap 283 2024-09-17T15:44:02.621 INFO:tasks.rados.rados.0.smithi203.stdout:2672: done (1 left) 2024-09-17T15:44:02.621 INFO:tasks.rados.rados.0.smithi203.stdout:2673: read oid 27 snap -1 2024-09-17T15:44:02.621 INFO:tasks.rados.rados.0.smithi203.stdout:2673: expect (ObjNum 931 snap 286 seq_num 931) 2024-09-17T15:44:02.621 INFO:tasks.rados.rados.0.smithi203.stdout:2674: write (excl) oid 12 current snap is 287 2024-09-17T15:44:02.621 INFO:tasks.rados.rados.0.smithi203.stdout:2674: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2024-09-17T15:44:02.638 INFO:tasks.rados.rados.0.smithi203.stdout:2674: writing smithi20328086-12 from 649001 to 1236580 tid 1 2024-09-17T15:44:02.655 INFO:tasks.rados.rados.0.smithi203.stdout:2674: writing smithi20328086-12 from 1898755 to 2432745 tid 2 2024-09-17T15:44:02.674 INFO:tasks.rados.rados.0.smithi203.stdout:2674: writing smithi20328086-12 from 3162941 to 3849001 tid 3 2024-09-17T15:44:02.674 INFO:tasks.rados.rados.0.smithi203.stdout:2675: snap_remove snap 278 2024-09-17T15:44:03.630 INFO:tasks.rados.rados.0.smithi203.stdout:2675: done (3 left) 2024-09-17T15:44:03.630 INFO:tasks.rados.rados.0.smithi203.stdout:2676: snap_remove snap 285 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2676: done (3 left) 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2677: read oid 41 snap -1 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2677: expect (ObjNum 924 snap 279 seq_num 924) 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2678: delete oid 42 current snap is 287 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2671: finishing copy_from to smithi20328086-20 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2671: got expected ENOENT (src dne) 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2671: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:44:04.639 INFO:tasks.rados.rados.0.smithi203.stdout:2673: expect (ObjNum 931 snap 286 seq_num 931) 2024-09-17T15:44:04.652 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 983545 2024-09-17T15:44:04.807 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3439851 2024-09-17T15:44:04.841 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1750085 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: finishing write tid 1 to smithi20328086-12 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: finishing write tid 2 to smithi20328086-12 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2671: done (4 left) 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2673: done (3 left) 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2678: done (2 left) 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2679: rollback oid 24 current snap is 287 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 281 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: finishing write tid 3 to smithi20328086-12 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: finishing write tid 4 to smithi20328086-12 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: finishing write tid 5 to smithi20328086-12 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: oid 12 updating version 0 to 1268 2024-09-17T15:44:04.873 INFO:tasks.rados.rados.0.smithi203.stdout:2674: oid 12 version 1268 is already newer than 1267 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:2674: oid 12 version 1268 is already newer than 1266 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:2674: oid 12 updating version 1268 to 1269 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1269 (ObjNum 943 snap 287 seq_num 943) dirty exists 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:2674: left oid 12 (ObjNum 943 snap 287 seq_num 943) 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:2677: expect (ObjNum 924 snap 279 seq_num 924) 2024-09-17T15:44:04.874 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:04.930 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3386471 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2679: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1172 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2674: done (2 left) 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2677: done (1 left) 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2679: done (0 left) 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2680: write oid 5 current snap is 287 2024-09-17T15:44:04.970 INFO:tasks.rados.rados.0.smithi203.stdout:2680: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-17T15:44:04.977 INFO:tasks.rados.rados.0.smithi203.stdout:2680: writing smithi20328086-5 from 682615 to 1420692 tid 1 2024-09-17T15:44:04.984 INFO:tasks.rados.rados.0.smithi203.stdout:2680: writing smithi20328086-5 from 2015719 to 2553378 tid 2 2024-09-17T15:44:04.991 INFO:tasks.rados.rados.0.smithi203.stdout:2680: writing smithi20328086-5 from 3217787 to 3638545 tid 3 2024-09-17T15:44:04.991 INFO:tasks.rados.rados.0.smithi203.stdout:2681: write oid 3 current snap is 287 2024-09-17T15:44:04.992 INFO:tasks.rados.rados.0.smithi203.stdout:2681: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-17T15:44:04.998 INFO:tasks.rados.rados.0.smithi203.stdout:2681: writing smithi20328086-3 from 699422 to 1112748 tid 1 2024-09-17T15:44:05.005 INFO:tasks.rados.rados.0.smithi203.stdout:2681: writing smithi20328086-3 from 1557848 to 2239165 tid 2 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2681: writing smithi20328086-3 from 3012504 to 3760887 tid 3 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2682: copy_from oid 23 from oid 26 current snap is 287 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2680: finishing write tid 1 to smithi20328086-5 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2680: finishing write tid 2 to smithi20328086-5 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2680: finishing write tid 3 to smithi20328086-5 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2680: finishing write tid 4 to smithi20328086-5 2024-09-17T15:44:05.015 INFO:tasks.rados.rados.0.smithi203.stdout:2680: finishing write tid 5 to smithi20328086-5 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: oid 5 updating version 0 to 1094 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: oid 5 updating version 1094 to 1095 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: oid 5 version 1095 is already newer than 1093 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: oid 5 version 1095 is already newer than 1092 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1095 (ObjNum 945 snap 287 seq_num 945) dirty exists 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: left oid 5 (ObjNum 945 snap 287 seq_num 945) 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2681: finishing write tid 1 to smithi20328086-3 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2681: finishing write tid 2 to smithi20328086-3 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2680: done (2 left) 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2683: write oid 15 current snap is 287 2024-09-17T15:44:05.016 INFO:tasks.rados.rados.0.smithi203.stdout:2683: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-17T15:44:05.023 INFO:tasks.rados.rados.0.smithi203.stdout:2683: writing smithi20328086-15 from 716229 to 1204804 tid 1 2024-09-17T15:44:05.030 INFO:tasks.rados.rados.0.smithi203.stdout:2683: writing smithi20328086-15 from 1616330 to 2157658 tid 2 2024-09-17T15:44:05.038 INFO:tasks.rados.rados.0.smithi203.stdout:2683: writing smithi20328086-15 from 2756280 to 3315935 tid 3 2024-09-17T15:44:05.039 INFO:tasks.rados.rados.0.smithi203.stdout:2684: write (excl) oid 11 current snap is 287 2024-09-17T15:44:05.039 INFO:tasks.rados.rados.0.smithi203.stdout:2684: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2024-09-17T15:44:05.046 INFO:tasks.rados.rados.0.smithi203.stdout:2684: writing smithi20328086-11 from 733036 to 1296860 tid 1 2024-09-17T15:44:05.054 INFO:tasks.rados.rados.0.smithi203.stdout:2684: writing smithi20328086-11 from 2074812 to 2759798 tid 2 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2684: writing smithi20328086-11 from 3183703 to 3933036 tid 3 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2685: copy_from oid 6 from oid 4 current snap is 287 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: finishing write tid 3 to smithi20328086-3 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: finishing write tid 4 to smithi20328086-3 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: finishing write tid 5 to smithi20328086-3 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: oid 3 updating version 0 to 1347 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: oid 3 version 1347 is already newer than 1346 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: oid 3 updating version 1347 to 1348 2024-09-17T15:44:05.062 INFO:tasks.rados.rados.0.smithi203.stdout:2681: oid 3 updating version 1348 to 1349 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1349 (ObjNum 946 snap 287 seq_num 946) dirty exists 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:2681: left oid 3 (ObjNum 946 snap 287 seq_num 946) 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:2683: finishing write tid 1 to smithi20328086-15 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:2681: done (4 left) 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:2686: write (excl) oid 32 current snap is 287 2024-09-17T15:44:05.063 INFO:tasks.rados.rados.0.smithi203.stdout:2686: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-09-17T15:44:05.069 INFO:tasks.rados.rados.0.smithi203.stdout:2686: writing smithi20328086-32 from 749843 to 1388916 tid 1 2024-09-17T15:44:05.076 INFO:tasks.rados.rados.0.smithi203.stdout:2686: writing smithi20328086-32 from 2133294 to 2678291 tid 2 2024-09-17T15:44:05.084 INFO:tasks.rados.rados.0.smithi203.stdout:2686: writing smithi20328086-32 from 3211126 to 3793325 tid 3 2024-09-17T15:44:05.084 INFO:tasks.rados.rados.0.smithi203.stdout:2687: snap_create 2024-09-17T15:44:05.084 INFO:tasks.rados.rados.0.smithi203.stdout:2683: finishing write tid 2 to smithi20328086-15 2024-09-17T15:44:05.084 INFO:tasks.rados.rados.0.smithi203.stdout:2682: finishing copy_from racing read to smithi20328086-23 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2682: finishing copy_from to smithi20328086-23 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1114 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: finishing write tid 3 to smithi20328086-15 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: finishing write tid 4 to smithi20328086-15 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: finishing write tid 5 to smithi20328086-15 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: oid 15 updating version 0 to 1272 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: oid 15 updating version 1272 to 1273 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: oid 15 version 1273 is already newer than 1270 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: oid 15 version 1273 is already newer than 1271 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1273 (ObjNum 947 snap 287 seq_num 947) dirty exists 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2683: left oid 15 (ObjNum 947 snap 287 seq_num 947) 2024-09-17T15:44:05.085 INFO:tasks.rados.rados.0.smithi203.stdout:2684: finishing write tid 1 to smithi20328086-11 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: finishing write tid 2 to smithi20328086-11 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2685: finishing copy_from to smithi20328086-6 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2685: got expected ENOENT (src dne) 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2685: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: finishing write tid 3 to smithi20328086-11 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: finishing write tid 4 to smithi20328086-11 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: finishing write tid 5 to smithi20328086-11 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: oid 11 updating version 0 to 1213 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: oid 11 version 1213 is already newer than 1212 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: oid 11 version 1213 is already newer than 1211 2024-09-17T15:44:05.086 INFO:tasks.rados.rados.0.smithi203.stdout:2684: oid 11 updating version 1213 to 1214 2024-09-17T15:44:05.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1214 (ObjNum 948 snap 287 seq_num 948) dirty exists 2024-09-17T15:44:05.087 INFO:tasks.rados.rados.0.smithi203.stdout:2684: left oid 11 (ObjNum 948 snap 287 seq_num 948) 2024-09-17T15:44:05.089 INFO:tasks.rados.rados.0.smithi203.stdout:2686: finishing write tid 1 to smithi20328086-32 2024-09-17T15:44:05.090 INFO:tasks.rados.rados.0.smithi203.stdout:2686: finishing write tid 2 to smithi20328086-32 2024-09-17T15:44:05.090 INFO:tasks.rados.rados.0.smithi203.stdout:2686: finishing write tid 3 to smithi20328086-32 2024-09-17T15:44:05.090 INFO:tasks.rados.rados.0.smithi203.stdout:2686: finishing write tid 4 to smithi20328086-32 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: finishing write tid 5 to smithi20328086-32 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: oid 32 updating version 0 to 1194 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: oid 32 updating version 1194 to 1195 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: oid 32 version 1195 is already newer than 1193 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: oid 32 updating version 1195 to 1196 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1196 (ObjNum 949 snap 287 seq_num 949) dirty exists 2024-09-17T15:44:05.091 INFO:tasks.rados.rados.0.smithi203.stdout:2686: left oid 32 (ObjNum 949 snap 287 seq_num 949) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2682: done (5 left) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2683: done (4 left) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2684: done (3 left) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2685: done (2 left) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2686: done (1 left) 2024-09-17T15:44:05.636 INFO:tasks.rados.rados.0.smithi203.stdout:2687: done (0 left) 2024-09-17T15:44:05.637 INFO:tasks.rados.rados.0.smithi203.stdout:2688: write (excl) oid 26 current snap is 288 2024-09-17T15:44:05.637 INFO:tasks.rados.rados.0.smithi203.stdout:2688: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-17T15:44:05.644 INFO:tasks.rados.rados.0.smithi203.stdout:2688: writing smithi20328086-26 from 766650 to 1480972 tid 1 2024-09-17T15:44:05.651 INFO:tasks.rados.rados.0.smithi203.stdout:2688: writing smithi20328086-26 from 2075423 to 2764078 tid 2 2024-09-17T15:44:05.658 INFO:tasks.rados.rados.0.smithi203.stdout:2688: writing smithi20328086-26 from 3522196 to 3966650 tid 3 2024-09-17T15:44:05.659 INFO:tasks.rados.rados.0.smithi203.stdout:2689: write (excl) oid 18 current snap is 288 2024-09-17T15:44:05.659 INFO:tasks.rados.rados.0.smithi203.stdout:2689: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-17T15:44:05.665 INFO:tasks.rados.rados.0.smithi203.stdout:2689: writing smithi20328086-18 from 783457 to 1289381 tid 1 2024-09-17T15:44:05.672 INFO:tasks.rados.rados.0.smithi203.stdout:2689: writing smithi20328086-18 from 1850258 to 2398924 tid 2 2024-09-17T15:44:05.679 INFO:tasks.rados.rados.0.smithi203.stdout:2689: writing smithi20328086-18 from 2865972 to 3587068 tid 3 2024-09-17T15:44:05.679 INFO:tasks.rados.rados.0.smithi203.stdout:2690: write oid 47 current snap is 288 2024-09-17T15:44:05.679 INFO:tasks.rados.rados.0.smithi203.stdout:2690: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-17T15:44:05.684 INFO:tasks.rados.rados.0.smithi203.stdout:2690: writing smithi20328086-47 from 400264 to 981437 tid 1 2024-09-17T15:44:05.690 INFO:tasks.rados.rados.0.smithi203.stdout:2690: writing smithi20328086-47 from 1508740 to 2000264 tid 2 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2691: copy_from oid 25 from oid 7 current snap is 288 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: finishing write tid 1 to smithi20328086-26 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: finishing write tid 2 to smithi20328086-26 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: finishing write tid 3 to smithi20328086-26 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: finishing write tid 4 to smithi20328086-26 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: finishing write tid 5 to smithi20328086-26 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: oid 26 updating version 0 to 1451 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: oid 26 version 1451 is already newer than 1450 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: oid 26 updating version 1451 to 1452 2024-09-17T15:44:05.691 INFO:tasks.rados.rados.0.smithi203.stdout:2688: oid 26 version 1452 is already newer than 1449 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1452 (ObjNum 950 snap 288 seq_num 950) dirty exists 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2688: left oid 26 (ObjNum 950 snap 288 seq_num 950) 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2689: finishing write tid 1 to smithi20328086-18 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2689: finishing write tid 2 to smithi20328086-18 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2688: done (3 left) 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2692: write (excl) oid 48 current snap is 288 2024-09-17T15:44:05.692 INFO:tasks.rados.rados.0.smithi203.stdout:2692: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-17T15:44:05.697 INFO:tasks.rados.rados.0.smithi203.stdout:2692: writing smithi20328086-48 from 417071 to 1073493 tid 1 2024-09-17T15:44:05.703 INFO:tasks.rados.rados.0.smithi203.stdout:2692: writing smithi20328086-48 from 1567222 to 2017071 tid 2 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2693: snap_create 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: finishing write tid 3 to smithi20328086-18 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: finishing write tid 4 to smithi20328086-18 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: finishing write tid 5 to smithi20328086-18 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: oid 18 updating version 0 to 851 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: oid 18 updating version 851 to 854 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: oid 18 version 854 is already newer than 852 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: oid 18 version 854 is already newer than 853 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 854 (ObjNum 951 snap 288 seq_num 951) dirty exists 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2689: left oid 18 (ObjNum 951 snap 288 seq_num 951) 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2690: finishing write tid 1 to smithi20328086-47 2024-09-17T15:44:05.704 INFO:tasks.rados.rados.0.smithi203.stdout:2690: finishing write tid 2 to smithi20328086-47 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: finishing write tid 3 to smithi20328086-47 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: finishing write tid 4 to smithi20328086-47 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: oid 47 updating version 0 to 855 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: oid 47 updating version 855 to 857 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: oid 47 version 857 is already newer than 856 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 857 (ObjNum 952 snap 288 seq_num 952) dirty exists 2024-09-17T15:44:05.705 INFO:tasks.rados.rados.0.smithi203.stdout:2690: left oid 47 (ObjNum 952 snap 288 seq_num 952) 2024-09-17T15:44:05.711 INFO:tasks.rados.rados.0.smithi203.stdout:2692: finishing write tid 1 to smithi20328086-48 2024-09-17T15:44:05.711 INFO:tasks.rados.rados.0.smithi203.stdout:2692: finishing write tid 2 to smithi20328086-48 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: finishing write tid 3 to smithi20328086-48 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: finishing write tid 4 to smithi20328086-48 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: oid 48 updating version 0 to 1341 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: oid 48 updating version 1341 to 1342 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: oid 48 version 1342 is already newer than 1340 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1342 (ObjNum 953 snap 288 seq_num 953) dirty exists 2024-09-17T15:44:05.712 INFO:tasks.rados.rados.0.smithi203.stdout:2692: left oid 48 (ObjNum 953 snap 288 seq_num 953) 2024-09-17T15:44:05.716 INFO:tasks.rados.rados.0.smithi203.stdout:2691: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:44:05.719 INFO:tasks.rados.rados.0.smithi203.stdout:2691: finishing copy_from to smithi20328086-25 2024-09-17T15:44:05.719 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1165 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:44:06.526 INFO:tasks.rados.rados.0.smithi203.stdout:2689: done (4 left) 2024-09-17T15:44:06.526 INFO:tasks.rados.rados.0.smithi203.stdout:2690: done (3 left) 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2691: done (2 left) 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2692: done (1 left) 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2693: done (0 left) 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2694: rollback oid 15 current snap is 289 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 281 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2695: write oid 22 current snap is 289 2024-09-17T15:44:06.527 INFO:tasks.rados.rados.0.smithi203.stdout:2695: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-17T15:44:06.533 INFO:tasks.rados.rados.0.smithi203.stdout:2695: writing smithi20328086-22 from 433878 to 1165549 tid 1 2024-09-17T15:44:06.538 INFO:tasks.rados.rados.0.smithi203.stdout:2695: writing smithi20328086-22 from 1909351 to 2033878 tid 2 2024-09-17T15:44:06.538 INFO:tasks.rados.rados.0.smithi203.stdout:2696: delete oid 21 current snap is 289 2024-09-17T15:44:06.538 INFO:tasks.rados.rados.0.smithi203.stdout:2694: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:44:06.538 INFO:tasks.rados.rados.0.smithi203.stdout:2694: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:44:06.538 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1275 (ObjNum 129 snap 0 seq_num 3606096400) dirty dne 2024-09-17T15:44:06.548 INFO:tasks.rados.rados.0.smithi203.stdout:2694: done (2 left) 2024-09-17T15:44:06.548 INFO:tasks.rados.rados.0.smithi203.stdout:2696: done (1 left) 2024-09-17T15:44:06.548 INFO:tasks.rados.rados.0.smithi203.stdout:2697: rollback oid 27 current snap is 289 2024-09-17T15:44:06.548 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 287 2024-09-17T15:44:06.549 INFO:tasks.rados.rados.0.smithi203.stdout:2698: delete oid 30 current snap is 289 2024-09-17T15:44:06.549 INFO:tasks.rados.rados.0.smithi203.stdout:2695: finishing write tid 1 to smithi20328086-22 2024-09-17T15:44:06.549 INFO:tasks.rados.rados.0.smithi203.stdout:2695: finishing write tid 2 to smithi20328086-22 2024-09-17T15:44:06.550 INFO:tasks.rados.rados.0.smithi203.stdout:2695: finishing write tid 3 to smithi20328086-22 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:2695: finishing write tid 4 to smithi20328086-22 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:2695: oid 22 updating version 0 to 1257 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:2695: oid 22 updating version 1257 to 1259 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:2695: oid 22 version 1259 is already newer than 1258 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1259 (ObjNum 954 snap 289 seq_num 954) dirty exists 2024-09-17T15:44:06.551 INFO:tasks.rados.rados.0.smithi203.stdout:2695: left oid 22 (ObjNum 954 snap 289 seq_num 954) 2024-09-17T15:44:06.558 INFO:tasks.rados.rados.0.smithi203.stdout:2695: done (2 left) 2024-09-17T15:44:06.558 INFO:tasks.rados.rados.0.smithi203.stdout:2698: done (1 left) 2024-09-17T15:44:06.558 INFO:tasks.rados.rados.0.smithi203.stdout:2699: write (excl) oid 11 current snap is 289 2024-09-17T15:44:06.559 INFO:tasks.rados.rados.0.smithi203.stdout:2699: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-17T15:44:06.566 INFO:tasks.rados.rados.0.smithi203.stdout:2699: writing smithi20328086-11 from 484299 to 1041717 tid 1 2024-09-17T15:44:06.575 INFO:tasks.rados.rados.0.smithi203.stdout:2699: writing smithi20328086-11 from 1684797 to 2084299 tid 2 2024-09-17T15:44:06.576 INFO:tasks.rados.rados.0.smithi203.stdout:2700: write oid 12 current snap is 289 2024-09-17T15:44:06.576 INFO:tasks.rados.rados.0.smithi203.stdout:2697: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:44:06.576 INFO:tasks.rados.rados.0.smithi203.stdout:2700: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-17T15:44:06.586 INFO:tasks.rados.rados.0.smithi203.stdout:2700: writing smithi20328086-12 from 501106 to 1250126 tid 1 2024-09-17T15:44:06.594 INFO:tasks.rados.rados.0.smithi203.stdout:2700: writing smithi20328086-12 from 1743279 to 2101106 tid 2 2024-09-17T15:44:06.594 INFO:tasks.rados.rados.0.smithi203.stdout:2701: read oid 23 snap 270 2024-09-17T15:44:06.594 INFO:tasks.rados.rados.0.smithi203.stdout:2701: expect deleted 2024-09-17T15:44:06.594 INFO:tasks.rados.rados.0.smithi203.stdout:2702: snap_create 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2697: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2697: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1352 (ObjNum 931 snap 286 seq_num 931) dirty exists 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: finishing write tid 1 to smithi20328086-11 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: finishing write tid 2 to smithi20328086-11 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: finishing write tid 3 to smithi20328086-11 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: finishing write tid 4 to smithi20328086-11 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: oid 11 updating version 0 to 1217 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: oid 11 version 1217 is already newer than 1216 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: oid 11 updating version 1217 to 1218 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1218 (ObjNum 957 snap 289 seq_num 957) dirty exists 2024-09-17T15:44:06.595 INFO:tasks.rados.rados.0.smithi203.stdout:2699: left oid 11 (ObjNum 957 snap 289 seq_num 957) 2024-09-17T15:44:06.601 INFO:tasks.rados.rados.0.smithi203.stdout:2700: finishing write tid 1 to smithi20328086-12 2024-09-17T15:44:06.601 INFO:tasks.rados.rados.0.smithi203.stdout:2700: finishing write tid 2 to smithi20328086-12 2024-09-17T15:44:06.602 INFO:tasks.rados.rados.0.smithi203.stdout:2700: finishing write tid 3 to smithi20328086-12 2024-09-17T15:44:06.602 INFO:tasks.rados.rados.0.smithi203.stdout:2700: finishing write tid 4 to smithi20328086-12 2024-09-17T15:44:06.603 INFO:tasks.rados.rados.0.smithi203.stdout:2700: oid 12 updating version 0 to 1277 2024-09-17T15:44:06.603 INFO:tasks.rados.rados.0.smithi203.stdout:2700: oid 12 updating version 1277 to 1278 2024-09-17T15:44:06.603 INFO:tasks.rados.rados.0.smithi203.stdout:2700: oid 12 updating version 1278 to 1279 2024-09-17T15:44:06.603 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1279 (ObjNum 958 snap 289 seq_num 958) dirty exists 2024-09-17T15:44:06.603 INFO:tasks.rados.rados.0.smithi203.stdout:2700: left oid 12 (ObjNum 958 snap 289 seq_num 958) 2024-09-17T15:44:07.527 INFO:tasks.rados.rados.0.smithi203.stdout:2697: done (4 left) 2024-09-17T15:44:07.527 INFO:tasks.rados.rados.0.smithi203.stdout:2699: done (3 left) 2024-09-17T15:44:07.527 INFO:tasks.rados.rados.0.smithi203.stdout:2700: done (2 left) 2024-09-17T15:44:07.528 INFO:tasks.rados.rados.0.smithi203.stdout:2701: done (1 left) 2024-09-17T15:44:07.528 INFO:tasks.rados.rados.0.smithi203.stdout:2702: done (0 left) 2024-09-17T15:44:07.528 INFO:tasks.rados.rados.0.smithi203.stdout:2703: write (excl) oid 13 current snap is 290 2024-09-17T15:44:07.528 INFO:tasks.rados.rados.0.smithi203.stdout:2703: seq_num 959 ranges {517913=424269,1401761=446989} 2024-09-17T15:44:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:2703: writing smithi20328086-13 from 517913 to 942182 tid 1 2024-09-17T15:44:07.544 INFO:tasks.rados.rados.0.smithi203.stdout:2703: writing smithi20328086-13 from 1401761 to 1848750 tid 2 2024-09-17T15:44:07.544 INFO:tasks.rados.rados.0.smithi203.stdout:2704: read oid 12 snap -1 2024-09-17T15:44:07.544 INFO:tasks.rados.rados.0.smithi203.stdout:2704: expect (ObjNum 958 snap 289 seq_num 958) 2024-09-17T15:44:07.544 INFO:tasks.rados.rados.0.smithi203.stdout:2705: write (excl) oid 45 current snap is 290 2024-09-17T15:44:07.544 INFO:tasks.rados.rados.0.smithi203.stdout:2705: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-17T15:44:07.553 INFO:tasks.rados.rados.0.smithi203.stdout:2705: writing smithi20328086-45 from 534720 to 1034238 tid 1 2024-09-17T15:44:07.564 INFO:tasks.rados.rados.0.smithi203.stdout:2705: writing smithi20328086-45 from 1460243 to 2134720 tid 2 2024-09-17T15:44:07.564 INFO:tasks.rados.rados.0.smithi203.stdout:2706: snap_remove snap 287 2024-09-17T15:44:08.541 INFO:tasks.rados.rados.0.smithi203.stdout:2706: done (3 left) 2024-09-17T15:44:08.541 INFO:tasks.rados.rados.0.smithi203.stdout:2707: snap_remove snap 289 2024-09-17T15:44:08.573 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:44:08.573 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:44:08.574 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:44:08.577 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:44:08.577 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:44:08.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:08.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: get_monmap_and_config 2024-09-17T15:44:08.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: build_initial_monmap 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: monmap: 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:08.709429+0000 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:08.709429+0000 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:08.705 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:08.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 Processor -- start 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 -- start start 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: init 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac409ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ac4068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ac4139f00 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:08.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac409d770 con 0x7f8ac4139f00 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ac409ed00 con 0x7f8ac4068db0 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac409efe0 con 0x7f8ac40689d0 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8acb73e640 10 monclient(hunting): authenticate will time out at 1981.675069s 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac409ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac9cdb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8ac4139f00 0x7f8ac413c2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37454/0 (socket says 172.21.15.86:37454) 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac9cdb640 1 -- 172.21.15.86:0/2657138474 learned_addr learned my addr 172.21.15.86:0/2657138474 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): get_auth_request con 0x7f8ac40689d0 auth_method 0 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:08.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient(hunting): handle_auth_done global_id 164396 payload 274 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 1 -- 172.21.15.86:0/2657138474 >> v1:172.21.15.203:6789/0 conn(0x7f8ac4068db0 legacy=0x7f8ac4069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 1 -- 172.21.15.86:0/2657138474 >> v1:172.21.15.86:6789/0 conn(0x7f8ac4139f00 legacy=0x7f8ac413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient: _finish_hunting 0 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 1 monclient: found mon.noname-b 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:08.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:08.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.706+0000 7f8ac94da640 1 -- 172.21.15.86:0/2657138474 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac40a04e0 con 0x7f8ac40689d0 2024-09-17T15:44:08.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.710+0000 7f8ac94da640 1 -- 172.21.15.86:0/2657138474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 msgr2=0x7f8ac409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:08.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.710+0000 7f8ac94da640 1 --2- 172.21.15.86:0/2657138474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac409ffa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00097f0 tx=0x7f8ac001c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75ca987cb899cc34 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:08.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 1 -- 172.21.15.86:0/2657138474 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00b0070 con 0x7f8ac40689d0 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: dump: 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.717 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: _finish_auth 0 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 5 monclient: authenticate success, global_id 164396 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 1 -- 172.21.15.86:0/2657138474 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac00ab600 con 0x7f8ac40689d0 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 4 set_mon_vals no callback set 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 1 -- 172.21.15.86:0/2657138474 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac00ab9d0 con 0x7f8ac40689d0 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8abbfff640 10 monclient: dump: 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: shutdown 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- 172.21.15.86:0/2657138474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 msgr2=0x7f8ac409ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 --2- 172.21.15.86:0/2657138474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac409ffa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8ac00097f0 tx=0x7f8ac001c430 comp rx=0 tx=0).stop 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- 172.21.15.86:0/2657138474 shutdown_connections 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 --2- 172.21.15.86:0/2657138474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac409ffa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- 172.21.15.86:0/2657138474 >> 172.21.15.86:0/2657138474 conn(0x7f8ac40a6440 msgr2=0x7f8ac40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:08.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- 172.21.15.86:0/2657138474 shutdown_connections 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- 172.21.15.86:0/2657138474 wait complete. 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: build_initial_monmap 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: monmap: 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:08.721 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 Processor -- start 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- start start 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: init 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-c con 0x7f8ac40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-b con 0x7f8ac4068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:08.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 0x7f8ac40724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): picked mon.noname-a con 0x7f8ac4139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac409be40 con 0x7f8ac4139f00 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac409bcc0 con 0x7f8ac4068db0 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8ac409bfc0 con 0x7f8ac40689d0 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8acb73e640 10 monclient(hunting): authenticate will time out at 1981.682343s 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:08.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 0x7f8ac40724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57232/0 (socket says 172.21.15.86:57232) 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 1 -- 172.21.15.86:0/3166146101 learned_addr learned my addr 172.21.15.86:0/3166146101 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): get_auth_request con 0x7f8ac40689d0 auth_method 0 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): get_auth_request con 0x7f8ac4139f00 auth_method 0 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:08.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 10 monclient(hunting): get_auth_request con 0x7f8ac4068db0 auth_method 0 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:08.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient(hunting): handle_auth_done global_id 184188 payload 824 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 msgr2=0x7f8ac4071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 msgr2=0x7f8ac4071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 msgr2=0x7f8ac4135720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 msgr2=0x7f8ac4071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac8cd9640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient: _finish_hunting 0 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 monclient: found mon.noname-a 2024-09-17T15:44:08.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac9cdb640 1 -- 172.21.15.86:0/3166146101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac411dac0 con 0x7f8ac4139f00 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.714+0000 7f8ac94da640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ac9cdb640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 msgr2=0x7f8ac40724a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ac9cdb640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 0x7f8ac40724a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00caf0 tx=0x7f8abc00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc00ade0 con 0x7f8ac4139f00 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: dump: 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:08.727 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8abc012400 con 0x7f8ac4139f00 2024-09-17T15:44:08.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8abc0127f0 con 0x7f8ac4139f00 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 10 monclient: dump: 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:08.729 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 5 monclient: authenticate success, global_id 184188 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _renew_subs 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8abb7fe640 4 set_mon_vals no callback set 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac411dda0 con 0x7f8ac4139f00 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _renew_subs 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac411c8e0 con 0x7f8ac4139f00 2024-09-17T15:44:08.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac409ffa0 con 0x7f8ac4139f00 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8abc012990 con 0x7f8ac4139f00 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 0x7f8aa803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(600..600 src has 438..600) ==== 4831+0+0 (secure 0 0 0) 0x7f8abc0504a0 con 0x7f8ac4139f00 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ac94da640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 0x7f8aa803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:08.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.718+0000 7f8ac94da640 10 monclient: get_auth_request con 0x7f8aa803bbe0 auth_method 0 2024-09-17T15:44:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.722+0000 7f8ac94da640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 0x7f8aa803e0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009740 tx=0x7f8ac00a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.722+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8abc007a60 con 0x7f8ac4139f00 2024-09-17T15:44:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.722+0000 7f8ab9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:08.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.722+0000 7f8ab9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.958+0000 7f8acb73e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.958+0000 7f8acb73e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.958+0000 7f8acb73e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.958+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ac406d4f0 con 0x7f8ac4139f00 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8ab9ffb640 1 -- 172.21.15.86:0/3166146101 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8abc01b920 con 0x7f8ac4139f00 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8ab9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8ab9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:08.959 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:08.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 msgr2=0x7f8aa803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:08.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 0x7f8aa803e0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009740 tx=0x7f8ac00a6000 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 10 monclient: shutdown 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 msgr2=0x7f8ac40724a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 0x7f8ac40724a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00caf0 tx=0x7f8abc00cef0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 shutdown_connections 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8aa803bbe0 0x7f8aa803e0a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ac4139f00 0x7f8ac40724a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ac4068db0 0x7f8ac4071f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 --2- 172.21.15.86:0/3166146101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ac40689d0 0x7f8ac4135720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:08.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 >> 172.21.15.86:0/3166146101 conn(0x7f8ac40a6440 msgr2=0x7f8ac413cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:08.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 shutdown_connections 2024-09-17T15:44:08.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:08.962+0000 7f8acb73e640 1 -- 172.21.15.86:0/3166146101 wait complete. 2024-09-17T15:44:09.549 INFO:tasks.rados.rados.0.smithi203.stdout:2707: done (3 left) 2024-09-17T15:44:09.549 INFO:tasks.rados.rados.0.smithi203.stdout:2708: write (excl) oid 27 current snap is 290 2024-09-17T15:44:09.549 INFO:tasks.rados.rados.0.smithi203.stdout:2708: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-17T15:44:09.560 INFO:tasks.rados.rados.0.smithi203.stdout:2708: writing smithi20328086-27 from 551527 to 1126294 tid 1 2024-09-17T15:44:09.574 INFO:tasks.rados.rados.0.smithi203.stdout:2708: writing smithi20328086-27 from 1918725 to 2151527 tid 2 2024-09-17T15:44:09.574 INFO:tasks.rados.rados.0.smithi203.stdout:2709: snap_remove snap 288 2024-09-17T15:44:10.564 INFO:tasks.rados.rados.0.smithi203.stdout:2709: done (4 left) 2024-09-17T15:44:10.564 INFO:tasks.rados.rados.0.smithi203.stdout:2710: write (excl) oid 38 current snap is 290 2024-09-17T15:44:10.565 INFO:tasks.rados.rados.0.smithi203.stdout:2710: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-17T15:44:10.582 INFO:tasks.rados.rados.0.smithi203.stdout:2710: writing smithi20328086-38 from 568334 to 1218350 tid 1 2024-09-17T15:44:10.596 INFO:tasks.rados.rados.0.smithi203.stdout:2710: writing smithi20328086-38 from 1860854 to 2168334 tid 2 2024-09-17T15:44:10.596 INFO:tasks.rados.rados.0.smithi203.stdout:2711: read oid 40 snap 284 2024-09-17T15:44:10.596 INFO:tasks.rados.rados.0.smithi203.stdout:2711: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:44:10.596 INFO:tasks.rados.rados.0.smithi203.stdout:2712: write (excl) oid 5 current snap is 290 2024-09-17T15:44:10.596 INFO:tasks.rados.rados.0.smithi203.stdout:2712: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-17T15:44:10.613 INFO:tasks.rados.rados.0.smithi203.stdout:2712: writing smithi20328086-5 from 585141 to 1310406 tid 1 2024-09-17T15:44:10.625 INFO:tasks.rados.rados.0.smithi203.stdout:2712: writing smithi20328086-5 from 1919336 to 2185141 tid 2 2024-09-17T15:44:10.625 INFO:tasks.rados.rados.0.smithi203.stdout:2713: delete oid 41 current snap is 290 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: finishing write tid 1 to smithi20328086-13 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: finishing write tid 2 to smithi20328086-13 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: finishing write tid 3 to smithi20328086-13 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: finishing write tid 4 to smithi20328086-13 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: oid 13 updating version 0 to 1120 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: oid 13 version 1120 is already newer than 1119 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: oid 13 updating version 1120 to 1121 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1121 (ObjNum 959 snap 290 seq_num 959) dirty exists 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2703: left oid 13 (ObjNum 959 snap 290 seq_num 959) 2024-09-17T15:44:10.626 INFO:tasks.rados.rados.0.smithi203.stdout:2704: expect (ObjNum 958 snap 289 seq_num 958) 2024-09-17T15:44:10.636 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 603278 2024-09-17T15:44:10.692 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 255437 2024-09-17T15:44:10.730 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:10.759 INFO:tasks.rados.rados.0.smithi203.stdout:2705: finishing write tid 1 to smithi20328086-45 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: finishing write tid 2 to smithi20328086-45 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: finishing write tid 3 to smithi20328086-45 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: finishing write tid 4 to smithi20328086-45 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: oid 45 updating version 0 to 1168 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: oid 45 updating version 1168 to 1169 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: oid 45 version 1169 is already newer than 1167 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1169 (ObjNum 960 snap 290 seq_num 960) dirty exists 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2705: left oid 45 (ObjNum 960 snap 290 seq_num 960) 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2708: finishing write tid 1 to smithi20328086-27 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2708: finishing write tid 2 to smithi20328086-27 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2708: finishing write tid 3 to smithi20328086-27 2024-09-17T15:44:10.760 INFO:tasks.rados.rados.0.smithi203.stdout:2708: finishing write tid 4 to smithi20328086-27 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2708: oid 27 updating version 0 to 1353 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2708: oid 27 updating version 1353 to 1355 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2708: oid 27 version 1355 is already newer than 1354 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1355 (ObjNum 961 snap 290 seq_num 961) dirty exists 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2708: left oid 27 (ObjNum 961 snap 290 seq_num 961) 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: finishing write tid 1 to smithi20328086-38 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: finishing write tid 2 to smithi20328086-38 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: finishing write tid 3 to smithi20328086-38 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: finishing write tid 4 to smithi20328086-38 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: oid 38 updating version 0 to 1123 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: oid 38 updating version 1123 to 1124 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: oid 38 version 1124 is already newer than 1122 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1124 (ObjNum 962 snap 290 seq_num 962) dirty exists 2024-09-17T15:44:10.761 INFO:tasks.rados.rados.0.smithi203.stdout:2710: left oid 38 (ObjNum 962 snap 290 seq_num 962) 2024-09-17T15:44:10.762 INFO:tasks.rados.rados.0.smithi203.stdout:2711: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:44:11.077 INFO:tasks.rados.rados.0.smithi203.stdout:2712: finishing write tid 1 to smithi20328086-5 2024-09-17T15:44:11.077 INFO:tasks.rados.rados.0.smithi203.stdout:2712: finishing write tid 2 to smithi20328086-5 2024-09-17T15:44:11.077 INFO:tasks.rados.rados.0.smithi203.stdout:2712: finishing write tid 3 to smithi20328086-5 2024-09-17T15:44:11.077 INFO:tasks.rados.rados.0.smithi203.stdout:2712: finishing write tid 4 to smithi20328086-5 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2712: oid 5 updating version 0 to 1097 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2712: oid 5 updating version 1097 to 1098 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2712: oid 5 version 1098 is already newer than 1096 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1098 (ObjNum 963 snap 290 seq_num 963) dirty exists 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2712: left oid 5 (ObjNum 963 snap 290 seq_num 963) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2703: done (7 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2704: done (6 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2705: done (5 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2708: done (4 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2710: done (3 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2711: done (2 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2712: done (1 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2713: done (0 left) 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2714: write oid 13 current snap is 290 2024-09-17T15:44:11.078 INFO:tasks.rados.rados.0.smithi203.stdout:2714: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-17T15:44:11.095 INFO:tasks.rados.rados.0.smithi203.stdout:2714: writing smithi20328086-13 from 618755 to 1094518 tid 1 2024-09-17T15:44:11.111 INFO:tasks.rados.rados.0.smithi203.stdout:2714: writing smithi20328086-13 from 1636300 to 2094296 tid 2 2024-09-17T15:44:11.111 INFO:tasks.rados.rados.0.smithi203.stdout:2715: snap_remove snap 284 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2715: done (1 left) 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2716: read oid 39 snap -1 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2716: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: finishing write tid 1 to smithi20328086-13 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: finishing write tid 2 to smithi20328086-13 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: finishing write tid 3 to smithi20328086-13 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: finishing write tid 4 to smithi20328086-13 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: oid 13 updating version 0 to 1126 2024-09-17T15:44:11.528 INFO:tasks.rados.rados.0.smithi203.stdout:2714: oid 13 updating version 1126 to 1127 2024-09-17T15:44:11.529 INFO:tasks.rados.rados.0.smithi203.stdout:2714: oid 13 version 1127 is already newer than 1125 2024-09-17T15:44:11.529 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1127 (ObjNum 965 snap 290 seq_num 965) dirty exists 2024-09-17T15:44:11.529 INFO:tasks.rados.rados.0.smithi203.stdout:2714: left oid 13 (ObjNum 965 snap 290 seq_num 965) 2024-09-17T15:44:11.529 INFO:tasks.rados.rados.0.smithi203.stdout:2714: done (1 left) 2024-09-17T15:44:11.529 INFO:tasks.rados.rados.0.smithi203.stdout:2717: snap_create 2024-09-17T15:44:11.536 INFO:tasks.rados.rados.0.smithi203.stdout:2716: expect (ObjNum 829 snap 254 seq_num 829) 2024-09-17T15:44:11.703 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2732576 2024-09-17T15:44:11.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:12.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: get_monmap_and_config 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: build_initial_monmap 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: monmap: 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:12.113422+0000 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:12.113422+0000 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 Processor -- start 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 -- start start 2024-09-17T15:44:12.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: init 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0ac80689d0 0x7f0ac809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac80689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 0x7f0ac8069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ac8134a40 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ac809d770 con 0x7f0ac8134a40 2024-09-17T15:44:12.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0ac809ed00 con 0x7f0ac80689d0 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac809efe0 con 0x7f0ac8068db0 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acf984640 10 monclient(hunting): authenticate will time out at 1985.078863s 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acdf21640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0ac8134a40 0x7f0ac8134ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37458/0 (socket says 172.21.15.86:37458) 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acdf21640 1 -- 172.21.15.86:0/1473936320 learned_addr learned my addr 172.21.15.86:0/1473936320 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 --2- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 0x7f0ac8069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): get_auth_request con 0x7f0ac8068db0 auth_method 0 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:12.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acd720640 1 --2- 172.21.15.86:0/1473936320 >> v2:172.21.15.203:3300/0 conn(0x7f0ac80689d0 0x7f0ac809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acd720640 10 monclient(hunting): get_auth_request con 0x7f0ac80689d0 auth_method 0 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acd720640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acd720640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0acd720640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient(hunting): handle_auth_done global_id 164402 payload 274 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 -- 172.21.15.86:0/1473936320 >> v2:172.21.15.203:3300/0 conn(0x7f0ac80689d0 msgr2=0x7f0ac809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 --2- 172.21.15.86:0/1473936320 >> v2:172.21.15.203:3300/0 conn(0x7f0ac80689d0 0x7f0ac809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 -- 172.21.15.86:0/1473936320 >> v1:172.21.15.86:6789/0 conn(0x7f0ac8134a40 legacy=0x7f0ac8134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient: _finish_hunting 0 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 monclient: found mon.noname-b 2024-09-17T15:44:12.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:12.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.110+0000 7f0accf1f640 1 -- 172.21.15.86:0/1473936320 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac80a04e0 con 0x7f0ac8068db0 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0accf1f640 1 -- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 msgr2=0x7f0ac8069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0accf1f640 1 --2- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 0x7f0ac8069190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097c0 tx=0x7f0abc01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f99f44b6a8375416 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 1 -- 172.21.15.86:0/1473936320 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0b2070 con 0x7f0ac8068db0 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: dump: 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: _finish_auth 0 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 1 -- 172.21.15.86:0/1473936320 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0abc0ad640 con 0x7f0ac8068db0 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0acf984640 5 monclient: authenticate success, global_id 164402 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0acf984640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0acf984640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 1 -- 172.21.15.86:0/1473936320 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0ada10 con 0x7f0ac8068db0 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0acf984640 4 set_mon_vals no callback set 2024-09-17T15:44:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.114+0000 7f0ac7fff640 10 monclient: dump: 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: shutdown 2024-09-17T15:44:12.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 msgr2=0x7f0ac8069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 0x7f0ac8069190 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097c0 tx=0x7f0abc01c3f0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- 172.21.15.86:0/1473936320 shutdown_connections 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- 172.21.15.86:0/1473936320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8068db0 0x7f0ac8069190 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- 172.21.15.86:0/1473936320 >> v2:172.21.15.203:3300/0 conn(0x7f0ac80689d0 0x7f0ac809ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- 172.21.15.86:0/1473936320 >> 172.21.15.86:0/1473936320 conn(0x7f0ac80a6440 msgr2=0x7f0ac80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- 172.21.15.86:0/1473936320 shutdown_connections 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- 172.21.15.86:0/1473936320 wait complete. 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: build_initial_monmap 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: monmap: 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:12.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 Processor -- start 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- start start 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: init 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-a con 0x7f0ac80689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:12.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac8068db0 0x7f0ac8133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-b con 0x7f0ac8068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): picked mon.noname-c con 0x7f0ac8134a40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac813fad0 con 0x7f0ac80689d0 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac813f950 con 0x7f0ac8068db0 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0ac813fc50 con 0x7f0ac8134a40 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acf984640 10 monclient(hunting): authenticate will time out at 1985.086213s 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57264/0 (socket says 172.21.15.86:57264) 2024-09-17T15:44:12.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 -- 172.21.15.86:0/1410688575 learned_addr learned my addr 172.21.15.86:0/1410688575 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58004/0 (socket says 172.21.15.86:58004) 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): get_auth_request con 0x7f0ac8134a40 auth_method 0 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): get_auth_request con 0x7f0ac80689d0 auth_method 0 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:12.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0accf1f640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac8068db0 0x7f0ac8133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0accf1f640 10 monclient(hunting): get_auth_request con 0x7f0ac8068db0 auth_method 0 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0accf1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0accf1f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0accf1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient(hunting): handle_auth_done global_id 164405 payload 824 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 -- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac8068db0 msgr2=0x7f0ac8133c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac8068db0 0x7f0ac8133c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 -- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 msgr2=0x7f0ac813af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient: _finish_hunting 0 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 monclient: found mon.noname-c 2024-09-17T15:44:12.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acdf21640 1 -- 172.21.15.86:0/1410688575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac813b8b0 con 0x7f0ac8134a40 2024-09-17T15:44:12.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.118+0000 7f0acd720640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acdf21640 1 -- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 msgr2=0x7f0ac811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acdf21640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c420 tx=0x7f0ab800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8017070 con 0x7f0ac8134a40 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: dump: 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ab80070d0 con 0x7f0ac8134a40 2024-09-17T15:44:12.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80074c0 con 0x7f0ac8134a40 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 10 monclient: dump: 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:12.130 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac77fe640 4 set_mon_vals no callback set 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 5 monclient: authenticate success, global_id 164405 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _renew_subs 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac811f600 con 0x7f0ac8134a40 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _renew_subs 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac811e0d0 con 0x7f0ac8134a40 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac809ffa0 con 0x7f0ac8134a40 2024-09-17T15:44:12.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0ab801a070 con 0x7f0ac8134a40 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 0x7f0ab403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(603..603 src has 438..603) ==== 4875+0+0 (secure 0 0 0) 0x7f0ab804f430 con 0x7f0ac8134a40 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acd720640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 0x7f0ab403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.122+0000 7f0acd720640 10 monclient: get_auth_request con 0x7f0ab403bbe0 auth_method 0 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.126+0000 7f0acd720640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 0x7f0ab403e0a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0003f10 tx=0x7f0ac00098d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.126+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0ab8007660 con 0x7f0ac8134a40 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.126+0000 7f0ac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:12.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.126+0000 7f0ac5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:12.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:12.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:12.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:12.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0ac811f140 con 0x7f0ac8134a40 2024-09-17T15:44:12.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0ac5ffb640 1 -- 172.21.15.86:0/1410688575 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f0ab801bbb0 con 0x7f0ac8134a40 2024-09-17T15:44:12.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0ac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:12.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0ac5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:12.363 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":154,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:12.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 msgr2=0x7f0ab403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:12.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 0x7f0ab403e0a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0003f10 tx=0x7f0ac00098d0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 10 monclient: shutdown 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 msgr2=0x7f0ac811c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c420 tx=0x7f0ab800c8e0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 shutdown_connections 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0ab403bbe0 0x7f0ab403e0a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ac8134a40 0x7f0ac811c8c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac8068db0 0x7f0ac8133c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 --2- 172.21.15.86:0/1410688575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ac80689d0 0x7f0ac813af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 >> 172.21.15.86:0/1410688575 conn(0x7f0ac80a6440 msgr2=0x7f0ac813c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 shutdown_connections 2024-09-17T15:44:12.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:12.366+0000 7f0acf984640 1 -- 172.21.15.86:0/1410688575 wait complete. 2024-09-17T15:44:12.530 INFO:tasks.rados.rados.0.smithi203.stdout:2716: done (1 left) 2024-09-17T15:44:12.530 INFO:tasks.rados.rados.0.smithi203.stdout:2717: done (0 left) 2024-09-17T15:44:12.530 INFO:tasks.rados.rados.0.smithi203.stdout:2718: rollback oid 11 current snap is 291 2024-09-17T15:44:12.530 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 270 2024-09-17T15:44:12.530 INFO:tasks.rados.rados.0.smithi203.stdout:2719: snap_create 2024-09-17T15:44:12.541 INFO:tasks.rados.rados.0.smithi203.stdout:2718: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:44:12.542 INFO:tasks.rados.rados.0.smithi203.stdout:2718: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:44:12.543 INFO:tasks.rados.rados.0.smithi203.stdout:2718: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:44:12.543 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1226 (ObjNum 858 snap 263 seq_num 858) dirty exists 2024-09-17T15:44:13.532 INFO:tasks.rados.rados.0.smithi203.stdout:2718: done (1 left) 2024-09-17T15:44:13.532 INFO:tasks.rados.rados.0.smithi203.stdout:2719: done (0 left) 2024-09-17T15:44:13.532 INFO:tasks.rados.rados.0.smithi203.stdout:2720: snap_remove snap 291 2024-09-17T15:44:14.558 INFO:tasks.rados.rados.0.smithi203.stdout:2720: done (0 left) 2024-09-17T15:44:14.558 INFO:tasks.rados.rados.0.smithi203.stdout:2721: copy_from oid 39 from oid 36 current snap is 292 2024-09-17T15:44:14.559 INFO:tasks.rados.rados.0.smithi203.stdout:2722: copy_from oid 26 from oid 24 current snap is 292 2024-09-17T15:44:14.559 INFO:tasks.rados.rados.0.smithi203.stdout:2723: rollback oid 31 current snap is 292 2024-09-17T15:44:14.559 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 290 2024-09-17T15:44:14.559 INFO:tasks.rados.rados.0.smithi203.stdout:2724: snap_create 2024-09-17T15:44:14.570 INFO:tasks.rados.rados.0.smithi203.stdout:2723: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:44:14.572 INFO:tasks.rados.rados.0.smithi203.stdout:2723: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:44:14.573 INFO:tasks.rados.rados.0.smithi203.stdout:2723: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:44:14.573 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1262 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:44:14.574 INFO:tasks.rados.rados.0.smithi203.stdout:2722: finishing copy_from to smithi20328086-26 2024-09-17T15:44:14.574 INFO:tasks.rados.rados.0.smithi203.stdout:2722: got expected ENOENT (src dne) 2024-09-17T15:44:14.575 INFO:tasks.rados.rados.0.smithi203.stdout:2722: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:44:14.599 INFO:tasks.rados.rados.0.smithi203.stdout:2721: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:44:14.600 INFO:tasks.rados.rados.0.smithi203.stdout:2721: finishing copy_from to smithi20328086-39 2024-09-17T15:44:14.600 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1429 (ObjNum 888 snap 271 seq_num 888) dirty exists 2024-09-17T15:44:15.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: get_monmap_and_config 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: build_initial_monmap 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: monmap: 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:15.517409+0000 2024-09-17T15:44:15.511 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:15.517409+0000 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:15.512 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 Processor -- start 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 -- start start 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: init 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed000689d0 0x7fed0009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-a con 0x7fed000689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed00068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fed00139f00 0x7fed0013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-c con 0x7fed00139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed0009d770 con 0x7fed00068db0 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fed0009ed00 con 0x7fed000689d0 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fed0009efe0 con 0x7fed00139f00 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed0855f640 10 monclient(hunting): authenticate will time out at 1988.483395s 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed000689d0 0x7fed0009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed000689d0 0x7fed0009ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58016/0 (socket says 172.21.15.86:58016) 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 -- 172.21.15.86:0/958056922 learned_addr learned my addr 172.21.15.86:0/958056922 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): get_auth_request con 0x7fed000689d0 auth_method 0 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient(hunting): handle_auth_done global_id 184194 payload 274 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed06afc640 1 --2- 172.21.15.86:0/958056922 >> v2:172.21.15.203:3300/0 conn(0x7fed00139f00 0x7fed0013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 -- 172.21.15.86:0/958056922 >> v2:172.21.15.203:3300/0 conn(0x7fed00139f00 msgr2=0x7fed0013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 --2- 172.21.15.86:0/958056922 >> v2:172.21.15.203:3300/0 conn(0x7fed00139f00 0x7fed0013c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 -- 172.21.15.86:0/958056922 >> v1:172.21.15.203:6789/0 conn(0x7fed00068db0 legacy=0x7fed000691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient: _finish_hunting 0 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 monclient: found mon.noname-a 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:15.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:15.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.514+0000 7fed062fb640 1 -- 172.21.15.86:0/958056922 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed000a04e0 con 0x7fed000689d0 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 -- 172.21.15.86:0/958056922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed000689d0 msgr2=0x7fed0009ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 --2- 172.21.15.86:0/958056922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed000689d0 0x7fed0009ffa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed0009f2c0 tx=0x7fecfc01eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95032159acfb1669 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 1 -- 172.21.15.86:0/958056922 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0b4070 con 0x7fed000689d0 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: dump: 2024-09-17T15:44:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: _finish_auth 0 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: _check_auth_tickets 2024-09-17T15:44:15.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 1 -- 172.21.15.86:0/958056922 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecfc0a7a30 con 0x7fed000689d0 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 5 monclient: authenticate success, global_id 184194 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 1 -- 172.21.15.86:0/958056922 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0a7e00 con 0x7fed000689d0 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 4 set_mon_vals no callback set 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed052f9640 10 monclient: dump: 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: shutdown 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- 172.21.15.86:0/958056922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed000689d0 msgr2=0x7fed0009ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- 172.21.15.86:0/958056922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed000689d0 0x7fed0009ffa0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed0009f2c0 tx=0x7fecfc01eab0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- 172.21.15.86:0/958056922 shutdown_connections 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- 172.21.15.86:0/958056922 >> v2:172.21.15.203:3300/0 conn(0x7fed00139f00 0x7fed0013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- 172.21.15.86:0/958056922 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed000689d0 0x7fed0009ffa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- 172.21.15.86:0/958056922 >> 172.21.15.86:0/958056922 conn(0x7fed000a6440 msgr2=0x7fed000a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- 172.21.15.86:0/958056922 shutdown_connections 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- 172.21.15.86:0/958056922 wait complete. 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: build_initial_monmap 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: monmap: 2024-09-17T15:44:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:15.524 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 Processor -- start 2024-09-17T15:44:15.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- start start 2024-09-17T15:44:15.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: init 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-c con 0x7fed000689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed00068db0 0x7fed00131f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-b con 0x7fed00068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed00139f00 0x7fed00132460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): picked mon.noname-a con 0x7fed00139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fed0009c610 con 0x7fed00139f00 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed0009c490 con 0x7fed00068db0 2024-09-17T15:44:15.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fed0009c790 con 0x7fed000689d0 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed0855f640 10 monclient(hunting): authenticate will time out at 1988.490563s 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed00139f00 0x7fed00132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57270/0 (socket says 172.21.15.86:57270) 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 -- 172.21.15.86:0/2087136765 learned_addr learned my addr 172.21.15.86:0/2087136765 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): get_auth_request con 0x7fed000689d0 auth_method 0 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): get_auth_request con 0x7fed00139f00 auth_method 0 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed06afc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed05afa640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed00068db0 0x7fed00131f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed05afa640 10 monclient(hunting): get_auth_request con 0x7fed00068db0 auth_method 0 2024-09-17T15:44:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed05afa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed05afa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed05afa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient(hunting): handle_auth_done global_id 164408 payload 824 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 -- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed00068db0 msgr2=0x7fed00131f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed00068db0 0x7fed00131f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 -- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed00139f00 msgr2=0x7fed00132460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed00139f00 0x7fed00132460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient: _finish_hunting 0 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 monclient: found mon.noname-c 2024-09-17T15:44:15.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.522+0000 7fed062fb640 1 -- 172.21.15.86:0/2087136765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0009d460 con 0x7fed000689d0 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed062fb640 1 -- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 msgr2=0x7fed001cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed062fb640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b2c0 tx=0x7fecfc01d2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0bf070 con 0x7fed000689d0 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: dump: 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.534 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecfc0ba440 con 0x7fed000689d0 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 5 monclient: authenticate success, global_id 164408 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _renew_subs 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:15.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed00073230 con 0x7fed000689d0 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0b4070 con 0x7fed000689d0 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf7fff640 4 set_mon_vals no callback set 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 10 monclient: dump: 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _renew_subs 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:15.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed00074050 con 0x7fed000689d0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed0009ffa0 con 0x7fed000689d0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fecfc0b93e0 con 0x7fed000689d0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 0x7fece403e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(606..606 src has 438..606) ==== 4799+0+0 (secure 0 0 0) 0x7fecfc0ea2e0 con 0x7fed000689d0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed05afa640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 0x7fece403e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.526+0000 7fed05afa640 10 monclient: get_auth_request con 0x7fece403bbc0 auth_method 0 2024-09-17T15:44:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.530+0000 7fed05afa640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 0x7fece403e080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fed0009e330 tx=0x7fecf0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.530+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fecfc0187c0 con 0x7fed000689d0 2024-09-17T15:44:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.530+0000 7fecf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.530+0000 7fecf6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:15.557 INFO:tasks.rados.rados.0.smithi203.stdout:2721: done (3 left) 2024-09-17T15:44:15.557 INFO:tasks.rados.rados.0.smithi203.stdout:2722: done (2 left) 2024-09-17T15:44:15.557 INFO:tasks.rados.rados.0.smithi203.stdout:2723: done (1 left) 2024-09-17T15:44:15.557 INFO:tasks.rados.rados.0.smithi203.stdout:2724: done (0 left) 2024-09-17T15:44:15.557 INFO:tasks.rados.rados.0.smithi203.stdout:2725: write oid 29 current snap is 293 2024-09-17T15:44:15.558 INFO:tasks.rados.rados.0.smithi203.stdout:2725: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-17T15:44:15.579 INFO:tasks.rados.rados.0.smithi203.stdout:2725: writing smithi20328086-29 from 635562 to 1302927 tid 1 2024-09-17T15:44:15.594 INFO:tasks.rados.rados.0.smithi203.stdout:2725: writing smithi20328086-29 from 2094782 to 2235562 tid 2 2024-09-17T15:44:15.594 INFO:tasks.rados.rados.0.smithi203.stdout:2726: snap_remove snap 281 2024-09-17T15:44:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fed0855f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fed0855f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fed0855f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed001d0060 con 0x7fed000689d0 2024-09-17T15:44:15.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fecf6ffd640 1 -- 172.21.15.86:0/2087136765 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fecfc0c1d20 con 0x7fed000689d0 2024-09-17T15:44:15.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fecf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:15.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.766+0000 7fecf6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:15.765 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 msgr2=0x7fece403e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 0x7fece403e080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fed0009e330 tx=0x7fecf0009040 comp rx=0 tx=0).stop 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 10 monclient: shutdown 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 msgr2=0x7fed001cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00b2c0 tx=0x7fecfc01d2f0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 shutdown_connections 2024-09-17T15:44:15.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fece403bbc0 0x7fece403e080 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed00139f00 0x7fed00132460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed00068db0 0x7fed00131f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 --2- 172.21.15.86:0/2087136765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed000689d0 0x7fed001cfb20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 >> 172.21.15.86:0/2087136765 conn(0x7fed000a6440 msgr2=0x7fed000a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 shutdown_connections 2024-09-17T15:44:15.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.770+0000 7fed0855f640 1 -- 172.21.15.86:0/2087136765 wait complete. 2024-09-17T15:44:15.783 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:44:15.783 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:44:15.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:44:15.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: get_monmap_and_config 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: build_initial_monmap 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: monmap: 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:15.908945+0000 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:15.908945+0000 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:15.903 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:15.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 Processor -- start 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 -- start start 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: init 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:44:15.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc31c0a4900 0x7fc31c06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-a con 0x7fc31c0a4900 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc31c06d0b0 0x7fc31c06d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-c con 0x7fc31c06d0b0 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 0x7fc31c13e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-b con 0x7fc31c13c4b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc31c0a7180 con 0x7fc31c0a4900 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc31c0a81b0 con 0x7fc31c06d0b0 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc31c06ed80 con 0x7fc31c13c4b0 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc322b43640 10 monclient(hunting): authenticate will time out at 1988.875028s 2024-09-17T15:44:15.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc31c0a4900 0x7fc31c06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 0x7fc31c13e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc31c0a4900 0x7fc31c06cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58024/0 (socket says 172.21.15.86:58024) 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 0x7fc31c13e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57286/0 (socket says 172.21.15.86:57286) 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 -- 172.21.15.86:0/3863926965 learned_addr learned my addr 172.21.15.86:0/3863926965 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): get_auth_request con 0x7fc31c0a4900 auth_method 0 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): get_auth_request con 0x7fc31c13c4b0 auth_method 0 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc31bfff640 1 --2- 172.21.15.86:0/3863926965 >> v2:172.21.15.203:3300/0 conn(0x7fc31c06d0b0 0x7fc31c06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3210e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc31bfff640 10 monclient(hunting): get_auth_request con 0x7fc31c06d0b0 auth_method 0 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient(hunting): handle_auth_done global_id 184200 payload 274 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 -- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 msgr2=0x7fc31c13e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:15.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 --2- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 0x7fc31c13e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 -- 172.21.15.86:0/3863926965 >> v2:172.21.15.203:3300/0 conn(0x7fc31c06d0b0 msgr2=0x7fc31c06d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 --2- 172.21.15.86:0/3863926965 >> v2:172.21.15.203:3300/0 conn(0x7fc31c06d0b0 0x7fc31c06d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient: _finish_hunting 0 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 monclient: found mon.noname-a 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:15.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.906+0000 7fc3208df640 1 -- 172.21.15.86:0/3863926965 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c09bf70 con 0x7fc31c0a4900 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc3208df640 1 -- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 msgr2=0x7fc31c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc3208df640 1 --2- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c06cb70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc31c09e4d0 tx=0x7fc30c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80f0ef84905f543 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 1 -- 172.21.15.86:0/3863926965 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0b0070 con 0x7fc31c0a4900 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 10 monclient: dump: 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.910+0000 7fc31b7fe640 1 -- 172.21.15.86:0/3863926965 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc30c0ab600 con 0x7fc31c0a4900 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:15.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31b7fe640 1 -- 172.21.15.86:0/3863926965 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0ab9d0 con 0x7fc31c0a4900 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31b7fe640 10 monclient: dump: 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 5 monclient: authenticate success, global_id 184200 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 4 set_mon_vals no callback set 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: shutdown 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 msgr2=0x7fc31c06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c06cb70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc31c09e4d0 tx=0x7fc30c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- 172.21.15.86:0/3863926965 shutdown_connections 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c13c4b0 0x7fc31c13e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- 172.21.15.86:0/3863926965 >> v2:172.21.15.203:3300/0 conn(0x7fc31c06d0b0 0x7fc31c06d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- 172.21.15.86:0/3863926965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c06cb70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- 172.21.15.86:0/3863926965 >> 172.21.15.86:0/3863926965 conn(0x7fc31c0a3e10 msgr2=0x7fc31c0a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- 172.21.15.86:0/3863926965 shutdown_connections 2024-09-17T15:44:15.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- 172.21.15.86:0/3863926965 wait complete. 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: build_initial_monmap 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: monmap: 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 Processor -- start 2024-09-17T15:44:15.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- start start 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: init 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c06d0b0 0x7fc31c13bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-c con 0x7fc31c06d0b0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-a con 0x7fc31c0a4900 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31c13c4b0 0x7fc31c0919c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): picked mon.noname-b con 0x7fc31c13c4b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c06d9d0 con 0x7fc31c0a4900 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c09c2c0 con 0x7fc31c13c4b0 2024-09-17T15:44:15.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc31c0a4ce0 con 0x7fc31c06d0b0 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc322b43640 10 monclient(hunting): authenticate will time out at 1988.882081s 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c06d0b0 0x7fc31c13bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58030/0 (socket says 172.21.15.86:58030) 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 -- 172.21.15.86:0/4243571164 learned_addr learned my addr 172.21.15.86:0/4243571164 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): get_auth_request con 0x7fc31c0a4900 auth_method 0 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): get_auth_request con 0x7fc31c06d0b0 auth_method 0 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient(hunting): handle_auth_done global_id 184203 payload 824 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3210e0640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31c13c4b0 0x7fc31c0919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c06d0b0 msgr2=0x7fc31c13bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c06d0b0 0x7fc31c13bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31c13c4b0 msgr2=0x7fc31c0919c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc31c13c4b0 0x7fc31c0919c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:15.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 monclient: found mon.noname-a 2024-09-17T15:44:15.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:15.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:15.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc31bfff640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31c095280 con 0x7fc31c0a4900 2024-09-17T15:44:15.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.914+0000 7fc3208df640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc31c06d0b0 0x7fc31c13bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc31bfff640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 msgr2=0x7fc31c091480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc31bfff640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d5f0 tx=0x7fc31000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310012800 con 0x7fc31c0a4900 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: dump: 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.925 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: _finish_auth 0 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: _check_auth_tickets 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc310019070 con 0x7fc31c0a4900 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310012be0 con 0x7fc31c0a4900 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 5 monclient: authenticate success, global_id 184203 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 10 monclient: dump: 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.927 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 monclient: _renew_subs 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc31affd640 4 set_mon_vals no callback set 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31c13e680 con 0x7fc31c0a4900 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 monclient: _renew_subs 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31c094fa0 con 0x7fc31c0a4900 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 0x7fc31c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c13ebb0 con 0x7fc31c091f00 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 0x7fc31c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient: get_auth_request con 0x7fc31c091f00 auth_method 0 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc31bfff640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d5f0 tx=0x7fc31000dab0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7fc31001e460 mgrmap(e 5) 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc31001e460 con 0x7fc31c0a4900 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 10 monclient(hunting): handle_auth_done global_id 184206 payload 824 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3208df640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc31c0922e0 con 0x7fc31c091f00 2024-09-17T15:44:15.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3197fa640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc30803bbb0 0x7fc30803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3210e0640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc30803bbb0 0x7fc30803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.918+0000 7fc3210e0640 10 monclient: get_auth_request con 0x7fc30803bbb0 auth_method 0 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3208df640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 msgr2=0x7fc31c070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3208df640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 0x7fc31c070fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc31c06fa40 tx=0x7fc30c019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0ba070 con 0x7fc31c091f00 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 10 monclient: dump: 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc30c0b1040 con 0x7fc31c091f00 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 msgr2=0x7fc31c070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 0x7fc31c070fc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc31c06fa40 tx=0x7fc30c019180 comp rx=0 tx=0).stop 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 msgr2=0x7fc31c070fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3197fa640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c091f00 0x7fc31c070fc0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc31c06fa40 tx=0x7fc30c019180 comp rx=0 tx=0).stop 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.922+0000 7fc3210e0640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc30803bbb0 0x7fc30803e070 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc31c093810 tx=0x7fc31400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc322b43640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc322b43640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 0x7fc31c0718e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:15.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc322b43640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc31c06d490 con 0x7fc31c071500 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 0x7fc31c0718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient: get_auth_request con 0x7fc31c071500 auth_method 0 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 10 monclient(hunting): handle_auth_done global_id 184209 payload 824 2024-09-17T15:44:15.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.926+0000 7fc3208df640 1 -- 172.21.15.86:0/4243571164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc31c093a80 con 0x7fc31c071500 2024-09-17T15:44:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3208df640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 msgr2=0x7fc31c0718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3208df640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 0x7fc31c0718e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc30c01cc80 tx=0x7fc30c0a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc30c0ba070 con 0x7fc31c071500 2024-09-17T15:44:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:15.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 10 monclient: dump: 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fc30c003a40 con 0x7fc31c071500 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:15.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 msgr2=0x7fc31c0718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 0x7fc31c0718e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc30c01cc80 tx=0x7fc30c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 msgr2=0x7fc31c0718e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c071500 0x7fc31c0718e0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc30c01cc80 tx=0x7fc30c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 59423, 2024-09-17T15:44:15.935 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:15.936 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:15.937 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:15.938 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:15.939 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.940 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:15.942 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:15.943 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:15.944 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.945 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:15.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:15.947 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:44:15.948 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:15.948 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:15.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:15.948 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:15.948 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc30803bbb0 msgr2=0x7fc30803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc30803bbb0 0x7fc30803e070 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc31c093810 tx=0x7fc31400b040 comp rx=0 tx=0).stop 2024-09-17T15:44:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 10 monclient: shutdown 2024-09-17T15:44:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 msgr2=0x7fc31c091480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 --2- 172.21.15.86:0/4243571164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc31c0a4900 0x7fc31c091480 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc31000d5f0 tx=0x7fc31000dab0 comp rx=0 tx=0).stop 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3210e0640 1 -- 172.21.15.86:0/4243571164 reap_dead start 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3197fa640 1 -- 172.21.15.86:0/4243571164 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(607..607 src has 438..607) ==== 4771+0+0 (unknown 0 0 0) 0x7fc310050270 con 0x7fc31c0a4900 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc3210e0640 1 -- 172.21.15.86:0/4243571164 reap_dead start 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 shutdown_connections 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 >> 172.21.15.86:0/4243571164 conn(0x7fc31c0a3e10 msgr2=0x7fc31c13d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 shutdown_connections 2024-09-17T15:44:15.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:15.930+0000 7fc322b43640 1 -- 172.21.15.86:0/4243571164 wait complete. 2024-09-17T15:44:15.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:15.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:44:15.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:44:16.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: get_monmap_and_config 2024-09-17T15:44:16.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: build_initial_monmap 2024-09-17T15:44:16.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: monmap: 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:16.068633+0000 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:16.068633+0000 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:16.063 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 Processor -- start 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 -- start start 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: init 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcec00a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcec00a5bd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:16.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcec009cd60 0x7fcec0140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-a con 0x7fcec009cd60 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec00a6280 con 0x7fcec00a50e0 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec009c0a0 con 0x7fcec00a5bd0 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcec009c380 con 0x7fcec009cd60 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec695f640 10 monclient(hunting): authenticate will time out at 1989.034839s 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcec009cd60 0x7fcec0140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcebffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcec00a50e0 0x7fcec00a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37462/0 (socket says 172.21.15.86:37462) 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcebffff640 1 -- 172.21.15.86:0/3510371734 learned_addr learned my addr 172.21.15.86:0/3510371734 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): get_auth_request con 0x7fcec009cd60 auth_method 0 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient(hunting): handle_auth_done global_id 184212 payload 274 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 1 -- 172.21.15.86:0/3510371734 >> v1:172.21.15.203:6789/0 conn(0x7fcec00a5bd0 legacy=0x7fcec013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 1 -- 172.21.15.86:0/3510371734 >> v1:172.21.15.86:6789/0 conn(0x7fcec00a50e0 legacy=0x7fcec00a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient: _finish_hunting 0 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 1 monclient: found mon.noname-a 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.066+0000 7fcec4efc640 1 -- 172.21.15.86:0/3510371734 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec009c660 con 0x7fcec009cd60 2024-09-17T15:44:16.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.070+0000 7fcec4efc640 1 -- 172.21.15.86:0/3510371734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 msgr2=0x7fcec0140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.070+0000 7fcec4efc640 1 --2- 172.21.15.86:0/3510371734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec0140a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fceb80097c0 tx=0x7fceb801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9aed69da460b5ec server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 1 -- 172.21.15.86:0/3510371734 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80b0070 con 0x7fcec009cd60 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: dump: 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.071 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 5 monclient: authenticate success, global_id 184212 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 1 -- 172.21.15.86:0/3510371734 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb80ab600 con 0x7fcec009cd60 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 4 set_mon_vals no callback set 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 1 -- 172.21.15.86:0/3510371734 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb80a4e60 con 0x7fcec009cd60 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebeffd640 10 monclient: dump: 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: shutdown 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- 172.21.15.86:0/3510371734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 msgr2=0x7fcec0140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 --2- 172.21.15.86:0/3510371734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec0140a20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fceb80097c0 tx=0x7fceb801c330 comp rx=0 tx=0).stop 2024-09-17T15:44:16.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- 172.21.15.86:0/3510371734 shutdown_connections 2024-09-17T15:44:16.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 --2- 172.21.15.86:0/3510371734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec0140a20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- 172.21.15.86:0/3510371734 >> 172.21.15.86:0/3510371734 conn(0x7fcec00aaae0 msgr2=0x7fcec00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:16.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- 172.21.15.86:0/3510371734 shutdown_connections 2024-09-17T15:44:16.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- 172.21.15.86:0/3510371734 wait complete. 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: build_initial_monmap 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: monmap: 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:16.078 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 Processor -- start 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- start start 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: init 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-a con 0x7fcec009cd60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec00a50e0 0x7fcec0134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-c con 0x7fcec00a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcec00a5bd0 0x7fcec01392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): picked mon.noname-b con 0x7fcec00a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:16.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcec0143900 con 0x7fcec009cd60 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcec0143780 con 0x7fcec00a5bd0 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcec0143a80 con 0x7fcec00a50e0 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec695f640 10 monclient(hunting): authenticate will time out at 1989.042566s 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec01340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58070/0 (socket says 172.21.15.86:58070) 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 -- 172.21.15.86:0/2299925337 learned_addr learned my addr 172.21.15.86:0/2299925337 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): get_auth_request con 0x7fcec009cd60 auth_method 0 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec00a50e0 0x7fcec0134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): get_auth_request con 0x7fcec00a50e0 auth_method 0 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec4efc640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcec00a5bd0 0x7fcec01392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient(hunting): handle_auth_done global_id 184215 payload 824 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec00a50e0 msgr2=0x7fcec0134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec00a50e0 0x7fcec0134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcec00a5bd0 msgr2=0x7fcec01392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcec00a5bd0 0x7fcec01392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 monclient: found mon.noname-a 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcebffff640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec013fc30 con 0x7fcec009cd60 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec4efc640 10 monclient: get_auth_request con 0x7fcec00a5bd0 auth_method 0 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.074+0000 7fcec4efc640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcec00a5bd0 0x7fcec01392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebffff640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 msgr2=0x7fcec01340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebffff640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec01340c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fceb000cc10 tx=0x7fceb000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0012400 con 0x7fcec009cd60 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: dump: 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: _finish_auth 0 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: _check_auth_tickets 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb00125e0 con 0x7fcec009cd60 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb00129d0 con 0x7fcec009cd60 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 5 monclient: authenticate success, global_id 184215 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 monclient: _renew_subs 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec013ca30 con 0x7fcec009cd60 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 10 monclient: dump: 2024-09-17T15:44:16.084 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebdffb640 4 set_mon_vals no callback set 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 monclient: _renew_subs 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec013c750 con 0x7fcec009cd60 2024-09-17T15:44:16.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 0x7fcec0139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcec00a54c0 con 0x7fcec0139810 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 0x7fcec0139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient: get_auth_request con 0x7fcec0139810 auth_method 0 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fceb0012bf0 con 0x7fcec009cd60 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fceac03bbe0 0x7fceac03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_done global_id 164420 payload 824 2024-09-17T15:44:16.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebf7fe640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcec00008e0 con 0x7fcec0139810 2024-09-17T15:44:16.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(607..607 src has 438..607) ==== 4771+0+0 (secure 0 0 0) 0x7fceb00505f0 con 0x7fcec009cd60 2024-09-17T15:44:16.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec4efc640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fceac03bbe0 0x7fceac03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.078+0000 7fcec4efc640 10 monclient: get_auth_request con 0x7fceac03bbe0 auth_method 0 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebf7fe640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 msgr2=0x7fcec0139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebf7fe640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 0x7fcec0139bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb400fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4003b80 con 0x7fcec0139810 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 10 monclient: dump: 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.089 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fceb4019070 con 0x7fcec0139810 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 msgr2=0x7fcec0139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 0x7fcec0139bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb400fc00 comp rx=0 tx=0).stop 2024-09-17T15:44:16.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 msgr2=0x7fcec0139bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcebcff9640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec0139810 0x7fcec0139bf0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb400fc00 comp rx=0 tx=0).stop 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.082+0000 7fcec4efc640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fceac03bbe0 0x7fceac03e0a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fceb80ab000 tx=0x7fceb8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcec695f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcec695f640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 0x7fcec0135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcec695f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcec013b570 con 0x7fcec013a130 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 0x7fcec0135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient: get_auth_request con 0x7fcec013a130 auth_method 0 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.086+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebf7fe640 10 monclient(hunting): handle_auth_done global_id 164423 payload 824 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebf7fe640 1 -- 172.21.15.86:0/2299925337 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcec0134b40 con 0x7fcec013a130 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebf7fe640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 msgr2=0x7fcec0135080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebf7fe640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 0x7fcec0135080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb4003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4006e00 con 0x7fcec013a130 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 10 monclient: dump: 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fceb4006e00 con 0x7fcec013a130 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 msgr2=0x7fcec0135080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 0x7fcec0135080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb4003770 comp rx=0 tx=0).stop 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 msgr2=0x7fcec0135080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:16.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcebcff9640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcec013a130 0x7fcec0135080 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcec0139090 tx=0x7fceb4003770 comp rx=0 tx=0).stop 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 159278, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:44:16.094 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:16.095 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:16.096 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:16.097 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:16.098 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:16.099 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.100 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:16.101 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:16.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:16.103 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:16.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.105 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:16.106 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fceac03bbe0 msgr2=0x7fceac03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fceac03bbe0 0x7fceac03e0a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fceb80ab000 tx=0x7fceb8008040 comp rx=0 tx=0).stop 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 10 monclient: shutdown 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 msgr2=0x7fcec01340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 --2- 172.21.15.86:0/2299925337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcec009cd60 0x7fcec01340c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fceb000cc10 tx=0x7fceb000a430 comp rx=0 tx=0).stop 2024-09-17T15:44:16.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec4efc640 1 -- 172.21.15.86:0/2299925337 reap_dead start 2024-09-17T15:44:16.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec4efc640 1 -- 172.21.15.86:0/2299925337 reap_dead start 2024-09-17T15:44:16.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 shutdown_connections 2024-09-17T15:44:16.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 >> 172.21.15.86:0/2299925337 conn(0x7fcec00aaae0 msgr2=0x7fcec00a77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:16.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 shutdown_connections 2024-09-17T15:44:16.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.090+0000 7fcec695f640 1 -- 172.21.15.86:0/2299925337 wait complete. 2024-09-17T15:44:16.109 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:16.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:44:16.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:44:16.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: get_monmap_and_config 2024-09-17T15:44:16.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: build_initial_monmap 2024-09-17T15:44:16.218 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: monmap: 2024-09-17T15:44:16.218 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:16.224100+0000 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:16.224100+0000 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:16.219 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:16.220 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:16.220 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:16.220 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:16.220 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 Processor -- start 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 -- start start 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: init 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-17T15:44:16.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 0x7fc9800a1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): picked mon.noname-c con 0x7fc980134d90 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc980135170 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc980135580 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc980138a00 con 0x7fc980135580 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc980073b10 con 0x7fc980135170 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc9800a21b0 con 0x7fc980134d90 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc986158640 10 monclient(hunting): authenticate will time out at 1989.189591s 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc980135580 0x7fc9801382f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37466/0 (socket says 172.21.15.86:37466) 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 learned_addr learned my addr 172.21.15.86:0/1125122462 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:16.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 0x7fc9800a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request con 0x7fc980134d90 auth_method 0 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 1 -- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 msgr2=0x7fc9800a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 1 -- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 msgr2=0x7fc9800a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 0x7fc9800a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 0x7fc9800a1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.222+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 >> v1:172.21.15.86:6789/0 conn(0x7fc980135580 legacy=0x7fc9801382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550923807 0 0) 0x7fc980138a00 con 0x7fc980135580 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): _negotiate 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): my global_id is 184218 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc968003530 con 0x7fc980135580 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3796511192 0 0) 0x7fc968003530 con 0x7fc980135580 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 msgr2=0x7fc9800a1130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 --2- 172.21.15.86:0/1125122462 >> v2:172.21.15.203:3300/0 conn(0x7fc980134d90 0x7fc9800a1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 >> v1:172.21.15.203:6789/0 conn(0x7fc980135170 legacy=0x7fc980137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 1 -- 172.21.15.86:0/1125122462 >> v1:172.21.15.203:6789/0 conn(0x7fc980135170 legacy=0x7fc980137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9800a1ed0 con 0x7fc980135580 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:44:16.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fc978001d30 con 0x7fc980135580 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: dump: 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 >> v1:172.21.15.86:6789/0 conn(0x7fc980135580 legacy=0x7fc9801382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc968003bd0 0x7fc968003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): picked mon.a con 0x7fc968003bd0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc968004810 0x7fc968004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): picked mon.b con 0x7fc968004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 0x7fc96800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): picked mon.c con 0x7fc9680099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc9800a21b0 con 0x7fc968003bd0 2024-09-17T15:44:16.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc96800c4e0 con 0x7fc968004810 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc96800c6e0 con 0x7fc9680099c0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 0x7fc96800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): get_auth_request con 0x7fc9680099c0 auth_method 0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc968003bd0 0x7fc968003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request con 0x7fc968003bd0 auth_method 0 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc968004810 0x7fc968004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 10 monclient(hunting): get_auth_request con 0x7fc968004810 auth_method 0 2024-09-17T15:44:16.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient(hunting): handle_auth_done global_id 184218 payload 293 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc968004810 msgr2=0x7fc968004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc968004810 0x7fc968004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc968003bd0 msgr2=0x7fc968003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc968003bd0 0x7fc968003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 monclient: found mon.c 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc968003530 con 0x7fc9680099c0 2024-09-17T15:44:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient: _finish_auth 0 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc986158640 5 monclient: authenticate success, global_id 184218 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc986158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc968003bd0 0x7fc968003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.226+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 reap_dead start 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97ffff640 1 -- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 msgr2=0x7fc96800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97ffff640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 0x7fc96800bdb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc978001860 tx=0x7fc97801d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5beb234fe54601e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97000f040 con 0x7fc9680099c0 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: dump: 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9780ad600 con 0x7fc9680099c0 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 1 -- 172.21.15.86:0/1125122462 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97800a6d0 con 0x7fc9680099c0 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97e7fc640 10 monclient: dump: 2024-09-17T15:44:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 4 set_mon_vals no callback set 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: shutdown 2024-09-17T15:44:16.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 msgr2=0x7fc96800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 0x7fc96800bdb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc978001860 tx=0x7fc97801d4a0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- 172.21.15.86:0/1125122462 shutdown_connections 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 --2- 172.21.15.86:0/1125122462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9680099c0 0x7fc96800bdb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- 172.21.15.86:0/1125122462 >> 172.21.15.86:0/1125122462 conn(0x7fc9800a2f40 msgr2=0x7fc9800a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- 172.21.15.86:0/1125122462 shutdown_connections 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- 172.21.15.86:0/1125122462 wait complete. 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: build_initial_monmap 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: monmap: 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 Processor -- start 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- start start 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: init 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 0x7fc9801cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): picked mon.noname-a con 0x7fc9680099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 0x7fc980140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): picked mon.noname-c con 0x7fc9801404f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801cc570 0x7fc9801cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): picked mon.noname-b con 0x7fc9801cc570 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc9800a14c0 con 0x7fc9680099c0 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9800a1340 con 0x7fc9801cc570 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc980137be0 con 0x7fc9801404f0 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc986158640 10 monclient(hunting): authenticate will time out at 1989.199370s 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 0x7fc9801cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 0x7fc980140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 0x7fc9801cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58090/0 (socket says 172.21.15.86:58090) 2024-09-17T15:44:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 0x7fc980140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57338/0 (socket says 172.21.15.86:57338) 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 -- 172.21.15.86:0/2451488022 learned_addr learned my addr 172.21.15.86:0/2451488022 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request con 0x7fc9680099c0 auth_method 0 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): get_auth_request con 0x7fc9801404f0 auth_method 0 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient(hunting): handle_auth_done global_id 184221 payload 824 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 msgr2=0x7fc980140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97ffff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801cc570 0x7fc9801cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 0x7fc980140950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801cc570 msgr2=0x7fc9801cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801cc570 0x7fc9801cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:16.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 monclient: found mon.noname-a 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97f7fe640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9800a53f0 con 0x7fc9680099c0 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.230+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9801404f0 0x7fc980140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97f7fe640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 msgr2=0x7fc9801cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97f7fe640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 0x7fc9801cfad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e590 tx=0x7fc97000ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97001b020 con 0x7fc9680099c0 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: dump: 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: _finish_auth 0 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: _check_auth_tickets 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 5 monclient: authenticate success, global_id 184221 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc970017a20 con 0x7fc9680099c0 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:16.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 monclient: _renew_subs 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970017e10 con 0x7fc9680099c0 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98009d460 con 0x7fc9680099c0 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97dffb640 4 set_mon_vals no callback set 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 10 monclient: dump: 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:16.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 monclient: _renew_subs 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98013aca0 con 0x7fc9680099c0 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:16.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9800a5960 con 0x7fc9801ccf30 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc97001e020 con 0x7fc9680099c0 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc9801ccf30 auth_method 0 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc96c03bb60 0x7fc96c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(607..607 src has 438..607) ==== 4771+0+0 (secure 0 0 0) 0x7fc970050190 con 0x7fc9680099c0 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:16.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:16.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97ffff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc96c03bb60 0x7fc96c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.234+0000 7fc97ffff640 10 monclient: get_auth_request con 0x7fc96c03bb60 auth_method 0 2024-09-17T15:44:16.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.238+0000 7fc97ffff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc96c03bb60 0x7fc96c03e020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc980135960 tx=0x7fc98006d7c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc9801ccf30 auth_method 0 2024-09-17T15:44:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:44:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:44:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:16.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.438+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:44:16.569 INFO:tasks.rados.rados.0.smithi203.stdout:2726: done (1 left) 2024-09-17T15:44:16.569 INFO:tasks.rados.rados.0.smithi203.stdout:2727: rollback oid 37 current snap is 293 2024-09-17T15:44:16.569 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 270 2024-09-17T15:44:16.569 INFO:tasks.rados.rados.0.smithi203.stdout:2725: finishing write tid 1 to smithi20328086-29 2024-09-17T15:44:16.569 INFO:tasks.rados.rados.0.smithi203.stdout:2725: finishing write tid 2 to smithi20328086-29 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: finishing write tid 3 to smithi20328086-29 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: finishing write tid 4 to smithi20328086-29 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: oid 29 updating version 0 to 1178 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: oid 29 updating version 1178 to 1180 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: oid 29 version 1180 is already newer than 1179 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1180 (ObjNum 966 snap 293 seq_num 966) dirty exists 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: left oid 29 (ObjNum 966 snap 293 seq_num 966) 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2725: done (1 left) 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2728: read oid 1 snap -1 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2728: expect (ObjNum 928 snap 283 seq_num 928) 2024-09-17T15:44:16.570 INFO:tasks.rados.rados.0.smithi203.stdout:2729: snap_remove snap 292 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc9801ccf30 auth_method 0 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:44:16.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:16.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:16.842+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:44:17.586 INFO:tasks.rados.rados.0.smithi203.stdout:2729: done (2 left) 2024-09-17T15:44:17.586 INFO:tasks.rados.rados.0.smithi203.stdout:2730: copy_from oid 42 from oid 43 current snap is 293 2024-09-17T15:44:17.586 INFO:tasks.rados.rados.0.smithi203.stdout:2728: expect (ObjNum 928 snap 283 seq_num 928) 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc9801ccf30 auth_method 0 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:17.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:44:17.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:44:17.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:17.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:17.646+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:44:17.694 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:17.750 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2727: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:44:17.805 INFO:tasks.rados.rados.0.smithi203.stdout:2727: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2727: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1432 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2730: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2730: finishing copy_from to smithi20328086-42 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 644 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2727: done (2 left) 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2728: done (1 left) 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2730: done (0 left) 2024-09-17T15:44:17.806 INFO:tasks.rados.rados.0.smithi203.stdout:2731: snap_create 2024-09-17T15:44:18.595 INFO:tasks.rados.rados.0.smithi203.stdout:2731: done (0 left) 2024-09-17T15:44:18.595 INFO:tasks.rados.rados.0.smithi203.stdout:2732: rollback oid 45 current snap is 294 2024-09-17T15:44:18.595 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 279 2024-09-17T15:44:18.596 INFO:tasks.rados.rados.0.smithi203.stdout:2733: read oid 9 snap -1 2024-09-17T15:44:18.596 INFO:tasks.rados.rados.0.smithi203.stdout:2733: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:44:18.596 INFO:tasks.rados.rados.0.smithi203.stdout:2734: snap_remove snap 293 2024-09-17T15:44:19.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.234+0000 7fc966ffd640 10 monclient: tick 2024-09-17T15:44:19.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.234+0000 7fc966ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:19.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.234+0000 7fc966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:19.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.234+0000 7fc966ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:19.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc9801ccf30 auth_method 0 2024-09-17T15:44:19.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 10 monclient(hunting): handle_auth_done global_id 194098 payload 824 2024-09-17T15:44:19.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.250+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc980137f70 con 0x7fc9801ccf30 2024-09-17T15:44:19.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:19.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740153b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:19.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9740070d0 con 0x7fc9801ccf30 2024-09-17T15:44:19.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 10 monclient: dump: 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.251 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc9740070d0 con 0x7fc9801ccf30 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740153b0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 msgr2=0x7fc98009d630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:19.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.254+0000 7fc967fff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9801ccf30 0x7fc98009d630 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740153b0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc986158640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:19.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc986158640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 0x7fc98009df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc986158640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:19.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9801cd310 con 0x7fc98009db70 2024-09-17T15:44:19.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 0x7fc98009df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient: get_auth_request con 0x7fc98009db70 auth_method 0 2024-09-17T15:44:19.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 10 monclient(hunting): handle_auth_done global_id 194101 payload 824 2024-09-17T15:44:19.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.262+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc98006d4f0 con 0x7fc98009db70 2024-09-17T15:44:19.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc97effd640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 msgr2=0x7fc98009df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:19.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc97effd640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 0x7fc98009df80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740014c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974025570 con 0x7fc98009db70 2024-09-17T15:44:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:44:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 10 monclient: dump: 2024-09-17T15:44:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fc974025570 con 0x7fc98009db70 2024-09-17T15:44:19.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:19.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:19.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 msgr2=0x7fc98009df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 0x7fc98009df80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740014c0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 msgr2=0x7fc98009df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:19.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc967fff640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc98009db70 0x7fc98009df80 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc9801cd630 tx=0x7fc9740014c0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.269 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:19.269 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:19.270 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:19.270 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:44:19.270 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:19.270 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10603, 2024-09-17T15:44:19.270 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:44:19.271 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:19.272 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:19.272 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:19.272 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:19.273 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:19.273 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:19.273 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:19.273 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:19.273 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:19.274 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:19.274 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:19.274 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:19.274 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:19.274 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:19.275 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:19.275 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:19.275 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:19.275 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:19.276 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:19.277 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:19.277 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:19.277 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:19.278 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:19.279 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:19.279 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:19.279 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:19.279 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:19.279 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:19.280 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:19.281 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:19.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:19.282 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:19.282 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:19.283 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:19.283 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:19.283 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:19.283 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:19.284 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.285 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:19.286 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:19.287 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:19.288 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:19.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:19.291 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:19.292 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:19.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc96c03bb60 msgr2=0x7fc96c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc96c03bb60 0x7fc96c03e020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc980135960 tx=0x7fc98006d7c0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 10 monclient: shutdown 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 msgr2=0x7fc9801cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 --2- 172.21.15.86:0/2451488022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9680099c0 0x7fc9801cfad0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e590 tx=0x7fc97000ea50 comp rx=0 tx=0).stop 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc97ffff640 1 -- 172.21.15.86:0/2451488022 reap_dead start 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc97ffff640 1 -- 172.21.15.86:0/2451488022 reap_dead start 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 shutdown_connections 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 >> 172.21.15.86:0/2451488022 conn(0x7fc9800a2f40 msgr2=0x7fc980136da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 shutdown_connections 2024-09-17T15:44:19.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.266+0000 7fc986158640 1 -- 172.21.15.86:0/2451488022 wait complete. 2024-09-17T15:44:19.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:19.295 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:44:19.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:44:19.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.422+0000 7f111ab69640 10 monclient: get_monmap_and_config 2024-09-17T15:44:19.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.422+0000 7f111ab69640 10 monclient: build_initial_monmap 2024-09-17T15:44:19.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.422+0000 7f111ab69640 10 monclient: monmap: 2024-09-17T15:44:19.420 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:19.426668+0000 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:19.426668+0000 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:19.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 Processor -- start 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 -- start start 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient: init 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 0x7f111409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-b con 0x7f11140a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11140a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f111413d280 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11140a6f40 con 0x7f11140a5450 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11140a65e0 con 0x7f111413d280 2024-09-17T15:44:19.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f111409c2a0 con 0x7f11140a5070 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f111ab69640 10 monclient(hunting): authenticate will time out at 1992.392814s 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f110bfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f11140a5450 0x7f11140a5830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58276/0 (socket says 172.21.15.86:58276) 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f110bfff640 1 -- 172.21.15.86:0/3128686509 learned_addr learned my addr 172.21.15.86:0/3128686509 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 1 --2- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 0x7f111409cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): get_auth_request con 0x7f11140a5070 auth_method 0 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:19.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient(hunting): handle_auth_done global_id 164429 payload 274 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 1 -- 172.21.15.86:0/3128686509 >> v1:172.21.15.203:6789/0 conn(0x7f111413d280 legacy=0x7f111413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 1 -- 172.21.15.86:0/3128686509 >> v1:172.21.15.86:6789/0 conn(0x7f11140a5450 legacy=0x7f11140a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient: _finish_hunting 0 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 1 monclient: found mon.noname-b 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.426+0000 7f1118905640 1 -- 172.21.15.86:0/3128686509 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f111409c580 con 0x7f11140a5070 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f1118905640 1 -- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 msgr2=0x7f111409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f1118905640 1 --2- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 0x7f111409cf60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f11100097f0 tx=0x7f111001c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a9dfe5a4bb8e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 1 -- 172.21.15.86:0/3128686509 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11100b0070 con 0x7f11140a5070 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: dump: 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.428 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 5 monclient: authenticate success, global_id 164429 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 1 -- 172.21.15.86:0/3128686509 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11100ab600 con 0x7f11140a5070 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:19.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 1 -- 172.21.15.86:0/3128686509 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11100ab9d0 con 0x7f11140a5070 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 4 set_mon_vals no callback set 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f110b7fe640 10 monclient: dump: 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: shutdown 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 msgr2=0x7f111409cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 --2- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 0x7f111409cf60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f11100097f0 tx=0x7f111001c430 comp rx=0 tx=0).stop 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- 172.21.15.86:0/3128686509 shutdown_connections 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 --2- 172.21.15.86:0/3128686509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5070 0x7f111409cf60 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- 172.21.15.86:0/3128686509 >> 172.21.15.86:0/3128686509 conn(0x7f11140742a0 msgr2=0x7f11140746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- 172.21.15.86:0/3128686509 shutdown_connections 2024-09-17T15:44:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- 172.21.15.86:0/3128686509 wait complete. 2024-09-17T15:44:19.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: build_initial_monmap 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: monmap: 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 Processor -- start 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- start start 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: init 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 0x7f1114138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-a con 0x7f11140a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-c con 0x7f11140a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f111413d280 0x7f111413a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): picked mon.noname-b con 0x7f111413d280 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1114143940 con 0x7f11140a5070 2024-09-17T15:44:19.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11141437c0 con 0x7f111413d280 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1114143ac0 con 0x7f11140a5450 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.430+0000 7f111ab69640 10 monclient(hunting): authenticate will time out at 1992.399917s 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 0x7f1114138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44154/0 (socket says 172.21.15.86:44154) 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 -- 172.21.15.86:0/3560049950 learned_addr learned my addr 172.21.15.86:0/3560049950 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): get_auth_request con 0x7f11140a5450 auth_method 0 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 0x7f1114138200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34474/0 (socket says 172.21.15.86:34474) 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): get_auth_request con 0x7f11140a5070 auth_method 0 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1118905640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1119106640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f111413d280 0x7f111413a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1119106640 10 monclient(hunting): get_auth_request con 0x7f111413d280 auth_method 0 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1119106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1119106640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f1119106640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient(hunting): handle_auth_done global_id 164432 payload 824 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 -- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f111413d280 msgr2=0x7f111413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f111413d280 0x7f111413a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 -- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 msgr2=0x7f1114138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 0x7f1114138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 monclient: found mon.noname-c 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 -- 172.21.15.86:0/3560049950 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11140a30e0 con 0x7f11140a5450 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 -- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 msgr2=0x7f111413a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110bfff640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f110400c4a0 tx=0x7f110400c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104017070 con 0x7f11140a5450 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: dump: 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.440 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: _finish_auth 0 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: _check_auth_tickets 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1104007070 con 0x7f11140a5450 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 5 monclient: authenticate success, global_id 164432 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _renew_subs 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1114121b80 con 0x7f11140a5450 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104007460 con 0x7f11140a5450 2024-09-17T15:44:19.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 10 monclient: dump: 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:19.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f110affd640 4 set_mon_vals no callback set 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _renew_subs 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f111413ce00 con 0x7f11140a5450 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111409cfe0 con 0x7f11140a5450 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.434+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f110401a070 con 0x7f11140a5450 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.438+0000 7f11097fa640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 0x7f110003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.438+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(610..610 src has 438..610) ==== 4787+0+0 (secure 0 0 0) 0x7f110404f490 con 0x7f11140a5450 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.438+0000 7f1118905640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 0x7f110003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:19.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.438+0000 7f1118905640 10 monclient: get_auth_request con 0x7f110003bbb0 auth_method 0 2024-09-17T15:44:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.438+0000 7f1118905640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 0x7f110003e070 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f111406e260 tx=0x7f11100a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.442+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1104007680 con 0x7f11140a5450 2024-09-17T15:44:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.442+0000 7f11097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:19.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.442+0000 7f11097fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:19.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.678+0000 7f111ab69640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:44:19.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.678+0000 7f111ab69640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:44:19.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.678+0000 7f111ab69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.678+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1114138e30 con 0x7f11140a5450 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=1405274487,config=3541143443,health=86150175,kv=2814306489,logm=3650697941}) 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=4205284161,config=3541143443,health=3081679109,kv=2732825231,logm=2732392261}) 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=1405274487,config=3541143443,health=86150175,kv=2814306489,logm=3650697941}) 2024-09-17T15:44:19.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=775235967,config=3541143443,health=86150175,kv=1440880585,logm=2324242790}) 2024-09-17T15:44:19.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f11097fa640 1 -- 172.21.15.86:0/3560049950 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f110401bbb0 con 0x7f11140a5450 2024-09-17T15:44:19.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f11097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:44:19.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f11097fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:19.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 msgr2=0x7f110003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 0x7f110003e070 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f111406e260 tx=0x7f11100a6000 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 10 monclient: shutdown 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 msgr2=0x7f111413a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f110400c4a0 tx=0x7f110400c960 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 shutdown_connections 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f110003bbb0 0x7f110003e070 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f111413d280 0x7f111413a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f11140a5450 0x7f111413a320 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 --2- 172.21.15.86:0/3560049950 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f11140a5070 0x7f1114138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.682+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 >> 172.21.15.86:0/3560049950 conn(0x7f11140742a0 msgr2=0x7f11140a3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.686+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 shutdown_connections 2024-09-17T15:44:19.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:19.686+0000 7f111ab69640 1 -- 172.21.15.86:0/3560049950 wait complete. 2024-09-17T15:44:19.695 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405986187}) 2024-09-17T15:44:19.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1932760917}) 2024-09-17T15:44:19.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405986187}) 2024-09-17T15:44:19.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925120607}) 2024-09-17T15:44:19.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:44:19.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235657938}) 2024-09-17T15:44:19.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448154595}) 2024-09-17T15:44:19.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:19.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235657938}) 2024-09-17T15:44:19.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:19.906+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712921163}) 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850617450}) 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497369048}) 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850617450}) 2024-09-17T15:44:20.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.186+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671764821}) 2024-09-17T15:44:20.479 INFO:tasks.rados.rados.0.smithi203.stdout:2734: done (2 left) 2024-09-17T15:44:20.479 INFO:tasks.rados.rados.0.smithi203.stdout:2735: write (excl) oid 21 current snap is 294 2024-09-17T15:44:20.479 INFO:tasks.rados.rados.0.smithi203.stdout:2735: seq_num 967 ranges {652369=742614,2153264=99105} 2024-09-17T15:44:20.501 INFO:tasks.rados.rados.0.smithi203.stdout:2735: writing smithi20328086-21 from 652369 to 1394983 tid 1 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2735: writing smithi20328086-21 from 2153264 to 2252369 tid 2 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2736: rollback oid 13 current snap is 294 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 13 to 279 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2732: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2732: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2732: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1172 (ObjNum 755 snap 233 seq_num 755) dirty exists 2024-09-17T15:44:20.516 INFO:tasks.rados.rados.0.smithi203.stdout:2733: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:44:20.518 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 446661 2024-09-17T15:44:20.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418522150}) 2024-09-17T15:44:20.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1234474450}) 2024-09-17T15:44:20.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418522150}) 2024-09-17T15:44:20.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.606+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1784716752}) 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090751283}) 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3700692952}) 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090751283}) 2024-09-17T15:44:20.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.614+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2084937930}) 2024-09-17T15:44:20.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193492376}) 2024-09-17T15:44:20.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109773661}) 2024-09-17T15:44:20.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193492376}) 2024-09-17T15:44:20.624 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.626+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2160189019}) 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416792983}) 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=701197754}) 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416792983}) 2024-09-17T15:44:20.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151898744}) 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801839171}) 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527041214}) 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801839171}) 2024-09-17T15:44:20.633 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185628003}) 2024-09-17T15:44:20.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679939786}) 2024-09-17T15:44:20.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1835920233}) 2024-09-17T15:44:20.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.635 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679939786}) 2024-09-17T15:44:20.636 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3601880019}) 2024-09-17T15:44:20.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448862137}) 2024-09-17T15:44:20.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548952488}) 2024-09-17T15:44:20.637 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.638+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448862137}) 2024-09-17T15:44:20.638 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220062035}) 2024-09-17T15:44:20.639 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356792914}) 2024-09-17T15:44:20.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2571507078}) 2024-09-17T15:44:20.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356792914}) 2024-09-17T15:44:20.640 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.642+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309706677}) 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215339628}) 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3988530309}) 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215339628}) 2024-09-17T15:44:20.642 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=904262025}) 2024-09-17T15:44:20.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189378297}) 2024-09-17T15:44:20.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=650368899}) 2024-09-17T15:44:20.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.644 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189378297}) 2024-09-17T15:44:20.645 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.646+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=313980621}) 2024-09-17T15:44:20.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383311685}) 2024-09-17T15:44:20.646 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4181475324}) 2024-09-17T15:44:20.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383311685}) 2024-09-17T15:44:20.647 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475029837}) 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83387415}) 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000052897}) 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83387415}) 2024-09-17T15:44:20.649 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.650+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417317518}) 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218281476}) 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86696443}) 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218281476}) 2024-09-17T15:44:20.651 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=296325553}) 2024-09-17T15:44:20.653 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978049763}) 2024-09-17T15:44:20.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35223600}) 2024-09-17T15:44:20.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.654+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1978049763}) 2024-09-17T15:44:20.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920103262}) 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330034667}) 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917334}) 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330034667}) 2024-09-17T15:44:20.656 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.658+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2868717478}) 2024-09-17T15:44:20.658 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251044712}) 2024-09-17T15:44:20.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287404215}) 2024-09-17T15:44:20.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251044712}) 2024-09-17T15:44:20.659 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1803555113}) 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243409946}) 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462966589}) 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243409946}) 2024-09-17T15:44:20.660 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.662+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3670729829}) 2024-09-17T15:44:20.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.662 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187752359}) 2024-09-17T15:44:20.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160002036}) 2024-09-17T15:44:20.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187752359}) 2024-09-17T15:44:20.663 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067409314}) 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014207804}) 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1462580243}) 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014207804}) 2024-09-17T15:44:20.665 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.666+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977655963}) 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463980542}) 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=473994474}) 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463980542}) 2024-09-17T15:44:20.667 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2616521819}) 2024-09-17T15:44:20.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057696085}) 2024-09-17T15:44:20.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21009100}) 2024-09-17T15:44:20.669 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.670+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057696085}) 2024-09-17T15:44:20.670 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078785908}) 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652247997}) 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510611444}) 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652247997}) 2024-09-17T15:44:20.672 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.674+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461516235}) 2024-09-17T15:44:20.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211177728}) 2024-09-17T15:44:20.674 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249299914}) 2024-09-17T15:44:20.675 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.675 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211177728}) 2024-09-17T15:44:20.675 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=695644262}) 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=130820240}) 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4180372411}) 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=130820240}) 2024-09-17T15:44:20.677 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.678+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760843492}) 2024-09-17T15:44:20.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163442019}) 2024-09-17T15:44:20.679 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699652347}) 2024-09-17T15:44:20.680 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.680 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163442019}) 2024-09-17T15:44:20.680 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.682+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452343622}) 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: finishing write tid 1 to smithi20328086-21 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: finishing write tid 2 to smithi20328086-21 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: finishing write tid 3 to smithi20328086-21 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: finishing write tid 4 to smithi20328086-21 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: oid 21 updating version 0 to 1471 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: oid 21 version 1471 is already newer than 1469 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: oid 21 version 1471 is already newer than 1470 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1471 (ObjNum 967 snap 294 seq_num 967) dirty exists 2024-09-17T15:44:20.681 INFO:tasks.rados.rados.0.smithi203.stdout:2735: left oid 21 (ObjNum 967 snap 294 seq_num 967) 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2736: finishing rollback tid 0 to smithi20328086-13 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2736: finishing rollback tid 1 to smithi20328086-13 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1129 (ObjNum 129 snap 0 seq_num 3606688368) dirty dne 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2732: done (3 left) 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2733: done (2 left) 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2735: done (1 left) 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2736: done (0 left) 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2737: write (excl) oid 2 current snap is 294 2024-09-17T15:44:20.682 INFO:tasks.rados.rados.0.smithi203.stdout:2737: seq_num 968 ranges {669176=417863,1811746=457430} 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745085201}) 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3027625730}) 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745085201}) 2024-09-17T15:44:20.683 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2142241840}) 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977933822}) 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011375872}) 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977933822}) 2024-09-17T15:44:20.684 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.686+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3854516474}) 2024-09-17T15:44:20.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544659868}) 2024-09-17T15:44:20.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777003558}) 2024-09-17T15:44:20.686 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544659868}) 2024-09-17T15:44:20.687 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513195704}) 2024-09-17T15:44:20.688 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225916294}) 2024-09-17T15:44:20.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1965889990}) 2024-09-17T15:44:20.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2225916294}) 2024-09-17T15:44:20.689 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.690+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492429928}) 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286910417}) 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2731929310}) 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286910417}) 2024-09-17T15:44:20.691 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2587258924}) 2024-09-17T15:44:20.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.694+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567353510}) 2024-09-17T15:44:20.693 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3302280166}) 2024-09-17T15:44:20.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567353510}) 2024-09-17T15:44:20.694 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080566609}) 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259043447}) 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253169122}) 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259043447}) 2024-09-17T15:44:20.696 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.698+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284654923}) 2024-09-17T15:44:20.698 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:44:20.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526478758}) 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493592372}) 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526478758}) 2024-09-17T15:44:20.699 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867106638}) 2024-09-17T15:44:20.700 INFO:tasks.rados.rados.0.smithi203.stdout:2737: writing smithi20328086-2 from 669176 to 1087039 tid 1 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37118630}) 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220787696}) 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37118630}) 2024-09-17T15:44:20.702 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.702+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994943882}) 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361489166}) 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983350985}) 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361489166}) 2024-09-17T15:44:20.704 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:20.706+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3803002706}) 2024-09-17T15:44:20.718 INFO:tasks.rados.rados.0.smithi203.stdout:2737: writing smithi20328086-2 from 1811746 to 2269176 tid 2 2024-09-17T15:44:20.718 INFO:tasks.rados.rados.0.smithi203.stdout:2738: read oid 36 snap -1 2024-09-17T15:44:20.719 INFO:tasks.rados.rados.0.smithi203.stdout:2738: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:44:20.719 INFO:tasks.rados.rados.0.smithi203.stdout:2739: read oid 10 snap -1 2024-09-17T15:44:20.719 INFO:tasks.rados.rados.0.smithi203.stdout:2739: expect deleted 2024-09-17T15:44:20.719 INFO:tasks.rados.rados.0.smithi203.stdout:2737: finishing write tid 1 to smithi20328086-2 2024-09-17T15:44:20.719 INFO:tasks.rados.rados.0.smithi203.stdout:2740: snap_remove snap 279 2024-09-17T15:44:20.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: get_monmap_and_config 2024-09-17T15:44:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: build_initial_monmap 2024-09-17T15:44:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: monmap: 2024-09-17T15:44:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:20.825877+0000 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:20.825877+0000 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:20.821 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:20.822 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:20.822 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 Processor -- start 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 -- start start 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: init 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-b con 0x7fd20c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd20c068db0 0x7fd20c069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-a con 0x7fd20c068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd20c134a40 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd20c09d770 con 0x7fd20c134a40 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd20c09ed00 con 0x7fd20c068db0 2024-09-17T15:44:20.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c09efe0 con 0x7fd20c0689d0 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd2113b8640 10 monclient(hunting): authenticate will time out at 1993.791441s 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44172/0 (socket says 172.21.15.86:44172) 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 -- 172.21.15.86:0/588318935 learned_addr learned my addr 172.21.15.86:0/588318935 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): get_auth_request con 0x7fd20c0689d0 auth_method 0 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 1 --2- 172.21.15.86:0/588318935 >> v2:172.21.15.86:3300/0 conn(0x7fd20c068db0 0x7fd20c069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): get_auth_request con 0x7fd20c068db0 auth_method 0 2024-09-17T15:44:20.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient(hunting): handle_auth_done global_id 164435 payload 274 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 -- 172.21.15.86:0/588318935 >> v2:172.21.15.86:3300/0 conn(0x7fd20c068db0 msgr2=0x7fd20c069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 --2- 172.21.15.86:0/588318935 >> v2:172.21.15.86:3300/0 conn(0x7fd20c068db0 0x7fd20c069190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 -- 172.21.15.86:0/588318935 >> v1:172.21.15.86:6789/0 conn(0x7fd20c134a40 legacy=0x7fd20c134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:20.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient: _finish_hunting 0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 monclient: found mon.noname-b 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.822+0000 7fd20affd640 1 -- 172.21.15.86:0/588318935 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c0a04e0 con 0x7fd20c0689d0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 -- 172.21.15.86:0/588318935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 msgr2=0x7fd20c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 --2- 172.21.15.86:0/588318935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00bb10 tx=0x7fd1fc01eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=739a96a105c5ac5d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 1 -- 172.21.15.86:0/588318935 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b4070 con 0x7fd20c0689d0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:20.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: dump: 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: _finish_auth 0 2024-09-17T15:44:20.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 5 monclient: authenticate success, global_id 164435 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 1 -- 172.21.15.86:0/588318935 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1fc0a7af0 con 0x7fd20c0689d0 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 4 set_mon_vals no callback set 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 1 -- 172.21.15.86:0/588318935 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0af440 con 0x7fd20c0689d0 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd209ffb640 10 monclient: dump: 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:20.832 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: shutdown 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- 172.21.15.86:0/588318935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 msgr2=0x7fd20c09ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- 172.21.15.86:0/588318935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00bb10 tx=0x7fd1fc01eb80 comp rx=0 tx=0).stop 2024-09-17T15:44:20.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- 172.21.15.86:0/588318935 shutdown_connections 2024-09-17T15:44:20.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- 172.21.15.86:0/588318935 >> v2:172.21.15.86:3300/0 conn(0x7fd20c068db0 0x7fd20c069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:20.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- 172.21.15.86:0/588318935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c0689d0 0x7fd20c09ffa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:20.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- 172.21.15.86:0/588318935 >> 172.21.15.86:0/588318935 conn(0x7fd20c0a6440 msgr2=0x7fd20c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:20.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- 172.21.15.86:0/588318935 shutdown_connections 2024-09-17T15:44:20.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- 172.21.15.86:0/588318935 wait complete. 2024-09-17T15:44:20.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: build_initial_monmap 2024-09-17T15:44:20.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: monmap: 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 Processor -- start 2024-09-17T15:44:20.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- start start 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: init 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-a con 0x7fd20c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c068db0 0x7fd20c13b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-c con 0x7fd20c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd20c134a40 0x7fd20c133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): picked mon.noname-b con 0x7fd20c134a40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c09c650 con 0x7fd20c0689d0 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:20.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd20c09c4d0 con 0x7fd20c134a40 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd20c09c7d0 con 0x7fd20c068db0 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd2113b8640 10 monclient(hunting): authenticate will time out at 1993.798862s 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34494/0 (socket says 172.21.15.86:34494) 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 -- 172.21.15.86:0/1825716687 learned_addr learned my addr 172.21.15.86:0/1825716687 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): get_auth_request con 0x7fd20c0689d0 auth_method 0 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:20.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:20.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:20.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c068db0 0x7fd20c13b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:20.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): get_auth_request con 0x7fd20c068db0 auth_method 0 2024-09-17T15:44:20.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:20.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:20.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient(hunting): handle_auth_done global_id 184230 payload 824 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 -- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c068db0 msgr2=0x7fd20c13b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c068db0 0x7fd20c13b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 -- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd20c134a40 msgr2=0x7fd20c133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd20c134a40 0x7fd20c133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient: _finish_hunting 0 2024-09-17T15:44:20.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 monclient: found mon.noname-a 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.830+0000 7fd20affd640 1 -- 172.21.15.86:0/1825716687 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20c13c840 con 0x7fd20c0689d0 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd20affd640 1 -- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 msgr2=0x7fd20c11cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd20affd640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b2c0 tx=0x7fd1fc01b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:20.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b4040 con 0x7fd20c0689d0 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: dump: 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.846 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:20.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1fc0afdc0 con 0x7fd20c0689d0 2024-09-17T15:44:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b8670 con 0x7fd20c0689d0 2024-09-17T15:44:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:20.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 10 monclient: dump: 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:20.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2097fa640 4 set_mon_vals no callback set 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 5 monclient: authenticate success, global_id 184230 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _renew_subs 2024-09-17T15:44:20.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd20c09d460 con 0x7fd20c0689d0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _renew_subs 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20c11e540 con 0x7fd20c0689d0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd1fc0b8810 con 0x7fd20c0689d0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd20c09ffa0 con 0x7fd20c0689d0 2024-09-17T15:44:20.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 0x7fd1f803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(611..611 src has 438..611) ==== 4815+0+0 (secure 0 0 0) 0x7fd1fc0ece00 con 0x7fd20c0689d0 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd20a7fc640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 0x7fd1f803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.834+0000 7fd20a7fc640 10 monclient: get_auth_request con 0x7fd1f803bbe0 auth_method 0 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.838+0000 7fd20a7fc640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 0x7fd1f803e0a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd20c11d880 tx=0x7fd200009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.838+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd1fc018800 con 0x7fd20c0689d0 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.838+0000 7fd1f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:20.838+0000 7fd1f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:21.024 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619679580}) 2024-09-17T15:44:21.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513547539}) 2024-09-17T15:44:21.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619679580}) 2024-09-17T15:44:21.025 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.026+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=204081345}) 2024-09-17T15:44:21.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764493284}) 2024-09-17T15:44:21.027 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792341660}) 2024-09-17T15:44:21.028 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.028 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1764493284}) 2024-09-17T15:44:21.028 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2143251013}) 2024-09-17T15:44:21.029 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989650711}) 2024-09-17T15:44:21.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242874845}) 2024-09-17T15:44:21.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=989650711}) 2024-09-17T15:44:21.030 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.030+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1291402948}) 2024-09-17T15:44:21.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420873292}) 2024-09-17T15:44:21.031 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2531401858}) 2024-09-17T15:44:21.032 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.032 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1420873292}) 2024-09-17T15:44:21.032 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.034+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3584720323}) 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408591258}) 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556687107}) 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408591258}) 2024-09-17T15:44:21.034 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3848366439}) 2024-09-17T15:44:21.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.036 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055718350}) 2024-09-17T15:44:21.037 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3375746114}) 2024-09-17T15:44:21.037 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.037 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055718350}) 2024-09-17T15:44:21.037 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.038+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3085235562}) 2024-09-17T15:44:21.039 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.039 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981666935}) 2024-09-17T15:44:21.039 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=381506515}) 2024-09-17T15:44:21.039 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.039 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981666935}) 2024-09-17T15:44:21.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2036098481}) 2024-09-17T15:44:21.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576650989}) 2024-09-17T15:44:21.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821163551}) 2024-09-17T15:44:21.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.041 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576650989}) 2024-09-17T15:44:21.042 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.042+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918444324}) 2024-09-17T15:44:21.043 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434945444}) 2024-09-17T15:44:21.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917027972}) 2024-09-17T15:44:21.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434945444}) 2024-09-17T15:44:21.044 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.046+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000403697}) 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4040048911}) 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1991364278}) 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4040048911}) 2024-09-17T15:44:21.046 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557749625}) 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859121451}) 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472559450}) 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859121451}) 2024-09-17T15:44:21.049 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.050+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009609147}) 2024-09-17T15:44:21.050 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289877375}) 2024-09-17T15:44:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157838623}) 2024-09-17T15:44:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289877375}) 2024-09-17T15:44:21.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.054+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3180597258}) 2024-09-17T15:44:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd2113b8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd2113b8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd2113b8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd20c06d4f0 con 0x7fd20c0689d0 2024-09-17T15:44:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd1f37fe640 1 -- 172.21.15.86:0/1825716687 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd1fc0eb7e0 con 0x7fd20c0689d0 2024-09-17T15:44:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd1f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.058+0000 7fd1f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:21.057 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:21.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 msgr2=0x7fd1f803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 0x7fd1f803e0a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd20c11d880 tx=0x7fd200009190 comp rx=0 tx=0).stop 2024-09-17T15:44:21.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 10 monclient: shutdown 2024-09-17T15:44:21.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 msgr2=0x7fd20c11cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc00b2c0 tx=0x7fd1fc01b920 comp rx=0 tx=0).stop 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 shutdown_connections 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd1f803bbe0 0x7fd1f803e0a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd20c134a40 0x7fd20c133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd20c068db0 0x7fd20c13b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 --2- 172.21.15.86:0/1825716687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd20c0689d0 0x7fd20c11cd40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 >> 172.21.15.86:0/1825716687 conn(0x7fd20c0a6440 msgr2=0x7fd20c0a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 shutdown_connections 2024-09-17T15:44:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.062+0000 7fd2113b8640 1 -- 172.21.15.86:0/1825716687 wait complete. 2024-09-17T15:44:21.074 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:44:21.074 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:44:21.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:44:21.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: get_monmap_and_config 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: monmap: 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:21.244581+0000 2024-09-17T15:44:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:21.244581+0000 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:21.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 Processor -- start 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 -- start start 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: init 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f951c0a01a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f951c0a0580 0x7f951c09f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-c con 0x7f951c0a0580 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f951c0a1380 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f951c074e00 con 0x7f951c0a1380 2024-09-17T15:44:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f951c09e350 con 0x7f951c0a01a0 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f951c09e630 con 0x7f951c0a0580 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95225eb640 10 monclient(hunting): authenticate will time out at 1994.210559s 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f9520b88640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f951c0a1380 0x7f951c13ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58294/0 (socket says 172.21.15.86:58294) 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f9520b88640 1 -- 172.21.15.86:0/538034899 learned_addr learned my addr 172.21.15.86:0/538034899 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 1 --2- 172.21.15.86:0/538034899 >> v2:172.21.15.203:3300/0 conn(0x7f951c0a0580 0x7f951c09f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): get_auth_request con 0x7f951c0a0580 auth_method 0 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.242+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 1 -- 172.21.15.86:0/538034899 >> v1:172.21.15.86:6789/0 conn(0x7f951c0a1380 legacy=0x7f951c13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288558017 0 0) 0x7f951c074e00 con 0x7f951c0a1380 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): my global_id is 184233 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9508003530 con 0x7f951c0a1380 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951bfff640 1 -- 172.21.15.86:0/538034899 >> v1:172.21.15.203:6789/0 conn(0x7f951c0a01a0 legacy=0x7f951c0a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2823403981 0 0) 0x7f9508003530 con 0x7f951c0a1380 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 >> v2:172.21.15.203:3300/0 conn(0x7f951c0a0580 msgr2=0x7f951c09f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 --2- 172.21.15.86:0/538034899 >> v2:172.21.15.203:3300/0 conn(0x7f951c0a0580 0x7f951c09f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 >> v1:172.21.15.203:6789/0 conn(0x7f951c0a01a0 legacy=0x7f951c0a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c09e910 con 0x7f951c0a1380 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f9514001d30 con 0x7f951c0a1380 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: dump: 2024-09-17T15:44:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 >> v1:172.21.15.86:6789/0 conn(0x7f951c0a1380 legacy=0x7f951c13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9508003b50 0x7f9508003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): picked mon.b con 0x7f9508003b50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): picked mon.a con 0x7f9508004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508005420 0x7f950800cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): picked mon.c con 0x7f9508005420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f951c09e630 con 0x7f9508004810 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f950800d440 con 0x7f9508003b50 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f950800d640 con 0x7f9508005420 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): get_auth_request con 0x7f9508004810 auth_method 0 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508005420 0x7f950800cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): get_auth_request con 0x7f9508005420 auth_method 0 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951bfff640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9508003b50 0x7f9508003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient(hunting): handle_auth_done global_id 184233 payload 293 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 -- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508005420 msgr2=0x7f950800cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508005420 0x7f950800cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 -- 172.21.15.86:0/538034899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9508003b50 msgr2=0x7f9508003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9508003b50 0x7f9508003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 monclient: found mon.a 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 1 -- 172.21.15.86:0/538034899 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9508003530 con 0x7f9508004810 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951bfff640 10 monclient: get_auth_request con 0x7f9508003b50 auth_method 0 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f9520b88640 1 -- 172.21.15.86:0/538034899 reap_dead start 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f951bfff640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9508003b50 0x7f9508003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95225eb640 5 monclient: authenticate success, global_id 184233 2024-09-17T15:44:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.246+0000 7f95225eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95137fe640 1 -- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 msgr2=0x7f9508004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95137fe640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 0x7f9508004c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9504009ab0 tx=0x7f950401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6db74f7084df356 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95040b2070 con 0x7f9508004810 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: dump: 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95040a56b0 con 0x7f9508004810 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 1 -- 172.21.15.86:0/538034899 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95040096b0 con 0x7f9508004810 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 4 set_mon_vals no callback set 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f951b7fe640 10 monclient: dump: 2024-09-17T15:44:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 10 monclient: shutdown 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 -- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 msgr2=0x7f9508004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 0x7f9508004c90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9504009ab0 tx=0x7f950401ca30 comp rx=0 tx=0).stop 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 -- 172.21.15.86:0/538034899 shutdown_connections 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 --2- 172.21.15.86:0/538034899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 -- 172.21.15.86:0/538034899 >> 172.21.15.86:0/538034899 conn(0x7f951c0a2f40 msgr2=0x7f951c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 -- 172.21.15.86:0/538034899 shutdown_connections 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 1 -- 172.21.15.86:0/538034899 wait complete. 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.250+0000 7f95225eb640 10 monclient: monmap: 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 Processor -- start 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- start start 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: init 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508004810 0x7f951c1d0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-c con 0x7f9508004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 0x7f951c134320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-a con 0x7f951c133ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f951c126ad0 0x7f951c126f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): picked mon.noname-b con 0x7f951c126ad0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f951c0a1000 con 0x7f951c133ea0 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f951c0a0e80 con 0x7f951c126ad0 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f951c0a1180 con 0x7f9508004810 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient(hunting): authenticate will time out at 1994.220598s 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508004810 0x7f951c1d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 0x7f951c134320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 0x7f951c134320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34512/0 (socket says 172.21.15.86:34512) 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 -- 172.21.15.86:0/1358482384 learned_addr learned my addr 172.21.15.86:0/1358482384 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): get_auth_request con 0x7f9508004810 auth_method 0 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): get_auth_request con 0x7f951c133ea0 auth_method 0 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient(hunting): handle_auth_done global_id 184236 payload 824 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f9520b88640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f951c126ad0 0x7f951c126f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508004810 msgr2=0x7f951c1d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508004810 0x7f951c1d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f951c126ad0 msgr2=0x7f951c126f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f951c126ad0 0x7f951c126f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 monclient: found mon.noname-a 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f951c128590 con 0x7f951c133ea0 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f9520b88640 10 monclient: get_auth_request con 0x7f951c126ad0 auth_method 0 2024-09-17T15:44:21.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951bfff640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508004810 0x7f951c1d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f9520b88640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f951c126ad0 0x7f951c126f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 msgr2=0x7f951c134320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95137fe640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 0x7f951c134320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f950401c860 tx=0x7f95040197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95040b2040 con 0x7f951c133ea0 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: dump: 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.266 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95040a5e00 con 0x7f951c133ea0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9504004260 con 0x7f951c133ea0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 5 monclient: authenticate success, global_id 184236 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _renew_subs 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f951c134860 con 0x7f951c133ea0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95197fa640 10 monclient: dump: 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f951affd640 4 set_mon_vals no callback set 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _renew_subs 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.254+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c12a050 con 0x7f951c133ea0 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95225eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95225eb640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 0x7f951c127460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f951c09d460 con 0x7f951c02e5d0 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 0x7f951c127460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient: get_auth_request con 0x7f951c02e5d0 auth_method 0 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f95040bf020 con 0x7f951c133ea0 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94fc03bbb0 0x7f94fc03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 10 monclient(hunting): handle_auth_done global_id 184239 payload 824 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(611..611 src has 438..611) ==== 4815+0+0 (secure 0 0 0) 0x7f95040e9fa0 con 0x7f951c133ea0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f951c02e9b0 con 0x7f951c02e5d0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f9520b88640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94fc03bbb0 0x7f94fc03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f9520b88640 10 monclient: get_auth_request con 0x7f94fc03bbb0 auth_method 0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 msgr2=0x7f951c127460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f951bfff640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 0x7f951c127460 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f951c06b120 tx=0x7f950c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c018020 con 0x7f951c02e5d0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 10 monclient: dump: 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f950c010d90 con 0x7f951c02e5d0 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 msgr2=0x7f951c127460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 0x7f951c127460 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f951c06b120 tx=0x7f950c007700 comp rx=0 tx=0).stop 2024-09-17T15:44:21.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 msgr2=0x7f951c127460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f95197fa640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c02e5d0 0x7f951c127460 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f951c06b120 tx=0x7f950c007700 comp rx=0 tx=0).stop 2024-09-17T15:44:21.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.258+0000 7f9520b88640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94fc03bbb0 0x7f94fc03e070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f951c0a2c60 tx=0x7f9514006040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f95225eb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:21.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f95225eb640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 0x7f951c064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f95225eb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f951c136b70 con 0x7f951c063e00 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 0x7f951c064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient: get_auth_request con 0x7f951c063e00 auth_method 0 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 10 monclient(hunting): handle_auth_done global_id 184242 payload 824 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.266+0000 7f951bfff640 1 -- 172.21.15.86:0/1358482384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f951c11efe0 con 0x7f951c063e00 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f951bfff640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 msgr2=0x7f951c064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f951bfff640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 0x7f951c064210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f950c019000 tx=0x7f950c0014c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950c018070 con 0x7f951c063e00 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 10 monclient: dump: 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f950c014b80 con 0x7f951c063e00 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 msgr2=0x7f951c064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 0x7f951c064210 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f950c019000 tx=0x7f950c0014c0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 msgr2=0x7f951c064210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95197fa640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c063e00 0x7f951c064210 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f950c019000 tx=0x7f950c0014c0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:21.276 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 64763, 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.277 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.278 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:21.279 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:21.280 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.281 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:21.282 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.283 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:21.284 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.285 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.288 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:21.289 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94fc03bbb0 msgr2=0x7f94fc03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f94fc03bbb0 0x7f94fc03e070 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f951c0a2c60 tx=0x7f9514006040 comp rx=0 tx=0).stop 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 10 monclient: shutdown 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 msgr2=0x7f951c134320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 --2- 172.21.15.86:0/1358482384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f951c133ea0 0x7f951c134320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f950401c860 tx=0x7f95040197d0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f9520b88640 1 -- 172.21.15.86:0/1358482384 reap_dead start 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f9520b88640 1 -- 172.21.15.86:0/1358482384 reap_dead start 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 shutdown_connections 2024-09-17T15:44:21.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 >> 172.21.15.86:0/1358482384 conn(0x7f951c0a2f40 msgr2=0x7f951c070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 shutdown_connections 2024-09-17T15:44:21.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.270+0000 7f95225eb640 1 -- 172.21.15.86:0/1358482384 wait complete. 2024-09-17T15:44:21.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:21.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:44:21.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:44:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.410+0000 7fd6a55cb640 10 monclient: get_monmap_and_config 2024-09-17T15:44:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.410+0000 7fd6a55cb640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.410+0000 7fd6a55cb640 10 monclient: monmap: 2024-09-17T15:44:21.409 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:21.415660+0000 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:21.415660+0000 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:21.410 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 Processor -- start 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 -- start start 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient: init 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6a00a50e0 0x7fd6a009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a00a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a00a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd6a013d2f0 0x7fd6a013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a013d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a00a6280 con 0x7fd6a00a54c0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd6a009d2a0 con 0x7fd6a00a50e0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd6a009c0a0 con 0x7fd6a013d2f0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd6a55cb640 10 monclient(hunting): authenticate will time out at 1994.381139s 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd6a00a50e0 0x7fd6a009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69e7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd6a00a54c0 0x7fd6a00a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58310/0 (socket says 172.21.15.86:58310) 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69e7fc640 1 -- 172.21.15.86:0/4271776977 learned_addr learned my addr 172.21.15.86:0/4271776977 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:21.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): get_auth_request con 0x7fd6a00a50e0 auth_method 0 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69f7fe640 1 --2- 172.21.15.86:0/4271776977 >> v2:172.21.15.203:3300/0 conn(0x7fd6a013d2f0 0x7fd6a013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69f7fe640 10 monclient(hunting): get_auth_request con 0x7fd6a013d2f0 auth_method 0 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient(hunting): handle_auth_done global_id 184245 payload 274 2024-09-17T15:44:21.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 -- 172.21.15.86:0/4271776977 >> v2:172.21.15.203:3300/0 conn(0x7fd6a013d2f0 msgr2=0x7fd6a013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 --2- 172.21.15.86:0/4271776977 >> v2:172.21.15.203:3300/0 conn(0x7fd6a013d2f0 0x7fd6a013f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 -- 172.21.15.86:0/4271776977 >> v1:172.21.15.86:6789/0 conn(0x7fd6a00a54c0 legacy=0x7fd6a00a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 monclient: found mon.noname-a 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.414+0000 7fd69effd640 1 -- 172.21.15.86:0/4271776977 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a009c380 con 0x7fd6a00a50e0 2024-09-17T15:44:21.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69effd640 1 -- 172.21.15.86:0/4271776977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 msgr2=0x7fd6a009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69effd640 1 --2- 172.21.15.86:0/4271776977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a009cd60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a84e0 tx=0x7fd68801c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bcafcd6082bbf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 1 -- 172.21.15.86:0/4271776977 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880b0070 con 0x7fd6a00a50e0 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: dump: 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.417 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 5 monclient: authenticate success, global_id 184245 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 1 -- 172.21.15.86:0/4271776977 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6880ab600 con 0x7fd6a00a50e0 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 1 -- 172.21.15.86:0/4271776977 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880ab9d0 con 0x7fd6a00a50e0 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd69dffb640 10 monclient: dump: 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 4 set_mon_vals no callback set 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 10 monclient: shutdown 2024-09-17T15:44:21.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- 172.21.15.86:0/4271776977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 msgr2=0x7fd6a009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/4271776977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a009cd60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a84e0 tx=0x7fd68801c310 comp rx=0 tx=0).stop 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- 172.21.15.86:0/4271776977 shutdown_connections 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/4271776977 >> v2:172.21.15.203:3300/0 conn(0x7fd6a013d2f0 0x7fd6a013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/4271776977 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a009cd60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- 172.21.15.86:0/4271776977 >> 172.21.15.86:0/4271776977 conn(0x7fd6a00aaae0 msgr2=0x7fd6a00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- 172.21.15.86:0/4271776977 shutdown_connections 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- 172.21.15.86:0/4271776977 wait complete. 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 10 monclient: monmap: 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:21.420 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:21.421 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 Processor -- start 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 1 -- start start 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.418+0000 7fd6a55cb640 10 monclient: init 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a01cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-a con 0x7fd6a00a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a00a54c0 0x7fd6a01d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a00a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a013d2f0 0x7fd6a01d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a0142650 con 0x7fd6a00a50e0 2024-09-17T15:44:21.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a00a6ab0 con 0x7fd6a00a54c0 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a01427d0 con 0x7fd6a013d2f0 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): authenticate will time out at 1994.388303s 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a01cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a01cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34540/0 (socket says 172.21.15.86:34540) 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 -- 172.21.15.86:0/2161587187 learned_addr learned my addr 172.21.15.86:0/2161587187 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a013d2f0 0x7fd6a01d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): get_auth_request con 0x7fd6a00a50e0 auth_method 0 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): get_auth_request con 0x7fd6a013d2f0 auth_method 0 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69e7fc640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a00a54c0 0x7fd6a01d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69e7fc640 10 monclient(hunting): get_auth_request con 0x7fd6a00a54c0 auth_method 0 2024-09-17T15:44:21.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient(hunting): handle_auth_done global_id 184248 payload 824 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a013d2f0 msgr2=0x7fd6a01d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a013d2f0 0x7fd6a01d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a00a54c0 msgr2=0x7fd6a01d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd6a00a54c0 0x7fd6a01d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 monclient: found mon.noname-a 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a00a1560 con 0x7fd6a00a50e0 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 msgr2=0x7fd6a01cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69effd640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a01cfc10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ae040 tx=0x7fd68801ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880b0050 con 0x7fd6a00a50e0 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: dump: 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6880bc070 con 0x7fd6a00a50e0 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 5 monclient: authenticate success, global_id 184248 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _renew_subs 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a00a3600 con 0x7fd6a00a50e0 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880b5a80 con 0x7fd6a00a50e0 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6977fe640 10 monclient: dump: 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd69d7fa640 4 set_mon_vals no callback set 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _renew_subs 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a009fb40 con 0x7fd6a00a50e0 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 0x7fd6a0132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.422+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a006a0e0 con 0x7fd6a0131e40 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 0x7fd6a0132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient: get_auth_request con 0x7fd6a0131e40 auth_method 0 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd6880b5c20 con 0x7fd6a00a50e0 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_done global_id 164447 payload 824 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6a009cd60 con 0x7fd6a0131e40 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68003bbe0 0x7fd68003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(611..611 src has 438..611) ==== 4815+0+0 (secure 0 0 0) 0x7fd6880ea270 con 0x7fd6a00a50e0 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69f7fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68003bbe0 0x7fd68003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69f7fe640 10 monclient: get_auth_request con 0x7fd68003bbe0 auth_method 0 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 msgr2=0x7fd6a0132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69e7fc640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 0x7fd6a0132220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a20e0 tx=0x7fd69000e960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690004e40 con 0x7fd6a0131e40 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 10 monclient: dump: 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd690018070 con 0x7fd6a0131e40 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 msgr2=0x7fd6a0132220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 0x7fd6a0132220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a20e0 tx=0x7fd69000e960 comp rx=0 tx=0).stop 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 msgr2=0x7fd6a0132220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd6977fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0131e40 0x7fd6a0132220 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a20e0 tx=0x7fd69000e960 comp rx=0 tx=0).stop 2024-09-17T15:44:21.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.426+0000 7fd69f7fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68003bbe0 0x7fd68003e0a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a4b90 tx=0x7fd698009820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6a55cb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 0x7fd6a0136300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6a55cb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a0074be0 con 0x7fd6a0132760 2024-09-17T15:44:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 0x7fd6a0136300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient: get_auth_request con 0x7fd6a0132760 auth_method 0 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 10 monclient(hunting): handle_auth_done global_id 164450 payload 824 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 1 -- 172.21.15.86:0/2161587187 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6a00a23c0 con 0x7fd6a0132760 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 msgr2=0x7fd6a0136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd69e7fc640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 0x7fd6a0136300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6a006c9a0 tx=0x7fd690004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6900066f0 con 0x7fd6a0132760 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.434+0000 7fd6977fe640 10 monclient: dump: 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fd6900066f0 con 0x7fd6a0132760 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 msgr2=0x7fd6a0136300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 0x7fd6a0136300 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6a006c9a0 tx=0x7fd690004950 comp rx=0 tx=0).stop 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 msgr2=0x7fd6a0136300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6977fe640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6a0132760 0x7fd6a0136300 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd6a006c9a0 tx=0x7fd690004950 comp rx=0 tx=0).stop 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:21.441 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 164622, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.442 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.443 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.444 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:21.445 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.446 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:21.447 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.448 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.449 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.450 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.452 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.453 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.454 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:21.455 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:21.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68003bbe0 msgr2=0x7fd68003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd68003bbe0 0x7fd68003e0a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a4b90 tx=0x7fd698009820 comp rx=0 tx=0).stop 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 10 monclient: shutdown 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 msgr2=0x7fd6a01cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 --2- 172.21.15.86:0/2161587187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6a00a50e0 0x7fd6a01cfc10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd6880ae040 tx=0x7fd68801ab50 comp rx=0 tx=0).stop 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd69f7fe640 1 -- 172.21.15.86:0/2161587187 reap_dead start 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd69f7fe640 1 -- 172.21.15.86:0/2161587187 reap_dead start 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 shutdown_connections 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 >> 172.21.15.86:0/2161587187 conn(0x7fd6a00aaae0 msgr2=0x7fd6a013c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 shutdown_connections 2024-09-17T15:44:21.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.438+0000 7fd6a55cb640 1 -- 172.21.15.86:0/2161587187 wait complete. 2024-09-17T15:44:21.457 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:21.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:44:21.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837404548}) 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3999726969}) 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837404548}) 2024-09-17T15:44:21.525 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.526+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483066663}) 2024-09-17T15:44:21.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612223828}) 2024-09-17T15:44:21.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351804385}) 2024-09-17T15:44:21.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612223828}) 2024-09-17T15:44:21.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.534+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014503109}) 2024-09-17T15:44:21.535 INFO:tasks.rados.rados.0.smithi203.stdout:2740: done (3 left) 2024-09-17T15:44:21.536 INFO:tasks.rados.rados.0.smithi203.stdout:2741: write (excl) oid 13 current snap is 294 2024-09-17T15:44:21.536 INFO:tasks.rados.rados.0.smithi203.stdout:2741: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-17T15:44:21.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465695897}) 2024-09-17T15:44:21.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429993373}) 2024-09-17T15:44:21.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465695897}) 2024-09-17T15:44:21.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.558+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687723147}) 2024-09-17T15:44:21.558 INFO:tasks.rados.rados.0.smithi203.stdout:2741: writing smithi20328086-13 from 685983 to 1179095 tid 1 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2741: writing smithi20328086-13 from 1870228 to 2285983 tid 2 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2742: read oid 17 snap -1 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2742: expect (ObjNum 921 snap 277 seq_num 921) 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2743: read oid 47 snap -1 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2743: expect (ObjNum 952 snap 288 seq_num 952) 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2744: write (excl) oid 33 current snap is 294 2024-09-17T15:44:21.576 INFO:tasks.rados.rados.0.smithi203.stdout:2744: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-17T15:44:21.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.590+0000 7fabfc444640 10 monclient: get_monmap_and_config 2024-09-17T15:44:21.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.590+0000 7fabfc444640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.590+0000 7fabfc444640 10 monclient: monmap: 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:21.594994+0000 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:21.594994+0000 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:21.589 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:21.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.590+0000 7fabfc444640 1 Processor -- start 2024-09-17T15:44:21.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 -- start start 2024-09-17T15:44:21.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient: init 2024-09-17T15:44:21.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-a con 0x7fabf4134b10 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf40a2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-b con 0x7fabf4134650 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fabf40a2b60 0x7fabf40a3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-c con 0x7fabf40a2b60 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fabf40a0010 con 0x7fabf4134b10 2024-09-17T15:44:21.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fabf40a1010 con 0x7fabf40a2b60 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fabf406ffd0 con 0x7fabf4134650 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfc444640 10 monclient(hunting): authenticate will time out at 1994.560729s 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34556/0 (socket says 172.21.15.86:34556) 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 -- 172.21.15.86:0/3148414784 learned_addr learned my addr 172.21.15.86:0/3148414784 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request con 0x7fabf4134b10 auth_method 0 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 --2- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf40a2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): get_auth_request con 0x7fabf4134650 auth_method 0 2024-09-17T15:44:21.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 -- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 msgr2=0x7fabf4134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 -- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 msgr2=0x7fabf4134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa9e1640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.203:3300/0 conn(0x7fabf40a2b60 0x7fabf40a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa9e1640 10 monclient(hunting): get_auth_request con 0x7fabf40a2b60 auth_method 0 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa9e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa9e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabfa9e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient(hunting): handle_auth_done global_id 164453 payload 274 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 -- 172.21.15.86:0/3148414784 >> v2:172.21.15.203:3300/0 conn(0x7fabf40a2b60 msgr2=0x7fabf40a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.203:3300/0 conn(0x7fabf40a2b60 0x7fabf40a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 -- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 msgr2=0x7fabf4134ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 monclient: found mon.noname-b 2024-09-17T15:44:21.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:21.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.594+0000 7fabf99df640 1 -- 172.21.15.86:0/3148414784 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf40702b0 con 0x7fabf4134650 2024-09-17T15:44:21.598 INFO:tasks.rados.rados.0.smithi203.stdout:2744: writing smithi20328086-33 from 702790 to 1271151 tid 1 2024-09-17T15:44:21.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf99df640 1 -- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 msgr2=0x7fabf40a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf99df640 1 --2- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf40a2620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabf4074820 tx=0x7fabe401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511636654e43039d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 1 -- 172.21.15.86:0/3148414784 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40b0070 con 0x7fabf4134650 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: dump: 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 5 monclient: authenticate success, global_id 164453 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 1 -- 172.21.15.86:0/3148414784 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe40ab600 con 0x7fabf4134650 2024-09-17T15:44:21.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 4 set_mon_vals no callback set 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 1 -- 172.21.15.86:0/3148414784 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40ab9d0 con 0x7fabf4134650 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabf91de640 10 monclient: dump: 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.606 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 10 monclient: shutdown 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 msgr2=0x7fabf40a2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 --2- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf40a2620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fabf4074820 tx=0x7fabe401c340 comp rx=0 tx=0).stop 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- 172.21.15.86:0/3148414784 shutdown_connections 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.203:3300/0 conn(0x7fabf40a2b60 0x7fabf40a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 --2- 172.21.15.86:0/3148414784 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf40a2620 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 --2- 172.21.15.86:0/3148414784 >> v2:172.21.15.86:3300/0 conn(0x7fabf4134b10 0x7fabf4134ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- 172.21.15.86:0/3148414784 >> 172.21.15.86:0/3148414784 conn(0x7fabf40a7210 msgr2=0x7fabf40a3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- 172.21.15.86:0/3148414784 shutdown_connections 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- 172.21.15.86:0/3148414784 wait complete. 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 10 monclient: monmap: 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:21.609 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 Processor -- start 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 1 -- start start 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.598+0000 7fabfc444640 10 monclient: init 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf40a2b60 0x7fabf41cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-b con 0x7fabf40a2b60 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf4132f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-c con 0x7fabf4134650 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabf4134b10 0x7fabf4133450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): picked mon.noname-a con 0x7fabf4134b10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:21.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fabf4141700 con 0x7fabf4134b10 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fabf40a04f0 con 0x7fabf40a2b60 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fabf4141880 con 0x7fabf4134650 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient(hunting): authenticate will time out at 1994.568205s 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf4132f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf4132f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44240/0 (socket says 172.21.15.86:44240) 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabf4134b10 0x7fabf4133450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 -- 172.21.15.86:0/1625033746 learned_addr learned my addr 172.21.15.86:0/1625033746 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): get_auth_request con 0x7fabf4134650 auth_method 0 2024-09-17T15:44:21.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): get_auth_request con 0x7fabf4134b10 auth_method 0 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf40a2b60 0x7fabf41cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa9e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request con 0x7fabf40a2b60 auth_method 0 2024-09-17T15:44:21.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa1e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfa1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient(hunting): handle_auth_done global_id 164456 payload 824 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf40a2b60 msgr2=0x7fabf41cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf40a2b60 0x7fabf41cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabf4134b10 msgr2=0x7fabf4133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabf4134b10 0x7fabf4133450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 monclient: found mon.noname-c 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf411e9b0 con 0x7fabf4134650 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 msgr2=0x7fabf4132f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf99df640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf4132f10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fabe40ae040 tx=0x7fabe40190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40b8020 con 0x7fabf4134650 2024-09-17T15:44:21.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: dump: 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.616 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe40b4620 con 0x7fabf4134650 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe40b0040 con 0x7fabf4134650 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 5 monclient: authenticate success, global_id 164456 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _renew_subs 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf4133c70 con 0x7fabf4134650 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabf89dd640 4 set_mon_vals no callback set 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabeaffd640 10 monclient: dump: 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.618 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _renew_subs 2024-09-17T15:44:21.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.602+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf413dfe0 con 0x7fabf4134650 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfc444640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfc444640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 0x7fabf413c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fabf40a2620 con 0x7fabf413c520 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fabe40bb020 con 0x7fabf4134650 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fabd803bb90 0x7fabd803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7fabe40e9bf0 con 0x7fabf4134650 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 0x7fabf413c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient: get_auth_request con 0x7fabf413c520 auth_method 0 2024-09-17T15:44:21.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa9e1640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fabd803bb90 0x7fabd803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa9e1640 10 monclient: get_auth_request con 0x7fabd803bb90 auth_method 0 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_done global_id 194122 payload 824 2024-09-17T15:44:21.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fabf411d3e0 con 0x7fabf413c520 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2744: writing smithi20328086-33 from 1812357 to 2302790 tid 2 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2745: rollback oid 21 current snap is 294 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 290 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2737: finishing write tid 2 to smithi20328086-2 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2737: finishing write tid 3 to smithi20328086-2 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2737: finishing write tid 4 to smithi20328086-2 2024-09-17T15:44:21.623 INFO:tasks.rados.rados.0.smithi203.stdout:2737: oid 2 updating version 0 to 1168 2024-09-17T15:44:21.624 INFO:tasks.rados.rados.0.smithi203.stdout:2737: oid 2 version 1168 is already newer than 1167 2024-09-17T15:44:21.624 INFO:tasks.rados.rados.0.smithi203.stdout:2737: oid 2 version 1168 is already newer than 1166 2024-09-17T15:44:21.624 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1168 (ObjNum 968 snap 294 seq_num 968) dirty exists 2024-09-17T15:44:21.624 INFO:tasks.rados.rados.0.smithi203.stdout:2737: left oid 2 (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:44:21.624 INFO:tasks.rados.rados.0.smithi203.stdout:2738: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:44:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 msgr2=0x7fabf413c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 0x7fabf413c900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf0011a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabfa9e1640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fabd803bb90 0x7fabd803e050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fabf413e2f0 tx=0x7fabec009780 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0015a80 con 0x7fabf413c520 2024-09-17T15:44:21.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.606+0000 7fabeaffd640 10 monclient: dump: 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.626 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fabf0015a80 con 0x7fabf413c520 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 msgr2=0x7fabf413c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 0x7fabf413c900 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf0011a20 comp rx=0 tx=0).stop 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 msgr2=0x7fabf413c900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.610+0000 7fabeaffd640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413c520 0x7fabf413c900 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf0011a20 comp rx=0 tx=0).stop 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.614+0000 7fabfc444640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:21.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.614+0000 7fabfc444640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 0x7fabf4137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.614+0000 7fabfc444640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.614+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fabf406eee0 con 0x7fabf413ce40 2024-09-17T15:44:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 0x7fabf4137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient: get_auth_request con 0x7fabf413ce40 auth_method 0 2024-09-17T15:44:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 10 monclient(hunting): handle_auth_done global_id 194125 payload 824 2024-09-17T15:44:21.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 1 -- 172.21.15.86:0/1625033746 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fabf406ddc0 con 0x7fabf413ce40 2024-09-17T15:44:21.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 msgr2=0x7fabf4137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabfa1e0640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 0x7fabf4137a70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.631 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076765140}) 2024-09-17T15:44:21.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557326523}) 2024-09-17T15:44:21.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076765140}) 2024-09-17T15:44:21.632 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903724387}) 2024-09-17T15:44:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf0006450 con 0x7fabf413ce40 2024-09-17T15:44:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:44:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 10 monclient: dump: 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fabf0006450 con 0x7fabf413ce40 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 msgr2=0x7fabf4137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.618+0000 7fabeaffd640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 0x7fabf4137a70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf00046d0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabeaffd640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 msgr2=0x7fabf4137a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabeaffd640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fabf413ce40 0x7fabf4137a70 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fabf0011560 tx=0x7fabf00046d0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:21.636 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 156, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 12959, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 7, 2024-09-17T15:44:21.637 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.638 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.639 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.640 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.641 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:21.642 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:21.643 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.644 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.646 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.647 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:21.648 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.649 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.650 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:21.651 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fabd803bb90 msgr2=0x7fabd803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fabd803bb90 0x7fabd803e050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fabf413e2f0 tx=0x7fabec009780 comp rx=0 tx=0).stop 2024-09-17T15:44:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 10 monclient: shutdown 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 msgr2=0x7fabf4132f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 --2- 172.21.15.86:0/1625033746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabf4134650 0x7fabf4132f10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fabe40ae040 tx=0x7fabe40190e0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfa9e1640 1 -- 172.21.15.86:0/1625033746 reap_dead start 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfa9e1640 1 -- 172.21.15.86:0/1625033746 reap_dead start 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 shutdown_connections 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 >> 172.21.15.86:0/1625033746 conn(0x7fabf40a7210 msgr2=0x7fabf4073b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 shutdown_connections 2024-09-17T15:44:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.622+0000 7fabfc444640 1 -- 172.21.15.86:0/1625033746 wait complete. 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879248497}) 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549663389}) 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879248497}) 2024-09-17T15:44:21.654 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.634+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3624994}) 2024-09-17T15:44:21.655 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:44:21.655 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:44:21.655 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:44:21.656 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:44:21.656 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:44:21.656 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:44:21.656 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:44:21.669 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:44:21.669 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:44:21.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:21.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099997861}) 2024-09-17T15:44:21.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1352266055}) 2024-09-17T15:44:21.714 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:44:21.715 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099997861}) 2024-09-17T15:44:21.715 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:44:21.718+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125169488}) 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: get_monmap_and_config 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: monmap: 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:21.789144+0000 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:21.789144+0000 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:21.783 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:21.784 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:21.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 Processor -- start 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 -- start start 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: init 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1fe00689d0 0x7f1fe006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fe0135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe009c8b0 con 0x7f1fe0135ae0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe0074500 con 0x7f1fe00689d0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe009cbe0 con 0x7f1fe0068db0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fe48d8640 10 monclient(hunting): authenticate will time out at 1994.754707s 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fded9d640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f1fe0135ae0 legacy=0x7f1fe0135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1fe00689d0 0x7f1fe006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1fe00689d0 0x7f1fe006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34578/0 (socket says 172.21.15.86:34578) 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 -- 172.21.15.86:0/2779505973 learned_addr learned my addr 172.21.15.86:0/2779505973 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): get_auth_request con 0x7f1fe00689d0 auth_method 0 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): get_auth_request con 0x7f1fe0068db0 auth_method 0 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient(hunting): handle_auth_done global_id 184257 payload 274 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 -- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 msgr2=0x7f1fe00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe00691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 -- 172.21.15.86:0/2779505973 >> v1:172.21.15.203:6789/0 conn(0x7f1fe0135ae0 legacy=0x7f1fe0135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 monclient: found mon.noname-a 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fde59c640 1 -- 172.21.15.86:0/2779505973 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe009ce70 con 0x7f1fe00689d0 2024-09-17T15:44:21.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.786+0000 7f1fddd9b640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe00691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fde59c640 1 -- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 msgr2=0x7f1fe006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fde59c640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe006f840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009e50 tx=0x7f1fd801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0090ceb6479f84 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 1 -- 172.21.15.86:0/2779505973 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80a56f0 con 0x7f1fe00689d0 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: dump: 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fe48d8640 5 monclient: authenticate success, global_id 184257 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 1 -- 172.21.15.86:0/2779505973 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd80a5890 con 0x7f1fe00689d0 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fe48d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:21.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fe48d8640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fe48d8640 4 set_mon_vals no callback set 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 1 -- 172.21.15.86:0/2779505973 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80a5c60 con 0x7f1fe00689d0 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.790+0000 7f1fdd59a640 10 monclient: dump: 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: shutdown 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 msgr2=0x7f1fe006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe006f840 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8009e50 tx=0x7f1fd801c890 comp rx=0 tx=0).stop 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- 172.21.15.86:0/2779505973 shutdown_connections 2024-09-17T15:44:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe00691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/2779505973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe006f840 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- 172.21.15.86:0/2779505973 >> 172.21.15.86:0/2779505973 conn(0x7f1fe00a6440 msgr2=0x7f1fe00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- 172.21.15.86:0/2779505973 shutdown_connections 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- 172.21.15.86:0/2779505973 wait complete. 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: build_initial_monmap 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: monmap: 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 Processor -- start 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- start start 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: init 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe01353c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 0x7f1fe009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe0135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe01409b0 con 0x7f1fe00689d0 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe0140830 con 0x7f1fe0135ae0 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe0140b30 con 0x7f1fe0068db0 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fe48d8640 10 monclient(hunting): authenticate will time out at 1994.761625s 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe01353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34592/0 (socket says 172.21.15.86:34592) 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): get_auth_request con 0x7f1fe0068db0 auth_method 0 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 -- 172.21.15.86:0/3307766786 learned_addr learned my addr 172.21.15.86:0/3307766786 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): get_auth_request con 0x7f1fe00689d0 auth_method 0 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fded9d640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 msgr2=0x7f1fe009d770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:21.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fded9d640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 0x7f1fe009d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient(hunting): handle_auth_done global_id 184260 payload 824 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 msgr2=0x7f1fe01353c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe01353c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 msgr2=0x7f1fe009d770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 0x7f1fe009d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient: _finish_hunting 0 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 monclient: found mon.noname-a 2024-09-17T15:44:21.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fde59c640 1 -- 172.21.15.86:0/3307766786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe009ecb0 con 0x7f1fe00689d0 2024-09-17T15:44:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.794+0000 7f1fddd9b640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe01353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fde59c640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 msgr2=0x7f1fe0134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fde59c640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1fd801d130 tx=0x7f1fd8019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80bc020 con 0x7f1fe00689d0 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: dump: 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd80b8880 con 0x7f1fe00689d0 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 5 monclient: authenticate success, global_id 184260 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _renew_subs 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe0121550 con 0x7f1fe00689d0 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80a56f0 con 0x7f1fe00689d0 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 10 monclient: dump: 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _renew_subs 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe00a3950 con 0x7f1fe00689d0 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fdcd99640 4 set_mon_vals no callback set 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe009d460 con 0x7f1fe00689d0 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1fd80bf020 con 0x7f1fe00689d0 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 0x7f1fc003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7f1fd80eaf00 con 0x7f1fe00689d0 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fddd9b640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 0x7f1fc003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.798+0000 7f1fddd9b640 10 monclient: get_auth_request con 0x7f1fc003bb60 auth_method 0 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.802+0000 7f1fddd9b640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 0x7f1fc003e020 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0120df0 tx=0x7f1fc8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.802+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1fd8016600 con 0x7f1fe00689d0 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.802+0000 7f1fd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:21.802+0000 7f1fd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:21.824 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:21.914 INFO:tasks.rados.rados.0.smithi203.stdout:2741: finishing write tid 1 to smithi20328086-13 2024-09-17T15:44:21.914 INFO:tasks.rados.rados.0.smithi203.stdout:2741: finishing write tid 2 to smithi20328086-13 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: finishing write tid 3 to smithi20328086-13 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: finishing write tid 4 to smithi20328086-13 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: oid 13 updating version 0 to 1131 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: oid 13 version 1131 is already newer than 1130 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: oid 13 updating version 1131 to 1132 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1132 (ObjNum 969 snap 294 seq_num 969) dirty exists 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2741: left oid 13 (ObjNum 969 snap 294 seq_num 969) 2024-09-17T15:44:21.915 INFO:tasks.rados.rados.0.smithi203.stdout:2743: expect (ObjNum 952 snap 288 seq_num 952) 2024-09-17T15:44:21.958 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fe48d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fe48d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fe48d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1fe006f840 con 0x7f1fe00689d0 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fd6ffd640 1 -- 172.21.15.86:0/3307766786 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1fd80039a0 con 0x7f1fe00689d0 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:22.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fd6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:22.020 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.022+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 msgr2=0x7f1fc003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 0x7f1fc003e020 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0120df0 tx=0x7f1fc8009190 comp rx=0 tx=0).stop 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 10 monclient: shutdown 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 msgr2=0x7f1fe0134e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1fd801d130 tx=0x7f1fd8019690 comp rx=0 tx=0).stop 2024-09-17T15:44:22.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 shutdown_connections 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1fc003bb60 0x7f1fc003e020 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe0135ae0 0x7f1fe009d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1fe0068db0 0x7f1fe01353c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 --2- 172.21.15.86:0/3307766786 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1fe00689d0 0x7f1fe0134e80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 >> 172.21.15.86:0/3307766786 conn(0x7f1fe00a6440 msgr2=0x7f1fe013d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 shutdown_connections 2024-09-17T15:44:22.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:22.026+0000 7f1fe48d8640 1 -- 172.21.15.86:0/3307766786 wait complete. 2024-09-17T15:44:22.053 INFO:tasks.rados.rados.0.smithi203.stdout:2742: expect (ObjNum 921 snap 277 seq_num 921) 2024-09-17T15:44:22.135 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2306427 2024-09-17T15:44:22.179 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1284756 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:2744: finishing write tid 1 to smithi20328086-33 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:2745: finishing rollback tid 0 to smithi20328086-21 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:2745: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1477 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:2744: finishing write tid 2 to smithi20328086-33 2024-09-17T15:44:22.231 INFO:tasks.rados.rados.0.smithi203.stdout:2744: finishing write tid 3 to smithi20328086-33 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2744: finishing write tid 4 to smithi20328086-33 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2744: oid 33 updating version 0 to 1100 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2744: oid 33 version 1100 is already newer than 1099 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2744: oid 33 updating version 1100 to 1101 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1101 (ObjNum 970 snap 294 seq_num 970) dirty exists 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2744: left oid 33 (ObjNum 970 snap 294 seq_num 970) 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2737: done (7 left) 2024-09-17T15:44:22.232 INFO:tasks.rados.rados.0.smithi203.stdout:2738: done (6 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2739: done (5 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2741: done (4 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2742: done (3 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2743: done (2 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2744: done (1 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2745: done (0 left) 2024-09-17T15:44:22.233 INFO:tasks.rados.rados.0.smithi203.stdout:2746: delete oid 26 current snap is 294 2024-09-17T15:44:22.238 INFO:tasks.rados.rados.0.smithi203.stdout:2746: done (0 left) 2024-09-17T15:44:22.238 INFO:tasks.rados.rados.0.smithi203.stdout:2747: read oid 36 snap 290 2024-09-17T15:44:22.238 INFO:tasks.rados.rados.0.smithi203.stdout:2747: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:44:22.238 INFO:tasks.rados.rados.0.smithi203.stdout:2748: snap_remove snap 270 2024-09-17T15:44:25.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: get_monmap_and_config 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: build_initial_monmap 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: monmap: 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:25.173263+0000 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:25.173263+0000 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:25.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:25.168 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 Processor -- start 2024-09-17T15:44:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 -- start start 2024-09-17T15:44:25.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: init 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83a00689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-b con 0x7f83a00694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 0x7f83a013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-a con 0x7f83a013aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a0074470 con 0x7f83a00689d0 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009c8b0 con 0x7f83a013aff0 2024-09-17T15:44:25.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83a009cb90 con 0x7f83a00694c0 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a6c24640 10 monclient(hunting): authenticate will time out at 1998.138895s 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a49c0640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f83a00689d0 legacy=0x7f83a0068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 0x7f83a013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44266/0 (socket says 172.21.15.86:44266) 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 -- 172.21.15.86:0/3075851735 learned_addr learned my addr 172.21.15.86:0/3075851735 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): get_auth_request con 0x7f83a00694c0 auth_method 0 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839f7fe640 1 -- 172.21.15.86:0/3075851735 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): get_auth_request con 0x7f83a013aff0 auth_method 0 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:25.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient(hunting): handle_auth_done global_id 164465 payload 274 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 -- 172.21.15.86:0/3075851735 >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 msgr2=0x7f83a013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 --2- 172.21.15.86:0/3075851735 >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 0x7f83a013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 -- 172.21.15.86:0/3075851735 >> v1:172.21.15.203:6789/0 conn(0x7f83a00689d0 legacy=0x7f83a0068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 monclient: found mon.noname-b 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:25.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f839ffff640 1 -- 172.21.15.86:0/3075851735 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a009ce70 con 0x7f83a00694c0 2024-09-17T15:44:25.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.170+0000 7f83a51c1640 1 --2- 172.21.15.86:0/3075851735 >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 0x7f83a013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839ffff640 1 -- 172.21.15.86:0/3075851735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 msgr2=0x7f83a006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839ffff640 1 --2- 172.21.15.86:0/3075851735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c680126a6a752e63 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 1 -- 172.21.15.86:0/3075851735 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83900b1070 con 0x7f83a00694c0 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: dump: 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.178 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 1 -- 172.21.15.86:0/3075851735 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83900ac620 con 0x7f83a00694c0 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f83a6c24640 5 monclient: authenticate success, global_id 164465 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f83a6c24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:25.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f83a6c24640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 1 -- 172.21.15.86:0/3075851735 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a009c2d0 con 0x7f83a00694c0 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f83a6c24640 4 set_mon_vals no callback set 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.174+0000 7f839f7fe640 10 monclient: dump: 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: shutdown 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- 172.21.15.86:0/3075851735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 msgr2=0x7f83a006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- 172.21.15.86:0/3075851735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8390009870 tx=0x7f839001c420 comp rx=0 tx=0).stop 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- 172.21.15.86:0/3075851735 shutdown_connections 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- 172.21.15.86:0/3075851735 >> v2:172.21.15.86:3300/0 conn(0x7f83a013aff0 0x7f83a013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- 172.21.15.86:0/3075851735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a006f840 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- 172.21.15.86:0/3075851735 >> 172.21.15.86:0/3075851735 conn(0x7f83a00a6440 msgr2=0x7f83a00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- 172.21.15.86:0/3075851735 shutdown_connections 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- 172.21.15.86:0/3075851735 wait complete. 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: build_initial_monmap 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: monmap: 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:25.182 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 Processor -- start 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- start start 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: init 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-a con 0x7f83a00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a0135a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-c con 0x7f83a00694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 0x7f83a011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): picked mon.noname-b con 0x7f83a013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:25.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83a0140610 con 0x7f83a00689d0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83a0140490 con 0x7f83a013aff0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83a0140790 con 0x7f83a00694c0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a6c24640 10 monclient(hunting): authenticate will time out at 1998.146567s 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a0135a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a51c1640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 msgr2=0x7f83a011c8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34610/0 (socket says 172.21.15.86:34610) 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a0135a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44272/0 (socket says 172.21.15.86:44272) 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 -- 172.21.15.86:0/4181313658 learned_addr learned my addr 172.21.15.86:0/4181313658 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:25.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): get_auth_request con 0x7f83a00689d0 auth_method 0 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a51c1640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 0x7f83a011c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): get_auth_request con 0x7f83a00694c0 auth_method 0 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:25.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient(hunting): handle_auth_done global_id 184266 payload 824 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 -- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 msgr2=0x7f83a0135a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a0135a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 -- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 msgr2=0x7f83a011c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 0x7f83a011c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient: _finish_hunting 0 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 monclient: found mon.noname-a 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:25.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.178+0000 7f83a49c0640 1 -- 172.21.15.86:0/4181313658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a009d460 con 0x7f83a00689d0 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a49c0640 1 -- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 msgr2=0x7f83a011ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a49c0640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f839400b530 tx=0x7f839400b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394013020 con 0x7f83a00689d0 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: dump: 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:25.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 5 monclient: authenticate success, global_id 184266 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8394005070 con 0x7f83a00689d0 2024-09-17T15:44:25.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _renew_subs 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394005460 con 0x7f83a00689d0 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a01366d0 con 0x7f83a00689d0 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839effd640 4 set_mon_vals no callback set 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 10 monclient: dump: 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:25.191 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _renew_subs 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a011e0a0 con 0x7f83a00689d0 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a006f840 con 0x7f83a00689d0 2024-09-17T15:44:25.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8394016020 con 0x7f83a00689d0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 0x7f838c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7f839404fbc0 con 0x7f83a00689d0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839ffff640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 0x7f838c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.182+0000 7f839ffff640 10 monclient: get_auth_request con 0x7f838c03bb10 auth_method 0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.186+0000 7f839ffff640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 0x7f838c03dfd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83900ac000 tx=0x7f83900a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.186+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f839401d020 con 0x7f83a00689d0 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.186+0000 7f839d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:25.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.186+0000 7f839d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:25.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f83a6c24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:25.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f83a6c24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:25.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f83a6c24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:25.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f83a006d4f0 con 0x7f83a00689d0 2024-09-17T15:44:25.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f839d7fa640 1 -- 172.21.15.86:0/4181313658 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f8394018c50 con 0x7f83a00689d0 2024-09-17T15:44:25.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f839d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:25.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.414+0000 7f839d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:25.415 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:25.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 msgr2=0x7f838c03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 0x7f838c03dfd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f83900ac000 tx=0x7f83900a7000 comp rx=0 tx=0).stop 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 10 monclient: shutdown 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 msgr2=0x7f83a011ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f839400b530 tx=0x7f839400b9f0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 shutdown_connections 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f838c03bb10 0x7f838c03dfd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83a013aff0 0x7f83a011c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83a00694c0 0x7f83a0135a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 --2- 172.21.15.86:0/4181313658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83a00689d0 0x7f83a011ebc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 >> 172.21.15.86:0/4181313658 conn(0x7f83a00a6440 msgr2=0x7f83a00a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 shutdown_connections 2024-09-17T15:44:25.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:25.418+0000 7f83a6c24640 1 -- 172.21.15.86:0/4181313658 wait complete. 2024-09-17T15:44:28.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: get_monmap_and_config 2024-09-17T15:44:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: build_initial_monmap 2024-09-17T15:44:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: monmap: 2024-09-17T15:44:28.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:28.565841+0000 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:28.565841+0000 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:28.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 Processor -- start 2024-09-17T15:44:28.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 -- start start 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: init 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 0x7f93e409cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-c con 0x7f93e40a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f93e40a5450 0x7f93e40a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-a con 0x7f93e40a5450 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93e4137e30 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93e40a6f40 con 0x7f93e4137e30 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:28.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f93e40a65e0 con 0x7f93e40a5450 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f93e409c2a0 con 0x7f93e40a5070 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.562+0000 7f93ebe76640 10 monclient(hunting): authenticate will time out at 2001.531846s 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9c12640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 msgr2=0x7f93e409cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9c12640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 0x7f93e409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93ea413640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f93e4137e30 0x7f93e4138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43494/0 (socket says 172.21.15.86:43494) 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f93e40a5450 0x7f93e40a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93ea413640 1 -- 172.21.15.86:0/3566275850 learned_addr learned my addr 172.21.15.86:0/3566275850 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): get_auth_request con 0x7f93e40a5450 auth_method 0 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:28.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient(hunting): handle_auth_done global_id 184269 payload 274 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 -- 172.21.15.86:0/3566275850 >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 msgr2=0x7f93e409cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 --2- 172.21.15.86:0/3566275850 >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 0x7f93e409cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 -- 172.21.15.86:0/3566275850 >> v1:172.21.15.86:6789/0 conn(0x7f93e4137e30 legacy=0x7f93e4138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient: _finish_hunting 0 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 monclient: found mon.noname-a 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.566+0000 7f93e9411640 1 -- 172.21.15.86:0/3566275850 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e409c580 con 0x7f93e40a5450 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 -- 172.21.15.86:0/3566275850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 msgr2=0x7f93e40a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 --2- 172.21.15.86:0/3566275850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e40a5860 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f93e406e200 tx=0x7f93d401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9d51e55a6591ad server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 1 -- 172.21.15.86:0/3566275850 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40b0070 con 0x7f93e40a5450 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: dump: 2024-09-17T15:44:28.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: _finish_auth 0 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: _check_auth_tickets 2024-09-17T15:44:28.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 1 -- 172.21.15.86:0/3566275850 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d40ab600 con 0x7f93e40a5450 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 5 monclient: authenticate success, global_id 184269 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 1 -- 172.21.15.86:0/3566275850 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40ab9d0 con 0x7f93e40a5450 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 4 set_mon_vals no callback set 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e8c10640 10 monclient: dump: 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:28.569 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: shutdown 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- 172.21.15.86:0/3566275850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 msgr2=0x7f93e40a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- 172.21.15.86:0/3566275850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e40a5860 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f93e406e200 tx=0x7f93d401c360 comp rx=0 tx=0).stop 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- 172.21.15.86:0/3566275850 shutdown_connections 2024-09-17T15:44:28.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- 172.21.15.86:0/3566275850 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e40a5860 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- 172.21.15.86:0/3566275850 >> v2:172.21.15.203:3300/0 conn(0x7f93e40a5070 0x7f93e409cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- 172.21.15.86:0/3566275850 >> 172.21.15.86:0/3566275850 conn(0x7f93e40742a0 msgr2=0x7f93e40746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- 172.21.15.86:0/3566275850 shutdown_connections 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- 172.21.15.86:0/3566275850 wait complete. 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: build_initial_monmap 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: monmap: 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.571 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:28.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:28.572 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:28.572 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 Processor -- start 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- start start 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: init 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93e40a5070 0x7f93e41cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-c con 0x7f93e40a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-a con 0x7f93e40a5450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 0x7f93e413c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): picked mon.noname-b con 0x7f93e4137e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4142ac0 con 0x7f93e40a5450 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f93e40a6a70 con 0x7f93e4137e30 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f93e4142c40 con 0x7f93e40a5070 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ebe76640 10 monclient(hunting): authenticate will time out at 2001.538854s 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93e40a5070 0x7f93e41cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60496/0 (socket says 172.21.15.86:60496) 2024-09-17T15:44:28.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 -- 172.21.15.86:0/1306426236 learned_addr learned my addr 172.21.15.86:0/1306426236 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): get_auth_request con 0x7f93e40a5450 auth_method 0 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): get_auth_request con 0x7f93e40a5070 auth_method 0 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ea413640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 msgr2=0x7f93e413c050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93ea413640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 0x7f93e413c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9c12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient(hunting): handle_auth_done global_id 184272 payload 824 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93e40a5070 msgr2=0x7f93e41cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93e40a5070 0x7f93e41cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 msgr2=0x7f93e413c050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 0x7f93e413c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient: _finish_hunting 0 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 monclient: found mon.noname-a 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.570+0000 7f93e9411640 1 -- 172.21.15.86:0/1306426236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e41218d0 con 0x7f93e40a5450 2024-09-17T15:44:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93e9411640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 msgr2=0x7f93e4136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93e9411640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f93d401c840 tx=0x7f93d4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40b9020 con 0x7f93e40a5450 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: dump: 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: _finish_auth 0 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d40b5890 con 0x7f93e40a5450 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 5 monclient: authenticate success, global_id 184272 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _renew_subs 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e4074b10 con 0x7f93e40a5450 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40b0040 con 0x7f93e40a5450 2024-09-17T15:44:28.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 10 monclient: dump: 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:28.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _renew_subs 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93dbfff640 4 set_mon_vals no callback set 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e41215f0 con 0x7f93e40a5450 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e409cfe0 con 0x7f93e40a5450 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f93d40bc020 con 0x7f93e40a5450 2024-09-17T15:44:28.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 0x7f93c803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7f93d40e9bb0 con 0x7f93e40a5450 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93e9c12640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 0x7f93c803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.574+0000 7f93e9c12640 10 monclient: get_auth_request con 0x7f93c803bb60 auth_method 0 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.578+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.578+0000 7f93e9c12640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 0x7f93c803e020 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f93e00093d0 tx=0x7f93e0009360 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.582+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f93d40c3020 con 0x7f93e40a5450 2024-09-17T15:44:28.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.582+0000 7f93da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:28.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.582+0000 7f93da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93ebe76640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93ebe76640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93ebe76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f93e4121110 con 0x7f93e40a5450 2024-09-17T15:44:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93da7fc640 1 -- 172.21.15.86:0/1306426236 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f93d4016070 con 0x7f93e40a5450 2024-09-17T15:44:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.814+0000 7f93da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:28.812 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":156,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 msgr2=0x7f93c803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 0x7f93c803e020 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f93e00093d0 tx=0x7f93e0009360 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 10 monclient: shutdown 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 msgr2=0x7f93e4136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f93d401c840 tx=0x7f93d4019100 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 shutdown_connections 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f93c803bb60 0x7f93c803e020 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f93e4137e30 0x7f93e413c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93e40a5450 0x7f93e4136f40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 --2- 172.21.15.86:0/1306426236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93e40a5070 0x7f93e41cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 >> 172.21.15.86:0/1306426236 conn(0x7f93e40742a0 msgr2=0x7f93e413f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 shutdown_connections 2024-09-17T15:44:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:28.818+0000 7f93ebe76640 1 -- 172.21.15.86:0/1306426236 wait complete. 2024-09-17T15:44:31.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:31.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: get_monmap_and_config 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: build_initial_monmap 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: monmap: 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:31.969066+0000 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:31.969066+0000 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:31.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:31.964 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:31.964 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:31.964 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:31.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 Processor -- start 2024-09-17T15:44:31.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 -- start start 2024-09-17T15:44:31.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: init 2024-09-17T15:44:31.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): picked mon.noname-b con 0x7f25600689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): picked mon.noname-c con 0x7f2560068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): picked mon.noname-a con 0x7f2560135ae0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2560074500 con 0x7f2560135ae0 2024-09-17T15:44:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f256009c8b0 con 0x7f2560068db0 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f256009cb90 con 0x7f25600689d0 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2567835640 10 monclient(hunting): authenticate will time out at 2004.934592s 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2564dd0640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 msgr2=0x7f25600691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2564dd0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60506/0 (socket says 172.21.15.86:60506) 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 learned_addr learned my addr 172.21.15.86:0/3242954061 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 10 monclient(hunting): get_auth_request con 0x7f2560135ae0 auth_method 0 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f25600689d0 auth_method 0 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:31.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:31.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:31.966+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:32.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 msgr2=0x7f25600691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:32.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:44:32.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 10 monclient(hunting): get_auth_request con 0x7f2560135ae0 auth_method 0 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f25600689d0 auth_method 0 2024-09-17T15:44:32.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:32.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:44:32.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:32.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:32.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:44:32.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.170+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:44:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 msgr2=0x7f25600691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:44:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f25600689d0 auth_method 0 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 10 monclient(hunting): get_auth_request con 0x7f2560135ae0 auth_method 0 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:32.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:44:32.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:32.574+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 msgr2=0x7f25600691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 10 monclient(hunting): get_auth_request con 0x7f2560135ae0 auth_method 0 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:33.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f25600689d0 auth_method 0 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:44:33.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:33.378+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): tick 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 monclient(hunting): continuing hunt 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f256006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f256006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 msgr2=0x7f25600691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:34.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.203:3300/0 conn(0x7f2560068db0 0x7f25600691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 -- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 msgr2=0x7f2560135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 --2- 172.21.15.86:0/3242954061 >> v2:172.21.15.86:3300/0 conn(0x7f2560135ae0 0x7f2560135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient: _add_conns ranks=[1,0] 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25440036a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f25440073d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 -- 172.21.15.86:0/3242954061 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256009ce70 con 0x7f25440073d0 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 1 -- 172.21.15.86:0/3242954061 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254400c130 con 0x7f25440036a0 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f2552ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.966+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.203:6789/0 conn(0x7f25440036a0 legacy=0x7f2544005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:34.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:34.970+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.86:6789/0 conn(0x7f25440073d0 legacy=0x7f2544009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:35.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:35.170+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.203:6789/0 conn(0x7f25440036a0 legacy=0x7f2544005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:35.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:35.574+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.203:6789/0 conn(0x7f25440036a0 legacy=0x7f2544005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:36.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.378+0000 7f25655d1640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.203:6789/0 conn(0x7f25440036a0 legacy=0x7f2544005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124716884 0 0) 0x7f256009ce70 con 0x7f25440073d0 2024-09-17T15:44:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 10 monclient(hunting): _negotiate 2024-09-17T15:44:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 10 monclient(hunting): my global_id is 184275 2024-09-17T15:44:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.690+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f254c003530 con 0x7f25440073d0 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3741168472 0 0) 0x7f254c003530 con 0x7f25440073d0 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.203:6789/0 conn(0x7f25440036a0 legacy=0x7f2544005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254400d390 con 0x7f25440073d0 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _finish_auth 0 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _check_auth_tickets 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f255c004510 con 0x7f25440073d0 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: dump: 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 >> v1:172.21.15.86:6789/0 conn(0x7f25440073d0 legacy=0x7f2544009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c003a60 0x7f254c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): picked mon.a con 0x7f254c003a60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f254c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): picked mon.c con 0x7f25600689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 0x7f254c0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): picked mon.b con 0x7f254c004e20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f254c00bc50 con 0x7f254c003a60 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f254c00be00 con 0x7f254c004e20 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f254c00c000 con 0x7f25600689d0 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2553fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 reap_dead start 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2565dd2640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 msgr2=0x7f254c0052a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:36.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f254c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2565dd2640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 0x7f254c0052a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): get_auth_request con 0x7f25600689d0 auth_method 0 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c003a60 0x7f254c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f254c003a60 auth_method 0 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:36.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient(hunting): handle_auth_done global_id 184275 payload 293 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 msgr2=0x7f254c0052a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 0x7f254c0052a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c003a60 msgr2=0x7f254c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c003a60 0x7f254c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient: _finish_hunting 0 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 monclient: found mon.c 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:36.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254400d390 con 0x7f25600689d0 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient: _finish_auth 0 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 10 monclient: _check_auth_tickets 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2564dd0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2567835640 5 monclient: authenticate success, global_id 184275 2024-09-17T15:44:36.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.698+0000 7f2567835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2564dd0640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f254c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2564dd0640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f254c004810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f255c003940 tx=0x7f255c01d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48030b465fa764b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c00d2c0 con 0x7f25600689d0 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: dump: 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:36.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f255c005230 con 0x7f25600689d0 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:36.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 4 set_mon_vals no callback set 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 1 -- 172.21.15.86:0/3242954061 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c005620 con 0x7f25600689d0 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2553fff640 10 monclient: dump: 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:36.713 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 10 monclient: shutdown 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 msgr2=0x7f254c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f254c004810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f255c003940 tx=0x7f255c01d2e0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- 172.21.15.86:0/3242954061 shutdown_connections 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f254c004e20 0x7f254c0052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f25600689d0 0x7f254c004810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 --2- 172.21.15.86:0/3242954061 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c003a60 0x7f254c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- 172.21.15.86:0/3242954061 >> 172.21.15.86:0/3242954061 conn(0x7f25600a6440 msgr2=0x7f25600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- 172.21.15.86:0/3242954061 shutdown_connections 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- 172.21.15.86:0/3242954061 wait complete. 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 10 monclient: build_initial_monmap 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 10 monclient: monmap: 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:36.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 Processor -- start 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 1 -- start start 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.706+0000 7f2567835640 10 monclient: init 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): picked mon.noname-c con 0x7f254c003a60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c004e20 0x7f2560134230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): picked mon.noname-a con 0x7f254c004e20 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 0x7f2560071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): picked mon.noname-b con 0x7f25600689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:36.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2560136280 con 0x7f254c004e20 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2560136100 con 0x7f25600689d0 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2560136400 con 0x7f254c003a60 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient(hunting): authenticate will time out at 2009.676322s 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2565dd2640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 msgr2=0x7f2560071050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2565dd2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 0x7f2560071050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43304/0 (socket says 172.21.15.86:43304) 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 -- 172.21.15.86:0/2452505238 learned_addr learned my addr 172.21.15.86:0/2452505238 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:36.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): get_auth_request con 0x7f254c003a60 auth_method 0 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c004e20 0x7f2560134230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): get_auth_request con 0x7f254c004e20 auth_method 0 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:36.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2564dd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient(hunting): handle_auth_done global_id 164474 payload 824 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 -- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 msgr2=0x7f2560071050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 0x7f2560071050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 -- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c004e20 msgr2=0x7f2560134230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c004e20 0x7f2560134230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient: _finish_hunting 0 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 monclient: found mon.noname-c 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 -- 172.21.15.86:0/2452505238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2560072b10 con 0x7f254c003a60 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 -- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 msgr2=0x7f25601cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25655d1640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2558004670 tx=0x7f25580156a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:36.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255801c030 con 0x7f254c003a60 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: dump: 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:36.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: _finish_auth 0 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2558015e70 con 0x7f254c003a60 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 5 monclient: authenticate success, global_id 164474 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _renew_subs 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2560121630 con 0x7f254c003a60 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2558018660 con 0x7f254c003a60 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:36.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2551ffb640 10 monclient: dump: 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f25537fe640 4 set_mon_vals no callback set 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _renew_subs 2024-09-17T15:44:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.710+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f25601d04e0 con 0x7f254c003a60 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f255801f030 con 0x7f254c003a60 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2567835640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2567835640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2567835640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2560121ae0 con 0x7f254c003a60 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2551ffb640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 0x7f254403ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7f2558051040 con 0x7f254c003a60 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2564dd0640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 0x7f254403ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:36.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.714+0000 7f2564dd0640 10 monclient: get_auth_request con 0x7f254403c7f0 auth_method 0 2024-09-17T15:44:36.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.718+0000 7f2564dd0640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 0x7f254403ecb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f255c006140 tx=0x7f255c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:36.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.718+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2558026070 con 0x7f254c003a60 2024-09-17T15:44:36.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.718+0000 7f2551ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:36.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.718+0000 7f2551ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:36.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2567835640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2567835640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2567835640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2560136790 con 0x7f254c003a60 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2551ffb640 1 -- 172.21.15.86:0/2452505238 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f25580203c0 con 0x7f254c003a60 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2551ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:36.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.958+0000 7f2551ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:36.956 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:36.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 msgr2=0x7f254403ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:36.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 0x7f254403ecb0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f255c006140 tx=0x7f255c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:44:36.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 10 monclient: shutdown 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 msgr2=0x7f25601cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2558004670 tx=0x7f25580156a0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 shutdown_connections 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f254403c7f0 0x7f254403ecb0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25600689d0 0x7f2560071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f254c004e20 0x7f2560134230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 --2- 172.21.15.86:0/2452505238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f254c003a60 0x7f25601cf660 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 >> 172.21.15.86:0/2452505238 conn(0x7f25600a6440 msgr2=0x7f2560135190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 shutdown_connections 2024-09-17T15:44:36.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:36.962+0000 7f2567835640 1 -- 172.21.15.86:0/2452505238 wait complete. 2024-09-17T15:44:36.975 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:44:36.975 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:44:36.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:44:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: get_monmap_and_config 2024-09-17T15:44:37.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: build_initial_monmap 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: monmap: 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:37.101035+0000 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:37.101035+0000 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:37.095 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 Processor -- start 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 -- start start 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: init 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a50e0 0x7f139809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-b con 0x7f13980a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13980a54c0 0x7f13980a58c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-a con 0x7f13980a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1398137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13980a6280 con 0x7f1398137ea0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f139809c0a0 con 0x7f13980a54c0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f139809c380 con 0x7f13980a50e0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139ea20640 10 monclient(hunting): authenticate will time out at 2010.067194s 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13980a54c0 0x7f13980a58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f13980a54c0 0x7f13980a58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60560/0 (socket says 172.21.15.86:60560) 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 -- 172.21.15.86:0/3970704035 learned_addr learned my addr 172.21.15.86:0/3970704035 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): get_auth_request con 0x7f13980a54c0 auth_method 0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a50e0 0x7f139809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f139cfbd640 1 --1- 172.21.15.86:0/3970704035 >> v1:172.21.15.86:6789/0 conn(0x7f1398137ea0 0x7f1398138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43508/0 (socket says 172.21.15.86:43508) 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): get_auth_request con 0x7f13980a50e0 auth_method 0 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient(hunting): handle_auth_done global_id 184281 payload 274 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 -- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a50e0 msgr2=0x7f139809cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 --2- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a50e0 0x7f139809cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 -- 172.21.15.86:0/3970704035 >> v1:172.21.15.86:6789/0 conn(0x7f1398137ea0 legacy=0x7f1398138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 monclient: found mon.noname-a 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.098+0000 7f13977fe640 1 -- 172.21.15.86:0/3970704035 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139809c660 con 0x7f13980a54c0 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.102+0000 7f13977fe640 1 -- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a54c0 msgr2=0x7f13980a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.102+0000 7f13977fe640 1 --2- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a54c0 0x7f13980a58c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f138c009870 tx=0x7f138c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baa6ff8769147d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 1 -- 172.21.15.86:0/3970704035 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0b0070 con 0x7f13980a54c0 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: dump: 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 1 -- 172.21.15.86:0/3970704035 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138c0a4e50 con 0x7f13980a54c0 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 1 -- 172.21.15.86:0/3970704035 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0ab820 con 0x7f13980a54c0 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 5 monclient: authenticate success, global_id 184281 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 4 set_mon_vals no callback set 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1396ffd640 10 monclient: dump: 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: shutdown 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a54c0 msgr2=0x7f13980a58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a54c0 0x7f13980a58c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f138c009870 tx=0x7f138c01c4b0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- 172.21.15.86:0/3970704035 shutdown_connections 2024-09-17T15:44:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a54c0 0x7f13980a58c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- 172.21.15.86:0/3970704035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a50e0 0x7f139809cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- 172.21.15.86:0/3970704035 >> 172.21.15.86:0/3970704035 conn(0x7f13980aaae0 msgr2=0x7f13980aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- 172.21.15.86:0/3970704035 shutdown_connections 2024-09-17T15:44:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- 172.21.15.86:0/3970704035 wait complete. 2024-09-17T15:44:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: build_initial_monmap 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: monmap: 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 Processor -- start 2024-09-17T15:44:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- start start 2024-09-17T15:44:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: init 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-a con 0x7f13980a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 0x7f1398137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-c con 0x7f13980a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1398137ea0 0x7f139813c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): picked mon.noname-b con 0x7f1398137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1398142cb0 con 0x7f13980a50e0 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1398142b30 con 0x7f1398137ea0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1398142e30 con 0x7f13980a54c0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139ea20640 10 monclient(hunting): authenticate will time out at 2010.074282s 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 0x7f1398137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60574/0 (socket says 172.21.15.86:60574) 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 0x7f1398137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:43320/0 (socket says 172.21.15.86:43320) 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 -- 172.21.15.86:0/3473787458 learned_addr learned my addr 172.21.15.86:0/3473787458 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): get_auth_request con 0x7f13980a50e0 auth_method 0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139cfbd640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1398137ea0 msgr2=0x7f139813c1b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): get_auth_request con 0x7f13980a54c0 auth_method 0 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f139cfbd640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1398137ea0 0x7f139813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient(hunting): handle_auth_done global_id 184284 payload 824 2024-09-17T15:44:37.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 msgr2=0x7f1398137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 0x7f1398137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1398137ea0 msgr2=0x7f139813c1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1398137ea0 0x7f139813c1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 monclient: found mon.noname-a 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f1397fff640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13981d0ff0 con 0x7f13980a50e0 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.106+0000 7f13977fe640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f13980a54c0 0x7f1398137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1397fff640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 msgr2=0x7f13981d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1397fff640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f138800efc0 tx=0x7f138800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388018020 con 0x7f13980a50e0 2024-09-17T15:44:37.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: dump: 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: _finish_auth 0 2024-09-17T15:44:37.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1388014680 con 0x7f13980a50e0 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388014a50 con 0x7f13980a50e0 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 5 monclient: authenticate success, global_id 184284 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 monclient: _renew_subs 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13981210f0 con 0x7f13980a50e0 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 10 monclient: dump: 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13967fc640 4 set_mon_vals no callback set 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 monclient: _renew_subs 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1398121990 con 0x7f13980a50e0 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:37.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 0x7f139813c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f139806a0e0 con 0x7f1398137540 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 0x7f139813c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient: get_auth_request con 0x7f1398137540 auth_method 0 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 10 monclient(hunting): handle_auth_done global_id 184287 payload 824 2024-09-17T15:44:37.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f138801b020 con 0x7f13980a50e0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f13977fe640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1398121540 con 0x7f1398137540 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f138403bb60 0x7f138403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7f1388050c80 con 0x7f13980a50e0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139cfbd640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f138403bb60 0x7f138403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.110+0000 7f139cfbd640 10 monclient: get_auth_request con 0x7f138403bb60 auth_method 0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f13977fe640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 msgr2=0x7f139813c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f13977fe640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 0x7f139813c6f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f138c0ad000 tx=0x7f138c01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0b0050 con 0x7f1398137540 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 10 monclient: dump: 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.123 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f138c0bc070 con 0x7f1398137540 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 msgr2=0x7f139813c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 0x7f139813c6f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f138c0ad000 tx=0x7f138c01acf0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 msgr2=0x7f139813c6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:37.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f1394ff9640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1398137540 0x7f139813c6f0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f138c0ad000 tx=0x7f138c01acf0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.114+0000 7f139cfbd640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f138403bb60 0x7f138403e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f13900048d0 tx=0x7f13900079f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f139ea20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:37.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f139ea20640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 0x7f13980a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f139ea20640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f139813d040 con 0x7f139813cc30 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 0x7f13980a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient: get_auth_request con 0x7f139813cc30 auth_method 0 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 10 monclient(hunting): handle_auth_done global_id 184290 payload 824 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.118+0000 7f13977fe640 1 -- 172.21.15.86:0/3473787458 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f139809cd60 con 0x7f139813cc30 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f13977fe640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 msgr2=0x7f13980a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f13977fe640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 0x7f13980a05f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f139806c9a0 tx=0x7f138c0a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138c0c1bd0 con 0x7f139813cc30 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 10 monclient: dump: 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.128 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f138c0b1040 con 0x7f139813cc30 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 msgr2=0x7f13980a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 0x7f13980a05f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f139806c9a0 tx=0x7f138c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 msgr2=0x7f13980a05f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f1394ff9640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f139813cc30 0x7f13980a05f0 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f139806c9a0 tx=0x7f138c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 160, 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 80615, 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:37.130 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:37.131 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:37.133 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.134 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.139 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.140 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.141 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:37.142 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f138403bb60 msgr2=0x7f138403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f138403bb60 0x7f138403e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f13900048d0 tx=0x7f13900079f0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 10 monclient: shutdown 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 msgr2=0x7f13981d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f138800efc0 tx=0x7f138800c490 comp rx=0 tx=0).stop 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139cfbd640 1 -- 172.21.15.86:0/3473787458 reap_dead start 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 shutdown_connections 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 --2- 172.21.15.86:0/3473787458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f13980a50e0 0x7f13981d00e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 >> 172.21.15.86:0/3473787458 conn(0x7f13980aaae0 msgr2=0x7f13980a1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:37.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 shutdown_connections 2024-09-17T15:44:37.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.122+0000 7f139ea20640 1 -- 172.21.15.86:0/3473787458 wait complete. 2024-09-17T15:44:37.145 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:44:37.145 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:44:37.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.258+0000 7efcc226c640 10 monclient: get_monmap_and_config 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.258+0000 7efcc226c640 10 monclient: build_initial_monmap 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.258+0000 7efcc226c640 10 monclient: monmap: 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:37.263488+0000 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:37.263488+0000 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:37.258 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 Processor -- start 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 -- start start 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient: init 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcbc0a4f70 0x7efcbc09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc0a4f70 addr v2:172.21.15.86:3300/0 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efcbc0a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 0x7efcbc13f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc13d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc09d430 con 0x7efcbc0a5350 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc09c7f0 con 0x7efcbc0a4f70 2024-09-17T15:44:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0a68d0 con 0x7efcbc13d180 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc226c640 10 monclient(hunting): authenticate will time out at 2010.229168s 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcbc0a4f70 0x7efcbc09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbb7fe640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7efcbc0a5350 legacy=0x7efcbc0a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcbc0a4f70 0x7efcbc09cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60594/0 (socket says 172.21.15.86:60594) 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 -- 172.21.15.86:0/3470871024 learned_addr learned my addr 172.21.15.86:0/3470871024 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): get_auth_request con 0x7efcbc0a4f70 auth_method 0 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc0809640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 msgr2=0x7efcbc13f570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcc0809640 1 --2- 172.21.15.86:0/3470871024 >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 0x7efcbc13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbaffd640 1 -- 172.21.15.86:0/3470871024 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient(hunting): handle_auth_done global_id 184293 payload 274 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 -- 172.21.15.86:0/3470871024 >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 msgr2=0x7efcbc13f570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 --2- 172.21.15.86:0/3470871024 >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 0x7efcbc13f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 -- 172.21.15.86:0/3470871024 >> v1:172.21.15.203:6789/0 conn(0x7efcbc0a5350 legacy=0x7efcbc0a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient: _finish_hunting 0 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 monclient: found mon.noname-a 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:37.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.262+0000 7efcbbfff640 1 -- 172.21.15.86:0/3470871024 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0a64a0 con 0x7efcbc0a4f70 2024-09-17T15:44:37.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbbfff640 1 -- 172.21.15.86:0/3470871024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 msgr2=0x7efcbc09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbbfff640 1 --2- 172.21.15.86:0/3470871024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc09cef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efcbc09c970 tx=0x7efcac01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3c1ac8d5154acd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 1 -- 172.21.15.86:0/3470871024 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0b2070 con 0x7efcbc0a4f70 2024-09-17T15:44:37.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: dump: 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: _finish_auth 0 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 1 -- 172.21.15.86:0/3470871024 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcac0ad640 con 0x7efcbc0a4f70 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 1 -- 172.21.15.86:0/3470871024 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0ada10 con 0x7efcbc0a4f70 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcbaffd640 10 monclient: dump: 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 5 monclient: authenticate success, global_id 184293 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 4 set_mon_vals no callback set 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 10 monclient: shutdown 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- 172.21.15.86:0/3470871024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 msgr2=0x7efcbc09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 --2- 172.21.15.86:0/3470871024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc09cef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efcbc09c970 tx=0x7efcac01c3d0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- 172.21.15.86:0/3470871024 shutdown_connections 2024-09-17T15:44:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 --2- 172.21.15.86:0/3470871024 >> v2:172.21.15.203:3300/0 conn(0x7efcbc13d180 0x7efcbc13f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 --2- 172.21.15.86:0/3470871024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc09cef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- 172.21.15.86:0/3470871024 >> 172.21.15.86:0/3470871024 conn(0x7efcbc0a7240 msgr2=0x7efcbc0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- 172.21.15.86:0/3470871024 shutdown_connections 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- 172.21.15.86:0/3470871024 wait complete. 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 10 monclient: build_initial_monmap 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 10 monclient: monmap: 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 Processor -- start 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 1 -- start start 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.266+0000 7efcc226c640 10 monclient: init 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc1320b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-a con 0x7efcbc0a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 0x7efcbc1325f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-c con 0x7efcbc0a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcbc13d180 0x7efcbc1cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): picked mon.noname-b con 0x7efcbc13d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc142450 con 0x7efcbc0a4f70 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc1422d0 con 0x7efcbc13d180 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc1425d0 con 0x7efcbc0a5350 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient(hunting): authenticate will time out at 2010.236273s 2024-09-17T15:44:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc1320b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 0x7efcbc1325f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc1320b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60598/0 (socket says 172.21.15.86:60598) 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 1 -- 172.21.15.86:0/2405415643 learned_addr learned my addr 172.21.15.86:0/2405415643 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc0809640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcbc13d180 msgr2=0x7efcbc1cfc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): get_auth_request con 0x7efcbc0a4f70 auth_method 0 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): get_auth_request con 0x7efcbc0a5350 auth_method 0 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc0809640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcbc13d180 0x7efcbc1cfc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient(hunting): handle_auth_done global_id 164483 payload 824 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcbc13d180 msgr2=0x7efcbc1cfc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efcbc13d180 0x7efcbc1cfc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 msgr2=0x7efcbc1320b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc1320b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 monclient: found mon.noname-c 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc13d9f0 con 0x7efcbc0a5350 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbbfff640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc0a4f70 0x7efcbc1320b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 msgr2=0x7efcbc1325f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcbb7fe640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 0x7efcbc1325f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb000b4a0 tx=0x7efcb000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0003de0 con 0x7efcbc0a5350 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:44:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: dump: 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: _finish_auth 0 2024-09-17T15:44:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 5 monclient: authenticate success, global_id 164483 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb0015070 con 0x7efcbc0a5350 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _renew_subs 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc0a3710 con 0x7efcbc0a5350 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb00052b0 con 0x7efcbc0a5350 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcb8ff9640 10 monclient: dump: 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcba7fc640 4 set_mon_vals no callback set 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _renew_subs 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc132b30 con 0x7efcbc0a5350 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.270+0000 7efcc226c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc226c640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 0x7efcbc1d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc1d0af0 con 0x7efcbc1d01d0 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7efcb0005490 con 0x7efcbc0a5350 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 0x7efcbc1d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient: get_auth_request con 0x7efcbc1d01d0 auth_method 0 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efca803bb10 0x7efca803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(612..612 src has 438..612) ==== 4859+0+0 (secure 0 0 0) 0x7efcb004f9e0 con 0x7efcbc0a5350 2024-09-17T15:44:37.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 10 monclient(hunting): handle_auth_done global_id 164486 payload 824 2024-09-17T15:44:37.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efcbc09cef0 con 0x7efcbc1d01d0 2024-09-17T15:44:37.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc0809640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efca803bb10 0x7efca803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc0809640 10 monclient: get_auth_request con 0x7efca803bb10 auth_method 0 2024-09-17T15:44:37.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 msgr2=0x7efcbc1d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcbbfff640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 0x7efcbc1d05b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcac01cc70 tx=0x7efcac01ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0bd020 con 0x7efcbc1d01d0 2024-09-17T15:44:37.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 10 monclient: dump: 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efcac0ade50 con 0x7efcbc1d01d0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 msgr2=0x7efcbc1d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 0x7efcbc1d05b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcac01cc70 tx=0x7efcac01ac10 comp rx=0 tx=0).stop 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 msgr2=0x7efcbc1d05b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcb8ff9640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1d01d0 0x7efcbc1d05b0 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efcac01cc70 tx=0x7efcac01ac10 comp rx=0 tx=0).stop 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.274+0000 7efcc0809640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efca803bb10 0x7efca803dfd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a2260 tx=0x7efcb4009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcc226c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcc226c640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 0x7efcbc1365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcc226c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc0a5ff0 con 0x7efcbc1361b0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 0x7efcbc1365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient: get_auth_request con 0x7efcbc1361b0 auth_method 0 2024-09-17T15:44:37.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 10 monclient(hunting): handle_auth_done global_id 164489 payload 824 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 1 -- 172.21.15.86:0/2405415643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efcbc069920 con 0x7efcbc1361b0 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 msgr2=0x7efcbc1365c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.282+0000 7efcbbfff640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 0x7efcbc1365c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c10 tx=0x7efcac019170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcac0bd070 con 0x7efcbc1361b0 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 10 monclient: dump: 2024-09-17T15:44:37.286 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7efcac0b8c90 con 0x7efcbc1361b0 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 msgr2=0x7efcbc1365c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 0x7efcbc1365c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c10 tx=0x7efcac019170 comp rx=0 tx=0).stop 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 msgr2=0x7efcbc1365c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcb8ff9640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1361b0 0x7efcbc1365c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 160, 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 180473, 2024-09-17T15:44:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:44:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:44:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:44:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:44:37.294 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:44:37.295 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:44:37.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:44:37.298 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.299 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:44:37.300 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:44:37.301 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efca803bb10 msgr2=0x7efca803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7efca803bb10 0x7efca803dfd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efcbc0a2260 tx=0x7efcb4009270 comp rx=0 tx=0).stop 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 10 monclient: shutdown 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 msgr2=0x7efcbc1325f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 0x7efcbc1325f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efcb000b4a0 tx=0x7efcb000b960 comp rx=0 tx=0).stop 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc0809640 1 -- 172.21.15.86:0/2405415643 reap_dead start 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 shutdown_connections 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 --2- 172.21.15.86:0/2405415643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc0a5350 0x7efcbc1325f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 >> 172.21.15.86:0/2405415643 conn(0x7efcbc0a7240 msgr2=0x7efcbc13c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:37.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 shutdown_connections 2024-09-17T15:44:37.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:37.286+0000 7efcc226c640 1 -- 172.21.15.86:0/2405415643 wait complete. 2024-09-17T15:44:37.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:44:37.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:44:37.681 INFO:tasks.rados.rados.0.smithi203.stdout:2748: done (1 left) 2024-09-17T15:44:37.681 INFO:tasks.rados.rados.0.smithi203.stdout:2749: write oid 11 current snap is 294 2024-09-17T15:44:37.681 INFO:tasks.rados.rados.0.smithi203.stdout:2749: seq_num 972 ranges {736404=718859,1929321=407083} 2024-09-17T15:44:37.704 INFO:tasks.rados.rados.0.smithi203.stdout:2749: writing smithi20328086-11 from 736404 to 1455263 tid 1 2024-09-17T15:44:37.721 INFO:tasks.rados.rados.0.smithi203.stdout:2749: writing smithi20328086-11 from 1929321 to 2336404 tid 2 2024-09-17T15:44:37.721 INFO:tasks.rados.rados.0.smithi203.stdout:2750: write oid 40 current snap is 294 2024-09-17T15:44:37.722 INFO:tasks.rados.rados.0.smithi203.stdout:2750: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-17T15:44:37.745 INFO:tasks.rados.rados.0.smithi203.stdout:2750: writing smithi20328086-40 from 753211 to 1547319 tid 1 2024-09-17T15:44:37.754 INFO:tasks.rados.rados.0.smithi203.stdout:2750: writing smithi20328086-40 from 1987803 to 2353211 tid 2 2024-09-17T15:44:37.755 INFO:tasks.rados.rados.0.smithi203.stdout:2751: rollback oid 21 current snap is 294 2024-09-17T15:44:37.755 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 21 to 290 2024-09-17T15:44:37.755 INFO:tasks.rados.rados.0.smithi203.stdout:2747: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:44:37.956 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2542526 2024-09-17T15:44:38.054 INFO:tasks.rados.rados.0.smithi203.stdout:2749: finishing write tid 1 to smithi20328086-11 2024-09-17T15:44:38.054 INFO:tasks.rados.rados.0.smithi203.stdout:2749: finishing write tid 2 to smithi20328086-11 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: finishing write tid 3 to smithi20328086-11 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: finishing write tid 4 to smithi20328086-11 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: oid 11 updating version 0 to 1235 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: oid 11 updating version 1235 to 1236 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: oid 11 updating version 1236 to 1237 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1237 (ObjNum 972 snap 294 seq_num 972) dirty exists 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2749: left oid 11 (ObjNum 972 snap 294 seq_num 972) 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2751: finishing rollback tid 1 to smithi20328086-21 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2750: finishing write tid 1 to smithi20328086-40 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2750: finishing write tid 2 to smithi20328086-40 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2750: finishing write tid 3 to smithi20328086-40 2024-09-17T15:44:38.055 INFO:tasks.rados.rados.0.smithi203.stdout:2750: finishing write tid 4 to smithi20328086-40 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2750: oid 40 updating version 0 to 1241 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2750: oid 40 version 1241 is already newer than 1239 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2750: oid 40 version 1241 is already newer than 1240 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1241 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2750: left oid 40 (ObjNum 973 snap 294 seq_num 973) 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2747: done (3 left) 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2749: done (2 left) 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2750: done (1 left) 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2751: done (0 left) 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2752: copy_from oid 34 from oid 33 current snap is 294 2024-09-17T15:44:38.056 INFO:tasks.rados.rados.0.smithi203.stdout:2753: delete oid 39 current snap is 294 2024-09-17T15:44:38.058 INFO:tasks.rados.rados.0.smithi203.stdout:2753: done (1 left) 2024-09-17T15:44:38.058 INFO:tasks.rados.rados.0.smithi203.stdout:2754: snap_create 2024-09-17T15:44:38.075 INFO:tasks.rados.rados.0.smithi203.stdout:2752: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:44:38.080 INFO:tasks.rados.rados.0.smithi203.stdout:2752: finishing copy_from to smithi20328086-34 2024-09-17T15:44:38.080 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1101 (ObjNum 970 snap 294 seq_num 970) dirty exists 2024-09-17T15:44:38.684 INFO:tasks.rados.rados.0.smithi203.stdout:2752: done (1 left) 2024-09-17T15:44:38.685 INFO:tasks.rados.rados.0.smithi203.stdout:2754: done (0 left) 2024-09-17T15:44:38.685 INFO:tasks.rados.rados.0.smithi203.stdout:2755: write (excl) oid 28 current snap is 295 2024-09-17T15:44:38.685 INFO:tasks.rados.rados.0.smithi203.stdout:2755: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-17T15:44:38.708 INFO:tasks.rados.rados.0.smithi203.stdout:2755: writing smithi20328086-28 from 786825 to 1447784 tid 1 2024-09-17T15:44:38.723 INFO:tasks.rados.rados.0.smithi203.stdout:2755: writing smithi20328086-28 from 2104767 to 2386825 tid 2 2024-09-17T15:44:38.724 INFO:tasks.rados.rados.0.smithi203.stdout:2756: snap_create 2024-09-17T15:44:38.724 INFO:tasks.rados.rados.0.smithi203.stdout:2755: finishing write tid 1 to smithi20328086-28 2024-09-17T15:44:38.730 INFO:tasks.rados.rados.0.smithi203.stdout:2755: finishing write tid 2 to smithi20328086-28 2024-09-17T15:44:38.732 INFO:tasks.rados.rados.0.smithi203.stdout:2755: finishing write tid 3 to smithi20328086-28 2024-09-17T15:44:38.732 INFO:tasks.rados.rados.0.smithi203.stdout:2755: finishing write tid 4 to smithi20328086-28 2024-09-17T15:44:38.732 INFO:tasks.rados.rados.0.smithi203.stdout:2755: oid 28 updating version 0 to 1490 2024-09-17T15:44:38.732 INFO:tasks.rados.rados.0.smithi203.stdout:2755: oid 28 version 1490 is already newer than 1488 2024-09-17T15:44:38.733 INFO:tasks.rados.rados.0.smithi203.stdout:2755: oid 28 version 1490 is already newer than 1489 2024-09-17T15:44:38.733 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1490 (ObjNum 975 snap 295 seq_num 975) dirty exists 2024-09-17T15:44:38.733 INFO:tasks.rados.rados.0.smithi203.stdout:2755: left oid 28 (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:44:39.687 INFO:tasks.rados.rados.0.smithi203.stdout:2755: done (1 left) 2024-09-17T15:44:39.687 INFO:tasks.rados.rados.0.smithi203.stdout:2756: done (0 left) 2024-09-17T15:44:39.687 INFO:tasks.rados.rados.0.smithi203.stdout:2757: copy_from oid 49 from oid 20 current snap is 296 2024-09-17T15:44:39.687 INFO:tasks.rados.rados.0.smithi203.stdout:2758: snap_create 2024-09-17T15:44:39.722 INFO:tasks.rados.rados.0.smithi203.stdout:2757: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:44:39.727 INFO:tasks.rados.rados.0.smithi203.stdout:2757: finishing copy_from to smithi20328086-49 2024-09-17T15:44:39.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1130 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:44:40.690 INFO:tasks.rados.rados.0.smithi203.stdout:2757: done (1 left) 2024-09-17T15:44:40.690 INFO:tasks.rados.rados.0.smithi203.stdout:2758: done (0 left) 2024-09-17T15:44:40.690 INFO:tasks.rados.rados.0.smithi203.stdout:2759: rollback oid 22 current snap is 297 2024-09-17T15:44:40.690 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 22 to 294 2024-09-17T15:44:40.690 INFO:tasks.rados.rados.0.smithi203.stdout:2760: snap_create 2024-09-17T15:44:40.701 INFO:tasks.rados.rados.0.smithi203.stdout:2759: finishing rollback tid 0 to smithi20328086-22 2024-09-17T15:44:40.701 INFO:tasks.rados.rados.0.smithi203.stdout:2759: finishing rollback tid 1 to smithi20328086-22 2024-09-17T15:44:40.702 INFO:tasks.rados.rados.0.smithi203.stdout:2759: finishing rollback tid 2 to smithi20328086-22 2024-09-17T15:44:40.702 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1265 (ObjNum 954 snap 289 seq_num 954) dirty exists 2024-09-17T15:44:41.536 INFO:tasks.rados.rados.0.smithi203.stdout:2759: done (1 left) 2024-09-17T15:44:41.536 INFO:tasks.rados.rados.0.smithi203.stdout:2760: done (0 left) 2024-09-17T15:44:41.536 INFO:tasks.rados.rados.0.smithi203.stdout:2761: snap_remove snap 290 2024-09-17T15:44:42.537 INFO:tasks.rados.rados.0.smithi203.stdout:2761: done (0 left) 2024-09-17T15:44:42.537 INFO:tasks.rados.rados.0.smithi203.stdout:2762: delete oid 12 current snap is 298 2024-09-17T15:44:42.555 INFO:tasks.rados.rados.0.smithi203.stdout:2762: done (0 left) 2024-09-17T15:44:42.555 INFO:tasks.rados.rados.0.smithi203.stdout:2763: snap_create 2024-09-17T15:44:43.550 INFO:tasks.rados.rados.0.smithi203.stdout:2763: done (0 left) 2024-09-17T15:44:43.550 INFO:tasks.rados.rados.0.smithi203.stdout:2764: read oid 8 snap -1 2024-09-17T15:44:43.550 INFO:tasks.rados.rados.0.smithi203.stdout:2764: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2765: read oid 47 snap -1 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2765: expect (ObjNum 952 snap 288 seq_num 952) 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2766: read oid 5 snap 294 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2766: expect (ObjNum 963 snap 290 seq_num 963) 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2767: rollback oid 42 current snap is 299 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 294 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2768: read oid 22 snap -1 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2768: expect (ObjNum 954 snap 289 seq_num 954) 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2769: read oid 7 snap -1 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2769: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2770: write oid 44 current snap is 299 2024-09-17T15:44:43.551 INFO:tasks.rados.rados.0.smithi203.stdout:2770: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-17T15:44:43.566 INFO:tasks.rados.rados.0.smithi203.stdout:2770: writing smithi20328086-44 from 420439 to 831896 tid 1 2024-09-17T15:44:43.584 INFO:tasks.rados.rados.0.smithi203.stdout:2770: writing smithi20328086-44 from 1421731 to 1901741 tid 2 2024-09-17T15:44:43.584 INFO:tasks.rados.rados.0.smithi203.stdout:2771: delete oid 43 current snap is 299 2024-09-17T15:44:43.584 INFO:tasks.rados.rados.0.smithi203.stdout:2769: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:44:43.675 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2132670 2024-09-17T15:44:43.944 INFO:tasks.rados.rados.0.smithi203.stdout:2768: expect (ObjNum 954 snap 289 seq_num 954) 2024-09-17T15:44:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:2766: expect (ObjNum 963 snap 290 seq_num 963) 2024-09-17T15:44:44.050 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 621328 2024-09-17T15:44:44.088 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:44.174 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2003962 2024-09-17T15:44:44.175 INFO:tasks.rados.rados.0.smithi203.stdout:2764: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:44:44.266 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 25936 2024-09-17T15:44:44.309 INFO:tasks.rados.rados.0.smithi203.stdout:2767: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:44:44.310 INFO:tasks.rados.rados.0.smithi203.stdout:2770: finishing write tid 1 to smithi20328086-44 2024-09-17T15:44:44.310 INFO:tasks.rados.rados.0.smithi203.stdout:2767: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:44:44.310 INFO:tasks.rados.rados.0.smithi203.stdout:2765: expect (ObjNum 952 snap 288 seq_num 952) 2024-09-17T15:44:44.310 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:44.353 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 19740 2024-09-17T15:44:44.431 INFO:tasks.rados.rados.0.smithi203.stdout:2767: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1358 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: finishing write tid 2 to smithi20328086-44 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: finishing write tid 3 to smithi20328086-44 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: finishing write tid 4 to smithi20328086-44 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: oid 44 updating version 0 to 1246 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: oid 44 updating version 1246 to 1247 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: oid 44 updating version 1247 to 1248 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1248 (ObjNum 977 snap 299 seq_num 977) dirty exists 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2770: left oid 44 (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2764: done (7 left) 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2765: done (6 left) 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2766: done (5 left) 2024-09-17T15:44:44.432 INFO:tasks.rados.rados.0.smithi203.stdout:2767: done (4 left) 2024-09-17T15:44:44.433 INFO:tasks.rados.rados.0.smithi203.stdout:2768: done (3 left) 2024-09-17T15:44:44.433 INFO:tasks.rados.rados.0.smithi203.stdout:2769: done (2 left) 2024-09-17T15:44:44.433 INFO:tasks.rados.rados.0.smithi203.stdout:2770: done (1 left) 2024-09-17T15:44:44.433 INFO:tasks.rados.rados.0.smithi203.stdout:2771: done (0 left) 2024-09-17T15:44:44.433 INFO:tasks.rados.rados.0.smithi203.stdout:2772: snap_create 2024-09-17T15:44:44.559 INFO:tasks.rados.rados.0.smithi203.stdout:2772: done (0 left) 2024-09-17T15:44:44.559 INFO:tasks.rados.rados.0.smithi203.stdout:2773: rollback oid 19 current snap is 300 2024-09-17T15:44:44.559 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 299 2024-09-17T15:44:44.559 INFO:tasks.rados.rados.0.smithi203.stdout:2774: delete oid 24 current snap is 300 2024-09-17T15:44:44.563 INFO:tasks.rados.rados.0.smithi203.stdout:2773: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:44:44.563 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:44:44.568 INFO:tasks.rados.rados.0.smithi203.stdout:2773: done (1 left) 2024-09-17T15:44:44.568 INFO:tasks.rados.rados.0.smithi203.stdout:2774: done (0 left) 2024-09-17T15:44:44.568 INFO:tasks.rados.rados.0.smithi203.stdout:2775: copy_from oid 19 from oid 28 current snap is 300 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2776: read oid 39 snap -1 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2776: expect deleted 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2777: read oid 30 snap 295 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2777: expect deleted 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2778: rollback oid 1 current snap is 300 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 299 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2779: read oid 31 snap -1 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2779: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2780: rollback oid 46 current snap is 300 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 295 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2781: read oid 44 snap -1 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2781: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:44:44.569 INFO:tasks.rados.rados.0.smithi203.stdout:2782: snap_remove snap 298 2024-09-17T15:44:45.565 INFO:tasks.rados.rados.0.smithi203.stdout:2782: done (7 left) 2024-09-17T15:44:45.565 INFO:tasks.rados.rados.0.smithi203.stdout:2783: rollback oid 7 current snap is 300 2024-09-17T15:44:45.565 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 297 2024-09-17T15:44:45.566 INFO:tasks.rados.rados.0.smithi203.stdout:2778: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:44:45.566 INFO:tasks.rados.rados.0.smithi203.stdout:2778: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:44:45.566 INFO:tasks.rados.rados.0.smithi203.stdout:2778: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:44:45.566 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1307 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:44:45.566 INFO:tasks.rados.rados.0.smithi203.stdout:2779: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:44:45.605 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 982001 2024-09-17T15:44:45.661 INFO:tasks.rados.rados.0.smithi203.stdout:2780: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:44:45.667 INFO:tasks.rados.rados.0.smithi203.stdout:2781: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:44:45.667 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 527624 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2780: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2780: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1361 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2775: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2775: finishing copy_from to smithi20328086-19 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1490 (ObjNum 975 snap 295 seq_num 975) dirty exists 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2775: done (7 left) 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2776: done (6 left) 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2777: done (5 left) 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2778: done (4 left) 2024-09-17T15:44:45.725 INFO:tasks.rados.rados.0.smithi203.stdout:2779: done (3 left) 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2780: done (2 left) 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2781: done (1 left) 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2784: snap_create 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2783: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2783: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:2783: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:44:45.726 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1171 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:44:46.580 INFO:tasks.rados.rados.0.smithi203.stdout:2783: done (1 left) 2024-09-17T15:44:46.580 INFO:tasks.rados.rados.0.smithi203.stdout:2784: done (0 left) 2024-09-17T15:44:46.581 INFO:tasks.rados.rados.0.smithi203.stdout:2785: snap_create 2024-09-17T15:44:47.581 INFO:tasks.rados.rados.0.smithi203.stdout:2785: done (0 left) 2024-09-17T15:44:47.581 INFO:tasks.rados.rados.0.smithi203.stdout:2786: snap_create 2024-09-17T15:44:48.589 INFO:tasks.rados.rados.0.smithi203.stdout:2786: done (0 left) 2024-09-17T15:44:48.589 INFO:tasks.rados.rados.0.smithi203.stdout:2787: rollback oid 4 current snap is 303 2024-09-17T15:44:48.589 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 295 2024-09-17T15:44:48.589 INFO:tasks.rados.rados.0.smithi203.stdout:2788: copy_from oid 26 from oid 17 current snap is 303 2024-09-17T15:44:48.590 INFO:tasks.rados.rados.0.smithi203.stdout:2789: read oid 20 snap 296 2024-09-17T15:44:48.590 INFO:tasks.rados.rados.0.smithi203.stdout:2789: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:44:48.590 INFO:tasks.rados.rados.0.smithi203.stdout:2790: snap_create 2024-09-17T15:44:48.590 INFO:tasks.rados.rados.0.smithi203.stdout:2787: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:44:48.590 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:44:48.600 INFO:tasks.rados.rados.0.smithi203.stdout:2789: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:44:48.685 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3128589 2024-09-17T15:44:48.832 INFO:tasks.rados.rados.0.smithi203.stdout:2788: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:44:48.832 INFO:tasks.rados.rados.0.smithi203.stdout:2788: finishing copy_from to smithi20328086-26 2024-09-17T15:44:48.832 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1192 (ObjNum 921 snap 277 seq_num 921) dirty exists 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2787: done (3 left) 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2788: done (2 left) 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2789: done (1 left) 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2790: done (0 left) 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2791: read oid 34 snap -1 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2791: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-17T15:44:49.605 INFO:tasks.rados.rados.0.smithi203.stdout:2792: snap_create 2024-09-17T15:44:49.619 INFO:tasks.rados.rados.0.smithi203.stdout:2791: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-17T15:44:49.640 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1998025 2024-09-17T15:44:49.642 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:49.684 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1106782 2024-09-17T15:44:50.601 INFO:tasks.rados.rados.0.smithi203.stdout:2791: done (1 left) 2024-09-17T15:44:50.601 INFO:tasks.rados.rados.0.smithi203.stdout:2792: done (0 left) 2024-09-17T15:44:50.601 INFO:tasks.rados.rados.0.smithi203.stdout:2793: delete oid 49 current snap is 305 2024-09-17T15:44:50.614 INFO:tasks.rados.rados.0.smithi203.stdout:2793: done (0 left) 2024-09-17T15:44:50.614 INFO:tasks.rados.rados.0.smithi203.stdout:2794: read oid 5 snap -1 2024-09-17T15:44:50.615 INFO:tasks.rados.rados.0.smithi203.stdout:2794: expect (ObjNum 963 snap 290 seq_num 963) 2024-09-17T15:44:50.615 INFO:tasks.rados.rados.0.smithi203.stdout:2795: rollback oid 40 current snap is 305 2024-09-17T15:44:50.615 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 301 2024-09-17T15:44:50.615 INFO:tasks.rados.rados.0.smithi203.stdout:2796: delete oid 16 current snap is 305 2024-09-17T15:44:50.620 INFO:tasks.rados.rados.0.smithi203.stdout:2796: done (2 left) 2024-09-17T15:44:50.620 INFO:tasks.rados.rados.0.smithi203.stdout:2797: snap_remove snap 303 2024-09-17T15:44:51.684 INFO:tasks.rados.rados.0.smithi203.stdout:2797: done (2 left) 2024-09-17T15:44:51.684 INFO:tasks.rados.rados.0.smithi203.stdout:2798: write oid 42 current snap is 305 2024-09-17T15:44:51.684 INFO:tasks.rados.rados.0.smithi203.stdout:2798: seq_num 982 ranges {504474=504055,1714141=631006} 2024-09-17T15:44:51.700 INFO:tasks.rados.rados.0.smithi203.stdout:2798: writing smithi20328086-42 from 504474 to 1008529 tid 1 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:2798: writing smithi20328086-42 from 1714141 to 2345147 tid 2 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:2799: read oid 17 snap -1 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:2799: expect (ObjNum 921 snap 277 seq_num 921) 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:2800: rollback oid 20 current snap is 305 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 304 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:2794: expect (ObjNum 963 snap 290 seq_num 963) 2024-09-17T15:44:51.723 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 331083 2024-09-17T15:44:51.772 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:51.867 INFO:tasks.rados.rados.0.smithi203.stdout:2795: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:44:51.867 INFO:tasks.rados.rados.0.smithi203.stdout:2795: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:44:51.867 INFO:tasks.rados.rados.0.smithi203.stdout:2795: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:44:51.867 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1252 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: finishing write tid 1 to smithi20328086-42 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: finishing write tid 2 to smithi20328086-42 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: finishing write tid 3 to smithi20328086-42 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: finishing write tid 4 to smithi20328086-42 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: oid 42 updating version 0 to 1362 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: oid 42 updating version 1362 to 1363 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: oid 42 updating version 1363 to 1364 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1364 (ObjNum 982 snap 305 seq_num 982) dirty exists 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2798: left oid 42 (ObjNum 982 snap 305 seq_num 982) 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:2799: expect (ObjNum 921 snap 277 seq_num 921) 2024-09-17T15:44:51.868 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2800: finishing rollback tid 0 to smithi20328086-20 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2800: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2800: finishing rollback tid 2 to smithi20328086-20 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1310 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2794: done (4 left) 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2795: done (3 left) 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2798: done (2 left) 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2799: done (1 left) 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2800: done (0 left) 2024-09-17T15:44:52.051 INFO:tasks.rados.rados.0.smithi203.stdout:2801: write (excl) oid 45 current snap is 305 2024-09-17T15:44:52.052 INFO:tasks.rados.rados.0.smithi203.stdout:2801: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-17T15:44:52.069 INFO:tasks.rados.rados.0.smithi203.stdout:2801: writing smithi20328086-45 from 521281 to 1100585 tid 1 2024-09-17T15:44:52.091 INFO:tasks.rados.rados.0.smithi203.stdout:2801: writing smithi20328086-45 from 1656270 to 2430934 tid 2 2024-09-17T15:44:52.091 INFO:tasks.rados.rados.0.smithi203.stdout:2802: copy_from oid 24 from oid 7 current snap is 305 2024-09-17T15:44:52.091 INFO:tasks.rados.rados.0.smithi203.stdout:2801: finishing write tid 1 to smithi20328086-45 2024-09-17T15:44:52.091 INFO:tasks.rados.rados.0.smithi203.stdout:2803: copy_from oid 39 from oid 44 current snap is 305 2024-09-17T15:44:52.091 INFO:tasks.rados.rados.0.smithi203.stdout:2804: copy_from oid 8 from oid 14 current snap is 305 2024-09-17T15:44:52.092 INFO:tasks.rados.rados.0.smithi203.stdout:2805: copy_from oid 46 from oid 42 current snap is 305 2024-09-17T15:44:52.092 INFO:tasks.rados.rados.0.smithi203.stdout:2806: snap_create 2024-09-17T15:44:52.093 INFO:tasks.rados.rados.0.smithi203.stdout:2804: finishing copy_from to smithi20328086-8 2024-09-17T15:44:52.093 INFO:tasks.rados.rados.0.smithi203.stdout:2804: got expected ENOENT (src dne) 2024-09-17T15:44:52.093 INFO:tasks.rados.rados.0.smithi203.stdout:2804: finishing copy_from racing read to smithi20328086-8 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: finishing write tid 2 to smithi20328086-45 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: finishing write tid 3 to smithi20328086-45 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: finishing write tid 4 to smithi20328086-45 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: oid 45 updating version 0 to 1175 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: oid 45 version 1175 is already newer than 1173 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:2801: oid 45 version 1175 is already newer than 1174 2024-09-17T15:44:52.101 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1175 (ObjNum 983 snap 305 seq_num 983) dirty exists 2024-09-17T15:44:52.102 INFO:tasks.rados.rados.0.smithi203.stdout:2801: left oid 45 (ObjNum 983 snap 305 seq_num 983) 2024-09-17T15:44:52.104 INFO:tasks.rados.rados.0.smithi203.stdout:2803: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:44:52.108 INFO:tasks.rados.rados.0.smithi203.stdout:2803: finishing copy_from to smithi20328086-39 2024-09-17T15:44:52.108 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1248 (ObjNum 977 snap 299 seq_num 977) dirty exists 2024-09-17T15:44:52.112 INFO:tasks.rados.rados.0.smithi203.stdout:2802: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:44:52.115 INFO:tasks.rados.rados.0.smithi203.stdout:2805: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:44:52.118 INFO:tasks.rados.rados.0.smithi203.stdout:2805: finishing copy_from to smithi20328086-46 2024-09-17T15:44:52.118 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1364 (ObjNum 982 snap 305 seq_num 982) dirty exists 2024-09-17T15:44:52.128 INFO:tasks.rados.rados.0.smithi203.stdout:2802: finishing copy_from to smithi20328086-24 2024-09-17T15:44:52.129 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1171 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2801: done (5 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2802: done (4 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2803: done (3 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2804: done (2 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2805: done (1 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2806: done (0 left) 2024-09-17T15:44:52.700 INFO:tasks.rados.rados.0.smithi203.stdout:2807: delete oid 31 current snap is 306 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2807: done (0 left) 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2808: rollback oid 18 current snap is 306 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 296 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2809: rollback oid 40 current snap is 306 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 297 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2810: read oid 2 snap 301 2024-09-17T15:44:52.713 INFO:tasks.rados.rados.0.smithi203.stdout:2810: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:44:52.714 INFO:tasks.rados.rados.0.smithi203.stdout:2811: read oid 22 snap -1 2024-09-17T15:44:52.714 INFO:tasks.rados.rados.0.smithi203.stdout:2811: expect (ObjNum 954 snap 289 seq_num 954) 2024-09-17T15:44:52.714 INFO:tasks.rados.rados.0.smithi203.stdout:2812: snap_create 2024-09-17T15:44:52.717 INFO:tasks.rados.rados.0.smithi203.stdout:2810: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:44:52.895 INFO:tasks.rados.rados.0.smithi203.stdout:2811: expect (ObjNum 954 snap 289 seq_num 954) 2024-09-17T15:44:52.948 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1796666 2024-09-17T15:44:52.962 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 734920 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2808: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2808: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2808: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 864 (ObjNum 951 snap 288 seq_num 951) dirty exists 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2809: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2809: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:2809: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:44:52.990 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1258 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:44:53.713 INFO:tasks.rados.rados.0.smithi203.stdout:2808: done (4 left) 2024-09-17T15:44:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:2809: done (3 left) 2024-09-17T15:44:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:2810: done (2 left) 2024-09-17T15:44:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:2811: done (1 left) 2024-09-17T15:44:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:2812: done (0 left) 2024-09-17T15:44:53.714 INFO:tasks.rados.rados.0.smithi203.stdout:2813: snap_create 2024-09-17T15:44:54.727 INFO:tasks.rados.rados.0.smithi203.stdout:2813: done (0 left) 2024-09-17T15:44:54.727 INFO:tasks.rados.rados.0.smithi203.stdout:2814: write (excl) oid 25 current snap is 308 2024-09-17T15:44:54.727 INFO:tasks.rados.rados.0.smithi203.stdout:2814: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-17T15:44:54.748 INFO:tasks.rados.rados.0.smithi203.stdout:2814: writing smithi20328086-25 from 554895 to 1284697 tid 1 2024-09-17T15:44:54.769 INFO:tasks.rados.rados.0.smithi203.stdout:2814: writing smithi20328086-25 from 1773234 to 2551567 tid 2 2024-09-17T15:44:54.769 INFO:tasks.rados.rados.0.smithi203.stdout:2815: snap_create 2024-09-17T15:44:54.769 INFO:tasks.rados.rados.0.smithi203.stdout:2814: finishing write tid 1 to smithi20328086-25 2024-09-17T15:44:54.779 INFO:tasks.rados.rados.0.smithi203.stdout:2814: finishing write tid 2 to smithi20328086-25 2024-09-17T15:44:54.779 INFO:tasks.rados.rados.0.smithi203.stdout:2814: finishing write tid 3 to smithi20328086-25 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:2814: finishing write tid 4 to smithi20328086-25 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:2814: oid 25 updating version 0 to 1182 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:2814: oid 25 updating version 1182 to 1183 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:2814: oid 25 version 1183 is already newer than 1181 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1183 (ObjNum 985 snap 308 seq_num 985) dirty exists 2024-09-17T15:44:54.780 INFO:tasks.rados.rados.0.smithi203.stdout:2814: left oid 25 (ObjNum 985 snap 308 seq_num 985) 2024-09-17T15:44:55.728 INFO:tasks.rados.rados.0.smithi203.stdout:2814: done (1 left) 2024-09-17T15:44:55.728 INFO:tasks.rados.rados.0.smithi203.stdout:2815: done (0 left) 2024-09-17T15:44:55.729 INFO:tasks.rados.rados.0.smithi203.stdout:2816: delete oid 49 current snap is 309 2024-09-17T15:44:55.736 INFO:tasks.rados.rados.0.smithi203.stdout:2816: done (0 left) 2024-09-17T15:44:55.736 INFO:tasks.rados.rados.0.smithi203.stdout:2817: write oid 42 current snap is 309 2024-09-17T15:44:55.736 INFO:tasks.rados.rados.0.smithi203.stdout:2817: seq_num 987 ranges {588509=480300,1773845=782002} 2024-09-17T15:44:55.753 INFO:tasks.rados.rados.0.smithi203.stdout:2817: writing smithi20328086-42 from 588509 to 1068809 tid 1 2024-09-17T15:44:55.778 INFO:tasks.rados.rados.0.smithi203.stdout:2817: writing smithi20328086-42 from 1773845 to 2555847 tid 2 2024-09-17T15:44:55.778 INFO:tasks.rados.rados.0.smithi203.stdout:2818: write oid 37 current snap is 309 2024-09-17T15:44:55.778 INFO:tasks.rados.rados.0.smithi203.stdout:2818: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-17T15:44:55.794 INFO:tasks.rados.rados.0.smithi203.stdout:2818: writing smithi20328086-37 from 605316 to 1160865 tid 1 2024-09-17T15:44:55.808 INFO:tasks.rados.rados.0.smithi203.stdout:2818: writing smithi20328086-37 from 1832327 to 2474340 tid 2 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2819: copy_from oid 38 from oid 26 current snap is 309 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: finishing write tid 1 to smithi20328086-42 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: finishing write tid 2 to smithi20328086-42 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: finishing write tid 3 to smithi20328086-42 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: finishing write tid 4 to smithi20328086-42 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: oid 42 updating version 0 to 1367 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: oid 42 version 1367 is already newer than 1366 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: oid 42 version 1367 is already newer than 1365 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1367 (ObjNum 987 snap 309 seq_num 987) dirty exists 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: left oid 42 (ObjNum 987 snap 309 seq_num 987) 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2818: finishing write tid 1 to smithi20328086-37 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2817: done (2 left) 2024-09-17T15:44:55.809 INFO:tasks.rados.rados.0.smithi203.stdout:2820: delete oid 1 current snap is 309 2024-09-17T15:44:55.823 INFO:tasks.rados.rados.0.smithi203.stdout:2820: done (2 left) 2024-09-17T15:44:55.823 INFO:tasks.rados.rados.0.smithi203.stdout:2821: delete oid 50 current snap is 309 2024-09-17T15:44:55.825 INFO:tasks.rados.rados.0.smithi203.stdout:2818: finishing write tid 2 to smithi20328086-37 2024-09-17T15:44:55.825 INFO:tasks.rados.rados.0.smithi203.stdout:2818: finishing write tid 3 to smithi20328086-37 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:2818: finishing write tid 4 to smithi20328086-37 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:2818: oid 37 updating version 0 to 1493 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:2818: oid 37 version 1493 is already newer than 1491 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:2818: oid 37 version 1493 is already newer than 1492 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1493 (ObjNum 988 snap 309 seq_num 988) dirty exists 2024-09-17T15:44:55.826 INFO:tasks.rados.rados.0.smithi203.stdout:2818: left oid 37 (ObjNum 988 snap 309 seq_num 988) 2024-09-17T15:44:55.835 INFO:tasks.rados.rados.0.smithi203.stdout:2818: done (2 left) 2024-09-17T15:44:55.835 INFO:tasks.rados.rados.0.smithi203.stdout:2821: done (1 left) 2024-09-17T15:44:55.835 INFO:tasks.rados.rados.0.smithi203.stdout:2822: write (excl) oid 11 current snap is 309 2024-09-17T15:44:55.835 INFO:tasks.rados.rados.0.smithi203.stdout:2822: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-17T15:44:55.847 INFO:tasks.rados.rados.0.smithi203.stdout:2822: writing smithi20328086-11 from 655737 to 1153386 tid 1 2024-09-17T15:44:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:2822: writing smithi20328086-11 from 1607773 to 2397113 tid 2 2024-09-17T15:44:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:2823: rollback oid 46 current snap is 309 2024-09-17T15:44:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 294 2024-09-17T15:44:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:2819: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:44:55.862 INFO:tasks.rados.rados.0.smithi203.stdout:2819: finishing copy_from to smithi20328086-38 2024-09-17T15:44:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1192 (ObjNum 921 snap 277 seq_num 921) dirty exists 2024-09-17T15:44:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:2822: finishing write tid 1 to smithi20328086-11 2024-09-17T15:44:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:2819: done (2 left) 2024-09-17T15:44:55.863 INFO:tasks.rados.rados.0.smithi203.stdout:2824: snap_create 2024-09-17T15:44:55.872 INFO:tasks.rados.rados.0.smithi203.stdout:2823: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:44:55.873 INFO:tasks.rados.rados.0.smithi203.stdout:2822: finishing write tid 2 to smithi20328086-11 2024-09-17T15:44:55.874 INFO:tasks.rados.rados.0.smithi203.stdout:2823: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:44:55.874 INFO:tasks.rados.rados.0.smithi203.stdout:2822: finishing write tid 3 to smithi20328086-11 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2823: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1370 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2822: finishing write tid 4 to smithi20328086-11 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2822: oid 11 updating version 0 to 1261 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2822: oid 11 updating version 1261 to 1262 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2822: oid 11 version 1262 is already newer than 1260 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1262 (ObjNum 991 snap 309 seq_num 991) dirty exists 2024-09-17T15:44:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:2822: left oid 11 (ObjNum 991 snap 309 seq_num 991) 2024-09-17T15:44:57.194 INFO:tasks.rados.rados.0.smithi203.stdout:2822: done (2 left) 2024-09-17T15:44:57.194 INFO:tasks.rados.rados.0.smithi203.stdout:2823: done (1 left) 2024-09-17T15:44:57.194 INFO:tasks.rados.rados.0.smithi203.stdout:2824: done (0 left) 2024-09-17T15:44:57.194 INFO:tasks.rados.rados.0.smithi203.stdout:2825: copy_from oid 6 from oid 12 current snap is 310 2024-09-17T15:44:57.194 INFO:tasks.rados.rados.0.smithi203.stdout:2826: snap_remove snap 299 2024-09-17T15:44:57.305 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:44:57.305 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:44:57.306 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:44:57.309 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:44:57.309 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:44:57.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: get_monmap_and_config 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: build_initial_monmap 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: monmap: 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:44:57.446366+0000 2024-09-17T15:44:57.440 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:44:57.446366+0000 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:44:57.441 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 Processor -- start 2024-09-17T15:44:57.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 -- start start 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: init 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-b con 0x7f1bac0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1bac068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1bac13aff0 0x7f1bac13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac13aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bac074470 con 0x7f1bac068db0 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1bac09c8b0 con 0x7f1bac13aff0 2024-09-17T15:44:57.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1bac09cb90 con 0x7f1bac0689d0 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.441+0000 7f1bb4120640 10 monclient(hunting): authenticate will time out at 2030.411858s 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50474/0 (socket says 172.21.15.86:50474) 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/1437961480 learned_addr learned my addr 172.21.15.86:0/1437961480 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): get_auth_request con 0x7f1bac0689d0 auth_method 0 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:57.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb26bd640 1 --2- 172.21.15.86:0/1437961480 >> v2:172.21.15.203:3300/0 conn(0x7f1bac13aff0 0x7f1bac13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb26bd640 10 monclient(hunting): get_auth_request con 0x7f1bac13aff0 auth_method 0 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb26bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb26bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb26bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_done global_id 164492 payload 274 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/1437961480 >> v2:172.21.15.203:3300/0 conn(0x7f1bac13aff0 msgr2=0x7f1bac13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 --2- 172.21.15.86:0/1437961480 >> v2:172.21.15.203:3300/0 conn(0x7f1bac13aff0 0x7f1bac13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/1437961480 >> v1:172.21.15.203:6789/0 conn(0x7f1bac068db0 legacy=0x7f1bac0691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient: _finish_hunting 0 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 monclient: found mon.noname-b 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:57.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:44:57.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.445+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/1437961480 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac09ce70 con 0x7f1bac0689d0 2024-09-17T15:44:57.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/1437961480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 msgr2=0x7f1bac06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:57.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 --2- 172.21.15.86:0/1437961480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8264b30373db8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:57.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 1 -- 172.21.15.86:0/1437961480 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c0b1070 con 0x7f1bac0689d0 2024-09-17T15:44:57.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: dump: 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:57.453 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: _finish_auth 0 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: _check_auth_tickets 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 1 -- 172.21.15.86:0/1437961480 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b9c0ac620 con 0x7f1bac0689d0 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 1 -- 172.21.15.86:0/1437961480 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac09c2d0 con 0x7f1bac0689d0 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 5 monclient: authenticate success, global_id 164492 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: get_monmap_and_config success 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 4 set_mon_vals no callback set 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb0eba640 10 monclient: dump: 2024-09-17T15:44:57.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: shutdown 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- 172.21.15.86:0/1437961480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 msgr2=0x7f1bac06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:57.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- 172.21.15.86:0/1437961480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009870 tx=0x7f1b9c01c420 comp rx=0 tx=0).stop 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- 172.21.15.86:0/1437961480 shutdown_connections 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- 172.21.15.86:0/1437961480 >> v2:172.21.15.203:3300/0 conn(0x7f1bac13aff0 0x7f1bac13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- 172.21.15.86:0/1437961480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac06f840 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- 172.21.15.86:0/1437961480 >> 172.21.15.86:0/1437961480 conn(0x7f1bac0a6440 msgr2=0x7f1bac0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- 172.21.15.86:0/1437961480 shutdown_connections 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- 172.21.15.86:0/1437961480 wait complete. 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: build_initial_monmap 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: monmap: 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:44:57.456 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 Processor -- start 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- start start 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: init 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: _reopen_session rank -1 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-c con 0x7f1bac0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bac068db0 0x7f1bac135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-b con 0x7f1bac068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 0x7f1bac11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): picked mon.noname-a con 0x7f1bac13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac140610 con 0x7f1bac13aff0 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac140490 con 0x7f1bac068db0 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): start opening mon connection 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1bac140790 con 0x7f1bac0689d0 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): _renew_subs 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb4120640 10 monclient(hunting): authenticate will time out at 2030.419371s 2024-09-17T15:44:57.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50488/0 (socket says 172.21.15.86:50488) 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 -- 172.21.15.86:0/3958657640 learned_addr learned my addr 172.21.15.86:0/3958657640 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): get_auth_request con 0x7f1bac0689d0 auth_method 0 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 0x7f1bac11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): get_auth_request con 0x7f1bac13aff0 auth_method 0 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:57.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb16bb640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bac068db0 0x7f1bac135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb16bb640 10 monclient(hunting): get_auth_request con 0x7f1bac068db0 auth_method 0 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb16bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb16bb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb16bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient(hunting): handle_auth_done global_id 184299 payload 824 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 -- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 msgr2=0x7f1bac11ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 -- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bac068db0 msgr2=0x7f1bac135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bac068db0 0x7f1bac135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient: _finish_hunting 0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 monclient: found mon.noname-a 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb26bd640 1 -- 172.21.15.86:0/3958657640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac120470 con 0x7f1bac13aff0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.449+0000 7f1bb1ebc640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb26bd640 1 -- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 msgr2=0x7f1bac11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb26bd640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 0x7f1bac11c8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4007dc0 tx=0x7f1ba400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba40039a0 con 0x7f1bac13aff0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:44:57.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: dump: 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: _finish_auth 0 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:44:57.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba4003b60 con 0x7f1bac13aff0 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4004020 con 0x7f1bac13aff0 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 10 monclient: dump: 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:44:57.463 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1babfff640 4 set_mon_vals no callback set 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 5 monclient: authenticate success, global_id 184299 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _renew_subs 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bac136780 con 0x7f1bac13aff0 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _renew_subs 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:57.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bac120190 con 0x7f1bac13aff0 2024-09-17T15:44:57.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bac06f840 con 0x7f1bac13aff0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.453+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1ba401b020 con 0x7f1bac13aff0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.457+0000 7f1baa7fc640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 0x7f1b9803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.457+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(632..632 src has 438..632) ==== 4739+0+0 (secure 0 0 0) 0x7f1ba4050120 con 0x7f1bac13aff0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.457+0000 7f1bb1ebc640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 0x7f1b9803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.457+0000 7f1bb1ebc640 10 monclient: get_auth_request con 0x7f1b9803bb60 auth_method 0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.457+0000 7f1bb1ebc640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 0x7f1b9803e020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0af040 tx=0x7f1b9c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.461+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1ba401c370 con 0x7f1bac13aff0 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.461+0000 7f1baa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:44:57.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.461+0000 7f1baa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:44:57.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.697+0000 7f1bb4120640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:44:57.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.697+0000 7f1bb4120640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:57.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.697+0000 7f1bb4120640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:44:57.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.697+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1bac11fbd0 con 0x7f1bac13aff0 2024-09-17T15:44:57.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1baa7fc640 1 -- 172.21.15.86:0/3958657640 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1ba4014420 con 0x7f1bac13aff0 2024-09-17T15:44:57.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1baa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:44:57.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1baa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:44:57.698 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:44:57.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 msgr2=0x7f1b9803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:57.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 0x7f1b9803e020 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0af040 tx=0x7f1b9c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 10 monclient: shutdown 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 msgr2=0x7f1bac11c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 0x7f1bac11c8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4007dc0 tx=0x7f1ba400e4b0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 shutdown_connections 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1b9803bb60 0x7f1b9803e020 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bac13aff0 0x7f1bac11c8c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1bac068db0 0x7f1bac135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 --2- 172.21.15.86:0/3958657640 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bac0689d0 0x7f1bac11ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 >> 172.21.15.86:0/3958657640 conn(0x7f1bac0a6440 msgr2=0x7f1bac0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 shutdown_connections 2024-09-17T15:44:57.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:44:57.701+0000 7f1bb4120640 1 -- 172.21.15.86:0/3958657640 wait complete. 2024-09-17T15:44:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:2826: done (1 left) 2024-09-17T15:44:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:2827: write oid 5 current snap is 310 2024-09-17T15:44:58.202 INFO:tasks.rados.rados.0.smithi203.stdout:2825: finishing copy_from to smithi20328086-6 2024-09-17T15:44:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:2825: got expected ENOENT (src dne) 2024-09-17T15:44:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:2825: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:44:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:2827: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-17T15:44:58.224 INFO:tasks.rados.rados.0.smithi203.stdout:2827: writing smithi20328086-5 from 672544 to 1245442 tid 1 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2827: writing smithi20328086-5 from 1666255 to 2199253 tid 2 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2825: done (1 left) 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2828: read oid 10 snap -1 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2828: expect deleted 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2829: snap_create 2024-09-17T15:44:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:2827: finishing write tid 1 to smithi20328086-5 2024-09-17T15:44:58.251 INFO:tasks.rados.rados.0.smithi203.stdout:2827: finishing write tid 2 to smithi20328086-5 2024-09-17T15:44:58.251 INFO:tasks.rados.rados.0.smithi203.stdout:2827: finishing write tid 3 to smithi20328086-5 2024-09-17T15:44:58.251 INFO:tasks.rados.rados.0.smithi203.stdout:2827: finishing write tid 4 to smithi20328086-5 2024-09-17T15:44:58.252 INFO:tasks.rados.rados.0.smithi203.stdout:2827: oid 5 updating version 0 to 1103 2024-09-17T15:44:58.252 INFO:tasks.rados.rados.0.smithi203.stdout:2827: oid 5 version 1103 is already newer than 1102 2024-09-17T15:44:58.252 INFO:tasks.rados.rados.0.smithi203.stdout:2827: oid 5 updating version 1103 to 1104 2024-09-17T15:44:58.252 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1104 (ObjNum 992 snap 310 seq_num 992) dirty exists 2024-09-17T15:44:58.252 INFO:tasks.rados.rados.0.smithi203.stdout:2827: left oid 5 (ObjNum 992 snap 310 seq_num 992) 2024-09-17T15:44:59.209 INFO:tasks.rados.rados.0.smithi203.stdout:2827: done (2 left) 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2828: done (1 left) 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2829: done (0 left) 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2830: read oid 3 snap -1 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2830: expect (ObjNum 946 snap 287 seq_num 946) 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2831: write oid 47 current snap is 311 2024-09-17T15:44:59.210 INFO:tasks.rados.rados.0.smithi203.stdout:2831: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-17T15:44:59.231 INFO:tasks.rados.rados.0.smithi203.stdout:2831: writing smithi20328086-47 from 689351 to 1337498 tid 1 2024-09-17T15:44:59.253 INFO:tasks.rados.rados.0.smithi203.stdout:2831: writing smithi20328086-47 from 2124737 to 2689351 tid 2 2024-09-17T15:44:59.254 INFO:tasks.rados.rados.0.smithi203.stdout:2832: write oid 19 current snap is 311 2024-09-17T15:44:59.254 INFO:tasks.rados.rados.0.smithi203.stdout:2832: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-17T15:44:59.272 INFO:tasks.rados.rados.0.smithi203.stdout:2832: writing smithi20328086-19 from 706158 to 1429554 tid 1 2024-09-17T15:44:59.286 INFO:tasks.rados.rados.0.smithi203.stdout:2832: writing smithi20328086-19 from 2183219 to 2706158 tid 2 2024-09-17T15:44:59.286 INFO:tasks.rados.rados.0.smithi203.stdout:2833: read oid 6 snap -1 2024-09-17T15:44:59.286 INFO:tasks.rados.rados.0.smithi203.stdout:2833: expect deleted 2024-09-17T15:44:59.286 INFO:tasks.rados.rados.0.smithi203.stdout:2834: snap_remove snap 302 2024-09-17T15:45:00.219 INFO:tasks.rados.rados.0.smithi203.stdout:2834: done (4 left) 2024-09-17T15:45:00.219 INFO:tasks.rados.rados.0.smithi203.stdout:2835: read oid 31 snap -1 2024-09-17T15:45:00.219 INFO:tasks.rados.rados.0.smithi203.stdout:2835: expect deleted 2024-09-17T15:45:00.219 INFO:tasks.rados.rados.0.smithi203.stdout:2836: snap_remove snap 310 2024-09-17T15:45:00.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: get_monmap_and_config 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: build_initial_monmap 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: monmap: 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:00.853573+0000 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:00.853573+0000 2024-09-17T15:45:00.847 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:00.848 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 Processor -- start 2024-09-17T15:45:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 -- start start 2024-09-17T15:45:00.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: init 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea1809ba70 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea18068c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181446b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-b con 0x7fea1809eea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1809f2d0 con 0x7fea1809ba70 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea1809dc20 con 0x7fea18068c30 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fea1809d7e0 con 0x7fea1809eea0 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea1e21a640 10 monclient(hunting): authenticate will time out at 2033.819200s 2024-09-17T15:45:00.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea177fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fea1809ba70 0x7fea1809be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34814/0 (socket says 172.21.15.86:34814) 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea177fe640 1 -- 172.21.15.86:0/2908607181 learned_addr learned my addr 172.21.15.86:0/2908607181 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): get_auth_request con 0x7fea1809eea0 auth_method 0 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient(hunting): handle_auth_done global_id 164498 payload 274 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 1 -- 172.21.15.86:0/2908607181 >> v1:172.21.15.203:6789/0 conn(0x7fea18068c30 legacy=0x7fea1809c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 1 -- 172.21.15.86:0/2908607181 >> v1:172.21.15.86:6789/0 conn(0x7fea1809ba70 legacy=0x7fea1809be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:00.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 1 monclient: found mon.noname-b 2024-09-17T15:45:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:00.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.849+0000 7fea17fff640 1 -- 172.21.15.86:0/2908607181 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea18146c10 con 0x7fea1809eea0 2024-09-17T15:45:00.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 -- 172.21.15.86:0/2908607181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 msgr2=0x7fea181446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:00.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 --2- 172.21.15.86:0/2908607181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181446b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fea100097f0 tx=0x7fea1001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43aa06ef90ee954 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:00.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 1 -- 172.21.15.86:0/2908607181 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100b0070 con 0x7fea1809eea0 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: dump: 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:00.856 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 1 -- 172.21.15.86:0/2908607181 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea100ab600 con 0x7fea1809eea0 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 1 -- 172.21.15.86:0/2908607181 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100ab9d0 con 0x7fea1809eea0 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 5 monclient: authenticate success, global_id 164498 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:00.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 4 set_mon_vals no callback set 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea167fc640 10 monclient: dump: 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: shutdown 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- 172.21.15.86:0/2908607181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 msgr2=0x7fea181446b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 --2- 172.21.15.86:0/2908607181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181446b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fea100097f0 tx=0x7fea1001c3a0 comp rx=0 tx=0).stop 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- 172.21.15.86:0/2908607181 shutdown_connections 2024-09-17T15:45:00.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 --2- 172.21.15.86:0/2908607181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181446b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- 172.21.15.86:0/2908607181 >> 172.21.15.86:0/2908607181 conn(0x7fea180a6440 msgr2=0x7fea180a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- 172.21.15.86:0/2908607181 shutdown_connections 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- 172.21.15.86:0/2908607181 wait complete. 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: build_initial_monmap 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: monmap: 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:00.860 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 Processor -- start 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- start start 2024-09-17T15:45:00.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: init 2024-09-17T15:45:00.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:00.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:45:00.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 0x7fea1813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-b con 0x7fea18068c30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 0x7fea1813bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-a con 0x7fea1809ba70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): picked mon.noname-c con 0x7fea1809eea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fea18148060 con 0x7fea1809ba70 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea18147ee0 con 0x7fea18068c30 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fea181481e0 con 0x7fea1809eea0 2024-09-17T15:45:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea1e21a640 10 monclient(hunting): authenticate will time out at 2033.826715s 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 0x7fea1813bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50146/0 (socket says 172.21.15.86:50146) 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 -- 172.21.15.86:0/1015830155 learned_addr learned my addr 172.21.15.86:0/1015830155 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): get_auth_request con 0x7fea1809eea0 auth_method 0 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): get_auth_request con 0x7fea1809ba70 auth_method 0 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:00.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea177fe640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 0x7fea1813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient(hunting): handle_auth_done global_id 164501 payload 824 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 -- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 msgr2=0x7fea1813baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 0x7fea1813baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 -- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 msgr2=0x7fea1813bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 0x7fea1813bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:00.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 monclient: found mon.noname-c 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea17fff640 1 -- 172.21.15.86:0/1015830155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea18125830 con 0x7fea1809eea0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea177fe640 10 monclient: get_auth_request con 0x7fea18068c30 auth_method 0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea177fe640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 0x7fea1813baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.857+0000 7fea16ffd640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 0x7fea1813bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea17fff640 1 -- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 msgr2=0x7fea181413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea17fff640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fea100ae040 tx=0x7fea1001abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100b0050 con 0x7fea1809eea0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:00.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: dump: 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: _finish_auth 0 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:00.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea100bc070 con 0x7fea1809eea0 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea100aac90 con 0x7fea1809eea0 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 10 monclient: dump: 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:00.872 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 5 monclient: authenticate success, global_id 164501 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _renew_subs 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea157fa640 4 set_mon_vals no callback set 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea18125b10 con 0x7fea1809eea0 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _renew_subs 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea18125550 con 0x7fea1809eea0 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:00.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea1809be50 con 0x7fea1809eea0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fea100c1460 con 0x7fea1809eea0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 0x7fe9fc03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(635..635 src has 438..635) ==== 4783+0+0 (secure 0 0 0) 0x7fea100eab30 con 0x7fea1809eea0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea177fe640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 0x7fe9fc03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.861+0000 7fea177fe640 10 monclient: get_auth_request con 0x7fe9fc03bb90 auth_method 0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.865+0000 7fea177fe640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 0x7fe9fc03e050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea00005b50 tx=0x7fea00009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.865+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fea100160b0 con 0x7fea1809eea0 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.865+0000 7fea0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:00.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:00.865+0000 7fea0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea1e21a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea1e21a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea1e21a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:01.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea1806d4f0 con 0x7fea1809eea0 2024-09-17T15:45:01.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea0ffff640 1 -- 172.21.15.86:0/1015830155 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fea100a9e70 con 0x7fea1809eea0 2024-09-17T15:45:01.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:01.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.093+0000 7fea0ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:01.093 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":160,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 msgr2=0x7fe9fc03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 0x7fe9fc03e050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea00005b50 tx=0x7fea00009040 comp rx=0 tx=0).stop 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 10 monclient: shutdown 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 msgr2=0x7fea181413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fea100ae040 tx=0x7fea1001abe0 comp rx=0 tx=0).stop 2024-09-17T15:45:01.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 shutdown_connections 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9fc03bb90 0x7fe9fc03e050 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1809eea0 0x7fea181413f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1809ba70 0x7fea1813bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 --2- 172.21.15.86:0/1015830155 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea18068c30 0x7fea1813baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 >> 172.21.15.86:0/1015830155 conn(0x7fea180a6440 msgr2=0x7fea180a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 shutdown_connections 2024-09-17T15:45:01.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:01.097+0000 7fea1e21a640 1 -- 172.21.15.86:0/1015830155 wait complete. 2024-09-17T15:45:01.220 INFO:tasks.rados.rados.0.smithi203.stdout:2836: done (5 left) 2024-09-17T15:45:01.220 INFO:tasks.rados.rados.0.smithi203.stdout:2837: copy_from oid 5 from oid 26 current snap is 311 2024-09-17T15:45:01.221 INFO:tasks.rados.rados.0.smithi203.stdout:2830: expect (ObjNum 946 snap 287 seq_num 946) 2024-09-17T15:45:01.328 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3780818 2024-09-17T15:45:01.487 INFO:tasks.rados.rados.0.smithi203.stdout:2831: finishing write tid 1 to smithi20328086-47 2024-09-17T15:45:01.487 INFO:tasks.rados.rados.0.smithi203.stdout:2831: finishing write tid 2 to smithi20328086-47 2024-09-17T15:45:01.487 INFO:tasks.rados.rados.0.smithi203.stdout:2831: finishing write tid 3 to smithi20328086-47 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2831: finishing write tid 4 to smithi20328086-47 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2831: oid 47 updating version 0 to 868 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2831: oid 47 updating version 868 to 870 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2831: oid 47 version 870 is already newer than 869 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 870 (ObjNum 993 snap 311 seq_num 993) dirty exists 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2831: left oid 47 (ObjNum 993 snap 311 seq_num 993) 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: finishing write tid 1 to smithi20328086-19 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: finishing write tid 2 to smithi20328086-19 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: finishing write tid 3 to smithi20328086-19 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: finishing write tid 4 to smithi20328086-19 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: oid 19 updating version 0 to 1494 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: oid 19 updating version 1494 to 1495 2024-09-17T15:45:01.488 INFO:tasks.rados.rados.0.smithi203.stdout:2832: oid 19 updating version 1495 to 1496 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1496 (ObjNum 994 snap 311 seq_num 994) dirty exists 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2832: left oid 19 (ObjNum 994 snap 311 seq_num 994) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2837: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2837: finishing copy_from to smithi20328086-5 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1192 (ObjNum 921 snap 277 seq_num 921) dirty exists 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2830: done (5 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2831: done (4 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2832: done (3 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2833: done (2 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2835: done (1 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2837: done (0 left) 2024-09-17T15:45:01.489 INFO:tasks.rados.rados.0.smithi203.stdout:2838: write oid 15 current snap is 311 2024-09-17T15:45:01.490 INFO:tasks.rados.rados.0.smithi203.stdout:2838: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-17T15:45:01.512 INFO:tasks.rados.rados.0.smithi203.stdout:2838: writing smithi20328086-15 from 722965 to 1521610 tid 1 2024-09-17T15:45:01.533 INFO:tasks.rados.rados.0.smithi203.stdout:2838: writing smithi20328086-15 from 2125348 to 2722965 tid 2 2024-09-17T15:45:01.533 INFO:tasks.rados.rados.0.smithi203.stdout:2839: write (excl) oid 12 current snap is 311 2024-09-17T15:45:01.533 INFO:tasks.rados.rados.0.smithi203.stdout:2839: seq_num 996 ranges {739772=590247,1900183=540336} 2024-09-17T15:45:01.550 INFO:tasks.rados.rados.0.smithi203.stdout:2839: writing smithi20328086-12 from 739772 to 1330019 tid 1 2024-09-17T15:45:01.562 INFO:tasks.rados.rados.0.smithi203.stdout:2839: writing smithi20328086-12 from 1900183 to 2440519 tid 2 2024-09-17T15:45:01.562 INFO:tasks.rados.rados.0.smithi203.stdout:2840: snap_remove snap 307 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2840: done (2 left) 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2841: read oid 39 snap -1 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2841: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2842: read oid 28 snap -1 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2842: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:45:02.564 INFO:tasks.rados.rados.0.smithi203.stdout:2843: copy_from oid 47 from oid 49 current snap is 311 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2844: read oid 42 snap -1 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2844: expect (ObjNum 987 snap 309 seq_num 987) 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2845: rollback oid 29 current snap is 311 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 29 to 304 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2846: delete oid 22 current snap is 311 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2838: finishing write tid 1 to smithi20328086-15 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2838: finishing write tid 2 to smithi20328086-15 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2838: finishing write tid 3 to smithi20328086-15 2024-09-17T15:45:02.565 INFO:tasks.rados.rados.0.smithi203.stdout:2838: finishing write tid 4 to smithi20328086-15 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2838: oid 15 updating version 0 to 1282 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2838: oid 15 version 1282 is already newer than 1281 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2838: oid 15 updating version 1282 to 1283 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1283 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2838: left oid 15 (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: finishing write tid 1 to smithi20328086-12 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: finishing write tid 2 to smithi20328086-12 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: finishing write tid 3 to smithi20328086-12 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: finishing write tid 4 to smithi20328086-12 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: oid 12 updating version 0 to 1286 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: oid 12 version 1286 is already newer than 1284 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: oid 12 version 1286 is already newer than 1285 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1286 (ObjNum 996 snap 311 seq_num 996) dirty exists 2024-09-17T15:45:02.566 INFO:tasks.rados.rados.0.smithi203.stdout:2839: left oid 12 (ObjNum 996 snap 311 seq_num 996) 2024-09-17T15:45:02.588 INFO:tasks.rados.rados.0.smithi203.stdout:2842: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:45:02.715 INFO:tasks.rados.rados.0.smithi203.stdout:2841: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:45:02.715 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 168271 2024-09-17T15:45:02.741 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:02.767 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2845: finishing rollback tid 0 to smithi20328086-29 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2843: finishing copy_from to smithi20328086-47 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2843: got expected ENOENT (src dne) 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2843: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2845: finishing rollback tid 1 to smithi20328086-29 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:2844: expect (ObjNum 987 snap 309 seq_num 987) 2024-09-17T15:45:02.794 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 169258 2024-09-17T15:45:02.849 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 267437 2024-09-17T15:45:02.960 INFO:tasks.rados.rados.0.smithi203.stdout:2845: finishing rollback tid 2 to smithi20328086-29 2024-09-17T15:45:02.960 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1186 (ObjNum 966 snap 293 seq_num 966) dirty exists 2024-09-17T15:45:02.960 INFO:tasks.rados.rados.0.smithi203.stdout:2838: done (7 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2839: done (6 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2841: done (5 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2842: done (4 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2843: done (3 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2844: done (2 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2845: done (1 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2846: done (0 left) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2847: copy_from oid 4 from oid 13 current snap is 311 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2848: rollback oid 40 current snap is 311 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 305 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2849: read oid 8 snap -1 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2849: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:45:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:2850: delete oid 45 current snap is 311 2024-09-17T15:45:02.968 INFO:tasks.rados.rados.0.smithi203.stdout:2849: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:45:02.999 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2986408 2024-09-17T15:45:03.009 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 906583 2024-09-17T15:45:03.086 INFO:tasks.rados.rados.0.smithi203.stdout:2848: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:45:03.086 INFO:tasks.rados.rados.0.smithi203.stdout:2848: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:45:03.086 INFO:tasks.rados.rados.0.smithi203.stdout:2848: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1276 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2847: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2847: finishing copy_from to smithi20328086-4 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1132 (ObjNum 969 snap 294 seq_num 969) dirty exists 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2847: done (3 left) 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2848: done (2 left) 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2849: done (1 left) 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2850: done (0 left) 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2851: read oid 1 snap -1 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2851: expect deleted 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2852: rollback oid 2 current snap is 311 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 304 2024-09-17T15:45:03.087 INFO:tasks.rados.rados.0.smithi203.stdout:2853: read oid 46 snap -1 2024-09-17T15:45:03.088 INFO:tasks.rados.rados.0.smithi203.stdout:2853: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:45:03.088 INFO:tasks.rados.rados.0.smithi203.stdout:2854: delete oid 11 current snap is 311 2024-09-17T15:45:03.095 INFO:tasks.rados.rados.0.smithi203.stdout:2852: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:45:03.095 INFO:tasks.rados.rados.0.smithi203.stdout:2851: done (3 left) 2024-09-17T15:45:03.095 INFO:tasks.rados.rados.0.smithi203.stdout:2854: done (2 left) 2024-09-17T15:45:03.095 INFO:tasks.rados.rados.0.smithi203.stdout:2855: write oid 16 current snap is 311 2024-09-17T15:45:03.095 INFO:tasks.rados.rados.0.smithi203.stdout:2855: seq_num 1000 ranges {407000=491243,1617758=547674,2595228=211772} 2024-09-17T15:45:03.100 INFO:tasks.rados.rados.0.smithi203.stdout:2855: writing smithi20328086-16 from 407000 to 898243 tid 1 2024-09-17T15:45:03.117 INFO:tasks.rados.rados.0.smithi203.stdout:2855: writing smithi20328086-16 from 1617758 to 2165432 tid 2 2024-09-17T15:45:03.128 INFO:tasks.rados.rados.0.smithi203.stdout:2855: writing smithi20328086-16 from 2595228 to 2807000 tid 3 2024-09-17T15:45:03.128 INFO:tasks.rados.rados.0.smithi203.stdout:2856: write oid 47 current snap is 311 2024-09-17T15:45:03.128 INFO:tasks.rados.rados.0.smithi203.stdout:2856: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-17T15:45:03.134 INFO:tasks.rados.rados.0.smithi203.stdout:2856: writing smithi20328086-47 from 423807 to 990299 tid 1 2024-09-17T15:45:03.140 INFO:tasks.rados.rados.0.smithi203.stdout:2856: writing smithi20328086-47 from 1676240 to 2083925 tid 2 2024-09-17T15:45:03.144 INFO:tasks.rados.rados.0.smithi203.stdout:2856: writing smithi20328086-47 from 2739004 to 2823807 tid 3 2024-09-17T15:45:03.144 INFO:tasks.rados.rados.0.smithi203.stdout:2857: rollback oid 18 current snap is 311 2024-09-17T15:45:03.144 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 296 2024-09-17T15:45:03.144 INFO:tasks.rados.rados.0.smithi203.stdout:2852: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:45:03.145 INFO:tasks.rados.rados.0.smithi203.stdout:2852: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:45:03.145 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1174 (ObjNum 968 snap 294 seq_num 968) dirty exists 2024-09-17T15:45:03.145 INFO:tasks.rados.rados.0.smithi203.stdout:2853: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:45:03.170 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:03.211 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1048878 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2855: finishing write tid 1 to smithi20328086-16 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2852: done (4 left) 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2853: done (3 left) 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2858: delete oid 33 current snap is 311 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2855: finishing write tid 2 to smithi20328086-16 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2855: finishing write tid 3 to smithi20328086-16 2024-09-17T15:45:03.236 INFO:tasks.rados.rados.0.smithi203.stdout:2855: finishing write tid 4 to smithi20328086-16 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: finishing write tid 5 to smithi20328086-16 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: oid 16 updating version 0 to 1498 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: oid 16 updating version 1498 to 1499 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: oid 16 updating version 1499 to 1500 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: oid 16 version 1500 is already newer than 1497 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1500 (ObjNum 1000 snap 311 seq_num 1000) dirty exists 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2855: left oid 16 (ObjNum 1000 snap 311 seq_num 1000) 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: finishing write tid 1 to smithi20328086-47 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: finishing write tid 2 to smithi20328086-47 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: finishing write tid 3 to smithi20328086-47 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: finishing write tid 4 to smithi20328086-47 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: finishing write tid 5 to smithi20328086-47 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: oid 47 updating version 0 to 882 2024-09-17T15:45:03.237 INFO:tasks.rados.rados.0.smithi203.stdout:2856: oid 47 version 882 is already newer than 879 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2856: oid 47 version 882 is already newer than 881 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2856: oid 47 version 882 is already newer than 880 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 882 (ObjNum 1001 snap 311 seq_num 1001) dirty exists 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2856: left oid 47 (ObjNum 1001 snap 311 seq_num 1001) 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2857: finishing rollback tid 0 to smithi20328086-18 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2857: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:2857: finishing rollback tid 2 to smithi20328086-18 2024-09-17T15:45:03.238 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 886 (ObjNum 951 snap 288 seq_num 951) dirty exists 2024-09-17T15:45:03.242 INFO:tasks.rados.rados.0.smithi203.stdout:2855: done (3 left) 2024-09-17T15:45:03.242 INFO:tasks.rados.rados.0.smithi203.stdout:2856: done (2 left) 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2857: done (1 left) 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2858: done (0 left) 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2859: read oid 45 snap -1 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2859: expect deleted 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2860: copy_from oid 22 from oid 12 current snap is 311 2024-09-17T15:45:03.243 INFO:tasks.rados.rados.0.smithi203.stdout:2861: snap_create 2024-09-17T15:45:03.259 INFO:tasks.rados.rados.0.smithi203.stdout:2860: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:45:03.261 INFO:tasks.rados.rados.0.smithi203.stdout:2860: finishing copy_from to smithi20328086-22 2024-09-17T15:45:03.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1286 (ObjNum 996 snap 311 seq_num 996) dirty exists 2024-09-17T15:45:03.578 INFO:tasks.rados.rados.0.smithi203.stdout:2859: done (2 left) 2024-09-17T15:45:03.578 INFO:tasks.rados.rados.0.smithi203.stdout:2860: done (1 left) 2024-09-17T15:45:03.578 INFO:tasks.rados.rados.0.smithi203.stdout:2861: done (0 left) 2024-09-17T15:45:03.578 INFO:tasks.rados.rados.0.smithi203.stdout:2862: write oid 31 current snap is 312 2024-09-17T15:45:03.578 INFO:tasks.rados.rados.0.smithi203.stdout:2862: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-17T15:45:03.594 INFO:tasks.rados.rados.0.smithi203.stdout:2862: writing smithi20328086-31 from 457421 to 1174411 tid 1 2024-09-17T15:45:03.607 INFO:tasks.rados.rados.0.smithi203.stdout:2862: writing smithi20328086-31 from 1676851 to 2088205 tid 2 2024-09-17T15:45:03.618 INFO:tasks.rados.rados.0.smithi203.stdout:2862: writing smithi20328086-31 from 2677497 to 2857421 tid 3 2024-09-17T15:45:03.618 INFO:tasks.rados.rados.0.smithi203.stdout:2863: write (excl) oid 44 current snap is 312 2024-09-17T15:45:03.619 INFO:tasks.rados.rados.0.smithi203.stdout:2863: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-17T15:45:03.631 INFO:tasks.rados.rados.0.smithi203.stdout:2863: writing smithi20328086-44 from 474228 to 982820 tid 1 2024-09-17T15:45:03.642 INFO:tasks.rados.rados.0.smithi203.stdout:2863: writing smithi20328086-44 from 1451686 to 2006698 tid 2 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2863: writing smithi20328086-44 from 2704920 to 2874228 tid 3 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2864: delete oid 16 current snap is 312 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2862: finishing write tid 1 to smithi20328086-31 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2862: finishing write tid 2 to smithi20328086-31 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2862: finishing write tid 3 to smithi20328086-31 2024-09-17T15:45:03.651 INFO:tasks.rados.rados.0.smithi203.stdout:2862: finishing write tid 4 to smithi20328086-31 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: finishing write tid 5 to smithi20328086-31 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: oid 31 updating version 0 to 1288 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: oid 31 updating version 1288 to 1290 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: oid 31 version 1290 is already newer than 1289 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: oid 31 version 1290 is already newer than 1287 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1290 (ObjNum 1003 snap 312 seq_num 1003) dirty exists 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2862: left oid 31 (ObjNum 1003 snap 312 seq_num 1003) 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2863: finishing write tid 1 to smithi20328086-44 2024-09-17T15:45:03.652 INFO:tasks.rados.rados.0.smithi203.stdout:2863: finishing write tid 2 to smithi20328086-44 2024-09-17T15:45:03.657 INFO:tasks.rados.rados.0.smithi203.stdout:2863: finishing write tid 3 to smithi20328086-44 2024-09-17T15:45:03.657 INFO:tasks.rados.rados.0.smithi203.stdout:2863: finishing write tid 4 to smithi20328086-44 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: finishing write tid 5 to smithi20328086-44 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: oid 44 updating version 0 to 1281 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: oid 44 updating version 1281 to 1283 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: oid 44 version 1283 is already newer than 1282 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: oid 44 version 1283 is already newer than 1280 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1283 (ObjNum 1004 snap 312 seq_num 1004) dirty exists 2024-09-17T15:45:03.658 INFO:tasks.rados.rados.0.smithi203.stdout:2863: left oid 44 (ObjNum 1004 snap 312 seq_num 1004) 2024-09-17T15:45:03.662 INFO:tasks.rados.rados.0.smithi203.stdout:2862: done (2 left) 2024-09-17T15:45:03.662 INFO:tasks.rados.rados.0.smithi203.stdout:2863: done (1 left) 2024-09-17T15:45:03.662 INFO:tasks.rados.rados.0.smithi203.stdout:2864: done (0 left) 2024-09-17T15:45:03.663 INFO:tasks.rados.rados.0.smithi203.stdout:2865: snap_create 2024-09-17T15:45:04.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:04.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.241+0000 7f3d2486c640 10 monclient: get_monmap_and_config 2024-09-17T15:45:04.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.241+0000 7f3d2486c640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.241+0000 7f3d2486c640 10 monclient: monmap: 2024-09-17T15:45:04.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:04.247069+0000 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:04.247069+0000 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:04.241 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:04.242 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:04.242 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:04.242 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 Processor -- start 2024-09-17T15:45:04.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 -- start start 2024-09-17T15:45:04.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient: init 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d200689d0 0x7f3d2006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d200689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d20068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 0x7f3d2013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d2013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2009c8b0 con 0x7f3d20068db0 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3d20074500 con 0x7f3d200689d0 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2009cb90 con 0x7f3d2013aff0 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d2486c640 10 monclient(hunting): authenticate will time out at 2037.213022s 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d200689d0 0x7f3d2006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1dd9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3d20068db0 0x7f3d20069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34822/0 (socket says 172.21.15.86:34822) 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d200689d0 0x7f3d2006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60724/0 (socket says 172.21.15.86:60724) 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1dd9b640 1 -- 172.21.15.86:0/3476581567 learned_addr learned my addr 172.21.15.86:0/3476581567 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): get_auth_request con 0x7f3d200689d0 auth_method 0 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1ed9d640 1 --2- 172.21.15.86:0/3476581567 >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 0x7f3d2013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_done global_id 184305 payload 274 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 -- 172.21.15.86:0/3476581567 >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 msgr2=0x7f3d2013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 --2- 172.21.15.86:0/3476581567 >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 0x7f3d2013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 -- 172.21.15.86:0/3476581567 >> v1:172.21.15.86:6789/0 conn(0x7f3d20068db0 legacy=0x7f3d20069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 monclient: found mon.noname-a 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1e59c640 1 -- 172.21.15.86:0/3476581567 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2009ce70 con 0x7f3d200689d0 2024-09-17T15:45:04.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1ed9d640 10 monclient: get_auth_request con 0x7f3d2013aff0 auth_method 0 2024-09-17T15:45:04.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.245+0000 7f3d1ed9d640 1 --2- 172.21.15.86:0/3476581567 >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 0x7f3d2013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1e59c640 1 -- 172.21.15.86:0/3476581567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d200689d0 msgr2=0x7f3d2006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1e59c640 1 --2- 172.21.15.86:0/3476581567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d200689d0 0x7f3d2006f840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3d20069a70 tx=0x7f3d1001c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2463caf315b3abde server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 1 -- 172.21.15.86:0/3476581567 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d100b0070 con 0x7f3d200689d0 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:04.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: dump: 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 1 -- 172.21.15.86:0/3476581567 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d100ab600 con 0x7f3d200689d0 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 5 monclient: authenticate success, global_id 184305 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 1 -- 172.21.15.86:0/3476581567 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2009c2d0 con 0x7f3d200689d0 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 4 set_mon_vals no callback set 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d1d59a640 10 monclient: dump: 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 10 monclient: shutdown 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- 172.21.15.86:0/3476581567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d200689d0 msgr2=0x7f3d2006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 --2- 172.21.15.86:0/3476581567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d200689d0 0x7f3d2006f840 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3d20069a70 tx=0x7f3d1001c300 comp rx=0 tx=0).stop 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- 172.21.15.86:0/3476581567 shutdown_connections 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 --2- 172.21.15.86:0/3476581567 >> v2:172.21.15.203:3300/0 conn(0x7f3d2013aff0 0x7f3d2013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 --2- 172.21.15.86:0/3476581567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d200689d0 0x7f3d2006f840 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- 172.21.15.86:0/3476581567 >> 172.21.15.86:0/3476581567 conn(0x7f3d200a6440 msgr2=0x7f3d200a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- 172.21.15.86:0/3476581567 shutdown_connections 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- 172.21.15.86:0/3476581567 wait complete. 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 10 monclient: monmap: 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 Processor -- start 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 1 -- start start 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.249+0000 7f3d2486c640 10 monclient: init 2024-09-17T15:45:04.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 0x7f3d201cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d200689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d20068db0 0x7f3d20135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d20068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d2013aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201405c0 con 0x7f3d2013aff0 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d20140440 con 0x7f3d20068db0 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3d20140740 con 0x7f3d200689d0 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient(hunting): authenticate will time out at 2037.220277s 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 0x7f3d201cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60738/0 (socket says 172.21.15.86:60738) 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 0x7f3d201cff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50162/0 (socket says 172.21.15.86:50162) 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 -- 172.21.15.86:0/1352740700 learned_addr learned my addr 172.21.15.86:0/1352740700 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): get_auth_request con 0x7f3d2013aff0 auth_method 0 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): get_auth_request con 0x7f3d200689d0 auth_method 0 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1dd9b640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d20068db0 0x7f3d20135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1dd9b640 10 monclient(hunting): get_auth_request con 0x7f3d20068db0 auth_method 0 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient(hunting): handle_auth_done global_id 184308 payload 824 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 -- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 msgr2=0x7f3d201cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 0x7f3d201cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 -- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d20068db0 msgr2=0x7f3d20135fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d20068db0 0x7f3d20135fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 -- 172.21.15.86:0/1352740700 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2009f170 con 0x7f3d2013aff0 2024-09-17T15:45:04.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 -- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 msgr2=0x7f3d20136520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1ed9d640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800cef0 tx=0x7f3d1800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1801a070 con 0x7f3d2013aff0 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: dump: 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1800da70 con 0x7f3d2013aff0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 5 monclient: authenticate success, global_id 184308 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _renew_subs 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2009d460 con 0x7f3d2013aff0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800de60 con 0x7f3d2013aff0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d0effd640 10 monclient: dump: 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d1cd99640 4 set_mon_vals no callback set 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _renew_subs 2024-09-17T15:45:04.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.253+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d20122a90 con 0x7f3d2013aff0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3d1801d070 con 0x7f3d2013aff0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d2486c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d2486c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d2486c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2006f840 con 0x7f3d2013aff0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d0effd640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 0x7f3d0003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(639..639 src has 438..639) ==== 4771+0+0 (secure 0 0 0) 0x7f3d180508f0 con 0x7f3d2013aff0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d1e59c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 0x7f3d0003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d1e59c640 10 monclient: get_auth_request con 0x7f3d0003bbb0 auth_method 0 2024-09-17T15:45:04.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.257+0000 7f3d1e59c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 0x7f3d0003e070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3d100ab000 tx=0x7f3d10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.261+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3d1804f2f0 con 0x7f3d2013aff0 2024-09-17T15:45:04.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.261+0000 7f3d0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.261+0000 7f3d0effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d2486c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d2486c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d2486c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3d200a29d0 con 0x7f3d2013aff0 2024-09-17T15:45:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d0effd640 1 -- 172.21.15.86:0/1352740700 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f3d1801ed90 con 0x7f3d2013aff0 2024-09-17T15:45:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:04.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.493+0000 7f3d0effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:04.492 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 msgr2=0x7f3d0003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 0x7f3d0003e070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f3d100ab000 tx=0x7f3d10008040 comp rx=0 tx=0).stop 2024-09-17T15:45:04.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 10 monclient: shutdown 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 msgr2=0x7f3d20136520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800cef0 tx=0x7f3d1800a710 comp rx=0 tx=0).stop 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 shutdown_connections 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3d0003bbb0 0x7f3d0003e070 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d2013aff0 0x7f3d20136520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d20068db0 0x7f3d20135fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 --2- 172.21.15.86:0/1352740700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d200689d0 0x7f3d201cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 >> 172.21.15.86:0/1352740700 conn(0x7f3d200a6440 msgr2=0x7f3d200a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 shutdown_connections 2024-09-17T15:45:04.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.497+0000 7f3d2486c640 1 -- 172.21.15.86:0/1352740700 wait complete. 2024-09-17T15:45:04.510 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:45:04.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:45:04.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:45:04.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: get_monmap_and_config 2024-09-17T15:45:04.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: monmap: 2024-09-17T15:45:04.633 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:04.640153+0000 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:04.640153+0000 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:04.634 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 Processor -- start 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 -- start start 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: init 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee640a50e0 0x7fee6409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-a con 0x7fee640a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 0x7fee640a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-b con 0x7fee640a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee64137ea0 0x7fee641382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-c con 0x7fee64137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fee640a6280 con 0x7fee640a50e0 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fee6409c0a0 con 0x7fee64137ea0 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fee6409c380 con 0x7fee640a54c0 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee6b923640 10 monclient(hunting): authenticate will time out at 2037.605692s 2024-09-17T15:45:04.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee640a50e0 0x7fee6409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 0x7fee640a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee640a50e0 0x7fee6409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60752/0 (socket says 172.21.15.86:60752) 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 -- 172.21.15.86:0/3223874633 learned_addr learned my addr 172.21.15.86:0/3223874633 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): get_auth_request con 0x7fee640a50e0 auth_method 0 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): get_auth_request con 0x7fee640a54c0 auth_method 0 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee69ec0640 1 --2- 172.21.15.86:0/3223874633 >> v2:172.21.15.203:3300/0 conn(0x7fee64137ea0 0x7fee641382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee69ec0640 10 monclient(hunting): get_auth_request con 0x7fee64137ea0 auth_method 0 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee69ec0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee69ec0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee69ec0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient(hunting): handle_auth_done global_id 184311 payload 274 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 -- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 msgr2=0x7fee640a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 0x7fee640a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 -- 172.21.15.86:0/3223874633 >> v2:172.21.15.203:3300/0 conn(0x7fee64137ea0 msgr2=0x7fee641382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 --2- 172.21.15.86:0/3223874633 >> v2:172.21.15.203:3300/0 conn(0x7fee64137ea0 0x7fee641382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 monclient: found mon.noname-a 2024-09-17T15:45:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee696bf640 1 -- 172.21.15.86:0/3223874633 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6409c660 con 0x7fee640a50e0 2024-09-17T15:45:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.637+0000 7fee68ebe640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 0x7fee640a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:04.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee696bf640 1 -- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 msgr2=0x7fee6409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee696bf640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee6409cd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee6406f8c0 tx=0x7fee6001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1d075d07296661 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 1 -- 172.21.15.86:0/3223874633 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600b2070 con 0x7fee640a50e0 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: dump: 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.647 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 1 -- 172.21.15.86:0/3223874633 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee600a9070 con 0x7fee640a50e0 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 5 monclient: authenticate success, global_id 184311 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 1 -- 172.21.15.86:0/3223874633 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600ad880 con 0x7fee640a50e0 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 4 set_mon_vals no callback set 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee5bfff640 10 monclient: dump: 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 10 monclient: shutdown 2024-09-17T15:45:04.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 -- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 msgr2=0x7fee6409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee6409cd60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fee6406f8c0 tx=0x7fee6001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 -- 172.21.15.86:0/3223874633 shutdown_connections 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 --2- 172.21.15.86:0/3223874633 >> v2:172.21.15.203:3300/0 conn(0x7fee64137ea0 0x7fee641382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee640a54c0 0x7fee640a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 --2- 172.21.15.86:0/3223874633 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee6409cd60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 -- 172.21.15.86:0/3223874633 >> 172.21.15.86:0/3223874633 conn(0x7fee640aaae0 msgr2=0x7fee640aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 -- 172.21.15.86:0/3223874633 shutdown_connections 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 1 -- 172.21.15.86:0/3223874633 wait complete. 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.641+0000 7fee6b923640 10 monclient: monmap: 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:04.650 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 Processor -- start 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- start start 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: init 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee641cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-a con 0x7fee640a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:04.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee640a54c0 0x7fee64136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-b con 0x7fee640a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee64137ea0 0x7fee641373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): picked mon.noname-c con 0x7fee64137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee640a6ae0 con 0x7fee640a50e0 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee640a6960 con 0x7fee640a54c0 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fee64141b20 con 0x7fee64137ea0 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient(hunting): authenticate will time out at 2037.612753s 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee641cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee64137ea0 0x7fee641373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee641cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60762/0 (socket says 172.21.15.86:60762) 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 -- 172.21.15.86:0/966968236 learned_addr learned my addr 172.21.15.86:0/966968236 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): get_auth_request con 0x7fee640a50e0 auth_method 0 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): get_auth_request con 0x7fee64137ea0 auth_method 0 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee640a54c0 0x7fee64136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient(hunting): handle_auth_done global_id 184314 payload 824 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee64137ea0 msgr2=0x7fee641373a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee64137ea0 0x7fee641373a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee640a54c0 msgr2=0x7fee64136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee640a54c0 0x7fee64136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 monclient: found mon.noname-a 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee64122900 con 0x7fee640a50e0 2024-09-17T15:45:04.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee68ebe640 10 monclient: get_auth_request con 0x7fee640a54c0 auth_method 0 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee640a54c0 0x7fee64136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee69ec0640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee64137ea0 0x7fee641373a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 msgr2=0x7fee641cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee696bf640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee641cfd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fee6001cc80 tx=0x7fee60019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600bc070 con 0x7fee640a50e0 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: dump: 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.655 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee600b3040 con 0x7fee640a50e0 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee600b2070 con 0x7fee640a50e0 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee59ffb640 10 monclient: dump: 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 5 monclient: authenticate success, global_id 184314 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _renew_subs 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee5b7fe640 4 set_mon_vals no callback set 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee640a45e0 con 0x7fee640a50e0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _renew_subs 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.645+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6413eb60 con 0x7fee640a50e0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee6b923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee6b923640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 0x7fee6413d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee6406a0e0 con 0x7fee6413d0a0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 0x7fee6413d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient: get_auth_request con 0x7fee6413d0a0 auth_method 0 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fee600bf070 con 0x7fee640a50e0 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 10 monclient(hunting): handle_auth_done global_id 184317 payload 824 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee640a58f0 con 0x7fee6413d0a0 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee4803bbb0 0x7fee4803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(639..639 src has 438..639) ==== 4771+0+0 (secure 0 0 0) 0x7fee600e93f0 con 0x7fee640a50e0 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee69ec0640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee4803bbb0 0x7fee4803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee69ec0640 10 monclient: get_auth_request con 0x7fee4803bbb0 auth_method 0 2024-09-17T15:45:04.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 msgr2=0x7fee6413d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 0x7fee6413d480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fee6413e710 tx=0x7fee54010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5400f040 con 0x7fee6413d0a0 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 10 monclient: dump: 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.662 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fee5401c400 con 0x7fee6413d0a0 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 msgr2=0x7fee6413d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 0x7fee6413d480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fee6413e710 tx=0x7fee54010c20 comp rx=0 tx=0).stop 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 msgr2=0x7fee6413d480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.649+0000 7fee59ffb640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d0a0 0x7fee6413d480 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fee6413e710 tx=0x7fee54010c20 comp rx=0 tx=0).stop 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.653+0000 7fee69ec0640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee4803bbb0 0x7fee4803e070 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fee6413ef90 tx=0x7fee5c009820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee6b923640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee6b923640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 0x7fee641392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee6b923640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fee64074be0 con 0x7fee6413d9c0 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 0x7fee641392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient: get_auth_request con 0x7fee6413d9c0 auth_method 0 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 10 monclient(hunting): handle_auth_done global_id 184320 payload 824 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.657+0000 7fee68ebe640 1 -- 172.21.15.86:0/966968236 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee6413dda0 con 0x7fee6413d9c0 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee68ebe640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 msgr2=0x7fee641392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee68ebe640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 0x7fee641392d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fee5401b000 tx=0x7fee540046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee54005dc0 con 0x7fee6413d9c0 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 10 monclient: dump: 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.665 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fee54025070 con 0x7fee6413d9c0 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 msgr2=0x7fee641392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 0x7fee641392d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fee5401b000 tx=0x7fee540046d0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 msgr2=0x7fee641392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee59ffb640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee6413d9c0 0x7fee641392d0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fee5401b000 tx=0x7fee540046d0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:04.666 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 108154, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:04.667 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:04.668 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:04.670 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.671 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.672 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.673 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:04.674 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.675 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:04.676 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.677 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.679 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:04.679 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee4803bbb0 msgr2=0x7fee4803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fee4803bbb0 0x7fee4803e070 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fee6413ef90 tx=0x7fee5c009820 comp rx=0 tx=0).stop 2024-09-17T15:45:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 10 monclient: shutdown 2024-09-17T15:45:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 msgr2=0x7fee641cfd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 --2- 172.21.15.86:0/966968236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee640a50e0 0x7fee641cfd70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fee6001cc80 tx=0x7fee60019180 comp rx=0 tx=0).stop 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee69ec0640 1 -- 172.21.15.86:0/966968236 reap_dead start 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee69ec0640 1 -- 172.21.15.86:0/966968236 reap_dead start 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 shutdown_connections 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 >> 172.21.15.86:0/966968236 conn(0x7fee640aaae0 msgr2=0x7fee640a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 shutdown_connections 2024-09-17T15:45:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.661+0000 7fee6b923640 1 -- 172.21.15.86:0/966968236 wait complete. 2024-09-17T15:45:04.681 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:04.681 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:45:04.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:45:04.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 10 monclient: get_monmap_and_config 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 10 monclient: monmap: 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:04.806585+0000 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:04.806585+0000 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:04.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:04.801 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:04.801 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:04.801 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 1 Processor -- start 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 1 -- start start 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.801+0000 7f28db788640 10 monclient: init 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): picked mon.noname-b con 0x7f28d409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 0x7f28d4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): picked mon.noname-c con 0x7f28d409e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28d4134f80 0x7f28d4135360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): picked mon.noname-a con 0x7f28d4134f80 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:04.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f28d409e7f0 con 0x7f28d4134f80 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f28d41358a0 con 0x7f28d409e090 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f28d40a2110 con 0x7f28d409d770 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28db788640 10 monclient(hunting): authenticate will time out at 2037.772535s 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28d4134f80 0x7f28d4135360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f28d4134f80 0x7f28d4135360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60800/0 (socket says 172.21.15.86:60800) 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 -- 172.21.15.86:0/3069983895 learned_addr learned my addr 172.21.15.86:0/3069983895 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): get_auth_request con 0x7f28d4134f80 auth_method 0 2024-09-17T15:45:04.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): get_auth_request con 0x7f28d409d770 auth_method 0 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 1 --2- 172.21.15.86:0/3069983895 >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 0x7f28d4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 10 monclient(hunting): get_auth_request con 0x7f28d409e090 auth_method 0 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient(hunting): handle_auth_done global_id 184323 payload 274 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 -- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 msgr2=0x7f28d409db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d409db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 -- 172.21.15.86:0/3069983895 >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 msgr2=0x7f28d4134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 --2- 172.21.15.86:0/3069983895 >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 0x7f28d4134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 monclient: found mon.noname-a 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:04.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9d25640 1 -- 172.21.15.86:0/3069983895 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d409ff00 con 0x7f28d4134f80 2024-09-17T15:45:04.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d9524640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d409db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:04.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.805+0000 7f28d8d23640 1 --2- 172.21.15.86:0/3069983895 >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 0x7f28d4134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28d9d25640 1 -- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 msgr2=0x7f28d4135360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28d9d25640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4135360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28d406e2e0 tx=0x7f28cc01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed564fe5c603e01 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 1 -- 172.21.15.86:0/3069983895 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0a5620 con 0x7f28d4134f80 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:04.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: dump: 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 5 monclient: authenticate success, global_id 184323 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 1 -- 172.21.15.86:0/3069983895 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28cc0a57c0 con 0x7f28d4134f80 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 1 -- 172.21.15.86:0/3069983895 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0a5b90 con 0x7f28d4134f80 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 4 set_mon_vals no callback set 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28cbfff640 10 monclient: dump: 2024-09-17T15:45:04.818 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: shutdown 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 msgr2=0x7f28d4135360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4135360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28d406e2e0 tx=0x7f28cc01c8b0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- 172.21.15.86:0/3069983895 shutdown_connections 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4135360 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- 172.21.15.86:0/3069983895 >> v2:172.21.15.203:3300/0 conn(0x7f28d409e090 0x7f28d4134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- 172.21.15.86:0/3069983895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d409db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- 172.21.15.86:0/3069983895 >> 172.21.15.86:0/3069983895 conn(0x7f28d40a2f40 msgr2=0x7f28d40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- 172.21.15.86:0/3069983895 shutdown_connections 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- 172.21.15.86:0/3069983895 wait complete. 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: monmap: 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:04.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 Processor -- start 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- start start 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: init 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d413acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): picked mon.noname-c con 0x7f28d409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28d409e090 0x7f28d4133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): picked mon.noname-b con 0x7f28d409e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): picked mon.noname-a con 0x7f28d4134f80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f28d409ce60 con 0x7f28d4134f80 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f28d409cce0 con 0x7f28d409e090 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f28d40a1480 con 0x7f28d409d770 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28db788640 10 monclient(hunting): authenticate will time out at 2037.779873s 2024-09-17T15:45:04.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.809+0000 7f28d9524640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d413acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d413acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50202/0 (socket says 172.21.15.86:50202) 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 -- 172.21.15.86:0/1152261445 learned_addr learned my addr 172.21.15.86:0/1152261445 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): get_auth_request con 0x7f28d409d770 auth_method 0 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): get_auth_request con 0x7f28d4134f80 auth_method 0 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28d409e090 0x7f28d4133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): get_auth_request con 0x7f28d409e090 auth_method 0 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient(hunting): handle_auth_done global_id 164516 payload 824 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28d409e090 msgr2=0x7f28d4133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28d409e090 0x7f28d4133ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 msgr2=0x7f28d4134410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4134410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 monclient: found mon.noname-c 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d409c680 con 0x7f28d409d770 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9d25640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f28d4134f80 0x7f28d4134410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 msgr2=0x7f28d413acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d9524640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d413acf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f28d000ed40 tx=0x7f28d000d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d0010a60 con 0x7f28d409d770 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: dump: 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28d0010c00 con 0x7f28d409d770 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 5 monclient: authenticate success, global_id 164516 2024-09-17T15:45:04.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 monclient: _renew_subs 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d4134950 con 0x7f28d409d770 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d000de30 con 0x7f28d409d770 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 10 monclient: dump: 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.827 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28cb7fe640 4 set_mon_vals no callback set 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 monclient: _renew_subs 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d4123010 con 0x7f28d409d770 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 0x7f28d4121490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f28d4123480 con 0x7f28d41220b0 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 0x7f28d4121490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient: get_auth_request con 0x7f28d41220b0 auth_method 0 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.813+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f28d000f3d0 con 0x7f28d409d770 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d8d23640 10 monclient(hunting): handle_auth_done global_id 164519 payload 824 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d8d23640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28d40a1800 con 0x7f28d41220b0 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f28b803bb60 0x7f28b803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(639..639 src has 438..639) ==== 4771+0+0 (secure 0 0 0) 0x7f28d00505f0 con 0x7f28d409d770 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d9d25640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f28b803bb60 0x7f28b803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d9d25640 10 monclient: get_auth_request con 0x7f28b803bb60 auth_method 0 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d8d23640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 msgr2=0x7f28d4121490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d8d23640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 0x7f28d4121490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f28d413bee0 tx=0x7f28c4010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401a030 con 0x7f28d41220b0 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 10 monclient: dump: 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f28c4004020 con 0x7f28d41220b0 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 msgr2=0x7f28d4121490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 0x7f28d4121490 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f28d413bee0 tx=0x7f28c4010c20 comp rx=0 tx=0).stop 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 msgr2=0x7f28d4121490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:04.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28c9ffb640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d41220b0 0x7f28d4121490 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.817+0000 7f28d9d25640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f28b803bb60 0x7f28b803e020 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f28d409f3d0 tx=0x7f28cc0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28db788640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28db788640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 0x7f28d402d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28db788640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f28d4123970 con 0x7f28d402d450 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 0x7f28d402d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient: get_auth_request con 0x7f28d402d450 auth_method 0 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 10 monclient(hunting): handle_auth_done global_id 164522 payload 824 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 1 -- 172.21.15.86:0/1152261445 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f28d41375f0 con 0x7f28d402d450 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 msgr2=0x7f28d402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28d8d23640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 0x7f28d402d830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f28d413bee0 tx=0x7f28c40046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c400f040 con 0x7f28d402d450 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:04.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 10 monclient: dump: 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f28c4003620 con 0x7f28d402d450 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:04.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 msgr2=0x7f28d402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 0x7f28d402d830 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f28d413bee0 tx=0x7f28c40046d0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 msgr2=0x7f28d402d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.825+0000 7f28c9ffb640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d402d450 0x7f28d402d830 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f28d413bee0 tx=0x7f28c40046d0 comp rx=0 tx=0).stop 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 208013, 2024-09-17T15:45:04.836 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.837 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.838 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:04.839 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:04.840 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:04.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.842 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.843 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:04.844 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.846 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.847 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:04.848 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:04.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:04.849 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:04.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:04.849 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:04.849 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:04.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f28b803bb60 msgr2=0x7f28b803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f28b803bb60 0x7f28b803e020 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f28d409f3d0 tx=0x7f28cc0a7070 comp rx=0 tx=0).stop 2024-09-17T15:45:04.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 10 monclient: shutdown 2024-09-17T15:45:04.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:04.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 msgr2=0x7f28d413acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 --2- 172.21.15.86:0/1152261445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f28d409d770 0x7f28d413acf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f28d000ed40 tx=0x7f28d000d720 comp rx=0 tx=0).stop 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28d9d25640 1 -- 172.21.15.86:0/1152261445 reap_dead start 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28d9d25640 1 -- 172.21.15.86:0/1152261445 reap_dead start 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 shutdown_connections 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 >> 172.21.15.86:0/1152261445 conn(0x7f28d40a2f40 msgr2=0x7f28d40a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 shutdown_connections 2024-09-17T15:45:04.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.829+0000 7f28db788640 1 -- 172.21.15.86:0/1152261445 wait complete. 2024-09-17T15:45:04.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:04.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:45:04.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:45:04.921 INFO:tasks.rados.rados.0.smithi203.stdout:2865: done (0 left) 2024-09-17T15:45:04.921 INFO:tasks.rados.rados.0.smithi203.stdout:2866: read oid 9 snap -1 2024-09-17T15:45:04.921 INFO:tasks.rados.rados.0.smithi203.stdout:2866: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:45:04.921 INFO:tasks.rados.rados.0.smithi203.stdout:2867: snap_create 2024-09-17T15:45:04.936 INFO:tasks.rados.rados.0.smithi203.stdout:2866: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:45:04.936 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:04.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.965+0000 7f6d7a2cd640 10 monclient: get_monmap_and_config 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.965+0000 7f6d7a2cd640 10 monclient: build_initial_monmap 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.965+0000 7f6d7a2cd640 10 monclient: monmap: 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:04.971640+0000 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:04.971640+0000 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:04.965 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:04.966 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:04.966 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 Processor -- start 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 -- start start 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient: init 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740a4900 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:04.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d7406d0b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d7406d490 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7409bf70 con 0x7f6d7406d490 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7409b3f0 con 0x7f6d7406d0b0 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d740a7180 con 0x7f6d740a4900 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7a2cd640 10 monclient(hunting): authenticate will time out at 2037.937312s 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7886a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6d7406d490 0x7f6d7413fbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34830/0 (socket says 172.21.15.86:34830) 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d7886a640 1 -- 172.21.15.86:0/4021390706 learned_addr learned my addr 172.21.15.86:0/4021390706 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): get_auth_request con 0x7f6d740a4900 auth_method 0 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:04.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 164525 payload 274 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 >> v1:172.21.15.203:6789/0 conn(0x7f6d7406d0b0 legacy=0x7f6d7413c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 >> v1:172.21.15.86:6789/0 conn(0x7f6d7406d490 legacy=0x7f6d7413fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 1 monclient: found mon.noname-b 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.969+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7409bc90 con 0x7f6d740a4900 2024-09-17T15:45:04.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.973+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 msgr2=0x7f6d7406cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send send error: (32) Broken pipe 2024-09-17T15:45:04.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.973+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f6d640097f0 tx=0x7f6d6401c3a0 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-17T15:45:04.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:04.973+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:05.026 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:05.084 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient: get_auth_request con 0x7f6d740a4900 auth_method 0 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 0 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 msgr2=0x7f6d7406cb70 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d740a4900 0x7f6d7406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:45:05.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 --2- 172.21.15.86:0/4021390706 >> v2:172.21.15.86:3300/0 conn(0x7f6d5c003510 0x7f6d5c005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f6d5c003510 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 --2- 172.21.15.86:0/4021390706 >> v2:172.21.15.203:3300/0 conn(0x7f6d5c0062c0 0x7f6d5c008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f6d5c0062c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6d5c009090 con 0x7f6d5c003510 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6d5c009240 con 0x7f6d5c0062c0 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d72ffd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> v2:172.21.15.86:3300/0 conn(0x7f6d5c003510 0x7f6d5c005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): get_auth_request con 0x7f6d5c003510 auth_method 0 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d737fe640 1 --2- 172.21.15.86:0/4021390706 >> v2:172.21.15.203:3300/0 conn(0x7f6d5c0062c0 0x7f6d5c008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d737fe640 10 monclient(hunting): get_auth_request con 0x7f6d5c0062c0 auth_method 0 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 164525 payload 293 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 >> v2:172.21.15.203:3300/0 conn(0x7f6d5c0062c0 msgr2=0x7f6d5c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> v2:172.21.15.203:3300/0 conn(0x7f6d5c0062c0 0x7f6d5c008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:05.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 monclient: found mon.noname-a 2024-09-17T15:45:05.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:05.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:05.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.177+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5c0094b0 con 0x7f6d5c003510 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d73fff640 1 -- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 msgr2=0x7f6d5c005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d73fff640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d5c005990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d64001ec0 tx=0x7f6d64009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76f052555bc76b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d640aa640 con 0x7f6d5c003510 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:05.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: dump: 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: _finish_auth 0 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:05.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d640aac40 con 0x7f6d5c003510 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 5 monclient: authenticate success, global_id 164525 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 1 -- 172.21.15.86:0/4021390706 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d640aa460 con 0x7f6d5c003510 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 4 set_mon_vals no callback set 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d72ffd640 10 monclient: dump: 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 10 monclient: shutdown 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 msgr2=0x7f6d5c005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d5c005990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d64001ec0 tx=0x7f6d64009190 comp rx=0 tx=0).stop 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7886a640 1 -- 172.21.15.86:0/4021390706 reap_dead start 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/4021390706 shutdown_connections 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/4021390706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d5c005990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/4021390706 >> 172.21.15.86:0/4021390706 conn(0x7f6d740a3e10 msgr2=0x7f6d740a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/4021390706 shutdown_connections 2024-09-17T15:45:05.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.181+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/4021390706 wait complete. 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient: build_initial_monmap 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient: monmap: 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 Processor -- start 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 -- start start 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient: init 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d74099410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-a con 0x7f6d5c003510 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d74099950 0x7f6d74093ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-c con 0x7f6d74099950 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74094410 0x7f6d74094870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): picked mon.noname-b con 0x7f6d74094410 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7406ed80 con 0x7f6d5c003510 2024-09-17T15:45:05.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7409c2c0 con 0x7f6d74094410 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d740a4ce0 con 0x7f6d74099950 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7a2cd640 10 monclient(hunting): authenticate will time out at 2038.153791s 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d74099410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d74099410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60830/0 (socket says 172.21.15.86:60830) 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 -- 172.21.15.86:0/3892155583 learned_addr learned my addr 172.21.15.86:0/3892155583 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): get_auth_request con 0x7f6d5c003510 auth_method 0 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d74099950 0x7f6d74093ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): get_auth_request con 0x7f6d74099950 auth_method 0 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7886a640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74094410 0x7f6d74094870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7886a640 10 monclient(hunting): get_auth_request con 0x7f6d74094410 auth_method 0 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7886a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7886a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d7886a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient(hunting): handle_auth_done global_id 184329 payload 824 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d74099950 msgr2=0x7f6d74093ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d74099950 0x7f6d74093ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74094410 msgr2=0x7f6d74094870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74094410 0x7f6d74094870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 monclient: found mon.noname-a 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.185+0000 7f6d73fff640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d74097cb0 con 0x7f6d5c003510 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d73fff640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 msgr2=0x7f6d74099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d73fff640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d74099410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6d64008010 tx=0x7f6d64019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64003820 con 0x7f6d5c003510 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.194 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: _finish_auth 0 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d64000e50 con 0x7f6d5c003510 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 5 monclient: authenticate success, global_id 184329 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 monclient: _renew_subs 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d74097100 con 0x7f6d5c003510 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d64003c40 con 0x7f6d5c003510 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 monclient: _renew_subs 2024-09-17T15:45:05.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d727fc640 4 set_mon_vals no callback set 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d74095950 con 0x7f6d5c003510 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d740a81b0 con 0x7f6d74097410 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6d640b6020 con 0x7f6d5c003510 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 10 monclient: get_auth_request con 0x7f6d74097410 auth_method 0 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 msgr2=0x7f6d74072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 msgr2=0x7f6d74072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(640..640 src has 438..640) ==== 4771+0+0 (secure 0 0 0) 0x7f6d640eb0c0 con 0x7f6d5c003510 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7886a640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.189+0000 7f6d7886a640 10 monclient: get_auth_request con 0x7f6d6003bbb0 auth_method 0 2024-09-17T15:45:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.193+0000 7f6d7886a640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d7406d870 tx=0x7f6d6c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient: get_auth_request con 0x7f6d74097410 auth_method 0 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 10 monclient(hunting): handle_auth_done global_id 204115 payload 824 2024-09-17T15:45:05.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.393+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d7406cb70 con 0x7f6d74097410 2024-09-17T15:45:05.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 msgr2=0x7f6d74072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d68013620 tx=0x7f6d68013ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6801c020 con 0x7f6d74097410 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6d68006c00 con 0x7f6d74097410 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 msgr2=0x7f6d74072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d68013620 tx=0x7f6d68013ae0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 msgr2=0x7f6d74072fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:05.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.397+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74097410 0x7f6d74072fd0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6d68013620 tx=0x7f6d68013ae0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d7a2cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:05.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 0x7f6d74073920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d7a2cd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7409a1a0 con 0x7f6d74073510 2024-09-17T15:45:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 0x7f6d74073920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient: get_auth_request con 0x7f6d74073510 auth_method 0 2024-09-17T15:45:05.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 10 monclient(hunting): handle_auth_done global_id 204118 payload 824 2024-09-17T15:45:05.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.405+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6d740977f0 con 0x7f6d74073510 2024-09-17T15:45:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d737fe640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 msgr2=0x7f6d74073920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d737fe640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 0x7f6d74073920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d6801d000 tx=0x7f6d68003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6801c030 con 0x7f6d74073510 2024-09-17T15:45:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 10 monclient: dump: 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.408 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f6d68007230 con 0x7f6d74073510 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 msgr2=0x7f6d74073920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 0x7f6d74073920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d6801d000 tx=0x7f6d68003770 comp rx=0 tx=0).stop 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 msgr2=0x7f6d74073920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:05.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d70ff9640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d74073510 0x7f6d74073920 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6d6801d000 tx=0x7f6d68003770 comp rx=0 tx=0).stop 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 8011, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:45:05.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:05.411 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:05.412 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:05.413 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:05.414 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.415 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:05.416 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:05.417 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:05.418 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:05.419 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.420 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:05.421 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:05.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:05.422 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:05.422 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 msgr2=0x7f6d6003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6d6003bbb0 0x7f6d6003e070 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f6d7406d870 tx=0x7f6d6c009040 comp rx=0 tx=0).stop 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 10 monclient: shutdown 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 msgr2=0x7f6d74099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7a2cd640 1 --2- 172.21.15.86:0/3892155583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d5c003510 0x7f6d74099410 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6d64008010 tx=0x7f6d64019140 comp rx=0 tx=0).stop 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.409+0000 7f6d7886a640 1 -- 172.21.15.86:0/3892155583 reap_dead start 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.413+0000 7f6d7886a640 1 -- 172.21.15.86:0/3892155583 reap_dead start 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.413+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 shutdown_connections 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.413+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 >> 172.21.15.86:0/3892155583 conn(0x7f6d740a3e10 msgr2=0x7f6d74069180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.413+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 shutdown_connections 2024-09-17T15:45:05.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.413+0000 7f6d7a2cd640 1 -- 172.21.15.86:0/3892155583 wait complete. 2024-09-17T15:45:05.438 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:05.438 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:45:05.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:45:05.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.565+0000 7f1c30672640 10 monclient: get_monmap_and_config 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.565+0000 7f1c30672640 10 monclient: build_initial_monmap 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.565+0000 7f1c30672640 10 monclient: monmap: 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:05.571492+0000 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:05.571492+0000 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:05.565 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:05.566 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:05.566 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:05.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 Processor -- start 2024-09-17T15:45:05.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 -- start start 2024-09-17T15:45:05.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient: init 2024-09-17T15:45:05.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c28138280 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-b con 0x7f1c2806fb00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2813d470 0x7f1c2813f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2813d470 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c2806dd30 con 0x7f1c28138280 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1c280a48e0 con 0x7f1c2813d470 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c2806e190 con 0x7f1c2806fb00 2024-09-17T15:45:05.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c30672640 10 monclient(hunting): authenticate will time out at 2038.537037s 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2813d470 0x7f1c2813f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2813d470 0x7f1c2813f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60846/0 (socket says 172.21.15.86:60846) 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50240/0 (socket says 172.21.15.86:50240) 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/2233456197 learned_addr learned my addr 172.21.15.86:0/2233456197 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): get_auth_request con 0x7f1c2813d470 auth_method 0 2024-09-17T15:45:05.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): get_auth_request con 0x7f1c2806fb00 auth_method 0 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_done global_id 184332 payload 274 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 msgr2=0x7f1c280a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/2233456197 >> v1:172.21.15.203:6789/0 conn(0x7f1c28138280 legacy=0x7f1c28138660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient: _finish_hunting 0 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 monclient: found mon.noname-a 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/2233456197 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c2806df70 con 0x7f1c2813d470 2024-09-17T15:45:05.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.569+0000 7f1c2dc0d640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.573+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2813d470 msgr2=0x7f1c2813f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.573+0000 7f1c2ec0f640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2813d470 0x7f1c2813f860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2001c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baba6f6060e5601f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 1 -- 172.21.15.86:0/2233456197 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200a55b0 con 0x7f1c2813d470 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: dump: 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.574 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: _finish_auth 0 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: _check_auth_tickets 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 1 -- 172.21.15.86:0/2233456197 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c200a5750 con 0x7f1c2813d470 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 5 monclient: authenticate success, global_id 184332 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 1 -- 172.21.15.86:0/2233456197 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200a5b20 con 0x7f1c2813d470 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 4 set_mon_vals no callback set 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2d40c640 10 monclient: dump: 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: shutdown 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2813d470 msgr2=0x7f1c2813f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2813d470 0x7f1c2813f860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009e50 tx=0x7f1c2001c770 comp rx=0 tx=0).stop 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- 172.21.15.86:0/2233456197 shutdown_connections 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2813d470 0x7f1c2813f860 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- 172.21.15.86:0/2233456197 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2806fb00 0x7f1c280a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- 172.21.15.86:0/2233456197 >> 172.21.15.86:0/2233456197 conn(0x7f1c280a8470 msgr2=0x7f1c280a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- 172.21.15.86:0/2233456197 shutdown_connections 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- 172.21.15.86:0/2233456197 wait complete. 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: build_initial_monmap 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: monmap: 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:05.582 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 Processor -- start 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- start start 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: init 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2806fb00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c28138280 0x7f1c281378a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-b con 0x7f1c28138280 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2813d470 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:05.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1c28142740 con 0x7f1c2806fb00 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c281425c0 con 0x7f1c28138280 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c281428c0 con 0x7f1c2813d470 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c30672640 10 monclient(hunting): authenticate will time out at 2038.547082s 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50256/0 (socket says 172.21.15.86:50256) 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60854/0 (socket says 172.21.15.86:60854) 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/3884898150 learned_addr learned my addr 172.21.15.86:0/3884898150 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:05.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): get_auth_request con 0x7f1c2813d470 auth_method 0 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): get_auth_request con 0x7f1c2806fb00 auth_method 0 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:05.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2dc0d640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c28138280 0x7f1c281378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2dc0d640 10 monclient(hunting): get_auth_request con 0x7f1c28138280 auth_method 0 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2dc0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2dc0d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2dc0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient(hunting): handle_auth_done global_id 164534 payload 824 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c28138280 msgr2=0x7f1c281378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c28138280 0x7f1c281378a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 msgr2=0x7f1c28137360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient: _finish_hunting 0 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 monclient: found mon.noname-c 2024-09-17T15:45:05.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/3884898150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c280a7410 con 0x7f1c2813d470 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.577+0000 7f1c2e40e640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c2ec0f640 1 -- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 msgr2=0x7f1c281cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c2ec0f640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c2001d010 tx=0x7f1c20019510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200bc020 con 0x7f1c2813d470 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: dump: 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.587 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: _finish_auth 0 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 5 monclient: authenticate success, global_id 164534 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _renew_subs 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c200b8880 con 0x7f1c2813d470 2024-09-17T15:45:05.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c2809df30 con 0x7f1c2813d470 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c200a55b0 con 0x7f1c2813d470 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c2cc0b640 4 set_mon_vals no callback set 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 10 monclient: dump: 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:05.589 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _renew_subs 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c2809f660 con 0x7f1c2813d470 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c28138660 con 0x7f1c2813d470 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1c200bf020 con 0x7f1c2813d470 2024-09-17T15:45:05.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 0x7f1c0c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(640..640 src has 438..640) ==== 4771+0+0 (secure 0 0 0) 0x7f1c200eadc0 con 0x7f1c2813d470 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c2e40e640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 0x7f1c0c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.581+0000 7f1c2e40e640 10 monclient: get_auth_request con 0x7f1c0c03bb60 auth_method 0 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.585+0000 7f1c2e40e640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 0x7f1c0c03e020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c18007fd0 tx=0x7f1c18007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.585+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1c200c6030 con 0x7f1c2813d470 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.589+0000 7f1c1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:05.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.589+0000 7f1c1effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:05.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.813+0000 7f1c30672640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:45:05.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.813+0000 7f1c30672640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:45:05.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.813+0000 7f1c30672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:05.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.813+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1c280a45d0 con 0x7f1c2813d470 2024-09-17T15:45:05.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c1effd640 1 -- 172.21.15.86:0/3884898150 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1c280a7410 con 0x7f1c2813d470 2024-09-17T15:45:05.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:45:05.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c1effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3271384393,config=3541143443,health=1845395070,kv=2794928421,logm=1598102811}) 2024-09-17T15:45:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=3715967173,config=3541143443,health=2193270098,kv=3591778543,logm=2725410239}) 2024-09-17T15:45:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3271384393,config=3541143443,health=1845395070,kv=2794928421,logm=1598102811}) 2024-09-17T15:45:05.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.817+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=11,logm=46} crc {auth=3777568510,config=3541143443,health=3514206583,kv=3591778543,logm=3736272311}) 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 msgr2=0x7f1c0c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 0x7f1c0c03e020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f1c18007fd0 tx=0x7f1c18007e40 comp rx=0 tx=0).stop 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 10 monclient: shutdown 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 msgr2=0x7f1c281cfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c2001d010 tx=0x7f1c20019510 comp rx=0 tx=0).stop 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 shutdown_connections 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c0c03bb60 0x7f1c0c03e020 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c2813d470 0x7f1c281cfcb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c28138280 0x7f1c281378a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 --2- 172.21.15.86:0/3884898150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2806fb00 0x7f1c28137360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:05.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 >> 172.21.15.86:0/3884898150 conn(0x7f1c280a8470 msgr2=0x7f1c280a0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:05.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 shutdown_connections 2024-09-17T15:45:05.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:05.817+0000 7f1c30672640 1 -- 172.21.15.86:0/3884898150 wait complete. 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392846455}) 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982145857}) 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3392846455}) 2024-09-17T15:45:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679471490}) 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683356885}) 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450092800}) 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683356885}) 2024-09-17T15:45:05.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.821+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430556552}) 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995776684}) 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271798483}) 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1995776684}) 2024-09-17T15:45:05.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3375401641}) 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633028645}) 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1308984078}) 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633028645}) 2024-09-17T15:45:05.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.825+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832885043}) 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321742852}) 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884345719}) 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=321742852}) 2024-09-17T15:45:05.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969295718}) 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3700561946}) 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024545346}) 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3700561946}) 2024-09-17T15:45:05.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.829+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=576020673}) 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794030420}) 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004883023}) 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794030420}) 2024-09-17T15:45:05.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87177558}) 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145821701}) 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327994074}) 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145821701}) 2024-09-17T15:45:05.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.833+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2377052236}) 2024-09-17T15:45:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816895741}) 2024-09-17T15:45:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1073802640}) 2024-09-17T15:45:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816895741}) 2024-09-17T15:45:05.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763891049}) 2024-09-17T15:45:05.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:45:05.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252721592}) 2024-09-17T15:45:05.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1320617893}) 2024-09-17T15:45:05.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1252721592}) 2024-09-17T15:45:05.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.837+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1386841884}) 2024-09-17T15:45:05.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575311448}) 2024-09-17T15:45:05.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1465334070}) 2024-09-17T15:45:05.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575311448}) 2024-09-17T15:45:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4194420025}) 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653104531}) 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61422677}) 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653104531}) 2024-09-17T15:45:05.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961623040}) 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639740146}) 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=703384788}) 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639740146}) 2024-09-17T15:45:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.841+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1482232778}) 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059296146}) 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542015113}) 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059296146}) 2024-09-17T15:45:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094563337}) 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337873850}) 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084752704}) 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3337873850}) 2024-09-17T15:45:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.845+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817102778}) 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395515920}) 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509553449}) 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395515920}) 2024-09-17T15:45:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561321487}) 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540185300}) 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716986236}) 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2540185300}) 2024-09-17T15:45:05.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.849+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954026325}) 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016077515}) 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624929058}) 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016077515}) 2024-09-17T15:45:05.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318563542}) 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804524815}) 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2371525111}) 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804524815}) 2024-09-17T15:45:05.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.853+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3321293493}) 2024-09-17T15:45:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188958346}) 2024-09-17T15:45:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=174168511}) 2024-09-17T15:45:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188958346}) 2024-09-17T15:45:05.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584633285}) 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640832831}) 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=727214945}) 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1640832831}) 2024-09-17T15:45:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.857+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2971914106}) 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797519089}) 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119374753}) 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797519089}) 2024-09-17T15:45:05.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288339966}) 2024-09-17T15:45:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807780044}) 2024-09-17T15:45:05.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659465219}) 2024-09-17T15:45:05.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807780044}) 2024-09-17T15:45:05.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.861+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1369955143}) 2024-09-17T15:45:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548113364}) 2024-09-17T15:45:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1767878755}) 2024-09-17T15:45:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1548113364}) 2024-09-17T15:45:05.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253007167}) 2024-09-17T15:45:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406903707}) 2024-09-17T15:45:05.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=589490591}) 2024-09-17T15:45:05.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406903707}) 2024-09-17T15:45:05.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.865+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281965902}) 2024-09-17T15:45:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420910845}) 2024-09-17T15:45:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911689071}) 2024-09-17T15:45:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420910845}) 2024-09-17T15:45:05.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2409404188}) 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258630848}) 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3001599076}) 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258630848}) 2024-09-17T15:45:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3090586974}) 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456880708}) 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284525292}) 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456880708}) 2024-09-17T15:45:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.869+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=363174175}) 2024-09-17T15:45:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905201554}) 2024-09-17T15:45:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872652315}) 2024-09-17T15:45:05.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905201554}) 2024-09-17T15:45:05.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219554153}) 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044519916}) 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255618760}) 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044519916}) 2024-09-17T15:45:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.873+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767620755}) 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121188268}) 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004025558}) 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121188268}) 2024-09-17T15:45:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576685582}) 2024-09-17T15:45:05.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049739303}) 2024-09-17T15:45:05.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656361375}) 2024-09-17T15:45:05.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049739303}) 2024-09-17T15:45:05.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2363406234}) 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143978309}) 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=494140749}) 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.877+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143978309}) 2024-09-17T15:45:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000454928}) 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468890401}) 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912349678}) 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468890401}) 2024-09-17T15:45:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382841414}) 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350808348}) 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=258242818}) 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350808348}) 2024-09-17T15:45:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.881+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639603203}) 2024-09-17T15:45:05.882 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.882 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946068450}) 2024-09-17T15:45:05.882 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515675195}) 2024-09-17T15:45:05.882 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.882 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1946068450}) 2024-09-17T15:45:05.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.885+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097105188}) 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664226329}) 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=928323759}) 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3664226329}) 2024-09-17T15:45:05.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.921+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708973708}) 2024-09-17T15:45:05.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052160022}) 2024-09-17T15:45:05.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492348930}) 2024-09-17T15:45:05.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:05.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052160022}) 2024-09-17T15:45:05.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:05.929+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=488413789}) 2024-09-17T15:45:05.930 INFO:tasks.rados.rados.0.smithi203.stdout:2866: done (1 left) 2024-09-17T15:45:05.931 INFO:tasks.rados.rados.0.smithi203.stdout:2867: done (0 left) 2024-09-17T15:45:05.931 INFO:tasks.rados.rados.0.smithi203.stdout:2868: snap_create 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191542387}) 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401650187}) 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191542387}) 2024-09-17T15:45:06.094 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.097+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018364872}) 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588423941}) 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991739060}) 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=588423941}) 2024-09-17T15:45:06.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.161+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352079313}) 2024-09-17T15:45:06.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105465295}) 2024-09-17T15:45:06.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817532641}) 2024-09-17T15:45:06.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2105465295}) 2024-09-17T15:45:06.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.165+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538874245}) 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255547489}) 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=354841668}) 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255547489}) 2024-09-17T15:45:06.447 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106475569}) 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.449+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330152608}) 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1706498703}) 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=330152608}) 2024-09-17T15:45:06.449 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2034247584}) 2024-09-17T15:45:06.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881871309}) 2024-09-17T15:45:06.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1869521735}) 2024-09-17T15:45:06.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881871309}) 2024-09-17T15:45:06.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.453+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730498454}) 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338913098}) 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4005112594}) 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338913098}) 2024-09-17T15:45:06.454 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567436994}) 2024-09-17T15:45:06.456 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619804228}) 2024-09-17T15:45:06.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2718826523}) 2024-09-17T15:45:06.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619804228}) 2024-09-17T15:45:06.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.457+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924637923}) 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117442928}) 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159236591}) 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117442928}) 2024-09-17T15:45:06.459 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.461+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125996834}) 2024-09-17T15:45:06.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035987903}) 2024-09-17T15:45:06.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3625411784}) 2024-09-17T15:45:06.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035987903}) 2024-09-17T15:45:06.462 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563174048}) 2024-09-17T15:45:06.463 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.463 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720715658}) 2024-09-17T15:45:06.463 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744011159}) 2024-09-17T15:45:06.463 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.463 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720715658}) 2024-09-17T15:45:06.464 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.465+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=953980964}) 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087223554}) 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842049460}) 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087223554}) 2024-09-17T15:45:06.466 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095164830}) 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531276642}) 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243889276}) 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531276642}) 2024-09-17T15:45:06.468 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.469+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3985980752}) 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758062348}) 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3943774960}) 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758062348}) 2024-09-17T15:45:06.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508532052}) 2024-09-17T15:45:06.472 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.472 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506897106}) 2024-09-17T15:45:06.472 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792033369}) 2024-09-17T15:45:06.472 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.472 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506897106}) 2024-09-17T15:45:06.473 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.473+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2245381890}) 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595749471}) 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993909590}) 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3595749471}) 2024-09-17T15:45:06.475 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651104204}) 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325337645}) 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268760575}) 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325337645}) 2024-09-17T15:45:06.477 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.477+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519864964}) 2024-09-17T15:45:06.479 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.479 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969907468}) 2024-09-17T15:45:06.480 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478949273}) 2024-09-17T15:45:06.480 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.480 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969907468}) 2024-09-17T15:45:06.480 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.481+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991201038}) 2024-09-17T15:45:06.481 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912710139}) 2024-09-17T15:45:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4130934122}) 2024-09-17T15:45:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912710139}) 2024-09-17T15:45:06.482 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1917132808}) 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168736687}) 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161235194}) 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2168736687}) 2024-09-17T15:45:06.484 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.485+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338258677}) 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122713533}) 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846552702}) 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122713533}) 2024-09-17T15:45:06.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700253941}) 2024-09-17T15:45:06.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627343702}) 2024-09-17T15:45:06.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=378290648}) 2024-09-17T15:45:06.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627343702}) 2024-09-17T15:45:06.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.489+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308020441}) 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681742486}) 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3650870092}) 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681742486}) 2024-09-17T15:45:06.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204330887}) 2024-09-17T15:45:06.492 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.493 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610378596}) 2024-09-17T15:45:06.493 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3000435800}) 2024-09-17T15:45:06.493 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.493 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610378596}) 2024-09-17T15:45:06.493 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.493+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459532398}) 2024-09-17T15:45:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519797126}) 2024-09-17T15:45:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967127345}) 2024-09-17T15:45:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.494 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2519797126}) 2024-09-17T15:45:06.495 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319627060}) 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001655041}) 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345252437}) 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4001655041}) 2024-09-17T15:45:06.496 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.497+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685651491}) 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=1661170122,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2092993741,mon_config_key=2445621393,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1259201597,osd_pg_creating=411286724,osd_snap=346377624}) 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=686890401,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2092993741,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=411286724,osd_snap=2489193687}) 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13} crc {logm=1661170122,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2092993741,mon_config_key=2445621393,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1259201597,osd_pg_creating=411286724,osd_snap=346377624}) 2024-09-17T15:45:06.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.501+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=350391990,mdsmap=2815901553,mgr=1375715255,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2092993741,mon_config_key=1773882147,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=411286724,osd_snap=190612186}) 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=98,osdmap=2} crc {osd_snap=2597454114,osdmap=720805386}) 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=99,osdmap=1} crc {osd_snap=3296924909,osdmap=307225901}) 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=98,osdmap=2} crc {osd_snap=2597454114,osdmap=720805386}) 2024-09-17T15:45:06.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3320747578}) 2024-09-17T15:45:06.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1423916239}) 2024-09-17T15:45:06.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2686429957}) 2024-09-17T15:45:06.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1423916239}) 2024-09-17T15:45:06.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.505+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3284823005}) 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=887485845}) 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2222941313}) 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=887485845}) 2024-09-17T15:45:06.506 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.509+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2426639}) 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614804920}) 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4090067231}) 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614804920}) 2024-09-17T15:45:06.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.513+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3804571749}) 2024-09-17T15:45:06.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.525+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:45:06.523 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.525+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=5} crc {osdmap=775321912}) 2024-09-17T15:45:06.523 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:45:06.525+0000 7fd082c95640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=5} crc {osdmap=2730608340}) 2024-09-17T15:45:06.542 INFO:tasks.rados.rados.0.smithi203.stdout:2868: done (0 left) 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2869: read oid 37 snap -1 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2869: expect (ObjNum 988 snap 309 seq_num 988) 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2870: read oid 15 snap -1 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2870: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2871: read oid 20 snap -1 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2871: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:45:06.543 INFO:tasks.rados.rados.0.smithi203.stdout:2872: snap_remove snap 314 2024-09-17T15:45:06.836 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:45:06.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:06.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: get_monmap_and_config 2024-09-17T15:45:06.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: build_initial_monmap 2024-09-17T15:45:06.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: monmap: 2024-09-17T15:45:06.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:06.973061+0000 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:06.973061+0000 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:06.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 Processor -- start 2024-09-17T15:45:06.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 -- start start 2024-09-17T15:45:06.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: init 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 0x7f038009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-c con 0x7f03800689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-b con 0x7f0380068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0380134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f038009d770 con 0x7f0380134a40 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f038009ed00 con 0x7f03800689d0 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f038009efe0 con 0x7f0380068db0 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:06.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f03850c0640 10 monclient(hunting): authenticate will time out at 2039.938664s 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50270/0 (socket says 172.21.15.86:50270) 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 -- 172.21.15.86:0/2938555552 learned_addr learned my addr 172.21.15.86:0/2938555552 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): get_auth_request con 0x7f0380068db0 auth_method 0 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient(hunting): handle_auth_done global_id 164537 payload 274 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 -- 172.21.15.86:0/2938555552 >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 msgr2=0x7f038009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:06.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037ed9d640 1 --2- 172.21.15.86:0/2938555552 >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 0x7f038009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 --2- 172.21.15.86:0/2938555552 >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 0x7f038009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 -- 172.21.15.86:0/2938555552 >> v1:172.21.15.203:6789/0 conn(0x7f0380134a40 legacy=0x7f0380134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 monclient: found mon.noname-b 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037e59c640 1 -- 172.21.15.86:0/2938555552 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03800a04e0 con 0x7f0380068db0 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037ed9d640 10 monclient: get_auth_request con 0x7f03800689d0 auth_method 0 2024-09-17T15:45:06.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.969+0000 7f037ed9d640 1 --2- 172.21.15.86:0/2938555552 >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 0x7f038009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:06.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037e59c640 1 -- 172.21.15.86:0/2938555552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 msgr2=0x7f0380069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037e59c640 1 --2- 172.21.15.86:0/2938555552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0374009e50 tx=0x7f037401c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73039c2b97f7254b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 1 -- 172.21.15.86:0/2938555552 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03740a57e0 con 0x7f0380068db0 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: dump: 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:06.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 1 -- 172.21.15.86:0/2938555552 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03740a5980 con 0x7f0380068db0 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 1 -- 172.21.15.86:0/2938555552 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03740a5d50 con 0x7f0380068db0 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f037dd9b640 10 monclient: dump: 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:06.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f03850c0640 5 monclient: authenticate success, global_id 164537 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f03850c0640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.973+0000 7f03850c0640 4 set_mon_vals no callback set 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: shutdown 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:06.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- 172.21.15.86:0/2938555552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 msgr2=0x7f0380069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- 172.21.15.86:0/2938555552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0374009e50 tx=0x7f037401c980 comp rx=0 tx=0).stop 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- 172.21.15.86:0/2938555552 shutdown_connections 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- 172.21.15.86:0/2938555552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0380068db0 0x7f0380069190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- 172.21.15.86:0/2938555552 >> v2:172.21.15.203:3300/0 conn(0x7f03800689d0 0x7f038009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- 172.21.15.86:0/2938555552 >> 172.21.15.86:0/2938555552 conn(0x7f03800a6440 msgr2=0x7f03800a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- 172.21.15.86:0/2938555552 shutdown_connections 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- 172.21.15.86:0/2938555552 wait complete. 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: build_initial_monmap 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: monmap: 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:06.978 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.978 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:06.978 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:06.978 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:06.978 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 Processor -- start 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- start start 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: init 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-c con 0x7f03800689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380068db0 0x7f0380133b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-a con 0x7f0380068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0380134a40 0x7f03801cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): picked mon.noname-b con 0x7f0380134a40 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f038013fad0 con 0x7f0380068db0 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f038013f950 con 0x7f0380134a40 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f038013fc50 con 0x7f03800689d0 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f03850c0640 10 monclient(hunting): authenticate will time out at 2039.945960s 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50282/0 (socket says 172.21.15.86:50282) 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 -- 172.21.15.86:0/2460485989 learned_addr learned my addr 172.21.15.86:0/2460485989 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): get_auth_request con 0x7f03800689d0 auth_method 0 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380068db0 0x7f0380133b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): get_auth_request con 0x7f0380068db0 auth_method 0 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037f59e640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0380134a40 0x7f03801cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037f59e640 10 monclient(hunting): get_auth_request con 0x7f0380134a40 auth_method 0 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037f59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient(hunting): handle_auth_done global_id 164540 payload 824 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 -- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0380134a40 msgr2=0x7f03801cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0380134a40 0x7f03801cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 -- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380068db0 msgr2=0x7f0380133b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380068db0 0x7f0380133b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 monclient: found mon.noname-c 2024-09-17T15:45:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.977+0000 7f037ed9d640 1 -- 172.21.15.86:0/2460485989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038013b7c0 con 0x7f03800689d0 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f037ed9d640 1 -- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 msgr2=0x7f038013ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f037ed9d640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f037000c820 tx=0x7f037000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0370015070 con 0x7f03800689d0 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: dump: 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0370007be0 con 0x7f03800689d0 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037000ce10 con 0x7f03800689d0 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 10 monclient: dump: 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 5 monclient: authenticate success, global_id 164540 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _renew_subs 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f037cd99640 4 set_mon_vals no callback set 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0380074390 con 0x7f03800689d0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _renew_subs 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f038009d460 con 0x7f03800689d0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f038009ffa0 con 0x7f03800689d0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0370018070 con 0x7f03800689d0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 0x7f036c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(642..642 src has 438..642) ==== 4771+0+0 (secure 0 0 0) 0x7f037004ff40 con 0x7f03800689d0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f037e59c640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 0x7f036c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.981+0000 7f037e59c640 10 monclient: get_auth_request con 0x7f036c03bb90 auth_method 0 2024-09-17T15:45:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.985+0000 7f037e59c640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 0x7f036c03e050 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f038006ed60 tx=0x7f03740a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.985+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0380074390 con 0x7f03800689d0 2024-09-17T15:45:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.985+0000 7f03677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:06.985+0000 7f03677fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:07.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03850c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:07.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03850c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:07.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03850c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f038011ed90 con 0x7f03800689d0 2024-09-17T15:45:07.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03677fe640 1 -- 172.21.15.86:0/2460485989 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0370011ba0 con 0x7f03800689d0 2024-09-17T15:45:07.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:07.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.229+0000 7f03677fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:07.227 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:07.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 msgr2=0x7f036c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 0x7f036c03e050 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f038006ed60 tx=0x7f03740a7070 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 10 monclient: shutdown 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 msgr2=0x7f038013ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f037000c820 tx=0x7f037000cce0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 shutdown_connections 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f036c03bb90 0x7f036c03e050 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0380134a40 0x7f03801cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0380068db0 0x7f0380133b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 --2- 172.21.15.86:0/2460485989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03800689d0 0x7f038013ad10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 >> 172.21.15.86:0/2460485989 conn(0x7f03800a6440 msgr2=0x7f038013c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 shutdown_connections 2024-09-17T15:45:07.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.233+0000 7f03850c0640 1 -- 172.21.15.86:0/2460485989 wait complete. 2024-09-17T15:45:07.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:45:07.246 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:45:07.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:45:07.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:45:07.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: get_monmap_and_config 2024-09-17T15:45:07.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: monmap: 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:07.372422+0000 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:07.372422+0000 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:07.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 Processor -- start 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 -- start start 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: init 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 0x7febec09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): picked mon.noname-b con 0x7febec0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7febec0a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febec13d2f0 0x7febec13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): picked mon.noname-a con 0x7febec13d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febec0a6280 con 0x7febec0a54c0 2024-09-17T15:45:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7febec09d2a0 con 0x7febec13d2f0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7febec09c0a0 con 0x7febec0a50e0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf4782640 10 monclient(hunting): authenticate will time out at 2040.337918s 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 0x7febec09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf1d1d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7febec0a54c0 0x7febec0a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34842/0 (socket says 172.21.15.86:34842) 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf1d1d640 1 -- 172.21.15.86:0/1917172482 learned_addr learned my addr 172.21.15.86:0/1917172482 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): get_auth_request con 0x7febec0a50e0 auth_method 0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 1 --2- 172.21.15.86:0/1917172482 >> v2:172.21.15.86:3300/0 conn(0x7febec13d2f0 0x7febec13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): get_auth_request con 0x7febec13d2f0 auth_method 0 2024-09-17T15:45:07.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient(hunting): handle_auth_done global_id 164543 payload 274 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 -- 172.21.15.86:0/1917172482 >> v2:172.21.15.86:3300/0 conn(0x7febec13d2f0 msgr2=0x7febec13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 --2- 172.21.15.86:0/1917172482 >> v2:172.21.15.86:3300/0 conn(0x7febec13d2f0 0x7febec13f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 -- 172.21.15.86:0/1917172482 >> v1:172.21.15.86:6789/0 conn(0x7febec0a54c0 legacy=0x7febec0a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 monclient: found mon.noname-b 2024-09-17T15:45:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.369+0000 7febf251e640 1 -- 172.21.15.86:0/1917172482 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec09c380 con 0x7febec0a50e0 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf251e640 1 -- 172.21.15.86:0/1917172482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 msgr2=0x7febec09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf251e640 1 --2- 172.21.15.86:0/1917172482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 0x7febec09cd60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7febe40097c0 tx=0x7febe401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56e784f697f7f27e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 1 -- 172.21.15.86:0/1917172482 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe40b0070 con 0x7febec0a50e0 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: dump: 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 1 -- 172.21.15.86:0/1917172482 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe40ab600 con 0x7febec0a50e0 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf4782640 5 monclient: authenticate success, global_id 164543 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf4782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 1 -- 172.21.15.86:0/1917172482 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe40a4e60 con 0x7febec0a50e0 2024-09-17T15:45:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf151c640 10 monclient: dump: 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf4782640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:07.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.373+0000 7febf4782640 4 set_mon_vals no callback set 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: shutdown 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- 172.21.15.86:0/1917172482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 msgr2=0x7febec09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- 172.21.15.86:0/1917172482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 0x7febec09cd60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7febe40097c0 tx=0x7febe401c330 comp rx=0 tx=0).stop 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- 172.21.15.86:0/1917172482 shutdown_connections 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- 172.21.15.86:0/1917172482 >> v2:172.21.15.86:3300/0 conn(0x7febec13d2f0 0x7febec13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- 172.21.15.86:0/1917172482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec0a50e0 0x7febec09cd60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- 172.21.15.86:0/1917172482 >> 172.21.15.86:0/1917172482 conn(0x7febec0aaae0 msgr2=0x7febec0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- 172.21.15.86:0/1917172482 shutdown_connections 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- 172.21.15.86:0/1917172482 wait complete. 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: monmap: 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 Processor -- start 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- start start 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: init 2024-09-17T15:45:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec0a50e0 0x7febec1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): picked mon.noname-b con 0x7febec0a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 0x7febec134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): picked mon.noname-a con 0x7febec0a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 0x7febec0a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): picked mon.noname-c con 0x7febec13d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7febec142650 con 0x7febec0a54c0 2024-09-17T15:45:07.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7febec0a6b00 con 0x7febec0a50e0 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7febec1427d0 con 0x7febec13d2f0 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf4782640 10 monclient(hunting): authenticate will time out at 2040.346099s 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 0x7febec134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 0x7febec0a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 0x7febec0a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50294/0 (socket says 172.21.15.86:50294) 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 0x7febec134600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60874/0 (socket says 172.21.15.86:60874) 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 learned_addr learned my addr 172.21.15.86:0/1403172256 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): get_auth_request con 0x7febec13d2f0 auth_method 0 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): get_auth_request con 0x7febec0a54c0 auth_method 0 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf251e640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec0a50e0 0x7febec1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf251e640 10 monclient(hunting): get_auth_request con 0x7febec0a50e0 auth_method 0 2024-09-17T15:45:07.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf251e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf251e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient(hunting): handle_auth_done global_id 164546 payload 824 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec0a50e0 msgr2=0x7febec1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7febec0a50e0 0x7febec1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 msgr2=0x7febec134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 0x7febec134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 monclient: found mon.noname-c 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec139b50 con 0x7febec13d2f0 2024-09-17T15:45:07.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.377+0000 7febf1d1d640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a54c0 0x7febec134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 msgr2=0x7febec0a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf2d1f640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 0x7febec0a1f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e880 tx=0x7febdc00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc017070 con 0x7febec13d2f0 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: dump: 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febdc00cb50 con 0x7febec13d2f0 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc013470 con 0x7febec13d2f0 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 10 monclient: dump: 2024-09-17T15:45:07.388 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf0d1b640 4 set_mon_vals no callback set 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 5 monclient: authenticate success, global_id 164546 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 monclient: _renew_subs 2024-09-17T15:45:07.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febec139e30 con 0x7febec13d2f0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 monclient: _renew_subs 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febec0a18d0 con 0x7febec13d2f0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 0x7febec0a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7febec06a0e0 con 0x7febec0a2460 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 0x7febec0a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient: get_auth_request con 0x7febec0a2460 auth_method 0 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7febdc01a070 con 0x7febec13d2f0 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 10 monclient(hunting): handle_auth_done global_id 184347 payload 824 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7febd803bbb0 0x7febd803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf251e640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febec13a200 con 0x7febec0a2460 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(642..642 src has 438..642) ==== 4771+0+0 (secure 0 0 0) 0x7febdc04ffb0 con 0x7febec13d2f0 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf1d1d640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7febd803bbb0 0x7febd803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.381+0000 7febf1d1d640 10 monclient: get_auth_request con 0x7febd803bbb0 auth_method 0 2024-09-17T15:45:07.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febf251e640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 msgr2=0x7febec0a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febf251e640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 0x7febec0a2840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7febe40ad000 tx=0x7febe40190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe40bb070 con 0x7febec0a2460 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 10 monclient: dump: 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7febe40abe20 con 0x7febec0a2460 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 msgr2=0x7febec0a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 0x7febec0a2840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7febe40ad000 tx=0x7febe40190d0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 msgr2=0x7febec0a2840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febe2ffd640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2460 0x7febec0a2840 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7febe40ad000 tx=0x7febe40190d0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.385+0000 7febf1d1d640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7febd803bbb0 0x7febd803e070 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7febec134eb0 tx=0x7febe8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf4782640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf4782640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 0x7febec09e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf4782640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7febec13d6d0 con 0x7febec0a2d80 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 0x7febec09e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient: get_auth_request con 0x7febec0a2d80 auth_method 0 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.389+0000 7febf251e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf251e640 10 monclient(hunting): handle_auth_done global_id 184350 payload 824 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf251e640 1 -- 172.21.15.86:0/1403172256 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febec09cd60 con 0x7febec0a2d80 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf251e640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 msgr2=0x7febec09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf251e640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 0x7febec09e5e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7febe401c370 tx=0x7febe4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe40cc020 con 0x7febec0a2d80 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 10 monclient: dump: 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7febe40b6440 con 0x7febec0a2d80 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 msgr2=0x7febec09e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 0x7febec09e5e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7febe401c370 tx=0x7febe4008040 comp rx=0 tx=0).stop 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 msgr2=0x7febec09e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febe2ffd640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febec0a2d80 0x7febec09e5e0 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7febe401c370 tx=0x7febe4008040 comp rx=0 tx=0).stop 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 110886, 2024-09-17T15:45:07.398 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.399 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.400 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:07.401 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:07.402 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:07.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.405 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.406 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:07.410 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:07.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7febd803bbb0 msgr2=0x7febd803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7febd803bbb0 0x7febd803e070 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7febec134eb0 tx=0x7febe8009040 comp rx=0 tx=0).stop 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 10 monclient: shutdown 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 msgr2=0x7febec0a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 --2- 172.21.15.86:0/1403172256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febec13d2f0 0x7febec0a1f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e880 tx=0x7febdc00ed40 comp rx=0 tx=0).stop 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 reap_dead start 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf2d1f640 1 -- 172.21.15.86:0/1403172256 reap_dead start 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 shutdown_connections 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 >> 172.21.15.86:0/1403172256 conn(0x7febec0aaae0 msgr2=0x7febec13c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 shutdown_connections 2024-09-17T15:45:07.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.393+0000 7febf4782640 1 -- 172.21.15.86:0/1403172256 wait complete. 2024-09-17T15:45:07.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:07.413 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:45:07.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: get_monmap_and_config 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: monmap: 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.529 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:07.536117+0000 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:07.536117+0000 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:07.530 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 Processor -- start 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 -- start start 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: init 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 0x7f2f3c0a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3c0a4970 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f3c09ae20 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:07.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2f3c13d530 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3c09b980 con 0x7f2f3c09ae20 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3c09bf40 con 0x7f2f3c13d530 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c06d1d0 con 0x7f2f3c0a4970 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f40908640 10 monclient(hunting): authenticate will time out at 2040.501628s 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 0x7f2f3c0a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f39d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2f3c09ae20 0x7f2f3c0a32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:34846/0 (socket says 172.21.15.86:34846) 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f39d9b640 1 -- 172.21.15.86:0/2805847493 learned_addr learned my addr 172.21.15.86:0/2805847493 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request con 0x7f2f3c0a4970 auth_method 0 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_done global_id 164549 payload 274 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 1 -- 172.21.15.86:0/2805847493 >> v1:172.21.15.203:6789/0 conn(0x7f2f3c13d530 legacy=0x7f2f3c13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 1 -- 172.21.15.86:0/2805847493 >> v1:172.21.15.86:6789/0 conn(0x7f2f3c09ae20 legacy=0x7f2f3c0a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 1 monclient: found mon.noname-b 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.533+0000 7f2f3a59c640 1 -- 172.21.15.86:0/2805847493 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c06cb70 con 0x7f2f3c0a4970 2024-09-17T15:45:07.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3a59c640 1 -- 172.21.15.86:0/2805847493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 msgr2=0x7f2f3c0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/2805847493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 0x7f2f3c0a2d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097f0 tx=0x7f2f3001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb7615ffbd677b8b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 1 -- 172.21.15.86:0/2805847493 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300b0070 con 0x7f2f3c0a4970 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: dump: 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.536 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 1 -- 172.21.15.86:0/2805847493 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f300ab600 con 0x7f2f3c0a4970 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 1 -- 172.21.15.86:0/2805847493 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300ab9d0 con 0x7f2f3c0a4970 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 5 monclient: authenticate success, global_id 164549 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 4 set_mon_vals no callback set 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f3959a640 10 monclient: dump: 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.537 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.538 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 10 monclient: shutdown 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 -- 172.21.15.86:0/2805847493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 msgr2=0x7f2f3c0a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 --2- 172.21.15.86:0/2805847493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 0x7f2f3c0a2d90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097f0 tx=0x7f2f3001c3a0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 -- 172.21.15.86:0/2805847493 shutdown_connections 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 --2- 172.21.15.86:0/2805847493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c0a4970 0x7f2f3c0a2d90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 -- 172.21.15.86:0/2805847493 >> 172.21.15.86:0/2805847493 conn(0x7f2f3c0a3e80 msgr2=0x7f2f3c0a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 -- 172.21.15.86:0/2805847493 shutdown_connections 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 1 -- 172.21.15.86:0/2805847493 wait complete. 2024-09-17T15:45:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.537+0000 7f2f40908640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient: monmap: 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 Processor -- start 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 -- start start 2024-09-17T15:45:07.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient: init 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 0x7f2f3c099af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3c09ae20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 0x7f2f3c091410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3c0a4970 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3c13d530 0x7f2f3c091950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3c13d530 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c09bc60 con 0x7f2f3c0a4970 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c0a4d50 con 0x7f2f3c13d530 2024-09-17T15:45:07.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c143a70 con 0x7f2f3c09ae20 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f40908640 10 monclient(hunting): authenticate will time out at 2040.508998s 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 0x7f2f3c091410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 0x7f2f3c099af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 0x7f2f3c099af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50314/0 (socket says 172.21.15.86:50314) 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 1 -- 172.21.15.86:0/3054447643 learned_addr learned my addr 172.21.15.86:0/3054447643 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request con 0x7f2f3c09ae20 auth_method 0 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): get_auth_request con 0x7f2f3c0a4970 auth_method 0 2024-09-17T15:45:07.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3ad9d640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3c13d530 0x7f2f3c091950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3ad9d640 10 monclient(hunting): get_auth_request con 0x7f2f3c13d530 auth_method 0 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient(hunting): handle_auth_done global_id 184353 payload 824 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 msgr2=0x7f2f3c099af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 0x7f2f3c099af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3c13d530 msgr2=0x7f2f3c091950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2f3c13d530 0x7f2f3c091950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 monclient: found mon.noname-a 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c093f50 con 0x7f2f3c0a4970 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09ae20 0x7f2f3c099af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 msgr2=0x7f2f3c091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.541+0000 7f2f39d9b640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 0x7f2f3c091410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00c3e0 tx=0x7f2f2c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c017070 con 0x7f2f3c0a4970 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: dump: 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.548 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2c0070d0 con 0x7f2f3c0a4970 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0074c0 con 0x7f2f3c0a4970 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 10 monclient: dump: 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f38d99640 4 set_mon_vals no callback set 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 5 monclient: authenticate success, global_id 184353 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 monclient: _renew_subs 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3c0a6040 con 0x7f2f3c0a4970 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 monclient: _renew_subs 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:45:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3c093c70 con 0x7f2f3c0a4970 2024-09-17T15:45:07.551 INFO:tasks.rados.rados.0.smithi203.stdout:2872: done (3 left) 2024-09-17T15:45:07.551 INFO:tasks.rados.rados.0.smithi203.stdout:2873: delete oid 11 current snap is 315 2024-09-17T15:45:07.551 INFO:tasks.rados.rados.0.smithi203.stdout:2869: expect (ObjNum 988 snap 309 seq_num 988) 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 0x7f2f3c09f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c092270 con 0x7f2f3c091e90 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 0x7f2f3c09f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient: get_auth_request con 0x7f2f3c091e90 auth_method 0 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f39d9b640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 0x7f2f3c091410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00c3e0 tx=0x7f2f2c00c8a0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f2f2c04f3c0 osd_map(643..643 src has 438..643) 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2f2c01a070 con 0x7f2f3c0a4970 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_done global_id 164555 payload 824 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3a59c640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f3c0a2e10 con 0x7f2f3c091e90 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f2affd640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bb60 0x7f2f1c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3ad9d640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bb60 0x7f2f1c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.545+0000 7f2f3ad9d640 10 monclient: get_auth_request con 0x7f2f1c03bb60 auth_method 0 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f3a59c640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 msgr2=0x7f2f3c09f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 0x7f2f3c09f980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097f0 tx=0x7f2f30019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300bc070 con 0x7f2f3c091e90 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 10 monclient: dump: 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f300b1040 con 0x7f2f3c091e90 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 msgr2=0x7f2f3c09f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 0x7f2f3c09f980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097f0 tx=0x7f2f30019140 comp rx=0 tx=0).stop 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 msgr2=0x7f2f3c09f980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f2affd640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c091e90 0x7f2f3c09f980 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f300097f0 tx=0x7f2f30019140 comp rx=0 tx=0).stop 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.549+0000 7f2f3ad9d640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bb60 0x7f2f1c03e020 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0a5250 tx=0x7f2f34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f40908640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f40908640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 0x7f2f3c0a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f40908640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c0927b0 con 0x7f2f3c09fec0 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 0x7f2f3c0a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient: get_auth_request con 0x7f2f3c09fec0 auth_method 0 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 10 monclient(hunting): handle_auth_done global_id 164558 payload 824 2024-09-17T15:45:07.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.553+0000 7f2f3a59c640 1 -- 172.21.15.86:0/3054447643 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f3c0a2b50 con 0x7f2f3c09fec0 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f3a59c640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 msgr2=0x7f2f3c0a02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f3a59c640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 0x7f2f3c0a02d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c10 tx=0x7f2f30009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f300b0040 con 0x7f2f3c09fec0 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 10 monclient: dump: 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.559 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f2f30003be0 con 0x7f2f3c09fec0 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 msgr2=0x7f2f3c0a02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 0x7f2f3c0a02d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c10 tx=0x7f2f30009190 comp rx=0 tx=0).stop 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 msgr2=0x7f2f3c0a02d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f3c09fec0 0x7f2f3c0a02d0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c10 tx=0x7f2f30009190 comp rx=0 tx=0).stop 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 210745, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:45:07.561 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.562 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.563 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:07.564 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.565 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.566 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.568 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.570 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:07.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:45:07.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:07.573 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:07.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bb60 msgr2=0x7f2f1c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2f1c03bb60 0x7f2f1c03e020 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c0a5250 tx=0x7f2f34009040 comp rx=0 tx=0).stop 2024-09-17T15:45:07.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 10 monclient: shutdown 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 msgr2=0x7f2f3c091410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 --2- 172.21.15.86:0/3054447643 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f3c0a4970 0x7f2f3c091410 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00c3e0 tx=0x7f2f2c00c8a0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f3ad9d640 1 -- 172.21.15.86:0/3054447643 reap_dead start 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f2affd640 1 -- 172.21.15.86:0/3054447643 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(643..643 src has 438..643) ==== 4767+0+0 (unknown 0 0 0) 0x7f2f2c04f3c0 con 0x7f2f3c0a4970 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f3ad9d640 1 -- 172.21.15.86:0/3054447643 reap_dead start 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 shutdown_connections 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 >> 172.21.15.86:0/3054447643 conn(0x7f2f3c0a3e80 msgr2=0x7f2f3c0a1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 shutdown_connections 2024-09-17T15:45:07.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.557+0000 7f2f40908640 1 -- 172.21.15.86:0/3054447643 wait complete. 2024-09-17T15:45:07.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:07.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:45:07.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:45:07.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: get_monmap_and_config 2024-09-17T15:45:07.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: monmap: 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:07.696380+0000 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:07.696380+0000 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:07.690 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 Processor -- start 2024-09-17T15:45:07.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 -- start start 2024-09-17T15:45:07.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: init 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 0x7f0b7c0a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-b con 0x7f0b7c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0b7c0a1590 0x7f0b7c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-c con 0x7f0b7c0a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b7c135f90 0x7f0b7c136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-a con 0x7f0b7c135f90 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0b7c0a2410 con 0x7f0b7c135f90 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0b7c0a2130 con 0x7f0b7c0a1590 2024-09-17T15:45:07.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7c0a26f0 con 0x7f0b7c0a0c70 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b83345640 10 monclient(hunting): authenticate will time out at 2040.661873s 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 0x7f0b7c0a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b7c135f90 0x7f0b7c136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b7c135f90 0x7f0b7c136420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60908/0 (socket says 172.21.15.86:60908) 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 -- 172.21.15.86:0/399137664 learned_addr learned my addr 172.21.15.86:0/399137664 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): get_auth_request con 0x7f0b7c135f90 auth_method 0 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 0x7f0b7c0a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50346/0 (socket says 172.21.15.86:50346) 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): get_auth_request con 0x7f0b7c0a0c70 auth_method 0 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b808e0640 1 --2- 172.21.15.86:0/399137664 >> v2:172.21.15.203:3300/0 conn(0x7f0b7c0a1590 0x7f0b7c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b808e0640 10 monclient(hunting): get_auth_request con 0x7f0b7c0a1590 auth_method 0 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b808e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b808e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b808e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient(hunting): handle_auth_done global_id 184356 payload 274 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 -- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 msgr2=0x7f0b7c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 --2- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 0x7f0b7c0a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 -- 172.21.15.86:0/399137664 >> v2:172.21.15.203:3300/0 conn(0x7f0b7c0a1590 msgr2=0x7f0b7c135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 --2- 172.21.15.86:0/399137664 >> v2:172.21.15.203:3300/0 conn(0x7f0b7c0a1590 0x7f0b7c135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 monclient: found mon.noname-a 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:45:07.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.693+0000 7f0b818e2640 1 -- 172.21.15.86:0/399137664 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7c09f9f0 con 0x7f0b7c135f90 2024-09-17T15:45:07.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b818e2640 1 -- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 msgr2=0x7f0b7c136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b818e2640 1 --2- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c136420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c06e2e0 tx=0x7f0b6c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548aea2b85fe98b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 1 -- 172.21.15.86:0/399137664 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0b1070 con 0x7f0b7c135f90 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: dump: 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 1 -- 172.21.15.86:0/399137664 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6c0ac620 con 0x7f0b7c135f90 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 1 -- 172.21.15.86:0/399137664 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0ac9f0 con 0x7f0b7c135f90 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 5 monclient: authenticate success, global_id 184356 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 4 set_mon_vals no callback set 2024-09-17T15:45:07.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b7bfff640 10 monclient: dump: 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 10 monclient: shutdown 2024-09-17T15:45:07.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 -- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 msgr2=0x7f0b7c136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 --2- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c136420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c06e2e0 tx=0x7f0b6c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 -- 172.21.15.86:0/399137664 shutdown_connections 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 --2- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c136420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 --2- 172.21.15.86:0/399137664 >> v2:172.21.15.203:3300/0 conn(0x7f0b7c0a1590 0x7f0b7c135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 --2- 172.21.15.86:0/399137664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a0c70 0x7f0b7c0a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 -- 172.21.15.86:0/399137664 >> 172.21.15.86:0/399137664 conn(0x7f0b7c0a2f40 msgr2=0x7f0b7c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 -- 172.21.15.86:0/399137664 shutdown_connections 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.697+0000 7f0b83345640 1 -- 172.21.15.86:0/399137664 wait complete. 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient: monmap: 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:07.707 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 Processor -- start 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 -- start start 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient: init 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c0a0c70 0x7f0b7c1cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-b con 0x7f0b7c0a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-c con 0x7f0b7c0a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c1cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): picked mon.noname-a con 0x7f0b7c135f90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c0a00d0 con 0x7f0b7c135f90 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c09ff50 con 0x7f0b7c0a0c70 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7c13f740 con 0x7f0b7c0a1590 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b83345640 10 monclient(hunting): authenticate will time out at 2040.669152s 2024-09-17T15:45:07.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c1cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50360/0 (socket says 172.21.15.86:50360) 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 -- 172.21.15.86:0/872834030 learned_addr learned my addr 172.21.15.86:0/872834030 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c1cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60920/0 (socket says 172.21.15.86:60920) 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): get_auth_request con 0x7f0b7c0a1590 auth_method 0 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): get_auth_request con 0x7f0b7c135f90 auth_method 0 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b810e1640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c0a0c70 0x7f0b7c1cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b810e1640 10 monclient(hunting): get_auth_request con 0x7f0b7c0a0c70 auth_method 0 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b810e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b810e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b810e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient(hunting): handle_auth_done global_id 164564 payload 824 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c0a0c70 msgr2=0x7f0b7c1cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c0a0c70 0x7f0b7c1cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 msgr2=0x7f0b7c1cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c1cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 monclient: found mon.noname-c 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7c09d460 con 0x7f0b7c0a1590 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b818e2640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7c135f90 0x7f0b7c1cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 msgr2=0x7f0b7c134fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.701+0000 7f0b808e0640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000d800 tx=0x7f0b7000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000b8d0 con 0x7f0b7c0a1590 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:07.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: dump: 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b70019070 con 0x7f0b7c0a1590 2024-09-17T15:45:07.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000bc70 con 0x7f0b7c0a1590 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 5 monclient: authenticate success, global_id 164564 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 monclient: _renew_subs 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7c122680 con 0x7f0b7c0a1590 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 10 monclient: dump: 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b7b7fe640 4 set_mon_vals no callback set 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 monclient: _renew_subs 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7c0734f0 con 0x7f0b7c0a1590 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 0x7f0b7c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c135a50 con 0x7f0b7c1354e0 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b808e0640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000d800 tx=0x7f0b7000dcc0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f0b70050310 osd_map(643..643 src has 438..643) 2024-09-17T15:45:07.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0b7001e460 con 0x7f0b7c0a1590 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 0x7f0b7c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient: get_auth_request con 0x7f0b7c1354e0 auth_method 0 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b79ffb640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6803bbb0 0x7f0b6803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b818e2640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6803bbb0 0x7f0b6803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b818e2640 10 monclient: get_auth_request con 0x7f0b6803bbb0 auth_method 0 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 10 monclient(hunting): handle_auth_done global_id 204139 payload 824 2024-09-17T15:45:07.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.705+0000 7f0b810e1640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b7c072720 con 0x7f0b7c1354e0 2024-09-17T15:45:07.717 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1090384 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b810e1640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 msgr2=0x7f0b7c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b810e1640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 0x7f0b7c02d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b740115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74011ea0 con 0x7f0b7c1354e0 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:07.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 10 monclient: dump: 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b818e2640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6803bbb0 0x7f0b6803e070 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0a40b0 tx=0x7f0b6c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0b740039a0 con 0x7f0b7c1354e0 2024-09-17T15:45:07.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 msgr2=0x7f0b7c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 0x7f0b7c02d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b740115a0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 msgr2=0x7f0b7c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.709+0000 7f0b79ffb640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c1354e0 0x7f0b7c02d450 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b740115a0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b83345640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b83345640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 0x7f0b7c02d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b83345640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7c1cd5d0 con 0x7f0b7c121160 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 0x7f0b7c02d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient: get_auth_request con 0x7f0b7c121160 auth_method 0 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 10 monclient(hunting): handle_auth_done global_id 204142 payload 824 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.713+0000 7f0b810e1640 1 -- 172.21.15.86:0/872834030 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0b7c1221a0 con 0x7f0b7c121160 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b810e1640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 msgr2=0x7f0b7c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b810e1640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 0x7f0b7c02d990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b74004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b74004250 con 0x7f0b7c121160 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 10 monclient: dump: 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f0b74004230 con 0x7f0b7c121160 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 msgr2=0x7f0b7c02d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 0x7f0b7c02d990 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b74004a50 comp rx=0 tx=0).stop 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 msgr2=0x7f0b7c02d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b7c121160 0x7f0b7c02d990 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0b74007f00 tx=0x7f0b74004a50 comp rx=0 tx=0).stop 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 162, 2024-09-17T15:45:07.724 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10319, 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.725 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.726 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:07.728 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:07.729 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.730 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.731 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.734 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.735 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:07.736 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:07.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:07.737 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:07.737 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:07.738 INFO:tasks.rados.rados.0.smithi203.stdout:2870: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:45:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6803bbb0 msgr2=0x7f0b6803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6803bbb0 0x7f0b6803e070 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f0b7c0a40b0 tx=0x7f0b6c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 10 monclient: shutdown 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 msgr2=0x7f0b7c134fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 --2- 172.21.15.86:0/872834030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7c0a1590 0x7f0b7c134fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000d800 tx=0x7f0b7000dcc0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b818e2640 1 -- 172.21.15.86:0/872834030 reap_dead start 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b79ffb640 1 -- 172.21.15.86:0/872834030 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(643..643 src has 438..643) ==== 4767+0+0 (unknown 0 0 0) 0x7f0b70050310 con 0x7f0b7c0a1590 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b818e2640 1 -- 172.21.15.86:0/872834030 reap_dead start 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 shutdown_connections 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 >> 172.21.15.86:0/872834030 conn(0x7f0b7c0a2f40 msgr2=0x7f0b7c13c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 shutdown_connections 2024-09-17T15:45:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.717+0000 7f0b83345640 1 -- 172.21.15.86:0/872834030 wait complete. 2024-09-17T15:45:07.740 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:45:07.740 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:45:07.740 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:45:07.740 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:45:07.740 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:45:07.740 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:45:07.740 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:45:07.741 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1227633 2024-09-17T15:45:07.752 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:45:07.752 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:45:07.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:07.756 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:07.813 INFO:tasks.rados.rados.0.smithi203.stdout:2871: expect (ObjNum 588 snap 174 seq_num 588) 2024-09-17T15:45:07.813 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:07.887 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 516037 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: get_monmap_and_config 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: monmap: 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:07.909742+0000 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:07.909742+0000 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:07.903 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:07.904 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:07.904 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:07.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 Processor -- start 2024-09-17T15:45:07.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 -- start start 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: init 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1c3c0689d0 0x7f1c3c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c0689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 0x7f1c3c135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c135ae0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3c074500 con 0x7f1c3c135ae0 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f1c3c09c8b0 con 0x7f1c3c0689d0 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3c09cb90 con 0x7f1c3c068db0 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c41dda640 10 monclient(hunting): authenticate will time out at 2040.875254s 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3bfff640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 msgr2=0x7f1c3c135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3bfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 0x7f1c3c135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50362/0 (socket says 172.21.15.86:50362) 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 -- 172.21.15.86:0/1029067897 learned_addr learned my addr 172.21.15.86:0/1029067897 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): get_auth_request con 0x7f1c3c068db0 auth_method 0 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3b7fe640 1 --2- 172.21.15.86:0/1029067897 >> v2:172.21.15.203:3300/0 conn(0x7f1c3c0689d0 0x7f1c3c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3b7fe640 10 monclient(hunting): get_auth_request con 0x7f1c3c0689d0 auth_method 0 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient(hunting): handle_auth_done global_id 164567 payload 274 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 -- 172.21.15.86:0/1029067897 >> v2:172.21.15.203:3300/0 conn(0x7f1c3c0689d0 msgr2=0x7f1c3c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 --2- 172.21.15.86:0/1029067897 >> v2:172.21.15.203:3300/0 conn(0x7f1c3c0689d0 0x7f1c3c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 -- 172.21.15.86:0/1029067897 >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 msgr2=0x7f1c3c135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 --2- 172.21.15.86:0/1029067897 >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 0x7f1c3c135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 monclient: found mon.noname-b 2024-09-17T15:45:07.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.905+0000 7f1c3affd640 1 -- 172.21.15.86:0/1029067897 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c09ce70 con 0x7f1c3c068db0 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 -- 172.21.15.86:0/1029067897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 msgr2=0x7f1c3c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 --2- 172.21.15.86:0/1029067897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c09cac0 tx=0x7f1c3001cda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699ab2472ff39217 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 1 -- 172.21.15.86:0/1029067897 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300b0070 con 0x7f1c3c068db0 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: dump: 2024-09-17T15:45:07.913 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 1 -- 172.21.15.86:0/1029067897 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c300ab600 con 0x7f1c3c068db0 2024-09-17T15:45:07.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 1 -- 172.21.15.86:0/1029067897 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300ab9d0 con 0x7f1c3c068db0 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 5 monclient: authenticate success, global_id 164567 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 4 set_mon_vals no callback set 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3a7fc640 10 monclient: dump: 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.915 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: shutdown 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- 172.21.15.86:0/1029067897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 msgr2=0x7f1c3c0691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- 172.21.15.86:0/1029067897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c09cac0 tx=0x7f1c3001cda0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- 172.21.15.86:0/1029067897 shutdown_connections 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- 172.21.15.86:0/1029067897 >> v2:172.21.15.86:3300/0 conn(0x7f1c3c135ae0 0x7f1c3c135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- 172.21.15.86:0/1029067897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c0691d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- 172.21.15.86:0/1029067897 >> v2:172.21.15.203:3300/0 conn(0x7f1c3c0689d0 0x7f1c3c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- 172.21.15.86:0/1029067897 >> 172.21.15.86:0/1029067897 conn(0x7f1c3c0a6440 msgr2=0x7f1c3c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- 172.21.15.86:0/1029067897 shutdown_connections 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- 172.21.15.86:0/1029067897 wait complete. 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: build_initial_monmap 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: monmap: 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 Processor -- start 2024-09-17T15:45:07.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- start start 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: init 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 0x7f1c3c1ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-c con 0x7f1c3c068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 0x7f1c3c1d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): picked mon.noname-b con 0x7f1c3c135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c09c2d0 con 0x7f1c3c0689d0 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3c09be70 con 0x7f1c3c135ae0 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c3c13f820 con 0x7f1c3c068db0 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c41dda640 10 monclient(hunting): authenticate will time out at 2040.882944s 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3b7fe640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 msgr2=0x7f1c3c1ccb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 0x7f1c3c1ccb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50372/0 (socket says 172.21.15.86:50372) 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 -- 172.21.15.86:0/2780509778 learned_addr learned my addr 172.21.15.86:0/2780509778 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): get_auth_request con 0x7f1c3c068db0 auth_method 0 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3bfff640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 0x7f1c3c1d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient(hunting): handle_auth_done global_id 164570 payload 824 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 -- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 msgr2=0x7f1c3c1d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 0x7f1c3c1d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 -- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 msgr2=0x7f1c3c1ccb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 0x7f1c3c1ccb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 monclient: found mon.noname-c 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3affd640 1 -- 172.21.15.86:0/2780509778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c3c0a39b0 con 0x7f1c3c068db0 2024-09-17T15:45:07.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3bfff640 10 monclient: get_auth_request con 0x7f1c3c135ae0 auth_method 0 2024-09-17T15:45:07.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.913+0000 7f1c3bfff640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 0x7f1c3c1d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c3affd640 1 -- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 msgr2=0x7f1c3c1350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c3affd640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c30019f70 tx=0x7f1c300190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c3affd640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c30019f70 tx=0x7f1c300190e0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f1c300b0070 mon_map magic: 0 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300b0070 con 0x7f1c3c068db0 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: dump: 2024-09-17T15:45:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:07.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c300b1040 con 0x7f1c3c068db0 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 5 monclient: authenticate success, global_id 164570 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _renew_subs 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c3c09d460 con 0x7f1c3c068db0 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c39ffb640 4 set_mon_vals no callback set 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _renew_subs 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c3c122ab0 con 0x7f1c3c068db0 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:07.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c3c0691d0 con 0x7f1c3c068db0 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c300aa9e0 con 0x7f1c3c068db0 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 10 monclient: dump: 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:07.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1c300aac00 con 0x7f1c3c068db0 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c23fff640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 0x7f1c2803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c3b7fe640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 0x7f1c2803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.917+0000 7f1c3b7fe640 10 monclient: get_auth_request con 0x7f1c2803bb10 auth_method 0 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.921+0000 7f1c3b7fe640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 0x7f1c2803dfd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009660 tx=0x7f1c34009930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.921+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:45:07.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:07.925+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(643..643 src has 438..643) ==== 4767+0+0 (secure 0 0 0) 0x7f1c300e9ee0 con 0x7f1c3c068db0 2024-09-17T15:45:07.978 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1813632 2024-09-17T15:45:08.023 INFO:tasks.rados.rados.0.smithi203.stdout:2869: done (3 left) 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2870: done (2 left) 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2871: done (1 left) 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2873: done (0 left) 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2874: read oid 29 snap -1 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2874: expect (ObjNum 966 snap 293 seq_num 966) 2024-09-17T15:45:08.024 INFO:tasks.rados.rados.0.smithi203.stdout:2875: snap_create 2024-09-17T15:45:08.030 INFO:tasks.rados.rados.0.smithi203.stdout:2874: expect (ObjNum 966 snap 293 seq_num 966) 2024-09-17T15:45:08.128 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:08.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.541+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f1c30016110 con 0x7f1c3c068db0 2024-09-17T15:45:08.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.541+0000 7f1c23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:08.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.541+0000 7f1c23fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:08.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c41dda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:08.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c41dda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:08.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c41dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:08.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1c3c06d4f0 con 0x7f1c3c068db0 2024-09-17T15:45:08.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c23fff640 1 -- 172.21.15.86:0/2780509778 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1c300e88f0 con 0x7f1c3c068db0 2024-09-17T15:45:08.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:08.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.765+0000 7f1c23fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:08.764 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 msgr2=0x7f1c2803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 0x7f1c2803dfd0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1c34009660 tx=0x7f1c34009930 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 10 monclient: shutdown 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 msgr2=0x7f1c3c1350f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c30019f70 tx=0x7f1c300190e0 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 shutdown_connections 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1c2803bb10 0x7f1c2803dfd0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c3c135ae0 0x7f1c3c1d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c3c068db0 0x7f1c3c1350f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 --2- 172.21.15.86:0/2780509778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c3c0689d0 0x7f1c3c1ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:08.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 >> 172.21.15.86:0/2780509778 conn(0x7f1c3c0a6440 msgr2=0x7f1c3c0a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:08.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 shutdown_connections 2024-09-17T15:45:08.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:08.769+0000 7f1c41dda640 1 -- 172.21.15.86:0/2780509778 wait complete. 2024-09-17T15:45:11.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: get_monmap_and_config 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: build_initial_monmap 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: monmap: 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:11.916962+0000 2024-09-17T15:45:11.910 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:11.916962+0000 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:11.911 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 Processor -- start 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 -- start start 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: init 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T15:45:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 0x7fc53009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-a con 0x7fc530069260 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc530074c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-b con 0x7fc53009c6c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc53009e410 con 0x7fc530074c30 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc53009d150 con 0x7fc530069260 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc53009ce80 con 0x7fc53009c6c0 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc537a59640 10 monclient(hunting): authenticate will time out at 2044.882444s 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc5357f5640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 msgr2=0x7fc53009ba70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:11.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc5357f5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 0x7fc53009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49832/0 (socket says 172.21.15.86:49832) 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 -- 172.21.15.86:0/3817970056 learned_addr learned my addr 172.21.15.86:0/3817970056 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): get_auth_request con 0x7fc53009c6c0 auth_method 0 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient(hunting): handle_auth_done global_id 164573 payload 274 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 -- 172.21.15.86:0/3817970056 >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 msgr2=0x7fc53009ba70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 --2- 172.21.15.86:0/3817970056 >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 0x7fc53009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 -- 172.21.15.86:0/3817970056 >> v1:172.21.15.203:6789/0 conn(0x7fc530074c30 legacy=0x7fc53009bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:11.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient: _finish_hunting 0 2024-09-17T15:45:11.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 monclient: found mon.noname-b 2024-09-17T15:45:11.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:11.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:11.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.913+0000 7fc535ff6640 1 -- 172.21.15.86:0/3817970056 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53009e6f0 con 0x7fc53009c6c0 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc535ff6640 1 -- 172.21.15.86:0/3817970056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 msgr2=0x7fc53013ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc535ff6640 1 --2- 172.21.15.86:0/3817970056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc52800d180 tx=0x7fc52801eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0f9202d4387ec6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 1 -- 172.21.15.86:0/3817970056 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52800b3b0 con 0x7fc53009c6c0 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: dump: 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:11.918 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 1 -- 172.21.15.86:0/3817970056 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5280a7750 con 0x7fc53009c6c0 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 5 monclient: authenticate success, global_id 164573 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 1 -- 172.21.15.86:0/3817970056 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280a7b20 con 0x7fc53009c6c0 2024-09-17T15:45:11.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 4 set_mon_vals no callback set 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc527fff640 10 monclient: dump: 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 10 monclient: shutdown 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 1 -- 172.21.15.86:0/3817970056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 msgr2=0x7fc53013ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.917+0000 7fc537a59640 1 --2- 172.21.15.86:0/3817970056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc52800d180 tx=0x7fc52801eaf0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- 172.21.15.86:0/3817970056 shutdown_connections 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 --2- 172.21.15.86:0/3817970056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc53013ba80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 --2- 172.21.15.86:0/3817970056 >> v2:172.21.15.86:3300/0 conn(0x7fc530069260 0x7fc53009ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- 172.21.15.86:0/3817970056 >> 172.21.15.86:0/3817970056 conn(0x7fc5300a6440 msgr2=0x7fc5300a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- 172.21.15.86:0/3817970056 shutdown_connections 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- 172.21.15.86:0/3817970056 wait complete. 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient: build_initial_monmap 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient: monmap: 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:11.921 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 Processor -- start 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- start start 2024-09-17T15:45:11.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient: init 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 0x7fc53011fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-b con 0x7fc530069260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 0x7fc53011dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-a con 0x7fc530074c30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): picked mon.noname-c con 0x7fc53009c6c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc53013e390 con 0x7fc530074c30 2024-09-17T15:45:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc53013e210 con 0x7fc530069260 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc53013e510 con 0x7fc53009c6c0 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc537a59640 10 monclient(hunting): authenticate will time out at 2044.889496s 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc534ff4640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 msgr2=0x7fc53011dd40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc534ff4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 0x7fc53011dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49838/0 (socket says 172.21.15.86:49838) 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 -- 172.21.15.86:0/440629072 learned_addr learned my addr 172.21.15.86:0/440629072 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): get_auth_request con 0x7fc53009c6c0 auth_method 0 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc5357f5640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 0x7fc53011fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient(hunting): handle_auth_done global_id 164576 payload 824 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 -- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 msgr2=0x7fc53011fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 0x7fc53011fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 -- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 msgr2=0x7fc53011dd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 0x7fc53011dd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient: _finish_hunting 0 2024-09-17T15:45:11.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 monclient: found mon.noname-c 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc535ff6640 1 -- 172.21.15.86:0/440629072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53011f340 con 0x7fc53009c6c0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc5357f5640 10 monclient: get_auth_request con 0x7fc530069260 auth_method 0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.921+0000 7fc5357f5640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 0x7fc53011fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc535ff6640 1 -- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 msgr2=0x7fc530136d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc535ff6640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5280b1040 tx=0x7fc52801b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5280be020 con 0x7fc53009c6c0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: dump: 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc528009a10 con 0x7fc53009c6c0 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 5 monclient: authenticate success, global_id 164576 2024-09-17T15:45:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _renew_subs 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5301216a0 con 0x7fc53009c6c0 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc528009e00 con 0x7fc53009c6c0 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 10 monclient: dump: 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:11.931 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _renew_subs 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc526ffd640 4 set_mon_vals no callback set 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc530137250 con 0x7fc53009c6c0 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53009ba70 con 0x7fc53009c6c0 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc5280c1020 con 0x7fc53009c6c0 2024-09-17T15:45:11.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 0x7fc51403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(643..643 src has 438..643) ==== 4767+0+0 (secure 0 0 0) 0x7fc5280ed180 con 0x7fc53009c6c0 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc5357f5640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 0x7fc51403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.925+0000 7fc5357f5640 10 monclient: get_auth_request con 0x7fc51403bb10 auth_method 0 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.929+0000 7fc5357f5640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 0x7fc51403dfd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc520003990 tx=0x7fc520008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.929+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc528018800 con 0x7fc53009c6c0 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.929+0000 7fc525ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:11.929+0000 7fc525ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:12.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc537a59640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:12.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc537a59640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:12.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc537a59640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:12.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc53006d4f0 con 0x7fc53009c6c0 2024-09-17T15:45:12.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc525ffb640 1 -- 172.21.15.86:0/440629072 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc5280a7e10 con 0x7fc53009c6c0 2024-09-17T15:45:12.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc525ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:12.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.153+0000 7fc525ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:12.153 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:12.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 msgr2=0x7fc51403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:12.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 0x7fc51403dfd0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc520003990 tx=0x7fc520008040 comp rx=0 tx=0).stop 2024-09-17T15:45:12.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 10 monclient: shutdown 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 msgr2=0x7fc530136d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5280b1040 tx=0x7fc52801b890 comp rx=0 tx=0).stop 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 shutdown_connections 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc51403bb10 0x7fc51403dfd0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc53009c6c0 0x7fc530136d10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc530074c30 0x7fc53011dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 --2- 172.21.15.86:0/440629072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc530069260 0x7fc53011fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 >> 172.21.15.86:0/440629072 conn(0x7fc5300a6440 msgr2=0x7fc530070870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 shutdown_connections 2024-09-17T15:45:12.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:12.157+0000 7fc537a59640 1 -- 172.21.15.86:0/440629072 wait complete. 2024-09-17T15:45:15.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: get_monmap_and_config 2024-09-17T15:45:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: build_initial_monmap 2024-09-17T15:45:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: monmap: 2024-09-17T15:45:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:15.309267+0000 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:15.309267+0000 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:15.303 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 Processor -- start 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 -- start start 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: init 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd49c0689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:15.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd49c0694c0 0x7fd49c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c0694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd49c13aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c09c8b0 con 0x7fd49c13aff0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd49c09cb90 con 0x7fd49c0689d0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd49c074500 con 0x7fd49c0694c0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd4a18a3640 10 monclient(hunting): authenticate will time out at 2048.274713s 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49b7fe640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7fd49c13aff0 legacy=0x7fd49c13d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd49c0694c0 0x7fd49c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd49c0694c0 0x7fd49c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:46712/0 (socket says 172.21.15.86:46712) 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 -- 172.21.15.86:0/213701971 learned_addr learned my addr 172.21.15.86:0/213701971 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49affd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd49c0689d0 0x7fd49c068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:33208/0 (socket says 172.21.15.86:33208) 2024-09-17T15:45:15.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request con 0x7fd49c0694c0 auth_method 0 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient(hunting): handle_auth_done global_id 204148 payload 274 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 -- 172.21.15.86:0/213701971 >> v1:172.21.15.203:6789/0 conn(0x7fd49c0689d0 legacy=0x7fd49c068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 -- 172.21.15.86:0/213701971 >> v1:172.21.15.86:6789/0 conn(0x7fd49c13aff0 legacy=0x7fd49c13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 monclient: found mon.noname-c 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:15.307 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:45:15.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.305+0000 7fd49a7fc640 1 -- 172.21.15.86:0/213701971 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c09ce70 con 0x7fd49c0694c0 2024-09-17T15:45:15.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49a7fc640 1 -- 172.21.15.86:0/213701971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 msgr2=0x7fd49c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:15.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/213701971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c06f840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd49c069a70 tx=0x7fd48c01c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4d74d26f2bb0f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:15.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 1 -- 172.21.15.86:0/213701971 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48c0b1070 con 0x7fd49c0694c0 2024-09-17T15:45:15.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:15.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: dump: 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:15.312 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: _finish_auth 0 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 1 -- 172.21.15.86:0/213701971 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd48c0ac600 con 0x7fd49c0694c0 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 1 -- 172.21.15.86:0/213701971 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49c09c2d0 con 0x7fd49c0694c0 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd499ffb640 10 monclient: dump: 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:15.313 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 5 monclient: authenticate success, global_id 204148 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 4 set_mon_vals no callback set 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: shutdown 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- 172.21.15.86:0/213701971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 msgr2=0x7fd49c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:15.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/213701971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c06f840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd49c069a70 tx=0x7fd48c01c400 comp rx=0 tx=0).stop 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- 172.21.15.86:0/213701971 shutdown_connections 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/213701971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c06f840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- 172.21.15.86:0/213701971 >> 172.21.15.86:0/213701971 conn(0x7fd49c0a6440 msgr2=0x7fd49c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- 172.21.15.86:0/213701971 shutdown_connections 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- 172.21.15.86:0/213701971 wait complete. 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: build_initial_monmap 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: monmap: 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:15.315 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.316 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:15.316 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:15.316 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:15.316 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 Processor -- start 2024-09-17T15:45:15.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- start start 2024-09-17T15:45:15.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: init 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd49c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c136260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-b con 0x7fd49c0694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 0x7fd49c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): picked mon.noname-a con 0x7fd49c13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c141c30 con 0x7fd49c13aff0 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c141ab0 con 0x7fd49c0694c0 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd49c141db0 con 0x7fd49c0689d0 2024-09-17T15:45:15.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd4a18a3640 10 monclient(hunting): authenticate will time out at 2048.283716s 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49b7fe640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 msgr2=0x7fd49c09d770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 0x7fd49c09d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49842/0 (socket says 172.21.15.86:49842) 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 1 -- 172.21.15.86:0/1826908899 learned_addr learned my addr 172.21.15.86:0/1826908899 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): get_auth_request con 0x7fd49c0689d0 auth_method 0 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:15.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.313+0000 7fd49affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c136260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 10 monclient(hunting): handle_auth_done global_id 164579 payload 824 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 msgr2=0x7fd49c136260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c136260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 msgr2=0x7fd49c09d770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 0x7fd49c09d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 monclient: found mon.noname-c 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 -- 172.21.15.86:0/1826908899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c1217c0 con 0x7fd49c0689d0 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49a7fc640 10 monclient: get_auth_request con 0x7fd49c0694c0 auth_method 0 2024-09-17T15:45:15.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 msgr2=0x7fd49c135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd49affd640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c460 tx=0x7fd49000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490017070 con 0x7fd49c0689d0 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: dump: 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:15.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 5 monclient: authenticate success, global_id 164579 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd490005820 con 0x7fd49c0689d0 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _renew_subs 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd490005c10 con 0x7fd49c0689d0 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:15.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c13e190 con 0x7fd49c0689d0 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4997fa640 4 set_mon_vals no callback set 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 10 monclient: dump: 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:15.326 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _renew_subs 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd49c09d460 con 0x7fd49c0689d0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49c06f840 con 0x7fd49c0689d0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd49001a070 con 0x7fd49c0689d0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.317+0000 7fd483fff640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(643..643 src has 438..643) ==== 4767+0+0 (secure 0 0 0) 0x7fd49004f500 con 0x7fd49c0689d0 2024-09-17T15:45:15.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 10 monclient: get_auth_request con 0x7fd48803bb10 auth_method 0 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 msgr2=0x7fd48803dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 msgr2=0x7fd48803dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.321+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.325+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd490005df0 con 0x7fd49c0689d0 2024-09-17T15:45:15.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.325+0000 7fd483fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:15.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.325+0000 7fd483fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.525+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.525+0000 7fd49a7fc640 10 monclient: get_auth_request con 0x7fd48803bb10 auth_method 0 2024-09-17T15:45:15.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.525+0000 7fd49a7fc640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd49c069940 tx=0x7fd48c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:15.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd4a18a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:15.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd4a18a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:15.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd4a18a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:15.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd49c073720 con 0x7fd49c0689d0 2024-09-17T15:45:15.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd483fff640 1 -- 172.21.15.86:0/1826908899 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd49001bc50 con 0x7fd49c0689d0 2024-09-17T15:45:15.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd483fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:15.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.557+0000 7fd483fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:15.555 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":162,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:15.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 msgr2=0x7fd48803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:15.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd49c069940 tx=0x7fd48c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 10 monclient: shutdown 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 msgr2=0x7fd49c135d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd49000c460 tx=0x7fd49000c920 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 shutdown_connections 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd48803bb10 0x7fd48803dfd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd49c13aff0 0x7fd49c09d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd49c0694c0 0x7fd49c136260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 --2- 172.21.15.86:0/1826908899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd49c0689d0 0x7fd49c135d20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 >> 172.21.15.86:0/1826908899 conn(0x7fd49c0a6440 msgr2=0x7fd49c0a23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 shutdown_connections 2024-09-17T15:45:15.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:15.561+0000 7fd4a18a3640 1 -- 172.21.15.86:0/1826908899 wait complete. 2024-09-17T15:45:18.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: get_monmap_and_config 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: build_initial_monmap 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: monmap: 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:18.712808+0000 2024-09-17T15:45:18.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:18.712808+0000 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:18.707 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 Processor -- start 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 -- start start 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: init 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde540a4150 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:18.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-a con 0x7fde54074340 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-b con 0x7fde54139b40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde54071c40 con 0x7fde540a4150 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fde54071300 con 0x7fde54074340 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fde54072570 con 0x7fde54139b40 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde58d01640 10 monclient(hunting): authenticate will time out at 2051.679018s 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde51d9b640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 msgr2=0x7fde54074750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde51d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:18.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57178/0 (socket says 172.21.15.86:57178) 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 learned_addr learned my addr 172.21.15.86:0/1121378854 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 10 monclient(hunting): get_auth_request con 0x7fde54139b40 auth_method 0 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:18.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.709+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:18.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.713+0000 7fde5259c640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.203:6789/0 conn(0x7fde540a4150 legacy=0x7fde54134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:18.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 msgr2=0x7fde54074750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:18.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 10 monclient(hunting): get_auth_request con 0x7fde54139b40 auth_method 0 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:18.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:18.913+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 10 monclient(hunting): get_auth_request con 0x7fde54139b40 auth_method 0 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:19.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:45:19.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:19.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 msgr2=0x7fde54074750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:19.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:19.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:45:19.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:19.317+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:45:20.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 msgr2=0x7fde54074750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 10 monclient(hunting): get_auth_request con 0x7fde54139b40 auth_method 0 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:20.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:20.121+0000 7fde52d9d640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): tick 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 msgr2=0x7fde5413bf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54139b40 0x7fde5413bf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:21.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 msgr2=0x7fde54074750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.86:3300/0 conn(0x7fde54074340 0x7fde54074750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.203:6789/0 conn(0x7fde540a4150 legacy=0x7fde54134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient: _add_conns ranks=[0,4] 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde34003770 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): picked mon.noname-c con 0x7fde340074a0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 -- 172.21.15.86:0/1121378854 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde5406ff40 con 0x7fde34003770 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 1 -- 172.21.15.86:0/1121378854 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fde3400b000 con 0x7fde340074a0 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde43fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde5259c640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.86:6789/0 conn(0x7fde34003770 legacy=0x7fde34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:21.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde51d9b640 10 monclient(hunting): get_auth_request con 0x7fde340074a0 auth_method 0 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.709+0000 7fde51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.713+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.713+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.713+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:21.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.713+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:21.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.913+0000 7fde5259c640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.86:6789/0 conn(0x7fde34003770 legacy=0x7fde34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 10 monclient(hunting): get_auth_request con 0x7fde340074a0 auth_method 0 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:21.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:21.917+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:45:22.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.317+0000 7fde5259c640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.86:6789/0 conn(0x7fde34003770 legacy=0x7fde34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 10 monclient(hunting): get_auth_request con 0x7fde340074a0 auth_method 0 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 msgr2=0x7fde34009960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:22.321+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:45:22.656 INFO:tasks.rados.rados.0.smithi203.stdout:2874: done (1 left) 2024-09-17T15:45:22.656 INFO:tasks.rados.rados.0.smithi203.stdout:2875: done (0 left) 2024-09-17T15:45:22.656 INFO:tasks.rados.rados.0.smithi203.stdout:2876: copy_from oid 5 from oid 39 current snap is 316 2024-09-17T15:45:22.656 INFO:tasks.rados.rados.0.smithi203.stdout:2877: write (excl) oid 13 current snap is 316 2024-09-17T15:45:22.656 INFO:tasks.rados.rados.0.smithi203.stdout:2877: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-09-17T15:45:22.677 INFO:tasks.rados.rados.0.smithi203.stdout:2877: writing smithi20328086-13 from 524649 to 1258988 tid 1 2024-09-17T15:45:22.699 INFO:tasks.rados.rados.0.smithi203.stdout:2877: writing smithi20328086-13 from 1910779 to 2613118 tid 2 2024-09-17T15:45:22.699 INFO:tasks.rados.rados.0.smithi203.stdout:2878: copy_from oid 10 from oid 42 current snap is 316 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2876: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2876: finishing copy_from to smithi20328086-5 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1248 (ObjNum 977 snap 299 seq_num 977) dirty exists 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2877: finishing write tid 1 to smithi20328086-13 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2876: done (2 left) 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2879: read oid 47 snap -1 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2879: expect (ObjNum 1001 snap 311 seq_num 1001) 2024-09-17T15:45:22.700 INFO:tasks.rados.rados.0.smithi203.stdout:2880: snap_remove snap 306 2024-09-17T15:45:23.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.121+0000 7fde5259c640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.86:6789/0 conn(0x7fde34003770 legacy=0x7fde34005bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> v2:172.21.15.203:3300/0 conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): get_auth_request con 0x7fde340074a0 auth_method 0 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient(hunting): handle_auth_done global_id 204151 payload 274 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> v1:172.21.15.86:6789/0 conn(0x7fde34003770 legacy=0x7fde34005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 1 monclient: found mon.noname-c 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:45:23.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.125+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde3400b1b0 con 0x7fde340074a0 2024-09-17T15:45:23.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde51d9b640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 msgr2=0x7fde34009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde51d9b640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde34009960 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde48003990 tx=0x7fde4801d1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eaef6d886e5dddc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 1 -- 172.21.15.86:0/1121378854 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480b3070 con 0x7fde340074a0 2024-09-17T15:45:23.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:23.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: dump: 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 5 monclient: authenticate success, global_id 204151 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 1 -- 172.21.15.86:0/1121378854 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde480ab560 con 0x7fde340074a0 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 4 set_mon_vals no callback set 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 1 -- 172.21.15.86:0/1121378854 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480ab950 con 0x7fde340074a0 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde5159a640 10 monclient: dump: 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.133 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 10 monclient: shutdown 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 1 -- 172.21.15.86:0/1121378854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 msgr2=0x7fde34009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.133+0000 7fde58d01640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde34009960 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde48003990 tx=0x7fde4801d1c0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde52d9d640 1 -- 172.21.15.86:0/1121378854 reap_dead start 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- 172.21.15.86:0/1121378854 shutdown_connections 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 --2- 172.21.15.86:0/1121378854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde34009960 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- 172.21.15.86:0/1121378854 >> 172.21.15.86:0/1121378854 conn(0x7fde540a3630 msgr2=0x7fde540a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- 172.21.15.86:0/1121378854 shutdown_connections 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- 172.21.15.86:0/1121378854 wait complete. 2024-09-17T15:45:23.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient: build_initial_monmap 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient: monmap: 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:23.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 Processor -- start 2024-09-17T15:45:23.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- start start 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient: init 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde541312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-b con 0x7fde340074a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-c con 0x7fde54131800 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 0x7fde541361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): picked mon.noname-a con 0x7fde54135da0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fde54070d10 con 0x7fde54135da0 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde54070b90 con 0x7fde340074a0 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fde54070e90 con 0x7fde54131800 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde58d01640 10 monclient(hunting): authenticate will time out at 2056.107818s 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde52d9d640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 msgr2=0x7fde541361e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.137+0000 7fde52d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 0x7fde541361e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57226/0 (socket says 172.21.15.86:57226) 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 -- 172.21.15.86:0/3700788436 learned_addr learned my addr 172.21.15.86:0/3700788436 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): get_auth_request con 0x7fde54131800 auth_method 0 2024-09-17T15:45:23.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde541312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): get_auth_request con 0x7fde340074a0 auth_method 0 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde5259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient(hunting): handle_auth_done global_id 164582 payload 824 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 -- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 msgr2=0x7fde541312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde541312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 -- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 msgr2=0x7fde541361e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 0x7fde541361e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 monclient: found mon.noname-c 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 -- 172.21.15.86:0/3700788436 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde540a5140 con 0x7fde54131800 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 -- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 msgr2=0x7fde54135860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde51d9b640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde48019dd0 tx=0x7fde48019f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480b3040 con 0x7fde54131800 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: dump: 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde48009430 con 0x7fde54131800 2024-09-17T15:45:23.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 5 monclient: authenticate success, global_id 164582 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 10 monclient: _renew_subs 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde541390e0 con 0x7fde54131800 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde480ae830 con 0x7fde54131800 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde50d99640 4 set_mon_vals no callback set 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde42ffd640 10 monclient: dump: 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 10 monclient: _renew_subs 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.141+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5413d170 con 0x7fde54131800 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde58d01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde58d01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde58d01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde540a67c0 con 0x7fde54131800 2024-09-17T15:45:23.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fde480b6070 con 0x7fde54131800 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde42ffd640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 0x7fde3403eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(644..644 src has 438..644) ==== 4739+0+0 (secure 0 0 0) 0x7fde480ea880 con 0x7fde54131800 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde5259c640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 0x7fde3403eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.145+0000 7fde5259c640 10 monclient: get_auth_request con 0x7fde3403ca20 auth_method 0 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.149+0000 7fde5259c640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 0x7fde3403eee0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fde44000f70 tx=0x7fde44009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.149+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fde48017140 con 0x7fde54131800 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.149+0000 7fde42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.149+0000 7fde42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde58d01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde58d01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde58d01640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde54074fe0 con 0x7fde54131800 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde42ffd640 1 -- 172.21.15.86:0/3700788436 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fde480eccd0 con 0x7fde54131800 2024-09-17T15:45:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.377+0000 7fde42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:23.377 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 msgr2=0x7fde3403eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 0x7fde3403eee0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fde44000f70 tx=0x7fde44009040 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 10 monclient: shutdown 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 msgr2=0x7fde54135860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fde48019dd0 tx=0x7fde48019f60 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 shutdown_connections 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde3403ca20 0x7fde3403eee0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde54135da0 0x7fde541361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde54131800 0x7fde54135860 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 --2- 172.21.15.86:0/3700788436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde340074a0 0x7fde541312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 >> 172.21.15.86:0/3700788436 conn(0x7fde540a3630 msgr2=0x7fde540a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 shutdown_connections 2024-09-17T15:45:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.381+0000 7fde58d01640 1 -- 172.21.15.86:0/3700788436 wait complete. 2024-09-17T15:45:23.395 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:45:23.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:45:23.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:45:23.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: get_monmap_and_config 2024-09-17T15:45:23.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: build_initial_monmap 2024-09-17T15:45:23.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: monmap: 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:23.520387+0000 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:23.520387+0000 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:23.514 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:23.515 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 Processor -- start 2024-09-17T15:45:23.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 -- start start 2024-09-17T15:45:23.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: init 2024-09-17T15:45:23.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:23.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcf940a0c70 0x7fcf940a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-c con 0x7fcf940a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-b con 0x7fcf940a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 0x7fcf94136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-a con 0x7fcf94135f90 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcf940a2410 con 0x7fcf94135f90 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcf940a2130 con 0x7fcf940a0c70 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf940a26f0 con 0x7fcf940a1590 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf98faa640 10 monclient(hunting): authenticate will time out at 2056.485939s 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf92d9d640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 msgr2=0x7fcf94136420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf92d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 0x7fcf94136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57230/0 (socket says 172.21.15.86:57230) 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 -- 172.21.15.86:0/2099842799 learned_addr learned my addr 172.21.15.86:0/2099842799 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): get_auth_request con 0x7fcf940a1590 auth_method 0 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf9259c640 1 --2- 172.21.15.86:0/2099842799 >> v2:172.21.15.203:3300/0 conn(0x7fcf940a0c70 0x7fcf940a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient(hunting): handle_auth_done global_id 164585 payload 274 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 -- 172.21.15.86:0/2099842799 >> v2:172.21.15.203:3300/0 conn(0x7fcf940a0c70 msgr2=0x7fcf940a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 --2- 172.21.15.86:0/2099842799 >> v2:172.21.15.203:3300/0 conn(0x7fcf940a0c70 0x7fcf940a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 -- 172.21.15.86:0/2099842799 >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 msgr2=0x7fcf94136420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 --2- 172.21.15.86:0/2099842799 >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 0x7fcf94136420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 monclient: found mon.noname-b 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.517+0000 7fcf91d9b640 1 -- 172.21.15.86:0/2099842799 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9409f9f0 con 0x7fcf940a1590 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf91d9b640 1 -- 172.21.15.86:0/2099842799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 msgr2=0x7fcf94135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf91d9b640 1 --2- 172.21.15.86:0/2099842799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcf9406d160 tx=0x7fcf8401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ada0c8f977717e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 1 -- 172.21.15.86:0/2099842799 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840a5770 con 0x7fcf940a1590 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: dump: 2024-09-17T15:45:23.524 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 1 -- 172.21.15.86:0/2099842799 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf840a5910 con 0x7fcf940a1590 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 1 -- 172.21.15.86:0/2099842799 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840a5ce0 con 0x7fcf940a1590 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 5 monclient: authenticate success, global_id 164585 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 4 set_mon_vals no callback set 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf9159a640 10 monclient: dump: 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.526 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 10 monclient: shutdown 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 -- 172.21.15.86:0/2099842799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 msgr2=0x7fcf94135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 --2- 172.21.15.86:0/2099842799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcf9406d160 tx=0x7fcf8401c8d0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 -- 172.21.15.86:0/2099842799 shutdown_connections 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 --2- 172.21.15.86:0/2099842799 >> v2:172.21.15.86:3300/0 conn(0x7fcf94135f90 0x7fcf94136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 --2- 172.21.15.86:0/2099842799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf940a1590 0x7fcf94135a50 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 --2- 172.21.15.86:0/2099842799 >> v2:172.21.15.203:3300/0 conn(0x7fcf940a0c70 0x7fcf940a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 -- 172.21.15.86:0/2099842799 >> 172.21.15.86:0/2099842799 conn(0x7fcf940a2f40 msgr2=0x7fcf940a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 -- 172.21.15.86:0/2099842799 shutdown_connections 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.521+0000 7fcf98faa640 1 -- 172.21.15.86:0/2099842799 wait complete. 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient: build_initial_monmap 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient: monmap: 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:23.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 Processor -- start 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 -- start start 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient: init 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf940a0c70 0x7fcf941cc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-b con 0x7fcf940a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf940a1590 0x7fcf94133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-a con 0x7fcf940a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 0x7fcf941cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): picked mon.noname-c con 0x7fcf94135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf940a00d0 con 0x7fcf940a1590 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf9409ff50 con 0x7fcf940a0c70 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf9413f740 con 0x7fcf94135f90 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf98faa640 10 monclient(hunting): authenticate will time out at 2056.493485s 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf91d9b640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf940a1590 msgr2=0x7fcf94133030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf91d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf940a1590 0x7fcf94133030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 0x7fcf941cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 0x7fcf941cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57242/0 (socket says 172.21.15.86:57242) 2024-09-17T15:45:23.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 learned_addr learned my addr 172.21.15.86:0/3110787867 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): get_auth_request con 0x7fcf94135f90 auth_method 0 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf9259c640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf940a0c70 0x7fcf941cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient(hunting): handle_auth_done global_id 164588 payload 824 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf940a0c70 msgr2=0x7fcf941cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf940a0c70 0x7fcf941cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf940a1590 msgr2=0x7fcf94133030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:23.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf940a1590 0x7fcf94133030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 monclient: found mon.noname-c 2024-09-17T15:45:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.525+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf9409d460 con 0x7fcf94135f90 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 msgr2=0x7fcf941cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf92d9d640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 0x7fcf941cfc50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00b440 tx=0x7fcf8c00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c017070 con 0x7fcf94135f90 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:23.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: dump: 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf8c0060d0 con 0x7fcf94135f90 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c0064a0 con 0x7fcf94135f90 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 5 monclient: authenticate success, global_id 164588 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 monclient: _renew_subs 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf9409f370 con 0x7fcf94135f90 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf90d99640 4 set_mon_vals no callback set 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: dump: 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 monclient: _renew_subs 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf940735b0 con 0x7fcf94135f90 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 0x7fcf941d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf9413bcd0 con 0x7fcf94133570 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 0x7fcf941d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient: get_auth_request con 0x7fcf94133570 auth_method 0 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcf8c01a070 con 0x7fcf94135f90 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 10 monclient(hunting): handle_auth_done global_id 164591 payload 824 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf9409ef20 con 0x7fcf94133570 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf7403bb60 0x7fcf7403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(644..644 src has 438..644) ==== 4739+0+0 (secure 0 0 0) 0x7fcf8c04f2b0 con 0x7fcf94135f90 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf91d9b640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf7403bb60 0x7fcf7403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf91d9b640 10 monclient: get_auth_request con 0x7fcf7403bb60 auth_method 0 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 msgr2=0x7fcf941d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf9259c640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 0x7fcf941d0190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00d4b0 tx=0x7fcf7c00d970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c016070 con 0x7fcf94133570 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.529+0000 7fcf8affd640 10 monclient: dump: 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf7c012b60 con 0x7fcf94133570 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 msgr2=0x7fcf941d0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 0x7fcf941d0190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00d4b0 tx=0x7fcf7c00d970 comp rx=0 tx=0).stop 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 msgr2=0x7fcf941d0190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf8affd640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94133570 0x7fcf941d0190 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00d4b0 tx=0x7fcf7c00d970 comp rx=0 tx=0).stop 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.533+0000 7fcf91d9b640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf7403bb60 0x7fcf7403e020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf840ab000 tx=0x7fcf840a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf98faa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf98faa640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 0x7fcf941d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf98faa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf941cd5d0 con 0x7fcf9409de50 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 0x7fcf941d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient: get_auth_request con 0x7fcf9409de50 auth_method 0 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 10 monclient(hunting): handle_auth_done global_id 164594 payload 824 2024-09-17T15:45:23.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.537+0000 7fcf9259c640 1 -- 172.21.15.86:0/3110787867 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf9406d4f0 con 0x7fcf9409de50 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf9259c640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 msgr2=0x7fcf941d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf9259c640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 0x7fcf941d06d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c018000 tx=0x7fcf7c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c016040 con 0x7fcf9409de50 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 10 monclient: dump: 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.545 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fcf7c0113c0 con 0x7fcf9409de50 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 msgr2=0x7fcf941d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 0x7fcf941d06d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c018000 tx=0x7fcf7c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 msgr2=0x7fcf941d06d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf8affd640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9409de50 0x7fcf941d06d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 166, 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 226729, 2024-09-17T15:45:23.547 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.548 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.549 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:23.550 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:23.551 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.552 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.553 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:23.554 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.555 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:23.557 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:23.558 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.559 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:23.560 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:23.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf7403bb60 msgr2=0x7fcf7403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf7403bb60 0x7fcf7403e020 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fcf840ab000 tx=0x7fcf840a7070 comp rx=0 tx=0).stop 2024-09-17T15:45:23.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 10 monclient: shutdown 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 msgr2=0x7fcf941cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 --2- 172.21.15.86:0/3110787867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf94135f90 0x7fcf941cfc50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00b440 tx=0x7fcf8c00b900 comp rx=0 tx=0).stop 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 reap_dead start 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf92d9d640 1 -- 172.21.15.86:0/3110787867 reap_dead start 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 shutdown_connections 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 >> 172.21.15.86:0/3110787867 conn(0x7fcf940a2f40 msgr2=0x7fcf940a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 shutdown_connections 2024-09-17T15:45:23.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.541+0000 7fcf98faa640 1 -- 172.21.15.86:0/3110787867 wait complete. 2024-09-17T15:45:23.562 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:45:23.563 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:45:23.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:45:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:2880: done (3 left) 2024-09-17T15:45:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:2881: read oid 29 snap -1 2024-09-17T15:45:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:2881: expect (ObjNum 966 snap 293 seq_num 966) 2024-09-17T15:45:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:2877: finishing write tid 2 to smithi20328086-13 2024-09-17T15:45:23.670 INFO:tasks.rados.rados.0.smithi203.stdout:2879: expect (ObjNum 1001 snap 311 seq_num 1001) 2024-09-17T15:45:23.671 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: get_monmap_and_config 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: build_initial_monmap 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: monmap: 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:23.692889+0000 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:23.692889+0000 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:23.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:23.687 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:23.687 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:23.687 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 Processor -- start 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 -- start start 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: init 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:45:23.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 0x7f05a80a1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-a con 0x7f05a8134a70 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05a8134e50 0x7f05a8135260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-c con 0x7f05a8134e50 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05a809a0d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a80a71d0 con 0x7f05a809a0d0 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f05a80a0360 con 0x7f05a8134a70 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f05a80a0080 con 0x7f05a8134e50 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05adf4a640 10 monclient(hunting): authenticate will time out at 2056.659089s 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a77fe640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 msgr2=0x7f05a80a1300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 0x7f05a80a1300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:23.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05a8134e50 0x7f05a8135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f05a8134e50 0x7f05a8135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:52568/0 (socket says 172.21.15.86:52568) 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f05a809a0d0 0x7f05a809a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:38794/0 (socket says 172.21.15.86:38794) 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 1 -- 172.21.15.86:0/1756273739 learned_addr learned my addr 172.21.15.86:0/1756273739 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): get_auth_request con 0x7f05a8134e50 auth_method 0 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.689+0000 7f05a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 10 monclient(hunting): handle_auth_done global_id 204157 payload 274 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 1 -- 172.21.15.86:0/1756273739 >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 msgr2=0x7f05a80a1300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 1 --2- 172.21.15.86:0/1756273739 >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 0x7f05a80a1300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 1 -- 172.21.15.86:0/1756273739 >> v1:172.21.15.203:6789/0 conn(0x7f05a809a0d0 legacy=0x7f05a809a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:23.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 1 monclient: found mon.noname-c 2024-09-17T15:45:23.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:45:23.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.693+0000 7f05a6ffd640 1 -- 172.21.15.86:0/1756273739 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a80a0640 con 0x7f05a8134e50 2024-09-17T15:45:23.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a6ffd640 1 -- 172.21.15.86:0/1756273739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a8134e50 msgr2=0x7f05a8135260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a6ffd640 1 --2- 172.21.15.86:0/1756273739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a8134e50 0x7f05a8135260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f05a8070500 tx=0x7f059c01c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3738cb41066dbfbb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 1 -- 172.21.15.86:0/1756273739 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0b2070 con 0x7f05a8134e50 2024-09-17T15:45:23.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: dump: 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 1 -- 172.21.15.86:0/1756273739 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059c0a9070 con 0x7f05a8134e50 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 1 -- 172.21.15.86:0/1756273739 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059c0ad880 con 0x7f05a8134e50 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 5 monclient: authenticate success, global_id 204157 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 4 set_mon_vals no callback set 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05a67fc640 10 monclient: dump: 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.696 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.697 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.697 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.697 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.697 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 10 monclient: shutdown 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- 172.21.15.86:0/1756273739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a8134e50 msgr2=0x7f05a8135260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 --2- 172.21.15.86:0/1756273739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a8134e50 0x7f05a8135260 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f05a8070500 tx=0x7f059c01c380 comp rx=0 tx=0).stop 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- 172.21.15.86:0/1756273739 shutdown_connections 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 --2- 172.21.15.86:0/1756273739 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a8134e50 0x7f05a8135260 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 --2- 172.21.15.86:0/1756273739 >> v2:172.21.15.86:3300/0 conn(0x7f05a8134a70 0x7f05a80a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- 172.21.15.86:0/1756273739 >> 172.21.15.86:0/1756273739 conn(0x7f05a80a3ef0 msgr2=0x7f05a80a42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- 172.21.15.86:0/1756273739 shutdown_connections 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- 172.21.15.86:0/1756273739 wait complete. 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 10 monclient: build_initial_monmap 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 10 monclient: monmap: 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:23.703 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 Processor -- start 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 1 -- start start 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.697+0000 7f05adf4a640 10 monclient: init 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a809a0d0 0x7f05a8130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-b con 0x7f05a809a0d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8134a70 0x7f05a813c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-a con 0x7f05a8134a70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 0x7f05a813c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): picked mon.noname-c con 0x7f05a8134e50 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f05a813f960 con 0x7f05a8134a70 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05a813f7e0 con 0x7f05a809a0d0 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f05a813fae0 con 0x7f05a8134e50 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): authenticate will time out at 2056.668121s 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a6ffd640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8134a70 msgr2=0x7f05a813c3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8134a70 0x7f05a813c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 0x7f05a813c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 0x7f05a813c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57280/0 (socket says 172.21.15.86:57280) 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 learned_addr learned my addr 172.21.15.86:0/3573767923 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:23.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): get_auth_request con 0x7f05a8134e50 auth_method 0 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a809a0d0 0x7f05a8130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient(hunting): handle_auth_done global_id 164597 payload 824 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a809a0d0 msgr2=0x7f05a8130ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a809a0d0 0x7f05a8130ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8134a70 msgr2=0x7f05a813c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05a8134a70 0x7f05a813c3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:45:23.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 monclient: found mon.noname-c 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a809b980 con 0x7f05a8134e50 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a77fe640 10 monclient: get_auth_request con 0x7f05a809a0d0 auth_method 0 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a809a0d0 0x7f05a8130ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 msgr2=0x7f05a813c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a7fff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 0x7f05a813c8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c7b0 tx=0x7f05a000cbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0017070 con 0x7f05a8134e50 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.707 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: _finish_auth 0 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 5 monclient: authenticate success, global_id 164597 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _renew_subs 2024-09-17T15:45:23.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a000a860 con 0x7f05a8134e50 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a81365a0 con 0x7f05a8134e50 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a000ac50 con 0x7f05a8134e50 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.709 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05a5ffb640 4 set_mon_vals no callback set 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _renew_subs 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a809cbc0 con 0x7f05a8134e50 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 0x7f05a81d0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8075070 con 0x7f05a81cff80 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.701+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f05a001a070 con 0x7f05a8134e50 2024-09-17T15:45:23.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb10 0x7f059403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 0x7f05a81d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient: get_auth_request con 0x7f05a81cff80 auth_method 0 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(645..645 src has 438..645) ==== 4783+0+0 (secure 0 0 0) 0x7f05a004f670 con 0x7f05a8134e50 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a6ffd640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb10 0x7f059403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a6ffd640 10 monclient: get_auth_request con 0x7f059403bb10 auth_method 0 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 10 monclient(hunting): handle_auth_done global_id 204160 payload 824 2024-09-17T15:45:23.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05a80a1300 con 0x7f05a81cff80 2024-09-17T15:45:23.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 msgr2=0x7f05a81d0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 0x7f05a81d0360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f059800cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f05a6ffd640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb10 0x7f059403dfd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f059c0ac000 tx=0x7f059c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598003dd0 con 0x7f05a81cff80 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.705+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0598003dd0 con 0x7f05a81cff80 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 msgr2=0x7f05a81d0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 0x7f05a81d0360 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f059800cbb0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 msgr2=0x7f05a81d0360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:23.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.709+0000 7f058ffff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81cff80 0x7f05a81d0360 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f059800cbb0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05adf4a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05adf4a640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 0x7f05a8071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05adf4a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f05a806e620 con 0x7f05a81d08a0 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 0x7f05a8071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient: get_auth_request con 0x7f05a81d08a0 auth_method 0 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:23.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 10 monclient(hunting): handle_auth_done global_id 204163 payload 824 2024-09-17T15:45:23.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.713+0000 7f05a77fe640 1 -- 172.21.15.86:0/3573767923 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05a809e4d0 con 0x7f05a81d08a0 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05a77fe640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 msgr2=0x7f05a8071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05a77fe640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 0x7f05a8071030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f05980059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059801c110 con 0x7f05a81d08a0 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:23.718 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f059801c110 con 0x7f05a81d08a0 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:45:23.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 msgr2=0x7f05a8071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 0x7f05a8071030 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f05980059e0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 msgr2=0x7f05a8071030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f058ffff640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05a81d08a0 0x7f05a8071030 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f059800c6f0 tx=0x7f05980059e0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:23.720 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 166, 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 26319, 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:45:23.721 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.722 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.723 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:45:23.724 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:45:23.725 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:45:23.726 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:45:23.727 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:45:23.728 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.729 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:45:23.730 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:45:23.731 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.732 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.734 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:45:23.735 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:45:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:45:23.736 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:45:23.736 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:45:23.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb10 msgr2=0x7f059403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f059403bb10 0x7f059403dfd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f059c0ac000 tx=0x7f059c008040 comp rx=0 tx=0).stop 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 10 monclient: shutdown 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 msgr2=0x7f05a813c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 --2- 172.21.15.86:0/3573767923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05a8134e50 0x7f05a813c8f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c7b0 tx=0x7f05a000cbb0 comp rx=0 tx=0).stop 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 reap_dead start 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05a7fff640 1 -- 172.21.15.86:0/3573767923 reap_dead start 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 shutdown_connections 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 >> 172.21.15.86:0/3573767923 conn(0x7f05a80a3ef0 msgr2=0x7f05a80689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:23.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 shutdown_connections 2024-09-17T15:45:23.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:23.717+0000 7f05adf4a640 1 -- 172.21.15.86:0/3573767923 wait complete. 2024-09-17T15:45:23.739 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:45:23.739 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:45:23.878 INFO:tasks.rados.rados.0.smithi203.stdout:2877: finishing write tid 3 to smithi20328086-13 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: finishing write tid 4 to smithi20328086-13 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: oid 13 updating version 0 to 1194 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: oid 13 updating version 1194 to 1195 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: oid 13 version 1195 is already newer than 1193 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1195 (ObjNum 1007 snap 316 seq_num 1007) dirty exists 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: left oid 13 (ObjNum 1007 snap 316 seq_num 1007) 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2878: finishing copy_from racing read to smithi20328086-10 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2878: finishing copy_from to smithi20328086-10 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1367 (ObjNum 987 snap 309 seq_num 987) dirty exists 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2877: done (3 left) 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2878: done (2 left) 2024-09-17T15:45:23.879 INFO:tasks.rados.rados.0.smithi203.stdout:2879: done (1 left) 2024-09-17T15:45:23.880 INFO:tasks.rados.rados.0.smithi203.stdout:2882: write (excl) oid 49 current snap is 316 2024-09-17T15:45:23.880 INFO:tasks.rados.rados.0.smithi203.stdout:2882: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-17T15:45:23.886 INFO:tasks.rados.rados.0.smithi203.stdout:2882: writing smithi20328086-49 from 541456 to 951044 tid 1 2024-09-17T15:45:23.893 INFO:tasks.rados.rados.0.smithi203.stdout:2882: writing smithi20328086-49 from 1569261 to 2131611 tid 2 2024-09-17T15:45:23.898 INFO:tasks.rados.rados.0.smithi203.stdout:2882: writing smithi20328086-49 from 2814612 to 2941456 tid 3 2024-09-17T15:45:23.898 INFO:tasks.rados.rados.0.smithi203.stdout:2883: rollback oid 12 current snap is 316 2024-09-17T15:45:23.898 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 12 to 304 2024-09-17T15:45:23.898 INFO:tasks.rados.rados.0.smithi203.stdout:2881: expect (ObjNum 966 snap 293 seq_num 966) 2024-09-17T15:45:23.898 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 275802 2024-09-17T15:45:23.961 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:24.077 INFO:tasks.rados.rados.0.smithi203.stdout:2882: finishing write tid 1 to smithi20328086-49 2024-09-17T15:45:24.077 INFO:tasks.rados.rados.0.smithi203.stdout:2881: done (2 left) 2024-09-17T15:45:24.077 INFO:tasks.rados.rados.0.smithi203.stdout:2884: read oid 11 snap -1 2024-09-17T15:45:24.077 INFO:tasks.rados.rados.0.smithi203.stdout:2884: expect deleted 2024-09-17T15:45:24.078 INFO:tasks.rados.rados.0.smithi203.stdout:2885: write (excl) oid 18 current snap is 316 2024-09-17T15:45:24.078 INFO:tasks.rados.rados.0.smithi203.stdout:2883: finishing rollback tid 0 to smithi20328086-12 2024-09-17T15:45:24.078 INFO:tasks.rados.rados.0.smithi203.stdout:2885: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-17T15:45:24.082 INFO:tasks.rados.rados.0.smithi203.stdout:2885: writing smithi20328086-18 from 558263 to 1043100 tid 1 2024-09-17T15:45:24.089 INFO:tasks.rados.rados.0.smithi203.stdout:2885: writing smithi20328086-18 from 1627743 to 2333751 tid 2 2024-09-17T15:45:24.090 INFO:tasks.rados.rados.0.smithi203.stdout:2886: snap_remove snap 315 2024-09-17T15:45:24.678 INFO:tasks.rados.rados.0.smithi203.stdout:2886: done (4 left) 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2887: rollback oid 6 current snap is 316 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 300 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: finishing write tid 2 to smithi20328086-49 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2883: finishing rollback tid 1 to smithi20328086-12 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1288 (ObjNum 49 snap 0 seq_num 1) dirty dne 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: finishing write tid 3 to smithi20328086-49 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: finishing write tid 4 to smithi20328086-49 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: finishing write tid 5 to smithi20328086-49 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: oid 49 updating version 0 to 1511 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: oid 49 updating version 1511 to 1512 2024-09-17T15:45:24.679 INFO:tasks.rados.rados.0.smithi203.stdout:2882: oid 49 version 1512 is already newer than 1509 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2882: oid 49 version 1512 is already newer than 1510 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1512 (ObjNum 1008 snap 316 seq_num 1008) dirty exists 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2882: left oid 49 (ObjNum 1008 snap 316 seq_num 1008) 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: finishing write tid 1 to smithi20328086-18 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: finishing write tid 2 to smithi20328086-18 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: finishing write tid 3 to smithi20328086-18 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2882: done (4 left) 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2883: done (3 left) 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2884: done (2 left) 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2888: rollback oid 37 current snap is 316 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 301 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: finishing write tid 4 to smithi20328086-18 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: oid 18 updating version 0 to 894 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: oid 18 version 894 is already newer than 893 2024-09-17T15:45:24.680 INFO:tasks.rados.rados.0.smithi203.stdout:2885: oid 18 version 894 is already newer than 892 2024-09-17T15:45:24.681 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 894 (ObjNum 1009 snap 316 seq_num 1009) dirty exists 2024-09-17T15:45:24.681 INFO:tasks.rados.rados.0.smithi203.stdout:2885: left oid 18 (ObjNum 1009 snap 316 seq_num 1009) 2024-09-17T15:45:24.681 INFO:tasks.rados.rados.0.smithi203.stdout:2885: done (2 left) 2024-09-17T15:45:24.681 INFO:tasks.rados.rados.0.smithi203.stdout:2889: write (excl) oid 48 current snap is 316 2024-09-17T15:45:24.681 INFO:tasks.rados.rados.0.smithi203.stdout:2889: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2024-09-17T15:45:24.685 INFO:tasks.rados.rados.0.smithi203.stdout:2889: writing smithi20328086-48 from 575070 to 1135156 tid 1 2024-09-17T15:45:24.706 INFO:tasks.rados.rados.0.smithi203.stdout:2889: writing smithi20328086-48 from 1686225 to 2252244 tid 2 2024-09-17T15:45:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:2889: writing smithi20328086-48 from 2869458 to 2975070 tid 3 2024-09-17T15:45:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:2890: delete oid 13 current snap is 316 2024-09-17T15:45:24.718 INFO:tasks.rados.rados.0.smithi203.stdout:2887: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:2888: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:2888: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:2888: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1504 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:2889: finishing write tid 1 to smithi20328086-48 2024-09-17T15:45:24.719 INFO:tasks.rados.rados.0.smithi203.stdout:2889: finishing write tid 2 to smithi20328086-48 2024-09-17T15:45:24.722 INFO:tasks.rados.rados.0.smithi203.stdout:2889: finishing write tid 3 to smithi20328086-48 2024-09-17T15:45:24.723 INFO:tasks.rados.rados.0.smithi203.stdout:2889: finishing write tid 4 to smithi20328086-48 2024-09-17T15:45:24.723 INFO:tasks.rados.rados.0.smithi203.stdout:2889: finishing write tid 5 to smithi20328086-48 2024-09-17T15:45:24.723 INFO:tasks.rados.rados.0.smithi203.stdout:2889: oid 48 updating version 0 to 1507 2024-09-17T15:45:24.723 INFO:tasks.rados.rados.0.smithi203.stdout:2889: oid 48 version 1507 is already newer than 1506 2024-09-17T15:45:24.723 INFO:tasks.rados.rados.0.smithi203.stdout:2889: oid 48 version 1507 is already newer than 1505 2024-09-17T15:45:24.724 INFO:tasks.rados.rados.0.smithi203.stdout:2889: oid 48 updating version 1507 to 1508 2024-09-17T15:45:24.724 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1508 (ObjNum 1010 snap 316 seq_num 1010) dirty exists 2024-09-17T15:45:24.724 INFO:tasks.rados.rados.0.smithi203.stdout:2889: left oid 48 (ObjNum 1010 snap 316 seq_num 1010) 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2887: done (3 left) 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2888: done (2 left) 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2889: done (1 left) 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2890: done (0 left) 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2891: write oid 14 current snap is 316 2024-09-17T15:45:24.728 INFO:tasks.rados.rados.0.smithi203.stdout:2891: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-17T15:45:24.734 INFO:tasks.rados.rados.0.smithi203.stdout:2891: writing smithi20328086-14 from 608684 to 1035621 tid 1 2024-09-17T15:45:24.743 INFO:tasks.rados.rados.0.smithi203.stdout:2891: writing smithi20328086-14 from 1803189 to 2372877 tid 2 2024-09-17T15:45:24.747 INFO:tasks.rados.rados.0.smithi203.stdout:2891: writing smithi20328086-14 from 2924304 to 3008684 tid 3 2024-09-17T15:45:24.747 INFO:tasks.rados.rados.0.smithi203.stdout:2892: snap_remove snap 301 2024-09-17T15:45:25.689 INFO:tasks.rados.rados.0.smithi203.stdout:2892: done (1 left) 2024-09-17T15:45:25.689 INFO:tasks.rados.rados.0.smithi203.stdout:2893: copy_from oid 29 from oid 15 current snap is 316 2024-09-17T15:45:25.690 INFO:tasks.rados.rados.0.smithi203.stdout:2894: write (excl) oid 39 current snap is 316 2024-09-17T15:45:25.690 INFO:tasks.rados.rados.0.smithi203.stdout:2894: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-17T15:45:25.702 INFO:tasks.rados.rados.0.smithi203.stdout:2894: writing smithi20328086-39 from 625491 to 1127677 tid 1 2024-09-17T15:45:25.717 INFO:tasks.rados.rados.0.smithi203.stdout:2894: writing smithi20328086-39 from 1861671 to 2575017 tid 2 2024-09-17T15:45:25.717 INFO:tasks.rados.rados.0.smithi203.stdout:2895: write (excl) oid 21 current snap is 316 2024-09-17T15:45:25.718 INFO:tasks.rados.rados.0.smithi203.stdout:2895: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-17T15:45:25.730 INFO:tasks.rados.rados.0.smithi203.stdout:2895: writing smithi20328086-21 from 642298 to 1219733 tid 1 2024-09-17T15:45:25.742 INFO:tasks.rados.rados.0.smithi203.stdout:2895: writing smithi20328086-21 from 1920153 to 2493510 tid 2 2024-09-17T15:45:25.748 INFO:tasks.rados.rados.0.smithi203.stdout:2895: writing smithi20328086-21 from 2979150 to 3042298 tid 3 2024-09-17T15:45:25.748 INFO:tasks.rados.rados.0.smithi203.stdout:2896: delete oid 16 current snap is 316 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: finishing write tid 1 to smithi20328086-14 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: finishing write tid 2 to smithi20328086-14 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: finishing write tid 3 to smithi20328086-14 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: finishing write tid 4 to smithi20328086-14 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: finishing write tid 5 to smithi20328086-14 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: oid 14 updating version 0 to 1312 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: oid 14 updating version 1312 to 1315 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: oid 14 version 1315 is already newer than 1313 2024-09-17T15:45:25.749 INFO:tasks.rados.rados.0.smithi203.stdout:2891: oid 14 version 1315 is already newer than 1314 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1315 (ObjNum 1012 snap 316 seq_num 1012) dirty exists 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2891: left oid 14 (ObjNum 1012 snap 316 seq_num 1012) 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2893: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: finishing write tid 1 to smithi20328086-39 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: finishing write tid 2 to smithi20328086-39 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2893: finishing copy_from to smithi20328086-29 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1283 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: finishing write tid 3 to smithi20328086-39 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: finishing write tid 4 to smithi20328086-39 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: oid 39 updating version 0 to 1510 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: oid 39 updating version 1510 to 1511 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:2894: oid 39 version 1511 is already newer than 1509 2024-09-17T15:45:25.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1511 (ObjNum 1013 snap 316 seq_num 1013) dirty exists 2024-09-17T15:45:25.751 INFO:tasks.rados.rados.0.smithi203.stdout:2894: left oid 39 (ObjNum 1013 snap 316 seq_num 1013) 2024-09-17T15:45:25.751 INFO:tasks.rados.rados.0.smithi203.stdout:2895: finishing write tid 1 to smithi20328086-21 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2895: finishing write tid 2 to smithi20328086-21 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2891: done (4 left) 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2893: done (3 left) 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2894: done (2 left) 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2896: done (1 left) 2024-09-17T15:45:25.752 INFO:tasks.rados.rados.0.smithi203.stdout:2897: delete oid 44 current snap is 316 2024-09-17T15:45:25.753 INFO:tasks.rados.rados.0.smithi203.stdout:2895: finishing write tid 3 to smithi20328086-21 2024-09-17T15:45:25.753 INFO:tasks.rados.rados.0.smithi203.stdout:2895: finishing write tid 4 to smithi20328086-21 2024-09-17T15:45:25.753 INFO:tasks.rados.rados.0.smithi203.stdout:2895: finishing write tid 5 to smithi20328086-21 2024-09-17T15:45:25.753 INFO:tasks.rados.rados.0.smithi203.stdout:2895: oid 21 updating version 0 to 1518 2024-09-17T15:45:25.753 INFO:tasks.rados.rados.0.smithi203.stdout:2895: oid 21 version 1518 is already newer than 1517 2024-09-17T15:45:25.754 INFO:tasks.rados.rados.0.smithi203.stdout:2895: oid 21 version 1518 is already newer than 1516 2024-09-17T15:45:25.754 INFO:tasks.rados.rados.0.smithi203.stdout:2895: oid 21 version 1518 is already newer than 1515 2024-09-17T15:45:25.754 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1518 (ObjNum 1014 snap 316 seq_num 1014) dirty exists 2024-09-17T15:45:25.754 INFO:tasks.rados.rados.0.smithi203.stdout:2895: left oid 21 (ObjNum 1014 snap 316 seq_num 1014) 2024-09-17T15:45:25.758 INFO:tasks.rados.rados.0.smithi203.stdout:2895: done (1 left) 2024-09-17T15:45:25.758 INFO:tasks.rados.rados.0.smithi203.stdout:2897: done (0 left) 2024-09-17T15:45:25.758 INFO:tasks.rados.rados.0.smithi203.stdout:2898: write (excl) oid 35 current snap is 316 2024-09-17T15:45:25.758 INFO:tasks.rados.rados.0.smithi203.stdout:2898: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-09-17T15:45:25.769 INFO:tasks.rados.rados.0.smithi203.stdout:2898: writing smithi20328086-35 from 692719 to 1095901 tid 1 2024-09-17T15:45:25.781 INFO:tasks.rados.rados.0.smithi203.stdout:2898: writing smithi20328086-35 from 1579246 to 2299930 tid 2 2024-09-17T15:45:25.789 INFO:tasks.rados.rados.0.smithi203.stdout:2898: writing smithi20328086-35 from 2828713 to 3092719 tid 3 2024-09-17T15:45:25.789 INFO:tasks.rados.rados.0.smithi203.stdout:2899: rollback oid 48 current snap is 316 2024-09-17T15:45:25.789 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 297 2024-09-17T15:45:25.790 INFO:tasks.rados.rados.0.smithi203.stdout:2898: finishing write tid 1 to smithi20328086-35 2024-09-17T15:45:25.790 INFO:tasks.rados.rados.0.smithi203.stdout:2898: finishing write tid 2 to smithi20328086-35 2024-09-17T15:45:25.790 INFO:tasks.rados.rados.0.smithi203.stdout:2900: snap_remove snap 295 2024-09-17T15:45:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2900: done (2 left) 2024-09-17T15:45:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2901: write oid 34 current snap is 316 2024-09-17T15:45:26.700 INFO:tasks.rados.rados.0.smithi203.stdout:2898: finishing write tid 3 to smithi20328086-35 2024-09-17T15:45:26.701 INFO:tasks.rados.rados.0.smithi203.stdout:2901: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-17T15:45:26.711 INFO:tasks.rados.rados.0.smithi203.stdout:2901: writing smithi20328086-34 from 709526 to 1187957 tid 1 2024-09-17T15:45:26.720 INFO:tasks.rados.rados.0.smithi203.stdout:2901: writing smithi20328086-34 from 1637728 to 2102070 tid 2 2024-09-17T15:45:26.730 INFO:tasks.rados.rados.0.smithi203.stdout:2901: writing smithi20328086-34 from 2856136 to 3109526 tid 3 2024-09-17T15:45:26.730 INFO:tasks.rados.rados.0.smithi203.stdout:2902: read oid 42 snap -1 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2902: expect (ObjNum 987 snap 309 seq_num 987) 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2903: rollback oid 41 current snap is 316 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: finishing write tid 4 to smithi20328086-35 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 309 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: finishing write tid 5 to smithi20328086-35 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: oid 35 updating version 0 to 1102 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: oid 35 updating version 1102 to 1103 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: oid 35 updating version 1103 to 1105 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: oid 35 version 1105 is already newer than 1104 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1105 (ObjNum 1017 snap 316 seq_num 1017) dirty exists 2024-09-17T15:45:26.731 INFO:tasks.rados.rados.0.smithi203.stdout:2898: left oid 35 (ObjNum 1017 snap 316 seq_num 1017) 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2899: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2899: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2899: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1514 (ObjNum 953 snap 288 seq_num 953) dirty exists 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2898: done (4 left) 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2899: done (3 left) 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2904: read oid 2 snap -1 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2904: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2905: write (excl) oid 38 current snap is 316 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2901: finishing write tid 1 to smithi20328086-34 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2901: finishing write tid 2 to smithi20328086-34 2024-09-17T15:45:26.732 INFO:tasks.rados.rados.0.smithi203.stdout:2905: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-17T15:45:26.745 INFO:tasks.rados.rados.0.smithi203.stdout:2905: writing smithi20328086-38 from 726333 to 1396366 tid 1 2024-09-17T15:45:26.758 INFO:tasks.rados.rados.0.smithi203.stdout:2905: writing smithi20328086-38 from 2096210 to 2820563 tid 2 2024-09-17T15:45:26.758 INFO:tasks.rados.rados.0.smithi203.stdout:2906: read oid 14 snap -1 2024-09-17T15:45:26.758 INFO:tasks.rados.rados.0.smithi203.stdout:2906: expect (ObjNum 1012 snap 316 seq_num 1012) 2024-09-17T15:45:26.758 INFO:tasks.rados.rados.0.smithi203.stdout:2907: write (excl) oid 37 current snap is 316 2024-09-17T15:45:26.758 INFO:tasks.rados.rados.0.smithi203.stdout:2907: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-17T15:45:26.770 INFO:tasks.rados.rados.0.smithi203.stdout:2907: writing smithi20328086-37 from 743140 to 1488422 tid 1 2024-09-17T15:45:26.779 INFO:tasks.rados.rados.0.smithi203.stdout:2907: writing smithi20328086-37 from 2154692 to 2622703 tid 2 2024-09-17T15:45:26.779 INFO:tasks.rados.rados.0.smithi203.stdout:2908: snap_remove snap 305 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2908: done (7 left) 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2909: delete oid 36 current snap is 316 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2903: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 0 (ObjNum 20433 snap 0 seq_num 4108903408) dirty dne 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2901: finishing write tid 3 to smithi20328086-34 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2901: finishing write tid 4 to smithi20328086-34 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2901: finishing write tid 5 to smithi20328086-34 2024-09-17T15:45:27.414 INFO:tasks.rados.rados.0.smithi203.stdout:2901: oid 34 updating version 0 to 1107 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:2901: oid 34 updating version 1107 to 1109 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:2901: oid 34 version 1109 is already newer than 1108 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:2901: oid 34 version 1109 is already newer than 1106 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1109 (ObjNum 1018 snap 316 seq_num 1018) dirty exists 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:2901: left oid 34 (ObjNum 1018 snap 316 seq_num 1018) 2024-09-17T15:45:27.415 INFO:tasks.rados.rados.0.smithi203.stdout:2902: expect (ObjNum 987 snap 309 seq_num 987) 2024-09-17T15:45:27.452 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2088465 2024-09-17T15:45:27.497 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1997104 2024-09-17T15:45:27.507 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 307593 2024-09-17T15:45:27.561 INFO:tasks.rados.rados.0.smithi203.stdout:2904: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:27.561 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:27.695 INFO:tasks.rados.rados.0.smithi203.stdout:2905: finishing write tid 1 to smithi20328086-38 2024-09-17T15:45:27.696 INFO:tasks.rados.rados.0.smithi203.stdout:2906: expect (ObjNum 1012 snap 316 seq_num 1012) 2024-09-17T15:45:27.727 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:27.763 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1747016 2024-09-17T15:45:27.783 INFO:tasks.rados.rados.0.smithi203.stdout:2905: finishing write tid 2 to smithi20328086-38 2024-09-17T15:45:27.783 INFO:tasks.rados.rados.0.smithi203.stdout:2905: finishing write tid 3 to smithi20328086-38 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2901: done (7 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2902: done (6 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2903: done (5 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2904: done (4 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2906: done (3 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2909: done (2 left) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2910: read oid 43 snap 311 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2910: expect deleted 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2911: read oid 12 snap 309 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2911: expect deleted 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2912: read oid 46 snap -1 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2912: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:45:27.784 INFO:tasks.rados.rados.0.smithi203.stdout:2913: rollback oid 9 current snap is 316 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 313 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2905: finishing write tid 4 to smithi20328086-38 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2905: oid 38 updating version 0 to 1370 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2905: oid 38 version 1370 is already newer than 1369 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2905: oid 38 updating version 1370 to 1371 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1371 (ObjNum 1019 snap 316 seq_num 1019) dirty exists 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2905: left oid 38 (ObjNum 1019 snap 316 seq_num 1019) 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: finishing write tid 1 to smithi20328086-37 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: finishing write tid 2 to smithi20328086-37 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: finishing write tid 3 to smithi20328086-37 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: finishing write tid 4 to smithi20328086-37 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: oid 37 updating version 0 to 1515 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: oid 37 updating version 1515 to 1517 2024-09-17T15:45:27.785 INFO:tasks.rados.rados.0.smithi203.stdout:2907: oid 37 version 1517 is already newer than 1516 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1517 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:2907: left oid 37 (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:2905: done (5 left) 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:2907: done (4 left) 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:2914: write (excl) oid 34 current snap is 316 2024-09-17T15:45:27.786 INFO:tasks.rados.rados.0.smithi203.stdout:2914: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-17T15:45:27.790 INFO:tasks.rados.rados.0.smithi203.stdout:2914: writing smithi20328086-34 from 776754 to 1272534 tid 1 2024-09-17T15:45:27.797 INFO:tasks.rados.rados.0.smithi203.stdout:2914: writing smithi20328086-34 from 1871656 to 2343336 tid 2 2024-09-17T15:45:27.802 INFO:tasks.rados.rados.0.smithi203.stdout:2914: writing smithi20328086-34 from 2965828 to 3176754 tid 3 2024-09-17T15:45:27.802 INFO:tasks.rados.rados.0.smithi203.stdout:2915: write oid 49 current snap is 316 2024-09-17T15:45:27.802 INFO:tasks.rados.rados.0.smithi203.stdout:2915: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-17T15:45:27.810 INFO:tasks.rados.rados.0.smithi203.stdout:2915: writing smithi20328086-49 from 793561 to 1364590 tid 1 2024-09-17T15:45:27.816 INFO:tasks.rados.rados.0.smithi203.stdout:2915: writing smithi20328086-49 from 1813785 to 2545476 tid 2 2024-09-17T15:45:27.821 INFO:tasks.rados.rados.0.smithi203.stdout:2915: writing smithi20328086-49 from 2993251 to 3193561 tid 3 2024-09-17T15:45:27.821 INFO:tasks.rados.rados.0.smithi203.stdout:2916: copy_from oid 29 from oid 22 current snap is 316 2024-09-17T15:45:27.821 INFO:tasks.rados.rados.0.smithi203.stdout:2912: expect (ObjNum 559 snap 166 seq_num 559) 2024-09-17T15:45:27.906 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2913: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2913: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2913: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1286 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2914: finishing write tid 1 to smithi20328086-34 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2914: finishing write tid 2 to smithi20328086-34 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2914: finishing write tid 3 to smithi20328086-34 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2914: finishing write tid 4 to smithi20328086-34 2024-09-17T15:45:27.983 INFO:tasks.rados.rados.0.smithi203.stdout:2914: finishing write tid 5 to smithi20328086-34 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: oid 34 updating version 0 to 1113 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: oid 34 version 1113 is already newer than 1112 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: oid 34 version 1113 is already newer than 1111 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: oid 34 version 1113 is already newer than 1110 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1113 (ObjNum 1022 snap 316 seq_num 1022) dirty exists 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: left oid 34 (ObjNum 1022 snap 316 seq_num 1022) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2910: done (6 left) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2911: done (5 left) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2912: done (4 left) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2913: done (3 left) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2914: done (2 left) 2024-09-17T15:45:27.984 INFO:tasks.rados.rados.0.smithi203.stdout:2917: snap_remove snap 308 2024-09-17T15:45:28.704 INFO:tasks.rados.rados.0.smithi203.stdout:2917: done (2 left) 2024-09-17T15:45:28.704 INFO:tasks.rados.rados.0.smithi203.stdout:2918: copy_from oid 36 from oid 45 current snap is 316 2024-09-17T15:45:28.704 INFO:tasks.rados.rados.0.smithi203.stdout:2915: finishing write tid 1 to smithi20328086-49 2024-09-17T15:45:28.704 INFO:tasks.rados.rados.0.smithi203.stdout:2915: finishing write tid 2 to smithi20328086-49 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: finishing write tid 3 to smithi20328086-49 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: finishing write tid 4 to smithi20328086-49 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: finishing write tid 5 to smithi20328086-49 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: oid 49 updating version 0 to 1526 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: oid 49 version 1526 is already newer than 1525 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: oid 49 version 1526 is already newer than 1523 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: oid 49 version 1526 is already newer than 1524 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1526 (ObjNum 1023 snap 316 seq_num 1023) dirty exists 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: left oid 49 (ObjNum 1023 snap 316 seq_num 1023) 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2916: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2916: finishing copy_from to smithi20328086-29 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1286 (ObjNum 996 snap 311 seq_num 996) dirty exists 2024-09-17T15:45:28.705 INFO:tasks.rados.rados.0.smithi203.stdout:2915: done (2 left) 2024-09-17T15:45:28.706 INFO:tasks.rados.rados.0.smithi203.stdout:2916: done (1 left) 2024-09-17T15:45:28.706 INFO:tasks.rados.rados.0.smithi203.stdout:2919: write oid 17 current snap is 316 2024-09-17T15:45:28.706 INFO:tasks.rados.rados.0.smithi203.stdout:2919: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-17T15:45:28.711 INFO:tasks.rados.rados.0.smithi203.stdout:2919: writing smithi20328086-17 from 410368 to 1056646 tid 1 2024-09-17T15:45:28.716 INFO:tasks.rados.rados.0.smithi203.stdout:2919: writing smithi20328086-17 from 1472267 to 1947616 tid 2 2024-09-17T15:45:28.722 INFO:tasks.rados.rados.0.smithi203.stdout:2919: writing smithi20328086-17 from 2504321 to 2909921 tid 3 2024-09-17T15:45:28.722 INFO:tasks.rados.rados.0.smithi203.stdout:2920: copy_from oid 38 from oid 32 current snap is 316 2024-09-17T15:45:28.722 INFO:tasks.rados.rados.0.smithi203.stdout:2918: finishing copy_from to smithi20328086-36 2024-09-17T15:45:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2918: got expected ENOENT (src dne) 2024-09-17T15:45:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2918: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:45:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2918: done (2 left) 2024-09-17T15:45:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2921: write (excl) oid 16 current snap is 316 2024-09-17T15:45:28.723 INFO:tasks.rados.rados.0.smithi203.stdout:2921: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-17T15:45:28.729 INFO:tasks.rados.rados.0.smithi203.stdout:2921: writing smithi20328086-16 from 427175 to 1148702 tid 1 2024-09-17T15:45:28.738 INFO:tasks.rados.rados.0.smithi203.stdout:2921: writing smithi20328086-16 from 1930749 to 2666109 tid 2 2024-09-17T15:45:28.738 INFO:tasks.rados.rados.0.smithi203.stdout:2922: copy_from oid 50 from oid 41 current snap is 316 2024-09-17T15:45:28.738 INFO:tasks.rados.rados.0.smithi203.stdout:2921: finishing write tid 1 to smithi20328086-16 2024-09-17T15:45:28.738 INFO:tasks.rados.rados.0.smithi203.stdout:2923: delete oid 10 current snap is 316 2024-09-17T15:45:28.741 INFO:tasks.rados.rados.0.smithi203.stdout:2922: finishing copy_from to smithi20328086-50 2024-09-17T15:45:28.741 INFO:tasks.rados.rados.0.smithi203.stdout:2922: got expected ENOENT (src dne) 2024-09-17T15:45:28.741 INFO:tasks.rados.rados.0.smithi203.stdout:2922: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:45:28.743 INFO:tasks.rados.rados.0.smithi203.stdout:2919: finishing write tid 1 to smithi20328086-17 2024-09-17T15:45:28.744 INFO:tasks.rados.rados.0.smithi203.stdout:2921: finishing write tid 2 to smithi20328086-16 2024-09-17T15:45:28.744 INFO:tasks.rados.rados.0.smithi203.stdout:2921: finishing write tid 3 to smithi20328086-16 2024-09-17T15:45:28.744 INFO:tasks.rados.rados.0.smithi203.stdout:2921: finishing write tid 4 to smithi20328086-16 2024-09-17T15:45:28.744 INFO:tasks.rados.rados.0.smithi203.stdout:2921: oid 16 updating version 0 to 1520 2024-09-17T15:45:28.745 INFO:tasks.rados.rados.0.smithi203.stdout:2921: oid 16 version 1520 is already newer than 1519 2024-09-17T15:45:28.745 INFO:tasks.rados.rados.0.smithi203.stdout:2921: oid 16 version 1520 is already newer than 1518 2024-09-17T15:45:28.745 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1520 (ObjNum 1025 snap 316 seq_num 1025) dirty exists 2024-09-17T15:45:28.745 INFO:tasks.rados.rados.0.smithi203.stdout:2921: left oid 16 (ObjNum 1025 snap 316 seq_num 1025) 2024-09-17T15:45:28.746 INFO:tasks.rados.rados.0.smithi203.stdout:2919: finishing write tid 2 to smithi20328086-17 2024-09-17T15:45:28.748 INFO:tasks.rados.rados.0.smithi203.stdout:2919: finishing write tid 3 to smithi20328086-17 2024-09-17T15:45:28.748 INFO:tasks.rados.rados.0.smithi203.stdout:2919: finishing write tid 4 to smithi20328086-17 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: finishing write tid 5 to smithi20328086-17 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: oid 17 updating version 0 to 1314 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: oid 17 updating version 1314 to 1315 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: oid 17 updating version 1315 to 1316 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: oid 17 updating version 1316 to 1317 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1317 (ObjNum 1024 snap 316 seq_num 1024) dirty exists 2024-09-17T15:45:28.749 INFO:tasks.rados.rados.0.smithi203.stdout:2919: left oid 17 (ObjNum 1024 snap 316 seq_num 1024) 2024-09-17T15:45:28.759 INFO:tasks.rados.rados.0.smithi203.stdout:2920: finishing copy_from racing read to smithi20328086-38 2024-09-17T15:45:28.760 INFO:tasks.rados.rados.0.smithi203.stdout:2919: done (4 left) 2024-09-17T15:45:28.760 INFO:tasks.rados.rados.0.smithi203.stdout:2921: done (3 left) 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2922: done (2 left) 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2923: done (1 left) 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2924: read oid 1 snap -1 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2924: expect deleted 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2925: read oid 19 snap -1 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2925: expect (ObjNum 994 snap 311 seq_num 994) 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2926: read oid 36 snap 309 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2926: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:45:28.761 INFO:tasks.rados.rados.0.smithi203.stdout:2927: snap_create 2024-09-17T15:45:28.764 INFO:tasks.rados.rados.0.smithi203.stdout:2920: finishing copy_from to smithi20328086-38 2024-09-17T15:45:28.764 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1196 (ObjNum 949 snap 287 seq_num 949) dirty exists 2024-09-17T15:45:28.766 INFO:tasks.rados.rados.0.smithi203.stdout:2925: expect (ObjNum 994 snap 311 seq_num 994) 2024-09-17T15:45:28.900 INFO:tasks.rados.rados.0.smithi203.stdout:2926: expect (ObjNum 888 snap 271 seq_num 888) 2024-09-17T15:45:28.911 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1264454 2024-09-17T15:45:28.979 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:29.727 INFO:tasks.rados.rados.0.smithi203.stdout:2920: done (4 left) 2024-09-17T15:45:29.727 INFO:tasks.rados.rados.0.smithi203.stdout:2924: done (3 left) 2024-09-17T15:45:29.727 INFO:tasks.rados.rados.0.smithi203.stdout:2925: done (2 left) 2024-09-17T15:45:29.728 INFO:tasks.rados.rados.0.smithi203.stdout:2926: done (1 left) 2024-09-17T15:45:29.728 INFO:tasks.rados.rados.0.smithi203.stdout:2927: done (0 left) 2024-09-17T15:45:29.728 INFO:tasks.rados.rados.0.smithi203.stdout:2928: delete oid 15 current snap is 317 2024-09-17T15:45:29.736 INFO:tasks.rados.rados.0.smithi203.stdout:2928: done (0 left) 2024-09-17T15:45:29.737 INFO:tasks.rados.rados.0.smithi203.stdout:2929: rollback oid 22 current snap is 317 2024-09-17T15:45:29.737 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 22 to 316 2024-09-17T15:45:29.737 INFO:tasks.rados.rados.0.smithi203.stdout:2930: copy_from oid 46 from oid 9 current snap is 317 2024-09-17T15:45:29.737 INFO:tasks.rados.rados.0.smithi203.stdout:2931: write oid 26 current snap is 317 2024-09-17T15:45:29.737 INFO:tasks.rados.rados.0.smithi203.stdout:2931: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-17T15:45:29.751 INFO:tasks.rados.rados.0.smithi203.stdout:2931: writing smithi20328086-26 from 477596 to 1141223 tid 1 2024-09-17T15:45:29.765 INFO:tasks.rados.rados.0.smithi203.stdout:2931: writing smithi20328086-26 from 1706195 to 2188882 tid 2 2024-09-17T15:45:29.778 INFO:tasks.rados.rados.0.smithi203.stdout:2931: writing smithi20328086-26 from 2614013 to 3181054 tid 3 2024-09-17T15:45:29.778 INFO:tasks.rados.rados.0.smithi203.stdout:2932: snap_create 2024-09-17T15:45:29.778 INFO:tasks.rados.rados.0.smithi203.stdout:2929: finishing rollback tid 0 to smithi20328086-22 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:2929: finishing rollback tid 1 to smithi20328086-22 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:2929: finishing rollback tid 2 to smithi20328086-22 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1293 (ObjNum 996 snap 311 seq_num 996) dirty exists 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:2931: finishing write tid 1 to smithi20328086-26 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:2930: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:45:29.779 INFO:tasks.rados.rados.0.smithi203.stdout:2931: finishing write tid 2 to smithi20328086-26 2024-09-17T15:45:29.780 INFO:tasks.rados.rados.0.smithi203.stdout:2930: finishing copy_from to smithi20328086-46 2024-09-17T15:45:29.780 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1286 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:45:29.787 INFO:tasks.rados.rados.0.smithi203.stdout:2931: finishing write tid 3 to smithi20328086-26 2024-09-17T15:45:29.787 INFO:tasks.rados.rados.0.smithi203.stdout:2931: finishing write tid 4 to smithi20328086-26 2024-09-17T15:45:29.787 INFO:tasks.rados.rados.0.smithi203.stdout:2931: finishing write tid 5 to smithi20328086-26 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:2931: oid 26 updating version 0 to 1530 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:2931: oid 26 updating version 1530 to 1531 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:2931: oid 26 updating version 1531 to 1532 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:2931: oid 26 updating version 1532 to 1533 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1533 (ObjNum 1028 snap 317 seq_num 1028) dirty exists 2024-09-17T15:45:29.788 INFO:tasks.rados.rados.0.smithi203.stdout:2931: left oid 26 (ObjNum 1028 snap 317 seq_num 1028) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2929: done (3 left) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2930: done (2 left) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2931: done (1 left) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2932: done (0 left) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2933: read oid 2 snap -1 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2933: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:30.735 INFO:tasks.rados.rados.0.smithi203.stdout:2934: delete oid 47 current snap is 318 2024-09-17T15:45:30.741 INFO:tasks.rados.rados.0.smithi203.stdout:2933: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:30.741 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 147937 2024-09-17T15:45:30.840 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2062039 2024-09-17T15:45:30.887 INFO:tasks.rados.rados.0.smithi203.stdout:2933: done (1 left) 2024-09-17T15:45:30.887 INFO:tasks.rados.rados.0.smithi203.stdout:2934: done (0 left) 2024-09-17T15:45:30.887 INFO:tasks.rados.rados.0.smithi203.stdout:2935: read oid 49 snap -1 2024-09-17T15:45:30.888 INFO:tasks.rados.rados.0.smithi203.stdout:2935: expect (ObjNum 1023 snap 316 seq_num 1023) 2024-09-17T15:45:30.888 INFO:tasks.rados.rados.0.smithi203.stdout:2936: delete oid 29 current snap is 318 2024-09-17T15:45:30.896 INFO:tasks.rados.rados.0.smithi203.stdout:2936: done (1 left) 2024-09-17T15:45:30.896 INFO:tasks.rados.rados.0.smithi203.stdout:2937: rollback oid 6 current snap is 318 2024-09-17T15:45:30.896 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 309 2024-09-17T15:45:30.896 INFO:tasks.rados.rados.0.smithi203.stdout:2938: rollback oid 40 current snap is 318 2024-09-17T15:45:30.896 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 317 2024-09-17T15:45:30.897 INFO:tasks.rados.rados.0.smithi203.stdout:2939: rollback oid 1 current snap is 318 2024-09-17T15:45:30.897 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 304 2024-09-17T15:45:30.897 INFO:tasks.rados.rados.0.smithi203.stdout:2940: snap_create 2024-09-17T15:45:30.900 INFO:tasks.rados.rados.0.smithi203.stdout:2935: expect (ObjNum 1023 snap 316 seq_num 1023) 2024-09-17T15:45:30.926 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2643064 2024-09-17T15:45:30.990 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:31.038 INFO:tasks.rados.rados.0.smithi203.stdout:2937: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:45:31.038 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:45:31.038 INFO:tasks.rados.rados.0.smithi203.stdout:2939: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:45:31.038 INFO:tasks.rados.rados.0.smithi203.stdout:2939: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:45:31.038 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1317 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:45:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:2938: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:45:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:2938: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:45:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:2938: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:45:31.039 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1323 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2935: done (4 left) 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2937: done (3 left) 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2938: done (2 left) 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2939: done (1 left) 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2940: done (0 left) 2024-09-17T15:45:31.743 INFO:tasks.rados.rados.0.smithi203.stdout:2941: delete oid 49 current snap is 319 2024-09-17T15:45:31.753 INFO:tasks.rados.rados.0.smithi203.stdout:2941: done (0 left) 2024-09-17T15:45:31.753 INFO:tasks.rados.rados.0.smithi203.stdout:2942: rollback oid 40 current snap is 319 2024-09-17T15:45:31.754 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 318 2024-09-17T15:45:31.754 INFO:tasks.rados.rados.0.smithi203.stdout:2943: rollback oid 46 current snap is 319 2024-09-17T15:45:31.754 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 309 2024-09-17T15:45:31.754 INFO:tasks.rados.rados.0.smithi203.stdout:2944: delete oid 38 current snap is 319 2024-09-17T15:45:31.772 INFO:tasks.rados.rados.0.smithi203.stdout:2944: done (2 left) 2024-09-17T15:45:31.772 INFO:tasks.rados.rados.0.smithi203.stdout:2945: read oid 28 snap -1 2024-09-17T15:45:31.772 INFO:tasks.rados.rados.0.smithi203.stdout:2945: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:45:31.773 INFO:tasks.rados.rados.0.smithi203.stdout:2946: delete oid 17 current snap is 319 2024-09-17T15:45:31.773 INFO:tasks.rados.rados.0.smithi203.stdout:2943: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:45:31.773 INFO:tasks.rados.rados.0.smithi203.stdout:2942: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:45:31.773 INFO:tasks.rados.rados.0.smithi203.stdout:2942: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:45:31.777 INFO:tasks.rados.rados.0.smithi203.stdout:2942: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:45:31.779 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1327 (ObjNum 973 snap 294 seq_num 973) dirty exists 2024-09-17T15:45:31.780 INFO:tasks.rados.rados.0.smithi203.stdout:2942: done (3 left) 2024-09-17T15:45:31.780 INFO:tasks.rados.rados.0.smithi203.stdout:2946: done (2 left) 2024-09-17T15:45:31.780 INFO:tasks.rados.rados.0.smithi203.stdout:2947: write oid 16 current snap is 319 2024-09-17T15:45:31.780 INFO:tasks.rados.rados.0.smithi203.stdout:2947: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-17T15:45:31.798 INFO:tasks.rados.rados.0.smithi203.stdout:2947: writing smithi20328086-16 from 578438 to 1009912 tid 1 2024-09-17T15:45:31.813 INFO:tasks.rados.rados.0.smithi203.stdout:2947: writing smithi20328086-16 from 1657087 to 2150781 tid 2 2024-09-17T15:45:31.831 INFO:tasks.rados.rados.0.smithi203.stdout:2947: writing smithi20328086-16 from 2778551 to 3378438 tid 3 2024-09-17T15:45:31.831 INFO:tasks.rados.rados.0.smithi203.stdout:2948: snap_create 2024-09-17T15:45:31.831 INFO:tasks.rados.rados.0.smithi203.stdout:2945: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:45:31.831 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:31.931 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:31.964 INFO:tasks.rados.rados.0.smithi203.stdout:2943: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:45:31.965 INFO:tasks.rados.rados.0.smithi203.stdout:2943: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:45:31.965 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1373 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:45:31.965 INFO:tasks.rados.rados.0.smithi203.stdout:2947: finishing write tid 1 to smithi20328086-16 2024-09-17T15:45:31.965 INFO:tasks.rados.rados.0.smithi203.stdout:2947: finishing write tid 2 to smithi20328086-16 2024-09-17T15:45:31.965 INFO:tasks.rados.rados.0.smithi203.stdout:2947: finishing write tid 3 to smithi20328086-16 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: finishing write tid 4 to smithi20328086-16 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: finishing write tid 5 to smithi20328086-16 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: oid 16 updating version 0 to 1524 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: oid 16 version 1524 is already newer than 1523 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: oid 16 version 1524 is already newer than 1522 2024-09-17T15:45:31.966 INFO:tasks.rados.rados.0.smithi203.stdout:2947: oid 16 version 1524 is already newer than 1521 2024-09-17T15:45:31.967 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1524 (ObjNum 1034 snap 319 seq_num 1034) dirty exists 2024-09-17T15:45:31.967 INFO:tasks.rados.rados.0.smithi203.stdout:2947: left oid 16 (ObjNum 1034 snap 319 seq_num 1034) 2024-09-17T15:45:32.415 INFO:tasks.rados.rados.0.smithi203.stdout:2943: done (3 left) 2024-09-17T15:45:32.415 INFO:tasks.rados.rados.0.smithi203.stdout:2945: done (2 left) 2024-09-17T15:45:32.415 INFO:tasks.rados.rados.0.smithi203.stdout:2947: done (1 left) 2024-09-17T15:45:32.416 INFO:tasks.rados.rados.0.smithi203.stdout:2948: done (0 left) 2024-09-17T15:45:32.416 INFO:tasks.rados.rados.0.smithi203.stdout:2949: write (excl) oid 50 current snap is 320 2024-09-17T15:45:32.416 INFO:tasks.rados.rados.0.smithi203.stdout:2949: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-17T15:45:32.429 INFO:tasks.rados.rados.0.smithi203.stdout:2949: writing smithi20328086-50 from 595245 to 1101968 tid 1 2024-09-17T15:45:32.444 INFO:tasks.rados.rados.0.smithi203.stdout:2949: writing smithi20328086-50 from 1715569 to 2352921 tid 2 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2949: writing smithi20328086-50 from 2805974 to 3368272 tid 3 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2950: read oid 10 snap 300 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2950: expect deleted 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2951: copy_from oid 24 from oid 2 current snap is 320 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2949: finishing write tid 1 to smithi20328086-50 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2949: finishing write tid 2 to smithi20328086-50 2024-09-17T15:45:32.457 INFO:tasks.rados.rados.0.smithi203.stdout:2952: snap_create 2024-09-17T15:45:32.469 INFO:tasks.rados.rados.0.smithi203.stdout:2949: finishing write tid 3 to smithi20328086-50 2024-09-17T15:45:32.470 INFO:tasks.rados.rados.0.smithi203.stdout:2949: finishing write tid 4 to smithi20328086-50 2024-09-17T15:45:32.470 INFO:tasks.rados.rados.0.smithi203.stdout:2949: finishing write tid 5 to smithi20328086-50 2024-09-17T15:45:32.470 INFO:tasks.rados.rados.0.smithi203.stdout:2949: oid 50 updating version 0 to 1201 2024-09-17T15:45:32.470 INFO:tasks.rados.rados.0.smithi203.stdout:2949: oid 50 version 1201 is already newer than 1200 2024-09-17T15:45:32.470 INFO:tasks.rados.rados.0.smithi203.stdout:2949: oid 50 version 1201 is already newer than 1198 2024-09-17T15:45:32.471 INFO:tasks.rados.rados.0.smithi203.stdout:2949: oid 50 version 1201 is already newer than 1199 2024-09-17T15:45:32.471 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1201 (ObjNum 1035 snap 320 seq_num 1035) dirty exists 2024-09-17T15:45:32.471 INFO:tasks.rados.rados.0.smithi203.stdout:2949: left oid 50 (ObjNum 1035 snap 320 seq_num 1035) 2024-09-17T15:45:32.473 INFO:tasks.rados.rados.0.smithi203.stdout:2951: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:45:32.478 INFO:tasks.rados.rados.0.smithi203.stdout:2951: finishing copy_from to smithi20328086-24 2024-09-17T15:45:32.478 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1174 (ObjNum 968 snap 294 seq_num 968) dirty exists 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2949: done (3 left) 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2950: done (2 left) 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2951: done (1 left) 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2952: done (0 left) 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2953: write oid 7 current snap is 321 2024-09-17T15:45:33.424 INFO:tasks.rados.rados.0.smithi203.stdout:2953: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-17T15:45:33.440 INFO:tasks.rados.rados.0.smithi203.stdout:2953: writing smithi20328086-7 from 612052 to 1194024 tid 1 2024-09-17T15:45:33.449 INFO:tasks.rados.rados.0.smithi203.stdout:2953: writing smithi20328086-7 from 1657698 to 2155061 tid 2 2024-09-17T15:45:33.465 INFO:tasks.rados.rados.0.smithi203.stdout:2953: writing smithi20328086-7 from 2717044 to 3412052 tid 3 2024-09-17T15:45:33.466 INFO:tasks.rados.rados.0.smithi203.stdout:2954: rollback oid 9 current snap is 321 2024-09-17T15:45:33.466 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 300 2024-09-17T15:45:33.466 INFO:tasks.rados.rados.0.smithi203.stdout:2953: finishing write tid 1 to smithi20328086-7 2024-09-17T15:45:33.466 INFO:tasks.rados.rados.0.smithi203.stdout:2953: finishing write tid 2 to smithi20328086-7 2024-09-17T15:45:33.466 INFO:tasks.rados.rados.0.smithi203.stdout:2955: delete oid 31 current snap is 321 2024-09-17T15:45:33.474 INFO:tasks.rados.rados.0.smithi203.stdout:2955: done (2 left) 2024-09-17T15:45:33.474 INFO:tasks.rados.rados.0.smithi203.stdout:2956: rollback oid 8 current snap is 321 2024-09-17T15:45:33.474 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 316 2024-09-17T15:45:33.474 INFO:tasks.rados.rados.0.smithi203.stdout:2957: snap_remove snap 311 2024-09-17T15:45:34.431 INFO:tasks.rados.rados.0.smithi203.stdout:2957: done (3 left) 2024-09-17T15:45:34.432 INFO:tasks.rados.rados.0.smithi203.stdout:2958: read oid 32 snap 294 2024-09-17T15:45:34.432 INFO:tasks.rados.rados.0.smithi203.stdout:2958: expect (ObjNum 949 snap 287 seq_num 949) 2024-09-17T15:45:34.432 INFO:tasks.rados.rados.0.smithi203.stdout:2959: snap_remove snap 296 2024-09-17T15:45:35.436 INFO:tasks.rados.rados.0.smithi203.stdout:2959: done (4 left) 2024-09-17T15:45:35.436 INFO:tasks.rados.rados.0.smithi203.stdout:2960: write (excl) oid 4 current snap is 321 2024-09-17T15:45:35.436 INFO:tasks.rados.rados.0.smithi203.stdout:2960: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-17T15:45:35.461 INFO:tasks.rados.rados.0.smithi203.stdout:2960: writing smithi20328086-4 from 645666 to 1378136 tid 1 2024-09-17T15:45:35.484 INFO:tasks.rados.rados.0.smithi203.stdout:2960: writing smithi20328086-4 from 2174662 to 2675694 tid 2 2024-09-17T15:45:35.493 INFO:tasks.rados.rados.0.smithi203.stdout:2960: writing smithi20328086-4 from 3288243 to 3445666 tid 3 2024-09-17T15:45:35.494 INFO:tasks.rados.rados.0.smithi203.stdout:2961: snap_remove snap 304 2024-09-17T15:45:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:2961: done (5 left) 2024-09-17T15:45:36.463 INFO:tasks.rados.rados.0.smithi203.stdout:2962: delete oid 33 current snap is 321 2024-09-17T15:45:36.467 INFO:tasks.rados.rados.0.smithi203.stdout:2953: finishing write tid 3 to smithi20328086-7 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: finishing write tid 4 to smithi20328086-7 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: finishing write tid 5 to smithi20328086-7 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: oid 7 updating version 0 to 1176 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: oid 7 updating version 1176 to 1177 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: oid 7 updating version 1177 to 1178 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: oid 7 version 1178 is already newer than 1175 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1178 (ObjNum 1036 snap 321 seq_num 1036) dirty exists 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2953: left oid 7 (ObjNum 1036 snap 321 seq_num 1036) 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2954: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2954: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:45:36.468 INFO:tasks.rados.rados.0.smithi203.stdout:2954: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1290 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:2956: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:2956: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:2956: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 918 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:45:36.469 INFO:tasks.rados.rados.0.smithi203.stdout:2958: expect (ObjNum 949 snap 287 seq_num 949) 2024-09-17T15:45:36.763 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2066592 2024-09-17T15:45:36.820 INFO:tasks.rados.rados.0.smithi203.stdout:2960: finishing write tid 1 to smithi20328086-4 2024-09-17T15:45:36.820 INFO:tasks.rados.rados.0.smithi203.stdout:2960: finishing write tid 2 to smithi20328086-4 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: finishing write tid 3 to smithi20328086-4 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: finishing write tid 4 to smithi20328086-4 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: finishing write tid 5 to smithi20328086-4 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: oid 4 updating version 0 to 1177 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: oid 4 updating version 1177 to 1179 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: oid 4 version 1179 is already newer than 1178 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: oid 4 updating version 1179 to 1180 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1180 (ObjNum 1038 snap 321 seq_num 1038) dirty exists 2024-09-17T15:45:36.821 INFO:tasks.rados.rados.0.smithi203.stdout:2960: left oid 4 (ObjNum 1038 snap 321 seq_num 1038) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2953: done (5 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2954: done (4 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2956: done (3 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2958: done (2 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2960: done (1 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2962: done (0 left) 2024-09-17T15:45:36.823 INFO:tasks.rados.rados.0.smithi203.stdout:2963: rollback oid 39 current snap is 321 2024-09-17T15:45:36.824 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 39 to 320 2024-09-17T15:45:36.824 INFO:tasks.rados.rados.0.smithi203.stdout:2964: delete oid 13 current snap is 321 2024-09-17T15:45:36.828 INFO:tasks.rados.rados.0.smithi203.stdout:2964: done (1 left) 2024-09-17T15:45:36.828 INFO:tasks.rados.rados.0.smithi203.stdout:2965: copy_from oid 47 from oid 16 current snap is 321 2024-09-17T15:45:36.828 INFO:tasks.rados.rados.0.smithi203.stdout:2966: write oid 27 current snap is 321 2024-09-17T15:45:36.828 INFO:tasks.rados.rados.0.smithi203.stdout:2966: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-17T15:45:36.848 INFO:tasks.rados.rados.0.smithi203.stdout:2966: writing smithi20328086-27 from 696087 to 1254304 tid 1 2024-09-17T15:45:36.869 INFO:tasks.rados.rados.0.smithi203.stdout:2966: writing smithi20328086-27 from 1833755 to 2482114 tid 2 2024-09-17T15:45:36.885 INFO:tasks.rados.rados.0.smithi203.stdout:2966: writing smithi20328086-27 from 3137806 to 3496087 tid 3 2024-09-17T15:45:36.885 INFO:tasks.rados.rados.0.smithi203.stdout:2967: read oid 40 snap -1 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2967: expect (ObjNum 973 snap 294 seq_num 973) 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2968: copy_from oid 33 from oid 12 current snap is 321 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2963: finishing rollback tid 0 to smithi20328086-39 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2963: finishing rollback tid 1 to smithi20328086-39 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2963: finishing rollback tid 2 to smithi20328086-39 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1527 (ObjNum 1013 snap 316 seq_num 1013) dirty exists 2024-09-17T15:45:36.886 INFO:tasks.rados.rados.0.smithi203.stdout:2965: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:2965: finishing copy_from to smithi20328086-47 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1524 (ObjNum 1034 snap 319 seq_num 1034) dirty exists 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:2966: finishing write tid 1 to smithi20328086-27 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:2966: finishing write tid 2 to smithi20328086-27 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:2963: done (4 left) 2024-09-17T15:45:36.887 INFO:tasks.rados.rados.0.smithi203.stdout:2965: done (3 left) 2024-09-17T15:45:36.888 INFO:tasks.rados.rados.0.smithi203.stdout:2969: snap_create 2024-09-17T15:45:36.888 INFO:tasks.rados.rados.0.smithi203.stdout:2968: finishing copy_from to smithi20328086-33 2024-09-17T15:45:36.888 INFO:tasks.rados.rados.0.smithi203.stdout:2968: got expected ENOENT (src dne) 2024-09-17T15:45:36.889 INFO:tasks.rados.rados.0.smithi203.stdout:2968: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:45:36.890 INFO:tasks.rados.rados.0.smithi203.stdout:2966: finishing write tid 3 to smithi20328086-27 2024-09-17T15:45:36.890 INFO:tasks.rados.rados.0.smithi203.stdout:2966: finishing write tid 4 to smithi20328086-27 2024-09-17T15:45:36.890 INFO:tasks.rados.rados.0.smithi203.stdout:2966: finishing write tid 5 to smithi20328086-27 2024-09-17T15:45:36.890 INFO:tasks.rados.rados.0.smithi203.stdout:2966: oid 27 updating version 0 to 1374 2024-09-17T15:45:36.890 INFO:tasks.rados.rados.0.smithi203.stdout:2966: oid 27 updating version 1374 to 1376 2024-09-17T15:45:36.891 INFO:tasks.rados.rados.0.smithi203.stdout:2966: oid 27 version 1376 is already newer than 1375 2024-09-17T15:45:36.891 INFO:tasks.rados.rados.0.smithi203.stdout:2966: oid 27 updating version 1376 to 1377 2024-09-17T15:45:36.891 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1377 (ObjNum 1041 snap 321 seq_num 1041) dirty exists 2024-09-17T15:45:36.891 INFO:tasks.rados.rados.0.smithi203.stdout:2966: left oid 27 (ObjNum 1041 snap 321 seq_num 1041) 2024-09-17T15:45:36.893 INFO:tasks.rados.rados.0.smithi203.stdout:2967: expect (ObjNum 973 snap 294 seq_num 973) 2024-09-17T15:45:36.930 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1336934 2024-09-17T15:45:36.985 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 510849 2024-09-17T15:45:37.059 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2127077 2024-09-17T15:45:37.415 INFO:tasks.rados.rados.0.smithi203.stdout:2966: done (3 left) 2024-09-17T15:45:37.415 INFO:tasks.rados.rados.0.smithi203.stdout:2967: done (2 left) 2024-09-17T15:45:37.416 INFO:tasks.rados.rados.0.smithi203.stdout:2968: done (1 left) 2024-09-17T15:45:37.416 INFO:tasks.rados.rados.0.smithi203.stdout:2969: done (0 left) 2024-09-17T15:45:37.416 INFO:tasks.rados.rados.0.smithi203.stdout:2970: snap_create 2024-09-17T15:45:38.418 INFO:tasks.rados.rados.0.smithi203.stdout:2970: done (0 left) 2024-09-17T15:45:38.419 INFO:tasks.rados.rados.0.smithi203.stdout:2971: copy_from oid 40 from oid 28 current snap is 323 2024-09-17T15:45:38.419 INFO:tasks.rados.rados.0.smithi203.stdout:2972: snap_create 2024-09-17T15:45:38.445 INFO:tasks.rados.rados.0.smithi203.stdout:2971: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:45:38.446 INFO:tasks.rados.rados.0.smithi203.stdout:2971: finishing copy_from to smithi20328086-40 2024-09-17T15:45:38.446 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1490 (ObjNum 975 snap 295 seq_num 975) dirty exists 2024-09-17T15:45:39.427 INFO:tasks.rados.rados.0.smithi203.stdout:2971: done (1 left) 2024-09-17T15:45:39.427 INFO:tasks.rados.rados.0.smithi203.stdout:2972: done (0 left) 2024-09-17T15:45:39.427 INFO:tasks.rados.rados.0.smithi203.stdout:2973: read oid 4 snap -1 2024-09-17T15:45:39.427 INFO:tasks.rados.rados.0.smithi203.stdout:2973: expect (ObjNum 1038 snap 321 seq_num 1038) 2024-09-17T15:45:39.428 INFO:tasks.rados.rados.0.smithi203.stdout:2974: write oid 22 current snap is 324 2024-09-17T15:45:39.428 INFO:tasks.rados.rados.0.smithi203.stdout:2974: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-17T15:45:39.451 INFO:tasks.rados.rados.0.smithi203.stdout:2974: writing smithi20328086-22 from 712894 to 1462713 tid 1 2024-09-17T15:45:39.472 INFO:tasks.rados.rados.0.smithi203.stdout:2974: writing smithi20328086-22 from 2008590 to 2800607 tid 2 2024-09-17T15:45:39.482 INFO:tasks.rados.rados.0.smithi203.stdout:2974: writing smithi20328086-22 from 3281582 to 3512894 tid 3 2024-09-17T15:45:39.482 INFO:tasks.rados.rados.0.smithi203.stdout:2975: rollback oid 27 current snap is 324 2024-09-17T15:45:39.482 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 316 2024-09-17T15:45:39.482 INFO:tasks.rados.rados.0.smithi203.stdout:2973: expect (ObjNum 1038 snap 321 seq_num 1038) 2024-09-17T15:45:39.495 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 771003 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: finishing write tid 1 to smithi20328086-22 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: finishing write tid 2 to smithi20328086-22 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: finishing write tid 3 to smithi20328086-22 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: finishing write tid 4 to smithi20328086-22 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: finishing write tid 5 to smithi20328086-22 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: oid 22 updating version 0 to 1296 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: oid 22 version 1296 is already newer than 1295 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: oid 22 updating version 1296 to 1298 2024-09-17T15:45:39.674 INFO:tasks.rados.rados.0.smithi203.stdout:2974: oid 22 version 1298 is already newer than 1297 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1298 (ObjNum 1042 snap 324 seq_num 1042) dirty exists 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2974: left oid 22 (ObjNum 1042 snap 324 seq_num 1042) 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2975: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2975: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2975: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1380 (ObjNum 961 snap 290 seq_num 961) dirty exists 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2973: done (2 left) 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2974: done (1 left) 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2975: done (0 left) 2024-09-17T15:45:39.675 INFO:tasks.rados.rados.0.smithi203.stdout:2976: snap_remove snap 294 2024-09-17T15:45:40.430 INFO:tasks.rados.rados.0.smithi203.stdout:2976: done (0 left) 2024-09-17T15:45:40.430 INFO:tasks.rados.rados.0.smithi203.stdout:2977: read oid 37 snap -1 2024-09-17T15:45:40.430 INFO:tasks.rados.rados.0.smithi203.stdout:2977: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:45:40.430 INFO:tasks.rados.rados.0.smithi203.stdout:2978: write (excl) oid 35 current snap is 324 2024-09-17T15:45:40.431 INFO:tasks.rados.rados.0.smithi203.stdout:2978: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-17T15:45:40.450 INFO:tasks.rados.rados.0.smithi203.stdout:2978: writing smithi20328086-35 from 729701 to 1154769 tid 1 2024-09-17T15:45:40.469 INFO:tasks.rados.rados.0.smithi203.stdout:2978: writing smithi20328086-35 from 1667072 to 2319100 tid 2 2024-09-17T15:45:40.483 INFO:tasks.rados.rados.0.smithi203.stdout:2978: writing smithi20328086-35 from 2909005 to 3394185 tid 3 2024-09-17T15:45:40.483 INFO:tasks.rados.rados.0.smithi203.stdout:2979: rollback oid 2 current snap is 324 2024-09-17T15:45:40.483 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 319 2024-09-17T15:45:40.483 INFO:tasks.rados.rados.0.smithi203.stdout:2977: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:45:40.528 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:40.583 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:40.637 INFO:tasks.rados.rados.0.smithi203.stdout:2978: finishing write tid 1 to smithi20328086-35 2024-09-17T15:45:40.637 INFO:tasks.rados.rados.0.smithi203.stdout:2978: finishing write tid 2 to smithi20328086-35 2024-09-17T15:45:40.637 INFO:tasks.rados.rados.0.smithi203.stdout:2978: finishing write tid 3 to smithi20328086-35 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: finishing write tid 4 to smithi20328086-35 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2979: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: finishing write tid 5 to smithi20328086-35 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: oid 35 updating version 0 to 1117 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: oid 35 version 1117 is already newer than 1114 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: oid 35 version 1117 is already newer than 1116 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: oid 35 version 1117 is already newer than 1115 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1117 (ObjNum 1043 snap 324 seq_num 1043) dirty exists 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2978: left oid 35 (ObjNum 1043 snap 324 seq_num 1043) 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2979: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2979: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1181 (ObjNum 968 snap 294 seq_num 968) dirty exists 2024-09-17T15:45:40.638 INFO:tasks.rados.rados.0.smithi203.stdout:2977: done (2 left) 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:2978: done (1 left) 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:2979: done (0 left) 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:2980: rollback oid 32 current snap is 324 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 322 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:2981: write oid 21 current snap is 324 2024-09-17T15:45:40.639 INFO:tasks.rados.rados.0.smithi203.stdout:2981: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-17T15:45:40.651 INFO:tasks.rados.rados.0.smithi203.stdout:2981: writing smithi20328086-21 from 746508 to 1246825 tid 1 2024-09-17T15:45:40.669 INFO:tasks.rados.rados.0.smithi203.stdout:2981: writing smithi20328086-21 from 2009201 to 2804887 tid 2 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2981: writing smithi20328086-21 from 3220075 to 3546508 tid 3 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2982: delete oid 7 current snap is 324 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2980: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2980: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2980: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1204 (ObjNum 949 snap 287 seq_num 949) dirty exists 2024-09-17T15:45:40.679 INFO:tasks.rados.rados.0.smithi203.stdout:2981: finishing write tid 1 to smithi20328086-21 2024-09-17T15:45:40.680 INFO:tasks.rados.rados.0.smithi203.stdout:2981: finishing write tid 2 to smithi20328086-21 2024-09-17T15:45:40.686 INFO:tasks.rados.rados.0.smithi203.stdout:2981: finishing write tid 3 to smithi20328086-21 2024-09-17T15:45:40.686 INFO:tasks.rados.rados.0.smithi203.stdout:2981: finishing write tid 4 to smithi20328086-21 2024-09-17T15:45:40.686 INFO:tasks.rados.rados.0.smithi203.stdout:2980: done (2 left) 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2982: done (1 left) 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2983: read oid 44 snap 317 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2983: expect deleted 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2984: read oid 31 snap -1 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2984: expect deleted 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2985: write oid 24 current snap is 324 2024-09-17T15:45:40.687 INFO:tasks.rados.rados.0.smithi203.stdout:2985: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-17T15:45:40.703 INFO:tasks.rados.rados.0.smithi203.stdout:2985: writing smithi20328086-24 from 780122 to 1430937 tid 1 2024-09-17T15:45:40.719 INFO:tasks.rados.rados.0.smithi203.stdout:2985: writing smithi20328086-24 from 2126165 to 2925520 tid 2 2024-09-17T15:45:40.719 INFO:tasks.rados.rados.0.smithi203.stdout:2986: write oid 27 current snap is 324 2024-09-17T15:45:40.720 INFO:tasks.rados.rados.0.smithi203.stdout:2986: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-17T15:45:40.733 INFO:tasks.rados.rados.0.smithi203.stdout:2986: writing smithi20328086-27 from 796929 to 1522993 tid 1 2024-09-17T15:45:40.743 INFO:tasks.rados.rados.0.smithi203.stdout:2986: writing smithi20328086-27 from 2184647 to 2844013 tid 2 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2986: writing smithi20328086-27 from 3302344 to 3596929 tid 3 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2987: rollback oid 33 current snap is 324 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 321 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: finishing write tid 5 to smithi20328086-21 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: oid 21 updating version 0 to 1559 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: oid 21 version 1559 is already newer than 1558 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: oid 21 updating version 1559 to 1560 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: oid 21 version 1560 is already newer than 1557 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1560 (ObjNum 1044 snap 324 seq_num 1044) dirty exists 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2981: left oid 21 (ObjNum 1044 snap 324 seq_num 1044) 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2985: finishing write tid 1 to smithi20328086-24 2024-09-17T15:45:40.749 INFO:tasks.rados.rados.0.smithi203.stdout:2985: finishing write tid 2 to smithi20328086-24 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: finishing write tid 3 to smithi20328086-24 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: finishing write tid 4 to smithi20328086-24 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: oid 24 updating version 0 to 1293 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: oid 24 version 1293 is already newer than 1292 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: oid 24 version 1293 is already newer than 1291 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1293 (ObjNum 1046 snap 324 seq_num 1046) dirty exists 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2985: left oid 24 (ObjNum 1046 snap 324 seq_num 1046) 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2986: finishing write tid 1 to smithi20328086-27 2024-09-17T15:45:40.750 INFO:tasks.rados.rados.0.smithi203.stdout:2981: done (5 left) 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:2983: done (4 left) 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:2984: done (3 left) 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:2985: done (2 left) 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:2988: delete oid 46 current snap is 324 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:2987: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:45:40.751 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1193 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:45:40.755 INFO:tasks.rados.rados.0.smithi203.stdout:2986: finishing write tid 2 to smithi20328086-27 2024-09-17T15:45:40.760 INFO:tasks.rados.rados.0.smithi203.stdout:2986: finishing write tid 3 to smithi20328086-27 2024-09-17T15:45:40.761 INFO:tasks.rados.rados.0.smithi203.stdout:2986: finishing write tid 4 to smithi20328086-27 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: finishing write tid 5 to smithi20328086-27 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: oid 27 updating version 0 to 1383 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: oid 27 version 1383 is already newer than 1382 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: oid 27 updating version 1383 to 1384 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: oid 27 version 1384 is already newer than 1381 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1384 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:45:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:2986: left oid 27 (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:45:40.763 INFO:tasks.rados.rados.0.smithi203.stdout:2986: done (2 left) 2024-09-17T15:45:40.763 INFO:tasks.rados.rados.0.smithi203.stdout:2987: done (1 left) 2024-09-17T15:45:40.763 INFO:tasks.rados.rados.0.smithi203.stdout:2988: done (0 left) 2024-09-17T15:45:40.763 INFO:tasks.rados.rados.0.smithi203.stdout:2989: write oid 6 current snap is 324 2024-09-17T15:45:40.763 INFO:tasks.rados.rados.0.smithi203.stdout:2989: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-17T15:45:40.769 INFO:tasks.rados.rados.0.smithi203.stdout:2989: writing smithi20328086-6 from 430543 to 907105 tid 1 2024-09-17T15:45:40.778 INFO:tasks.rados.rados.0.smithi203.stdout:2989: writing smithi20328086-6 from 1385258 to 2048293 tid 2 2024-09-17T15:45:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:2989: writing smithi20328086-6 from 2840837 to 3510002 tid 3 2024-09-17T15:45:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:2990: snap_remove snap 316 2024-09-17T15:45:41.431 INFO:tasks.rados.rados.0.smithi203.stdout:2990: done (1 left) 2024-09-17T15:45:41.431 INFO:tasks.rados.rados.0.smithi203.stdout:2991: rollback oid 35 current snap is 324 2024-09-17T15:45:41.431 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 309 2024-09-17T15:45:41.431 INFO:tasks.rados.rados.0.smithi203.stdout:2989: finishing write tid 1 to smithi20328086-6 2024-09-17T15:45:41.431 INFO:tasks.rados.rados.0.smithi203.stdout:2989: finishing write tid 2 to smithi20328086-6 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: finishing write tid 3 to smithi20328086-6 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: finishing write tid 4 to smithi20328086-6 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: finishing write tid 5 to smithi20328086-6 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: oid 6 updating version 0 to 1321 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: oid 6 version 1321 is already newer than 1320 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: oid 6 version 1321 is already newer than 1319 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: oid 6 version 1321 is already newer than 1318 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1321 (ObjNum 1049 snap 324 seq_num 1049) dirty exists 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: left oid 6 (ObjNum 1049 snap 324 seq_num 1049) 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2989: done (1 left) 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2992: write (excl) oid 24 current snap is 324 2024-09-17T15:45:41.432 INFO:tasks.rados.rados.0.smithi203.stdout:2992: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-17T15:45:41.439 INFO:tasks.rados.rados.0.smithi203.stdout:2992: writing smithi20328086-24 from 447350 to 1115514 tid 1 2024-09-17T15:45:41.446 INFO:tasks.rados.rados.0.smithi203.stdout:2992: writing smithi20328086-24 from 1560093 to 1966786 tid 2 2024-09-17T15:45:41.456 INFO:tasks.rados.rados.0.smithi203.stdout:2992: writing smithi20328086-24 from 2584613 to 3065050 tid 3 2024-09-17T15:45:41.456 INFO:tasks.rados.rados.0.smithi203.stdout:2993: snap_remove snap 323 2024-09-17T15:45:42.710 INFO:tasks.rados.rados.0.smithi203.stdout:2993: done (2 left) 2024-09-17T15:45:42.710 INFO:tasks.rados.rados.0.smithi203.stdout:2994: write oid 6 current snap is 324 2024-09-17T15:45:42.710 INFO:tasks.rados.rados.0.smithi203.stdout:2994: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-09-17T15:45:42.722 INFO:tasks.rados.rados.0.smithi203.stdout:2994: writing smithi20328086-6 from 464157 to 1207570 tid 1 2024-09-17T15:45:42.733 INFO:tasks.rados.rados.0.smithi203.stdout:2994: writing smithi20328086-6 from 1618575 to 2285279 tid 2 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2994: writing smithi20328086-6 from 3012036 to 3420098 tid 3 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2995: rollback oid 9 current snap is 324 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 300 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2991: finishing rollback tid 0 to smithi20328086-35 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2991: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1119 (ObjNum 33 snap 0 seq_num 3892342784) dirty dne 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2992: finishing write tid 1 to smithi20328086-24 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2992: finishing write tid 2 to smithi20328086-24 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2992: finishing write tid 3 to smithi20328086-24 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2992: finishing write tid 4 to smithi20328086-24 2024-09-17T15:45:42.746 INFO:tasks.rados.rados.0.smithi203.stdout:2992: finishing write tid 5 to smithi20328086-24 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: oid 24 updating version 0 to 1297 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: oid 24 version 1297 is already newer than 1294 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: oid 24 version 1297 is already newer than 1296 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: oid 24 version 1297 is already newer than 1295 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1297 (ObjNum 1050 snap 324 seq_num 1050) dirty exists 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: left oid 24 (ObjNum 1050 snap 324 seq_num 1050) 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2994: finishing write tid 1 to smithi20328086-6 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2991: done (3 left) 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2992: done (2 left) 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2996: read oid 12 snap 312 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2996: expect (ObjNum 996 snap 311 seq_num 996) 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2997: copy_from oid 31 from oid 49 current snap is 324 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2994: finishing write tid 2 to smithi20328086-6 2024-09-17T15:45:42.747 INFO:tasks.rados.rados.0.smithi203.stdout:2998: snap_create 2024-09-17T15:45:42.748 INFO:tasks.rados.rados.0.smithi203.stdout:2997: finishing copy_from to smithi20328086-31 2024-09-17T15:45:42.748 INFO:tasks.rados.rados.0.smithi203.stdout:2997: got expected ENOENT (src dne) 2024-09-17T15:45:42.748 INFO:tasks.rados.rados.0.smithi203.stdout:2997: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:45:42.757 INFO:tasks.rados.rados.0.smithi203.stdout:2996: expect (ObjNum 996 snap 311 seq_num 996) 2024-09-17T15:45:42.757 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:42.862 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: finishing write tid 3 to smithi20328086-6 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: finishing write tid 4 to smithi20328086-6 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: finishing write tid 5 to smithi20328086-6 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: oid 6 updating version 0 to 1322 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: oid 6 updating version 1322 to 1325 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: oid 6 version 1325 is already newer than 1324 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: oid 6 version 1325 is already newer than 1323 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1325 (ObjNum 1051 snap 324 seq_num 1051) dirty exists 2024-09-17T15:45:42.892 INFO:tasks.rados.rados.0.smithi203.stdout:2994: left oid 6 (ObjNum 1051 snap 324 seq_num 1051) 2024-09-17T15:45:42.893 INFO:tasks.rados.rados.0.smithi203.stdout:2995: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:45:42.893 INFO:tasks.rados.rados.0.smithi203.stdout:2995: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:45:42.893 INFO:tasks.rados.rados.0.smithi203.stdout:2995: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:45:42.893 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1300 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2994: done (4 left) 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2995: done (3 left) 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2996: done (2 left) 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2997: done (1 left) 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2998: done (0 left) 2024-09-17T15:45:43.725 INFO:tasks.rados.rados.0.smithi203.stdout:2999: snap_create 2024-09-17T15:45:43.739 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:45:43.740 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:45:43.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:45:43.743 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:45:43.743 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:45:43.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: get_monmap_and_config 2024-09-17T15:45:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: build_initial_monmap 2024-09-17T15:45:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: monmap: 2024-09-17T15:45:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:43.868 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:43.870859+0000 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:43.870859+0000 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 1 Processor -- start 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 1 -- start start 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: init 2024-09-17T15:45:43.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-a con 0x7fcf580a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcf580a54c0 0x7fcf580a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf580a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf58137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.865+0000 7fcf604a6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5809d350 con 0x7fcf58137ea0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcf580a6530 con 0x7fcf580a50e0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fcf5809c150 con 0x7fcf580a54c0 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:43.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf604a6640 10 monclient(hunting): authenticate will time out at 2076.836028s 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51288/0 (socket says 172.21.15.86:51288) 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 -- 172.21.15.86:0/132629339 learned_addr learned my addr 172.21.15.86:0/132629339 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 10 monclient(hunting): get_auth_request con 0x7fcf580a50e0 auth_method 0 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 -- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 msgr2=0x7fcf5809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 -- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 msgr2=0x7fcf5809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 --2- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5e242640 1 --2- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:43.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 --2- 172.21.15.86:0/132629339 >> v2:172.21.15.203:3300/0 conn(0x7fcf580a54c0 0x7fcf580a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): get_auth_request con 0x7fcf580a54c0 auth_method 0 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient(hunting): handle_auth_done global_id 204166 payload 274 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 -- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 msgr2=0x7fcf5809ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 --2- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 -- 172.21.15.86:0/132629339 >> v1:172.21.15.203:6789/0 conn(0x7fcf58137ea0 legacy=0x7fcf58138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient: _finish_hunting 0 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 monclient: found mon.noname-c 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:43.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:45:43.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.869+0000 7fcf5da41640 1 -- 172.21.15.86:0/132629339 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5809c430 con 0x7fcf580a54c0 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5da41640 1 -- 172.21.15.86:0/132629339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 msgr2=0x7fcf580a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5da41640 1 --2- 172.21.15.86:0/132629339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf580a58d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf580a6220 tx=0x7fcf4801c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f5362d339af7c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 1 -- 172.21.15.86:0/132629339 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf480b0070 con 0x7fcf580a54c0 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: dump: 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.880 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: _finish_auth 0 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: _check_auth_tickets 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 1 -- 172.21.15.86:0/132629339 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf480ab600 con 0x7fcf580a54c0 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 5 monclient: authenticate success, global_id 204166 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 4 set_mon_vals no callback set 2024-09-17T15:45:43.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 1 -- 172.21.15.86:0/132629339 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf480ab9d0 con 0x7fcf580a54c0 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf5d240640 10 monclient: dump: 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 10 monclient: shutdown 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- 172.21.15.86:0/132629339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 msgr2=0x7fcf580a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 --2- 172.21.15.86:0/132629339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf580a58d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcf580a6220 tx=0x7fcf4801c3d0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- 172.21.15.86:0/132629339 shutdown_connections 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 --2- 172.21.15.86:0/132629339 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf580a58d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 --2- 172.21.15.86:0/132629339 >> v2:172.21.15.86:3300/0 conn(0x7fcf580a50e0 0x7fcf5809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- 172.21.15.86:0/132629339 >> 172.21.15.86:0/132629339 conn(0x7fcf580aaae0 msgr2=0x7fcf580aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- 172.21.15.86:0/132629339 shutdown_connections 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- 172.21.15.86:0/132629339 wait complete. 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 10 monclient: build_initial_monmap 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 10 monclient: monmap: 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:43.883 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 Processor -- start 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 1 -- start start 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.873+0000 7fcf604a6640 10 monclient: init 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:45:43.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-a con 0x7fcf580a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf581d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-b con 0x7fcf580a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 0x7fcf5813c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): picked mon.noname-c con 0x7fcf58137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58142cb0 con 0x7fcf580a50e0 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58142b30 con 0x7fcf580a54c0 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf58142e30 con 0x7fcf58137ea0 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient(hunting): authenticate will time out at 2076.844214s 2024-09-17T15:45:43.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51300/0 (socket says 172.21.15.86:51300) 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 -- 172.21.15.86:0/1871652398 learned_addr learned my addr 172.21.15.86:0/1871652398 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 10 monclient(hunting): get_auth_request con 0x7fcf580a50e0 auth_method 0 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 0x7fcf5813c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): get_auth_request con 0x7fcf58137ea0 auth_method 0 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 msgr2=0x7fcf581d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 msgr2=0x7fcf581d0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:43.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5e242640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5da41640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf581d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient(hunting): handle_auth_done global_id 164600 payload 824 2024-09-17T15:45:43.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 msgr2=0x7fcf581d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf581d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 msgr2=0x7fcf581d0170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient: _finish_hunting 0 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 monclient: found mon.noname-c 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 -- 172.21.15.86:0/1871652398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5813ddb0 con 0x7fcf58137ea0 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5da41640 10 monclient: get_auth_request con 0x7fcf580a54c0 auth_method 0 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5da41640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf581d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 msgr2=0x7fcf5813c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ea43640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 0x7fcf5813c2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c410 tx=0x7fcf5400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:43.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54017070 con 0x7fcf58137ea0 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: dump: 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: _finish_auth 0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: _check_auth_tickets 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf54005230 con 0x7fcf58137ea0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 5 monclient: authenticate success, global_id 164600 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _renew_subs 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5813e910 con 0x7fcf58137ea0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf540117a0 con 0x7fcf58137ea0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf4effd640 10 monclient: dump: 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf5ca3f640 4 set_mon_vals no callback set 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _renew_subs 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.877+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf58121260 con 0x7fcf58137ea0 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf604a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf604a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf604a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5809ce10 con 0x7fcf58137ea0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcf5401a070 con 0x7fcf58137ea0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf4effd640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 0x7fcf3c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(665..665 src has 438..665) ==== 4771+0+0 (secure 0 0 0) 0x7fcf5404f250 con 0x7fcf58137ea0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf5e242640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 0x7fcf3c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf5e242640 10 monclient: get_auth_request con 0x7fcf3c03bb60 auth_method 0 2024-09-17T15:45:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.881+0000 7fcf5e242640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 0x7fcf3c03e020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcf500090f0 tx=0x7fcf50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:43.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.885+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf540102e0 con 0x7fcf58137ea0 2024-09-17T15:45:43.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.885+0000 7fcf4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:43.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:43.885+0000 7fcf4effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:44.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcf580a9f40 con 0x7fcf58137ea0 2024-09-17T15:45:44.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf4effd640 1 -- 172.21.15.86:0/1871652398 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fcf54004ab0 con 0x7fcf58137ea0 2024-09-17T15:45:44.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:44.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf4effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:44.118 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 msgr2=0x7fcf3c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 0x7fcf3c03e020 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcf500090f0 tx=0x7fcf50009040 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 10 monclient: shutdown 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 msgr2=0x7fcf5813c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 0x7fcf5813c2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c410 tx=0x7fcf5400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 shutdown_connections 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcf3c03bb60 0x7fcf3c03e020 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf58137ea0 0x7fcf5813c2f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf580a54c0 0x7fcf581d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 --2- 172.21.15.86:0/1871652398 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf580a50e0 0x7fcf581d0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 >> 172.21.15.86:0/1871652398 conn(0x7fcf580aaae0 msgr2=0x7fcf5809e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 shutdown_connections 2024-09-17T15:45:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:44.121+0000 7fcf604a6640 1 -- 172.21.15.86:0/1871652398 wait complete. 2024-09-17T15:45:44.733 INFO:tasks.rados.rados.0.smithi203.stdout:2999: done (0 left) 2024-09-17T15:45:44.734 INFO:tasks.rados.rados.0.smithi203.stdout:3000: read oid 35 snap -1 2024-09-17T15:45:44.734 INFO:tasks.rados.rados.0.smithi203.stdout:3000: expect deleted 2024-09-17T15:45:44.734 INFO:tasks.rados.rados.0.smithi203.stdout:3001: snap_remove snap 318 2024-09-17T15:45:45.736 INFO:tasks.rados.rados.0.smithi203.stdout:3001: done (1 left) 2024-09-17T15:45:45.737 INFO:tasks.rados.rados.0.smithi203.stdout:3002: rollback oid 7 current snap is 326 2024-09-17T15:45:45.737 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 297 2024-09-17T15:45:45.737 INFO:tasks.rados.rados.0.smithi203.stdout:3000: done (1 left) 2024-09-17T15:45:45.737 INFO:tasks.rados.rados.0.smithi203.stdout:3003: snap_remove snap 319 2024-09-17T15:45:46.739 INFO:tasks.rados.rados.0.smithi203.stdout:3003: done (1 left) 2024-09-17T15:45:46.739 INFO:tasks.rados.rados.0.smithi203.stdout:3004: write oid 35 current snap is 326 2024-09-17T15:45:46.739 INFO:tasks.rados.rados.0.smithi203.stdout:3004: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-17T15:45:46.755 INFO:tasks.rados.rados.0.smithi203.stdout:3004: writing smithi20328086-35 from 480964 to 899626 tid 1 2024-09-17T15:45:46.773 INFO:tasks.rados.rados.0.smithi203.stdout:3004: writing smithi20328086-35 from 1560704 to 1971066 tid 2 2024-09-17T15:45:46.791 INFO:tasks.rados.rados.0.smithi203.stdout:3004: writing smithi20328086-35 from 2523106 to 3142440 tid 3 2024-09-17T15:45:46.791 INFO:tasks.rados.rados.0.smithi203.stdout:3005: read oid 30 snap -1 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3005: expect deleted 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3006: read oid 6 snap 325 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3006: expect (ObjNum 1051 snap 324 seq_num 1051) 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3007: read oid 11 snap -1 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3007: expect deleted 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3008: rollback oid 50 current snap is 326 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 313 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3002: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:3002: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:45:46.792 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1184 (ObjNum 816 snap 254 seq_num 816) dirty exists 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3004: finishing write tid 1 to smithi20328086-35 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3004: finishing write tid 2 to smithi20328086-35 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3002: done (5 left) 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3009: read oid 16 snap -1 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3009: expect (ObjNum 1034 snap 319 seq_num 1034) 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3010: read oid 25 snap 297 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3010: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:45:46.793 INFO:tasks.rados.rados.0.smithi203.stdout:3011: snap_create 2024-09-17T15:45:46.801 INFO:tasks.rados.rados.0.smithi203.stdout:3006: expect (ObjNum 1051 snap 324 seq_num 1051) 2024-09-17T15:45:46.878 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2305172 2024-09-17T15:45:46.960 INFO:tasks.rados.rados.0.smithi203.stdout:3009: expect (ObjNum 1034 snap 319 seq_num 1034) 2024-09-17T15:45:46.960 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:47.087 INFO:tasks.rados.rados.0.smithi203.stdout:3008: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3008: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1206 (ObjNum 129 snap 0 seq_num 3606746240) dirty dne 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: finishing write tid 3 to smithi20328086-35 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: finishing write tid 4 to smithi20328086-35 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: finishing write tid 5 to smithi20328086-35 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: oid 35 updating version 0 to 1120 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: oid 35 updating version 1120 to 1121 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: oid 35 updating version 1121 to 1122 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: oid 35 updating version 1122 to 1123 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1123 (ObjNum 1052 snap 326 seq_num 1052) dirty exists 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3004: left oid 35 (ObjNum 1052 snap 326 seq_num 1052) 2024-09-17T15:45:47.088 INFO:tasks.rados.rados.0.smithi203.stdout:3010: expect (ObjNum 816 snap 254 seq_num 816) 2024-09-17T15:45:47.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:47.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: get_monmap_and_config 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: build_initial_monmap 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: monmap: 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:47.273192+0000 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:47.273192+0000 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:47.267 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 Processor -- start 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 -- start start 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: init 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f82e80689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f82e8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f82e8135ae0 0x7f82e8135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f82e8135ae0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f82e8074500 con 0x7f82e80689d0 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f82e809c8b0 con 0x7f82e8135ae0 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f82e809cb90 con 0x7f82e8068db0 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82ed1e9640 10 monclient(hunting): authenticate will time out at 2080.239097s 2024-09-17T15:45:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58232/0 (socket says 172.21.15.86:58232) 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 -- 172.21.15.86:0/3734901035 learned_addr learned my addr 172.21.15.86:0/3734901035 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 10 monclient(hunting): get_auth_request con 0x7f82e80689d0 auth_method 0 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): get_auth_request con 0x7f82e8068db0 auth_method 0 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 1 -- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 msgr2=0x7f82e806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 1 -- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 msgr2=0x7f82e806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e759e640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.203:3300/0 conn(0x7f82e8135ae0 0x7f82e8135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e759e640 10 monclient(hunting): get_auth_request con 0x7f82e8135ae0 auth_method 0 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient(hunting): handle_auth_done global_id 164603 payload 274 2024-09-17T15:45:47.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 -- 172.21.15.86:0/3734901035 >> v2:172.21.15.203:3300/0 conn(0x7f82e8135ae0 msgr2=0x7f82e8135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.203:3300/0 conn(0x7f82e8135ae0 0x7f82e8135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 -- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 msgr2=0x7f82e806f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient: _finish_hunting 0 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 monclient: found mon.noname-b 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.268+0000 7f82e659c640 1 -- 172.21.15.86:0/3734901035 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e809ce70 con 0x7f82e8068db0 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.272+0000 7f82e659c640 1 -- 172.21.15.86:0/3734901035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 msgr2=0x7f82e80691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.272+0000 7f82e659c640 1 --2- 172.21.15.86:0/3734901035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82e809cac0 tx=0x7f82dc01cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e0ded6670a3da6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.272+0000 7f82e5d9b640 1 -- 172.21.15.86:0/3734901035 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0b0070 con 0x7f82e8068db0 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: dump: 2024-09-17T15:45:47.280 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.281 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 1 -- 172.21.15.86:0/3734901035 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc0ab600 con 0x7f82e8068db0 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 5 monclient: authenticate success, global_id 164603 2024-09-17T15:45:47.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 4 set_mon_vals no callback set 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 1 -- 172.21.15.86:0/3734901035 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0ab9d0 con 0x7f82e8068db0 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e5d9b640 10 monclient: dump: 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:47.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: shutdown 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- 172.21.15.86:0/3734901035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 msgr2=0x7f82e80691d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/3734901035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82e809cac0 tx=0x7f82dc01cd80 comp rx=0 tx=0).stop 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- 172.21.15.86:0/3734901035 shutdown_connections 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.203:3300/0 conn(0x7f82e8135ae0 0x7f82e8135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/3734901035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e80691d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/3734901035 >> v2:172.21.15.86:3300/0 conn(0x7f82e80689d0 0x7f82e806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- 172.21.15.86:0/3734901035 >> 172.21.15.86:0/3734901035 conn(0x7f82e80a6440 msgr2=0x7f82e80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- 172.21.15.86:0/3734901035 shutdown_connections 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- 172.21.15.86:0/3734901035 wait complete. 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: build_initial_monmap 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: monmap: 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:47.285 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 Processor -- start 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- start start 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: init 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-a con 0x7f82e80689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-c con 0x7f82e8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e8135ae0 0x7f82e81332c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): picked mon.noname-b con 0x7f82e8135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f82e809be10 con 0x7f82e80689d0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f82e809c2d0 con 0x7f82e8135ae0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f82e813f820 con 0x7f82e8068db0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82ed1e9640 10 monclient(hunting): authenticate will time out at 2080.246161s 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:58246/0 (socket says 172.21.15.86:58246) 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51304/0 (socket says 172.21.15.86:51304) 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 -- 172.21.15.86:0/1778230104 learned_addr learned my addr 172.21.15.86:0/1778230104 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): get_auth_request con 0x7f82e8068db0 auth_method 0 2024-09-17T15:45:47.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 10 monclient(hunting): get_auth_request con 0x7f82e80689d0 auth_method 0 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 msgr2=0x7f82e81cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 msgr2=0x7f82e81cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e759e640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e8135ae0 0x7f82e81332c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e759e640 10 monclient(hunting): get_auth_request con 0x7f82e8135ae0 auth_method 0 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient(hunting): handle_auth_done global_id 164606 payload 824 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e8135ae0 msgr2=0x7f82e81332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e8135ae0 0x7f82e81332c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 msgr2=0x7f82e81cdd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient: _finish_hunting 0 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 monclient: found mon.noname-c 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:47.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.276+0000 7f82e659c640 1 -- 172.21.15.86:0/1778230104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e811dea0 con 0x7f82e8068db0 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82e659c640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 msgr2=0x7f82e8132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82e659c640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82dc01a9d0 tx=0x7f82dc0190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0b0040 con 0x7f82e8068db0 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: dump: 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.290 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 5 monclient: authenticate success, global_id 164606 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _renew_subs 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc0aa5b0 con 0x7f82e8068db0 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e809d460 con 0x7f82e8068db0 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82dc0aa9a0 con 0x7f82e8068db0 2024-09-17T15:45:47.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82e559a640 4 set_mon_vals no callback set 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 10 monclient: dump: 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:47.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _renew_subs 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e811e740 con 0x7f82e8068db0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e80691d0 con 0x7f82e8068db0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f82dc0aab40 con 0x7f82e8068db0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 0x7f82d403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(668..668 src has 438..668) ==== 4815+0+0 (secure 0 0 0) 0x7f82dc0e9bf0 con 0x7f82e8068db0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 0x7f82d403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:47.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.280+0000 7f82e6d9d640 10 monclient: get_auth_request con 0x7f82d403bb60 auth_method 0 2024-09-17T15:45:47.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.284+0000 7f82e6d9d640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 0x7f82d403e020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f82d8006aa0 tx=0x7f82d8006a30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:47.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.284+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f82e811dea0 con 0x7f82e8068db0 2024-09-17T15:45:47.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.284+0000 7f82cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:47.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.284+0000 7f82cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:47.423 INFO:tasks.rados.rados.0.smithi203.stdout:3004: done (7 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3005: done (6 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3006: done (5 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3007: done (4 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3008: done (3 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3009: done (2 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3010: done (1 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3011: done (0 left) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3012: copy_from oid 37 from oid 10 current snap is 327 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3013: read oid 2 snap -1 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3013: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:47.424 INFO:tasks.rados.rados.0.smithi203.stdout:3014: snap_remove snap 326 2024-09-17T15:45:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82ed1e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82ed1e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82ed1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f82e8133810 con 0x7f82e8068db0 2024-09-17T15:45:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82cf7fe640 1 -- 172.21.15.86:0/1778230104 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f82dc0160f0 con 0x7f82e8068db0 2024-09-17T15:45:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.520+0000 7f82cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:47.521 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 msgr2=0x7f82d403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 0x7f82d403e020 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f82d8006aa0 tx=0x7f82d8006a30 comp rx=0 tx=0).stop 2024-09-17T15:45:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 10 monclient: shutdown 2024-09-17T15:45:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 msgr2=0x7f82e8132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82dc01a9d0 tx=0x7f82dc0190c0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 shutdown_connections 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f82d403bb60 0x7f82d403e020 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f82e8135ae0 0x7f82e81332c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82e8068db0 0x7f82e8132d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 --2- 172.21.15.86:0/1778230104 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82e80689d0 0x7f82e81cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 >> 172.21.15.86:0/1778230104 conn(0x7f82e80a6440 msgr2=0x7f82e80a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 shutdown_connections 2024-09-17T15:45:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:47.524+0000 7f82ed1e9640 1 -- 172.21.15.86:0/1778230104 wait complete. 2024-09-17T15:45:48.829 INFO:tasks.rados.rados.0.smithi203.stdout:3014: done (2 left) 2024-09-17T15:45:48.829 INFO:tasks.rados.rados.0.smithi203.stdout:3015: rollback oid 44 current snap is 327 2024-09-17T15:45:48.829 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 300 2024-09-17T15:45:48.829 INFO:tasks.rados.rados.0.smithi203.stdout:3013: expect (ObjNum 968 snap 294 seq_num 968) 2024-09-17T15:45:48.936 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3012: finishing copy_from to smithi20328086-37 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3012: got expected ENOENT (src dne) 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3012: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3015: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3015: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1492 (ObjNum 977 snap 299 seq_num 977) dirty exists 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3012: done (2 left) 2024-09-17T15:45:48.985 INFO:tasks.rados.rados.0.smithi203.stdout:3013: done (1 left) 2024-09-17T15:45:48.986 INFO:tasks.rados.rados.0.smithi203.stdout:3015: done (0 left) 2024-09-17T15:45:48.986 INFO:tasks.rados.rados.0.smithi203.stdout:3016: write oid 7 current snap is 327 2024-09-17T15:45:48.986 INFO:tasks.rados.rados.0.smithi203.stdout:3016: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-17T15:45:49.001 INFO:tasks.rados.rados.0.smithi203.stdout:3016: writing smithi20328086-7 from 497771 to 991682 tid 1 2024-09-17T15:45:49.020 INFO:tasks.rados.rados.0.smithi203.stdout:3016: writing smithi20328086-7 from 1619186 to 2173206 tid 2 2024-09-17T15:45:49.039 INFO:tasks.rados.rados.0.smithi203.stdout:3016: writing smithi20328086-7 from 2950529 to 3381135 tid 3 2024-09-17T15:45:49.039 INFO:tasks.rados.rados.0.smithi203.stdout:3017: write oid 21 current snap is 327 2024-09-17T15:45:49.039 INFO:tasks.rados.rados.0.smithi203.stdout:3017: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-17T15:45:49.051 INFO:tasks.rados.rados.0.smithi203.stdout:3017: writing smithi20328086-21 from 514578 to 1083738 tid 1 2024-09-17T15:45:49.062 INFO:tasks.rados.rados.0.smithi203.stdout:3017: writing smithi20328086-21 from 1677668 to 2091699 tid 2 2024-09-17T15:45:49.071 INFO:tasks.rados.rados.0.smithi203.stdout:3017: writing smithi20328086-21 from 2577952 to 3219830 tid 3 2024-09-17T15:45:49.072 INFO:tasks.rados.rados.0.smithi203.stdout:3018: write (excl) oid 41 current snap is 327 2024-09-17T15:45:49.072 INFO:tasks.rados.rados.0.smithi203.stdout:3018: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-17T15:45:49.081 INFO:tasks.rados.rados.0.smithi203.stdout:3018: writing smithi20328086-41 from 531385 to 1175794 tid 1 2024-09-17T15:45:49.088 INFO:tasks.rados.rados.0.smithi203.stdout:3018: writing smithi20328086-41 from 1736150 to 2293839 tid 2 2024-09-17T15:45:49.096 INFO:tasks.rados.rados.0.smithi203.stdout:3018: writing smithi20328086-41 from 3005375 to 3574878 tid 3 2024-09-17T15:45:49.097 INFO:tasks.rados.rados.0.smithi203.stdout:3019: snap_remove snap 297 2024-09-17T15:45:49.835 INFO:tasks.rados.rados.0.smithi203.stdout:3016: finishing write tid 1 to smithi20328086-7 2024-09-17T15:45:49.835 INFO:tasks.rados.rados.0.smithi203.stdout:3016: finishing write tid 2 to smithi20328086-7 2024-09-17T15:45:49.835 INFO:tasks.rados.rados.0.smithi203.stdout:3016: finishing write tid 3 to smithi20328086-7 2024-09-17T15:45:49.835 INFO:tasks.rados.rados.0.smithi203.stdout:3016: finishing write tid 4 to smithi20328086-7 2024-09-17T15:45:49.835 INFO:tasks.rados.rados.0.smithi203.stdout:3016: finishing write tid 5 to smithi20328086-7 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3016: oid 7 updating version 0 to 1186 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3016: oid 7 updating version 1186 to 1187 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3016: oid 7 updating version 1187 to 1188 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3016: oid 7 version 1188 is already newer than 1185 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1188 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3016: left oid 7 (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: finishing write tid 1 to smithi20328086-21 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: finishing write tid 2 to smithi20328086-21 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: finishing write tid 3 to smithi20328086-21 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: finishing write tid 4 to smithi20328086-21 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: finishing write tid 5 to smithi20328086-21 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: oid 21 updating version 0 to 1578 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: oid 21 version 1578 is already newer than 1576 2024-09-17T15:45:49.836 INFO:tasks.rados.rados.0.smithi203.stdout:3017: oid 21 version 1578 is already newer than 1577 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3017: oid 21 updating version 1578 to 1579 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1579 (ObjNum 1054 snap 327 seq_num 1054) dirty exists 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3017: left oid 21 (ObjNum 1054 snap 327 seq_num 1054) 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: finishing write tid 1 to smithi20328086-41 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: finishing write tid 2 to smithi20328086-41 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: finishing write tid 3 to smithi20328086-41 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: finishing write tid 4 to smithi20328086-41 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: finishing write tid 5 to smithi20328086-41 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: oid 41 updating version 0 to 654 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: oid 41 version 654 is already newer than 652 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: oid 41 version 654 is already newer than 653 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: oid 41 updating version 654 to 655 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 655 (ObjNum 1055 snap 327 seq_num 1055) dirty exists 2024-09-17T15:45:49.837 INFO:tasks.rados.rados.0.smithi203.stdout:3018: left oid 41 (ObjNum 1055 snap 327 seq_num 1055) 2024-09-17T15:45:49.838 INFO:tasks.rados.rados.0.smithi203.stdout:3016: done (3 left) 2024-09-17T15:45:49.838 INFO:tasks.rados.rados.0.smithi203.stdout:3017: done (2 left) 2024-09-17T15:45:49.838 INFO:tasks.rados.rados.0.smithi203.stdout:3018: done (1 left) 2024-09-17T15:45:49.838 INFO:tasks.rados.rados.0.smithi203.stdout:3019: done (0 left) 2024-09-17T15:45:49.838 INFO:tasks.rados.rados.0.smithi203.stdout:3020: snap_remove snap 309 2024-09-17T15:45:50.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:50.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: get_monmap_and_config 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: build_initial_monmap 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: monmap: 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:50.677456+0000 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:50.677456+0000 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:50.672 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:50.672 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 Processor -- start 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 -- start start 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: init 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-17T15:45:50.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 0x7f2e8009ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-b con 0x7f2e800a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e800a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e8013d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8009d350 con 0x7f2e800a54c0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e800a6530 con 0x7f2e8013d2f0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e8009c150 con 0x7f2e800a50e0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e861ff640 10 monclient(hunting): authenticate will time out at 2083.643305s 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 0x7f2e8009ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2e800a54c0 0x7f2e800a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58356/0 (socket says 172.21.15.86:58356) 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7effd640 1 -- 172.21.15.86:0/1565869877 learned_addr learned my addr 172.21.15.86:0/1565869877 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): get_auth_request con 0x7f2e800a50e0 auth_method 0 2024-09-17T15:45:50.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient(hunting): handle_auth_done global_id 164609 payload 274 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 1 -- 172.21.15.86:0/1565869877 >> v1:172.21.15.203:6789/0 conn(0x7f2e8013d2f0 legacy=0x7f2e8013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 1 -- 172.21.15.86:0/1565869877 >> v1:172.21.15.86:6789/0 conn(0x7f2e800a54c0 legacy=0x7f2e800a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 1 monclient: found mon.noname-b 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.672+0000 7f2e7f7fe640 1 -- 172.21.15.86:0/1565869877 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8009c430 con 0x7f2e800a50e0 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7f7fe640 1 -- 172.21.15.86:0/1565869877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 msgr2=0x7f2e8009ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7f7fe640 1 --2- 172.21.15.86:0/1565869877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 0x7f2e8009ce10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e8009c770 tx=0x7f2e7001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1484195d872a8439 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 1 -- 172.21.15.86:0/1565869877 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700b0070 con 0x7f2e800a50e0 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: dump: 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:50.678 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 1 -- 172.21.15.86:0/1565869877 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e700ab600 con 0x7f2e800a50e0 2024-09-17T15:45:50.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 1 -- 172.21.15.86:0/1565869877 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700ab9d0 con 0x7f2e800a50e0 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7e7fc640 10 monclient: dump: 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 5 monclient: authenticate success, global_id 164609 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 4 set_mon_vals no callback set 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: shutdown 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- 172.21.15.86:0/1565869877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 msgr2=0x7f2e8009ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 --2- 172.21.15.86:0/1565869877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 0x7f2e8009ce10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e8009c770 tx=0x7f2e7001c360 comp rx=0 tx=0).stop 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- 172.21.15.86:0/1565869877 shutdown_connections 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 --2- 172.21.15.86:0/1565869877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a50e0 0x7f2e8009ce10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- 172.21.15.86:0/1565869877 >> 172.21.15.86:0/1565869877 conn(0x7f2e800aaae0 msgr2=0x7f2e800aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- 172.21.15.86:0/1565869877 shutdown_connections 2024-09-17T15:45:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- 172.21.15.86:0/1565869877 wait complete. 2024-09-17T15:45:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: build_initial_monmap 2024-09-17T15:45:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: monmap: 2024-09-17T15:45:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:50.685 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 Processor -- start 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- start start 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: init 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:45:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e800a50e0 0x7f2e80138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-b con 0x7f2e800a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-c con 0x7f2e800a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80143af0 con 0x7f2e8013d2f0 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e80143970 con 0x7f2e800a50e0 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2e80143c70 con 0x7f2e800a54c0 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e861ff640 10 monclient(hunting): authenticate will time out at 2083.650431s 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51686/0 (socket says 172.21.15.86:51686) 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 -- 172.21.15.86:0/794630862 learned_addr learned my addr 172.21.15.86:0/794630862 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): get_auth_request con 0x7f2e800a54c0 auth_method 0 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request con 0x7f2e8013d2f0 auth_method 0 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:50.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 msgr2=0x7f2e800a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 msgr2=0x7f2e800a20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7ffff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient(hunting): handle_auth_done global_id 164612 payload 824 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7f7fe640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e800a50e0 0x7f2e80138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e800a50e0 msgr2=0x7f2e80138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e800a50e0 0x7f2e80138000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 msgr2=0x7f2e800a20c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient: _finish_hunting 0 2024-09-17T15:45:50.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 monclient: found mon.noname-c 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.680+0000 7f2e7effd640 1 -- 172.21.15.86:0/794630862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e80140700 con 0x7f2e800a54c0 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e7effd640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 msgr2=0x7f2e800a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e7effd640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800c470 tx=0x7f2e6800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e68017070 con 0x7f2e800a54c0 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: dump: 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: _finish_auth 0 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: _check_auth_tickets 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e680070d0 con 0x7f2e800a54c0 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e680074c0 con 0x7f2e800a54c0 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 5 monclient: authenticate success, global_id 164612 2024-09-17T15:45:50.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _renew_subs 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e800a18d0 con 0x7f2e800a54c0 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e7dffb640 4 set_mon_vals no callback set 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 10 monclient: dump: 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:50.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _renew_subs 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e800a3eb0 con 0x7f2e800a54c0 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8009ce10 con 0x7f2e800a54c0 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2e6801a070 con 0x7f2e800a54c0 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 0x7f2e6003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:50.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(671..671 src has 438..671) ==== 4799+0+0 (secure 0 0 0) 0x7f2e6804f9b0 con 0x7f2e800a54c0 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e7f7fe640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 0x7f2e6003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.684+0000 7f2e7f7fe640 10 monclient: get_auth_request con 0x7f2e6003bb60 auth_method 0 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.688+0000 7f2e7f7fe640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 0x7f2e6003e020 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2e8006f9b0 tx=0x7f2e70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.688+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2e68021020 con 0x7f2e800a54c0 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.688+0000 7f2e77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:50.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.688+0000 7f2e77fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:50.844 INFO:tasks.rados.rados.0.smithi203.stdout:3020: done (0 left) 2024-09-17T15:45:50.844 INFO:tasks.rados.rados.0.smithi203.stdout:3021: copy_from oid 30 from oid 50 current snap is 327 2024-09-17T15:45:50.845 INFO:tasks.rados.rados.0.smithi203.stdout:3022: write (excl) oid 22 current snap is 327 2024-09-17T15:45:50.845 INFO:tasks.rados.rados.0.smithi203.stdout:3022: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-17T15:45:50.855 INFO:tasks.rados.rados.0.smithi203.stdout:3022: writing smithi20328086-22 from 548192 to 1267850 tid 1 2024-09-17T15:45:50.863 INFO:tasks.rados.rados.0.smithi203.stdout:3022: writing smithi20328086-22 from 1678279 to 2095979 tid 2 2024-09-17T15:45:50.873 INFO:tasks.rados.rados.0.smithi203.stdout:3022: writing smithi20328086-22 from 2516445 to 3297220 tid 3 2024-09-17T15:45:50.873 INFO:tasks.rados.rados.0.smithi203.stdout:3023: write oid 11 current snap is 327 2024-09-17T15:45:50.873 INFO:tasks.rados.rados.0.smithi203.stdout:3021: finishing copy_from to smithi20328086-30 2024-09-17T15:45:50.874 INFO:tasks.rados.rados.0.smithi203.stdout:3021: got expected ENOENT (src dne) 2024-09-17T15:45:50.874 INFO:tasks.rados.rados.0.smithi203.stdout:3023: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-17T15:45:50.882 INFO:tasks.rados.rados.0.smithi203.stdout:3023: writing smithi20328086-11 from 564999 to 1076259 tid 1 2024-09-17T15:45:50.892 INFO:tasks.rados.rados.0.smithi203.stdout:3023: writing smithi20328086-11 from 1853114 to 2414472 tid 2 2024-09-17T15:45:50.902 INFO:tasks.rados.rados.0.smithi203.stdout:3023: writing smithi20328086-11 from 3060221 to 3652268 tid 3 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3021: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3024: read oid 31 snap 321 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3024: expect deleted 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3021: done (3 left) 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3025: write (excl) oid 48 current snap is 327 2024-09-17T15:45:50.903 INFO:tasks.rados.rados.0.smithi203.stdout:3025: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-17T15:45:50.911 INFO:tasks.rados.rados.0.smithi203.stdout:3025: writing smithi20328086-48 from 581806 to 1168315 tid 1 2024-09-17T15:45:50.921 INFO:tasks.rados.rados.0.smithi203.stdout:3025: writing smithi20328086-48 from 1911596 to 2332965 tid 2 2024-09-17T15:45:50.930 INFO:tasks.rados.rados.0.smithi203.stdout:3025: writing smithi20328086-48 from 3087644 to 3490963 tid 3 2024-09-17T15:45:50.930 INFO:tasks.rados.rados.0.smithi203.stdout:3026: snap_remove snap 320 2024-09-17T15:45:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2e800a4b80 con 0x7f2e800a54c0 2024-09-17T15:45:50.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e77fff640 1 -- 172.21.15.86:0/794630862 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2e800a18d0 con 0x7f2e800a54c0 2024-09-17T15:45:50.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:50.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e77fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:50.947 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":28,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:50.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 msgr2=0x7f2e6003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:50.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 0x7f2e6003e020 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2e8006f9b0 tx=0x7f2e70008040 comp rx=0 tx=0).stop 2024-09-17T15:45:50.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 10 monclient: shutdown 2024-09-17T15:45:50.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:50.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 msgr2=0x7f2e800a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2e6800c470 tx=0x7f2e6800c930 comp rx=0 tx=0).stop 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 shutdown_connections 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2e6003bb60 0x7f2e6003e020 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e8013d2f0 0x7f2e800a20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e800a54c0 0x7f2e800a1b80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 --2- 172.21.15.86:0/794630862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e800a50e0 0x7f2e80138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.948+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 >> 172.21.15.86:0/794630862 conn(0x7f2e800aaae0 msgr2=0x7f2e800a0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.952+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 shutdown_connections 2024-09-17T15:45:50.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:50.952+0000 7f2e861ff640 1 -- 172.21.15.86:0/794630862 wait complete. 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3026: done (4 left) 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3027: snap_create 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: finishing write tid 1 to smithi20328086-22 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: finishing write tid 2 to smithi20328086-22 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: finishing write tid 3 to smithi20328086-22 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: finishing write tid 4 to smithi20328086-22 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: finishing write tid 5 to smithi20328086-22 2024-09-17T15:45:51.846 INFO:tasks.rados.rados.0.smithi203.stdout:3022: oid 22 updating version 0 to 1299 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3022: oid 22 updating version 1299 to 1302 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3022: oid 22 version 1302 is already newer than 1300 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3022: oid 22 version 1302 is already newer than 1301 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1302 (ObjNum 1056 snap 327 seq_num 1056) dirty exists 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3022: left oid 22 (ObjNum 1056 snap 327 seq_num 1056) 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: finishing write tid 1 to smithi20328086-11 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: finishing write tid 2 to smithi20328086-11 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: finishing write tid 3 to smithi20328086-11 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: finishing write tid 4 to smithi20328086-11 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: finishing write tid 5 to smithi20328086-11 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: oid 11 updating version 0 to 1496 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: oid 11 version 1496 is already newer than 1494 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: oid 11 version 1496 is already newer than 1495 2024-09-17T15:45:51.847 INFO:tasks.rados.rados.0.smithi203.stdout:3023: oid 11 version 1496 is already newer than 1493 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1496 (ObjNum 1057 snap 327 seq_num 1057) dirty exists 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3023: left oid 11 (ObjNum 1057 snap 327 seq_num 1057) 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: finishing write tid 1 to smithi20328086-48 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: finishing write tid 2 to smithi20328086-48 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: finishing write tid 3 to smithi20328086-48 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: finishing write tid 4 to smithi20328086-48 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: finishing write tid 5 to smithi20328086-48 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: oid 48 updating version 0 to 1530 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: oid 48 updating version 1530 to 1531 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: oid 48 version 1531 is already newer than 1528 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: oid 48 version 1531 is already newer than 1529 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1531 (ObjNum 1058 snap 327 seq_num 1058) dirty exists 2024-09-17T15:45:51.848 INFO:tasks.rados.rados.0.smithi203.stdout:3025: left oid 48 (ObjNum 1058 snap 327 seq_num 1058) 2024-09-17T15:45:52.424 INFO:tasks.rados.rados.0.smithi203.stdout:3022: done (4 left) 2024-09-17T15:45:52.424 INFO:tasks.rados.rados.0.smithi203.stdout:3023: done (3 left) 2024-09-17T15:45:52.424 INFO:tasks.rados.rados.0.smithi203.stdout:3024: done (2 left) 2024-09-17T15:45:52.425 INFO:tasks.rados.rados.0.smithi203.stdout:3025: done (1 left) 2024-09-17T15:45:52.425 INFO:tasks.rados.rados.0.smithi203.stdout:3027: done (0 left) 2024-09-17T15:45:52.425 INFO:tasks.rados.rados.0.smithi203.stdout:3028: snap_remove snap 300 2024-09-17T15:45:53.426 INFO:tasks.rados.rados.0.smithi203.stdout:3028: done (0 left) 2024-09-17T15:45:53.426 INFO:tasks.rados.rados.0.smithi203.stdout:3029: read oid 38 snap -1 2024-09-17T15:45:53.426 INFO:tasks.rados.rados.0.smithi203.stdout:3029: expect deleted 2024-09-17T15:45:53.426 INFO:tasks.rados.rados.0.smithi203.stdout:3030: snap_remove snap 324 2024-09-17T15:45:53.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:54.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: get_monmap_and_config 2024-09-17T15:45:54.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: build_initial_monmap 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: monmap: 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:54.105072+0000 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:54.105072+0000 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:54.099 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:54.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 Processor -- start 2024-09-17T15:45:54.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 -- start start 2024-09-17T15:45:54.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: init 2024-09-17T15:45:54.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f5e600689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e60068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5e6013aff0 0x7f5e6013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f5e6013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e60074470 con 0x7f5e60068db0 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6009c8b0 con 0x7f5e6013aff0 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6009cb90 con 0x7f5e600689d0 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:54.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e66dc2640 10 monclient(hunting): authenticate will time out at 2087.070607s 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:51702/0 (socket says 172.21.15.86:51702) 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 -- 172.21.15.86:0/3184681551 learned_addr learned my addr 172.21.15.86:0/3184681551 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e5ffff640 1 --1- 172.21.15.86:0/3184681551 >> v1:172.21.15.86:6789/0 conn(0x7f5e60068db0 0x7f5e600691d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58370/0 (socket says 172.21.15.86:58370) 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): get_auth_request con 0x7f5e600689d0 auth_method 0 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient(hunting): handle_auth_done global_id 164615 payload 274 2024-09-17T15:45:54.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e6535f640 1 --2- 172.21.15.86:0/3184681551 >> v2:172.21.15.203:3300/0 conn(0x7f5e6013aff0 0x7f5e6013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 -- 172.21.15.86:0/3184681551 >> v2:172.21.15.203:3300/0 conn(0x7f5e6013aff0 msgr2=0x7f5e6013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/3184681551 >> v2:172.21.15.203:3300/0 conn(0x7f5e6013aff0 0x7f5e6013d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 -- 172.21.15.86:0/3184681551 >> v1:172.21.15.86:6789/0 conn(0x7f5e60068db0 legacy=0x7f5e600691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient: _finish_hunting 0 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 monclient: found mon.noname-b 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.100+0000 7f5e64b5e640 1 -- 172.21.15.86:0/3184681551 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6009ce70 con 0x7f5e600689d0 2024-09-17T15:45:54.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e64b5e640 1 -- 172.21.15.86:0/3184681551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 msgr2=0x7f5e6006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:54.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/3184681551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009e50 tx=0x7f5e5801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d333e9ca9dbd695 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:54.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 1 -- 172.21.15.86:0/3184681551 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6009c2d0 con 0x7f5e600689d0 2024-09-17T15:45:54.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:54.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:45:54.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: dump: 2024-09-17T15:45:54.116 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:54.116 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:54.117 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:54.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 1 -- 172.21.15.86:0/3184681551 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e580a5750 con 0x7f5e600689d0 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 1 -- 172.21.15.86:0/3184681551 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e580a5b20 con 0x7f5e600689d0 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e5f7fe640 10 monclient: dump: 2024-09-17T15:45:54.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.120 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e66dc2640 5 monclient: authenticate success, global_id 164615 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e66dc2640 10 monclient: get_monmap_and_config success 2024-09-17T15:45:54.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.104+0000 7f5e66dc2640 4 set_mon_vals no callback set 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: shutdown 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- 172.21.15.86:0/3184681551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 msgr2=0x7f5e6006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/3184681551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009e50 tx=0x7f5e5801c8f0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- 172.21.15.86:0/3184681551 shutdown_connections 2024-09-17T15:45:54.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/3184681551 >> v2:172.21.15.203:3300/0 conn(0x7f5e6013aff0 0x7f5e6013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/3184681551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e600689d0 0x7f5e6006f840 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- 172.21.15.86:0/3184681551 >> 172.21.15.86:0/3184681551 conn(0x7f5e600a6440 msgr2=0x7f5e600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- 172.21.15.86:0/3184681551 shutdown_connections 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- 172.21.15.86:0/3184681551 wait complete. 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: build_initial_monmap 2024-09-17T15:45:54.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: monmap: 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.124 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:45:54.125 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:45:54.125 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:45:54.125 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 Processor -- start 2024-09-17T15:45:54.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- start start 2024-09-17T15:45:54.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: init 2024-09-17T15:45:54.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:54.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:45:54.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-a con 0x7f5e600689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e60068db0 0x7f5e6011cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f5e60068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 0x7f5e60135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f5e6013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5e601405c0 con 0x7f5e600689d0 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e60140440 con 0x7f5e60068db0 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5e60140740 con 0x7f5e6013aff0 2024-09-17T15:45:54.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e66dc2640 10 monclient(hunting): authenticate will time out at 2087.077852s 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:60640/0 (socket says 172.21.15.86:60640) 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 -- 172.21.15.86:0/4235404133 learned_addr learned my addr 172.21.15.86:0/4235404133 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 0x7f5e60135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 10 monclient(hunting): get_auth_request con 0x7f5e600689d0 auth_method 0 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): get_auth_request con 0x7f5e6013aff0 auth_method 0 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 msgr2=0x7f5e6011ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 msgr2=0x7f5e6011ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:45:54.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e5ffff640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e60068db0 0x7f5e6011cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e5ffff640 10 monclient(hunting): get_auth_request con 0x7f5e60068db0 auth_method 0 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient(hunting): handle_auth_done global_id 164618 payload 824 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e60068db0 msgr2=0x7f5e6011cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e60068db0 0x7f5e6011cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 msgr2=0x7f5e6011ee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient: _finish_hunting 0 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 monclient: found mon.noname-c 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:45:54.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.108+0000 7f5e6535f640 1 -- 172.21.15.86:0/4235404133 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e600a3540 con 0x7f5e6013aff0 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e6535f640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 msgr2=0x7f5e60135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e6535f640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 0x7f5e60135c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c570 tx=0x7f5e5000ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50015070 con 0x7f5e6013aff0 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:45:54.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: dump: 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 5 monclient: authenticate success, global_id 164618 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e50004220 con 0x7f5e6013aff0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _renew_subs 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50011810 con 0x7f5e6013aff0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6013d380 con 0x7f5e6013aff0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5effd640 4 set_mon_vals no callback set 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:45:54.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 10 monclient: dump: 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:45:54.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _renew_subs 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6011fd60 con 0x7f5e6013aff0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6006f840 con 0x7f5e6013aff0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5e500119b0 con 0x7f5e6013aff0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 0x7f5e4c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(675..675 src has 438..675) ==== 4831+0+0 (secure 0 0 0) 0x7f5e5004fe20 con 0x7f5e6013aff0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 0x7f5e4c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.112+0000 7f5e64b5e640 10 monclient: get_auth_request con 0x7f5e4c03bb60 auth_method 0 2024-09-17T15:45:54.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.116+0000 7f5e64b5e640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 0x7f5e4c03e020 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009e50 tx=0x7f5e580a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:45:54.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.116+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5e50003b40 con 0x7f5e6013aff0 2024-09-17T15:45:54.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.116+0000 7f5e5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:45:54.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.116+0000 7f5e5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:45:54.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e66dc2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:45:54.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e66dc2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e66dc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:45:54.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e60120140 con 0x7f5e6013aff0 2024-09-17T15:45:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e5d7fa640 1 -- 172.21.15.86:0/4235404133 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f5e50018360 con 0x7f5e6013aff0 2024-09-17T15:45:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:45:54.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.356+0000 7f5e5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:45:54.356 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":166,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:45:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 msgr2=0x7f5e4c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 0x7f5e4c03e020 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5e58009e50 tx=0x7f5e580a7070 comp rx=0 tx=0).stop 2024-09-17T15:45:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 10 monclient: shutdown 2024-09-17T15:45:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:45:54.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 msgr2=0x7f5e60135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 0x7f5e60135c70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000c570 tx=0x7f5e5000ca30 comp rx=0 tx=0).stop 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 shutdown_connections 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5e4c03bb60 0x7f5e4c03e020 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e6013aff0 0x7f5e60135c70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e60068db0 0x7f5e6011cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 --2- 172.21.15.86:0/4235404133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e600689d0 0x7f5e6011ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 >> 172.21.15.86:0/4235404133 conn(0x7f5e600a6440 msgr2=0x7f5e600a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:45:54.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 shutdown_connections 2024-09-17T15:45:54.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:54.360+0000 7f5e66dc2640 1 -- 172.21.15.86:0/4235404133 wait complete. 2024-09-17T15:45:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3030: done (1 left) 2024-09-17T15:45:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3031: write (excl) oid 20 current snap is 328 2024-09-17T15:45:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3031: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-09-17T15:45:54.446 INFO:tasks.rados.rados.0.smithi203.stdout:3031: writing smithi20328086-20 from 598613 to 1260371 tid 1 2024-09-17T15:45:54.458 INFO:tasks.rados.rados.0.smithi203.stdout:3031: writing smithi20328086-20 from 1970078 to 2535105 tid 2 2024-09-17T15:45:54.469 INFO:tasks.rados.rados.0.smithi203.stdout:3031: writing smithi20328086-20 from 3115067 to 3729658 tid 3 2024-09-17T15:45:54.469 INFO:tasks.rados.rados.0.smithi203.stdout:3032: copy_from oid 4 from oid 1 current snap is 328 2024-09-17T15:45:54.470 INFO:tasks.rados.rados.0.smithi203.stdout:3031: finishing write tid 1 to smithi20328086-20 2024-09-17T15:45:54.470 INFO:tasks.rados.rados.0.smithi203.stdout:3031: finishing write tid 2 to smithi20328086-20 2024-09-17T15:45:54.470 INFO:tasks.rados.rados.0.smithi203.stdout:3029: done (2 left) 2024-09-17T15:45:54.470 INFO:tasks.rados.rados.0.smithi203.stdout:3033: write (excl) oid 15 current snap is 328 2024-09-17T15:45:54.470 INFO:tasks.rados.rados.0.smithi203.stdout:3033: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-17T15:45:54.482 INFO:tasks.rados.rados.0.smithi203.stdout:3033: writing smithi20328086-15 from 615420 to 1352427 tid 1 2024-09-17T15:45:54.491 INFO:tasks.rados.rados.0.smithi203.stdout:3033: writing smithi20328086-15 from 1912207 to 2337245 tid 2 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3033: writing smithi20328086-15 from 3026137 to 3568353 tid 3 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3034: read oid 47 snap -1 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3034: expect (ObjNum 1034 snap 319 seq_num 1034) 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3035: delete oid 3 current snap is 328 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3031: finishing write tid 3 to smithi20328086-20 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3031: finishing write tid 4 to smithi20328086-20 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3031: finishing write tid 5 to smithi20328086-20 2024-09-17T15:45:54.501 INFO:tasks.rados.rados.0.smithi203.stdout:3031: oid 20 updating version 0 to 1327 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:3031: oid 20 version 1327 is already newer than 1326 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:3031: oid 20 updating version 1327 to 1328 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:3031: oid 20 updating version 1328 to 1329 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1329 (ObjNum 1059 snap 328 seq_num 1059) dirty exists 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:3031: left oid 20 (ObjNum 1059 snap 328 seq_num 1059) 2024-09-17T15:45:54.502 INFO:tasks.rados.rados.0.smithi203.stdout:3033: finishing write tid 1 to smithi20328086-15 2024-09-17T15:45:54.506 INFO:tasks.rados.rados.0.smithi203.stdout:3032: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:45:54.507 INFO:tasks.rados.rados.0.smithi203.stdout:3033: finishing write tid 2 to smithi20328086-15 2024-09-17T15:45:54.512 INFO:tasks.rados.rados.0.smithi203.stdout:3033: finishing write tid 3 to smithi20328086-15 2024-09-17T15:45:54.512 INFO:tasks.rados.rados.0.smithi203.stdout:3033: finishing write tid 4 to smithi20328086-15 2024-09-17T15:45:54.513 INFO:tasks.rados.rados.0.smithi203.stdout:3033: finishing write tid 5 to smithi20328086-15 2024-09-17T15:45:54.513 INFO:tasks.rados.rados.0.smithi203.stdout:3033: oid 15 updating version 0 to 1290 2024-09-17T15:45:54.513 INFO:tasks.rados.rados.0.smithi203.stdout:3033: oid 15 updating version 1290 to 1292 2024-09-17T15:45:54.513 INFO:tasks.rados.rados.0.smithi203.stdout:3033: oid 15 version 1292 is already newer than 1291 2024-09-17T15:45:54.513 INFO:tasks.rados.rados.0.smithi203.stdout:3033: oid 15 updating version 1292 to 1293 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1293 (ObjNum 1060 snap 328 seq_num 1060) dirty exists 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3033: left oid 15 (ObjNum 1060 snap 328 seq_num 1060) 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3031: done (4 left) 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3033: done (3 left) 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3035: done (2 left) 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3036: write (excl) oid 44 current snap is 328 2024-09-17T15:45:54.514 INFO:tasks.rados.rados.0.smithi203.stdout:3036: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-17T15:45:54.524 INFO:tasks.rados.rados.0.smithi203.stdout:3036: writing smithi20328086-44 from 649034 to 1136539 tid 1 2024-09-17T15:45:54.529 INFO:tasks.rados.rados.0.smithi203.stdout:3036: writing smithi20328086-44 from 1629171 to 2057878 tid 2 2024-09-17T15:45:54.538 INFO:tasks.rados.rados.0.smithi203.stdout:3036: writing smithi20328086-44 from 2680983 to 3245743 tid 3 2024-09-17T15:45:54.543 INFO:tasks.rados.rados.0.smithi203.stdout:3036: writing smithi20328086-44 from 3798768 to 3849034 tid 4 2024-09-17T15:45:54.543 INFO:tasks.rados.rados.0.smithi203.stdout:3037: snap_remove snap 322 2024-09-17T15:45:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:3037: done (3 left) 2024-09-17T15:45:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:3038: snap_create 2024-09-17T15:45:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:3032: finishing copy_from to smithi20328086-4 2024-09-17T15:45:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1317 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:45:55.439 INFO:tasks.rados.rados.0.smithi203.stdout:3034: expect (ObjNum 1034 snap 319 seq_num 1034) 2024-09-17T15:45:55.466 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1647601 2024-09-17T15:45:55.558 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2806889 2024-09-17T15:45:55.568 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 613100 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 1 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 2 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 3 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 4 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 5 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: finishing write tid 6 to smithi20328086-44 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: oid 44 updating version 0 to 1497 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: oid 44 updating version 1497 to 1499 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: oid 44 version 1499 is already newer than 1498 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: oid 44 updating version 1499 to 1500 2024-09-17T15:45:55.607 INFO:tasks.rados.rados.0.smithi203.stdout:3036: oid 44 updating version 1500 to 1501 2024-09-17T15:45:55.608 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1501 (ObjNum 1062 snap 328 seq_num 1062) dirty exists 2024-09-17T15:45:55.608 INFO:tasks.rados.rados.0.smithi203.stdout:3036: left oid 44 (ObjNum 1062 snap 328 seq_num 1062) 2024-09-17T15:45:56.457 INFO:tasks.rados.rados.0.smithi203.stdout:3032: done (3 left) 2024-09-17T15:45:56.458 INFO:tasks.rados.rados.0.smithi203.stdout:3034: done (2 left) 2024-09-17T15:45:56.458 INFO:tasks.rados.rados.0.smithi203.stdout:3036: done (1 left) 2024-09-17T15:45:56.458 INFO:tasks.rados.rados.0.smithi203.stdout:3038: done (0 left) 2024-09-17T15:45:56.458 INFO:tasks.rados.rados.0.smithi203.stdout:3039: delete oid 42 current snap is 329 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:3039: done (0 left) 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:3040: copy_from oid 48 from oid 38 current snap is 329 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:3041: read oid 39 snap -1 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:3041: expect (ObjNum 1013 snap 316 seq_num 1013) 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:3042: rollback oid 7 current snap is 329 2024-09-17T15:45:56.472 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 327 2024-09-17T15:45:56.473 INFO:tasks.rados.rados.0.smithi203.stdout:3043: copy_from oid 35 from oid 9 current snap is 329 2024-09-17T15:45:56.473 INFO:tasks.rados.rados.0.smithi203.stdout:3044: write (excl) oid 41 current snap is 329 2024-09-17T15:45:56.473 INFO:tasks.rados.rados.0.smithi203.stdout:3044: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-17T15:45:56.486 INFO:tasks.rados.rados.0.smithi203.stdout:3044: writing smithi20328086-41 from 682648 to 1320651 tid 1 2024-09-17T15:45:56.498 INFO:tasks.rados.rados.0.smithi203.stdout:3044: writing smithi20328086-41 from 2029782 to 2462158 tid 2 2024-09-17T15:45:56.509 INFO:tasks.rados.rados.0.smithi203.stdout:3044: writing smithi20328086-41 from 3019476 to 3606780 tid 3 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3045: rollback oid 26 current snap is 329 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 26 to 327 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3040: finishing copy_from to smithi20328086-48 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3040: got expected ENOENT (src dne) 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3040: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3042: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3042: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3042: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1191 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:45:56.510 INFO:tasks.rados.rados.0.smithi203.stdout:3041: expect (ObjNum 1013 snap 316 seq_num 1013) 2024-09-17T15:45:56.573 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2656579 2024-09-17T15:45:56.626 INFO:tasks.rados.rados.0.smithi203.stdout:3044: finishing write tid 1 to smithi20328086-41 2024-09-17T15:45:56.626 INFO:tasks.rados.rados.0.smithi203.stdout:3044: finishing write tid 2 to smithi20328086-41 2024-09-17T15:45:56.626 INFO:tasks.rados.rados.0.smithi203.stdout:3040: done (5 left) 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3041: done (4 left) 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3042: done (3 left) 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3046: delete oid 2 current snap is 329 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3043: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3045: finishing rollback tid 0 to smithi20328086-26 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: finishing write tid 3 to smithi20328086-41 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: finishing write tid 4 to smithi20328086-41 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: finishing write tid 5 to smithi20328086-41 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: oid 41 updating version 0 to 657 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: oid 41 updating version 657 to 659 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: oid 41 version 659 is already newer than 658 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:3044: oid 41 version 659 is already newer than 656 2024-09-17T15:45:56.627 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 659 (ObjNum 1064 snap 329 seq_num 1064) dirty exists 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:3044: left oid 41 (ObjNum 1064 snap 329 seq_num 1064) 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:3045: finishing rollback tid 1 to smithi20328086-26 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:3045: finishing rollback tid 2 to smithi20328086-26 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1601 (ObjNum 1028 snap 317 seq_num 1028) dirty exists 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:3043: finishing copy_from to smithi20328086-35 2024-09-17T15:45:56.628 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1300 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:45:56.631 INFO:tasks.rados.rados.0.smithi203.stdout:3043: done (3 left) 2024-09-17T15:45:56.632 INFO:tasks.rados.rados.0.smithi203.stdout:3044: done (2 left) 2024-09-17T15:45:56.632 INFO:tasks.rados.rados.0.smithi203.stdout:3045: done (1 left) 2024-09-17T15:45:56.632 INFO:tasks.rados.rados.0.smithi203.stdout:3046: done (0 left) 2024-09-17T15:45:56.632 INFO:tasks.rados.rados.0.smithi203.stdout:3047: snap_create 2024-09-17T15:45:57.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:45:57.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: get_monmap_and_config 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: build_initial_monmap 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: monmap: 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:45:57.513400+0000 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:45:57.513400+0000 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:45:57.507 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:45:57.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:45:57.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 Processor -- start 2024-09-17T15:45:57.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 -- start start 2024-09-17T15:45:57.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: init 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: _reopen_session rank -1 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): picked mon.noname-c con 0x7f520c0689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f520c068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f520c13aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f520c09c8b0 con 0x7f520c068db0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f520c09cb90 con 0x7f520c13aff0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): start opening mon connection 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f520c074500 con 0x7f520c0689d0 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): _renew_subs 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f5213450640 10 monclient(hunting): authenticate will time out at 2090.478970s 2024-09-17T15:45:57.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52109eb640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 0x7f520c069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58382/0 (socket says 172.21.15.86:58382) 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 learned_addr learned my addr 172.21.15.86:0/2458990571 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:57.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.508+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:45:57.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 legacy=0x7f520c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:57.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.512+0000 7f52119ed640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.203:6789/0 conn(0x7f520c13aff0 legacy=0x7f520c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:45:57.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:57.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:45:57.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:57.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:57.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:57.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:45:57.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:57.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:57.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:57.712+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:58.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.116+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:45:58.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.832+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 legacy=0x7f520c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:45:58.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.832+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 legacy=0x7f520c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:45:58.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.832+0000 7f52109eb640 1 --1- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 0x7f520c069200 :-1 s=OPENED pgs=30 cs=1 l=1).handle_message read tag failed 2024-09-17T15:45:58.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.832+0000 7f52109eb640 1 --1- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 0x7f520c069200 :-1 s=OPENED pgs=30 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:45:58.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.832+0000 7f5203fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:45:58.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:45:58.920+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:00.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): tick 2024-09-17T15:46:00.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:00.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:00.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:00.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f520c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f520c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.203:6789/0 conn(0x7f520c13aff0 legacy=0x7f520c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f520c068db0 legacy=0x7f520c069200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient: _add_conns ranks=[3,2] 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f51f80036a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f51f80061e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f51f8008da0 con 0x7f51f80036a0 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51f8008f50 con 0x7f51f80061e0 2024-09-17T15:46:00.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f5202ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f80061e0 auth_method 0 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f80036a0 auth_method 0 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:00.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:00.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:00.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:00.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:00.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f80036a0 auth_method 0 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f80061e0 auth_method 0 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:00.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:00.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:00.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:01.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:01.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f80036a0 auth_method 0 2024-09-17T15:46:01.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f80061e0 auth_method 0 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:01.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:01.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:01.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:01.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f80061e0 auth_method 0 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:01.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:01.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:01.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f80036a0 auth_method 0 2024-09-17T15:46:01.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:01.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:01.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:01.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:01.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:01.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:01.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:01.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:03.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): tick 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 msgr2=0x7f51f80086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f80061e0 0x7f51f80086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 msgr2=0x7f51f8005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f80036a0 0x7f51f8005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f52119ed640 1 -- 172.21.15.86:0/2458990571 reap_dead start 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f520c0689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:03.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f51f8003af0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51f8006bd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f520c09ce70 con 0x7f51f8006bd0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f51f8008da0 con 0x7f51f8003af0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f51f8008f50 con 0x7f520c0689d0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f5202ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.508+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8003af0 auth_method 0 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:03.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:03.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:03.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:03.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:03.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:03.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.512+0000 7f52119ed640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.203:6789/0 conn(0x7f51f8006bd0 legacy=0x7f51f8007060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:03.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8003af0 auth_method 0 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:46:03.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:03.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:03.716+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8003af0 auth_method 0 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.120+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8003af0 auth_method 0 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:04.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:04.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:04.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:04.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f520c0689d0 auth_method 0 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:04.924+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): tick 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 msgr2=0x7f51f800b400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.203:3300/0 conn(0x7f520c0689d0 0x7f51f800b400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 msgr2=0x7f51f8006690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> v2:172.21.15.86:3300/0 conn(0x7f51f8003af0 0x7f51f8006690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.203:6789/0 conn(0x7f51f8006bd0 legacy=0x7f51f8007060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient: _add_conns ranks=[0,2] 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51f800d9d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f51f8011640 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f8008850 con 0x7f51f800d9d0 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.508+0000 7f5202ffd640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51f8008da0 con 0x7f51f8011640 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f5202ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:06.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8011640 auth_method 0 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.512+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f51f800d9d0 legacy=0x7f51f800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:06.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:06.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8011640 auth_method 0 2024-09-17T15:46:06.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:06.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:06.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:06.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:06.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:06.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:06.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:06.716+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8011640 auth_method 0 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:07.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.120+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f8011640 auth_method 0 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:07.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:07.924+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:09.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 47312137 0 0) 0x7f51f8008850 con 0x7f51f800d9d0 2024-09-17T15:46:09.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 10 monclient(hunting): my global_id is 214098 2024-09-17T15:46:09.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.344+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51f4003530 con 0x7f51f800d9d0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 846922865 0 0) 0x7f51f4003530 con 0x7f51f800d9d0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 msgr2=0x7f51f8013a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f8011640 0x7f51f8013a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f8015240 con 0x7f51f800d9d0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f520400f040 con 0x7f51f800d9d0 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5213450640 5 monclient: authenticate success, global_id 214098 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5213450640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: dump: 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 >> v1:172.21.15.86:6789/0 conn(0x7f51f800d9d0 legacy=0x7f51f800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 1 -- 172.21.15.86:0/2458990571 reap_dead start 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 0x7f51f4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): picked mon.a con 0x7f51f4003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4004810 0x7f51f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): picked mon.c con 0x7f51f4004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f51f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): picked mon.b con 0x7f51f40053d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f51f8008da0 con 0x7f51f4003a30 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51f400c260 con 0x7f51f40053d0 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51f400c460 con 0x7f51f4004810 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5203fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f5213450640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 0x7f51f4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f4003a30 auth_method 0 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4004810 0x7f51f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f4004810 auth_method 0 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f51f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 10 monclient(hunting): get_auth_request con 0x7f51f40053d0 auth_method 0 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient(hunting): handle_auth_done global_id 214098 payload 293 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4004810 msgr2=0x7f51f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4004810 0x7f51f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 msgr2=0x7f51f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f51f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 monclient: found mon.a 2024-09-17T15:46:09.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f4003530 con 0x7f51f4003a30 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52111ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.352+0000 7f52119ed640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f51f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f52111ec640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 msgr2=0x7f51f4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f52111ec640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 0x7f51f4003e30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5204000ee0 tx=0x7f520401e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9228b43313fc9135 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204004040 con 0x7f51f4003a30 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: dump: 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52040a84e0 con 0x7f51f4003a30 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 1 -- 172.21.15.86:0/2458990571 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520401a2d0 con 0x7f51f4003a30 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5203fff640 10 monclient: dump: 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.361 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 4 set_mon_vals no callback set 2024-09-17T15:46:09.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 10 monclient: shutdown 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 1 -- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 msgr2=0x7f51f4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.360+0000 7f5213450640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 0x7f51f4003e30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5204000ee0 tx=0x7f520401e830 comp rx=0 tx=0).stop 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- 172.21.15.86:0/2458990571 shutdown_connections 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f51f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4004810 0x7f51f4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- 172.21.15.86:0/2458990571 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4003a30 0x7f51f4003e30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- 172.21.15.86:0/2458990571 >> 172.21.15.86:0/2458990571 conn(0x7f520c0a6440 msgr2=0x7f520c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- 172.21.15.86:0/2458990571 shutdown_connections 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- 172.21.15.86:0/2458990571 wait complete. 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient: monmap: 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 Processor -- start 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- start start 2024-09-17T15:46:09.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient: init 2024-09-17T15:46:09.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:09.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): picked mon.noname-c con 0x7f51f4003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 0x7f520c134b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): picked mon.noname-a con 0x7f51f4004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f520c1350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): picked mon.noname-b con 0x7f51f40053d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1362f0 con 0x7f51f4004810 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f520c09c2d0 con 0x7f51f40053d0 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f520c136470 con 0x7f51f4003a30 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f5213450640 10 monclient(hunting): authenticate will time out at 2102.335213s 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 0x7f520c134b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42700/0 (socket says 172.21.15.86:42700) 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 -- 172.21.15.86:0/780277852 learned_addr learned my addr 172.21.15.86:0/780277852 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): get_auth_request con 0x7f51f4003a30 auth_method 0 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): get_auth_request con 0x7f51f4004810 auth_method 0 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52119ed640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f520c1350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52119ed640 10 monclient(hunting): get_auth_request con 0x7f51f40053d0 auth_method 0 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52119ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52119ed640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52119ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient(hunting): handle_auth_done global_id 164621 payload 824 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 -- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 msgr2=0x7f520c1350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f520c1350d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 -- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 msgr2=0x7f520c134b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 0x7f520c134b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 monclient: found mon.noname-c 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52111ec640 1 -- 172.21.15.86:0/780277852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c126d00 con 0x7f51f4003a30 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.364+0000 7f52109eb640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 0x7f520c134b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f52111ec640 1 -- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 msgr2=0x7f520c1d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f52111ec640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f520401f0d0 tx=0x7f51f4005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204004000 con 0x7f51f4003a30 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: dump: 2024-09-17T15:46:09.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520401a0d0 con 0x7f51f4003a30 2024-09-17T15:46:09.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52040a88d0 con 0x7f51f4003a30 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 10 monclient: dump: 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 5 monclient: authenticate success, global_id 164621 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _renew_subs 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5202ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c09d460 con 0x7f51f4003a30 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _renew_subs 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1d0970 con 0x7f51f4003a30 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f52040c2060 con 0x7f51f4003a30 2024-09-17T15:46:09.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c136800 con 0x7f51f4003a30 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.368+0000 7f5201ffb640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 0x7f51f803fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.372+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(678..678 src has 438..678) ==== 4835+0+0 (secure 0 0 0) 0x7f52040eb5f0 con 0x7f51f4003a30 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.372+0000 7f52109eb640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 0x7f51f803fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.372+0000 7f52109eb640 10 monclient: get_auth_request con 0x7f51f803d880 auth_method 0 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.372+0000 7f52109eb640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 0x7f51f803fd40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f520c125d30 tx=0x7f5208009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.376+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f520c126d00 con 0x7f51f4003a30 2024-09-17T15:46:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.376+0000 7f5201ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.376+0000 7f5201ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5213450640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:09.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5213450640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:09.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5213450640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f520c125240 con 0x7f51f4003a30 2024-09-17T15:46:09.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5201ffb640 1 -- 172.21.15.86:0/780277852 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f52040b7040 con 0x7f51f4003a30 2024-09-17T15:46:09.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5201ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:09.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.620+0000 7f5201ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.620 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 msgr2=0x7f51f803fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 0x7f51f803fd40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f520c125d30 tx=0x7f5208009040 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 10 monclient: shutdown 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 msgr2=0x7f520c1d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f520401f0d0 tx=0x7f51f4005840 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 shutdown_connections 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51f803d880 0x7f51f803fd40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51f40053d0 0x7f520c1350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51f4004810 0x7f520c134b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 --2- 172.21.15.86:0/780277852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51f4003a30 0x7f520c1d02c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 >> 172.21.15.86:0/780277852 conn(0x7f520c0a6440 msgr2=0x7f520c13a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 shutdown_connections 2024-09-17T15:46:09.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.624+0000 7f5213450640 1 -- 172.21.15.86:0/780277852 wait complete. 2024-09-17T15:46:09.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:46:09.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:46:09.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: monmap: 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:09.768313+0000 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:09.768313+0000 2024-09-17T15:46:09.763 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:09.764 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 Processor -- start 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 -- start start 2024-09-17T15:46:09.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: init 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f605009d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f605009e260 0x7f6050139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f605009e260 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f605013a440 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:09.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605009ff00 con 0x7f605013a440 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605009ec30 con 0x7f605009d770 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f605009e7f0 con 0x7f605009e260 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6056ab7640 10 monclient(hunting): authenticate will time out at 2102.733844s 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6055054640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f605013a440 0x7f605013c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39848/0 (socket says 172.21.15.86:39848) 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 learned_addr learned my addr 172.21.15.86:0/3674901216 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 1 --2- 172.21.15.86:0/3674901216 >> v2:172.21.15.203:3300/0 conn(0x7f605009e260 0x7f6050139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): get_auth_request con 0x7f605009e260 auth_method 0 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.764+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 >> v1:172.21.15.86:6789/0 conn(0x7f605013a440 legacy=0x7f605013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801718612 0 0) 0x7f605009ff00 con 0x7f605013a440 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): my global_id is 214104 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6038003530 con 0x7f605013a440 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3474366595 0 0) 0x7f6038003530 con 0x7f605013a440 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 >> v2:172.21.15.203:3300/0 conn(0x7f605009e260 msgr2=0x7f6050139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 --2- 172.21.15.86:0/3674901216 >> v2:172.21.15.203:3300/0 conn(0x7f605009e260 0x7f6050139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 >> v1:172.21.15.203:6789/0 conn(0x7f605009d770 legacy=0x7f605009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605009ef10 con 0x7f605013a440 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f6048001df0 con 0x7f605013a440 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: dump: 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.772 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 >> v1:172.21.15.86:6789/0 conn(0x7f605013a440 legacy=0x7f605013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6038003a80 0x7f6038003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): picked mon.a con 0x7f6038003a80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6038004810 0x7f6038004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): picked mon.b con 0x7f6038004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 0x7f603800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): picked mon.c con 0x7f60380053d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f603800c210 con 0x7f6038003a80 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f603800c3c0 con 0x7f6038004810 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f603800e7d0 con 0x7f60380053d0 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f604f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 0x7f603800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): get_auth_request con 0x7f60380053d0 auth_method 0 2024-09-17T15:46:09.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6038003a80 0x7f6038003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): get_auth_request con 0x7f6038003a80 auth_method 0 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:09.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient(hunting): handle_auth_done global_id 214104 payload 293 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6038004810 msgr2=0x7f6038004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6038004810 0x7f6038004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6038003a80 msgr2=0x7f6038003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6038003a80 0x7f6038003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 monclient: found mon.c 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6038003530 con 0x7f60380053d0 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6056ab7640 5 monclient: authenticate success, global_id 214104 2024-09-17T15:46:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6056ab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.768+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 reap_dead start 2024-09-17T15:46:09.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6055054640 1 -- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 msgr2=0x7f603800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6055054640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 0x7f603800b9d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60500a2c80 tx=0x7f604801d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b004c9f554c118b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604800a860 con 0x7f60380053d0 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: dump: 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.784 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480072f0 con 0x7f60380053d0 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 1 -- 172.21.15.86:0/3674901216 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60480076e0 con 0x7f60380053d0 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 4 set_mon_vals no callback set 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604f7fe640 10 monclient: dump: 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.785 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: shutdown 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 msgr2=0x7f603800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 0x7f603800b9d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f60500a2c80 tx=0x7f604801d720 comp rx=0 tx=0).stop 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- 172.21.15.86:0/3674901216 shutdown_connections 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 --2- 172.21.15.86:0/3674901216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60380053d0 0x7f603800b9d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- 172.21.15.86:0/3674901216 >> 172.21.15.86:0/3674901216 conn(0x7f60500a2f40 msgr2=0x7f60500a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- 172.21.15.86:0/3674901216 shutdown_connections 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- 172.21.15.86:0/3674901216 wait complete. 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: monmap: 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:09.787 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 Processor -- start 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- start start 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: init 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60380053d0 0x7f60501356f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-b con 0x7f60380053d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 0x7f6050072410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-c con 0x7f6050071f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 0x7f60501d1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): picked mon.noname-a con 0x7f6050072950 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f605009dee0 con 0x7f6050072950 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f605009dd60 con 0x7f60380053d0 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f605009e060 con 0x7f6050071f90 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6056ab7640 10 monclient(hunting): authenticate will time out at 2102.743853s 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f604ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 0x7f6050072410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.772+0000 7f6055054640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 0x7f60501d1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 0x7f6050072410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42722/0 (socket says 172.21.15.86:42722) 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 -- 172.21.15.86:0/2231108496 learned_addr learned my addr 172.21.15.86:0/2231108496 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): get_auth_request con 0x7f6050071f90 auth_method 0 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 0x7f60501d1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:42972/0 (socket says 172.21.15.86:42972) 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): get_auth_request con 0x7f6050072950 auth_method 0 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6054853640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60380053d0 0x7f60501356f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6054853640 10 monclient(hunting): get_auth_request con 0x7f60380053d0 auth_method 0 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6054853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6054853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6054853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient(hunting): handle_auth_done global_id 164624 payload 824 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60380053d0 msgr2=0x7f60501356f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60380053d0 0x7f60501356f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 msgr2=0x7f60501d1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 0x7f60501d1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 monclient: found mon.noname-c 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605011d7e0 con 0x7f6050071f90 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6055054640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050072950 0x7f60501d1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 msgr2=0x7f6050072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604ffff640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 0x7f6050072410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f604400e7c0 tx=0x7f604400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044017070 con 0x7f6050071f90 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: dump: 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.792 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f604400ee70 con 0x7f6050071f90 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 5 monclient: authenticate success, global_id 164624 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604400cd70 con 0x7f6050071f90 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604d7fa640 10 monclient: dump: 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f604effd640 4 set_mon_vals no callback set 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 10 monclient: _renew_subs 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605011dac0 con 0x7f6050071f90 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 10 monclient: _renew_subs 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6050074300 con 0x7f6050071f90 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.776+0000 7f6056ab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6056ab7640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 0x7f60501382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60500a1480 con 0x7f6050137ef0 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 0x7f60501382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient: get_auth_request con 0x7f6050137ef0 auth_method 0 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f604401a070 con 0x7f6050071f90 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 10 monclient(hunting): handle_auth_done global_id 214110 payload 824 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f605009d460 con 0x7f6050137ef0 2024-09-17T15:46:09.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f603c03bb60 0x7f603c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(678..678 src has 438..678) ==== 4835+0+0 (secure 0 0 0) 0x7f604404fdb0 con 0x7f6050071f90 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6055054640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f603c03bb60 0x7f603c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6055054640 10 monclient: get_auth_request con 0x7f603c03bb60 auth_method 0 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 msgr2=0x7f60501382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6054853640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 0x7f60501382d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604000f910 con 0x7f6050137ef0 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 10 monclient: dump: 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.796 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f604000f910 con 0x7f6050137ef0 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 msgr2=0x7f60501382d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 0x7f60501382d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040011a80 comp rx=0 tx=0).stop 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 msgr2=0x7f60501382d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f604d7fa640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050137ef0 0x7f60501382d0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040011a80 comp rx=0 tx=0).stop 2024-09-17T15:46:09.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.780+0000 7f6055054640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f603c03bb60 0x7f603c03e020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f605009d2c0 tx=0x7f605009f980 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6056ab7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6056ab7640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 0x7f60501d6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6056ab7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6050134a90 con 0x7f6050138810 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 0x7f60501d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient: get_auth_request con 0x7f6050138810 auth_method 0 2024-09-17T15:46:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 10 monclient(hunting): handle_auth_done global_id 214113 payload 824 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 1 -- 172.21.15.86:0/2231108496 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f605006d4f0 con 0x7f6050138810 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 msgr2=0x7f60501d6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.788+0000 7f6054853640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 0x7f60501d6270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60400069e0 con 0x7f6050138810 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 10 monclient: dump: 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f60400069e0 con 0x7f6050138810 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 msgr2=0x7f60501d6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 0x7f60501d6270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040003700 comp rx=0 tx=0).stop 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 msgr2=0x7f60501d6270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f604d7fa640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6050138810 0x7f60501d6270 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60400115c0 tx=0x7f6040003700 comp rx=0 tx=0).stop 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 25967, 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:09.802 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.803 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.804 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:09.805 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.806 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:09.807 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.809 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.810 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:09.812 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.814 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:09.814 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f603c03bb60 msgr2=0x7f603c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f603c03bb60 0x7f603c03e020 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f605009d2c0 tx=0x7f605009f980 comp rx=0 tx=0).stop 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 10 monclient: shutdown 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 msgr2=0x7f6050072410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 --2- 172.21.15.86:0/2231108496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6050071f90 0x7f6050072410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f604400e7c0 tx=0x7f604400ec80 comp rx=0 tx=0).stop 2024-09-17T15:46:09.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6055054640 1 -- 172.21.15.86:0/2231108496 reap_dead start 2024-09-17T15:46:09.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6055054640 1 -- 172.21.15.86:0/2231108496 reap_dead start 2024-09-17T15:46:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 shutdown_connections 2024-09-17T15:46:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 >> 172.21.15.86:0/2231108496 conn(0x7f60500a2f40 msgr2=0x7f605013c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 shutdown_connections 2024-09-17T15:46:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.792+0000 7f6056ab7640 1 -- 172.21.15.86:0/2231108496 wait complete. 2024-09-17T15:46:09.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:09.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:46:09.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:09.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: get_monmap_and_config 2024-09-17T15:46:09.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: monmap: 2024-09-17T15:46:09.933 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:09.940187+0000 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:09.940187+0000 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:09.934 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 Processor -- start 2024-09-17T15:46:09.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 -- start start 2024-09-17T15:46:09.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: init 2024-09-17T15:46:09.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bc40a50e0 0x7f2bc409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc40a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2bc40a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bc413d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc40a6280 con 0x7f2bc40a54c0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc409c0a0 con 0x7f2bc413d2f0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc409d2a0 con 0x7f2bc40a50e0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc9842640 10 monclient(hunting): authenticate will time out at 2102.905815s 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2bc40a50e0 0x7f2bc409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc27fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2bc40a54c0 0x7f2bc40a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39850/0 (socket says 172.21.15.86:39850) 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc27fc640 1 -- 172.21.15.86:0/3750104380 learned_addr learned my addr 172.21.15.86:0/3750104380 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): get_auth_request con 0x7f2bc40a50e0 auth_method 0 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_done global_id 214116 payload 274 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/3750104380 >> v1:172.21.15.203:6789/0 conn(0x7f2bc413d2f0 legacy=0x7f2bc413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/3750104380 >> v1:172.21.15.86:6789/0 conn(0x7f2bc40a54c0 legacy=0x7f2bc40a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:09.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:09.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.936+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/3750104380 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc409c3d0 con 0x7f2bc40a50e0 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/3750104380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc40a50e0 msgr2=0x7f2bc409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/3750104380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc40a50e0 0x7f2bc409cd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009d90 tx=0x7f2bac01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc8641d472a86d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 1 -- 172.21.15.86:0/3750104380 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0b2070 con 0x7f2bc40a50e0 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: dump: 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.940 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 1 -- 172.21.15.86:0/3750104380 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bac0a5860 con 0x7f2bc40a50e0 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 5 monclient: authenticate success, global_id 214116 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 1 -- 172.21.15.86:0/3750104380 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0a5c30 con 0x7f2bc40a50e0 2024-09-17T15:46:09.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc1ffb640 10 monclient: dump: 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:09.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 4 set_mon_vals no callback set 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 10 monclient: shutdown 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 -- 172.21.15.86:0/3750104380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc40a50e0 msgr2=0x7f2bc409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 --2- 172.21.15.86:0/3750104380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc40a50e0 0x7f2bc409cd60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009d90 tx=0x7f2bac01c830 comp rx=0 tx=0).stop 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 -- 172.21.15.86:0/3750104380 shutdown_connections 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 --2- 172.21.15.86:0/3750104380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc40a50e0 0x7f2bc409cd60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 -- 172.21.15.86:0/3750104380 >> 172.21.15.86:0/3750104380 conn(0x7f2bc40aaae0 msgr2=0x7f2bc40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 -- 172.21.15.86:0/3750104380 shutdown_connections 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 1 -- 172.21.15.86:0/3750104380 wait complete. 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 10 monclient: build_initial_monmap 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.940+0000 7f2bc9842640 10 monclient: monmap: 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:09.947 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 Processor -- start 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- start start 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: init 2024-09-17T15:46:09.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc40a50e0 0x7f2bc41cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc40a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 0x7f2bc41381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 0x7f2bc413d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc41439b0 con 0x7f2bc413d2f0 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4143830 con 0x7f2bc40a50e0 2024-09-17T15:46:09.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc4143b30 con 0x7f2bc40a54c0 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient(hunting): authenticate will time out at 2102.912815s 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 0x7f2bc41381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 0x7f2bc413d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 0x7f2bc41381c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42726/0 (socket says 172.21.15.86:42726) 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 1 -- 172.21.15.86:0/103271105 learned_addr learned my addr 172.21.15.86:0/103271105 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): get_auth_request con 0x7f2bc40a54c0 auth_method 0 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): get_auth_request con 0x7f2bc413d2f0 auth_method 0 2024-09-17T15:46:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc40a50e0 0x7f2bc41cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient(hunting): handle_auth_done global_id 214119 payload 824 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 msgr2=0x7f2bc41381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 0x7f2bc41381c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc40a50e0 msgr2=0x7f2bc41cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc40a50e0 0x7f2bc41cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 monclient: found mon.noname-a 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc4140ab0 con 0x7f2bc413d2f0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc2ffd640 10 monclient: get_auth_request con 0x7f2bc40a50e0 auth_method 0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc27fc640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc40a54c0 0x7f2bc41381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2bc40a50e0 0x7f2bc41cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 msgr2=0x7f2bc413d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc37fe640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 0x7f2bc413d960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c3e0 tx=0x7f2bbc00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc017070 con 0x7f2bc413d2f0 2024-09-17T15:46:09.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: dump: 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:09.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bbc003820 con 0x7f2bc413d2f0 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc003c10 con 0x7f2bc413d2f0 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 5 monclient: authenticate success, global_id 214119 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _renew_subs 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc40a18d0 con 0x7f2bc413d2f0 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc17fa640 4 set_mon_vals no callback set 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bbbfff640 10 monclient: dump: 2024-09-17T15:46:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _renew_subs 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.944+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc4121b40 con 0x7f2bc413d2f0 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc9842640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc9842640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 0x7f2bc413e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc413e7c0 con 0x7f2bc413dea0 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 0x7f2bc413e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient: get_auth_request con 0x7f2bc413dea0 auth_method 0 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2bbc01a070 con 0x7f2bc413d2f0 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_done global_id 164630 payload 824 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bc409cd60 con 0x7f2bc413dea0 2024-09-17T15:46:09.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ba803bb60 0x7f2ba803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(678..678 src has 438..678) ==== 4835+0+0 (secure 0 0 0) 0x7f2bbc04f1f0 con 0x7f2bc413d2f0 2024-09-17T15:46:09.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc27fc640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ba803bb60 0x7f2ba803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc27fc640 10 monclient: get_auth_request con 0x7f2ba803bb60 auth_method 0 2024-09-17T15:46:09.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 msgr2=0x7f2bc413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 0x7f2bc413e280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0af000 tx=0x7f2bac0195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0bd070 con 0x7f2bc413dea0 2024-09-17T15:46:09.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 10 monclient: dump: 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2bac0b8850 con 0x7f2bc413dea0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 msgr2=0x7f2bc413e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 0x7f2bc413e280 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0af000 tx=0x7f2bac0195d0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 msgr2=0x7f2bc413e280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.948+0000 7f2bbbfff640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc413dea0 0x7f2bc413e280 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.952+0000 7f2bc27fc640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ba803bb60 0x7f2ba803e020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007d60 tx=0x7f2bb4007cb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc9842640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc9842640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 0x7f2bc41396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc9842640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc4074be0 con 0x7f2bc41392d0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 0x7f2bc41396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient: get_auth_request con 0x7f2bc41392d0 auth_method 0 2024-09-17T15:46:09.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 10 monclient(hunting): handle_auth_done global_id 164633 payload 824 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.956+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/103271105 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2bc413fd80 con 0x7f2bc41392d0 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc2ffd640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 msgr2=0x7f2bc41396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc2ffd640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 0x7f2bc41396e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2bac01c660 tx=0x7f2bac009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bac0b2040 con 0x7f2bc41392d0 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:09.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 10 monclient: dump: 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f2bac004720 con 0x7f2bc41392d0 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:09.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 msgr2=0x7f2bc41396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 0x7f2bc41396e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2bac01c660 tx=0x7f2bac009750 comp rx=0 tx=0).stop 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 msgr2=0x7f2bc41396e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bbbfff640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2bc41392d0 0x7f2bc41396e0 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2bac01c660 tx=0x7f2bac009750 comp rx=0 tx=0).stop 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 273147, 2024-09-17T15:46:09.964 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.965 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.966 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:09.967 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.969 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:09.970 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:09.972 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:09.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.975 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:09.976 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:09.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:09.977 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:09.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:09.977 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:09.977 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ba803bb60 msgr2=0x7f2ba803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ba803bb60 0x7f2ba803e020 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007d60 tx=0x7f2bb4007cb0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 10 monclient: shutdown 2024-09-17T15:46:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 msgr2=0x7f2bc413d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 --2- 172.21.15.86:0/103271105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2bc413d2f0 0x7f2bc413d960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c3e0 tx=0x7f2bbc00c8a0 comp rx=0 tx=0).stop 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 reap_dead start 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc37fe640 1 -- 172.21.15.86:0/103271105 reap_dead start 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 shutdown_connections 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 >> 172.21.15.86:0/103271105 conn(0x7f2bc40aaae0 msgr2=0x7f2bc40a3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 shutdown_connections 2024-09-17T15:46:09.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:09.960+0000 7f2bc9842640 1 -- 172.21.15.86:0/103271105 wait complete. 2024-09-17T15:46:09.979 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:09.979 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:46:09.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:10.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: monmap: 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:10.102346+0000 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:10.102346+0000 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:10.096 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:10.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:10.097 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:10.097 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:10.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 Processor -- start 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 -- start start 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: init 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48140a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): picked mon.noname-b con 0x7f48140a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 0x7f4814135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): picked mon.noname-c con 0x7f48140a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): picked mon.noname-a con 0x7f4814135f90 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f48140a2410 con 0x7f4814135f90 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f48140a2130 con 0x7f48140a1590 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f48140a26f0 con 0x7f48140a0c70 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4818809640 10 monclient(hunting): authenticate will time out at 2103.067797s 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.096+0000 7f4812d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48140a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43012/0 (socket says 172.21.15.86:43012) 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): get_auth_request con 0x7f48140a0c70 auth_method 0 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 1 -- 172.21.15.86:0/1066943204 learned_addr learned my addr 172.21.15.86:0/1066943204 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): get_auth_request con 0x7f4814135f90 auth_method 0 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4811d9b640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 0x7f4814135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient(hunting): handle_auth_done global_id 164636 payload 274 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 -- 172.21.15.86:0/1066943204 >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 msgr2=0x7f4814135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 0x7f4814135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 -- 172.21.15.86:0/1066943204 >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 msgr2=0x7f4814136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 monclient: found mon.noname-b 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:10.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f481259c640 1 -- 172.21.15.86:0/1066943204 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f481409f9f0 con 0x7f48140a0c70 2024-09-17T15:46:10.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4812d9d640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:10.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4811d9b640 10 monclient: get_auth_request con 0x7f48140a1590 auth_method 0 2024-09-17T15:46:10.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.100+0000 7f4811d9b640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 0x7f4814135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481259c640 1 -- 172.21.15.86:0/1066943204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 msgr2=0x7f48140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481259c640 1 --2- 172.21.15.86:0/1066943204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48140a1050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f481406e2e0 tx=0x7f480401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924ecce99da5199d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 1 -- 172.21.15.86:0/1066943204 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48040b2070 con 0x7f48140a0c70 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: dump: 2024-09-17T15:46:10.106 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 1 -- 172.21.15.86:0/1066943204 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48040a5890 con 0x7f48140a0c70 2024-09-17T15:46:10.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 5 monclient: authenticate success, global_id 164636 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 1 -- 172.21.15.86:0/1066943204 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48040a5c60 con 0x7f48140a0c70 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 4 set_mon_vals no callback set 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f481159a640 10 monclient: dump: 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: shutdown 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- 172.21.15.86:0/1066943204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 msgr2=0x7f48140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 --2- 172.21.15.86:0/1066943204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48140a1050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f481406e2e0 tx=0x7f480401c7f0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- 172.21.15.86:0/1066943204 shutdown_connections 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.86:3300/0 conn(0x7f4814135f90 0x7f4814136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 --2- 172.21.15.86:0/1066943204 >> v2:172.21.15.203:3300/0 conn(0x7f48140a1590 0x7f4814135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 --2- 172.21.15.86:0/1066943204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48140a1050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- 172.21.15.86:0/1066943204 >> 172.21.15.86:0/1066943204 conn(0x7f48140a2f40 msgr2=0x7f48140a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- 172.21.15.86:0/1066943204 shutdown_connections 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- 172.21.15.86:0/1066943204 wait complete. 2024-09-17T15:46:10.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: monmap: 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 Processor -- start 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 -- start start 2024-09-17T15:46:10.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: init 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.104+0000 7f4818809640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48141cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): picked mon.noname-c con 0x7f48140a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 0x7f4814134ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): picked mon.noname-a con 0x7f48140a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135f90 0x7f481411cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): picked mon.noname-b con 0x7f4814135f90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f48140a00d0 con 0x7f48140a1590 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f481409ff50 con 0x7f4814135f90 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f481413f740 con 0x7f48140a0c70 2024-09-17T15:46:10.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): authenticate will time out at 2103.076062s 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 0x7f4814134ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48141cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48141cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42778/0 (socket says 172.21.15.86:42778) 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 0x7f4814134ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43018/0 (socket says 172.21.15.86:43018) 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 1 -- 172.21.15.86:0/4007564939 learned_addr learned my addr 172.21.15.86:0/4007564939 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): get_auth_request con 0x7f48140a0c70 auth_method 0 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): get_auth_request con 0x7f48140a1590 auth_method 0 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient(hunting): handle_auth_done global_id 214125 payload 824 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4812d9d640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135f90 0x7f481411cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 msgr2=0x7f48141cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48140a0c70 0x7f48141cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135f90 msgr2=0x7f481411cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135f90 0x7f481411cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 monclient: found mon.noname-a 2024-09-17T15:46:10.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48141202c0 con 0x7f48140a1590 2024-09-17T15:46:10.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 msgr2=0x7f4814134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4811d9b640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 0x7f4814134ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00c7f0 tx=0x7f47fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc015070 con 0x7f48140a1590 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: dump: 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47fc004220 con 0x7f48140a1590 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 5 monclient: authenticate success, global_id 214125 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient: _renew_subs 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f481411f080 con 0x7f48140a1590 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc007830 con 0x7f48140a1590 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f480affd640 10 monclient: dump: 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4810d99640 4 set_mon_vals no callback set 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient: _renew_subs 2024-09-17T15:46:10.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4814073570 con 0x7f48140a1590 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 0x7f481402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.108+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4814135a50 con 0x7f4814135400 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f47fc007a10 con 0x7f48140a1590 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f47f403bb60 0x7f47f403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 0x7f481402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(678..678 src has 438..678) ==== 4835+0+0 (secure 0 0 0) 0x7f47fc04ff20 con 0x7f48140a1590 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient: get_auth_request con 0x7f4814135400 auth_method 0 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f4812d9d640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f47f403bb60 0x7f47f403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f4812d9d640 10 monclient: get_auth_request con 0x7f47f403bb60 auth_method 0 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 10 monclient(hunting): handle_auth_done global_id 204187 payload 824 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4814138600 con 0x7f4814135400 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 msgr2=0x7f481402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f481259c640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 0x7f481402d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f480401aea0 tx=0x7f4804019590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f4812d9d640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f47f403bb60 0x7f47f403e020 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f480c009630 tx=0x7f480c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48040be070 con 0x7f4814135400 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.112+0000 7f480affd640 10 monclient: dump: 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f48040b3040 con 0x7f4814135400 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 msgr2=0x7f481402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 0x7f481402d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f480401aea0 tx=0x7f4804019590 comp rx=0 tx=0).stop 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 msgr2=0x7f481402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.116+0000 7f480affd640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4814135400 0x7f481402d450 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f480401aea0 tx=0x7f4804019590 comp rx=0 tx=0).stop 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f4818809640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f4818809640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 0x7f481402dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f4818809640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48141209c0 con 0x7f481402d990 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 0x7f481402dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient: get_auth_request con 0x7f481402d990 auth_method 0 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 10 monclient(hunting): handle_auth_done global_id 204190 payload 824 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.120+0000 7f481259c640 1 -- 172.21.15.86:0/4007564939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f481406d4f0 con 0x7f481402d990 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f481259c640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 msgr2=0x7f481402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f481259c640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 0x7f481402dda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48040bd000 tx=0x7f48040a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48040b2040 con 0x7f481402d990 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 10 monclient: dump: 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.126 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f48040ade20 con 0x7f481402d990 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 msgr2=0x7f481402dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 0x7f481402dda0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48040bd000 tx=0x7f48040a7000 comp rx=0 tx=0).stop 2024-09-17T15:46:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 msgr2=0x7f481402dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f480affd640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f481402d990 0x7f481402dda0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f48040bd000 tx=0x7f48040a7000 comp rx=0 tx=0).stop 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72725, 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:10.128 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:10.129 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:10.130 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:10.133 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:10.134 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.137 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.138 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:10.139 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:46:10.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:10.140 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:10.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:10.140 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:10.140 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f47f403bb60 msgr2=0x7f47f403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f47f403bb60 0x7f47f403e020 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f480c009630 tx=0x7f480c008040 comp rx=0 tx=0).stop 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 10 monclient: shutdown 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 msgr2=0x7f4814134ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 --2- 172.21.15.86:0/4007564939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48140a1590 0x7f4814134ec0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00c7f0 tx=0x7f47fc00ccb0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4812d9d640 1 -- 172.21.15.86:0/4007564939 reap_dead start 2024-09-17T15:46:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4812d9d640 1 -- 172.21.15.86:0/4007564939 reap_dead start 2024-09-17T15:46:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 shutdown_connections 2024-09-17T15:46:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 >> 172.21.15.86:0/4007564939 conn(0x7f48140a2f40 msgr2=0x7f481413c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 shutdown_connections 2024-09-17T15:46:10.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.124+0000 7f4818809640 1 -- 172.21.15.86:0/4007564939 wait complete. 2024-09-17T15:46:10.142 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:10.142 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:46:10.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:46:10.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: get_monmap_and_config 2024-09-17T15:46:10.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: monmap: 2024-09-17T15:46:10.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:10.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:10.262313+0000 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:10.262313+0000 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:10.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:10.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:10.258 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:10.258 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:10.258 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 Processor -- start 2024-09-17T15:46:10.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 -- start start 2024-09-17T15:46:10.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: init 2024-09-17T15:46:10.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:46:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95d00689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 0x7f95d0069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-b con 0x7f95d0068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:10.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 0x7f95d013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-c con 0x7f95d0139f00 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95d009d770 con 0x7f95d00689d0 2024-09-17T15:46:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f95d009ed00 con 0x7f95d0139f00 2024-09-17T15:46:10.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f95d009efe0 con 0x7f95d0068db0 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95d5398640 10 monclient(hunting): authenticate will time out at 2103.227757s 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95ceffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f95d00689d0 0x7f95d009ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39854/0 (socket says 172.21.15.86:39854) 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95ceffd640 1 -- 172.21.15.86:0/1335621291 learned_addr learned my addr 172.21.15.86:0/1335621291 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:10.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.256+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 0x7f95d0069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): get_auth_request con 0x7f95d0068db0 auth_method 0 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95cf7fe640 1 --2- 172.21.15.86:0/1335621291 >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 0x7f95d013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient(hunting): handle_auth_done global_id 164642 payload 274 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 1 -- 172.21.15.86:0/1335621291 >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 msgr2=0x7f95d013c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/1335621291 >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 0x7f95d013c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 1 -- 172.21.15.86:0/1335621291 >> v1:172.21.15.86:6789/0 conn(0x7f95d00689d0 legacy=0x7f95d009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:10.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 1 monclient: found mon.noname-b 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95ce7fc640 1 -- 172.21.15.86:0/1335621291 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d00696d0 con 0x7f95d0068db0 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95cf7fe640 10 monclient: get_auth_request con 0x7f95d0139f00 auth_method 0 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.260+0000 7f95cf7fe640 1 --2- 172.21.15.86:0/1335621291 >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 0x7f95d013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ce7fc640 1 -- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 msgr2=0x7f95d0069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 0x7f95d0069190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f95c40097c0 tx=0x7f95c401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b93ae906b06c1fc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 1 -- 172.21.15.86:0/1335621291 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40b0070 con 0x7f95d0068db0 2024-09-17T15:46:10.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:10.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: dump: 2024-09-17T15:46:10.284 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.284 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.285 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.286 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.286 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 1 -- 172.21.15.86:0/1335621291 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c40ab600 con 0x7f95d0068db0 2024-09-17T15:46:10.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 1 -- 172.21.15.86:0/1335621291 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40ab9d0 con 0x7f95d0068db0 2024-09-17T15:46:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cdffb640 10 monclient: dump: 2024-09-17T15:46:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 5 monclient: authenticate success, global_id 164642 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 4 set_mon_vals no callback set 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: shutdown 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 msgr2=0x7f95d0069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 0x7f95d0069190 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f95c40097c0 tx=0x7f95c401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- 172.21.15.86:0/1335621291 shutdown_connections 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- 172.21.15.86:0/1335621291 >> v2:172.21.15.203:3300/0 conn(0x7f95d0139f00 0x7f95d013c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- 172.21.15.86:0/1335621291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0068db0 0x7f95d0069190 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- 172.21.15.86:0/1335621291 >> 172.21.15.86:0/1335621291 conn(0x7f95d00a6440 msgr2=0x7f95d00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- 172.21.15.86:0/1335621291 shutdown_connections 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- 172.21.15.86:0/1335621291 wait complete. 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: build_initial_monmap 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: monmap: 2024-09-17T15:46:10.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 Processor -- start 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- start start 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: init 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:10.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-a con 0x7f95d00689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95d0068db0 0x7f95d0134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-b con 0x7f95d0068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0139f00 0x7f95d0072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): picked mon.noname-c con 0x7f95d0139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f95d013f3e0 con 0x7f95d00689d0 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95d013f260 con 0x7f95d0068db0 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f95d013f560 con 0x7f95d0139f00 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:10.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95d5398640 10 monclient(hunting): authenticate will time out at 2103.235056s 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43024/0 (socket says 172.21.15.86:43024) 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 -- 172.21.15.86:0/2921754606 learned_addr learned my addr 172.21.15.86:0/2921754606 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): get_auth_request con 0x7f95d00689d0 auth_method 0 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0139f00 0x7f95d0072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): get_auth_request con 0x7f95d0139f00 auth_method 0 2024-09-17T15:46:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95d0068db0 0x7f95d0134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient(hunting): handle_auth_done global_id 214128 payload 824 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 -- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0139f00 msgr2=0x7f95d0072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0139f00 0x7f95d0072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 -- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95d0068db0 msgr2=0x7f95d0134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95d0068db0 0x7f95d0134ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:10.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.264+0000 7f95ceffd640 1 -- 172.21.15.86:0/2921754606 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d011d7d0 con 0x7f95d00689d0 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95ceffd640 1 -- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 msgr2=0x7f95d011ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95ceffd640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95c000b430 tx=0x7f95c000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0015020 con 0x7f95d00689d0 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: dump: 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.295 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 5 monclient: authenticate success, global_id 214128 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c0011a60 con 0x7f95d00689d0 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _renew_subs 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d011e640 con 0x7f95d00689d0 2024-09-17T15:46:10.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0011e50 con 0x7f95d00689d0 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 10 monclient: dump: 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:10.297 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _renew_subs 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95cd7fa640 4 set_mon_vals no callback set 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d011e360 con 0x7f95d00689d0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d0069190 con 0x7f95d00689d0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f95c0018020 con 0x7f95d00689d0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 0x7f95bc03e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(678..678 src has 438..678) ==== 4835+0+0 (secure 0 0 0) 0x7f95c004ff40 con 0x7f95d00689d0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 0x7f95bc03e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.268+0000 7f95ce7fc640 10 monclient: get_auth_request con 0x7f95bc03bd00 auth_method 0 2024-09-17T15:46:10.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.272+0000 7f95ce7fc640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 0x7f95bc03e1c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f95c40ab000 tx=0x7f95c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:10.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.272+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f95c001f020 con 0x7f95d00689d0 2024-09-17T15:46:10.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.276+0000 7f95b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:10.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.276+0000 7f95b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:10.335 INFO:tasks.rados.rados.0.smithi203.stdout:3047: done (0 left) 2024-09-17T15:46:10.335 INFO:tasks.rados.rados.0.smithi203.stdout:3048: write oid 19 current snap is 330 2024-09-17T15:46:10.335 INFO:tasks.rados.rados.0.smithi203.stdout:3048: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-09-17T15:46:10.354 INFO:tasks.rados.rados.0.smithi203.stdout:3048: writing smithi20328086-19 from 716262 to 1221116 tid 1 2024-09-17T15:46:10.374 INFO:tasks.rados.rados.0.smithi203.stdout:3048: writing smithi20328086-19 from 1863099 to 2582791 tid 2 2024-09-17T15:46:10.382 INFO:tasks.rados.rados.0.smithi203.stdout:3048: writing smithi20328086-19 from 3074322 to 3800523 tid 3 2024-09-17T15:46:10.382 INFO:tasks.rados.rados.0.smithi203.stdout:3049: write (excl) oid 40 current snap is 330 2024-09-17T15:46:10.382 INFO:tasks.rados.rados.0.smithi203.stdout:3048: finishing write tid 1 to smithi20328086-19 2024-09-17T15:46:10.382 INFO:tasks.rados.rados.0.smithi203.stdout:3049: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-09-17T15:46:10.389 INFO:tasks.rados.rados.0.smithi203.stdout:3049: writing smithi20328086-40 from 733069 to 1313172 tid 1 2024-09-17T15:46:10.396 INFO:tasks.rados.rados.0.smithi203.stdout:3049: writing smithi20328086-40 from 1921581 to 2501284 tid 2 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3049: writing smithi20328086-40 from 3218098 to 3755571 tid 3 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3050: copy_from oid 39 from oid 50 current snap is 330 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3048: finishing write tid 2 to smithi20328086-19 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3048: finishing write tid 3 to smithi20328086-19 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3048: finishing write tid 4 to smithi20328086-19 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3048: finishing write tid 5 to smithi20328086-19 2024-09-17T15:46:10.405 INFO:tasks.rados.rados.0.smithi203.stdout:3048: oid 19 updating version 0 to 1537 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3048: oid 19 updating version 1537 to 1539 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3048: oid 19 updating version 1539 to 1540 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3048: oid 19 version 1540 is already newer than 1538 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1540 (ObjNum 1066 snap 330 seq_num 1066) dirty exists 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3048: left oid 19 (ObjNum 1066 snap 330 seq_num 1066) 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3048: done (2 left) 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3051: read oid 10 snap -1 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3051: expect deleted 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3052: read oid 20 snap -1 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3052: expect (ObjNum 1059 snap 328 seq_num 1059) 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3053: write oid 32 current snap is 330 2024-09-17T15:46:10.406 INFO:tasks.rados.rados.0.smithi203.stdout:3053: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-17T15:46:10.416 INFO:tasks.rados.rados.0.smithi203.stdout:3053: writing smithi20328086-32 from 749876 to 1405228 tid 1 2024-09-17T15:46:10.425 INFO:tasks.rados.rados.0.smithi203.stdout:3053: writing smithi20328086-32 from 1863710 to 2587071 tid 2 2024-09-17T15:46:10.433 INFO:tasks.rados.rados.0.smithi203.stdout:3053: writing smithi20328086-32 from 3012815 to 3761560 tid 3 2024-09-17T15:46:10.433 INFO:tasks.rados.rados.0.smithi203.stdout:3054: copy_from oid 11 from oid 27 current snap is 330 2024-09-17T15:46:10.433 INFO:tasks.rados.rados.0.smithi203.stdout:3049: finishing write tid 1 to smithi20328086-40 2024-09-17T15:46:10.433 INFO:tasks.rados.rados.0.smithi203.stdout:3049: finishing write tid 2 to smithi20328086-40 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3050: finishing copy_from to smithi20328086-39 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3050: got expected ENOENT (src dne) 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3050: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: finishing write tid 3 to smithi20328086-40 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: finishing write tid 4 to smithi20328086-40 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: finishing write tid 5 to smithi20328086-40 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: oid 40 updating version 0 to 1502 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: oid 40 updating version 1502 to 1503 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: oid 40 updating version 1503 to 1504 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: oid 40 updating version 1504 to 1505 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1505 (ObjNum 1067 snap 330 seq_num 1067) dirty exists 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3049: left oid 40 (ObjNum 1067 snap 330 seq_num 1067) 2024-09-17T15:46:10.434 INFO:tasks.rados.rados.0.smithi203.stdout:3052: expect (ObjNum 1059 snap 328 seq_num 1059) 2024-09-17T15:46:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.524+0000 7f95d5398640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:46:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.524+0000 7f95d5398640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:46:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.524+0000 7f95d5398640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:10.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.524+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f95d01ce6d0 con 0x7f95d00689d0 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f95b77fe640 1 -- 172.21.15.86:0/2921754606 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f95c000fcd0 con 0x7f95d00689d0 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f95b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:46:10.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f95b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:10.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=1907390336,config=1802735181,health=1845395070,kv=3591778543,logm=2213992071}) 2024-09-17T15:46:10.527 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=32,config=2,health=12,kv=10,logm=44} crc {auth=2367906074,config=3541143443,health=1845395070,kv=3591778543,logm=1958366825}) 2024-09-17T15:46:10.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=1907390336,config=1802735181,health=1845395070,kv=3591778543,logm=2213992071}) 2024-09-17T15:46:10.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.528+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=32,config=2,health=12,kv=11,logm=43} crc {auth=2367906074,config=3541143443,health=1845395070,kv=2552736648,logm=1474822735}) 2024-09-17T15:46:10.528 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 msgr2=0x7f95bc03e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 0x7f95bc03e1c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f95c40ab000 tx=0x7f95c4008040 comp rx=0 tx=0).stop 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 10 monclient: shutdown 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 msgr2=0x7f95d011ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f95c000b430 tx=0x7f95c000b8f0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 shutdown_connections 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f95bc03bd00 0x7f95bc03e1c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f95d0139f00 0x7f95d0072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95d0068db0 0x7f95d0134ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 --2- 172.21.15.86:0/2921754606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95d00689d0 0x7f95d011ce40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 >> 172.21.15.86:0/2921754606 conn(0x7f95d00a6440 msgr2=0x7f95d013ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:10.530 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.530 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257084268}) 2024-09-17T15:46:10.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 shutdown_connections 2024-09-17T15:46:10.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f95d5398640 1 -- 172.21.15.86:0/2921754606 wait complete. 2024-09-17T15:46:10.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2431430519}) 2024-09-17T15:46:10.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257084268}) 2024-09-17T15:46:10.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3783098687}) 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010837561}) 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115818975}) 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010837561}) 2024-09-17T15:46:10.532 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.532+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3654857079}) 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051295794}) 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668822857}) 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051295794}) 2024-09-17T15:46:10.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.536+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983707570}) 2024-09-17T15:46:10.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450667469}) 2024-09-17T15:46:10.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=725286920}) 2024-09-17T15:46:10.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450667469}) 2024-09-17T15:46:10.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042935962}) 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096384905}) 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475963374}) 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096384905}) 2024-09-17T15:46:10.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.540+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3127705952}) 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049009345}) 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109194698}) 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049009345}) 2024-09-17T15:46:10.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576401364}) 2024-09-17T15:46:10.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730453145}) 2024-09-17T15:46:10.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689655960}) 2024-09-17T15:46:10.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730453145}) 2024-09-17T15:46:10.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.544+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439198504}) 2024-09-17T15:46:10.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:46:10.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893609815}) 2024-09-17T15:46:10.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245683341}) 2024-09-17T15:46:10.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893609815}) 2024-09-17T15:46:10.547 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965631861}) 2024-09-17T15:46:10.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.549 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920509780}) 2024-09-17T15:46:10.549 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150417431}) 2024-09-17T15:46:10.549 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.549 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920509780}) 2024-09-17T15:46:10.549 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.548+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=39196712}) 2024-09-17T15:46:10.551 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638559216}) 2024-09-17T15:46:10.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952030015}) 2024-09-17T15:46:10.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638559216}) 2024-09-17T15:46:10.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262246587}) 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373525185}) 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2906190297}) 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373525185}) 2024-09-17T15:46:10.553 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.552+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830803697}) 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170737613}) 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=368895859}) 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170737613}) 2024-09-17T15:46:10.555 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3254676678}) 2024-09-17T15:46:10.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835289802}) 2024-09-17T15:46:10.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4028390264}) 2024-09-17T15:46:10.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.556+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835289802}) 2024-09-17T15:46:10.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1795718542}) 2024-09-17T15:46:10.559 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.559 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2711001947}) 2024-09-17T15:46:10.559 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=439036685}) 2024-09-17T15:46:10.559 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2711001947}) 2024-09-17T15:46:10.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772209066}) 2024-09-17T15:46:10.561 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.560+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.561 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839482987}) 2024-09-17T15:46:10.561 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3452104646}) 2024-09-17T15:46:10.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839482987}) 2024-09-17T15:46:10.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3896000951}) 2024-09-17T15:46:10.563 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.563 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012751407}) 2024-09-17T15:46:10.563 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010613076}) 2024-09-17T15:46:10.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012751407}) 2024-09-17T15:46:10.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.564+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068217332}) 2024-09-17T15:46:10.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116225775}) 2024-09-17T15:46:10.566 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509173730}) 2024-09-17T15:46:10.566 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.566 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116225775}) 2024-09-17T15:46:10.566 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650406129}) 2024-09-17T15:46:10.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732992065}) 2024-09-17T15:46:10.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3231001062}) 2024-09-17T15:46:10.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732992065}) 2024-09-17T15:46:10.568 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.568+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550201410}) 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024322261}) 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=760626277}) 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024322261}) 2024-09-17T15:46:10.570 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897370073}) 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558155148}) 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=512631112}) 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1558155148}) 2024-09-17T15:46:10.572 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.572+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2839536582}) 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344099819}) 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114102578}) 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344099819}) 2024-09-17T15:46:10.574 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1056885927}) 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728524438}) 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392334760}) 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728524438}) 2024-09-17T15:46:10.576 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.576+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3886548764}) 2024-09-17T15:46:10.577 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660894754}) 2024-09-17T15:46:10.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=66963607}) 2024-09-17T15:46:10.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=660894754}) 2024-09-17T15:46:10.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752151181}) 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318101360}) 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122411028}) 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3318101360}) 2024-09-17T15:46:10.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.580+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021578096}) 2024-09-17T15:46:10.581 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.581 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3930517504}) 2024-09-17T15:46:10.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912076529}) 2024-09-17T15:46:10.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3930517504}) 2024-09-17T15:46:10.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265575180}) 2024-09-17T15:46:10.583 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.583 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688267555}) 2024-09-17T15:46:10.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058286298}) 2024-09-17T15:46:10.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688267555}) 2024-09-17T15:46:10.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.584+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1077579872}) 2024-09-17T15:46:10.677 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2861347 2024-09-17T15:46:10.698 INFO:tasks.rados.rados.0.smithi203.stdout:3053: finishing write tid 1 to smithi20328086-32 2024-09-17T15:46:10.698 INFO:tasks.rados.rados.0.smithi203.stdout:3049: done (5 left) 2024-09-17T15:46:10.698 INFO:tasks.rados.rados.0.smithi203.stdout:3050: done (4 left) 2024-09-17T15:46:10.698 INFO:tasks.rados.rados.0.smithi203.stdout:3051: done (3 left) 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3052: done (2 left) 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3055: delete oid 41 current snap is 330 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: finishing write tid 2 to smithi20328086-32 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: finishing write tid 3 to smithi20328086-32 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: finishing write tid 4 to smithi20328086-32 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: finishing write tid 5 to smithi20328086-32 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: oid 32 updating version 0 to 1209 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: oid 32 version 1209 is already newer than 1207 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: oid 32 updating version 1209 to 1210 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: oid 32 version 1210 is already newer than 1208 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1210 (ObjNum 1068 snap 330 seq_num 1068) dirty exists 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3053: left oid 32 (ObjNum 1068 snap 330 seq_num 1068) 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3054: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:46:10.699 INFO:tasks.rados.rados.0.smithi203.stdout:3054: finishing copy_from to smithi20328086-11 2024-09-17T15:46:10.700 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1384 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3053: done (2 left) 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3054: done (1 left) 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3055: done (0 left) 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3056: copy_from oid 29 from oid 37 current snap is 330 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3057: copy_from oid 41 from oid 2 current snap is 330 2024-09-17T15:46:10.705 INFO:tasks.rados.rados.0.smithi203.stdout:3058: snap_remove snap 328 2024-09-17T15:46:10.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058474434}) 2024-09-17T15:46:10.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=213716779}) 2024-09-17T15:46:10.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:10.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058474434}) 2024-09-17T15:46:10.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:10.800+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703871181}) 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912736742}) 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3404365889}) 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912736742}) 2024-09-17T15:46:11.132 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.132+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289404632}) 2024-09-17T15:46:11.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060537982}) 2024-09-17T15:46:11.134 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276849114}) 2024-09-17T15:46:11.135 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.135 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1060537982}) 2024-09-17T15:46:11.135 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390622918}) 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105083751}) 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074328627}) 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105083751}) 2024-09-17T15:46:11.137 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.136+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=340297045}) 2024-09-17T15:46:11.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249354249}) 2024-09-17T15:46:11.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2144647917}) 2024-09-17T15:46:11.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.139 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249354249}) 2024-09-17T15:46:11.140 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.140+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2660796778}) 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407205772}) 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325439163}) 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1407205772}) 2024-09-17T15:46:11.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597523253}) 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3614216548}) 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837586914}) 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3614216548}) 2024-09-17T15:46:11.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.144+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952545967}) 2024-09-17T15:46:11.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360509431}) 2024-09-17T15:46:11.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1416052150}) 2024-09-17T15:46:11.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360509431}) 2024-09-17T15:46:11.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=868982078}) 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340257969}) 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851980973}) 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340257969}) 2024-09-17T15:46:11.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.148+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483799361}) 2024-09-17T15:46:11.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077402250}) 2024-09-17T15:46:11.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315302048}) 2024-09-17T15:46:11.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077402250}) 2024-09-17T15:46:11.152 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.152+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433615168}) 2024-09-17T15:46:11.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869474220}) 2024-09-17T15:46:11.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2142565118}) 2024-09-17T15:46:11.154 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.154 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869474220}) 2024-09-17T15:46:11.154 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1802572455}) 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240429518}) 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1348279705}) 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240429518}) 2024-09-17T15:46:11.156 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.156+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285728202}) 2024-09-17T15:46:11.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181902028}) 2024-09-17T15:46:11.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=161316749}) 2024-09-17T15:46:11.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181902028}) 2024-09-17T15:46:11.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047699964}) 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615258279}) 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602810173}) 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3615258279}) 2024-09-17T15:46:11.161 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.160+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2435760292}) 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755474227}) 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4047632505}) 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2755474227}) 2024-09-17T15:46:11.163 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328414511}) 2024-09-17T15:46:11.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414149652}) 2024-09-17T15:46:11.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1098171196}) 2024-09-17T15:46:11.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.164+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.165 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2414149652}) 2024-09-17T15:46:11.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1027188438}) 2024-09-17T15:46:11.167 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428899796}) 2024-09-17T15:46:11.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3138138488}) 2024-09-17T15:46:11.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428899796}) 2024-09-17T15:46:11.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.168+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1156728751}) 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853897881}) 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2449644999}) 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853897881}) 2024-09-17T15:46:11.170 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2393794578}) 2024-09-17T15:46:11.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.172 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009243878}) 2024-09-17T15:46:11.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=401037184}) 2024-09-17T15:46:11.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009243878}) 2024-09-17T15:46:11.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.172+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1471071020}) 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814865725}) 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341367544}) 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1814865725}) 2024-09-17T15:46:11.175 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788979958}) 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643591752}) 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2654611503}) 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.176+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643591752}) 2024-09-17T15:46:11.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2104520634}) 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250143921}) 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795382634}) 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2250143921}) 2024-09-17T15:46:11.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.180+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=74407993}) 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691599254}) 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521232542}) 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691599254}) 2024-09-17T15:46:11.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485710996}) 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996384168}) 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339976681}) 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996384168}) 2024-09-17T15:46:11.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.184+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3306212667}) 2024-09-17T15:46:11.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737125911}) 2024-09-17T15:46:11.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663123309}) 2024-09-17T15:46:11.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737125911}) 2024-09-17T15:46:11.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492833661}) 2024-09-17T15:46:11.188 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226955316}) 2024-09-17T15:46:11.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2188368426}) 2024-09-17T15:46:11.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226955316}) 2024-09-17T15:46:11.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.188+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=599784604}) 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493667233}) 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225605839}) 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493667233}) 2024-09-17T15:46:11.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.192+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871932050}) 2024-09-17T15:46:11.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640723651}) 2024-09-17T15:46:11.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3120312173}) 2024-09-17T15:46:11.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640723651}) 2024-09-17T15:46:11.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.196+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951840196}) 2024-09-17T15:46:11.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200955097}) 2024-09-17T15:46:11.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302878142}) 2024-09-17T15:46:11.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200955097}) 2024-09-17T15:46:11.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640067066}) 2024-09-17T15:46:11.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049147440}) 2024-09-17T15:46:11.205 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567155179}) 2024-09-17T15:46:11.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.204+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049147440}) 2024-09-17T15:46:11.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3131811316}) 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386563177}) 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286086100}) 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386563177}) 2024-09-17T15:46:11.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.208+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945685794}) 2024-09-17T15:46:11.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155018203}) 2024-09-17T15:46:11.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183720152}) 2024-09-17T15:46:11.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.211 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155018203}) 2024-09-17T15:46:11.211 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2189299518}) 2024-09-17T15:46:11.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581141532}) 2024-09-17T15:46:11.213 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2020435016}) 2024-09-17T15:46:11.213 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.213 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581141532}) 2024-09-17T15:46:11.213 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.212+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852386765}) 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287756708}) 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3875291741}) 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287756708}) 2024-09-17T15:46:11.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=751682348}) 2024-09-17T15:46:11.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141510149}) 2024-09-17T15:46:11.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.216+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2041059153}) 2024-09-17T15:46:11.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141510149}) 2024-09-17T15:46:11.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143320472}) 2024-09-17T15:46:11.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225580247}) 2024-09-17T15:46:11.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793309450}) 2024-09-17T15:46:11.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225580247}) 2024-09-17T15:46:11.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.220+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784612160}) 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379488258}) 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987270927}) 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3379488258}) 2024-09-17T15:46:11.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317723112}) 2024-09-17T15:46:11.224 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.224 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244501611}) 2024-09-17T15:46:11.224 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927888874}) 2024-09-17T15:46:11.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244501611}) 2024-09-17T15:46:11.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.224+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571795769}) 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265090474}) 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924565458}) 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265090474}) 2024-09-17T15:46:11.227 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168138468}) 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866019018}) 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=920319034}) 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866019018}) 2024-09-17T15:46:11.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.228+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515044336}) 2024-09-17T15:46:11.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101802516}) 2024-09-17T15:46:11.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1660117770}) 2024-09-17T15:46:11.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101802516}) 2024-09-17T15:46:11.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.232+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680871580}) 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=430555042,mdsmap=2815901553,mgr=333540546}) 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=89,mdsmap=3,mgr=7,mgr_command_descs=1} crc {logm=1206940350,mdsmap=2815901553,mgr=1115345915,mgr_command_descs=1770323759}) 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=430555042,mdsmap=2815901553,mgr=333540546}) 2024-09-17T15:46:11.234 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=2694826438,mdsmap=2815901553,mgr=1738947347}) 2024-09-17T15:46:11.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=619735488,mon_config_key=639531838,monmap=4191199434,nvmeofgw=3140514009,osd_metadata=3616902286,osd_pg_creating=407488823,osd_snap=4083605105}) 2024-09-17T15:46:11.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=57} crc {mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3079010025,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=407488823,osd_snap=3695179853}) 2024-09-17T15:46:11.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=619735488,mon_config_key=639531838,monmap=4191199434,nvmeofgw=3140514009,osd_metadata=3616902286,osd_pg_creating=407488823,osd_snap=4083605105}) 2024-09-17T15:46:11.238 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.236+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=59} crc {mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3079010025,mon_config_key=577667152,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=407488823,osd_snap=1638813126}) 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=69,osdmap=31} crc {osd_snap=1222694788,osdmap=2911647968}) 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=67,osdmap=33} crc {osd_snap=1188577173,osdmap=3131147609}) 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=69,osdmap=31} crc {osd_snap=1222694788,osdmap=2911647968}) 2024-09-17T15:46:11.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.240+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=66,osdmap=34} crc {osd_snap=119096522,osdmap=2176841373}) 2024-09-17T15:46:11.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1734249789}) 2024-09-17T15:46:11.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3407993909}) 2024-09-17T15:46:11.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1734249789}) 2024-09-17T15:46:11.242 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=883655133}) 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1261839293}) 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3684192723}) 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1261839293}) 2024-09-17T15:46:11.244 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.244+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2150332694}) 2024-09-17T15:46:11.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3194678033}) 2024-09-17T15:46:11.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=689537529}) 2024-09-17T15:46:11.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3194678033}) 2024-09-17T15:46:11.250 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.248+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=231582558}) 2024-09-17T15:46:11.258 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.258 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=214798625}) 2024-09-17T15:46:11.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=24} crc {osdmap=2099269714}) 2024-09-17T15:46:11.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:11.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=24} crc {osdmap=214798625}) 2024-09-17T15:46:11.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:11.260+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=22} crc {osdmap=3875382359}) 2024-09-17T15:46:11.339 INFO:tasks.rados.rados.0.smithi203.stdout:3058: done (2 left) 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3059: delete oid 38 current snap is 330 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3057: finishing copy_from to smithi20328086-41 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3057: got expected ENOENT (src dne) 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3057: finishing copy_from racing read to smithi20328086-41 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3056: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:3056: finishing copy_from to smithi20328086-29 2024-09-17T15:46:11.340 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1517 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3056: done (2 left) 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3057: done (1 left) 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3059: done (0 left) 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3060: rollback oid 1 current snap is 330 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 325 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3061: write (excl) oid 50 current snap is 330 2024-09-17T15:46:11.361 INFO:tasks.rados.rados.0.smithi203.stdout:3061: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-17T15:46:11.376 INFO:tasks.rados.rados.0.smithi203.stdout:3061: writing smithi20328086-50 from 400297 to 881396 tid 1 2024-09-17T15:46:11.395 INFO:tasks.rados.rados.0.smithi203.stdout:3061: writing smithi20328086-50 from 1639156 to 2000297 tid 2 2024-09-17T15:46:11.395 INFO:tasks.rados.rados.0.smithi203.stdout:3062: rollback oid 20 current snap is 330 2024-09-17T15:46:11.395 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 20 to 321 2024-09-17T15:46:11.395 INFO:tasks.rados.rados.0.smithi203.stdout:3060: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:46:11.396 INFO:tasks.rados.rados.0.smithi203.stdout:3060: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:46:11.396 INFO:tasks.rados.rados.0.smithi203.stdout:3060: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:46:11.396 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1332 (ObjNum 928 snap 283 seq_num 928) dirty exists 2024-09-17T15:46:11.397 INFO:tasks.rados.rados.0.smithi203.stdout:3061: finishing write tid 1 to smithi20328086-50 2024-09-17T15:46:11.397 INFO:tasks.rados.rados.0.smithi203.stdout:3060: done (2 left) 2024-09-17T15:46:11.397 INFO:tasks.rados.rados.0.smithi203.stdout:3063: snap_remove snap 329 2024-09-17T15:46:11.546 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:46:11.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:11.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: get_monmap_and_config 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: build_initial_monmap 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: monmap: 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:11.681363+0000 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:11.681363+0000 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:11.675 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:11.676 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:11.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 Processor -- start 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 -- start start 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: init 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec6c073fc0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec6c09bc00 0x7fec6c13d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c09bc00 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:11.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec6c074ab0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c09d210 con 0x7fec6c074ab0 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6c09c3c0 con 0x7fec6c073fc0 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fec6c09c6a0 con 0x7fec6c09bc00 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7337d640 10 monclient(hunting): authenticate will time out at 2104.646860s 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec6c09bc00 0x7fec6c13d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7191a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fec6c074ab0 0x7fec6c13f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39868/0 (socket says 172.21.15.86:39868) 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec7191a640 1 -- 172.21.15.86:0/1284482815 learned_addr learned my addr 172.21.15.86:0/1284482815 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): get_auth_request con 0x7fec6c09bc00 auth_method 0 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:11.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient(hunting): handle_auth_done global_id 214131 payload 274 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 1 -- 172.21.15.86:0/1284482815 >> v1:172.21.15.203:6789/0 conn(0x7fec6c073fc0 legacy=0x7fec6c0743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 1 -- 172.21.15.86:0/1284482815 >> v1:172.21.15.86:6789/0 conn(0x7fec6c074ab0 legacy=0x7fec6c13f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient: _finish_hunting 0 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 1 monclient: found mon.noname-a 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:11.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:11.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.676+0000 7fec70918640 1 -- 172.21.15.86:0/1284482815 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c09c980 con 0x7fec6c09bc00 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec70918640 1 -- 172.21.15.86:0/1284482815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 msgr2=0x7fec6c13d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec70918640 1 --2- 172.21.15.86:0/1284482815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c13d010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0097c0 tx=0x7fec5c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6e56cce2523b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 1 -- 172.21.15.86:0/1284482815 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0b0070 con 0x7fec6c09bc00 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: dump: 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:11.685 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 1 -- 172.21.15.86:0/1284482815 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec5c0ab600 con 0x7fec6c09bc00 2024-09-17T15:46:11.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec7337d640 5 monclient: authenticate success, global_id 214131 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec7337d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec7337d640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec7337d640 4 set_mon_vals no callback set 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 1 -- 172.21.15.86:0/1284482815 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0ab9d0 con 0x7fec6c09bc00 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.680+0000 7fec63fff640 10 monclient: dump: 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:11.687 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: shutdown 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- 172.21.15.86:0/1284482815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 msgr2=0x7fec6c13d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 --2- 172.21.15.86:0/1284482815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c13d010 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0097c0 tx=0x7fec5c01c360 comp rx=0 tx=0).stop 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- 172.21.15.86:0/1284482815 shutdown_connections 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 --2- 172.21.15.86:0/1284482815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c13d010 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- 172.21.15.86:0/1284482815 >> 172.21.15.86:0/1284482815 conn(0x7fec6c0a6440 msgr2=0x7fec6c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- 172.21.15.86:0/1284482815 shutdown_connections 2024-09-17T15:46:11.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- 172.21.15.86:0/1284482815 wait complete. 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: build_initial_monmap 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: monmap: 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:11.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 Processor -- start 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- start start 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: init 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c073fc0 0x7fec6c11fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-b con 0x7fec6c073fc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-c con 0x7fec6c074ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c1cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): picked mon.noname-a con 0x7fec6c09bc00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c143830 con 0x7fec6c09bc00 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec6c1436b0 con 0x7fec6c073fc0 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fec6c1439b0 con 0x7fec6c074ab0 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7337d640 10 monclient(hunting): authenticate will time out at 2104.653976s 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c1cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42814/0 (socket says 172.21.15.86:42814) 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 -- 172.21.15.86:0/2035768178 learned_addr learned my addr 172.21.15.86:0/2035768178 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): get_auth_request con 0x7fec6c074ab0 auth_method 0 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:11.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): get_auth_request con 0x7fec6c09bc00 auth_method 0 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:11.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec71119640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c073fc0 0x7fec6c11fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec71119640 10 monclient(hunting): get_auth_request con 0x7fec6c073fc0 auth_method 0 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec71119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec71119640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec71119640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient(hunting): handle_auth_done global_id 164648 payload 824 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 -- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c073fc0 msgr2=0x7fec6c11fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c073fc0 0x7fec6c11fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 -- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 msgr2=0x7fec6c1cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c1cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient: _finish_hunting 0 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 monclient: found mon.noname-c 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec70918640 1 -- 172.21.15.86:0/2035768178 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6c121590 con 0x7fec6c074ab0 2024-09-17T15:46:11.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.684+0000 7fec7191a640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c1cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:11.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec70918640 1 -- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 msgr2=0x7fec6c137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:11.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec70918640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0ae040 tx=0x7fec5c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0b0050 con 0x7fec6c074ab0 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: dump: 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:11.697 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec5c0bc070 con 0x7fec6c074ab0 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 5 monclient: authenticate success, global_id 164648 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _renew_subs 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec6c120720 con 0x7fec6c074ab0 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0b5c80 con 0x7fec6c074ab0 2024-09-17T15:46:11.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 10 monclient: dump: 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:11.699 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _renew_subs 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec637fe640 4 set_mon_vals no callback set 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec6c140110 con 0x7fec6c074ab0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec6c121150 con 0x7fec6c074ab0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fec5c0ab4c0 con 0x7fec6c074ab0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 0x7fec5003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(680..680 src has 438..680) ==== 4879+0+0 (secure 0 0 0) 0x7fec5c0ea8e0 con 0x7fec6c074ab0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec71119640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 0x7fec5003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:11.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.688+0000 7fec71119640 10 monclient: get_auth_request con 0x7fec5003bb90 auth_method 0 2024-09-17T15:46:11.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.692+0000 7fec71119640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 0x7fec5003e050 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fec68001dc0 tx=0x7fec68009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:11.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.692+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fec5c016070 con 0x7fec6c074ab0 2024-09-17T15:46:11.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.692+0000 7fec61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:11.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.692+0000 7fec61ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec6c074420 con 0x7fec6c074ab0 2024-09-17T15:46:11.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec61ffb640 1 -- 172.21.15.86:0/2035768178 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fec5c0c1c10 con 0x7fec6c074ab0 2024-09-17T15:46:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:11.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec61ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:11.927 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 msgr2=0x7fec5003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 0x7fec5003e050 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fec68001dc0 tx=0x7fec68009040 comp rx=0 tx=0).stop 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 10 monclient: shutdown 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 msgr2=0x7fec6c137c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:11.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fec5c0ae040 tx=0x7fec5c019100 comp rx=0 tx=0).stop 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 shutdown_connections 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fec5003bb90 0x7fec5003e050 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec6c09bc00 0x7fec6c1cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6c074ab0 0x7fec6c137c10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 --2- 172.21.15.86:0/2035768178 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c073fc0 0x7fec6c11fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 >> 172.21.15.86:0/2035768178 conn(0x7fec6c0a6440 msgr2=0x7fec6c0a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 shutdown_connections 2024-09-17T15:46:11.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:11.928+0000 7fec7337d640 1 -- 172.21.15.86:0/2035768178 wait complete. 2024-09-17T15:46:11.946 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:46:11.946 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:46:11.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:46:11.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: get_monmap_and_config 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: monmap: 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:12.068795+0000 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:12.068795+0000 2024-09-17T15:46:12.062 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:12.063 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 Processor -- start 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 -- start start 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: init 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:12.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7cf80a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7cf813d2f0 0x7f7cf813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf813d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf80a6280 con 0x7f7cf80a54c0 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf809d2a0 con 0x7f7cf813d2f0 2024-09-17T15:46:12.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf809c0a0 con 0x7f7cf80a50e0 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfe6ea640 10 monclient(hunting): authenticate will time out at 2105.034328s 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7ceffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7cf80a54c0 0x7f7cf80a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39870/0 (socket says 172.21.15.86:39870) 2024-09-17T15:46:12.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7ceffff640 1 -- 172.21.15.86:0/237099237 learned_addr learned my addr 172.21.15.86:0/237099237 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request con 0x7f7cf80a50e0 auth_method 0 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfcc87640 1 --2- 172.21.15.86:0/237099237 >> v2:172.21.15.203:3300/0 conn(0x7f7cf813d2f0 0x7f7cf813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfcc87640 10 monclient(hunting): get_auth_request con 0x7f7cf813d2f0 auth_method 0 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfcc87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfcc87640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cfcc87640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_done global_id 164651 payload 274 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 -- 172.21.15.86:0/237099237 >> v2:172.21.15.203:3300/0 conn(0x7f7cf813d2f0 msgr2=0x7f7cf813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 --2- 172.21.15.86:0/237099237 >> v2:172.21.15.203:3300/0 conn(0x7f7cf813d2f0 0x7f7cf813f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 -- 172.21.15.86:0/237099237 >> v1:172.21.15.86:6789/0 conn(0x7f7cf80a54c0 legacy=0x7f7cf80a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 monclient: found mon.noname-b 2024-09-17T15:46:12.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:12.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.064+0000 7f7cf7fff640 1 -- 172.21.15.86:0/237099237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf809c380 con 0x7f7cf80a50e0 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf7fff640 1 -- 172.21.15.86:0/237099237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 msgr2=0x7f7cf809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf7fff640 1 --2- 172.21.15.86:0/237099237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf809cd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4622c9841481a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 1 -- 172.21.15.86:0/237099237 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b0070 con 0x7f7cf80a50e0 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:12.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: dump: 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 1 -- 172.21.15.86:0/237099237 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80ab600 con 0x7f7cf80a50e0 2024-09-17T15:46:12.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cfe6ea640 5 monclient: authenticate success, global_id 164651 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cfe6ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cfe6ea640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cfe6ea640 4 set_mon_vals no callback set 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 1 -- 172.21.15.86:0/237099237 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80a4e60 con 0x7f7cf80a50e0 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.068+0000 7f7cf77fe640 10 monclient: dump: 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: shutdown 2024-09-17T15:46:12.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/237099237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 msgr2=0x7f7cf809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/237099237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf809cd60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80097c0 tx=0x7f7ce801c330 comp rx=0 tx=0).stop 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/237099237 shutdown_connections 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/237099237 >> v2:172.21.15.203:3300/0 conn(0x7f7cf813d2f0 0x7f7cf813f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/237099237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf809cd60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/237099237 >> 172.21.15.86:0/237099237 conn(0x7f7cf80aaae0 msgr2=0x7f7cf80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/237099237 shutdown_connections 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/237099237 wait complete. 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: monmap: 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.085 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 Processor -- start 2024-09-17T15:46:12.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- start start 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: init 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 0x7f7cf8134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf80a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cf813d2f0 0x7f7cf80a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8142650 con 0x7f7cf80a54c0 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf80a6b00 con 0x7f7cf813d2f0 2024-09-17T15:46:12.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf81427d0 con 0x7f7cf80a50e0 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfe6ea640 10 monclient(hunting): authenticate will time out at 2105.042200s 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 0x7f7cf8134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf81340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42842/0 (socket says 172.21.15.86:42842) 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 0x7f7cf8134600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43058/0 (socket says 172.21.15.86:43058) 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 -- 172.21.15.86:0/1786598196 learned_addr learned my addr 172.21.15.86:0/1786598196 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request con 0x7f7cf80a50e0 auth_method 0 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): get_auth_request con 0x7f7cf80a54c0 auth_method 0 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cfcc87640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cf813d2f0 0x7f7cf80a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_done global_id 164654 payload 824 2024-09-17T15:46:12.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cf813d2f0 msgr2=0x7f7cf80a1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cf813d2f0 0x7f7cf80a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 msgr2=0x7f7cf8134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 0x7f7cf8134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 monclient: found mon.noname-c 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7cf7fff640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf8139b50 con 0x7f7cf80a50e0 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.072+0000 7f7ceffff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a54c0 0x7f7cf8134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf7fff640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 msgr2=0x7f7cf81340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf7fff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf81340c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7ce801a9e0 tx=0x7f7ce80190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b9070 con 0x7f7cf80a50e0 2024-09-17T15:46:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: dump: 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80b5690 con 0x7f7cf80a50e0 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b0040 con 0x7f7cf80a50e0 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 5 monclient: authenticate success, global_id 164654 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 monclient: _renew_subs 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf8139e30 con 0x7f7cf80a50e0 2024-09-17T15:46:12.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf67fc640 4 set_mon_vals no callback set 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 10 monclient: dump: 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 monclient: _renew_subs 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf80a18d0 con 0x7f7cf80a50e0 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 0x7f7cf80a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf806a0e0 con 0x7f7cf80a2460 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 0x7f7cf80a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7ce80bc070 con 0x7f7cf80a50e0 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient: get_auth_request con 0x7f7cf80a2460 auth_method 0 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 10 monclient(hunting): handle_auth_done global_id 214140 payload 824 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd803bbb0 0x7f7cd803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7ceffff640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cf813da90 con 0x7f7cf80a2460 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(680..680 src has 438..680) ==== 4879+0+0 (secure 0 0 0) 0x7f7ce80ea160 con 0x7f7cf80a50e0 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfcc87640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd803bbb0 0x7f7cd803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.076+0000 7f7cfcc87640 10 monclient: get_auth_request con 0x7f7cd803bbb0 auth_method 0 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7ceffff640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 msgr2=0x7f7cf80a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7ceffff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 0x7f7cf80a2840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0007270 tx=0x7f7ce0007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce000fce0 con 0x7f7cf80a2460 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 10 monclient: dump: 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ce000fce0 con 0x7f7cf80a2460 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 msgr2=0x7f7cf80a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 0x7f7cf80a2840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0007270 tx=0x7f7ce0007730 comp rx=0 tx=0).stop 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 msgr2=0x7f7cf80a2840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cf57fa640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2460 0x7f7cf80a2840 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.080+0000 7f7cfcc87640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd803bbb0 0x7f7cd803e070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7cf813a850 tx=0x7f7cf000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7cfe6ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:12.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 0x7f7cf809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7cfe6ea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf813a200 con 0x7f7cf80a2d80 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 0x7f7cf809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient: get_auth_request con 0x7f7cf80a2d80 auth_method 0 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.084+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7ceffff640 10 monclient(hunting): handle_auth_done global_id 214143 payload 824 2024-09-17T15:46:12.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7ceffff640 1 -- 172.21.15.86:0/1786598196 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7cf813d6d0 con 0x7f7cf80a2d80 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7ceffff640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 msgr2=0x7f7cf809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7ceffff640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 0x7f7cf809e5e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0017000 tx=0x7f7ce0004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0006450 con 0x7f7cf80a2d80 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 10 monclient: dump: 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.102 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f7ce0006450 con 0x7f7cf80a2d80 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 msgr2=0x7f7cf809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 0x7f7cf809e5e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0017000 tx=0x7f7ce0004950 comp rx=0 tx=0).stop 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 msgr2=0x7f7cf809e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cf57fa640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7cf80a2d80 0x7f7cf809e5e0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0017000 tx=0x7f7ce0004950 comp rx=0 tx=0).stop 2024-09-17T15:46:12.103 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28263, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:46:12.104 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.105 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.106 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:12.107 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.108 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:12.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.111 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.113 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.114 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.115 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:12.116 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd803bbb0 msgr2=0x7f7cd803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7cd803bbb0 0x7f7cd803e070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7cf813a850 tx=0x7f7cf000b040 comp rx=0 tx=0).stop 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 10 monclient: shutdown 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 msgr2=0x7f7cf81340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 --2- 172.21.15.86:0/1786598196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7cf80a50e0 0x7f7cf81340c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7ce801a9e0 tx=0x7f7ce80190d0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfcc87640 1 -- 172.21.15.86:0/1786598196 reap_dead start 2024-09-17T15:46:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfcc87640 1 -- 172.21.15.86:0/1786598196 reap_dead start 2024-09-17T15:46:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 shutdown_connections 2024-09-17T15:46:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 >> 172.21.15.86:0/1786598196 conn(0x7f7cf80aaae0 msgr2=0x7f7cf813c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 shutdown_connections 2024-09-17T15:46:12.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.088+0000 7f7cfe6ea640 1 -- 172.21.15.86:0/1786598196 wait complete. 2024-09-17T15:46:12.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:12.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:46:12.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:12.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: get_monmap_and_config 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: monmap: 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:12.240282+0000 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:12.240282+0000 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:12.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:12.235 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:12.235 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:12.235 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 Processor -- start 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 -- start start 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: init 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22d40a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f22d40a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f22d413e190 0x7f22d4140580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-c con 0x7f22d413e190 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d40a68d0 con 0x7f22d40a4f70 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22d40a64a0 con 0x7f22d40a5350 2024-09-17T15:46:12.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f22d409c7f0 con 0x7f22d413e190 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d974c640 10 monclient(hunting): authenticate will time out at 2105.205806s 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f22d40a4f70 0x7f22d409cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39884/0 (socket says 172.21.15.86:39884) 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d2ffd640 1 -- 172.21.15.86:0/2657780404 learned_addr learned my addr 172.21.15.86:0/2657780404 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 1 --2- 172.21.15.86:0/2657780404 >> v2:172.21.15.203:3300/0 conn(0x7f22d413e190 0x7f22d4140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): get_auth_request con 0x7f22d413e190 auth_method 0 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient(hunting): handle_auth_done global_id 204199 payload 274 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 1 -- 172.21.15.86:0/2657780404 >> v1:172.21.15.203:6789/0 conn(0x7f22d40a5350 legacy=0x7f22d40a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 1 -- 172.21.15.86:0/2657780404 >> v1:172.21.15.86:6789/0 conn(0x7f22d40a4f70 legacy=0x7f22d409cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 1 monclient: found mon.noname-c 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:46:12.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.236+0000 7f22d37fe640 1 -- 172.21.15.86:0/2657780404 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d409c230 con 0x7f22d413e190 2024-09-17T15:46:12.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d37fe640 1 -- 172.21.15.86:0/2657780404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d413e190 msgr2=0x7f22d4140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d37fe640 1 --2- 172.21.15.86:0/2657780404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d413e190 0x7f22d4140580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22c40097c0 tx=0x7f22c401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7073f4485787720d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 1 -- 172.21.15.86:0/2657780404 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c40b0070 con 0x7f22d413e190 2024-09-17T15:46:12.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: dump: 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.253 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 1 -- 172.21.15.86:0/2657780404 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c40ab600 con 0x7f22d413e190 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 1 -- 172.21.15.86:0/2657780404 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c40ab9d0 con 0x7f22d413e190 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d1ffb640 10 monclient: dump: 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 5 monclient: authenticate success, global_id 204199 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 4 set_mon_vals no callback set 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 10 monclient: shutdown 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- 172.21.15.86:0/2657780404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d413e190 msgr2=0x7f22d4140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 --2- 172.21.15.86:0/2657780404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d413e190 0x7f22d4140580 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22c40097c0 tx=0x7f22c401c330 comp rx=0 tx=0).stop 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- 172.21.15.86:0/2657780404 shutdown_connections 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 --2- 172.21.15.86:0/2657780404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d413e190 0x7f22d4140580 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- 172.21.15.86:0/2657780404 >> 172.21.15.86:0/2657780404 conn(0x7f22d40a7240 msgr2=0x7f22d40a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- 172.21.15.86:0/2657780404 shutdown_connections 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- 172.21.15.86:0/2657780404 wait complete. 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 10 monclient: monmap: 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:12.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 Processor -- start 2024-09-17T15:46:12.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 1 -- start start 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.244+0000 7f22d974c640 10 monclient: init 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 0x7f22d4138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-a con 0x7f22d40a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d40a5350 0x7f22d409f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-b con 0x7f22d40a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d413e190 0x7f22d409fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): picked mon.noname-c con 0x7f22d413e190 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4143b80 con 0x7f22d40a4f70 2024-09-17T15:46:12.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f22d4143a00 con 0x7f22d40a5350 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f22d4143d00 con 0x7f22d413e190 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient(hunting): authenticate will time out at 2105.216265s 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 0x7f22d4138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 0x7f22d4138750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43088/0 (socket says 172.21.15.86:43088) 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 -- 172.21.15.86:0/426980573 learned_addr learned my addr 172.21.15.86:0/426980573 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): get_auth_request con 0x7f22d40a4f70 auth_method 0 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d413e190 0x7f22d409fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): get_auth_request con 0x7f22d413e190 auth_method 0 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d27fc640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d40a5350 0x7f22d409f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d27fc640 10 monclient(hunting): get_auth_request con 0x7f22d40a5350 auth_method 0 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient(hunting): handle_auth_done global_id 214146 payload 824 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d413e190 msgr2=0x7f22d409fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d413e190 0x7f22d409fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d40a5350 msgr2=0x7f22d409f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f22d40a5350 0x7f22d409f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4137010 con 0x7f22d40a4f70 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 msgr2=0x7f22d4138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d2ffd640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 0x7f22d4138750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00cc10 tx=0x7f22cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc012400 con 0x7f22d40a4f70 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: dump: 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22cc0125e0 con 0x7f22d40a4f70 2024-09-17T15:46:12.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 5 monclient: authenticate success, global_id 214146 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _renew_subs 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22d4135af0 con 0x7f22d40a4f70 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc0129d0 con 0x7f22d40a4f70 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22cb7fe640 10 monclient: dump: 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.264 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d17fa640 4 set_mon_vals no callback set 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _renew_subs 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.248+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d40a1760 con 0x7f22d40a4f70 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d974c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d974c640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 0x7f22d40a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f22d40a07a0 con 0x7f22d40a03c0 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 0x7f22d40a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient: get_auth_request con 0x7f22d40a03c0 auth_method 0 2024-09-17T15:46:12.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f22cc012bf0 con 0x7f22d40a4f70 2024-09-17T15:46:12.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 10 monclient(hunting): handle_auth_done global_id 164660 payload 824 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22d409cef0 con 0x7f22d40a03c0 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f22b403bb60 0x7f22b403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(680..680 src has 438..680) ==== 4879+0+0 (secure 0 0 0) 0x7f22cc0506f0 con 0x7f22d40a4f70 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d37fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f22b403bb60 0x7f22b403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d37fe640 10 monclient: get_auth_request con 0x7f22b403bb60 auth_method 0 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 msgr2=0x7f22d40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d27fc640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 0x7f22d40a1930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc00e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc007450 con 0x7f22d40a03c0 2024-09-17T15:46:12.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 10 monclient: dump: 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.269 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22bc018070 con 0x7f22d40a03c0 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 msgr2=0x7f22d40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 0x7f22d40a1930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc00e8b0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 msgr2=0x7f22d40a1930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22cb7fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a03c0 0x7f22d40a1930 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc00e8b0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.252+0000 7f22d37fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f22b403bb60 0x7f22b403e020 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f22c40ab000 tx=0x7f22c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d974c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d974c640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 0x7f22d40a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d974c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f22d40a5ff0 con 0x7f22d40a1e70 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 0x7f22d40a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient: get_auth_request con 0x7f22d40a1e70 auth_method 0 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 10 monclient(hunting): handle_auth_done global_id 164663 payload 824 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.260+0000 7f22d27fc640 1 -- 172.21.15.86:0/426980573 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22d4136be0 con 0x7f22d40a1e70 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d27fc640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 msgr2=0x7f22d40a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d27fc640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 0x7f22d40a2280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc01d9c0 con 0x7f22d40a1e70 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 10 monclient: dump: 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.274 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f22bc01d9c0 con 0x7f22d40a1e70 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:12.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 msgr2=0x7f22d40a2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 0x7f22d40a2280 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc003770 comp rx=0 tx=0).stop 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 msgr2=0x7f22d40a2280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22cb7fe640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f22d40a1e70 0x7f22d40a2280 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f22d4137500 tx=0x7f22bc003770 comp rx=0 tx=0).stop 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:12.276 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 275451, 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:12.277 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.278 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.279 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.280 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:12.281 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.282 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:12.283 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.284 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.286 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:12.287 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.291 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:12.292 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:12.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f22b403bb60 msgr2=0x7f22b403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f22b403bb60 0x7f22b403e020 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f22c40ab000 tx=0x7f22c4008040 comp rx=0 tx=0).stop 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 10 monclient: shutdown 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 msgr2=0x7f22d4138750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 --2- 172.21.15.86:0/426980573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22d40a4f70 0x7f22d4138750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00cc10 tx=0x7f22cc00a430 comp rx=0 tx=0).stop 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d37fe640 1 -- 172.21.15.86:0/426980573 reap_dead start 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d37fe640 1 -- 172.21.15.86:0/426980573 reap_dead start 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 shutdown_connections 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 >> 172.21.15.86:0/426980573 conn(0x7f22d40a7240 msgr2=0x7f22d413d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 shutdown_connections 2024-09-17T15:46:12.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.264+0000 7f22d974c640 1 -- 172.21.15.86:0/426980573 wait complete. 2024-09-17T15:46:12.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:12.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:46:12.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:12.368 INFO:tasks.rados.rados.0.smithi203.stdout:3063: done (2 left) 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3064: rollback oid 4 current snap is 330 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3062: finishing rollback tid 0 to smithi20328086-20 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3062: finishing rollback tid 1 to smithi20328086-20 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3062: finishing rollback tid 2 to smithi20328086-20 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1335 (ObjNum 588 snap 174 seq_num 588) dirty exists 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: finishing write tid 2 to smithi20328086-50 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: finishing write tid 3 to smithi20328086-50 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: finishing write tid 4 to smithi20328086-50 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: oid 50 updating version 0 to 1211 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: oid 50 updating version 1211 to 1213 2024-09-17T15:46:12.369 INFO:tasks.rados.rados.0.smithi203.stdout:3061: oid 50 version 1213 is already newer than 1212 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1213 (ObjNum 1071 snap 330 seq_num 1071) dirty exists 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:3061: left oid 50 (ObjNum 1071 snap 330 seq_num 1071) 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 325 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:3061: done (2 left) 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:3062: done (1 left) 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:3065: write oid 6 current snap is 330 2024-09-17T15:46:12.370 INFO:tasks.rados.rados.0.smithi203.stdout:3065: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-09-17T15:46:12.386 INFO:tasks.rados.rados.0.smithi203.stdout:3065: writing smithi20328086-6 from 417104 to 1089805 tid 1 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3065: writing smithi20328086-6 from 1813991 to 2017104 tid 2 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3066: snap_create 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3064: finishing rollback tid 0 to smithi20328086-4 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3064: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3064: finishing rollback tid 2 to smithi20328086-4 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1320 (ObjNum 1038 snap 321 seq_num 1038) dirty exists 2024-09-17T15:46:12.400 INFO:tasks.rados.rados.0.smithi203.stdout:3065: finishing write tid 1 to smithi20328086-6 2024-09-17T15:46:12.407 INFO:tasks.rados.rados.0.smithi203.stdout:3065: finishing write tid 2 to smithi20328086-6 2024-09-17T15:46:12.407 INFO:tasks.rados.rados.0.smithi203.stdout:3065: finishing write tid 3 to smithi20328086-6 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:3065: finishing write tid 4 to smithi20328086-6 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:3065: oid 6 updating version 0 to 1336 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:3065: oid 6 updating version 1336 to 1338 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:3065: oid 6 version 1338 is already newer than 1337 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1338 (ObjNum 1072 snap 330 seq_num 1072) dirty exists 2024-09-17T15:46:12.408 INFO:tasks.rados.rados.0.smithi203.stdout:3065: left oid 6 (ObjNum 1072 snap 330 seq_num 1072) 2024-09-17T15:46:12.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: get_monmap_and_config 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: monmap: 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:12.422448+0000 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:12.422448+0000 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:12.417 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:12.417 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 Processor -- start 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 -- start start 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: init 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff6b009d770 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff6b009e260 0x7ff6b0139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-c con 0x7ff6b009e260 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff6b013a440 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b009ff00 con 0x7ff6b009d770 2024-09-17T15:46:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b009ec30 con 0x7ff6b013a440 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff6b009e7f0 con 0x7ff6b009e260 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.416+0000 7ff6b4899640 10 monclient(hunting): authenticate will time out at 2105.387978s 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6ae59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff6b009d770 0x7ff6b009db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39898/0 (socket says 172.21.15.86:39898) 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6ae59c640 1 -- 172.21.15.86:0/2953620098 learned_addr learned my addr 172.21.15.86:0/2953620098 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 1 --2- 172.21.15.86:0/2953620098 >> v2:172.21.15.203:3300/0 conn(0x7ff6b009e260 0x7ff6b0139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): get_auth_request con 0x7ff6b009e260 auth_method 0 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient(hunting): handle_auth_done global_id 204205 payload 274 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 1 -- 172.21.15.86:0/2953620098 >> v1:172.21.15.203:6789/0 conn(0x7ff6b013a440 legacy=0x7ff6b013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 1 -- 172.21.15.86:0/2953620098 >> v1:172.21.15.86:6789/0 conn(0x7ff6b009d770 legacy=0x7ff6b009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 1 monclient: found mon.noname-c 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:46:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.420+0000 7ff6add9b640 1 -- 172.21.15.86:0/2953620098 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b009ef10 con 0x7ff6b009e260 2024-09-17T15:46:12.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6add9b640 1 -- 172.21.15.86:0/2953620098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b009e260 msgr2=0x7ff6b0139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6add9b640 1 --2- 172.21.15.86:0/2953620098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b009e260 0x7ff6b0139f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009d90 tx=0x7ff6a401c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de85454a04fe5a21 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 1 -- 172.21.15.86:0/2953620098 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40ad070 con 0x7ff6b009e260 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: dump: 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.423 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 1 -- 172.21.15.86:0/2953620098 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a40a9750 con 0x7ff6b009e260 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 1 -- 172.21.15.86:0/2953620098 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40a9b20 con 0x7ff6b009e260 2024-09-17T15:46:12.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6ad59a640 10 monclient: dump: 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 5 monclient: authenticate success, global_id 204205 2024-09-17T15:46:12.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:12.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 4 set_mon_vals no callback set 2024-09-17T15:46:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 10 monclient: shutdown 2024-09-17T15:46:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 -- 172.21.15.86:0/2953620098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b009e260 msgr2=0x7ff6b0139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 --2- 172.21.15.86:0/2953620098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b009e260 0x7ff6b0139f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009d90 tx=0x7ff6a401c830 comp rx=0 tx=0).stop 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 -- 172.21.15.86:0/2953620098 shutdown_connections 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 --2- 172.21.15.86:0/2953620098 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b009e260 0x7ff6b0139f00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 -- 172.21.15.86:0/2953620098 >> 172.21.15.86:0/2953620098 conn(0x7ff6b00a2f40 msgr2=0x7ff6b00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 -- 172.21.15.86:0/2953620098 shutdown_connections 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 -- 172.21.15.86:0/2953620098 wait complete. 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 10 monclient: monmap: 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.424+0000 7ff6b4899640 1 Processor -- start 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- start start 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: init 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 0x7ff6b00a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-c con 0x7ff6b009d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:12.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 0x7ff6b0134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-a con 0x7ff6b009e260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b013a440 0x7ff6b01353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): picked mon.noname-b con 0x7ff6b013a440 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b01408f0 con 0x7ff6b009e260 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b0140770 con 0x7ff6b013a440 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b0140a70 con 0x7ff6b009d770 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient(hunting): authenticate will time out at 2105.396637s 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 0x7ff6b0134e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 0x7ff6b00a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 0x7ff6b0134e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43096/0 (socket says 172.21.15.86:43096) 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 0x7ff6b00a0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42856/0 (socket says 172.21.15.86:42856) 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 -- 172.21.15.86:0/1125710188 learned_addr learned my addr 172.21.15.86:0/1125710188 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): get_auth_request con 0x7ff6b009e260 auth_method 0 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): get_auth_request con 0x7ff6b009d770 auth_method 0 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b013a440 0x7ff6b01353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): get_auth_request con 0x7ff6b013a440 auth_method 0 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient(hunting): handle_auth_done global_id 214149 payload 824 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 msgr2=0x7ff6b00a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 0x7ff6b00a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b013a440 msgr2=0x7ff6b01353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b013a440 0x7ff6b01353a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 monclient: found mon.noname-a 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b0072b80 con 0x7ff6b009e260 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6b009d770 0x7ff6b00a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 msgr2=0x7ff6b0134e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6add9b640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 0x7ff6b0134e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6a401d0d0 tx=0x7ff6a40195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40ad040 con 0x7ff6b009e260 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:12.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: dump: 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a40b2a70 con 0x7ff6b009e260 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 5 monclient: authenticate success, global_id 214149 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a40b2e20 con 0x7ff6b009e260 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _renew_subs 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b0121830 con 0x7ff6b009e260 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6acd99640 4 set_mon_vals no callback set 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff69effd640 10 monclient: dump: 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.439 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _renew_subs 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.428+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b0120cd0 con 0x7ff6b009e260 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6b4899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6b4899640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 0x7ff6b01d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b009db50 con 0x7ff6b01cfc30 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff6a40b0070 con 0x7ff6b009e260 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 0x7ff6b01d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient: get_auth_request con 0x7ff6b01cfc30 auth_method 0 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff69effd640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff69003bb90 0x7ff69003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(681..681 src has 438..681) ==== 4879+0+0 (secure 0 0 0) 0x7ff6a40eb1d0 con 0x7ff6b009e260 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6aed9d640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff69003bb90 0x7ff69003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6aed9d640 10 monclient: get_auth_request con 0x7ff69003bb90 auth_method 0 2024-09-17T15:46:12.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_done global_id 204211 payload 824 2024-09-17T15:46:12.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.432+0000 7ff6ae59c640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6b0121090 con 0x7ff6b01cfc30 2024-09-17T15:46:12.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff6ae59c640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 msgr2=0x7ff6b01d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 0x7ff6b01d0010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a800fce0 con 0x7ff6b01cfc30 2024-09-17T15:46:12.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 10 monclient: dump: 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff6aed9d640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff69003bb90 0x7ff69003e050 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00a2c80 tx=0x7ff6a0007d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6a800fce0 con 0x7ff6b01cfc30 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 msgr2=0x7ff6b01d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 0x7ff6b01d0010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8007760 comp rx=0 tx=0).stop 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 msgr2=0x7ff6b01d0010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.436+0000 7ff69effd640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01cfc30 0x7ff6b01d0010 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8007760 comp rx=0 tx=0).stop 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6b4899640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6b4899640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 0x7ff6b0137ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6b4899640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b006d4f0 con 0x7ff6b01d0550 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 0x7ff6b0137ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient: get_auth_request con 0x7ff6b01d0550 auth_method 0 2024-09-17T15:46:12.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 10 monclient(hunting): handle_auth_done global_id 204214 payload 824 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.440+0000 7ff6ae59c640 1 -- 172.21.15.86:0/1125710188 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6b01d0960 con 0x7ff6b01d0550 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6ae59c640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 msgr2=0x7ff6b0137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6ae59c640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 0x7ff6b0137ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8006030 con 0x7ff6b01d0550 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 10 monclient: dump: 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7ff6a8006030 con 0x7ff6b01d0550 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 msgr2=0x7ff6b0137ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 0x7ff6b0137ef0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8004950 comp rx=0 tx=0).stop 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 msgr2=0x7ff6b0137ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:12.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff69effd640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6b01d0550 0x7ff6b0137ef0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80072a0 tx=0x7ff6a8004950 comp rx=0 tx=0).stop 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 178, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 75045, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:46:12.449 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.450 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.451 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:12.452 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:12.453 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.454 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.455 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:12.456 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.457 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:12.458 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:12.459 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.460 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:12.461 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff69003bb90 msgr2=0x7ff69003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff69003bb90 0x7ff69003e050 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00a2c80 tx=0x7ff6a0007d10 comp rx=0 tx=0).stop 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 10 monclient: shutdown 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 msgr2=0x7ff6b0134e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6b4899640 1 --2- 172.21.15.86:0/1125710188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6b009e260 0x7ff6b0134e60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff6a401d0d0 tx=0x7ff6a40195d0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.444+0000 7ff6aed9d640 1 -- 172.21.15.86:0/1125710188 reap_dead start 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.448+0000 7ff6aed9d640 1 -- 172.21.15.86:0/1125710188 reap_dead start 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.448+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 shutdown_connections 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.448+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 >> 172.21.15.86:0/1125710188 conn(0x7ff6b00a2f40 msgr2=0x7ff6b013d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.448+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 shutdown_connections 2024-09-17T15:46:12.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.448+0000 7ff6b4899640 1 -- 172.21.15.86:0/1125710188 wait complete. 2024-09-17T15:46:12.464 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:12.464 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:46:12.464 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:46:12.464 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:46:12.464 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:46:12.465 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:46:12.465 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:46:12.479 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:46:12.480 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:46:12.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:12.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: get_monmap_and_config 2024-09-17T15:46:12.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: monmap: 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:12.601201+0000 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:12.601201+0000 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:12.595 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:12.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 Processor -- start 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 -- start start 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: init 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83740689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83740694c0 0x7f837406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-a con 0x7f83740694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:12.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f837413aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f837409c8b0 con 0x7f837413aff0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f837409cb90 con 0x7f83740689d0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8374074500 con 0x7f83740694c0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837c2d8640 10 monclient(hunting): authenticate will time out at 2105.566703s 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83740694c0 0x7f837406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837a875640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f837413aff0 0x7f837413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39906/0 (socket says 172.21.15.86:39906) 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837a875640 1 -- 172.21.15.86:0/3026842655 learned_addr learned my addr 172.21.15.86:0/3026842655 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f837a074640 1 -- 172.21.15.86:0/3026842655 >> v1:172.21.15.203:6789/0 conn(0x7f83740689d0 legacy=0x7f8374068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): get_auth_request con 0x7f83740694c0 auth_method 0 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient(hunting): handle_auth_done global_id 214152 payload 274 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 1 -- 172.21.15.86:0/3026842655 >> v1:172.21.15.203:6789/0 conn(0x7f83740689d0 legacy=0x7f8374068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 1 -- 172.21.15.86:0/3026842655 >> v1:172.21.15.86:6789/0 conn(0x7f837413aff0 legacy=0x7f837413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 1 monclient: found mon.noname-a 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.596+0000 7f8379873640 1 -- 172.21.15.86:0/3026842655 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837409ce70 con 0x7f83740694c0 2024-09-17T15:46:12.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379873640 1 -- 172.21.15.86:0/3026842655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 msgr2=0x7f837406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379873640 1 --2- 172.21.15.86:0/3026842655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f837406f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8374069a70 tx=0x7f837001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2626f622773777e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 1 -- 172.21.15.86:0/3026842655 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700b2070 con 0x7f83740694c0 2024-09-17T15:46:12.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: dump: 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.603 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 1 -- 172.21.15.86:0/3026842655 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83700a5930 con 0x7f83740694c0 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f837c2d8640 5 monclient: authenticate success, global_id 214152 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f837c2d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 1 -- 172.21.15.86:0/3026842655 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837409c2d0 con 0x7f83740694c0 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f837c2d8640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f837c2d8640 4 set_mon_vals no callback set 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.600+0000 7f8379072640 10 monclient: dump: 2024-09-17T15:46:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: shutdown 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- 172.21.15.86:0/3026842655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 msgr2=0x7f837406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 --2- 172.21.15.86:0/3026842655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f837406f840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8374069a70 tx=0x7f837001c7f0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- 172.21.15.86:0/3026842655 shutdown_connections 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 --2- 172.21.15.86:0/3026842655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f837406f840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- 172.21.15.86:0/3026842655 >> 172.21.15.86:0/3026842655 conn(0x7f83740a6440 msgr2=0x7f83740a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- 172.21.15.86:0/3026842655 shutdown_connections 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- 172.21.15.86:0/3026842655 wait complete. 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: build_initial_monmap 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: monmap: 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:12.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 Processor -- start 2024-09-17T15:46:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- start start 2024-09-17T15:46:12.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: init 2024-09-17T15:46:12.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:12.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:12.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83740689d0 0x7f83741d0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-c con 0x7f83740689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-a con 0x7f83740694c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 0x7f8374070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): picked mon.noname-b con 0x7f837413aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8374141c30 con 0x7f83740694c0 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8374141ab0 con 0x7f837413aff0 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8374141db0 con 0x7f83740689d0 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837c2d8640 10 monclient(hunting): authenticate will time out at 2105.573930s 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83740689d0 0x7f83741d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43112/0 (socket says 172.21.15.86:43112) 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 -- 172.21.15.86:0/67243885 learned_addr learned my addr 172.21.15.86:0/67243885 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): get_auth_request con 0x7f83740689d0 auth_method 0 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a875640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 msgr2=0x7f8374070aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a875640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 0x7f8374070aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): get_auth_request con 0x7f83740694c0 auth_method 0 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:12.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f837a074640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient(hunting): handle_auth_done global_id 214155 payload 824 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83740689d0 msgr2=0x7f83741d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83740689d0 0x7f83741d0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 msgr2=0x7f8374070aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 0x7f8374070aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient: _finish_hunting 0 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 monclient: found mon.noname-a 2024-09-17T15:46:12.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.604+0000 7f8379873640 1 -- 172.21.15.86:0/67243885 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8374121610 con 0x7f83740694c0 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8379873640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 msgr2=0x7f8374135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8379873640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83700b0040 tx=0x7f8370019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700bd020 con 0x7f83740694c0 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: dump: 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.614 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 5 monclient: authenticate success, global_id 214155 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _renew_subs 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837409f5b0 con 0x7f83740694c0 2024-09-17T15:46:12.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83700b8880 con 0x7f83740694c0 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83700b2040 con 0x7f83740694c0 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 10 monclient: dump: 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.616 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _renew_subs 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8378871640 4 set_mon_vals no callback set 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8374121330 con 0x7f83740694c0 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f837406f840 con 0x7f83740694c0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f83700c0020 con 0x7f83740694c0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 0x7f835803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(681..681 src has 438..681) ==== 4879+0+0 (secure 0 0 0) 0x7f83700eb040 con 0x7f83740694c0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837a074640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 0x7f835803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.608+0000 7f837a074640 10 monclient: get_auth_request con 0x7f835803bb10 auth_method 0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.612+0000 7f837a074640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 0x7f835803dfd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8368009380 tx=0x7f8368009310 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:12.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.612+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f83700c7030 con 0x7f83740694c0 2024-09-17T15:46:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.612+0000 7f8366ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:12.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.612+0000 7f8366ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.844+0000 7f837c2d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.844+0000 7f837c2d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.844+0000 7f837c2d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:12.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.844+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8374120fb0 con 0x7f83740694c0 2024-09-17T15:46:12.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f8366ffd640 1 -- 172.21.15.86:0/67243885 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8370016500 con 0x7f83740694c0 2024-09-17T15:46:12.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f8366ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:12.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f8366ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:12.846 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 msgr2=0x7f835803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 0x7f835803dfd0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8368009380 tx=0x7f8368009310 comp rx=0 tx=0).stop 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 10 monclient: shutdown 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 msgr2=0x7f8374135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:12.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83700b0040 tx=0x7f8370019590 comp rx=0 tx=0).stop 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 shutdown_connections 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f835803bb10 0x7f835803dfd0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f837413aff0 0x7f8374070aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83740694c0 0x7f8374135c70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 --2- 172.21.15.86:0/67243885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83740689d0 0x7f83741d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 >> 172.21.15.86:0/67243885 conn(0x7f83740a6440 msgr2=0x7f837409d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 shutdown_connections 2024-09-17T15:46:12.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:12.848+0000 7f837c2d8640 1 -- 172.21.15.86:0/67243885 wait complete. 2024-09-17T15:46:15.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: get_monmap_and_config 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: build_initial_monmap 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: monmap: 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:16.001514+0000 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:16.001514+0000 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:15.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:15.996 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 Processor -- start 2024-09-17T15:46:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 -- start start 2024-09-17T15:46:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: init 2024-09-17T15:46:15.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd428073fc0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 0x7fd42813d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-c con 0x7fd42809bc00 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-b con 0x7fd428074ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42809d210 con 0x7fd428073fc0 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd42809fbe0 con 0x7fd42809bc00 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd42809c3c0 con 0x7fd428074ab0 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:15.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42f667640 10 monclient(hunting): authenticate will time out at 2108.967064s 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42cc02640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 msgr2=0x7fd42813d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42d403640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd428073fc0 legacy=0x7fd4280743a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42cc02640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 0x7fd42813d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42874/0 (socket says 172.21.15.86:42874) 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 -- 172.21.15.86:0/1640169605 learned_addr learned my addr 172.21.15.86:0/1640169605 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): get_auth_request con 0x7fd428074ab0 auth_method 0 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:16.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient(hunting): handle_auth_done global_id 164672 payload 274 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 -- 172.21.15.86:0/1640169605 >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 msgr2=0x7fd42813d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 --2- 172.21.15.86:0/1640169605 >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 0x7fd42813d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 -- 172.21.15.86:0/1640169605 >> v1:172.21.15.203:6789/0 conn(0x7fd428073fc0 legacy=0x7fd4280743a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient: _finish_hunting 0 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 monclient: found mon.noname-b 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:16.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:15.996+0000 7fd42dc04640 1 -- 172.21.15.86:0/1640169605 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42809c6a0 con 0x7fd428074ab0 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42dc04640 1 -- 172.21.15.86:0/1640169605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 msgr2=0x7fd42813f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42dc04640 1 --2- 172.21.15.86:0/1640169605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd420009e50 tx=0x7fd42001d340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd29adf7a2171e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 1 -- 172.21.15.86:0/1640169605 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200a5740 con 0x7fd428074ab0 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: dump: 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 1 -- 172.21.15.86:0/1640169605 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4200a58e0 con 0x7fd428074ab0 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 1 -- 172.21.15.86:0/1640169605 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200a5cb0 con 0x7fd428074ab0 2024-09-17T15:46:16.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd427fff640 10 monclient: dump: 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 5 monclient: authenticate success, global_id 164672 2024-09-17T15:46:16.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 4 set_mon_vals no callback set 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: shutdown 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- 172.21.15.86:0/1640169605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 msgr2=0x7fd42813f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- 172.21.15.86:0/1640169605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd420009e50 tx=0x7fd42001d340 comp rx=0 tx=0).stop 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- 172.21.15.86:0/1640169605 shutdown_connections 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- 172.21.15.86:0/1640169605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd42813f560 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- 172.21.15.86:0/1640169605 >> v2:172.21.15.203:3300/0 conn(0x7fd42809bc00 0x7fd42813d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- 172.21.15.86:0/1640169605 >> 172.21.15.86:0/1640169605 conn(0x7fd4280a6440 msgr2=0x7fd4280a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- 172.21.15.86:0/1640169605 shutdown_connections 2024-09-17T15:46:16.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- 172.21.15.86:0/1640169605 wait complete. 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: build_initial_monmap 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: monmap: 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:16.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 Processor -- start 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- start start 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: init 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-a con 0x7fd428073fc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd428120c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:16.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-c con 0x7fd428074ab0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 0x7fd428137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): picked mon.noname-b con 0x7fd42809bc00 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd4281424d0 con 0x7fd428073fc0 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd428068d80 con 0x7fd42809bc00 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd428142650 con 0x7fd428074ab0 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42f667640 10 monclient(hunting): authenticate will time out at 2108.975117s 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:43114/0 (socket says 172.21.15.86:43114) 2024-09-17T15:46:16.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 -- 172.21.15.86:0/37916391 learned_addr learned my addr 172.21.15.86:0/37916391 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): get_auth_request con 0x7fd428073fc0 auth_method 0 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd428120c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): get_auth_request con 0x7fd428074ab0 auth_method 0 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:16.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42dc04640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 msgr2=0x7fd428137ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42dc04640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 0x7fd428137ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient(hunting): handle_auth_done global_id 214158 payload 824 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 msgr2=0x7fd428120c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd428120c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 msgr2=0x7fd428137ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 0x7fd428137ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient: _finish_hunting 0 2024-09-17T15:46:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 monclient: found mon.noname-a 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42d403640 1 -- 172.21.15.86:0/37916391 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4281200e0 con 0x7fd428073fc0 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.004+0000 7fd42cc02640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd428120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42d403640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 msgr2=0x7fd42811eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42d403640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd41800b580 tx=0x7fd41800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418016070 con 0x7fd428073fc0 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: dump: 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:16.019 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 5 monclient: authenticate success, global_id 214158 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4180050d0 con 0x7fd428073fc0 2024-09-17T15:46:16.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4180054c0 con 0x7fd428073fc0 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _renew_subs 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd428122440 con 0x7fd428073fc0 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd4277fe640 4 set_mon_vals no callback set 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 10 monclient: dump: 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _renew_subs 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42811fa30 con 0x7fd428073fc0 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd428074420 con 0x7fd428073fc0 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd418019070 con 0x7fd428073fc0 2024-09-17T15:46:16.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 0x7fd41403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(681..681 src has 438..681) ==== 4879+0+0 (secure 0 0 0) 0x7fd41804f2e0 con 0x7fd428073fc0 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42cc02640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 0x7fd41403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.008+0000 7fd42cc02640 10 monclient: get_auth_request con 0x7fd41403bb10 auth_method 0 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.012+0000 7fd42cc02640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 0x7fd41403dfd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd41c006e50 tx=0x7fd41c006da0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.012+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd41800fb90 con 0x7fd428073fc0 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.012+0000 7fd425ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:16.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.012+0000 7fd425ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.244+0000 7fd42f667640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:16.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.244+0000 7fd42f667640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.244+0000 7fd42f667640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.244+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd428121f40 con 0x7fd428073fc0 2024-09-17T15:46:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd425ffb640 1 -- 172.21.15.86:0/37916391 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd41804f080 con 0x7fd428073fc0 2024-09-17T15:46:16.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd425ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:16.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd425ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:16.247 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 msgr2=0x7fd41403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 0x7fd41403dfd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fd41c006e50 tx=0x7fd41c006da0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 10 monclient: shutdown 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 msgr2=0x7fd42811eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd41800b580 tx=0x7fd41800ba40 comp rx=0 tx=0).stop 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 shutdown_connections 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd41403bb10 0x7fd41403dfd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd42809bc00 0x7fd428137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd428074ab0 0x7fd428120c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 --2- 172.21.15.86:0/37916391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd428073fc0 0x7fd42811eb30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 >> 172.21.15.86:0/37916391 conn(0x7fd4280a6440 msgr2=0x7fd42813ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 shutdown_connections 2024-09-17T15:46:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:16.248+0000 7fd42f667640 1 -- 172.21.15.86:0/37916391 wait complete. 2024-09-17T15:46:19.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:19.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: get_monmap_and_config 2024-09-17T15:46:19.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: build_initial_monmap 2024-09-17T15:46:19.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: monmap: 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:19.398030+0000 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:19.398030+0000 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:19.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 Processor -- start 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 -- start start 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: init 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2bc0689d0 0x7fd2bc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2bc0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 0x7fd2bc069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2bc068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2bc135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:19.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2bc09c8b0 con 0x7fd2bc135ae0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd2bc09cb90 con 0x7fd2bc0689d0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd2bc09ce70 con 0x7fd2bc068db0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c37a6640 10 monclient(hunting): authenticate will time out at 2112.363584s 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2bc0689d0 0x7fd2bc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c0d41640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 msgr2=0x7fd2bc069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1d43640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fd2bc135ae0 legacy=0x7fd2bc135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c0d41640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 0x7fd2bc069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2bc0689d0 0x7fd2bc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49246/0 (socket says 172.21.15.86:49246) 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 1 -- 172.21.15.86:0/315534206 learned_addr learned my addr 172.21.15.86:0/315534206 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): get_auth_request con 0x7fd2bc0689d0 auth_method 0 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.392+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 10 monclient(hunting): handle_auth_done global_id 214161 payload 274 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 1 -- 172.21.15.86:0/315534206 >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 msgr2=0x7fd2bc069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 1 --2- 172.21.15.86:0/315534206 >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 0x7fd2bc069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 1 -- 172.21.15.86:0/315534206 >> v1:172.21.15.203:6789/0 conn(0x7fd2bc135ae0 legacy=0x7fd2bc135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 10 monclient: _finish_hunting 0 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 1 monclient: found mon.noname-a 2024-09-17T15:46:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.396+0000 7fd2c1542640 1 -- 172.21.15.86:0/315534206 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc069740 con 0x7fd2bc0689d0 2024-09-17T15:46:19.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c1542640 1 -- 172.21.15.86:0/315534206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 msgr2=0x7fd2bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:19.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c1542640 1 --2- 172.21.15.86:0/315534206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc06f840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0a05c0 tx=0x7fd2ac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ee4cf6ea460010 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 1 -- 172.21.15.86:0/315534206 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0b0070 con 0x7fd2bc0689d0 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: dump: 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:19.400 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 1 -- 172.21.15.86:0/315534206 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ac0ab600 con 0x7fd2bc0689d0 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:19.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 1 -- 172.21.15.86:0/315534206 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0ab9d0 con 0x7fd2bc0689d0 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2bbfff640 10 monclient: dump: 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 5 monclient: authenticate success, global_id 214161 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 4 set_mon_vals no callback set 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 10 monclient: shutdown 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- 172.21.15.86:0/315534206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 msgr2=0x7fd2bc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/315534206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc06f840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0a05c0 tx=0x7fd2ac01c490 comp rx=0 tx=0).stop 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- 172.21.15.86:0/315534206 shutdown_connections 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/315534206 >> v2:172.21.15.203:3300/0 conn(0x7fd2bc068db0 0x7fd2bc069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/315534206 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc06f840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- 172.21.15.86:0/315534206 >> 172.21.15.86:0/315534206 conn(0x7fd2bc0a6440 msgr2=0x7fd2bc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- 172.21.15.86:0/315534206 shutdown_connections 2024-09-17T15:46:19.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- 172.21.15.86:0/315534206 wait complete. 2024-09-17T15:46:19.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 10 monclient: build_initial_monmap 2024-09-17T15:46:19.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 10 monclient: monmap: 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 Processor -- start 2024-09-17T15:46:19.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 1 -- start start 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.400+0000 7fd2c37a6640 10 monclient: init 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-a con 0x7fd2bc0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 0x7fd2bc131070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-b con 0x7fd2bc068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2bc135ae0 0x7fd2bc1315b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): picked mon.noname-c con 0x7fd2bc135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc09db00 con 0x7fd2bc0689d0 2024-09-17T15:46:19.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc09d980 con 0x7fd2bc068db0 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd2bc09dc80 con 0x7fd2bc135ae0 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient(hunting): authenticate will time out at 2112.372431s 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2bc135ae0 0x7fd2bc1315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:49258/0 (socket says 172.21.15.86:49258) 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 -- 172.21.15.86:0/1358452342 learned_addr learned my addr 172.21.15.86:0/1358452342 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): get_auth_request con 0x7fd2bc0689d0 auth_method 0 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:19.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c0d41640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 msgr2=0x7fd2bc131070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): get_auth_request con 0x7fd2bc135ae0 auth_method 0 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c0d41640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 0x7fd2bc131070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1d43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:19.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient(hunting): handle_auth_done global_id 214164 payload 824 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2bc135ae0 msgr2=0x7fd2bc1315b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2bc135ae0 0x7fd2bc1315b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 msgr2=0x7fd2bc131070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 0x7fd2bc131070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient: _finish_hunting 0 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 monclient: found mon.noname-a 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 -- 172.21.15.86:0/1358452342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc134e90 con 0x7fd2bc0689d0 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 msgr2=0x7fd2bc133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c1542640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac01a0c0 tx=0x7fd2ac019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:19.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0b9020 con 0x7fd2bc0689d0 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: dump: 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ac0b1040 con 0x7fd2bc0689d0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 5 monclient: authenticate success, global_id 214164 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _renew_subs 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc09d460 con 0x7fd2bc0689d0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0b0050 con 0x7fd2bc0689d0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:19.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2b9ffb640 10 monclient: dump: 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2bb7fe640 4 set_mon_vals no callback set 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _renew_subs 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.404+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc132b30 con 0x7fd2bc0689d0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd2ac0bc020 con 0x7fd2bc0689d0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c37a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c37a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c37a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2bc06f840 con 0x7fd2bc0689d0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2b9ffb640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 0x7fd2a803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(681..681 src has 438..681) ==== 4879+0+0 (secure 0 0 0) 0x7fd2ac0e9d40 con 0x7fd2bc0689d0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c0d41640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 0x7fd2a803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:19.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.408+0000 7fd2c0d41640 10 monclient: get_auth_request con 0x7fd2a803bb90 auth_method 0 2024-09-17T15:46:19.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.412+0000 7fd2c0d41640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 0x7fd2a803e050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc1326e0 tx=0x7fd2b0009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:19.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.412+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd2ac0c3030 con 0x7fd2bc0689d0 2024-09-17T15:46:19.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.412+0000 7fd2b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:19.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.412+0000 7fd2b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:19.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.644+0000 7fd2c37a6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:19.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd2bc06d4f0 con 0x7fd2bc0689d0 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2b9ffb640 1 -- 172.21.15.86:0/1358452342 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd2ac016110 con 0x7fd2bc0689d0 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2b9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:19.646 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:19.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 msgr2=0x7fd2a803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 0x7fd2a803e050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc1326e0 tx=0x7fd2b0009270 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 10 monclient: shutdown 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 msgr2=0x7fd2bc133690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac01a0c0 tx=0x7fd2ac019230 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 shutdown_connections 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd2a803bb90 0x7fd2a803e050 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2bc135ae0 0x7fd2bc1315b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2bc068db0 0x7fd2bc131070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 --2- 172.21.15.86:0/1358452342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2bc0689d0 0x7fd2bc133690 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 >> 172.21.15.86:0/1358452342 conn(0x7fd2bc0a6440 msgr2=0x7fd2bc13d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 shutdown_connections 2024-09-17T15:46:19.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:19.648+0000 7fd2c37a6640 1 -- 172.21.15.86:0/1358452342 wait complete. 2024-09-17T15:46:22.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:22.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: get_monmap_and_config 2024-09-17T15:46:22.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: build_initial_monmap 2024-09-17T15:46:22.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: monmap: 2024-09-17T15:46:22.795 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:22.795 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:22.801791+0000 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:22.801791+0000 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:22.796 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 Processor -- start 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 -- start start 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: init 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-c con 0x7f6cf80689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cf8068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cf8139f00 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf809d770 con 0x7f6cf8068db0 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf809ed00 con 0x7f6cf8139f00 2024-09-17T15:46:22.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf809efe0 con 0x7f6cf80689d0 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cffbf3640 10 monclient(hunting): authenticate will time out at 2115.767358s 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cfe190640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f6cf8139f00 legacy=0x7f6cf813c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cfd18e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6cf8068db0 0x7f6cf80691c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:55746/0 (socket says 172.21.15.86:55746) 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cfd98f640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 msgr2=0x7f6cf809ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 learned_addr learned my addr 172.21.15.86:0/149324484 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.799+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:22.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:22.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.86:6789/0 conn(0x7f6cf8068db0 legacy=0x7f6cf80691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:22.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.003+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 msgr2=0x7f6cf809ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:23.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.003+0000 7f6cfe190640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6cf8139f00 legacy=0x7f6cf813c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:23.192 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.003+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:23.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.407+0000 7f6cfe190640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6cf8139f00 legacy=0x7f6cf813c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:23.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.407+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 msgr2=0x7f6cf809ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:23.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:23.407+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:24.211+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 msgr2=0x7f6cf809ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:24.211+0000 7f6cfe190640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6cf8139f00 legacy=0x7f6cf813c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:24.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:24.211+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): tick 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 msgr2=0x7f6cf809ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cf80689d0 0x7f6cf809ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6cf8139f00 legacy=0x7f6cf813c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.86:6789/0 conn(0x7f6cf8068db0 legacy=0x7f6cf80691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient: _add_conns ranks=[2,3] 2024-09-17T15:46:25.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6cdc005960 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc008490 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf809efe0 con 0x7f6cdc008490 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6cdc00b010 con 0x7f6cdc005960 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6ceb7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.799+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cdc008490 auth_method 0 2024-09-17T15:46:25.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request con 0x7f6cdc005960 auth_method 0 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:25.803+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request con 0x7f6cdc005960 auth_method 0 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cdc008490 auth_method 0 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.007+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request con 0x7f6cdc005960 auth_method 0 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cdc008490 auth_method 0 2024-09-17T15:46:26.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:26.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:26.411+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request con 0x7f6cdc005960 auth_method 0 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:27.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cdc008490 auth_method 0 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:27.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:27.215+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:46:28.406 INFO:tasks.rados.rados.0.smithi203.stdout:3064: done (2 left) 2024-09-17T15:46:28.406 INFO:tasks.rados.rados.0.smithi203.stdout:3065: done (1 left) 2024-09-17T15:46:28.406 INFO:tasks.rados.rados.0.smithi203.stdout:3066: done (0 left) 2024-09-17T15:46:28.406 INFO:tasks.rados.rados.0.smithi203.stdout:3067: snap_create 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient(hunting): tick 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 monclient(hunting): continuing hunt 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 msgr2=0x7f6cdc007db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc005960 0x7f6cdc007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 msgr2=0x7f6cdc00a950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008490 0x7f6cdc00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6cfe190640 1 -- 172.21.15.86:0/149324484 reap_dead start 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cdc005da0 0x7f6cdc0088c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc005da0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008e00 0x7f6cdc009290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.799+0000 7f6ceb7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6cdc008e00 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cdc009830 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cf80a04e0 con 0x7f6cdc009830 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6cf809efe0 con 0x7f6cdc008e00 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 1 -- 172.21.15.86:0/149324484 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6cdc00b010 con 0x7f6cdc005da0 2024-09-17T15:46:28.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6ceb7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cdc005da0 msgr2=0x7f6cdc0088c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cdc005da0 0x7f6cdc0088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.86:3300/0 conn(0x7f6cdc008e00 0x7f6cdc009290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cdc008e00 auth_method 0 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_done global_id 214167 payload 274 2024-09-17T15:46:28.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cdc005da0 msgr2=0x7f6cdc0088c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> v2:172.21.15.203:3300/0 conn(0x7f6cdc005da0 0x7f6cdc0088c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.86:6789/0 conn(0x7f6cdc009830 legacy=0x7f6cdc009cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 monclient: found mon.noname-a 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:28.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.803+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf809ed00 con 0x7f6cdc008e00 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 msgr2=0x7f6cdc009290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 0x7f6cdc009290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6cf809d2d0 tx=0x7f6cf401d1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792da46d625754c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 msgr2=0x7f6cdc009290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 msgr2=0x7f6cdc009290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 0x7f6cdc009290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6cf809d2d0 tx=0x7f6cf401d1c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:28.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 0x7f6cdc009290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6cf809d2d0 tx=0x7f6cf401d1c0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:46:28.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 msgr2=0x7f6cdc009290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 0x7f6cdc009290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient: _add_conns ranks=[1,2] 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ce4003510 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 0x7f6ce4009900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ce4007440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ce400a1a0 con 0x7f6ce4003510 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6ce400b3a0 con 0x7f6ce4007440 2024-09-17T15:46:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfc98d640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd98f640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6ce4003510 legacy=0x7f6ce4005990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 0x7f6ce4009900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6ce4007440 auth_method 0 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.811+0000 7f6cfd18e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:28.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_done global_id 214167 payload 293 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> v1:172.21.15.203:6789/0 conn(0x7f6ce4003510 legacy=0x7f6ce4005990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 1 monclient: found mon.noname-b 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:28.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce400b610 con 0x7f6ce4007440 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 msgr2=0x7f6ce4009900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 0x7f6ce4009900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009800 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e08496acbf749fa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40ab070 con 0x7f6ce4007440 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: dump: 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:28.813 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: _finish_auth 0 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: _check_auth_tickets 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf4004bf0 con 0x7f6ce4007440 2024-09-17T15:46:28.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cffbf3640 5 monclient: authenticate success, global_id 214167 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cffbf3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cffbf3640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cffbf3640 4 set_mon_vals no callback set 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 1 -- 172.21.15.86:0/149324484 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4005040 con 0x7f6ce4007440 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.815+0000 7f6cfc98d640 10 monclient: dump: 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:28.816 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: shutdown 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 msgr2=0x7f6ce4009900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ce4007440 0x7f6ce4009900 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009800 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfe190640 1 -- 172.21.15.86:0/149324484 reap_dead start 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- 172.21.15.86:0/149324484 shutdown_connections 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/149324484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cdc008e00 0x7f6cdc009290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- 172.21.15.86:0/149324484 >> 172.21.15.86:0/149324484 conn(0x7f6cf80a6440 msgr2=0x7f6cf80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- 172.21.15.86:0/149324484 shutdown_connections 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- 172.21.15.86:0/149324484 wait complete. 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: build_initial_monmap 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: monmap: 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 Processor -- start 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- start start 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: init 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-c con 0x7f6cdc008e00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-a con 0x7f6cf81d4a60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 0x7f6cf8125540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): picked mon.noname-b con 0x7f6cf81250d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8134dd0 con 0x7f6cf81d4a60 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6cf8134c50 con 0x7f6cf81250d0 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6cf8134f50 con 0x7f6cdc008e00 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cffbf3640 10 monclient(hunting): authenticate will time out at 2121.787317s 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfe190640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 msgr2=0x7f6cf8125540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfe190640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 0x7f6cf8125540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:35938/0 (socket says 172.21.15.86:35938) 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48402/0 (socket says 172.21.15.86:48402) 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 1 -- 172.21.15.86:0/2338226935 learned_addr learned my addr 172.21.15.86:0/2338226935 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request con 0x7f6cdc008e00 auth_method 0 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request con 0x7f6cf81d4a60 auth_method 0 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd98f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.819+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 10 monclient(hunting): handle_auth_done global_id 214170 payload 824 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 -- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 msgr2=0x7f6cf8073ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 -- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 msgr2=0x7f6cf8125540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 0x7f6cf8125540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 monclient: found mon.noname-a 2024-09-17T15:46:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 -- 172.21.15.86:0/2338226935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cf8127fe0 con 0x7f6cf81d4a60 2024-09-17T15:46:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 -- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 msgr2=0x7f6cf81d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cfd18e640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc008460 tx=0x7f6cf4019f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40c8070 con 0x7f6cf81d4a60 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: dump: 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cf40baa40 con 0x7f6cf81d4a60 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf40badf0 con 0x7f6cf81d4a60 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 10 monclient: dump: 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 5 monclient: authenticate success, global_id 214170 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 monclient: _renew_subs 2024-09-17T15:46:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cf809d460 con 0x7f6cf81d4a60 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cebfff640 4 set_mon_vals no callback set 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 monclient: _renew_subs 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cf81267e0 con 0x7f6cf81d4a60 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6cf40cd450 con 0x7f6cf81d4a60 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cea7fc640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 0x7f6cdc03f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.823+0000 7f6cffbf3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cffbf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cf81267e0 con 0x7f6cf81d4a60 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(682..682 src has 438..682) ==== 4851+0+0 (secure 0 0 0) 0x7f6cf40ec260 con 0x7f6cf81d4a60 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 0x7f6cdc03f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cfd98f640 10 monclient: get_auth_request con 0x7f6cdc03d330 auth_method 0 2024-09-17T15:46:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.827+0000 7f6cfd98f640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 0x7f6cdc03f7f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6cec003dc0 tx=0x7f6cec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.831+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6cf4016f10 con 0x7f6cf81d4a60 2024-09-17T15:46:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.831+0000 7f6cea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:28.831+0000 7f6cea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3067: done (0 left) 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3068: copy_from oid 36 from oid 23 current snap is 332 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3069: copy_from oid 21 from oid 49 current snap is 332 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3070: rollback oid 44 current snap is 332 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 331 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3071: rollback oid 4 current snap is 332 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 4 to 327 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3072: read oid 48 snap -1 2024-09-17T15:46:28.845 INFO:tasks.rados.rados.0.smithi203.stdout:3072: expect (ObjNum 1058 snap 327 seq_num 1058) 2024-09-17T15:46:28.846 INFO:tasks.rados.rados.0.smithi203.stdout:3073: delete oid 39 current snap is 332 2024-09-17T15:46:28.846 INFO:tasks.rados.rados.0.smithi203.stdout:3069: finishing copy_from to smithi20328086-21 2024-09-17T15:46:28.846 INFO:tasks.rados.rados.0.smithi203.stdout:3069: got expected ENOENT (src dne) 2024-09-17T15:46:28.846 INFO:tasks.rados.rados.0.smithi203.stdout:3069: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:46:28.865 INFO:tasks.rados.rados.0.smithi203.stdout:3070: finishing rollback tid 0 to smithi20328086-44 2024-09-17T15:46:28.865 INFO:tasks.rados.rados.0.smithi203.stdout:3071: finishing rollback tid 0 to smithi20328086-4 2024-09-17T15:46:28.865 INFO:tasks.rados.rados.0.smithi203.stdout:3070: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:46:28.868 INFO:tasks.rados.rados.0.smithi203.stdout:3072: expect (ObjNum 1058 snap 327 seq_num 1058) 2024-09-17T15:46:29.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cffbf3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:29.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cffbf3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:29.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cffbf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6cf81267e0 con 0x7f6cf81d4a60 2024-09-17T15:46:29.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cea7fc640 1 -- 172.21.15.86:0/2338226935 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f6cf40cccd0 con 0x7f6cf81d4a60 2024-09-17T15:46:29.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:29.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.083+0000 7f6cea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:29.080 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":182,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 msgr2=0x7f6cdc03f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 0x7f6cdc03f7f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6cec003dc0 tx=0x7f6cec008040 comp rx=0 tx=0).stop 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 10 monclient: shutdown 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 msgr2=0x7f6cf81d4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc008460 tx=0x7f6cf4019f60 comp rx=0 tx=0).stop 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 shutdown_connections 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cdc03d330 0x7f6cdc03f7f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6cf81250d0 0x7f6cf8125540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cf81d4a60 0x7f6cf81d4e60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 --2- 172.21.15.86:0/2338226935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cdc008e00 0x7f6cf8073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 >> 172.21.15.86:0/2338226935 conn(0x7f6cf80a6440 msgr2=0x7f6cf80a1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 shutdown_connections 2024-09-17T15:46:29.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.087+0000 7f6cffbf3640 1 -- 172.21.15.86:0/2338226935 wait complete. 2024-09-17T15:46:29.099 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:46:29.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:46:29.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:29.130 INFO:tasks.rados.rados.0.smithi203.stdout:3068: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:46:29.130 INFO:tasks.rados.rados.0.smithi203.stdout:3070: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:46:29.130 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1508 (ObjNum 1062 snap 328 seq_num 1062) dirty exists 2024-09-17T15:46:29.130 INFO:tasks.rados.rados.0.smithi203.stdout:3071: finishing rollback tid 1 to smithi20328086-4 2024-09-17T15:46:29.130 INFO:tasks.rados.rados.0.smithi203.stdout:3071: finishing rollback tid 2 to smithi20328086-4 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1323 (ObjNum 1038 snap 321 seq_num 1038) dirty exists 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3068: finishing copy_from to smithi20328086-36 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1114 (ObjNum 559 snap 166 seq_num 559) dirty exists 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3068: done (5 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3069: done (4 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3070: done (3 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3071: done (2 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3072: done (1 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3073: done (0 left) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3074: read oid 29 snap -1 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3074: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:46:29.131 INFO:tasks.rados.rados.0.smithi203.stdout:3075: write oid 2 current snap is 332 2024-09-17T15:46:29.132 INFO:tasks.rados.rados.0.smithi203.stdout:3075: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-17T15:46:29.144 INFO:tasks.rados.rados.0.smithi203.stdout:3075: writing smithi20328086-2 from 450718 to 873917 tid 1 2024-09-17T15:46:29.164 INFO:tasks.rados.rados.0.smithi203.stdout:3075: writing smithi20328086-2 from 1414602 to 2050718 tid 2 2024-09-17T15:46:29.164 INFO:tasks.rados.rados.0.smithi203.stdout:3076: delete oid 28 current snap is 332 2024-09-17T15:46:29.164 INFO:tasks.rados.rados.0.smithi203.stdout:3074: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:46:29.164 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:29.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 10 monclient: get_monmap_and_config 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 10 monclient: build_initial_monmap 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 10 monclient: monmap: 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:29.222994+0000 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:29.222994+0000 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:29.217 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:29.218 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:29.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 1 Processor -- start 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 1 -- start start 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.219+0000 7f60cac01640 10 monclient: init 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60c40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60c40749c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:29.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-b con 0x7f60c41428d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c4071640 con 0x7f60c40689d0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60c40718b0 con 0x7f60c40749c0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f60c4068db0 con 0x7f60c41428d0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60cac01640 10 monclient(hunting): authenticate will time out at 2122.188593s 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c3fff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f60c40749c0 legacy=0x7f60c41411b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c899d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f60c40689d0 0x7f60c40742b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56734/0 (socket says 172.21.15.86:56734) 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c899d640 1 -- 172.21.15.86:0/2545947439 learned_addr learned my addr 172.21.15.86:0/2545947439 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): get_auth_request con 0x7f60c41428d0 auth_method 0 2024-09-17T15:46:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient(hunting): handle_auth_done global_id 164684 payload 274 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 1 -- 172.21.15.86:0/2545947439 >> v1:172.21.15.203:6789/0 conn(0x7f60c40749c0 legacy=0x7f60c41411b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 1 -- 172.21.15.86:0/2545947439 >> v1:172.21.15.86:6789/0 conn(0x7f60c40689d0 legacy=0x7f60c40742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 1 monclient: found mon.noname-b 2024-09-17T15:46:29.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:29.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.223+0000 7f60c919e640 1 -- 172.21.15.86:0/2545947439 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c413c780 con 0x7f60c41428d0 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c919e640 1 -- 172.21.15.86:0/2545947439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 msgr2=0x7f60c4144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c919e640 1 --2- 172.21.15.86:0/2545947439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c4144cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009820 tx=0x7f60b401c170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=916b3fd8eaaee06f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 1 -- 172.21.15.86:0/2545947439 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b40b0070 con 0x7f60c41428d0 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: dump: 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.225 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 1 -- 172.21.15.86:0/2545947439 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b40ab600 con 0x7f60c41428d0 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 1 -- 172.21.15.86:0/2545947439 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b40ab9d0 con 0x7f60c41428d0 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60c37fe640 10 monclient: dump: 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 5 monclient: authenticate success, global_id 164684 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 4 set_mon_vals no callback set 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 10 monclient: shutdown 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:29.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- 172.21.15.86:0/2545947439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 msgr2=0x7f60c4144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 --2- 172.21.15.86:0/2545947439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c4144cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009820 tx=0x7f60b401c170 comp rx=0 tx=0).stop 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- 172.21.15.86:0/2545947439 shutdown_connections 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 --2- 172.21.15.86:0/2545947439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c4144cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- 172.21.15.86:0/2545947439 >> 172.21.15.86:0/2545947439 conn(0x7f60c40a3710 msgr2=0x7f60c40a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- 172.21.15.86:0/2545947439 shutdown_connections 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- 172.21.15.86:0/2545947439 wait complete. 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 10 monclient: build_initial_monmap 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 10 monclient: monmap: 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:29.229 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 Processor -- start 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 1 -- start start 2024-09-17T15:46:29.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.227+0000 7f60cac01640 10 monclient: init 2024-09-17T15:46:29.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60c40689d0 0x7f60c41d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-b con 0x7f60c40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c409ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-c con 0x7f60c41428d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 0x7f60c413a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): picked mon.noname-a con 0x7f60c40a0520 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60c4147ba0 con 0x7f60c40a0520 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f60c4147a20 con 0x7f60c40689d0 2024-09-17T15:46:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f60c4147d20 con 0x7f60c41428d0 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient(hunting): authenticate will time out at 2122.196369s 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 0x7f60c413a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c409ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c899d640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60c40689d0 msgr2=0x7f60c41d44d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c899d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60c40689d0 0x7f60c41d44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 0x7f60c413a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48414/0 (socket says 172.21.15.86:48414) 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c409ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:35968/0 (socket says 172.21.15.86:35968) 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 learned_addr learned my addr 172.21.15.86:0/1462669704 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): get_auth_request con 0x7f60c40a0520 auth_method 0 2024-09-17T15:46:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): get_auth_request con 0x7f60c41428d0 auth_method 0 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient(hunting): handle_auth_done global_id 214173 payload 824 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 msgr2=0x7f60c409ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c409ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60c40689d0 msgr2=0x7f60c41d44d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60c40689d0 0x7f60c41d44d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 monclient: found mon.noname-a 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c413bec0 con 0x7f60c40a0520 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c3fff640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60c41428d0 0x7f60c409ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 msgr2=0x7f60c413a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c919e640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 0x7f60c413a500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f60b40ae040 tx=0x7f60b401a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b40b0050 con 0x7f60c40a0520 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b40bc070 con 0x7f60c40a0520 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b40b7670 con 0x7f60c40a0520 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 5 monclient: authenticate success, global_id 214173 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _renew_subs 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60c2ffd640 4 set_mon_vals no callback set 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c40a09d0 con 0x7f60c40a0520 2024-09-17T15:46:29.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _renew_subs 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c40750b0 con 0x7f60c40a0520 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.231+0000 7f60cac01640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 0x7f60c413aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60c4074330 con 0x7f60c40749c0 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 0x7f60c413aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient: get_auth_request con 0x7f60c40749c0 auth_method 0 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f60b40b7850 con 0x7f60c40a0520 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 10 monclient(hunting): handle_auth_done global_id 214176 payload 824 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60c409d2c0 con 0x7f60c40749c0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f60b003bb10 0x7f60b003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(683..683 src has 438..683) ==== 4851+0+0 (secure 0 0 0) 0x7f60b40ea5a0 con 0x7f60c40a0520 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c3fff640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f60b003bb10 0x7f60b003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c3fff640 10 monclient: get_auth_request con 0x7f60b003bb10 auth_method 0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 msgr2=0x7f60c413aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c899d640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 0x7f60c413aa40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00c9f0 tx=0x7f60bc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc012400 con 0x7f60c40749c0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T15:46:29.243 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60bc00ae80 con 0x7f60c40749c0 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:29.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 msgr2=0x7f60c413aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 0x7f60c413aa40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00c9f0 tx=0x7f60bc00ceb0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 msgr2=0x7f60c413aa40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:29.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c17fa640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40749c0 0x7f60c413aa40 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f60bc00c9f0 tx=0x7f60bc00ceb0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.235+0000 7f60c3fff640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f60b003bb10 0x7f60b003dfd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f60c4142510 tx=0x7f60b8003b50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60cac01640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:29.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60cac01640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 0x7f60c4134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60cac01640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60c413b390 con 0x7f60c413af80 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 0x7f60c4134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient: get_auth_request con 0x7f60c413af80 auth_method 0 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 10 monclient(hunting): handle_auth_done global_id 214179 payload 824 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 1 -- 172.21.15.86:0/1462669704 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60c409d5e0 con 0x7f60c413af80 2024-09-17T15:46:29.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 msgr2=0x7f60c4134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c899d640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 0x7f60c4134c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60bc015000 tx=0x7f60bc004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60bc012af0 con 0x7f60c413af80 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.243+0000 7f60c17fa640 10 monclient: dump: 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4536 (secure 0 0 0) 0x7f60bc012af0 con 0x7f60c413af80 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 msgr2=0x7f60c4134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 0x7f60c4134c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60bc015000 tx=0x7f60bc004cc0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 msgr2=0x7f60c4134c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c17fa640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c413af80 0x7f60c4134c60 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f60bc015000 tx=0x7f60bc004cc0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.249 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 182, 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 45418, 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:29.250 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.251 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.252 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:29.253 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.254 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.255 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.256 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:29.262 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f60b003bb10 msgr2=0x7f60b003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f60b003bb10 0x7f60b003dfd0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f60c4142510 tx=0x7f60b8003b50 comp rx=0 tx=0).stop 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 10 monclient: shutdown 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 msgr2=0x7f60c413a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 --2- 172.21.15.86:0/1462669704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60c40a0520 0x7f60c413a500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f60b40ae040 tx=0x7f60b401a9b0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 reap_dead start 2024-09-17T15:46:29.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60c919e640 1 -- 172.21.15.86:0/1462669704 reap_dead start 2024-09-17T15:46:29.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 shutdown_connections 2024-09-17T15:46:29.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 >> 172.21.15.86:0/1462669704 conn(0x7f60c40a3710 msgr2=0x7f60c4072250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:29.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 shutdown_connections 2024-09-17T15:46:29.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.247+0000 7f60cac01640 1 -- 172.21.15.86:0/1462669704 wait complete. 2024-09-17T15:46:29.264 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:46:29.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:46:29.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:29.272 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:29.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: get_monmap_and_config 2024-09-17T15:46:29.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: build_initial_monmap 2024-09-17T15:46:29.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: monmap: 2024-09-17T15:46:29.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:29.384469+0000 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:29.384469+0000 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:29.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:29.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:29.380 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:29.380 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:29.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 Processor -- start 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 -- start start 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: init 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e300a50e0 0x7f3e3009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-a con 0x7f3e300a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-b con 0x7f3e300a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3e30137ea0 0x7f3e301382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-c con 0x7f3e30137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3e300a6280 con 0x7f3e300a50e0 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3e3009c0a0 con 0x7f3e30137ea0 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e3009c380 con 0x7f3e300a54c0 2024-09-17T15:46:29.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e38383640 10 monclient(hunting): authenticate will time out at 2122.349956s 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e300a50e0 0x7f3e3009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:35980/0 (socket says 172.21.15.86:35980) 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e36920640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f3e30137ea0 msgr2=0x7f3e301382f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 learned_addr learned my addr 172.21.15.86:0/3676749366 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): get_auth_request con 0x7f3e300a54c0 auth_method 0 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): get_auth_request con 0x7f3e300a50e0 auth_method 0 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> v2:172.21.15.203:3300/0 conn(0x7f3e30137ea0 0x7f3e301382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient(hunting): handle_auth_done global_id 164690 payload 274 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> v2:172.21.15.203:3300/0 conn(0x7f3e30137ea0 msgr2=0x7f3e301382f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> v2:172.21.15.203:3300/0 conn(0x7f3e30137ea0 0x7f3e301382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> v2:172.21.15.86:3300/0 conn(0x7f3e300a50e0 msgr2=0x7f3e3009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> v2:172.21.15.86:3300/0 conn(0x7f3e300a50e0 0x7f3e3009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 monclient: found mon.noname-b 2024-09-17T15:46:29.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:29.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3009c660 con 0x7f3e300a54c0 2024-09-17T15:46:29.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.383+0000 7f3e3611f640 1 --2- 172.21.15.86:0/3676749366 >> v2:172.21.15.86:3300/0 conn(0x7f3e300a50e0 0x7f3e3009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:29.389 INFO:tasks.rados.rados.0.smithi203.stdout:3075: finishing write tid 1 to smithi20328086-2 2024-09-17T15:46:29.389 INFO:tasks.rados.rados.0.smithi203.stdout:3075: finishing write tid 2 to smithi20328086-2 2024-09-17T15:46:29.389 INFO:tasks.rados.rados.0.smithi203.stdout:3075: finishing write tid 3 to smithi20328086-2 2024-09-17T15:46:29.389 INFO:tasks.rados.rados.0.smithi203.stdout:3075: finishing write tid 4 to smithi20328086-2 2024-09-17T15:46:29.389 INFO:tasks.rados.rados.0.smithi203.stdout:3075: oid 2 updating version 0 to 1194 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3075: oid 2 updating version 1194 to 1195 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3075: oid 2 version 1195 is already newer than 1193 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1195 (ObjNum 1074 snap 332 seq_num 1074) dirty exists 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3075: left oid 2 (ObjNum 1074 snap 332 seq_num 1074) 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3074: done (2 left) 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3075: done (1 left) 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3076: done (0 left) 2024-09-17T15:46:29.390 INFO:tasks.rados.rados.0.smithi203.stdout:3077: write (excl) oid 31 current snap is 332 2024-09-17T15:46:29.391 INFO:tasks.rados.rados.0.smithi203.stdout:3077: seq_num 1076 ranges {484332=573697,1531566=552766} 2024-09-17T15:46:29.399 INFO:tasks.rados.rados.0.smithi203.stdout:3077: writing smithi20328086-31 from 484332 to 1058029 tid 1 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 msgr2=0x7f3e300a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006f8c0 tx=0x7f3e2001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b51b6aeee1a7fd0c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 msgr2=0x7f3e300a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 msgr2=0x7f3e300a58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006f8c0 tx=0x7f3e2001c400 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006f8c0 tx=0x7f3e2001c400 comp rx=0 tx=0).stop 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient: ms_handle_reset current mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 msgr2=0x7f3e300a58f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e300a54c0 0x7f3e300a58f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient: _add_conns ranks=[0,1] 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e1c003510 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e1c0074a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1c00b410 con 0x7f3e1c003510 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1c00c610 con 0x7f3e1c0074a0 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f3e200b2070 con 0x7f3e300a54c0 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): discarding stray monitor message mon_map magic: 0 2024-09-17T15:46:29.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3e200ad680 con 0x7f3e300a54c0 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3511d640 10 monclient(hunting): discarding stray monitor message config(0 keys) 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.387+0000 7f3e3591e640 1 -- 172.21.15.86:0/3676749366 >> v1:172.21.15.203:6789/0 conn(0x7f3e1c0074a0 legacy=0x7f3e1c009960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3611f640 1 -- 172.21.15.86:0/3676749366 >> v1:172.21.15.86:6789/0 conn(0x7f3e1c003510 legacy=0x7f3e1c005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984538120 0 0) 0x7f3e1c00b410 con 0x7f3e1c003510 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): _negotiate 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> v1:172.21.15.86:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f3e1c00b5e0 con 0x7f3e1c003510 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3146703338 0 0) 0x7f3e1c00b5e0 con 0x7f3e1c003510 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 >> v1:172.21.15.203:6789/0 conn(0x7f3e1c0074a0 legacy=0x7f3e1c009960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:29.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c00d8d0 con 0x7f3e1c003510 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _finish_auth 0 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _check_auth_tickets 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f3e2c009be0 con 0x7f3e1c003510 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: dump: 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 >> v1:172.21.15.86:6789/0 conn(0x7f3e1c003510 legacy=0x7f3e1c005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 -- 172.21.15.86:0/3676749366 reap_dead start 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 0x7f3e1c00bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): picked mon.b con 0x7f3e1c00bbe0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00c830 0x7f3e1c009930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): picked mon.c con 0x7f3e1c00c830 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): picked mon.a con 0x7f3e1c00cc10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:29.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c00a2e0 con 0x7f3e1c00cc10 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e1c00a460 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e1c00a660 con 0x7f3e1c00c830 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3511d640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): get_auth_request con 0x7f3e1c00cc10 auth_method 0 2024-09-17T15:46:29.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3611f640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 msgr2=0x7f3e1c00bfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00c830 0x7f3e1c009930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3611f640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 0x7f3e1c00bfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): get_auth_request con 0x7f3e1c00c830 auth_method 0 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:29.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient(hunting): handle_auth_done global_id 164690 payload 293 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00c830 msgr2=0x7f3e1c009930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00c830 0x7f3e1c009930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 msgr2=0x7f3e1c00bfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 0x7f3e1c00bfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 monclient: found mon.a 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 1 -- 172.21.15.86:0/3676749366 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1c00b5e0 con 0x7f3e1c00cc10 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient: _finish_auth 0 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 10 monclient: _check_auth_tickets 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e36920640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e38383640 5 monclient: authenticate success, global_id 164690 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.391+0000 7f3e38383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 msgr2=0x7f3e1c00d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3e300a84e0 tx=0x7f3e20008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde6e4fb8bedf4d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3e300a84e0 tx=0x7f3e20008030 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f3e28015640 mon_map magic: 0 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e280099b0 con 0x7f3e1c00cc10 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 10 monclient: dump: 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.413 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e28009e80 con 0x7f3e1c00cc10 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 4 set_mon_vals no callback set 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: shutdown 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:29.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 msgr2=0x7f3e1c00d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3e300a84e0 tx=0x7f3e20008030 comp rx=0 tx=0).stop 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 1 -- 172.21.15.86:0/3676749366 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f3e28015640 con 0x7f3e1c00cc10 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3511d640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- 172.21.15.86:0/3676749366 shutdown_connections 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00cc10 0x7f3e1c00d090 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00c830 0x7f3e1c009930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- 172.21.15.86:0/3676749366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00bbe0 0x7f3e1c00bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- 172.21.15.86:0/3676749366 >> 172.21.15.86:0/3676749366 conn(0x7f3e300aaae0 msgr2=0x7f3e300aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- 172.21.15.86:0/3676749366 shutdown_connections 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- 172.21.15.86:0/3676749366 wait complete. 2024-09-17T15:46:29.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: build_initial_monmap 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: monmap: 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:29.416 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 Processor -- start 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- start start 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: init 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1c00bbe0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00c830 0x7f3e301371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-b con 0x7f3e1c00c830 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00cc10 0x7f3e3013c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): picked mon.noname-c con 0x7f3e1c00cc10 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:29.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e300a6ae0 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e300a6960 con 0x7f3e1c00c830 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e300a58f0 con 0x7f3e1c00cc10 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e38383640 10 monclient(hunting): authenticate will time out at 2122.363013s 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:48484/0 (socket says 172.21.15.86:48484) 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 -- 172.21.15.86:0/536217466 learned_addr learned my addr 172.21.15.86:0/536217466 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): get_auth_request con 0x7f3e1c00bbe0 auth_method 0 2024-09-17T15:46:29.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00cc10 0x7f3e3013c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3591e640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00c830 msgr2=0x7f3e301371a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): get_auth_request con 0x7f3e1c00cc10 auth_method 0 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3591e640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00c830 0x7f3e301371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient(hunting): handle_auth_done global_id 214185 payload 824 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00cc10 msgr2=0x7f3e3013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00cc10 0x7f3e3013c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00c830 msgr2=0x7f3e301371a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e1c00c830 0x7f3e301371a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient: _finish_hunting 0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 monclient: found mon.noname-a 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e3611f640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e300a18d0 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.395+0000 7f3e36920640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e1c00cc10 0x7f3e3013c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3611f640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 msgr2=0x7f3e301cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3611f640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00eb00 tx=0x7f3e2c00c150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c017020 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:29.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: dump: 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 5 monclient: authenticate success, global_id 214185 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e2c00cce0 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 monclient: _renew_subs 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e30122b40 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c0136b0 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:29.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 10 monclient: dump: 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3491c640 4 set_mon_vals no callback set 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 monclient: _renew_subs 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3013dda0 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3e2c01a020 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 0x7f3e3013c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e3006a0e0 con 0x7f3e301376e0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 0x7f3e3013c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient: get_auth_request con 0x7f3e301376e0 auth_method 0 2024-09-17T15:46:29.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e1403bb60 0x7f3e1403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(683..683 src has 438..683) ==== 4851+0+0 (secure 0 0 0) 0x7f3e2c050640 con 0x7f3e1c00bbe0 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 10 monclient(hunting): handle_auth_done global_id 164696 payload 824 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e3591e640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e30122430 con 0x7f3e301376e0 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e36920640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e1403bb60 0x7f3e1403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.399+0000 7f3e36920640 10 monclient: get_auth_request con 0x7f3e1403bb60 auth_method 0 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e3591e640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 msgr2=0x7f3e3013c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e3591e640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 0x7f3e3013c820 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e300aa620 tx=0x7f3e200191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200be030 con 0x7f3e301376e0 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 10 monclient: dump: 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.427 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e20009490 con 0x7f3e301376e0 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 msgr2=0x7f3e3013c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 0x7f3e3013c820 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e300aa620 tx=0x7f3e200191a0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 msgr2=0x7f3e3013c820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:29.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e26ffd640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e301376e0 0x7f3e3013c820 secure :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3e300aa620 tx=0x7f3e200191a0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.403+0000 7f3e36920640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e1403bb60 0x7f3e1403e020 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e30138280 tx=0x7f3e28005000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.430 INFO:tasks.rados.rados.0.smithi203.stdout:3077: writing smithi20328086-31 from 1531566 to 2084332 tid 2 2024-09-17T15:46:29.430 INFO:tasks.rados.rados.0.smithi203.stdout:3078: write oid 26 current snap is 332 2024-09-17T15:46:29.430 INFO:tasks.rados.rados.0.smithi203.stdout:3078: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-09-17T15:46:29.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e38383640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e38383640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 0x7f3e301392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e38383640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e3013d160 con 0x7f3e3013cd60 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 0x7f3e301392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient: get_auth_request con 0x7f3e3013cd60 auth_method 0 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:29.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 10 monclient(hunting): handle_auth_done global_id 164699 payload 824 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.411+0000 7f3e3591e640 1 -- 172.21.15.86:0/536217466 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e300ab7d0 con 0x7f3e3013cd60 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e3591e640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 msgr2=0x7f3e301392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e3591e640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 0x7f3e301392d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006c9a0 tx=0x7f3e200a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e200be040 con 0x7f3e3013cd60 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 10 monclient: dump: 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f3e200c37c0 con 0x7f3e3013cd60 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 msgr2=0x7f3e301392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 0x7f3e301392d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006c9a0 tx=0x7f3e200a9000 comp rx=0 tx=0).stop 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 msgr2=0x7f3e301392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e26ffd640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3013cd60 0x7f3e301392d0 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3e3006c9a0 tx=0x7f3e200a9000 comp rx=0 tx=0).stop 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:29.435 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 182, 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 292603, 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:29.436 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.437 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.438 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:29.439 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:29.440 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:29.441 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:29.442 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.443 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:29.444 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:29.445 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.447 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:29.448 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.449 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:46:29.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:29.451 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e1403bb60 msgr2=0x7f3e1403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e1403bb60 0x7f3e1403e020 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e30138280 tx=0x7f3e28005000 comp rx=0 tx=0).stop 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 10 monclient: shutdown 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 msgr2=0x7f3e301cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00eb00 tx=0x7f3e2c00c150 comp rx=0 tx=0).stop 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e36920640 1 -- 172.21.15.86:0/536217466 reap_dead start 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 shutdown_connections 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 --2- 172.21.15.86:0/536217466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e1c00bbe0 0x7f3e301cfc40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 >> 172.21.15.86:0/536217466 conn(0x7f3e300aaae0 msgr2=0x7f3e3013e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 shutdown_connections 2024-09-17T15:46:29.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:29.415+0000 7f3e38383640 1 -- 172.21.15.86:0/536217466 wait complete. 2024-09-17T15:46:29.455 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:46:29.456 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:46:29.456 INFO:tasks.rados.rados.0.smithi203.stdout:3078: writing smithi20328086-26 from 501139 to 1150085 tid 1 2024-09-17T15:46:29.456 INFO:tasks.rados.rados.0.smithi203.stdout:3078: writing smithi20328086-26 from 1873695 to 2101139 tid 2 2024-09-17T15:46:29.456 INFO:tasks.rados.rados.0.smithi203.stdout:3079: read oid 10 snap -1 2024-09-17T15:46:29.456 INFO:tasks.rados.rados.0.smithi203.stdout:3079: expect deleted 2024-09-17T15:46:29.456 INFO:tasks.rados.rados.0.smithi203.stdout:3080: snap_create 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: finishing write tid 1 to smithi20328086-31 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: finishing write tid 2 to smithi20328086-31 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: finishing write tid 3 to smithi20328086-31 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: finishing write tid 4 to smithi20328086-31 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: oid 31 updating version 0 to 1304 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: oid 31 version 1304 is already newer than 1303 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: oid 31 updating version 1304 to 1305 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1305 (ObjNum 1076 snap 332 seq_num 1076) dirty exists 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3077: left oid 31 (ObjNum 1076 snap 332 seq_num 1076) 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3078: finishing write tid 1 to smithi20328086-26 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3078: finishing write tid 2 to smithi20328086-26 2024-09-17T15:46:29.457 INFO:tasks.rados.rados.0.smithi203.stdout:3078: finishing write tid 3 to smithi20328086-26 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:3078: finishing write tid 4 to smithi20328086-26 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:3078: oid 26 updating version 0 to 1608 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:3078: oid 26 updating version 1608 to 1609 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:3078: oid 26 updating version 1609 to 1610 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1610 (ObjNum 1077 snap 332 seq_num 1077) dirty exists 2024-09-17T15:46:29.458 INFO:tasks.rados.rados.0.smithi203.stdout:3078: left oid 26 (ObjNum 1077 snap 332 seq_num 1077) 2024-09-17T15:46:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:3077: done (3 left) 2024-09-17T15:46:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:3078: done (2 left) 2024-09-17T15:46:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:3079: done (1 left) 2024-09-17T15:46:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:3080: done (0 left) 2024-09-17T15:46:30.428 INFO:tasks.rados.rados.0.smithi203.stdout:3081: snap_remove snap 330 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3081: done (0 left) 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3082: read oid 44 snap -1 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3082: expect (ObjNum 1062 snap 328 seq_num 1062) 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3083: copy_from oid 31 from oid 42 current snap is 333 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3084: read oid 21 snap -1 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3084: expect (ObjNum 1054 snap 327 seq_num 1054) 2024-09-17T15:46:31.437 INFO:tasks.rados.rados.0.smithi203.stdout:3085: snap_create 2024-09-17T15:46:31.448 INFO:tasks.rados.rados.0.smithi203.stdout:3083: finishing copy_from to smithi20328086-31 2024-09-17T15:46:31.448 INFO:tasks.rados.rados.0.smithi203.stdout:3083: got expected ENOENT (src dne) 2024-09-17T15:46:31.448 INFO:tasks.rados.rados.0.smithi203.stdout:3083: finishing copy_from racing read to smithi20328086-31 2024-09-17T15:46:31.462 INFO:tasks.rados.rados.0.smithi203.stdout:3082: expect (ObjNum 1062 snap 328 seq_num 1062) 2024-09-17T15:46:31.565 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:31.653 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2942730 2024-09-17T15:46:31.668 INFO:tasks.rados.rados.0.smithi203.stdout:3084: expect (ObjNum 1054 snap 327 seq_num 1054) 2024-09-17T15:46:31.718 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3317153 2024-09-17T15:46:31.757 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3159373 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3082: done (3 left) 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3083: done (2 left) 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3084: done (1 left) 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3085: done (0 left) 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3086: write (excl) oid 6 current snap is 334 2024-09-17T15:46:32.451 INFO:tasks.rados.rados.0.smithi203.stdout:3086: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-09-17T15:46:32.468 INFO:tasks.rados.rados.0.smithi203.stdout:3086: writing smithi20328086-6 from 517946 to 1242141 tid 1 2024-09-17T15:46:32.481 INFO:tasks.rados.rados.0.smithi203.stdout:3086: writing smithi20328086-6 from 1932177 to 2117946 tid 2 2024-09-17T15:46:32.481 INFO:tasks.rados.rados.0.smithi203.stdout:3087: write oid 27 current snap is 334 2024-09-17T15:46:32.481 INFO:tasks.rados.rados.0.smithi203.stdout:3086: finishing write tid 1 to smithi20328086-6 2024-09-17T15:46:32.481 INFO:tasks.rados.rados.0.smithi203.stdout:3087: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-17T15:46:32.501 INFO:tasks.rados.rados.0.smithi203.stdout:3087: writing smithi20328086-27 from 534753 to 1334197 tid 1 2024-09-17T15:46:32.511 INFO:tasks.rados.rados.0.smithi203.stdout:3087: writing smithi20328086-27 from 1990659 to 2134753 tid 2 2024-09-17T15:46:32.511 INFO:tasks.rados.rados.0.smithi203.stdout:3088: snap_remove snap 325 2024-09-17T15:46:34.322 INFO:tasks.rados.rados.0.smithi203.stdout:3088: done (2 left) 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3089: write (excl) oid 20 current snap is 334 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: finishing write tid 2 to smithi20328086-6 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: finishing write tid 3 to smithi20328086-6 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: finishing write tid 4 to smithi20328086-6 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: oid 6 updating version 0 to 1341 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: oid 6 version 1341 is already newer than 1339 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: oid 6 version 1341 is already newer than 1340 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1341 (ObjNum 1078 snap 334 seq_num 1078) dirty exists 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3086: left oid 6 (ObjNum 1078 snap 334 seq_num 1078) 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3087: finishing write tid 1 to smithi20328086-27 2024-09-17T15:46:34.323 INFO:tasks.rados.rados.0.smithi203.stdout:3089: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-17T15:46:34.340 INFO:tasks.rados.rados.0.smithi203.stdout:3089: writing smithi20328086-20 from 551560 to 1142606 tid 1 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3089: writing smithi20328086-20 from 1765494 to 2151560 tid 2 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3086: done (2 left) 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3090: copy_from oid 12 from oid 28 current snap is 334 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: finishing write tid 2 to smithi20328086-27 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: finishing write tid 3 to smithi20328086-27 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: finishing write tid 4 to smithi20328086-27 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: oid 27 updating version 0 to 1389 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: oid 27 version 1389 is already newer than 1388 2024-09-17T15:46:34.358 INFO:tasks.rados.rados.0.smithi203.stdout:3087: oid 27 updating version 1389 to 1390 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1390 (ObjNum 1079 snap 334 seq_num 1079) dirty exists 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3087: left oid 27 (ObjNum 1079 snap 334 seq_num 1079) 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3089: finishing write tid 1 to smithi20328086-20 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3087: done (2 left) 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3091: read oid 4 snap -1 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3091: expect (ObjNum 1038 snap 321 seq_num 1038) 2024-09-17T15:46:34.359 INFO:tasks.rados.rados.0.smithi203.stdout:3092: delete oid 23 current snap is 334 2024-09-17T15:46:34.360 INFO:tasks.rados.rados.0.smithi203.stdout:3090: finishing copy_from to smithi20328086-12 2024-09-17T15:46:34.360 INFO:tasks.rados.rados.0.smithi203.stdout:3090: got expected ENOENT (src dne) 2024-09-17T15:46:34.361 INFO:tasks.rados.rados.0.smithi203.stdout:3090: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:46:34.367 INFO:tasks.rados.rados.0.smithi203.stdout:3089: finishing write tid 2 to smithi20328086-20 2024-09-17T15:46:34.369 INFO:tasks.rados.rados.0.smithi203.stdout:3089: finishing write tid 3 to smithi20328086-20 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: finishing write tid 4 to smithi20328086-20 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: oid 20 updating version 0 to 1343 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: oid 20 updating version 1343 to 1344 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: oid 20 version 1344 is already newer than 1342 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1344 (ObjNum 1080 snap 334 seq_num 1080) dirty exists 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: left oid 20 (ObjNum 1080 snap 334 seq_num 1080) 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3089: done (3 left) 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3090: done (2 left) 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3092: done (1 left) 2024-09-17T15:46:34.370 INFO:tasks.rados.rados.0.smithi203.stdout:3093: delete oid 31 current snap is 334 2024-09-17T15:46:34.373 INFO:tasks.rados.rados.0.smithi203.stdout:3091: expect (ObjNum 1038 snap 321 seq_num 1038) 2024-09-17T15:46:34.394 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1280636 2024-09-17T15:46:34.518 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 384590 2024-09-17T15:46:34.566 INFO:tasks.rados.rados.0.smithi203.stdout:3091: done (1 left) 2024-09-17T15:46:34.566 INFO:tasks.rados.rados.0.smithi203.stdout:3093: done (0 left) 2024-09-17T15:46:34.566 INFO:tasks.rados.rados.0.smithi203.stdout:3094: copy_from oid 33 from oid 8 current snap is 334 2024-09-17T15:46:34.567 INFO:tasks.rados.rados.0.smithi203.stdout:3095: copy_from oid 1 from oid 48 current snap is 334 2024-09-17T15:46:34.567 INFO:tasks.rados.rados.0.smithi203.stdout:3096: snap_remove snap 317 2024-09-17T15:46:35.464 INFO:tasks.rados.rados.0.smithi203.stdout:3096: done (2 left) 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3097: read oid 19 snap 327 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3097: expect (ObjNum 994 snap 311 seq_num 994) 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3098: copy_from oid 18 from oid 25 current snap is 334 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3095: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3094: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3095: finishing copy_from to smithi20328086-1 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1531 (ObjNum 1058 snap 327 seq_num 1058) dirty exists 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3094: finishing copy_from to smithi20328086-33 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 918 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3094: done (3 left) 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3095: done (2 left) 2024-09-17T15:46:35.465 INFO:tasks.rados.rados.0.smithi203.stdout:3099: snap_remove snap 331 2024-09-17T15:46:36.474 INFO:tasks.rados.rados.0.smithi203.stdout:3099: done (2 left) 2024-09-17T15:46:36.474 INFO:tasks.rados.rados.0.smithi203.stdout:3100: delete oid 36 current snap is 334 2024-09-17T15:46:36.474 INFO:tasks.rados.rados.0.smithi203.stdout:3097: expect (ObjNum 994 snap 311 seq_num 994) 2024-09-17T15:46:36.474 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:36.530 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 199826 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:3098: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:3098: finishing copy_from to smithi20328086-18 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 1183 (ObjNum 985 snap 308 seq_num 985) dirty exists 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:3097: done (2 left) 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:3098: done (1 left) 2024-09-17T15:46:36.613 INFO:tasks.rados.rados.0.smithi203.stdout:3100: done (0 left) 2024-09-17T15:46:36.614 INFO:tasks.rados.rados.0.smithi203.stdout:3101: write oid 16 current snap is 334 2024-09-17T15:46:36.614 INFO:tasks.rados.rados.0.smithi203.stdout:3101: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-17T15:46:36.631 INFO:tasks.rados.rados.0.smithi203.stdout:3101: writing smithi20328086-16 from 618788 to 1110830 tid 1 2024-09-17T15:46:36.649 INFO:tasks.rados.rados.0.smithi203.stdout:3101: writing smithi20328086-16 from 1883069 to 2218788 tid 2 2024-09-17T15:46:36.649 INFO:tasks.rados.rados.0.smithi203.stdout:3102: snap_create 2024-09-17T15:46:36.649 INFO:tasks.rados.rados.0.smithi203.stdout:3101: finishing write tid 1 to smithi20328086-16 2024-09-17T15:46:36.658 INFO:tasks.rados.rados.0.smithi203.stdout:3101: finishing write tid 2 to smithi20328086-16 2024-09-17T15:46:36.658 INFO:tasks.rados.rados.0.smithi203.stdout:3101: finishing write tid 3 to smithi20328086-16 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:3101: finishing write tid 4 to smithi20328086-16 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:3101: oid 16 updating version 0 to 1566 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:3101: oid 16 updating version 1566 to 1568 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:3101: oid 16 version 1568 is already newer than 1567 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1568 (ObjNum 1084 snap 334 seq_num 1084) dirty exists 2024-09-17T15:46:36.659 INFO:tasks.rados.rados.0.smithi203.stdout:3101: left oid 16 (ObjNum 1084 snap 334 seq_num 1084) 2024-09-17T15:46:37.486 INFO:tasks.rados.rados.0.smithi203.stdout:3101: done (1 left) 2024-09-17T15:46:37.486 INFO:tasks.rados.rados.0.smithi203.stdout:3102: done (0 left) 2024-09-17T15:46:37.486 INFO:tasks.rados.rados.0.smithi203.stdout:3103: rollback oid 15 current snap is 335 2024-09-17T15:46:37.486 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 313 2024-09-17T15:46:37.486 INFO:tasks.rados.rados.0.smithi203.stdout:3104: delete oid 35 current snap is 335 2024-09-17T15:46:37.501 INFO:tasks.rados.rados.0.smithi203.stdout:3104: done (1 left) 2024-09-17T15:46:37.501 INFO:tasks.rados.rados.0.smithi203.stdout:3105: write (excl) oid 36 current snap is 335 2024-09-17T15:46:37.501 INFO:tasks.rados.rados.0.smithi203.stdout:3105: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-17T15:46:37.518 INFO:tasks.rados.rados.0.smithi203.stdout:3105: writing smithi20328086-36 from 652402 to 1294942 tid 1 2024-09-17T15:46:37.535 INFO:tasks.rados.rados.0.smithi203.stdout:3105: writing smithi20328086-36 from 1883680 to 2252402 tid 2 2024-09-17T15:46:37.535 INFO:tasks.rados.rados.0.smithi203.stdout:3106: read oid 28 snap 312 2024-09-17T15:46:37.535 INFO:tasks.rados.rados.0.smithi203.stdout:3106: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:3107: snap_create 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:3103: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:3103: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:3103: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1296 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:46:37.536 INFO:tasks.rados.rados.0.smithi203.stdout:3105: finishing write tid 1 to smithi20328086-36 2024-09-17T15:46:37.551 INFO:tasks.rados.rados.0.smithi203.stdout:3105: finishing write tid 2 to smithi20328086-36 2024-09-17T15:46:37.551 INFO:tasks.rados.rados.0.smithi203.stdout:3105: finishing write tid 3 to smithi20328086-36 2024-09-17T15:46:37.551 INFO:tasks.rados.rados.0.smithi203.stdout:3105: finishing write tid 4 to smithi20328086-36 2024-09-17T15:46:37.551 INFO:tasks.rados.rados.0.smithi203.stdout:3105: oid 36 updating version 0 to 1635 2024-09-17T15:46:37.551 INFO:tasks.rados.rados.0.smithi203.stdout:3105: oid 36 updating version 1635 to 1636 2024-09-17T15:46:37.552 INFO:tasks.rados.rados.0.smithi203.stdout:3105: oid 36 updating version 1636 to 1637 2024-09-17T15:46:37.552 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1637 (ObjNum 1086 snap 335 seq_num 1086) dirty exists 2024-09-17T15:46:37.552 INFO:tasks.rados.rados.0.smithi203.stdout:3105: left oid 36 (ObjNum 1086 snap 335 seq_num 1086) 2024-09-17T15:46:37.552 INFO:tasks.rados.rados.0.smithi203.stdout:3106: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:46:37.552 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:38.494 INFO:tasks.rados.rados.0.smithi203.stdout:3103: done (3 left) 2024-09-17T15:46:38.495 INFO:tasks.rados.rados.0.smithi203.stdout:3105: done (2 left) 2024-09-17T15:46:38.495 INFO:tasks.rados.rados.0.smithi203.stdout:3106: done (1 left) 2024-09-17T15:46:38.495 INFO:tasks.rados.rados.0.smithi203.stdout:3107: done (0 left) 2024-09-17T15:46:38.495 INFO:tasks.rados.rados.0.smithi203.stdout:3108: delete oid 21 current snap is 336 2024-09-17T15:46:38.503 INFO:tasks.rados.rados.0.smithi203.stdout:3108: done (0 left) 2024-09-17T15:46:38.503 INFO:tasks.rados.rados.0.smithi203.stdout:3109: read oid 23 snap -1 2024-09-17T15:46:38.503 INFO:tasks.rados.rados.0.smithi203.stdout:3109: expect deleted 2024-09-17T15:46:38.504 INFO:tasks.rados.rados.0.smithi203.stdout:3110: write (excl) oid 14 current snap is 336 2024-09-17T15:46:38.504 INFO:tasks.rados.rados.0.smithi203.stdout:3110: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-09-17T15:46:38.523 INFO:tasks.rados.rados.0.smithi203.stdout:3110: writing smithi20328086-14 from 686016 to 1195407 tid 1 2024-09-17T15:46:38.541 INFO:tasks.rados.rados.0.smithi203.stdout:3110: writing smithi20328086-14 from 1716997 to 2286016 tid 2 2024-09-17T15:46:38.541 INFO:tasks.rados.rados.0.smithi203.stdout:3111: delete oid 1 current snap is 336 2024-09-17T15:46:38.542 INFO:tasks.rados.rados.0.smithi203.stdout:3110: finishing write tid 1 to smithi20328086-14 2024-09-17T15:46:38.554 INFO:tasks.rados.rados.0.smithi203.stdout:3110: finishing write tid 2 to smithi20328086-14 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:3110: finishing write tid 3 to smithi20328086-14 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:3110: finishing write tid 4 to smithi20328086-14 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:3110: oid 14 updating version 0 to 1534 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:3110: oid 14 version 1534 is already newer than 1533 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:3110: oid 14 version 1534 is already newer than 1532 2024-09-17T15:46:38.555 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1534 (ObjNum 1088 snap 336 seq_num 1088) dirty exists 2024-09-17T15:46:38.556 INFO:tasks.rados.rados.0.smithi203.stdout:3110: left oid 14 (ObjNum 1088 snap 336 seq_num 1088) 2024-09-17T15:46:38.556 INFO:tasks.rados.rados.0.smithi203.stdout:3109: done (2 left) 2024-09-17T15:46:38.556 INFO:tasks.rados.rados.0.smithi203.stdout:3110: done (1 left) 2024-09-17T15:46:38.556 INFO:tasks.rados.rados.0.smithi203.stdout:3111: done (0 left) 2024-09-17T15:46:38.556 INFO:tasks.rados.rados.0.smithi203.stdout:3112: snap_create 2024-09-17T15:46:38.846 INFO:tasks.rados.rados.0.smithi203.stdout:3112: done (0 left) 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3113: copy_from oid 25 from oid 29 current snap is 337 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3114: read oid 6 snap 332 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3114: expect (ObjNum 1072 snap 330 seq_num 1072) 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3115: copy_from oid 20 from oid 17 current snap is 337 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3116: read oid 2 snap -1 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3116: expect (ObjNum 1074 snap 332 seq_num 1074) 2024-09-17T15:46:38.847 INFO:tasks.rados.rados.0.smithi203.stdout:3117: snap_create 2024-09-17T15:46:38.855 INFO:tasks.rados.rados.0.smithi203.stdout:3116: expect (ObjNum 1074 snap 332 seq_num 1074) 2024-09-17T15:46:38.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1386814 2024-09-17T15:46:38.894 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:38.962 INFO:tasks.rados.rados.0.smithi203.stdout:3114: expect (ObjNum 1072 snap 330 seq_num 1072) 2024-09-17T15:46:38.963 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 55279 2024-09-17T15:46:39.005 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 681161 2024-09-17T15:46:39.036 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:39.074 INFO:tasks.rados.rados.0.smithi203.stdout:3115: finishing copy_from to smithi20328086-20 2024-09-17T15:46:39.075 INFO:tasks.rados.rados.0.smithi203.stdout:3115: got expected ENOENT (src dne) 2024-09-17T15:46:39.075 INFO:tasks.rados.rados.0.smithi203.stdout:3115: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:46:39.075 INFO:tasks.rados.rados.0.smithi203.stdout:3113: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:46:39.075 INFO:tasks.rados.rados.0.smithi203.stdout:3113: finishing copy_from to smithi20328086-25 2024-09-17T15:46:39.075 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1517 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:46:39.856 INFO:tasks.rados.rados.0.smithi203.stdout:3113: done (4 left) 2024-09-17T15:46:39.856 INFO:tasks.rados.rados.0.smithi203.stdout:3114: done (3 left) 2024-09-17T15:46:39.856 INFO:tasks.rados.rados.0.smithi203.stdout:3115: done (2 left) 2024-09-17T15:46:39.856 INFO:tasks.rados.rados.0.smithi203.stdout:3116: done (1 left) 2024-09-17T15:46:39.856 INFO:tasks.rados.rados.0.smithi203.stdout:3117: done (0 left) 2024-09-17T15:46:39.857 INFO:tasks.rados.rados.0.smithi203.stdout:3118: delete oid 25 current snap is 338 2024-09-17T15:46:39.869 INFO:tasks.rados.rados.0.smithi203.stdout:3118: done (0 left) 2024-09-17T15:46:39.869 INFO:tasks.rados.rados.0.smithi203.stdout:3119: write (excl) oid 22 current snap is 338 2024-09-17T15:46:39.869 INFO:tasks.rados.rados.0.smithi203.stdout:3119: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-17T15:46:39.892 INFO:tasks.rados.rados.0.smithi203.stdout:3119: writing smithi20328086-22 from 736437 to 1471575 tid 1 2024-09-17T15:46:39.907 INFO:tasks.rados.rados.0.smithi203.stdout:3119: writing smithi20328086-22 from 2176090 to 2336437 tid 2 2024-09-17T15:46:39.907 INFO:tasks.rados.rados.0.smithi203.stdout:3120: snap_remove snap 333 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3120: done (1 left) 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3121: copy_from oid 29 from oid 20 current snap is 338 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: finishing write tid 1 to smithi20328086-22 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: finishing write tid 2 to smithi20328086-22 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: finishing write tid 3 to smithi20328086-22 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: finishing write tid 4 to smithi20328086-22 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: oid 22 updating version 0 to 1308 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: oid 22 updating version 1308 to 1309 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:3119: oid 22 version 1309 is already newer than 1307 2024-09-17T15:46:40.927 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1309 (ObjNum 1091 snap 338 seq_num 1091) dirty exists 2024-09-17T15:46:40.928 INFO:tasks.rados.rados.0.smithi203.stdout:3119: left oid 22 (ObjNum 1091 snap 338 seq_num 1091) 2024-09-17T15:46:40.928 INFO:tasks.rados.rados.0.smithi203.stdout:3119: done (1 left) 2024-09-17T15:46:40.928 INFO:tasks.rados.rados.0.smithi203.stdout:3122: snap_remove snap 312 2024-09-17T15:46:41.868 INFO:tasks.rados.rados.0.smithi203.stdout:3122: done (1 left) 2024-09-17T15:46:41.868 INFO:tasks.rados.rados.0.smithi203.stdout:3123: delete oid 6 current snap is 338 2024-09-17T15:46:41.868 INFO:tasks.rados.rados.0.smithi203.stdout:3121: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:46:41.868 INFO:tasks.rados.rados.0.smithi203.stdout:3121: finishing copy_from to smithi20328086-29 2024-09-17T15:46:41.868 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1344 (ObjNum 1080 snap 334 seq_num 1080) dirty exists 2024-09-17T15:46:41.884 INFO:tasks.rados.rados.0.smithi203.stdout:3121: done (1 left) 2024-09-17T15:46:41.885 INFO:tasks.rados.rados.0.smithi203.stdout:3123: done (0 left) 2024-09-17T15:46:41.885 INFO:tasks.rados.rados.0.smithi203.stdout:3124: snap_create 2024-09-17T15:46:43.118 INFO:tasks.rados.rados.0.smithi203.stdout:3124: done (0 left) 2024-09-17T15:46:43.118 INFO:tasks.rados.rados.0.smithi203.stdout:3125: snap_create 2024-09-17T15:46:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:3125: done (0 left) 2024-09-17T15:46:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:3126: rollback oid 47 current snap is 340 2024-09-17T15:46:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 334 2024-09-17T15:46:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:3127: write (excl) oid 3 current snap is 340 2024-09-17T15:46:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:3127: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-17T15:46:43.868 INFO:tasks.rados.rados.0.smithi203.stdout:3127: writing smithi20328086-3 from 770051 to 1255687 tid 1 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3127: writing smithi20328086-3 from 1776701 to 2287748 tid 2 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3128: delete oid 16 current snap is 340 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3126: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3126: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3126: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1527 (ObjNum 1034 snap 319 seq_num 1034) dirty exists 2024-09-17T15:46:43.886 INFO:tasks.rados.rados.0.smithi203.stdout:3127: finishing write tid 1 to smithi20328086-3 2024-09-17T15:46:43.894 INFO:tasks.rados.rados.0.smithi203.stdout:3127: finishing write tid 2 to smithi20328086-3 2024-09-17T15:46:43.895 INFO:tasks.rados.rados.0.smithi203.stdout:3127: finishing write tid 3 to smithi20328086-3 2024-09-17T15:46:43.895 INFO:tasks.rados.rados.0.smithi203.stdout:3127: finishing write tid 4 to smithi20328086-3 2024-09-17T15:46:43.895 INFO:tasks.rados.rados.0.smithi203.stdout:3127: oid 3 updating version 0 to 1392 2024-09-17T15:46:43.895 INFO:tasks.rados.rados.0.smithi203.stdout:3127: oid 3 version 1392 is already newer than 1391 2024-09-17T15:46:43.895 INFO:tasks.rados.rados.0.smithi203.stdout:3127: oid 3 updating version 1392 to 1393 2024-09-17T15:46:43.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1393 (ObjNum 1093 snap 340 seq_num 1093) dirty exists 2024-09-17T15:46:43.896 INFO:tasks.rados.rados.0.smithi203.stdout:3127: left oid 3 (ObjNum 1093 snap 340 seq_num 1093) 2024-09-17T15:46:43.897 INFO:tasks.rados.rados.0.smithi203.stdout:3126: done (2 left) 2024-09-17T15:46:43.897 INFO:tasks.rados.rados.0.smithi203.stdout:3127: done (1 left) 2024-09-17T15:46:43.897 INFO:tasks.rados.rados.0.smithi203.stdout:3128: done (0 left) 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3129: read oid 30 snap -1 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3129: expect deleted 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3130: rollback oid 9 current snap is 340 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 335 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3131: write oid 22 current snap is 340 2024-09-17T15:46:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3131: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-17T15:46:43.917 INFO:tasks.rados.rados.0.smithi203.stdout:3131: writing smithi20328086-22 from 403665 to 1156152 tid 1 2024-09-17T15:46:43.932 INFO:tasks.rados.rados.0.smithi203.stdout:3131: writing smithi20328086-22 from 1610018 to 2124734 tid 2 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:3132: delete oid 10 current snap is 340 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:3130: finishing rollback tid 0 to smithi20328086-9 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:3130: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:3130: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1521 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:46:43.933 INFO:tasks.rados.rados.0.smithi203.stdout:3131: finishing write tid 1 to smithi20328086-22 2024-09-17T15:46:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:3129: done (3 left) 2024-09-17T15:46:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:3130: done (2 left) 2024-09-17T15:46:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:3132: done (1 left) 2024-09-17T15:46:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:3133: write (excl) oid 40 current snap is 340 2024-09-17T15:46:43.938 INFO:tasks.rados.rados.0.smithi203.stdout:3133: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-17T15:46:43.950 INFO:tasks.rados.rados.0.smithi203.stdout:3133: writing smithi20328086-40 from 437279 to 940264 tid 1 2024-09-17T15:46:43.965 INFO:tasks.rados.rados.0.smithi203.stdout:3133: writing smithi20328086-40 from 1610629 to 2129014 tid 2 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3134: rollback oid 24 current snap is 340 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 336 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: finishing write tid 2 to smithi20328086-22 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: finishing write tid 3 to smithi20328086-22 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: finishing write tid 4 to smithi20328086-22 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: oid 22 updating version 0 to 1312 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: oid 22 version 1312 is already newer than 1310 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: oid 22 version 1312 is already newer than 1311 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1312 (ObjNum 1095 snap 340 seq_num 1095) dirty exists 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: left oid 22 (ObjNum 1095 snap 340 seq_num 1095) 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3133: finishing write tid 1 to smithi20328086-40 2024-09-17T15:46:43.966 INFO:tasks.rados.rados.0.smithi203.stdout:3131: done (2 left) 2024-09-17T15:46:43.967 INFO:tasks.rados.rados.0.smithi203.stdout:3135: write (excl) oid 46 current snap is 340 2024-09-17T15:46:43.967 INFO:tasks.rados.rados.0.smithi203.stdout:3135: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-09-17T15:46:43.978 INFO:tasks.rados.rados.0.smithi203.stdout:3135: writing smithi20328086-46 from 454086 to 1032320 tid 1 2024-09-17T15:46:43.990 INFO:tasks.rados.rados.0.smithi203.stdout:3135: writing smithi20328086-46 from 1669111 to 2331154 tid 2 2024-09-17T15:46:43.990 INFO:tasks.rados.rados.0.smithi203.stdout:3136: write oid 41 current snap is 340 2024-09-17T15:46:43.990 INFO:tasks.rados.rados.0.smithi203.stdout:3136: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-17T15:46:43.998 INFO:tasks.rados.rados.0.smithi203.stdout:3136: writing smithi20328086-41 from 470893 to 1124376 tid 1 2024-09-17T15:46:44.006 INFO:tasks.rados.rados.0.smithi203.stdout:3136: writing smithi20328086-41 from 1727593 to 2249647 tid 2 2024-09-17T15:46:44.007 INFO:tasks.rados.rados.0.smithi203.stdout:3137: write oid 42 current snap is 340 2024-09-17T15:46:44.007 INFO:tasks.rados.rados.0.smithi203.stdout:3137: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-17T15:46:44.014 INFO:tasks.rados.rados.0.smithi203.stdout:3137: writing smithi20328086-42 from 487700 to 1216432 tid 1 2024-09-17T15:46:44.022 INFO:tasks.rados.rados.0.smithi203.stdout:3137: writing smithi20328086-42 from 1786075 to 2451787 tid 2 2024-09-17T15:46:44.022 INFO:tasks.rados.rados.0.smithi203.stdout:3138: write (excl) oid 5 current snap is 340 2024-09-17T15:46:44.022 INFO:tasks.rados.rados.0.smithi203.stdout:3133: finishing write tid 2 to smithi20328086-40 2024-09-17T15:46:44.023 INFO:tasks.rados.rados.0.smithi203.stdout:3138: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-09-17T15:46:44.027 INFO:tasks.rados.rados.0.smithi203.stdout:3138: writing smithi20328086-5 from 504507 to 908488 tid 1 2024-09-17T15:46:44.032 INFO:tasks.rados.rados.0.smithi203.stdout:3138: writing smithi20328086-5 from 1328204 to 1737574 tid 2 2024-09-17T15:46:44.037 INFO:tasks.rados.rados.0.smithi203.stdout:3138: writing smithi20328086-5 from 2252362 to 2504507 tid 3 2024-09-17T15:46:44.037 INFO:tasks.rados.rados.0.smithi203.stdout:3139: delete oid 4 current snap is 340 2024-09-17T15:46:44.037 INFO:tasks.rados.rados.0.smithi203.stdout:3133: finishing write tid 3 to smithi20328086-40 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3134: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3133: finishing write tid 4 to smithi20328086-40 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3133: oid 40 updating version 0 to 1511 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3133: oid 40 version 1511 is already newer than 1510 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3133: oid 40 version 1511 is already newer than 1509 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1511 (ObjNum 1097 snap 340 seq_num 1097) dirty exists 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3133: left oid 40 (ObjNum 1097 snap 340 seq_num 1097) 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3134: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3134: finishing rollback tid 2 to smithi20328086-24 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1524 (ObjNum 1050 snap 324 seq_num 1050) dirty exists 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3135: finishing write tid 1 to smithi20328086-46 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3135: finishing write tid 2 to smithi20328086-46 2024-09-17T15:46:44.038 INFO:tasks.rados.rados.0.smithi203.stdout:3135: finishing write tid 3 to smithi20328086-46 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3135: finishing write tid 4 to smithi20328086-46 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3135: oid 46 updating version 0 to 1394 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3135: oid 46 updating version 1394 to 1396 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3135: oid 46 version 1396 is already newer than 1395 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1396 (ObjNum 1098 snap 340 seq_num 1098) dirty exists 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3135: left oid 46 (ObjNum 1098 snap 340 seq_num 1098) 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: finishing write tid 1 to smithi20328086-41 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: finishing write tid 2 to smithi20328086-41 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: finishing write tid 3 to smithi20328086-41 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: finishing write tid 4 to smithi20328086-41 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: oid 41 updating version 0 to 661 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: oid 41 updating version 661 to 662 2024-09-17T15:46:44.039 INFO:tasks.rados.rados.0.smithi203.stdout:3136: oid 41 updating version 662 to 663 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 663 (ObjNum 1099 snap 340 seq_num 1099) dirty exists 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3136: left oid 41 (ObjNum 1099 snap 340 seq_num 1099) 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: finishing write tid 1 to smithi20328086-42 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: finishing write tid 2 to smithi20328086-42 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: finishing write tid 3 to smithi20328086-42 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: finishing write tid 4 to smithi20328086-42 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: oid 42 updating version 0 to 1397 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: oid 42 updating version 1397 to 1398 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: oid 42 updating version 1398 to 1399 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1399 (ObjNum 1100 snap 340 seq_num 1100) dirty exists 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3137: left oid 42 (ObjNum 1100 snap 340 seq_num 1100) 2024-09-17T15:46:44.040 INFO:tasks.rados.rados.0.smithi203.stdout:3138: finishing write tid 1 to smithi20328086-5 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: finishing write tid 2 to smithi20328086-5 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: finishing write tid 3 to smithi20328086-5 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: finishing write tid 4 to smithi20328086-5 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: finishing write tid 5 to smithi20328086-5 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: oid 5 updating version 0 to 1251 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: oid 5 updating version 1251 to 1252 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: oid 5 version 1252 is already newer than 1249 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: oid 5 version 1252 is already newer than 1250 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1252 (ObjNum 1101 snap 340 seq_num 1101) dirty exists 2024-09-17T15:46:44.042 INFO:tasks.rados.rados.0.smithi203.stdout:3138: left oid 5 (ObjNum 1101 snap 340 seq_num 1101) 2024-09-17T15:46:44.043 INFO:tasks.rados.rados.0.smithi203.stdout:3133: done (6 left) 2024-09-17T15:46:44.043 INFO:tasks.rados.rados.0.smithi203.stdout:3134: done (5 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3135: done (4 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3136: done (3 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3137: done (2 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3138: done (1 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3139: done (0 left) 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3140: rollback oid 34 current snap is 340 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 321 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3141: read oid 13 snap -1 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3141: expect deleted 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3142: write oid 3 current snap is 340 2024-09-17T15:46:44.044 INFO:tasks.rados.rados.0.smithi203.stdout:3142: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-17T15:46:44.050 INFO:tasks.rados.rados.0.smithi203.stdout:3142: writing smithi20328086-3 from 538121 to 1208953 tid 1 2024-09-17T15:46:44.057 INFO:tasks.rados.rados.0.smithi203.stdout:3142: writing smithi20328086-3 from 1961521 to 2374560 tid 2 2024-09-17T15:46:44.058 INFO:tasks.rados.rados.0.smithi203.stdout:3143: delete oid 12 current snap is 340 2024-09-17T15:46:44.058 INFO:tasks.rados.rados.0.smithi203.stdout:3140: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:46:44.058 INFO:tasks.rados.rados.0.smithi203.stdout:3140: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:46:44.058 INFO:tasks.rados.rados.0.smithi203.stdout:3140: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:46:44.058 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1304 (ObjNum 1022 snap 316 seq_num 1022) dirty exists 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3140: done (3 left) 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3141: done (2 left) 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3143: done (1 left) 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3144: read oid 38 snap -1 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3144: expect deleted 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3145: write (excl) oid 16 current snap is 340 2024-09-17T15:46:44.061 INFO:tasks.rados.rados.0.smithi203.stdout:3145: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-17T15:46:44.066 INFO:tasks.rados.rados.0.smithi203.stdout:3145: writing smithi20328086-16 from 571735 to 993065 tid 1 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3145: writing smithi20328086-16 from 1562132 to 1978840 tid 2 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3146: read oid 5 snap 334 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3146: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3147: read oid 46 snap -1 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3147: expect (ObjNum 1098 snap 340 seq_num 1098) 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3148: read oid 40 snap 327 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3148: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3149: snap_create 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3142: finishing write tid 1 to smithi20328086-3 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3142: finishing write tid 2 to smithi20328086-3 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3142: finishing write tid 3 to smithi20328086-3 2024-09-17T15:46:44.073 INFO:tasks.rados.rados.0.smithi203.stdout:3142: finishing write tid 4 to smithi20328086-3 2024-09-17T15:46:44.074 INFO:tasks.rados.rados.0.smithi203.stdout:3142: oid 3 updating version 0 to 1402 2024-09-17T15:46:44.074 INFO:tasks.rados.rados.0.smithi203.stdout:3142: oid 3 version 1402 is already newer than 1401 2024-09-17T15:46:44.074 INFO:tasks.rados.rados.0.smithi203.stdout:3142: oid 3 version 1402 is already newer than 1400 2024-09-17T15:46:44.074 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1402 (ObjNum 1103 snap 340 seq_num 1103) dirty exists 2024-09-17T15:46:44.074 INFO:tasks.rados.rados.0.smithi203.stdout:3142: left oid 3 (ObjNum 1103 snap 340 seq_num 1103) 2024-09-17T15:46:44.077 INFO:tasks.rados.rados.0.smithi203.stdout:3145: finishing write tid 1 to smithi20328086-16 2024-09-17T15:46:44.077 INFO:tasks.rados.rados.0.smithi203.stdout:3146: expect (ObjNum 977 snap 299 seq_num 977) 2024-09-17T15:46:44.153 INFO:tasks.rados.rados.0.smithi203.stdout:3147: expect (ObjNum 1098 snap 340 seq_num 1098) 2024-09-17T15:46:44.155 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1594205 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: finishing write tid 2 to smithi20328086-16 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: finishing write tid 3 to smithi20328086-16 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: finishing write tid 4 to smithi20328086-16 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: oid 16 updating version 0 to 1582 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: oid 16 version 1582 is already newer than 1581 2024-09-17T15:46:44.201 INFO:tasks.rados.rados.0.smithi203.stdout:3145: oid 16 updating version 1582 to 1583 2024-09-17T15:46:44.202 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1583 (ObjNum 1105 snap 340 seq_num 1105) dirty exists 2024-09-17T15:46:44.202 INFO:tasks.rados.rados.0.smithi203.stdout:3145: left oid 16 (ObjNum 1105 snap 340 seq_num 1105) 2024-09-17T15:46:44.202 INFO:tasks.rados.rados.0.smithi203.stdout:3148: expect (ObjNum 975 snap 295 seq_num 975) 2024-09-17T15:46:44.270 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 796594 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3142: done (6 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3144: done (5 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3145: done (4 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3146: done (3 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3147: done (2 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3148: done (1 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3149: done (0 left) 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3150: copy_from oid 32 from oid 20 current snap is 341 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3151: write (excl) oid 48 current snap is 341 2024-09-17T15:46:44.894 INFO:tasks.rados.rados.0.smithi203.stdout:3151: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-17T15:46:44.903 INFO:tasks.rados.rados.0.smithi203.stdout:3151: writing smithi20328086-48 from 588542 to 1085121 tid 1 2024-09-17T15:46:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:3151: writing smithi20328086-48 from 1620614 to 2297333 tid 2 2024-09-17T15:46:44.912 INFO:tasks.rados.rados.0.smithi203.stdout:3152: read oid 36 snap -1 2024-09-17T15:46:44.912 INFO:tasks.rados.rados.0.smithi203.stdout:3152: expect (ObjNum 1086 snap 335 seq_num 1086) 2024-09-17T15:46:44.912 INFO:tasks.rados.rados.0.smithi203.stdout:3153: write oid 10 current snap is 341 2024-09-17T15:46:44.912 INFO:tasks.rados.rados.0.smithi203.stdout:3153: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-09-17T15:46:44.921 INFO:tasks.rados.rados.0.smithi203.stdout:3153: writing smithi20328086-10 from 605349 to 1177177 tid 1 2024-09-17T15:46:44.927 INFO:tasks.rados.rados.0.smithi203.stdout:3153: writing smithi20328086-10 from 1679096 to 2099473 tid 2 2024-09-17T15:46:44.928 INFO:tasks.rados.rados.0.smithi203.stdout:3154: write oid 43 current snap is 341 2024-09-17T15:46:44.928 INFO:tasks.rados.rados.0.smithi203.stdout:3154: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-17T15:46:44.936 INFO:tasks.rados.rados.0.smithi203.stdout:3154: writing smithi20328086-43 from 622156 to 1269233 tid 1 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3154: writing smithi20328086-43 from 1737578 to 2417966 tid 2 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3155: read oid 3 snap -1 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3155: expect (ObjNum 1103 snap 340 seq_num 1103) 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3156: read oid 1 snap -1 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3156: expect deleted 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3157: rollback oid 40 current snap is 341 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 336 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:3152: expect (ObjNum 1086 snap 335 seq_num 1086) 2024-09-17T15:46:44.944 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 606355 2024-09-17T15:46:44.998 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2193649 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3151: finishing write tid 1 to smithi20328086-48 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3152: done (7 left) 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3158: rollback oid 15 current snap is 341 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 335 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3151: finishing write tid 2 to smithi20328086-48 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3151: finishing write tid 3 to smithi20328086-48 2024-09-17T15:46:45.018 INFO:tasks.rados.rados.0.smithi203.stdout:3150: finishing copy_from racing read to smithi20328086-32 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3151: finishing write tid 4 to smithi20328086-48 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3151: oid 48 updating version 0 to 1586 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3151: oid 48 updating version 1586 to 1587 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3151: oid 48 version 1587 is already newer than 1585 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1587 (ObjNum 1106 snap 341 seq_num 1106) dirty exists 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3151: left oid 48 (ObjNum 1106 snap 341 seq_num 1106) 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3150: finishing copy_from to smithi20328086-32 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1344 (ObjNum 1080 snap 334 seq_num 1080) dirty exists 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: finishing write tid 1 to smithi20328086-10 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: finishing write tid 2 to smithi20328086-10 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: finishing write tid 3 to smithi20328086-10 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: finishing write tid 4 to smithi20328086-10 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: oid 10 updating version 0 to 1374 2024-09-17T15:46:45.019 INFO:tasks.rados.rados.0.smithi203.stdout:3153: oid 10 updating version 1374 to 1376 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3153: oid 10 version 1376 is already newer than 1375 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1376 (ObjNum 1107 snap 341 seq_num 1107) dirty exists 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3153: left oid 10 (ObjNum 1107 snap 341 seq_num 1107) 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: finishing write tid 1 to smithi20328086-43 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: finishing write tid 2 to smithi20328086-43 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: finishing write tid 3 to smithi20328086-43 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: finishing write tid 4 to smithi20328086-43 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: oid 43 updating version 0 to 665 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: oid 43 version 665 is already newer than 664 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: oid 43 updating version 665 to 666 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 666 (ObjNum 1108 snap 341 seq_num 1108) dirty exists 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3154: left oid 43 (ObjNum 1108 snap 341 seq_num 1108) 2024-09-17T15:46:45.020 INFO:tasks.rados.rados.0.smithi203.stdout:3155: expect (ObjNum 1103 snap 340 seq_num 1103) 2024-09-17T15:46:45.029 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1412410 2024-09-17T15:46:45.056 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:45.096 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:45.136 INFO:tasks.rados.rados.0.smithi203.stdout:3157: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:46:45.136 INFO:tasks.rados.rados.0.smithi203.stdout:3157: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:46:45.136 INFO:tasks.rados.rados.0.smithi203.stdout:3157: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1514 (ObjNum 1067 snap 330 seq_num 1067) dirty exists 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3150: done (7 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3151: done (6 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3153: done (5 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3154: done (4 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3155: done (3 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3156: done (2 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3157: done (1 left) 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3159: read oid 1 snap -1 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3159: expect deleted 2024-09-17T15:46:45.137 INFO:tasks.rados.rados.0.smithi203.stdout:3160: rollback oid 3 current snap is 341 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 337 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3158: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3158: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3158: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1299 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3158: done (2 left) 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3161: write oid 24 current snap is 341 2024-09-17T15:46:45.138 INFO:tasks.rados.rados.0.smithi203.stdout:3161: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-17T15:46:45.144 INFO:tasks.rados.rados.0.smithi203.stdout:3161: writing smithi20328086-24 from 638963 to 1361289 tid 1 2024-09-17T15:46:45.150 INFO:tasks.rados.rados.0.smithi203.stdout:3161: writing smithi20328086-24 from 2079707 to 2503753 tid 2 2024-09-17T15:46:45.151 INFO:tasks.rados.rados.0.smithi203.stdout:3162: write oid 41 current snap is 341 2024-09-17T15:46:45.151 INFO:tasks.rados.rados.0.smithi203.stdout:3162: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-09-17T15:46:45.158 INFO:tasks.rados.rados.0.smithi203.stdout:3162: writing smithi20328086-41 from 655770 to 1169698 tid 1 2024-09-17T15:46:45.166 INFO:tasks.rados.rados.0.smithi203.stdout:3162: writing smithi20328086-41 from 1854542 to 2538599 tid 2 2024-09-17T15:46:45.166 INFO:tasks.rados.rados.0.smithi203.stdout:3163: snap_remove snap 321 2024-09-17T15:46:45.167 INFO:tasks.rados.rados.0.smithi203.stdout:3160: finishing rollback tid 0 to smithi20328086-3 2024-09-17T15:46:45.167 INFO:tasks.rados.rados.0.smithi203.stdout:3160: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:46:45.167 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1404 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3159: done (4 left) 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3160: done (3 left) 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3163: done (2 left) 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3164: rollback oid 40 current snap is 341 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: finishing write tid 1 to smithi20328086-24 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: finishing write tid 2 to smithi20328086-24 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: finishing write tid 3 to smithi20328086-24 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: finishing write tid 4 to smithi20328086-24 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: oid 24 updating version 0 to 1526 2024-09-17T15:46:45.895 INFO:tasks.rados.rados.0.smithi203.stdout:3161: oid 24 updating version 1526 to 1527 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3161: oid 24 version 1527 is already newer than 1525 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1527 (ObjNum 1109 snap 341 seq_num 1109) dirty exists 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3161: left oid 24 (ObjNum 1109 snap 341 seq_num 1109) 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: finishing write tid 1 to smithi20328086-41 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: finishing write tid 2 to smithi20328086-41 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: finishing write tid 3 to smithi20328086-41 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: finishing write tid 4 to smithi20328086-41 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: oid 41 updating version 0 to 669 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: oid 41 version 669 is already newer than 667 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: oid 41 version 669 is already newer than 668 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 669 (ObjNum 1110 snap 341 seq_num 1110) dirty exists 2024-09-17T15:46:45.896 INFO:tasks.rados.rados.0.smithi203.stdout:3162: left oid 41 (ObjNum 1110 snap 341 seq_num 1110) 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 338 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:3161: done (2 left) 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:3162: done (1 left) 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:3165: copy_from oid 47 from oid 2 current snap is 341 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:3166: write (excl) oid 32 current snap is 341 2024-09-17T15:46:45.897 INFO:tasks.rados.rados.0.smithi203.stdout:3166: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-17T15:46:45.906 INFO:tasks.rados.rados.0.smithi203.stdout:3166: writing smithi20328086-32 from 672577 to 1261754 tid 1 2024-09-17T15:46:45.917 INFO:tasks.rados.rados.0.smithi203.stdout:3166: writing smithi20328086-32 from 1913024 to 2340739 tid 2 2024-09-17T15:46:45.917 INFO:tasks.rados.rados.0.smithi203.stdout:3167: delete oid 31 current snap is 341 2024-09-17T15:46:45.923 INFO:tasks.rados.rados.0.smithi203.stdout:3164: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:46:45.926 INFO:tasks.rados.rados.0.smithi203.stdout:3166: finishing write tid 1 to smithi20328086-32 2024-09-17T15:46:45.927 INFO:tasks.rados.rados.0.smithi203.stdout:3167: done (3 left) 2024-09-17T15:46:45.928 INFO:tasks.rados.rados.0.smithi203.stdout:3168: read oid 8 snap -1 2024-09-17T15:46:45.928 INFO:tasks.rados.rados.0.smithi203.stdout:3168: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:46:45.928 INFO:tasks.rados.rados.0.smithi203.stdout:3169: snap_remove snap 332 2024-09-17T15:46:47.253 INFO:tasks.rados.rados.0.smithi203.stdout:3169: done (4 left) 2024-09-17T15:46:47.253 INFO:tasks.rados.rados.0.smithi203.stdout:3170: snap_remove snap 327 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3170: done (4 left) 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3171: rollback oid 34 current snap is 341 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 337 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: finishing write tid 2 to smithi20328086-32 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: finishing write tid 3 to smithi20328086-32 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: finishing write tid 4 to smithi20328086-32 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: oid 32 updating version 0 to 1346 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: oid 32 version 1346 is already newer than 1345 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:3166: oid 32 updating version 1346 to 1347 2024-09-17T15:46:48.260 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1347 (ObjNum 1111 snap 341 seq_num 1111) dirty exists 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:3166: left oid 32 (ObjNum 1111 snap 341 seq_num 1111) 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:3164: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:3164: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1517 (ObjNum 1067 snap 330 seq_num 1067) dirty exists 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:3165: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:46:48.261 INFO:tasks.rados.rados.0.smithi203.stdout:3168: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:46:48.370 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 478839 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3165: finishing copy_from to smithi20328086-47 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1195 (ObjNum 1074 snap 332 seq_num 1074) dirty exists 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3171: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3171: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3171: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1307 (ObjNum 1022 snap 316 seq_num 1022) dirty exists 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3164: done (4 left) 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3165: done (3 left) 2024-09-17T15:46:48.411 INFO:tasks.rados.rados.0.smithi203.stdout:3166: done (2 left) 2024-09-17T15:46:48.412 INFO:tasks.rados.rados.0.smithi203.stdout:3168: done (1 left) 2024-09-17T15:46:48.412 INFO:tasks.rados.rados.0.smithi203.stdout:3171: done (0 left) 2024-09-17T15:46:48.412 INFO:tasks.rados.rados.0.smithi203.stdout:3172: read oid 23 snap -1 2024-09-17T15:46:48.412 INFO:tasks.rados.rados.0.smithi203.stdout:3172: expect deleted 2024-09-17T15:46:48.412 INFO:tasks.rados.rados.0.smithi203.stdout:3173: snap_create 2024-09-17T15:46:48.857 INFO:tasks.rados.rados.0.smithi203.stdout:3172: done (1 left) 2024-09-17T15:46:48.857 INFO:tasks.rados.rados.0.smithi203.stdout:3173: done (0 left) 2024-09-17T15:46:48.857 INFO:tasks.rados.rados.0.smithi203.stdout:3174: rollback oid 36 current snap is 342 2024-09-17T15:46:48.857 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 36 to 340 2024-09-17T15:46:48.857 INFO:tasks.rados.rados.0.smithi203.stdout:3175: write oid 20 current snap is 342 2024-09-17T15:46:48.858 INFO:tasks.rados.rados.0.smithi203.stdout:3175: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-17T15:46:48.881 INFO:tasks.rados.rados.0.smithi203.stdout:3175: writing smithi20328086-20 from 706191 to 1445866 tid 1 2024-09-17T15:46:48.897 INFO:tasks.rados.rados.0.smithi203.stdout:3175: writing smithi20328086-20 from 1913635 to 2345019 tid 2 2024-09-17T15:46:48.897 INFO:tasks.rados.rados.0.smithi203.stdout:3176: snap_remove snap 337 2024-09-17T15:46:49.457 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:46:49.457 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:46:49.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:46:49.460 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:46:49.460 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:46:49.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:49.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: get_monmap_and_config 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: build_initial_monmap 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: monmap: 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:49.596888+0000 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:49.596888+0000 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:49.591 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:49.592 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:49.592 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:49.592 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 Processor -- start 2024-09-17T15:46:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 -- start start 2024-09-17T15:46:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: init 2024-09-17T15:46:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:49.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffa3c0689d0 0x7ffa3c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-a con 0x7ffa3c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-c con 0x7ffa3c068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa3c135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c09c8b0 con 0x7ffa3c135ae0 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ffa3c09cb90 con 0x7ffa3c0689d0 2024-09-17T15:46:49.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ffa3c09ce70 con 0x7ffa3c068db0 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa43a90640 10 monclient(hunting): authenticate will time out at 2142.562733s 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffa3c0689d0 0x7ffa3c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4202d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ffa3c135ae0 0x7ffa3c135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38852/0 (socket says 172.21.15.86:38852) 2024-09-17T15:46:49.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4202d640 1 -- 172.21.15.86:0/1658534559 learned_addr learned my addr 172.21.15.86:0/1658534559 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:49.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): get_auth_request con 0x7ffa3c0689d0 auth_method 0 2024-09-17T15:46:49.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:49.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:49.597 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:49.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 1 --2- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:49.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:49.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 10 monclient(hunting): get_auth_request con 0x7ffa3c068db0 auth_method 0 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 1 -- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 msgr2=0x7ffa3c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 1 -- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 msgr2=0x7ffa3c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 1 --2- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4102b640 1 --2- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient(hunting): handle_auth_done global_id 214188 payload 274 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 -- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 msgr2=0x7ffa3c069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 --2- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 -- 172.21.15.86:0/1658534559 >> v1:172.21.15.86:6789/0 conn(0x7ffa3c135ae0 legacy=0x7ffa3c135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:49.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient: _finish_hunting 0 2024-09-17T15:46:49.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 monclient: found mon.noname-a 2024-09-17T15:46:49.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:49.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:49.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.595+0000 7ffa4182c640 1 -- 172.21.15.86:0/1658534559 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c069740 con 0x7ffa3c0689d0 2024-09-17T15:46:49.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4182c640 1 -- 172.21.15.86:0/1658534559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0689d0 msgr2=0x7ffa3c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4182c640 1 --2- 172.21.15.86:0/1658534559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0689d0 0x7ffa3c06f840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0a05c0 tx=0x7ffa2c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3482e62e29a14fd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 1 -- 172.21.15.86:0/1658534559 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0b0070 con 0x7ffa3c0689d0 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: dump: 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:49.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: _finish_auth 0 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:49.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 1 -- 172.21.15.86:0/1658534559 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c0ab600 con 0x7ffa3c0689d0 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 1 -- 172.21.15.86:0/1658534559 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0ab9d0 con 0x7ffa3c0689d0 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa43a90640 5 monclient: authenticate success, global_id 214188 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa43a90640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa43a90640 4 set_mon_vals no callback set 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:49.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.599+0000 7ffa4082a640 10 monclient: dump: 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:49.609 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: shutdown 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- 172.21.15.86:0/1658534559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0689d0 msgr2=0x7ffa3c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- 172.21.15.86:0/1658534559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0689d0 0x7ffa3c06f840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0a05c0 tx=0x7ffa2c01c3d0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- 172.21.15.86:0/1658534559 shutdown_connections 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- 172.21.15.86:0/1658534559 >> v2:172.21.15.203:3300/0 conn(0x7ffa3c068db0 0x7ffa3c069200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- 172.21.15.86:0/1658534559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c0689d0 0x7ffa3c06f840 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- 172.21.15.86:0/1658534559 >> 172.21.15.86:0/1658534559 conn(0x7ffa3c0a6440 msgr2=0x7ffa3c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- 172.21.15.86:0/1658534559 shutdown_connections 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- 172.21.15.86:0/1658534559 wait complete. 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: build_initial_monmap 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: monmap: 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.611 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 Processor -- start 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- start start 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: init 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 0x7ffa3c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-c con 0x7ffa3c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-a con 0x7ffa3c068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa3c135ae0 0x7ffa3c09e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): picked mon.noname-b con 0x7ffa3c135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c140a40 con 0x7ffa3c068db0 2024-09-17T15:46:49.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:49.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c09c2d0 con 0x7ffa3c135ae0 2024-09-17T15:46:49.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:49.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c140bc0 con 0x7ffa3c0689d0 2024-09-17T15:46:49.613 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:49.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa43a90640 10 monclient(hunting): authenticate will time out at 2142.570253s 2024-09-17T15:46:49.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 0x7ffa3c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35560/0 (socket says 172.21.15.86:35560) 2024-09-17T15:46:49.614 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 1 -- 172.21.15.86:0/2151613601 learned_addr learned my addr 172.21.15.86:0/2151613601 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): get_auth_request con 0x7ffa3c068db0 auth_method 0 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): get_auth_request con 0x7ffa3c0689d0 auth_method 0 2024-09-17T15:46:49.615 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:49.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4202d640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa3c135ae0 0x7ffa3c09e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4202d640 10 monclient(hunting): get_auth_request con 0x7ffa3c135ae0 auth_method 0 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4202d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:49.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4202d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4202d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient(hunting): handle_auth_done global_id 164702 payload 824 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 -- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa3c135ae0 msgr2=0x7ffa3c09e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa3c135ae0 0x7ffa3c09e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 -- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 msgr2=0x7ffa3c09dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient: _finish_hunting 0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 monclient: found mon.noname-c 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4182c640 1 -- 172.21.15.86:0/2151613601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c121270 con 0x7ffa3c0689d0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.603+0000 7ffa4102b640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa4182c640 1 -- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 msgr2=0x7ffa3c09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa4182c640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 0x7ffa3c09d770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0ab000 tx=0x7ffa2c019170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0b8030 con 0x7ffa3c0689d0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:49.619 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: dump: 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:49.620 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c0ab400 con 0x7ffa3c0689d0 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 5 monclient: authenticate success, global_id 164702 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _renew_subs 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:49.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c121550 con 0x7ffa3c0689d0 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0b0040 con 0x7ffa3c0689d0 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 10 monclient: dump: 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa337fe640 4 set_mon_vals no callback set 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _renew_subs 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c0a2880 con 0x7ffa3c0689d0 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:49.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c06f840 con 0x7ffa3c0689d0 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ffa2c0b4840 con 0x7ffa3c0689d0 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 0x7ffa2003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(702..702 src has 438..702) ==== 4815+0+0 (secure 0 0 0) 0x7ffa2c0e94c0 con 0x7ffa3c0689d0 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa4102b640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 0x7ffa2003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:49.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.607+0000 7ffa4102b640 10 monclient: get_auth_request con 0x7ffa2003bb60 auth_method 0 2024-09-17T15:46:49.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.611+0000 7ffa4102b640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 0x7ffa2003e020 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c1d1180 tx=0x7ffa38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:49.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.611+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ffa2c0c2060 con 0x7ffa3c0689d0 2024-09-17T15:46:49.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.611+0000 7ffa327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:49.625 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.611+0000 7ffa327fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:49.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:49.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:49.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffa3c09ea10 con 0x7ffa3c0689d0 2024-09-17T15:46:49.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa327fc640 1 -- 172.21.15.86:0/2151613601 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ffa2c0a4e20 con 0x7ffa3c0689d0 2024-09-17T15:46:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:49.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa327fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:49.844 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":182,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:49.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 msgr2=0x7ffa2003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:49.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 0x7ffa2003e020 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c1d1180 tx=0x7ffa38008040 comp rx=0 tx=0).stop 2024-09-17T15:46:49.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 10 monclient: shutdown 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 msgr2=0x7ffa3c09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 0x7ffa3c09d770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c0ab000 tx=0x7ffa2c019170 comp rx=0 tx=0).stop 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 shutdown_connections 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ffa2003bb60 0x7ffa2003e020 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffa3c135ae0 0x7ffa3c09e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa3c068db0 0x7ffa3c09dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 --2- 172.21.15.86:0/2151613601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa3c0689d0 0x7ffa3c09d770 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:49.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.847+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 >> 172.21.15.86:0/2151613601 conn(0x7ffa3c0a6440 msgr2=0x7ffa3c06ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:49.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.851+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 shutdown_connections 2024-09-17T15:46:49.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:49.851+0000 7ffa43a90640 1 -- 172.21.15.86:0/2151613601 wait complete. 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3176: done (2 left) 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3177: snap_create 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3174: finishing rollback tid 0 to smithi20328086-36 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3174: finishing rollback tid 1 to smithi20328086-36 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3174: finishing rollback tid 2 to smithi20328086-36 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1669 (ObjNum 1086 snap 335 seq_num 1086) dirty exists 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: finishing write tid 1 to smithi20328086-20 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: finishing write tid 2 to smithi20328086-20 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: finishing write tid 3 to smithi20328086-20 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: finishing write tid 4 to smithi20328086-20 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: oid 20 updating version 0 to 1539 2024-09-17T15:46:49.949 INFO:tasks.rados.rados.0.smithi203.stdout:3175: oid 20 version 1539 is already newer than 1537 2024-09-17T15:46:49.950 INFO:tasks.rados.rados.0.smithi203.stdout:3175: oid 20 version 1539 is already newer than 1538 2024-09-17T15:46:49.950 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1539 (ObjNum 1113 snap 342 seq_num 1113) dirty exists 2024-09-17T15:46:49.950 INFO:tasks.rados.rados.0.smithi203.stdout:3175: left oid 20 (ObjNum 1113 snap 342 seq_num 1113) 2024-09-17T15:46:50.871 INFO:tasks.rados.rados.0.smithi203.stdout:3174: done (2 left) 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3175: done (1 left) 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3177: done (0 left) 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3178: read oid 25 snap -1 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3178: expect deleted 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3179: rollback oid 6 current snap is 343 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 335 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3180: read oid 3 snap -1 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3180: expect deleted 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3181: copy_from oid 19 from oid 38 current snap is 343 2024-09-17T15:46:50.872 INFO:tasks.rados.rados.0.smithi203.stdout:3182: snap_remove snap 336 2024-09-17T15:46:52.061 INFO:tasks.rados.rados.0.smithi203.stdout:3182: done (4 left) 2024-09-17T15:46:52.061 INFO:tasks.rados.rados.0.smithi203.stdout:3183: write (excl) oid 22 current snap is 343 2024-09-17T15:46:52.061 INFO:tasks.rados.rados.0.smithi203.stdout:3183: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-17T15:46:52.081 INFO:tasks.rados.rados.0.smithi203.stdout:3183: writing smithi20328086-22 from 722998 to 1137922 tid 1 2024-09-17T15:46:52.098 INFO:tasks.rados.rados.0.smithi203.stdout:3183: writing smithi20328086-22 from 1572117 to 2147159 tid 2 2024-09-17T15:46:52.099 INFO:tasks.rados.rados.0.smithi203.stdout:3184: snap_remove snap 334 2024-09-17T15:46:52.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: get_monmap_and_config 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: build_initial_monmap 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: monmap: 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:52.998115+0000 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:52.998115+0000 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:52.992 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:52.993 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:52.993 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:52.993 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:52.993 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:52.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 Processor -- start 2024-09-17T15:46:52.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 -- start start 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: init 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 0x7f81a006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-c con 0x7f81a00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81a0068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81a013aff0 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a009c8b0 con 0x7f81a0068db0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a009cb90 con 0x7f81a013aff0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f81a0074500 con 0x7f81a00689d0 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:52.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a7642640 10 monclient(hunting): authenticate will time out at 2145.963581s 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a4bdd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f81a0068db0 0x7f81a0069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38864/0 (socket says 172.21.15.86:38864) 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a4bdd640 1 -- 172.21.15.86:0/2152598678 learned_addr learned my addr 172.21.15.86:0/2152598678 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.994+0000 7f81a53de640 1 --2- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 0x7f81a006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 10 monclient(hunting): get_auth_request con 0x7f81a00689d0 auth_method 0 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 -- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 msgr2=0x7f81a006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 -- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 msgr2=0x7f81a006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 --2- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 0x7f81a006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:52.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 --2- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 0x7f81a006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 1 -- 172.21.15.86:0/2152598678 >> v1:172.21.15.86:6789/0 conn(0x7f81a0068db0 legacy=0x7f81a0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969815026 0 0) 0x7f81a009c8b0 con 0x7f81a0068db0 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): _negotiate 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:53.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): my global_id is 214194 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f818c003530 con 0x7f81a0068db0 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 >> v1:172.21.15.203:6789/0 conn(0x7f81a013aff0 legacy=0x7f81a013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4231717055 0 0) 0x7f818c003530 con 0x7f81a0068db0 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 msgr2=0x7f81a006f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 --2- 172.21.15.86:0/2152598678 >> v2:172.21.15.203:3300/0 conn(0x7f81a00689d0 0x7f81a006f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 >> v1:172.21.15.203:6789/0 conn(0x7f81a013aff0 legacy=0x7f81a013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:46:53.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a009ce70 con 0x7f81a0068db0 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _finish_auth 0 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f819c0019a0 con 0x7f81a0068db0 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a7642640 5 monclient: authenticate success, global_id 214194 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a7642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:53.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: dump: 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.004 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 >> v1:172.21.15.86:6789/0 conn(0x7f81a0068db0 legacy=0x7f81a0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c003b20 0x7f818c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): picked mon.b con 0x7f818c003b20 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:53.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c004810 0x7f818c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): picked mon.c con 0x7f818c004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 0x7f818c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): picked mon.a con 0x7f818c0099c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f818c00c3a0 con 0x7f818c0099c0 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f818c00c550 con 0x7f818c003b20 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f818c00c750 con 0x7f818c004810 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f8197fff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a7642640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:53.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c004810 0x7f818c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 0x7f818c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): get_auth_request con 0x7f818c004810 auth_method 0 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): get_auth_request con 0x7f818c0099c0 auth_method 0 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c003b20 0x7f818c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient(hunting): handle_auth_done global_id 214194 payload 293 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c004810 msgr2=0x7f818c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c004810 0x7f818c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c003b20 msgr2=0x7f818c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c003b20 0x7f818c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient: _finish_hunting 0 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 monclient: found mon.a 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:53.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c003530 con 0x7f818c0099c0 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient: _finish_auth 0 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 10 monclient: _check_auth_tickets 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a5bdf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a4bdd640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c004810 0x7f818c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 10 monclient: get_auth_request con 0x7f818c003b20 auth_method 0 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:52.998+0000 7f81a53de640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c003b20 0x7f818c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 reap_dead start 2024-09-17T15:46:53.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a5bdf640 1 -- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 msgr2=0x7f818c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:53.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a5bdf640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 0x7f818c00bdb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8198002090 tx=0x7f819801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c02333278d76b8b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:53.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819c00c0e0 con 0x7f818c0099c0 2024-09-17T15:46:53.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:53.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:53.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: dump: 2024-09-17T15:46:53.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:53.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:53.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:53.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81980a9070 con 0x7f818c0099c0 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 1 -- 172.21.15.86:0/2152598678 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8198001dc0 con 0x7f818c0099c0 2024-09-17T15:46:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 4 set_mon_vals no callback set 2024-09-17T15:46:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:53.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f8197fff640 10 monclient: dump: 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:53.015 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.016 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:53.016 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:53.016 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:53.016 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 10 monclient: shutdown 2024-09-17T15:46:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 1 -- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 msgr2=0x7f818c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.002+0000 7f81a7642640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 0x7f818c00bdb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8198002090 tx=0x7f819801c900 comp rx=0 tx=0).stop 2024-09-17T15:46:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- 172.21.15.86:0/2152598678 shutdown_connections 2024-09-17T15:46:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 --2- 172.21.15.86:0/2152598678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f818c0099c0 0x7f818c00bdb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- 172.21.15.86:0/2152598678 >> 172.21.15.86:0/2152598678 conn(0x7f81a00a6440 msgr2=0x7f81a00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- 172.21.15.86:0/2152598678 shutdown_connections 2024-09-17T15:46:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- 172.21.15.86:0/2152598678 wait complete. 2024-09-17T15:46:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient: build_initial_monmap 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient: monmap: 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 Processor -- start 2024-09-17T15:46:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- start start 2024-09-17T15:46:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient: init 2024-09-17T15:46:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-c con 0x7f818c0099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-a con 0x7f81a009dcb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a01d04e0 0x7f81a01d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): picked mon.noname-b con 0x7f81a01d04e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f81a00691c0 con 0x7f81a009dcb0 2024-09-17T15:46:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f81a006fab0 con 0x7f81a01d04e0 2024-09-17T15:46:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f81a0069340 con 0x7f818c0099c0 2024-09-17T15:46:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a7642640 10 monclient(hunting): authenticate will time out at 2145.973763s 2024-09-17T15:46:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52206/0 (socket says 172.21.15.86:52206) 2024-09-17T15:46:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 -- 172.21.15.86:0/2294195600 learned_addr learned my addr 172.21.15.86:0/2294195600 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): get_auth_request con 0x7f818c0099c0 auth_method 0 2024-09-17T15:46:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35568/0 (socket says 172.21.15.86:35568) 2024-09-17T15:46:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): get_auth_request con 0x7f81a009dcb0 auth_method 0 2024-09-17T15:46:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a5bdf640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a01d04e0 0x7f81a01d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient(hunting): handle_auth_done global_id 164705 payload 824 2024-09-17T15:46:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 -- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a01d04e0 msgr2=0x7f81a01d0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a01d04e0 0x7f81a01d0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 -- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 msgr2=0x7f81a009e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient: _finish_hunting 0 2024-09-17T15:46:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 monclient: found mon.noname-c 2024-09-17T15:46:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a53de640 1 -- 172.21.15.86:0/2294195600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a009ee00 con 0x7f818c0099c0 2024-09-17T15:46:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.006+0000 7f81a4bdd640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a53de640 1 -- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 msgr2=0x7f81a009d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a53de640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f819000f530 tx=0x7f819000f930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190013a20 con 0x7f818c0099c0 2024-09-17T15:46:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: dump: 2024-09-17T15:46:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.046 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:53.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8190013bc0 con 0x7f818c0099c0 2024-09-17T15:46:53.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:53.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190011380 con 0x7f818c0099c0 2024-09-17T15:46:53.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:53.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:53.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 10 monclient: dump: 2024-09-17T15:46:53.049 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:53.049 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:53.049 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.049 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:53.050 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:53.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81977fe640 4 set_mon_vals no callback set 2024-09-17T15:46:53.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 5 monclient: authenticate success, global_id 164705 2024-09-17T15:46:53.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _renew_subs 2024-09-17T15:46:53.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:53.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a009d460 con 0x7f818c0099c0 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _renew_subs 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a0132cd0 con 0x7f818c0099c0 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:53.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:53.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a009f280 con 0x7f818c0099c0 2024-09-17T15:46:53.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8190012070 con 0x7f818c0099c0 2024-09-17T15:46:53.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 0x7f818403e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:53.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(705..705 src has 438..705) ==== 4815+0+0 (secure 0 0 0) 0x7f819004fd00 con 0x7f818c0099c0 2024-09-17T15:46:53.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a4bdd640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 0x7f818403e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:53.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.010+0000 7f81a4bdd640 10 monclient: get_auth_request con 0x7f818403bbc0 auth_method 0 2024-09-17T15:46:53.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.014+0000 7f81a4bdd640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 0x7f818403e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f819c0010f0 tx=0x7f819c001ad0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:53.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.014+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f81a009d460 con 0x7f818c0099c0 2024-09-17T15:46:53.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.014+0000 7f8195ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:53.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.014+0000 7f8195ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3184: done (5 left) 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3185: read oid 49 snap -1 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3185: expect deleted 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3186: read oid 30 snap -1 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3186: expect deleted 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3187: copy_from oid 48 from oid 35 current snap is 343 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3181: finishing copy_from to smithi20328086-19 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3181: got expected ENOENT (src dne) 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3181: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:46:53.068 INFO:tasks.rados.rados.0.smithi203.stdout:3179: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3179: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1541 (ObjNum 1078 snap 334 seq_num 1078) dirty exists 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: finishing write tid 1 to smithi20328086-22 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: finishing write tid 2 to smithi20328086-22 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: finishing write tid 3 to smithi20328086-22 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: finishing write tid 4 to smithi20328086-22 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: oid 22 updating version 0 to 1314 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: oid 22 updating version 1314 to 1315 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: oid 22 version 1315 is already newer than 1313 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1315 (ObjNum 1114 snap 343 seq_num 1114) dirty exists 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3183: left oid 22 (ObjNum 1114 snap 343 seq_num 1114) 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3178: done (7 left) 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3179: done (6 left) 2024-09-17T15:46:53.069 INFO:tasks.rados.rados.0.smithi203.stdout:3180: done (5 left) 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3181: done (4 left) 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3183: done (3 left) 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3188: read oid 44 snap 335 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3188: expect (ObjNum 1062 snap 328 seq_num 1062) 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3189: write oid 20 current snap is 343 2024-09-17T15:46:53.070 INFO:tasks.rados.rados.0.smithi203.stdout:3189: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-09-17T15:46:53.089 INFO:tasks.rados.rados.0.smithi203.stdout:3189: writing smithi20328086-20 from 739805 to 1229978 tid 1 2024-09-17T15:46:53.105 INFO:tasks.rados.rados.0.smithi203.stdout:3189: writing smithi20328086-20 from 1630599 to 2065652 tid 2 2024-09-17T15:46:53.116 INFO:tasks.rados.rados.0.smithi203.stdout:3189: writing smithi20328086-20 from 2636284 to 2739805 tid 3 2024-09-17T15:46:53.116 INFO:tasks.rados.rados.0.smithi203.stdout:3190: snap_create 2024-09-17T15:46:53.116 INFO:tasks.rados.rados.0.smithi203.stdout:3188: expect (ObjNum 1062 snap 328 seq_num 1062) 2024-09-17T15:46:53.225 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:53.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.258+0000 7f81a7642640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:53.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.258+0000 7f81a7642640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:53.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.258+0000 7f81a7642640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:53.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.258+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f81a00a35c0 con 0x7f818c0099c0 2024-09-17T15:46:53.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f8195ffb640 1 -- 172.21.15.86:0/2294195600 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8190017020 con 0x7f818c0099c0 2024-09-17T15:46:53.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f8195ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:53.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f8195ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:53.258 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":182,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:53.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 msgr2=0x7f818403e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 0x7f818403e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f819c0010f0 tx=0x7f819c001ad0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 10 monclient: shutdown 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 msgr2=0x7f81a009d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f819000f530 tx=0x7f819000f930 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 shutdown_connections 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f818403bbc0 0x7f818403e080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a01d04e0 0x7f81a01d0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81a009dcb0 0x7f81a009e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 --2- 172.21.15.86:0/2294195600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f818c0099c0 0x7f81a009d770 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 >> 172.21.15.86:0/2294195600 conn(0x7f81a00a6440 msgr2=0x7f81a0130b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 shutdown_connections 2024-09-17T15:46:53.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:53.262+0000 7f81a7642640 1 -- 172.21.15.86:0/2294195600 wait complete. 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3187: finishing copy_from to smithi20328086-48 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3187: got expected ENOENT (src dne) 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3187: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: finishing write tid 1 to smithi20328086-20 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: finishing write tid 2 to smithi20328086-20 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: finishing write tid 3 to smithi20328086-20 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: finishing write tid 4 to smithi20328086-20 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: finishing write tid 5 to smithi20328086-20 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: oid 20 updating version 0 to 1545 2024-09-17T15:46:53.342 INFO:tasks.rados.rados.0.smithi203.stdout:3189: oid 20 version 1545 is already newer than 1543 2024-09-17T15:46:53.343 INFO:tasks.rados.rados.0.smithi203.stdout:3189: oid 20 version 1545 is already newer than 1544 2024-09-17T15:46:53.343 INFO:tasks.rados.rados.0.smithi203.stdout:3189: oid 20 version 1545 is already newer than 1542 2024-09-17T15:46:53.343 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1545 (ObjNum 1115 snap 343 seq_num 1115) dirty exists 2024-09-17T15:46:53.343 INFO:tasks.rados.rados.0.smithi203.stdout:3189: left oid 20 (ObjNum 1115 snap 343 seq_num 1115) 2024-09-17T15:46:53.857 INFO:tasks.rados.rados.0.smithi203.stdout:3185: done (5 left) 2024-09-17T15:46:53.857 INFO:tasks.rados.rados.0.smithi203.stdout:3186: done (4 left) 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3187: done (3 left) 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3188: done (2 left) 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3189: done (1 left) 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3190: done (0 left) 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3191: write (excl) oid 49 current snap is 344 2024-09-17T15:46:53.858 INFO:tasks.rados.rados.0.smithi203.stdout:3191: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-17T15:46:53.876 INFO:tasks.rados.rados.0.smithi203.stdout:3191: writing smithi20328086-49 from 756612 to 1322034 tid 1 2024-09-17T15:46:53.895 INFO:tasks.rados.rados.0.smithi203.stdout:3191: writing smithi20328086-49 from 2089081 to 2667792 tid 2 2024-09-17T15:46:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:3192: read oid 1 snap 313 2024-09-17T15:46:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:3192: expect deleted 2024-09-17T15:46:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:3193: write oid 47 current snap is 344 2024-09-17T15:46:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:3191: finishing write tid 1 to smithi20328086-49 2024-09-17T15:46:53.933 INFO:tasks.rados.rados.0.smithi203.stdout:3193: seq_num 1117 ranges {773419=640671,2147563=438722} 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3193: writing smithi20328086-47 from 773419 to 1414090 tid 1 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3193: writing smithi20328086-47 from 2147563 to 2586285 tid 2 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3194: rollback oid 6 current snap is 344 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 340 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3191: finishing write tid 2 to smithi20328086-49 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3191: finishing write tid 3 to smithi20328086-49 2024-09-17T15:46:53.934 INFO:tasks.rados.rados.0.smithi203.stdout:3191: finishing write tid 4 to smithi20328086-49 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3191: oid 49 updating version 0 to 1676 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3191: oid 49 updating version 1676 to 1678 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3191: oid 49 version 1678 is already newer than 1677 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1678 (ObjNum 1116 snap 344 seq_num 1116) dirty exists 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3191: left oid 49 (ObjNum 1116 snap 344 seq_num 1116) 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3193: finishing write tid 1 to smithi20328086-47 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3191: done (3 left) 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3192: done (2 left) 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3195: write oid 15 current snap is 344 2024-09-17T15:46:53.935 INFO:tasks.rados.rados.0.smithi203.stdout:3195: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-17T15:46:53.939 INFO:tasks.rados.rados.0.smithi203.stdout:3195: writing smithi20328086-15 from 790226 to 1222499 tid 1 2024-09-17T15:46:53.952 INFO:tasks.rados.rados.0.smithi203.stdout:3195: writing smithi20328086-15 from 1806045 to 2388425 tid 2 2024-09-17T15:46:53.952 INFO:tasks.rados.rados.0.smithi203.stdout:3196: copy_from oid 36 from oid 23 current snap is 344 2024-09-17T15:46:53.952 INFO:tasks.rados.rados.0.smithi203.stdout:3193: finishing write tid 2 to smithi20328086-47 2024-09-17T15:46:53.952 INFO:tasks.rados.rados.0.smithi203.stdout:3193: finishing write tid 3 to smithi20328086-47 2024-09-17T15:46:53.952 INFO:tasks.rados.rados.0.smithi203.stdout:3194: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: finishing write tid 4 to smithi20328086-47 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: oid 47 updating version 0 to 1530 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: oid 47 version 1530 is already newer than 1529 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: oid 47 version 1530 is already newer than 1528 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1530 (ObjNum 1117 snap 344 seq_num 1117) dirty exists 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: left oid 47 (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3194: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1547 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3195: finishing write tid 1 to smithi20328086-15 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3193: done (3 left) 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3194: done (2 left) 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3197: read oid 9 snap -1 2024-09-17T15:46:53.953 INFO:tasks.rados.rados.0.smithi203.stdout:3197: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:46:53.954 INFO:tasks.rados.rados.0.smithi203.stdout:3198: copy_from oid 24 from oid 37 current snap is 344 2024-09-17T15:46:53.954 INFO:tasks.rados.rados.0.smithi203.stdout:3199: write (excl) oid 34 current snap is 344 2024-09-17T15:46:53.954 INFO:tasks.rados.rados.0.smithi203.stdout:3199: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-09-17T15:46:53.961 INFO:tasks.rados.rados.0.smithi203.stdout:3199: writing smithi20328086-34 from 407033 to 914555 tid 1 2024-09-17T15:46:53.970 INFO:tasks.rados.rados.0.smithi203.stdout:3199: writing smithi20328086-34 from 1464527 to 1906918 tid 2 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3199: writing smithi20328086-34 from 2345976 to 2807033 tid 3 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3200: copy_from oid 5 from oid 11 current snap is 344 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3196: finishing copy_from to smithi20328086-36 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3196: got expected ENOENT (src dne) 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3196: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3195: finishing write tid 2 to smithi20328086-15 2024-09-17T15:46:53.978 INFO:tasks.rados.rados.0.smithi203.stdout:3195: finishing write tid 3 to smithi20328086-15 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: finishing write tid 4 to smithi20328086-15 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: oid 15 updating version 0 to 1300 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: oid 15 updating version 1300 to 1302 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: oid 15 version 1302 is already newer than 1301 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1302 (ObjNum 1118 snap 344 seq_num 1118) dirty exists 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: left oid 15 (ObjNum 1118 snap 344 seq_num 1118) 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3195: done (5 left) 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3196: done (4 left) 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3201: snap_create 2024-09-17T15:46:53.979 INFO:tasks.rados.rados.0.smithi203.stdout:3197: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:46:54.010 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3177342 2024-09-17T15:46:54.081 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 139232 2024-09-17T15:46:54.138 INFO:tasks.rados.rados.0.smithi203.stdout:3199: finishing write tid 1 to smithi20328086-34 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: finishing write tid 2 to smithi20328086-34 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: finishing write tid 3 to smithi20328086-34 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: finishing write tid 4 to smithi20328086-34 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3198: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: finishing write tid 5 to smithi20328086-34 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: oid 34 updating version 0 to 1311 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: oid 34 version 1311 is already newer than 1309 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: oid 34 version 1311 is already newer than 1308 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: oid 34 version 1311 is already newer than 1310 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1311 (ObjNum 1119 snap 344 seq_num 1119) dirty exists 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3199: left oid 34 (ObjNum 1119 snap 344 seq_num 1119) 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3198: finishing copy_from to smithi20328086-24 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1517 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:46:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3200: finishing copy_from racing read to smithi20328086-5 2024-09-17T15:46:54.140 INFO:tasks.rados.rados.0.smithi203.stdout:3200: finishing copy_from to smithi20328086-5 2024-09-17T15:46:54.140 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1384 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:46:54.869 INFO:tasks.rados.rados.0.smithi203.stdout:3197: done (4 left) 2024-09-17T15:46:54.869 INFO:tasks.rados.rados.0.smithi203.stdout:3198: done (3 left) 2024-09-17T15:46:54.869 INFO:tasks.rados.rados.0.smithi203.stdout:3199: done (2 left) 2024-09-17T15:46:54.869 INFO:tasks.rados.rados.0.smithi203.stdout:3200: done (1 left) 2024-09-17T15:46:54.870 INFO:tasks.rados.rados.0.smithi203.stdout:3201: done (0 left) 2024-09-17T15:46:54.870 INFO:tasks.rados.rados.0.smithi203.stdout:3202: delete oid 30 current snap is 345 2024-09-17T15:46:54.877 INFO:tasks.rados.rados.0.smithi203.stdout:3202: done (0 left) 2024-09-17T15:46:54.877 INFO:tasks.rados.rados.0.smithi203.stdout:3203: rollback oid 45 current snap is 345 2024-09-17T15:46:54.877 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 313 2024-09-17T15:46:54.877 INFO:tasks.rados.rados.0.smithi203.stdout:3204: snap_remove snap 343 2024-09-17T15:46:55.874 INFO:tasks.rados.rados.0.smithi203.stdout:3204: done (1 left) 2024-09-17T15:46:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:3205: copy_from oid 12 from oid 7 current snap is 345 2024-09-17T15:46:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:3203: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:46:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:46:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:3203: done (1 left) 2024-09-17T15:46:55.875 INFO:tasks.rados.rados.0.smithi203.stdout:3206: snap_create 2024-09-17T15:46:55.909 INFO:tasks.rados.rados.0.smithi203.stdout:3205: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:46:55.909 INFO:tasks.rados.rados.0.smithi203.stdout:3205: finishing copy_from to smithi20328086-12 2024-09-17T15:46:55.909 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1191 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:46:56.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:56.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: get_monmap_and_config 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: monmap: 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:56.409762+0000 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:56.409762+0000 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:56.404 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:56.405 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:56.405 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 Processor -- start 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 -- start start 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: init 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e6009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-b con 0x7f9e60069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9e60074c30 0x7f9e6009bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-a con 0x7f9e60074c30 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e6009c4f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6009e410 con 0x7f9e6009c4f0 2024-09-17T15:46:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9e6009d150 con 0x7f9e60074c30 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6009ce80 con 0x7f9e60069260 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e64918640 10 monclient(hunting): authenticate will time out at 2149.375242s 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5ed9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9e6009c4f0 0x7f9e6013aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38874/0 (socket says 172.21.15.86:38874) 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5ed9d640 1 -- 172.21.15.86:0/309800687 learned_addr learned my addr 172.21.15.86:0/309800687 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/309800687 >> v2:172.21.15.86:3300/0 conn(0x7f9e60074c30 0x7f9e6009bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e6009ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): get_auth_request con 0x7f9e60074c30 auth_method 0 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): get_auth_request con 0x7f9e60069260 auth_method 0 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_done global_id 214200 payload 274 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 msgr2=0x7f9e6009ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e6009ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/309800687 >> v1:172.21.15.86:6789/0 conn(0x7f9e6009c4f0 legacy=0x7f9e6013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.406+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/309800687 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6009e6f0 con 0x7f9e60074c30 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 msgr2=0x7f9e6009bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e6009bfb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e4801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9e1d82ccd1a45 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 1 -- 172.21.15.86:0/309800687 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480ad070 con 0x7f9e60074c30 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: dump: 2024-09-17T15:46:56.411 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 1 -- 172.21.15.86:0/309800687 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e480a95f0 con 0x7f9e60074c30 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 1 -- 172.21.15.86:0/309800687 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480a99c0 con 0x7f9e60074c30 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 5 monclient: authenticate success, global_id 214200 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 4 set_mon_vals no callback set 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5d59a640 10 monclient: dump: 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.413 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: shutdown 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 msgr2=0x7f9e6009bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e6009bfb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009d90 tx=0x7f9e4801c830 comp rx=0 tx=0).stop 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- 172.21.15.86:0/309800687 shutdown_connections 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e6009bfb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- 172.21.15.86:0/309800687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e6009ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- 172.21.15.86:0/309800687 >> 172.21.15.86:0/309800687 conn(0x7f9e600a6440 msgr2=0x7f9e600a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- 172.21.15.86:0/309800687 shutdown_connections 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- 172.21.15.86:0/309800687 wait complete. 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: monmap: 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:56.415 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 Processor -- start 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- start start 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: init 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e601cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-c con 0x7f9e60069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-a con 0x7f9e60074c30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 0x7f9e60139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6009c4f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6013e710 con 0x7f9e60074c30 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6013e590 con 0x7f9e6009c4f0 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6013e890 con 0x7f9e60069260 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e64918640 10 monclient(hunting): authenticate will time out at 2149.382683s 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e601cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35582/0 (socket says 172.21.15.86:35582) 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/2922348195 learned_addr learned my addr 172.21.15.86:0/2922348195 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): get_auth_request con 0x7f9e60074c30 auth_method 0 2024-09-17T15:46:56.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): get_auth_request con 0x7f9e60069260 auth_method 0 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient(hunting): handle_auth_done global_id 214203 payload 824 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5ed9d640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 0x7f9e60139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 msgr2=0x7f9e601cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e601cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 msgr2=0x7f9e60139ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 0x7f9e60139ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 monclient: found mon.noname-a 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/2922348195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6009d460 con 0x7f9e60074c30 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5ed9d640 10 monclient: get_auth_request con 0x7f9e6009c4f0 auth_method 0 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5ed9d640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 0x7f9e60139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.414+0000 7f9e5e59c640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e601cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:56.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e5dd9b640 1 -- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 msgr2=0x7f9e60133c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e5dd9b640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e4801d0d0 tx=0x7f9e480195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480ad040 con 0x7f9e60074c30 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: dump: 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e480b2870 con 0x7f9e60074c30 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 5 monclient: authenticate success, global_id 214203 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _renew_subs 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6011e310 con 0x7f9e60074c30 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480b2c40 con 0x7f9e60074c30 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 10 monclient: dump: 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e5cd99640 4 set_mon_vals no callback set 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _renew_subs 2024-09-17T15:46:56.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6013b870 con 0x7f9e60074c30 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e6009ba70 con 0x7f9e60074c30 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9e480b0070 con 0x7f9e60074c30 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 0x7f9e4003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(709..709 src has 438..709) ==== 4799+0+0 (secure 0 0 0) 0x7f9e480ea870 con 0x7f9e60074c30 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e5e59c640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 0x7f9e4003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.418+0000 7f9e5e59c640 10 monclient: get_auth_request con 0x7f9e4003bb60 auth_method 0 2024-09-17T15:46:56.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.422+0000 7f9e5e59c640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 0x7f9e4003e020 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009f50 tx=0x7f9e58009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.422+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9e6009d460 con 0x7f9e60074c30 2024-09-17T15:46:56.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.422+0000 7f9e56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.422+0000 7f9e56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e64918640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e64918640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e64918640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e6006d4f0 con 0x7f9e60074c30 2024-09-17T15:46:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e56ffd640 1 -- 172.21.15.86:0/2922348195 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9e48016540 con 0x7f9e60074c30 2024-09-17T15:46:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:56.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.654+0000 7f9e56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:56.654 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 msgr2=0x7f9e4003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 0x7f9e4003e020 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009f50 tx=0x7f9e58009190 comp rx=0 tx=0).stop 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 10 monclient: shutdown 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 msgr2=0x7f9e60133c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9e4801d0d0 tx=0x7f9e480195d0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 shutdown_connections 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9e4003bb60 0x7f9e4003e020 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e6009c4f0 0x7f9e60139ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e60074c30 0x7f9e60133c40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 --2- 172.21.15.86:0/2922348195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e60069260 0x7f9e601cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 >> 172.21.15.86:0/2922348195 conn(0x7f9e600a6440 msgr2=0x7f9e60070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 shutdown_connections 2024-09-17T15:46:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.658+0000 7f9e64918640 1 -- 172.21.15.86:0/2922348195 wait complete. 2024-09-17T15:46:56.673 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:46:56.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:46:56.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: get_monmap_and_config 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: monmap: 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:56.802397+0000 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:56.802397+0000 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:56.797 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 1 Processor -- start 2024-09-17T15:46:56.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 1 -- start start 2024-09-17T15:46:56.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: init 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 0x7fa9e409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e40a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.798+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa9e40a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 0x7fa9e413f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e413d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e40a6280 con 0x7fa9e40a54c0 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e409d2a0 con 0x7fa9e40a50e0 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e409c0a0 con 0x7fa9e413d2f0 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ebacc640 10 monclient(hunting): authenticate will time out at 2149.768073s 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 0x7fa9e413f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 0x7fa9e409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9067640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa9e40a54c0 0x7fa9e40a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38886/0 (socket says 172.21.15.86:38886) 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9067640 1 -- 172.21.15.86:0/3845866063 learned_addr learned my addr 172.21.15.86:0/3845866063 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): get_auth_request con 0x7fa9e413d2f0 auth_method 0 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): get_auth_request con 0x7fa9e40a50e0 auth_method 0 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient(hunting): handle_auth_done global_id 164714 payload 274 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 -- 172.21.15.86:0/3845866063 >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 msgr2=0x7fa9e409cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 --2- 172.21.15.86:0/3845866063 >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 0x7fa9e409cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 -- 172.21.15.86:0/3845866063 >> v1:172.21.15.86:6789/0 conn(0x7fa9e40a54c0 legacy=0x7fa9e40a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 monclient: found mon.noname-b 2024-09-17T15:46:56.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:56.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9ea069640 1 -- 172.21.15.86:0/3845866063 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e409c380 con 0x7fa9e413d2f0 2024-09-17T15:46:56.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.802+0000 7fa9e9868640 1 --2- 172.21.15.86:0/3845866063 >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 0x7fa9e409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 1 -- 172.21.15.86:0/3845866063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 msgr2=0x7fa9e413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 1 --2- 172.21.15.86:0/3845866063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 0x7fa9e413f6e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00097c0 tx=0x7fa9e001c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14d83f87af054c59 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 1 -- 172.21.15.86:0/3845866063 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00b0070 con 0x7fa9e413d2f0 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:56.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: dump: 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 5 monclient: authenticate success, global_id 164714 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 1 -- 172.21.15.86:0/3845866063 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e00ab600 con 0x7fa9e413d2f0 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 1 -- 172.21.15.86:0/3845866063 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00ab9d0 con 0x7fa9e413d2f0 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e8866640 10 monclient: dump: 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.806 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 4 set_mon_vals no callback set 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: shutdown 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- 172.21.15.86:0/3845866063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 msgr2=0x7fa9e413f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/3845866063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 0x7fa9e413f6e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00097c0 tx=0x7fa9e001c2d0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- 172.21.15.86:0/3845866063 shutdown_connections 2024-09-17T15:46:56.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/3845866063 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e413d2f0 0x7fa9e413f6e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/3845866063 >> v2:172.21.15.86:3300/0 conn(0x7fa9e40a50e0 0x7fa9e409cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- 172.21.15.86:0/3845866063 >> 172.21.15.86:0/3845866063 conn(0x7fa9e40aaae0 msgr2=0x7fa9e40aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- 172.21.15.86:0/3845866063 shutdown_connections 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- 172.21.15.86:0/3845866063 wait complete. 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: monmap: 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:56.809 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:56.809 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:56.809 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 Processor -- start 2024-09-17T15:46:56.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- start start 2024-09-17T15:46:56.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: init 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 0x7fa9e4121340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9e40a54c0 0x7fa9e41381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e40a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 0x7fa9e41cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e4142650 con 0x7fa9e413d2f0 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40a6b00 con 0x7fa9e40a54c0 2024-09-17T15:46:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e41427d0 con 0x7fa9e40a50e0 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ebacc640 10 monclient(hunting): authenticate will time out at 2149.775830s 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 0x7fa9e4121340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 0x7fa9e41cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 0x7fa9e4121340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52220/0 (socket says 172.21.15.86:52220) 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 1 -- 172.21.15.86:0/2830338623 learned_addr learned my addr 172.21.15.86:0/2830338623 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 10 monclient(hunting): get_auth_request con 0x7fa9e40a50e0 auth_method 0 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9e9868640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 10 monclient(hunting): get_auth_request con 0x7fa9e413d2f0 auth_method 0 2024-09-17T15:46:56.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.806+0000 7fa9ea069640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient(hunting): handle_auth_done global_id 214209 payload 824 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 msgr2=0x7fa9e4121340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 0x7fa9e4121340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9067640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9e40a54c0 0x7fa9e41381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9e40a54c0 msgr2=0x7fa9e41381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9e40a54c0 0x7fa9e41381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 monclient: found mon.noname-a 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e4120e80 con 0x7fa9e413d2f0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9067640 10 monclient: get_auth_request con 0x7fa9e40a54c0 auth_method 0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9e40a50e0 0x7fa9e4121340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9067640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9e40a54c0 0x7fa9e41381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 msgr2=0x7fa9e41cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ea069640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 0x7fa9e41cfb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00ab000 tx=0x7fa9e00190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00ba070 con 0x7fa9e413d2f0 2024-09-17T15:46:56.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: dump: 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e00b5890 con 0x7fa9e413d2f0 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00b0040 con 0x7fa9e413d2f0 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 10 monclient: dump: 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.819 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 5 monclient: authenticate success, global_id 214209 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 monclient: _renew_subs 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9dbfff640 4 set_mon_vals no callback set 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e413f8a0 con 0x7fa9e413d2f0 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 monclient: _renew_subs 2024-09-17T15:46:56.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e4122860 con 0x7fa9e413d2f0 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 0x7fa9e41d0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e406a0e0 con 0x7fa9e41386f0 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 0x7fa9e41d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient: get_auth_request con 0x7fa9e41386f0 auth_method 0 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa9e00b3070 con 0x7fa9e413d2f0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 10 monclient(hunting): handle_auth_done global_id 214212 payload 824 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.810+0000 7fa9e9868640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9e40a18d0 con 0x7fa9e41386f0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9c803bb60 0x7fa9c803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(709..709 src has 438..709) ==== 4799+0+0 (secure 0 0 0) 0x7fa9e00e9d10 con 0x7fa9e413d2f0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9e9067640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9c803bb60 0x7fa9c803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9e9067640 10 monclient: get_auth_request con 0x7fa9c803bb60 auth_method 0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9e9868640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 msgr2=0x7fa9e41d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9e9868640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 0x7fa9e41d0070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc00e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc019070 con 0x7fa9e41386f0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 10 monclient: dump: 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9dc015c10 con 0x7fa9e41386f0 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 msgr2=0x7fa9e41d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 0x7fa9e41d0070 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc00e8c0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 msgr2=0x7fa9e41d0070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9da7fc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e41386f0 0x7fa9e41d0070 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc00e8c0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.814+0000 7fa9e9067640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9c803bb60 0x7fa9c803e020 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007fd0 tx=0x7fa9d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9ebacc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:56.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 0x7fa9e41d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9ebacc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e4074be0 con 0x7fa9e409fc90 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 0x7fa9e41d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient: get_auth_request con 0x7fa9e409fc90 auth_method 0 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 10 monclient(hunting): handle_auth_done global_id 214215 payload 824 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.822+0000 7fa9e9868640 1 -- 172.21.15.86:0/2830338623 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9e41dae80 con 0x7fa9e409fc90 2024-09-17T15:46:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9e9868640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 msgr2=0x7fa9e41d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9e9868640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 0x7fa9e41d05b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9dc0147c0 con 0x7fa9e409fc90 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 10 monclient: dump: 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.827 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fa9dc0147c0 con 0x7fa9e409fc90 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 msgr2=0x7fa9e41d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 0x7fa9e41d05b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc004ac0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 msgr2=0x7fa9e41d05b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9da7fc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e409fc90 0x7fa9e41d05b0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc004980 tx=0x7fa9dc004ac0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.828 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 73001, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:46:56.829 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:56.830 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:56.831 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:56.832 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:56.833 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:56.834 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:56.837 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:56.838 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:56.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:56.840 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:56.841 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:56.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9c803bb60 msgr2=0x7fa9c803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa9c803bb60 0x7fa9c803e020 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4007fd0 tx=0x7fa9d4009040 comp rx=0 tx=0).stop 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 10 monclient: shutdown 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 msgr2=0x7fa9e41cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 --2- 172.21.15.86:0/2830338623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9e413d2f0 0x7fa9e41cfb30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00ab000 tx=0x7fa9e00190d0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 reap_dead start 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ea069640 1 -- 172.21.15.86:0/2830338623 reap_dead start 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 shutdown_connections 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 >> 172.21.15.86:0/2830338623 conn(0x7fa9e40aaae0 msgr2=0x7fa9e40a3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 shutdown_connections 2024-09-17T15:46:56.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.826+0000 7fa9ebacc640 1 -- 172.21.15.86:0/2830338623 wait complete. 2024-09-17T15:46:56.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:56.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:46:56.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:56.880 INFO:tasks.rados.rados.0.smithi203.stdout:3205: done (1 left) 2024-09-17T15:46:56.880 INFO:tasks.rados.rados.0.smithi203.stdout:3206: done (0 left) 2024-09-17T15:46:56.880 INFO:tasks.rados.rados.0.smithi203.stdout:3207: delete oid 34 current snap is 346 2024-09-17T15:46:56.885 INFO:tasks.rados.rados.0.smithi203.stdout:3207: done (0 left) 2024-09-17T15:46:56.885 INFO:tasks.rados.rados.0.smithi203.stdout:3208: rollback oid 10 current snap is 346 2024-09-17T15:46:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 313 2024-09-17T15:46:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:3209: read oid 14 snap 335 2024-09-17T15:46:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:3209: expect (ObjNum 1012 snap 316 seq_num 1012) 2024-09-17T15:46:56.886 INFO:tasks.rados.rados.0.smithi203.stdout:3210: delete oid 49 current snap is 346 2024-09-17T15:46:56.888 INFO:tasks.rados.rados.0.smithi203.stdout:3208: finishing rollback tid 0 to smithi20328086-10 2024-09-17T15:46:56.889 INFO:tasks.rados.rados.0.smithi203.stdout:3208: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:46:56.889 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1378 (ObjNum 129 snap 0 seq_num 3605691776) dirty dne 2024-09-17T15:46:56.892 INFO:tasks.rados.rados.0.smithi203.stdout:3208: done (2 left) 2024-09-17T15:46:56.892 INFO:tasks.rados.rados.0.smithi203.stdout:3210: done (1 left) 2024-09-17T15:46:56.892 INFO:tasks.rados.rados.0.smithi203.stdout:3211: rollback oid 7 current snap is 346 2024-09-17T15:46:56.892 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 7 to 341 2024-09-17T15:46:56.893 INFO:tasks.rados.rados.0.smithi203.stdout:3212: read oid 41 snap -1 2024-09-17T15:46:56.893 INFO:tasks.rados.rados.0.smithi203.stdout:3212: expect (ObjNum 1110 snap 341 seq_num 1110) 2024-09-17T15:46:56.893 INFO:tasks.rados.rados.0.smithi203.stdout:3213: write (excl) oid 30 current snap is 346 2024-09-17T15:46:56.893 INFO:tasks.rados.rados.0.smithi203.stdout:3213: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-09-17T15:46:56.908 INFO:tasks.rados.rados.0.smithi203.stdout:3213: writing smithi20328086-30 from 474261 to 882779 tid 1 2024-09-17T15:46:56.926 INFO:tasks.rados.rados.0.smithi203.stdout:3213: writing smithi20328086-30 from 1582102 to 2031831 tid 2 2024-09-17T15:46:56.940 INFO:tasks.rados.rados.0.smithi203.stdout:3213: writing smithi20328086-30 from 2739315 to 2874261 tid 3 2024-09-17T15:46:56.940 INFO:tasks.rados.rados.0.smithi203.stdout:3214: snap_create 2024-09-17T15:46:56.940 INFO:tasks.rados.rados.0.smithi203.stdout:3212: expect (ObjNum 1110 snap 341 seq_num 1110) 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.966+0000 7f9428bd4640 10 monclient: get_monmap_and_config 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.966+0000 7f9428bd4640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient: monmap: 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:56.972003+0000 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:56.972003+0000 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:56.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:56.967 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:56.967 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:56.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 Processor -- start 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 -- start start 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient: init 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f94240a0c70 0x7f94240a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f94240a0c70 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f94240a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 0x7f9424136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-b con 0x7f9424135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f94240a2410 con 0x7f94240a1590 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f94240a2130 con 0x7f94240a0c70 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94240a26f0 con 0x7f9424135f90 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9428bd4640 10 monclient(hunting): authenticate will time out at 2149.937390s 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35620/0 (socket says 172.21.15.86:35620) 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 0x7f9424136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 1 -- 172.21.15.86:0/1697917092 learned_addr learned my addr 172.21.15.86:0/1697917092 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): get_auth_request con 0x7f94240a1590 auth_method 0 2024-09-17T15:46:56.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): get_auth_request con 0x7f9424135f90 auth_method 0 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f942259c640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.203:3300/0 conn(0x7f94240a0c70 0x7f94240a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f942259c640 10 monclient(hunting): get_auth_request con 0x7f94240a0c70 auth_method 0 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f942259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f942259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f942259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient(hunting): handle_auth_done global_id 164720 payload 274 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 -- 172.21.15.86:0/1697917092 >> v2:172.21.15.203:3300/0 conn(0x7f94240a0c70 msgr2=0x7f94240a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.203:3300/0 conn(0x7f94240a0c70 0x7f94240a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 -- 172.21.15.86:0/1697917092 >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 msgr2=0x7f9424135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 monclient: found mon.noname-b 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:56.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9422d9d640 1 -- 172.21.15.86:0/1697917092 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942409f9f0 con 0x7f9424135f90 2024-09-17T15:46:56.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.970+0000 7f9421d9b640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:56.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9422d9d640 1 -- 172.21.15.86:0/1697917092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 msgr2=0x7f9424136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9422d9d640 1 --2- 172.21.15.86:0/1697917092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 0x7f9424136420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f942406d160 tx=0x7f941c01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cc129b9a8309cea server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 1 -- 172.21.15.86:0/1697917092 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c0b2070 con 0x7f9424135f90 2024-09-17T15:46:56.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: dump: 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 1 -- 172.21.15.86:0/1697917092 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941c0a5890 con 0x7f9424135f90 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 5 monclient: authenticate success, global_id 164720 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 1 -- 172.21.15.86:0/1697917092 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f941c0a5c60 con 0x7f9424135f90 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 4 set_mon_vals no callback set 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f942159a640 10 monclient: dump: 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.978 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 10 monclient: shutdown 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 -- 172.21.15.86:0/1697917092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 msgr2=0x7f9424136420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 --2- 172.21.15.86:0/1697917092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 0x7f9424136420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f942406d160 tx=0x7f941c01c7f0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 -- 172.21.15.86:0/1697917092 shutdown_connections 2024-09-17T15:46:56.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 --2- 172.21.15.86:0/1697917092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9424135f90 0x7f9424136420 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.86:3300/0 conn(0x7f94240a1590 0x7f9424135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 --2- 172.21.15.86:0/1697917092 >> v2:172.21.15.203:3300/0 conn(0x7f94240a0c70 0x7f94240a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 -- 172.21.15.86:0/1697917092 >> 172.21.15.86:0/1697917092 conn(0x7f94240a2f40 msgr2=0x7f94240a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 -- 172.21.15.86:0/1697917092 shutdown_connections 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 1 -- 172.21.15.86:0/1697917092 wait complete. 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 10 monclient: build_initial_monmap 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.974+0000 7f9428bd4640 10 monclient: monmap: 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:56.980 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 Processor -- start 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 -- start start 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient: init 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94240a0c70 0x7f94241cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-c con 0x7f94240a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 0x7f9424134db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-a con 0x7f94240a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:56.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424135f90 0x7f942411c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): picked mon.noname-b con 0x7f9424135f90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f94240a00d0 con 0x7f94240a1590 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f942409ff50 con 0x7f9424135f90 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f942413f740 con 0x7f94240a0c70 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9428bd4640 10 monclient(hunting): authenticate will time out at 2149.944890s 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94240a0c70 0x7f94241cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 0x7f9424134db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 0x7f9424134db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35634/0 (socket says 172.21.15.86:35634) 2024-09-17T15:46:56.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 -- 172.21.15.86:0/565090025 learned_addr learned my addr 172.21.15.86:0/565090025 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94240a0c70 0x7f94241cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52234/0 (socket says 172.21.15.86:52234) 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): get_auth_request con 0x7f94240a1590 auth_method 0 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): get_auth_request con 0x7f94240a0c70 auth_method 0 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9422d9d640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424135f90 0x7f942411c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9422d9d640 10 monclient(hunting): get_auth_request con 0x7f9424135f90 auth_method 0 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9422d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9422d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9422d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient(hunting): handle_auth_done global_id 214221 payload 824 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94240a0c70 msgr2=0x7f94241cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94240a0c70 0x7f94241cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424135f90 msgr2=0x7f942411c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9424135f90 0x7f942411c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:46:56.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 monclient: found mon.noname-a 2024-09-17T15:46:56.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:56.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9424120750 con 0x7f94240a1590 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 msgr2=0x7f9424134db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9421d9b640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 0x7f9424134db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f941800e9d0 tx=0x7f941800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418017070 con 0x7f94240a1590 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: dump: 2024-09-17T15:46:56.989 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:56.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f941800cc70 con 0x7f94240a1590 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418013670 con 0x7f94240a1590 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9412ffd640 10 monclient: dump: 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.991 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.978+0000 7f9420d99640 4 set_mon_vals no callback set 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 5 monclient: authenticate success, global_id 214221 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 monclient: _renew_subs 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942411f520 con 0x7f94240a1590 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 monclient: _renew_subs 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94241ce8f0 con 0x7f94240a1590 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 0x7f942402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f942411e3f0 con 0x7f942411ef50 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 0x7f942402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient: get_auth_request con 0x7f942411ef50 auth_method 0 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f941801a070 con 0x7f94240a1590 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 10 monclient(hunting): handle_auth_done global_id 164726 payload 824 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9424135a30 con 0x7f942411ef50 2024-09-17T15:46:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940403bb60 0x7f940403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(710..710 src has 438..710) ==== 4771+0+0 (secure 0 0 0) 0x7f941804fe90 con 0x7f94240a1590 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9422d9d640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940403bb60 0x7f940403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9422d9d640 10 monclient: get_auth_request con 0x7f940403bb60 auth_method 0 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 msgr2=0x7f942402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f942259c640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 0x7f942402d450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f941400bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414018070 con 0x7f942411ef50 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 10 monclient: dump: 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.994 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9414010ab0 con 0x7f942411ef50 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 msgr2=0x7f942402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 0x7f942402d450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f941400bba0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 msgr2=0x7f942402d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:56.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.982+0000 7f9412ffd640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f942411ef50 0x7f942402d450 secure :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f941400bba0 comp rx=0 tx=0).stop 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.986+0000 7f9422d9d640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940403bb60 0x7f940403e020 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ac000 tx=0x7f941c0a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f9428bd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f9428bd4640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 0x7f942402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f9428bd4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f94240726b0 con 0x7f94241352f0 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 0x7f942402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient: get_auth_request con 0x7f94241352f0 auth_method 0 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:56.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 10 monclient(hunting): handle_auth_done global_id 164729 payload 824 2024-09-17T15:46:56.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.990+0000 7f942259c640 1 -- 172.21.15.86:0/565090025 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f942406d4f0 con 0x7f94241352f0 2024-09-17T15:46:56.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f942259c640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 msgr2=0x7f942402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:56.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f942259c640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 0x7f942402d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f9414004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414014840 con 0x7f94241352f0 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 10 monclient: dump: 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:56.998 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f9414014840 con 0x7f94241352f0 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 msgr2=0x7f942402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 0x7f942402d990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f9414004740 comp rx=0 tx=0).stop 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 msgr2=0x7f942402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9412ffd640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94241352f0 0x7f942402d990 secure :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f941400b6e0 tx=0x7f9414004740 comp rx=0 tx=0).stop 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 320182, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:57.013 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:57.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940403bb60 msgr2=0x7f940403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f940403bb60 0x7f940403e020 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ac000 tx=0x7f941c0a7070 comp rx=0 tx=0).stop 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 10 monclient: shutdown 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 msgr2=0x7f9424134db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 --2- 172.21.15.86:0/565090025 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94240a1590 0x7f9424134db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f941800e9d0 tx=0x7f941800ee90 comp rx=0 tx=0).stop 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9422d9d640 1 -- 172.21.15.86:0/565090025 reap_dead start 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9422d9d640 1 -- 172.21.15.86:0/565090025 reap_dead start 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 shutdown_connections 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 >> 172.21.15.86:0/565090025 conn(0x7f94240a2f40 msgr2=0x7f942413c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 shutdown_connections 2024-09-17T15:46:57.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:56.994+0000 7f9428bd4640 1 -- 172.21.15.86:0/565090025 wait complete. 2024-09-17T15:46:57.016 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:57.016 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:46:57.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:57.081 INFO:tasks.rados.rados.0.smithi203.stdout:3211: finishing rollback tid 0 to smithi20328086-7 2024-09-17T15:46:57.082 INFO:tasks.rados.rados.0.smithi203.stdout:3211: finishing rollback tid 1 to smithi20328086-7 2024-09-17T15:46:57.082 INFO:tasks.rados.rados.0.smithi203.stdout:3211: finishing rollback tid 2 to smithi20328086-7 2024-09-17T15:46:57.082 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1198 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:46:57.082 INFO:tasks.rados.rados.0.smithi203.stdout:3209: expect (ObjNum 1012 snap 316 seq_num 1012) 2024-09-17T15:46:57.082 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:57.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.138+0000 7fd716baf640 10 monclient: get_monmap_and_config 2024-09-17T15:46:57.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.138+0000 7fd716baf640 10 monclient: build_initial_monmap 2024-09-17T15:46:57.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.138+0000 7fd716baf640 10 monclient: monmap: 2024-09-17T15:46:57.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:57.143632+0000 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:57.143632+0000 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:57.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 Processor -- start 2024-09-17T15:46:57.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 -- start start 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient: init 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7100a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 0x7fd71013af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-a con 0x7fd7100a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 0x7fd71013d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-b con 0x7fd71013b450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7100a2130 con 0x7fd7100a0c70 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd7100a2410 con 0x7fd7100a1760 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd7100a26f0 con 0x7fd71013b450 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd716baf640 10 monclient(hunting): authenticate will time out at 2150.109403s 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 0x7fd71013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 0x7fd71013d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71494b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd7100a0c70 0x7fd7100a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38894/0 (socket says 172.21.15.86:38894) 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71494b640 1 -- 172.21.15.86:0/996086539 learned_addr learned my addr 172.21.15.86:0/996086539 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): get_auth_request con 0x7fd7100a1760 auth_method 0 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): get_auth_request con 0x7fd71013b450 auth_method 0 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient(hunting): handle_auth_done global_id 164732 payload 274 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 -- 172.21.15.86:0/996086539 >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 msgr2=0x7fd71013af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 --2- 172.21.15.86:0/996086539 >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 0x7fd71013af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 -- 172.21.15.86:0/996086539 >> v1:172.21.15.86:6789/0 conn(0x7fd7100a0c70 legacy=0x7fd7100a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient: _finish_hunting 0 2024-09-17T15:46:57.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 monclient: found mon.noname-b 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd71514c640 1 -- 172.21.15.86:0/996086539 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71009f9f0 con 0x7fd71013b450 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.142+0000 7fd70ffff640 1 --2- 172.21.15.86:0/996086539 >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 0x7fd71013af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd71514c640 1 -- 172.21.15.86:0/996086539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 msgr2=0x7fd71013d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd71514c640 1 --2- 172.21.15.86:0/996086539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 0x7fd71013d840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd708009d90 tx=0x7fd70801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4692f6789a9c3752 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 1 -- 172.21.15.86:0/996086539 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b2070 con 0x7fd71013b450 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: dump: 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.146 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 5 monclient: authenticate success, global_id 164732 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 1 -- 172.21.15.86:0/996086539 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7080a5890 con 0x7fd71013b450 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:57.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 1 -- 172.21.15.86:0/996086539 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080a5c60 con 0x7fd71013b450 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd70f7fe640 10 monclient: dump: 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 4 set_mon_vals no callback set 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 10 monclient: shutdown 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 1 -- 172.21.15.86:0/996086539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 msgr2=0x7fd71013d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.146+0000 7fd716baf640 1 --2- 172.21.15.86:0/996086539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 0x7fd71013d840 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd708009d90 tx=0x7fd70801c830 comp rx=0 tx=0).stop 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- 172.21.15.86:0/996086539 shutdown_connections 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 --2- 172.21.15.86:0/996086539 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71013b450 0x7fd71013d840 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 --2- 172.21.15.86:0/996086539 >> v2:172.21.15.86:3300/0 conn(0x7fd7100a1760 0x7fd71013af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- 172.21.15.86:0/996086539 >> 172.21.15.86:0/996086539 conn(0x7fd7100a2f40 msgr2=0x7fd7100a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- 172.21.15.86:0/996086539 shutdown_connections 2024-09-17T15:46:57.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- 172.21.15.86:0/996086539 wait complete. 2024-09-17T15:46:57.150 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1381829 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient: build_initial_monmap 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient: monmap: 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:57.155 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 Processor -- start 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- start start 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient: init 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 0x7fd71011ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-c con 0x7fd7100a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7100a1760 0x7fd710135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-b con 0x7fd7100a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 0x7fd71011c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): picked mon.noname-a con 0x7fd71013b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd710140720 con 0x7fd71013b450 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd7101405a0 con 0x7fd7100a1760 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd7101408a0 con 0x7fd7100a0c70 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd716baf640 10 monclient(hunting): authenticate will time out at 2150.118105s 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 0x7fd71011ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 0x7fd71011c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 0x7fd71011ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:52256/0 (socket says 172.21.15.86:52256) 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 learned_addr learned my addr 172.21.15.86:0/3654155172 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): get_auth_request con 0x7fd7100a0c70 auth_method 0 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): get_auth_request con 0x7fd71013b450 auth_method 0 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7100a1760 0x7fd710135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 10 monclient(hunting): get_auth_request con 0x7fd7100a1760 auth_method 0 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient(hunting): handle_auth_done global_id 214227 payload 824 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 msgr2=0x7fd71011ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 0x7fd71011ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7100a1760 msgr2=0x7fd710135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7100a1760 0x7fd710135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient: _finish_hunting 0 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 monclient: found mon.noname-a 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71013bc40 con 0x7fd71013b450 2024-09-17T15:46:57.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7100a0c70 0x7fd71011ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.150+0000 7fd70ffff640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7100a1760 0x7fd710135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 msgr2=0x7fd71011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71514c640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 0x7fd71011c8c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7080ac000 tx=0x7fd7080195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b2050 con 0x7fd71013b450 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: dump: 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.160 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7080b3040 con 0x7fd71013b450 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 5 monclient: authenticate success, global_id 214227 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 monclient: _renew_subs 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71013b8a0 con 0x7fd71013b450 2024-09-17T15:46:57.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080ada60 con 0x7fd71013b450 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 10 monclient: dump: 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.162 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70effd640 4 set_mon_vals no callback set 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 monclient: _renew_subs 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd710120230 con 0x7fd71013b450 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 0x7fd71011d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd71011d720 con 0x7fd71011ce00 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd7080adc00 con 0x7fd71013b450 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 0x7fd71011d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient: get_auth_request con 0x7fd71011ce00 auth_method 0 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd6f003bbe0 0x7fd6f003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(710..710 src has 438..710) ==== 4771+0+0 (secure 0 0 0) 0x7fd7080ea180 con 0x7fd71013b450 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70ffff640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd6f003bbe0 0x7fd6f003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 10 monclient(hunting): handle_auth_done global_id 224107 payload 824 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7100a1050 con 0x7fd71011ce00 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.154+0000 7fd70ffff640 10 monclient: get_auth_request con 0x7fd6f003bbe0 auth_method 0 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 msgr2=0x7fd71011d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 0x7fd71011d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7000072a0 tx=0x7fd700007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70ffff640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd6f003bbe0 0x7fd6f003e0a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd71011fba0 tx=0x7fd6f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70000fde0 con 0x7fd71011ce00 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 10 monclient: dump: 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.166 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd70000fde0 con 0x7fd71011ce00 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:57.167 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 msgr2=0x7fd71011d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 0x7fd71011d1e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd7000072a0 tx=0x7fd700007760 comp rx=0 tx=0).stop 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 msgr2=0x7fd71011d1e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.158+0000 7fd70d7fa640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd71011ce00 0x7fd71011d1e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd716baf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd716baf640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd716baf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd71011fd50 con 0x7fd710136ef0 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient: get_auth_request con 0x7fd710136ef0 auth_method 0 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 10 monclient(hunting): handle_auth_done global_id 224110 payload 824 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd71006d4f0 con 0x7fd710136ef0 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 msgr2=0x7fd710137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 msgr2=0x7fd710137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fd700017000 tx=0x7fd700004950 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:46:57.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.166+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:46:57.169 INFO:tasks.rados.rados.0.smithi203.stdout:3213: finishing write tid 1 to smithi20328086-30 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: finishing write tid 2 to smithi20328086-30 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: finishing write tid 3 to smithi20328086-30 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: finishing write tid 4 to smithi20328086-30 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: finishing write tid 5 to smithi20328086-30 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: oid 30 updating version 0 to 1306 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: oid 30 version 1306 is already newer than 1303 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: oid 30 version 1306 is already newer than 1305 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: oid 30 version 1306 is already newer than 1304 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1306 (ObjNum 1123 snap 346 seq_num 1123) dirty exists 2024-09-17T15:46:57.170 INFO:tasks.rados.rados.0.smithi203.stdout:3213: left oid 30 (ObjNum 1123 snap 346 seq_num 1123) 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: get_auth_request con 0x7fd710136ef0 auth_method 0 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: _init_auth method 2 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: _init_auth already have auth, reseting 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: handle_auth_reply_more payload 9 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-17T15:46:57.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.370+0000 7fd71494b640 10 monclient: handle_auth_done global_id 224110 payload 293 2024-09-17T15:46:57.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd71494b640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 msgr2=0x7fd710137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd71494b640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd700017000 tx=0x7fd700009030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70002e070 con 0x7fd710136ef0 2024-09-17T15:46:57.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 10 monclient: dump: 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.371 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fd700000e50 con 0x7fd710136ef0 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 msgr2=0x7fd710137300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd700017000 tx=0x7fd700009030 comp rx=0 tx=0).stop 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 msgr2=0x7fd710137300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:57.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd70d7fa640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd710136ef0 0x7fd710137300 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd700017000 tx=0x7fd700009030 comp rx=0 tx=0).stop 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7823, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:46:57.373 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.374 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.375 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:57.376 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:57.377 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.378 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.379 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.380 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.381 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:57.382 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:57.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:57.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:57.385 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:57.385 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:57.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd6f003bbe0 msgr2=0x7fd6f003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd6f003bbe0 0x7fd6f003e0a0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd71011fba0 tx=0x7fd6f8008040 comp rx=0 tx=0).stop 2024-09-17T15:46:57.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 10 monclient: shutdown 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 msgr2=0x7fd71011c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 --2- 172.21.15.86:0/3654155172 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71013b450 0x7fd71011c8c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7080ac000 tx=0x7fd7080195d0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 reap_dead start 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd71514c640 1 -- 172.21.15.86:0/3654155172 reap_dead start 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 shutdown_connections 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 >> 172.21.15.86:0/3654155172 conn(0x7fd7100a2f40 msgr2=0x7fd71013c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 shutdown_connections 2024-09-17T15:46:57.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.374+0000 7fd716baf640 1 -- 172.21.15.86:0/3654155172 wait complete. 2024-09-17T15:46:57.397 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:57.397 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:46:57.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:46:57.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: get_monmap_and_config 2024-09-17T15:46:57.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: build_initial_monmap 2024-09-17T15:46:57.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: monmap: 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:57.530489+0000 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:57.530489+0000 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:57.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 Processor -- start 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 -- start start 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: init 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f03d4073fc0 0x7f03d40743a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-c con 0x7f03d4073fc0 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03d409bc00 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d4138400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-b con 0x7f03d40748e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d409d210 con 0x7f03d409bc00 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f03d409fbe0 con 0x7f03d4073fc0 2024-09-17T15:46:57.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f03d4074d50 con 0x7f03d40748e0 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.526+0000 7f03d8a19640 10 monclient(hunting): authenticate will time out at 2150.495937s 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d1d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f03d409bc00 0x7f03d4137cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38910/0 (socket says 172.21.15.86:38910) 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d1d9b640 1 -- 172.21.15.86:0/2814349150 learned_addr learned my addr 172.21.15.86:0/2814349150 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d4138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): get_auth_request con 0x7f03d40748e0 auth_method 0 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d259c640 1 --2- 172.21.15.86:0/2814349150 >> v2:172.21.15.203:3300/0 conn(0x7f03d4073fc0 0x7f03d40743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d259c640 10 monclient(hunting): get_auth_request con 0x7f03d4073fc0 auth_method 0 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_done global_id 164738 payload 274 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 -- 172.21.15.86:0/2814349150 >> v2:172.21.15.203:3300/0 conn(0x7f03d4073fc0 msgr2=0x7f03d40743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/2814349150 >> v2:172.21.15.203:3300/0 conn(0x7f03d4073fc0 0x7f03d40743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 -- 172.21.15.86:0/2814349150 >> v1:172.21.15.86:6789/0 conn(0x7f03d409bc00 legacy=0x7f03d4137cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 monclient: found mon.noname-b 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:57.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.530+0000 7f03d2d9d640 1 -- 172.21.15.86:0/2814349150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d409c3c0 con 0x7f03d40748e0 2024-09-17T15:46:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 -- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 msgr2=0x7f03d4138400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d4138400 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009d90 tx=0x7f03cc01d290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a82da4eff89d445 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 1 -- 172.21.15.86:0/2814349150 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0b2070 con 0x7f03d40748e0 2024-09-17T15:46:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: dump: 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: _finish_auth 0 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: _check_auth_tickets 2024-09-17T15:46:57.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 1 -- 172.21.15.86:0/2814349150 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03cc0a5780 con 0x7f03d40748e0 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 1 -- 172.21.15.86:0/2814349150 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0a5b50 con 0x7f03d40748e0 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 5 monclient: authenticate success, global_id 164738 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 4 set_mon_vals no callback set 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d159a640 10 monclient: dump: 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: shutdown 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 msgr2=0x7f03d4138400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d4138400 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009d90 tx=0x7f03cc01d290 comp rx=0 tx=0).stop 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- 172.21.15.86:0/2814349150 shutdown_connections 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- 172.21.15.86:0/2814349150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d4138400 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- 172.21.15.86:0/2814349150 >> v2:172.21.15.203:3300/0 conn(0x7f03d4073fc0 0x7f03d40743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- 172.21.15.86:0/2814349150 >> 172.21.15.86:0/2814349150 conn(0x7f03d40a6440 msgr2=0x7f03d40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- 172.21.15.86:0/2814349150 shutdown_connections 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- 172.21.15.86:0/2814349150 wait complete. 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: build_initial_monmap 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: monmap: 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:57.538 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 Processor -- start 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- start start 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: init 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03d4073fc0 0x7f03d413f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-b con 0x7f03d4073fc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-c con 0x7f03d40748e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): picked mon.noname-a con 0x7f03d409bc00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4138ac0 con 0x7f03d409bc00 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4138940 con 0x7f03d4073fc0 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f03d4142850 con 0x7f03d40748e0 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d8a19640 10 monclient(hunting): authenticate will time out at 2150.503251s 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35658/0 (socket says 172.21.15.86:35658) 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 -- 172.21.15.86:0/499604800 learned_addr learned my addr 172.21.15.86:0/499604800 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): get_auth_request con 0x7f03d409bc00 auth_method 0 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): get_auth_request con 0x7f03d40748e0 auth_method 0 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d259c640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03d4073fc0 0x7f03d413f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d259c640 10 monclient(hunting): get_auth_request con 0x7f03d4073fc0 auth_method 0 2024-09-17T15:46:57.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient(hunting): handle_auth_done global_id 214230 payload 824 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 -- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 msgr2=0x7f03d413f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d413f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 -- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03d4073fc0 msgr2=0x7f03d413f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03d4073fc0 0x7f03d413f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.534+0000 7f03d2d9d640 1 -- 172.21.15.86:0/499604800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d413b810 con 0x7f03d409bc00 2024-09-17T15:46:57.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d2d9d640 1 -- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 msgr2=0x7f03d4139e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:57.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d2d9d640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03cc01c660 tx=0x7f03cc0195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0b2050 con 0x7f03d409bc00 2024-09-17T15:46:57.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: dump: 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03cc0b3040 con 0x7f03d409bc00 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 5 monclient: authenticate success, global_id 214230 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _renew_subs 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d40a3ae0 con 0x7f03d409bc00 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0ada60 con 0x7f03d409bc00 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 10 monclient: dump: 2024-09-17T15:46:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c3fff640 4 set_mon_vals no callback set 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _renew_subs 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d413d890 con 0x7f03d409bc00 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:57.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d413fb20 con 0x7f03d409bc00 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f03cc0c0020 con 0x7f03d409bc00 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 0x7f03b403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(710..710 src has 438..710) ==== 4771+0+0 (secure 0 0 0) 0x7f03cc0ea830 con 0x7f03d409bc00 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d259c640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 0x7f03b403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.538+0000 7f03d259c640 10 monclient: get_auth_request con 0x7f03b403bb60 auth_method 0 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.542+0000 7f03d259c640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 0x7f03b403e020 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f03d413dd80 tx=0x7f03c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.542+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f03d40a3ae0 con 0x7f03d409bc00 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.542+0000 7f03c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.542+0000 7f03c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:57.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.774+0000 7f03d8a19640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:46:57.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.774+0000 7f03d8a19640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:46:57.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.774+0000 7f03d8a19640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:57.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.774+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f03d40743a0 con 0x7f03d409bc00 2024-09-17T15:46:57.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03c2ffd640 1 -- 172.21.15.86:0/499604800 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f03cc016540 con 0x7f03d409bc00 2024-09-17T15:46:57.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:46:57.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:57.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1464120576,config=3541143443,health=2246248209,kv=3591778543,logm=3022927040}) 2024-09-17T15:46:57.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=4040126969,config=3541143443,health=849142597,kv=2898202873,logm=3032430327}) 2024-09-17T15:46:57.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=1464120576,config=3541143443,health=2246248209,kv=3591778543,logm=3022927040}) 2024-09-17T15:46:57.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=10,kv=10,logm=52} crc {auth=2569557562,config=3541143443,health=1459784005,kv=2732825231,logm=669346514}) 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 msgr2=0x7f03b403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 0x7f03b403e020 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f03d413dd80 tx=0x7f03c4008040 comp rx=0 tx=0).stop 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 10 monclient: shutdown 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 msgr2=0x7f03d4139e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f03cc01c660 tx=0x7f03cc0195d0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 shutdown_connections 2024-09-17T15:46:57.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f03b403bb60 0x7f03b403e020 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d409bc00 0x7f03d4139e60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d40748e0 0x7f03d413f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 --2- 172.21.15.86:0/499604800 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03d4073fc0 0x7f03d413f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 >> 172.21.15.86:0/499604800 conn(0x7f03d40a6440 msgr2=0x7f03d40a3080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 shutdown_connections 2024-09-17T15:46:57.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f03d8a19640 1 -- 172.21.15.86:0/499604800 wait complete. 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.778+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438976270}) 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127867149}) 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438976270}) 2024-09-17T15:46:57.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984476276}) 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240266160}) 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804725015}) 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240266160}) 2024-09-17T15:46:57.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.782+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333785103}) 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764028248}) 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4007157679}) 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764028248}) 2024-09-17T15:46:57.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.786+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415121465}) 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147887766}) 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008082403}) 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1147887766}) 2024-09-17T15:46:57.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126333717}) 2024-09-17T15:46:57.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893121851}) 2024-09-17T15:46:57.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3475101289}) 2024-09-17T15:46:57.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=893121851}) 2024-09-17T15:46:57.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.790+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261932752}) 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513550120}) 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4284625427}) 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2513550120}) 2024-09-17T15:46:57.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.794+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433520681}) 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086308367}) 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308151065}) 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086308367}) 2024-09-17T15:46:57.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3584277419}) 2024-09-17T15:46:57.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146990059}) 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620837610}) 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146990059}) 2024-09-17T15:46:57.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629087398}) 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126158891}) 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.798+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1760407840}) 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126158891}) 2024-09-17T15:46:57.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492765453}) 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217283047}) 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3015800366}) 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217283047}) 2024-09-17T15:46:57.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.802+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161255912}) 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886856489}) 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1419232188}) 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886856489}) 2024-09-17T15:46:57.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3763044326}) 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224607438}) 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957599223}) 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224607438}) 2024-09-17T15:46:57.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.806+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245025009}) 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689451256}) 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4079039614}) 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689451256}) 2024-09-17T15:46:57.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066056011}) 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136779827}) 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=332203980}) 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136779827}) 2024-09-17T15:46:57.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.810+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1966030326}) 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723669457}) 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2834814161}) 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723669457}) 2024-09-17T15:46:57.810 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848430920}) 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472633917}) 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196151090}) 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472633917}) 2024-09-17T15:46:57.812 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.814+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657890329}) 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893669881}) 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186948893}) 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893669881}) 2024-09-17T15:46:57.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2999834168}) 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360218403}) 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=754536208}) 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360218403}) 2024-09-17T15:46:57.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.818+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871052620}) 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504785796}) 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307397818}) 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504785796}) 2024-09-17T15:46:57.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508625442}) 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103553771}) 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903577713}) 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4103553771}) 2024-09-17T15:46:57.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.822+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865099263}) 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730226097}) 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3518261869}) 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730226097}) 2024-09-17T15:46:57.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1724848355}) 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003642136}) 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1469473334}) 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003642136}) 2024-09-17T15:46:57.824 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.826+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2425670445}) 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619136000}) 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025684468}) 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619136000}) 2024-09-17T15:46:57.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267594944}) 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242389209}) 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1974684677}) 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242389209}) 2024-09-17T15:46:57.828 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3159901965}) 2024-09-17T15:46:57.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865968056}) 2024-09-17T15:46:57.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167828405}) 2024-09-17T15:46:57.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.830+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865968056}) 2024-09-17T15:46:57.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2029786664}) 2024-09-17T15:46:57.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279487046}) 2024-09-17T15:46:57.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1650676236}) 2024-09-17T15:46:57.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279487046}) 2024-09-17T15:46:57.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375463572}) 2024-09-17T15:46:57.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697381990}) 2024-09-17T15:46:57.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442795653}) 2024-09-17T15:46:57.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697381990}) 2024-09-17T15:46:57.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.834+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817375486}) 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598002740}) 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2226452261}) 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2598002740}) 2024-09-17T15:46:57.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742071787}) 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472606123}) 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649629381}) 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472606123}) 2024-09-17T15:46:57.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.838+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1936676718}) 2024-09-17T15:46:57.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051192198}) 2024-09-17T15:46:57.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846544173}) 2024-09-17T15:46:57.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1051192198}) 2024-09-17T15:46:57.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112374370}) 2024-09-17T15:46:57.840 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.840 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050962306}) 2024-09-17T15:46:57.840 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3846872150}) 2024-09-17T15:46:57.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050962306}) 2024-09-17T15:46:57.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.842+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975502727}) 2024-09-17T15:46:57.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837223373}) 2024-09-17T15:46:57.880 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=611428903}) 2024-09-17T15:46:57.880 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.880 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837223373}) 2024-09-17T15:46:57.880 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.882+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932397967}) 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=574187031}) 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528492978}) 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=574187031}) 2024-09-17T15:46:57.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.890+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63223284}) 2024-09-17T15:46:57.954 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.955 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068223931}) 2024-09-17T15:46:57.955 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2096224628}) 2024-09-17T15:46:57.955 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:57.955 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068223931}) 2024-09-17T15:46:57.955 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:57.958+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=475726758}) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3209: done (4 left) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3211: done (3 left) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3212: done (2 left) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3213: done (1 left) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3214: done (0 left) 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3215: rollback oid 15 current snap is 347 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 339 2024-09-17T15:46:58.100 INFO:tasks.rados.rados.0.smithi203.stdout:3216: write oid 11 current snap is 347 2024-09-17T15:46:58.101 INFO:tasks.rados.rados.0.smithi203.stdout:3216: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-17T15:46:58.114 INFO:tasks.rados.rados.0.smithi203.stdout:3216: writing smithi20328086-11 from 491068 to 974835 tid 1 2024-09-17T15:46:58.131 INFO:tasks.rados.rados.0.smithi203.stdout:3216: writing smithi20328086-11 from 1640584 to 2233971 tid 2 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3216: writing smithi20328086-11 from 2650385 to 2891068 tid 3 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3217: copy_from oid 19 from oid 17 current snap is 347 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3215: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3215: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3215: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1309 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3216: finishing write tid 1 to smithi20328086-11 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3216: finishing write tid 2 to smithi20328086-11 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3215: done (2 left) 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3218: write (excl) oid 41 current snap is 347 2024-09-17T15:46:58.142 INFO:tasks.rados.rados.0.smithi203.stdout:3218: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-17T15:46:58.156 INFO:tasks.rados.rados.0.smithi203.stdout:3218: writing smithi20328086-41 from 507875 to 1066891 tid 1 2024-09-17T15:46:58.175 INFO:tasks.rados.rados.0.smithi203.stdout:3218: writing smithi20328086-41 from 1699066 to 2436111 tid 2 2024-09-17T15:46:58.175 INFO:tasks.rados.rados.0.smithi203.stdout:3219: write oid 48 current snap is 347 2024-09-17T15:46:58.175 INFO:tasks.rados.rados.0.smithi203.stdout:3217: finishing copy_from to smithi20328086-19 2024-09-17T15:46:58.175 INFO:tasks.rados.rados.0.smithi203.stdout:3217: got expected ENOENT (src dne) 2024-09-17T15:46:58.175 INFO:tasks.rados.rados.0.smithi203.stdout:3219: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-17T15:46:58.189 INFO:tasks.rados.rados.0.smithi203.stdout:3219: writing smithi20328086-48 from 524682 to 1275300 tid 1 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3219: writing smithi20328086-48 from 1757548 to 2354604 tid 2 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3220: write (excl) oid 24 current snap is 347 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: finishing write tid 3 to smithi20328086-11 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3217: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: finishing write tid 4 to smithi20328086-11 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: finishing write tid 5 to smithi20328086-11 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: oid 11 updating version 0 to 1520 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: oid 11 updating version 1520 to 1521 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: oid 11 version 1521 is already newer than 1518 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:3216: oid 11 version 1521 is already newer than 1519 2024-09-17T15:46:58.203 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1521 (ObjNum 1124 snap 347 seq_num 1124) dirty exists 2024-09-17T15:46:58.204 INFO:tasks.rados.rados.0.smithi203.stdout:3216: left oid 11 (ObjNum 1124 snap 347 seq_num 1124) 2024-09-17T15:46:58.204 INFO:tasks.rados.rados.0.smithi203.stdout:3218: finishing write tid 1 to smithi20328086-41 2024-09-17T15:46:58.204 INFO:tasks.rados.rados.0.smithi203.stdout:3220: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-17T15:46:58.212 INFO:tasks.rados.rados.0.smithi203.stdout:3220: writing smithi20328086-24 from 541489 to 967356 tid 1 2024-09-17T15:46:58.224 INFO:tasks.rados.rados.0.smithi203.stdout:3220: writing smithi20328086-24 from 1416030 to 2156744 tid 2 2024-09-17T15:46:58.232 INFO:tasks.rados.rados.0.smithi203.stdout:3220: writing smithi20328086-24 from 2732654 to 2941489 tid 3 2024-09-17T15:46:58.232 INFO:tasks.rados.rados.0.smithi203.stdout:3216: done (4 left) 2024-09-17T15:46:58.232 INFO:tasks.rados.rados.0.smithi203.stdout:3217: done (3 left) 2024-09-17T15:46:58.232 INFO:tasks.rados.rados.0.smithi203.stdout:3221: read oid 40 snap 313 2024-09-17T15:46:58.232 INFO:tasks.rados.rados.0.smithi203.stdout:3221: expect (ObjNum 973 snap 294 seq_num 973) 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3222: read oid 33 snap -1 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3222: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3223: read oid 28 snap 335 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3223: expect deleted 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3224: read oid 9 snap -1 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3224: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3225: read oid 35 snap 340 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3225: expect deleted 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3226: delete oid 10 current snap is 347 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3218: finishing write tid 2 to smithi20328086-41 2024-09-17T15:46:58.233 INFO:tasks.rados.rados.0.smithi203.stdout:3218: finishing write tid 3 to smithi20328086-41 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3218: finishing write tid 4 to smithi20328086-41 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3218: oid 41 updating version 0 to 670 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3218: oid 41 updating version 670 to 671 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3218: oid 41 updating version 671 to 672 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 672 (ObjNum 1125 snap 347 seq_num 1125) dirty exists 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3218: left oid 41 (ObjNum 1125 snap 347 seq_num 1125) 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3219: finishing write tid 1 to smithi20328086-48 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3219: finishing write tid 2 to smithi20328086-48 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3219: finishing write tid 3 to smithi20328086-48 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3219: finishing write tid 4 to smithi20328086-48 2024-09-17T15:46:58.234 INFO:tasks.rados.rados.0.smithi203.stdout:3219: oid 48 updating version 0 to 1622 2024-09-17T15:46:58.235 INFO:tasks.rados.rados.0.smithi203.stdout:3219: oid 48 version 1622 is already newer than 1621 2024-09-17T15:46:58.235 INFO:tasks.rados.rados.0.smithi203.stdout:3219: oid 48 updating version 1622 to 1623 2024-09-17T15:46:58.235 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1623 (ObjNum 1126 snap 347 seq_num 1126) dirty exists 2024-09-17T15:46:58.235 INFO:tasks.rados.rados.0.smithi203.stdout:3219: left oid 48 (ObjNum 1126 snap 347 seq_num 1126) 2024-09-17T15:46:58.235 INFO:tasks.rados.rados.0.smithi203.stdout:3220: finishing write tid 1 to smithi20328086-24 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3218: done (8 left) 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3219: done (7 left) 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3223: done (6 left) 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3225: done (5 left) 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3226: done (4 left) 2024-09-17T15:46:58.236 INFO:tasks.rados.rados.0.smithi203.stdout:3227: snap_remove snap 342 2024-09-17T15:46:58.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425066785}) 2024-09-17T15:46:58.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=884499407}) 2024-09-17T15:46:58.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425066785}) 2024-09-17T15:46:58.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834650522}) 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798802175}) 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366916478}) 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798802175}) 2024-09-17T15:46:58.341 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.342+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851903274}) 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176684436}) 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056941199}) 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176684436}) 2024-09-17T15:46:58.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184335285}) 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304782907}) 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929968762}) 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304782907}) 2024-09-17T15:46:58.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.346+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2639663113}) 2024-09-17T15:46:58.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939433927}) 2024-09-17T15:46:58.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808008943}) 2024-09-17T15:46:58.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3939433927}) 2024-09-17T15:46:58.348 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2628751046}) 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949970637}) 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1973657744}) 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=949970637}) 2024-09-17T15:46:58.350 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.350+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2148948918}) 2024-09-17T15:46:58.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.352 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103932073}) 2024-09-17T15:46:58.352 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727993260}) 2024-09-17T15:46:58.352 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.352 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103932073}) 2024-09-17T15:46:58.352 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.354+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083949854}) 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919400221}) 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284392064}) 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919400221}) 2024-09-17T15:46:58.354 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2457888081}) 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735720904}) 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3662240607}) 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735720904}) 2024-09-17T15:46:58.356 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.358+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328325572}) 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523668101}) 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442441943}) 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523668101}) 2024-09-17T15:46:58.358 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1818298838}) 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523758265}) 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807528767}) 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523758265}) 2024-09-17T15:46:58.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.362+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153770755}) 2024-09-17T15:46:58.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356651661}) 2024-09-17T15:46:58.363 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221770695}) 2024-09-17T15:46:58.363 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.363 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356651661}) 2024-09-17T15:46:58.363 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334690416}) 2024-09-17T15:46:58.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.365 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306281546}) 2024-09-17T15:46:58.365 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094152579}) 2024-09-17T15:46:58.365 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.365 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306281546}) 2024-09-17T15:46:58.365 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.366+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713094069}) 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367326913}) 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016577671}) 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367326913}) 2024-09-17T15:46:58.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387283154}) 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129196221}) 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4220175760}) 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4129196221}) 2024-09-17T15:46:58.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.370+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=250655966}) 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523774035}) 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2577943747}) 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523774035}) 2024-09-17T15:46:58.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4202326900}) 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220578087}) 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119432002}) 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220578087}) 2024-09-17T15:46:58.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.374+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=384893245}) 2024-09-17T15:46:58.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881623159}) 2024-09-17T15:46:58.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1410280442}) 2024-09-17T15:46:58.376 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.376 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881623159}) 2024-09-17T15:46:58.376 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2147677768}) 2024-09-17T15:46:58.377 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.377 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337348234}) 2024-09-17T15:46:58.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457716909}) 2024-09-17T15:46:58.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337348234}) 2024-09-17T15:46:58.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.378+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=508609998}) 2024-09-17T15:46:58.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140014362}) 2024-09-17T15:46:58.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845480060}) 2024-09-17T15:46:58.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140014362}) 2024-09-17T15:46:58.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.382+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827500597}) 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146120668}) 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2263534424}) 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146120668}) 2024-09-17T15:46:58.382 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=649259629}) 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232640687}) 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261510575}) 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232640687}) 2024-09-17T15:46:58.384 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.386+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3908124071}) 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792095593}) 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831658158}) 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792095593}) 2024-09-17T15:46:58.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.394+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1128242062}) 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869731432}) 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1232212705}) 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1869731432}) 2024-09-17T15:46:58.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136304936}) 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163613519}) 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4028243103}) 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3163613519}) 2024-09-17T15:46:58.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.398+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=399812914}) 2024-09-17T15:46:58.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160364626}) 2024-09-17T15:46:58.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3852986335}) 2024-09-17T15:46:58.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160364626}) 2024-09-17T15:46:58.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227860805}) 2024-09-17T15:46:58.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621033586}) 2024-09-17T15:46:58.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3297147059}) 2024-09-17T15:46:58.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3621033586}) 2024-09-17T15:46:58.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.402+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037996382}) 2024-09-17T15:46:58.403 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.403 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492094663}) 2024-09-17T15:46:58.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535397344}) 2024-09-17T15:46:58.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3492094663}) 2024-09-17T15:46:58.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=931840504}) 2024-09-17T15:46:58.405 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743464924}) 2024-09-17T15:46:58.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.406+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=923978542}) 2024-09-17T15:46:58.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743464924}) 2024-09-17T15:46:58.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703267999}) 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126951435}) 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199872035}) 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126951435}) 2024-09-17T15:46:58.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.410+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2535473927}) 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552281951}) 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655791227}) 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552281951}) 2024-09-17T15:46:58.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239284680}) 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=574304014}) 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438731373}) 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=574304014}) 2024-09-17T15:46:58.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.414+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112485765}) 2024-09-17T15:46:58.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020777086}) 2024-09-17T15:46:58.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462122834}) 2024-09-17T15:46:58.415 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.415 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2020777086}) 2024-09-17T15:46:58.415 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850333399}) 2024-09-17T15:46:58.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306943838}) 2024-09-17T15:46:58.417 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4270563198}) 2024-09-17T15:46:58.417 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.417 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306943838}) 2024-09-17T15:46:58.417 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.418+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525185511}) 2024-09-17T15:46:58.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165462440}) 2024-09-17T15:46:58.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557357947}) 2024-09-17T15:46:58.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165462440}) 2024-09-17T15:46:58.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081220286}) 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121716373}) 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664945736}) 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121716373}) 2024-09-17T15:46:58.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.422+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617556612}) 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453042059}) 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019837120}) 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453042059}) 2024-09-17T15:46:58.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664361196}) 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857714048}) 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314391194}) 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857714048}) 2024-09-17T15:46:58.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.426+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179869643}) 2024-09-17T15:46:58.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415972321}) 2024-09-17T15:46:58.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417295017}) 2024-09-17T15:46:58.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415972321}) 2024-09-17T15:46:58.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2400675654}) 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878643773}) 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1180012997}) 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878643773}) 2024-09-17T15:46:58.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.430+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1893540715}) 2024-09-17T15:46:58.430 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10} crc {logm=2048558102,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1086912755,mon_config_key=1974893097}) 2024-09-17T15:46:58.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=2} crc {logm=1919410798,mdsmap=2815901553,mgr=2332442456,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4020059582,mon_config_key=2429078719,monmap=2119989493}) 2024-09-17T15:46:58.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10} crc {logm=2048558102,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1086912755,mon_config_key=1974893097}) 2024-09-17T15:46:58.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=8} crc {logm=527546442,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3834771353,mon_config_key=574182990}) 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=85} crc {mon_config_key=830496287,monmap=4191199434,nvmeofgw=598337686,osd_metadata=2564601247,osd_pg_creating=1427281147,osd_snap=2421617947}) 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=85} crc {mon_config_key=830496287,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3734937928,osd_pg_creating=1427281147,osd_snap=3463712793}) 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=85} crc {mon_config_key=830496287,monmap=4191199434,nvmeofgw=598337686,osd_metadata=2564601247,osd_pg_creating=1427281147,osd_snap=2421617947}) 2024-09-17T15:46:58.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.434+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=3,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=85} crc {mon_config_key=830496287,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_snap=3817307734}) 2024-09-17T15:46:58.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.435 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3415329332,osdmap=4063441331}) 2024-09-17T15:46:58.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=275606335,osdmap=2877370026}) 2024-09-17T15:46:58.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=43,osdmap=57} crc {osd_snap=3415329332,osdmap=4063441331}) 2024-09-17T15:46:58.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.438+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=42,osdmap=58} crc {osd_snap=829068349,osdmap=543573633}) 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1358684379}) 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2336360352}) 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1358684379}) 2024-09-17T15:46:58.438 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1195706012}) 2024-09-17T15:46:58.440 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.440 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=454792884}) 2024-09-17T15:46:58.440 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1004811250}) 2024-09-17T15:46:58.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=454792884}) 2024-09-17T15:46:58.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.442+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3987561236}) 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1256809167}) 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3809616311}) 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1256809167}) 2024-09-17T15:46:58.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.446+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=897703858}) 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3097271262}) 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=240159620}) 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=47} crc {osdmap=3097271262}) 2024-09-17T15:46:58.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:46:58.458+0000 7f56a93d1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=2440625720}) 2024-09-17T15:46:58.796 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:46:58.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:46:58.858 INFO:tasks.rados.rados.0.smithi203.stdout:3227: done (4 left) 2024-09-17T15:46:58.858 INFO:tasks.rados.rados.0.smithi203.stdout:3228: read oid 10 snap -1 2024-09-17T15:46:58.858 INFO:tasks.rados.rados.0.smithi203.stdout:3228: expect deleted 2024-09-17T15:46:58.858 INFO:tasks.rados.rados.0.smithi203.stdout:3229: write (excl) oid 8 current snap is 347 2024-09-17T15:46:58.858 INFO:tasks.rados.rados.0.smithi203.stdout:3229: seq_num 1129 ranges {575103=576365,1932994=744383} 2024-09-17T15:46:58.868 INFO:tasks.rados.rados.0.smithi203.stdout:3229: writing smithi20328086-8 from 575103 to 1151468 tid 1 2024-09-17T15:46:58.881 INFO:tasks.rados.rados.0.smithi203.stdout:3229: writing smithi20328086-8 from 1932994 to 2677377 tid 2 2024-09-17T15:46:58.881 INFO:tasks.rados.rados.0.smithi203.stdout:3230: rollback oid 23 current snap is 347 2024-09-17T15:46:58.881 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 345 2024-09-17T15:46:58.882 INFO:tasks.rados.rados.0.smithi203.stdout:3231: delete oid 6 current snap is 347 2024-09-17T15:46:58.885 INFO:tasks.rados.rados.0.smithi203.stdout:3220: finishing write tid 2 to smithi20328086-24 2024-09-17T15:46:58.885 INFO:tasks.rados.rados.0.smithi203.stdout:3221: expect (ObjNum 973 snap 294 seq_num 973) 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: get_monmap_and_config 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: build_initial_monmap 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: monmap: 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:58.940 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:58.946357+0000 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:58.946357+0000 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:58.941 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 Processor -- start 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 -- start start 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: init 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c54131710 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2c5413aef0 0x7f2c5413d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-a con 0x7f2c5413aef0 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:58.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2c5413d820 0x7f2c5413fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-c con 0x7f2c5413d820 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c54132200 con 0x7f2c54131710 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f2c540a4370 con 0x7f2c5413aef0 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2c5409de90 con 0x7f2c5413d820 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.942+0000 7f2c59c33640 10 monclient(hunting): authenticate will time out at 2151.911949s 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2c5413aef0 0x7f2c5413d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2c5413aef0 0x7f2c5413d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54150/0 (socket says 172.21.15.86:54150) 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 -- 172.21.15.86:0/3546816818 learned_addr learned my addr 172.21.15.86:0/3546816818 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:58.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): get_auth_request con 0x7f2c5413aef0 auth_method 0 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 1 --2- 172.21.15.86:0/3546816818 >> v2:172.21.15.203:3300/0 conn(0x7f2c5413d820 0x7f2c5413fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): get_auth_request con 0x7f2c5413d820 auth_method 0 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:58.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient(hunting): handle_auth_done global_id 214233 payload 274 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 -- 172.21.15.86:0/3546816818 >> v2:172.21.15.203:3300/0 conn(0x7f2c5413d820 msgr2=0x7f2c5413fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 --2- 172.21.15.86:0/3546816818 >> v2:172.21.15.203:3300/0 conn(0x7f2c5413d820 0x7f2c5413fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 -- 172.21.15.86:0/3546816818 >> v1:172.21.15.203:6789/0 conn(0x7f2c54131710 legacy=0x7f2c54131af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:58.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.946+0000 7f2c52ffd640 1 -- 172.21.15.86:0/3546816818 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c54074710 con 0x7f2c5413aef0 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 1 -- 172.21.15.86:0/3546816818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413aef0 msgr2=0x7f2c5413d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 1 --2- 172.21.15.86:0/3546816818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413aef0 0x7f2c5413d2e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2c5409d2c0 tx=0x7f2c4401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8732ea2222c2166 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 1 -- 172.21.15.86:0/3546816818 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c440b0070 con 0x7f2c5413aef0 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: dump: 2024-09-17T15:46:58.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: _finish_auth 0 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: _check_auth_tickets 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 1 -- 172.21.15.86:0/3546816818 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c440ab600 con 0x7f2c5413aef0 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 1 -- 172.21.15.86:0/3546816818 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c440ab9d0 con 0x7f2c5413aef0 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 5 monclient: authenticate success, global_id 214233 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 4 set_mon_vals no callback set 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c527fc640 10 monclient: dump: 2024-09-17T15:46:58.953 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:58.954 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: shutdown 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- 172.21.15.86:0/3546816818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413aef0 msgr2=0x7f2c5413d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- 172.21.15.86:0/3546816818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413aef0 0x7f2c5413d2e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2c5409d2c0 tx=0x7f2c4401c3a0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- 172.21.15.86:0/3546816818 shutdown_connections 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- 172.21.15.86:0/3546816818 >> v2:172.21.15.203:3300/0 conn(0x7f2c5413d820 0x7f2c5413fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- 172.21.15.86:0/3546816818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413aef0 0x7f2c5413d2e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- 172.21.15.86:0/3546816818 >> 172.21.15.86:0/3546816818 conn(0x7f2c5406d170 msgr2=0x7f2c540a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- 172.21.15.86:0/3546816818 shutdown_connections 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- 172.21.15.86:0/3546816818 wait complete. 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: build_initial_monmap 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: monmap: 2024-09-17T15:46:58.955 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 Processor -- start 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- start start 2024-09-17T15:46:58.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: init 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c54131710 0x7f2c54134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-c con 0x7f2c54131710 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c5413aef0 0x7f2c54139b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-b con 0x7f2c5413aef0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): picked mon.noname-a con 0x7f2c5413d820 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2c54142970 con 0x7f2c5413d820 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c54068db0 con 0x7f2c5413aef0 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:58.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2c54142af0 con 0x7f2c54131710 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c59c33640 10 monclient(hunting): authenticate will time out at 2151.919181s 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c54131710 0x7f2c54134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54156/0 (socket says 172.21.15.86:54156) 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 -- 172.21.15.86:0/2191382264 learned_addr learned my addr 172.21.15.86:0/2191382264 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): get_auth_request con 0x7f2c5413d820 auth_method 0 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): get_auth_request con 0x7f2c54131710 auth_method 0 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:58.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c5413aef0 0x7f2c54139b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 10 monclient(hunting): get_auth_request con 0x7f2c5413aef0 auth_method 0 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient(hunting): handle_auth_done global_id 214236 payload 824 2024-09-17T15:46:58.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 -- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c54131710 msgr2=0x7f2c54134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c54131710 0x7f2c54134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 -- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c5413aef0 msgr2=0x7f2c54139b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c5413aef0 0x7f2c54139b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient: _finish_hunting 0 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 monclient: found mon.noname-a 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:58.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.950+0000 7f2c53fff640 1 -- 172.21.15.86:0/2191382264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c5413aa30 con 0x7f2c5413d820 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c53fff640 1 -- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 msgr2=0x7f2c541cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c53fff640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00efc0 tx=0x7f2c4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c018020 con 0x7f2c5413d820 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: dump: 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: _finish_auth 0 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c4c014680 con 0x7f2c5413d820 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 5 monclient: authenticate success, global_id 214236 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _renew_subs 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:58.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c5406efb0 con 0x7f2c5413d820 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c014a70 con 0x7f2c5413d820 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 10 monclient: dump: 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:58.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _renew_subs 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c54135070 con 0x7f2c5413d820 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c51ffb640 4 set_mon_vals no callback set 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c54131af0 con 0x7f2c5413d820 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2c4c01b020 con 0x7f2c5413d820 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 0x7f2c4003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f2c4c050e50 con 0x7f2c5413d820 2024-09-17T15:46:58.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c537fe640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 0x7f2c4003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:58.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.954+0000 7f2c537fe640 10 monclient: get_auth_request con 0x7f2c4003bbb0 auth_method 0 2024-09-17T15:46:58.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.958+0000 7f2c537fe640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 0x7f2c4003e070 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2c48007fd0 tx=0x7f2c48007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:58.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.958+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2c4c022030 con 0x7f2c5413d820 2024-09-17T15:46:58.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.958+0000 7f2c3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:58.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:58.958+0000 7f2c3bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:59.006 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:59.052 INFO:tasks.rados.rados.0.smithi203.stdout:3220: finishing write tid 3 to smithi20328086-24 2024-09-17T15:46:59.053 INFO:tasks.rados.rados.0.smithi203.stdout:3222: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:46:59.177 INFO:tasks.rados.rados.0.smithi203.stdout:3224: expect (ObjNum 940 snap 287 seq_num 940) 2024-09-17T15:46:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c59c33640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:46:59.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c59c33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c59c33640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c541386e0 con 0x7f2c5413d820 2024-09-17T15:46:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c3bfff640 1 -- 172.21.15.86:0/2191382264 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2c4c010040 con 0x7f2c5413d820 2024-09-17T15:46:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:46:59.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c3bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:59.192 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:46:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.194+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 msgr2=0x7f2c4003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 0x7f2c4003e070 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2c48007fd0 tx=0x7f2c48007e40 comp rx=0 tx=0).stop 2024-09-17T15:46:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 10 monclient: shutdown 2024-09-17T15:46:59.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 msgr2=0x7f2c541cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00efc0 tx=0x7f2c4c00c490 comp rx=0 tx=0).stop 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 shutdown_connections 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2c4003bbb0 0x7f2c4003e070 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c5413d820 0x7f2c541cfc00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c5413aef0 0x7f2c54139b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 --2- 172.21.15.86:0/2191382264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c54131710 0x7f2c54134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 >> 172.21.15.86:0/2191382264 conn(0x7f2c5406d170 msgr2=0x7f2c540a3540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 shutdown_connections 2024-09-17T15:46:59.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.198+0000 7f2c59c33640 1 -- 172.21.15.86:0/2191382264 wait complete. 2024-09-17T15:46:59.210 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:46:59.210 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:46:59.211 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:46:59.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:46:59.277 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:46:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: get_monmap_and_config 2024-09-17T15:46:59.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: monmap: 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:59.332818+0000 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:59.332818+0000 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:59.327 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:59.328 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:59.328 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 Processor -- start 2024-09-17T15:46:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 -- start start 2024-09-17T15:46:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: init 2024-09-17T15:46:59.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f77e80a4f70 0x7f77e809cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-c con 0x7f77e80a4f70 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77e80a5350 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77e813d180 0x7f77e813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-a con 0x7f77e813d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77e809d430 con 0x7f77e80a5350 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f77e809c7f0 con 0x7f77e813d180 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f77e80a68d0 con 0x7f77e80a4f70 2024-09-17T15:46:59.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ef9cc640 10 monclient(hunting): authenticate will time out at 2152.298219s 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77e813d180 0x7f77e813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77e813d180 0x7f77e813f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54168/0 (socket says 172.21.15.86:54168) 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 -- 172.21.15.86:0/182335862 learned_addr learned my addr 172.21.15.86:0/182335862 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): get_auth_request con 0x7f77e813d180 auth_method 0 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 1 --2- 172.21.15.86:0/182335862 >> v2:172.21.15.203:3300/0 conn(0x7f77e80a4f70 0x7f77e809cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): get_auth_request con 0x7f77e80a4f70 auth_method 0 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient(hunting): handle_auth_done global_id 214239 payload 274 2024-09-17T15:46:59.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 -- 172.21.15.86:0/182335862 >> v2:172.21.15.203:3300/0 conn(0x7f77e80a4f70 msgr2=0x7f77e809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 --2- 172.21.15.86:0/182335862 >> v2:172.21.15.203:3300/0 conn(0x7f77e80a4f70 0x7f77e809cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 -- 172.21.15.86:0/182335862 >> v1:172.21.15.203:6789/0 conn(0x7f77e80a5350 legacy=0x7f77e80a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 monclient: found mon.noname-a 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.330+0000 7f77edf69640 1 -- 172.21.15.86:0/182335862 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e80a64a0 con 0x7f77e813d180 2024-09-17T15:46:59.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77edf69640 1 -- 172.21.15.86:0/182335862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e813d180 msgr2=0x7f77e813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77edf69640 1 --2- 172.21.15.86:0/182335862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e813d180 0x7f77e813f570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f77e809c970 tx=0x7f77e001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c16596c2638630f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 1 -- 172.21.15.86:0/182335862 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00a5700 con 0x7f77e813d180 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: dump: 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.340 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 1 -- 172.21.15.86:0/182335862 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e00a58a0 con 0x7f77e813d180 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 5 monclient: authenticate success, global_id 214239 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:59.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 4 set_mon_vals no callback set 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 1 -- 172.21.15.86:0/182335862 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e00a5c70 con 0x7f77e813d180 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77dffff640 10 monclient: dump: 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.342 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 10 monclient: shutdown 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 -- 172.21.15.86:0/182335862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e813d180 msgr2=0x7f77e813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/182335862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e813d180 0x7f77e813f570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f77e809c970 tx=0x7f77e001c880 comp rx=0 tx=0).stop 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 -- 172.21.15.86:0/182335862 shutdown_connections 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/182335862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e813d180 0x7f77e813f570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/182335862 >> v2:172.21.15.203:3300/0 conn(0x7f77e80a4f70 0x7f77e809cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.334+0000 7f77ef9cc640 1 -- 172.21.15.86:0/182335862 >> 172.21.15.86:0/182335862 conn(0x7f77e80a7240 msgr2=0x7f77e80a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- 172.21.15.86:0/182335862 shutdown_connections 2024-09-17T15:46:59.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- 172.21.15.86:0/182335862 wait complete. 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient: monmap: 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:59.344 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 Processor -- start 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- start start 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient: init 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-a con 0x7f77e80a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77e80a5350 0x7f77e80a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-b con 0x7f77e80a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e813d180 0x7f77e8138220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): picked mon.noname-c con 0x7f77e813d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:59.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8142450 con 0x7f77e80a4f70 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81422d0 con 0x7f77e80a5350 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f77e81425d0 con 0x7f77e813d180 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ef9cc640 10 monclient(hunting): authenticate will time out at 2152.305466s 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e813d180 0x7f77e8138220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54176/0 (socket says 172.21.15.86:54176) 2024-09-17T15:46:59.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 -- 172.21.15.86:0/3513158861 learned_addr learned my addr 172.21.15.86:0/3513158861 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): get_auth_request con 0x7f77e80a4f70 auth_method 0 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): get_auth_request con 0x7f77e813d180 auth_method 0 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.347 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77e80a5350 0x7f77e80a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77edf69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 10 monclient(hunting): get_auth_request con 0x7f77e80a5350 auth_method 0 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient(hunting): handle_auth_done global_id 214242 payload 824 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e813d180 msgr2=0x7f77e8138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e813d180 0x7f77e8138220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77e80a5350 msgr2=0x7f77e80a31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77e80a5350 0x7f77e80a31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 monclient: found mon.noname-a 2024-09-17T15:46:59.348 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ed768640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e8122330 con 0x7f77e80a4f70 2024-09-17T15:46:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.338+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f77e80a5350 0x7f77e80a31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:46:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ed768640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 msgr2=0x7f77e81213d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.349 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ed768640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77e400e930 tx=0x7f77e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: finishing write tid 4 to smithi20328086-24 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: finishing write tid 5 to smithi20328086-24 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: oid 24 updating version 0 to 1531 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: oid 24 version 1531 is already newer than 1529 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: oid 24 version 1531 is already newer than 1530 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: oid 24 version 1531 is already newer than 1528 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1531 (ObjNum 1127 snap 347 seq_num 1127) dirty exists 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: left oid 24 (ObjNum 1127 snap 347 seq_num 1127) 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3229: finishing write tid 1 to smithi20328086-8 2024-09-17T15:46:59.351 INFO:tasks.rados.rados.0.smithi203.stdout:3220: done (7 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3221: done (6 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3222: done (5 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3224: done (4 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3228: done (3 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3231: done (2 left) 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3232: snap_create 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3230: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1345 (ObjNum 33 snap 0 seq_num 3605972592) dirty dne 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: finishing write tid 2 to smithi20328086-8 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: finishing write tid 3 to smithi20328086-8 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: finishing write tid 4 to smithi20328086-8 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: oid 8 updating version 0 to 1531 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: oid 8 updating version 1531 to 1533 2024-09-17T15:46:59.352 INFO:tasks.rados.rados.0.smithi203.stdout:3229: oid 8 version 1533 is already newer than 1532 2024-09-17T15:46:59.353 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1533 (ObjNum 1129 snap 347 seq_num 1129) dirty exists 2024-09-17T15:46:59.353 INFO:tasks.rados.rados.0.smithi203.stdout:3229: left oid 8 (ObjNum 1129 snap 347 seq_num 1129) 2024-09-17T15:46:59.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4017070 con 0x7f77e80a4f70 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: dump: 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.357 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 5 monclient: authenticate success, global_id 214242 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e400cc10 con 0x7f77e80a4f70 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 monclient: _renew_subs 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4013470 con 0x7f77e80a4f70 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e813da50 con 0x7f77e80a4f70 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77df7fe640 4 set_mon_vals no callback set 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 10 monclient: dump: 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 monclient: _renew_subs 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e8122bd0 con 0x7f77e80a4f70 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 0x7f77e81d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8122780 con 0x7f77e81387c0 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 0x7f77e81d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient: get_auth_request con 0x7f77e81387c0 auth_method 0 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 10 monclient(hunting): handle_auth_done global_id 214245 payload 824 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ecf67640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77e809cef0 con 0x7f77e81387c0 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f77e401a070 con 0x7f77e80a4f70 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.342+0000 7f77ddffb640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77cc03bbe0 0x7f77cc03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f77e4050080 con 0x7f77e80a4f70 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77edf69640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77cc03bbe0 0x7f77cc03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77edf69640 10 monclient: get_auth_request con 0x7f77cc03bbe0 auth_method 0 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ecf67640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 msgr2=0x7f77e81d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 0x7f77e81d01f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e500 tx=0x7f77d800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8013800 con 0x7f77e81387c0 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 10 monclient: dump: 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.362 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f77d8013800 con 0x7f77e81387c0 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 msgr2=0x7f77e81d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 0x7f77e81d01f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e500 tx=0x7f77d800e9c0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 msgr2=0x7f77e81d01f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77ddffb640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e81387c0 0x7f77e81d01f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.346+0000 7f77edf69640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77cc03bbe0 0x7f77cc03e0a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f77e809d9c0 tx=0x7f77e00a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ef9cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 0x7f77e802d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ef9cc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f77e80a5ff0 con 0x7f77e802d450 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 0x7f77e802d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient: get_auth_request con 0x7f77e802d450 auth_method 0 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 10 monclient(hunting): handle_auth_done global_id 214248 payload 824 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 1 -- 172.21.15.86:0/3513158861 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77e809f630 con 0x7f77e802d450 2024-09-17T15:46:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 msgr2=0x7f77e802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ecf67640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 0x7f77e802d830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e500 tx=0x7f77d8003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8006e00 con 0x7f77e802d450 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 10 monclient: dump: 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f77d8006e00 con 0x7f77e802d450 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 msgr2=0x7f77e802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 0x7f77e802d830 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e500 tx=0x7f77d8003770 comp rx=0 tx=0).stop 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 msgr2=0x7f77e802d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.354+0000 7f77ddffb640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e802d450 0x7f77e802d830 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e500 tx=0x7f77d8003770 comp rx=0 tx=0).stop 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 75529, 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:59.379 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:59.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77cc03bbe0 msgr2=0x7f77cc03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f77cc03bbe0 0x7f77cc03e0a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f77e809d9c0 tx=0x7f77e00a7070 comp rx=0 tx=0).stop 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 10 monclient: shutdown 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 msgr2=0x7f77e81213d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f77e400e930 tx=0x7f77e400edf0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77edf69640 1 -- 172.21.15.86:0/3513158861 reap_dead start 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 shutdown_connections 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 --2- 172.21.15.86:0/3513158861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e80a4f70 0x7f77e81213d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 >> 172.21.15.86:0/3513158861 conn(0x7f77e80a7240 msgr2=0x7f77e813b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 shutdown_connections 2024-09-17T15:46:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.358+0000 7f77ef9cc640 1 -- 172.21.15.86:0/3513158861 wait complete. 2024-09-17T15:46:59.382 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:59.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:46:59.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.498+0000 7f304d758640 10 monclient: get_monmap_and_config 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.498+0000 7f304d758640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient: monmap: 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:59.503983+0000 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:59.503983+0000 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:59.498 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:59.499 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:59.499 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:59.499 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:59.499 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 Processor -- start 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 -- start start 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient: init 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f304809d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f304809e260 0x7f3048139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): picked mon.noname-a con 0x7f304809e260 addr v2:172.21.15.86:3300/0 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f304813a440 addr v1:172.21.15.86:6789/0 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f304809ff00 con 0x7f304813a440 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f304809ec30 con 0x7f304809d770 2024-09-17T15:46:59.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f304809e7f0 con 0x7f304809e260 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f304d758640 10 monclient(hunting): authenticate will time out at 2152.469471s 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f304809e260 0x7f3048139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30477fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f304813a440 0x7f304813c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45184/0 (socket says 172.21.15.86:45184) 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30477fe640 1 -- 172.21.15.86:0/2498325776 learned_addr learned my addr 172.21.15.86:0/2498325776 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): get_auth_request con 0x7f304809e260 auth_method 0 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient(hunting): handle_auth_done global_id 214251 payload 274 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 1 -- 172.21.15.86:0/2498325776 >> v1:172.21.15.203:6789/0 conn(0x7f304809d770 legacy=0x7f304809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 1 -- 172.21.15.86:0/2498325776 >> v1:172.21.15.86:6789/0 conn(0x7f304813a440 legacy=0x7f304813c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 1 monclient: found mon.noname-a 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.502+0000 7f30467fc640 1 -- 172.21.15.86:0/2498325776 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304809ef10 con 0x7f304809e260 2024-09-17T15:46:59.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f30467fc640 1 -- 172.21.15.86:0/2498325776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809e260 msgr2=0x7f3048139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f30467fc640 1 --2- 172.21.15.86:0/2498325776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809e260 0x7f3048139f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f303c0097c0 tx=0x7f303c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d25a02e8df4df1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 1 -- 172.21.15.86:0/2498325776 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0b0070 con 0x7f304809e260 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: dump: 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 1 -- 172.21.15.86:0/2498325776 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f303c0ab600 con 0x7f304809e260 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 1 -- 172.21.15.86:0/2498325776 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303c0ab9d0 con 0x7f304809e260 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 5 monclient: authenticate success, global_id 214251 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 4 set_mon_vals no callback set 2024-09-17T15:46:59.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f3045ffb640 10 monclient: dump: 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 10 monclient: shutdown 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 -- 172.21.15.86:0/2498325776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809e260 msgr2=0x7f3048139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 --2- 172.21.15.86:0/2498325776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809e260 0x7f3048139f00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f303c0097c0 tx=0x7f303c01c360 comp rx=0 tx=0).stop 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 -- 172.21.15.86:0/2498325776 shutdown_connections 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 --2- 172.21.15.86:0/2498325776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809e260 0x7f3048139f00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 -- 172.21.15.86:0/2498325776 >> 172.21.15.86:0/2498325776 conn(0x7f30480a2f40 msgr2=0x7f30480a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 -- 172.21.15.86:0/2498325776 shutdown_connections 2024-09-17T15:46:59.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.506+0000 7f304d758640 1 -- 172.21.15.86:0/2498325776 wait complete. 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient: monmap: 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.515 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 Processor -- start 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 -- start start 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient: init 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809d770 0x7f30481348c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): picked mon.noname-a con 0x7f304809d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f304809e260 0x7f3048134e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): picked mon.noname-b con 0x7f304809e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:59.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 0x7f3048135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): picked mon.noname-c con 0x7f304813a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f30481408f0 con 0x7f304809d770 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3048140770 con 0x7f304809e260 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3048140a70 con 0x7f304813a440 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f304d758640 10 monclient(hunting): authenticate will time out at 2152.477670s 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809d770 0x7f30481348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 0x7f3048135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 0x7f3048135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48976/0 (socket says 172.21.15.86:48976) 2024-09-17T15:46:59.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 learned_addr learned my addr 172.21.15.86:0/2934715057 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): get_auth_request con 0x7f304813a440 auth_method 0 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): get_auth_request con 0x7f304809d770 auth_method 0 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30467fc640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f304809e260 0x7f3048134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30467fc640 10 monclient(hunting): get_auth_request con 0x7f304809e260 auth_method 0 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30467fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient(hunting): handle_auth_done global_id 164750 payload 824 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f304809e260 msgr2=0x7f3048134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f304809e260 0x7f3048134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809d770 msgr2=0x7f30481348c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809d770 0x7f30481348c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 monclient: found mon.noname-c 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:46:59.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3048072f70 con 0x7f304813a440 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.510+0000 7f3046ffd640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304809d770 0x7f30481348c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 msgr2=0x7f3048135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f30477fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 0x7f3048135340 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f304000caf0 tx=0x7f3040005b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040017070 con 0x7f304813a440 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: dump: 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f304000ade0 con 0x7f304813a440 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3040012640 con 0x7f304813a440 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 5 monclient: authenticate success, global_id 164750 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 monclient: _renew_subs 2024-09-17T15:46:59.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3048073250 con 0x7f304813a440 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3044ff9640 4 set_mon_vals no callback set 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 10 monclient: dump: 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.522 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 monclient: _renew_subs 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3048071af0 con 0x7f304813a440 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 0x7f3048130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f304813ad00 con 0x7f30480a0ff0 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 0x7f3048130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient: get_auth_request con 0x7f30480a0ff0 auth_method 0 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f304001a070 con 0x7f304813a440 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 10 monclient(hunting): handle_auth_done global_id 164753 payload 824 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f3046ffd640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f304809db50 con 0x7f30480a0ff0 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f303403bb90 0x7f303403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f304004fc60 con 0x7f304813a440 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f30467fc640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f303403bb90 0x7f303403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.514+0000 7f30467fc640 10 monclient: get_auth_request con 0x7f303403bb90 auth_method 0 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f3046ffd640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 msgr2=0x7f3048130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f3046ffd640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 0x7f3048130a00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f303800fae0 tx=0x7f3038004510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303800ddc0 con 0x7f30480a0ff0 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 10 monclient: dump: 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f303800ddc0 con 0x7f30480a0ff0 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 msgr2=0x7f3048130a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 0x7f3048130a00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f303800fae0 tx=0x7f3038004510 comp rx=0 tx=0).stop 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 msgr2=0x7f3048130a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f302f7fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f30480a0ff0 0x7f3048130a00 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f303800fae0 tx=0x7f3038004510 comp rx=0 tx=0).stop 2024-09-17T15:46:59.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.518+0000 7f30467fc640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f303403bb90 0x7f303403e050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f303c0ab000 tx=0x7f303c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f304d758640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f304d758640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 0x7f3048131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f304d758640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3048131890 con 0x7f3048130f40 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 0x7f3048131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient: get_auth_request con 0x7f3048130f40 auth_method 0 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 10 monclient(hunting): handle_auth_done global_id 164756 payload 824 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.522+0000 7f3046ffd640 1 -- 172.21.15.86:0/2934715057 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3048072bf0 con 0x7f3048130f40 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f3046ffd640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 msgr2=0x7f3048131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f3046ffd640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 0x7f3048131350 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3038018000 tx=0x7f3038003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30380047f0 con 0x7f3048130f40 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 10 monclient: dump: 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.529 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f30380047f0 con 0x7f3048130f40 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 msgr2=0x7f3048131350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 0x7f3048131350 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3038018000 tx=0x7f3038003770 comp rx=0 tx=0).stop 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 msgr2=0x7f3048131350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f302f7fe640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3048130f40 0x7f3048131350 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3038018000 tx=0x7f3038003770 comp rx=0 tx=0).stop 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 322714, 2024-09-17T15:46:59.531 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.532 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.533 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.534 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:59.535 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.536 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.540 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.541 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:46:59.543 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 7 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:59.544 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.545 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:59.545 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f303403bb90 msgr2=0x7f303403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f303403bb90 0x7f303403e050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f303c0ab000 tx=0x7f303c008040 comp rx=0 tx=0).stop 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 10 monclient: shutdown 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 msgr2=0x7f3048135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 --2- 172.21.15.86:0/2934715057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304813a440 0x7f3048135340 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f304000caf0 tx=0x7f3040005b60 comp rx=0 tx=0).stop 2024-09-17T15:46:59.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 reap_dead start 2024-09-17T15:46:59.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f30477fe640 1 -- 172.21.15.86:0/2934715057 reap_dead start 2024-09-17T15:46:59.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 shutdown_connections 2024-09-17T15:46:59.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 >> 172.21.15.86:0/2934715057 conn(0x7f30480a2f40 msgr2=0x7f304813d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 shutdown_connections 2024-09-17T15:46:59.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.526+0000 7f304d758640 1 -- 172.21.15.86:0/2934715057 wait complete. 2024-09-17T15:46:59.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:59.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:46:59.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:46:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: get_monmap_and_config 2024-09-17T15:46:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: monmap: 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:46:59.666505+0000 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:46:59.666505+0000 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:46:59.661 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 Processor -- start 2024-09-17T15:46:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 -- start start 2024-09-17T15:46:59.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: init 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2cf409d770 0x7f2cf409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf409d770 addr v2:172.21.15.203:3300/0 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf409e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cf4134f80 addr v1:172.21.15.203:6789/0 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf409ff00 con 0x7f2cf4134f80 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf409e7f0 con 0x7f2cf409d770 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf40a2110 con 0x7f2cf409e090 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.662+0000 7f2cf96d5640 10 monclient(hunting): authenticate will time out at 2152.631958s 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:48994/0 (socket says 172.21.15.86:48994) 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 -- 172.21.15.86:0/3443042523 learned_addr learned my addr 172.21.15.86:0/3443042523 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request con 0x7f2cf409e090 auth_method 0 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf2ffd640 1 --2- 172.21.15.86:0/3443042523 >> v2:172.21.15.203:3300/0 conn(0x7f2cf409d770 0x7f2cf409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf2ffd640 10 monclient(hunting): get_auth_request con 0x7f2cf409d770 auth_method 0 2024-09-17T15:46:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_done global_id 164759 payload 274 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 -- 172.21.15.86:0/3443042523 >> v2:172.21.15.203:3300/0 conn(0x7f2cf409d770 msgr2=0x7f2cf409db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/3443042523 >> v2:172.21.15.203:3300/0 conn(0x7f2cf409d770 0x7f2cf409db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 -- 172.21.15.86:0/3443042523 >> v1:172.21.15.203:6789/0 conn(0x7f2cf4134f80 legacy=0x7f2cf413bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 monclient: found mon.noname-b 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:46:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.666+0000 7f2cf27fc640 1 -- 172.21.15.86:0/3443042523 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf409ec30 con 0x7f2cf409e090 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 1 -- 172.21.15.86:0/3443042523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 msgr2=0x7f2cf4134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/3443042523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85841039ac7c719e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 1 -- 172.21.15.86:0/3443042523 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80b0070 con 0x7f2cf409e090 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: dump: 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 5 monclient: authenticate success, global_id 164759 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 1 -- 172.21.15.86:0/3443042523 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce80ab600 con 0x7f2cf409e090 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: get_monmap_and_config success 2024-09-17T15:46:59.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 4 set_mon_vals no callback set 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 1 -- 172.21.15.86:0/3443042523 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80ab9d0 con 0x7f2cf409e090 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf1ffb640 10 monclient: dump: 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: shutdown 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- 172.21.15.86:0/3443042523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 msgr2=0x7f2cf4134a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/3443042523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce801c420 comp rx=0 tx=0).stop 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- 172.21.15.86:0/3443042523 shutdown_connections 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/3443042523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf409e090 0x7f2cf4134a40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/3443042523 >> v2:172.21.15.203:3300/0 conn(0x7f2cf409d770 0x7f2cf409db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- 172.21.15.86:0/3443042523 >> 172.21.15.86:0/3443042523 conn(0x7f2cf40a2f40 msgr2=0x7f2cf40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- 172.21.15.86:0/3443042523 shutdown_connections 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- 172.21.15.86:0/3443042523 wait complete. 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: build_initial_monmap 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: monmap: 2024-09-17T15:46:59.674 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 Processor -- start 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- start start 2024-09-17T15:46:59.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: init 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: _reopen_session rank -1 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 0x7f2cf40a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf409d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf409e090 0x7f2cf4133d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf409e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf4134f80 0x7f2cf41cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf4134f80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf409cce0 con 0x7f2cf409d770 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf409e4c0 con 0x7f2cf409e090 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf409ce60 con 0x7f2cf4134f80 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): _renew_subs 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf96d5640 10 monclient(hunting): authenticate will time out at 2152.639268s 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf4134f80 0x7f2cf41cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 0x7f2cf40a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf4134f80 0x7f2cf41cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49006/0 (socket says 172.21.15.86:49006) 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 0x7f2cf40a0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54228/0 (socket says 172.21.15.86:54228) 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 1 -- 172.21.15.86:0/787185646 learned_addr learned my addr 172.21.15.86:0/787185646 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): get_auth_request con 0x7f2cf4134f80 auth_method 0 2024-09-17T15:46:59.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): get_auth_request con 0x7f2cf409d770 auth_method 0 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf409e090 0x7f2cf4133d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request con 0x7f2cf409e090 auth_method 0 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient(hunting): handle_auth_done global_id 214257 payload 824 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf4134f80 msgr2=0x7f2cf41cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2cf4134f80 0x7f2cf41cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf409e090 msgr2=0x7f2cf4133d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf409e090 0x7f2cf4133d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 monclient: found mon.noname-a 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf2ffd640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf413bdc0 con 0x7f2cf409d770 2024-09-17T15:46:59.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.670+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf409e090 0x7f2cf4133d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf2ffd640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 msgr2=0x7f2cf40a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf2ffd640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 0x7f2cf40a0ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400d5f0 tx=0x7f2ce400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4012800 con 0x7f2cf409d770 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: dump: 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.683 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 5 monclient: authenticate success, global_id 214257 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce4019070 con 0x7f2cf409d770 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:46:59.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 monclient: _renew_subs 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4012bc0 con 0x7f2cf409d770 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf406f6e0 con 0x7f2cf409d770 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf17fa640 4 set_mon_vals no callback set 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 10 monclient: dump: 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.685 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 monclient: _renew_subs 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf411f6e0 con 0x7f2cf409d770 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 0x7f2cf41ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf409db50 con 0x7f2cf4134280 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2ce401e460 con 0x7f2cf409d770 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 0x7f2cf41ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient: get_auth_request con 0x7f2cf4134280 auth_method 0 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ce003bb60 0x7f2ce003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f2ce40503a0 con 0x7f2cf409d770 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf37fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ce003bb60 0x7f2ce003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.674+0000 7f2cf37fe640 10 monclient: get_auth_request con 0x7f2ce003bb60 auth_method 0 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_done global_id 224140 payload 824 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cf27fc640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cf40008e0 con 0x7f2cf4134280 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cf27fc640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 msgr2=0x7f2cf41ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 0x7f2cf41ce3a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80ad000 tx=0x7f2ce801ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cf37fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ce003bb60 0x7f2ce003e020 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40a1ac0 tx=0x7f2cec003d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80b0040 con 0x7f2cf4134280 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 10 monclient: dump: 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ce80b5660 con 0x7f2cf4134280 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 msgr2=0x7f2cf41ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 0x7f2cf41ce3a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80ad000 tx=0x7f2ce801ac60 comp rx=0 tx=0).stop 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 msgr2=0x7f2cf41ce3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.678+0000 7f2cdb7fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf4134280 0x7f2cf41ce3a0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80ad000 tx=0x7f2ce801ac60 comp rx=0 tx=0).stop 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf96d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 0x7f2cf4135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf96d5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf41cecc0 con 0x7f2cf41ce8e0 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 0x7f2cf4135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient: get_auth_request con 0x7f2cf41ce8e0 auth_method 0 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:46:59.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 10 monclient(hunting): handle_auth_done global_id 224143 payload 824 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.686+0000 7f2cf27fc640 1 -- 172.21.15.86:0/787185646 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cf411ee40 con 0x7f2cf41ce8e0 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf27fc640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 msgr2=0x7f2cf4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf27fc640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 0x7f2cf4135ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce80191c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80b0040 con 0x7f2cf41ce8e0 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 10 monclient: dump: 2024-09-17T15:46:59.692 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f2ce80bed00 con 0x7f2cf41ce8e0 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 msgr2=0x7f2cf4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 0x7f2cf4135ee0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce80191c0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 msgr2=0x7f2cf4135ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cdb7fe640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cf41ce8e0 0x7f2cf4135ee0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009870 tx=0x7f2ce80191c0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 184, 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10139, 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:46:59.694 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.695 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.696 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:46:59.697 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:46:59.698 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:46:59.699 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.700 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:46:59.701 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:46:59.703 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.704 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:46:59.705 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ce003bb60 msgr2=0x7f2ce003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2ce003bb60 0x7f2ce003e020 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40a1ac0 tx=0x7f2cec003d40 comp rx=0 tx=0).stop 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 10 monclient: shutdown 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 msgr2=0x7f2cf40a0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 --2- 172.21.15.86:0/787185646 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2cf409d770 0x7f2cf40a0ca0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400d5f0 tx=0x7f2ce400dab0 comp rx=0 tx=0).stop 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf37fe640 1 -- 172.21.15.86:0/787185646 reap_dead start 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf37fe640 1 -- 172.21.15.86:0/787185646 reap_dead start 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 shutdown_connections 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 >> 172.21.15.86:0/787185646 conn(0x7f2cf40a2f40 msgr2=0x7f2cf4070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:46:59.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 shutdown_connections 2024-09-17T15:46:59.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:46:59.690+0000 7f2cf96d5640 1 -- 172.21.15.86:0/787185646 wait complete. 2024-09-17T15:46:59.708 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:46:59.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-17T15:46:59.708 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-17T15:46:59.708 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-17T15:46:59.708 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-17T15:46:59.709 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:46:59.709 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:46:59.720 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:46:59.720 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:46:59.720 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:46:59.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:47:14.721 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:47:14.722 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:47:14.722 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:47:14.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: get_monmap_and_config 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: build_initial_monmap 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: monmap: 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:47:14.853983+0000 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:47:14.853983+0000 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:47:14.849 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 1 Processor -- start 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 1 -- start start 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: init 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T15:47:14.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f54300689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f5430068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.850+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5430134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f543009d770 con 0x7f5430134a40 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f5430068db0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542effd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f54300691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f543009efe0 con 0x7f54300689d0 2024-09-17T15:47:14.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f5435e7d640 10 monclient(hunting): authenticate will time out at 2167.820064s 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542effd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f5430134a40 0x7f5430134ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.86:35042/0 (socket says 172.21.15.86:35042) 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:55160/0 (socket says 172.21.15.86:55160) 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 learned_addr learned my addr 172.21.15.86:0/892359529 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f54300689d0 auth_method 0 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:14.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:14.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:14.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:14.854+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f5430134a40 legacy=0x7f5430134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f54300689d0 auth_method 0 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f54300691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:15.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:15.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:15.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f54300691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:15.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f54300689d0 auth_method 0 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:15.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:15.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:16.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f54300691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:16.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:16.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f54300689d0 auth_method 0 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:16.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:16.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): tick 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 msgr2=0x7f543009ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:17.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f54300689d0 0x7f543009ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f54300691c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f54300691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f5430134a40 legacy=0x7f5430134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient: _add_conns ranks=[2,0] 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f541c003720 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f541c0062a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54300a04e0 con 0x7f541c0062a0 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:17.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f541c003720 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:17.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:17.854+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:18.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:18.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:18.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:18.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.058+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:18.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:18.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:18.462+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:19.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:19.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:19.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:19.266+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): tick 2024-09-17T15:47:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:20.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f541c003720 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 reap_dead start 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f5430068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f541c0062a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c00b090 con 0x7f541c0062a0 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:20.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f5430068db0 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f543009efe0 con 0x7f541c003720 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f541c00a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:20.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:20.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:20.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f541c00a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:21.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:21.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f541c00a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:21.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:21.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c003720 auth_method 0 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:22.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f541c00a040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:22.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:22.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:23.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): tick 2024-09-17T15:47:23.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:23.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:23.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:23.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 msgr2=0x7f541c005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c003720 0x7f541c005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 msgr2=0x7f541c00a040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f5430068db0 0x7f541c00a040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c0062a0 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient: _add_conns ranks=[1,4] 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f541c00a650 addr v1:172.21.15.203:6789/0 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f541c00abc0 addr v2:172.21.15.203:3300/0 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54300a04e0 con 0x7f541c00a650 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f541c00abc0 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:23.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00abc0 auth_method 0 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:23.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:23.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:23.857+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 legacy=0x7f541c00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00abc0 auth_method 0 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:24.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.057+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00abc0 auth_method 0 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:24.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:24.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.461+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.557+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 legacy=0x7f541c00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:47:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.557+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 legacy=0x7f541c00f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:47:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.557+0000 7f542f7fe640 1 --1- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 0x7f541c00f4e0 :-1 s=OPENED pgs=42 cs=1 l=1).handle_message read tag failed 2024-09-17T15:47:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.557+0000 7f542f7fe640 1 --1- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 0x7f541c00f4e0 :-1 s=OPENED pgs=42 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:47:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:24.557+0000 7f542e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.203:6789/0 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00abc0 auth_method 0 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:47:25.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:25.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:25.265+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:26.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): tick 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 msgr2=0x7f541c012c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00abc0 0x7f541c012c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c00a650 legacy=0x7f541c00f4e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 reap_dead start 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:26.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f541c00a040 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f541c013220 addr v1:172.21.15.203:6789/0 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f541c013640 addr v2:172.21.15.86:3300/0 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c00b090 con 0x7f541c013220 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f541c013640 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f543009efe0 con 0x7f541c00a040 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 msgr2=0x7f541c013af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542ffff640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c00a040 auth_method 0 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.853+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:26.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:26.857+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:27.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 msgr2=0x7f541c013af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:27.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542ffff640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c00a040 auth_method 0 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:27.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.057+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 msgr2=0x7f541c013af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542ffff640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c00a040 auth_method 0 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:27.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:27.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:27.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:27.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:27.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:27.461+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:47:28.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 msgr2=0x7f541c013af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:28.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542ffff640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c00a040 auth_method 0 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:28.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:28.265+0000 7f542f7fe640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:47:29.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.561+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-17T15:47:29.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.561+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 legacy=0x7f541c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:47:29.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.561+0000 7f542effd640 1 --1- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 0x7f541c008740 :-1 s=OPENED pgs=45 cs=1 l=1).handle_message read tag failed 2024-09-17T15:47:29.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.561+0000 7f542effd640 1 --1- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 0x7f541c008740 :-1 s=OPENED pgs=45 cs=1 l=1).fault on lossy channel, failing 2024-09-17T15:47:29.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.561+0000 7f542e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.203:6789/0 2024-09-17T15:47:29.726 INFO:tasks.ceph.mon.c.smithi086.stderr:2024-09-17T15:47:29.725+0000 7f5b58735640 -1 mon.c@2(electing) e1 get_health_metrics reporting 2 slow ops, oldest is log(1 entries from seq 88 at 2024-09-17T15:46:59.529561+0000) 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): tick 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 monclient(hunting): continuing hunt 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 msgr2=0x7f541c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00a040 0x7f541c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:29.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 msgr2=0x7f541c013af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.86:3300/0 conn(0x7f541c013640 0x7f541c013af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.203:6789/0 conn(0x7f541c013220 legacy=0x7f541c008740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient: _add_conns ranks=[0,4] 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f541c00d4d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f541c00e080 addr v2:172.21.15.203:3300/0 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c014100 con 0x7f541c00d4d0 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 1 -- 172.21.15.86:0/892359529 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f543009ed00 con 0x7f541c00e080 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542d7fa640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c00d4d0 legacy=0x7f541c00d8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:29.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00e080 auth_method 0 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 msgr2=0x7f541c011540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 msgr2=0x7f541c011540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:47:29.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:29.853+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542f7fe640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c00d4d0 legacy=0x7f541c00d8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> v2:172.21.15.203:3300/0 conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f541c00e080 auth_method 0 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient(hunting): handle_auth_done global_id 224146 payload 274 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> v1:172.21.15.86:6789/0 conn(0x7f541c00d4d0 legacy=0x7f541c00d8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 1 monclient: found mon.noname-c 2024-09-17T15:47:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:47:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.057+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c012bc0 con 0x7f541c00e080 2024-09-17T15:47:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.065+0000 7f542effd640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f541c00e080 msgr2=0x7f541c011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.065+0000 7f542effd640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f541c00e080 0x7f541c011540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5420003ec0 tx=0x7f542001de40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1eb0ae894b1dbc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 1 -- 172.21.15.86:0/892359529 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200ac070 con 0x7f541c00e080 2024-09-17T15:47:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: dump: 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.080 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 5 monclient: authenticate success, global_id 224146 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:47:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 1 -- 172.21.15.86:0/892359529 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f542000c040 con 0x7f541c00e080 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 1 -- 172.21.15.86:0/892359529 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54200b0640 con 0x7f541c00e080 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542e7fc640 10 monclient: dump: 2024-09-17T15:47:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.084 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.084 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 10 monclient: get_monmap_and_config success 2024-09-17T15:47:30.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 4 set_mon_vals no callback set 2024-09-17T15:47:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 10 monclient: shutdown 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 -- 172.21.15.86:0/892359529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f541c00e080 msgr2=0x7f541c011540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f541c00e080 0x7f541c011540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5420003ec0 tx=0x7f542001de40 comp rx=0 tx=0).stop 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f542ffff640 1 -- 172.21.15.86:0/892359529 reap_dead start 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 -- 172.21.15.86:0/892359529 shutdown_connections 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 --2- 172.21.15.86:0/892359529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f541c00e080 0x7f541c011540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 -- 172.21.15.86:0/892359529 >> 172.21.15.86:0/892359529 conn(0x7f54300a6440 msgr2=0x7f54300a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 -- 172.21.15.86:0/892359529 shutdown_connections 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.081+0000 7f5435e7d640 1 -- 172.21.15.86:0/892359529 wait complete. 2024-09-17T15:47:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient: build_initial_monmap 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient: monmap: 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:47:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 Processor -- start 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 -- start start 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient: init 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-c con 0x7f541c00e080 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 0x7f54301330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-b con 0x7f5430132c80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:47:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 0x7f543011cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): picked mon.noname-a con 0x7f543011c8c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f54301351e0 con 0x7f543011c8c0 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5430135060 con 0x7f5430132c80 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5430135360 con 0x7f541c00e080 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f5435e7d640 10 monclient(hunting): authenticate will time out at 2183.054427s 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542ffff640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 msgr2=0x7f543011cce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 0x7f543011cce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40670/0 (socket says 172.21.15.86:40670) 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 -- 172.21.15.86:0/289085321 learned_addr learned my addr 172.21.15.86:0/289085321 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): get_auth_request con 0x7f541c00e080 auth_method 0 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 0x7f54301330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 10 monclient(hunting): get_auth_request con 0x7f5430132c80 auth_method 0 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient(hunting): handle_auth_done global_id 164765 payload 824 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 -- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 msgr2=0x7f54301330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 0x7f54301330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 -- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 msgr2=0x7f543011cce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 0x7f543011cce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542f7fe640 1 -- 172.21.15.86:0/289085321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5430133620 con 0x7f541c00e080 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.085+0000 7f542effd640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 0x7f54301330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f542f7fe640 1 -- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 msgr2=0x7f54301cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f542f7fe640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5424003910 tx=0x7f5424003940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424018040 con 0x7f541c00e080 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:47:30.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: dump: 2024-09-17T15:47:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f542400ed30 con 0x7f541c00e080 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542401e660 con 0x7f541c00e080 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 10 monclient: dump: 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.102 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 5 monclient: authenticate success, global_id 164765 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _renew_subs 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543009d460 con 0x7f541c00e080 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f542dffb640 4 set_mon_vals no callback set 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _renew_subs 2024-09-17T15:47:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5430120640 con 0x7f541c00e080 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f542402e020 con 0x7f541c00e080 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543009efe0 con 0x7f541c00e080 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 0x7f541c040320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f54240535c0 con 0x7f541c00e080 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f542effd640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 0x7f541c040320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.089+0000 7f542effd640 10 monclient: get_auth_request con 0x7f541c03de60 auth_method 0 2024-09-17T15:47:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.093+0000 7f542effd640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 0x7f541c040320 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fa0 tx=0x7f54200a8070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.097+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5424051fa0 con 0x7f541c00e080 2024-09-17T15:47:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.097+0000 7f5417fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.097+0000 7f5417fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:47:30.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5435e7d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:47:30.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5435e7d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:47:30.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5435e7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f54301cbae0 con 0x7f541c00e080 2024-09-17T15:47:30.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5417fff640 1 -- 172.21.15.86:0/289085321 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5424027780 con 0x7f541c00e080 2024-09-17T15:47:30.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5417fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:47:30.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.341+0000 7f5417fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:47:30.342 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":186,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:47:30.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 msgr2=0x7f541c040320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 0x7f541c040320 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f541c009fa0 tx=0x7f54200a8070 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 10 monclient: shutdown 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 msgr2=0x7f54301cb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5424003910 tx=0x7f5424003940 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 shutdown_connections 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f541c03de60 0x7f541c040320 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f543011c8c0 0x7f543011cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5430132c80 0x7f54301330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 --2- 172.21.15.86:0/289085321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f541c00e080 0x7f54301cb5a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 >> 172.21.15.86:0/289085321 conn(0x7f54300a6440 msgr2=0x7f54300a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 shutdown_connections 2024-09-17T15:47:30.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.345+0000 7f5435e7d640 1 -- 172.21.15.86:0/289085321 wait complete. 2024-09-17T15:47:30.361 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:47:30.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:47:30.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:47:30.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: get_monmap_and_config 2024-09-17T15:47:30.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: build_initial_monmap 2024-09-17T15:47:30.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: monmap: 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:47:30.489125+0000 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:47:30.489125+0000 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:30.485 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:47:30.486 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 Processor -- start 2024-09-17T15:47:30.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 -- start start 2024-09-17T15:47:30.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: init 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 0x7f59b00a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-a con 0x7f59b00a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59b00a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f59b0136160 0x7f59b013df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-c con 0x7f59b0136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b00a2130 con 0x7f59b00a1590 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f59b00a2410 con 0x7f59b00a0c70 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f59b00a26f0 con 0x7f59b0136160 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59b53e6640 10 monclient(hunting): authenticate will time out at 2183.454610s 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59aeffd640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 msgr2=0x7f59b00a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59aeffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 0x7f59b00a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f59b0136160 0x7f59b013df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f59b0136160 0x7f59b013df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.86:47682/0 (socket says 172.21.15.86:47682) 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 1 -- 172.21.15.86:0/217021885 learned_addr learned my addr 172.21.15.86:0/217021885 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): get_auth_request con 0x7f59b0136160 auth_method 0 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.485+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59ae7fc640 1 -- 172.21.15.86:0/217021885 >> v1:172.21.15.203:6789/0 conn(0x7f59b00a1590 legacy=0x7f59b0135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 10 monclient(hunting): handle_auth_done global_id 224152 payload 274 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 1 -- 172.21.15.86:0/217021885 >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 msgr2=0x7f59b00a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 1 --2- 172.21.15.86:0/217021885 >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 0x7f59b00a1050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 1 -- 172.21.15.86:0/217021885 >> v1:172.21.15.203:6789/0 conn(0x7f59b00a1590 legacy=0x7f59b0135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 1 monclient: found mon.noname-c 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.489+0000 7f59af7fe640 1 -- 172.21.15.86:0/217021885 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b009f9f0 con 0x7f59b0136160 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59af7fe640 1 -- 172.21.15.86:0/217021885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b0136160 msgr2=0x7f59b013df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59af7fe640 1 --2- 172.21.15.86:0/217021885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b0136160 0x7f59b013df30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59b006f840 tx=0x7f59a801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65f112b2d4426fe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 1 -- 172.21.15.86:0/217021885 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a80aa6e0 con 0x7f59b0136160 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: dump: 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.503 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 1 -- 172.21.15.86:0/217021885 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a80af070 con 0x7f59b0136160 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 5 monclient: authenticate success, global_id 224152 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 10 monclient: get_monmap_and_config success 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 1 -- 172.21.15.86:0/217021885 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a80aaaa0 con 0x7f59b0136160 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 4 set_mon_vals no callback set 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59adffb640 10 monclient: dump: 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.505 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 10 monclient: shutdown 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 -- 172.21.15.86:0/217021885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b0136160 msgr2=0x7f59b013df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 --2- 172.21.15.86:0/217021885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b0136160 0x7f59b013df30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59b006f840 tx=0x7f59a801c480 comp rx=0 tx=0).stop 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 -- 172.21.15.86:0/217021885 shutdown_connections 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 --2- 172.21.15.86:0/217021885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b0136160 0x7f59b013df30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 --2- 172.21.15.86:0/217021885 >> v2:172.21.15.86:3300/0 conn(0x7f59b00a0c70 0x7f59b00a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 -- 172.21.15.86:0/217021885 >> 172.21.15.86:0/217021885 conn(0x7f59b00a2f40 msgr2=0x7f59b00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 -- 172.21.15.86:0/217021885 shutdown_connections 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.493+0000 7f59b53e6640 1 -- 172.21.15.86:0/217021885 wait complete. 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient: build_initial_monmap 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient: monmap: 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:30.507 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 Processor -- start 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 -- start start 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient: init 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:30.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59b00a0c70 0x7f59b012ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-a con 0x7f59b00a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b00a1590 0x7f59b012f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-b con 0x7f59b00a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 0x7f59b0136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): picked mon.noname-c con 0x7f59b0136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f59b00a00d0 con 0x7f59b00a0c70 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59b009ff50 con 0x7f59b00a1590 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59b0140c90 con 0x7f59b0136160 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59b53e6640 10 monclient(hunting): authenticate will time out at 2183.465298s 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59aeffd640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59b00a0c70 msgr2=0x7f59b012ed40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59aeffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59b00a0c70 0x7f59b012ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 0x7f59b0136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 0x7f59b0136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40678/0 (socket says 172.21.15.86:40678) 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 learned_addr learned my addr 172.21.15.86:0/3255433303 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): get_auth_request con 0x7f59b0136160 auth_method 0 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59ae7fc640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b00a1590 0x7f59b012f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59ae7fc640 10 monclient(hunting): get_auth_request con 0x7f59b00a1590 auth_method 0 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient(hunting): handle_auth_done global_id 164768 payload 824 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b00a1590 msgr2=0x7f59b012f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59b00a1590 0x7f59b012f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59b00a0c70 msgr2=0x7f59b012ed40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59b00a0c70 0x7f59b012ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 monclient: found mon.noname-c 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.497+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b013a520 con 0x7f59b0136160 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 msgr2=0x7f59b0136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59af7fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 0x7f59b0136ef0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f59a80ac040 tx=0x7f59a801acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:47:30.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a80bd5f0 con 0x7f59b0136160 2024-09-17T15:47:30.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:47:30.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: dump: 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.518 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a80bd790 con 0x7f59b0136160 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a80bdb80 con 0x7f59b0136160 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 5 monclient: authenticate success, global_id 164768 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 monclient: _renew_subs 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b0138780 con 0x7f59b0136160 2024-09-17T15:47:30.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59ad7fa640 4 set_mon_vals no callback set 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 10 monclient: dump: 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 monclient: _renew_subs 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b013b4e0 con 0x7f59b0136160 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 0x7f59b0137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59b013a740 con 0x7f59b01392e0 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 0x7f59b0137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient: get_auth_request con 0x7f59b01392e0 auth_method 0 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f59a80a9070 con 0x7f59b0136160 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 10 monclient(hunting): handle_auth_done global_id 164771 payload 824 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59aeffd640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59b009d3e0 con 0x7f59b01392e0 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f599003bb60 0x7f599003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (secure 0 0 0) 0x7f59a80ea8f0 con 0x7f59b0136160 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.501+0000 7f59ae7fc640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f599003bb60 0x7f599003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59ae7fc640 10 monclient: get_auth_request con 0x7f599003bb60 auth_method 0 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59aeffd640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 msgr2=0x7f59b0137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59aeffd640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 0x7f59b0137430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f59a000bcc0 tx=0x7f59a0009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0005560 con 0x7f59b01392e0 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 10 monclient: dump: 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.525 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59a0005560 con 0x7f59b01392e0 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 msgr2=0x7f59b0137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 0x7f59b0137430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f59a000bcc0 tx=0x7f59a0009f30 comp rx=0 tx=0).stop 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 msgr2=0x7f59b0137430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59a77fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b01392e0 0x7f59b0137430 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f59a000bcc0 tx=0x7f59a0009f30 comp rx=0 tx=0).stop 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.505+0000 7f59ae7fc640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f599003bb60 0x7f599003e020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f59b00702c0 tx=0x7f5998009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 0x7f59b0137d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59b01382c0 con 0x7f59b0137970 2024-09-17T15:47:30.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 0x7f59b0137d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient: get_auth_request con 0x7f59b0137970 auth_method 0 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 10 monclient(hunting): handle_auth_done global_id 164774 payload 824 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 1 -- 172.21.15.86:0/3255433303 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59b006d4f0 con 0x7f59b0137970 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 msgr2=0x7f59b0137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.527 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59aeffd640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 0x7f59b0137d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59a0014000 tx=0x7f59a0005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0019040 con 0x7f59b0137970 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 10 monclient: dump: 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f59a001ac30 con 0x7f59b0137970 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 msgr2=0x7f59b0137d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 0x7f59b0137d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59a0014000 tx=0x7f59a0005e20 comp rx=0 tx=0).stop 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 msgr2=0x7f59b0137d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:30.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59a77fe640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0137970 0x7f59b0137d80 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f59a0014000 tx=0x7f59a0005e20 comp rx=0 tx=0).stop 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 186, 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 353701, 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:47:30.530 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.531 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.532 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:47:30.533 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.534 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:47:30.535 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.536 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.537 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:47:30.538 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.540 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.541 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:47:30.542 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f599003bb60 msgr2=0x7f599003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f599003bb60 0x7f599003e020 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f59b00702c0 tx=0x7f5998009040 comp rx=0 tx=0).stop 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 10 monclient: shutdown 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 msgr2=0x7f59b0136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 --2- 172.21.15.86:0/3255433303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59b0136160 0x7f59b0136ef0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f59a80ac040 tx=0x7f59a801acc0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 reap_dead start 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59af7fe640 1 -- 172.21.15.86:0/3255433303 reap_dead start 2024-09-17T15:47:30.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 shutdown_connections 2024-09-17T15:47:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 >> 172.21.15.86:0/3255433303 conn(0x7f59b00a2f40 msgr2=0x7f59b00a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 shutdown_connections 2024-09-17T15:47:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.513+0000 7f59b53e6640 1 -- 172.21.15.86:0/3255433303 wait complete. 2024-09-17T15:47:30.545 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:47:30.545 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:47:30.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:47:30.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: get_monmap_and_config 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: build_initial_monmap 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: monmap: 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:47:30.672460+0000 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:47:30.672460+0000 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:47:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:47:30.668 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:47:30.668 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 Processor -- start 2024-09-17T15:47:30.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 -- start start 2024-09-17T15:47:30.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: init 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c000a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c000a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 0x7f1c0013df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c00136160 addr v2:172.21.15.86:3300/0 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c000a2130 con 0x7f1c000a1590 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1c000a2410 con 0x7f1c00136160 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c000a26f0 con 0x7f1c000a0c70 2024-09-17T15:47:30.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c0809c640 10 monclient(hunting): authenticate will time out at 2183.637990s 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c06639640 1 -- >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 msgr2=0x7f1c0013df30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05637640 1 -- >> v1:172.21.15.86:6789/0 conn(0x7f1c000a1590 legacy=0x7f1c00135a50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.86:6789/0 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c06639640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 0x7f1c0013df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40708/0 (socket says 172.21.15.86:40708) 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 -- 172.21.15.86:0/1464878162 learned_addr learned my addr 172.21.15.86:0/1464878162 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): get_auth_request con 0x7f1c000a0c70 auth_method 0 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient(hunting): handle_auth_done global_id 164777 payload 274 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 -- 172.21.15.86:0/1464878162 >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 msgr2=0x7f1c0013df30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 --2- 172.21.15.86:0/1464878162 >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 0x7f1c0013df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 -- 172.21.15.86:0/1464878162 >> v1:172.21.15.86:6789/0 conn(0x7f1c000a1590 legacy=0x7f1c00135a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 monclient: found mon.noname-b 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.669+0000 7f1c05e38640 1 -- 172.21.15.86:0/1464878162 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0009f9f0 con 0x7f1c000a0c70 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c05e38640 1 -- 172.21.15.86:0/1464878162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 msgr2=0x7f1c000a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c05e38640 1 --2- 172.21.15.86:0/1464878162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009870 tx=0x7f1bf401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccb93f4978f24a3b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 1 -- 172.21.15.86:0/1464878162 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b1070 con 0x7f1c000a0c70 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: dump: 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 1 -- 172.21.15.86:0/1464878162 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf40a8050 con 0x7f1c000a0c70 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 1 -- 172.21.15.86:0/1464878162 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40ac860 con 0x7f1c000a0c70 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c04e36640 10 monclient: dump: 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c0809c640 5 monclient: authenticate success, global_id 164777 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c0809c640 10 monclient: get_monmap_and_config success 2024-09-17T15:47:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.673+0000 7f1c0809c640 4 set_mon_vals no callback set 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: shutdown 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- 172.21.15.86:0/1464878162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 msgr2=0x7f1c000a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- 172.21.15.86:0/1464878162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009870 tx=0x7f1bf401c440 comp rx=0 tx=0).stop 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- 172.21.15.86:0/1464878162 shutdown_connections 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- 172.21.15.86:0/1464878162 >> v2:172.21.15.86:3300/0 conn(0x7f1c00136160 0x7f1c0013df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- 172.21.15.86:0/1464878162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c000a0c70 0x7f1c000a1050 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- 172.21.15.86:0/1464878162 >> 172.21.15.86:0/1464878162 conn(0x7f1c000a2f40 msgr2=0x7f1c000a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- 172.21.15.86:0/1464878162 shutdown_connections 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- 172.21.15.86:0/1464878162 wait complete. 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: build_initial_monmap 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: monmap: 2024-09-17T15:47:30.683 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 Processor -- start 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- start start 2024-09-17T15:47:30.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: init 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: _reopen_session rank -1 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c000a0c70 0x7f1c001cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-a con 0x7f1c000a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c000a1590 0x7f1c00135510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-b con 0x7f1c000a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): picked mon.noname-c con 0x7f1c00136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1c000a00d0 con 0x7f1c000a0c70 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0009ff50 con 0x7f1c000a1590 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0009d770 con 0x7f1c00136160 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): _renew_subs 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05e38640 1 -- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c000a0c70 msgr2=0x7f1c001cbdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3300/0 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c0809c640 10 monclient(hunting): authenticate will time out at 2183.646132s 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05e38640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c000a0c70 0x7f1c001cbdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40710/0 (socket says 172.21.15.86:40710) 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 learned_addr learned my addr 172.21.15.86:0/2283349443 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): get_auth_request con 0x7f1c00136160 auth_method 0 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05637640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c000a1590 0x7f1c00135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05637640 10 monclient(hunting): get_auth_request con 0x7f1c000a1590 auth_method 0 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05637640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05637640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c05637640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient(hunting): handle_auth_done global_id 164780 payload 824 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c000a1590 msgr2=0x7f1c00135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c000a1590 0x7f1c00135510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c000a0c70 msgr2=0x7f1c001cbdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c000a0c70 0x7f1c001cbdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient: _finish_hunting 0 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 monclient: found mon.noname-c 2024-09-17T15:47:30.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.677+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0009d460 con 0x7f1c00136160 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 msgr2=0x7f1c0013b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c06639640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b410 tx=0x7f1bf800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c06639640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b410 tx=0x7f1bf800b8d0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f1bf80064c0 mon_map magic: 0 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8017070 con 0x7f1c00136160 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: dump: 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.688 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: _finish_auth 0 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 5 monclient: authenticate success, global_id 164780 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf80060d0 con 0x7f1c00136160 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf3fff640 4 set_mon_vals no callback set 2024-09-17T15:47:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 monclient: _renew_subs 2024-09-17T15:47:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0011de80 con 0x7f1c00136160 2024-09-17T15:47:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 monclient: _renew_subs 2024-09-17T15:47:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:47:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0011d5e0 con 0x7f1c00136160 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 0x7f1c0013bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c000a1050 con 0x7f1c00135a50 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80064c0 con 0x7f1c00136160 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 10 monclient: dump: 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1bf801a070 con 0x7f1c00136160 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 0x7f1c0013bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient: get_auth_request con 0x7f1c00135a50 auth_method 0 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be403bb40 0x7f1be403e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 10 monclient(hunting): handle_auth_done global_id 224161 payload 824 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05e38640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c0013dcc0 con 0x7f1c00135a50 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05637640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be403bb40 0x7f1be403e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.681+0000 7f1c05637640 10 monclient: get_auth_request con 0x7f1be403bb40 auth_method 0 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1c05e38640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 msgr2=0x7f1c0013bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1c05e38640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 0x7f1c0013bbe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40ae000 tx=0x7f1bf401ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b1040 con 0x7f1c00135a50 2024-09-17T15:47:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 10 monclient: dump: 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1c05637640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be403bb40 0x7f1be403e000 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c0011d510 tx=0x7f1bfc0094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1bf40a8040 con 0x7f1c00135a50 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 msgr2=0x7f1c0013bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 0x7f1c0013bbe0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40ae000 tx=0x7f1bf401ac80 comp rx=0 tx=0).stop 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 msgr2=0x7f1c0013bbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.685+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c00135a50 0x7f1c0013bbe0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c0809c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c0809c640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 0x7f1c00136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c0809c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0013c500 con 0x7f1c0013c120 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 0x7f1c00136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:47:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient: get_auth_request con 0x7f1c0013c120 auth_method 0 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 10 monclient(hunting): handle_auth_done global_id 224164 payload 824 2024-09-17T15:47:30.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.693+0000 7f1c05e38640 1 -- 172.21.15.86:0/2283349443 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c0011da20 con 0x7f1c0013c120 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c05e38640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 msgr2=0x7f1c00136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c05e38640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 0x7f1c00136ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401c270 tx=0x7f1bf40191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf40b1040 con 0x7f1c0013c120 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 10 monclient: dump: 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:47:30.699 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f1bf4003620 con 0x7f1c0013c120 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 msgr2=0x7f1c00136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 0x7f1c00136ef0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401c270 tx=0x7f1bf40191e0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 msgr2=0x7f1c00136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:47:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0013c120 0x7f1c00136ef0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1bf401c270 tx=0x7f1bf40191e0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 186, 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 41146, 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:47:30.701 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.702 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.703 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:47:30.704 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:47:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.706 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.707 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.708 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.712 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:47:30.713 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:47:30.714 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.715 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:47:30.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:47:30.717 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:47:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be403bb40 msgr2=0x7f1be403e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1be403bb40 0x7f1be403e000 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c0011d510 tx=0x7f1bfc0094d0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 10 monclient: shutdown 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 msgr2=0x7f1c0013b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c06639640 1 -- 172.21.15.86:0/2283349443 reap_dead start 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b410 tx=0x7f1bf800b8d0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1bf27fc640 1 -- 172.21.15.86:0/2283349443 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(712..712 src has 438..712) ==== 4811+0+0 (unknown 0 0 0) 0x7f1bf804f2c0 con 0x7f1c00136160 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 shutdown_connections 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 --2- 172.21.15.86:0/2283349443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c00136160 0x7f1c0013b6a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 >> 172.21.15.86:0/2283349443 conn(0x7f1c000a2f40 msgr2=0x7f1c00072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 shutdown_connections 2024-09-17T15:47:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:47:30.697+0000 7f1c0809c640 1 -- 172.21.15.86:0/2283349443 wait complete. 2024-09-17T15:47:30.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:47:30.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3229: done (2 left) 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3230: done (1 left) 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3232: done (0 left) 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3233: copy_from oid 9 from oid 7 current snap is 348 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3234: rollback oid 19 current snap is 348 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 341 2024-09-17T15:47:31.086 INFO:tasks.rados.rados.0.smithi203.stdout:3235: snap_create 2024-09-17T15:47:31.110 INFO:tasks.rados.rados.0.smithi203.stdout:3234: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:47:31.110 INFO:tasks.rados.rados.0.smithi203.stdout:3234: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:47:31.110 INFO:tasks.rados.rados.0.smithi203.stdout:3234: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:47:31.110 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1629 (ObjNum 1066 snap 330 seq_num 1066) dirty exists 2024-09-17T15:47:31.121 INFO:tasks.rados.rados.0.smithi203.stdout:3233: finishing copy_from racing read to smithi20328086-9 2024-09-17T15:47:31.125 INFO:tasks.rados.rados.0.smithi203.stdout:3233: finishing copy_from to smithi20328086-9 2024-09-17T15:47:31.126 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1198 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:47:32.097 INFO:tasks.rados.rados.0.smithi203.stdout:3233: done (2 left) 2024-09-17T15:47:32.097 INFO:tasks.rados.rados.0.smithi203.stdout:3234: done (1 left) 2024-09-17T15:47:32.097 INFO:tasks.rados.rados.0.smithi203.stdout:3235: done (0 left) 2024-09-17T15:47:32.097 INFO:tasks.rados.rados.0.smithi203.stdout:3236: delete oid 14 current snap is 349 2024-09-17T15:47:32.107 INFO:tasks.rados.rados.0.smithi203.stdout:3236: done (0 left) 2024-09-17T15:47:32.107 INFO:tasks.rados.rados.0.smithi203.stdout:3237: snap_create 2024-09-17T15:47:33.095 INFO:tasks.rados.rados.0.smithi203.stdout:3237: done (0 left) 2024-09-17T15:47:33.095 INFO:tasks.rados.rados.0.smithi203.stdout:3238: rollback oid 32 current snap is 350 2024-09-17T15:47:33.095 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 338 2024-09-17T15:47:33.095 INFO:tasks.rados.rados.0.smithi203.stdout:3239: rollback oid 45 current snap is 350 2024-09-17T15:47:33.096 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 340 2024-09-17T15:47:33.096 INFO:tasks.rados.rados.0.smithi203.stdout:3240: snap_create 2024-09-17T15:47:33.099 INFO:tasks.rados.rados.0.smithi203.stdout:3239: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:47:33.099 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:47:33.110 INFO:tasks.rados.rados.0.smithi203.stdout:3238: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:47:33.113 INFO:tasks.rados.rados.0.smithi203.stdout:3238: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:47:33.113 INFO:tasks.rados.rados.0.smithi203.stdout:3238: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:47:33.113 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1350 (ObjNum 1068 snap 330 seq_num 1068) dirty exists 2024-09-17T15:47:34.097 INFO:tasks.rados.rados.0.smithi203.stdout:3238: done (2 left) 2024-09-17T15:47:34.097 INFO:tasks.rados.rados.0.smithi203.stdout:3239: done (1 left) 2024-09-17T15:47:34.097 INFO:tasks.rados.rados.0.smithi203.stdout:3240: done (0 left) 2024-09-17T15:47:34.098 INFO:tasks.rados.rados.0.smithi203.stdout:3241: snap_create 2024-09-17T15:47:34.567 INFO:tasks.rados.rados.0.smithi203.stdout:3241: done (0 left) 2024-09-17T15:47:34.567 INFO:tasks.rados.rados.0.smithi203.stdout:3242: write oid 13 current snap is 352 2024-09-17T15:47:34.567 INFO:tasks.rados.rados.0.smithi203.stdout:3242: seq_num 1132 ranges {625524=402112,1592087=608063,2869769=155755} 2024-09-17T15:47:34.584 INFO:tasks.rados.rados.0.smithi203.stdout:3242: writing smithi20328086-13 from 625524 to 1027636 tid 1 2024-09-17T15:47:34.603 INFO:tasks.rados.rados.0.smithi203.stdout:3242: writing smithi20328086-13 from 1592087 to 2200150 tid 2 2024-09-17T15:47:34.617 INFO:tasks.rados.rados.0.smithi203.stdout:3242: writing smithi20328086-13 from 2869769 to 3025524 tid 3 2024-09-17T15:47:34.617 INFO:tasks.rados.rados.0.smithi203.stdout:3243: rollback oid 6 current snap is 352 2024-09-17T15:47:34.617 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 338 2024-09-17T15:47:34.618 INFO:tasks.rados.rados.0.smithi203.stdout:3242: finishing write tid 1 to smithi20328086-13 2024-09-17T15:47:34.618 INFO:tasks.rados.rados.0.smithi203.stdout:3242: finishing write tid 2 to smithi20328086-13 2024-09-17T15:47:34.618 INFO:tasks.rados.rados.0.smithi203.stdout:3244: snap_create 2024-09-17T15:47:34.628 INFO:tasks.rados.rados.0.smithi203.stdout:3243: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:47:34.629 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1547 (ObjNum 33 snap 0 seq_num 3586005048) dirty dne 2024-09-17T15:47:34.630 INFO:tasks.rados.rados.0.smithi203.stdout:3242: finishing write tid 3 to smithi20328086-13 2024-09-17T15:47:34.630 INFO:tasks.rados.rados.0.smithi203.stdout:3242: finishing write tid 4 to smithi20328086-13 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: finishing write tid 5 to smithi20328086-13 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: oid 13 updating version 0 to 1380 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: oid 13 updating version 1380 to 1381 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: oid 13 version 1381 is already newer than 1379 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: oid 13 updating version 1381 to 1382 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1382 (ObjNum 1132 snap 352 seq_num 1132) dirty exists 2024-09-17T15:47:34.631 INFO:tasks.rados.rados.0.smithi203.stdout:3242: left oid 13 (ObjNum 1132 snap 352 seq_num 1132) 2024-09-17T15:47:35.575 INFO:tasks.rados.rados.0.smithi203.stdout:3242: done (2 left) 2024-09-17T15:47:35.575 INFO:tasks.rados.rados.0.smithi203.stdout:3243: done (1 left) 2024-09-17T15:47:35.575 INFO:tasks.rados.rados.0.smithi203.stdout:3244: done (0 left) 2024-09-17T15:47:35.576 INFO:tasks.rados.rados.0.smithi203.stdout:3245: delete oid 39 current snap is 353 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3245: done (0 left) 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3246: read oid 22 snap -1 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3246: expect (ObjNum 1114 snap 343 seq_num 1114) 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3247: rollback oid 37 current snap is 353 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 340 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3248: read oid 30 snap -1 2024-09-17T15:47:35.585 INFO:tasks.rados.rados.0.smithi203.stdout:3248: expect (ObjNum 1123 snap 346 seq_num 1123) 2024-09-17T15:47:35.586 INFO:tasks.rados.rados.0.smithi203.stdout:3249: write (excl) oid 2 current snap is 353 2024-09-17T15:47:35.586 INFO:tasks.rados.rados.0.smithi203.stdout:3249: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-17T15:47:35.609 INFO:tasks.rados.rados.0.smithi203.stdout:3249: writing smithi20328086-2 from 659138 to 1328101 tid 1 2024-09-17T15:47:35.631 INFO:tasks.rados.rados.0.smithi203.stdout:3249: writing smithi20328086-2 from 2109051 to 2720783 tid 2 2024-09-17T15:47:35.631 INFO:tasks.rados.rados.0.smithi203.stdout:3250: snap_remove snap 347 2024-09-17T15:47:36.586 INFO:tasks.rados.rados.0.smithi203.stdout:3250: done (4 left) 2024-09-17T15:47:36.586 INFO:tasks.rados.rados.0.smithi203.stdout:3251: write (excl) oid 36 current snap is 353 2024-09-17T15:47:36.586 INFO:tasks.rados.rados.0.smithi203.stdout:3251: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-09-17T15:47:36.607 INFO:tasks.rados.rados.0.smithi203.stdout:3251: writing smithi20328086-36 from 675945 to 1420157 tid 1 2024-09-17T15:47:36.632 INFO:tasks.rados.rados.0.smithi203.stdout:3251: writing smithi20328086-36 from 2167533 to 2922923 tid 2 2024-09-17T15:47:36.632 INFO:tasks.rados.rados.0.smithi203.stdout:3252: delete oid 42 current snap is 353 2024-09-17T15:47:36.632 INFO:tasks.rados.rados.0.smithi203.stdout:3246: expect (ObjNum 1114 snap 343 seq_num 1114) 2024-09-17T15:47:36.680 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1481143 2024-09-17T15:47:36.761 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1564506 2024-09-17T15:47:36.863 INFO:tasks.rados.rados.0.smithi203.stdout:3248: expect (ObjNum 1123 snap 346 seq_num 1123) 2024-09-17T15:47:36.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:3247: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:3247: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:3247: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1634 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:3249: finishing write tid 1 to smithi20328086-2 2024-09-17T15:47:36.949 INFO:tasks.rados.rados.0.smithi203.stdout:3249: finishing write tid 2 to smithi20328086-2 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: finishing write tid 3 to smithi20328086-2 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: finishing write tid 4 to smithi20328086-2 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: oid 2 updating version 0 to 1199 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: oid 2 updating version 1199 to 1201 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: oid 2 version 1201 is already newer than 1200 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1201 (ObjNum 1134 snap 353 seq_num 1134) dirty exists 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3249: left oid 2 (ObjNum 1134 snap 353 seq_num 1134) 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: finishing write tid 1 to smithi20328086-36 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: finishing write tid 2 to smithi20328086-36 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: finishing write tid 3 to smithi20328086-36 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: finishing write tid 4 to smithi20328086-36 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: oid 36 updating version 0 to 1684 2024-09-17T15:47:36.950 INFO:tasks.rados.rados.0.smithi203.stdout:3251: oid 36 version 1684 is already newer than 1682 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3251: oid 36 version 1684 is already newer than 1683 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1684 (ObjNum 1135 snap 353 seq_num 1135) dirty exists 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3251: left oid 36 (ObjNum 1135 snap 353 seq_num 1135) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3246: done (5 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3247: done (4 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3248: done (3 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3249: done (2 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3251: done (1 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3252: done (0 left) 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3253: read oid 47 snap -1 2024-09-17T15:47:36.951 INFO:tasks.rados.rados.0.smithi203.stdout:3253: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:47:36.952 INFO:tasks.rados.rados.0.smithi203.stdout:3254: write (excl) oid 22 current snap is 353 2024-09-17T15:47:36.952 INFO:tasks.rados.rados.0.smithi203.stdout:3254: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-09-17T15:47:36.968 INFO:tasks.rados.rados.0.smithi203.stdout:3254: writing smithi20328086-22 from 709559 to 1204269 tid 1 2024-09-17T15:47:36.992 INFO:tasks.rados.rados.0.smithi203.stdout:3254: writing smithi20328086-22 from 1884497 to 2643556 tid 2 2024-09-17T15:47:36.992 INFO:tasks.rados.rados.0.smithi203.stdout:3255: write (excl) oid 41 current snap is 353 2024-09-17T15:47:36.992 INFO:tasks.rados.rados.0.smithi203.stdout:3255: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-09-17T15:47:37.010 INFO:tasks.rados.rados.0.smithi203.stdout:3255: writing smithi20328086-41 from 726366 to 1296325 tid 1 2024-09-17T15:47:37.021 INFO:tasks.rados.rados.0.smithi203.stdout:3255: writing smithi20328086-41 from 1826626 to 2329343 tid 2 2024-09-17T15:47:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:3255: writing smithi20328086-41 from 2801601 to 3126366 tid 3 2024-09-17T15:47:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:3256: read oid 13 snap 313 2024-09-17T15:47:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:3256: expect (ObjNum 969 snap 294 seq_num 969) 2024-09-17T15:47:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:3257: write (excl) oid 45 current snap is 353 2024-09-17T15:47:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:3257: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-17T15:47:37.040 INFO:tasks.rados.rados.0.smithi203.stdout:3257: writing smithi20328086-45 from 743173 to 1388381 tid 1 2024-09-17T15:47:37.049 INFO:tasks.rados.rados.0.smithi203.stdout:3257: writing smithi20328086-45 from 1885108 to 2647836 tid 2 2024-09-17T15:47:37.050 INFO:tasks.rados.rados.0.smithi203.stdout:3258: rollback oid 11 current snap is 353 2024-09-17T15:47:37.050 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 345 2024-09-17T15:47:37.050 INFO:tasks.rados.rados.0.smithi203.stdout:3253: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:47:37.187 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1259457 2024-09-17T15:47:37.213 INFO:tasks.rados.rados.0.smithi203.stdout:3254: finishing write tid 1 to smithi20328086-22 2024-09-17T15:47:37.213 INFO:tasks.rados.rados.0.smithi203.stdout:3253: done (5 left) 2024-09-17T15:47:37.213 INFO:tasks.rados.rados.0.smithi203.stdout:3259: read oid 39 snap -1 2024-09-17T15:47:37.213 INFO:tasks.rados.rados.0.smithi203.stdout:3259: expect deleted 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3260: read oid 35 snap -1 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3260: expect deleted 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3261: copy_from oid 12 from oid 36 current snap is 353 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: finishing write tid 2 to smithi20328086-22 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: finishing write tid 3 to smithi20328086-22 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: finishing write tid 4 to smithi20328086-22 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: oid 22 updating version 0 to 1318 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: oid 22 version 1318 is already newer than 1317 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:3254: oid 22 version 1318 is already newer than 1316 2024-09-17T15:47:37.214 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1318 (ObjNum 1137 snap 353 seq_num 1137) dirty exists 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3254: left oid 22 (ObjNum 1137 snap 353 seq_num 1137) 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: finishing write tid 1 to smithi20328086-41 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: finishing write tid 2 to smithi20328086-41 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: finishing write tid 3 to smithi20328086-41 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: finishing write tid 4 to smithi20328086-41 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: finishing write tid 5 to smithi20328086-41 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: oid 41 updating version 0 to 674 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: oid 41 version 674 is already newer than 673 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: oid 41 updating version 674 to 675 2024-09-17T15:47:37.215 INFO:tasks.rados.rados.0.smithi203.stdout:3255: oid 41 updating version 675 to 676 2024-09-17T15:47:37.216 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 676 (ObjNum 1138 snap 353 seq_num 1138) dirty exists 2024-09-17T15:47:37.216 INFO:tasks.rados.rados.0.smithi203.stdout:3255: left oid 41 (ObjNum 1138 snap 353 seq_num 1138) 2024-09-17T15:47:37.216 INFO:tasks.rados.rados.0.smithi203.stdout:3256: expect (ObjNum 969 snap 294 seq_num 969) 2024-09-17T15:47:37.216 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1469935 2024-09-17T15:47:37.235 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 962968 2024-09-17T15:47:37.275 INFO:tasks.rados.rados.0.smithi203.stdout:3257: finishing write tid 1 to smithi20328086-45 2024-09-17T15:47:37.275 INFO:tasks.rados.rados.0.smithi203.stdout:3257: finishing write tid 2 to smithi20328086-45 2024-09-17T15:47:37.275 INFO:tasks.rados.rados.0.smithi203.stdout:3254: done (7 left) 2024-09-17T15:47:37.275 INFO:tasks.rados.rados.0.smithi203.stdout:3255: done (6 left) 2024-09-17T15:47:37.275 INFO:tasks.rados.rados.0.smithi203.stdout:3256: done (5 left) 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3262: delete oid 22 current snap is 353 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3257: finishing write tid 3 to smithi20328086-45 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3257: finishing write tid 4 to smithi20328086-45 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3257: oid 45 updating version 0 to 1325 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3257: oid 45 updating version 1325 to 1326 2024-09-17T15:47:37.276 INFO:tasks.rados.rados.0.smithi203.stdout:3257: oid 45 updating version 1326 to 1327 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1327 (ObjNum 1139 snap 353 seq_num 1139) dirty exists 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:3257: left oid 45 (ObjNum 1139 snap 353 seq_num 1139) 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:3258: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:3258: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:3258: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:47:37.277 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1524 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:47:37.278 INFO:tasks.rados.rados.0.smithi203.stdout:3261: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:47:37.278 INFO:tasks.rados.rados.0.smithi203.stdout:3261: finishing copy_from to smithi20328086-12 2024-09-17T15:47:37.278 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1684 (ObjNum 1135 snap 353 seq_num 1135) dirty exists 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3257: done (5 left) 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3258: done (4 left) 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3259: done (3 left) 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3260: done (2 left) 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3261: done (1 left) 2024-09-17T15:47:37.280 INFO:tasks.rados.rados.0.smithi203.stdout:3262: done (0 left) 2024-09-17T15:47:37.281 INFO:tasks.rados.rados.0.smithi203.stdout:3263: rollback oid 50 current snap is 353 2024-09-17T15:47:37.281 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 341 2024-09-17T15:47:37.281 INFO:tasks.rados.rados.0.smithi203.stdout:3264: write oid 4 current snap is 353 2024-09-17T15:47:37.281 INFO:tasks.rados.rados.0.smithi203.stdout:3264: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-17T15:47:37.287 INFO:tasks.rados.rados.0.smithi203.stdout:3264: writing smithi20328086-4 from 776787 to 1288846 tid 1 2024-09-17T15:47:37.294 INFO:tasks.rados.rados.0.smithi203.stdout:3264: writing smithi20328086-4 from 1718425 to 2484822 tid 2 2024-09-17T15:47:37.299 INFO:tasks.rados.rados.0.smithi203.stdout:3264: writing smithi20328086-4 from 3000223 to 3176787 tid 3 2024-09-17T15:47:37.299 INFO:tasks.rados.rados.0.smithi203.stdout:3265: read oid 22 snap -1 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3265: expect deleted 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3266: read oid 37 snap -1 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3266: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3267: snap_create 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3263: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3263: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3263: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1353 (ObjNum 1071 snap 330 seq_num 1071) dirty exists 2024-09-17T15:47:37.300 INFO:tasks.rados.rados.0.smithi203.stdout:3264: finishing write tid 1 to smithi20328086-4 2024-09-17T15:47:37.302 INFO:tasks.rados.rados.0.smithi203.stdout:3264: finishing write tid 2 to smithi20328086-4 2024-09-17T15:47:37.303 INFO:tasks.rados.rados.0.smithi203.stdout:3264: finishing write tid 3 to smithi20328086-4 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: finishing write tid 4 to smithi20328086-4 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: finishing write tid 5 to smithi20328086-4 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: oid 4 updating version 0 to 1331 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: oid 4 version 1331 is already newer than 1329 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: oid 4 version 1331 is already newer than 1330 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: oid 4 version 1331 is already newer than 1328 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1331 (ObjNum 1141 snap 353 seq_num 1141) dirty exists 2024-09-17T15:47:37.304 INFO:tasks.rados.rados.0.smithi203.stdout:3264: left oid 4 (ObjNum 1141 snap 353 seq_num 1141) 2024-09-17T15:47:37.309 INFO:tasks.rados.rados.0.smithi203.stdout:3266: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:47:37.331 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1739288 2024-09-17T15:47:37.356 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3263: done (4 left) 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3264: done (3 left) 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3265: done (2 left) 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3266: done (1 left) 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3267: done (0 left) 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3268: write (excl) oid 12 current snap is 354 2024-09-17T15:47:37.591 INFO:tasks.rados.rados.0.smithi203.stdout:3268: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-17T15:47:37.607 INFO:tasks.rados.rados.0.smithi203.stdout:3268: writing smithi20328086-12 from 793594 to 1380902 tid 1 2024-09-17T15:47:37.621 INFO:tasks.rados.rados.0.smithi203.stdout:3268: writing smithi20328086-12 from 2060554 to 2570609 tid 2 2024-09-17T15:47:37.621 INFO:tasks.rados.rados.0.smithi203.stdout:3269: delete oid 48 current snap is 354 2024-09-17T15:47:37.630 INFO:tasks.rados.rados.0.smithi203.stdout:3268: finishing write tid 1 to smithi20328086-12 2024-09-17T15:47:37.630 INFO:tasks.rados.rados.0.smithi203.stdout:3268: finishing write tid 2 to smithi20328086-12 2024-09-17T15:47:37.631 INFO:tasks.rados.rados.0.smithi203.stdout:3269: done (1 left) 2024-09-17T15:47:37.631 INFO:tasks.rados.rados.0.smithi203.stdout:3270: copy_from oid 46 from oid 1 current snap is 354 2024-09-17T15:47:37.631 INFO:tasks.rados.rados.0.smithi203.stdout:3271: copy_from oid 49 from oid 31 current snap is 354 2024-09-17T15:47:37.631 INFO:tasks.rados.rados.0.smithi203.stdout:3272: write oid 44 current snap is 354 2024-09-17T15:47:37.631 INFO:tasks.rados.rados.0.smithi203.stdout:3272: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-09-17T15:47:37.637 INFO:tasks.rados.rados.0.smithi203.stdout:3272: writing smithi20328086-44 from 427208 to 1165014 tid 1 2024-09-17T15:47:37.644 INFO:tasks.rados.rados.0.smithi203.stdout:3272: writing smithi20328086-44 from 1777518 to 2291242 tid 2 2024-09-17T15:47:37.649 INFO:tasks.rados.rados.0.smithi203.stdout:3272: writing smithi20328086-44 from 2966139 to 3227208 tid 3 2024-09-17T15:47:37.649 INFO:tasks.rados.rados.0.smithi203.stdout:3273: snap_remove snap 313 2024-09-17T15:47:38.600 INFO:tasks.rados.rados.0.smithi203.stdout:3273: done (4 left) 2024-09-17T15:47:38.600 INFO:tasks.rados.rados.0.smithi203.stdout:3274: snap_remove snap 340 2024-09-17T15:47:39.601 INFO:tasks.rados.rados.0.smithi203.stdout:3274: done (4 left) 2024-09-17T15:47:39.601 INFO:tasks.rados.rados.0.smithi203.stdout:3275: snap_remove snap 350 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3275: done (4 left) 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3276: delete oid 9 current snap is 354 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: finishing write tid 3 to smithi20328086-12 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: finishing write tid 4 to smithi20328086-12 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: oid 12 updating version 0 to 1685 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: oid 12 updating version 1685 to 1687 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: oid 12 version 1687 is already newer than 1686 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1687 (ObjNum 1142 snap 354 seq_num 1142) dirty exists 2024-09-17T15:47:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3268: left oid 12 (ObjNum 1142 snap 354 seq_num 1142) 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3270: finishing copy_from to smithi20328086-46 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3270: got expected ENOENT (src dne) 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3271: finishing copy_from to smithi20328086-49 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3271: got expected ENOENT (src dne) 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3270: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3271: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: finishing write tid 1 to smithi20328086-44 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: finishing write tid 2 to smithi20328086-44 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: finishing write tid 3 to smithi20328086-44 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: finishing write tid 4 to smithi20328086-44 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: finishing write tid 5 to smithi20328086-44 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: oid 44 updating version 0 to 1527 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: oid 44 version 1527 is already newer than 1526 2024-09-17T15:47:40.815 INFO:tasks.rados.rados.0.smithi203.stdout:3272: oid 44 version 1527 is already newer than 1525 2024-09-17T15:47:40.816 INFO:tasks.rados.rados.0.smithi203.stdout:3272: oid 44 updating version 1527 to 1528 2024-09-17T15:47:40.816 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1528 (ObjNum 1144 snap 354 seq_num 1144) dirty exists 2024-09-17T15:47:40.816 INFO:tasks.rados.rados.0.smithi203.stdout:3272: left oid 44 (ObjNum 1144 snap 354 seq_num 1144) 2024-09-17T15:47:40.820 INFO:tasks.rados.rados.0.smithi203.stdout:3268: done (4 left) 2024-09-17T15:47:40.821 INFO:tasks.rados.rados.0.smithi203.stdout:3270: done (3 left) 2024-09-17T15:47:40.821 INFO:tasks.rados.rados.0.smithi203.stdout:3271: done (2 left) 2024-09-17T15:47:40.821 INFO:tasks.rados.rados.0.smithi203.stdout:3272: done (1 left) 2024-09-17T15:47:40.821 INFO:tasks.rados.rados.0.smithi203.stdout:3276: done (0 left) 2024-09-17T15:47:40.821 INFO:tasks.rados.rados.0.smithi203.stdout:3277: snap_create 2024-09-17T15:47:41.629 INFO:tasks.rados.rados.0.smithi203.stdout:3277: done (0 left) 2024-09-17T15:47:41.629 INFO:tasks.rados.rados.0.smithi203.stdout:3278: read oid 27 snap -1 2024-09-17T15:47:41.629 INFO:tasks.rados.rados.0.smithi203.stdout:3278: expect (ObjNum 1079 snap 334 seq_num 1079) 2024-09-17T15:47:41.629 INFO:tasks.rados.rados.0.smithi203.stdout:3279: delete oid 26 current snap is 355 2024-09-17T15:47:41.639 INFO:tasks.rados.rados.0.smithi203.stdout:3278: expect (ObjNum 1079 snap 334 seq_num 1079) 2024-09-17T15:47:41.736 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:41.796 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:41.856 INFO:tasks.rados.rados.0.smithi203.stdout:3278: done (1 left) 2024-09-17T15:47:41.857 INFO:tasks.rados.rados.0.smithi203.stdout:3279: done (0 left) 2024-09-17T15:47:41.857 INFO:tasks.rados.rados.0.smithi203.stdout:3280: delete oid 31 current snap is 355 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3280: done (0 left) 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3281: copy_from oid 47 from oid 40 current snap is 355 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3282: read oid 36 snap -1 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3282: expect (ObjNum 1135 snap 353 seq_num 1135) 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3283: write (excl) oid 46 current snap is 355 2024-09-17T15:47:41.866 INFO:tasks.rados.rados.0.smithi203.stdout:3283: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-17T15:47:41.886 INFO:tasks.rados.rados.0.smithi203.stdout:3283: writing smithi20328086-46 from 494436 to 1249591 tid 1 2024-09-17T15:47:41.907 INFO:tasks.rados.rados.0.smithi203.stdout:3283: writing smithi20328086-46 from 2011446 to 2532508 tid 2 2024-09-17T15:47:41.918 INFO:tasks.rados.rados.0.smithi203.stdout:3283: writing smithi20328086-46 from 3075831 to 3294436 tid 3 2024-09-17T15:47:41.918 INFO:tasks.rados.rados.0.smithi203.stdout:3284: write oid 28 current snap is 355 2024-09-17T15:47:41.918 INFO:tasks.rados.rados.0.smithi203.stdout:3284: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-17T15:47:41.929 INFO:tasks.rados.rados.0.smithi203.stdout:3284: writing smithi20328086-28 from 511243 to 941647 tid 1 2024-09-17T15:47:41.943 INFO:tasks.rados.rados.0.smithi203.stdout:3284: writing smithi20328086-28 from 1669928 to 2334648 tid 2 2024-09-17T15:47:41.952 INFO:tasks.rados.rados.0.smithi203.stdout:3284: writing smithi20328086-28 from 3103254 to 3311243 tid 3 2024-09-17T15:47:41.952 INFO:tasks.rados.rados.0.smithi203.stdout:3285: snap_create 2024-09-17T15:47:41.952 INFO:tasks.rados.rados.0.smithi203.stdout:3282: expect (ObjNum 1135 snap 353 seq_num 1135) 2024-09-17T15:47:42.148 INFO:tasks.rados.rados.0.smithi203.stdout:3283: finishing write tid 1 to smithi20328086-46 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3281: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3281: finishing copy_from to smithi20328086-47 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1517 (ObjNum 1067 snap 330 seq_num 1067) dirty exists 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: finishing write tid 2 to smithi20328086-46 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: finishing write tid 3 to smithi20328086-46 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: finishing write tid 4 to smithi20328086-46 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: finishing write tid 5 to smithi20328086-46 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: oid 46 updating version 0 to 1407 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: oid 46 updating version 1407 to 1408 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: oid 46 updating version 1408 to 1409 2024-09-17T15:47:42.149 INFO:tasks.rados.rados.0.smithi203.stdout:3283: oid 46 version 1409 is already newer than 1406 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1409 (ObjNum 1148 snap 355 seq_num 1148) dirty exists 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3283: left oid 46 (ObjNum 1148 snap 355 seq_num 1148) 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: finishing write tid 1 to smithi20328086-28 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: finishing write tid 2 to smithi20328086-28 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: finishing write tid 3 to smithi20328086-28 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: finishing write tid 4 to smithi20328086-28 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: finishing write tid 5 to smithi20328086-28 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: oid 28 updating version 0 to 1698 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: oid 28 version 1698 is already newer than 1697 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: oid 28 updating version 1698 to 1700 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: oid 28 version 1700 is already newer than 1699 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1700 (ObjNum 1149 snap 355 seq_num 1149) dirty exists 2024-09-17T15:47:42.150 INFO:tasks.rados.rados.0.smithi203.stdout:3284: left oid 28 (ObjNum 1149 snap 355 seq_num 1149) 2024-09-17T15:47:42.637 INFO:tasks.rados.rados.0.smithi203.stdout:3281: done (4 left) 2024-09-17T15:47:42.637 INFO:tasks.rados.rados.0.smithi203.stdout:3282: done (3 left) 2024-09-17T15:47:42.637 INFO:tasks.rados.rados.0.smithi203.stdout:3283: done (2 left) 2024-09-17T15:47:42.638 INFO:tasks.rados.rados.0.smithi203.stdout:3284: done (1 left) 2024-09-17T15:47:42.638 INFO:tasks.rados.rados.0.smithi203.stdout:3285: done (0 left) 2024-09-17T15:47:42.638 INFO:tasks.rados.rados.0.smithi203.stdout:3286: snap_remove snap 349 2024-09-17T15:47:43.640 INFO:tasks.rados.rados.0.smithi203.stdout:3286: done (0 left) 2024-09-17T15:47:43.640 INFO:tasks.rados.rados.0.smithi203.stdout:3287: snap_remove snap 338 2024-09-17T15:47:44.577 INFO:tasks.rados.rados.0.smithi203.stdout:3287: done (0 left) 2024-09-17T15:47:44.577 INFO:tasks.rados.rados.0.smithi203.stdout:3288: write (excl) oid 22 current snap is 356 2024-09-17T15:47:44.577 INFO:tasks.rados.rados.0.smithi203.stdout:3288: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-17T15:47:44.585 INFO:tasks.rados.rados.0.smithi203.stdout:3288: writing smithi20328086-22 from 528050 to 1033703 tid 1 2024-09-17T15:47:44.591 INFO:tasks.rados.rados.0.smithi203.stdout:3288: writing smithi20328086-22 from 1612057 to 2136788 tid 2 2024-09-17T15:47:44.598 INFO:tasks.rados.rados.0.smithi203.stdout:3288: writing smithi20328086-22 from 2614324 to 3181727 tid 3 2024-09-17T15:47:44.598 INFO:tasks.rados.rados.0.smithi203.stdout:3289: snap_remove snap 346 2024-09-17T15:47:45.586 INFO:tasks.rados.rados.0.smithi203.stdout:3289: done (1 left) 2024-09-17T15:47:45.586 INFO:tasks.rados.rados.0.smithi203.stdout:3290: snap_create 2024-09-17T15:47:45.586 INFO:tasks.rados.rados.0.smithi203.stdout:3288: finishing write tid 1 to smithi20328086-22 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: finishing write tid 2 to smithi20328086-22 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: finishing write tid 3 to smithi20328086-22 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: finishing write tid 4 to smithi20328086-22 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: finishing write tid 5 to smithi20328086-22 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: oid 22 updating version 0 to 1323 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: oid 22 version 1323 is already newer than 1320 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: oid 22 version 1323 is already newer than 1322 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: oid 22 version 1323 is already newer than 1321 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1323 (ObjNum 1150 snap 356 seq_num 1150) dirty exists 2024-09-17T15:47:45.587 INFO:tasks.rados.rados.0.smithi203.stdout:3288: left oid 22 (ObjNum 1150 snap 356 seq_num 1150) 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3288: done (1 left) 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3290: done (0 left) 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3291: read oid 7 snap -1 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3291: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3292: rollback oid 40 current snap is 357 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 352 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3293: rollback oid 19 current snap is 357 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 352 2024-09-17T15:47:46.596 INFO:tasks.rados.rados.0.smithi203.stdout:3294: rollback oid 25 current snap is 357 2024-09-17T15:47:46.597 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 351 2024-09-17T15:47:46.597 INFO:tasks.rados.rados.0.smithi203.stdout:3295: read oid 12 snap -1 2024-09-17T15:47:46.597 INFO:tasks.rados.rados.0.smithi203.stdout:3295: expect (ObjNum 1142 snap 354 seq_num 1142) 2024-09-17T15:47:46.597 INFO:tasks.rados.rados.0.smithi203.stdout:3296: write (excl) oid 5 current snap is 357 2024-09-17T15:47:46.597 INFO:tasks.rados.rados.0.smithi203.stdout:3296: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-17T15:47:46.616 INFO:tasks.rados.rados.0.smithi203.stdout:3296: writing smithi20328086-5 from 544857 to 1125759 tid 1 2024-09-17T15:47:46.635 INFO:tasks.rados.rados.0.smithi203.stdout:3296: writing smithi20328086-5 from 1670539 to 2338928 tid 2 2024-09-17T15:47:46.648 INFO:tasks.rados.rados.0.smithi203.stdout:3296: writing smithi20328086-5 from 3041747 to 3344857 tid 3 2024-09-17T15:47:46.648 INFO:tasks.rados.rados.0.smithi203.stdout:3297: read oid 21 snap 335 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:3297: expect (ObjNum 1054 snap 327 seq_num 1054) 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:3298: rollback oid 16 current snap is 357 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 344 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:3294: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1518 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:47:46.649 INFO:tasks.rados.rados.0.smithi203.stdout:3291: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:47:46.796 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:46.908 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:47.019 INFO:tasks.rados.rados.0.smithi203.stdout:3292: finishing rollback tid 0 to smithi20328086-40 2024-09-17T15:47:47.019 INFO:tasks.rados.rados.0.smithi203.stdout:3295: expect (ObjNum 1142 snap 354 seq_num 1142) 2024-09-17T15:47:47.085 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:47.204 INFO:tasks.rados.rados.0.smithi203.stdout:3292: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:47:47.204 INFO:tasks.rados.rados.0.smithi203.stdout:3292: finishing rollback tid 2 to smithi20328086-40 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1531 (ObjNum 1067 snap 330 seq_num 1067) dirty exists 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3293: finishing rollback tid 0 to smithi20328086-19 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3293: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3296: finishing write tid 1 to smithi20328086-5 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3293: finishing rollback tid 2 to smithi20328086-19 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1680 (ObjNum 1066 snap 330 seq_num 1066) dirty exists 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3296: finishing write tid 2 to smithi20328086-5 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3296: finishing write tid 3 to smithi20328086-5 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3296: finishing write tid 4 to smithi20328086-5 2024-09-17T15:47:47.205 INFO:tasks.rados.rados.0.smithi203.stdout:3296: finishing write tid 5 to smithi20328086-5 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3296: oid 5 updating version 0 to 1385 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3296: oid 5 updating version 1385 to 1386 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3296: oid 5 updating version 1386 to 1388 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3296: oid 5 version 1388 is already newer than 1387 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1388 (ObjNum 1151 snap 357 seq_num 1151) dirty exists 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3296: left oid 5 (ObjNum 1151 snap 357 seq_num 1151) 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3298: finishing rollback tid 0 to smithi20328086-16 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3298: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3298: finishing rollback tid 2 to smithi20328086-16 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1684 (ObjNum 1105 snap 340 seq_num 1105) dirty exists 2024-09-17T15:47:47.206 INFO:tasks.rados.rados.0.smithi203.stdout:3297: expect (ObjNum 1054 snap 327 seq_num 1054) 2024-09-17T15:47:47.231 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2658944 2024-09-17T15:47:47.287 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2130759 2024-09-17T15:47:47.330 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 136493 2024-09-17T15:47:47.402 INFO:tasks.rados.rados.0.smithi203.stdout:3291: done (7 left) 2024-09-17T15:47:47.402 INFO:tasks.rados.rados.0.smithi203.stdout:3292: done (6 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3293: done (5 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3294: done (4 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3295: done (3 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3296: done (2 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3297: done (1 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3298: done (0 left) 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3299: write oid 29 current snap is 357 2024-09-17T15:47:47.403 INFO:tasks.rados.rados.0.smithi203.stdout:3299: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-17T15:47:47.422 INFO:tasks.rados.rados.0.smithi203.stdout:3299: writing smithi20328086-29 from 561664 to 1217815 tid 1 2024-09-17T15:47:47.437 INFO:tasks.rados.rados.0.smithi203.stdout:3299: writing smithi20328086-29 from 1729021 to 2257421 tid 2 2024-09-17T15:47:47.445 INFO:tasks.rados.rados.0.smithi203.stdout:3299: writing smithi20328086-29 from 2669170 to 3361664 tid 3 2024-09-17T15:47:47.445 INFO:tasks.rados.rados.0.smithi203.stdout:3300: delete oid 13 current snap is 357 2024-09-17T15:47:47.446 INFO:tasks.rados.rados.0.smithi203.stdout:3299: finishing write tid 1 to smithi20328086-29 2024-09-17T15:47:47.446 INFO:tasks.rados.rados.0.smithi203.stdout:3299: finishing write tid 2 to smithi20328086-29 2024-09-17T15:47:47.453 INFO:tasks.rados.rados.0.smithi203.stdout:3300: done (1 left) 2024-09-17T15:47:47.453 INFO:tasks.rados.rados.0.smithi203.stdout:3301: read oid 31 snap -1 2024-09-17T15:47:47.453 INFO:tasks.rados.rados.0.smithi203.stdout:3301: expect deleted 2024-09-17T15:47:47.453 INFO:tasks.rados.rados.0.smithi203.stdout:3302: delete oid 42 current snap is 357 2024-09-17T15:47:47.454 INFO:tasks.rados.rados.0.smithi203.stdout:3299: finishing write tid 3 to smithi20328086-29 2024-09-17T15:47:47.455 INFO:tasks.rados.rados.0.smithi203.stdout:3299: finishing write tid 4 to smithi20328086-29 2024-09-17T15:47:47.455 INFO:tasks.rados.rados.0.smithi203.stdout:3299: finishing write tid 5 to smithi20328086-29 2024-09-17T15:47:47.455 INFO:tasks.rados.rados.0.smithi203.stdout:3299: oid 29 updating version 0 to 1535 2024-09-17T15:47:47.456 INFO:tasks.rados.rados.0.smithi203.stdout:3299: oid 29 version 1535 is already newer than 1534 2024-09-17T15:47:47.456 INFO:tasks.rados.rados.0.smithi203.stdout:3299: oid 29 updating version 1535 to 1536 2024-09-17T15:47:47.456 INFO:tasks.rados.rados.0.smithi203.stdout:3299: oid 29 version 1536 is already newer than 1533 2024-09-17T15:47:47.456 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1536 (ObjNum 1152 snap 357 seq_num 1152) dirty exists 2024-09-17T15:47:47.456 INFO:tasks.rados.rados.0.smithi203.stdout:3299: left oid 29 (ObjNum 1152 snap 357 seq_num 1152) 2024-09-17T15:47:47.458 INFO:tasks.rados.rados.0.smithi203.stdout:3299: done (2 left) 2024-09-17T15:47:47.458 INFO:tasks.rados.rados.0.smithi203.stdout:3301: done (1 left) 2024-09-17T15:47:47.458 INFO:tasks.rados.rados.0.smithi203.stdout:3302: done (0 left) 2024-09-17T15:47:47.458 INFO:tasks.rados.rados.0.smithi203.stdout:3303: write (excl) oid 19 current snap is 357 2024-09-17T15:47:47.458 INFO:tasks.rados.rados.0.smithi203.stdout:3303: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-17T15:47:47.466 INFO:tasks.rados.rados.0.smithi203.stdout:3303: writing smithi20328086-19 from 612085 to 1093983 tid 1 2024-09-17T15:47:47.474 INFO:tasks.rados.rados.0.smithi203.stdout:3303: writing smithi20328086-19 from 1788114 to 2463841 tid 2 2024-09-17T15:47:47.480 INFO:tasks.rados.rados.0.smithi203.stdout:3303: writing smithi20328086-19 from 3035086 to 3412085 tid 3 2024-09-17T15:47:47.480 INFO:tasks.rados.rados.0.smithi203.stdout:3304: write (excl) oid 32 current snap is 357 2024-09-17T15:47:47.480 INFO:tasks.rados.rados.0.smithi203.stdout:3303: finishing write tid 1 to smithi20328086-19 2024-09-17T15:47:47.480 INFO:tasks.rados.rados.0.smithi203.stdout:3303: finishing write tid 2 to smithi20328086-19 2024-09-17T15:47:47.480 INFO:tasks.rados.rados.0.smithi203.stdout:3304: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-09-17T15:47:47.487 INFO:tasks.rados.rados.0.smithi203.stdout:3304: writing smithi20328086-32 from 628892 to 1302392 tid 1 2024-09-17T15:47:47.494 INFO:tasks.rados.rados.0.smithi203.stdout:3304: writing smithi20328086-32 from 1962949 to 2498687 tid 2 2024-09-17T15:47:47.499 INFO:tasks.rados.rados.0.smithi203.stdout:3304: writing smithi20328086-32 from 3178862 to 3428892 tid 3 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3305: rollback oid 27 current snap is 357 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 354 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: finishing write tid 3 to smithi20328086-19 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: finishing write tid 4 to smithi20328086-19 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: finishing write tid 5 to smithi20328086-19 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: oid 19 updating version 0 to 1685 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: oid 19 updating version 1685 to 1686 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: oid 19 updating version 1686 to 1688 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: oid 19 version 1688 is already newer than 1687 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1688 (ObjNum 1155 snap 357 seq_num 1155) dirty exists 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: left oid 19 (ObjNum 1155 snap 357 seq_num 1155) 2024-09-17T15:47:47.500 INFO:tasks.rados.rados.0.smithi203.stdout:3303: done (2 left) 2024-09-17T15:47:47.501 INFO:tasks.rados.rados.0.smithi203.stdout:3306: rollback oid 34 current snap is 357 2024-09-17T15:47:47.501 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 335 2024-09-17T15:47:47.501 INFO:tasks.rados.rados.0.smithi203.stdout:3307: write (excl) oid 8 current snap is 357 2024-09-17T15:47:47.501 INFO:tasks.rados.rados.0.smithi203.stdout:3307: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-09-17T15:47:47.507 INFO:tasks.rados.rados.0.smithi203.stdout:3307: writing smithi20328086-8 from 645699 to 1394448 tid 1 2024-09-17T15:47:47.514 INFO:tasks.rados.rados.0.smithi203.stdout:3307: writing smithi20328086-8 from 2021431 to 2700827 tid 2 2024-09-17T15:47:47.520 INFO:tasks.rados.rados.0.smithi203.stdout:3307: writing smithi20328086-8 from 3206285 to 3445699 tid 3 2024-09-17T15:47:47.520 INFO:tasks.rados.rados.0.smithi203.stdout:3308: copy_from oid 14 from oid 48 current snap is 357 2024-09-17T15:47:47.520 INFO:tasks.rados.rados.0.smithi203.stdout:3306: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:47:47.520 INFO:tasks.rados.rados.0.smithi203.stdout:3309: snap_remove snap 344 2024-09-17T15:47:47.603 INFO:tasks.rados.rados.0.smithi203.stdout:3309: done (5 left) 2024-09-17T15:47:47.604 INFO:tasks.rados.rados.0.smithi203.stdout:3310: write oid 45 current snap is 357 2024-09-17T15:47:47.604 INFO:tasks.rados.rados.0.smithi203.stdout:3310: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-17T15:47:47.610 INFO:tasks.rados.rados.0.smithi203.stdout:3310: writing smithi20328086-45 from 662506 to 1086504 tid 1 2024-09-17T15:47:47.615 INFO:tasks.rados.rados.0.smithi203.stdout:3310: writing smithi20328086-45 from 1563560 to 2102967 tid 2 2024-09-17T15:47:47.622 INFO:tasks.rados.rados.0.smithi203.stdout:3310: writing smithi20328086-45 from 2717355 to 3207640 tid 3 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3311: copy_from oid 35 from oid 25 current snap is 357 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3304: finishing write tid 1 to smithi20328086-32 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3304: finishing write tid 2 to smithi20328086-32 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3306: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1314 (ObjNum 1022 snap 316 seq_num 1022) dirty exists 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3305: finishing rollback tid 0 to smithi20328086-27 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3305: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3306: done (6 left) 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3312: delete oid 16 current snap is 357 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3308: finishing copy_from to smithi20328086-14 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3308: got expected ENOENT (src dne) 2024-09-17T15:47:47.623 INFO:tasks.rados.rados.0.smithi203.stdout:3308: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: finishing write tid 3 to smithi20328086-32 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: finishing write tid 4 to smithi20328086-32 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: finishing write tid 5 to smithi20328086-32 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: oid 32 updating version 0 to 1355 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: oid 32 version 1355 is already newer than 1354 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: oid 32 updating version 1355 to 1357 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: oid 32 version 1357 is already newer than 1356 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1357 (ObjNum 1156 snap 357 seq_num 1156) dirty exists 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3304: left oid 32 (ObjNum 1156 snap 357 seq_num 1156) 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3305: finishing rollback tid 2 to smithi20328086-27 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1412 (ObjNum 1079 snap 334 seq_num 1079) dirty exists 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3307: finishing write tid 1 to smithi20328086-8 2024-09-17T15:47:47.624 INFO:tasks.rados.rados.0.smithi203.stdout:3307: finishing write tid 2 to smithi20328086-8 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: finishing write tid 3 to smithi20328086-8 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: finishing write tid 4 to smithi20328086-8 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: finishing write tid 5 to smithi20328086-8 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: oid 8 updating version 0 to 1534 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: oid 8 updating version 1534 to 1537 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: oid 8 version 1537 is already newer than 1535 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: oid 8 version 1537 is already newer than 1536 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1537 (ObjNum 1157 snap 357 seq_num 1157) dirty exists 2024-09-17T15:47:47.625 INFO:tasks.rados.rados.0.smithi203.stdout:3307: left oid 8 (ObjNum 1157 snap 357 seq_num 1157) 2024-09-17T15:47:47.627 INFO:tasks.rados.rados.0.smithi203.stdout:3310: finishing write tid 1 to smithi20328086-45 2024-09-17T15:47:47.629 INFO:tasks.rados.rados.0.smithi203.stdout:3310: finishing write tid 2 to smithi20328086-45 2024-09-17T15:47:47.629 INFO:tasks.rados.rados.0.smithi203.stdout:3311: finishing copy_from to smithi20328086-35 2024-09-17T15:47:47.629 INFO:tasks.rados.rados.0.smithi203.stdout:3311: got expected ENOENT (src dne) 2024-09-17T15:47:47.629 INFO:tasks.rados.rados.0.smithi203.stdout:3311: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:47:47.634 INFO:tasks.rados.rados.0.smithi203.stdout:3304: done (6 left) 2024-09-17T15:47:47.634 INFO:tasks.rados.rados.0.smithi203.stdout:3305: done (5 left) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3307: done (4 left) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3308: done (3 left) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3311: done (2 left) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3312: done (1 left) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3313: read oid 15 snap -1 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3313: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3314: read oid 31 snap -1 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3314: expect deleted 2024-09-17T15:47:47.635 INFO:tasks.rados.rados.0.smithi203.stdout:3315: delete oid 13 current snap is 357 2024-09-17T15:47:47.636 INFO:tasks.rados.rados.0.smithi203.stdout:3310: finishing write tid 3 to smithi20328086-45 2024-09-17T15:47:47.636 INFO:tasks.rados.rados.0.smithi203.stdout:3310: finishing write tid 4 to smithi20328086-45 2024-09-17T15:47:47.636 INFO:tasks.rados.rados.0.smithi203.stdout:3310: finishing write tid 5 to smithi20328086-45 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:3310: oid 45 updating version 0 to 1332 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:3310: oid 45 updating version 1332 to 1335 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:3310: oid 45 version 1335 is already newer than 1333 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:3310: oid 45 version 1335 is already newer than 1334 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1335 (ObjNum 1158 snap 357 seq_num 1158) dirty exists 2024-09-17T15:47:47.637 INFO:tasks.rados.rados.0.smithi203.stdout:3310: left oid 45 (ObjNum 1158 snap 357 seq_num 1158) 2024-09-17T15:47:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:3310: done (3 left) 2024-09-17T15:47:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:3314: done (2 left) 2024-09-17T15:47:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:3315: done (1 left) 2024-09-17T15:47:47.642 INFO:tasks.rados.rados.0.smithi203.stdout:3316: delete oid 26 current snap is 357 2024-09-17T15:47:47.644 INFO:tasks.rados.rados.0.smithi203.stdout:3313: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:47:47.652 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2014414 2024-09-17T15:47:47.722 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 634914 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3313: done (1 left) 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3316: done (0 left) 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3317: copy_from oid 50 from oid 28 current snap is 357 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3318: read oid 25 snap -1 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3318: expect deleted 2024-09-17T15:47:47.748 INFO:tasks.rados.rados.0.smithi203.stdout:3319: snap_create 2024-09-17T15:47:47.770 INFO:tasks.rados.rados.0.smithi203.stdout:3317: finishing copy_from racing read to smithi20328086-50 2024-09-17T15:47:47.772 INFO:tasks.rados.rados.0.smithi203.stdout:3317: finishing copy_from to smithi20328086-50 2024-09-17T15:47:47.772 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1700 (ObjNum 1149 snap 355 seq_num 1149) dirty exists 2024-09-17T15:47:48.612 INFO:tasks.rados.rados.0.smithi203.stdout:3317: done (2 left) 2024-09-17T15:47:48.612 INFO:tasks.rados.rados.0.smithi203.stdout:3318: done (1 left) 2024-09-17T15:47:48.612 INFO:tasks.rados.rados.0.smithi203.stdout:3319: done (0 left) 2024-09-17T15:47:48.612 INFO:tasks.rados.rados.0.smithi203.stdout:3320: write oid 41 current snap is 358 2024-09-17T15:47:48.612 INFO:tasks.rados.rados.0.smithi203.stdout:3320: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-09-17T15:47:48.628 INFO:tasks.rados.rados.0.smithi203.stdout:3320: writing smithi20328086-41 from 729734 to 1454728 tid 1 2024-09-17T15:47:48.642 INFO:tasks.rados.rados.0.smithi203.stdout:3320: writing smithi20328086-41 from 2197488 to 2627880 tid 2 2024-09-17T15:47:48.654 INFO:tasks.rados.rados.0.smithi203.stdout:3320: writing smithi20328086-41 from 3227047 to 3529734 tid 3 2024-09-17T15:47:48.654 INFO:tasks.rados.rados.0.smithi203.stdout:3321: rollback oid 22 current snap is 358 2024-09-17T15:47:48.655 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 22 to 345 2024-09-17T15:47:48.655 INFO:tasks.rados.rados.0.smithi203.stdout:3320: finishing write tid 1 to smithi20328086-41 2024-09-17T15:47:48.655 INFO:tasks.rados.rados.0.smithi203.stdout:3320: finishing write tid 2 to smithi20328086-41 2024-09-17T15:47:48.655 INFO:tasks.rados.rados.0.smithi203.stdout:3322: write (excl) oid 50 current snap is 358 2024-09-17T15:47:48.655 INFO:tasks.rados.rados.0.smithi203.stdout:3322: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-09-17T15:47:48.667 INFO:tasks.rados.rados.0.smithi203.stdout:3322: writing smithi20328086-50 from 746541 to 1146784 tid 1 2024-09-17T15:47:48.679 INFO:tasks.rados.rados.0.smithi203.stdout:3322: writing smithi20328086-50 from 1739617 to 2430020 tid 2 2024-09-17T15:47:48.688 INFO:tasks.rados.rados.0.smithi203.stdout:3322: writing smithi20328086-50 from 3138117 to 3546541 tid 3 2024-09-17T15:47:48.688 INFO:tasks.rados.rados.0.smithi203.stdout:3323: snap_remove snap 351 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3323: done (3 left) 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3324: copy_from oid 1 from oid 29 current snap is 358 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: finishing write tid 3 to smithi20328086-41 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: finishing write tid 4 to smithi20328086-41 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: finishing write tid 5 to smithi20328086-41 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: oid 41 updating version 0 to 677 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: oid 41 updating version 677 to 678 2024-09-17T15:47:49.581 INFO:tasks.rados.rados.0.smithi203.stdout:3320: oid 41 updating version 678 to 680 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3320: oid 41 version 680 is already newer than 679 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 680 (ObjNum 1162 snap 358 seq_num 1162) dirty exists 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3320: left oid 41 (ObjNum 1162 snap 358 seq_num 1162) 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3321: finishing rollback tid 0 to smithi20328086-22 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3321: finishing rollback tid 1 to smithi20328086-22 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3321: finishing rollback tid 2 to smithi20328086-22 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1326 (ObjNum 1114 snap 343 seq_num 1114) dirty exists 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: finishing write tid 1 to smithi20328086-50 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: finishing write tid 2 to smithi20328086-50 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: finishing write tid 3 to smithi20328086-50 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: finishing write tid 4 to smithi20328086-50 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: finishing write tid 5 to smithi20328086-50 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: oid 50 updating version 0 to 1702 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: oid 50 version 1702 is already newer than 1701 2024-09-17T15:47:49.582 INFO:tasks.rados.rados.0.smithi203.stdout:3322: oid 50 updating version 1702 to 1703 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3322: oid 50 updating version 1703 to 1704 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1704 (ObjNum 1163 snap 358 seq_num 1163) dirty exists 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3322: left oid 50 (ObjNum 1163 snap 358 seq_num 1163) 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3320: done (3 left) 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3321: done (2 left) 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3322: done (1 left) 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3325: copy_from oid 30 from oid 17 current snap is 358 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3326: write oid 19 current snap is 358 2024-09-17T15:47:49.583 INFO:tasks.rados.rados.0.smithi203.stdout:3326: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-17T15:47:49.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: writing smithi20328086-19 from 763348 to 1355193 tid 1 2024-09-17T15:47:49.599 INFO:tasks.rados.rados.0.smithi203.stdout:3326: writing smithi20328086-19 from 1914452 to 2348513 tid 2 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:3326: writing smithi20328086-19 from 2881893 to 3556163 tid 3 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:3327: rollback oid 50 current snap is 358 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 356 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:3325: finishing copy_from to smithi20328086-30 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:3325: got expected ENOENT (src dne) 2024-09-17T15:47:49.610 INFO:tasks.rados.rados.0.smithi203.stdout:3325: finishing copy_from racing read to smithi20328086-30 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3325: done (3 left) 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3328: read oid 10 snap -1 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3328: expect deleted 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3329: read oid 14 snap -1 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3329: expect deleted 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3330: read oid 11 snap -1 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3330: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:47:49.611 INFO:tasks.rados.rados.0.smithi203.stdout:3331: snap_remove snap 348 2024-09-17T15:47:50.590 INFO:tasks.rados.rados.0.smithi203.stdout:3331: done (6 left) 2024-09-17T15:47:50.591 INFO:tasks.rados.rados.0.smithi203.stdout:3332: delete oid 12 current snap is 358 2024-09-17T15:47:50.591 INFO:tasks.rados.rados.0.smithi203.stdout:3327: finishing rollback tid 0 to smithi20328086-50 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3327: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: finishing write tid 1 to smithi20328086-19 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3324: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3327: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1707 (ObjNum 1071 snap 330 seq_num 1071) dirty exists 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3324: finishing copy_from to smithi20328086-1 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1536 (ObjNum 1152 snap 357 seq_num 1152) dirty exists 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: finishing write tid 2 to smithi20328086-19 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: finishing write tid 3 to smithi20328086-19 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: finishing write tid 4 to smithi20328086-19 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: finishing write tid 5 to smithi20328086-19 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: oid 19 updating version 0 to 1704 2024-09-17T15:47:50.592 INFO:tasks.rados.rados.0.smithi203.stdout:3326: oid 19 updating version 1704 to 1705 2024-09-17T15:47:50.593 INFO:tasks.rados.rados.0.smithi203.stdout:3326: oid 19 version 1705 is already newer than 1699 2024-09-17T15:47:50.593 INFO:tasks.rados.rados.0.smithi203.stdout:3326: oid 19 updating version 1705 to 1706 2024-09-17T15:47:50.593 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1706 (ObjNum 1164 snap 358 seq_num 1164) dirty exists 2024-09-17T15:47:50.593 INFO:tasks.rados.rados.0.smithi203.stdout:3326: left oid 19 (ObjNum 1164 snap 358 seq_num 1164) 2024-09-17T15:47:50.593 INFO:tasks.rados.rados.0.smithi203.stdout:3330: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:47:50.721 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-17T15:47:50.721 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T15:47:50.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T15:47:50.724 INFO:tasks.ceph.mon.a:Started 2024-09-17T15:47:50.724 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-17T15:47:50.724 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-17T15:47:50.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:47:50.765 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3324: done (6 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3326: done (5 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3327: done (4 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3328: done (3 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3329: done (2 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3330: done (1 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3332: done (0 left) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3333: read oid 4 snap -1 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3333: expect (ObjNum 1141 snap 353 seq_num 1141) 2024-09-17T15:47:50.837 INFO:tasks.rados.rados.0.smithi203.stdout:3334: snap_remove snap 357 2024-09-17T15:48:05.725 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-17T15:48:05.726 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-17T15:48:05.726 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:48:05.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:05.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: get_monmap_and_config 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: build_initial_monmap 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: monmap: 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:05.861829+0000 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:05.861829+0000 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:05.858 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 Processor -- start 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 -- start start 2024-09-17T15:48:05.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: init 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e140689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-c con 0x7f5e14068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5e14135ae0 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1409c8b0 con 0x7f5e14135ae0 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5e14074500 con 0x7f5e14068db0 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1409cbe0 con 0x7f5e140689d0 2024-09-17T15:48:05.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1893f640 10 monclient(hunting): authenticate will time out at 2218.827275s 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e12d9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5e14135ae0 0x7f5e14135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50514/0 (socket says 172.21.15.86:50514) 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e12d9d640 1 -- 172.21.15.86:0/2781633233 learned_addr learned my addr 172.21.15.86:0/2781633233 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:05.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e14068db0 auth_method 0 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.856+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:05.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:05.860+0000 7f5e12d9d640 1 -- 172.21.15.86:0/2781633233 >> v1:172.21.15.86:6789/0 conn(0x7f5e14135ae0 legacy=0x7f5e14135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e14068db0 auth_method 0 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:06.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.060+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:06.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:06.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:06.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:06.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e14068db0 auth_method 0 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:06.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:06.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:06.464+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:07.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e14068db0 auth_method 0 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:07.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:07.268+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:08.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): tick 2024-09-17T15:48:08.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e1406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e1406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 msgr2=0x7f5e140691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:08.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.203:3300/0 conn(0x7f5e14068db0 0x7f5e140691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 -- 172.21.15.86:0/2781633233 >> v1:172.21.15.86:6789/0 conn(0x7f5e14135ae0 legacy=0x7f5e14135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient: _add_conns ranks=[1,3] 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5df4003750 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f5df4007480 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:08.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 -- 172.21.15.86:0/2781633233 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1409ce70 con 0x7f5df4003750 2024-09-17T15:48:08.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:08.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 1 -- 172.21.15.86:0/2781633233 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5e1409cbe0 con 0x7f5df4007480 2024-09-17T15:48:08.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e0bfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:08.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5df4007480 auth_method 0 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:08.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:08.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:08.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:08.860+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> v1:172.21.15.203:6789/0 conn(0x7f5df4003750 legacy=0x7f5df4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5df4007480 auth_method 0 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:09.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:09.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:09.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.064+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5df4007480 auth_method 0 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:09.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:09.468+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5df4007480 auth_method 0 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:10.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.272+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088140167 0 0) 0x7f5e1409ce70 con 0x7f5df4003750 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 10 monclient(hunting): _negotiate 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 10 monclient(hunting): my global_id is 224167 2024-09-17T15:48:10.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.784+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e00003530 con 0x7f5df4003750 2024-09-17T15:48:10.787 INFO:tasks.rados.rados.0.smithi203.stdout:3334: done (1 left) 2024-09-17T15:48:10.787 INFO:tasks.rados.rados.0.smithi203.stdout:3335: snap_create 2024-09-17T15:48:10.787 INFO:tasks.rados.rados.0.smithi203.stdout:3333: expect (ObjNum 1141 snap 353 seq_num 1141) 2024-09-17T15:48:10.787 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4109154562 0 0) 0x7f5e00003530 con 0x7f5df4003750 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 msgr2=0x7f5df4009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 --2- 172.21.15.86:0/2781633233 >> v2:172.21.15.86:3300/0 conn(0x7f5df4007480 0x7f5df4009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _finish_hunting 0 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df400b070 con 0x7f5df4003750 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _finish_auth 0 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _check_auth_tickets 2024-09-17T15:48:10.790 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f5e0400f040 con 0x7f5df4003750 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: dump: 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 >> v1:172.21.15.203:6789/0 conn(0x7f5df4003750 legacy=0x7f5df4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 1 -- 172.21.15.86:0/2781633233 reap_dead start 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e00003b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): picked mon.c con 0x7f5e140689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 0x7f5e00004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): picked mon.b con 0x7f5e00004180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 0x7f5e0000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): picked mon.a con 0x7f5e00004f50 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1409cbe0 con 0x7f5e00004f50 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0000bf10 con 0x7f5e00004180 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0000c090 con 0x7f5e140689d0 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1159a640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e00003b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 0x7f5e00004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 0x7f5e0000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e00004180 auth_method 0 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): get_auth_request con 0x7f5e00004f50 auth_method 0 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:48:10.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient(hunting): handle_auth_done global_id 224167 payload 293 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 msgr2=0x7f5e00004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 0x7f5e00004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 msgr2=0x7f5e0000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 0x7f5e0000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 monclient: found mon.c 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:10.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e00003530 con 0x7f5e140689d0 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient: _finish_auth 0 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 10 monclient: _check_auth_tickets 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1893f640 5 monclient: authenticate success, global_id 224167 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e1893f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 0x7f5e00004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:10.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.788+0000 7f5e12d9d640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 0x7f5e0000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1259c640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e00003b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1259c640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e00003b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5e04000ee0 tx=0x7f5e040200e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f2ff18f6c7aa066 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e040af070 con 0x7f5e140689d0 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: dump: 2024-09-17T15:48:10.797 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e040a9d70 con 0x7f5e140689d0 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:10.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 1 -- 172.21.15.86:0/2781633233 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e040142f0 con 0x7f5e140689d0 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 4 set_mon_vals no callback set 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1159a640 10 monclient: dump: 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:10.799 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:10.800 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:10.800 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 10 monclient: shutdown 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 -- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e00003b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e00003b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5e04000ee0 tx=0x7f5e040200e0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 -- 172.21.15.86:0/2781633233 shutdown_connections 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004f50 0x7f5e0000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004180 0x7f5e00004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 --2- 172.21.15.86:0/2781633233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e00003b20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 -- 172.21.15.86:0/2781633233 >> 172.21.15.86:0/2781633233 conn(0x7f5e140a6440 msgr2=0x7f5e140a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 -- 172.21.15.86:0/2781633233 shutdown_connections 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.796+0000 7f5e1893f640 1 -- 172.21.15.86:0/2781633233 wait complete. 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient: build_initial_monmap 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient: monmap: 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:10.804 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 Processor -- start 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 -- start start 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient: init 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-a con 0x7f5e00004180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004f50 0x7f5e141cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-b con 0x7f5e00004f50 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): picked mon.noname-c con 0x7f5e140689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5e1409c0c0 con 0x7f5e00004180 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e14069830 con 0x7f5e00004f50 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1409c240 con 0x7f5e140689d0 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1893f640 10 monclient(hunting): authenticate will time out at 2223.770728s 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:54142/0 (socket says 172.21.15.86:54142) 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 -- 172.21.15.86:0/1537050695 learned_addr learned my addr 172.21.15.86:0/1537050695 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): get_auth_request con 0x7f5e140689d0 auth_method 0 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 10 monclient(hunting): get_auth_request con 0x7f5e00004180 auth_method 0 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 msgr2=0x7f5e141cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:48:10.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 msgr2=0x7f5e141cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004f50 0x7f5e141cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e1259c640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request con 0x7f5e00004f50 auth_method 0 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e11d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient(hunting): handle_auth_done global_id 164783 payload 824 2024-09-17T15:48:10.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004f50 msgr2=0x7f5e141cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004f50 0x7f5e141cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 msgr2=0x7f5e141cd050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 monclient: found mon.noname-c 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.800+0000 7f5e12d9d640 1 -- 172.21.15.86:0/1537050695 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e1409ee80 con 0x7f5e140689d0 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.804+0000 7f5e12d9d640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e14123620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.804+0000 7f5e12d9d640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009fa0 tx=0x7f5e0c00f1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c00fdf0 con 0x7f5e140689d0 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: dump: 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:10.829 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: _finish_auth 0 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e0c018070 con 0x7f5e140689d0 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 5 monclient: authenticate success, global_id 164783 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _renew_subs 2024-09-17T15:48:10.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e1409d460 con 0x7f5e140689d0 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c0135f0 con 0x7f5e140689d0 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 10 monclient: dump: 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:10.831 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _renew_subs 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e10d99640 4 set_mon_vals no callback set 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:10.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e14127160 con 0x7f5e140689d0 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e140692f0 con 0x7f5e140689d0 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5e0c0137d0 con 0x7f5e140689d0 2024-09-17T15:48:10.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 0x7f5df403e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:10.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.828+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(734..734 src has 696..734) ==== 4847+0+0 (secure 0 0 0) 0x7f5e0c050040 con 0x7f5e140689d0 2024-09-17T15:48:10.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.832+0000 7f5e1259c640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 0x7f5df403e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:10.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.832+0000 7f5e1259c640 10 monclient: get_auth_request con 0x7f5df403c430 auth_method 0 2024-09-17T15:48:10.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.832+0000 7f5e1259c640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 0x7f5df403e8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400e920 tx=0x7f5e04010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:10.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.836+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5e1409ee80 con 0x7f5e140689d0 2024-09-17T15:48:10.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.836+0000 7f5e0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:10.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:10.836+0000 7f5e0affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:10.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 623888 2024-09-17T15:48:10.910 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1959678 2024-09-17T15:48:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:11.068+0000 7f5e1893f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:11.068+0000 7f5e1893f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:11.068+0000 7f5e1893f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:11.068+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5e1406d4f0 con 0x7f5e140689d0 2024-09-17T15:48:13.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:13.804+0000 7f5e09ffb640 10 monclient: tick 2024-09-17T15:48:13.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:13.804+0000 7f5e09ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:13.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:13.804+0000 7f5e09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:13.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:13.804+0000 7f5e09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:14.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:14.800+0000 7f5e09ffb640 10 monclient: tick 2024-09-17T15:48:14.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:14.800+0000 7f5e09ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:14.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:14.800+0000 7f5e09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:15.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.800+0000 7f5e09ffb640 10 monclient: tick 2024-09-17T15:48:15.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.800+0000 7f5e09ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:15.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.800+0000 7f5e09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:15.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.808+0000 7f5e0affd640 1 -- 172.21.15.86:0/1537050695 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5e0c012d60 con 0x7f5e140689d0 2024-09-17T15:48:15.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.808+0000 7f5e0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:15.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.808+0000 7f5e0affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:15.810 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":192,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:15.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 msgr2=0x7f5df403e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:15.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 0x7f5df403e8f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5e0400e920 tx=0x7f5e04010040 comp rx=0 tx=0).stop 2024-09-17T15:48:15.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 10 monclient: shutdown 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 msgr2=0x7f5e14123620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009fa0 tx=0x7f5e0c00f1d0 comp rx=0 tx=0).stop 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 shutdown_connections 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5df403c430 0x7f5df403e8f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e140689d0 0x7f5e14123620 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e00004f50 0x7f5e141cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 --2- 172.21.15.86:0/1537050695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e00004180 0x7f5e141cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:15.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 >> 172.21.15.86:0/1537050695 conn(0x7f5e140a6440 msgr2=0x7f5e14135210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:15.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 shutdown_connections 2024-09-17T15:48:15.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:15.812+0000 7f5e1893f640 1 -- 172.21.15.86:0/1537050695 wait complete. 2024-09-17T15:48:17.319 INFO:tasks.rados.rados.0.smithi203.stdout:3333: done (1 left) 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3335: done (0 left) 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3336: read oid 10 snap 341 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3336: expect (ObjNum 1107 snap 341 seq_num 1107) 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3337: read oid 4 snap -1 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3337: expect (ObjNum 1141 snap 353 seq_num 1141) 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3338: write oid 45 current snap is 359 2024-09-17T15:48:17.320 INFO:tasks.rados.rados.0.smithi203.stdout:3338: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-09-17T15:48:17.341 INFO:tasks.rados.rados.0.smithi203.stdout:3338: writing smithi20328086-45 from 796962 to 1539305 tid 1 2024-09-17T15:48:17.355 INFO:tasks.rados.rados.0.smithi203.stdout:3338: writing smithi20328086-45 from 2031416 to 2469146 tid 2 2024-09-17T15:48:17.368 INFO:tasks.rados.rados.0.smithi203.stdout:3338: writing smithi20328086-45 from 2936739 to 3349906 tid 3 2024-09-17T15:48:17.368 INFO:tasks.rados.rados.0.smithi203.stdout:3339: write oid 15 current snap is 359 2024-09-17T15:48:17.368 INFO:tasks.rados.rados.0.smithi203.stdout:3339: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-17T15:48:17.378 INFO:tasks.rados.rados.0.smithi203.stdout:3339: writing smithi20328086-15 from 413769 to 831361 tid 1 2024-09-17T15:48:17.392 INFO:tasks.rados.rados.0.smithi203.stdout:3339: writing smithi20328086-15 from 1573545 to 2271286 tid 2 2024-09-17T15:48:17.402 INFO:tasks.rados.rados.0.smithi203.stdout:3339: writing smithi20328086-15 from 2847809 to 3472248 tid 3 2024-09-17T15:48:17.402 INFO:tasks.rados.rados.0.smithi203.stdout:3340: delete oid 17 current snap is 359 2024-09-17T15:48:17.402 INFO:tasks.rados.rados.0.smithi203.stdout:3337: expect (ObjNum 1141 snap 353 seq_num 1141) 2024-09-17T15:48:17.437 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:17.505 INFO:tasks.rados.rados.0.smithi203.stdout:3336: expect (ObjNum 1107 snap 341 seq_num 1107) 2024-09-17T15:48:17.514 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2573389 2024-09-17T15:48:17.563 INFO:tasks.rados.rados.0.smithi203.stdout:3338: finishing write tid 1 to smithi20328086-45 2024-09-17T15:48:17.563 INFO:tasks.rados.rados.0.smithi203.stdout:3338: finishing write tid 2 to smithi20328086-45 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: finishing write tid 3 to smithi20328086-45 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: finishing write tid 4 to smithi20328086-45 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: finishing write tid 5 to smithi20328086-45 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: oid 45 updating version 0 to 1338 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: oid 45 version 1338 is already newer than 1337 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: oid 45 updating version 1338 to 1339 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: oid 45 version 1339 is already newer than 1336 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1339 (ObjNum 1166 snap 359 seq_num 1166) dirty exists 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3338: left oid 45 (ObjNum 1166 snap 359 seq_num 1166) 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3339: finishing write tid 1 to smithi20328086-15 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3339: finishing write tid 2 to smithi20328086-15 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3339: finishing write tid 3 to smithi20328086-15 2024-09-17T15:48:17.564 INFO:tasks.rados.rados.0.smithi203.stdout:3339: finishing write tid 4 to smithi20328086-15 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: finishing write tid 5 to smithi20328086-15 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: oid 15 updating version 0 to 1690 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: oid 15 version 1690 is already newer than 1689 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: oid 15 updating version 1690 to 1691 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: oid 15 updating version 1691 to 1692 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1692 (ObjNum 1167 snap 359 seq_num 1167) dirty exists 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: left oid 15 (ObjNum 1167 snap 359 seq_num 1167) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3336: done (4 left) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3337: done (3 left) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3338: done (2 left) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3339: done (1 left) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3340: done (0 left) 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3341: write (excl) oid 44 current snap is 359 2024-09-17T15:48:17.565 INFO:tasks.rados.rados.0.smithi203.stdout:3341: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-09-17T15:48:17.575 INFO:tasks.rados.rados.0.smithi203.stdout:3341: writing smithi20328086-44 from 447383 to 1015473 tid 1 2024-09-17T15:48:17.591 INFO:tasks.rados.rados.0.smithi203.stdout:3341: writing smithi20328086-44 from 1690509 to 2391919 tid 2 2024-09-17T15:48:17.604 INFO:tasks.rados.rados.0.smithi203.stdout:3341: writing smithi20328086-44 from 2902655 to 3549638 tid 3 2024-09-17T15:48:17.605 INFO:tasks.rados.rados.0.smithi203.stdout:3342: write oid 49 current snap is 359 2024-09-17T15:48:17.605 INFO:tasks.rados.rados.0.smithi203.stdout:3342: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-17T15:48:17.616 INFO:tasks.rados.rados.0.smithi203.stdout:3342: writing smithi20328086-49 from 464190 to 1107529 tid 1 2024-09-17T15:48:17.626 INFO:tasks.rados.rados.0.smithi203.stdout:3342: writing smithi20328086-49 from 1748991 to 2194059 tid 2 2024-09-17T15:48:17.637 INFO:tasks.rados.rados.0.smithi203.stdout:3342: writing smithi20328086-49 from 2930078 to 3504686 tid 3 2024-09-17T15:48:17.637 INFO:tasks.rados.rados.0.smithi203.stdout:3343: snap_remove snap 353 2024-09-17T15:48:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:3343: done (2 left) 2024-09-17T15:48:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:3344: copy_from oid 16 from oid 33 current snap is 359 2024-09-17T15:48:18.328 INFO:tasks.rados.rados.0.smithi203.stdout:3341: finishing write tid 1 to smithi20328086-44 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: finishing write tid 2 to smithi20328086-44 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: finishing write tid 3 to smithi20328086-44 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: finishing write tid 4 to smithi20328086-44 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: finishing write tid 5 to smithi20328086-44 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: oid 44 updating version 0 to 1539 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: oid 44 updating version 1539 to 1540 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: oid 44 updating version 1540 to 1541 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: oid 44 version 1541 is already newer than 1538 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1541 (ObjNum 1169 snap 359 seq_num 1169) dirty exists 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3341: left oid 44 (ObjNum 1169 snap 359 seq_num 1169) 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3342: finishing write tid 1 to smithi20328086-49 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3342: finishing write tid 2 to smithi20328086-49 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3342: finishing write tid 3 to smithi20328086-49 2024-09-17T15:48:18.329 INFO:tasks.rados.rados.0.smithi203.stdout:3342: finishing write tid 4 to smithi20328086-49 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: finishing write tid 5 to smithi20328086-49 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: oid 49 updating version 0 to 1729 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: oid 49 updating version 1729 to 1730 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: oid 49 version 1730 is already newer than 1728 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: oid 49 updating version 1730 to 1731 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1731 (ObjNum 1170 snap 359 seq_num 1170) dirty exists 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: left oid 49 (ObjNum 1170 snap 359 seq_num 1170) 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3341: done (2 left) 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3342: done (1 left) 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3345: read oid 28 snap 345 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3345: expect deleted 2024-09-17T15:48:18.330 INFO:tasks.rados.rados.0.smithi203.stdout:3346: snap_create 2024-09-17T15:48:18.355 INFO:tasks.rados.rados.0.smithi203.stdout:3344: finishing copy_from racing read to smithi20328086-16 2024-09-17T15:48:18.355 INFO:tasks.rados.rados.0.smithi203.stdout:3344: finishing copy_from to smithi20328086-16 2024-09-17T15:48:18.355 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 918 (ObjNum 551 snap 161 seq_num 551) dirty exists 2024-09-17T15:48:18.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:18.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: get_monmap_and_config 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: build_initial_monmap 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: monmap: 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:18.965764+0000 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:18.965764+0000 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:18.961 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:18.962 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:18.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 Processor -- start 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 -- start start 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: init 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 0x7fb5b406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5b4068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5b413aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b4074470 con 0x7fb5b413aff0 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b409c8b0 con 0x7fb5b4068db0 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b409cbe0 con 0x7fb5b40689d0 2024-09-17T15:48:18.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b8aea640 10 monclient(hunting): authenticate will time out at 2231.931325s 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 0x7fb5b406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b2d9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb5b413aff0 0x7fb5b413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40738/0 (socket says 172.21.15.86:40738) 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/3359598305 learned_addr learned my addr 172.21.15.86:0/3359598305 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): get_auth_request con 0x7fb5b40689d0 auth_method 0 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient(hunting): handle_auth_done global_id 164786 payload 274 2024-09-17T15:48:18.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 1 -- 172.21.15.86:0/3359598305 >> v1:172.21.15.203:6789/0 conn(0x7fb5b4068db0 legacy=0x7fb5b40691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 1 -- 172.21.15.86:0/3359598305 >> v1:172.21.15.86:6789/0 conn(0x7fb5b413aff0 legacy=0x7fb5b413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 1 monclient: found mon.noname-b 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:18.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.963+0000 7fb5b259c640 1 -- 172.21.15.86:0/3359598305 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b409ce70 con 0x7fb5b40689d0 2024-09-17T15:48:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b259c640 1 -- 172.21.15.86:0/3359598305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 msgr2=0x7fb5b406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:18.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b259c640 1 --2- 172.21.15.86:0/3359598305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 0x7fb5b406f840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0097f0 tx=0x7fb59c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=691cb4081270a89 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 1 -- 172.21.15.86:0/3359598305 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0b0070 con 0x7fb5b40689d0 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: dump: 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:18.971 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: _finish_auth 0 2024-09-17T15:48:18.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 1 -- 172.21.15.86:0/3359598305 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c0ab600 con 0x7fb5b40689d0 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 5 monclient: authenticate success, global_id 164786 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 1 -- 172.21.15.86:0/3359598305 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b409c2d0 con 0x7fb5b40689d0 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 4 set_mon_vals no callback set 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:18.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b159a640 10 monclient: dump: 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: shutdown 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- 172.21.15.86:0/3359598305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 msgr2=0x7fb5b406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/3359598305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 0x7fb5b406f840 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0097f0 tx=0x7fb59c01c3a0 comp rx=0 tx=0).stop 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- 172.21.15.86:0/3359598305 shutdown_connections 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/3359598305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b40689d0 0x7fb5b406f840 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- 172.21.15.86:0/3359598305 >> 172.21.15.86:0/3359598305 conn(0x7fb5b40a6440 msgr2=0x7fb5b40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- 172.21.15.86:0/3359598305 shutdown_connections 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- 172.21.15.86:0/3359598305 wait complete. 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: build_initial_monmap 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: monmap: 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:18.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 Processor -- start 2024-09-17T15:48:18.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- start start 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: init 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 0x7fb5b4072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b40689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 0x7fb5b4135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): picked mon.noname-a con 0x7fb5b413aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b4141ca0 con 0x7fb5b413aff0 2024-09-17T15:48:18.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b4141b20 con 0x7fb5b40689d0 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b4141e20 con 0x7fb5b4068db0 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b8aea640 10 monclient(hunting): authenticate will time out at 2231.939025s 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 0x7fb5b4135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56196/0 (socket says 172.21.15.86:56196) 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 0x7fb5b4135c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59050/0 (socket says 172.21.15.86:59050) 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/807989548 learned_addr learned my addr 172.21.15.86:0/807989548 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:18.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): get_auth_request con 0x7fb5b413aff0 auth_method 0 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): get_auth_request con 0x7fb5b4068db0 auth_method 0 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:18.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b259c640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 0x7fb5b4072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient(hunting): handle_auth_done global_id 234099 payload 824 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 msgr2=0x7fb5b4135c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 0x7fb5b4135c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 msgr2=0x7fb5b4072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 0x7fb5b4072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:48:18.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:18.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:18.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/807989548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b409dc50 con 0x7fb5b413aff0 2024-09-17T15:48:18.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b259c640 10 monclient: get_auth_request con 0x7fb5b40689d0 auth_method 0 2024-09-17T15:48:18.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.971+0000 7fb5b259c640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 0x7fb5b4072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:48:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b2d9d640 1 -- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 msgr2=0x7fb5b41d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b2d9d640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c420 tx=0x7fb5ac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:18.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac017070 con 0x7fb5b413aff0 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: dump: 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.984 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0070d0 con 0x7fb5b413aff0 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0074c0 con 0x7fb5b413aff0 2024-09-17T15:48:18.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 10 monclient: dump: 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:18.986 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b0d99640 4 set_mon_vals no callback set 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 5 monclient: authenticate success, global_id 234099 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _renew_subs 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b409d460 con 0x7fb5b413aff0 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _renew_subs 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b4073700 con 0x7fb5b413aff0 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:18.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5b406f840 con 0x7fb5b413aff0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb5ac01a070 con 0x7fb5b413aff0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 0x7fb59403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.975+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(736..736 src has 696..736) ==== 4863+0+0 (secure 0 0 0) 0x7fb5ac04f400 con 0x7fb5b413aff0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.979+0000 7fb5b259c640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 0x7fb59403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.979+0000 7fb5b259c640 10 monclient: get_auth_request con 0x7fb59403bbb0 auth_method 0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.979+0000 7fb5b259c640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 0x7fb59403e070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ab000 tx=0x7fb59c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.983+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fb5ac007660 con 0x7fb5b413aff0 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.983+0000 7fb5aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:18.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:18.983+0000 7fb5aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5b8aea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5b8aea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5b8aea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:19.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb5b41216b0 con 0x7fb5b413aff0 2024-09-17T15:48:19.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5aaffd640 1 -- 172.21.15.86:0/807989548 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb5ac04f080 con 0x7fb5b413aff0 2024-09-17T15:48:19.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:19.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.211+0000 7fb5aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:19.211 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 msgr2=0x7fb59403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 0x7fb59403e070 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb59c0ab000 tx=0x7fb59c008040 comp rx=0 tx=0).stop 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 10 monclient: shutdown 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 msgr2=0x7fb5b41d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c420 tx=0x7fb5ac00c8e0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 shutdown_connections 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb59403bbb0 0x7fb59403e070 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5b413aff0 0x7fb5b41d0130 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5b4068db0 0x7fb5b4135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 --2- 172.21.15.86:0/807989548 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5b40689d0 0x7fb5b4072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 >> 172.21.15.86:0/807989548 conn(0x7fb5b40a6440 msgr2=0x7fb5b413c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 shutdown_connections 2024-09-17T15:48:19.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.215+0000 7fb5b8aea640 1 -- 172.21.15.86:0/807989548 wait complete. 2024-09-17T15:48:19.228 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:48:19.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:48:19.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.351+0000 7f7b22b47640 10 monclient: get_monmap_and_config 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.351+0000 7f7b22b47640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.351+0000 7f7b22b47640 10 monclient: monmap: 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:19.354975+0000 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:19.354975+0000 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.350 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:19.351 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:19.351 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:19.351 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:19.351 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:19.351 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.351+0000 7f7b22b47640 1 Processor -- start 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 -- start start 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient: init 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c0a01a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b1c0a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c0a1380 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b1c074e00 con 0x7f7b1c0a0580 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1c09e350 con 0x7f7b1c0a1380 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c09e630 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b22b47640 10 monclient(hunting): authenticate will time out at 2232.320902s 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59062/0 (socket says 172.21.15.86:59062) 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56210/0 (socket says 172.21.15.86:56210) 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 -- 172.21.15.86:0/1599149147 learned_addr learned my addr 172.21.15.86:0/1599149147 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a01a0 auth_method 0 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a1380 auth_method 0 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient(hunting): handle_auth_done global_id 164792 payload 274 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 -- 172.21.15.86:0/1599149147 >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 msgr2=0x7f7b1c13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1599149147 >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 -- 172.21.15.86:0/1599149147 >> v1:172.21.15.203:6789/0 conn(0x7f7b1c0a0580 legacy=0x7f7b1c0a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 monclient: found mon.noname-b 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b208e3640 1 -- 172.21.15.86:0/1599149147 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c09e910 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.355+0000 7f7b210e4640 1 --2- 172.21.15.86:0/1599149147 >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b208e3640 1 -- 172.21.15.86:0/1599149147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 msgr2=0x7f7b1c09f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1599149147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7b18008120 tx=0x7f7b1801c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8993b3f5d9cb3fc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 1 -- 172.21.15.86:0/1599149147 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800b040 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: dump: 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 1 -- 172.21.15.86:0/1599149147 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b18004230 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 1 -- 172.21.15.86:0/1599149147 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18003620 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 5 monclient: authenticate success, global_id 164792 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 4 set_mon_vals no callback set 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b137fe640 10 monclient: dump: 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.360 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 10 monclient: shutdown 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 -- 172.21.15.86:0/1599149147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 msgr2=0x7f7b1c09f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1599149147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7b18008120 tx=0x7f7b1801c6c0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 -- 172.21.15.86:0/1599149147 shutdown_connections 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1599149147 >> v2:172.21.15.86:3300/0 conn(0x7f7b1c0a1380 0x7f7b1c13ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1599149147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a01a0 0x7f7b1c09f010 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 -- 172.21.15.86:0/1599149147 >> 172.21.15.86:0/1599149147 conn(0x7f7b1c0a2f40 msgr2=0x7f7b1c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 -- 172.21.15.86:0/1599149147 shutdown_connections 2024-09-17T15:48:19.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 1 -- 172.21.15.86:0/1599149147 wait complete. 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.359+0000 7f7b22b47640 10 monclient: monmap: 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 Processor -- start 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- start start 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: init 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b1c0a01a0 0x7f7b1c11fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-b con 0x7f7b1c0a01a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-c con 0x7f7b1c0a0580 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c0a1380 0x7f7b1c136cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): picked mon.noname-a con 0x7f7b1c0a1380 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c13e360 con 0x7f7b1c0a1380 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c13e1e0 con 0x7f7b1c0a01a0 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1c13e4e0 con 0x7f7b1c0a0580 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient(hunting): authenticate will time out at 2232.328699s 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c0a1380 0x7f7b1c136cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59066/0 (socket says 172.21.15.86:59066) 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 -- 172.21.15.86:0/1886291751 learned_addr learned my addr 172.21.15.86:0/1886291751 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a0580 auth_method 0 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c0a1380 0x7f7b1c136cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56214/0 (socket says 172.21.15.86:56214) 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): get_auth_request con 0x7f7b1c0a1380 auth_method 0 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b210e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b1c0a01a0 0x7f7b1c11fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient(hunting): handle_auth_done global_id 164795 payload 824 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b1c0a01a0 msgr2=0x7f7b1c11fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b1c0a01a0 0x7f7b1c11fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c0a1380 msgr2=0x7f7b1c136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c0a1380 0x7f7b1c136cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 monclient: found mon.noname-c 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c09d460 con 0x7f7b1c0a0580 2024-09-17T15:48:19.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b208e3640 10 monclient: get_auth_request con 0x7f7b1c0a01a0 auth_method 0 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b1c0a01a0 0x7f7b1c11fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 msgr2=0x7f7b1c11dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b13fff640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00c410 tx=0x7f7b0c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0119a0 con 0x7f7b1c0a0580 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: dump: 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.373 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 5 monclient: authenticate success, global_id 164795 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c017070 con 0x7f7b1c0a0580 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _renew_subs 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c11f2f0 con 0x7f7b1c0a0580 2024-09-17T15:48:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c011d80 con 0x7f7b1c0a0580 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b117fa640 10 monclient: dump: 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.375 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _renew_subs 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c120db0 con 0x7f7b1c0a0580 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.363+0000 7f7b12ffd640 4 set_mon_vals no callback set 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b22b47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 0x7f7b1c1cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c09f010 con 0x7f7b1c137200 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 0x7f7b1c1cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient: get_auth_request con 0x7f7b1c137200 auth_method 0 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 10 monclient(hunting): handle_auth_done global_id 234108 payload 824 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b1c0008e0 con 0x7f7b1c137200 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7b0c01c460 con 0x7f7b1c0a0580 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b0803bbe0 0x7f7b0803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(736..736 src has 696..736) ==== 4863+0+0 (secure 0 0 0) 0x7f7b0c04fd70 con 0x7f7b1c0a0580 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b210e4640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b0803bbe0 0x7f7b0803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b210e4640 10 monclient: get_auth_request con 0x7f7b0803bbe0 auth_method 0 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 msgr2=0x7f7b1c1cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 0x7f7b1c1cb880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b180ae000 tx=0x7f7b18019460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18004320 con 0x7f7b1c137200 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 10 monclient: dump: 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b18004320 con 0x7f7b1c137200 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 msgr2=0x7f7b1c1cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 0x7f7b1c1cb880 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b180ae000 tx=0x7f7b18019460 comp rx=0 tx=0).stop 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 msgr2=0x7f7b1c1cb880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.367+0000 7f7b117fa640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c137200 0x7f7b1c1cb880 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.371+0000 7f7b210e4640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b0803bbe0 0x7f7b0803e0a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400bde0 tx=0x7f7b1400d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b22b47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:19.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 0x7f7b1c1cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b22b47640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1cc710 con 0x7f7b1c1cbdc0 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 0x7f7b1c1cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient: get_auth_request con 0x7f7b1c1cbdc0 auth_method 0 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 10 monclient(hunting): handle_auth_done global_id 234111 payload 824 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.375+0000 7f7b208e3640 1 -- 172.21.15.86:0/1886291751 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b1c11ede0 con 0x7f7b1c1cbdc0 2024-09-17T15:48:19.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b208e3640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 msgr2=0x7f7b1c1cc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b208e3640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 0x7f7b1c1cc1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cf60 tx=0x7f7b180a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b180a69c0 con 0x7f7b1c1cbdc0 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 10 monclient: dump: 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f7b180a69c0 con 0x7f7b1c1cbdc0 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 msgr2=0x7f7b1c1cc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 0x7f7b1c1cc1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cf60 tx=0x7f7b180a9000 comp rx=0 tx=0).stop 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 msgr2=0x7f7b1c1cc1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b117fa640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b1c1cbdc0 0x7f7b1c1cc1d0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cf60 tx=0x7f7b180a9000 comp rx=0 tx=0).stop 2024-09-17T15:48:19.383 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 28575, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:48:19.384 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.386 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:19.387 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.388 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.389 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.390 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.392 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:19.393 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.395 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:19.396 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b0803bbe0 msgr2=0x7f7b0803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b0803bbe0 0x7f7b0803e0a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7b1400bde0 tx=0x7f7b1400d040 comp rx=0 tx=0).stop 2024-09-17T15:48:19.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 10 monclient: shutdown 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 msgr2=0x7f7b1c11dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00c410 tx=0x7f7b0c00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b210e4640 1 -- 172.21.15.86:0/1886291751 reap_dead start 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 shutdown_connections 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 --2- 172.21.15.86:0/1886291751 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b1c0a0580 0x7f7b1c11dd90 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 >> 172.21.15.86:0/1886291751 conn(0x7f7b1c0a2f40 msgr2=0x7f7b1c0729c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 shutdown_connections 2024-09-17T15:48:19.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.379+0000 7f7b22b47640 1 -- 172.21.15.86:0/1886291751 wait complete. 2024-09-17T15:48:19.398 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:19.399 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:48:19.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.511+0000 7fc6b869b640 10 monclient: get_monmap_and_config 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.511+0000 7fc6b869b640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.511+0000 7fc6b869b640 10 monclient: monmap: 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:19.515898+0000 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:19.515898+0000 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:19.512 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:19.512 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:19.512 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:19.512 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:19.512 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 Processor -- start 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 -- start start 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient: init 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6b00a01a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:19.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6b00a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc6b009f010 0x7fc6b013ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-c con 0x7fc6b009f010 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b0074e00 con 0x7fc6b00a01a0 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b00a1a90 con 0x7fc6b00a0580 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b009e350 con 0x7fc6b009f010 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b869b640 10 monclient(hunting): authenticate will time out at 2232.481399s 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6437640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc6b00a01a0 0x7fc6b00a0c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40750/0 (socket says 172.21.15.86:40750) 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6437640 1 -- 172.21.15.86:0/1432880740 learned_addr learned my addr 172.21.15.86:0/1432880740 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6c38640 1 --2- 172.21.15.86:0/1432880740 >> v2:172.21.15.203:3300/0 conn(0x7fc6b009f010 0x7fc6b013ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6c38640 10 monclient(hunting): get_auth_request con 0x7fc6b009f010 auth_method 0 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6c38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6c38640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6c38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.515+0000 7fc6b6437640 1 -- 172.21.15.86:0/1432880740 >> v1:172.21.15.86:6789/0 conn(0x7fc6b00a01a0 legacy=0x7fc6b00a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3774189577 0 0) 0x7fc6b0074e00 con 0x7fc6b00a01a0 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): _negotiate 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): my global_id is 234114 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc69c003530 con 0x7fc6b00a01a0 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1432880740 >> v1:172.21.15.203:6789/0 conn(0x7fc6b00a0580 legacy=0x7fc6b00a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4172832563 0 0) 0x7fc69c003530 con 0x7fc6b00a01a0 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 >> v2:172.21.15.203:3300/0 conn(0x7fc6b009f010 msgr2=0x7fc6b013ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 --2- 172.21.15.86:0/1432880740 >> v2:172.21.15.203:3300/0 conn(0x7fc6b009f010 0x7fc6b013ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 >> v1:172.21.15.203:6789/0 conn(0x7fc6b00a0580 legacy=0x7fc6b00a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b009e630 con 0x7fc6b00a01a0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7fc6ac0019b0 con 0x7fc6b00a01a0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b869b640 5 monclient: authenticate success, global_id 234114 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b869b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: dump: 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:48:19.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 >> v1:172.21.15.86:6789/0 conn(0x7fc6b00a01a0 legacy=0x7fc6b00a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c003b20 0x7fc69c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): picked mon.c con 0x7fc69c003b20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 0x7fc69c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): picked mon.a con 0x7fc69c004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc69c0053d0 0x7fc69c007a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): picked mon.b con 0x7fc69c0053d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b009e350 con 0x7fc69c004810 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c0082f0 con 0x7fc69c0053d0 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc69c0084f0 con 0x7fc69c003b20 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5435640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b869b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 0x7fc69c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c003b20 0x7fc69c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): get_auth_request con 0x7fc69c004810 auth_method 0 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): get_auth_request con 0x7fc69c003b20 auth_method 0 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:19.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_done global_id 234114 payload 293 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c003b20 msgr2=0x7fc69c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c003b20 0x7fc69c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6c38640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc69c0053d0 0x7fc69c007a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1432880740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc69c0053d0 msgr2=0x7fc69c007a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc69c0053d0 0x7fc69c007a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 monclient: found mon.a 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6c38640 1 -- 172.21.15.86:0/1432880740 reap_dead start 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1432880740 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c003530 con 0x7fc69c004810 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b5c36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.519+0000 7fc6b6437640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c003b20 0x7fc69c003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 msgr2=0x7fc69c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 0x7fc69c004c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001990 tx=0x7fc6a401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d351103890bf0314 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40016c0 con 0x7fc69c004810 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: dump: 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.531 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a4007350 con 0x7fc69c004810 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 1 -- 172.21.15.86:0/1432880740 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40057d0 con 0x7fc69c004810 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 4 set_mon_vals no callback set 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5435640 10 monclient: dump: 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.533 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: shutdown 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 msgr2=0x7fc69c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 0x7fc69c004c90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001990 tx=0x7fc6a401c4f0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- 172.21.15.86:0/1432880740 shutdown_connections 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1432880740 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc69c004810 0x7fc69c004c90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- 172.21.15.86:0/1432880740 >> 172.21.15.86:0/1432880740 conn(0x7fc6b00a2f40 msgr2=0x7fc6b00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- 172.21.15.86:0/1432880740 shutdown_connections 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- 172.21.15.86:0/1432880740 wait complete. 2024-09-17T15:48:19.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: monmap: 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 Processor -- start 2024-09-17T15:48:19.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- start start 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: init 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 0x7fc6b01cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-c con 0x7fc69c004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b01cc000 0x7fc6b01224f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-b con 0x7fc6b01cc000 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b01cc460 0x7fc6b0122a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): picked mon.noname-a con 0x7fc6b01cc460 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b00a1000 con 0x7fc6b01cc460 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b00a0e80 con 0x7fc6b01cc000 2024-09-17T15:48:19.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b00a1180 con 0x7fc69c004810 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b869b640 10 monclient(hunting): authenticate will time out at 2232.491543s 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 0x7fc6b01cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 0x7fc6b01cbac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59082/0 (socket says 172.21.15.86:59082) 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 -- 172.21.15.86:0/1462706308 learned_addr learned my addr 172.21.15.86:0/1462706308 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b01cc460 0x7fc6b0122a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): get_auth_request con 0x7fc69c004810 auth_method 0 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): get_auth_request con 0x7fc6b01cc460 auth_method 0 2024-09-17T15:48:19.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6c38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b01cc000 0x7fc6b01224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 10 monclient(hunting): handle_auth_done global_id 164798 payload 824 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b01cc000 msgr2=0x7fc6b01224f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b01cc000 0x7fc6b01224f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b01cc460 msgr2=0x7fc6b0122a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.538 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.523+0000 7fc6b6437640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b01cc460 0x7fc6b0122a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 1 monclient: found mon.noname-c 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b0125ff0 con 0x7fc69c004810 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient: get_auth_request con 0x7fc6b01cc000 auth_method 0 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6c38640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6b01cc460 0x7fc6b0122a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc6b01cc000 0x7fc6b01224f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 msgr2=0x7fc6b01cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b6437640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 0x7fc6b01cbac0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0014e0 tx=0x7fc6ac007720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00c0e0 con 0x7fc69c004810 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:19.539 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: dump: 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 5 monclient: authenticate success, global_id 164798 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 monclient: _renew_subs 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ac018070 con 0x7fc69c004810 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b0073b70 con 0x7fc69c004810 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0133f0 con 0x7fc69c004810 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 10 monclient: dump: 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.541 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b4c34640 4 set_mon_vals no callback set 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 monclient: _renew_subs 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b0125d10 con 0x7fc69c004810 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 0x7fc6b002d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b00a1510 con 0x7fc6b002d450 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc6ac013610 con 0x7fc69c004810 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 0x7fc6b002d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient: get_auth_request con 0x7fc6b002d450 auth_method 0 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(736..736 src has 696..736) ==== 4863+0+0 (secure 0 0 0) 0x7fc6ac04ff60 con 0x7fc69c004810 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_done global_id 164801 payload 824 2024-09-17T15:48:19.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.527+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6b011efe0 con 0x7fc6b002d450 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6b6c38640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6b6c38640 10 monclient: get_auth_request con 0x7fc69403bb60 auth_method 0 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 msgr2=0x7fc6b002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 0x7fc6b002d830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009000 tx=0x7fc69c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40bc020 con 0x7fc6b002d450 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 10 monclient: dump: 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.544 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6a4007690 con 0x7fc6b002d450 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 msgr2=0x7fc6b002d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 0x7fc6b002d830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009000 tx=0x7fc69c005840 comp rx=0 tx=0).stop 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 msgr2=0x7fc6b002d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6a2ffd640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002d450 0x7fc6b002d830 secure :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009000 tx=0x7fc69c005840 comp rx=0 tx=0).stop 2024-09-17T15:48:19.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.531+0000 7fc6b6c38640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc6b006e1a0 tx=0x7fc6a800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 0x7fc6b002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b01247a0 con 0x7fc6b002dd70 2024-09-17T15:48:19.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 0x7fc6b002e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient: get_auth_request con 0x7fc6b002dd70 auth_method 0 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 10 monclient(hunting): handle_auth_done global_id 164804 payload 824 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1462706308 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6b0124ad0 con 0x7fc6b002dd70 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 msgr2=0x7fc6b002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b5c36640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 0x7fc6b002e180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001120 tx=0x7fc6a4007c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40bc030 con 0x7fc6b002dd70 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 10 monclient: dump: 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fc6a40c3020 con 0x7fc6b002dd70 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 msgr2=0x7fc6b002e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 0x7fc6b002e180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001120 tx=0x7fc6a4007c30 comp rx=0 tx=0).stop 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 msgr2=0x7fc6b002e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6a2ffd640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6b002dd70 0x7fc6b002e180 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001120 tx=0x7fc6a4007c30 comp rx=0 tx=0).stop 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:48:19.550 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 402727, 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.551 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.552 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:19.553 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:19.554 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.555 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.558 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.560 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.561 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.562 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:19.563 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 msgr2=0x7fc69403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc69403bb60 0x7fc69403e020 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc6b006e1a0 tx=0x7fc6a800b040 comp rx=0 tx=0).stop 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 10 monclient: shutdown 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 msgr2=0x7fc6b01cbac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.539+0000 7fc6b869b640 1 --2- 172.21.15.86:0/1462706308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc69c004810 0x7fc6b01cbac0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0014e0 tx=0x7fc6ac007720 comp rx=0 tx=0).stop 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b6c38640 1 -- 172.21.15.86:0/1462706308 reap_dead start 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b6c38640 1 -- 172.21.15.86:0/1462706308 reap_dead start 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 shutdown_connections 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 >> 172.21.15.86:0/1462706308 conn(0x7fc6b00a2f40 msgr2=0x7fc6b00740a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 shutdown_connections 2024-09-17T15:48:19.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.543+0000 7fc6b869b640 1 -- 172.21.15.86:0/1462706308 wait complete. 2024-09-17T15:48:19.565 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:19.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:48:19.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: get_monmap_and_config 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: monmap: 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:19.677158+0000 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:19.677158+0000 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.672 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:19.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 Processor -- start 2024-09-17T15:48:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 -- start start 2024-09-17T15:48:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: init 2024-09-17T15:48:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f379009d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f379009e260 0x7f3790139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): picked mon.noname-c con 0x7f379009e260 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): picked mon.noname-b con 0x7f379013a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f379009ff00 con 0x7f379009d770 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f379009e7f0 con 0x7f379009e260 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f37900a2110 con 0x7f379013a440 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3796866640 10 monclient(hunting): authenticate will time out at 2232.642706s 2024-09-17T15:48:19.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59098/0 (socket says 172.21.15.86:59098) 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 -- 172.21.15.86:0/4062243607 learned_addr learned my addr 172.21.15.86:0/4062243607 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): get_auth_request con 0x7f379013a440 auth_method 0 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f378f7fe640 1 --2- 172.21.15.86:0/4062243607 >> v2:172.21.15.203:3300/0 conn(0x7f379009e260 0x7f3790139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f378f7fe640 10 monclient(hunting): get_auth_request con 0x7f379009e260 auth_method 0 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f378f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f378f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f378f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient(hunting): handle_auth_done global_id 164807 payload 274 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 -- 172.21.15.86:0/4062243607 >> v2:172.21.15.203:3300/0 conn(0x7f379009e260 msgr2=0x7f3790139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 --2- 172.21.15.86:0/4062243607 >> v2:172.21.15.203:3300/0 conn(0x7f379009e260 0x7f3790139f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 -- 172.21.15.86:0/4062243607 >> v1:172.21.15.203:6789/0 conn(0x7f379009d770 legacy=0x7f379009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 monclient: found mon.noname-b 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.675+0000 7f3794e03640 1 -- 172.21.15.86:0/4062243607 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379009ec30 con 0x7f379013a440 2024-09-17T15:48:19.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f3794e03640 1 -- 172.21.15.86:0/4062243607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 msgr2=0x7f379013c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f3794e03640 1 --2- 172.21.15.86:0/4062243607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f378000d150 tx=0x7f378001eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5ec0402fc4f2e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 1 -- 172.21.15.86:0/4062243607 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378000b2d0 con 0x7f379013a440 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: dump: 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.686 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f3796866640 5 monclient: authenticate success, global_id 164807 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 1 -- 172.21.15.86:0/4062243607 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378000b470 con 0x7f379013a440 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f3796866640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.679+0000 7f378effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 4 set_mon_vals no callback set 2024-09-17T15:48:19.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378effd640 1 -- 172.21.15.86:0/4062243607 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37800a78d0 con 0x7f379013a440 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378effd640 10 monclient: dump: 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: shutdown 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- 172.21.15.86:0/4062243607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 msgr2=0x7f379013c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- 172.21.15.86:0/4062243607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f378000d150 tx=0x7f378001eaf0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- 172.21.15.86:0/4062243607 shutdown_connections 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- 172.21.15.86:0/4062243607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379013a440 0x7f379013c830 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- 172.21.15.86:0/4062243607 >> v2:172.21.15.203:3300/0 conn(0x7f379009e260 0x7f3790139f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- 172.21.15.86:0/4062243607 >> 172.21.15.86:0/4062243607 conn(0x7f37900a2f40 msgr2=0x7f37900a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- 172.21.15.86:0/4062243607 shutdown_connections 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- 172.21.15.86:0/4062243607 wait complete. 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: monmap: 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.689 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 Processor -- start 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- start start 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: init 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:48:19.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f379009d770 0x7f3790134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): picked mon.noname-a con 0x7f379009d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 0x7f379011e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): picked mon.noname-c con 0x7f379009e260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379013a440 0x7f379011eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): picked mon.noname-b con 0x7f379013a440 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f37900a1850 con 0x7f379009d770 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37900a16d0 con 0x7f379013a440 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f37900a19d0 con 0x7f379009e260 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3796866640 10 monclient(hunting): authenticate will time out at 2232.649966s 2024-09-17T15:48:19.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 0x7f379011e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f379009d770 0x7f3790134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 0x7f379011e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59104/0 (socket says 172.21.15.86:59104) 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 -- 172.21.15.86:0/2507979808 learned_addr learned my addr 172.21.15.86:0/2507979808 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): get_auth_request con 0x7f379009e260 auth_method 0 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): get_auth_request con 0x7f379009d770 auth_method 0 2024-09-17T15:48:19.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3794e03640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379013a440 0x7f379011eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3794e03640 10 monclient(hunting): get_auth_request con 0x7f379013a440 auth_method 0 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3794e03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3794e03640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f3794e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient(hunting): handle_auth_done global_id 164810 payload 824 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379013a440 msgr2=0x7f379011eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379013a440 0x7f379011eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f379009d770 msgr2=0x7f3790134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f379009d770 0x7f3790134ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378f7fe640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f379009d460 con 0x7f379009e260 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.683+0000 7f378ffff640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f379009d770 0x7f3790134ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378f7fe640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 msgr2=0x7f379011e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378f7fe640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 0x7f379011e5e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f378800d800 tx=0x7f378800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378800b8d0 con 0x7f379009e260 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: dump: 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.695 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 5 monclient: authenticate success, global_id 164810 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 monclient: _renew_subs 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3788019070 con 0x7f379009e260 2024-09-17T15:48:19.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37901200a0 con 0x7f379009e260 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378800bcf0 con 0x7f379009e260 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 10 monclient: dump: 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.697 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378e7fc640 4 set_mon_vals no callback set 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 monclient: _renew_subs 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37901351e0 con 0x7f379009e260 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 0x7f379011f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37901d0f70 con 0x7f3790120c00 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f378801e460 con 0x7f379009e260 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f377003bb90 0x7f377003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(736..736 src has 696..736) ==== 4863+0+0 (secure 0 0 0) 0x7f37880504d0 con 0x7f379009e260 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 0x7f379011f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient: get_auth_request con 0x7f3790120c00 auth_method 0 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3794e03640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f377003bb90 0x7f377003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f3794e03640 10 monclient: get_auth_request con 0x7f377003bb90 auth_method 0 2024-09-17T15:48:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 10 monclient(hunting): handle_auth_done global_id 224182 payload 824 2024-09-17T15:48:19.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.687+0000 7f378ffff640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f379013a920 con 0x7f3790120c00 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378ffff640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 msgr2=0x7f379011f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378ffff640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 0x7f379011f060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778009e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f3794e03640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f377003bb90 0x7f377003e050 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f378000d150 tx=0x7f37800a9070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778017070 con 0x7f3790120c00 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 10 monclient: dump: 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.702 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f377800cd30 con 0x7f3790120c00 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 msgr2=0x7f379011f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 0x7f379011f060 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778009e70 comp rx=0 tx=0).stop 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 msgr2=0x7f379011f060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.691+0000 7f378cff9640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3790120c00 0x7f379011f060 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778009e70 comp rx=0 tx=0).stop 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f3796866640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f3796866640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 0x7f37900716f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f3796866640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f37901c5430 con 0x7f379011f5a0 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 0x7f37900716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient: get_auth_request con 0x7f379011f5a0 auth_method 0 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 10 monclient(hunting): handle_auth_done global_id 224185 payload 824 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.699+0000 7f378ffff640 1 -- 172.21.15.86:0/2507979808 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3790135410 con 0x7f379011f5a0 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378ffff640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 msgr2=0x7f37900716f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378ffff640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 0x7f37900716f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778017040 con 0x7f379011f5a0 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 10 monclient: dump: 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.705 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f3778003900 con 0x7f379011f5a0 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 msgr2=0x7f37900716f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 0x7f37900716f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778004700 comp rx=0 tx=0).stop 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 msgr2=0x7f37900716f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:19.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f378cff9640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f379011f5a0 0x7f37900716f0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f377800eb10 tx=0x7f3778004700 comp rx=0 tx=0).stop 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 90153, 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.707 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.708 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.709 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:19.710 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:19.711 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.712 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.713 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:19.714 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.715 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.716 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:19.717 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.718 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:19.719 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:19.720 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f377003bb90 msgr2=0x7f377003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f377003bb90 0x7f377003e050 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f378000d150 tx=0x7f37800a9070 comp rx=0 tx=0).stop 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 10 monclient: shutdown 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 msgr2=0x7f379011e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 --2- 172.21.15.86:0/2507979808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f379009e260 0x7f379011e5e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f378800d800 tx=0x7f378800dcc0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3794e03640 1 -- 172.21.15.86:0/2507979808 reap_dead start 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3794e03640 1 -- 172.21.15.86:0/2507979808 reap_dead start 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 shutdown_connections 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 >> 172.21.15.86:0/2507979808 conn(0x7f37900a2f40 msgr2=0x7f37901c3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 shutdown_connections 2024-09-17T15:48:19.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.703+0000 7f3796866640 1 -- 172.21.15.86:0/2507979808 wait complete. 2024-09-17T15:48:19.723 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:19.723 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:48:19.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:48:19.810 INFO:tasks.rados.rados.0.smithi203.stdout:3344: done (2 left) 2024-09-17T15:48:19.811 INFO:tasks.rados.rados.0.smithi203.stdout:3345: done (1 left) 2024-09-17T15:48:19.811 INFO:tasks.rados.rados.0.smithi203.stdout:3346: done (0 left) 2024-09-17T15:48:19.811 INFO:tasks.rados.rados.0.smithi203.stdout:3347: delete oid 3 current snap is 360 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3347: done (0 left) 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3348: read oid 33 snap 358 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3348: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3349: read oid 37 snap -1 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3349: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:48:19.823 INFO:tasks.rados.rados.0.smithi203.stdout:3350: delete oid 30 current snap is 360 2024-09-17T15:48:19.834 INFO:tasks.rados.rados.0.smithi203.stdout:3349: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.839+0000 7ff9d93f3640 10 monclient: get_monmap_and_config 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.839+0000 7ff9d93f3640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.839+0000 7ff9d93f3640 10 monclient: monmap: 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:19.842894+0000 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:19.842894+0000 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:19.842 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 Processor -- start 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 -- start start 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient: init 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff9d40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d40a2dc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9d40a31a0 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:19.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d40a4650 con 0x7ff9d40689d0 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d409dc00 con 0x7ff9d40a31a0 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d40a41d0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d93f3640 10 monclient(hunting): authenticate will time out at 2232.808784s 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59118/0 (socket says 172.21.15.86:59118) 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2670672205 learned_addr learned my addr 172.21.15.86:0/2670672205 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): get_auth_request con 0x7ff9d40a2dc0 auth_method 0 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d2ffd640 1 --1- 172.21.15.86:0/2670672205 >> v1:172.21.15.86:6789/0 conn(0x7ff9d40689d0 0x7ff9d40a26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40756/0 (socket says 172.21.15.86:40756) 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_done global_id 164813 payload 274 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2670672205 >> v1:172.21.15.203:6789/0 conn(0x7ff9d40a31a0 legacy=0x7ff9d413c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2670672205 >> v1:172.21.15.86:6789/0 conn(0x7ff9d40689d0 legacy=0x7ff9d40a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 monclient: found mon.noname-b 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.843+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2670672205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d409e2d0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2670672205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 msgr2=0x7ff9d413a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 1 --2- 172.21.15.86:0/2670672205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800bb40 tx=0x7ff9c801eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b9c581d5c49e52f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 1 -- 172.21.15.86:0/2670672205 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80b4070 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: dump: 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.851 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 5 monclient: authenticate success, global_id 164813 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 1 -- 172.21.15.86:0/2670672205 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c80a79f0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 1 -- 172.21.15.86:0/2670672205 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80a7dc0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d1ffb640 10 monclient: dump: 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 4 set_mon_vals no callback set 2024-09-17T15:48:19.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: shutdown 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2670672205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 msgr2=0x7ff9d413a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2670672205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800bb40 tx=0x7ff9c801eb10 comp rx=0 tx=0).stop 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2670672205 shutdown_connections 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2670672205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d413a4a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2670672205 >> 172.21.15.86:0/2670672205 conn(0x7ff9d40a72a0 msgr2=0x7ff9d40a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2670672205 shutdown_connections 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2670672205 wait complete. 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: build_initial_monmap 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: monmap: 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:19.854 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 Processor -- start 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- start start 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: init 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 0x7ff9d41d0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9d40689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-c con 0x7ff9d40a2dc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9d40a31a0 0x7ff9d4134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): picked mon.noname-b con 0x7ff9d40a31a0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d409c400 con 0x7ff9d40689d0 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d409c280 con 0x7ff9d40a31a0 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d409c580 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d93f3640 10 monclient(hunting): authenticate will time out at 2232.815684s 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 0x7ff9d41d0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59128/0 (socket says 172.21.15.86:59128) 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2964153662 learned_addr learned my addr 172.21.15.86:0/2964153662 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): get_auth_request con 0x7ff9d40a2dc0 auth_method 0 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): get_auth_request con 0x7ff9d40689d0 auth_method 0 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.847+0000 7ff9d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d37fe640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9d40a31a0 0x7ff9d4134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request con 0x7ff9d40a31a0 auth_method 0 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 10 monclient(hunting): handle_auth_done global_id 164816 payload 824 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9d40a31a0 msgr2=0x7ff9d4134e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9d40a31a0 0x7ff9d4134e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 msgr2=0x7ff9d41d0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 0x7ff9d41d0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 monclient: found mon.noname-c 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2964153662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d409f2c0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d2ffd640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 0x7ff9d41d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 -- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 msgr2=0x7ff9d41348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d27fc640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800bb40 tx=0x7ff9c801d350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80bf070 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:19.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: dump: 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:19.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 5 monclient: authenticate success, global_id 164816 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c80afc40 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _renew_subs 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d406f060 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80b4070 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 10 monclient: dump: 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.861 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d17fa640 4 set_mon_vals no callback set 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _renew_subs 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d413c3f0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:19.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d40a26b0 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff9c80c3430 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.851+0000 7ff9bb7fe640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 0x7ff9c003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.855+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(737..737 src has 696..737) ==== 4835+0+0 (secure 0 0 0) 0x7ff9c80ea050 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.855+0000 7ff9d2ffd640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 0x7ff9c003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.855+0000 7ff9d2ffd640 10 monclient: get_auth_request con 0x7ff9c003bb90 auth_method 0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.855+0000 7ff9d2ffd640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 0x7ff9c003e050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff9d413dbb0 tx=0x7ff9c4004fc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:19.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.859+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff9d406f060 con 0x7ff9d40a2dc0 2024-09-17T15:48:19.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.859+0000 7ff9bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:19.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:19.859+0000 7ff9bb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:19.955 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1838722 2024-09-17T15:48:20.015 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2981031 2024-09-17T15:48:20.021 INFO:tasks.rados.rados.0.smithi203.stdout:3348: expect (ObjNum 551 snap 161 seq_num 551) 2024-09-17T15:48:20.059 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:20.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.091+0000 7ff9d93f3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:48:20.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.091+0000 7ff9d93f3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:48:20.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.091+0000 7ff9d93f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:20.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.091+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff9d4121700 con 0x7ff9d40a2dc0 2024-09-17T15:48:20.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9bb7fe640 1 -- 172.21.15.86:0/2964153662 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff9c8018820 con 0x7ff9d40a2dc0 2024-09-17T15:48:20.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:48:20.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9bb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 msgr2=0x7ff9c003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 0x7ff9c003e050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff9d413dbb0 tx=0x7ff9c4004fc0 comp rx=0 tx=0).stop 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 10 monclient: shutdown 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 msgr2=0x7ff9d41348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff9c800bb40 tx=0x7ff9c801d350 comp rx=0 tx=0).stop 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 shutdown_connections 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff9c003bb90 0x7ff9c003e050 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff9d40a31a0 0x7ff9d4134e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:20.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9d40a2dc0 0x7ff9d41348e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:20.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 --2- 172.21.15.86:0/2964153662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9d40689d0 0x7ff9d41d0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:20.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 >> 172.21.15.86:0/2964153662 conn(0x7ff9d40a72a0 msgr2=0x7ff9d40a4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:20.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 shutdown_connections 2024-09-17T15:48:20.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:20.095+0000 7ff9d93f3640 1 -- 172.21.15.86:0/2964153662 wait complete. 2024-09-17T15:48:20.105 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:20.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:48:20.151 INFO:tasks.rados.rados.0.smithi203.stdout:3348: done (2 left) 2024-09-17T15:48:20.151 INFO:tasks.rados.rados.0.smithi203.stdout:3349: done (1 left) 2024-09-17T15:48:20.151 INFO:tasks.rados.rados.0.smithi203.stdout:3350: done (0 left) 2024-09-17T15:48:20.151 INFO:tasks.rados.rados.0.smithi203.stdout:3351: write (excl) oid 50 current snap is 360 2024-09-17T15:48:20.151 INFO:tasks.rados.rados.0.smithi203.stdout:3351: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-17T15:48:20.169 INFO:tasks.rados.rados.0.smithi203.stdout:3351: writing smithi20328086-50 from 514611 to 1100050 tid 1 2024-09-17T15:48:20.180 INFO:tasks.rados.rados.0.smithi203.stdout:3351: writing smithi20328086-50 from 1524437 to 2116832 tid 2 2024-09-17T15:48:20.199 INFO:tasks.rados.rados.0.smithi203.stdout:3351: writing smithi20328086-50 from 2895994 to 3304418 tid 3 2024-09-17T15:48:20.199 INFO:tasks.rados.rados.0.smithi203.stdout:3352: read oid 8 snap -1 2024-09-17T15:48:20.199 INFO:tasks.rados.rados.0.smithi203.stdout:3352: expect (ObjNum 1157 snap 357 seq_num 1157) 2024-09-17T15:48:20.199 INFO:tasks.rados.rados.0.smithi203.stdout:3353: write oid 49 current snap is 360 2024-09-17T15:48:20.199 INFO:tasks.rados.rados.0.smithi203.stdout:3353: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-17T15:48:20.213 INFO:tasks.rados.rados.0.smithi203.stdout:3353: writing smithi20328086-49 from 531418 to 1192106 tid 1 2024-09-17T15:48:20.225 INFO:tasks.rados.rados.0.smithi203.stdout:3353: writing smithi20328086-49 from 1982919 to 2435325 tid 2 2024-09-17T15:48:20.236 INFO:tasks.rados.rados.0.smithi203.stdout:3353: writing smithi20328086-49 from 3039770 to 3731418 tid 3 2024-09-17T15:48:20.236 INFO:tasks.rados.rados.0.smithi203.stdout:3354: read oid 31 snap 355 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3354: expect deleted 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3355: write oid 1 current snap is 360 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: finishing write tid 1 to smithi20328086-50 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: finishing write tid 2 to smithi20328086-50 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: finishing write tid 3 to smithi20328086-50 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: finishing write tid 4 to smithi20328086-50 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: finishing write tid 5 to smithi20328086-50 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: oid 50 updating version 0 to 1708 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: oid 50 updating version 1708 to 1709 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: oid 50 updating version 1709 to 1711 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: oid 50 version 1711 is already newer than 1710 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1711 (ObjNum 1173 snap 360 seq_num 1173) dirty exists 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3351: left oid 50 (ObjNum 1173 snap 360 seq_num 1173) 2024-09-17T15:48:20.237 INFO:tasks.rados.rados.0.smithi203.stdout:3352: expect (ObjNum 1157 snap 357 seq_num 1157) 2024-09-17T15:48:20.238 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:20.268 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=4095889108,config=3541143443,health=2171044315,kv=3591778543,logm=841876297}) 2024-09-17T15:48:20.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=1,health=11,kv=11,logm=48} crc {auth=1796052787,config=555582061,health=2622424524,kv=3591778543,logm=1619697623}) 2024-09-17T15:48:20.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=10,kv=11,logm=51} crc {auth=4095889108,config=3541143443,health=2171044315,kv=3591778543,logm=841876297}) 2024-09-17T15:48:20.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.271+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=3696177982,config=3541143443,health=3219210987,kv=3591778543,logm=2195696588}) 2024-09-17T15:48:20.303 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:20.425 INFO:tasks.rados.rados.0.smithi203.stdout:3353: finishing write tid 1 to smithi20328086-49 2024-09-17T15:48:20.425 INFO:tasks.rados.rados.0.smithi203.stdout:3353: finishing write tid 2 to smithi20328086-49 2024-09-17T15:48:20.425 INFO:tasks.rados.rados.0.smithi203.stdout:3353: finishing write tid 3 to smithi20328086-49 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: finishing write tid 4 to smithi20328086-49 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: finishing write tid 5 to smithi20328086-49 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: oid 49 updating version 0 to 1736 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: oid 49 updating version 1736 to 1738 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: oid 49 version 1738 is already newer than 1735 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: oid 49 version 1738 is already newer than 1737 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1738 (ObjNum 1174 snap 360 seq_num 1174) dirty exists 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3353: left oid 49 (ObjNum 1174 snap 360 seq_num 1174) 2024-09-17T15:48:20.426 INFO:tasks.rados.rados.0.smithi203.stdout:3355: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2024-09-17T15:48:20.436 INFO:tasks.rados.rados.0.smithi203.stdout:3355: writing smithi20328086-1 from 548225 to 1284162 tid 1 2024-09-17T15:48:20.447 INFO:tasks.rados.rados.0.smithi203.stdout:3355: writing smithi20328086-1 from 1925048 to 2521112 tid 2 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3355: writing smithi20328086-1 from 2950840 to 3498161 tid 3 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3355: finishing write tid 1 to smithi20328086-1 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3355: finishing write tid 2 to smithi20328086-1 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3351: done (4 left) 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3352: done (3 left) 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3353: done (2 left) 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3354: done (1 left) 2024-09-17T15:48:20.456 INFO:tasks.rados.rados.0.smithi203.stdout:3356: delete oid 39 current snap is 360 2024-09-17T15:48:20.460 INFO:tasks.rados.rados.0.smithi203.stdout:3356: done (1 left) 2024-09-17T15:48:20.460 INFO:tasks.rados.rados.0.smithi203.stdout:3357: read oid 38 snap -1 2024-09-17T15:48:20.460 INFO:tasks.rados.rados.0.smithi203.stdout:3357: expect deleted 2024-09-17T15:48:20.460 INFO:tasks.rados.rados.0.smithi203.stdout:3358: write (excl) oid 40 current snap is 360 2024-09-17T15:48:20.460 INFO:tasks.rados.rados.0.smithi203.stdout:3358: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-17T15:48:20.466 INFO:tasks.rados.rados.0.smithi203.stdout:3358: writing smithi20328086-40 from 581839 to 1068274 tid 1 2024-09-17T15:48:20.473 INFO:tasks.rados.rados.0.smithi203.stdout:3358: writing smithi20328086-40 from 1642012 to 2241745 tid 2 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3358: writing smithi20328086-40 from 3005686 to 3575551 tid 3 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3359: copy_from oid 49 from oid 37 current snap is 360 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: finishing write tid 3 to smithi20328086-1 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: finishing write tid 4 to smithi20328086-1 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: finishing write tid 5 to smithi20328086-1 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: oid 1 updating version 0 to 1550 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: oid 1 updating version 1550 to 1551 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: oid 1 version 1551 is already newer than 1549 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:3355: oid 1 updating version 1551 to 1552 2024-09-17T15:48:20.481 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1552 (ObjNum 1175 snap 360 seq_num 1175) dirty exists 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3355: left oid 1 (ObjNum 1175 snap 360 seq_num 1175) 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3355: done (3 left) 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3357: done (2 left) 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3360: rollback oid 32 current snap is 360 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 359 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3358: finishing write tid 1 to smithi20328086-40 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3358: finishing write tid 2 to smithi20328086-40 2024-09-17T15:48:20.482 INFO:tasks.rados.rados.0.smithi203.stdout:3361: snap_create 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567756671}) 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1863991491}) 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1567756671}) 2024-09-17T15:48:20.483 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628600073}) 2024-09-17T15:48:20.492 INFO:tasks.rados.rados.0.smithi203.stdout:3358: finishing write tid 3 to smithi20328086-40 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: finishing write tid 4 to smithi20328086-40 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: finishing write tid 5 to smithi20328086-40 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: oid 40 updating version 0 to 1547 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: oid 40 updating version 1547 to 1550 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: oid 40 version 1550 is already newer than 1548 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:3358: oid 40 version 1550 is already newer than 1549 2024-09-17T15:48:20.493 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1550 (ObjNum 1177 snap 360 seq_num 1177) dirty exists 2024-09-17T15:48:20.494 INFO:tasks.rados.rados.0.smithi203.stdout:3358: left oid 40 (ObjNum 1177 snap 360 seq_num 1177) 2024-09-17T15:48:20.499 INFO:tasks.rados.rados.0.smithi203.stdout:3359: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:48:20.499 INFO:tasks.rados.rados.0.smithi203.stdout:3360: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:48:20.500 INFO:tasks.rados.rados.0.smithi203.stdout:3360: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:48:20.500 INFO:tasks.rados.rados.0.smithi203.stdout:3359: finishing copy_from to smithi20328086-49 2024-09-17T15:48:20.500 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1634 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:48:20.501 INFO:tasks.rados.rados.0.smithi203.stdout:3360: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:48:20.501 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1714 (ObjNum 1156 snap 357 seq_num 1156) dirty exists 2024-09-17T15:48:20.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354828423}) 2024-09-17T15:48:20.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193129495}) 2024-09-17T15:48:20.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354828423}) 2024-09-17T15:48:20.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452518974}) 2024-09-17T15:48:20.746 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191934783}) 2024-09-17T15:48:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645618992}) 2024-09-17T15:48:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191934783}) 2024-09-17T15:48:20.747 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.747+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608657122}) 2024-09-17T15:48:20.749 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662487969}) 2024-09-17T15:48:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3961152829}) 2024-09-17T15:48:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662487969}) 2024-09-17T15:48:20.750 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.751+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90748693}) 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1845659303}) 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570912453}) 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1845659303}) 2024-09-17T15:48:20.752 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055115171}) 2024-09-17T15:48:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050978530}) 2024-09-17T15:48:20.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942408438}) 2024-09-17T15:48:20.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050978530}) 2024-09-17T15:48:20.755 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.755+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525676007}) 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741279979}) 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626542157}) 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741279979}) 2024-09-17T15:48:20.757 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.759+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449689006}) 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863960657}) 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226875703}) 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863960657}) 2024-09-17T15:48:20.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658450784}) 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166372384}) 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793026683}) 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166372384}) 2024-09-17T15:48:20.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.763+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014503312}) 2024-09-17T15:48:20.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755574661}) 2024-09-17T15:48:20.763 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555508016}) 2024-09-17T15:48:20.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755574661}) 2024-09-17T15:48:20.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1132242284}) 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229055608}) 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286526054}) 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1229055608}) 2024-09-17T15:48:20.766 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.767+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458304784}) 2024-09-17T15:48:20.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397389042}) 2024-09-17T15:48:20.769 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054391380}) 2024-09-17T15:48:20.769 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.769 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397389042}) 2024-09-17T15:48:20.769 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53522842}) 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404741104}) 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1776507475}) 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404741104}) 2024-09-17T15:48:20.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.771+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556449693}) 2024-09-17T15:48:20.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870036400}) 2024-09-17T15:48:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244698222}) 2024-09-17T15:48:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870036400}) 2024-09-17T15:48:20.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3945976822}) 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441230689}) 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1799483382}) 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.775+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1441230689}) 2024-09-17T15:48:20.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3130730139}) 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426368443}) 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1266860993}) 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426368443}) 2024-09-17T15:48:20.777 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.779+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583105513}) 2024-09-17T15:48:20.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776603854}) 2024-09-17T15:48:20.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602244713}) 2024-09-17T15:48:20.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.779 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776603854}) 2024-09-17T15:48:20.780 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2716461833}) 2024-09-17T15:48:20.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230522816}) 2024-09-17T15:48:20.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3677333073}) 2024-09-17T15:48:20.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230522816}) 2024-09-17T15:48:20.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.783+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995896434}) 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060450875}) 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823492858}) 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2060450875}) 2024-09-17T15:48:20.784 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904628643}) 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213597193}) 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3431774784}) 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=213597193}) 2024-09-17T15:48:20.786 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.787+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909359518}) 2024-09-17T15:48:20.788 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.788 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030475429}) 2024-09-17T15:48:20.788 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698105883}) 2024-09-17T15:48:20.788 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4030475429}) 2024-09-17T15:48:20.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=196565883}) 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568948494}) 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=568903905}) 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568948494}) 2024-09-17T15:48:20.791 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.791+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433243928}) 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389764067}) 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893653433}) 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389764067}) 2024-09-17T15:48:20.793 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.795+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=301576377}) 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3672956208}) 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969914547}) 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3672956208}) 2024-09-17T15:48:20.795 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706520322}) 2024-09-17T15:48:20.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140889660}) 2024-09-17T15:48:20.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308271087}) 2024-09-17T15:48:20.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140889660}) 2024-09-17T15:48:20.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.799+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167856267}) 2024-09-17T15:48:20.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382283058}) 2024-09-17T15:48:20.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177403314}) 2024-09-17T15:48:20.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382283058}) 2024-09-17T15:48:20.800 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980569147}) 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755491373}) 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509580581}) 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755491373}) 2024-09-17T15:48:20.802 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.803+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892184035}) 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526194086}) 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037613152}) 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526194086}) 2024-09-17T15:48:20.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042376291}) 2024-09-17T15:48:20.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551260672}) 2024-09-17T15:48:20.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=996176351}) 2024-09-17T15:48:20.806 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551260672}) 2024-09-17T15:48:20.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.807+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4017141935}) 2024-09-17T15:48:20.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192126967}) 2024-09-17T15:48:20.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4103708005}) 2024-09-17T15:48:20.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192126967}) 2024-09-17T15:48:20.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715533793}) 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.811+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251673911}) 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866099111}) 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251673911}) 2024-09-17T15:48:20.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1710461236}) 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103993129}) 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596410809}) 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103993129}) 2024-09-17T15:48:20.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.815+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224780687}) 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250406319}) 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140518519}) 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250406319}) 2024-09-17T15:48:20.818 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.819+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12165703}) 2024-09-17T15:48:20.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486085576}) 2024-09-17T15:48:20.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186375313}) 2024-09-17T15:48:20.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486085576}) 2024-09-17T15:48:20.826 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3030006377}) 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309814379}) 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2473391685}) 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309814379}) 2024-09-17T15:48:20.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894586692}) 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839979814}) 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518767486}) 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839979814}) 2024-09-17T15:48:20.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942368104}) 2024-09-17T15:48:20.848 INFO:tasks.rados.rados.0.smithi203.stdout:3358: done (3 left) 2024-09-17T15:48:20.848 INFO:tasks.rados.rados.0.smithi203.stdout:3359: done (2 left) 2024-09-17T15:48:20.848 INFO:tasks.rados.rados.0.smithi203.stdout:3360: done (1 left) 2024-09-17T15:48:20.848 INFO:tasks.rados.rados.0.smithi203.stdout:3361: done (0 left) 2024-09-17T15:48:20.848 INFO:tasks.rados.rados.0.smithi203.stdout:3362: delete oid 26 current snap is 361 2024-09-17T15:48:20.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398543849}) 2024-09-17T15:48:20.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2183595880}) 2024-09-17T15:48:20.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398543849}) 2024-09-17T15:48:20.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506907036}) 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361013450}) 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955303584}) 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3361013450}) 2024-09-17T15:48:20.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1119443218}) 2024-09-17T15:48:20.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146901583}) 2024-09-17T15:48:20.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823610680}) 2024-09-17T15:48:20.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2146901583}) 2024-09-17T15:48:20.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3002657421}) 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216273133}) 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21854862}) 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216273133}) 2024-09-17T15:48:20.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.859+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2193411113}) 2024-09-17T15:48:20.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210502273}) 2024-09-17T15:48:20.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1791436090}) 2024-09-17T15:48:20.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210502273}) 2024-09-17T15:48:20.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.863+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258786549}) 2024-09-17T15:48:20.861 INFO:tasks.rados.rados.0.smithi203.stdout:3362: done (0 left) 2024-09-17T15:48:20.861 INFO:tasks.rados.rados.0.smithi203.stdout:3363: read oid 41 snap -1 2024-09-17T15:48:20.861 INFO:tasks.rados.rados.0.smithi203.stdout:3363: expect (ObjNum 1162 snap 358 seq_num 1162) 2024-09-17T15:48:20.862 INFO:tasks.rados.rados.0.smithi203.stdout:3364: rollback oid 38 current snap is 361 2024-09-17T15:48:20.862 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 359 2024-09-17T15:48:20.862 INFO:tasks.rados.rados.0.smithi203.stdout:3365: delete oid 40 current snap is 361 2024-09-17T15:48:20.862 INFO:tasks.rados.rados.0.smithi203.stdout:3364: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:48:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:48:20.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4224873574}) 2024-09-17T15:48:20.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=480921293}) 2024-09-17T15:48:20.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4224873574}) 2024-09-17T15:48:20.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396143014}) 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249177799}) 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032299645}) 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249177799}) 2024-09-17T15:48:20.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.867+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294475313}) 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086430987}) 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1153264895}) 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086430987}) 2024-09-17T15:48:20.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980742047}) 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772557301}) 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148940394}) 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772557301}) 2024-09-17T15:48:20.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494496057}) 2024-09-17T15:48:20.872 INFO:tasks.rados.rados.0.smithi203.stdout:3364: done (2 left) 2024-09-17T15:48:20.872 INFO:tasks.rados.rados.0.smithi203.stdout:3365: done (1 left) 2024-09-17T15:48:20.872 INFO:tasks.rados.rados.0.smithi203.stdout:3366: rollback oid 45 current snap is 361 2024-09-17T15:48:20.872 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 352 2024-09-17T15:48:20.873 INFO:tasks.rados.rados.0.smithi203.stdout:3367: snap_remove snap 360 2024-09-17T15:48:20.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510958955}) 2024-09-17T15:48:20.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166562778}) 2024-09-17T15:48:20.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2510958955}) 2024-09-17T15:48:20.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1172489343}) 2024-09-17T15:48:20.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780621245}) 2024-09-17T15:48:20.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1856925527}) 2024-09-17T15:48:20.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=780621245}) 2024-09-17T15:48:20.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2066500590}) 2024-09-17T15:48:20.878 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757127627}) 2024-09-17T15:48:20.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234131591}) 2024-09-17T15:48:20.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757127627}) 2024-09-17T15:48:20.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=354662200}) 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448079874}) 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2325708275}) 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448079874}) 2024-09-17T15:48:20.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370319179}) 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1773909364}) 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509083847}) 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1773909364}) 2024-09-17T15:48:20.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715245004}) 2024-09-17T15:48:20.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988092139}) 2024-09-17T15:48:20.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397256363}) 2024-09-17T15:48:20.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2988092139}) 2024-09-17T15:48:20.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506861945}) 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499081572}) 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3762986515}) 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499081572}) 2024-09-17T15:48:20.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.887+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274115834}) 2024-09-17T15:48:20.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848239750}) 2024-09-17T15:48:20.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=308943141}) 2024-09-17T15:48:20.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848239750}) 2024-09-17T15:48:20.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985242197}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542318886}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2439555045}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542318886}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.891+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583257376}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446794202}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667922091}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446794202}) 2024-09-17T15:48:20.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441045195}) 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112584098}) 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795320251}) 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112584098}) 2024-09-17T15:48:20.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3602384484}) 2024-09-17T15:48:20.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063140526}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1170827002}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063140526}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.895+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166574352}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533259469}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906523874}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533259469}) 2024-09-17T15:48:20.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295379227}) 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5210364}) 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695458440}) 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=5210364}) 2024-09-17T15:48:20.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.899+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=521350061}) 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656609816}) 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299918920}) 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656609816}) 2024-09-17T15:48:20.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015974776}) 2024-09-17T15:48:20.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872271975}) 2024-09-17T15:48:20.902 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626601957}) 2024-09-17T15:48:20.902 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.902 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872271975}) 2024-09-17T15:48:20.902 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.903+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3539560660}) 2024-09-17T15:48:20.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897259148}) 2024-09-17T15:48:20.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911442318}) 2024-09-17T15:48:20.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897259148}) 2024-09-17T15:48:20.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=176231915}) 2024-09-17T15:48:20.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454885582}) 2024-09-17T15:48:20.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687154895}) 2024-09-17T15:48:20.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454885582}) 2024-09-17T15:48:20.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.907+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1194606657}) 2024-09-17T15:48:20.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642811915}) 2024-09-17T15:48:20.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741062704}) 2024-09-17T15:48:20.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642811915}) 2024-09-17T15:48:20.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192176511}) 2024-09-17T15:48:20.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989424497}) 2024-09-17T15:48:20.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533704834}) 2024-09-17T15:48:20.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2989424497}) 2024-09-17T15:48:20.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250826255}) 2024-09-17T15:48:20.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010389494}) 2024-09-17T15:48:20.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3741186342}) 2024-09-17T15:48:20.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3010389494}) 2024-09-17T15:48:20.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.911+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=21557627}) 2024-09-17T15:48:20.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198691844}) 2024-09-17T15:48:20.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3105274662}) 2024-09-17T15:48:20.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198691844}) 2024-09-17T15:48:20.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733526633}) 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948656497}) 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796301692}) 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=948656497}) 2024-09-17T15:48:20.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.915+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587808658}) 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419513137}) 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1816507493}) 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419513137}) 2024-09-17T15:48:20.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275407794}) 2024-09-17T15:48:20.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768542490}) 2024-09-17T15:48:20.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=237599526}) 2024-09-17T15:48:20.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2768542490}) 2024-09-17T15:48:20.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.919+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122255892}) 2024-09-17T15:48:20.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124636697}) 2024-09-17T15:48:20.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125247391}) 2024-09-17T15:48:20.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124636697}) 2024-09-17T15:48:20.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1780496051}) 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962635938}) 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088364372}) 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962635938}) 2024-09-17T15:48:20.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235872870}) 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733549432}) 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449190585}) 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733549432}) 2024-09-17T15:48:20.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.923+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258909217}) 2024-09-17T15:48:20.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948622736}) 2024-09-17T15:48:20.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657771813}) 2024-09-17T15:48:20.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948622736}) 2024-09-17T15:48:20.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987585095}) 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363273155}) 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941525058}) 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2363273155}) 2024-09-17T15:48:20.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.927+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986805909}) 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575146758}) 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371109781}) 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3575146758}) 2024-09-17T15:48:20.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932845743}) 2024-09-17T15:48:20.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903253025}) 2024-09-17T15:48:20.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157348734}) 2024-09-17T15:48:20.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903253025}) 2024-09-17T15:48:20.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.931+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3471112588}) 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067008386}) 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115604919}) 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067008386}) 2024-09-17T15:48:20.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=483410892}) 2024-09-17T15:48:20.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155417581}) 2024-09-17T15:48:20.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788430693}) 2024-09-17T15:48:20.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155417581}) 2024-09-17T15:48:20.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657459112}) 2024-09-17T15:48:20.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286032874}) 2024-09-17T15:48:20.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967282327}) 2024-09-17T15:48:20.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4286032874}) 2024-09-17T15:48:20.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.935+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1159894253}) 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9} crc {logm=1577553936,mdsmap=3164546943,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=915208412,mon_config_key=1807879747}) 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=6} crc {logm=3240868159,mdsmap=2815901553,mgr=579208892,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3385794233,mon_config_key=347247062}) 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9} crc {logm=1577553936,mdsmap=3164546943,mgr=1115345915,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=915208412,mon_config_key=1807879747}) 2024-09-17T15:48:20.945 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=3} crc {logm=1950230672,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3667221254,mon_config_key=3388444897}) 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=1934961422,monmap=4191199434,nvmeofgw=3517310312,osd_metadata=4043941930,osd_pg_creating=2283336739,osd_snap=1939029387}) 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=5,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=82} crc {mon_config_key=3440763142,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2283336739,osd_snap=2365644626}) 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.947+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=1934961422,monmap=4191199434,nvmeofgw=3517310312,osd_metadata=4043941930,osd_pg_creating=2283336739,osd_snap=1939029387}) 2024-09-17T15:48:20.947 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=5,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=83} crc {mon_config_key=3440763142,monmap=4191199435,nvmeofgw=598337686,osd_metadata=4043941930,osd_snap=2370013245}) 2024-09-17T15:48:20.948 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.948 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=4055795005,osdmap=4263319776}) 2024-09-17T15:48:20.948 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=2531276733,osdmap=3171534977}) 2024-09-17T15:48:20.948 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.948 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=4055795005,osdmap=4263319776}) 2024-09-17T15:48:20.949 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.951+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=56,osdmap=44} crc {osd_snap=2531276733,osdmap=949034023}) 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=1585420830}) 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=2423444454}) 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=1585420830}) 2024-09-17T15:48:20.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:48:20.959+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=32} crc {osdmap=3817060857}) 2024-09-17T15:48:21.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:48:21.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:21.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: get_monmap_and_config 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: monmap: 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:21.241778+0000 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:21.241778+0000 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:21.237 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:21.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 Processor -- start 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 -- start start 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: init 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f784c0689d0 0x7f784c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-a con 0x7f784c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f784c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f784c13aff0 0x7f784c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-c con 0x7f784c13aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f784c09c8b0 con 0x7f784c068db0 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f784c074500 con 0x7f784c0689d0 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f784c09cb90 con 0x7f784c13aff0 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f78530ad640 10 monclient(hunting): authenticate will time out at 2234.207435s 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f784c0689d0 0x7f784c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f784c0689d0 0x7f784c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56278/0 (socket says 172.21.15.86:56278) 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 -- 172.21.15.86:0/1364397590 learned_addr learned my addr 172.21.15.86:0/1364397590 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): get_auth_request con 0x7f784c0689d0 auth_method 0 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient(hunting): handle_auth_done global_id 234126 payload 274 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f785164a640 1 --2- 172.21.15.86:0/1364397590 >> v2:172.21.15.203:3300/0 conn(0x7f784c13aff0 0x7f784c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 -- 172.21.15.86:0/1364397590 >> v2:172.21.15.203:3300/0 conn(0x7f784c13aff0 msgr2=0x7f784c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 --2- 172.21.15.86:0/1364397590 >> v2:172.21.15.203:3300/0 conn(0x7f784c13aff0 0x7f784c13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 -- 172.21.15.86:0/1364397590 >> v1:172.21.15.203:6789/0 conn(0x7f784c068db0 legacy=0x7f784c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.239+0000 7f7850e49640 1 -- 172.21.15.86:0/1364397590 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784c09ce70 con 0x7f784c0689d0 2024-09-17T15:48:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 -- 172.21.15.86:0/1364397590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 msgr2=0x7f784c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 --2- 172.21.15.86:0/1364397590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c06f840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f784c069a70 tx=0x7f784401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d61c7b787a31ce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 1 -- 172.21.15.86:0/1364397590 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c09c2d0 con 0x7f784c0689d0 2024-09-17T15:48:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: dump: 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 5 monclient: authenticate success, global_id 234126 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 1 -- 172.21.15.86:0/1364397590 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78440a5750 con 0x7f784c0689d0 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 4 set_mon_vals no callback set 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 1 -- 172.21.15.86:0/1364397590 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440a5b20 con 0x7f784c0689d0 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784b7fe640 10 monclient: dump: 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: shutdown 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- 172.21.15.86:0/1364397590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 msgr2=0x7f784c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- 172.21.15.86:0/1364397590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c06f840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f784c069a70 tx=0x7f784401c8b0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- 172.21.15.86:0/1364397590 shutdown_connections 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- 172.21.15.86:0/1364397590 >> v2:172.21.15.203:3300/0 conn(0x7f784c13aff0 0x7f784c13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- 172.21.15.86:0/1364397590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c06f840 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- 172.21.15.86:0/1364397590 >> 172.21.15.86:0/1364397590 conn(0x7f784c0a6440 msgr2=0x7f784c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- 172.21.15.86:0/1364397590 shutdown_connections 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- 172.21.15.86:0/1364397590 wait complete. 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: monmap: 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:21.248 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 Processor -- start 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- start start 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: init 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-a con 0x7f784c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:21.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f784c068db0 0x7f784c0737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-b con 0x7f784c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 0x7f784c132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): picked mon.noname-c con 0x7f784c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f784c1405c0 con 0x7f784c0689d0 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f784c140440 con 0x7f784c068db0 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f784c140740 con 0x7f784c13aff0 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f78530ad640 10 monclient(hunting): authenticate will time out at 2234.214869s 2024-09-17T15:48:21.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56286/0 (socket says 172.21.15.86:56286) 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 -- 172.21.15.86:0/2229138762 learned_addr learned my addr 172.21.15.86:0/2229138762 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): get_auth_request con 0x7f784c0689d0 auth_method 0 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 0x7f784c132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): get_auth_request con 0x7f784c13aff0 auth_method 0 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f784c068db0 0x7f784c0737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): get_auth_request con 0x7f784c068db0 auth_method 0 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f784bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient(hunting): handle_auth_done global_id 234129 payload 824 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 -- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 msgr2=0x7f784c132ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 0x7f784c132ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 -- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f784c068db0 msgr2=0x7f784c0737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f784c068db0 0x7f784c0737d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f7850e49640 1 -- 172.21.15.86:0/2229138762 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f784c072660 con 0x7f784c0689d0 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.247+0000 7f785164a640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 0x7f784c132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f7850e49640 1 -- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 msgr2=0x7f784c071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f7850e49640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78440af040 tx=0x7f7844019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440bc020 con 0x7f784c0689d0 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: dump: 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78440b8880 con 0x7f784c0689d0 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78440b8c50 con 0x7f784c0689d0 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 10 monclient: dump: 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.259 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 5 monclient: authenticate success, global_id 234129 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f784affd640 4 set_mon_vals no callback set 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _renew_subs 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f784c09d460 con 0x7f784c0689d0 2024-09-17T15:48:21.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _renew_subs 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f784c073d10 con 0x7f784c0689d0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f784c06f840 con 0x7f784c0689d0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f78440bf020 con 0x7f784c0689d0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 0x7f782c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(738..738 src has 696..738) ==== 4835+0+0 (secure 0 0 0) 0x7f78440eb120 con 0x7f784c0689d0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f784bfff640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 0x7f782c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.251+0000 7f784bfff640 10 monclient: get_auth_request con 0x7f782c03bbb0 auth_method 0 2024-09-17T15:48:21.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.255+0000 7f784bfff640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 0x7f782c03e070 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f784c13d180 tx=0x7f7834009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.255+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f78440165c0 con 0x7f784c0689d0 2024-09-17T15:48:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.255+0000 7f78497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.255+0000 7f78497fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:21.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f784c11d990 con 0x7f784c0689d0 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78497fa640 1 -- 172.21.15.86:0/2229138762 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f78440b73d0 con 0x7f784c0689d0 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:21.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78497fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:21.493 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:21.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 msgr2=0x7f782c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 0x7f782c03e070 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f784c13d180 tx=0x7f7834009040 comp rx=0 tx=0).stop 2024-09-17T15:48:21.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 10 monclient: shutdown 2024-09-17T15:48:21.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 msgr2=0x7f784c071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.495+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f78440af040 tx=0x7f7844019650 comp rx=0 tx=0).stop 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 shutdown_connections 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f782c03bbb0 0x7f782c03e070 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f784c13aff0 0x7f784c132ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f784c068db0 0x7f784c0737d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 --2- 172.21.15.86:0/2229138762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f784c0689d0 0x7f784c071670 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 >> 172.21.15.86:0/2229138762 conn(0x7f784c0a6440 msgr2=0x7f784c135c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 shutdown_connections 2024-09-17T15:48:21.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.499+0000 7f78530ad640 1 -- 172.21.15.86:0/2229138762 wait complete. 2024-09-17T15:48:21.511 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:48:21.511 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:48:21.512 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:48:21.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: get_monmap_and_config 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: monmap: 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:21.632813+0000 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:21.632813+0000 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.628 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:21.629 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:21.629 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 Processor -- start 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 -- start start 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: init 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-b con 0x7fcdd40a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcdd40a1590 0x7fcdd4135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-a con 0x7fcdd40a1590 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdd4135f90 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdd40a2130 con 0x7fcdd4135f90 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd40a2410 con 0x7fcdd40a1590 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd40a26f0 con 0x7fcdd40a0c70 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcddb026640 10 monclient(hunting): authenticate will time out at 2234.598299s 2024-09-17T15:48:21.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcdd40a1590 0x7fcdd4135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59140/0 (socket says 172.21.15.86:59140) 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcdd40a1590 0x7fcdd4135a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56302/0 (socket says 172.21.15.86:56302) 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 1 -- 172.21.15.86:0/1360003532 learned_addr learned my addr 172.21.15.86:0/1360003532 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): get_auth_request con 0x7fcdd40a0c70 auth_method 0 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): get_auth_request con 0x7fcdd40a1590 auth_method 0 2024-09-17T15:48:21.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_done global_id 234132 payload 274 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 msgr2=0x7fcdd40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1360003532 >> v1:172.21.15.203:6789/0 conn(0x7fcdd4135f90 legacy=0x7fcdd413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1360003532 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd409f9f0 con 0x7fcdd40a1590 2024-09-17T15:48:21.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.631+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:21.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 msgr2=0x7fcdd4135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd3fff640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4135a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80097c0 tx=0x7fcdc801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accd8c67a2464adc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 1 -- 172.21.15.86:0/1360003532 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80b0070 con 0x7fcdd40a1590 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: dump: 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 1 -- 172.21.15.86:0/1360003532 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc80ab600 con 0x7fcdd40a1590 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 1 -- 172.21.15.86:0/1360003532 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80ab9d0 con 0x7fcdd40a1590 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 5 monclient: authenticate success, global_id 234132 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 4 set_mon_vals no callback set 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcdd37fe640 10 monclient: dump: 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.644 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 10 monclient: shutdown 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 -- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 msgr2=0x7fcdd4135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4135a50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80097c0 tx=0x7fcdc801c360 comp rx=0 tx=0).stop 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 -- 172.21.15.86:0/1360003532 shutdown_connections 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4135a50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 --2- 172.21.15.86:0/1360003532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd40a0c70 0x7fcdd40a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.635+0000 7fcddb026640 1 -- 172.21.15.86:0/1360003532 >> 172.21.15.86:0/1360003532 conn(0x7fcdd40a2f40 msgr2=0x7fcdd40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- 172.21.15.86:0/1360003532 shutdown_connections 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- 172.21.15.86:0/1360003532 wait complete. 2024-09-17T15:48:21.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient: monmap: 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 Processor -- start 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- start start 2024-09-17T15:48:21.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient: init 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdd40a0c70 0x7fcdd412ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-b con 0x7fcdd40a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-a con 0x7fcdd40a1590 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd4135f90 0x7fcdd4137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): picked mon.noname-c con 0x7fcdd4135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd409ce60 con 0x7fcdd40a1590 2024-09-17T15:48:21.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd409cce0 con 0x7fcdd40a0c70 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd409ff50 con 0x7fcdd4135f90 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcddb026640 10 monclient(hunting): authenticate will time out at 2234.605501s 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56306/0 (socket says 172.21.15.86:56306) 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1013682164 learned_addr learned my addr 172.21.15.86:0/1013682164 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): get_auth_request con 0x7fcdd40a1590 auth_method 0 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd4135f90 0x7fcdd4137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): get_auth_request con 0x7fcdd4135f90 auth_method 0 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd95c3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdd40a0c70 0x7fcdd412ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient(hunting): handle_auth_done global_id 234135 payload 824 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd4135f90 msgr2=0x7fcdd4137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcdd4135f90 0x7fcdd4137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdd40a0c70 msgr2=0x7fcdd412ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdd40a0c70 0x7fcdd412ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd4139a10 con 0x7fcdd40a1590 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd8dc2640 10 monclient: get_auth_request con 0x7fcdd40a0c70 auth_method 0 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.639+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcdd40a0c70 0x7fcdd412ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd3fff640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 msgr2=0x7fcdd4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd3fff640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4136ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80ae040 tx=0x7fcdc8019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80b0040 con 0x7fcdd40a1590 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: dump: 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc80b9070 con 0x7fcdd40a1590 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80aaa10 con 0x7fcdd40a1590 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 5 monclient: authenticate success, global_id 234135 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 monclient: _renew_subs 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd409d460 con 0x7fcdd40a1590 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 10 monclient: dump: 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.652 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd2ffd640 4 set_mon_vals no callback set 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 monclient: _renew_subs 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd413ad50 con 0x7fcdd40a1590 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 0x7fcdd4137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd40a1050 con 0x7fcdd413b6a0 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 0x7fcdd4137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient: get_auth_request con 0x7fcdd413b6a0 auth_method 0 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcdc80ab4c0 con 0x7fcdd40a1590 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_done global_id 234138 payload 824 2024-09-17T15:48:21.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdd41395b0 con 0x7fcdd413b6a0 2024-09-17T15:48:21.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcdc003bb60 0x7fcdc003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(738..738 src has 696..738) ==== 4835+0+0 (secure 0 0 0) 0x7fcdc80e9dc0 con 0x7fcdd40a1590 2024-09-17T15:48:21.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd95c3640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcdc003bb60 0x7fcdc003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd95c3640 10 monclient: get_auth_request con 0x7fcdc003bb60 auth_method 0 2024-09-17T15:48:21.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 msgr2=0x7fcdd4137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.643+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 0x7fcdd4137970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc007cc0 tx=0x7fcdcc00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0147e0 con 0x7fcdd413b6a0 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 10 monclient: dump: 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.658 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcdcc0147e0 con 0x7fcdd413b6a0 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 msgr2=0x7fcdd4137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 0x7fcdd4137970 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc007cc0 tx=0x7fcdcc00f520 comp rx=0 tx=0).stop 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 msgr2=0x7fcdd4137970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd17fa640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd413b6a0 0x7fcdd4137970 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc007cc0 tx=0x7fcdcc00f520 comp rx=0 tx=0).stop 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.647+0000 7fcdd95c3640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcdc003bb60 0x7fcdc003e020 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4004ef0 tx=0x7fcdc4007d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcddb026640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcddb026640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 0x7fcdd4137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcddb026640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd406d4f0 con 0x7fcdd412f4d0 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 0x7fcdd4137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient: get_auth_request con 0x7fcdd412f4d0 auth_method 0 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 10 monclient(hunting): handle_auth_done global_id 234141 payload 824 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.651+0000 7fcdd8dc2640 1 -- 172.21.15.86:0/1013682164 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcdd41d9290 con 0x7fcdd412f4d0 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd8dc2640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 msgr2=0x7fcdd4137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd8dc2640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 0x7fcdd4137eb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc019000 tx=0x7fcdcc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc01fd80 con 0x7fcdd412f4d0 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 10 monclient: dump: 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fcdcc0039a0 con 0x7fcdd412f4d0 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:21.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 msgr2=0x7fcdd4137eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 0x7fcdd4137eb0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc019000 tx=0x7fcdcc004a50 comp rx=0 tx=0).stop 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 msgr2=0x7fcdd4137eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd17fa640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd412f4d0 0x7fcdd4137eb0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:21.662 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 30850, 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.663 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:21.664 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.665 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:21.666 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:21.667 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.668 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.670 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.672 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.673 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:21.674 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:21.675 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcdc003bb60 msgr2=0x7fcdc003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcdc003bb60 0x7fcdc003e020 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4004ef0 tx=0x7fcdc4007d10 comp rx=0 tx=0).stop 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 10 monclient: shutdown 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 msgr2=0x7fcdd4136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 --2- 172.21.15.86:0/1013682164 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcdd40a1590 0x7fcdd4136ef0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcdc80ae040 tx=0x7fcdc8019100 comp rx=0 tx=0).stop 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd95c3640 1 -- 172.21.15.86:0/1013682164 reap_dead start 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcdd95c3640 1 -- 172.21.15.86:0/1013682164 reap_dead start 2024-09-17T15:48:21.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 shutdown_connections 2024-09-17T15:48:21.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 >> 172.21.15.86:0/1013682164 conn(0x7fcdd40a2f40 msgr2=0x7fcdd40a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 shutdown_connections 2024-09-17T15:48:21.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.655+0000 7fcddb026640 1 -- 172.21.15.86:0/1013682164 wait complete. 2024-09-17T15:48:21.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:21.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:48:21.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: get_monmap_and_config 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: monmap: 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:21.796940+0000 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:21.796940+0000 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.792 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:21.793 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 Processor -- start 2024-09-17T15:48:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 -- start start 2024-09-17T15:48:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: init 2024-09-17T15:48:21.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-b con 0x7f7db80a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7db80a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 0x7f7db813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-a con 0x7f7db8136160 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db80a2130 con 0x7f7db80a1590 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7db80a2410 con 0x7f7db8136160 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7db80a26f0 con 0x7f7db80a0c70 2024-09-17T15:48:21.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7dbd6b2640 10 monclient(hunting): authenticate will time out at 2234.762398s 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 0x7f7db813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59154/0 (socket says 172.21.15.86:59154) 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3151330919 learned_addr learned my addr 172.21.15.86:0/3151330919 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): get_auth_request con 0x7f7db80a0c70 auth_method 0 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): get_auth_request con 0x7f7db8136160 auth_method 0 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_done global_id 164828 payload 274 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3151330919 >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 msgr2=0x7f7db813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 --2- 172.21.15.86:0/3151330919 >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 0x7f7db813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3151330919 >> v1:172.21.15.203:6789/0 conn(0x7f7db80a1590 legacy=0x7f7db8135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 monclient: found mon.noname-b 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3151330919 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db809f9f0 con 0x7f7db80a0c70 2024-09-17T15:48:21.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.795+0000 7f7db77fe640 1 --2- 172.21.15.86:0/3151330919 >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 0x7f7db813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3151330919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 msgr2=0x7f7db80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db6ffd640 1 --2- 172.21.15.86:0/3151330919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009870 tx=0x7f7da801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd175ad19c5e5f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.802 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 1 -- 172.21.15.86:0/3151330919 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80b1070 con 0x7f7db80a0c70 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: dump: 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.803 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 1 -- 172.21.15.86:0/3151330919 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da80ac620 con 0x7f7db80a0c70 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 1 -- 172.21.15.86:0/3151330919 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80ac9f0 con 0x7f7db80a0c70 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7db5ffb640 10 monclient: dump: 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7dbd6b2640 5 monclient: authenticate success, global_id 164828 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7dbd6b2640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.799+0000 7f7dbd6b2640 4 set_mon_vals no callback set 2024-09-17T15:48:21.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: shutdown 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3151330919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 msgr2=0x7f7db80a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3151330919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f7da8009870 tx=0x7f7da801c420 comp rx=0 tx=0).stop 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3151330919 shutdown_connections 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3151330919 >> v2:172.21.15.86:3300/0 conn(0x7f7db8136160 0x7f7db813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3151330919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db80a0c70 0x7f7db80a1050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3151330919 >> 172.21.15.86:0/3151330919 conn(0x7f7db80a2f40 msgr2=0x7f7db80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3151330919 shutdown_connections 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3151330919 wait complete. 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: monmap: 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:21.806 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 Processor -- start 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- start start 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: init 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 0x7f7db813d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-a con 0x7f7db80a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7db80a1590 0x7f7db8135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-b con 0x7f7db80a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 0x7f7db809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): picked mon.noname-c con 0x7f7db8136160 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7db80a00d0 con 0x7f7db80a0c70 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7db809ff50 con 0x7f7db80a1590 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8140c90 con 0x7f7db8136160 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7dbd6b2640 10 monclient(hunting): authenticate will time out at 2234.769627s 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 0x7f7db813d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 0x7f7db809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 0x7f7db813d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56328/0 (socket says 172.21.15.86:56328) 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 0x7f7db809d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59158/0 (socket says 172.21.15.86:59158) 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3899165982 learned_addr learned my addr 172.21.15.86:0/3899165982 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): get_auth_request con 0x7f7db80a0c70 auth_method 0 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): get_auth_request con 0x7f7db8136160 auth_method 0 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db67fc640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7db80a1590 0x7f7db8135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db67fc640 10 monclient(hunting): get_auth_request con 0x7f7db80a1590 auth_method 0 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient(hunting): handle_auth_done global_id 234147 payload 824 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 msgr2=0x7f7db809d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 0x7f7db809d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7db80a1590 msgr2=0x7f7db8135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7db80a1590 0x7f7db8135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db806fbb0 con 0x7f7db80a0c70 2024-09-17T15:48:21.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.803+0000 7f7db77fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8136160 0x7f7db809d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db6ffd640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 msgr2=0x7f7db813d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db6ffd640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 0x7f7db813d780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7da801c250 tx=0x7f7da801ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80bb020 con 0x7f7db80a0c70 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: dump: 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.816 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da80b7420 con 0x7f7db80a0c70 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80b1040 con 0x7f7db80a0c70 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 5 monclient: authenticate success, global_id 234147 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 monclient: _renew_subs 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db8070100 con 0x7f7db80a0c70 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db57fa640 4 set_mon_vals no callback set 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 10 monclient: dump: 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.818 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 monclient: _renew_subs 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db8121fb0 con 0x7f7db80a0c70 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 0x7f7db809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8121b80 con 0x7f7db809f460 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 0x7f7db809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient: get_auth_request con 0x7f7db809f460 auth_method 0 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7da80be020 con 0x7f7db80a0c70 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 10 monclient(hunting): handle_auth_done global_id 164834 payload 824 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7db80a1050 con 0x7f7db809f460 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7da403bb60 0x7f7da403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(738..738 src has 696..738) ==== 4835+0+0 (secure 0 0 0) 0x7f7da80ea040 con 0x7f7db80a0c70 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db77fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7da403bb60 0x7f7da403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db77fe640 10 monclient: get_auth_request con 0x7f7da403bb60 auth_method 0 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.807+0000 7f7db67fc640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 msgr2=0x7f7db809dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7db67fc640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 0x7f7db809dcb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00e450 tx=0x7f7dac00e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac007450 con 0x7f7db809f460 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 10 monclient: dump: 2024-09-17T15:48:21.821 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.822 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7dac007450 con 0x7f7db809f460 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 msgr2=0x7f7db809dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 0x7f7db809dcb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00e450 tx=0x7f7dac00e910 comp rx=0 tx=0).stop 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 msgr2=0x7f7db809dcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7d9f7fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db809f460 0x7f7db809dcb0 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f7dac00e450 tx=0x7f7dac00e910 comp rx=0 tx=0).stop 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.811+0000 7f7db77fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7da403bb60 0x7f7da403e020 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7db809ed30 tx=0x7f7db000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7dbd6b2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 0x7f7db809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7dbd6b2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7db813e0d0 con 0x7f7db8135b40 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 0x7f7db809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient: get_auth_request con 0x7f7db8135b40 auth_method 0 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 10 monclient(hunting): handle_auth_done global_id 164837 payload 824 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.815+0000 7f7db67fc640 1 -- 172.21.15.86:0/3899165982 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7db806d4f0 con 0x7f7db8135b40 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7db67fc640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 msgr2=0x7f7db809e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7db67fc640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 0x7f7db809e1f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7dac009010 tx=0x7f7dac003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dac01e960 con 0x7f7db8135b40 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 10 monclient: dump: 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.825 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.826 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f7dac01e960 con 0x7f7db8135b40 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 msgr2=0x7f7db809e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 0x7f7db809e1f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7dac009010 tx=0x7f7dac003770 comp rx=0 tx=0).stop 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 msgr2=0x7f7db809e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7d9f7fe640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7db8135b40 0x7f7db809e1f0 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7dac009010 tx=0x7f7dac003770 comp rx=0 tx=0).stop 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:21.827 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 405007, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:21.828 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:21.829 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:21.831 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:21.832 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:21.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:21.834 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:21.836 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.838 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:21.839 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:21.840 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7da403bb60 msgr2=0x7f7da403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7da403bb60 0x7f7da403e020 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7db809ed30 tx=0x7f7db000b040 comp rx=0 tx=0).stop 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 10 monclient: shutdown 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 msgr2=0x7f7db813d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 --2- 172.21.15.86:0/3899165982 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7db80a0c70 0x7f7db813d780 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7da801c250 tx=0x7f7da801ac60 comp rx=0 tx=0).stop 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7db77fe640 1 -- 172.21.15.86:0/3899165982 reap_dead start 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7db77fe640 1 -- 172.21.15.86:0/3899165982 reap_dead start 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 shutdown_connections 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 >> 172.21.15.86:0/3899165982 conn(0x7f7db80a2f40 msgr2=0x7f7db8071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 shutdown_connections 2024-09-17T15:48:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.819+0000 7f7dbd6b2640 1 -- 172.21.15.86:0/3899165982 wait complete. 2024-09-17T15:48:21.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:21.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:48:21.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:48:21.844 INFO:tasks.rados.rados.0.smithi203.stdout:3367: done (2 left) 2024-09-17T15:48:21.844 INFO:tasks.rados.rados.0.smithi203.stdout:3368: delete oid 17 current snap is 361 2024-09-17T15:48:21.844 INFO:tasks.rados.rados.0.smithi203.stdout:3363: expect (ObjNum 1162 snap 358 seq_num 1162) 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: get_monmap_and_config 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: monmap: 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.960 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:21.964446+0000 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:21.964446+0000 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 Processor -- start 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 -- start start 2024-09-17T15:48:21.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: init 2024-09-17T15:48:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50b409d770 0x7f50b409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-a con 0x7f50b409d770 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f50b409e090 0x7f50b4134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-c con 0x7f50b409e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50b4134f80 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50b409ff00 con 0x7f50b4134f80 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f50b409e7f0 con 0x7f50b409d770 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f50b40a2110 con 0x7f50b409e090 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b8c3c640 10 monclient(hunting): authenticate will time out at 2234.930131s 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50b409d770 0x7f50b409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50b409d770 0x7f50b409db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56330/0 (socket says 172.21.15.86:56330) 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 -- 172.21.15.86:0/299550888 learned_addr learned my addr 172.21.15.86:0/299550888 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): get_auth_request con 0x7f50b409d770 auth_method 0 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b1d9b640 1 --2- 172.21.15.86:0/299550888 >> v2:172.21.15.203:3300/0 conn(0x7f50b409e090 0x7f50b4134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient(hunting): handle_auth_done global_id 234150 payload 274 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 -- 172.21.15.86:0/299550888 >> v2:172.21.15.203:3300/0 conn(0x7f50b409e090 msgr2=0x7f50b4134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 --2- 172.21.15.86:0/299550888 >> v2:172.21.15.203:3300/0 conn(0x7f50b409e090 0x7f50b4134a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 -- 172.21.15.86:0/299550888 >> v1:172.21.15.203:6789/0 conn(0x7f50b4134f80 legacy=0x7f50b413bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.963+0000 7f50b259c640 1 -- 172.21.15.86:0/299550888 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b409ec30 con 0x7f50b409d770 2024-09-17T15:48:21.973 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b259c640 1 -- 172.21.15.86:0/299550888 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b409d770 msgr2=0x7f50b409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b259c640 1 --2- 172.21.15.86:0/299550888 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b409d770 0x7f50b409db50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50b4074de0 tx=0x7f50a401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14730f824a3299da server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 1 -- 172.21.15.86:0/299550888 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a40b0070 con 0x7f50b409d770 2024-09-17T15:48:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: dump: 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 1 -- 172.21.15.86:0/299550888 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a40ab600 con 0x7f50b409d770 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 1 -- 172.21.15.86:0/299550888 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a40ab9d0 con 0x7f50b409d770 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b159a640 10 monclient: dump: 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.976 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 5 monclient: authenticate success, global_id 234150 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 4 set_mon_vals no callback set 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 10 monclient: shutdown 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 -- 172.21.15.86:0/299550888 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b409d770 msgr2=0x7f50b409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/299550888 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b409d770 0x7f50b409db50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50b4074de0 tx=0x7f50a401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 -- 172.21.15.86:0/299550888 shutdown_connections 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/299550888 >> v2:172.21.15.203:3300/0 conn(0x7f50b409e090 0x7f50b4134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/299550888 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b409d770 0x7f50b409db50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 -- 172.21.15.86:0/299550888 >> 172.21.15.86:0/299550888 conn(0x7f50b40a2f40 msgr2=0x7f50b40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 -- 172.21.15.86:0/299550888 shutdown_connections 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.967+0000 7f50b8c3c640 1 -- 172.21.15.86:0/299550888 wait complete. 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient: build_initial_monmap 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient: monmap: 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 Processor -- start 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 -- start start 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient: init 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b409d770 0x7f50b41cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-c con 0x7f50b409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b409e090 0x7f50b4133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-b con 0x7f50b409e090 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 0x7f50b413a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): picked mon.noname-a con 0x7f50b4134f80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f50b409cce0 con 0x7f50b4134f80 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50b409e4a0 con 0x7f50b409e090 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f50b409ce60 con 0x7f50b409d770 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b8c3c640 10 monclient(hunting): authenticate will time out at 2234.937408s 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b409d770 0x7f50b41cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 0x7f50b413a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b409d770 0x7f50b41cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59180/0 (socket says 172.21.15.86:59180) 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 1 -- 172.21.15.86:0/3714798107 learned_addr learned my addr 172.21.15.86:0/3714798107 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): get_auth_request con 0x7f50b409d770 auth_method 0 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): get_auth_request con 0x7f50b4134f80 auth_method 0 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b1d9b640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b409e090 0x7f50b4133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b1d9b640 10 monclient(hunting): get_auth_request con 0x7f50b409e090 auth_method 0 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient(hunting): handle_auth_done global_id 234153 payload 824 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b409d770 msgr2=0x7f50b41cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b409d770 0x7f50b41cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b409e090 msgr2=0x7f50b4133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:21.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b409e090 0x7f50b4133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 monclient: found mon.noname-a 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.971+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b411d430 con 0x7f50b4134f80 2024-09-17T15:48:21.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 msgr2=0x7f50b413a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b2d9d640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 0x7f50b413a080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00c410 tx=0x7f50ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac017070 con 0x7f50b4134f80 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: dump: 2024-09-17T15:48:21.985 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.986 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.986 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.986 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.986 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.986 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ac003800 con 0x7f50b4134f80 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac003bf0 con 0x7f50b4134f80 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:21.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 10 monclient: dump: 2024-09-17T15:48:21.989 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:21.989 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:21.989 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.989 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:21.989 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:21.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 5 monclient: authenticate success, global_id 234153 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 monclient: _renew_subs 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b0d99640 4 set_mon_vals no callback set 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b413c6a0 con 0x7f50b4134f80 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 monclient: _renew_subs 2024-09-17T15:48:21.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b411c8c0 con 0x7f50b4134f80 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 0x7f50b413a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50b411d080 con 0x7f50b41341d0 2024-09-17T15:48:21.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f50ac01a070 con 0x7f50b4134f80 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 0x7f50b413a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient: get_auth_request con 0x7f50b41341d0 auth_method 0 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:21.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f509403bbb0 0x7f509403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(739..739 src has 696..739) ==== 4879+0+0 (secure 0 0 0) 0x7f50ac04f380 con 0x7f50b4134f80 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 10 monclient(hunting): handle_auth_done global_id 224200 payload 824 2024-09-17T15:48:21.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b259c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50b409db50 con 0x7f50b41341d0 2024-09-17T15:48:21.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b1d9b640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f509403bbb0 0x7f509403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:21.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.975+0000 7f50b1d9b640 10 monclient: get_auth_request con 0x7f509403bbb0 auth_method 0 2024-09-17T15:48:21.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50b259c640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 msgr2=0x7f50b413a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50b259c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 0x7f50b413a5c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50a401aa90 tx=0x7f50a401ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50b1d9b640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f509403bbb0 0x7f509403e070 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f50a80070b0 tx=0x7f50a8007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a40b0040 con 0x7f50b41341d0 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:22.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 10 monclient: dump: 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50a40b6670 con 0x7f50b41341d0 2024-09-17T15:48:22.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 msgr2=0x7f50b413a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 0x7f50b413a5c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50a401aa90 tx=0x7f50a401ac20 comp rx=0 tx=0).stop 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 msgr2=0x7f50b413a5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.979+0000 7f50a2ffd640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b41341d0 0x7f50b413a5c0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50a401aa90 tx=0x7f50a401ac20 comp rx=0 tx=0).stop 2024-09-17T15:48:22.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b8c3c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 0x7f50b4135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b8c3c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50b413af10 con 0x7f50b413ab00 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 0x7f50b4135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient: get_auth_request con 0x7f50b413ab00 auth_method 0 2024-09-17T15:48:22.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:22.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:22.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:22.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 10 monclient(hunting): handle_auth_done global_id 224203 payload 824 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 1 -- 172.21.15.86:0/3714798107 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50b413cb60 con 0x7f50b413ab00 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 msgr2=0x7f50b4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50b259c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 0x7f50b4135ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f50a4000c10 tx=0x7f50a4019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a40b0070 con 0x7f50b413ab00 2024-09-17T15:48:22.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.987+0000 7f50a2ffd640 10 monclient: dump: 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.007 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f50a40b6d10 con 0x7f50b413ab00 2024-09-17T15:48:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 msgr2=0x7f50b4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 0x7f50b4135ee0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f50a4000c10 tx=0x7f50a4019180 comp rx=0 tx=0).stop 2024-09-17T15:48:22.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 msgr2=0x7f50b4135ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50a2ffd640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50b413ab00 0x7f50b4135ee0 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f50a4000c10 tx=0x7f50a4019180 comp rx=0 tx=0).stop 2024-09-17T15:48:22.010 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:22.010 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:22.010 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 194, 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 92441, 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:22.011 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:48:22.012 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:22.012 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.012 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:48:22.012 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:22.012 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:22.013 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:22.015 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:22.016 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:22.017 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:22.018 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:22.019 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:22.020 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:22.021 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:22.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:22.024 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:22.025 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:22.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:22.027 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.029 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.030 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:22.031 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:22.032 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:22.033 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f509403bbb0 msgr2=0x7f509403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f509403bbb0 0x7f509403e070 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f50a80070b0 tx=0x7f50a8007040 comp rx=0 tx=0).stop 2024-09-17T15:48:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 10 monclient: shutdown 2024-09-17T15:48:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 msgr2=0x7f50b413a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 --2- 172.21.15.86:0/3714798107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b4134f80 0x7f50b413a080 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00c410 tx=0x7f50ac00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 reap_dead start 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b2d9d640 1 -- 172.21.15.86:0/3714798107 reap_dead start 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 shutdown_connections 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 >> 172.21.15.86:0/3714798107 conn(0x7f50b40a2f40 msgr2=0x7f50b4071ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 shutdown_connections 2024-09-17T15:48:22.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:21.991+0000 7f50b8c3c640 1 -- 172.21.15.86:0/3714798107 wait complete. 2024-09-17T15:48:22.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:48:22.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:48:22.039 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:48:22.039 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:48:22.039 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:48:22.039 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:48:22.040 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:48:22.061 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:48:22.061 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:48:22.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:22.061 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:22.152 INFO:tasks.rados.rados.0.smithi203.stdout:3366: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:48:22.152 INFO:tasks.rados.rados.0.smithi203.stdout:3366: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:48:22.152 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1341 (ObjNum 33 snap 0 seq_num 3605912672) dirty dne 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3363: done (2 left) 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3366: done (1 left) 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3368: done (0 left) 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3369: read oid 7 snap -1 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3369: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3370: read oid 17 snap -1 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3370: expect deleted 2024-09-17T15:48:22.153 INFO:tasks.rados.rados.0.smithi203.stdout:3371: snap_create 2024-09-17T15:48:22.155 INFO:tasks.rados.rados.0.smithi203.stdout:3369: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.183+0000 7ff40026a640 10 monclient: get_monmap_and_config 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.183+0000 7ff40026a640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.183+0000 7ff40026a640 10 monclient: monmap: 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:22.187304+0000 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:22.187304+0000 2024-09-17T15:48:22.182 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:22.183 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.183+0000 7ff40026a640 1 Processor -- start 2024-09-17T15:48:22.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 -- start start 2024-09-17T15:48:22.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient: init 2024-09-17T15:48:22.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f8069260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 0x7ff3f809bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f8074c30 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 0x7ff3f813aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f809c4f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff3f809d150 con 0x7ff3f8074c30 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7ff3f809ce80 con 0x7ff3f809c4f0 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f8069640 con 0x7ff3f8069260 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff40026a640 10 monclient(hunting): authenticate will time out at 2235.152688s 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe807640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 msgr2=0x7ff3f813aa70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe807640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 0x7ff3f813aa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 0x7ff3f809bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59190/0 (socket says 172.21.15.86:59190) 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 -- 172.21.15.86:0/1122587292 learned_addr learned my addr 172.21.15.86:0/1122587292 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): get_auth_request con 0x7ff3f8074c30 auth_method 0 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): get_auth_request con 0x7ff3f8069260 auth_method 0 2024-09-17T15:48:22.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient(hunting): handle_auth_done global_id 164843 payload 274 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 -- 172.21.15.86:0/1122587292 >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 msgr2=0x7ff3f813aa70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 --2- 172.21.15.86:0/1122587292 >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 0x7ff3f813aa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 -- 172.21.15.86:0/1122587292 >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 msgr2=0x7ff3f809bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:22.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 --2- 172.21.15.86:0/1122587292 >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 0x7ff3f809bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 monclient: found mon.noname-b 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fe006640 1 -- 172.21.15.86:0/1122587292 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f809e410 con 0x7ff3f8069260 2024-09-17T15:48:22.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.187+0000 7ff3fd805640 1 --2- 172.21.15.86:0/1122587292 >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 0x7ff3f809bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fe006640 1 -- 172.21.15.86:0/1122587292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 msgr2=0x7ff3f809ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fe006640 1 --2- 172.21.15.86:0/1122587292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff3f809fbd0 tx=0x7ff3f401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21be5f06aba8714a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 1 -- 172.21.15.86:0/1122587292 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40b0070 con 0x7ff3f8069260 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: dump: 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 1 -- 172.21.15.86:0/1122587292 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f40ab600 con 0x7ff3f8069260 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 1 -- 172.21.15.86:0/1122587292 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40ab9d0 con 0x7ff3f8069260 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 5 monclient: authenticate success, global_id 164843 2024-09-17T15:48:22.194 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 4 set_mon_vals no callback set 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff3fd004640 10 monclient: dump: 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.195 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 10 monclient: shutdown 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- 172.21.15.86:0/1122587292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 msgr2=0x7ff3f809ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 --2- 172.21.15.86:0/1122587292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff3f809fbd0 tx=0x7ff3f401c400 comp rx=0 tx=0).stop 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- 172.21.15.86:0/1122587292 shutdown_connections 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 --2- 172.21.15.86:0/1122587292 >> v2:172.21.15.203:3300/0 conn(0x7ff3f809c4f0 0x7ff3f813aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 --2- 172.21.15.86:0/1122587292 >> v2:172.21.15.86:3300/0 conn(0x7ff3f8074c30 0x7ff3f809bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 --2- 172.21.15.86:0/1122587292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f8069260 0x7ff3f809ba70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- 172.21.15.86:0/1122587292 >> 172.21.15.86:0/1122587292 conn(0x7ff3f80a6440 msgr2=0x7ff3f80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- 172.21.15.86:0/1122587292 shutdown_connections 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- 172.21.15.86:0/1122587292 wait complete. 2024-09-17T15:48:22.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 10 monclient: build_initial_monmap 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 10 monclient: monmap: 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 Processor -- start 2024-09-17T15:48:22.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 1 -- start start 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.191+0000 7ff40026a640 10 monclient: init 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-a con 0x7ff3f8069260 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 0x7ff3f813c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-b con 0x7ff3f8074c30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f809c4f0 0x7ff3f813c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): picked mon.noname-c con 0x7ff3f809c4f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f813b480 con 0x7ff3f8069260 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff3f813b300 con 0x7ff3f8074c30 2024-09-17T15:48:22.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff3f813b600 con 0x7ff3f809c4f0 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient(hunting): authenticate will time out at 2235.160135s 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fd805640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 msgr2=0x7ff3f813c0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fd805640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 0x7ff3f813c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56352/0 (socket says 172.21.15.86:56352) 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 -- 172.21.15.86:0/3667513944 learned_addr learned my addr 172.21.15.86:0/3667513944 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): get_auth_request con 0x7ff3f8069260 auth_method 0 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f809c4f0 0x7ff3f813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): get_auth_request con 0x7ff3f809c4f0 auth_method 0 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient(hunting): handle_auth_done global_id 234159 payload 824 2024-09-17T15:48:22.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 -- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f809c4f0 msgr2=0x7ff3f813c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f809c4f0 0x7ff3f813c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 -- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 msgr2=0x7ff3f813c0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 0x7ff3f813c0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient: _finish_hunting 0 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 monclient: found mon.noname-a 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:22.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 -- 172.21.15.86:0/3667513944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3f81247d0 con 0x7ff3f8069260 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 -- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 msgr2=0x7ff3f81377e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fe006640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40ae040 tx=0x7ff3f40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40abe60 con 0x7ff3f8069260 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: dump: 2024-09-17T15:48:22.203 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:22.204 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f40aa5b0 con 0x7ff3f8069260 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f40b0040 con 0x7ff3f8069260 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 5 monclient: authenticate success, global_id 234159 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _renew_subs 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 -- 172.21.15.86:0/3667513944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3f8125030 con 0x7ff3f8069260 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3fc803640 4 set_mon_vals no callback set 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff3eeffd640 10 monclient: dump: 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:22.205 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _renew_subs 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.195+0000 7ff40026a640 1 -- 172.21.15.86:0/3667513944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3f8074420 con 0x7ff3f8069260 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3ecff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3ecff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.206 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3ecff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3f813ba60 con 0x7ff3f8069260 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff3f40aa960 con 0x7ff3f8069260 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3eeffd640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 0x7ff3dc03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(739..739 src has 696..739) ==== 4879+0+0 (secure 0 0 0) 0x7ff3f40e97d0 con 0x7ff3f8069260 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3fd805640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 0x7ff3dc03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3fd805640 10 monclient: get_auth_request con 0x7ff3dc03bb40 auth_method 0 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.199+0000 7ff3fd805640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 0x7ff3dc03e000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff3f813c050 tx=0x7ff3e8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.203+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff3f40c2060 con 0x7ff3f8069260 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.203+0000 7ff3eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:22.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.203+0000 7ff3eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:22.324 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2291974 2024-09-17T15:48:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3ecff9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3ecff9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3ecff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff3f806d4f0 con 0x7ff3f8069260 2024-09-17T15:48:22.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3eeffd640 1 -- 172.21.15.86:0/3667513944 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff3f4016110 con 0x7ff3f8069260 2024-09-17T15:48:22.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:22.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.435+0000 7ff3eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:22.434 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 msgr2=0x7ff3dc03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 0x7ff3dc03e000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff3f813c050 tx=0x7ff3e8009190 comp rx=0 tx=0).stop 2024-09-17T15:48:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 10 monclient: shutdown 2024-09-17T15:48:22.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 msgr2=0x7ff3f81377e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff3f40ae040 tx=0x7ff3f40191a0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 shutdown_connections 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff3dc03bb40 0x7ff3dc03e000 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff3f809c4f0 0x7ff3f813c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3f8074c30 0x7ff3f813c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 --2- 172.21.15.86:0/3667513944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff3f8069260 0x7ff3f81377e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 >> 172.21.15.86:0/3667513944 conn(0x7ff3f80a6440 msgr2=0x7ff3f806f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 shutdown_connections 2024-09-17T15:48:22.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:22.439+0000 7ff3ecff9640 1 -- 172.21.15.86:0/3667513944 wait complete. 2024-09-17T15:48:22.440 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2795562 2024-09-17T15:48:25.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:25.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.583+0000 7f04b422e640 10 monclient: get_monmap_and_config 2024-09-17T15:48:25.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.583+0000 7f04b422e640 10 monclient: build_initial_monmap 2024-09-17T15:48:25.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.583+0000 7f04b422e640 10 monclient: monmap: 2024-09-17T15:48:25.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:25.587418+0000 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:25.587418+0000 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:25.583 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 Processor -- start 2024-09-17T15:48:25.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 -- start start 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient: init 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04ac0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04ac0694c0 0x7f04ac06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac0694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 0x7f04ac13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ac074470 con 0x7f04ac0689d0 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f04ac09c900 con 0x7f04ac0694c0 2024-09-17T15:48:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac09cb90 con 0x7f04ac13aff0 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b422e640 10 monclient(hunting): authenticate will time out at 2238.552897s 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f04ac0694c0 0x7f04ac06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b1fca640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f04ac0689d0 0x7f04ac068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40766/0 (socket says 172.21.15.86:40766) 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 0x7f04ac13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b1fca640 1 -- 172.21.15.86:0/2780012994 learned_addr learned my addr 172.21.15.86:0/2780012994 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): get_auth_request con 0x7f04ac13aff0 auth_method 0 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:25.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): get_auth_request con 0x7f04ac0694c0 auth_method 0 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient(hunting): handle_auth_done global_id 234162 payload 274 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 -- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 msgr2=0x7f04ac13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:25.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 0x7f04ac13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 -- 172.21.15.86:0/2780012994 >> v1:172.21.15.86:6789/0 conn(0x7f04ac0689d0 legacy=0x7f04ac068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 monclient: found mon.noname-a 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b17c9640 1 -- 172.21.15.86:0/2780012994 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac09ce70 con 0x7f04ac0694c0 2024-09-17T15:48:25.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.587+0000 7f04b27cb640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 0x7f04ac13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b17c9640 1 -- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac0694c0 msgr2=0x7f04ac06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b17c9640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac0694c0 0x7f04ac06f840 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f049c0097c0 tx=0x7f049c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555f992c99056a88 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 1 -- 172.21.15.86:0/2780012994 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b0070 con 0x7f04ac0694c0 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: dump: 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.591 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: _finish_auth 0 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: _check_auth_tickets 2024-09-17T15:48:25.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 1 -- 172.21.15.86:0/2780012994 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c0ab600 con 0x7f04ac0694c0 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 1 -- 172.21.15.86:0/2780012994 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04ac09c2d0 con 0x7f04ac0694c0 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 5 monclient: authenticate success, global_id 234162 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 4 set_mon_vals no callback set 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b0fc8640 10 monclient: dump: 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.593 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 10 monclient: shutdown 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 -- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac0694c0 msgr2=0x7f04ac06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac0694c0 0x7f04ac06f840 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f049c0097c0 tx=0x7f049c01c360 comp rx=0 tx=0).stop 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 -- 172.21.15.86:0/2780012994 shutdown_connections 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac13aff0 0x7f04ac13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 --2- 172.21.15.86:0/2780012994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac0694c0 0x7f04ac06f840 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 -- 172.21.15.86:0/2780012994 >> 172.21.15.86:0/2780012994 conn(0x7f04ac0a6440 msgr2=0x7f04ac0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 -- 172.21.15.86:0/2780012994 shutdown_connections 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 1 -- 172.21.15.86:0/2780012994 wait complete. 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 10 monclient: build_initial_monmap 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.591+0000 7f04b422e640 10 monclient: monmap: 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:25.595 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:25.596 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 Processor -- start 2024-09-17T15:48:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- start start 2024-09-17T15:48:25.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: init 2024-09-17T15:48:25.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 0x7f04ac135e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-b con 0x7f04ac0689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-c con 0x7f04ac0694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): picked mon.noname-a con 0x7f04ac13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac140610 con 0x7f04ac13aff0 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04ac140490 con 0x7f04ac0689d0 2024-09-17T15:48:25.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f04ac140790 con 0x7f04ac0694c0 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient(hunting): authenticate will time out at 2238.560731s 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b1fca640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 msgr2=0x7f04ac135e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b1fca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 0x7f04ac135e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59220/0 (socket says 172.21.15.86:59220) 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 -- 172.21.15.86:0/978481587 learned_addr learned my addr 172.21.15.86:0/978481587 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:56368/0 (socket says 172.21.15.86:56368) 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): get_auth_request con 0x7f04ac0694c0 auth_method 0 2024-09-17T15:48:25.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): get_auth_request con 0x7f04ac13aff0 auth_method 0 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:25.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient(hunting): handle_auth_done global_id 164852 payload 824 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 -- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 msgr2=0x7f04ac135e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 0x7f04ac135e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 -- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 msgr2=0x7f04ac09dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 monclient: found mon.noname-c 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 -- 172.21.15.86:0/978481587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ac09f0b0 con 0x7f04ac0694c0 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b27cb640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 -- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 msgr2=0x7f04ac09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:25.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b17c9640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f049c01cc00 tx=0x7f049c01aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b0050 con 0x7f04ac0694c0 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: dump: 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.604 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c0bc070 con 0x7f04ac0694c0 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 5 monclient: authenticate success, global_id 164852 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c0b5a40 con 0x7f04ac0694c0 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _renew_subs 2024-09-17T15:48:25.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ac1227c0 con 0x7f04ac0694c0 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a27fc640 10 monclient: dump: 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:25.606 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04a3fff640 4 set_mon_vals no callback set 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _renew_subs 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.595+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ac1224e0 con 0x7f04ac0694c0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b422e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b422e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b422e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ac06f840 con 0x7f04ac0694c0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f049c0b5c60 con 0x7f04ac0694c0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04a27fc640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 0x7f049003e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(739..739 src has 696..739) ==== 4879+0+0 (secure 0 0 0) 0x7f049c0ea790 con 0x7f04ac0694c0 2024-09-17T15:48:25.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b1fca640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 0x7f049003e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.599+0000 7f04b1fca640 10 monclient: get_auth_request con 0x7f049003bb40 auth_method 0 2024-09-17T15:48:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.603+0000 7f04b1fca640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 0x7f049003e000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f04a8004820 tx=0x7f04a8007b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.603+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f049c016070 con 0x7f04ac0694c0 2024-09-17T15:48:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.603+0000 7f04a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:25.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.603+0000 7f04a27fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:25.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:25.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:25.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:25.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f04ac13add0 con 0x7f04ac0694c0 2024-09-17T15:48:25.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04a27fc640 1 -- 172.21.15.86:0/978481587 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f049c0c1c10 con 0x7f04ac0694c0 2024-09-17T15:48:25.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:25.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04a27fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:25.837 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:25.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 msgr2=0x7f049003e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 0x7f049003e000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f04a8004820 tx=0x7f04a8007b30 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 10 monclient: shutdown 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 msgr2=0x7f04ac09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f049c01cc00 tx=0x7f049c01aba0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 shutdown_connections 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f049003bb40 0x7f049003e000 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04ac13aff0 0x7f04ac09dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04ac0694c0 0x7f04ac09d770 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 --2- 172.21.15.86:0/978481587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04ac0689d0 0x7f04ac135e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 >> 172.21.15.86:0/978481587 conn(0x7f04ac0a6440 msgr2=0x7f04ac0a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 shutdown_connections 2024-09-17T15:48:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:25.839+0000 7f04b422e640 1 -- 172.21.15.86:0/978481587 wait complete. 2024-09-17T15:48:28.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:28.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: get_monmap_and_config 2024-09-17T15:48:28.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: build_initial_monmap 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: monmap: 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:28.985500+0000 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:28.985500+0000 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:28.981 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 Processor -- start 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 -- start start 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: init 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ebc0689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ebc0694c0 0x7f4ebc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc0694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 0x7f4ebc13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:28.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc13aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ebc09c8b0 con 0x7f4ebc0689d0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4ebc074500 con 0x7f4ebc0694c0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4ebc09cb90 con 0x7f4ebc13aff0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ec16b9640 10 monclient(hunting): authenticate will time out at 2241.950949s 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ebaffd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f4ebc0689d0 legacy=0x7f4ebc068db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ebb7fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 msgr2=0x7f4ebc13d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4ebb7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 0x7f4ebc13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ebc0694c0 0x7f4ebc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4ebc0694c0 0x7f4ebc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44118/0 (socket says 172.21.15.86:44118) 2024-09-17T15:48:28.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 -- 172.21.15.86:0/2353598321 learned_addr learned my addr 172.21.15.86:0/2353598321 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): get_auth_request con 0x7f4ebc0694c0 auth_method 0 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient(hunting): handle_auth_done global_id 234168 payload 274 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 -- 172.21.15.86:0/2353598321 >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 msgr2=0x7f4ebc13d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 --2- 172.21.15.86:0/2353598321 >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 0x7f4ebc13d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 -- 172.21.15.86:0/2353598321 >> v1:172.21.15.203:6789/0 conn(0x7f4ebc0689d0 legacy=0x7f4ebc068db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 monclient: found mon.noname-a 2024-09-17T15:48:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:28.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:28.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.983+0000 7f4eba7fc640 1 -- 172.21.15.86:0/2353598321 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc09ce70 con 0x7f4ebc0694c0 2024-09-17T15:48:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eba7fc640 1 -- 172.21.15.86:0/2353598321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc0694c0 msgr2=0x7f4ebc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:28.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eba7fc640 1 --2- 172.21.15.86:0/2353598321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc06f840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc06fdb0 tx=0x7f4ea401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a3d92fd9831f62 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 1 -- 172.21.15.86:0/2353598321 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc09c2d0 con 0x7f4ebc0694c0 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 10 monclient: dump: 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:28.991 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4eb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4ec16b9640 5 monclient: authenticate success, global_id 234168 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.987+0000 7f4ec16b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 1 -- 172.21.15.86:0/2353598321 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ea40a5770 con 0x7f4ebc0694c0 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 1 -- 172.21.15.86:0/2353598321 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea40a5b40 con 0x7f4ebc0694c0 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eb9ffb640 10 monclient: dump: 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:28.992 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 4 set_mon_vals no callback set 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: shutdown 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2353598321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc0694c0 msgr2=0x7f4ebc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:28.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2353598321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc06f840 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc06fdb0 tx=0x7f4ea401c8d0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2353598321 shutdown_connections 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2353598321 >> v2:172.21.15.203:3300/0 conn(0x7f4ebc13aff0 0x7f4ebc13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2353598321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc06f840 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2353598321 >> 172.21.15.86:0/2353598321 conn(0x7f4ebc0a6440 msgr2=0x7f4ebc0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2353598321 shutdown_connections 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2353598321 wait complete. 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: build_initial_monmap 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: monmap: 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 Processor -- start 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- start start 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: init 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:28.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc0694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc13aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc1405c0 con 0x7f4ebc13aff0 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc140440 con 0x7f4ebc0694c0 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4ebc140740 con 0x7f4ebc0689d0 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ec16b9640 10 monclient(hunting): authenticate will time out at 2241.958333s 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eba7fc640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 msgr2=0x7f4ebc135ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:28.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4eba7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc135ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:49670/0 (socket says 172.21.15.86:49670) 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:44126/0 (socket says 172.21.15.86:44126) 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 1 -- 172.21.15.86:0/2937021016 learned_addr learned my addr 172.21.15.86:0/2937021016 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): get_auth_request con 0x7f4ebc0689d0 auth_method 0 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): get_auth_request con 0x7f4ebc13aff0 auth_method 0 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient(hunting): handle_auth_done global_id 234171 payload 824 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 -- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 msgr2=0x7f4ebc1cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 -- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 msgr2=0x7f4ebc135ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc135ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:48:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 monclient: found mon.noname-a 2024-09-17T15:48:28.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:28.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:28.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebb7fe640 1 -- 172.21.15.86:0/2937021016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc121680 con 0x7f4ebc13aff0 2024-09-17T15:48:28.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.991+0000 7f4ebaffd640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ebb7fe640 1 -- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 msgr2=0x7f4ebc136430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ebb7fe640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400b3e0 tx=0x7f4eb400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb4017070 con 0x7f4ebc13aff0 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: dump: 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: _finish_auth 0 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 5 monclient: authenticate success, global_id 234171 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eb40060d0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _renew_subs 2024-09-17T15:48:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc1228a0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40064c0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 10 monclient: dump: 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:29.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _renew_subs 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb97fa640 4 set_mon_vals no callback set 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc13b460 con 0x7f4ebc13aff0 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ebc06f840 con 0x7f4ebc13aff0 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4eb401a070 con 0x7f4ebc13aff0 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 0x7f4e9c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:29.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(739..739 src has 696..739) ==== 4879+0+0 (secure 0 0 0) 0x7f4eb404f0b0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ebaffd640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 0x7f4e9c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.995+0000 7f4ebaffd640 10 monclient: get_auth_request con 0x7f4e9c03bb10 auth_method 0 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.999+0000 7f4ebaffd640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 0x7f4e9c03dfd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4eac006f10 tx=0x7f4eac006e60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.999+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f4eb40066e0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.999+0000 7f4eb37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:28.999+0000 7f4eb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:29.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4ec16b9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:29.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4ec16b9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:29.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4ec16b9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:29.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4ebc06d4f0 con 0x7f4ebc13aff0 2024-09-17T15:48:29.232 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4eb37fe640 1 -- 172.21.15.86:0/2937021016 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f4eb401bb10 con 0x7f4ebc13aff0 2024-09-17T15:48:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4eb37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4eb37fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:29.233 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":194,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.235+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 msgr2=0x7f4e9c03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 0x7f4e9c03dfd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4eac006f10 tx=0x7f4eac006e60 comp rx=0 tx=0).stop 2024-09-17T15:48:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 10 monclient: shutdown 2024-09-17T15:48:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:29.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 msgr2=0x7f4ebc136430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400b3e0 tx=0x7f4eb400b8a0 comp rx=0 tx=0).stop 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 shutdown_connections 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4e9c03bb10 0x7f4e9c03dfd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ebc13aff0 0x7f4ebc136430 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ebc0694c0 0x7f4ebc135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 --2- 172.21.15.86:0/2937021016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ebc0689d0 0x7f4ebc1cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 >> 172.21.15.86:0/2937021016 conn(0x7f4ebc0a6440 msgr2=0x7f4ebc0a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 shutdown_connections 2024-09-17T15:48:29.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:29.239+0000 7f4ec16b9640 1 -- 172.21.15.86:0/2937021016 wait complete. 2024-09-17T15:48:32.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: get_monmap_and_config 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: build_initial_monmap 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: monmap: 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:32.385793+0000 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:32.385793+0000 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:32.381 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:32.382 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:32.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 Processor -- start 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 -- start start 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: init 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59f409ba70 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59f4068c30 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-a con 0x7f59f409eea0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:32.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f409f2d0 con 0x7f59f409ba70 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f409dc20 con 0x7f59f4068c30 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f59f409d7e0 con 0x7f59f409eea0 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f8b5b640 10 monclient(hunting): authenticate will time out at 2245.351260s 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f59f409ba70 0x7f59f409be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48592/0 (socket says 172.21.15.86:48592) 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 learned_addr learned my addr 172.21.15.86:0/1097537762 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59f409eea0 auth_method 0 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:32.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.203:6789/0 conn(0x7f59f4068c30 legacy=0x7f59f409c560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.383+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:32.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.86:6789/0 conn(0x7f59f409ba70 legacy=0x7f59f409be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.203:6789/0 conn(0x7f59f4068c30 legacy=0x7f59f409c560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59f409eea0 auth_method 0 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:48:32.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:32.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:32.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.587+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.203:6789/0 conn(0x7f59f4068c30 legacy=0x7f59f409c560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59f409eea0 auth_method 0 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:32.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:32.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:32.991+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59f409eea0 auth_method 0 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:33.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:33.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:33.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:33.795+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.203:6789/0 conn(0x7f59f4068c30 legacy=0x7f59f409c560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:35.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): tick 2024-09-17T15:48:35.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:35.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 msgr2=0x7f59f41446b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59f409eea0 0x7f59f41446b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.203:6789/0 conn(0x7f59f4068c30 legacy=0x7f59f409c560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.86:6789/0 conn(0x7f59f409ba70 legacy=0x7f59f409be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient: _add_conns ranks=[2,4] 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f59d4003730 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f59d4006260 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f59d4008db0 con 0x7f59d4006260 2024-09-17T15:48:35.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59d4008f60 con 0x7f59d4003730 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59e3fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 msgr2=0x7f59d4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.383+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.387+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:35.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.587+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 msgr2=0x7f59d4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:35.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.587+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:35.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:35.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.591+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:48:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.991+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 msgr2=0x7f59d4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.991+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:35.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:35.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:35.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:35.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:35.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:35.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:35.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:35.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:35.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:35.995+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:48:36.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.795+0000 7f59f1d9b640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 msgr2=0x7f59d4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:36.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.795+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:48:36.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:48:36.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:36.799+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:48:38.217 INFO:tasks.rados.rados.0.smithi203.stdout:3369: done (2 left) 2024-09-17T15:48:38.217 INFO:tasks.rados.rados.0.smithi203.stdout:3370: done (1 left) 2024-09-17T15:48:38.217 INFO:tasks.rados.rados.0.smithi203.stdout:3371: done (0 left) 2024-09-17T15:48:38.217 INFO:tasks.rados.rados.0.smithi203.stdout:3372: write oid 33 current snap is 362 2024-09-17T15:48:38.217 INFO:tasks.rados.rados.0.smithi203.stdout:3372: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-17T15:48:38.235 INFO:tasks.rados.rados.0.smithi203.stdout:3372: writing smithi20328086-33 from 649067 to 1152851 tid 1 2024-09-17T15:48:38.258 INFO:tasks.rados.rados.0.smithi203.stdout:3372: writing smithi20328086-33 from 1875940 to 2483011 tid 2 2024-09-17T15:48:38.278 INFO:tasks.rados.rados.0.smithi203.stdout:3372: writing smithi20328086-33 from 3115378 to 3846684 tid 3 2024-09-17T15:48:38.278 INFO:tasks.rados.rados.0.smithi203.stdout:3373: copy_from oid 4 from oid 48 current snap is 362 2024-09-17T15:48:38.278 INFO:tasks.rados.rados.0.smithi203.stdout:3372: finishing write tid 1 to smithi20328086-33 2024-09-17T15:48:38.278 INFO:tasks.rados.rados.0.smithi203.stdout:3372: finishing write tid 2 to smithi20328086-33 2024-09-17T15:48:38.278 INFO:tasks.rados.rados.0.smithi203.stdout:3374: snap_remove snap 358 2024-09-17T15:48:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 10 monclient(hunting): tick 2024-09-17T15:48:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:48:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 monclient(hunting): continuing hunt 2024-09-17T15:48:38.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 msgr2=0x7f59d4008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.203:3300/0 conn(0x7f59d4006260 0x7f59d4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59f2d9d640 1 -- 172.21.15.86:0/1097537762 reap_dead start 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.383+0000 7f59e3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f59d4003730 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d4006710 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59d400aba0 0x7f59d400cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f59d400aba0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59f4146c10 con 0x7f59d4006710 2024-09-17T15:48:38.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f59d4008db0 con 0x7f59d400aba0 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 1 -- 172.21.15.86:0/1097537762 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59d4008f60 con 0x7f59d4003730 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59e3fff640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59d400aba0 0x7f59d400cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59d400aba0 auth_method 0 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient(hunting): handle_auth_done global_id 164858 payload 274 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59d400aba0 msgr2=0x7f59d400cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:38.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59d400aba0 0x7f59d400cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> v1:172.21.15.86:6789/0 conn(0x7f59d4006710 legacy=0x7f59d4009480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 monclient: found mon.noname-b 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:38.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.387+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f409dc20 con 0x7f59d4003730 2024-09-17T15:48:38.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f259c640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f259c640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f59e8007fd0 tx=0x7f59e801d1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57d3fe2ede5ae746 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 1 -- 172.21.15.86:0/1097537762 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e80ab070 con 0x7f59d4003730 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: dump: 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 1 -- 172.21.15.86:0/1097537762 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e80a5e30 con 0x7f59d4003730 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 1 -- 172.21.15.86:0/1097537762 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e80af620 con 0x7f59d4003730 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 5 monclient: authenticate success, global_id 164858 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 4 set_mon_vals no callback set 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:38.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f159a640 10 monclient: dump: 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.395 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 10 monclient: shutdown 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 msgr2=0x7f59d4005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f59e8007fd0 tx=0x7f59e801d1f0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1097537762 shutdown_connections 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1097537762 >> v2:172.21.15.86:3300/0 conn(0x7f59d400aba0 0x7f59d400cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1097537762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d4003730 0x7f59d4005b80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1097537762 >> 172.21.15.86:0/1097537762 conn(0x7f59f40a6440 msgr2=0x7f59f40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1097537762 shutdown_connections 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.395+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1097537762 wait complete. 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient: build_initial_monmap 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient: monmap: 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:38.399 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 Processor -- start 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 -- start start 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient: init 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:48:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-a con 0x7f59d4003730 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 0x7f59f41d41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-b con 0x7f59d4006710 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d400aba0 0x7f59f4143230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): picked mon.noname-c con 0x7f59d400aba0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f59f409c9f0 con 0x7f59d4003730 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59f409c870 con 0x7f59d4006710 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f59f413c240 con 0x7f59d400aba0 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f8b5b640 10 monclient(hunting): authenticate will time out at 2251.365832s 2024-09-17T15:48:38.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f1d9b640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 msgr2=0x7f59f41d41e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f1d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 0x7f59f41d41e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35520/0 (socket says 172.21.15.86:35520) 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 -- 172.21.15.86:0/1596855456 learned_addr learned my addr 172.21.15.86:0/1596855456 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): get_auth_request con 0x7f59d4003730 auth_method 0 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d400aba0 0x7f59f4143230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request con 0x7f59d400aba0 auth_method 0 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient(hunting): handle_auth_done global_id 234177 payload 824 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 -- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d400aba0 msgr2=0x7f59f4143230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d400aba0 0x7f59f4143230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 -- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 msgr2=0x7f59f41d41e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 0x7f59f41d41e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 monclient: found mon.noname-a 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.399+0000 7f59f259c640 1 -- 172.21.15.86:0/1596855456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f4074470 con 0x7f59d4003730 2024-09-17T15:48:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f259c640 1 -- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 msgr2=0x7f59f4145850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f259c640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59e80010f0 tx=0x7f59e8019fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e80be020 con 0x7f59d4003730 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: dump: 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.406 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e80053d0 con 0x7f59d4003730 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 5 monclient: authenticate success, global_id 234177 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _renew_subs 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f4125650 con 0x7f59d4003730 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e80ab040 con 0x7f59d4003730 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 10 monclient: dump: 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f0d99640 4 set_mon_vals no callback set 2024-09-17T15:48:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _renew_subs 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f4125930 con 0x7f59d4003730 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f59e80c1020 con 0x7f59d4003730 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f409d460 con 0x7f59d4003730 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 0x7f59d403f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(740..740 src has 696..740) ==== 4851+0+0 (secure 0 0 0) 0x7f59e80eb160 con 0x7f59d4003730 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 0x7f59d403f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.403+0000 7f59f1d9b640 10 monclient: get_auth_request con 0x7f59d403d410 auth_method 0 2024-09-17T15:48:38.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.407+0000 7f59f1d9b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 0x7f59d403f8d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f59f4125860 tx=0x7f59e400a3b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.411+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f59e80c8030 con 0x7f59d4003730 2024-09-17T15:48:38.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.411+0000 7f59e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.411+0000 7f59e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:38.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.643+0000 7f59f8b5b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.643+0000 7f59f8b5b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.643+0000 7f59f8b5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.643+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f59f413c5d0 con 0x7f59d4003730 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59e2ffd640 1 -- 172.21.15.86:0/1596855456 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f59f4074470 con 0x7f59d4003730 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:38.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:38.644 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":198,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 msgr2=0x7f59d403f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 0x7f59d403f8d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f59f4125860 tx=0x7f59e400a3b0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 10 monclient: shutdown 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 msgr2=0x7f59f4145850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59e80010f0 tx=0x7f59e8019fe0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 shutdown_connections 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f59d403d410 0x7f59d403f8d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f59d400aba0 0x7f59f4143230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4006710 0x7f59f41d41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 --2- 172.21.15.86:0/1596855456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f59d4003730 0x7f59f4145850 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 >> 172.21.15.86:0/1596855456 conn(0x7f59f40a6440 msgr2=0x7f59f40a2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 shutdown_connections 2024-09-17T15:48:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.647+0000 7f59f8b5b640 1 -- 172.21.15.86:0/1596855456 wait complete. 2024-09-17T15:48:38.663 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:48:38.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:48:38.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:48:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: get_monmap_and_config 2024-09-17T15:48:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: build_initial_monmap 2024-09-17T15:48:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: monmap: 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:38.792154+0000 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:38.792154+0000 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:38.789 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.791 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 Processor -- start 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 -- start start 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: init 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63080a50e0 0x7f630809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-a con 0x7f63080a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 0x7f63080a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-b con 0x7f63080a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 0x7f63081382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-c con 0x7f6308137ea0 addr v2:172.21.15.203:3300/0 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f63080a6280 con 0x7f63080a50e0 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f630809c0a0 con 0x7f6308137ea0 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f630809c380 con 0x7f63080a54c0 2024-09-17T15:48:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630fdea640 10 monclient(hunting): authenticate will time out at 2251.757516s 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63080a50e0 0x7f630809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 0x7f63080a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63080a50e0 0x7f630809cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35536/0 (socket says 172.21.15.86:35536) 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 -- 172.21.15.86:0/2608606817 learned_addr learned my addr 172.21.15.86:0/2608606817 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630e387640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 msgr2=0x7f63081382f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): get_auth_request con 0x7f63080a50e0 auth_method 0 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): get_auth_request con 0x7f63080a54c0 auth_method 0 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630e387640 1 --2- 172.21.15.86:0/2608606817 >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 0x7f63081382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient(hunting): handle_auth_done global_id 234180 payload 274 2024-09-17T15:48:38.795 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 -- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 msgr2=0x7f63080a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 0x7f63080a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 -- 172.21.15.86:0/2608606817 >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 msgr2=0x7f63081382f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 --2- 172.21.15.86:0/2608606817 >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 0x7f63081382f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 monclient: found mon.noname-a 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630db86640 1 -- 172.21.15.86:0/2608606817 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630809c660 con 0x7f63080a50e0 2024-09-17T15:48:38.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.791+0000 7f630d385640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 0x7f63080a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630db86640 1 -- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a50e0 msgr2=0x7f630809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630db86640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a50e0 0x7f630809cd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f630806f8c0 tx=0x7f630001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eacdf7c945654b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 1 -- 172.21.15.86:0/2608606817 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000a5710 con 0x7f63080a50e0 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:38.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: dump: 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 5 monclient: authenticate success, global_id 234180 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 1 -- 172.21.15.86:0/2608606817 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63000a58b0 con 0x7f63080a50e0 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 1 -- 172.21.15.86:0/2608606817 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000a5c80 con 0x7f63080a50e0 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 4 set_mon_vals no callback set 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630cb84640 10 monclient: dump: 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.805 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 10 monclient: shutdown 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 -- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a50e0 msgr2=0x7f630809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a50e0 0x7f630809cd60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f630806f8c0 tx=0x7f630001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 -- 172.21.15.86:0/2608606817 shutdown_connections 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 --2- 172.21.15.86:0/2608606817 >> v2:172.21.15.203:3300/0 conn(0x7f6308137ea0 0x7f63081382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63080a54c0 0x7f63080a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 --2- 172.21.15.86:0/2608606817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a50e0 0x7f630809cd60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 -- 172.21.15.86:0/2608606817 >> 172.21.15.86:0/2608606817 conn(0x7f63080aaae0 msgr2=0x7f63080aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 -- 172.21.15.86:0/2608606817 shutdown_connections 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 1 -- 172.21.15.86:0/2608606817 wait complete. 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 10 monclient: build_initial_monmap 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.795+0000 7f630fdea640 10 monclient: monmap: 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:38.807 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 Processor -- start 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- start start 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: init 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 0x7f63081cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-a con 0x7f63080a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 0x7f63081370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-c con 0x7f6308137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63081375e0 0x7f630813c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): picked mon.noname-b con 0x7f63081375e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:38.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f63080a6ae0 con 0x7f63080a54c0 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f63080a6960 con 0x7f63081375e0 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6308141b20 con 0x7f6308137ea0 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient(hunting): authenticate will time out at 2251.764689s 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 0x7f63081370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 0x7f63081cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 0x7f63081370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55368/0 (socket says 172.21.15.86:55368) 2024-09-17T15:48:38.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 0x7f63081cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35548/0 (socket says 172.21.15.86:35548) 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 1 -- 172.21.15.86:0/1993368281 learned_addr learned my addr 172.21.15.86:0/1993368281 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): get_auth_request con 0x7f6308137ea0 auth_method 0 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): get_auth_request con 0x7f63080a54c0 auth_method 0 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630e387640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63081375e0 msgr2=0x7f630813c1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630e387640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63081375e0 0x7f630813c1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient(hunting): handle_auth_done global_id 234183 payload 824 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 msgr2=0x7f63081370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 0x7f63081370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63081375e0 msgr2=0x7f630813c1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63081375e0 0x7f630813c1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 monclient: found mon.noname-a 2024-09-17T15:48:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63081221c0 con 0x7f63080a54c0 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630d385640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6308137ea0 0x7f63081370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 msgr2=0x7f63081cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630db86640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 0x7f63081cfd90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63000ab000 tx=0x7f6300019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000bc030 con 0x7f63080a54c0 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: dump: 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.812 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63000b8680 con 0x7f63080a54c0 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63000a5710 con 0x7f63080a54c0 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 5 monclient: authenticate success, global_id 234183 2024-09-17T15:48:38.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _renew_subs 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63081224a0 con 0x7f63080a54c0 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f63067fc640 10 monclient: dump: 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.814 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f6307fff640 4 set_mon_vals no callback set 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _renew_subs 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.799+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63081d02d0 con 0x7f63080a54c0 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630fdea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630fdea640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 0x7f630813cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f630806a0e0 con 0x7f630813c720 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 0x7f630813cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient: get_auth_request con 0x7f630813c720 auth_method 0 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f63000b8a90 con 0x7f63080a54c0 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 10 monclient(hunting): handle_auth_done global_id 234186 payload 824 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f630813d040 con 0x7f630813c720 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62f403bb70 0x7f62f403e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(740..740 src has 696..740) ==== 4851+0+0 (secure 0 0 0) 0x7f63000ea440 con 0x7f63080a54c0 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630e387640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62f403bb70 0x7f62f403e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630e387640 10 monclient: get_auth_request con 0x7f62f403bb70 auth_method 0 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 msgr2=0x7f630813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f630d385640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 0x7f630813cb00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc014860 con 0x7f630813c720 2024-09-17T15:48:38.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 10 monclient: dump: 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f62fc014860 con 0x7f630813c720 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 msgr2=0x7f630813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 0x7f630813cb00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc00f610 comp rx=0 tx=0).stop 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 msgr2=0x7f630813cb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.803+0000 7f63067fc640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630813c720 0x7f630813cb00 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc00f610 comp rx=0 tx=0).stop 2024-09-17T15:48:38.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.807+0000 7f630e387640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62f403bb70 0x7f62f403e030 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f62f8003940 tx=0x7f62f8006cc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630fdea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630fdea640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 0x7f63081396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630fdea640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6308121e30 con 0x7f63081392d0 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 0x7f63081396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient: get_auth_request con 0x7f63081392d0 auth_method 0 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 10 monclient(hunting): handle_auth_done global_id 234189 payload 824 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.811+0000 7f630d385640 1 -- 172.21.15.86:0/1993368281 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6308122ac0 con 0x7f63081392d0 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630d385640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 msgr2=0x7f63081396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630d385640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 0x7f63081396e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62fc004230 con 0x7f63081392d0 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 10 monclient: dump: 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.821 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f62fc004230 con 0x7f63081392d0 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 msgr2=0x7f63081396e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 0x7f63081396e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 msgr2=0x7f63081396e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:38.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f63067fc640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63081392d0 0x7f63081396e0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f63081211f0 tx=0x7f62fc0046d0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 198, 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 48010, 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:38.823 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.824 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.825 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:38.826 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.827 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.828 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:38.829 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.831 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.832 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.833 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.834 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:38.835 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:38.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62f403bb70 msgr2=0x7f62f403e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62f403bb70 0x7f62f403e030 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f62f8003940 tx=0x7f62f8006cc0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 10 monclient: shutdown 2024-09-17T15:48:38.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 msgr2=0x7f63081cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 --2- 172.21.15.86:0/1993368281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63080a54c0 0x7f63081cfd90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f63000ab000 tx=0x7f6300019670 comp rx=0 tx=0).stop 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630e387640 1 -- 172.21.15.86:0/1993368281 reap_dead start 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630e387640 1 -- 172.21.15.86:0/1993368281 reap_dead start 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 shutdown_connections 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 >> 172.21.15.86:0/1993368281 conn(0x7f63080aaae0 msgr2=0x7f63080a3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 shutdown_connections 2024-09-17T15:48:38.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.815+0000 7f630fdea640 1 -- 172.21.15.86:0/1993368281 wait complete. 2024-09-17T15:48:38.837 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:48:38.837 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:48:38.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 10 monclient: get_monmap_and_config 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 10 monclient: build_initial_monmap 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 10 monclient: monmap: 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:38.958860+0000 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:38.958860+0000 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:38.954 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:38.955 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 1 Processor -- start 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 1 -- start start 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.955+0000 7f669b76f640 10 monclient: init 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f66940a0c80 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66940a1060 addr v1:172.21.15.86:6789/0 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66940a14d0 0x7f669413e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-a con 0x7f66940a14d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6694136550 con 0x7f66940a1060 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669406d550 con 0x7f66940a0c80 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f669406d830 con 0x7f66940a14d0 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669b76f640 10 monclient(hunting): authenticate will time out at 2251.924231s 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6698d0a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f66940a1060 0x7f669413ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:60960/0 (socket says 172.21.15.86:60960) 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f669950b640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f66940a0c80 legacy=0x7f6694135e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66940a14d0 0x7f669413e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6698d0a640 1 -- 172.21.15.86:0/3425976729 learned_addr learned my addr 172.21.15.86:0/3425976729 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): get_auth_request con 0x7f66940a14d0 auth_method 0 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient(hunting): handle_auth_done global_id 234192 payload 274 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 1 -- 172.21.15.86:0/3425976729 >> v1:172.21.15.203:6789/0 conn(0x7f66940a0c80 legacy=0x7f6694135e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 1 -- 172.21.15.86:0/3425976729 >> v1:172.21.15.86:6789/0 conn(0x7f66940a1060 legacy=0x7f669413ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 1 monclient: found mon.noname-a 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.959 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.959+0000 7f6699d0c640 1 -- 172.21.15.86:0/3425976729 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669409dbd0 con 0x7f66940a14d0 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 -- 172.21.15.86:0/3425976729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a14d0 msgr2=0x7f669413e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 --2- 172.21.15.86:0/3425976729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a14d0 0x7f669413e4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f668c009dc0 tx=0x7f668c01c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e2557dcd2c8ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 1 -- 172.21.15.86:0/3425976729 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0ac070 con 0x7f66940a14d0 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: dump: 2024-09-17T15:48:38.961 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 1 -- 172.21.15.86:0/3425976729 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c0a6740 con 0x7f66940a14d0 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 1 -- 172.21.15.86:0/3425976729 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c0a6b10 con 0x7f66940a14d0 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6687fff640 10 monclient: dump: 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.963 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 5 monclient: authenticate success, global_id 234192 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 4 set_mon_vals no callback set 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: shutdown 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- 172.21.15.86:0/3425976729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a14d0 msgr2=0x7f669413e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 --2- 172.21.15.86:0/3425976729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a14d0 0x7f669413e4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f668c009dc0 tx=0x7f668c01c810 comp rx=0 tx=0).stop 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- 172.21.15.86:0/3425976729 shutdown_connections 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 --2- 172.21.15.86:0/3425976729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a14d0 0x7f669413e4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- 172.21.15.86:0/3425976729 >> 172.21.15.86:0/3425976729 conn(0x7f66940a3e80 msgr2=0x7f66940a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- 172.21.15.86:0/3425976729 shutdown_connections 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- 172.21.15.86:0/3425976729 wait complete. 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: build_initial_monmap 2024-09-17T15:48:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: monmap: 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:38.965 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 Processor -- start 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- start start 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: init 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66940a0c80 0x7f6694134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-b con 0x7f66940a0c80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:38.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 0x7f669409b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-a con 0x7f66940a1060 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 0x7f669409ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): picked mon.noname-c con 0x7f66940a14d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f66941419d0 con 0x7f66940a1060 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6694141850 con 0x7f66940a0c80 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6694141b50 con 0x7f66940a14d0 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669b76f640 10 monclient(hunting): authenticate will time out at 2251.931264s 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 0x7f669409ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669950b640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66940a0c80 msgr2=0x7f6694134d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:48:38.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 0x7f669409ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:55376/0 (socket says 172.21.15.86:55376) 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 -- 172.21.15.86:0/174949416 learned_addr learned my addr 172.21.15.86:0/174949416 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): get_auth_request con 0x7f66940a14d0 auth_method 0 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 0x7f669409b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f669950b640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66940a0c80 0x7f6694134d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): get_auth_request con 0x7f66940a1060 auth_method 0 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient(hunting): handle_auth_done global_id 234195 payload 824 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 msgr2=0x7f669409ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 0x7f669409ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66940a0c80 msgr2=0x7f6694134d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66940a0c80 0x7f6694134d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient: _finish_hunting 0 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 monclient: found mon.noname-a 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6698d0a640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669413c6c0 con 0x7f66940a1060 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.963+0000 7f6699d0c640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66940a14d0 0x7f669409ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6698d0a640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 msgr2=0x7f669409b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6698d0a640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 0x7f669409b4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f669000c410 tx=0x7f669000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6690017070 con 0x7f66940a1060 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: dump: 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.974 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: _finish_auth 0 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66900070d0 con 0x7f66940a1060 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 5 monclient: authenticate success, global_id 234195 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 monclient: _renew_subs 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66940a35a0 con 0x7f66940a1060 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66900074c0 con 0x7f66940a1060 2024-09-17T15:48:38.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 10 monclient: dump: 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6686ffd640 4 set_mon_vals no callback set 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 monclient: _renew_subs 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f669413c3e0 con 0x7f66940a1060 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 0x7f6694136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f66940a3af0 con 0x7f669409bf50 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 0x7f6694136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient: get_auth_request con 0x7f669409bf50 auth_method 0 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f669001a070 con 0x7f66940a1060 2024-09-17T15:48:38.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 10 monclient(hunting): handle_auth_done global_id 164873 payload 824 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f669950b640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f669409c330 con 0x7f669409bf50 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f667803bb10 0x7f667803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(740..740 src has 696..740) ==== 4851+0+0 (secure 0 0 0) 0x7f669004f370 con 0x7f66940a1060 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.967+0000 7f6699d0c640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f667803bb10 0x7f667803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6699d0c640 10 monclient: get_auth_request con 0x7f667803bb10 auth_method 0 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f669950b640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 msgr2=0x7f6694136950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f669950b640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 0x7f6694136950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f668800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6688005270 con 0x7f669409bf50 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 10 monclient: dump: 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6688015070 con 0x7f669409bf50 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:38.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:38.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 msgr2=0x7f6694136950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 0x7f6694136950 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f668800b910 comp rx=0 tx=0).stop 2024-09-17T15:48:38.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 msgr2=0x7f6694136950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:38.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6685ffb640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f669409bf50 0x7f6694136950 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f668800b910 comp rx=0 tx=0).stop 2024-09-17T15:48:38.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.971+0000 7f6699d0c640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f667803bb10 0x7f667803dfd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f668c009bd0 tx=0x7f668c0a7000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 0x7f66941372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6694135ec0 con 0x7f6694136e90 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 0x7f66941372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient: get_auth_request con 0x7f6694136e90 auth_method 0 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 10 monclient(hunting): handle_auth_done global_id 164876 payload 824 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 1 -- 172.21.15.86:0/174949416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f669413bf00 con 0x7f6694136e90 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 msgr2=0x7f66941372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669950b640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 0x7f66941372a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f6688005a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668801ae60 con 0x7f6694136e90 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 10 monclient: dump: 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f6688010480 con 0x7f6694136e90 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 msgr2=0x7f66941372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 0x7f66941372a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f6688005a30 comp rx=0 tx=0).stop 2024-09-17T15:48:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 msgr2=0x7f66941372a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6685ffb640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6694136e90 0x7f66941372a0 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f668800b450 tx=0x7f6688005a30 comp rx=0 tx=0).stop 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 198, 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 422166, 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:48:38.985 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.986 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:48:38.988 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:48:38.989 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.990 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.991 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.992 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.994 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:48:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:48:38.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:48:38.997 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:48:38.998 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:48:38.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f667803bb10 msgr2=0x7f667803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:38.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f667803bb10 0x7f667803dfd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f668c009bd0 tx=0x7f668c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:48:38.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 10 monclient: shutdown 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 msgr2=0x7f669409b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 --2- 172.21.15.86:0/174949416 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66940a1060 0x7f669409b4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f669000c410 tx=0x7f669000c8d0 comp rx=0 tx=0).stop 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6699d0c640 1 -- 172.21.15.86:0/174949416 reap_dead start 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f6699d0c640 1 -- 172.21.15.86:0/174949416 reap_dead start 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 shutdown_connections 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 >> 172.21.15.86:0/174949416 conn(0x7f66940a3e80 msgr2=0x7f6694073a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 shutdown_connections 2024-09-17T15:48:39.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:38.979+0000 7f669b76f640 1 -- 172.21.15.86:0/174949416 wait complete. 2024-09-17T15:48:39.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:48:39.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:48:39.226 INFO:tasks.rados.rados.0.smithi203.stdout:3374: done (2 left) 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3375: rollback oid 34 current snap is 362 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 355 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3373: finishing copy_from to smithi20328086-4 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3373: got expected ENOENT (src dne) 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3373: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: finishing write tid 3 to smithi20328086-33 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: finishing write tid 4 to smithi20328086-33 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: finishing write tid 5 to smithi20328086-33 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: oid 33 updating version 0 to 1390 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: oid 33 version 1390 is already newer than 1389 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: oid 33 updating version 1390 to 1391 2024-09-17T15:48:39.227 INFO:tasks.rados.rados.0.smithi203.stdout:3372: oid 33 updating version 1391 to 1392 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1392 (ObjNum 1181 snap 362 seq_num 1181) dirty exists 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3372: left oid 33 (ObjNum 1181 snap 362 seq_num 1181) 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3372: done (2 left) 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3373: done (1 left) 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3376: read oid 26 snap -1 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3376: expect deleted 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3377: read oid 32 snap -1 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3377: expect (ObjNum 1156 snap 357 seq_num 1156) 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:3378: rollback oid 47 current snap is 362 2024-09-17T15:48:39.228 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 352 2024-09-17T15:48:39.229 INFO:tasks.rados.rados.0.smithi203.stdout:3379: delete oid 4 current snap is 362 2024-09-17T15:48:39.247 INFO:tasks.rados.rados.0.smithi203.stdout:3376: done (4 left) 2024-09-17T15:48:39.247 INFO:tasks.rados.rados.0.smithi203.stdout:3379: done (3 left) 2024-09-17T15:48:39.247 INFO:tasks.rados.rados.0.smithi203.stdout:3380: write oid 23 current snap is 362 2024-09-17T15:48:39.247 INFO:tasks.rados.rados.0.smithi203.stdout:3380: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-17T15:48:39.267 INFO:tasks.rados.rados.0.smithi203.stdout:3380: writing smithi20328086-23 from 682681 to 1336963 tid 1 2024-09-17T15:48:39.281 INFO:tasks.rados.rados.0.smithi203.stdout:3380: writing smithi20328086-23 from 1876551 to 2487291 tid 2 2024-09-17T15:48:39.294 INFO:tasks.rados.rados.0.smithi203.stdout:3380: writing smithi20328086-23 from 3053871 to 3524074 tid 3 2024-09-17T15:48:39.294 INFO:tasks.rados.rados.0.smithi203.stdout:3381: read oid 36 snap -1 2024-09-17T15:48:39.294 INFO:tasks.rados.rados.0.smithi203.stdout:3381: expect (ObjNum 1135 snap 353 seq_num 1135) 2024-09-17T15:48:39.294 INFO:tasks.rados.rados.0.smithi203.stdout:3382: rollback oid 9 current snap is 362 2024-09-17T15:48:39.294 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 9 to 335 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3375: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3378: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3375: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1316 (ObjNum 1201 snap 0 seq_num 4127638664) dirty dne 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3375: done (5 left) 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3383: write (excl) oid 3 current snap is 362 2024-09-17T15:48:39.295 INFO:tasks.rados.rados.0.smithi203.stdout:3383: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2024-09-17T15:48:39.309 INFO:tasks.rados.rados.0.smithi203.stdout:3383: writing smithi20328086-3 from 699488 to 1429019 tid 1 2024-09-17T15:48:39.317 INFO:tasks.rados.rados.0.smithi203.stdout:3383: writing smithi20328086-3 from 1935033 to 2405784 tid 2 2024-09-17T15:48:39.327 INFO:tasks.rados.rados.0.smithi203.stdout:3383: writing smithi20328086-3 from 3081294 to 3762769 tid 3 2024-09-17T15:48:39.327 INFO:tasks.rados.rados.0.smithi203.stdout:3384: write oid 41 current snap is 362 2024-09-17T15:48:39.327 INFO:tasks.rados.rados.0.smithi203.stdout:3384: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-09-17T15:48:39.334 INFO:tasks.rados.rados.0.smithi203.stdout:3384: writing smithi20328086-41 from 716295 to 1121075 tid 1 2024-09-17T15:48:39.341 INFO:tasks.rados.rados.0.smithi203.stdout:3384: writing smithi20328086-41 from 1593515 to 2207924 tid 2 2024-09-17T15:48:39.347 INFO:tasks.rados.rados.0.smithi203.stdout:3384: writing smithi20328086-41 from 2708717 to 3201464 tid 3 2024-09-17T15:48:39.352 INFO:tasks.rados.rados.0.smithi203.stdout:3384: writing smithi20328086-41 from 3640519 to 3916295 tid 4 2024-09-17T15:48:39.352 INFO:tasks.rados.rados.0.smithi203.stdout:3385: write (excl) oid 19 current snap is 362 2024-09-17T15:48:39.352 INFO:tasks.rados.rados.0.smithi203.stdout:3385: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-17T15:48:39.360 INFO:tasks.rados.rados.0.smithi203.stdout:3385: writing smithi20328086-19 from 733102 to 1329484 tid 1 2024-09-17T15:48:39.367 INFO:tasks.rados.rados.0.smithi203.stdout:3385: writing smithi20328086-19 from 1768350 to 2526417 tid 2 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3385: writing smithi20328086-19 from 3136140 to 3840159 tid 3 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3386: read oid 45 snap 341 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3386: expect deleted 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3387: copy_from oid 27 from oid 7 current snap is 362 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3388: read oid 34 snap 361 2024-09-17T15:48:39.376 INFO:tasks.rados.rados.0.smithi203.stdout:3388: expect (ObjNum 1022 snap 316 seq_num 1022) 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:3389: rollback oid 16 current snap is 362 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 16 to 335 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:3378: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:3378: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1540 (ObjNum 1117 snap 344 seq_num 1117) dirty exists 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:3377: expect (ObjNum 1156 snap 357 seq_num 1156) 2024-09-17T15:48:39.377 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:39.575 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:39.718 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: finishing write tid 1 to smithi20328086-23 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: finishing write tid 2 to smithi20328086-23 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3382: finishing rollback tid 1 to smithi20328086-9 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3382: finishing rollback tid 2 to smithi20328086-9 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1538 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: finishing write tid 3 to smithi20328086-23 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: finishing write tid 4 to smithi20328086-23 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: finishing write tid 5 to smithi20328086-23 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: oid 23 updating version 0 to 1555 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: oid 23 version 1555 is already newer than 1554 2024-09-17T15:48:39.862 INFO:tasks.rados.rados.0.smithi203.stdout:3380: oid 23 updating version 1555 to 1556 2024-09-17T15:48:39.863 INFO:tasks.rados.rados.0.smithi203.stdout:3380: oid 23 version 1556 is already newer than 1553 2024-09-17T15:48:39.863 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1556 (ObjNum 1183 snap 362 seq_num 1183) dirty exists 2024-09-17T15:48:39.863 INFO:tasks.rados.rados.0.smithi203.stdout:3380: left oid 23 (ObjNum 1183 snap 362 seq_num 1183) 2024-09-17T15:48:39.863 INFO:tasks.rados.rados.0.smithi203.stdout:3381: expect (ObjNum 1135 snap 353 seq_num 1135) 2024-09-17T15:48:39.909 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:39.972 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2149735 2024-09-17T15:48:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:3383: finishing write tid 1 to smithi20328086-3 2024-09-17T15:48:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:3383: finishing write tid 2 to smithi20328086-3 2024-09-17T15:48:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:3383: finishing write tid 3 to smithi20328086-3 2024-09-17T15:48:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:3383: finishing write tid 4 to smithi20328086-3 2024-09-17T15:48:39.993 INFO:tasks.rados.rados.0.smithi203.stdout:3383: finishing write tid 5 to smithi20328086-3 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3383: oid 3 updating version 0 to 1414 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3383: oid 3 updating version 1414 to 1415 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3383: oid 3 updating version 1415 to 1416 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3383: oid 3 version 1416 is already newer than 1413 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1416 (ObjNum 1184 snap 362 seq_num 1184) dirty exists 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3383: left oid 3 (ObjNum 1184 snap 362 seq_num 1184) 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 1 to smithi20328086-41 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 2 to smithi20328086-41 2024-09-17T15:48:39.994 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 3 to smithi20328086-41 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 4 to smithi20328086-41 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 5 to smithi20328086-41 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: finishing write tid 6 to smithi20328086-41 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: oid 41 updating version 0 to 683 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: oid 41 updating version 683 to 685 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: oid 41 version 685 is already newer than 684 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: oid 41 version 685 is already newer than 682 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: oid 41 version 685 is already newer than 681 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 685 (ObjNum 1185 snap 362 seq_num 1185) dirty exists 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3384: left oid 41 (ObjNum 1185 snap 362 seq_num 1185) 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3385: finishing write tid 1 to smithi20328086-19 2024-09-17T15:48:39.995 INFO:tasks.rados.rados.0.smithi203.stdout:3388: expect (ObjNum 1022 snap 316 seq_num 1022) 2024-09-17T15:48:40.101 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2954082 2024-09-17T15:48:40.172 INFO:tasks.rados.rados.0.smithi203.stdout:3385: finishing write tid 2 to smithi20328086-19 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: finishing write tid 3 to smithi20328086-19 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: finishing write tid 4 to smithi20328086-19 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: finishing write tid 5 to smithi20328086-19 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: oid 19 updating version 0 to 1730 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: oid 19 updating version 1730 to 1731 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: oid 19 updating version 1731 to 1733 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: oid 19 version 1733 is already newer than 1732 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1733 (ObjNum 1186 snap 362 seq_num 1186) dirty exists 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3385: left oid 19 (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3389: finishing rollback tid 0 to smithi20328086-16 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3377: done (11 left) 2024-09-17T15:48:40.173 INFO:tasks.rados.rados.0.smithi203.stdout:3378: done (10 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3380: done (9 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3381: done (8 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3382: done (7 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3383: done (6 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3384: done (5 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3385: done (4 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3386: done (3 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3388: done (2 left) 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3390: write oid 35 current snap is 362 2024-09-17T15:48:40.174 INFO:tasks.rados.rados.0.smithi203.stdout:3390: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-17T15:48:40.184 INFO:tasks.rados.rados.0.smithi203.stdout:3390: writing smithi20328086-35 from 749909 to 1421540 tid 1 2024-09-17T15:48:40.195 INFO:tasks.rados.rados.0.smithi203.stdout:3390: writing smithi20328086-35 from 2110479 to 2728557 tid 2 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3390: writing smithi20328086-35 from 3163563 to 3678854 tid 3 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3391: read oid 11 snap -1 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3391: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3392: copy_from oid 33 from oid 46 current snap is 362 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3389: finishing rollback tid 1 to smithi20328086-16 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3389: finishing rollback tid 2 to smithi20328086-16 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1737 (ObjNum 1084 snap 334 seq_num 1084) dirty exists 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3387: finishing copy_from racing read to smithi20328086-27 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3387: finishing copy_from to smithi20328086-27 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1198 (ObjNum 1053 snap 327 seq_num 1053) dirty exists 2024-09-17T15:48:40.204 INFO:tasks.rados.rados.0.smithi203.stdout:3390: finishing write tid 1 to smithi20328086-35 2024-09-17T15:48:40.205 INFO:tasks.rados.rados.0.smithi203.stdout:3390: finishing write tid 2 to smithi20328086-35 2024-09-17T15:48:40.205 INFO:tasks.rados.rados.0.smithi203.stdout:3387: done (4 left) 2024-09-17T15:48:40.205 INFO:tasks.rados.rados.0.smithi203.stdout:3389: done (3 left) 2024-09-17T15:48:40.205 INFO:tasks.rados.rados.0.smithi203.stdout:3393: snap_create 2024-09-17T15:48:40.210 INFO:tasks.rados.rados.0.smithi203.stdout:3391: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: finishing write tid 3 to smithi20328086-35 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: finishing write tid 4 to smithi20328086-35 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: finishing write tid 5 to smithi20328086-35 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: oid 35 updating version 0 to 1319 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: oid 35 version 1319 is already newer than 1318 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: oid 35 version 1319 is already newer than 1317 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: oid 35 updating version 1319 to 1320 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1320 (ObjNum 1187 snap 362 seq_num 1187) dirty exists 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3390: left oid 35 (ObjNum 1187 snap 362 seq_num 1187) 2024-09-17T15:48:40.436 INFO:tasks.rados.rados.0.smithi203.stdout:3392: finishing copy_from racing read to smithi20328086-33 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3392: finishing copy_from to smithi20328086-33 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1409 (ObjNum 1148 snap 355 seq_num 1148) dirty exists 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3390: done (3 left) 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3391: done (2 left) 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3392: done (1 left) 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3393: done (0 left) 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3394: rollback oid 15 current snap is 363 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 355 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3395: write oid 32 current snap is 363 2024-09-17T15:48:40.437 INFO:tasks.rados.rados.0.smithi203.stdout:3395: seq_num 1188 ranges {766716=746880,2168961=761736,3474633=442916} 2024-09-17T15:48:40.452 INFO:tasks.rados.rados.0.smithi203.stdout:3395: writing smithi20328086-32 from 766716 to 1513596 tid 1 2024-09-17T15:48:40.468 INFO:tasks.rados.rados.0.smithi203.stdout:3395: writing smithi20328086-32 from 2168961 to 2930697 tid 2 2024-09-17T15:48:40.479 INFO:tasks.rados.rados.0.smithi203.stdout:3395: writing smithi20328086-32 from 3474633 to 3917549 tid 3 2024-09-17T15:48:40.479 INFO:tasks.rados.rados.0.smithi203.stdout:3394: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:48:40.479 INFO:tasks.rados.rados.0.smithi203.stdout:3394: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:48:40.479 INFO:tasks.rados.rados.0.smithi203.stdout:3394: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:48:40.479 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1696 (ObjNum 995 snap 311 seq_num 995) dirty exists 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3395: finishing write tid 1 to smithi20328086-32 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3395: finishing write tid 2 to smithi20328086-32 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3396: read oid 27 snap 341 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3396: expect (ObjNum 1079 snap 334 seq_num 1079) 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3394: done (2 left) 2024-09-17T15:48:40.480 INFO:tasks.rados.rados.0.smithi203.stdout:3397: delete oid 23 current snap is 363 2024-09-17T15:48:40.487 INFO:tasks.rados.rados.0.smithi203.stdout:3395: finishing write tid 3 to smithi20328086-32 2024-09-17T15:48:40.488 INFO:tasks.rados.rados.0.smithi203.stdout:3397: done (2 left) 2024-09-17T15:48:40.488 INFO:tasks.rados.rados.0.smithi203.stdout:3398: write oid 31 current snap is 363 2024-09-17T15:48:40.488 INFO:tasks.rados.rados.0.smithi203.stdout:3398: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-17T15:48:40.493 INFO:tasks.rados.rados.0.smithi203.stdout:3398: writing smithi20328086-31 from 400330 to 897708 tid 1 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3398: writing smithi20328086-31 from 1485925 to 2000330 tid 2 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3399: read oid 34 snap -1 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3399: expect deleted 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3400: copy_from oid 11 from oid 45 current snap is 363 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: finishing write tid 4 to smithi20328086-32 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: finishing write tid 5 to smithi20328086-32 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: oid 32 updating version 0 to 1716 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: oid 32 updating version 1716 to 1717 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: oid 32 version 1717 is already newer than 1715 2024-09-17T15:48:40.500 INFO:tasks.rados.rados.0.smithi203.stdout:3395: oid 32 updating version 1717 to 1718 2024-09-17T15:48:40.501 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1718 (ObjNum 1188 snap 363 seq_num 1188) dirty exists 2024-09-17T15:48:40.501 INFO:tasks.rados.rados.0.smithi203.stdout:3395: left oid 32 (ObjNum 1188 snap 363 seq_num 1188) 2024-09-17T15:48:40.501 INFO:tasks.rados.rados.0.smithi203.stdout:3396: expect (ObjNum 1079 snap 334 seq_num 1079) 2024-09-17T15:48:40.663 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: finishing write tid 1 to smithi20328086-31 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3400: finishing copy_from to smithi20328086-11 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3400: got expected ENOENT (src dne) 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3400: finishing copy_from racing read to smithi20328086-11 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: finishing write tid 2 to smithi20328086-31 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: finishing write tid 3 to smithi20328086-31 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: finishing write tid 4 to smithi20328086-31 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: oid 31 updating version 0 to 1328 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: oid 31 version 1328 is already newer than 1327 2024-09-17T15:48:40.724 INFO:tasks.rados.rados.0.smithi203.stdout:3398: oid 31 updating version 1328 to 1329 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1329 (ObjNum 1190 snap 363 seq_num 1190) dirty exists 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3398: left oid 31 (ObjNum 1190 snap 363 seq_num 1190) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3395: done (4 left) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3396: done (3 left) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3398: done (2 left) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3399: done (1 left) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3400: done (0 left) 2024-09-17T15:48:40.725 INFO:tasks.rados.rados.0.smithi203.stdout:3401: delete oid 37 current snap is 363 2024-09-17T15:48:40.730 INFO:tasks.rados.rados.0.smithi203.stdout:3401: done (0 left) 2024-09-17T15:48:40.732 INFO:tasks.rados.rados.0.smithi203.stdout:3402: write oid 46 current snap is 363 2024-09-17T15:48:40.732 INFO:tasks.rados.rados.0.smithi203.stdout:3402: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-17T15:48:40.747 INFO:tasks.rados.rados.0.smithi203.stdout:3402: writing smithi20328086-46 from 433944 to 1081820 tid 1 2024-09-17T15:48:40.761 INFO:tasks.rados.rados.0.smithi203.stdout:3402: writing smithi20328086-46 from 1486536 to 2033944 tid 2 2024-09-17T15:48:40.761 INFO:tasks.rados.rados.0.smithi203.stdout:3403: write (excl) oid 21 current snap is 363 2024-09-17T15:48:40.762 INFO:tasks.rados.rados.0.smithi203.stdout:3403: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-17T15:48:40.778 INFO:tasks.rados.rados.0.smithi203.stdout:3403: writing smithi20328086-21 from 450751 to 1173876 tid 1 2024-09-17T15:48:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:3403: writing smithi20328086-21 from 1945018 to 2050751 tid 2 2024-09-17T15:48:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:3404: read oid 39 snap -1 2024-09-17T15:48:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:3404: expect deleted 2024-09-17T15:48:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:3405: write oid 32 current snap is 363 2024-09-17T15:48:40.789 INFO:tasks.rados.rados.0.smithi203.stdout:3405: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-17T15:48:40.798 INFO:tasks.rados.rados.0.smithi203.stdout:3405: writing smithi20328086-32 from 467558 to 982285 tid 1 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3405: writing smithi20328086-32 from 1719853 to 2067558 tid 2 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3406: copy_from oid 14 from oid 6 current snap is 363 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3407: read oid 38 snap 361 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3407: expect deleted 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3408: delete oid 42 current snap is 363 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3402: finishing write tid 1 to smithi20328086-46 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3402: finishing write tid 2 to smithi20328086-46 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3402: finishing write tid 3 to smithi20328086-46 2024-09-17T15:48:40.808 INFO:tasks.rados.rados.0.smithi203.stdout:3402: finishing write tid 4 to smithi20328086-46 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3402: oid 46 updating version 0 to 1419 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3402: oid 46 version 1419 is already newer than 1417 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3402: oid 46 version 1419 is already newer than 1418 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1419 (ObjNum 1192 snap 363 seq_num 1192) dirty exists 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3402: left oid 46 (ObjNum 1192 snap 363 seq_num 1192) 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: finishing write tid 1 to smithi20328086-21 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: finishing write tid 2 to smithi20328086-21 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: finishing write tid 3 to smithi20328086-21 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: finishing write tid 4 to smithi20328086-21 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: oid 21 updating version 0 to 1743 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: oid 21 version 1743 is already newer than 1741 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:3403: oid 21 version 1743 is already newer than 1742 2024-09-17T15:48:40.809 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1743 (ObjNum 1193 snap 363 seq_num 1193) dirty exists 2024-09-17T15:48:40.810 INFO:tasks.rados.rados.0.smithi203.stdout:3403: left oid 21 (ObjNum 1193 snap 363 seq_num 1193) 2024-09-17T15:48:40.810 INFO:tasks.rados.rados.0.smithi203.stdout:3405: finishing write tid 1 to smithi20328086-32 2024-09-17T15:48:40.810 INFO:tasks.rados.rados.0.smithi203.stdout:3406: finishing copy_from to smithi20328086-14 2024-09-17T15:48:40.810 INFO:tasks.rados.rados.0.smithi203.stdout:3406: got expected ENOENT (src dne) 2024-09-17T15:48:40.810 INFO:tasks.rados.rados.0.smithi203.stdout:3406: finishing copy_from racing read to smithi20328086-14 2024-09-17T15:48:40.813 INFO:tasks.rados.rados.0.smithi203.stdout:3402: done (6 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3403: done (5 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3404: done (4 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3406: done (3 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3407: done (2 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3408: done (1 left) 2024-09-17T15:48:40.814 INFO:tasks.rados.rados.0.smithi203.stdout:3409: snap_remove snap 355 2024-09-17T15:48:40.829 INFO:tasks.rados.rados.0.smithi203.stdout:3409: done (1 left) 2024-09-17T15:48:40.829 INFO:tasks.rados.rados.0.smithi203.stdout:3410: copy_from oid 1 from oid 49 current snap is 363 2024-09-17T15:48:40.829 INFO:tasks.rados.rados.0.smithi203.stdout:3405: finishing write tid 2 to smithi20328086-32 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: finishing write tid 3 to smithi20328086-32 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: finishing write tid 4 to smithi20328086-32 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: oid 32 updating version 0 to 1721 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: oid 32 version 1721 is already newer than 1720 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: oid 32 version 1721 is already newer than 1719 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1721 (ObjNum 1194 snap 363 seq_num 1194) dirty exists 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: left oid 32 (ObjNum 1194 snap 363 seq_num 1194) 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3405: done (1 left) 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3411: write oid 3 current snap is 363 2024-09-17T15:48:40.830 INFO:tasks.rados.rados.0.smithi203.stdout:3411: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-17T15:48:40.836 INFO:tasks.rados.rados.0.smithi203.stdout:3411: writing smithi20328086-3 from 501172 to 1166397 tid 1 2024-09-17T15:48:40.842 INFO:tasks.rados.rados.0.smithi203.stdout:3411: writing smithi20328086-3 from 1720464 to 2101172 tid 2 2024-09-17T15:48:40.842 INFO:tasks.rados.rados.0.smithi203.stdout:3412: read oid 12 snap -1 2024-09-17T15:48:40.842 INFO:tasks.rados.rados.0.smithi203.stdout:3412: expect deleted 2024-09-17T15:48:40.842 INFO:tasks.rados.rados.0.smithi203.stdout:3413: snap_remove snap 339 2024-09-17T15:48:41.840 INFO:tasks.rados.rados.0.smithi203.stdout:3413: done (3 left) 2024-09-17T15:48:41.841 INFO:tasks.rados.rados.0.smithi203.stdout:3414: write (excl) oid 26 current snap is 363 2024-09-17T15:48:41.841 INFO:tasks.rados.rados.0.smithi203.stdout:3414: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-17T15:48:41.859 INFO:tasks.rados.rados.0.smithi203.stdout:3414: writing smithi20328086-26 from 517979 to 1258453 tid 1 2024-09-17T15:48:41.871 INFO:tasks.rados.rados.0.smithi203.stdout:3414: writing smithi20328086-26 from 1778946 to 2117979 tid 2 2024-09-17T15:48:41.871 INFO:tasks.rados.rados.0.smithi203.stdout:3415: snap_create 2024-09-17T15:48:41.871 INFO:tasks.rados.rados.0.smithi203.stdout:3411: finishing write tid 1 to smithi20328086-3 2024-09-17T15:48:41.871 INFO:tasks.rados.rados.0.smithi203.stdout:3411: finishing write tid 2 to smithi20328086-3 2024-09-17T15:48:41.871 INFO:tasks.rados.rados.0.smithi203.stdout:3411: finishing write tid 3 to smithi20328086-3 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3411: finishing write tid 4 to smithi20328086-3 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3411: oid 3 updating version 0 to 1421 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3411: oid 3 updating version 1421 to 1422 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3411: oid 3 version 1422 is already newer than 1420 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1422 (ObjNum 1196 snap 363 seq_num 1196) dirty exists 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3411: left oid 3 (ObjNum 1196 snap 363 seq_num 1196) 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3410: finishing copy_from racing read to smithi20328086-1 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3410: finishing copy_from to smithi20328086-1 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1634 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:48:41.872 INFO:tasks.rados.rados.0.smithi203.stdout:3414: finishing write tid 1 to smithi20328086-26 2024-09-17T15:48:41.878 INFO:tasks.rados.rados.0.smithi203.stdout:3414: finishing write tid 2 to smithi20328086-26 2024-09-17T15:48:41.879 INFO:tasks.rados.rados.0.smithi203.stdout:3414: finishing write tid 3 to smithi20328086-26 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:3414: finishing write tid 4 to smithi20328086-26 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:3414: oid 26 updating version 0 to 1748 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:3414: oid 26 updating version 1748 to 1750 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:3414: oid 26 version 1750 is already newer than 1749 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1750 (ObjNum 1197 snap 363 seq_num 1197) dirty exists 2024-09-17T15:48:41.880 INFO:tasks.rados.rados.0.smithi203.stdout:3414: left oid 26 (ObjNum 1197 snap 363 seq_num 1197) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3410: done (4 left) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3411: done (3 left) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3412: done (2 left) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3414: done (1 left) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3415: done (0 left) 2024-09-17T15:48:42.846 INFO:tasks.rados.rados.0.smithi203.stdout:3416: snap_remove snap 335 2024-09-17T15:48:43.848 INFO:tasks.rados.rados.0.smithi203.stdout:3416: done (0 left) 2024-09-17T15:48:43.849 INFO:tasks.rados.rados.0.smithi203.stdout:3417: snap_remove snap 354 2024-09-17T15:48:44.849 INFO:tasks.rados.rados.0.smithi203.stdout:3417: done (0 left) 2024-09-17T15:48:44.849 INFO:tasks.rados.rados.0.smithi203.stdout:3418: read oid 15 snap -1 2024-09-17T15:48:44.850 INFO:tasks.rados.rados.0.smithi203.stdout:3418: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:48:44.850 INFO:tasks.rados.rados.0.smithi203.stdout:3419: snap_remove snap 352 2024-09-17T15:48:45.834 INFO:tasks.rados.rados.0.smithi203.stdout:3419: done (1 left) 2024-09-17T15:48:45.834 INFO:tasks.rados.rados.0.smithi203.stdout:3420: read oid 11 snap -1 2024-09-17T15:48:45.834 INFO:tasks.rados.rados.0.smithi203.stdout:3420: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:48:45.834 INFO:tasks.rados.rados.0.smithi203.stdout:3421: write oid 36 current snap is 364 2024-09-17T15:48:45.834 INFO:tasks.rados.rados.0.smithi203.stdout:3418: expect (ObjNum 995 snap 311 seq_num 995) 2024-09-17T15:48:45.913 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 548864 2024-09-17T15:48:45.974 INFO:tasks.rados.rados.0.smithi203.stdout:3420: expect (ObjNum 1047 snap 324 seq_num 1047) 2024-09-17T15:48:45.975 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:46.052 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 650989 2024-09-17T15:48:46.174 INFO:tasks.rados.rados.0.smithi203.stdout:3421: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-17T15:48:46.189 INFO:tasks.rados.rados.0.smithi203.stdout:3421: writing smithi20328086-36 from 534786 to 950509 tid 1 2024-09-17T15:48:46.209 INFO:tasks.rados.rados.0.smithi203.stdout:3421: writing smithi20328086-36 from 1437428 to 2134786 tid 2 2024-09-17T15:48:46.209 INFO:tasks.rados.rados.0.smithi203.stdout:3418: done (2 left) 2024-09-17T15:48:46.209 INFO:tasks.rados.rados.0.smithi203.stdout:3420: done (1 left) 2024-09-17T15:48:46.209 INFO:tasks.rados.rados.0.smithi203.stdout:3422: snap_remove snap 361 2024-09-17T15:48:46.841 INFO:tasks.rados.rados.0.smithi203.stdout:3422: done (1 left) 2024-09-17T15:48:46.841 INFO:tasks.rados.rados.0.smithi203.stdout:3423: snap_remove snap 359 2024-09-17T15:48:47.850 INFO:tasks.rados.rados.0.smithi203.stdout:3423: done (1 left) 2024-09-17T15:48:47.850 INFO:tasks.rados.rados.0.smithi203.stdout:3424: delete oid 32 current snap is 364 2024-09-17T15:48:47.850 INFO:tasks.rados.rados.0.smithi203.stdout:3421: finishing write tid 1 to smithi20328086-36 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: finishing write tid 2 to smithi20328086-36 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: finishing write tid 3 to smithi20328086-36 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: finishing write tid 4 to smithi20328086-36 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: oid 36 updating version 0 to 1764 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: oid 36 updating version 1764 to 1765 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: oid 36 updating version 1765 to 1766 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1766 (ObjNum 1198 snap 364 seq_num 1198) dirty exists 2024-09-17T15:48:47.851 INFO:tasks.rados.rados.0.smithi203.stdout:3421: left oid 36 (ObjNum 1198 snap 364 seq_num 1198) 2024-09-17T15:48:47.878 INFO:tasks.rados.rados.0.smithi203.stdout:3421: done (1 left) 2024-09-17T15:48:47.878 INFO:tasks.rados.rados.0.smithi203.stdout:3424: done (0 left) 2024-09-17T15:48:47.878 INFO:tasks.rados.rados.0.smithi203.stdout:3425: write (excl) oid 12 current snap is 364 2024-09-17T15:48:47.878 INFO:tasks.rados.rados.0.smithi203.stdout:3425: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-17T15:48:47.898 INFO:tasks.rados.rados.0.smithi203.stdout:3425: writing smithi20328086-12 from 568400 to 1134621 tid 1 2024-09-17T15:48:47.915 INFO:tasks.rados.rados.0.smithi203.stdout:3425: writing smithi20328086-12 from 1838039 to 2168400 tid 2 2024-09-17T15:48:47.915 INFO:tasks.rados.rados.0.smithi203.stdout:3426: write (excl) oid 34 current snap is 364 2024-09-17T15:48:47.915 INFO:tasks.rados.rados.0.smithi203.stdout:3425: finishing write tid 1 to smithi20328086-12 2024-09-17T15:48:47.915 INFO:tasks.rados.rados.0.smithi203.stdout:3426: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-17T15:48:47.934 INFO:tasks.rados.rados.0.smithi203.stdout:3426: writing smithi20328086-34 from 585207 to 1226677 tid 1 2024-09-17T15:48:47.947 INFO:tasks.rados.rados.0.smithi203.stdout:3426: writing smithi20328086-34 from 1896521 to 2185207 tid 2 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3427: copy_from oid 40 from oid 47 current snap is 364 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: finishing write tid 2 to smithi20328086-12 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3428: snap_create 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: finishing write tid 3 to smithi20328086-12 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: finishing write tid 4 to smithi20328086-12 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: oid 12 updating version 0 to 1699 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: oid 12 version 1699 is already newer than 1698 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: oid 12 version 1699 is already newer than 1697 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1699 (ObjNum 1200 snap 364 seq_num 1200) dirty exists 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3425: left oid 12 (ObjNum 1200 snap 364 seq_num 1200) 2024-09-17T15:48:47.948 INFO:tasks.rados.rados.0.smithi203.stdout:3426: finishing write tid 1 to smithi20328086-34 2024-09-17T15:48:47.954 INFO:tasks.rados.rados.0.smithi203.stdout:3426: finishing write tid 2 to smithi20328086-34 2024-09-17T15:48:47.954 INFO:tasks.rados.rados.0.smithi203.stdout:3426: finishing write tid 3 to smithi20328086-34 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:3426: finishing write tid 4 to smithi20328086-34 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:3426: oid 34 updating version 0 to 1322 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:3426: oid 34 updating version 1322 to 1323 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:3426: oid 34 version 1323 is already newer than 1321 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1323 (ObjNum 1201 snap 364 seq_num 1201) dirty exists 2024-09-17T15:48:47.955 INFO:tasks.rados.rados.0.smithi203.stdout:3426: left oid 34 (ObjNum 1201 snap 364 seq_num 1201) 2024-09-17T15:48:47.969 INFO:tasks.rados.rados.0.smithi203.stdout:3427: finishing copy_from racing read to smithi20328086-40 2024-09-17T15:48:47.972 INFO:tasks.rados.rados.0.smithi203.stdout:3427: finishing copy_from to smithi20328086-40 2024-09-17T15:48:47.973 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 1540 (ObjNum 1117 snap 344 seq_num 1117) dirty exists 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3425: done (3 left) 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3426: done (2 left) 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3427: done (1 left) 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3428: done (0 left) 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3429: copy_from oid 7 from oid 32 current snap is 365 2024-09-17T15:48:48.856 INFO:tasks.rados.rados.0.smithi203.stdout:3430: delete oid 24 current snap is 365 2024-09-17T15:48:48.865 INFO:tasks.rados.rados.0.smithi203.stdout:3429: finishing copy_from to smithi20328086-7 2024-09-17T15:48:48.866 INFO:tasks.rados.rados.0.smithi203.stdout:3429: got expected ENOENT (src dne) 2024-09-17T15:48:48.866 INFO:tasks.rados.rados.0.smithi203.stdout:3429: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:48:48.871 INFO:tasks.rados.rados.0.smithi203.stdout:3429: done (1 left) 2024-09-17T15:48:48.872 INFO:tasks.rados.rados.0.smithi203.stdout:3430: done (0 left) 2024-09-17T15:48:48.872 INFO:tasks.rados.rados.0.smithi203.stdout:3431: write (excl) oid 24 current snap is 365 2024-09-17T15:48:48.872 INFO:tasks.rados.rados.0.smithi203.stdout:3431: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-17T15:48:48.890 INFO:tasks.rados.rados.0.smithi203.stdout:3431: writing smithi20328086-24 from 618821 to 1127142 tid 1 2024-09-17T15:48:48.907 INFO:tasks.rados.rados.0.smithi203.stdout:3431: writing smithi20328086-24 from 1613485 to 2144562 tid 2 2024-09-17T15:48:48.907 INFO:tasks.rados.rados.0.smithi203.stdout:3432: delete oid 33 current snap is 365 2024-09-17T15:48:48.907 INFO:tasks.rados.rados.0.smithi203.stdout:3431: finishing write tid 1 to smithi20328086-24 2024-09-17T15:48:48.913 INFO:tasks.rados.rados.0.smithi203.stdout:3432: done (1 left) 2024-09-17T15:48:48.913 INFO:tasks.rados.rados.0.smithi203.stdout:3433: write (excl) oid 41 current snap is 365 2024-09-17T15:48:48.913 INFO:tasks.rados.rados.0.smithi203.stdout:3433: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-17T15:48:48.934 INFO:tasks.rados.rados.0.smithi203.stdout:3433: writing smithi20328086-41 from 652435 to 1311254 tid 1 2024-09-17T15:48:48.948 INFO:tasks.rados.rados.0.smithi203.stdout:3433: writing smithi20328086-41 from 1730449 to 2252435 tid 2 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3434: read oid 48 snap -1 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3434: expect deleted 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3435: rollback oid 49 current snap is 365 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 363 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: finishing write tid 2 to smithi20328086-24 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: finishing write tid 3 to smithi20328086-24 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: finishing write tid 4 to smithi20328086-24 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: oid 24 updating version 0 to 1541 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: oid 24 version 1541 is already newer than 1540 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: oid 24 updating version 1541 to 1542 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1542 (ObjNum 1203 snap 365 seq_num 1203) dirty exists 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: left oid 24 (ObjNum 1203 snap 365 seq_num 1203) 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3433: finishing write tid 1 to smithi20328086-41 2024-09-17T15:48:48.949 INFO:tasks.rados.rados.0.smithi203.stdout:3431: done (3 left) 2024-09-17T15:48:48.950 INFO:tasks.rados.rados.0.smithi203.stdout:3436: delete oid 40 current snap is 365 2024-09-17T15:48:48.957 INFO:tasks.rados.rados.0.smithi203.stdout:3434: done (3 left) 2024-09-17T15:48:48.957 INFO:tasks.rados.rados.0.smithi203.stdout:3436: done (2 left) 2024-09-17T15:48:48.957 INFO:tasks.rados.rados.0.smithi203.stdout:3437: read oid 45 snap -1 2024-09-17T15:48:48.957 INFO:tasks.rados.rados.0.smithi203.stdout:3437: expect deleted 2024-09-17T15:48:48.957 INFO:tasks.rados.rados.0.smithi203.stdout:3438: snap_remove snap 356 2024-09-17T15:48:49.867 INFO:tasks.rados.rados.0.smithi203.stdout:3438: done (3 left) 2024-09-17T15:48:49.867 INFO:tasks.rados.rados.0.smithi203.stdout:3439: rollback oid 2 current snap is 365 2024-09-17T15:48:49.867 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 345 2024-09-17T15:48:49.867 INFO:tasks.rados.rados.0.smithi203.stdout:3433: finishing write tid 2 to smithi20328086-41 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: finishing write tid 3 to smithi20328086-41 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: finishing write tid 4 to smithi20328086-41 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: oid 41 updating version 0 to 688 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: oid 41 version 688 is already newer than 687 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: oid 41 version 688 is already newer than 686 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 688 (ObjNum 1205 snap 365 seq_num 1205) dirty exists 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: left oid 41 (ObjNum 1205 snap 365 seq_num 1205) 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3435: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3435: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3435: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1776 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:48:49.868 INFO:tasks.rados.rados.0.smithi203.stdout:3433: done (3 left) 2024-09-17T15:48:49.869 INFO:tasks.rados.rados.0.smithi203.stdout:3435: done (2 left) 2024-09-17T15:48:49.869 INFO:tasks.rados.rados.0.smithi203.stdout:3437: done (1 left) 2024-09-17T15:48:49.869 INFO:tasks.rados.rados.0.smithi203.stdout:3440: read oid 44 snap -1 2024-09-17T15:48:49.869 INFO:tasks.rados.rados.0.smithi203.stdout:3440: expect (ObjNum 1169 snap 359 seq_num 1169) 2024-09-17T15:48:49.869 INFO:tasks.rados.rados.0.smithi203.stdout:3441: snap_remove snap 341 2024-09-17T15:48:51.114 INFO:tasks.rados.rados.0.smithi203.stdout:3441: done (2 left) 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:3442: delete oid 49 current snap is 365 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:3439: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:3439: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:3439: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1204 (ObjNum 1074 snap 332 seq_num 1074) dirty exists 2024-09-17T15:48:51.115 INFO:tasks.rados.rados.0.smithi203.stdout:3440: expect (ObjNum 1169 snap 359 seq_num 1169) 2024-09-17T15:48:51.256 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:51.340 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:51.425 INFO:tasks.rados.rados.0.smithi203.stdout:3439: done (2 left) 2024-09-17T15:48:51.425 INFO:tasks.rados.rados.0.smithi203.stdout:3440: done (1 left) 2024-09-17T15:48:51.425 INFO:tasks.rados.rados.0.smithi203.stdout:3442: done (0 left) 2024-09-17T15:48:51.425 INFO:tasks.rados.rados.0.smithi203.stdout:3443: write (excl) oid 48 current snap is 365 2024-09-17T15:48:51.425 INFO:tasks.rados.rados.0.smithi203.stdout:3443: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-17T15:48:51.444 INFO:tasks.rados.rados.0.smithi203.stdout:3443: writing smithi20328086-48 from 702856 to 1187422 tid 1 2024-09-17T15:48:51.463 INFO:tasks.rados.rados.0.smithi203.stdout:3443: writing smithi20328086-48 from 1789542 to 2302856 tid 2 2024-09-17T15:48:51.463 INFO:tasks.rados.rados.0.smithi203.stdout:3444: delete oid 33 current snap is 365 2024-09-17T15:48:51.463 INFO:tasks.rados.rados.0.smithi203.stdout:3443: finishing write tid 1 to smithi20328086-48 2024-09-17T15:48:51.466 INFO:tasks.rados.rados.0.smithi203.stdout:3444: done (1 left) 2024-09-17T15:48:51.466 INFO:tasks.rados.rados.0.smithi203.stdout:3445: delete oid 50 current snap is 365 2024-09-17T15:48:51.471 INFO:tasks.rados.rados.0.smithi203.stdout:3443: finishing write tid 2 to smithi20328086-48 2024-09-17T15:48:51.472 INFO:tasks.rados.rados.0.smithi203.stdout:3443: finishing write tid 3 to smithi20328086-48 2024-09-17T15:48:51.472 INFO:tasks.rados.rados.0.smithi203.stdout:3443: finishing write tid 4 to smithi20328086-48 2024-09-17T15:48:51.472 INFO:tasks.rados.rados.0.smithi203.stdout:3443: oid 48 updating version 0 to 1802 2024-09-17T15:48:51.473 INFO:tasks.rados.rados.0.smithi203.stdout:3443: oid 48 updating version 1802 to 1804 2024-09-17T15:48:51.473 INFO:tasks.rados.rados.0.smithi203.stdout:3443: oid 48 version 1804 is already newer than 1803 2024-09-17T15:48:51.473 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1804 (ObjNum 1208 snap 365 seq_num 1208) dirty exists 2024-09-17T15:48:51.473 INFO:tasks.rados.rados.0.smithi203.stdout:3443: left oid 48 (ObjNum 1208 snap 365 seq_num 1208) 2024-09-17T15:48:51.474 INFO:tasks.rados.rados.0.smithi203.stdout:3443: done (1 left) 2024-09-17T15:48:51.475 INFO:tasks.rados.rados.0.smithi203.stdout:3445: done (0 left) 2024-09-17T15:48:51.475 INFO:tasks.rados.rados.0.smithi203.stdout:3446: read oid 39 snap -1 2024-09-17T15:48:51.475 INFO:tasks.rados.rados.0.smithi203.stdout:3446: expect deleted 2024-09-17T15:48:51.475 INFO:tasks.rados.rados.0.smithi203.stdout:3447: delete oid 18 current snap is 365 2024-09-17T15:48:51.483 INFO:tasks.rados.rados.0.smithi203.stdout:3446: done (1 left) 2024-09-17T15:48:51.483 INFO:tasks.rados.rados.0.smithi203.stdout:3447: done (0 left) 2024-09-17T15:48:51.483 INFO:tasks.rados.rados.0.smithi203.stdout:3448: rollback oid 11 current snap is 365 2024-09-17T15:48:51.483 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 362 2024-09-17T15:48:51.484 INFO:tasks.rados.rados.0.smithi203.stdout:3449: write (excl) oid 48 current snap is 365 2024-09-17T15:48:51.484 INFO:tasks.rados.rados.0.smithi203.stdout:3449: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-17T15:48:51.495 INFO:tasks.rados.rados.0.smithi203.stdout:3449: writing smithi20328086-48 from 770084 to 1271999 tid 1 2024-09-17T15:48:51.505 INFO:tasks.rados.rados.0.smithi203.stdout:3449: writing smithi20328086-48 from 2023470 to 2370084 tid 2 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3450: copy_from oid 46 from oid 40 current snap is 365 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3448: finishing rollback tid 0 to smithi20328086-11 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3448: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3448: finishing rollback tid 2 to smithi20328086-11 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1604 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3449: finishing write tid 1 to smithi20328086-48 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3448: done (2 left) 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3451: read oid 19 snap -1 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3451: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:48:51.506 INFO:tasks.rados.rados.0.smithi203.stdout:3452: delete oid 5 current snap is 365 2024-09-17T15:48:51.507 INFO:tasks.rados.rados.0.smithi203.stdout:3450: finishing copy_from to smithi20328086-46 2024-09-17T15:48:51.507 INFO:tasks.rados.rados.0.smithi203.stdout:3450: got expected ENOENT (src dne) 2024-09-17T15:48:51.507 INFO:tasks.rados.rados.0.smithi203.stdout:3450: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:48:51.513 INFO:tasks.rados.rados.0.smithi203.stdout:3450: done (3 left) 2024-09-17T15:48:51.513 INFO:tasks.rados.rados.0.smithi203.stdout:3452: done (2 left) 2024-09-17T15:48:51.513 INFO:tasks.rados.rados.0.smithi203.stdout:3453: write (excl) oid 15 current snap is 365 2024-09-17T15:48:51.514 INFO:tasks.rados.rados.0.smithi203.stdout:3453: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-17T15:48:51.523 INFO:tasks.rados.rados.0.smithi203.stdout:3453: writing smithi20328086-15 from 403698 to 1056111 tid 1 2024-09-17T15:48:51.536 INFO:tasks.rados.rados.0.smithi203.stdout:3453: writing smithi20328086-15 from 1740434 to 2403698 tid 2 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3454: rollback oid 49 current snap is 365 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 363 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: finishing write tid 2 to smithi20328086-48 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: finishing write tid 3 to smithi20328086-48 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: finishing write tid 4 to smithi20328086-48 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: oid 48 updating version 0 to 1805 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: oid 48 updating version 1805 to 1806 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: oid 48 updating version 1806 to 1807 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1807 (ObjNum 1212 snap 365 seq_num 1212) dirty exists 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: left oid 48 (ObjNum 1212 snap 365 seq_num 1212) 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3449: done (3 left) 2024-09-17T15:48:51.537 INFO:tasks.rados.rados.0.smithi203.stdout:3455: read oid 5 snap -1 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3455: expect deleted 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3456: read oid 26 snap -1 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3456: expect (ObjNum 1197 snap 363 seq_num 1197) 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3457: read oid 12 snap 345 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3457: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3458: copy_from oid 20 from oid 22 current snap is 365 2024-09-17T15:48:51.538 INFO:tasks.rados.rados.0.smithi203.stdout:3451: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:48:51.742 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1947815 2024-09-17T15:48:51.871 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3123353 2024-09-17T15:48:51.890 INFO:tasks.rados.rados.0.smithi203.stdout:3453: finishing write tid 1 to smithi20328086-15 2024-09-17T15:48:51.890 INFO:tasks.rados.rados.0.smithi203.stdout:3454: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:48:51.890 INFO:tasks.rados.rados.0.smithi203.stdout:3454: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:48:51.890 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1785 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:48:51.891 INFO:tasks.rados.rados.0.smithi203.stdout:3456: expect (ObjNum 1197 snap 363 seq_num 1197) 2024-09-17T15:48:51.912 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:51.950 INFO:tasks.rados.rados.0.smithi203.stdout:3453: finishing write tid 2 to smithi20328086-15 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: finishing write tid 3 to smithi20328086-15 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: finishing write tid 4 to smithi20328086-15 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: oid 15 updating version 0 to 1701 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: oid 15 version 1701 is already newer than 1700 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: oid 15 updating version 1701 to 1702 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1702 (ObjNum 1214 snap 365 seq_num 1214) dirty exists 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3453: left oid 15 (ObjNum 1214 snap 365 seq_num 1214) 2024-09-17T15:48:51.951 INFO:tasks.rados.rados.0.smithi203.stdout:3457: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:48:52.068 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 681573 2024-09-17T15:48:52.169 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:52.281 INFO:tasks.rados.rados.0.smithi203.stdout:3458: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:48:52.281 INFO:tasks.rados.rados.0.smithi203.stdout:3458: finishing copy_from to smithi20328086-20 2024-09-17T15:48:52.281 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1326 (ObjNum 1114 snap 343 seq_num 1114) dirty exists 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3451: done (6 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3453: done (5 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3454: done (4 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3455: done (3 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3456: done (2 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3457: done (1 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3458: done (0 left) 2024-09-17T15:48:52.282 INFO:tasks.rados.rados.0.smithi203.stdout:3459: delete oid 17 current snap is 365 2024-09-17T15:48:52.286 INFO:tasks.rados.rados.0.smithi203.stdout:3459: done (0 left) 2024-09-17T15:48:52.286 INFO:tasks.rados.rados.0.smithi203.stdout:3460: snap_create 2024-09-17T15:48:53.996 INFO:tasks.rados.rados.0.smithi203.stdout:3460: done (0 left) 2024-09-17T15:48:53.997 INFO:tasks.rados.rados.0.smithi203.stdout:3461: snap_create 2024-09-17T15:48:54.144 INFO:tasks.rados.rados.0.smithi203.stdout:3461: done (0 left) 2024-09-17T15:48:54.144 INFO:tasks.rados.rados.0.smithi203.stdout:3462: write (excl) oid 6 current snap is 367 2024-09-17T15:48:54.145 INFO:tasks.rados.rados.0.smithi203.stdout:3462: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-17T15:48:54.158 INFO:tasks.rados.rados.0.smithi203.stdout:3462: writing smithi20328086-6 from 437312 to 840223 tid 1 2024-09-17T15:48:54.179 INFO:tasks.rados.rados.0.smithi203.stdout:3462: writing smithi20328086-6 from 1341045 to 2037794 tid 2 2024-09-17T15:48:54.179 INFO:tasks.rados.rados.0.smithi203.stdout:3463: snap_create 2024-09-17T15:48:54.179 INFO:tasks.rados.rados.0.smithi203.stdout:3462: finishing write tid 1 to smithi20328086-6 2024-09-17T15:48:54.188 INFO:tasks.rados.rados.0.smithi203.stdout:3462: finishing write tid 2 to smithi20328086-6 2024-09-17T15:48:54.188 INFO:tasks.rados.rados.0.smithi203.stdout:3462: finishing write tid 3 to smithi20328086-6 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:3462: finishing write tid 4 to smithi20328086-6 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:3462: oid 6 updating version 0 to 1636 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:3462: oid 6 version 1636 is already newer than 1635 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:3462: oid 6 updating version 1636 to 1637 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1637 (ObjNum 1216 snap 367 seq_num 1216) dirty exists 2024-09-17T15:48:54.189 INFO:tasks.rados.rados.0.smithi203.stdout:3462: left oid 6 (ObjNum 1216 snap 367 seq_num 1216) 2024-09-17T15:48:55.144 INFO:tasks.rados.rados.0.smithi203.stdout:3462: done (1 left) 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3463: done (0 left) 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3464: read oid 26 snap 366 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3464: expect (ObjNum 1197 snap 363 seq_num 1197) 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3465: copy_from oid 12 from oid 11 current snap is 368 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3466: read oid 34 snap -1 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3466: expect (ObjNum 1201 snap 364 seq_num 1201) 2024-09-17T15:48:55.145 INFO:tasks.rados.rados.0.smithi203.stdout:3467: snap_remove snap 362 2024-09-17T15:48:55.837 INFO:tasks.rados.rados.0.smithi203.stdout:3467: done (3 left) 2024-09-17T15:48:55.837 INFO:tasks.rados.rados.0.smithi203.stdout:3468: write oid 27 current snap is 368 2024-09-17T15:48:55.837 INFO:tasks.rados.rados.0.smithi203.stdout:3468: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-09-17T15:48:55.855 INFO:tasks.rados.rados.0.smithi203.stdout:3468: writing smithi20328086-27 from 454119 to 1048632 tid 1 2024-09-17T15:48:55.872 INFO:tasks.rados.rados.0.smithi203.stdout:3468: writing smithi20328086-27 from 1515880 to 2072640 tid 2 2024-09-17T15:48:55.872 INFO:tasks.rados.rados.0.smithi203.stdout:3469: snap_remove snap 345 2024-09-17T15:48:56.856 INFO:tasks.rados.rados.0.smithi203.stdout:3469: done (4 left) 2024-09-17T15:48:56.856 INFO:tasks.rados.rados.0.smithi203.stdout:3470: write oid 37 current snap is 368 2024-09-17T15:48:56.857 INFO:tasks.rados.rados.0.smithi203.stdout:3470: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-17T15:48:56.878 INFO:tasks.rados.rados.0.smithi203.stdout:3470: writing smithi20328086-37 from 470926 to 1140688 tid 1 2024-09-17T15:48:56.898 INFO:tasks.rados.rados.0.smithi203.stdout:3470: writing smithi20328086-37 from 1574362 to 2274780 tid 2 2024-09-17T15:48:56.898 INFO:tasks.rados.rados.0.smithi203.stdout:3471: write oid 16 current snap is 368 2024-09-17T15:48:56.898 INFO:tasks.rados.rados.0.smithi203.stdout:3464: expect (ObjNum 1197 snap 363 seq_num 1197) 2024-09-17T15:48:56.953 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1913999 2024-09-17T15:48:56.971 INFO:tasks.rados.rados.0.smithi203.stdout:3466: expect (ObjNum 1201 snap 364 seq_num 1201) 2024-09-17T15:48:56.971 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 220365 2024-09-17T15:48:56.998 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 585828 2024-09-17T15:48:57.014 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:57.038 INFO:tasks.rados.rados.0.smithi203.stdout:3465: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3465: finishing copy_from to smithi20328086-12 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1604 (ObjNum 1047 snap 324 seq_num 1047) dirty exists 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: finishing write tid 1 to smithi20328086-27 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: finishing write tid 2 to smithi20328086-27 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: finishing write tid 3 to smithi20328086-27 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: finishing write tid 4 to smithi20328086-27 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: oid 27 updating version 0 to 1425 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: oid 27 version 1425 is already newer than 1423 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: oid 27 version 1425 is already newer than 1424 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1425 (ObjNum 1217 snap 368 seq_num 1217) dirty exists 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3468: left oid 27 (ObjNum 1217 snap 368 seq_num 1217) 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3470: finishing write tid 1 to smithi20328086-37 2024-09-17T15:48:57.039 INFO:tasks.rados.rados.0.smithi203.stdout:3470: finishing write tid 2 to smithi20328086-37 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: finishing write tid 3 to smithi20328086-37 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: finishing write tid 4 to smithi20328086-37 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: oid 37 updating version 0 to 1814 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: oid 37 updating version 1814 to 1819 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: oid 37 updating version 1819 to 1820 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1820 (ObjNum 1218 snap 368 seq_num 1218) dirty exists 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3470: left oid 37 (ObjNum 1218 snap 368 seq_num 1218) 2024-09-17T15:48:57.040 INFO:tasks.rados.rados.0.smithi203.stdout:3471: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-17T15:48:57.058 INFO:tasks.rados.rados.0.smithi203.stdout:3471: writing smithi20328086-16 from 487733 to 1232744 tid 1 2024-09-17T15:48:57.075 INFO:tasks.rados.rados.0.smithi203.stdout:3471: writing smithi20328086-16 from 1632844 to 2193273 tid 2 2024-09-17T15:48:57.075 INFO:tasks.rados.rados.0.smithi203.stdout:3464: done (5 left) 2024-09-17T15:48:57.075 INFO:tasks.rados.rados.0.smithi203.stdout:3465: done (4 left) 2024-09-17T15:48:57.075 INFO:tasks.rados.rados.0.smithi203.stdout:3466: done (3 left) 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3468: done (2 left) 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3470: done (1 left) 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3472: read oid 35 snap -1 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3472: expect (ObjNum 1187 snap 362 seq_num 1187) 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3473: write oid 34 current snap is 368 2024-09-17T15:48:57.076 INFO:tasks.rados.rados.0.smithi203.stdout:3473: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-17T15:48:57.090 INFO:tasks.rados.rados.0.smithi203.stdout:3473: writing smithi20328086-34 from 504540 to 924800 tid 1 2024-09-17T15:48:57.105 INFO:tasks.rados.rados.0.smithi203.stdout:3473: writing smithi20328086-34 from 1574973 to 2279060 tid 2 2024-09-17T15:48:57.105 INFO:tasks.rados.rados.0.smithi203.stdout:3474: write oid 20 current snap is 368 2024-09-17T15:48:57.105 INFO:tasks.rados.rados.0.smithi203.stdout:3474: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-09-17T15:48:57.114 INFO:tasks.rados.rados.0.smithi203.stdout:3474: writing smithi20328086-20 from 521347 to 1016856 tid 1 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3474: writing smithi20328086-20 from 1633455 to 2197553 tid 2 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3475: delete oid 11 current snap is 368 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3471: finishing write tid 1 to smithi20328086-16 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3471: finishing write tid 2 to smithi20328086-16 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3471: finishing write tid 3 to smithi20328086-16 2024-09-17T15:48:57.126 INFO:tasks.rados.rados.0.smithi203.stdout:3471: finishing write tid 4 to smithi20328086-16 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:3471: oid 16 updating version 0 to 1823 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:3471: oid 16 version 1823 is already newer than 1822 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:3471: oid 16 updating version 1823 to 1824 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 16 v 1824 (ObjNum 1219 snap 368 seq_num 1219) dirty exists 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:3471: left oid 16 (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:3472: expect (ObjNum 1187 snap 362 seq_num 1187) 2024-09-17T15:48:57.127 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:57.271 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2939428 2024-09-17T15:48:57.286 INFO:tasks.rados.rados.0.smithi203.stdout:3473: finishing write tid 1 to smithi20328086-34 2024-09-17T15:48:57.286 INFO:tasks.rados.rados.0.smithi203.stdout:3473: finishing write tid 2 to smithi20328086-34 2024-09-17T15:48:57.286 INFO:tasks.rados.rados.0.smithi203.stdout:3473: finishing write tid 3 to smithi20328086-34 2024-09-17T15:48:57.286 INFO:tasks.rados.rados.0.smithi203.stdout:3473: finishing write tid 4 to smithi20328086-34 2024-09-17T15:48:57.286 INFO:tasks.rados.rados.0.smithi203.stdout:3473: oid 34 updating version 0 to 1326 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3473: oid 34 version 1326 is already newer than 1325 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3473: oid 34 version 1326 is already newer than 1324 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1326 (ObjNum 1220 snap 368 seq_num 1220) dirty exists 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3473: left oid 34 (ObjNum 1220 snap 368 seq_num 1220) 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: finishing write tid 1 to smithi20328086-20 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: finishing write tid 2 to smithi20328086-20 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: finishing write tid 3 to smithi20328086-20 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: finishing write tid 4 to smithi20328086-20 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: oid 20 updating version 0 to 1639 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: oid 20 version 1639 is already newer than 1638 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: oid 20 updating version 1639 to 1640 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1640 (ObjNum 1221 snap 368 seq_num 1221) dirty exists 2024-09-17T15:48:57.287 INFO:tasks.rados.rados.0.smithi203.stdout:3474: left oid 20 (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3471: done (4 left) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3472: done (3 left) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3473: done (2 left) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3474: done (1 left) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3475: done (0 left) 2024-09-17T15:48:57.288 INFO:tasks.rados.rados.0.smithi203.stdout:3476: snap_remove snap 367 2024-09-17T15:48:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:3476: done (0 left) 2024-09-17T15:48:58.242 INFO:tasks.rados.rados.0.smithi203.stdout:3477: write (excl) oid 17 current snap is 368 2024-09-17T15:48:58.243 INFO:tasks.rados.rados.0.smithi203.stdout:3477: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-17T15:48:58.255 INFO:tasks.rados.rados.0.smithi203.stdout:3477: writing smithi20328086-17 from 554961 to 1200968 tid 1 2024-09-17T15:48:58.266 INFO:tasks.rados.rados.0.smithi203.stdout:3477: writing smithi20328086-17 from 1750419 to 2201833 tid 2 2024-09-17T15:48:58.266 INFO:tasks.rados.rados.0.smithi203.stdout:3478: read oid 19 snap 363 2024-09-17T15:48:58.266 INFO:tasks.rados.rados.0.smithi203.stdout:3478: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:48:58.266 INFO:tasks.rados.rados.0.smithi203.stdout:3479: write oid 22 current snap is 368 2024-09-17T15:48:58.266 INFO:tasks.rados.rados.0.smithi203.stdout:3479: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-17T15:48:58.277 INFO:tasks.rados.rados.0.smithi203.stdout:3479: writing smithi20328086-22 from 571768 to 1009377 tid 1 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3479: writing smithi20328086-22 from 1808901 to 2520326 tid 2 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3480: read oid 6 snap 363 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3480: expect deleted 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3481: write oid 13 current snap is 368 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3477: finishing write tid 1 to smithi20328086-17 2024-09-17T15:48:58.293 INFO:tasks.rados.rados.0.smithi203.stdout:3478: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:48:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:3477: finishing write tid 2 to smithi20328086-17 2024-09-17T15:48:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:3477: finishing write tid 3 to smithi20328086-17 2024-09-17T15:48:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:3477: finishing write tid 4 to smithi20328086-17 2024-09-17T15:48:58.604 INFO:tasks.rados.rados.0.smithi203.stdout:3477: oid 17 updating version 0 to 1620 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3477: oid 17 version 1620 is already newer than 1619 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3477: oid 17 version 1620 is already newer than 1618 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1620 (ObjNum 1223 snap 368 seq_num 1223) dirty exists 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3477: left oid 17 (ObjNum 1223 snap 368 seq_num 1223) 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: finishing write tid 1 to smithi20328086-22 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: finishing write tid 2 to smithi20328086-22 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: finishing write tid 3 to smithi20328086-22 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: finishing write tid 4 to smithi20328086-22 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: oid 22 updating version 0 to 1332 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: oid 22 version 1332 is already newer than 1331 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:3479: oid 22 version 1332 is already newer than 1330 2024-09-17T15:48:58.605 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1332 (ObjNum 1224 snap 368 seq_num 1224) dirty exists 2024-09-17T15:48:58.606 INFO:tasks.rados.rados.0.smithi203.stdout:3479: left oid 22 (ObjNum 1224 snap 368 seq_num 1224) 2024-09-17T15:48:58.606 INFO:tasks.rados.rados.0.smithi203.stdout:3481: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-17T15:48:58.615 INFO:tasks.rados.rados.0.smithi203.stdout:3481: writing smithi20328086-13 from 588575 to 1101433 tid 1 2024-09-17T15:48:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:3481: writing smithi20328086-13 from 1867383 to 2322466 tid 2 2024-09-17T15:48:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:3477: done (4 left) 2024-09-17T15:48:58.628 INFO:tasks.rados.rados.0.smithi203.stdout:3478: done (3 left) 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3479: done (2 left) 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3480: done (1 left) 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3482: read oid 22 snap -1 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3482: expect (ObjNum 1224 snap 368 seq_num 1224) 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3483: write oid 45 current snap is 368 2024-09-17T15:48:58.629 INFO:tasks.rados.rados.0.smithi203.stdout:3483: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-17T15:48:58.640 INFO:tasks.rados.rados.0.smithi203.stdout:3483: writing smithi20328086-45 from 605382 to 1193489 tid 1 2024-09-17T15:48:58.655 INFO:tasks.rados.rados.0.smithi203.stdout:3483: writing smithi20328086-45 from 1925865 to 2605382 tid 2 2024-09-17T15:48:58.655 INFO:tasks.rados.rados.0.smithi203.stdout:3484: write oid 28 current snap is 368 2024-09-17T15:48:58.655 INFO:tasks.rados.rados.0.smithi203.stdout:3481: finishing write tid 1 to smithi20328086-13 2024-09-17T15:48:58.656 INFO:tasks.rados.rados.0.smithi203.stdout:3484: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-17T15:48:58.667 INFO:tasks.rados.rados.0.smithi203.stdout:3484: writing smithi20328086-28 from 622189 to 1285545 tid 1 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3484: writing smithi20328086-28 from 1984347 to 2443099 tid 2 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3485: snap_create 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: finishing write tid 2 to smithi20328086-13 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: finishing write tid 3 to smithi20328086-13 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: finishing write tid 4 to smithi20328086-13 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: oid 13 updating version 0 to 1384 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: oid 13 updating version 1384 to 1385 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: oid 13 updating version 1385 to 1386 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1386 (ObjNum 1225 snap 368 seq_num 1225) dirty exists 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3481: left oid 13 (ObjNum 1225 snap 368 seq_num 1225) 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:3482: expect (ObjNum 1224 snap 368 seq_num 1224) 2024-09-17T15:48:58.679 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: finishing write tid 1 to smithi20328086-45 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: finishing write tid 2 to smithi20328086-45 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: finishing write tid 3 to smithi20328086-45 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: finishing write tid 4 to smithi20328086-45 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: oid 45 updating version 0 to 1345 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: oid 45 version 1345 is already newer than 1344 2024-09-17T15:48:58.869 INFO:tasks.rados.rados.0.smithi203.stdout:3483: oid 45 version 1345 is already newer than 1343 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1345 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3483: left oid 45 (ObjNum 1226 snap 368 seq_num 1226) 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: finishing write tid 1 to smithi20328086-28 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: finishing write tid 2 to smithi20328086-28 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: finishing write tid 3 to smithi20328086-28 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: finishing write tid 4 to smithi20328086-28 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: oid 28 updating version 0 to 1799 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: oid 28 version 1799 is already newer than 1798 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: oid 28 version 1799 is already newer than 1797 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1799 (ObjNum 1227 snap 368 seq_num 1227) dirty exists 2024-09-17T15:48:58.870 INFO:tasks.rados.rados.0.smithi203.stdout:3484: left oid 28 (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:48:59.002 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:48:59.002 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:48:59.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:48:59.005 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:48:59.005 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:48:59.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: get_monmap_and_config 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: build_initial_monmap 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: monmap: 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:48:59.137866+0000 2024-09-17T15:48:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:48:59.137866+0000 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:48:59.134 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 Processor -- start 2024-09-17T15:48:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 -- start start 2024-09-17T15:48:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: init 2024-09-17T15:48:59.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-a con 0x7fde6c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde6c068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-b con 0x7fde6c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde6c074470 con 0x7fde6c068db0 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fde6c09c900 con 0x7fde6c0689d0 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fde6c09cb90 con 0x7fde6c13aff0 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:59.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde7430d640 10 monclient(hunting): authenticate will time out at 2272.103364s 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35000/0 (socket says 172.21.15.86:35000) 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42094/0 (socket says 172.21.15.86:42094) 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 1 -- 172.21.15.86:0/1892155147 learned_addr learned my addr 172.21.15.86:0/1892155147 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): get_auth_request con 0x7fde6c0689d0 auth_method 0 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): get_auth_request con 0x7fde6c13aff0 auth_method 0 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:59.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient(hunting): handle_auth_done global_id 164879 payload 274 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 -- 172.21.15.86:0/1892155147 >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 msgr2=0x7fde6c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 --2- 172.21.15.86:0/1892155147 >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 -- 172.21.15.86:0/1892155147 >> v1:172.21.15.203:6789/0 conn(0x7fde6c068db0 legacy=0x7fde6c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient: _finish_hunting 0 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 monclient: found mon.noname-b 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde728aa640 1 -- 172.21.15.86:0/1892155147 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c09ce70 con 0x7fde6c13aff0 2024-09-17T15:48:59.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.134+0000 7fde720a9640 1 --2- 172.21.15.86:0/1892155147 >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde728aa640 1 -- 172.21.15.86:0/1892155147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 msgr2=0x7fde6c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde728aa640 1 --2- 172.21.15.86:0/1892155147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d90 tx=0x7fde6401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=649846b580350065 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 1 -- 172.21.15.86:0/1892155147 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde640b2070 con 0x7fde6c13aff0 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: dump: 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:59.143 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: _finish_auth 0 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: _check_auth_tickets 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 1 -- 172.21.15.86:0/1892155147 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde640a5890 con 0x7fde6c13aff0 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:59.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 1 -- 172.21.15.86:0/1892155147 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c09c2d0 con 0x7fde6c13aff0 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 5 monclient: authenticate success, global_id 164879 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: get_monmap_and_config success 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 4 set_mon_vals no callback set 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde710a7640 10 monclient: dump: 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.145 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: shutdown 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- 172.21.15.86:0/1892155147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 msgr2=0x7fde6c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- 172.21.15.86:0/1892155147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fde64009d90 tx=0x7fde6401c830 comp rx=0 tx=0).stop 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- 172.21.15.86:0/1892155147 shutdown_connections 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- 172.21.15.86:0/1892155147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c13aff0 0x7fde6c13d3e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- 172.21.15.86:0/1892155147 >> v2:172.21.15.86:3300/0 conn(0x7fde6c0689d0 0x7fde6c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- 172.21.15.86:0/1892155147 >> 172.21.15.86:0/1892155147 conn(0x7fde6c0a6440 msgr2=0x7fde6c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- 172.21.15.86:0/1892155147 shutdown_connections 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- 172.21.15.86:0/1892155147 wait complete. 2024-09-17T15:48:59.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: build_initial_monmap 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: monmap: 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 Processor -- start 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- start start 2024-09-17T15:48:59.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: init 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: _reopen_session rank -1 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-c con 0x7fde6c0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde6c068db0 0x7fde6c135060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-a con 0x7fde6c068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde6c13aff0 0x7fde6c071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): picked mon.noname-b con 0x7fde6c13aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c140670 con 0x7fde6c068db0 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fde6c1404f0 con 0x7fde6c13aff0 2024-09-17T15:48:59.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fde6c1407f0 con 0x7fde6c0689d0 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): _renew_subs 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde7430d640 10 monclient(hunting): authenticate will time out at 2272.110438s 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42108/0 (socket says 172.21.15.86:42108) 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 -- 172.21.15.86:0/2057919685 learned_addr learned my addr 172.21.15.86:0/2057919685 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): get_auth_request con 0x7fde6c0689d0 auth_method 0 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:59.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde6c068db0 0x7fde6c135060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): get_auth_request con 0x7fde6c068db0 auth_method 0 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde718a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient(hunting): handle_auth_done global_id 164882 payload 824 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde728aa640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde6c13aff0 0x7fde6c071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 -- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde6c13aff0 msgr2=0x7fde6c071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde6c13aff0 0x7fde6c071080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 -- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde6c068db0 msgr2=0x7fde6c135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:59.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde6c068db0 0x7fde6c135060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient: _finish_hunting 0 2024-09-17T15:48:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 monclient: found mon.noname-c 2024-09-17T15:48:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:48:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.142+0000 7fde720a9640 1 -- 172.21.15.86:0/2057919685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6c0a3850 con 0x7fde6c0689d0 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde720a9640 1 -- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 msgr2=0x7fde6c1cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde720a9640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e9d0 tx=0x7fde6800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68017070 con 0x7fde6c0689d0 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: dump: 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: _finish_auth 0 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: _check_auth_tickets 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6800cc70 con 0x7fde6c0689d0 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde68013670 con 0x7fde6c0689d0 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:48:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 10 monclient: dump: 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 5 monclient: authenticate success, global_id 164882 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _renew_subs 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6c09d460 con 0x7fde6c0689d0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde708a6640 4 set_mon_vals no callback set 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _renew_subs 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde6c121a10 con 0x7fde6c0689d0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde6c1215c0 con 0x7fde6c0689d0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fde6801a070 con 0x7fde6c0689d0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 0x7fde5003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(759..759 src has 696..759) ==== 4831+0+0 (secure 0 0 0) 0x7fde6804fee0 con 0x7fde6c0689d0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde718a8640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 0x7fde5003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:48:59.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.146+0000 7fde718a8640 10 monclient: get_auth_request con 0x7fde5003bbe0 auth_method 0 2024-09-17T15:48:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.150+0000 7fde718a8640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 0x7fde5003e0a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde6c121940 tx=0x7fde60007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:48:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.150+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fde6c1215c0 con 0x7fde6c0689d0 2024-09-17T15:48:59.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.150+0000 7fde5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:48:59.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.150+0000 7fde5effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:48:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde7430d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:48:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde7430d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde7430d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:48:59.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde6c06f840 con 0x7fde6c0689d0 2024-09-17T15:48:59.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde5effd640 1 -- 172.21.15.86:0/2057919685 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fde68051020 con 0x7fde6c0689d0 2024-09-17T15:48:59.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:48:59.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.386+0000 7fde5effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:48:59.388 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":198,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:48:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 msgr2=0x7fde5003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 0x7fde5003e0a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fde6c121940 tx=0x7fde60007e40 comp rx=0 tx=0).stop 2024-09-17T15:48:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 10 monclient: shutdown 2024-09-17T15:48:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:48:59.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 msgr2=0x7fde6c1cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:48:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fde6800e9d0 tx=0x7fde6800ee90 comp rx=0 tx=0).stop 2024-09-17T15:48:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 shutdown_connections 2024-09-17T15:48:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fde5003bbe0 0x7fde5003e0a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fde6c13aff0 0x7fde6c071080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde6c068db0 0x7fde6c135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 --2- 172.21.15.86:0/2057919685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde6c0689d0 0x7fde6c1cfb70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:48:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 >> 172.21.15.86:0/2057919685 conn(0x7fde6c0a6440 msgr2=0x7fde6c0a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:48:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 shutdown_connections 2024-09-17T15:48:59.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:48:59.390+0000 7fde7430d640 1 -- 172.21.15.86:0/2057919685 wait complete. 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3481: done (4 left) 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3482: done (3 left) 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3483: done (2 left) 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3484: done (1 left) 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3485: done (0 left) 2024-09-17T15:48:59.688 INFO:tasks.rados.rados.0.smithi203.stdout:3486: snap_remove snap 366 2024-09-17T15:49:00.695 INFO:tasks.rados.rados.0.smithi203.stdout:3486: done (0 left) 2024-09-17T15:49:00.695 INFO:tasks.rados.rados.0.smithi203.stdout:3487: snap_remove snap 364 2024-09-17T15:49:00.840 INFO:tasks.rados.rados.0.smithi203.stdout:3487: done (0 left) 2024-09-17T15:49:00.840 INFO:tasks.rados.rados.0.smithi203.stdout:3488: snap_create 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3488: done (0 left) 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3489: copy_from oid 23 from oid 33 current snap is 370 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3490: read oid 29 snap -1 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3490: expect (ObjNum 1152 snap 357 seq_num 1152) 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3491: read oid 13 snap 365 2024-09-17T15:49:01.845 INFO:tasks.rados.rados.0.smithi203.stdout:3491: expect deleted 2024-09-17T15:49:01.846 INFO:tasks.rados.rados.0.smithi203.stdout:3492: rollback oid 31 current snap is 370 2024-09-17T15:49:01.846 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 368 2024-09-17T15:49:01.846 INFO:tasks.rados.rados.0.smithi203.stdout:3493: write (excl) oid 21 current snap is 370 2024-09-17T15:49:01.846 INFO:tasks.rados.rados.0.smithi203.stdout:3493: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-17T15:49:01.868 INFO:tasks.rados.rados.0.smithi203.stdout:3493: writing smithi20328086-21 from 638996 to 1377601 tid 1 2024-09-17T15:49:01.891 INFO:tasks.rados.rados.0.smithi203.stdout:3493: writing smithi20328086-21 from 1926476 to 2638996 tid 2 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3494: snap_create 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3489: finishing copy_from to smithi20328086-23 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3489: got expected ENOENT (src dne) 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3489: finishing copy_from racing read to smithi20328086-23 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3492: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3492: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3492: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1335 (ObjNum 1190 snap 363 seq_num 1190) dirty exists 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:3490: expect (ObjNum 1152 snap 357 seq_num 1152) 2024-09-17T15:49:01.892 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:01.974 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 322435 2024-09-17T15:49:02.027 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:02.080 INFO:tasks.rados.rados.0.smithi203.stdout:3493: finishing write tid 1 to smithi20328086-21 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: finishing write tid 2 to smithi20328086-21 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: finishing write tid 3 to smithi20328086-21 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: finishing write tid 4 to smithi20328086-21 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: oid 21 updating version 0 to 1809 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: oid 21 version 1809 is already newer than 1807 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: oid 21 version 1809 is already newer than 1808 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1809 (ObjNum 1228 snap 370 seq_num 1228) dirty exists 2024-09-17T15:49:02.081 INFO:tasks.rados.rados.0.smithi203.stdout:3493: left oid 21 (ObjNum 1228 snap 370 seq_num 1228) 2024-09-17T15:49:02.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: get_monmap_and_config 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: build_initial_monmap 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: monmap: 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:02.541319+0000 2024-09-17T15:49:02.537 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:02.541319+0000 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:02.538 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 Processor -- start 2024-09-17T15:49:02.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 -- start start 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: init 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d080689d0 0x7f6d0806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-a con 0x7f6d080689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 0x7f6d080691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-b con 0x7f6d08068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d08135ae0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d0809c8b0 con 0x7f6d08135ae0 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6d08074500 con 0x7f6d080689d0 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0809cbe0 con 0x7f6d08068db0 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0f545640 10 monclient(hunting): authenticate will time out at 2275.506835s 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d080689d0 0x7f6d0806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 0x7f6d080691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d080689d0 0x7f6d0806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35022/0 (socket says 172.21.15.86:35022) 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/941058673 learned_addr learned my addr 172.21.15.86:0/941058673 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): get_auth_request con 0x7f6d080689d0 auth_method 0 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 0x7f6d080691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42116/0 (socket says 172.21.15.86:42116) 2024-09-17T15:49:02.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): get_auth_request con 0x7f6d08068db0 auth_method 0 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0cae0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_done global_id 234204 payload 274 2024-09-17T15:49:02.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 msgr2=0x7f6d080691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 0x7f6d080691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/941058673 >> v1:172.21.15.203:6789/0 conn(0x7f6d08135ae0 legacy=0x7f6d08135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient: _finish_hunting 0 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 monclient: found mon.noname-a 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.544 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.538+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/941058673 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0809ce70 con 0x7f6d080689d0 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 msgr2=0x7f6d0806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0d2e1640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d0806f840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009e50 tx=0x7f6d0001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfeb42577d00534 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 1 -- 172.21.15.86:0/941058673 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000a56f0 con 0x7f6d080689d0 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: dump: 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 1 -- 172.21.15.86:0/941058673 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d000a5890 con 0x7f6d080689d0 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0f545640 5 monclient: authenticate success, global_id 234204 2024-09-17T15:49:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0f545640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0f545640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d0f545640 4 set_mon_vals no callback set 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.542+0000 7f6d07fff640 1 -- 172.21.15.86:0/941058673 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000a5c60 con 0x7f6d080689d0 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d07fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d07fff640 10 monclient: dump: 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: shutdown 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 msgr2=0x7f6d0806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d0806f840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009e50 tx=0x7f6d0001c890 comp rx=0 tx=0).stop 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- 172.21.15.86:0/941058673 shutdown_connections 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08068db0 0x7f6d080691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- 172.21.15.86:0/941058673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d0806f840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- 172.21.15.86:0/941058673 >> 172.21.15.86:0/941058673 conn(0x7f6d080a6440 msgr2=0x7f6d080a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- 172.21.15.86:0/941058673 shutdown_connections 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- 172.21.15.86:0/941058673 wait complete. 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: build_initial_monmap 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: monmap: 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 Processor -- start 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- start start 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: init 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-a con 0x7f6d080689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d08068db0 0x7f6d08133e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-b con 0x7f6d08068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08135ae0 0x7f6d081343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): picked mon.noname-c con 0x7f6d08135ae0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6d081409b0 con 0x7f6d080689d0 2024-09-17T15:49:02.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6d08140830 con 0x7f6d08068db0 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6d08140b30 con 0x7f6d08135ae0 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0f545640 10 monclient(hunting): authenticate will time out at 2275.514363s 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35034/0 (socket says 172.21.15.86:35034) 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/4281494474 learned_addr learned my addr 172.21.15.86:0/4281494474 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): get_auth_request con 0x7f6d080689d0 auth_method 0 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08135ae0 0x7f6d081343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): get_auth_request con 0x7f6d08135ae0 auth_method 0 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0dae2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0cae0640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d08068db0 0x7f6d08133e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0cae0640 10 monclient(hunting): get_auth_request con 0x7f6d08068db0 auth_method 0 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0cae0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0cae0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0cae0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient(hunting): handle_auth_done global_id 234207 payload 824 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08135ae0 msgr2=0x7f6d081343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08135ae0 0x7f6d081343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d08068db0 msgr2=0x7f6d08133e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d08068db0 0x7f6d08133e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient: _finish_hunting 0 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 monclient: found mon.noname-a 2024-09-17T15:49:02.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.546+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/4281494474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0811e4b0 con 0x7f6d080689d0 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0d2e1640 1 -- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 msgr2=0x7f6d081d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0d2e1640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d0001d130 tx=0x7f6d00019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000bc020 con 0x7f6d080689d0 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: dump: 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.559 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d000b88a0 con 0x7f6d080689d0 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 5 monclient: authenticate success, global_id 234207 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _renew_subs 2024-09-17T15:49:02.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d081d0600 con 0x7f6d080689d0 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d000a56f0 con 0x7f6d080689d0 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 10 monclient: dump: 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.561 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _renew_subs 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d077fe640 4 set_mon_vals no callback set 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0811f680 con 0x7f6d080689d0 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:02.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0806f840 con 0x7f6d080689d0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6d000bf020 con 0x7f6d080689d0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 0x7f6cf403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(763..763 src has 696..763) ==== 4755+0+0 (secure 0 0 0) 0x7f6d000eaac0 con 0x7f6d080689d0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0cae0640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 0x7f6cf403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.550+0000 7f6d0cae0640 10 monclient: get_auth_request con 0x7f6cf403bb90 auth_method 0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.554+0000 7f6d0cae0640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 0x7f6cf403e050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6d08074990 tx=0x7f6cfc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.554+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6d000c6070 con 0x7f6d080689d0 2024-09-17T15:49:02.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.554+0000 7f6d05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:02.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.554+0000 7f6d05ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:02.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d0f545640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:02.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d0f545640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:02.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d0f545640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.792 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6d0806d4f0 con 0x7f6d080689d0 2024-09-17T15:49:02.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d05ffb640 1 -- 172.21.15.86:0/4281494474 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6d00016600 con 0x7f6d080689d0 2024-09-17T15:49:02.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:02.793 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.794+0000 7f6d05ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:02.793 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 msgr2=0x7f6cf403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 0x7f6cf403e050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6d08074990 tx=0x7f6cfc008040 comp rx=0 tx=0).stop 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 10 monclient: shutdown 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 msgr2=0x7f6d081d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d0001d130 tx=0x7f6d00019690 comp rx=0 tx=0).stop 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 shutdown_connections 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6cf403bb90 0x7f6cf403e050 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d08135ae0 0x7f6d081343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d08068db0 0x7f6d08133e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 --2- 172.21.15.86:0/4281494474 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d080689d0 0x7f6d081d00c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 >> 172.21.15.86:0/4281494474 conn(0x7f6d080a6440 msgr2=0x7f6d08134c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:02.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 shutdown_connections 2024-09-17T15:49:02.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.798+0000 7f6d0f545640 1 -- 172.21.15.86:0/4281494474 wait complete. 2024-09-17T15:49:02.812 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:49:02.812 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:49:02.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3489: done (5 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3490: done (4 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3491: done (3 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3492: done (2 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3493: done (1 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3494: done (0 left) 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3495: write (excl) oid 11 current snap is 371 2024-09-17T15:49:02.853 INFO:tasks.rados.rados.0.smithi203.stdout:3495: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-17T15:49:02.867 INFO:tasks.rados.rados.0.smithi203.stdout:3495: writing smithi20328086-11 from 655803 to 1069657 tid 1 2024-09-17T15:49:02.882 INFO:tasks.rados.rados.0.smithi203.stdout:3495: writing smithi20328086-11 from 1584958 to 2047379 tid 2 2024-09-17T15:49:02.882 INFO:tasks.rados.rados.0.smithi203.stdout:3496: write (excl) oid 42 current snap is 371 2024-09-17T15:49:02.882 INFO:tasks.rados.rados.0.smithi203.stdout:3496: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-17T15:49:02.898 INFO:tasks.rados.rados.0.smithi203.stdout:3496: writing smithi20328086-42 from 672610 to 1161713 tid 1 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3496: writing smithi20328086-42 from 1643440 to 2365872 tid 2 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3497: delete oid 36 current snap is 371 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3495: finishing write tid 1 to smithi20328086-11 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3495: finishing write tid 2 to smithi20328086-11 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3495: finishing write tid 3 to smithi20328086-11 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3495: finishing write tid 4 to smithi20328086-11 2024-09-17T15:49:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3495: oid 11 updating version 0 to 1627 2024-09-17T15:49:02.914 INFO:tasks.rados.rados.0.smithi203.stdout:3495: oid 11 updating version 1627 to 1629 2024-09-17T15:49:02.914 INFO:tasks.rados.rados.0.smithi203.stdout:3495: oid 11 version 1629 is already newer than 1628 2024-09-17T15:49:02.914 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1629 (ObjNum 1229 snap 371 seq_num 1229) dirty exists 2024-09-17T15:49:02.914 INFO:tasks.rados.rados.0.smithi203.stdout:3495: left oid 11 (ObjNum 1229 snap 371 seq_num 1229) 2024-09-17T15:49:02.914 INFO:tasks.rados.rados.0.smithi203.stdout:3496: finishing write tid 1 to smithi20328086-42 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: finishing write tid 2 to smithi20328086-42 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: finishing write tid 3 to smithi20328086-42 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: finishing write tid 4 to smithi20328086-42 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: oid 42 updating version 0 to 1427 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: oid 42 updating version 1427 to 1428 2024-09-17T15:49:02.919 INFO:tasks.rados.rados.0.smithi203.stdout:3496: oid 42 version 1428 is already newer than 1426 2024-09-17T15:49:02.920 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1428 (ObjNum 1230 snap 371 seq_num 1230) dirty exists 2024-09-17T15:49:02.920 INFO:tasks.rados.rados.0.smithi203.stdout:3496: left oid 42 (ObjNum 1230 snap 371 seq_num 1230) 2024-09-17T15:49:02.920 INFO:tasks.rados.rados.0.smithi203.stdout:3495: done (2 left) 2024-09-17T15:49:02.920 INFO:tasks.rados.rados.0.smithi203.stdout:3496: done (1 left) 2024-09-17T15:49:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:3497: done (0 left) 2024-09-17T15:49:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:3498: write (excl) oid 24 current snap is 371 2024-09-17T15:49:02.921 INFO:tasks.rados.rados.0.smithi203.stdout:3498: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-17T15:49:02.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 10 monclient: get_monmap_and_config 2024-09-17T15:49:02.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 10 monclient: build_initial_monmap 2024-09-17T15:49:02.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 10 monclient: monmap: 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:02.939131+0000 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:02.939131+0000 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:02.935 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.936 INFO:tasks.rados.rados.0.smithi203.stdout:3498: writing smithi20328086-24 from 706224 to 1462178 tid 1 2024-09-17T15:49:02.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 1 Processor -- start 2024-09-17T15:49:02.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 1 -- start start 2024-09-17T15:49:02.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.934+0000 7fbb34ea5640 10 monclient: init 2024-09-17T15:49:02.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb300a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fbb300a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-a con 0x7fbb3013b450 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb300a2130 con 0x7fbb300a0c70 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fbb300a2410 con 0x7fbb3013b450 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbb300a26f0 con 0x7fbb300a1760 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:02.939 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb34ea5640 10 monclient(hunting): authenticate will time out at 2275.904502s 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42142/0 (socket says 172.21.15.86:42142) 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35042/0 (socket says 172.21.15.86:35042) 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/3010472186 learned_addr learned my addr 172.21.15.86:0/3010472186 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request con 0x7fbb300a1760 auth_method 0 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): get_auth_request con 0x7fbb3013b450 auth_method 0 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.940 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_done global_id 164891 payload 274 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/3010472186 >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 msgr2=0x7fbb3013d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/3010472186 >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/3010472186 >> v1:172.21.15.203:6789/0 conn(0x7fbb300a0c70 legacy=0x7fbb300a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient: _finish_hunting 0 2024-09-17T15:49:02.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 monclient: found mon.noname-b 2024-09-17T15:49:02.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:02.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:02.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/3010472186 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3009f9f0 con 0x7fbb300a1760 2024-09-17T15:49:02.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.938+0000 7fbb2ed9d640 1 --2- 172.21.15.86:0/3010472186 >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/3010472186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 msgr2=0x7fbb3013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/3010472186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59639b8bcf8cfdf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 1 -- 172.21.15.86:0/3010472186 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240b1070 con 0x7fbb300a1760 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: dump: 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.947 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: _finish_auth 0 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 1 -- 172.21.15.86:0/3010472186 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb240ac620 con 0x7fbb300a1760 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 5 monclient: authenticate success, global_id 164891 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:02.948 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 4 set_mon_vals no callback set 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 1 -- 172.21.15.86:0/3010472186 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ac9f0 con 0x7fbb300a1760 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb2d59a640 10 monclient: dump: 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.949 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 10 monclient: shutdown 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- 172.21.15.86:0/3010472186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 msgr2=0x7fbb3013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/3010472186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009870 tx=0x7fbb2401c420 comp rx=0 tx=0).stop 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- 172.21.15.86:0/3010472186 shutdown_connections 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/3010472186 >> v2:172.21.15.86:3300/0 conn(0x7fbb3013b450 0x7fbb3013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/3010472186 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb3013af10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- 172.21.15.86:0/3010472186 >> 172.21.15.86:0/3010472186 conn(0x7fbb300a2f40 msgr2=0x7fbb300a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:02.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- 172.21.15.86:0/3010472186 shutdown_connections 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- 172.21.15.86:0/3010472186 wait complete. 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 10 monclient: build_initial_monmap 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 10 monclient: monmap: 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 Processor -- start 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 1 -- start start 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.942+0000 7fbb34ea5640 10 monclient: init 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 0x7fbb301cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-a con 0x7fbb300a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb301d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-c con 0x7fbb300a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb3013b450 0x7fbb301d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): picked mon.noname-b con 0x7fbb3013b450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30140720 con 0x7fbb300a0c70 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbb301405a0 con 0x7fbb3013b450 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fbb301408a0 con 0x7fbb300a1760 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient(hunting): authenticate will time out at 2275.912160s 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 0x7fbb301cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 0x7fbb301cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35058/0 (socket says 172.21.15.86:35058) 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 -- 172.21.15.86:0/1085902552 learned_addr learned my addr 172.21.15.86:0/1085902552 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): get_auth_request con 0x7fbb300a0c70 auth_method 0 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.953 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb301d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request con 0x7fbb300a1760 auth_method 0 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2ed9d640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb3013b450 0x7fbb301d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient(hunting): handle_auth_done global_id 234213 payload 824 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 msgr2=0x7fbb301d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb300a1760 0x7fbb301d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb3013b450 msgr2=0x7fbb301d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb3013b450 0x7fbb301d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 monclient: found mon.noname-a 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb30135b70 con 0x7fbb300a0c70 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2ed9d640 10 monclient: get_auth_request con 0x7fbb3013b450 auth_method 0 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2ed9d640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb3013b450 0x7fbb301d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 msgr2=0x7fbb301cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2e59c640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 0x7fbb301cfd30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c7f0 tx=0x7fbb2000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20016020 con 0x7fbb300a0c70 2024-09-17T15:49:02.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: dump: 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: _finish_auth 0 2024-09-17T15:49:02.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb2000ac60 con 0x7fbb300a0c70 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20012660 con 0x7fbb300a0c70 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb1effd640 10 monclient: dump: 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb2cd99640 4 set_mon_vals no callback set 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 5 monclient: authenticate success, global_id 234213 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _renew_subs 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3013ba80 con 0x7fbb300a0c70 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _renew_subs 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:02.958 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.946+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb301349e0 con 0x7fbb300a0c70 2024-09-17T15:49:02.960 INFO:tasks.rados.rados.0.smithi203.stdout:3498: writing smithi20328086-24 from 2160404 to 2706224 tid 2 2024-09-17T15:49:02.960 INFO:tasks.rados.rados.0.smithi203.stdout:3499: snap_create 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: finishing write tid 1 to smithi20328086-24 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: finishing write tid 2 to smithi20328086-24 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: finishing write tid 3 to smithi20328086-24 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: finishing write tid 4 to smithi20328086-24 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: oid 24 updating version 0 to 1544 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: oid 24 updating version 1544 to 1545 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: oid 24 version 1545 is already newer than 1543 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1545 (ObjNum 1232 snap 371 seq_num 1232) dirty exists 2024-09-17T15:49:02.961 INFO:tasks.rados.rados.0.smithi203.stdout:3498: left oid 24 (ObjNum 1232 snap 371 seq_num 1232) 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb34ea5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 0x7fbb3012edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30135510 con 0x7fbb3012e9e0 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 0x7fbb3012edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient: get_auth_request con 0x7fbb3012e9e0 auth_method 0 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fbb20019020 con 0x7fbb300a0c70 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_done global_id 234216 payload 824 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbb1003bb60 0x7fbb1003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb300a1050 con 0x7fbb3012e9e0 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(764..764 src has 696..764) ==== 4755+0+0 (secure 0 0 0) 0x7fbb2004fff0 con 0x7fbb300a0c70 2024-09-17T15:49:02.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2ed9d640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbb1003bb60 0x7fbb1003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2ed9d640 10 monclient: get_auth_request con 0x7fbb1003bb60 auth_method 0 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 msgr2=0x7fbb3012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 0x7fbb3012edc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb3006d160 tx=0x7fbb2401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240bb070 con 0x7fbb3012e9e0 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 10 monclient: dump: 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb240b6460 con 0x7fbb3012e9e0 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 msgr2=0x7fbb3012edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 0x7fbb3012edc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb3006d160 tx=0x7fbb2401ac60 comp rx=0 tx=0).stop 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 msgr2=0x7fbb3012edc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.950+0000 7fbb1effd640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012e9e0 0x7fbb3012edc0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbb3006d160 tx=0x7fbb2401ac60 comp rx=0 tx=0).stop 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.954+0000 7fbb2ed9d640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbb1003bb60 0x7fbb1003e020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009580 tx=0x7fbb28008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb34ea5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 0x7fbb30136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb34ea5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3006d4f0 con 0x7fbb3012f300 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 0x7fbb30136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient: get_auth_request con 0x7fbb3012f300 auth_method 0 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 10 monclient(hunting): handle_auth_done global_id 234219 payload 824 2024-09-17T15:49:02.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.958+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/1085902552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb3012f710 con 0x7fbb3012f300 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb2dd9b640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 msgr2=0x7fbb30136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb2dd9b640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 0x7fbb30136ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb2401c460 tx=0x7fbb240191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240c6070 con 0x7fbb3012f300 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 10 monclient: dump: 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:02.969 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7fbb240b6810 con 0x7fbb3012f300 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 msgr2=0x7fbb30136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 0x7fbb30136ef0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbb2401c460 tx=0x7fbb240191c0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 msgr2=0x7fbb30136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:02.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb1effd640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb3012f300 0x7fbb30136ef0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 72157, 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:02.971 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:02.972 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:02.973 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:02.974 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:02.975 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:02.976 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.977 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.978 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:02.979 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.980 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:02.982 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:02.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:02.983 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:02.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:02.983 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:02.983 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:02.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbb1003bb60 msgr2=0x7fbb1003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fbb1003bb60 0x7fbb1003e020 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009580 tx=0x7fbb28008040 comp rx=0 tx=0).stop 2024-09-17T15:49:02.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 10 monclient: shutdown 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 msgr2=0x7fbb301cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 --2- 172.21.15.86:0/1085902552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb300a0c70 0x7fbb301cfd30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000c7f0 tx=0x7fbb2000ccb0 comp rx=0 tx=0).stop 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb2ed9d640 1 -- 172.21.15.86:0/1085902552 reap_dead start 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb2ed9d640 1 -- 172.21.15.86:0/1085902552 reap_dead start 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 shutdown_connections 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 >> 172.21.15.86:0/1085902552 conn(0x7fbb300a2f40 msgr2=0x7fbb300a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 shutdown_connections 2024-09-17T15:49:02.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:02.962+0000 7fbb34ea5640 1 -- 172.21.15.86:0/1085902552 wait complete. 2024-09-17T15:49:02.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:02.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:49:02.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: get_monmap_and_config 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: build_initial_monmap 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: monmap: 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:03.100742+0000 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:03.100742+0000 2024-09-17T15:49:03.096 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:03.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 Processor -- start 2024-09-17T15:49:03.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 -- start start 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: init 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab7c09d770 0x7fab7c09db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-a con 0x7fab7c09d770 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fab7c09e090 0x7fab7c134a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-c con 0x7fab7c09e090 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 0x7fab7c135410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-b con 0x7fab7c134f80 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fab7c09e7f0 con 0x7fab7c09d770 2024-09-17T15:49:03.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fab7c0a2110 con 0x7fab7c09e090 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c09ff00 con 0x7fab7c134f80 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab808ff640 10 monclient(hunting): authenticate will time out at 2276.066701s 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab7c09d770 0x7fab7c09db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 0x7fab7c135410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fab7c09d770 0x7fab7c09db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35070/0 (socket says 172.21.15.86:35070) 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 -- 172.21.15.86:0/2015658442 learned_addr learned my addr 172.21.15.86:0/2015658442 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): get_auth_request con 0x7fab7c09d770 auth_method 0 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): get_auth_request con 0x7fab7c134f80 auth_method 0 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab79d9b640 1 --2- 172.21.15.86:0/2015658442 >> v2:172.21.15.203:3300/0 conn(0x7fab7c09e090 0x7fab7c134a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab79d9b640 10 monclient(hunting): get_auth_request con 0x7fab7c09e090 auth_method 0 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient(hunting): handle_auth_done global_id 234222 payload 274 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 -- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 msgr2=0x7fab7c135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 0x7fab7c135410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 -- 172.21.15.86:0/2015658442 >> v2:172.21.15.203:3300/0 conn(0x7fab7c09e090 msgr2=0x7fab7c134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 --2- 172.21.15.86:0/2015658442 >> v2:172.21.15.203:3300/0 conn(0x7fab7c09e090 0x7fab7c134a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 monclient: found mon.noname-a 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7a59c640 1 -- 172.21.15.86:0/2015658442 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c09ec30 con 0x7fab7c09d770 2024-09-17T15:49:03.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.098+0000 7fab7ad9d640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 0x7fab7c135410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:03.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7a59c640 1 -- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c09d770 msgr2=0x7fab7c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7a59c640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c09d770 0x7fab7c09db50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab7c074de0 tx=0x7fab6c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610f4798600f25bd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 1 -- 172.21.15.86:0/2015658442 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0b0070 con 0x7fab7c09d770 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: dump: 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.110 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: _finish_auth 0 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 1 -- 172.21.15.86:0/2015658442 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab6c0ab600 con 0x7fab7c09d770 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab808ff640 5 monclient: authenticate success, global_id 234222 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab808ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab808ff640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 1 -- 172.21.15.86:0/2015658442 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0ab9d0 con 0x7fab7c09d770 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab808ff640 4 set_mon_vals no callback set 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.102+0000 7fab7959a640 10 monclient: dump: 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: shutdown 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c09d770 msgr2=0x7fab7c09db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c09d770 0x7fab7c09db50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fab7c074de0 tx=0x7fab6c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- 172.21.15.86:0/2015658442 shutdown_connections 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134f80 0x7fab7c135410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- 172.21.15.86:0/2015658442 >> v2:172.21.15.203:3300/0 conn(0x7fab7c09e090 0x7fab7c134a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- 172.21.15.86:0/2015658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c09d770 0x7fab7c09db50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- 172.21.15.86:0/2015658442 >> 172.21.15.86:0/2015658442 conn(0x7fab7c0a2f40 msgr2=0x7fab7c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- 172.21.15.86:0/2015658442 shutdown_connections 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- 172.21.15.86:0/2015658442 wait complete. 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: build_initial_monmap 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: monmap: 2024-09-17T15:49:03.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 Processor -- start 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- start start 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: init 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:03.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c09d770 0x7fab7c1cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-b con 0x7fab7c09d770 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c09e090 0x7fab7c133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-c con 0x7fab7c09e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 0x7fab7c138e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): picked mon.noname-a con 0x7fab7c134f80 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c09cce0 con 0x7fab7c134f80 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c09e4a0 con 0x7fab7c09d770 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c09ce60 con 0x7fab7c09e090 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:03.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab808ff640 10 monclient(hunting): authenticate will time out at 2276.073600s 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 0x7fab7c138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c09e090 0x7fab7c133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 0x7fab7c138e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35076/0 (socket says 172.21.15.86:35076) 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 learned_addr learned my addr 172.21.15.86:0/4255209775 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): get_auth_request con 0x7fab7c09e090 auth_method 0 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): get_auth_request con 0x7fab7c134f80 auth_method 0 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c09d770 0x7fab7c1cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient(hunting): handle_auth_done global_id 234225 payload 824 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c09e090 msgr2=0x7fab7c133c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c09e090 0x7fab7c133c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c09d770 msgr2=0x7fab7c1cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c09d770 0x7fab7c1cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:49:03.117 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 monclient: found mon.noname-a 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c11e310 con 0x7fab7c134f80 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7a59c640 10 monclient: get_auth_request con 0x7fab7c09d770 auth_method 0 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab7c09d770 0x7fab7c1cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.106+0000 7fab79d9b640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c09e090 0x7fab7c133c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 msgr2=0x7fab7c138e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7ad9d640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 0x7fab7c138e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fab7400efc0 tx=0x7fab7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74018030 con 0x7fab7c134f80 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:03.118 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: dump: 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: _finish_auth 0 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab740039a0 con 0x7fab7c134f80 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74003d70 con 0x7fab7c134f80 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 10 monclient: dump: 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.120 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 5 monclient: authenticate success, global_id 234225 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 monclient: _renew_subs 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c11e5f0 con 0x7fab7c134f80 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab78d99640 4 set_mon_vals no callback set 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 monclient: _renew_subs 2024-09-17T15:49:03.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c09d460 con 0x7fab7c134f80 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 0x7fab7c139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c1cc580 con 0x7fab7c134180 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 0x7fab7c139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient: get_auth_request con 0x7fab7c134180 auth_method 0 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fab7401b070 con 0x7fab7c134f80 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 10 monclient(hunting): handle_auth_done global_id 164903 payload 824 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab7c134a40 con 0x7fab7c134180 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab5c03bbb0 0x7fab5c03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(764..764 src has 696..764) ==== 4755+0+0 (secure 0 0 0) 0x7fab740508d0 con 0x7fab7c134f80 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab79d9b640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab5c03bbb0 0x7fab5c03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab79d9b640 10 monclient: get_auth_request con 0x7fab5c03bbb0 auth_method 0 2024-09-17T15:49:03.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 msgr2=0x7fab7c139350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.110+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 0x7fab7c139350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ad000 tx=0x7fab6c01ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0ba020 con 0x7fab7c134180 2024-09-17T15:49:03.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 10 monclient: dump: 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fab6c009490 con 0x7fab7c134180 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:03.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 msgr2=0x7fab7c139350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 0x7fab7c139350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ad000 tx=0x7fab6c01ac20 comp rx=0 tx=0).stop 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 msgr2=0x7fab7c139350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab6affd640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c134180 0x7fab7c139350 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ad000 tx=0x7fab6c01ac20 comp rx=0 tx=0).stop 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.114+0000 7fab79d9b640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab5c03bbb0 0x7fab5c03e070 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fab7c13a480 tx=0x7fab70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab808ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab808ff640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 0x7fab7c072700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab808ff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c139ca0 con 0x7fab7c139890 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 0x7fab7c072700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient: get_auth_request con 0x7fab7c139890 auth_method 0 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 10 monclient(hunting): handle_auth_done global_id 164906 payload 824 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.118+0000 7fab7a59c640 1 -- 172.21.15.86:0/4255209775 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab7c11df00 con 0x7fab7c139890 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab7a59c640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 msgr2=0x7fab7c072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab7a59c640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 0x7fab7c072700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fab6c000c10 tx=0x7fab6c0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0ba070 con 0x7fab7c139890 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 10 monclient: dump: 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.128 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fab6c003620 con 0x7fab7c139890 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 msgr2=0x7fab7c072700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 0x7fab7c072700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fab6c000c10 tx=0x7fab6c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 msgr2=0x7fab7c072700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab6affd640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c139890 0x7fab7c072700 secure :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fab6c000c10 tx=0x7fab6c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 446310, 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.130 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:03.131 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:03.133 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:03.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.135 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.136 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.137 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:03.138 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.139 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:49:03.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:03.141 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:03.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:03.142 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:03.142 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab5c03bbb0 msgr2=0x7fab5c03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fab5c03bbb0 0x7fab5c03e070 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fab7c13a480 tx=0x7fab70008040 comp rx=0 tx=0).stop 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 10 monclient: shutdown 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 msgr2=0x7fab7c138e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 --2- 172.21.15.86:0/4255209775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c134f80 0x7fab7c138e10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fab7400efc0 tx=0x7fab7400c490 comp rx=0 tx=0).stop 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 reap_dead start 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab7ad9d640 1 -- 172.21.15.86:0/4255209775 reap_dead start 2024-09-17T15:49:03.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 shutdown_connections 2024-09-17T15:49:03.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 >> 172.21.15.86:0/4255209775 conn(0x7fab7c0a2f40 msgr2=0x7fab7c0a53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:03.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 shutdown_connections 2024-09-17T15:49:03.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.122+0000 7fab808ff640 1 -- 172.21.15.86:0/4255209775 wait complete. 2024-09-17T15:49:03.144 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:03.144 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:49:03.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.250+0000 7f3c1947e640 10 monclient: get_monmap_and_config 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.250+0000 7f3c1947e640 10 monclient: build_initial_monmap 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.250+0000 7f3c1947e640 10 monclient: monmap: 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.251 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:03.255893+0000 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:03.255893+0000 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:03.252 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 Processor -- start 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 -- start start 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient: init 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c140a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c140a1590 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 0x7f3c1413df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c14136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c140a2130 con 0x7f3c140a1590 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3c140a2410 con 0x7f3c14136160 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3c140a26f0 con 0x7f3c140a0c70 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c1947e640 10 monclient(hunting): authenticate will time out at 2276.221494s 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c127fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3c140a1590 0x7f3c14135a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53026/0 (socket says 172.21.15.86:53026) 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42202/0 (socket says 172.21.15.86:42202) 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c127fc640 1 -- 172.21.15.86:0/789836491 learned_addr learned my addr 172.21.15.86:0/789836491 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): get_auth_request con 0x7f3c140a0c70 auth_method 0 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c137fe640 1 --2- 172.21.15.86:0/789836491 >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 0x7f3c1413df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_done global_id 164909 payload 274 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 -- 172.21.15.86:0/789836491 >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 msgr2=0x7f3c1413df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 --2- 172.21.15.86:0/789836491 >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 0x7f3c1413df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 -- 172.21.15.86:0/789836491 >> v1:172.21.15.86:6789/0 conn(0x7f3c140a1590 legacy=0x7f3c14135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 monclient: found mon.noname-b 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c12ffd640 1 -- 172.21.15.86:0/789836491 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1409f9f0 con 0x7f3c140a0c70 2024-09-17T15:49:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c137fe640 10 monclient: get_auth_request con 0x7f3c14136160 auth_method 0 2024-09-17T15:49:03.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.254+0000 7f3c137fe640 1 --2- 172.21.15.86:0/789836491 >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 0x7f3c1413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c12ffd640 1 -- 172.21.15.86:0/789836491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 msgr2=0x7f3c140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c12ffd640 1 --2- 172.21.15.86:0/789836491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3c040097c0 tx=0x7f3c0401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d68d80486d90e35 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 1 -- 172.21.15.86:0/789836491 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c040b0070 con 0x7f3c140a0c70 2024-09-17T15:49:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: dump: 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:03.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 5 monclient: authenticate success, global_id 164909 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 1 -- 172.21.15.86:0/789836491 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c040ab600 con 0x7f3c140a0c70 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 4 set_mon_vals no callback set 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 1 -- 172.21.15.86:0/789836491 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c040ab9d0 con 0x7f3c140a0c70 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c11ffb640 10 monclient: dump: 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.261 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 10 monclient: shutdown 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 -- 172.21.15.86:0/789836491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 msgr2=0x7f3c140a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 --2- 172.21.15.86:0/789836491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3c040097c0 tx=0x7f3c0401c360 comp rx=0 tx=0).stop 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 -- 172.21.15.86:0/789836491 shutdown_connections 2024-09-17T15:49:03.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 --2- 172.21.15.86:0/789836491 >> v2:172.21.15.203:3300/0 conn(0x7f3c14136160 0x7f3c1413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 --2- 172.21.15.86:0/789836491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c140a1050 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 -- 172.21.15.86:0/789836491 >> 172.21.15.86:0/789836491 conn(0x7f3c140a2f40 msgr2=0x7f3c140a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 -- 172.21.15.86:0/789836491 shutdown_connections 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 1 -- 172.21.15.86:0/789836491 wait complete. 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 10 monclient: build_initial_monmap 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.258+0000 7f3c1947e640 10 monclient: monmap: 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:03.263 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:03.264 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 Processor -- start 2024-09-17T15:49:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- start start 2024-09-17T15:49:03.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: init 2024-09-17T15:49:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:03.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c141cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c140a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c140a1590 0x7f3c14134b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c140a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c14136160 0x7f3c141350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c14136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3c140a00d0 con 0x7f3c14136160 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1409ff50 con 0x7f3c140a1590 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3c14140c90 con 0x7f3c140a0c70 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient(hunting): authenticate will time out at 2276.228933s 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c141cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c141cfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42212/0 (socket says 172.21.15.86:42212) 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 -- 172.21.15.86:0/3666605515 learned_addr learned my addr 172.21.15.86:0/3666605515 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): get_auth_request con 0x7f3c140a0c70 auth_method 0 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c14136160 0x7f3c141350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): get_auth_request con 0x7f3c14136160 auth_method 0 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient(hunting): handle_auth_done global_id 164912 payload 824 2024-09-17T15:49:03.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c140a1590 0x7f3c14134b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c140a1590 msgr2=0x7f3c14134b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c140a1590 0x7f3c14134b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c14136160 msgr2=0x7f3c141350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c14136160 0x7f3c141350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 monclient: found mon.noname-c 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c14121550 con 0x7f3c140a0c70 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 msgr2=0x7f3c141cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c12ffd640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c141cfec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3c040ae040 tx=0x7f3c0401aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c040b0050 con 0x7f3c140a0c70 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: dump: 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 5 monclient: authenticate success, global_id 164912 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _renew_subs 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c040bc070 con 0x7f3c140a0c70 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c14070100 con 0x7f3c140a0c70 2024-09-17T15:49:03.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c040b5a80 con 0x7f3c140a0c70 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3bfb7fe640 10 monclient: dump: 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:03.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _renew_subs 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c117fa640 4 set_mon_vals no callback set 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.262+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c14121270 con 0x7f3c140a0c70 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c1947e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c1947e640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c14120ef0 con 0x7f3c1411d820 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3c040b5c60 con 0x7f3c140a0c70 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3bfb7fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0003bb60 0x7f3c0003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(764..764 src has 696..764) ==== 4755+0+0 (secure 0 0 0) 0x7f3c040ea800 con 0x7f3c140a0c70 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 10 monclient: get_auth_request con 0x7f3c1411d820 auth_method 0 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c137fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0003bb60 0x7f3c0003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.266+0000 7f3c137fe640 10 monclient: get_auth_request con 0x7f3c0003bb60 auth_method 0 2024-09-17T15:49:03.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.270+0000 7f3c137fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0003bb60 0x7f3c0003e020 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00d540 tx=0x7f3c0c0076d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 10 monclient: get_auth_request con 0x7f3c1411d820 auth_method 0 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:49:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:49:03.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:03.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.470+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:49:03.854 INFO:tasks.rados.rados.0.smithi203.stdout:3498: done (1 left) 2024-09-17T15:49:03.854 INFO:tasks.rados.rados.0.smithi203.stdout:3499: done (0 left) 2024-09-17T15:49:03.854 INFO:tasks.rados.rados.0.smithi203.stdout:3500: snap_remove snap 368 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 10 monclient: get_auth_request con 0x7f3c1411d820 auth_method 0 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:49:03.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:03.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:03.874+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:49:04.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient: get_auth_request con 0x7f3c1411d820 auth_method 0 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 10 monclient(hunting): handle_auth_done global_id 244099 payload 824 2024-09-17T15:49:04.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.678+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c1406fbb0 con 0x7f3c1411d820 2024-09-17T15:49:04.679 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c08011620 tx=0x7f3c08011ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:04.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018070 con 0x7f3c1411d820 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 10 monclient: dump: 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3c0800fbc0 con 0x7f3c1411d820 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3c08011620 tx=0x7f3c08011ae0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 msgr2=0x7f3c1411dc00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:04.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.682+0000 7f3bfb7fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411d820 0x7f3c1411dc00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c1947e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:04.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c1947e640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 0x7f3c1413b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c1947e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:04.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c140a1050 con 0x7f3c1411e140 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 0x7f3c1413b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient: get_auth_request con 0x7f3c1411e140 auth_method 0 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:04.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:04.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:04.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:04.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 10 monclient(hunting): handle_auth_done global_id 244102 payload 824 2024-09-17T15:49:04.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.690+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c1406d4f0 con 0x7f3c1411e140 2024-09-17T15:49:04.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c127fc640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 msgr2=0x7f3c1413b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:04.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c127fc640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 0x7f3c1413b300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c08011620 tx=0x7f3c08004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:04.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018040 con 0x7f3c1411e140 2024-09-17T15:49:04.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 10 monclient: dump: 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.693 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.694 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.694 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.694 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.694 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f3c0801d660 con 0x7f3c1411e140 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 msgr2=0x7f3c1413b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 0x7f3c1413b300 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c08011620 tx=0x7f3c08004a20 comp rx=0 tx=0).stop 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 msgr2=0x7f3c1413b300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3bfb7fe640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c1411e140 0x7f3c1413b300 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.695 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 5603, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:04.696 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:04.697 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:04.699 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:04.700 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:04.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:04.704 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.705 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.706 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:04.707 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.708 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:04.709 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:04.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0003bb60 msgr2=0x7f3c0003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3c0003bb60 0x7f3c0003e020 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c00d540 tx=0x7f3c0c0076d0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 10 monclient: shutdown 2024-09-17T15:49:04.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 msgr2=0x7f3c141cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 --2- 172.21.15.86:0/3666605515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c140a0c70 0x7f3c141cfec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f3c040ae040 tx=0x7f3c0401aba0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c137fe640 1 -- 172.21.15.86:0/3666605515 reap_dead start 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c137fe640 1 -- 172.21.15.86:0/3666605515 reap_dead start 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 shutdown_connections 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 >> 172.21.15.86:0/3666605515 conn(0x7f3c140a2f40 msgr2=0x7f3c140712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 shutdown_connections 2024-09-17T15:49:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.694+0000 7f3c1947e640 1 -- 172.21.15.86:0/3666605515 wait complete. 2024-09-17T15:49:04.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:04.718 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:49:04.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:49:04.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.842+0000 7fcf1515a640 10 monclient: get_monmap_and_config 2024-09-17T15:49:04.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.842+0000 7fcf1515a640 10 monclient: build_initial_monmap 2024-09-17T15:49:04.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient: monmap: 2024-09-17T15:49:04.843 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:04.848042+0000 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:04.848042+0000 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:04.844 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 Processor -- start 2024-09-17T15:49:04.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 -- start start 2024-09-17T15:49:04.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient: init 2024-09-17T15:49:04.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf10131780 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf1009d850 0x7fcf1013cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1009d850 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf1013d480 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf10132270 con 0x7fcf10131780 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1009dca0 con 0x7fcf1013d480 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fcf100a31f0 con 0x7fcf1009d850 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:04.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf1515a640 10 monclient(hunting): authenticate will time out at 2277.813642s 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0ed9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcf10131780 0x7fcf10131b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53036/0 (socket says 172.21.15.86:53036) 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0ed9d640 1 -- 172.21.15.86:0/2535286794 learned_addr learned my addr 172.21.15.86:0/2535286794 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf1009d850 0x7fcf1013cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): get_auth_request con 0x7fcf1009d850 auth_method 0 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:04.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_done global_id 234231 payload 274 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 1 -- 172.21.15.86:0/2535286794 >> v1:172.21.15.203:6789/0 conn(0x7fcf1013d480 legacy=0x7fcf1013f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 1 -- 172.21.15.86:0/2535286794 >> v1:172.21.15.86:6789/0 conn(0x7fcf10131780 legacy=0x7fcf10131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 1 monclient: found mon.noname-a 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:04.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.846+0000 7fcf0e59c640 1 -- 172.21.15.86:0/2535286794 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1009c5d0 con 0x7fcf1009d850 2024-09-17T15:49:04.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0e59c640 1 -- 172.21.15.86:0/2535286794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf1009d850 msgr2=0x7fcf1013cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:04.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0e59c640 1 --2- 172.21.15.86:0/2535286794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf1009d850 0x7fcf1013cf40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf040097c0 tx=0x7fcf0401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd91703af87c15d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 1 -- 172.21.15.86:0/2535286794 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf040b0070 con 0x7fcf1009d850 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: dump: 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 1 -- 172.21.15.86:0/2535286794 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf040ab600 con 0x7fcf1009d850 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 1 -- 172.21.15.86:0/2535286794 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf040ab9d0 con 0x7fcf1009d850 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 5 monclient: authenticate success, global_id 234231 2024-09-17T15:49:04.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 4 set_mon_vals no callback set 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf0dd9b640 10 monclient: dump: 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 10 monclient: shutdown 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 -- 172.21.15.86:0/2535286794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf1009d850 msgr2=0x7fcf1013cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 --2- 172.21.15.86:0/2535286794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf1009d850 0x7fcf1013cf40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcf040097c0 tx=0x7fcf0401c3f0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 -- 172.21.15.86:0/2535286794 shutdown_connections 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 --2- 172.21.15.86:0/2535286794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf1009d850 0x7fcf1013cf40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 -- 172.21.15.86:0/2535286794 >> 172.21.15.86:0/2535286794 conn(0x7fcf1006cf70 msgr2=0x7fcf1006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 -- 172.21.15.86:0/2535286794 shutdown_connections 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 1 -- 172.21.15.86:0/2535286794 wait complete. 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 10 monclient: build_initial_monmap 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.850+0000 7fcf1515a640 10 monclient: monmap: 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:04.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 Processor -- start 2024-09-17T15:49:04.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- start start 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: init 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf1009d850 0x7fcf10130ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1009d850 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf10131780 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 0x7fcf1009f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1013d480 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fcf10143930 con 0x7fcf10131780 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcf101437b0 con 0x7fcf1009d850 2024-09-17T15:49:04.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fcf10143ab0 con 0x7fcf1013d480 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient(hunting): authenticate will time out at 2277.820746s 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35100/0 (socket says 172.21.15.86:35100) 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 0x7fcf1009f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 -- 172.21.15.86:0/1080405468 learned_addr learned my addr 172.21.15.86:0/1080405468 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): get_auth_request con 0x7fcf10131780 auth_method 0 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:04.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): get_auth_request con 0x7fcf1013d480 auth_method 0 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:04.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0ed9d640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf1009d850 0x7fcf10130ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0ed9d640 10 monclient(hunting): get_auth_request con 0x7fcf1009d850 auth_method 0 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient(hunting): handle_auth_done global_id 234234 payload 824 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 -- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 msgr2=0x7fcf1009f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 0x7fcf1009f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 -- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf1009d850 msgr2=0x7fcf10130ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf1009d850 0x7fcf10130ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 monclient: found mon.noname-a 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 -- 172.21.15.86:0/1080405468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf101393f0 con 0x7fcf10131780 2024-09-17T15:49:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0f59e640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 0x7fcf1009f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:04.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 -- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 msgr2=0x7fcf1009ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:04.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0e59c640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf0401c220 tx=0x7fcf04019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf040b9020 con 0x7fcf10131780 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: dump: 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf040b5890 con 0x7fcf10131780 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf040b0040 con 0x7fcf10131780 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcef77fe640 10 monclient: dump: 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.868 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 5 monclient: authenticate success, global_id 234234 2024-09-17T15:49:04.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _renew_subs 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf100a0930 con 0x7fcf10131780 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf0d59a640 4 set_mon_vals no callback set 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _renew_subs 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.854+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf10138e30 con 0x7fcf10131780 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf1515a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf1515a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf1515a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf10131b60 con 0x7fcf10131780 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fcf040bc020 con 0x7fcf10131780 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcef77fe640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 0x7fcefc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(765..765 src has 696..765) ==== 4755+0+0 (secure 0 0 0) 0x7fcf040ea1b0 con 0x7fcf10131780 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf0ed9d640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 0x7fcefc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.858+0000 7fcf0ed9d640 10 monclient: get_auth_request con 0x7fcefc03bb60 auth_method 0 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.862+0000 7fcf0ed9d640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 0x7fcefc03e020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcf08007fd0 tx=0x7fcf08009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.862+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fcf040c3030 con 0x7fcf10131780 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.862+0000 7fcef77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:04.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:04.862+0000 7fcef77fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:04.905 INFO:tasks.rados.rados.0.smithi203.stdout:3500: done (0 left) 2024-09-17T15:49:04.906 INFO:tasks.rados.rados.0.smithi203.stdout:3501: snap_create 2024-09-17T15:49:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.102+0000 7fcf1515a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:49:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.102+0000 7fcf1515a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:49:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.102+0000 7fcf1515a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:05.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.102+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcf1006de20 con 0x7fcf10131780 2024-09-17T15:49:05.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcef77fe640 1 -- 172.21.15.86:0/1080405468 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcf04016070 con 0x7fcf10131780 2024-09-17T15:49:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcef77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:49:05.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcef77fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=10,logm=52} crc {auth=4115003856,config=555582061,health=4226895542,kv=1452539945,logm=2746753534}) 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=10,kv=11,logm=47} crc {auth=3417714694,config=3541143443,health=949179588,kv=3591778543,logm=2254163233}) 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=10,logm=52} crc {auth=4115003856,config=555582061,health=4226895542,kv=1452539945,logm=2746753534}) 2024-09-17T15:49:05.105 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=1940227138,config=3541143443,health=4226895542,kv=3591778543,logm=3153352433}) 2024-09-17T15:49:05.107 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.107 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104117375}) 2024-09-17T15:49:05.107 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757614885}) 2024-09-17T15:49:05.107 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.107 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104117375}) 2024-09-17T15:49:05.108 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.106+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413916400}) 2024-09-17T15:49:05.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 msgr2=0x7fcefc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:05.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 0x7fcefc03e020 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcf08007fd0 tx=0x7fcf08009040 comp rx=0 tx=0).stop 2024-09-17T15:49:05.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 10 monclient: shutdown 2024-09-17T15:49:05.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 msgr2=0x7fcf1009ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcf0401c220 tx=0x7fcf04019190 comp rx=0 tx=0).stop 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 shutdown_connections 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fcefc03bb60 0x7fcefc03e020 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf1013d480 0x7fcf1009f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf10131780 0x7fcf1009ee70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 --2- 172.21.15.86:0/1080405468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcf1009d850 0x7fcf10130ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 >> 172.21.15.86:0/1080405468 conn(0x7fcf1006cf70 msgr2=0x7fcf10134550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 shutdown_connections 2024-09-17T15:49:05.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:05.106+0000 7fcf1515a640 1 -- 172.21.15.86:0/1080405468 wait complete. 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931390561}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697346449}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931390561}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.110+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573532551}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426183854}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102728405}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426183854}) 2024-09-17T15:49:05.113 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1097366673}) 2024-09-17T15:49:05.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530973575}) 2024-09-17T15:49:05.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1745006904}) 2024-09-17T15:49:05.114 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530973575}) 2024-09-17T15:49:05.115 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.114+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801650012}) 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272642244}) 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741821504}) 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1272642244}) 2024-09-17T15:49:05.116 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3824136253}) 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597198176}) 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=791122158}) 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=597198176}) 2024-09-17T15:49:05.118 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.118+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=646916367}) 2024-09-17T15:49:05.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.119 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809332127}) 2024-09-17T15:49:05.120 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010176315}) 2024-09-17T15:49:05.120 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.120 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809332127}) 2024-09-17T15:49:05.120 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666964569}) 2024-09-17T15:49:05.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646619296}) 2024-09-17T15:49:05.121 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391309505}) 2024-09-17T15:49:05.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646619296}) 2024-09-17T15:49:05.122 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=495989790}) 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555658188}) 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094550057}) 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555658188}) 2024-09-17T15:49:05.123 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.122+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309493614}) 2024-09-17T15:49:05.124 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280746773}) 2024-09-17T15:49:05.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634428458}) 2024-09-17T15:49:05.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4280746773}) 2024-09-17T15:49:05.125 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830209928}) 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099536887}) 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=390774611}) 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099536887}) 2024-09-17T15:49:05.126 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.126+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2775854559}) 2024-09-17T15:49:05.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:49:05.128 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668576385}) 2024-09-17T15:49:05.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3795831559}) 2024-09-17T15:49:05.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1668576385}) 2024-09-17T15:49:05.129 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.130+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332723210}) 2024-09-17T15:49:05.299 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166810348}) 2024-09-17T15:49:05.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1640710290}) 2024-09-17T15:49:05.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166810348}) 2024-09-17T15:49:05.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.302+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288516865}) 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325006435}) 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854652605}) 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3325006435}) 2024-09-17T15:49:05.731 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.730+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3589408318}) 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248938303}) 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1067749376}) 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3248938303}) 2024-09-17T15:49:05.733 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=664017999}) 2024-09-17T15:49:05.735 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.735 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583856364}) 2024-09-17T15:49:05.735 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.734+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163289522}) 2024-09-17T15:49:05.735 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2583856364}) 2024-09-17T15:49:05.736 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=404527456}) 2024-09-17T15:49:05.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091697283}) 2024-09-17T15:49:05.737 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659602030}) 2024-09-17T15:49:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091697283}) 2024-09-17T15:49:05.738 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.738+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547272362}) 2024-09-17T15:49:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.739 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819794224}) 2024-09-17T15:49:05.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179029433}) 2024-09-17T15:49:05.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819794224}) 2024-09-17T15:49:05.740 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2307110668}) 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731042739}) 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122588747}) 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731042739}) 2024-09-17T15:49:05.742 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.742+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977519283}) 2024-09-17T15:49:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206298265}) 2024-09-17T15:49:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075108849}) 2024-09-17T15:49:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.744 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3206298265}) 2024-09-17T15:49:05.745 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.746+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398939993}) 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847142481}) 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270408854}) 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2847142481}) 2024-09-17T15:49:05.754 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.754+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204142161}) 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410873593}) 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=866228705}) 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2410873593}) 2024-09-17T15:49:05.756 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917386349}) 2024-09-17T15:49:05.758 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386379151}) 2024-09-17T15:49:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076625294}) 2024-09-17T15:49:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386379151}) 2024-09-17T15:49:05.759 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.758+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987573195}) 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182400715}) 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3758568652}) 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182400715}) 2024-09-17T15:49:05.761 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.762+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=273285035}) 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107603617}) 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922721239}) 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107603617}) 2024-09-17T15:49:05.764 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.766+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4058766653}) 2024-09-17T15:49:05.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720811985}) 2024-09-17T15:49:05.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1485465611}) 2024-09-17T15:49:05.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.767 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720811985}) 2024-09-17T15:49:05.768 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=804624467}) 2024-09-17T15:49:05.770 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252801800}) 2024-09-17T15:49:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094532670}) 2024-09-17T15:49:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252801800}) 2024-09-17T15:49:05.771 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.770+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164340031}) 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168068776}) 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1073699446}) 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168068776}) 2024-09-17T15:49:05.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75026493}) 2024-09-17T15:49:05.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.774+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415573653}) 2024-09-17T15:49:05.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3998308711}) 2024-09-17T15:49:05.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.775 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415573653}) 2024-09-17T15:49:05.776 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812017368}) 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084792525}) 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893273838}) 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084792525}) 2024-09-17T15:49:05.778 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.778+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870646148}) 2024-09-17T15:49:05.780 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.780 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338100002}) 2024-09-17T15:49:05.780 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3223565030}) 2024-09-17T15:49:05.780 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338100002}) 2024-09-17T15:49:05.781 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558117238}) 2024-09-17T15:49:05.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354269527}) 2024-09-17T15:49:05.782 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851210172}) 2024-09-17T15:49:05.783 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.783 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354269527}) 2024-09-17T15:49:05.783 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.782+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=437940666}) 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91226235}) 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=404908060}) 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=91226235}) 2024-09-17T15:49:05.785 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040304110}) 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312880926}) 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1191787206}) 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312880926}) 2024-09-17T15:49:05.787 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.786+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543674786}) 2024-09-17T15:49:05.789 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224799030}) 2024-09-17T15:49:05.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935645755}) 2024-09-17T15:49:05.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224799030}) 2024-09-17T15:49:05.790 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.790+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751003520}) 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243509390}) 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31361103}) 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243509390}) 2024-09-17T15:49:05.792 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743349630}) 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559905078}) 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727038834}) 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559905078}) 2024-09-17T15:49:05.794 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.794+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=392097774}) 2024-09-17T15:49:05.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881811891}) 2024-09-17T15:49:05.796 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631432397}) 2024-09-17T15:49:05.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881811891}) 2024-09-17T15:49:05.797 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1351130692}) 2024-09-17T15:49:05.798 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731309637}) 2024-09-17T15:49:05.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3065219618}) 2024-09-17T15:49:05.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731309637}) 2024-09-17T15:49:05.799 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.798+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=891070860}) 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479312459}) 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024207095}) 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479312459}) 2024-09-17T15:49:05.801 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3751227762}) 2024-09-17T15:49:05.803 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.803 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928949046}) 2024-09-17T15:49:05.803 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3383787372}) 2024-09-17T15:49:05.803 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928949046}) 2024-09-17T15:49:05.804 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.802+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139942998}) 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747606692}) 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024817945}) 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747606692}) 2024-09-17T15:49:05.805 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3113219058}) 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357328441}) 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2140229162}) 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=357328441}) 2024-09-17T15:49:05.807 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.806+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753312973}) 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640663018}) 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148191919}) 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640663018}) 2024-09-17T15:49:05.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309452293}) 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835486559}) 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118810108}) 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=835486559}) 2024-09-17T15:49:05.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.810+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=26819288}) 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621586296}) 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451491981}) 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621586296}) 2024-09-17T15:49:05.813 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850395938}) 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674394515}) 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552355338}) 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=674394515}) 2024-09-17T15:49:05.815 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.814+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809993200}) 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256263605}) 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312784692}) 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256263605}) 2024-09-17T15:49:05.817 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683029467}) 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554382792}) 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088408907}) 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554382792}) 2024-09-17T15:49:05.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.818+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530120132}) 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632927929}) 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2229863834}) 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1632927929}) 2024-09-17T15:49:05.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994083593}) 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878089145}) 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1559070644}) 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878089145}) 2024-09-17T15:49:05.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.822+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474745225}) 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238758370}) 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2020570978}) 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=238758370}) 2024-09-17T15:49:05.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421674251}) 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347679289}) 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3412917233}) 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3347679289}) 2024-09-17T15:49:05.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.826+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569388598}) 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270848815}) 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057224919}) 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270848815}) 2024-09-17T15:49:05.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2176465339}) 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277897896}) 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3728086249}) 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277897896}) 2024-09-17T15:49:05.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.830+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1355210659}) 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654659585}) 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2432029682}) 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654659585}) 2024-09-17T15:49:05.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.834+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188033206}) 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145102608}) 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3038817360}) 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=145102608}) 2024-09-17T15:49:05.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.838+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427064396}) 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2856050243}) 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972798060}) 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2856050243}) 2024-09-17T15:49:05.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3237836590}) 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204718411}) 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1921958612}) 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204718411}) 2024-09-17T15:49:05.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.842+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625537613}) 2024-09-17T15:49:05.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427719996}) 2024-09-17T15:49:05.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128270794}) 2024-09-17T15:49:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427719996}) 2024-09-17T15:49:05.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134200485}) 2024-09-17T15:49:05.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85658705}) 2024-09-17T15:49:05.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2658616760}) 2024-09-17T15:49:05.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85658705}) 2024-09-17T15:49:05.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.846+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016642012}) 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933647227}) 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321576467}) 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2933647227}) 2024-09-17T15:49:05.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412824590}) 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138043417}) 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853020635}) 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138043417}) 2024-09-17T15:49:05.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.850+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912078214}) 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280064266}) 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=87139985}) 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280064266}) 2024-09-17T15:49:05.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2512169166}) 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624843279}) 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2240263715}) 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624843279}) 2024-09-17T15:49:05.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.854+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298551125}) 2024-09-17T15:49:05.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511345898}) 2024-09-17T15:49:05.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1514346774}) 2024-09-17T15:49:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511345898}) 2024-09-17T15:49:05.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742089874}) 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525261602}) 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608471248}) 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2525261602}) 2024-09-17T15:49:05.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376813855}) 2024-09-17T15:49:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529696152}) 2024-09-17T15:49:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2458346649}) 2024-09-17T15:49:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.858+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3529696152}) 2024-09-17T15:49:05.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605059738}) 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330136574}) 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184046847}) 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1330136574}) 2024-09-17T15:49:05.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573183753}) 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350919554}) 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3919265653}) 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350919554}) 2024-09-17T15:49:05.863 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.862+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428090312}) 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999124262}) 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2762684877}) 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2999124262}) 2024-09-17T15:49:05.865 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860233055}) 2024-09-17T15:49:05.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311388955}) 2024-09-17T15:49:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3639998467}) 2024-09-17T15:49:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311388955}) 2024-09-17T15:49:05.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.866+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1233083994}) 2024-09-17T15:49:05.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822308140}) 2024-09-17T15:49:05.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37475539}) 2024-09-17T15:49:05.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3822308140}) 2024-09-17T15:49:05.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156853420}) 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284916449}) 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3947089689}) 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2284916449}) 2024-09-17T15:49:05.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.870+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959386971}) 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462752895}) 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3908082586}) 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462752895}) 2024-09-17T15:49:05.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3881058011}) 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722705761}) 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529257252}) 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722705761}) 2024-09-17T15:49:05.874 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.874+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935131910}) 2024-09-17T15:49:05.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481955527}) 2024-09-17T15:49:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312230305}) 2024-09-17T15:49:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481955527}) 2024-09-17T15:49:05.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009589771}) 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500207256}) 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052882532}) 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1500207256}) 2024-09-17T15:49:05.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3640068211}) 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610497901}) 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2586629207}) 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610497901}) 2024-09-17T15:49:05.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850231896}) 2024-09-17T15:49:05.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766767187}) 2024-09-17T15:49:05.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4027012230}) 2024-09-17T15:49:05.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2766767187}) 2024-09-17T15:49:05.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449108472}) 2024-09-17T15:49:05.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143904749}) 2024-09-17T15:49:05.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489844942}) 2024-09-17T15:49:05.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143904749}) 2024-09-17T15:49:05.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=996283190}) 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214220621}) 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169929268}) 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4214220621}) 2024-09-17T15:49:05.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094591422}) 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736543533}) 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384915704}) 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3736543533}) 2024-09-17T15:49:05.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1640092630}) 2024-09-17T15:49:05.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410751522}) 2024-09-17T15:49:05.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479233033}) 2024-09-17T15:49:05.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410751522}) 2024-09-17T15:49:05.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1980431653}) 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=12,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=3} crc {logm=926314876,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgrstat=3184912752,mon_config_key=3673412594,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3734937928,osd_pg_creating=3799706863,osd_snap=2830078793}) 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=47,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2} crc {logm=1866084990,mdsmap=2815901553,mgr=1842696677,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3184912752,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=3799706863,osd_snap=3725691980}) 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=12,mon_config_key=12,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=3} crc {logm=926314876,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgrstat=3184912752,mon_config_key=3673412594,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3734937928,osd_pg_creating=3799706863,osd_snap=2830078793}) 2024-09-17T15:49:05.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=2} crc {logm=4236693024,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3184912752,mon_config_key=2443226221,monmap=4191199434,nvmeofgw=598337686,osd_metadata=3616902286,osd_snap=3725691980}) 2024-09-17T15:49:05.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=1887328037}) 2024-09-17T15:49:05.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=92118984}) 2024-09-17T15:49:05.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=1887328037}) 2024-09-17T15:49:05.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=2417606473}) 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=3424921729,osdmap=105016487}) 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=3505758410,osdmap=2085345507}) 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=3424921729,osdmap=105016487}) 2024-09-17T15:49:05.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=576172536,osdmap=4125349836}) 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=1552819849}) 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=1174027961}) 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=1552819849}) 2024-09-17T15:49:05.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:05.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1375545218}) 2024-09-17T15:49:05.912 INFO:tasks.rados.rados.0.smithi203.stdout:3501: done (0 left) 2024-09-17T15:49:05.913 INFO:tasks.rados.rados.0.smithi203.stdout:3502: delete oid 17 current snap is 373 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3502: done (0 left) 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3503: read oid 28 snap -1 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3503: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3504: read oid 5 snap -1 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3504: expect deleted 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3505: copy_from oid 19 from oid 17 current snap is 373 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3506: read oid 6 snap -1 2024-09-17T15:49:05.920 INFO:tasks.rados.rados.0.smithi203.stdout:3506: expect (ObjNum 1216 snap 367 seq_num 1216) 2024-09-17T15:49:05.921 INFO:tasks.rados.rados.0.smithi203.stdout:3507: snap_remove snap 369 2024-09-17T15:49:06.129 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:49:06.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:06.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: get_monmap_and_config 2024-09-17T15:49:06.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: monmap: 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:06.261733+0000 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:06.261733+0000 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:06.258 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:06.259 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:06.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 Processor -- start 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 -- start start 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: init 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-17T15:49:06.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f914c0689d0 0x7f914c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-a con 0x7f914c0689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f914c068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-b con 0x7f914c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914c074470 con 0x7f914c068db0 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f914c09c900 con 0x7f914c0689d0 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f914c09cb90 con 0x7f914c13aff0 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9153df5640 10 monclient(hunting): authenticate will time out at 2279.227210s 2024-09-17T15:49:06.262 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f914c0689d0 0x7f914c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151390640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f914c068db0 0x7f914c069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53048/0 (socket says 172.21.15.86:53048) 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151390640 1 -- 172.21.15.86:0/3696090744 learned_addr learned my addr 172.21.15.86:0/3696090744 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): get_auth_request con 0x7f914c13aff0 auth_method 0 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): get_auth_request con 0x7f914c0689d0 auth_method 0 2024-09-17T15:49:06.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient(hunting): handle_auth_done global_id 164918 payload 274 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 -- 172.21.15.86:0/3696090744 >> v2:172.21.15.86:3300/0 conn(0x7f914c0689d0 msgr2=0x7f914c06f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 --2- 172.21.15.86:0/3696090744 >> v2:172.21.15.86:3300/0 conn(0x7f914c0689d0 0x7f914c06f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 -- 172.21.15.86:0/3696090744 >> v1:172.21.15.86:6789/0 conn(0x7f914c068db0 legacy=0x7f914c069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 monclient: found mon.noname-b 2024-09-17T15:49:06.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:06.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.258+0000 7f9152392640 1 -- 172.21.15.86:0/3696090744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c09ce70 con 0x7f914c13aff0 2024-09-17T15:49:06.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 1 -- 172.21.15.86:0/3696090744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 msgr2=0x7f914c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 1 --2- 172.21.15.86:0/3696090744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c13d3e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9144009d90 tx=0x7f914401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f329e9373c789c1b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 1 -- 172.21.15.86:0/3696090744 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91440b2070 con 0x7f914c13aff0 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: dump: 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.271 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 5 monclient: authenticate success, global_id 164918 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 1 -- 172.21.15.86:0/3696090744 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91440a5890 con 0x7f914c13aff0 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 1 -- 172.21.15.86:0/3696090744 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c09c2d0 con 0x7f914c13aff0 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 4 set_mon_vals no callback set 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9150b8f640 10 monclient: dump: 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.273 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: shutdown 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- 172.21.15.86:0/3696090744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 msgr2=0x7f914c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- 172.21.15.86:0/3696090744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c13d3e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9144009d90 tx=0x7f914401c830 comp rx=0 tx=0).stop 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- 172.21.15.86:0/3696090744 shutdown_connections 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- 172.21.15.86:0/3696090744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c13d3e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- 172.21.15.86:0/3696090744 >> v2:172.21.15.86:3300/0 conn(0x7f914c0689d0 0x7f914c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- 172.21.15.86:0/3696090744 >> 172.21.15.86:0/3696090744 conn(0x7f914c0a6440 msgr2=0x7f914c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- 172.21.15.86:0/3696090744 shutdown_connections 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- 172.21.15.86:0/3696090744 wait complete. 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: monmap: 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 Processor -- start 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- start start 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: init 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-a con 0x7f914c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 0x7f914c136400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-b con 0x7f914c068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): picked mon.noname-c con 0x7f914c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f914c140670 con 0x7f914c0689d0 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f914c1404f0 con 0x7f914c068db0 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f914c1407f0 con 0x7f914c13aff0 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9153df5640 10 monclient(hunting): authenticate will time out at 2279.234462s 2024-09-17T15:49:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35120/0 (socket says 172.21.15.86:35120) 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 -- 172.21.15.86:0/842919376 learned_addr learned my addr 172.21.15.86:0/842919376 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): get_auth_request con 0x7f914c0689d0 auth_method 0 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): get_auth_request con 0x7f914c13aff0 auth_method 0 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151390640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 0x7f914c136400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient(hunting): handle_auth_done global_id 234240 payload 824 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 -- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 msgr2=0x7f914c071e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c071e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 -- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 msgr2=0x7f914c136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 0x7f914c136400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 monclient: found mon.noname-a 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151b91640 1 -- 172.21.15.86:0/842919376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c13d2a0 con 0x7f914c0689d0 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151390640 10 monclient: get_auth_request con 0x7f914c068db0 auth_method 0 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9151390640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 0x7f914c136400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.266+0000 7f9152392640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:06.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9151b91640 1 -- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 msgr2=0x7f914c135ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9151b91640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f914800e9b0 tx=0x7f914800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148017070 con 0x7f914c0689d0 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: dump: 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.281 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914800cbb0 con 0x7f914c0689d0 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148013470 con 0x7f914c0689d0 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:06.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 10 monclient: dump: 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 5 monclient: authenticate success, global_id 234240 2024-09-17T15:49:06.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _renew_subs 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91437fe640 4 set_mon_vals no callback set 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c122970 con 0x7f914c0689d0 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _renew_subs 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c073810 con 0x7f914c0689d0 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c06f840 con 0x7f914c0689d0 2024-09-17T15:49:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f914801a070 con 0x7f914c0689d0 2024-09-17T15:49:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 0x7f913003e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(767..767 src has 696..767) ==== 4771+0+0 (secure 0 0 0) 0x7f9148050030 con 0x7f914c0689d0 2024-09-17T15:49:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9151390640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 0x7f913003e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.270+0000 7f9151390640 10 monclient: get_auth_request con 0x7f913003bbc0 auth_method 0 2024-09-17T15:49:06.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.274+0000 7f9151390640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 0x7f913003e080 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f913c007fd0 tx=0x7f913c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.274+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f914c13d2a0 con 0x7f914c0689d0 2024-09-17T15:49:06.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.274+0000 7f91427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.274+0000 7f91427fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:06.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f9153df5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:06.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f9153df5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:06.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f9153df5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f914c06d4f0 con 0x7f914c0689d0 2024-09-17T15:49:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f91427fc640 1 -- 172.21.15.86:0/842919376 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9148023d00 con 0x7f914c0689d0 2024-09-17T15:49:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f91427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.510+0000 7f91427fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:06.510 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 msgr2=0x7f913003e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 0x7f913003e080 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f913c007fd0 tx=0x7f913c009040 comp rx=0 tx=0).stop 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 10 monclient: shutdown 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 msgr2=0x7f914c135ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f914800e9b0 tx=0x7f914800ee70 comp rx=0 tx=0).stop 2024-09-17T15:49:06.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 shutdown_connections 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f913003bbc0 0x7f913003e080 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f914c13aff0 0x7f914c071e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f914c068db0 0x7f914c136400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 --2- 172.21.15.86:0/842919376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f914c0689d0 0x7f914c135ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 >> 172.21.15.86:0/842919376 conn(0x7f914c0a6440 msgr2=0x7f914c09e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 shutdown_connections 2024-09-17T15:49:06.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.514+0000 7f9153df5640 1 -- 172.21.15.86:0/842919376 wait complete. 2024-09-17T15:49:06.528 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:49:06.528 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:49:06.528 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:49:06.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:06.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: get_monmap_and_config 2024-09-17T15:49:06.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: monmap: 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:06.656107+0000 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:06.656107+0000 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:06.652 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 Processor -- start 2024-09-17T15:49:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 -- start start 2024-09-17T15:49:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: init 2024-09-17T15:49:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f9fdc073fe0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9fdc074900 0x7f9fdc0a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f9fdc074900 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f9fdc0a07b0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9fdc0704f0 con 0x7f9fdc0a07b0 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9fdc0707d0 con 0x7f9fdc074900 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9fdc070ab0 con 0x7f9fdc073fe0 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fe1ecf640 10 monclient(hunting): authenticate will time out at 2279.621784s 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42252/0 (socket says 172.21.15.86:42252) 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35122/0 (socket says 172.21.15.86:35122) 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/4162387059 learned_addr learned my addr 172.21.15.86:0/4162387059 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request con 0x7f9fdc073fe0 auth_method 0 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): get_auth_request con 0x7f9fdc0a07b0 auth_method 0 2024-09-17T15:49:06.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdaffd640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.203:3300/0 conn(0x7f9fdc074900 0x7f9fdc0a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdaffd640 10 monclient(hunting): get_auth_request con 0x7f9fdc074900 auth_method 0 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_done global_id 164924 payload 274 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/4162387059 >> v2:172.21.15.203:3300/0 conn(0x7f9fdc074900 msgr2=0x7f9fdc0a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.203:3300/0 conn(0x7f9fdc074900 0x7f9fdc0a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/4162387059 >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 msgr2=0x7f9fdc0a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 monclient: found mon.noname-b 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/4162387059 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc09fe80 con 0x7f9fdc073fe0 2024-09-17T15:49:06.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.654+0000 7f9fdbfff640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:06.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/4162387059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 msgr2=0x7f9fdc0743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/4162387059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0a46f0 tx=0x7f9fd401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6684c27711e4cccd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 1 -- 172.21.15.86:0/4162387059 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40a55c0 con 0x7f9fdc073fe0 2024-09-17T15:49:06.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: dump: 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 1 -- 172.21.15.86:0/4162387059 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fd40a5760 con 0x7f9fdc073fe0 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 5 monclient: authenticate success, global_id 164924 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 1 -- 172.21.15.86:0/4162387059 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40a5b30 con 0x7f9fdc073fe0 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fda7fc640 10 monclient: dump: 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.667 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 4 set_mon_vals no callback set 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 10 monclient: shutdown 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/4162387059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 msgr2=0x7f9fdc0743c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/4162387059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc0a46f0 tx=0x7f9fd401c940 comp rx=0 tx=0).stop 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/4162387059 shutdown_connections 2024-09-17T15:49:06.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.86:3300/0 conn(0x7f9fdc0a07b0 0x7f9fdc0a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/4162387059 >> v2:172.21.15.203:3300/0 conn(0x7f9fdc074900 0x7f9fdc0a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/4162387059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc073fe0 0x7f9fdc0743c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/4162387059 >> 172.21.15.86:0/4162387059 conn(0x7f9fdc0a3630 msgr2=0x7f9fdc0a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/4162387059 shutdown_connections 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.658+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/4162387059 wait complete. 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient: monmap: 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 Processor -- start 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 -- start start 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient: init 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 0x7f9fdc1cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f9fdc073fe0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 0x7f9fdc13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f9fdc074900 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fdc0a07b0 0x7f9fdc13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f9fdc0a07b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc138260 con 0x7f9fdc073fe0 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc0a33d0 con 0x7f9fdc0a07b0 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9fdc1418d0 con 0x7f9fdc074900 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fe1ecf640 10 monclient(hunting): authenticate will time out at 2279.629111s 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 0x7f9fdc1cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 0x7f9fdc13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 0x7f9fdc1cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35134/0 (socket says 172.21.15.86:35134) 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/3327130378 learned_addr learned my addr 172.21.15.86:0/3327130378 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request con 0x7f9fdc073fe0 auth_method 0 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): get_auth_request con 0x7f9fdc074900 auth_method 0 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdbfff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fdc0a07b0 0x7f9fdc13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdbfff640 10 monclient(hunting): get_auth_request con 0x7f9fdc0a07b0 auth_method 0 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient(hunting): handle_auth_done global_id 164927 payload 824 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fdc0a07b0 msgr2=0x7f9fdc13e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fdc0a07b0 0x7f9fdc13e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 msgr2=0x7f9fdc1cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 0x7f9fdc1cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 monclient: found mon.noname-c 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc0a68c0 con 0x7f9fdc074900 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc073fe0 0x7f9fdc1cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 msgr2=0x7f9fdc13df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.662+0000 7f9fdaffd640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 0x7f9fdc13df10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400d5f0 tx=0x7f9fc400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4012800 con 0x7f9fdc074900 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: dump: 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.674 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc4019070 con 0x7f9fdc074900 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 5 monclient: authenticate success, global_id 164927 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 monclient: _renew_subs 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc13e990 con 0x7f9fdc074900 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4012bc0 con 0x7f9fdc074900 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd9ffb640 4 set_mon_vals no callback set 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: dump: 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 monclient: _renew_subs 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc13c140 con 0x7f9fdc074900 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 0x7f9fdc139040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc13bec0 con 0x7f9fdc138c60 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 0x7f9fdc139040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient: get_auth_request con 0x7f9fdc138c60 auth_method 0 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9fc401e460 con 0x7f9fdc074900 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_done global_id 234249 payload 824 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fdc0743c0 con 0x7f9fdc138c60 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9fbc03bb60 0x7f9fbc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(767..767 src has 696..767) ==== 4771+0+0 (secure 0 0 0) 0x7f9fc4050320 con 0x7f9fdc074900 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdbfff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9fbc03bb60 0x7f9fbc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdbfff640 10 monclient: get_auth_request con 0x7f9fbc03bb60 auth_method 0 2024-09-17T15:49:06.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 msgr2=0x7f9fdc139040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 0x7f9fdc139040 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401d1e0 tx=0x7f9fd40196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40adad0 con 0x7f9fdc138c60 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.666+0000 7f9fd3fff640 10 monclient: dump: 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9fd40adad0 con 0x7f9fdc138c60 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 msgr2=0x7f9fdc139040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 0x7f9fdc139040 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401d1e0 tx=0x7f9fd40196e0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 msgr2=0x7f9fdc139040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fd3fff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc138c60 0x7f9fdc139040 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401d1e0 tx=0x7f9fd40196e0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.670+0000 7f9fdbfff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9fbc03bb60 0x7f9fbc03e020 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc070420 tx=0x7f9fcc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fe1ecf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 0x7f9fdc0725b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fe1ecf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc13b1e0 con 0x7f9fdc139580 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 0x7f9fdc0725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient: get_auth_request con 0x7f9fdc139580 auth_method 0 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 10 monclient(hunting): handle_auth_done global_id 234252 payload 824 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.674+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/3327130378 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fdc13ca40 con 0x7f9fdc139580 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fdb7fe640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 msgr2=0x7f9fdc0725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fdb7fe640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 0x7f9fdc0725b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc077790 tx=0x7f9fd40a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4003db0 con 0x7f9fdc139580 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:06.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 10 monclient: dump: 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f9fd4003db0 con 0x7f9fdc139580 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:06.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 msgr2=0x7f9fdc0725b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 0x7f9fdc0725b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc077790 tx=0x7f9fd40a7000 comp rx=0 tx=0).stop 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 msgr2=0x7f9fdc0725b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fd3fff640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fdc139580 0x7f9fdc0725b0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc077790 tx=0x7f9fd40a7000 comp rx=0 tx=0).stop 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 75873, 2024-09-17T15:49:06.685 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.686 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.687 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:06.688 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.689 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.690 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:06.693 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:49:06.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.697 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:06.697 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:06.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9fbc03bb60 msgr2=0x7f9fbc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9fbc03bb60 0x7f9fbc03e020 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc070420 tx=0x7f9fcc008040 comp rx=0 tx=0).stop 2024-09-17T15:49:06.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 10 monclient: shutdown 2024-09-17T15:49:06.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 msgr2=0x7f9fdc13df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 0x7f9fdc13df10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400d5f0 tx=0x7f9fc400dab0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fdbfff640 1 -- 172.21.15.86:0/3327130378 reap_dead start 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 shutdown_connections 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 --2- 172.21.15.86:0/3327130378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fdc074900 0x7f9fdc13df10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 >> 172.21.15.86:0/3327130378 conn(0x7f9fdc0a3630 msgr2=0x7f9fdc0a4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 shutdown_connections 2024-09-17T15:49:06.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.678+0000 7f9fe1ecf640 1 -- 172.21.15.86:0/3327130378 wait complete. 2024-09-17T15:49:06.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:06.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:49:06.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 10 monclient: get_monmap_and_config 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 10 monclient: monmap: 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:06.818802+0000 2024-09-17T15:49:06.814 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:06.818802+0000 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:06.815 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 1 Processor -- start 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 1 -- start start 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.814+0000 7f4b835fb640 10 monclient: init 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b7c0a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7c0a1760 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4b7c13b450 0x7f4b7c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7c13b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7c0a2130 con 0x7f4b7c0a0c70 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f4b7c0a2410 con 0x7f4b7c0a1760 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4b7c0a26f0 con 0x7f4b7c13b450 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b835fb640 10 monclient(hunting): authenticate will time out at 2279.784253s 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81397640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4b7c0a0c70 0x7f4b7c0a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53058/0 (socket says 172.21.15.86:53058) 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81397640 1 -- 172.21.15.86:0/2330431358 learned_addr learned my addr 172.21.15.86:0/2330431358 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 10 monclient(hunting): get_auth_request con 0x7f4b7c0a1760 auth_method 0 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 1 -- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 msgr2=0x7f4b7c13af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 1 -- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 msgr2=0x7f4b7c13af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b80b96640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.203:3300/0 conn(0x7f4b7c13b450 0x7f4b7c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): get_auth_request con 0x7f4b7c13b450 auth_method 0 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient(hunting): handle_auth_done global_id 244114 payload 274 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 -- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 msgr2=0x7f4b7c13af10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 -- 172.21.15.86:0/2330431358 >> v1:172.21.15.86:6789/0 conn(0x7f4b7c0a0c70 legacy=0x7f4b7c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 monclient: found mon.noname-c 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:49:06.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.818+0000 7f4b81b98640 1 -- 172.21.15.86:0/2330431358 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c09f9f0 con 0x7f4b7c13b450 2024-09-17T15:49:06.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b81b98640 1 -- 172.21.15.86:0/2330431358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c13b450 msgr2=0x7f4b7c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b81b98640 1 --2- 172.21.15.86:0/2330431358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c13b450 0x7f4b7c13d840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c06f840 tx=0x7f4b6c01c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65ea2d764af636e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 1 -- 172.21.15.86:0/2330431358 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0b0070 con 0x7f4b7c13b450 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: dump: 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.824 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 1 -- 172.21.15.86:0/2330431358 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6c0ab600 con 0x7f4b7c13b450 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b835fb640 5 monclient: authenticate success, global_id 244114 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b835fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b835fb640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 1 -- 172.21.15.86:0/2330431358 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0ab9d0 con 0x7f4b7c13b450 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b835fb640 4 set_mon_vals no callback set 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.822+0000 7f4b7bfff640 10 monclient: dump: 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: shutdown 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- 172.21.15.86:0/2330431358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c13b450 msgr2=0x7f4b7c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- 172.21.15.86:0/2330431358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c13b450 0x7f4b7c13d840 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c06f840 tx=0x7f4b6c01c340 comp rx=0 tx=0).stop 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- 172.21.15.86:0/2330431358 shutdown_connections 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- 172.21.15.86:0/2330431358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c13b450 0x7f4b7c13d840 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- 172.21.15.86:0/2330431358 >> v2:172.21.15.86:3300/0 conn(0x7f4b7c0a1760 0x7f4b7c13af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- 172.21.15.86:0/2330431358 >> 172.21.15.86:0/2330431358 conn(0x7f4b7c0a2f40 msgr2=0x7f4b7c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- 172.21.15.86:0/2330431358 shutdown_connections 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- 172.21.15.86:0/2330431358 wait complete. 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: monmap: 2024-09-17T15:49:06.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:06.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 Processor -- start 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- start start 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: init 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c0a0c70 0x7f4b7c13a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7c0a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c0a1760 0x7f4b7c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7c0a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 0x7f4b7c09dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7c13b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c140720 con 0x7f4b7c13b450 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7c1405a0 con 0x7f4b7c0a0c70 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c1408a0 con 0x7f4b7c0a1760 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b835fb640 10 monclient(hunting): authenticate will time out at 2279.793837s 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 0x7f4b7c09dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c0a1760 0x7f4b7c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 0x7f4b7c09dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35168/0 (socket says 172.21.15.86:35168) 2024-09-17T15:49:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 learned_addr learned my addr 172.21.15.86:0/3920793124 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): get_auth_request con 0x7f4b7c13b450 auth_method 0 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): get_auth_request con 0x7f4b7c0a1760 auth_method 0 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81397640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c0a0c70 0x7f4b7c13a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81397640 10 monclient(hunting): get_auth_request con 0x7f4b7c0a0c70 auth_method 0 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81397640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81397640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81397640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient(hunting): handle_auth_done global_id 234258 payload 824 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c0a1760 msgr2=0x7f4b7c09d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c0a1760 0x7f4b7c09d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c0a0c70 msgr2=0x7f4b7c13a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b7c0a0c70 0x7f4b7c13a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 monclient: found mon.noname-a 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7c138960 con 0x7f4b7c13b450 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.826+0000 7f4b80b96640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c0a1760 0x7f4b7c09d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 msgr2=0x7f4b7c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81b98640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 0x7f4b7c09dcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0ae040 tx=0x7f4b6c0190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0b8070 con 0x7f4b7c13b450 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: dump: 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.838 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6c0b4630 con 0x7f4b7c13b450 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 5 monclient: authenticate success, global_id 234258 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 monclient: _renew_subs 2024-09-17T15:49:06.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c138c40 con 0x7f4b7c13b450 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0b0040 con 0x7f4b7c13b450 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 10 monclient: dump: 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 monclient: _renew_subs 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b7b7fe640 4 set_mon_vals no callback set 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7c09d460 con 0x7f4b7c13b450 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 0x7f4b7c09e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c0a1050 con 0x7f4b7c1324e0 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4b6c0bb070 con 0x7f4b7c13b450 2024-09-17T15:49:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 0x7f4b7c09e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient: get_auth_request con 0x7f4b7c1324e0 auth_method 0 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4b6803bbb0 0x7f4b6803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(767..767 src has 696..767) ==== 4771+0+0 (secure 0 0 0) 0x7f4b6c0e9c30 con 0x7f4b7c13b450 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 10 monclient(hunting): handle_auth_done global_id 164933 payload 824 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b81397640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b7c13ccf0 con 0x7f4b7c1324e0 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b80b96640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4b6803bbb0 0x7f4b6803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.830+0000 7f4b80b96640 10 monclient: get_auth_request con 0x7f4b6803bbb0 auth_method 0 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b81397640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 msgr2=0x7f4b7c09e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b81397640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 0x7f4b7c09e1f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400ebc0 tx=0x7f4b7400c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74017020 con 0x7f4b7c1324e0 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 10 monclient: dump: 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4b7400cdf0 con 0x7f4b7c1324e0 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 msgr2=0x7f4b7c09e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 0x7f4b7c09e1f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400ebc0 tx=0x7f4b7400c4b0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 msgr2=0x7f4b7c09e1f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b79ffb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c1324e0 0x7f4b7c09e1f0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4b7400ebc0 tx=0x7f4b7400c4b0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.834+0000 7f4b80b96640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4b6803bbb0 0x7f4b6803e070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c074900 tx=0x7f4b70008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b835fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b835fb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 0x7f4b7c06fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b835fb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7c06d4f0 con 0x7f4b7c06f9b0 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 0x7f4b7c06fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient: get_auth_request con 0x7f4b7c06f9b0 auth_method 0 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 10 monclient(hunting): handle_auth_done global_id 164936 payload 824 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.838+0000 7f4b81397640 1 -- 172.21.15.86:0/3920793124 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4b7c1385e0 con 0x7f4b7c06f9b0 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b81397640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 msgr2=0x7f4b7c06fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b81397640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 0x7f4b7c06fd90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4b74018000 tx=0x7f4b74004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74017040 con 0x7f4b7c06f9b0 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 10 monclient: dump: 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f4b7401e220 con 0x7f4b7c06f9b0 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:06.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 msgr2=0x7f4b7c06fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 0x7f4b7c06fd90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4b74018000 tx=0x7f4b74004cd0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 msgr2=0x7f4b7c06fd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b79ffb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b7c06f9b0 0x7f4b7c06fd90 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4b74018000 tx=0x7f4b74004cd0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:06.849 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 450030, 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.850 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.851 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:06.852 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:06.853 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.854 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:06.855 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:06.856 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.857 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:06.858 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:06.859 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.860 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:06.861 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:06.862 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:06.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4b6803bbb0 msgr2=0x7f4b6803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4b6803bbb0 0x7f4b6803e070 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c074900 tx=0x7f4b70008040 comp rx=0 tx=0).stop 2024-09-17T15:49:06.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 10 monclient: shutdown 2024-09-17T15:49:06.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:06.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 msgr2=0x7f4b7c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 --2- 172.21.15.86:0/3920793124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b7c13b450 0x7f4b7c09dcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c0ae040 tx=0x7f4b6c0190e0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 reap_dead start 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b81b98640 1 -- 172.21.15.86:0/3920793124 reap_dead start 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 shutdown_connections 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 >> 172.21.15.86:0/3920793124 conn(0x7f4b7c0a2f40 msgr2=0x7f4b7c135de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 shutdown_connections 2024-09-17T15:49:06.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.842+0000 7f4b835fb640 1 -- 172.21.15.86:0/3920793124 wait complete. 2024-09-17T15:49:06.864 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:06.864 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:49:06.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3507: done (4 left) 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3508: copy_from oid 29 from oid 25 current snap is 373 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3505: finishing copy_from to smithi20328086-19 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3505: got expected ENOENT (src dne) 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3505: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:49:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3506: expect (ObjNum 1216 snap 367 seq_num 1216) 2024-09-17T15:49:06.914 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:06.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: get_monmap_and_config 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: build_initial_monmap 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: monmap: 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:06.989339+0000 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:06.989339+0000 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:06.985 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:06.986 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:06.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 Processor -- start 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 -- start start 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: init 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f160c0a50e0 0x7f160c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f160c0a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f160c0a54c0 0x7f160c0a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f160c0a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f160c137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f160c0a6280 con 0x7f160c137ea0 2024-09-17T15:49:06.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f160c09c0a0 con 0x7f160c0a50e0 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f160c09c380 con 0x7f160c0a54c0 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f1611ba2640 10 monclient(hunting): authenticate will time out at 2279.954757s 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f160c0a50e0 0x7f160c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f160c0a50e0 0x7f160c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35170/0 (socket says 172.21.15.86:35170) 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 -- 172.21.15.86:0/1409246995 learned_addr learned my addr 172.21.15.86:0/1409246995 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): get_auth_request con 0x7f160c0a50e0 auth_method 0 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 1 --2- 172.21.15.86:0/1409246995 >> v2:172.21.15.203:3300/0 conn(0x7f160c0a54c0 0x7f160c0a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): get_auth_request con 0x7f160c0a54c0 auth_method 0 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient(hunting): handle_auth_done global_id 234261 payload 274 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 -- 172.21.15.86:0/1409246995 >> v2:172.21.15.203:3300/0 conn(0x7f160c0a54c0 msgr2=0x7f160c0a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 --2- 172.21.15.86:0/1409246995 >> v2:172.21.15.203:3300/0 conn(0x7f160c0a54c0 0x7f160c0a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 -- 172.21.15.86:0/1409246995 >> v1:172.21.15.203:6789/0 conn(0x7f160c137ea0 legacy=0x7f160c138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:06.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.986+0000 7f160b7fe640 1 -- 172.21.15.86:0/1409246995 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c09c660 con 0x7f160c0a50e0 2024-09-17T15:49:06.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160b7fe640 1 -- 172.21.15.86:0/1409246995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 msgr2=0x7f160c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:06.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160b7fe640 1 --2- 172.21.15.86:0/1409246995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c09cd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f160c06cc20 tx=0x7f160401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45aac7ec87ad1dec server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:06.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 1 -- 172.21.15.86:0/1409246995 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040a5550 con 0x7f160c0a50e0 2024-09-17T15:49:06.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:06.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: dump: 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.000 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 1 -- 172.21.15.86:0/1409246995 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16040a56f0 con 0x7f160c0a50e0 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f1611ba2640 5 monclient: authenticate success, global_id 234261 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f1611ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:07.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f1611ba2640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f1611ba2640 4 set_mon_vals no callback set 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 1 -- 172.21.15.86:0/1409246995 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040a5ac0 con 0x7f160c0a50e0 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.990+0000 7f160a7fc640 10 monclient: dump: 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: shutdown 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- 172.21.15.86:0/1409246995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 msgr2=0x7f160c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- 172.21.15.86:0/1409246995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c09cd60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f160c06cc20 tx=0x7f160401c940 comp rx=0 tx=0).stop 2024-09-17T15:49:07.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- 172.21.15.86:0/1409246995 shutdown_connections 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- 172.21.15.86:0/1409246995 >> v2:172.21.15.203:3300/0 conn(0x7f160c0a54c0 0x7f160c0a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- 172.21.15.86:0/1409246995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c09cd60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- 172.21.15.86:0/1409246995 >> 172.21.15.86:0/1409246995 conn(0x7f160c0aaae0 msgr2=0x7f160c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- 172.21.15.86:0/1409246995 shutdown_connections 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- 172.21.15.86:0/1409246995 wait complete. 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: build_initial_monmap 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: monmap: 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:07.004 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:07.006 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 Processor -- start 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- start start 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: init 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:07.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c1d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f160c0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c0a54c0 0x7f160c137100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f160c0a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 0x7f160c13c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f160c137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f160c142cb0 con 0x7f160c0a50e0 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f160c142b30 con 0x7f160c0a54c0 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f160c142e30 con 0x7f160c137ea0 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f1611ba2640 10 monclient(hunting): authenticate will time out at 2279.962122s 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c1d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 0x7f160c13c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 0x7f160c13c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42302/0 (socket says 172.21.15.86:42302) 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 1 -- 172.21.15.86:0/2524851183 learned_addr learned my addr 172.21.15.86:0/2524851183 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c1d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35186/0 (socket says 172.21.15.86:35186) 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): get_auth_request con 0x7f160c137ea0 auth_method 0 2024-09-17T15:49:07.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): get_auth_request con 0x7f160c0a50e0 auth_method 0 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c0a54c0 0x7f160c137100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 10 monclient(hunting): get_auth_request con 0x7f160c0a54c0 auth_method 0 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient(hunting): handle_auth_done global_id 234264 payload 824 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 msgr2=0x7f160c13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 0x7f160c13c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c0a54c0 msgr2=0x7f160c137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c0a54c0 0x7f160c137100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:49:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160b7fe640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f160c13dce0 con 0x7f160c0a50e0 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160bfff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f160c137ea0 0x7f160c13c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.994+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c0a54c0 0x7f160c137100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160b7fe640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 msgr2=0x7f160c1d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160b7fe640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c1d0050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16040ab000 tx=0x7f16040196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040bc070 con 0x7f160c0a50e0 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: dump: 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.012 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16040b8880 con 0x7f160c0a50e0 2024-09-17T15:49:07.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16040a5550 con 0x7f160c0a50e0 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 5 monclient: authenticate success, global_id 234264 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 monclient: _renew_subs 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f160c0a4590 con 0x7f160c0a50e0 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 10 monclient: dump: 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.014 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1609ffb640 4 set_mon_vals no callback set 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 monclient: _renew_subs 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f160c1211f0 con 0x7f160c0a50e0 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 0x7f160c13c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f160c06a0e0 con 0x7f160c137640 2024-09-17T15:49:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f16040b8cf0 con 0x7f160c0a50e0 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 0x7f160c13c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient: get_auth_request con 0x7f160c137640 auth_method 0 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15ec03bbb0 0x7f15ec03e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(768..768 src has 696..768) ==== 4799+0+0 (secure 0 0 0) 0x7f16040ea380 con 0x7f160c0a50e0 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 10 monclient(hunting): handle_auth_done global_id 244126 payload 824 2024-09-17T15:49:07.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160bfff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15ec03bbb0 0x7f15ec03e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160affd640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f160c121b90 con 0x7f160c137640 2024-09-17T15:49:07.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:06.998+0000 7f160bfff640 10 monclient: get_auth_request con 0x7f15ec03bbb0 auth_method 0 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f160affd640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 msgr2=0x7f160c13c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 0x7f160c13c820 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f160c0a43d0 tx=0x7f15f4011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011ea0 con 0x7f160c137640 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 10 monclient: dump: 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.019 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f160bfff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15ec03bbb0 0x7f15ec03e070 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f160c13e2a0 tx=0x7f15fc007e80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f15f400fe50 con 0x7f160c137640 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 msgr2=0x7f160c13c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 0x7f160c13c820 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f160c0a43d0 tx=0x7f15f4011c90 comp rx=0 tx=0).stop 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 msgr2=0x7f160c13c820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.002+0000 7f1603fff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c137640 0x7f160c13c820 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f160c0a43d0 tx=0x7f15f4011c90 comp rx=0 tx=0).stop 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f1611ba2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f1611ba2640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 0x7f160c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f1611ba2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f160c13d170 con 0x7f160c13cd60 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 0x7f160c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient: get_auth_request con 0x7f160c13cd60 auth_method 0 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 10 monclient(hunting): handle_auth_done global_id 244129 payload 824 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.010+0000 7f160affd640 1 -- 172.21.15.86:0/2524851183 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f160c09cd60 con 0x7f160c13cd60 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f160affd640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 msgr2=0x7f160c1392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f160affd640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 0x7f160c1392d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15f401c000 tx=0x7f15f40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011ea0 con 0x7f160c13cd60 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 10 monclient: dump: 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.022 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f15f4015870 con 0x7f160c13cd60 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 msgr2=0x7f160c1392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 0x7f160c1392d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15f401c000 tx=0x7f15f40046d0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 msgr2=0x7f160c1392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:07.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1603fff640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f160c13cd60 0x7f160c1392d0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f15f401c000 tx=0x7f15f40046d0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 200, 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7923, 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:07.024 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:07.025 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:07.026 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:07.027 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:07.028 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:07.029 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:07.030 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.032 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:07.034 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:07.035 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:07.037 INFO:tasks.rados.rados.0.smithi203.stdout:3503: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:49:07.037 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15ec03bbb0 msgr2=0x7f15ec03e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f15ec03bbb0 0x7f15ec03e070 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f160c13e2a0 tx=0x7f15fc007e80 comp rx=0 tx=0).stop 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 10 monclient: shutdown 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 msgr2=0x7f160c1d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 --2- 172.21.15.86:0/2524851183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f160c0a50e0 0x7f160c1d0050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f16040ab000 tx=0x7f16040196e0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f160bfff640 1 -- 172.21.15.86:0/2524851183 reap_dead start 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f160bfff640 1 -- 172.21.15.86:0/2524851183 reap_dead start 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 shutdown_connections 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 >> 172.21.15.86:0/2524851183 conn(0x7f160c0aaae0 msgr2=0x7f160c0a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 shutdown_connections 2024-09-17T15:49:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.014+0000 7f1611ba2640 1 -- 172.21.15.86:0/2524851183 wait complete. 2024-09-17T15:49:07.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:07.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:49:07.038 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:49:07.039 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:49:07.039 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:49:07.039 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:49:07.039 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:49:07.052 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:49:07.052 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:49:07.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:07.098 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3508: finishing copy_from to smithi20328086-29 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3508: got expected ENOENT (src dne) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3508: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3503: done (4 left) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3504: done (3 left) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3505: done (2 left) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3506: done (1 left) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3508: done (0 left) 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:3509: rollback oid 46 current snap is 373 2024-09-17T15:49:07.136 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 370 2024-09-17T15:49:07.137 INFO:tasks.rados.rados.0.smithi203.stdout:3510: write oid 23 current snap is 373 2024-09-17T15:49:07.137 INFO:tasks.rados.rados.0.smithi203.stdout:3510: seq_num 1234 ranges {739838=506452,1877368=613417} 2024-09-17T15:49:07.143 INFO:tasks.rados.rados.0.smithi203.stdout:3510: writing smithi20328086-23 from 739838 to 1246290 tid 1 2024-09-17T15:49:07.155 INFO:tasks.rados.rados.0.smithi203.stdout:3510: writing smithi20328086-23 from 1877368 to 2490785 tid 2 2024-09-17T15:49:07.155 INFO:tasks.rados.rados.0.smithi203.stdout:3511: write (excl) oid 6 current snap is 373 2024-09-17T15:49:07.155 INFO:tasks.rados.rados.0.smithi203.stdout:3511: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: get_monmap_and_config 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: build_initial_monmap 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: monmap: 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.165 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:07.170024+0000 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:07.170024+0000 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:07.166 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 Processor -- start 2024-09-17T15:49:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 -- start start 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: init 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d5c0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 0x7f5d5c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5c0694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5c13aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d5c09c8b0 con 0x7f5d5c0689d0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5d5c074500 con 0x7f5d5c13aff0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.166+0000 7f5d613a4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5d5c09cb90 con 0x7f5d5c0694c0 2024-09-17T15:49:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d613a4640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d613a4640 10 monclient(hunting): authenticate will time out at 2280.136015s 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35196/0 (socket says 172.21.15.86:35196) 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/1003185857 learned_addr learned my addr 172.21.15.86:0/1003185857 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5a7fc640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 msgr2=0x7f5d5c06f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5d5c13aff0 auth_method 0 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5a7fc640 1 --2- 172.21.15.86:0/1003185857 >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 0x7f5d5c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_done global_id 234267 payload 274 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/1003185857 >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 msgr2=0x7f5d5c06f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 --2- 172.21.15.86:0/1003185857 >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 0x7f5d5c06f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/1003185857 >> v1:172.21.15.86:6789/0 conn(0x7f5d5c0689d0 legacy=0x7f5d5c068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:07.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.170+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/1003185857 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c09ce70 con 0x7f5d5c13aff0 2024-09-17T15:49:07.172 INFO:tasks.rados.rados.0.smithi203.stdout:3511: writing smithi20328086-6 from 756645 to 1338346 tid 1 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/1003185857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c13aff0 msgr2=0x7f5d5c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 --2- 172.21.15.86:0/1003185857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c069940 tx=0x7f5d5401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38b836aea93c1911 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 1 -- 172.21.15.86:0/1003185857 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c09c2d0 con 0x7f5d5c13aff0 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: dump: 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.173 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 1 -- 172.21.15.86:0/1003185857 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d540a5750 con 0x7f5d5c13aff0 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 5 monclient: authenticate success, global_id 234267 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 4 set_mon_vals no callback set 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 1 -- 172.21.15.86:0/1003185857 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d540a5b20 con 0x7f5d5c13aff0 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d59ffb640 10 monclient: dump: 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.175 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: shutdown 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- 172.21.15.86:0/1003185857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c13aff0 msgr2=0x7f5d5c13d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- 172.21.15.86:0/1003185857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c069940 tx=0x7f5d5401c8b0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- 172.21.15.86:0/1003185857 shutdown_connections 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- 172.21.15.86:0/1003185857 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c13aff0 0x7f5d5c13d3e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- 172.21.15.86:0/1003185857 >> v2:172.21.15.203:3300/0 conn(0x7f5d5c0694c0 0x7f5d5c06f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- 172.21.15.86:0/1003185857 >> 172.21.15.86:0/1003185857 conn(0x7f5d5c0a6440 msgr2=0x7f5d5c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- 172.21.15.86:0/1003185857 shutdown_connections 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- 172.21.15.86:0/1003185857 wait complete. 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: build_initial_monmap 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: monmap: 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:07.176 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:07.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 Processor -- start 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- start start 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: init 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:07.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 0x7f5d5c132db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-b con 0x7f5d5c0694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5c13aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c1405f0 con 0x7f5d5c0689d0 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c140470 con 0x7f5d5c0694c0 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5c140770 con 0x7f5d5c13aff0 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d613a4640 10 monclient(hunting): authenticate will time out at 2280.143001s 2024-09-17T15:49:07.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5a7fc640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 msgr2=0x7f5d5c132db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 0x7f5d5c132db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42312/0 (socket says 172.21.15.86:42312) 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35210/0 (socket says 172.21.15.86:35210) 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/309092281 learned_addr learned my addr 172.21.15.86:0/309092281 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): get_auth_request con 0x7f5d5c13aff0 auth_method 0 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): get_auth_request con 0x7f5d5c0689d0 auth_method 0 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:07.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient(hunting): handle_auth_done global_id 164942 payload 824 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 msgr2=0x7f5d5c132db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 0x7f5d5c132db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 msgr2=0x7f5d5c132870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/309092281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c11d120 con 0x7f5d5c13aff0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.174+0000 7f5d5affd640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d5b7fe640 1 -- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 msgr2=0x7f5d5c070c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d5b7fe640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5d5401d150 tx=0x7f5d54019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d540bc020 con 0x7f5d5c13aff0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: dump: 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.185 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d540b8880 con 0x7f5d5c13aff0 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 5 monclient: authenticate success, global_id 164942 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _renew_subs 2024-09-17T15:49:07.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5c11d400 con 0x7f5d5c13aff0 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d540b8c50 con 0x7f5d5c13aff0 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 10 monclient: dump: 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d597fa640 4 set_mon_vals no callback set 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _renew_subs 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5c09d460 con 0x7f5d5c13aff0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5d540bf020 con 0x7f5d5c13aff0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5c06f840 con 0x7f5d5c13aff0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 0x7f5d4803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(768..768 src has 696..768) ==== 4799+0+0 (secure 0 0 0) 0x7f5d540eadc0 con 0x7f5d5c13aff0 2024-09-17T15:49:07.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d5affd640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 0x7f5d4803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:07.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.178+0000 7f5d5affd640 10 monclient: get_auth_request con 0x7f5d4803bb60 auth_method 0 2024-09-17T15:49:07.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.182+0000 7f5d5affd640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 0x7f5d4803e020 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c007fd0 tx=0x7f5d4c007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:07.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.186+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5d540c6070 con 0x7f5d5c13aff0 2024-09-17T15:49:07.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.186+0000 7f5d437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:07.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.186+0000 7f5d437fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3511: writing smithi20328086-6 from 1935850 to 2409278 tid 2 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3512: copy_from oid 29 from oid 24 current snap is 373 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3509: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3509: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3509: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1431 (ObjNum 1192 snap 363 seq_num 1192) dirty exists 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3510: finishing write tid 1 to smithi20328086-23 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3510: finishing write tid 2 to smithi20328086-23 2024-09-17T15:49:07.191 INFO:tasks.rados.rados.0.smithi203.stdout:3510: finishing write tid 3 to smithi20328086-23 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: finishing write tid 4 to smithi20328086-23 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: oid 23 updating version 0 to 1641 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: oid 23 updating version 1641 to 1642 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: oid 23 updating version 1642 to 1643 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1643 (ObjNum 1234 snap 373 seq_num 1234) dirty exists 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: left oid 23 (ObjNum 1234 snap 373 seq_num 1234) 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3511: finishing write tid 1 to smithi20328086-6 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3509: done (3 left) 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3510: done (2 left) 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3513: delete oid 27 current snap is 373 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3511: finishing write tid 2 to smithi20328086-6 2024-09-17T15:49:07.192 INFO:tasks.rados.rados.0.smithi203.stdout:3511: finishing write tid 3 to smithi20328086-6 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: finishing write tid 4 to smithi20328086-6 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: oid 6 updating version 0 to 1644 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: oid 6 updating version 1644 to 1645 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: oid 6 updating version 1645 to 1646 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1646 (ObjNum 1235 snap 373 seq_num 1235) dirty exists 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: left oid 6 (ObjNum 1235 snap 373 seq_num 1235) 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3511: done (2 left) 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3513: done (1 left) 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3514: copy_from oid 48 from oid 9 current snap is 373 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3515: copy_from oid 47 from oid 36 current snap is 373 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3516: write (excl) oid 12 current snap is 373 2024-09-17T15:49:07.193 INFO:tasks.rados.rados.0.smithi203.stdout:3516: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-09-17T15:49:07.208 INFO:tasks.rados.rados.0.smithi203.stdout:3516: writing smithi20328086-12 from 790259 to 1522458 tid 1 2024-09-17T15:49:07.217 INFO:tasks.rados.rados.0.smithi203.stdout:3516: writing smithi20328086-12 from 1936461 to 2413558 tid 2 2024-09-17T15:49:07.217 INFO:tasks.rados.rados.0.smithi203.stdout:3517: delete oid 3 current snap is 373 2024-09-17T15:49:07.217 INFO:tasks.rados.rados.0.smithi203.stdout:3515: finishing copy_from to smithi20328086-47 2024-09-17T15:49:07.217 INFO:tasks.rados.rados.0.smithi203.stdout:3515: got expected ENOENT (src dne) 2024-09-17T15:49:07.217 INFO:tasks.rados.rados.0.smithi203.stdout:3515: finishing copy_from racing read to smithi20328086-47 2024-09-17T15:49:07.218 INFO:tasks.rados.rados.0.smithi203.stdout:3512: finishing copy_from racing read to smithi20328086-29 2024-09-17T15:49:07.218 INFO:tasks.rados.rados.0.smithi203.stdout:3512: finishing copy_from to smithi20328086-29 2024-09-17T15:49:07.218 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1545 (ObjNum 1232 snap 371 seq_num 1232) dirty exists 2024-09-17T15:49:07.222 INFO:tasks.rados.rados.0.smithi203.stdout:3512: done (4 left) 2024-09-17T15:49:07.222 INFO:tasks.rados.rados.0.smithi203.stdout:3515: done (3 left) 2024-09-17T15:49:07.222 INFO:tasks.rados.rados.0.smithi203.stdout:3517: done (2 left) 2024-09-17T15:49:07.222 INFO:tasks.rados.rados.0.smithi203.stdout:3518: write (excl) oid 33 current snap is 373 2024-09-17T15:49:07.222 INFO:tasks.rados.rados.0.smithi203.stdout:3518: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-17T15:49:07.230 INFO:tasks.rados.rados.0.smithi203.stdout:3518: writing smithi20328086-33 from 423873 to 906570 tid 1 2024-09-17T15:49:07.242 INFO:tasks.rados.rados.0.smithi203.stdout:3518: writing smithi20328086-33 from 1653425 to 2134191 tid 2 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3518: writing smithi20328086-33 from 2691441 to 2823873 tid 3 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3519: rollback oid 10 current snap is 373 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 363 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3516: finishing write tid 1 to smithi20328086-12 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3516: finishing write tid 2 to smithi20328086-12 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3516: finishing write tid 3 to smithi20328086-12 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3516: finishing write tid 4 to smithi20328086-12 2024-09-17T15:49:07.249 INFO:tasks.rados.rados.0.smithi203.stdout:3516: oid 12 updating version 0 to 1704 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3516: oid 12 updating version 1704 to 1705 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3516: oid 12 version 1705 is already newer than 1703 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1705 (ObjNum 1237 snap 373 seq_num 1237) dirty exists 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3516: left oid 12 (ObjNum 1237 snap 373 seq_num 1237) 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3514: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3518: finishing write tid 1 to smithi20328086-33 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3514: finishing copy_from to smithi20328086-48 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1538 (ObjNum 940 snap 287 seq_num 940) dirty exists 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3518: finishing write tid 2 to smithi20328086-33 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3514: done (3 left) 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3516: done (2 left) 2024-09-17T15:49:07.250 INFO:tasks.rados.rados.0.smithi203.stdout:3520: read oid 20 snap -1 2024-09-17T15:49:07.251 INFO:tasks.rados.rados.0.smithi203.stdout:3520: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:49:07.251 INFO:tasks.rados.rados.0.smithi203.stdout:3521: write (excl) oid 32 current snap is 373 2024-09-17T15:49:07.251 INFO:tasks.rados.rados.0.smithi203.stdout:3521: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-17T15:49:07.257 INFO:tasks.rados.rados.0.smithi203.stdout:3521: writing smithi20328086-32 from 440680 to 1114979 tid 1 2024-09-17T15:49:07.267 INFO:tasks.rados.rados.0.smithi203.stdout:3521: writing smithi20328086-32 from 1711907 to 2336331 tid 2 2024-09-17T15:49:07.267 INFO:tasks.rados.rados.0.smithi203.stdout:3522: rollback oid 5 current snap is 373 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 371 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3519: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 0 (ObjNum 817 snap 0 seq_num 2231607252) dirty dne 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3519: done (4 left) 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3523: copy_from oid 12 from oid 11 current snap is 373 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: finishing write tid 3 to smithi20328086-33 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: finishing write tid 4 to smithi20328086-33 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: finishing write tid 5 to smithi20328086-33 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: oid 33 updating version 0 to 1412 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: oid 33 updating version 1412 to 1413 2024-09-17T15:49:07.268 INFO:tasks.rados.rados.0.smithi203.stdout:3518: oid 33 updating version 1413 to 1414 2024-09-17T15:49:07.269 INFO:tasks.rados.rados.0.smithi203.stdout:3518: oid 33 updating version 1414 to 1415 2024-09-17T15:49:07.269 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1415 (ObjNum 1239 snap 373 seq_num 1239) dirty exists 2024-09-17T15:49:07.269 INFO:tasks.rados.rados.0.smithi203.stdout:3518: left oid 33 (ObjNum 1239 snap 373 seq_num 1239) 2024-09-17T15:49:07.269 INFO:tasks.rados.rados.0.smithi203.stdout:3520: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:49:07.269 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:07.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d613a4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:07.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d613a4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d613a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5d5c06d4f0 con 0x7f5d5c13aff0 2024-09-17T15:49:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d437fe640 1 -- 172.21.15.86:0/309092281 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f5d540165c0 con 0x7f5d5c13aff0 2024-09-17T15:49:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.418+0000 7f5d437fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:07.419 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 msgr2=0x7f5d4803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 0x7f5d4803e020 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c007fd0 tx=0x7f5d4c007e40 comp rx=0 tx=0).stop 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 10 monclient: shutdown 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 msgr2=0x7f5d5c070c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5d5401d150 tx=0x7f5d54019650 comp rx=0 tx=0).stop 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 shutdown_connections 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5d4803bb60 0x7f5d4803e020 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5d5c13aff0 0x7f5d5c070c30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5d5c0694c0 0x7f5d5c132db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 --2- 172.21.15.86:0/309092281 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5d5c0689d0 0x7f5d5c132870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 >> 172.21.15.86:0/309092281 conn(0x7f5d5c0a6440 msgr2=0x7f5d5c13c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 shutdown_connections 2024-09-17T15:49:07.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:07.422+0000 7f5d613a4640 1 -- 172.21.15.86:0/309092281 wait complete. 2024-09-17T15:49:07.430 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:3521: finishing write tid 1 to smithi20328086-32 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:3522: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1411 (ObjNum 33 snap 0 seq_num 3586005048) dirty dne 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:3521: finishing write tid 2 to smithi20328086-32 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:3521: finishing write tid 3 to smithi20328086-32 2024-09-17T15:49:07.490 INFO:tasks.rados.rados.0.smithi203.stdout:3521: finishing write tid 4 to smithi20328086-32 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3521: oid 32 updating version 0 to 1726 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3521: oid 32 version 1726 is already newer than 1724 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3521: oid 32 version 1726 is already newer than 1725 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1726 (ObjNum 1240 snap 373 seq_num 1240) dirty exists 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3521: left oid 32 (ObjNum 1240 snap 373 seq_num 1240) 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3523: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3523: finishing copy_from to smithi20328086-12 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1629 (ObjNum 1229 snap 371 seq_num 1229) dirty exists 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3518: done (4 left) 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3520: done (3 left) 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3521: done (2 left) 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3522: done (1 left) 2024-09-17T15:49:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:3523: done (0 left) 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3524: rollback oid 42 current snap is 373 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 371 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3525: rollback oid 18 current snap is 373 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 18 to 370 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3526: rollback oid 17 current snap is 373 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 17 to 363 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3527: read oid 30 snap -1 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3527: expect deleted 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3528: write oid 11 current snap is 373 2024-09-17T15:49:07.492 INFO:tasks.rados.rados.0.smithi203.stdout:3528: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-17T15:49:07.498 INFO:tasks.rados.rados.0.smithi203.stdout:3528: writing smithi20328086-11 from 457487 to 1207035 tid 1 2024-09-17T15:49:07.506 INFO:tasks.rados.rados.0.smithi203.stdout:3528: writing smithi20328086-11 from 1770389 to 2254824 tid 2 2024-09-17T15:49:07.511 INFO:tasks.rados.rados.0.smithi203.stdout:3528: writing smithi20328086-11 from 2746287 to 2857487 tid 3 2024-09-17T15:49:07.511 INFO:tasks.rados.rados.0.smithi203.stdout:3529: write oid 36 current snap is 373 2024-09-17T15:49:07.511 INFO:tasks.rados.rados.0.smithi203.stdout:3529: seq_num 1242 ranges {474294=424797,1428871=628093,2657357=216937} 2024-09-17T15:49:07.517 INFO:tasks.rados.rados.0.smithi203.stdout:3529: writing smithi20328086-36 from 474294 to 899091 tid 1 2024-09-17T15:49:07.525 INFO:tasks.rados.rados.0.smithi203.stdout:3529: writing smithi20328086-36 from 1428871 to 2056964 tid 2 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:3529: writing smithi20328086-36 from 2657357 to 2874294 tid 3 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:3530: rollback oid 49 current snap is 373 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 365 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:3526: finishing rollback tid 1 to smithi20328086-17 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 17 v 1631 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:3524: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:3525: finishing rollback tid 1 to smithi20328086-18 2024-09-17T15:49:07.533 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 0 (ObjNum 33 snap 0 seq_num 3605941056) dirty dne 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3524: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3524: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1436 (ObjNum 1230 snap 371 seq_num 1230) dirty exists 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: finishing write tid 1 to smithi20328086-11 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: finishing write tid 2 to smithi20328086-11 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: finishing write tid 3 to smithi20328086-11 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: finishing write tid 4 to smithi20328086-11 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: finishing write tid 5 to smithi20328086-11 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: oid 11 updating version 0 to 1636 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: oid 11 updating version 1636 to 1637 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: oid 11 updating version 1637 to 1639 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: oid 11 version 1639 is already newer than 1638 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1639 (ObjNum 1241 snap 373 seq_num 1241) dirty exists 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3528: left oid 11 (ObjNum 1241 snap 373 seq_num 1241) 2024-09-17T15:49:07.534 INFO:tasks.rados.rados.0.smithi203.stdout:3529: finishing write tid 1 to smithi20328086-36 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3524: done (6 left) 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3525: done (5 left) 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3526: done (4 left) 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3527: done (3 left) 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3528: done (2 left) 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3531: delete oid 17 current snap is 373 2024-09-17T15:49:07.535 INFO:tasks.rados.rados.0.smithi203.stdout:3529: finishing write tid 2 to smithi20328086-36 2024-09-17T15:49:07.536 INFO:tasks.rados.rados.0.smithi203.stdout:3531: done (2 left) 2024-09-17T15:49:07.536 INFO:tasks.rados.rados.0.smithi203.stdout:3532: snap_create 2024-09-17T15:49:07.541 INFO:tasks.rados.rados.0.smithi203.stdout:3529: finishing write tid 3 to smithi20328086-36 2024-09-17T15:49:07.547 INFO:tasks.rados.rados.0.smithi203.stdout:3529: finishing write tid 4 to smithi20328086-36 2024-09-17T15:49:07.548 INFO:tasks.rados.rados.0.smithi203.stdout:3529: finishing write tid 5 to smithi20328086-36 2024-09-17T15:49:07.548 INFO:tasks.rados.rados.0.smithi203.stdout:3529: oid 36 updating version 0 to 1820 2024-09-17T15:49:07.548 INFO:tasks.rados.rados.0.smithi203.stdout:3529: oid 36 updating version 1820 to 1822 2024-09-17T15:49:07.548 INFO:tasks.rados.rados.0.smithi203.stdout:3529: oid 36 version 1822 is already newer than 1821 2024-09-17T15:49:07.549 INFO:tasks.rados.rados.0.smithi203.stdout:3529: oid 36 updating version 1822 to 1823 2024-09-17T15:49:07.549 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1823 (ObjNum 1242 snap 373 seq_num 1242) dirty exists 2024-09-17T15:49:07.549 INFO:tasks.rados.rados.0.smithi203.stdout:3529: left oid 36 (ObjNum 1242 snap 373 seq_num 1242) 2024-09-17T15:49:07.549 INFO:tasks.rados.rados.0.smithi203.stdout:3530: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:49:07.550 INFO:tasks.rados.rados.0.smithi203.stdout:3530: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:49:07.551 INFO:tasks.rados.rados.0.smithi203.stdout:3530: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:49:07.551 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1827 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:49:10.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:10.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: get_monmap_and_config 2024-09-17T15:49:10.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: build_initial_monmap 2024-09-17T15:49:10.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: monmap: 2024-09-17T15:49:10.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:10.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:10.569671+0000 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:10.569671+0000 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:10.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 Processor -- start 2024-09-17T15:49:10.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 -- start start 2024-09-17T15:49:10.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: init 2024-09-17T15:49:10.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88780689d0 0x7f887806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): picked mon.noname-a con 0x7f88780689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88780691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): picked mon.noname-b con 0x7f8878068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 0x7f8878135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): picked mon.noname-c con 0x7f8878135ae0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8878074500 con 0x7f88780689d0 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f887809c8b0 con 0x7f8878135ae0 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f887809cb90 con 0x7f8878068db0 2024-09-17T15:49:10.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887e558640 10 monclient(hunting): authenticate will time out at 2283.535183s 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88780691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88780689d0 0x7f887806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88780689d0 0x7f887806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35098/0 (socket says 172.21.15.86:35098) 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 -- 172.21.15.86:0/3877938987 learned_addr learned my addr 172.21.15.86:0/3877938987 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): get_auth_request con 0x7f88780689d0 auth_method 0 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887caf5640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 msgr2=0x7f8878135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): get_auth_request con 0x7f8878068db0 auth_method 0 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f887caf5640 1 --2- 172.21.15.86:0/3877938987 >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 0x7f8878135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:10.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient(hunting): handle_auth_done global_id 234273 payload 274 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 -- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 msgr2=0x7f88780691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 --2- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88780691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 -- 172.21.15.86:0/3877938987 >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 msgr2=0x7f8878135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 --2- 172.21.15.86:0/3877938987 >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 0x7f8878135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 monclient: found mon.noname-a 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:10.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.566+0000 7f8877fff640 1 -- 172.21.15.86:0/3877938987 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887809ce70 con 0x7f88780689d0 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.570+0000 7f8877fff640 1 -- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 msgr2=0x7f887806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.570+0000 7f8877fff640 1 --2- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f887806f840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f887809cac0 tx=0x7f886001cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd90e7ecb294c545 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 1 -- 172.21.15.86:0/3877938987 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600b0070 con 0x7f88780689d0 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: dump: 2024-09-17T15:49:10.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 1 -- 172.21.15.86:0/3877938987 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88600ab600 con 0x7f88780689d0 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 5 monclient: authenticate success, global_id 234273 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 1 -- 172.21.15.86:0/3877938987 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600ab9d0 con 0x7f88780689d0 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 4 set_mon_vals no callback set 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:10.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8876ffd640 10 monclient: dump: 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: shutdown 2024-09-17T15:49:10.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 msgr2=0x7f887806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f887806f840 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f887809cac0 tx=0x7f886001cda0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- 172.21.15.86:0/3877938987 shutdown_connections 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- 172.21.15.86:0/3877938987 >> v2:172.21.15.203:3300/0 conn(0x7f8878135ae0 0x7f8878135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88780691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- 172.21.15.86:0/3877938987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f887806f840 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- 172.21.15.86:0/3877938987 >> 172.21.15.86:0/3877938987 conn(0x7f88780a6440 msgr2=0x7f88780a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- 172.21.15.86:0/3877938987 shutdown_connections 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- 172.21.15.86:0/3877938987 wait complete. 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: build_initial_monmap 2024-09-17T15:49:10.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: monmap: 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:10.581 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 Processor -- start 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- start start 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: init 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): picked mon.noname-a con 0x7f88780689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88781ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): picked mon.noname-c con 0x7f8878068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 0x7f88781cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): picked mon.noname-b con 0x7f8878135ae0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f887809c2d0 con 0x7f88780689d0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f887809be70 con 0x7f8878135ae0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f887813f820 con 0x7f8878068db0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887e558640 10 monclient(hunting): authenticate will time out at 2283.542364s 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88781ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35102/0 (socket says 172.21.15.86:35102) 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 -- 172.21.15.86:0/1492067583 learned_addr learned my addr 172.21.15.86:0/1492067583 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): get_auth_request con 0x7f88780689d0 auth_method 0 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:10.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): get_auth_request con 0x7f8878068db0 auth_method 0 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887caf5640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 msgr2=0x7f88781cd030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f887caf5640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 0x7f88781cd030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:10.584 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient(hunting): handle_auth_done global_id 234276 payload 824 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 msgr2=0x7f88781ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88781ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 msgr2=0x7f88781cd030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 0x7f88781cd030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 monclient: found mon.noname-a 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f8877fff640 1 -- 172.21.15.86:0/1492067583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88781cd570 con 0x7f88780689d0 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.574+0000 7f88777fe640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88781ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:10.585 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8877fff640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 msgr2=0x7f8878133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:10.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8877fff640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f88600ae040 tx=0x7f88600190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:10.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600bb070 con 0x7f88780689d0 2024-09-17T15:49:10.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:10.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:10.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: dump: 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: _finish_auth 0 2024-09-17T15:49:10.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 5 monclient: authenticate success, global_id 234276 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88600aa5b0 con 0x7f88780689d0 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _renew_subs 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887809f390 con 0x7f88780689d0 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88600b0040 con 0x7f88780689d0 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 10 monclient: dump: 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:10.589 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f88767fc640 4 set_mon_vals no callback set 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _renew_subs 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f887813bd80 con 0x7f88780689d0 2024-09-17T15:49:10.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88780691d0 con 0x7f88780689d0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f88600be070 con 0x7f88780689d0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 0x7f885803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(768..768 src has 696..768) ==== 4799+0+0 (secure 0 0 0) 0x7f88600e9640 con 0x7f88780689d0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f88777fe640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 0x7f885803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.578+0000 7f88777fe640 10 monclient: get_auth_request con 0x7f885803bb10 auth_method 0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.582+0000 7f88777fe640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 0x7f885803dfd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f88781d10d0 tx=0x7f886800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.582+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f8860016110 con 0x7f88780689d0 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.582+0000 7f8874ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.582+0000 7f8874ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:10.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f887e558640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:10.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f887e558640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:10.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f887e558640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:10.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f887806d4f0 con 0x7f88780689d0 2024-09-17T15:49:10.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f8874ff9640 1 -- 172.21.15.86:0/1492067583 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f88600b53d0 con 0x7f88780689d0 2024-09-17T15:49:10.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f8874ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:10.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.818+0000 7f8874ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:10.820 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:10.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 msgr2=0x7f885803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:10.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 0x7f885803dfd0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f88781d10d0 tx=0x7f886800b040 comp rx=0 tx=0).stop 2024-09-17T15:49:10.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 10 monclient: shutdown 2024-09-17T15:49:10.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 msgr2=0x7f8878133030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f88600ae040 tx=0x7f88600190e0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 shutdown_connections 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f885803bb10 0x7f885803dfd0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8878135ae0 0x7f88781cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8878068db0 0x7f88781ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 --2- 172.21.15.86:0/1492067583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88780689d0 0x7f8878133030 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 >> 172.21.15.86:0/1492067583 conn(0x7f88780a6440 msgr2=0x7f88780a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 shutdown_connections 2024-09-17T15:49:10.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:10.822+0000 7f887e558640 1 -- 172.21.15.86:0/1492067583 wait complete. 2024-09-17T15:49:13.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:13.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 10 monclient: get_monmap_and_config 2024-09-17T15:49:13.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 10 monclient: build_initial_monmap 2024-09-17T15:49:13.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 10 monclient: monmap: 2024-09-17T15:49:13.970 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:13.974647+0000 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:13.974647+0000 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:13.971 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 1 Processor -- start 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 1 -- start start 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.970+0000 7f98bca3e640 10 monclient: init 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 0x7f98b809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-c con 0x7f98b80a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98b80a5450 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98b813d280 0x7f98b813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-a con 0x7f98b813d280 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:13.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b80a6f40 con 0x7f98b80a5450 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f98b80a65e0 con 0x7f98b813d280 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f98b809c2a0 con 0x7f98b80a5070 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98bca3e640 10 monclient(hunting): authenticate will time out at 2286.940549s 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98b813d280 0x7f98b813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b659c640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 msgr2=0x7f98b809cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b5d9b640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f98b80a5450 legacy=0x7f98b80a5860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98b813d280 0x7f98b813f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35112/0 (socket says 172.21.15.86:35112) 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 -- 172.21.15.86:0/3261452766 learned_addr learned my addr 172.21.15.86:0/3261452766 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:13.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b659c640 1 --2- 172.21.15.86:0/3261452766 >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 0x7f98b809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): get_auth_request con 0x7f98b813d280 auth_method 0 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_done global_id 234279 payload 274 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 -- 172.21.15.86:0/3261452766 >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 msgr2=0x7f98b809cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:13.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 --2- 172.21.15.86:0/3261452766 >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 0x7f98b809cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 -- 172.21.15.86:0/3261452766 >> v1:172.21.15.203:6789/0 conn(0x7f98b80a5450 legacy=0x7f98b80a5860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 monclient: found mon.noname-a 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:13.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.974+0000 7f98b6d9d640 1 -- 172.21.15.86:0/3261452766 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b809c580 con 0x7f98b813d280 2024-09-17T15:49:13.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b6d9d640 1 -- 172.21.15.86:0/3261452766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b813d280 msgr2=0x7f98b813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:13.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b6d9d640 1 --2- 172.21.15.86:0/3261452766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b813d280 0x7f98b813f670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f98b806e200 tx=0x7f98b001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1d7ee3cc75ba73 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:13.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 1 -- 172.21.15.86:0/3261452766 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b00a5550 con 0x7f98b813d280 2024-09-17T15:49:13.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:13.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: dump: 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:13.980 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: _finish_auth 0 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 1 -- 172.21.15.86:0/3261452766 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98b00a56f0 con 0x7f98b813d280 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 5 monclient: authenticate success, global_id 234279 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 1 -- 172.21.15.86:0/3261452766 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98b00a5ac0 con 0x7f98b813d280 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 4 set_mon_vals no callback set 2024-09-17T15:49:13.981 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98b559a640 10 monclient: dump: 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:13.982 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 10 monclient: shutdown 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 -- 172.21.15.86:0/3261452766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b813d280 msgr2=0x7f98b813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 --2- 172.21.15.86:0/3261452766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b813d280 0x7f98b813f670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f98b806e200 tx=0x7f98b001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 -- 172.21.15.86:0/3261452766 shutdown_connections 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 --2- 172.21.15.86:0/3261452766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b813d280 0x7f98b813f670 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 --2- 172.21.15.86:0/3261452766 >> v2:172.21.15.203:3300/0 conn(0x7f98b80a5070 0x7f98b809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 -- 172.21.15.86:0/3261452766 >> 172.21.15.86:0/3261452766 conn(0x7f98b80742a0 msgr2=0x7f98b80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.978+0000 7f98bca3e640 1 -- 172.21.15.86:0/3261452766 shutdown_connections 2024-09-17T15:49:13.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 -- 172.21.15.86:0/3261452766 wait complete. 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient: build_initial_monmap 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient: monmap: 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:13.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 Processor -- start 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 -- start start 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient: init 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-a con 0x7f98b80a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 0x7f98b8137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-b con 0x7f98b80a5450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b813d280 0x7f98b813b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): picked mon.noname-c con 0x7f98b813d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8142760 con 0x7f98b80a5070 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81425e0 con 0x7f98b80a5450 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f98b81428e0 con 0x7f98b813d280 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98bca3e640 10 monclient(hunting): authenticate will time out at 2286.949968s 2024-09-17T15:49:13.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b5d9b640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 msgr2=0x7f98b8137fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b5d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 0x7f98b8137fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b813d280 0x7f98b813b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35114/0 (socket says 172.21.15.86:35114) 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 -- 172.21.15.86:0/21578883 learned_addr learned my addr 172.21.15.86:0/21578883 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): get_auth_request con 0x7f98b80a5070 auth_method 0 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): get_auth_request con 0x7f98b813d280 auth_method 0 2024-09-17T15:49:13.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient(hunting): handle_auth_done global_id 234282 payload 824 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 -- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b813d280 msgr2=0x7f98b813b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b813d280 0x7f98b813b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 -- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 msgr2=0x7f98b8137fb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 0x7f98b8137fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 monclient: found mon.noname-a 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.982+0000 7f98b659c640 1 -- 172.21.15.86:0/21578883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b8122660 con 0x7f98b80a5070 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98b659c640 1 -- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 msgr2=0x7f98b8121140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98b659c640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98a800b560 tx=0x7f98a800ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8013020 con 0x7f98b80a5070 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:13.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: dump: 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a8005070 con 0x7f98b80a5070 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 5 monclient: authenticate success, global_id 234282 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _renew_subs 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b80a1860 con 0x7f98b80a5070 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8005460 con 0x7f98b80a5070 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:13.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 10 monclient: dump: 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98b4d99640 4 set_mon_vals no callback set 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _renew_subs 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b8122940 con 0x7f98b80a5070 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b809cfe0 con 0x7f98b80a5070 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f98a8016020 con 0x7f98b80a5070 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 0x7f989803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(768..768 src has 696..768) ==== 4799+0+0 (secure 0 0 0) 0x7f98a804fcf0 con 0x7f98b80a5070 2024-09-17T15:49:13.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98b5d9b640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 0x7f989803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:13.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.986+0000 7f98b5d9b640 10 monclient: get_auth_request con 0x7f989803bb10 auth_method 0 2024-09-17T15:49:13.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.990+0000 7f98b5d9b640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 0x7f989803dfd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f98ac006fd0 tx=0x7f98ac006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:13.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.990+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f98a801d020 con 0x7f98b80a5070 2024-09-17T15:49:13.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.990+0000 7f98a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:13.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:13.990+0000 7f98a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98bca3e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98bca3e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98bca3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f98b813bde0 con 0x7f98b80a5070 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98a6ffd640 1 -- 172.21.15.86:0/21578883 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f98a800fdc0 con 0x7f98b80a5070 2024-09-17T15:49:14.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:14.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.222+0000 7f98a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:14.222 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":200,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 msgr2=0x7f989803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 0x7f989803dfd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f98ac006fd0 tx=0x7f98ac006d40 comp rx=0 tx=0).stop 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 10 monclient: shutdown 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 msgr2=0x7f98b8121140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:14.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98a800b560 tx=0x7f98a800ba20 comp rx=0 tx=0).stop 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 shutdown_connections 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f989803bb10 0x7f989803dfd0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b813d280 0x7f98b813b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98b80a5450 0x7f98b8137fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 --2- 172.21.15.86:0/21578883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b80a5070 0x7f98b8121140 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 >> 172.21.15.86:0/21578883 conn(0x7f98b80742a0 msgr2=0x7f98b80a05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 shutdown_connections 2024-09-17T15:49:14.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:14.226+0000 7f98bca3e640 1 -- 172.21.15.86:0/21578883 wait complete. 2024-09-17T15:49:17.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:17.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: get_monmap_and_config 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: build_initial_monmap 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: monmap: 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:17.370078+0000 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:17.370078+0000 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:17.366 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:17.367 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:17.367 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:17.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 Processor -- start 2024-09-17T15:49:17.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 -- start start 2024-09-17T15:49:17.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: init 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1800a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1800a5b60 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-b con 0x7fa18009cf60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1800a6f40 con 0x7fa1800a5070 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1800a65e0 con 0x7fa1800a5b60 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa18009d390 con 0x7fa18009cf60 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa1882a5640 10 monclient(hunting): authenticate will time out at 2290.335845s 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.366+0000 7fa185840640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fa1800a5b60 legacy=0x7fa18013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186041640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa1800a5070 0x7fa1800a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38828/0 (socket says 172.21.15.86:38828) 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 learned_addr learned my addr 172.21.15.86:0/2045220766 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa18009cf60 auth_method 0 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:17.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:17.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.370+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.86:6789/0 conn(0x7fa1800a5070 legacy=0x7fa1800a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:17.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.203:6789/0 conn(0x7fa1800a5b60 legacy=0x7fa18013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:17.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:17.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa18009cf60 auth_method 0 2024-09-17T15:49:17.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:17.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.574+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa18009cf60 auth_method 0 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:17.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:17.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:17.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:49:17.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:17.978+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.203:6789/0 conn(0x7fa1800a5b60 legacy=0x7fa18013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:18.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.203:6789/0 conn(0x7fa1800a5b60 legacy=0x7fa18013d280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa18009cf60 auth_method 0 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:18.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:18.782+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:49:20.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): tick 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 msgr2=0x7fa1801409b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa18009cf60 0x7fa1801409b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.203:6789/0 conn(0x7fa1800a5b60 legacy=0x7fa18013d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.86:6789/0 conn(0x7fa1800a5070 legacy=0x7fa1800a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient: _add_conns ranks=[3,4] 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:20.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa16c003760 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa16c0062d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa18009d390 con 0x7fa16c003760 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c008e50 con 0x7fa16c0062d0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa17ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 msgr2=0x7fa16c008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa185840640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 10 monclient(hunting): get_auth_request con 0x7fa16c003760 auth_method 0 2024-09-17T15:49:20.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:20.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.370+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:20.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 10 monclient(hunting): get_auth_request con 0x7fa16c003760 auth_method 0 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 msgr2=0x7fa16c008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:20.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.574+0000 7fa185840640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:49:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 msgr2=0x7fa16c008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:20.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa185840640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 10 monclient(hunting): get_auth_request con 0x7fa16c003760 auth_method 0 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:20.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:20.978+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 msgr2=0x7fa16c008790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa185840640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 10 monclient(hunting): get_auth_request con 0x7fa16c003760 auth_method 0 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:49:21.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:49:21.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:21.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:21.782+0000 7fa186041640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:49:22.981 INFO:tasks.rados.rados.0.smithi203.stdout:3529: done (2 left) 2024-09-17T15:49:22.981 INFO:tasks.rados.rados.0.smithi203.stdout:3530: done (1 left) 2024-09-17T15:49:22.982 INFO:tasks.rados.rados.0.smithi203.stdout:3532: done (0 left) 2024-09-17T15:49:22.982 INFO:tasks.rados.rados.0.smithi203.stdout:3533: rollback oid 1 current snap is 374 2024-09-17T15:49:22.982 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 365 2024-09-17T15:49:22.982 INFO:tasks.rados.rados.0.smithi203.stdout:3534: write (excl) oid 25 current snap is 374 2024-09-17T15:49:22.982 INFO:tasks.rados.rados.0.smithi203.stdout:3534: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-17T15:49:22.991 INFO:tasks.rados.rados.0.smithi203.stdout:3534: writing smithi20328086-25 from 507908 to 1083203 tid 1 2024-09-17T15:49:23.004 INFO:tasks.rados.rados.0.smithi203.stdout:3534: writing smithi20328086-25 from 1829482 to 2461244 tid 2 2024-09-17T15:49:23.004 INFO:tasks.rados.rados.0.smithi203.stdout:3535: snap_remove snap 371 2024-09-17T15:49:23.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): tick 2024-09-17T15:49:23.369 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 monclient(hunting): continuing hunt 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 msgr2=0x7fa16c008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c0062d0 0x7fa16c008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 msgr2=0x7fa16c005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c003760 0x7fa16c005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 reap_dead start 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1800a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 0x7fa16c006780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fa16c003bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:23.370 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c006cc0 0x7fa16c007170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fa16c006cc0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa18009c2a0 con 0x7fa1800a5070 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa18009d390 con 0x7fa16c006cc0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 1 -- 172.21.15.86:0/2045220766 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c008e50 con 0x7fa16c003bd0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa17ffff640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa185840640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 msgr2=0x7fa16c006780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa185840640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 0x7fa16c006780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.86:3300/0 conn(0x7fa16c006cc0 0x7fa16c007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa16c006cc0 auth_method 0 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.371 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient(hunting): handle_auth_done global_id 234285 payload 274 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 msgr2=0x7fa16c006780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 0x7fa16c006780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> v1:172.21.15.86:6789/0 conn(0x7fa1800a5070 legacy=0x7fa16c00b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 monclient: found mon.noname-a 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.369+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1800a65e0 con 0x7fa16c006cc0 2024-09-17T15:49:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa186842640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa16c006cc0 msgr2=0x7fa16c007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa186842640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa16c006cc0 0x7fa16c007170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa18006fd30 tx=0x7fa17001d1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bd99e72290afe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 1 -- 172.21.15.86:0/2045220766 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700b3070 con 0x7fa16c006cc0 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: dump: 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 1 -- 172.21.15.86:0/2045220766 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1700ae410 con 0x7fa16c006cc0 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 5 monclient: authenticate success, global_id 234285 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 1 -- 172.21.15.86:0/2045220766 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700ae600 con 0x7fa16c006cc0 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 4 set_mon_vals no callback set 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa18503f640 10 monclient: dump: 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.380 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 10 monclient: shutdown 2024-09-17T15:49:23.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 -- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa16c006cc0 msgr2=0x7fa16c007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa16c006cc0 0x7fa16c007170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa18006fd30 tx=0x7fa17001d1d0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 -- 172.21.15.86:0/2045220766 shutdown_connections 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 --2- 172.21.15.86:0/2045220766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa16c006cc0 0x7fa16c007170 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 --2- 172.21.15.86:0/2045220766 >> v2:172.21.15.203:3300/0 conn(0x7fa16c003bd0 0x7fa16c006780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 -- 172.21.15.86:0/2045220766 >> 172.21.15.86:0/2045220766 conn(0x7fa1800742a0 msgr2=0x7fa1800746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 -- 172.21.15.86:0/2045220766 shutdown_connections 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.377+0000 7fa1882a5640 1 -- 172.21.15.86:0/2045220766 wait complete. 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient: build_initial_monmap 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient: monmap: 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:23.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 Processor -- start 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 -- start start 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient: init 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:49:23.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-c con 0x7fa16c003bd0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 0x7fa1801d0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-b con 0x7fa16c006cc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): picked mon.noname-a con 0x7fa1800a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800a69e0 con 0x7fa1800a5070 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa1800a6860 con 0x7fa16c006cc0 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa180138640 con 0x7fa16c003bd0 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa1882a5640 10 monclient(hunting): authenticate will time out at 2296.350164s 2024-09-17T15:49:23.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa185840640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 msgr2=0x7fa1801d0970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa185840640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 0x7fa1801d0970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40984/0 (socket says 172.21.15.86:40984) 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57992/0 (socket says 172.21.15.86:57992) 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 1 -- 172.21.15.86:0/3407168983 learned_addr learned my addr 172.21.15.86:0/3407168983 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): get_auth_request con 0x7fa16c003bd0 auth_method 0 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): get_auth_request con 0x7fa1800a5070 auth_method 0 2024-09-17T15:49:23.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient(hunting): handle_auth_done global_id 234288 payload 824 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 -- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 msgr2=0x7fa1801d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 -- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 msgr2=0x7fa1801d0970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:23.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 0x7fa1801d0970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 monclient: found mon.noname-a 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186842640 1 -- 172.21.15.86:0/3407168983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa180129480 con 0x7fa1800a5070 2024-09-17T15:49:23.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.381+0000 7fa186041640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa186842640 1 -- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 msgr2=0x7fa180125660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa186842640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa1700b0000 tx=0x7fa17001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700bf070 con 0x7fa1800a5070 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: dump: 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1700b4040 con 0x7fa1800a5070 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 5 monclient: authenticate success, global_id 234288 2024-09-17T15:49:23.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _renew_subs 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1800a1860 con 0x7fa1800a5070 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700b3050 con 0x7fa1800a5070 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 10 monclient: dump: 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.393 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa18483e640 4 set_mon_vals no callback set 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _renew_subs 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1801291a0 con 0x7fa1800a5070 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa1700c2070 con 0x7fa1800a5070 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1801389f0 con 0x7fa1800a5070 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 0x7fa16c03f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(769..769 src has 696..769) ==== 4771+0+0 (secure 0 0 0) 0x7fa1700eaca0 con 0x7fa1800a5070 2024-09-17T15:49:23.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa186041640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 0x7fa16c03f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.385+0000 7fa186041640 10 monclient: get_auth_request con 0x7fa16c03d230 auth_method 0 2024-09-17T15:49:23.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.389+0000 7fa186041640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 0x7fa16c03f6f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa178007be0 tx=0x7fa178009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.393+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa170016f80 con 0x7fa1800a5070 2024-09-17T15:49:23.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.393+0000 7fa17effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.393+0000 7fa17effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:23.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:23.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:23.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.620 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa180128be0 con 0x7fa1800a5070 2024-09-17T15:49:23.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa17effd640 1 -- 172.21.15.86:0/3407168983 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa1700b8df0 con 0x7fa1800a5070 2024-09-17T15:49:23.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa17effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:23.621 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa17effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:23.621 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":204,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:23.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 msgr2=0x7fa16c03f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.622 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 0x7fa16c03f6f0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa178007be0 tx=0x7fa178009040 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 10 monclient: shutdown 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 msgr2=0x7fa180125660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa1700b0000 tx=0x7fa17001ba10 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 shutdown_connections 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa16c03d230 0x7fa16c03f6f0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1800a5070 0x7fa180125660 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa16c006cc0 0x7fa1801d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 --2- 172.21.15.86:0/3407168983 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa16c003bd0 0x7fa1801d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 >> 172.21.15.86:0/3407168983 conn(0x7fa1800742a0 msgr2=0x7fa1800a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 shutdown_connections 2024-09-17T15:49:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.621+0000 7fa1882a5640 1 -- 172.21.15.86:0/3407168983 wait complete. 2024-09-17T15:49:23.638 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:49:23.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:49:23.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: get_monmap_and_config 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: build_initial_monmap 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: monmap: 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:23.761554+0000 2024-09-17T15:49:23.757 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:23.761554+0000 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:23.758 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 Processor -- start 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 -- start start 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: init 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0db0134d90 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 0x7f0db013cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-c con 0x7f0db00a1130 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0db013d4a0 0x7f0db013f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-a con 0x7f0db013d4a0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0db0135880 con 0x7f0db0134d90 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0db00a21b0 con 0x7f0db013d4a0 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0db00a1540 con 0x7f0db00a1130 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db65d7640 10 monclient(hunting): authenticate will time out at 2296.727099s 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0db013d4a0 0x7f0db013f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0daf7fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 msgr2=0x7f0db013cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0daf7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 0x7f0db013cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0db013d4a0 0x7f0db013f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58006/0 (socket says 172.21.15.86:58006) 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 -- 172.21.15.86:0/2601944219 learned_addr learned my addr 172.21.15.86:0/2601944219 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:23.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): get_auth_request con 0x7f0db013d4a0 auth_method 0 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0daffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0db0134d90 0x7f0db0135170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53418/0 (socket says 172.21.15.86:53418) 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient(hunting): handle_auth_done global_id 234291 payload 274 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 -- 172.21.15.86:0/2601944219 >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 msgr2=0x7f0db013cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 --2- 172.21.15.86:0/2601944219 >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 0x7f0db013cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 -- 172.21.15.86:0/2601944219 >> v1:172.21.15.86:6789/0 conn(0x7f0db0134d90 legacy=0x7f0db0135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 monclient: found mon.noname-a 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.757+0000 7f0db4b74640 1 -- 172.21.15.86:0/2601944219 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db0073b10 con 0x7f0db013d4a0 2024-09-17T15:49:23.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.761+0000 7f0db4b74640 1 -- 172.21.15.86:0/2601944219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013d4a0 msgr2=0x7f0db013f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.761+0000 7f0db4b74640 1 --2- 172.21.15.86:0/2601944219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013d4a0 0x7f0db013f890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009e50 tx=0x7f0da801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e5baec2d117431 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.761+0000 7f0daeffd640 1 -- 172.21.15.86:0/2601944219 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80a5720 con 0x7f0db013d4a0 2024-09-17T15:49:23.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.761+0000 7f0daeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: dump: 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.768 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 1 -- 172.21.15.86:0/2601944219 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da80a58c0 con 0x7f0db013d4a0 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 5 monclient: authenticate success, global_id 234291 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 1 -- 172.21.15.86:0/2601944219 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80a5c90 con 0x7f0db013d4a0 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 4 set_mon_vals no callback set 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daeffd640 10 monclient: dump: 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: shutdown 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- 172.21.15.86:0/2601944219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013d4a0 msgr2=0x7f0db013f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- 172.21.15.86:0/2601944219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013d4a0 0x7f0db013f890 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0da8009e50 tx=0x7f0da801c8a0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- 172.21.15.86:0/2601944219 shutdown_connections 2024-09-17T15:49:23.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- 172.21.15.86:0/2601944219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013d4a0 0x7f0db013f890 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- 172.21.15.86:0/2601944219 >> v2:172.21.15.203:3300/0 conn(0x7f0db00a1130 0x7f0db013cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- 172.21.15.86:0/2601944219 >> 172.21.15.86:0/2601944219 conn(0x7f0db00a2f40 msgr2=0x7f0db00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- 172.21.15.86:0/2601944219 shutdown_connections 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- 172.21.15.86:0/2601944219 wait complete. 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: build_initial_monmap 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: monmap: 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.771 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 Processor -- start 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- start start 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: init 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 0x7f0db01cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-a con 0x7f0db00a1130 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0db0134d90 0x7f0db0133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-b con 0x7f0db0134d90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:23.772 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 0x7f0db013aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): picked mon.noname-c con 0x7f0db013d4a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0db0142770 con 0x7f0db00a1130 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0db01425f0 con 0x7f0db0134d90 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0db01428f0 con 0x7f0db013d4a0 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db65d7640 10 monclient(hunting): authenticate will time out at 2296.734384s 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 0x7f0db01cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daf7fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0db0134d90 msgr2=0x7f0db0133bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daf7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0db0134d90 0x7f0db0133bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.773 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 0x7f0db01cfa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58010/0 (socket says 172.21.15.86:58010) 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 1 -- 172.21.15.86:0/3525656716 learned_addr learned my addr 172.21.15.86:0/3525656716 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): get_auth_request con 0x7f0db00a1130 auth_method 0 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 0x7f0db013aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): get_auth_request con 0x7f0db013d4a0 auth_method 0 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.774 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient(hunting): handle_auth_done global_id 164957 payload 824 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0db0134d90 msgr2=0x7f0db0133bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0db0134d90 0x7f0db0133bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 msgr2=0x7f0db01cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 0x7f0db01cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 monclient: found mon.noname-c 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0db4b74640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db013d080 con 0x7f0db013d4a0 2024-09-17T15:49:23.775 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.765+0000 7f0daffff640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db00a1130 0x7f0db01cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:23.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db4b74640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 msgr2=0x7f0db013aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db4b74640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 0x7f0db013aaf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0db006eca0 tx=0x7f0da8019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.779 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80bc030 con 0x7f0db013d4a0 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: dump: 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da80b8680 con 0x7f0db013d4a0 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da80a5720 con 0x7f0db013d4a0 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 10 monclient: dump: 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.781 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dae7fc640 4 set_mon_vals no callback set 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 5 monclient: authenticate success, global_id 164957 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 monclient: _renew_subs 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db013ccc0 con 0x7f0db013d4a0 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 monclient: _renew_subs 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:23.782 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db011e260 con 0x7f0db013d4a0 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 0x7f0db013b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0db0135170 con 0x7f0db01340f0 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 0x7f0db013b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient: get_auth_request con 0x7f0db01340f0 auth_method 0 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0da80bf070 con 0x7f0db013d4a0 2024-09-17T15:49:23.783 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 10 monclient(hunting): handle_auth_done global_id 234297 payload 824 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daffff640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0db00008e0 con 0x7f0db01340f0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d9003bb10 0x7f0d9003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(769..769 src has 696..769) ==== 4771+0+0 (secure 0 0 0) 0x7f0da80ead40 con 0x7f0db013d4a0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daf7fe640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d9003bb10 0x7f0d9003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.769+0000 7f0daf7fe640 10 monclient: get_auth_request con 0x7f0d9003bb10 auth_method 0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0daffff640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 msgr2=0x7f0db013b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0daffff640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 0x7f0db013b030 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0da00072a0 tx=0x7f0da00076a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da000fde0 con 0x7f0db01340f0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 10 monclient: dump: 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.784 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0da000fde0 con 0x7f0db01340f0 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 msgr2=0x7f0db013b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 0x7f0db013b030 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0da00072a0 tx=0x7f0da00076a0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.785 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 msgr2=0x7f0db013b030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0dacff9640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db01340f0 0x7f0db013b030 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0da00072a0 tx=0x7f0da00076a0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.773+0000 7f0daf7fe640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d9003bb10 0x7f0d9003dfd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0db013c930 tx=0x7f0d98006d90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0db65d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0db65d7640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 0x7f0db0135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0db65d7640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0db013b980 con 0x7f0db013b570 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 0x7f0db0135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient: get_auth_request con 0x7f0db013b570 auth_method 0 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.786 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 10 monclient(hunting): handle_auth_done global_id 234300 payload 824 2024-09-17T15:49:23.787 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.777+0000 7f0daffff640 1 -- 172.21.15.86:0/3525656716 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0db011e4a0 con 0x7f0db013b570 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0daffff640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 msgr2=0x7f0db0135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0daffff640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 0x7f0db0135a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0da0017000 tx=0x7f0da0004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da0006030 con 0x7f0db013b570 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 10 monclient: dump: 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.788 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f0da0006030 con 0x7f0db013b570 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 msgr2=0x7f0db0135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 0x7f0db0135a50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0da0017000 tx=0x7f0da0004940 comp rx=0 tx=0).stop 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 msgr2=0x7f0db0135a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0dacff9640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0db013b570 0x7f0db0135a50 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0da0017000 tx=0x7f0da0004940 comp rx=0 tx=0).stop 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 204, 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 92977, 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:23.790 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.791 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:23.793 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.794 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.796 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:23.799 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:23.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.801 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:23.802 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d9003bb10 msgr2=0x7f0d9003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0d9003bb10 0x7f0d9003dfd0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0db013c930 tx=0x7f0d98006d90 comp rx=0 tx=0).stop 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 10 monclient: shutdown 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 msgr2=0x7f0db013aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db4b74640 1 -- 172.21.15.86:0/3525656716 reap_dead start 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 0x7f0db013aaf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0db006eca0 tx=0x7f0da8019640 comp rx=0 tx=0).stop 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 shutdown_connections 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 --2- 172.21.15.86:0/3525656716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0db013d4a0 0x7f0db013aaf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 >> 172.21.15.86:0/3525656716 conn(0x7f0db00a2f40 msgr2=0x7f0db00a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 shutdown_connections 2024-09-17T15:49:23.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.781+0000 7f0db65d7640 1 -- 172.21.15.86:0/3525656716 wait complete. 2024-09-17T15:49:23.805 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:49:23.805 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:49:23.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: get_monmap_and_config 2024-09-17T15:49:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: build_initial_monmap 2024-09-17T15:49:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: monmap: 2024-09-17T15:49:23.920 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:23.924098+0000 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:23.924098+0000 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 Processor -- start 2024-09-17T15:49:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 -- start start 2024-09-17T15:49:23.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: init 2024-09-17T15:49:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-17T15:49:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f100a50e0 0x7f8f1009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-a con 0x7f8f100a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f100a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 0x7f8f1013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-c con 0x7f8f1013d2f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f100a6280 con 0x7f8f100a54c0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1009d2a0 con 0x7f8f100a50e0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1009c0a0 con 0x7f8f1013d2f0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f14def640 10 monclient(hunting): authenticate will time out at 2296.889529s 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f100a50e0 0x7f8f1009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f100a50e0 0x7f8f1009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58046/0 (socket says 172.21.15.86:58046) 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0ed9d640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 msgr2=0x7f8f1013f6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0dd9b640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f8f100a54c0 legacy=0x7f8f100a58d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 -- 172.21.15.86:0/952086310 learned_addr learned my addr 172.21.15.86:0/952086310 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request con 0x7f8f100a50e0 auth_method 0 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0ed9d640 1 --2- 172.21.15.86:0/952086310 >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 0x7f8f1013f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 234303 payload 274 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 -- 172.21.15.86:0/952086310 >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 msgr2=0x7f8f1013f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/952086310 >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 0x7f8f1013f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 -- 172.21.15.86:0/952086310 >> v1:172.21.15.203:6789/0 conn(0x7f8f100a54c0 legacy=0x7f8f100a58d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 monclient: found mon.noname-a 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.921+0000 7f8f0e59c640 1 -- 172.21.15.86:0/952086310 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1009c380 con 0x7f8f100a50e0 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0e59c640 1 -- 172.21.15.86:0/952086310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f100a50e0 msgr2=0x7f8f1009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/952086310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f100a50e0 0x7f8f1009cd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a84e0 tx=0x7f8f0801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22aacec4ed2ef928 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 1 -- 172.21.15.86:0/952086310 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f080a5640 con 0x7f8f100a50e0 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: dump: 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.930 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 1 -- 172.21.15.86:0/952086310 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f080a57e0 con 0x7f8f100a50e0 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 1 -- 172.21.15.86:0/952086310 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f080a5bb0 con 0x7f8f100a50e0 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 5 monclient: authenticate success, global_id 234303 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 4 set_mon_vals no callback set 2024-09-17T15:49:23.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f0d59a640 10 monclient: dump: 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 10 monclient: shutdown 2024-09-17T15:49:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 -- 172.21.15.86:0/952086310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f100a50e0 msgr2=0x7f8f1009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 --2- 172.21.15.86:0/952086310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f100a50e0 0x7f8f1009cd60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a84e0 tx=0x7f8f0801c8a0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 -- 172.21.15.86:0/952086310 shutdown_connections 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 --2- 172.21.15.86:0/952086310 >> v2:172.21.15.203:3300/0 conn(0x7f8f1013d2f0 0x7f8f1013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 --2- 172.21.15.86:0/952086310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f100a50e0 0x7f8f1009cd60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 -- 172.21.15.86:0/952086310 >> 172.21.15.86:0/952086310 conn(0x7f8f100aaae0 msgr2=0x7f8f100aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 -- 172.21.15.86:0/952086310 shutdown_connections 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.925+0000 7f8f14def640 1 -- 172.21.15.86:0/952086310 wait complete. 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: build_initial_monmap 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: monmap: 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 Processor -- start 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 -- start start 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: init 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:49:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 0x7f8f100a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-c con 0x7f8f100a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f100a54c0 0x7f8f100a2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-b con 0x7f8f100a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 0x7f8f100a2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): picked mon.noname-a con 0x7f8f1013d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f8f10142650 con 0x7f8f1013d2f0 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f100a6ad0 con 0x7f8f100a54c0 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8f101427d0 con 0x7f8f100a50e0 2024-09-17T15:49:23.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient(hunting): authenticate will time out at 2296.897007s 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 0x7f8f100a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 0x7f8f100a2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 0x7f8f100a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:40996/0 (socket says 172.21.15.86:40996) 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 1 -- 172.21.15.86:0/2726069365 learned_addr learned my addr 172.21.15.86:0/2726069365 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request con 0x7f8f100a50e0 auth_method 0 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0dd9b640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f100a54c0 msgr2=0x7f8f100a2650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request con 0x7f8f1013d2f0 auth_method 0 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0dd9b640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f100a54c0 0x7f8f100a2650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient(hunting): handle_auth_done global_id 234306 payload 824 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 msgr2=0x7f8f100a2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 0x7f8f100a2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f100a54c0 msgr2=0x7f8f100a2650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f100a54c0 0x7f8f100a2650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:23.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient: _finish_hunting 0 2024-09-17T15:49:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 monclient: found mon.noname-a 2024-09-17T15:49:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f101394e0 con 0x7f8f1013d2f0 2024-09-17T15:49:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f100a50e0 0x7f8f100a2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 msgr2=0x7f8f100a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0ed9d640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 0x7f8f100a2b90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f00003940 tx=0x7f8f0000b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00016070 con 0x7f8f1013d2f0 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: dump: 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.941 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f00005070 con 0x7f8f1013d2f0 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 5 monclient: authenticate success, global_id 234306 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: _renew_subs 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1013a5f0 con 0x7f8f1013d2f0 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00005460 con 0x7f8f1013d2f0 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f06ffd640 10 monclient: dump: 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.943 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 10 monclient: _renew_subs 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f100a18d0 con 0x7f8f1013d2f0 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.929+0000 7f8f0cd99640 4 set_mon_vals no callback set 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 0x7f8f101d01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1009cd60 con 0x7f8f101cfdc0 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 0x7f8f101d01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient: get_auth_request con 0x7f8f101cfdc0 auth_method 0 2024-09-17T15:49:23.944 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8f00019070 con 0x7f8f1013d2f0 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 164963 payload 824 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f100008e0 con 0x7f8f101cfdc0 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ef403bb10 0x7f8ef403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(769..769 src has 696..769) ==== 4771+0+0 (secure 0 0 0) 0x7f8f0004f1b0 con 0x7f8f1013d2f0 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0dd9b640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ef403bb10 0x7f8ef403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0dd9b640 10 monclient: get_auth_request con 0x7f8ef403bb10 auth_method 0 2024-09-17T15:49:23.945 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 msgr2=0x7f8f101d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 0x7f8f101d01a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f0801af50 tx=0x7f8f08019640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f080add20 con 0x7f8f101cfdc0 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.933+0000 7f8f06ffd640 10 monclient: dump: 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.946 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8f080add20 con 0x7f8f101cfdc0 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 msgr2=0x7f8f101d01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 0x7f8f101d01a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f0801af50 tx=0x7f8f08019640 comp rx=0 tx=0).stop 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 msgr2=0x7f8f101d01a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f06ffd640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101cfdc0 0x7f8f101d01a0 secure :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8f0801af50 tx=0x7f8f08019640 comp rx=0 tx=0).stop 2024-09-17T15:49:23.947 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.937+0000 7f8f0dd9b640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ef403bb10 0x7f8ef403dfd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8006fd0 tx=0x7f8ef8006d40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f14def640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f14def640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 0x7f8f1009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f14def640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1013a270 con 0x7f8f101d06e0 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 0x7f8f1009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient: get_auth_request con 0x7f8f101d06e0 auth_method 0 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 10 monclient(hunting): handle_auth_done global_id 164966 payload 824 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.941+0000 7f8f0e59c640 1 -- 172.21.15.86:0/2726069365 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f1013a420 con 0x7f8f101d06e0 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f0e59c640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 msgr2=0x7f8f1009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f0e59c640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 0x7f8f1009e5e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a4be0 tx=0x7f8f080a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08004450 con 0x7f8f101d06e0 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 10 monclient: dump: 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:23.950 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f8f08003e60 con 0x7f8f101d06e0 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 msgr2=0x7f8f1009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 0x7f8f1009e5e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a4be0 tx=0x7f8f080a7000 comp rx=0 tx=0).stop 2024-09-17T15:49:23.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 msgr2=0x7f8f1009e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f06ffd640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f101d06e0 0x7f8f1009e5e0 secure :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8f100a4be0 tx=0x7f8f080a7000 comp rx=0 tx=0).stop 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 204, 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 467133, 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:23.952 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.954 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.955 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:23.956 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:23.957 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.958 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:23.959 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:23.962 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:23.963 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:49:23.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:23.965 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ef403bb10 msgr2=0x7f8ef403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f8ef403bb10 0x7f8ef403dfd0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8006fd0 tx=0x7f8ef8006d40 comp rx=0 tx=0).stop 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 10 monclient: shutdown 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 msgr2=0x7f8f100a2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 --2- 172.21.15.86:0/2726069365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1013d2f0 0x7f8f100a2b90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f00003940 tx=0x7f8f0000b820 comp rx=0 tx=0).stop 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 reap_dead start 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f0ed9d640 1 -- 172.21.15.86:0/2726069365 reap_dead start 2024-09-17T15:49:23.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 shutdown_connections 2024-09-17T15:49:23.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 >> 172.21.15.86:0/2726069365 conn(0x7f8f100aaae0 msgr2=0x7f8f1013f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:23.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 shutdown_connections 2024-09-17T15:49:23.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:23.945+0000 7f8f14def640 1 -- 172.21.15.86:0/2726069365 wait complete. 2024-09-17T15:49:23.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:49:23.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:49:23.998 INFO:tasks.rados.rados.0.smithi203.stdout:3535: done (2 left) 2024-09-17T15:49:23.998 INFO:tasks.rados.rados.0.smithi203.stdout:3536: read oid 40 snap -1 2024-09-17T15:49:23.998 INFO:tasks.rados.rados.0.smithi203.stdout:3536: expect deleted 2024-09-17T15:49:23.998 INFO:tasks.rados.rados.0.smithi203.stdout:3537: write (excl) oid 26 current snap is 374 2024-09-17T15:49:23.998 INFO:tasks.rados.rados.0.smithi203.stdout:3533: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:3533: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:3533: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1649 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:3534: finishing write tid 1 to smithi20328086-25 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:3534: finishing write tid 2 to smithi20328086-25 2024-09-17T15:49:23.999 INFO:tasks.rados.rados.0.smithi203.stdout:3537: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-09-17T15:49:24.015 INFO:tasks.rados.rados.0.smithi203.stdout:3537: writing smithi20328086-26 from 524715 to 1175259 tid 1 2024-09-17T15:49:24.033 INFO:tasks.rados.rados.0.smithi203.stdout:3537: writing smithi20328086-26 from 1887964 to 2379737 tid 2 2024-09-17T15:49:24.033 INFO:tasks.rados.rados.0.smithi203.stdout:3533: done (3 left) 2024-09-17T15:49:24.033 INFO:tasks.rados.rados.0.smithi203.stdout:3538: snap_create 2024-09-17T15:49:24.033 INFO:tasks.rados.rados.0.smithi203.stdout:3534: finishing write tid 3 to smithi20328086-25 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:3534: finishing write tid 4 to smithi20328086-25 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:3534: oid 25 updating version 0 to 1547 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:3534: oid 25 version 1547 is already newer than 1546 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:3534: oid 25 updating version 1547 to 1548 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1548 (ObjNum 1244 snap 374 seq_num 1244) dirty exists 2024-09-17T15:49:24.034 INFO:tasks.rados.rados.0.smithi203.stdout:3534: left oid 25 (ObjNum 1244 snap 374 seq_num 1244) 2024-09-17T15:49:24.035 INFO:tasks.rados.rados.0.smithi203.stdout:3537: finishing write tid 1 to smithi20328086-26 2024-09-17T15:49:24.042 INFO:tasks.rados.rados.0.smithi203.stdout:3537: finishing write tid 2 to smithi20328086-26 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: finishing write tid 3 to smithi20328086-26 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: finishing write tid 4 to smithi20328086-26 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: oid 26 updating version 0 to 1833 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: oid 26 version 1833 is already newer than 1829 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: oid 26 version 1833 is already newer than 1832 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1833 (ObjNum 1245 snap 374 seq_num 1245) dirty exists 2024-09-17T15:49:24.043 INFO:tasks.rados.rados.0.smithi203.stdout:3537: left oid 26 (ObjNum 1245 snap 374 seq_num 1245) 2024-09-17T15:49:25.003 INFO:tasks.rados.rados.0.smithi203.stdout:3534: done (3 left) 2024-09-17T15:49:25.003 INFO:tasks.rados.rados.0.smithi203.stdout:3536: done (2 left) 2024-09-17T15:49:25.003 INFO:tasks.rados.rados.0.smithi203.stdout:3537: done (1 left) 2024-09-17T15:49:25.003 INFO:tasks.rados.rados.0.smithi203.stdout:3538: done (0 left) 2024-09-17T15:49:25.004 INFO:tasks.rados.rados.0.smithi203.stdout:3539: snap_remove snap 372 2024-09-17T15:49:25.845 INFO:tasks.rados.rados.0.smithi203.stdout:3539: done (0 left) 2024-09-17T15:49:25.845 INFO:tasks.rados.rados.0.smithi203.stdout:3540: snap_remove snap 363 2024-09-17T15:49:26.851 INFO:tasks.rados.rados.0.smithi203.stdout:3540: done (0 left) 2024-09-17T15:49:26.851 INFO:tasks.rados.rados.0.smithi203.stdout:3541: snap_create 2024-09-17T15:49:27.848 INFO:tasks.rados.rados.0.smithi203.stdout:3541: done (0 left) 2024-09-17T15:49:27.848 INFO:tasks.rados.rados.0.smithi203.stdout:3542: read oid 18 snap -1 2024-09-17T15:49:27.848 INFO:tasks.rados.rados.0.smithi203.stdout:3542: expect deleted 2024-09-17T15:49:27.848 INFO:tasks.rados.rados.0.smithi203.stdout:3543: snap_create 2024-09-17T15:49:28.855 INFO:tasks.rados.rados.0.smithi203.stdout:3542: done (1 left) 2024-09-17T15:49:28.855 INFO:tasks.rados.rados.0.smithi203.stdout:3543: done (0 left) 2024-09-17T15:49:28.856 INFO:tasks.rados.rados.0.smithi203.stdout:3544: copy_from oid 2 from oid 49 current snap is 377 2024-09-17T15:49:28.856 INFO:tasks.rados.rados.0.smithi203.stdout:3545: read oid 45 snap -1 2024-09-17T15:49:28.856 INFO:tasks.rados.rados.0.smithi203.stdout:3545: expect (ObjNum 1226 snap 368 seq_num 1226) 2024-09-17T15:49:28.856 INFO:tasks.rados.rados.0.smithi203.stdout:3546: copy_from oid 22 from oid 6 current snap is 377 2024-09-17T15:49:28.856 INFO:tasks.rados.rados.0.smithi203.stdout:3547: snap_create 2024-09-17T15:49:28.870 INFO:tasks.rados.rados.0.smithi203.stdout:3545: expect (ObjNum 1226 snap 368 seq_num 1226) 2024-09-17T15:49:28.870 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:29.000 INFO:tasks.rados.rados.0.smithi203.stdout:3544: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:49:29.000 INFO:tasks.rados.rados.0.smithi203.stdout:3546: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:49:29.000 INFO:tasks.rados.rados.0.smithi203.stdout:3544: finishing copy_from to smithi20328086-2 2024-09-17T15:49:29.000 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1827 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:49:29.001 INFO:tasks.rados.rados.0.smithi203.stdout:3546: finishing copy_from to smithi20328086-22 2024-09-17T15:49:29.001 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1646 (ObjNum 1235 snap 373 seq_num 1235) dirty exists 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3544: done (3 left) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3545: done (2 left) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3546: done (1 left) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3547: done (0 left) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3548: read oid 24 snap 370 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3548: expect (ObjNum 1203 snap 365 seq_num 1203) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3549: read oid 25 snap -1 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3549: expect (ObjNum 1244 snap 374 seq_num 1244) 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3550: read oid 43 snap -1 2024-09-17T15:49:29.864 INFO:tasks.rados.rados.0.smithi203.stdout:3550: expect (ObjNum 1108 snap 341 seq_num 1108) 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3551: read oid 15 snap -1 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3551: expect (ObjNum 1214 snap 365 seq_num 1214) 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3552: read oid 19 snap -1 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3552: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3553: write oid 34 current snap is 378 2024-09-17T15:49:29.865 INFO:tasks.rados.rados.0.smithi203.stdout:3553: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-17T15:49:29.888 INFO:tasks.rados.rados.0.smithi203.stdout:3553: writing smithi20328086-34 from 541522 to 1267315 tid 1 2024-09-17T15:49:29.909 INFO:tasks.rados.rados.0.smithi203.stdout:3553: writing smithi20328086-34 from 1946446 to 2581877 tid 2 2024-09-17T15:49:29.909 INFO:tasks.rados.rados.0.smithi203.stdout:3554: delete oid 36 current snap is 378 2024-09-17T15:49:29.909 INFO:tasks.rados.rados.0.smithi203.stdout:3550: expect (ObjNum 1108 snap 341 seq_num 1108) 2024-09-17T15:49:29.922 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1824030 2024-09-17T15:49:29.935 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 671142 2024-09-17T15:49:29.982 INFO:tasks.rados.rados.0.smithi203.stdout:3548: expect (ObjNum 1203 snap 365 seq_num 1203) 2024-09-17T15:49:29.982 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.006 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 13485 2024-09-17T15:49:30.051 INFO:tasks.rados.rados.0.smithi203.stdout:3552: expect (ObjNum 1186 snap 362 seq_num 1186) 2024-09-17T15:49:30.099 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2429815 2024-09-17T15:49:30.181 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1438535 2024-09-17T15:49:30.261 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 838453 2024-09-17T15:49:30.354 INFO:tasks.rados.rados.0.smithi203.stdout:3551: expect (ObjNum 1214 snap 365 seq_num 1214) 2024-09-17T15:49:30.354 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.385 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.416 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.447 INFO:tasks.rados.rados.0.smithi203.stdout:3549: expect (ObjNum 1244 snap 374 seq_num 1244) 2024-09-17T15:49:30.470 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.499 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: finishing write tid 1 to smithi20328086-34 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: finishing write tid 2 to smithi20328086-34 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: finishing write tid 3 to smithi20328086-34 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: finishing write tid 4 to smithi20328086-34 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: oid 34 updating version 0 to 1328 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: oid 34 version 1328 is already newer than 1327 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: oid 34 updating version 1328 to 1329 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1329 (ObjNum 1246 snap 378 seq_num 1246) dirty exists 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3553: left oid 34 (ObjNum 1246 snap 378 seq_num 1246) 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3548: done (6 left) 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3549: done (5 left) 2024-09-17T15:49:30.529 INFO:tasks.rados.rados.0.smithi203.stdout:3550: done (4 left) 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3551: done (3 left) 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3552: done (2 left) 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3553: done (1 left) 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3554: done (0 left) 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3555: write (excl) oid 36 current snap is 378 2024-09-17T15:49:30.530 INFO:tasks.rados.rados.0.smithi203.stdout:3555: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-09-17T15:49:30.547 INFO:tasks.rados.rados.0.smithi203.stdout:3555: writing smithi20328086-36 from 575136 to 1167780 tid 1 2024-09-17T15:49:30.566 INFO:tasks.rados.rados.0.smithi203.stdout:3555: writing smithi20328086-36 from 1663410 to 2302510 tid 2 2024-09-17T15:49:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:3555: writing smithi20328086-36 from 2705542 to 2975136 tid 3 2024-09-17T15:49:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:3556: rollback oid 46 current snap is 378 2024-09-17T15:49:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 46 to 374 2024-09-17T15:49:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:3555: finishing write tid 1 to smithi20328086-36 2024-09-17T15:49:30.577 INFO:tasks.rados.rados.0.smithi203.stdout:3555: finishing write tid 2 to smithi20328086-36 2024-09-17T15:49:30.578 INFO:tasks.rados.rados.0.smithi203.stdout:3557: snap_remove snap 365 2024-09-17T15:49:30.878 INFO:tasks.rados.rados.0.smithi203.stdout:3557: done (2 left) 2024-09-17T15:49:30.878 INFO:tasks.rados.rados.0.smithi203.stdout:3558: read oid 7 snap -1 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3558: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3559: snap_create 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: finishing write tid 3 to smithi20328086-36 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: finishing write tid 4 to smithi20328086-36 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: finishing write tid 5 to smithi20328086-36 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: oid 36 updating version 0 to 1853 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: oid 36 version 1853 is already newer than 1852 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: oid 36 version 1853 is already newer than 1851 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: oid 36 version 1853 is already newer than 1850 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1853 (ObjNum 1248 snap 378 seq_num 1248) dirty exists 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3555: left oid 36 (ObjNum 1248 snap 378 seq_num 1248) 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3556: finishing rollback tid 0 to smithi20328086-46 2024-09-17T15:49:30.879 INFO:tasks.rados.rados.0.smithi203.stdout:3556: finishing rollback tid 1 to smithi20328086-46 2024-09-17T15:49:30.880 INFO:tasks.rados.rados.0.smithi203.stdout:3556: finishing rollback tid 2 to smithi20328086-46 2024-09-17T15:49:30.880 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1439 (ObjNum 1192 snap 363 seq_num 1192) dirty exists 2024-09-17T15:49:30.892 INFO:tasks.rados.rados.0.smithi203.stdout:3558: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:30.892 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 121630 2024-09-17T15:49:31.055 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:31.166 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:31.872 INFO:tasks.rados.rados.0.smithi203.stdout:3555: done (3 left) 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3556: done (2 left) 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3558: done (1 left) 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3559: done (0 left) 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3560: read oid 31 snap -1 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3560: expect (ObjNum 1190 snap 363 seq_num 1190) 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3561: copy_from oid 22 from oid 45 current snap is 379 2024-09-17T15:49:31.873 INFO:tasks.rados.rados.0.smithi203.stdout:3562: snap_create 2024-09-17T15:49:31.889 INFO:tasks.rados.rados.0.smithi203.stdout:3560: expect (ObjNum 1190 snap 363 seq_num 1190) 2024-09-17T15:49:31.889 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:31.945 INFO:tasks.rados.rados.0.smithi203.stdout:3561: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:49:31.945 INFO:tasks.rados.rados.0.smithi203.stdout:3561: finishing copy_from to smithi20328086-22 2024-09-17T15:49:31.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1345 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:49:32.876 INFO:tasks.rados.rados.0.smithi203.stdout:3560: done (2 left) 2024-09-17T15:49:32.876 INFO:tasks.rados.rados.0.smithi203.stdout:3561: done (1 left) 2024-09-17T15:49:32.876 INFO:tasks.rados.rados.0.smithi203.stdout:3562: done (0 left) 2024-09-17T15:49:32.876 INFO:tasks.rados.rados.0.smithi203.stdout:3563: read oid 7 snap -1 2024-09-17T15:49:32.877 INFO:tasks.rados.rados.0.smithi203.stdout:3563: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:32.877 INFO:tasks.rados.rados.0.smithi203.stdout:3564: write (excl) oid 35 current snap is 380 2024-09-17T15:49:32.877 INFO:tasks.rados.rados.0.smithi203.stdout:3564: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-17T15:49:32.897 INFO:tasks.rados.rados.0.smithi203.stdout:3564: writing smithi20328086-35 from 591943 to 1259836 tid 1 2024-09-17T15:49:32.917 INFO:tasks.rados.rados.0.smithi203.stdout:3564: writing smithi20328086-35 from 1721892 to 2504650 tid 2 2024-09-17T15:49:32.917 INFO:tasks.rados.rados.0.smithi203.stdout:3565: write oid 18 current snap is 380 2024-09-17T15:49:32.917 INFO:tasks.rados.rados.0.smithi203.stdout:3565: seq_num 1250 ranges {608750=743142,1780374=642769} 2024-09-17T15:49:32.935 INFO:tasks.rados.rados.0.smithi203.stdout:3565: writing smithi20328086-18 from 608750 to 1351892 tid 1 2024-09-17T15:49:32.947 INFO:tasks.rados.rados.0.smithi203.stdout:3565: writing smithi20328086-18 from 1780374 to 2423143 tid 2 2024-09-17T15:49:32.947 INFO:tasks.rados.rados.0.smithi203.stdout:3566: write (excl) oid 41 current snap is 380 2024-09-17T15:49:32.947 INFO:tasks.rados.rados.0.smithi203.stdout:3566: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-17T15:49:32.956 INFO:tasks.rados.rados.0.smithi203.stdout:3566: writing smithi20328086-41 from 625557 to 1043948 tid 1 2024-09-17T15:49:32.969 INFO:tasks.rados.rados.0.smithi203.stdout:3566: writing smithi20328086-41 from 1838856 to 2625283 tid 2 2024-09-17T15:49:32.969 INFO:tasks.rados.rados.0.smithi203.stdout:3567: rollback oid 43 current snap is 380 2024-09-17T15:49:32.969 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 374 2024-09-17T15:49:32.969 INFO:tasks.rados.rados.0.smithi203.stdout:3563: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:33.060 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2064808 2024-09-17T15:49:33.257 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1775668 2024-09-17T15:49:33.313 INFO:tasks.rados.rados.0.smithi203.stdout:3564: finishing write tid 1 to smithi20328086-35 2024-09-17T15:49:33.313 INFO:tasks.rados.rados.0.smithi203.stdout:3564: finishing write tid 2 to smithi20328086-35 2024-09-17T15:49:33.313 INFO:tasks.rados.rados.0.smithi203.stdout:3564: finishing write tid 3 to smithi20328086-35 2024-09-17T15:49:33.313 INFO:tasks.rados.rados.0.smithi203.stdout:3564: finishing write tid 4 to smithi20328086-35 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3564: oid 35 updating version 0 to 1330 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3564: oid 35 updating version 1330 to 1332 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3564: oid 35 version 1332 is already newer than 1331 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1332 (ObjNum 1249 snap 380 seq_num 1249) dirty exists 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3564: left oid 35 (ObjNum 1249 snap 380 seq_num 1249) 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: finishing write tid 1 to smithi20328086-18 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: finishing write tid 2 to smithi20328086-18 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: finishing write tid 3 to smithi20328086-18 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: finishing write tid 4 to smithi20328086-18 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: oid 18 updating version 0 to 1544 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: oid 18 version 1544 is already newer than 1542 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: oid 18 version 1544 is already newer than 1543 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 1544 (ObjNum 1250 snap 380 seq_num 1250) dirty exists 2024-09-17T15:49:33.314 INFO:tasks.rados.rados.0.smithi203.stdout:3565: left oid 18 (ObjNum 1250 snap 380 seq_num 1250) 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: finishing write tid 1 to smithi20328086-41 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: finishing write tid 2 to smithi20328086-41 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: finishing write tid 3 to smithi20328086-41 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: finishing write tid 4 to smithi20328086-41 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: oid 41 updating version 0 to 691 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: oid 41 version 691 is already newer than 689 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: oid 41 version 691 is already newer than 690 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 691 (ObjNum 1251 snap 380 seq_num 1251) dirty exists 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3566: left oid 41 (ObjNum 1251 snap 380 seq_num 1251) 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3567: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3567: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3567: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 694 (ObjNum 1108 snap 341 seq_num 1108) dirty exists 2024-09-17T15:49:33.315 INFO:tasks.rados.rados.0.smithi203.stdout:3563: done (4 left) 2024-09-17T15:49:33.316 INFO:tasks.rados.rados.0.smithi203.stdout:3564: done (3 left) 2024-09-17T15:49:33.316 INFO:tasks.rados.rados.0.smithi203.stdout:3565: done (2 left) 2024-09-17T15:49:33.316 INFO:tasks.rados.rados.0.smithi203.stdout:3566: done (1 left) 2024-09-17T15:49:33.316 INFO:tasks.rados.rados.0.smithi203.stdout:3567: done (0 left) 2024-09-17T15:49:33.316 INFO:tasks.rados.rados.0.smithi203.stdout:3568: delete oid 13 current snap is 380 2024-09-17T15:49:33.320 INFO:tasks.rados.rados.0.smithi203.stdout:3568: done (0 left) 2024-09-17T15:49:33.320 INFO:tasks.rados.rados.0.smithi203.stdout:3569: rollback oid 33 current snap is 380 2024-09-17T15:49:33.320 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 377 2024-09-17T15:49:33.320 INFO:tasks.rados.rados.0.smithi203.stdout:3570: write (excl) oid 6 current snap is 380 2024-09-17T15:49:33.320 INFO:tasks.rados.rados.0.smithi203.stdout:3570: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-09-17T15:49:33.331 INFO:tasks.rados.rados.0.smithi203.stdout:3570: writing smithi20328086-6 from 659171 to 1228060 tid 1 2024-09-17T15:49:33.343 INFO:tasks.rados.rados.0.smithi203.stdout:3570: writing smithi20328086-6 from 1839467 to 2629563 tid 2 2024-09-17T15:49:33.343 INFO:tasks.rados.rados.0.smithi203.stdout:3571: write (excl) oid 21 current snap is 380 2024-09-17T15:49:33.343 INFO:tasks.rados.rados.0.smithi203.stdout:3571: seq_num 1254 ranges {675978=644138,1897949=650107} 2024-09-17T15:49:33.356 INFO:tasks.rados.rados.0.smithi203.stdout:3571: writing smithi20328086-21 from 675978 to 1320116 tid 1 2024-09-17T15:49:33.369 INFO:tasks.rados.rados.0.smithi203.stdout:3571: writing smithi20328086-21 from 1897949 to 2548056 tid 2 2024-09-17T15:49:33.369 INFO:tasks.rados.rados.0.smithi203.stdout:3572: write oid 14 current snap is 380 2024-09-17T15:49:33.369 INFO:tasks.rados.rados.0.smithi203.stdout:3572: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-17T15:49:33.378 INFO:tasks.rados.rados.0.smithi203.stdout:3572: writing smithi20328086-14 from 692785 to 1128525 tid 1 2024-09-17T15:49:33.457 INFO:tasks.rados.rados.0.smithi203.stdout:3572: writing smithi20328086-14 from 1672784 to 2466549 tid 2 2024-09-17T15:49:33.457 INFO:tasks.rados.rados.0.smithi203.stdout:3572: writing smithi20328086-14 from 2897503 to 3092785 tid 3 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3573: rollback oid 49 current snap is 380 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 49 to 373 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3574: delete oid 50 current snap is 380 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3569: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3569: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3569: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1418 (ObjNum 1239 snap 373 seq_num 1239) dirty exists 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: finishing write tid 1 to smithi20328086-6 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: finishing write tid 2 to smithi20328086-6 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: finishing write tid 3 to smithi20328086-6 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: finishing write tid 4 to smithi20328086-6 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: oid 6 updating version 0 to 1652 2024-09-17T15:49:33.458 INFO:tasks.rados.rados.0.smithi203.stdout:3570: oid 6 version 1652 is already newer than 1651 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3570: oid 6 version 1652 is already newer than 1650 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1652 (ObjNum 1253 snap 380 seq_num 1253) dirty exists 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3570: left oid 6 (ObjNum 1253 snap 380 seq_num 1253) 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: finishing write tid 1 to smithi20328086-21 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: finishing write tid 2 to smithi20328086-21 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3572: finishing write tid 1 to smithi20328086-14 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: finishing write tid 3 to smithi20328086-21 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: finishing write tid 4 to smithi20328086-21 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: oid 21 updating version 0 to 1865 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: oid 21 updating version 1865 to 1867 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: oid 21 version 1867 is already newer than 1866 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1867 (ObjNum 1254 snap 380 seq_num 1254) dirty exists 2024-09-17T15:49:33.459 INFO:tasks.rados.rados.0.smithi203.stdout:3571: left oid 21 (ObjNum 1254 snap 380 seq_num 1254) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3569: done (5 left) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3570: done (4 left) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3571: done (3 left) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3574: done (2 left) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3575: rollback oid 32 current snap is 380 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 373 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3576: read oid 33 snap -1 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3576: expect (ObjNum 1239 snap 373 seq_num 1239) 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3577: write oid 1 current snap is 380 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3577: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3577: writing smithi20328086-1 from 726399 to 1312637 tid 1 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3577: writing smithi20328086-1 from 2073395 to 2870829 tid 2 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3578: copy_from oid 39 from oid 6 current snap is 380 2024-09-17T15:49:33.460 INFO:tasks.rados.rados.0.smithi203.stdout:3572: finishing write tid 2 to smithi20328086-14 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: finishing write tid 3 to smithi20328086-14 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: finishing write tid 4 to smithi20328086-14 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: finishing write tid 5 to smithi20328086-14 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: oid 14 updating version 0 to 1654 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: oid 14 version 1654 is already newer than 1653 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: oid 14 updating version 1654 to 1655 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: oid 14 updating version 1655 to 1656 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1656 (ObjNum 1255 snap 380 seq_num 1255) dirty exists 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: left oid 14 (ObjNum 1255 snap 380 seq_num 1255) 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3572: done (5 left) 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3579: delete oid 12 current snap is 380 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3573: finishing rollback tid 0 to smithi20328086-49 2024-09-17T15:49:33.461 INFO:tasks.rados.rados.0.smithi203.stdout:3573: finishing rollback tid 1 to smithi20328086-49 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:3573: finishing rollback tid 2 to smithi20328086-49 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1871 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:3575: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:3575: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:3575: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1729 (ObjNum 1240 snap 373 seq_num 1240) dirty exists 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:3576: expect (ObjNum 1239 snap 373 seq_num 1239) 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1355895 2024-09-17T15:49:33.462 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:33.502 INFO:tasks.rados.rados.0.smithi203.stdout:3577: finishing write tid 1 to smithi20328086-1 2024-09-17T15:49:33.502 INFO:tasks.rados.rados.0.smithi203.stdout:3577: finishing write tid 2 to smithi20328086-1 2024-09-17T15:49:33.502 INFO:tasks.rados.rados.0.smithi203.stdout:3577: finishing write tid 3 to smithi20328086-1 2024-09-17T15:49:33.502 INFO:tasks.rados.rados.0.smithi203.stdout:3577: finishing write tid 4 to smithi20328086-1 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3577: oid 1 updating version 0 to 1658 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3577: oid 1 updating version 1658 to 1659 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3577: oid 1 version 1659 is already newer than 1657 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1659 (ObjNum 1257 snap 380 seq_num 1257) dirty exists 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3577: left oid 1 (ObjNum 1257 snap 380 seq_num 1257) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3578: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3578: finishing copy_from to smithi20328086-39 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1652 (ObjNum 1253 snap 380 seq_num 1253) dirty exists 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3573: done (5 left) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3575: done (4 left) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3576: done (3 left) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3577: done (2 left) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3578: done (1 left) 2024-09-17T15:49:33.503 INFO:tasks.rados.rados.0.smithi203.stdout:3579: done (0 left) 2024-09-17T15:49:33.504 INFO:tasks.rados.rados.0.smithi203.stdout:3580: snap_remove snap 374 2024-09-17T15:49:33.894 INFO:tasks.rados.rados.0.smithi203.stdout:3580: done (0 left) 2024-09-17T15:49:33.894 INFO:tasks.rados.rados.0.smithi203.stdout:3581: snap_remove snap 379 2024-09-17T15:49:34.904 INFO:tasks.rados.rados.0.smithi203.stdout:3581: done (0 left) 2024-09-17T15:49:34.904 INFO:tasks.rados.rados.0.smithi203.stdout:3582: delete oid 12 current snap is 380 2024-09-17T15:49:34.914 INFO:tasks.rados.rados.0.smithi203.stdout:3582: done (0 left) 2024-09-17T15:49:34.914 INFO:tasks.rados.rados.0.smithi203.stdout:3583: snap_remove snap 378 2024-09-17T15:49:35.849 INFO:tasks.rados.rados.0.smithi203.stdout:3583: done (0 left) 2024-09-17T15:49:35.849 INFO:tasks.rados.rados.0.smithi203.stdout:3584: snap_create 2024-09-17T15:49:36.849 INFO:tasks.rados.rados.0.smithi203.stdout:3584: done (0 left) 2024-09-17T15:49:36.850 INFO:tasks.rados.rados.0.smithi203.stdout:3585: rollback oid 23 current snap is 381 2024-09-17T15:49:36.850 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 370 2024-09-17T15:49:36.850 INFO:tasks.rados.rados.0.smithi203.stdout:3586: snap_create 2024-09-17T15:49:36.858 INFO:tasks.rados.rados.0.smithi203.stdout:3585: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:49:36.858 INFO:tasks.rados.rados.0.smithi203.stdout:3585: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:49:36.858 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1661 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:49:37.851 INFO:tasks.rados.rados.0.smithi203.stdout:3585: done (1 left) 2024-09-17T15:49:37.852 INFO:tasks.rados.rados.0.smithi203.stdout:3586: done (0 left) 2024-09-17T15:49:37.852 INFO:tasks.rados.rados.0.smithi203.stdout:3587: write (excl) oid 41 current snap is 382 2024-09-17T15:49:37.852 INFO:tasks.rados.rados.0.smithi203.stdout:3587: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-17T15:49:37.870 INFO:tasks.rados.rados.0.smithi203.stdout:3587: writing smithi20328086-41 from 776820 to 1188805 tid 1 2024-09-17T15:49:37.891 INFO:tasks.rados.rados.0.smithi203.stdout:3587: writing smithi20328086-41 from 1848841 to 2393602 tid 2 2024-09-17T15:49:37.904 INFO:tasks.rados.rados.0.smithi203.stdout:3587: writing smithi20328086-41 from 2918265 to 3176820 tid 3 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3588: read oid 46 snap -1 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3588: expect (ObjNum 1192 snap 363 seq_num 1192) 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3589: read oid 28 snap 380 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3589: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3590: read oid 29 snap 377 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3590: expect (ObjNum 1232 snap 371 seq_num 1232) 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3591: snap_create 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3587: finishing write tid 1 to smithi20328086-41 2024-09-17T15:49:37.905 INFO:tasks.rados.rados.0.smithi203.stdout:3587: finishing write tid 2 to smithi20328086-41 2024-09-17T15:49:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:3587: finishing write tid 3 to smithi20328086-41 2024-09-17T15:49:37.910 INFO:tasks.rados.rados.0.smithi203.stdout:3587: finishing write tid 4 to smithi20328086-41 2024-09-17T15:49:37.911 INFO:tasks.rados.rados.0.smithi203.stdout:3587: finishing write tid 5 to smithi20328086-41 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:3587: oid 41 updating version 0 to 696 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:3587: oid 41 version 696 is already newer than 695 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:3587: oid 41 updating version 696 to 698 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:3587: oid 41 version 698 is already newer than 697 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 698 (ObjNum 1260 snap 382 seq_num 1260) dirty exists 2024-09-17T15:49:37.912 INFO:tasks.rados.rados.0.smithi203.stdout:3587: left oid 41 (ObjNum 1260 snap 382 seq_num 1260) 2024-09-17T15:49:37.914 INFO:tasks.rados.rados.0.smithi203.stdout:3589: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:49:38.007 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2037336 2024-09-17T15:49:38.019 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1297328 2024-09-17T15:49:38.045 INFO:tasks.rados.rados.0.smithi203.stdout:3588: expect (ObjNum 1192 snap 363 seq_num 1192) 2024-09-17T15:49:38.098 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:38.125 INFO:tasks.rados.rados.0.smithi203.stdout:3590: expect (ObjNum 1232 snap 371 seq_num 1232) 2024-09-17T15:49:38.125 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:38.163 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:38.859 INFO:tasks.rados.rados.0.smithi203.stdout:3587: done (4 left) 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3588: done (3 left) 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3589: done (2 left) 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3590: done (1 left) 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3591: done (0 left) 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3592: write (excl) oid 14 current snap is 383 2024-09-17T15:49:38.860 INFO:tasks.rados.rados.0.smithi203.stdout:3592: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-17T15:49:38.878 INFO:tasks.rados.rados.0.smithi203.stdout:3592: writing smithi20328086-14 from 793627 to 1280861 tid 1 2024-09-17T15:49:38.892 INFO:tasks.rados.rados.0.smithi203.stdout:3592: writing smithi20328086-14 from 1790970 to 2195742 tid 2 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3592: writing smithi20328086-14 from 2829335 to 3193627 tid 3 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3593: rollback oid 8 current snap is 383 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 380 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3592: finishing write tid 1 to smithi20328086-14 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3592: finishing write tid 2 to smithi20328086-14 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3594: copy_from oid 18 from oid 11 current snap is 383 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3595: read oid 38 snap 382 2024-09-17T15:49:38.909 INFO:tasks.rados.rados.0.smithi203.stdout:3595: expect deleted 2024-09-17T15:49:38.910 INFO:tasks.rados.rados.0.smithi203.stdout:3596: read oid 20 snap 380 2024-09-17T15:49:38.910 INFO:tasks.rados.rados.0.smithi203.stdout:3596: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:49:38.910 INFO:tasks.rados.rados.0.smithi203.stdout:3597: snap_remove snap 377 2024-09-17T15:49:39.867 INFO:tasks.rados.rados.0.smithi203.stdout:3597: done (5 left) 2024-09-17T15:49:39.867 INFO:tasks.rados.rados.0.smithi203.stdout:3598: snap_remove snap 370 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3598: done (5 left) 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3599: snap_create 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: finishing write tid 3 to smithi20328086-14 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: finishing write tid 4 to smithi20328086-14 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: finishing write tid 5 to smithi20328086-14 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: oid 14 updating version 0 to 1665 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: oid 14 version 1665 is already newer than 1663 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: oid 14 version 1665 is already newer than 1664 2024-09-17T15:49:41.079 INFO:tasks.rados.rados.0.smithi203.stdout:3592: oid 14 version 1665 is already newer than 1662 2024-09-17T15:49:41.080 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1665 (ObjNum 1261 snap 383 seq_num 1261) dirty exists 2024-09-17T15:49:41.080 INFO:tasks.rados.rados.0.smithi203.stdout:3592: left oid 14 (ObjNum 1261 snap 383 seq_num 1261) 2024-09-17T15:49:41.080 INFO:tasks.rados.rados.0.smithi203.stdout:3596: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:49:41.189 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:3593: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:3593: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:3593: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1547 (ObjNum 1157 snap 357 seq_num 1157) dirty exists 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:3594: finishing copy_from racing read to smithi20328086-18 2024-09-17T15:49:41.308 INFO:tasks.rados.rados.0.smithi203.stdout:3594: finishing copy_from to smithi20328086-18 2024-09-17T15:49:41.309 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 1639 (ObjNum 1241 snap 373 seq_num 1241) dirty exists 2024-09-17T15:49:41.877 INFO:tasks.rados.rados.0.smithi203.stdout:3592: done (5 left) 2024-09-17T15:49:41.877 INFO:tasks.rados.rados.0.smithi203.stdout:3593: done (4 left) 2024-09-17T15:49:41.877 INFO:tasks.rados.rados.0.smithi203.stdout:3594: done (3 left) 2024-09-17T15:49:41.877 INFO:tasks.rados.rados.0.smithi203.stdout:3595: done (2 left) 2024-09-17T15:49:41.878 INFO:tasks.rados.rados.0.smithi203.stdout:3596: done (1 left) 2024-09-17T15:49:41.878 INFO:tasks.rados.rados.0.smithi203.stdout:3599: done (0 left) 2024-09-17T15:49:41.878 INFO:tasks.rados.rados.0.smithi203.stdout:3600: snap_create 2024-09-17T15:49:42.890 INFO:tasks.rados.rados.0.smithi203.stdout:3600: done (0 left) 2024-09-17T15:49:42.891 INFO:tasks.rados.rados.0.smithi203.stdout:3601: rollback oid 27 current snap is 385 2024-09-17T15:49:42.891 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 382 2024-09-17T15:49:42.891 INFO:tasks.rados.rados.0.smithi203.stdout:3602: write oid 38 current snap is 385 2024-09-17T15:49:42.891 INFO:tasks.rados.rados.0.smithi203.stdout:3602: seq_num 1262 ranges {410434=678836,1565805=548430,2573111=637323} 2024-09-17T15:49:42.908 INFO:tasks.rados.rados.0.smithi203.stdout:3602: writing smithi20328086-38 from 410434 to 1089270 tid 1 2024-09-17T15:49:42.925 INFO:tasks.rados.rados.0.smithi203.stdout:3602: writing smithi20328086-38 from 1565805 to 2114235 tid 2 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3602: writing smithi20328086-38 from 2573111 to 3210434 tid 3 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3603: copy_from oid 25 from oid 34 current snap is 385 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3601: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3602: finishing write tid 1 to smithi20328086-38 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3602: finishing write tid 2 to smithi20328086-38 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3601: done (2 left) 2024-09-17T15:49:42.942 INFO:tasks.rados.rados.0.smithi203.stdout:3604: delete oid 27 current snap is 385 2024-09-17T15:49:42.949 INFO:tasks.rados.rados.0.smithi203.stdout:3604: done (2 left) 2024-09-17T15:49:42.949 INFO:tasks.rados.rados.0.smithi203.stdout:3605: rollback oid 1 current snap is 385 2024-09-17T15:49:42.949 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 380 2024-09-17T15:49:42.949 INFO:tasks.rados.rados.0.smithi203.stdout:3606: delete oid 30 current snap is 385 2024-09-17T15:49:42.950 INFO:tasks.rados.rados.0.smithi203.stdout:3602: finishing write tid 3 to smithi20328086-38 2024-09-17T15:49:42.950 INFO:tasks.rados.rados.0.smithi203.stdout:3602: finishing write tid 4 to smithi20328086-38 2024-09-17T15:49:42.950 INFO:tasks.rados.rados.0.smithi203.stdout:3602: finishing write tid 5 to smithi20328086-38 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:3602: oid 38 updating version 0 to 1389 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:3602: oid 38 updating version 1389 to 1391 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:3602: oid 38 version 1391 is already newer than 1390 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:3602: oid 38 version 1391 is already newer than 1388 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1391 (ObjNum 1262 snap 385 seq_num 1262) dirty exists 2024-09-17T15:49:42.951 INFO:tasks.rados.rados.0.smithi203.stdout:3602: left oid 38 (ObjNum 1262 snap 385 seq_num 1262) 2024-09-17T15:49:42.960 INFO:tasks.rados.rados.0.smithi203.stdout:3602: done (3 left) 2024-09-17T15:49:42.960 INFO:tasks.rados.rados.0.smithi203.stdout:3606: done (2 left) 2024-09-17T15:49:42.960 INFO:tasks.rados.rados.0.smithi203.stdout:3607: delete oid 11 current snap is 385 2024-09-17T15:49:42.966 INFO:tasks.rados.rados.0.smithi203.stdout:3603: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:49:42.967 INFO:tasks.rados.rados.0.smithi203.stdout:3607: done (2 left) 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3608: rollback oid 32 current snap is 385 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 380 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3609: copy_from oid 48 from oid 2 current snap is 385 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3610: rollback oid 5 current snap is 385 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 375 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3611: read oid 16 snap -1 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3611: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3612: write (excl) oid 31 current snap is 385 2024-09-17T15:49:42.968 INFO:tasks.rados.rados.0.smithi203.stdout:3612: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-09-17T15:49:42.982 INFO:tasks.rados.rados.0.smithi203.stdout:3612: writing smithi20328086-31 from 477662 to 1057494 tid 1 2024-09-17T15:49:43.000 INFO:tasks.rados.rados.0.smithi203.stdout:3612: writing smithi20328086-31 from 1683380 to 2239148 tid 2 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3612: writing smithi20328086-31 from 2966450 to 3277662 tid 3 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3613: delete oid 29 current snap is 385 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3610: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3605: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3605: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3605: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1668 (ObjNum 1257 snap 380 seq_num 1257) dirty exists 2024-09-17T15:49:43.012 INFO:tasks.rados.rados.0.smithi203.stdout:3608: finishing rollback tid 0 to smithi20328086-32 2024-09-17T15:49:43.013 INFO:tasks.rados.rados.0.smithi203.stdout:3608: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:49:43.013 INFO:tasks.rados.rados.0.smithi203.stdout:3608: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:49:43.013 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1732 (ObjNum 1240 snap 373 seq_num 1240) dirty exists 2024-09-17T15:49:43.013 INFO:tasks.rados.rados.0.smithi203.stdout:3611: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:49:43.065 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:43.120 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 961243 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:3603: finishing copy_from to smithi20328086-25 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1329 (ObjNum 1246 snap 378 seq_num 1246) dirty exists 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:3612: finishing write tid 1 to smithi20328086-31 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:3609: finishing copy_from racing read to smithi20328086-48 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:3609: finishing copy_from to smithi20328086-48 2024-09-17T15:49:43.148 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1827 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: finishing write tid 2 to smithi20328086-31 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: finishing write tid 3 to smithi20328086-31 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: finishing write tid 4 to smithi20328086-31 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: finishing write tid 5 to smithi20328086-31 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: oid 31 updating version 0 to 1650 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: oid 31 version 1650 is already newer than 1649 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: oid 31 version 1650 is already newer than 1647 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: oid 31 version 1650 is already newer than 1648 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1650 (ObjNum 1266 snap 385 seq_num 1266) dirty exists 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3612: left oid 31 (ObjNum 1266 snap 385 seq_num 1266) 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3603: done (7 left) 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3605: done (6 left) 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3608: done (5 left) 2024-09-17T15:49:43.149 INFO:tasks.rados.rados.0.smithi203.stdout:3609: done (4 left) 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3610: done (3 left) 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3611: done (2 left) 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3612: done (1 left) 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3613: done (0 left) 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3614: read oid 27 snap 380 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3614: expect deleted 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3615: write (excl) oid 15 current snap is 385 2024-09-17T15:49:43.150 INFO:tasks.rados.rados.0.smithi203.stdout:3615: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-17T15:49:43.162 INFO:tasks.rados.rados.0.smithi203.stdout:3615: writing smithi20328086-15 from 511276 to 1241606 tid 1 2024-09-17T15:49:43.175 INFO:tasks.rados.rados.0.smithi203.stdout:3615: writing smithi20328086-15 from 1800344 to 2359781 tid 2 2024-09-17T15:49:43.187 INFO:tasks.rados.rados.0.smithi203.stdout:3615: writing smithi20328086-15 from 3021296 to 3311276 tid 3 2024-09-17T15:49:43.187 INFO:tasks.rados.rados.0.smithi203.stdout:3616: read oid 35 snap -1 2024-09-17T15:49:43.187 INFO:tasks.rados.rados.0.smithi203.stdout:3616: expect (ObjNum 1249 snap 380 seq_num 1249) 2024-09-17T15:49:43.187 INFO:tasks.rados.rados.0.smithi203.stdout:3617: snap_create 2024-09-17T15:49:43.187 INFO:tasks.rados.rados.0.smithi203.stdout:3615: finishing write tid 1 to smithi20328086-15 2024-09-17T15:49:43.188 INFO:tasks.rados.rados.0.smithi203.stdout:3615: finishing write tid 2 to smithi20328086-15 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: finishing write tid 3 to smithi20328086-15 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: finishing write tid 4 to smithi20328086-15 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: finishing write tid 5 to smithi20328086-15 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: oid 15 updating version 0 to 1710 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: oid 15 version 1710 is already newer than 1709 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: oid 15 version 1710 is already newer than 1707 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: oid 15 version 1710 is already newer than 1708 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1710 (ObjNum 1268 snap 385 seq_num 1268) dirty exists 2024-09-17T15:49:43.194 INFO:tasks.rados.rados.0.smithi203.stdout:3615: left oid 15 (ObjNum 1268 snap 385 seq_num 1268) 2024-09-17T15:49:43.202 INFO:tasks.rados.rados.0.smithi203.stdout:3616: expect (ObjNum 1249 snap 380 seq_num 1249) 2024-09-17T15:49:43.203 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:43.299 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3614: done (3 left) 2024-09-17T15:49:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3615: done (2 left) 2024-09-17T15:49:43.898 INFO:tasks.rados.rados.0.smithi203.stdout:3616: done (1 left) 2024-09-17T15:49:43.899 INFO:tasks.rados.rados.0.smithi203.stdout:3617: done (0 left) 2024-09-17T15:49:43.899 INFO:tasks.rados.rados.0.smithi203.stdout:3618: read oid 43 snap -1 2024-09-17T15:49:43.899 INFO:tasks.rados.rados.0.smithi203.stdout:3618: expect (ObjNum 1108 snap 341 seq_num 1108) 2024-09-17T15:49:43.899 INFO:tasks.rados.rados.0.smithi203.stdout:3619: snap_create 2024-09-17T15:49:43.912 INFO:tasks.rados.rados.0.smithi203.stdout:3618: expect (ObjNum 1108 snap 341 seq_num 1108) 2024-09-17T15:49:43.930 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:43.958 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:43.969 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:49:43.969 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:49:43.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:49:43.972 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:49:43.972 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:49:43.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: get_monmap_and_config 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: build_initial_monmap 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: monmap: 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:44.109299+0000 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:44.109299+0000 2024-09-17T15:49:44.105 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:44.106 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 Processor -- start 2024-09-17T15:49:44.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 -- start start 2024-09-17T15:49:44.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: init 2024-09-17T15:49:44.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:44.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a6c0689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c0694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 0x7f9a6c13d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c13aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6c074470 con 0x7f9a6c0689d0 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9a6c09c8b0 con 0x7f9a6c13aff0 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6c09cb90 con 0x7f9a6c0694c0 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a73f70640 10 monclient(hunting): authenticate will time out at 2317.074777s 2024-09-17T15:49:44.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a71d0c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9a6c0689d0 0x7f9a6c068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48440/0 (socket says 172.21.15.86:48440) 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a71d0c640 1 -- 172.21.15.86:0/3706408878 learned_addr learned my addr 172.21.15.86:0/3706408878 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): get_auth_request con 0x7f9a6c0694c0 auth_method 0 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient(hunting): handle_auth_done global_id 164969 payload 274 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7250d640 1 --2- 172.21.15.86:0/3706408878 >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 0x7f9a6c13d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 -- 172.21.15.86:0/3706408878 >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 msgr2=0x7f9a6c13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:44.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 --2- 172.21.15.86:0/3706408878 >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 0x7f9a6c13d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 -- 172.21.15.86:0/3706408878 >> v1:172.21.15.86:6789/0 conn(0x7f9a6c0689d0 legacy=0x7f9a6c068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient: _finish_hunting 0 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 monclient: found mon.noname-b 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7150b640 1 -- 172.21.15.86:0/3706408878 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c09ce70 con 0x7f9a6c0694c0 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7250d640 10 monclient: get_auth_request con 0x7f9a6c13aff0 auth_method 0 2024-09-17T15:49:44.111 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.105+0000 7f9a7250d640 1 --2- 172.21.15.86:0/3706408878 >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 0x7f9a6c13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:44.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.109+0000 7f9a7150b640 1 -- 172.21.15.86:0/3706408878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 msgr2=0x7f9a6c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:44.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.109+0000 7f9a7150b640 1 --2- 172.21.15.86:0/3706408878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c06f840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9a680097c0 tx=0x7f9a6801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9d7684272a0f68a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:44.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.109+0000 7f9a70d0a640 1 -- 172.21.15.86:0/3706408878 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680b0070 con 0x7f9a6c0694c0 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.109+0000 7f9a70d0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: dump: 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:44.113 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: _finish_auth 0 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: _check_auth_tickets 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 1 -- 172.21.15.86:0/3706408878 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a680ab600 con 0x7f9a6c0694c0 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 5 monclient: authenticate success, global_id 164969 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 1 -- 172.21.15.86:0/3706408878 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c09c2d0 con 0x7f9a6c0694c0 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 4 set_mon_vals no callback set 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:44.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a70d0a640 10 monclient: dump: 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: shutdown 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- 172.21.15.86:0/3706408878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 msgr2=0x7f9a6c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:44.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- 172.21.15.86:0/3706408878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c06f840 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9a680097c0 tx=0x7f9a6801c360 comp rx=0 tx=0).stop 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- 172.21.15.86:0/3706408878 shutdown_connections 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- 172.21.15.86:0/3706408878 >> v2:172.21.15.203:3300/0 conn(0x7f9a6c13aff0 0x7f9a6c13d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- 172.21.15.86:0/3706408878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c06f840 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- 172.21.15.86:0/3706408878 >> 172.21.15.86:0/3706408878 conn(0x7f9a6c0a6440 msgr2=0x7f9a6c0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- 172.21.15.86:0/3706408878 shutdown_connections 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- 172.21.15.86:0/3706408878 wait complete. 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: build_initial_monmap 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: monmap: 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:44.116 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:44.117 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:44.117 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.121 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 Processor -- start 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- start start 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: init 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-a con 0x7f9a6c0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-c con 0x7f9a6c0694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): picked mon.noname-b con 0x7f9a6c13aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c140610 con 0x7f9a6c0689d0 2024-09-17T15:49:44.122 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6c140490 con 0x7f9a6c13aff0 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a6c140790 con 0x7f9a6c0694c0 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a73f70640 10 monclient(hunting): authenticate will time out at 2317.082214s 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:46802/0 (socket says 172.21.15.86:46802) 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 -- 172.21.15.86:0/2439676843 learned_addr learned my addr 172.21.15.86:0/2439676843 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): get_auth_request con 0x7f9a6c0689d0 auth_method 0 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:44.123 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): get_auth_request con 0x7f9a6c0694c0 auth_method 0 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 10 monclient(hunting): get_auth_request con 0x7f9a6c13aff0 auth_method 0 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient(hunting): handle_auth_done global_id 234309 payload 824 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 msgr2=0x7f9a6c071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 msgr2=0x7f9a6c071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 msgr2=0x7f9a6c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 msgr2=0x7f9a6c071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7250d640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:44.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient: _finish_hunting 0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 monclient: found mon.noname-a 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a71d0c640 1 -- 172.21.15.86:0/2439676843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a6c073650 con 0x7f9a6c0689d0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.113+0000 7f9a7150b640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a71d0c640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 msgr2=0x7f9a6c1cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a71d0c640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c470 tx=0x7f9a5c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c007070 con 0x7f9a6c0689d0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: dump: 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:44.126 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5c007210 con 0x7f9a6c0689d0 2024-09-17T15:49:44.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c007600 con 0x7f9a6c0689d0 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 10 monclient: dump: 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a637fe640 4 set_mon_vals no callback set 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 5 monclient: authenticate success, global_id 234309 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _renew_subs 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6c0a3340 con 0x7f9a6c0689d0 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _renew_subs 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6c122410 con 0x7f9a6c0689d0 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:44.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6c06f840 con 0x7f9a6c0689d0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9a5c01b400 con 0x7f9a6c0689d0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 0x7f9a5003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(790..790 src has 696..790) ==== 4787+0+0 (secure 0 0 0) 0x7f9a5c04fd00 con 0x7f9a6c0689d0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a7150b640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 0x7f9a5003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.117+0000 7f9a7150b640 10 monclient: get_auth_request con 0x7f9a5003bb60 auth_method 0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.121+0000 7f9a7150b640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 0x7f9a5003e020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9a680ab000 tx=0x7f9a68008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.121+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9a6c0a3340 con 0x7f9a6c0689d0 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.121+0000 7f9a627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:44.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.121+0000 7f9a627fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:44.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:44.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:44.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:44.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9a6c121f30 con 0x7f9a6c0689d0 2024-09-17T15:49:44.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a627fc640 1 -- 172.21.15.86:0/2439676843 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f9a5c010d10 con 0x7f9a6c0689d0 2024-09-17T15:49:44.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:44.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a627fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:44.357 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":204,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 msgr2=0x7f9a5003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 0x7f9a5003e020 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9a680ab000 tx=0x7f9a68008040 comp rx=0 tx=0).stop 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 10 monclient: shutdown 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 msgr2=0x7f9a6c1cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c00c470 tx=0x7f9a5c00c930 comp rx=0 tx=0).stop 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 shutdown_connections 2024-09-17T15:49:44.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a5003bb60 0x7f9a5003e020 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a6c13aff0 0x7f9a6c071db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a6c0694c0 0x7f9a6c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 --2- 172.21.15.86:0/2439676843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a6c0689d0 0x7f9a6c1cfc20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 >> 172.21.15.86:0/2439676843 conn(0x7f9a6c0a6440 msgr2=0x7f9a6c09e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 shutdown_connections 2024-09-17T15:49:44.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:44.357+0000 7f9a73f70640 1 -- 172.21.15.86:0/2439676843 wait complete. 2024-09-17T15:49:44.910 INFO:tasks.rados.rados.0.smithi203.stdout:3618: done (1 left) 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:3619: done (0 left) 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:3620: rollback oid 37 current snap is 387 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 37 to 383 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:3621: rollback oid 8 current snap is 387 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 373 2024-09-17T15:49:44.911 INFO:tasks.rados.rados.0.smithi203.stdout:3622: snap_create 2024-09-17T15:49:44.926 INFO:tasks.rados.rados.0.smithi203.stdout:3620: finishing rollback tid 0 to smithi20328086-37 2024-09-17T15:49:44.926 INFO:tasks.rados.rados.0.smithi203.stdout:3620: finishing rollback tid 1 to smithi20328086-37 2024-09-17T15:49:44.927 INFO:tasks.rados.rados.0.smithi203.stdout:3620: finishing rollback tid 2 to smithi20328086-37 2024-09-17T15:49:44.927 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1851 (ObjNum 1218 snap 368 seq_num 1218) dirty exists 2024-09-17T15:49:44.927 INFO:tasks.rados.rados.0.smithi203.stdout:3621: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:49:44.928 INFO:tasks.rados.rados.0.smithi203.stdout:3621: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:49:44.928 INFO:tasks.rados.rados.0.smithi203.stdout:3621: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:49:44.928 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1642 (ObjNum 1157 snap 357 seq_num 1157) dirty exists 2024-09-17T15:49:46.115 INFO:tasks.rados.rados.0.smithi203.stdout:3620: done (2 left) 2024-09-17T15:49:46.115 INFO:tasks.rados.rados.0.smithi203.stdout:3621: done (1 left) 2024-09-17T15:49:46.115 INFO:tasks.rados.rados.0.smithi203.stdout:3622: done (0 left) 2024-09-17T15:49:46.115 INFO:tasks.rados.rados.0.smithi203.stdout:3623: write oid 9 current snap is 388 2024-09-17T15:49:46.115 INFO:tasks.rados.rados.0.smithi203.stdout:3623: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-17T15:49:46.130 INFO:tasks.rados.rados.0.smithi203.stdout:3623: writing smithi20328086-9 from 528083 to 933662 tid 1 2024-09-17T15:49:46.144 INFO:tasks.rados.rados.0.smithi203.stdout:3623: writing smithi20328086-9 from 1342473 to 1761921 tid 2 2024-09-17T15:49:46.164 INFO:tasks.rados.rados.0.smithi203.stdout:3623: writing smithi20328086-9 from 2248719 to 2982668 tid 3 2024-09-17T15:49:46.164 INFO:tasks.rados.rados.0.smithi203.stdout:3623: finishing write tid 1 to smithi20328086-9 2024-09-17T15:49:46.164 INFO:tasks.rados.rados.0.smithi203.stdout:3623: finishing write tid 2 to smithi20328086-9 2024-09-17T15:49:46.164 INFO:tasks.rados.rados.0.smithi203.stdout:3624: read oid 27 snap -1 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3624: expect deleted 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3625: copy_from oid 35 from oid 1 current snap is 388 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3626: read oid 18 snap -1 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3626: expect (ObjNum 1241 snap 373 seq_num 1241) 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3627: rollback oid 5 current snap is 388 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 385 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3628: read oid 7 snap -1 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3628: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3629: write (excl) oid 43 current snap is 388 2024-09-17T15:49:46.165 INFO:tasks.rados.rados.0.smithi203.stdout:3629: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-17T15:49:46.182 INFO:tasks.rados.rados.0.smithi203.stdout:3629: writing smithi20328086-43 from 544890 to 1142071 tid 1 2024-09-17T15:49:46.199 INFO:tasks.rados.rados.0.smithi203.stdout:3629: writing smithi20328086-43 from 1917308 to 2480414 tid 2 2024-09-17T15:49:46.208 INFO:tasks.rados.rados.0.smithi203.stdout:3629: writing smithi20328086-43 from 3076142 to 3344890 tid 3 2024-09-17T15:49:46.209 INFO:tasks.rados.rados.0.smithi203.stdout:3630: snap_remove snap 384 2024-09-17T15:49:47.132 INFO:tasks.rados.rados.0.smithi203.stdout:3630: done (7 left) 2024-09-17T15:49:47.132 INFO:tasks.rados.rados.0.smithi203.stdout:3631: snap_create 2024-09-17T15:49:47.132 INFO:tasks.rados.rados.0.smithi203.stdout:3627: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:49:47.132 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 0 (ObjNum 33 snap 0 seq_num 3606556592) dirty dne 2024-09-17T15:49:47.133 INFO:tasks.rados.rados.0.smithi203.stdout:3626: expect (ObjNum 1241 snap 373 seq_num 1241) 2024-09-17T15:49:47.215 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1924056 2024-09-17T15:49:47.276 INFO:tasks.rados.rados.0.smithi203.stdout:3628: expect (ObjNum 1053 snap 327 seq_num 1053) 2024-09-17T15:49:47.276 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:47.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:47.387 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:47.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: get_monmap_and_config 2024-09-17T15:49:47.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: build_initial_monmap 2024-09-17T15:49:47.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: monmap: 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:47.505553+0000 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:47.505553+0000 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:47.502 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:47.503 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:47.503 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:47.503 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:47.503 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 Processor -- start 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 -- start start 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: init 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74f80a5070 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:47.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f74f80a5b60 0x7f74f809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-c con 0x7f74f80a5b60 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-b con 0x7f74f813d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74f80a6f40 con 0x7f74f80a5070 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f74f80a65e0 con 0x7f74f80a5b60 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f74f809c2a0 con 0x7f74f813d280 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74ff1ad640 10 monclient(hunting): authenticate will time out at 2320.471049s 2024-09-17T15:49:47.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59800/0 (socket says 172.21.15.86:59800) 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 -- 172.21.15.86:0/4220826955 learned_addr learned my addr 172.21.15.86:0/4220826955 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): get_auth_request con 0x7f74f813d280 auth_method 0 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74f7fff640 1 --2- 172.21.15.86:0/4220826955 >> v2:172.21.15.203:3300/0 conn(0x7f74f80a5b60 0x7f74f809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74f7fff640 10 monclient(hunting): get_auth_request con 0x7f74f80a5b60 auth_method 0 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient(hunting): handle_auth_done global_id 164975 payload 274 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 -- 172.21.15.86:0/4220826955 >> v2:172.21.15.203:3300/0 conn(0x7f74f80a5b60 msgr2=0x7f74f809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 --2- 172.21.15.86:0/4220826955 >> v2:172.21.15.203:3300/0 conn(0x7f74f80a5b60 0x7f74f809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 -- 172.21.15.86:0/4220826955 >> v1:172.21.15.203:6789/0 conn(0x7f74f80a5070 legacy=0x7f74f80a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient: _finish_hunting 0 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 monclient: found mon.noname-b 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:47.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.501+0000 7f74fd74a640 1 -- 172.21.15.86:0/4220826955 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f809c580 con 0x7f74f813d280 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.505+0000 7f74fd74a640 1 -- 172.21.15.86:0/4220826955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 msgr2=0x7f74f813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.505+0000 7f74fd74a640 1 --2- 172.21.15.86:0/4220826955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e50 tx=0x7f74f001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa99486f6063d1eb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 1 -- 172.21.15.86:0/4220826955 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f00a5530 con 0x7f74f813d280 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: dump: 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 1 -- 172.21.15.86:0/4220826955 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74f00a56d0 con 0x7f74f813d280 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 1 -- 172.21.15.86:0/4220826955 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f00a5aa0 con 0x7f74f813d280 2024-09-17T15:49:47.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 5 monclient: authenticate success, global_id 164975 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 4 set_mon_vals no callback set 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f77fe640 10 monclient: dump: 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:47.518 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: shutdown 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- 172.21.15.86:0/4220826955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 msgr2=0x7f74f813f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/4220826955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74f0009e50 tx=0x7f74f001c8f0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- 172.21.15.86:0/4220826955 shutdown_connections 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/4220826955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f813d280 0x7f74f813f670 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/4220826955 >> v2:172.21.15.203:3300/0 conn(0x7f74f80a5b60 0x7f74f809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- 172.21.15.86:0/4220826955 >> 172.21.15.86:0/4220826955 conn(0x7f74f80742a0 msgr2=0x7f74f80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- 172.21.15.86:0/4220826955 shutdown_connections 2024-09-17T15:49:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- 172.21.15.86:0/4220826955 wait complete. 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: build_initial_monmap 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: monmap: 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.520 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 Processor -- start 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- start start 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: init 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74f80a5070 0x7f74f8121020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-a con 0x7f74f80a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-c con 0x7f74f80a5b60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f813d280 0x7f74f813b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): picked mon.noname-b con 0x7f74f813d280 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:47.521 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f74f8142760 con 0x7f74f80a5070 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f74f81425e0 con 0x7f74f813d280 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f74f81428e0 con 0x7f74f80a5b60 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74ff1ad640 10 monclient(hunting): authenticate will time out at 2320.478449s 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59812/0 (socket says 172.21.15.86:59812) 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74f80a5070 0x7f74f8121020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 -- 172.21.15.86:0/1462001778 learned_addr learned my addr 172.21.15.86:0/1462001778 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): get_auth_request con 0x7f74f80a5b60 auth_method 0 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:47.522 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): get_auth_request con 0x7f74f80a5070 auth_method 0 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:47.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fd74a640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f813d280 0x7f74f813b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fcf49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fd74a640 10 monclient(hunting): get_auth_request con 0x7f74f813d280 auth_method 0 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fd74a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fd74a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74fd74a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient(hunting): handle_auth_done global_id 164978 payload 824 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 -- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f813d280 msgr2=0x7f74f813b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f813d280 0x7f74f813b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 -- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74f80a5070 msgr2=0x7f74f8121020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74f80a5070 0x7f74f8121020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 monclient: found mon.noname-c 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:47.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.509+0000 7f74f7fff640 1 -- 172.21.15.86:0/1462001778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f80a1860 con 0x7f74f80a5b60 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f7fff640 1 -- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 msgr2=0x7f74f81d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f7fff640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f74e800d800 tx=0x7f74e800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8018020 con 0x7f74f80a5b60 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: dump: 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:47.525 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e8009850 con 0x7f74f80a5b60 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e8009c00 con 0x7f74f80a5b60 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 10 monclient: dump: 2024-09-17T15:49:47.526 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:47.527 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 5 monclient: authenticate success, global_id 164978 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f67fc640 4 set_mon_vals no callback set 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _renew_subs 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f8122260 con 0x7f74f80a5b60 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _renew_subs 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f81c78e0 con 0x7f74f80a5b60 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f8122900 con 0x7f74f80a5b60 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f74e801b020 con 0x7f74f80a5b60 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 0x7f74d803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(793..793 src has 696..793) ==== 4799+0+0 (secure 0 0 0) 0x7f74e8050550 con 0x7f74f80a5b60 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74fcf49640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 0x7f74d803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.513+0000 7f74fcf49640 10 monclient: get_auth_request con 0x7f74d803bbb0 auth_method 0 2024-09-17T15:49:47.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.517+0000 7f74fcf49640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 0x7f74d803e070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f74e0007fd0 tx=0x7f74e0007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:47.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.517+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f74e8022030 con 0x7f74f80a5b60 2024-09-17T15:49:47.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.517+0000 7f74f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:47.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.517+0000 7f74f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:47.574 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2761820 2024-09-17T15:49:47.599 INFO:tasks.rados.rados.0.smithi203.stdout:3623: finishing write tid 3 to smithi20328086-9 2024-09-17T15:49:47.599 INFO:tasks.rados.rados.0.smithi203.stdout:3623: finishing write tid 4 to smithi20328086-9 2024-09-17T15:49:47.599 INFO:tasks.rados.rados.0.smithi203.stdout:3623: finishing write tid 5 to smithi20328086-9 2024-09-17T15:49:47.599 INFO:tasks.rados.rados.0.smithi203.stdout:3623: oid 9 updating version 0 to 1552 2024-09-17T15:49:47.599 INFO:tasks.rados.rados.0.smithi203.stdout:3623: oid 9 version 1552 is already newer than 1551 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3623: oid 9 version 1552 is already newer than 1550 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3623: oid 9 updating version 1552 to 1553 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1553 (ObjNum 1269 snap 388 seq_num 1269) dirty exists 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3623: left oid 9 (ObjNum 1269 snap 388 seq_num 1269) 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: finishing write tid 1 to smithi20328086-43 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: finishing write tid 2 to smithi20328086-43 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: finishing write tid 3 to smithi20328086-43 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: finishing write tid 4 to smithi20328086-43 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: finishing write tid 5 to smithi20328086-43 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: oid 43 updating version 0 to 700 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: oid 43 version 700 is already newer than 699 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: oid 43 updating version 700 to 701 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:3629: oid 43 updating version 701 to 702 2024-09-17T15:49:47.600 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 702 (ObjNum 1270 snap 388 seq_num 1270) dirty exists 2024-09-17T15:49:47.601 INFO:tasks.rados.rados.0.smithi203.stdout:3629: left oid 43 (ObjNum 1270 snap 388 seq_num 1270) 2024-09-17T15:49:47.601 INFO:tasks.rados.rados.0.smithi203.stdout:3625: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:49:47.601 INFO:tasks.rados.rados.0.smithi203.stdout:3625: finishing copy_from to smithi20328086-35 2024-09-17T15:49:47.601 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1668 (ObjNum 1257 snap 380 seq_num 1257) dirty exists 2024-09-17T15:49:47.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:47.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:47.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:47.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f74f809cf60 con 0x7f74f80a5b60 2024-09-17T15:49:47.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74f57fa640 1 -- 172.21.15.86:0/1462001778 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f74f80a1860 con 0x7f74f80a5b60 2024-09-17T15:49:47.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:47.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:47.753 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":204,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 msgr2=0x7f74d803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 0x7f74d803e070 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f74e0007fd0 tx=0x7f74e0007e40 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 10 monclient: shutdown 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 msgr2=0x7f74f81d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f74e800d800 tx=0x7f74e800dcc0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 shutdown_connections 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f74d803bbb0 0x7f74d803e070 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74f813d280 0x7f74f813b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74f80a5b60 0x7f74f81d0040 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 --2- 172.21.15.86:0/1462001778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74f80a5070 0x7f74f8121020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:47.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 >> 172.21.15.86:0/1462001778 conn(0x7f74f80742a0 msgr2=0x7f74f80a3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:47.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 shutdown_connections 2024-09-17T15:49:47.756 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:47.753+0000 7f74ff1ad640 1 -- 172.21.15.86:0/1462001778 wait complete. 2024-09-17T15:49:48.129 INFO:tasks.rados.rados.0.smithi203.stdout:3623: done (7 left) 2024-09-17T15:49:48.129 INFO:tasks.rados.rados.0.smithi203.stdout:3624: done (6 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3625: done (5 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3626: done (4 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3627: done (3 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3628: done (2 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3629: done (1 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3631: done (0 left) 2024-09-17T15:49:48.130 INFO:tasks.rados.rados.0.smithi203.stdout:3632: delete oid 37 current snap is 389 2024-09-17T15:49:48.140 INFO:tasks.rados.rados.0.smithi203.stdout:3632: done (0 left) 2024-09-17T15:49:48.140 INFO:tasks.rados.rados.0.smithi203.stdout:3633: read oid 50 snap -1 2024-09-17T15:49:48.140 INFO:tasks.rados.rados.0.smithi203.stdout:3633: expect deleted 2024-09-17T15:49:48.140 INFO:tasks.rados.rados.0.smithi203.stdout:3634: delete oid 14 current snap is 389 2024-09-17T15:49:48.149 INFO:tasks.rados.rados.0.smithi203.stdout:3633: done (1 left) 2024-09-17T15:49:48.149 INFO:tasks.rados.rados.0.smithi203.stdout:3634: done (0 left) 2024-09-17T15:49:48.149 INFO:tasks.rados.rados.0.smithi203.stdout:3635: copy_from oid 12 from oid 28 current snap is 389 2024-09-17T15:49:48.149 INFO:tasks.rados.rados.0.smithi203.stdout:3636: rollback oid 38 current snap is 389 2024-09-17T15:49:48.150 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 38 to 388 2024-09-17T15:49:48.150 INFO:tasks.rados.rados.0.smithi203.stdout:3637: snap_create 2024-09-17T15:49:48.167 INFO:tasks.rados.rados.0.smithi203.stdout:3636: finishing rollback tid 0 to smithi20328086-38 2024-09-17T15:49:48.168 INFO:tasks.rados.rados.0.smithi203.stdout:3636: finishing rollback tid 1 to smithi20328086-38 2024-09-17T15:49:48.168 INFO:tasks.rados.rados.0.smithi203.stdout:3636: finishing rollback tid 2 to smithi20328086-38 2024-09-17T15:49:48.168 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1394 (ObjNum 1262 snap 385 seq_num 1262) dirty exists 2024-09-17T15:49:48.172 INFO:tasks.rados.rados.0.smithi203.stdout:3635: finishing copy_from racing read to smithi20328086-12 2024-09-17T15:49:48.175 INFO:tasks.rados.rados.0.smithi203.stdout:3635: finishing copy_from to smithi20328086-12 2024-09-17T15:49:48.175 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 12 v 1799 (ObjNum 1227 snap 368 seq_num 1227) dirty exists 2024-09-17T15:49:49.151 INFO:tasks.rados.rados.0.smithi203.stdout:3635: done (2 left) 2024-09-17T15:49:49.151 INFO:tasks.rados.rados.0.smithi203.stdout:3636: done (1 left) 2024-09-17T15:49:49.151 INFO:tasks.rados.rados.0.smithi203.stdout:3637: done (0 left) 2024-09-17T15:49:49.151 INFO:tasks.rados.rados.0.smithi203.stdout:3638: snap_create 2024-09-17T15:49:50.152 INFO:tasks.rados.rados.0.smithi203.stdout:3638: done (0 left) 2024-09-17T15:49:50.152 INFO:tasks.rados.rados.0.smithi203.stdout:3639: read oid 33 snap -1 2024-09-17T15:49:50.152 INFO:tasks.rados.rados.0.smithi203.stdout:3639: expect (ObjNum 1239 snap 373 seq_num 1239) 2024-09-17T15:49:50.152 INFO:tasks.rados.rados.0.smithi203.stdout:3640: write oid 15 current snap is 391 2024-09-17T15:49:50.153 INFO:tasks.rados.rados.0.smithi203.stdout:3640: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2024-09-17T15:49:50.169 INFO:tasks.rados.rados.0.smithi203.stdout:3640: writing smithi20328086-15 from 595311 to 1018239 tid 1 2024-09-17T15:49:50.191 INFO:tasks.rados.rados.0.smithi203.stdout:3640: writing smithi20328086-15 from 1576401 to 2286834 tid 2 2024-09-17T15:49:50.208 INFO:tasks.rados.rados.0.smithi203.stdout:3640: writing smithi20328086-15 from 3042058 to 3395311 tid 3 2024-09-17T15:49:50.208 INFO:tasks.rados.rados.0.smithi203.stdout:3641: delete oid 42 current snap is 391 2024-09-17T15:49:50.208 INFO:tasks.rados.rados.0.smithi203.stdout:3639: expect (ObjNum 1239 snap 373 seq_num 1239) 2024-09-17T15:49:50.264 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: finishing write tid 1 to smithi20328086-15 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: finishing write tid 2 to smithi20328086-15 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: finishing write tid 3 to smithi20328086-15 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: finishing write tid 4 to smithi20328086-15 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: finishing write tid 5 to smithi20328086-15 2024-09-17T15:49:50.342 INFO:tasks.rados.rados.0.smithi203.stdout:3640: oid 15 updating version 0 to 1801 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3640: oid 15 updating version 1801 to 1802 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3640: oid 15 updating version 1802 to 1803 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3640: oid 15 version 1803 is already newer than 1800 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1803 (ObjNum 1273 snap 391 seq_num 1273) dirty exists 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3640: left oid 15 (ObjNum 1273 snap 391 seq_num 1273) 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3639: done (2 left) 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3640: done (1 left) 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3641: done (0 left) 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3642: copy_from oid 4 from oid 27 current snap is 391 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3643: write (excl) oid 30 current snap is 391 2024-09-17T15:49:50.343 INFO:tasks.rados.rados.0.smithi203.stdout:3643: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-17T15:49:50.359 INFO:tasks.rados.rados.0.smithi203.stdout:3643: writing smithi20328086-30 from 628925 to 1202351 tid 1 2024-09-17T15:49:50.375 INFO:tasks.rados.rados.0.smithi203.stdout:3643: writing smithi20328086-30 from 1693365 to 2407467 tid 2 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3643: writing smithi20328086-30 from 3096904 to 3428925 tid 3 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3644: snap_create 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3642: finishing copy_from to smithi20328086-4 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3642: got expected ENOENT (src dne) 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3642: finishing copy_from racing read to smithi20328086-4 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3643: finishing write tid 1 to smithi20328086-30 2024-09-17T15:49:50.390 INFO:tasks.rados.rados.0.smithi203.stdout:3643: finishing write tid 2 to smithi20328086-30 2024-09-17T15:49:50.395 INFO:tasks.rados.rados.0.smithi203.stdout:3643: finishing write tid 3 to smithi20328086-30 2024-09-17T15:49:50.395 INFO:tasks.rados.rados.0.smithi203.stdout:3643: finishing write tid 4 to smithi20328086-30 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: finishing write tid 5 to smithi20328086-30 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: oid 30 updating version 0 to 1804 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: oid 30 updating version 1804 to 1805 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: oid 30 updating version 1805 to 1807 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: oid 30 version 1807 is already newer than 1806 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 30 v 1807 (ObjNum 1275 snap 391 seq_num 1275) dirty exists 2024-09-17T15:49:50.396 INFO:tasks.rados.rados.0.smithi203.stdout:3643: left oid 30 (ObjNum 1275 snap 391 seq_num 1275) 2024-09-17T15:49:50.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:50.851 INFO:tasks.rados.rados.0.smithi203.stdout:3642: done (2 left) 2024-09-17T15:49:50.851 INFO:tasks.rados.rados.0.smithi203.stdout:3643: done (1 left) 2024-09-17T15:49:50.851 INFO:tasks.rados.rados.0.smithi203.stdout:3644: done (0 left) 2024-09-17T15:49:50.851 INFO:tasks.rados.rados.0.smithi203.stdout:3645: write oid 35 current snap is 392 2024-09-17T15:49:50.851 INFO:tasks.rados.rados.0.smithi203.stdout:3645: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-09-17T15:49:50.867 INFO:tasks.rados.rados.0.smithi203.stdout:3645: writing smithi20328086-35 from 645732 to 1294407 tid 1 2024-09-17T15:49:50.880 INFO:tasks.rados.rados.0.smithi203.stdout:3645: writing smithi20328086-35 from 1751847 to 2325960 tid 2 2024-09-17T15:49:50.894 INFO:tasks.rados.rados.0.smithi203.stdout:3645: writing smithi20328086-35 from 3124327 to 3445732 tid 3 2024-09-17T15:49:50.894 INFO:tasks.rados.rados.0.smithi203.stdout:3646: snap_remove snap 373 2024-09-17T15:49:50.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: get_monmap_and_config 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: build_initial_monmap 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: monmap: 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:50.910025+0000 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:50.910025+0000 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:50.906 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:50.907 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:50.907 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 Processor -- start 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 -- start start 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: init 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 0x7f6bec09ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f6bec0689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6bec068db0 0x7f6bec069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f6bec068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:50.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bec134a40 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bec09d770 con 0x7f6bec134a40 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6bec09ed00 con 0x7f6bec068db0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6bec09efe0 con 0x7f6bec0689d0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf2e8d640 10 monclient(hunting): authenticate will time out at 2323.875371s 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf142a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6bec134a40 0x7f6bec134ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48444/0 (socket says 172.21.15.86:48444) 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6bec068db0 0x7f6bec069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf142a640 1 -- 172.21.15.86:0/2172077889 learned_addr learned my addr 172.21.15.86:0/2172077889 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 0x7f6bec09ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): get_auth_request con 0x7f6bec068db0 auth_method 0 2024-09-17T15:49:50.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): get_auth_request con 0x7f6bec0689d0 auth_method 0 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 10 monclient(hunting): handle_auth_done global_id 234315 payload 274 2024-09-17T15:49:50.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 1 -- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 msgr2=0x7f6bec09ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.905+0000 7f6be3fff640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 0x7f6bec09ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 1 -- 172.21.15.86:0/2172077889 >> v1:172.21.15.86:6789/0 conn(0x7f6bec134a40 legacy=0x7f6bec134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 1 monclient: found mon.noname-a 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6be3fff640 1 -- 172.21.15.86:0/2172077889 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec0a04e0 con 0x7f6bec068db0 2024-09-17T15:49:50.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.909+0000 7f6bf0c29640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 0x7f6bec09ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be3fff640 1 -- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec068db0 msgr2=0x7f6bec069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be3fff640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec068db0 0x7f6bec069190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6be80097c0 tx=0x7f6be801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c086436c5b62fd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 1 -- 172.21.15.86:0/2172077889 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80b2070 con 0x7f6bec068db0 2024-09-17T15:49:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:50.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: dump: 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 1 -- 172.21.15.86:0/2172077889 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be80ad640 con 0x7f6bec068db0 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 1 -- 172.21.15.86:0/2172077889 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be80ada10 con 0x7f6bec068db0 2024-09-17T15:49:50.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 5 monclient: authenticate success, global_id 234315 2024-09-17T15:49:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:50.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 4 set_mon_vals no callback set 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be37fe640 10 monclient: dump: 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: shutdown 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec068db0 msgr2=0x7f6bec069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec068db0 0x7f6bec069190 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6be80097c0 tx=0x7f6be801c3f0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/2172077889 shutdown_connections 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec068db0 0x7f6bec069190 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/2172077889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec0689d0 0x7f6bec09ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/2172077889 >> 172.21.15.86:0/2172077889 conn(0x7f6bec0a6440 msgr2=0x7f6bec0a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/2172077889 shutdown_connections 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/2172077889 wait complete. 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: build_initial_monmap 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: monmap: 2024-09-17T15:49:50.921 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 Processor -- start 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- start start 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: init 2024-09-17T15:49:50.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 0x7f6bec1cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-a con 0x7f6bec0689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 0x7f6bec133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-b con 0x7f6bec068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): picked mon.noname-c con 0x7f6bec134a40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec13db00 con 0x7f6bec0689d0 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6bec13d980 con 0x7f6bec068db0 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:50.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6bec13dc80 con 0x7f6bec134a40 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf2e8d640 10 monclient(hunting): authenticate will time out at 2323.882558s 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 0x7f6bec1cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59586/0 (socket says 172.21.15.86:59586) 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 -- 172.21.15.86:0/3376012039 learned_addr learned my addr 172.21.15.86:0/3376012039 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): get_auth_request con 0x7f6bec134a40 auth_method 0 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): get_auth_request con 0x7f6bec0689d0 auth_method 0 2024-09-17T15:49:50.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be3fff640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 0x7f6bec133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient(hunting): handle_auth_done global_id 164984 payload 824 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 -- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 msgr2=0x7f6bec133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:50.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 0x7f6bec133ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 -- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 msgr2=0x7f6bec1cdfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 0x7f6bec1cdfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient: _finish_hunting 0 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 monclient: found mon.noname-c 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf142a640 1 -- 172.21.15.86:0/3376012039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bec1ceea0 con 0x7f6bec134a40 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be3fff640 10 monclient: get_auth_request con 0x7f6bec068db0 auth_method 0 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6be3fff640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 0x7f6bec133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:50.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.913+0000 7f6bf0c29640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 0x7f6bec1cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf142a640 1 -- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 msgr2=0x7f6bec139ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf142a640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6be400caf0 tx=0x7f6be400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be4012400 con 0x7f6bec134a40 2024-09-17T15:49:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:50.928 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: dump: 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be40125c0 con 0x7f6bec134a40 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 5 monclient: authenticate success, global_id 164984 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _renew_subs 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bec11c8c0 con 0x7f6bec134a40 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be40129b0 con 0x7f6bec134a40 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 10 monclient: dump: 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _renew_subs 2024-09-17T15:49:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bec0a2930 con 0x7f6bec134a40 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be2ffd640 4 set_mon_vals no callback set 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bec09d460 con 0x7f6bec134a40 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6be4012bd0 con 0x7f6bec134a40 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 0x7f6bd803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(797..797 src has 696..797) ==== 4771+0+0 (secure 0 0 0) 0x7f6be40506d0 con 0x7f6bec134a40 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf0c29640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 0x7f6bd803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.917+0000 7f6bf0c29640 10 monclient: get_auth_request con 0x7f6bd803bb60 auth_method 0 2024-09-17T15:49:50.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.921+0000 7f6bf0c29640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 0x7f6bd803e020 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009580 tx=0x7f6bdc009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.921+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6be401ac60 con 0x7f6bec134a40 2024-09-17T15:49:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.921+0000 7f6be17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:50.921+0000 7f6be17fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:51.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:51.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:51.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:51.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6bec09ffa0 con 0x7f6bec134a40 2024-09-17T15:49:51.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6be17fa640 1 -- 172.21.15.86:0/3376012039 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6be404f0d0 con 0x7f6bec134a40 2024-09-17T15:49:51.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6be17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:51.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6be17fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:51.161 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:51.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 msgr2=0x7f6bd803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 0x7f6bd803e020 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc009580 tx=0x7f6bdc009190 comp rx=0 tx=0).stop 2024-09-17T15:49:51.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 10 monclient: shutdown 2024-09-17T15:49:51.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 msgr2=0x7f6bec139ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6be400caf0 tx=0x7f6be400a060 comp rx=0 tx=0).stop 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 shutdown_connections 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bd803bb60 0x7f6bd803e020 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6bec134a40 0x7f6bec139ca0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6bec068db0 0x7f6bec133ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 --2- 172.21.15.86:0/3376012039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6bec0689d0 0x7f6bec1cdfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 >> 172.21.15.86:0/3376012039 conn(0x7f6bec0a6440 msgr2=0x7f6bec074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 shutdown_connections 2024-09-17T15:49:51.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.161+0000 7f6bf2e8d640 1 -- 172.21.15.86:0/3376012039 wait complete. 2024-09-17T15:49:51.180 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:49:51.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:49:51.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: get_monmap_and_config 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: monmap: 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:51.296176+0000 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:51.296176+0000 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:51.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:51.293 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:51.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 Processor -- start 2024-09-17T15:49:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 -- start start 2024-09-17T15:49:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: init 2024-09-17T15:49:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b240a0c70 0x7f7b240a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-a con 0x7f7b240a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b240a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 0x7f7b2413df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-c con 0x7f7b24136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b240a2130 con 0x7f7b240a1590 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7b240a2410 con 0x7f7b240a0c70 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7b240a26f0 con 0x7f7b24136160 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b2a434640 10 monclient(hunting): authenticate will time out at 2324.261749s 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b240a0c70 0x7f7b240a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b240a0c70 0x7f7b240a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57584/0 (socket says 172.21.15.86:57584) 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 -- 172.21.15.86:0/2226652912 learned_addr learned my addr 172.21.15.86:0/2226652912 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): get_auth_request con 0x7f7b240a0c70 auth_method 0 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b289d1640 1 --2- 172.21.15.86:0/2226652912 >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 0x7f7b2413df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient(hunting): handle_auth_done global_id 234321 payload 274 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 -- 172.21.15.86:0/2226652912 >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 msgr2=0x7f7b2413df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 --2- 172.21.15.86:0/2226652912 >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 0x7f7b2413df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.297 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 -- 172.21.15.86:0/2226652912 >> v1:172.21.15.203:6789/0 conn(0x7f7b240a1590 legacy=0x7f7b24135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 monclient: found mon.noname-a 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b23fff640 1 -- 172.21.15.86:0/2226652912 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2409f9f0 con 0x7f7b240a0c70 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b289d1640 10 monclient: get_auth_request con 0x7f7b24136160 auth_method 0 2024-09-17T15:49:51.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.293+0000 7f7b289d1640 1 --2- 172.21.15.86:0/2226652912 >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 0x7f7b2413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b23fff640 1 -- 172.21.15.86:0/2226652912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b240a0c70 msgr2=0x7f7b240a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b23fff640 1 --2- 172.21.15.86:0/2226652912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b240a0c70 0x7f7b240a1050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b2406f840 tx=0x7f7b1801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede9c0ede34f54f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 1 -- 172.21.15.86:0/2226652912 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b180b1070 con 0x7f7b240a0c70 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: dump: 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.300 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 1 -- 172.21.15.86:0/2226652912 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b180ac620 con 0x7f7b240a0c70 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 5 monclient: authenticate success, global_id 234321 2024-09-17T15:49:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 1 -- 172.21.15.86:0/2226652912 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b180ac9f0 con 0x7f7b240a0c70 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 4 set_mon_vals no callback set 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b22ffd640 10 monclient: dump: 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.302 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 10 monclient: shutdown 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 -- 172.21.15.86:0/2226652912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b240a0c70 msgr2=0x7f7b240a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 --2- 172.21.15.86:0/2226652912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b240a0c70 0x7f7b240a1050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7b2406f840 tx=0x7f7b1801c3e0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 -- 172.21.15.86:0/2226652912 shutdown_connections 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 --2- 172.21.15.86:0/2226652912 >> v2:172.21.15.203:3300/0 conn(0x7f7b24136160 0x7f7b2413df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 --2- 172.21.15.86:0/2226652912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b240a0c70 0x7f7b240a1050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 -- 172.21.15.86:0/2226652912 >> 172.21.15.86:0/2226652912 conn(0x7f7b240a2f40 msgr2=0x7f7b240a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 -- 172.21.15.86:0/2226652912 shutdown_connections 2024-09-17T15:49:51.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.297+0000 7f7b2a434640 1 -- 172.21.15.86:0/2226652912 wait complete. 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient: monmap: 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 Processor -- start 2024-09-17T15:49:51.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 -- start start 2024-09-17T15:49:51.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient: init 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 0x7f7b2412ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-c con 0x7f7b240a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b240a1590 0x7f7b24136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-b con 0x7f7b240a1590 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 0x7f7b24137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): picked mon.noname-a con 0x7f7b24136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b240a00d0 con 0x7f7b24136160 2024-09-17T15:49:51.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2409ff50 con 0x7f7b240a1590 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7b24140c90 con 0x7f7b240a0c70 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b2a434640 10 monclient(hunting): authenticate will time out at 2324.269154s 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 0x7f7b2412ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 0x7f7b24137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 0x7f7b2412ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59602/0 (socket says 172.21.15.86:59602) 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 1 -- 172.21.15.86:0/4021445056 learned_addr learned my addr 172.21.15.86:0/4021445056 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): get_auth_request con 0x7f7b240a0c70 auth_method 0 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): get_auth_request con 0x7f7b24136160 auth_method 0 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient(hunting): handle_auth_done global_id 234324 payload 824 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 msgr2=0x7f7b2412ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b237fe640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b240a1590 0x7f7b24136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 0x7f7b2412ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b240a1590 msgr2=0x7f7b24136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b240a1590 0x7f7b24136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 monclient: found mon.noname-a 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b24138680 con 0x7f7b24136160 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b23fff640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b240a0c70 0x7f7b2412ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b237fe640 10 monclient: get_auth_request con 0x7f7b240a1590 auth_method 0 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b237fe640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b240a1590 0x7f7b24136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 msgr2=0x7f7b24137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.301+0000 7f7b289d1640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 0x7f7b24137430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00cdd0 tx=0x7f7b1c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c01a070 con 0x7f7b24136160 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: dump: 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b1c00d880 con 0x7f7b24136160 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c00dc70 con 0x7f7b24136160 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 10 monclient: dump: 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.315 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b227fc640 4 set_mon_vals no callback set 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 5 monclient: authenticate success, global_id 234324 2024-09-17T15:49:51.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 monclient: _renew_subs 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2406fb00 con 0x7f7b24136160 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 monclient: _renew_subs 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2413a700 con 0x7f7b24136160 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 0x7f7b24137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b240a1050 con 0x7f7b2412f4d0 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 0x7f7b24137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient: get_auth_request con 0x7f7b2412f4d0 auth_method 0 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7b1c01d070 con 0x7f7b24136160 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 10 monclient(hunting): handle_auth_done global_id 234327 payload 824 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b2413b470 con 0x7f7b2412f4d0 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b1003bbb0 0x7f7b1003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(797..797 src has 696..797) ==== 4771+0+0 (secure 0 0 0) 0x7f7b1c050610 con 0x7f7b24136160 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b237fe640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b1003bbb0 0x7f7b1003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b237fe640 10 monclient: get_auth_request con 0x7f7b1003bbb0 auth_method 0 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 msgr2=0x7f7b24137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.305+0000 7f7b23fff640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 0x7f7b24137970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cc80 tx=0x7f7b18019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b237fe640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b1003bbb0 0x7f7b1003e070 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7b140016e0 tx=0x7f7b14009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b180bd070 con 0x7f7b2412f4d0 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 10 monclient: dump: 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.319 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b180c2450 con 0x7f7b2412f4d0 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 msgr2=0x7f7b24137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 0x7f7b24137970 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cc80 tx=0x7f7b18019180 comp rx=0 tx=0).stop 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 msgr2=0x7f7b24137970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.309+0000 7f7b20ff9640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b2412f4d0 0x7f7b24137970 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801cc80 tx=0x7f7b18019180 comp rx=0 tx=0).stop 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b2a434640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b2a434640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 0x7f7b24071220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b2a434640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7b241382c0 con 0x7f7b24137eb0 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 0x7f7b24071220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient: get_auth_request con 0x7f7b24137eb0 auth_method 0 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 10 monclient(hunting): handle_auth_done global_id 234330 payload 824 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 1 -- 172.21.15.86:0/4021445056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b2413a220 con 0x7f7b24137eb0 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 msgr2=0x7f7b24071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b23fff640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 0x7f7b24071220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b2406ff90 tx=0x7f7b18008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b180b1040 con 0x7f7b24137eb0 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 10 monclient: dump: 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.323 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f7b180b6440 con 0x7f7b24137eb0 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 msgr2=0x7f7b24071220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 0x7f7b24071220 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b2406ff90 tx=0x7f7b18008040 comp rx=0 tx=0).stop 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 msgr2=0x7f7b24071220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.317+0000 7f7b20ff9640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24137eb0 0x7f7b24071220 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7b2406ff90 tx=0x7f7b18008040 comp rx=0 tx=0).stop 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 120512, 2024-09-17T15:49:51.325 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.326 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.327 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:51.328 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:51.329 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.330 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:51.331 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.332 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:51.333 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.334 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.335 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:51.336 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.337 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:51.338 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b1003bbb0 msgr2=0x7f7b1003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7b1003bbb0 0x7f7b1003e070 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7b140016e0 tx=0x7f7b14009040 comp rx=0 tx=0).stop 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 10 monclient: shutdown 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 msgr2=0x7f7b24137430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 --2- 172.21.15.86:0/4021445056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b24136160 0x7f7b24137430 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00cdd0 tx=0x7f7b1c00a710 comp rx=0 tx=0).stop 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 reap_dead start 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b289d1640 1 -- 172.21.15.86:0/4021445056 reap_dead start 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 shutdown_connections 2024-09-17T15:49:51.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 >> 172.21.15.86:0/4021445056 conn(0x7f7b240a2f40 msgr2=0x7f7b240a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 shutdown_connections 2024-09-17T15:49:51.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.321+0000 7f7b2a434640 1 -- 172.21.15.86:0/4021445056 wait complete. 2024-09-17T15:49:51.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:51.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:49:51.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:51.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.445+0000 7f7e814c1640 10 monclient: get_monmap_and_config 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.445+0000 7f7e814c1640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.445+0000 7f7e814c1640 10 monclient: monmap: 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:51.451880+0000 2024-09-17T15:49:51.448 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:51.451880+0000 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:51.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 Processor -- start 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 -- start start 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient: init 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e7c0a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 0x7f7e7c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7e7c0a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7e7c13b450 0x7f7e7c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7e7c13b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7c0a2130 con 0x7f7e7c0a0c70 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7e7c0a2410 con 0x7f7e7c13b450 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c0a26f0 con 0x7f7e7c0a1760 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e814c1640 10 monclient(hunting): authenticate will time out at 2324.417288s 2024-09-17T15:49:51.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7e7c0a0c70 0x7f7e7c0a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48446/0 (socket says 172.21.15.86:48446) 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 0x7f7e7c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7affd640 1 -- 172.21.15.86:0/3205032590 learned_addr learned my addr 172.21.15.86:0/3205032590 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): get_auth_request con 0x7f7e7c0a1760 auth_method 0 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7b7fe640 1 --2- 172.21.15.86:0/3205032590 >> v2:172.21.15.203:3300/0 conn(0x7f7e7c13b450 0x7f7e7c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7b7fe640 10 monclient(hunting): get_auth_request con 0x7f7e7c13b450 auth_method 0 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient(hunting): handle_auth_done global_id 164990 payload 274 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 -- 172.21.15.86:0/3205032590 >> v2:172.21.15.203:3300/0 conn(0x7f7e7c13b450 msgr2=0x7f7e7c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 --2- 172.21.15.86:0/3205032590 >> v2:172.21.15.203:3300/0 conn(0x7f7e7c13b450 0x7f7e7c13d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 -- 172.21.15.86:0/3205032590 >> v1:172.21.15.86:6789/0 conn(0x7f7e7c0a0c70 legacy=0x7f7e7c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 monclient: found mon.noname-b 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:51.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.449+0000 7f7e727fc640 1 -- 172.21.15.86:0/3205032590 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c09f9f0 con 0x7f7e7c0a1760 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e727fc640 1 -- 172.21.15.86:0/3205032590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 msgr2=0x7f7e7c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e727fc640 1 --2- 172.21.15.86:0/3205032590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 0x7f7e7c13af10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7e640097c0 tx=0x7f7e6401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7ca589c6bd92093 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 1 -- 172.21.15.86:0/3205032590 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640b0070 con 0x7f7e7c0a1760 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:51.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: dump: 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 5 monclient: authenticate success, global_id 164990 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 1 -- 172.21.15.86:0/3205032590 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e640ab600 con 0x7f7e7c0a1760 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 4 set_mon_vals no callback set 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 1 -- 172.21.15.86:0/3205032590 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640ab9d0 con 0x7f7e7c0a1760 2024-09-17T15:49:51.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e7a7fc640 10 monclient: dump: 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 10 monclient: shutdown 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- 172.21.15.86:0/3205032590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 msgr2=0x7f7e7c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 --2- 172.21.15.86:0/3205032590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 0x7f7e7c13af10 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7e640097c0 tx=0x7f7e6401c360 comp rx=0 tx=0).stop 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- 172.21.15.86:0/3205032590 shutdown_connections 2024-09-17T15:49:51.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 --2- 172.21.15.86:0/3205032590 >> v2:172.21.15.203:3300/0 conn(0x7f7e7c13b450 0x7f7e7c13d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 --2- 172.21.15.86:0/3205032590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a1760 0x7f7e7c13af10 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- 172.21.15.86:0/3205032590 >> 172.21.15.86:0/3205032590 conn(0x7f7e7c0a2f40 msgr2=0x7f7e7c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- 172.21.15.86:0/3205032590 shutdown_connections 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- 172.21.15.86:0/3205032590 wait complete. 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 10 monclient: monmap: 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:51.469 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 Processor -- start 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 1 -- start start 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.453+0000 7f7e814c1640 10 monclient: init 2024-09-17T15:49:51.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 0x7f7e7c1cfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7e7c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7e7c0a1760 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c13b450 0x7f7e7c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7e7c13b450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c140720 con 0x7f7e7c0a1760 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c1405a0 con 0x7f7e7c13b450 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c1408a0 con 0x7f7e7c0a0c70 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient(hunting): authenticate will time out at 2324.424464s 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 0x7f7e7c1cfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 0x7f7e7c1cfda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59618/0 (socket says 172.21.15.86:59618) 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57626/0 (socket says 172.21.15.86:57626) 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 1 -- 172.21.15.86:0/502634270 learned_addr learned my addr 172.21.15.86:0/502634270 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): get_auth_request con 0x7f7e7c0a0c70 auth_method 0 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): get_auth_request con 0x7f7e7c0a1760 auth_method 0 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7b7fe640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c13b450 0x7f7e7c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient(hunting): handle_auth_done global_id 234333 payload 824 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 msgr2=0x7f7e7c1cfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 0x7f7e7c1cfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c13b450 msgr2=0x7f7e7c09d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c13b450 0x7f7e7c09d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 monclient: found mon.noname-a 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c09f170 con 0x7f7e7c0a1760 2024-09-17T15:49:51.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7b7fe640 10 monclient: get_auth_request con 0x7f7e7c13b450 auth_method 0 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7affd640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c0a0c70 0x7f7e7c1cfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e7b7fe640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c13b450 0x7f7e7c09d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 msgr2=0x7f7e7c135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e727fc640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e640ab000 tx=0x7f7e64019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640b8030 con 0x7f7e7c0a1760 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: dump: 2024-09-17T15:49:51.476 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 5 monclient: authenticate success, global_id 234333 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _renew_subs 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c122270 con 0x7f7e7c0a1760 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e640b4620 con 0x7f7e7c0a1760 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e640b0040 con 0x7f7e7c0a1760 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e73fff640 10 monclient: dump: 2024-09-17T15:49:51.478 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _renew_subs 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e79ffb640 4 set_mon_vals no callback set 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.457+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c121f90 con 0x7f7e7c0a1760 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e814c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e814c1640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 0x7f7e7c09dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c0a1050 con 0x7f7e7c1363c0 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 0x7f7e7c09dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient: get_auth_request con 0x7f7e7c1363c0 auth_method 0 2024-09-17T15:49:51.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e727fc640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e640ab000 tx=0x7f7e64019100 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f7e640e95c0 osd_map(797..797 src has 696..797) 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7e640ab4c0 con 0x7f7e7c0a1760 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 10 monclient(hunting): handle_auth_done global_id 164996 payload 824 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e7c121c00 con 0x7f7e7c1363c0 2024-09-17T15:49:51.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e5c03bb60 0x7f7e5c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7b7fe640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e5c03bb60 0x7f7e5c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7b7fe640 10 monclient: get_auth_request con 0x7f7e5c03bb60 auth_method 0 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 msgr2=0x7f7e7c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7affd640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 0x7f7e7c09dcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0072a0 tx=0x7f7e6c007760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00fde0 con 0x7f7e7c1363c0 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 10 monclient: dump: 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.482 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e6c00fde0 con 0x7f7e7c1363c0 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 msgr2=0x7f7e7c09dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 0x7f7e7c09dcb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0072a0 tx=0x7f7e6c007760 comp rx=0 tx=0).stop 2024-09-17T15:49:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 msgr2=0x7f7e7c09dcb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e73fff640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c1363c0 0x7f7e7c09dcb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.461+0000 7f7e7b7fe640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e5c03bb60 0x7f7e5c03e020 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7e74007440 tx=0x7f7e7400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e814c1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:51.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e814c1640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 0x7f7e7c06f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e814c1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7c09e5d0 con 0x7f7e7c09e1f0 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 0x7f7e7c06f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient: get_auth_request con 0x7f7e7c09e1f0 auth_method 0 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 10 monclient(hunting): handle_auth_done global_id 164999 payload 824 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 1 -- 172.21.15.86:0/502634270 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e7c122670 con 0x7f7e7c09e1f0 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 msgr2=0x7f7e7c06f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e7affd640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 0x7f7e7c06f8f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0072a0 tx=0x7f7e6c004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c006450 con 0x7f7e7c09e1f0 2024-09-17T15:49:51.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 10 monclient: dump: 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.489 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f7e6c006450 con 0x7f7e7c09e1f0 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 msgr2=0x7f7e7c06f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 0x7f7e7c06f8f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0072a0 tx=0x7f7e6c004950 comp rx=0 tx=0).stop 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 msgr2=0x7f7e7c06f8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.469+0000 7f7e73fff640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e7c09e1f0 0x7f7e7c06f8f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:51.490 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 494656, 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:51.491 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.492 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.493 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.494 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:51.495 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.496 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:51.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.499 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.501 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.502 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:51.503 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.505 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:51.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.507 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.507 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:51.507 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e5c03bb60 msgr2=0x7f7e5c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7e5c03bb60 0x7f7e5c03e020 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7e74007440 tx=0x7f7e7400b040 comp rx=0 tx=0).stop 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 10 monclient: shutdown 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 msgr2=0x7f7e7c135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 --2- 172.21.15.86:0/502634270 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7c0a1760 0x7f7e7c135e80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7e640ab000 tx=0x7f7e64019100 comp rx=0 tx=0).stop 2024-09-17T15:49:51.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e7b7fe640 1 -- 172.21.15.86:0/502634270 reap_dead start 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e73fff640 1 -- 172.21.15.86:0/502634270 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(797..797 src has 696..797) ==== 4771+0+0 (unknown 0 0 0) 0x7f7e640e95c0 con 0x7f7e7c0a1760 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e7b7fe640 1 -- 172.21.15.86:0/502634270 reap_dead start 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 shutdown_connections 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 >> 172.21.15.86:0/502634270 conn(0x7f7e7c0a2f40 msgr2=0x7f7e7c0a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 shutdown_connections 2024-09-17T15:49:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.473+0000 7f7e814c1640 1 -- 172.21.15.86:0/502634270 wait complete. 2024-09-17T15:49:51.510 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:51.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:49:51.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: get_monmap_and_config 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: monmap: 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:51.628111+0000 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:51.628111+0000 2024-09-17T15:49:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:51.625 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 Processor -- start 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 -- start start 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: init 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0100a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0100a5bd0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa01009cd60 0x7fa010140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): picked mon.noname-a con 0x7fa01009cd60 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0100a6280 con 0x7fa0100a50e0 2024-09-17T15:49:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa01009c0a0 con 0x7fa0100a5bd0 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa01009c380 con 0x7fa01009cd60 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa018585640 10 monclient(hunting): authenticate will time out at 2324.593557s 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016321640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa0100a50e0 0x7fa0100a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48454/0 (socket says 172.21.15.86:48454) 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016321640 1 -- 172.21.15.86:0/360699041 learned_addr learned my addr 172.21.15.86:0/360699041 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 1 --2- 172.21.15.86:0/360699041 >> v2:172.21.15.86:3300/0 conn(0x7fa01009cd60 0x7fa010140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): get_auth_request con 0x7fa01009cd60 auth_method 0 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient(hunting): handle_auth_done global_id 234336 payload 274 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 1 -- 172.21.15.86:0/360699041 >> v1:172.21.15.203:6789/0 conn(0x7fa0100a5bd0 legacy=0x7fa01013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 1 -- 172.21.15.86:0/360699041 >> v1:172.21.15.86:6789/0 conn(0x7fa0100a50e0 legacy=0x7fa0100a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 1 monclient: found mon.noname-a 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.625+0000 7fa016b22640 1 -- 172.21.15.86:0/360699041 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01009c660 con 0x7fa01009cd60 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa016b22640 1 -- 172.21.15.86:0/360699041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01009cd60 msgr2=0x7fa010140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa016b22640 1 --2- 172.21.15.86:0/360699041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01009cd60 0x7fa010140a20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0040097c0 tx=0x7fa00401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270bd42756fbf39f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 1 -- 172.21.15.86:0/360699041 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0040b0070 con 0x7fa01009cd60 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:51.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: dump: 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 1 -- 172.21.15.86:0/360699041 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0040ab600 con 0x7fa01009cd60 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 5 monclient: authenticate success, global_id 234336 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 4 set_mon_vals no callback set 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 1 -- 172.21.15.86:0/360699041 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0040a4e60 con 0x7fa01009cd60 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa01531f640 10 monclient: dump: 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 10 monclient: shutdown 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 -- 172.21.15.86:0/360699041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01009cd60 msgr2=0x7fa010140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 --2- 172.21.15.86:0/360699041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01009cd60 0x7fa010140a20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0040097c0 tx=0x7fa00401c330 comp rx=0 tx=0).stop 2024-09-17T15:49:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 -- 172.21.15.86:0/360699041 shutdown_connections 2024-09-17T15:49:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 --2- 172.21.15.86:0/360699041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa01009cd60 0x7fa010140a20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 -- 172.21.15.86:0/360699041 >> 172.21.15.86:0/360699041 conn(0x7fa0100aaae0 msgr2=0x7fa0100aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 -- 172.21.15.86:0/360699041 shutdown_connections 2024-09-17T15:49:51.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.629+0000 7fa018585640 1 -- 172.21.15.86:0/360699041 wait complete. 2024-09-17T15:49:51.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient: build_initial_monmap 2024-09-17T15:49:51.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient: monmap: 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 Processor -- start 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 -- start start 2024-09-17T15:49:51.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient: init 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 0x7fa010137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): picked mon.noname-c con 0x7fa01009cd60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 0x7fa01013d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): picked mon.noname-a con 0x7fa0100a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0100a5bd0 0x7fa01013d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): picked mon.noname-b con 0x7fa0100a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa010143900 con 0x7fa0100a50e0 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa010143780 con 0x7fa0100a5bd0 2024-09-17T15:49:51.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa010143a80 con 0x7fa01009cd60 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa018585640 10 monclient(hunting): authenticate will time out at 2324.601427s 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 0x7fa010137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 0x7fa01013d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 0x7fa010137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59656/0 (socket says 172.21.15.86:59656) 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 0x7fa01013d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57648/0 (socket says 172.21.15.86:57648) 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 learned_addr learned my addr 172.21.15.86:0/3246501340 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): get_auth_request con 0x7fa01009cd60 auth_method 0 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): get_auth_request con 0x7fa0100a50e0 auth_method 0 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016b22640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0100a5bd0 0x7fa01013d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016b22640 10 monclient(hunting): get_auth_request con 0x7fa0100a5bd0 auth_method 0 2024-09-17T15:49:51.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016b22640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016b22640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient(hunting): handle_auth_done global_id 234339 payload 824 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 msgr2=0x7fa010137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 0x7fa010137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0100a5bd0 msgr2=0x7fa01013d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0100a5bd0 0x7fa01013d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient: _finish_hunting 0 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 monclient: found mon.noname-a 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa015b20640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa010121500 con 0x7fa0100a50e0 2024-09-17T15:49:51.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.633+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa01009cd60 0x7fa010137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa015b20640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 msgr2=0x7fa01013d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa015b20640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 0x7fa01013d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa00800c3e0 tx=0x7fa00800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0080119a0 con 0x7fa0100a50e0 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: dump: 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.645 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: _finish_auth 0 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:51.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 5 monclient: authenticate success, global_id 234339 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa008017070 con 0x7fa0100a50e0 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa008011d80 con 0x7fa0100a50e0 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 10 monclient: dump: 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.647 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa014b1e640 4 set_mon_vals no callback set 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 monclient: _renew_subs 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0101217e0 con 0x7fa0100a50e0 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 monclient: _renew_subs 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01013f370 con 0x7fa0100a50e0 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0100a54c0 con 0x7fa010138330 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa00801c460 con 0x7fa0100a50e0 2024-09-17T15:49:51.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 10 monclient: get_auth_request con 0x7fa010138330 auth_method 0 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 msgr2=0x7fa01013db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ffc03bbe0 0x7f9ffc03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 msgr2=0x7fa01013db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(797..797 src has 696..797) ==== 4771+0+0 (secure 0 0 0) 0x7fa00804fce0 con 0x7fa0100a50e0 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016b22640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ffc03bbe0 0x7f9ffc03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.637+0000 7fa016b22640 10 monclient: get_auth_request con 0x7f9ffc03bbe0 auth_method 0 2024-09-17T15:49:51.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.641+0000 7fa016b22640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ffc03bbe0 0x7f9ffc03e0a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa0100aa620 tx=0x7fa0040a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient: get_auth_request con 0x7fa010138330 auth_method 0 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:49:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 10 monclient(hunting): handle_auth_done global_id 254104 payload 824 2024-09-17T15:49:51.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.841+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0100a2240 con 0x7fa010138330 2024-09-17T15:49:51.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 msgr2=0x7fa01013db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa000011480 tx=0x7fa000011940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa000019030 con 0x7fa010138330 2024-09-17T15:49:51.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 10 monclient: dump: 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.845 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa000015890 con 0x7fa010138330 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 msgr2=0x7fa01013db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa000011480 tx=0x7fa000011940 comp rx=0 tx=0).stop 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 msgr2=0x7fa01013db40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.845+0000 7fa00effd640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa010138330 0x7fa01013db40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa018585640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:51.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa018585640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 0x7fa0100a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:51.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa018585640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:51.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa0101400e0 con 0x7fa01013e080 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 0x7fa0100a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient: get_auth_request con 0x7fa01013e080 auth_method 0 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:51.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:51.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:51.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 10 monclient(hunting): handle_auth_done global_id 254107 payload 824 2024-09-17T15:49:51.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.853+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa01013e490 con 0x7fa01013e080 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa016321640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 msgr2=0x7fa0100a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa016321640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 0x7fa0100a05f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa000011480 tx=0x7fa000005490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00001e870 con 0x7fa01013e080 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 10 monclient: dump: 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fa00001e870 con 0x7fa01013e080 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 msgr2=0x7fa0100a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 0x7fa0100a05f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa000011480 tx=0x7fa000005490 comp rx=0 tx=0).stop 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 msgr2=0x7fa0100a05f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:51.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa00effd640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa01013e080 0x7fa0100a05f0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa000011480 tx=0x7fa000005490 comp rx=0 tx=0).stop 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7803, 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.861 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.862 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.863 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.864 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:51.865 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:51.866 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.867 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.868 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:51.869 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.871 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:51.872 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:51.873 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ffc03bbe0 msgr2=0x7f9ffc03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9ffc03bbe0 0x7f9ffc03e0a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa0100aa620 tx=0x7fa0040a6000 comp rx=0 tx=0).stop 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 10 monclient: shutdown 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 msgr2=0x7fa01013d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 --2- 172.21.15.86:0/3246501340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0100a50e0 0x7fa01013d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa00800c3e0 tx=0x7fa00800c8a0 comp rx=0 tx=0).stop 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa016b22640 1 -- 172.21.15.86:0/3246501340 reap_dead start 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa016b22640 1 -- 172.21.15.86:0/3246501340 reap_dead start 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 shutdown_connections 2024-09-17T15:49:51.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 >> 172.21.15.86:0/3246501340 conn(0x7fa0100aaae0 msgr2=0x7fa0100a2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 shutdown_connections 2024-09-17T15:49:51.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:51.857+0000 7fa018585640 1 -- 172.21.15.86:0/3246501340 wait complete. 2024-09-17T15:49:51.875 INFO:tasks.rados.rados.0.smithi203.stdout:3646: done (1 left) 2024-09-17T15:49:51.875 INFO:tasks.rados.rados.0.smithi203.stdout:3647: write oid 8 current snap is 392 2024-09-17T15:49:51.875 INFO:tasks.rados.rados.0.smithi203.stdout:3645: finishing write tid 1 to smithi20328086-35 2024-09-17T15:49:51.875 INFO:tasks.rados.rados.0.smithi203.stdout:3645: finishing write tid 2 to smithi20328086-35 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: finishing write tid 3 to smithi20328086-35 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: finishing write tid 4 to smithi20328086-35 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: finishing write tid 5 to smithi20328086-35 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: oid 35 updating version 0 to 1672 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: oid 35 version 1672 is already newer than 1670 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: oid 35 version 1672 is already newer than 1671 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: oid 35 version 1672 is already newer than 1669 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1672 (ObjNum 1276 snap 392 seq_num 1276) dirty exists 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3645: left oid 35 (ObjNum 1276 snap 392 seq_num 1276) 2024-09-17T15:49:51.876 INFO:tasks.rados.rados.0.smithi203.stdout:3647: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-17T15:49:51.883 INFO:tasks.rados.rados.0.smithi203.stdout:3647: writing smithi20328086-8 from 662539 to 1386463 tid 1 2024-09-17T15:49:51.886 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:51.887 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:49:51.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:49:51.907 INFO:tasks.rados.rados.0.smithi203.stdout:3647: writing smithi20328086-8 from 2093976 to 2811747 tid 2 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3647: writing smithi20328086-8 from 3435397 to 3462539 tid 3 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3645: done (1 left) 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3648: rollback oid 42 current snap is 392 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 387 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3647: finishing write tid 1 to smithi20328086-8 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3647: finishing write tid 2 to smithi20328086-8 2024-09-17T15:49:51.916 INFO:tasks.rados.rados.0.smithi203.stdout:3649: delete oid 29 current snap is 392 2024-09-17T15:49:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:3647: finishing write tid 3 to smithi20328086-8 2024-09-17T15:49:51.919 INFO:tasks.rados.rados.0.smithi203.stdout:3647: finishing write tid 4 to smithi20328086-8 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3648: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: finishing write tid 5 to smithi20328086-8 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: oid 8 updating version 0 to 1644 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: oid 8 updating version 1644 to 1646 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: oid 8 version 1646 is already newer than 1645 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: oid 8 version 1646 is already newer than 1643 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1646 (ObjNum 1277 snap 392 seq_num 1277) dirty exists 2024-09-17T15:49:51.921 INFO:tasks.rados.rados.0.smithi203.stdout:3647: left oid 8 (ObjNum 1277 snap 392 seq_num 1277) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3648: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1442 (ObjNum 1230 snap 371 seq_num 1230) dirty exists 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3647: done (2 left) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3648: done (1 left) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3649: done (0 left) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3650: read oid 49 snap -1 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3650: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3651: read oid 21 snap -1 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3651: expect (ObjNum 1254 snap 380 seq_num 1254) 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3652: rollback oid 6 current snap is 392 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 6 to 390 2024-09-17T15:49:51.922 INFO:tasks.rados.rados.0.smithi203.stdout:3653: snap_create 2024-09-17T15:49:51.932 INFO:tasks.rados.rados.0.smithi203.stdout:3650: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:49:51.932 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:49:52.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 10 monclient: get_monmap_and_config 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 10 monclient: build_initial_monmap 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 10 monclient: monmap: 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:52.014919+0000 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:52.014919+0000 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.011 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:52.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:52.012 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:52.012 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:52.012 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:52.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 1 Processor -- start 2024-09-17T15:49:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 1 -- start start 2024-09-17T15:49:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.009+0000 7f1927883640 10 monclient: init 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19200689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19200694c0 0x7f192006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): picked mon.noname-a con 0x7f19200694c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f192013aff0 0x7f192013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): picked mon.noname-c con 0x7f192013aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f192009c8b0 con 0x7f19200689d0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1920074500 con 0x7f19200694c0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f192009cb90 con 0x7f192013aff0 2024-09-17T15:49:52.015 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1927883640 10 monclient(hunting): authenticate will time out at 2324.980416s 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19200694c0 0x7f192006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f192561f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f19200689d0 0x7f1920068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48468/0 (socket says 172.21.15.86:48468) 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f192561f640 1 -- 172.21.15.86:0/3318701038 learned_addr learned my addr 172.21.15.86:0/3318701038 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): get_auth_request con 0x7f19200694c0 auth_method 0 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:52.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1925e20640 1 --2- 172.21.15.86:0/3318701038 >> v2:172.21.15.203:3300/0 conn(0x7f192013aff0 0x7f192013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1925e20640 10 monclient(hunting): get_auth_request con 0x7f192013aff0 auth_method 0 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1925e20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1925e20640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1925e20640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient(hunting): handle_auth_done global_id 234342 payload 274 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 -- 172.21.15.86:0/3318701038 >> v2:172.21.15.203:3300/0 conn(0x7f192013aff0 msgr2=0x7f192013d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 --2- 172.21.15.86:0/3318701038 >> v2:172.21.15.203:3300/0 conn(0x7f192013aff0 0x7f192013d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 -- 172.21.15.86:0/3318701038 >> v1:172.21.15.86:6789/0 conn(0x7f19200689d0 legacy=0x7f1920068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:52.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient: _finish_hunting 0 2024-09-17T15:49:52.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 monclient: found mon.noname-a 2024-09-17T15:49:52.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:52.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.013+0000 7f1924e1e640 1 -- 172.21.15.86:0/3318701038 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192009ce70 con 0x7f19200694c0 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 1 -- 172.21.15.86:0/3318701038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200694c0 msgr2=0x7f192006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 1 --2- 172.21.15.86:0/3318701038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200694c0 0x7f192006f840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f192006fdb0 tx=0x7f191801c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8090f8315c32f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 1 -- 172.21.15.86:0/3318701038 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19180b2070 con 0x7f19200694c0 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: dump: 2024-09-17T15:49:52.024 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: _finish_auth 0 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 1 -- 172.21.15.86:0/3318701038 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19180a5870 con 0x7f19200694c0 2024-09-17T15:49:52.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 1 -- 172.21.15.86:0/3318701038 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192009c2d0 con 0x7f19200694c0 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 5 monclient: authenticate success, global_id 234342 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 4 set_mon_vals no callback set 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1917fff640 10 monclient: dump: 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.026 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: shutdown 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- 172.21.15.86:0/3318701038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200694c0 msgr2=0x7f192006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- 172.21.15.86:0/3318701038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200694c0 0x7f192006f840 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f192006fdb0 tx=0x7f191801c7d0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- 172.21.15.86:0/3318701038 shutdown_connections 2024-09-17T15:49:52.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- 172.21.15.86:0/3318701038 >> v2:172.21.15.203:3300/0 conn(0x7f192013aff0 0x7f192013d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- 172.21.15.86:0/3318701038 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200694c0 0x7f192006f840 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- 172.21.15.86:0/3318701038 >> 172.21.15.86:0/3318701038 conn(0x7f19200a6440 msgr2=0x7f19200a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- 172.21.15.86:0/3318701038 shutdown_connections 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- 172.21.15.86:0/3318701038 wait complete. 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: build_initial_monmap 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: monmap: 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 Processor -- start 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- start start 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: init 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): picked mon.noname-a con 0x7f19200689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): picked mon.noname-c con 0x7f19200694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f192013aff0 0x7f1920135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): picked mon.noname-b con 0x7f192013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f19201405c0 con 0x7f19200689d0 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1920140440 con 0x7f192013aff0 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1920140740 con 0x7f19200694c0 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1927883640 10 monclient(hunting): authenticate will time out at 2324.987450s 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57658/0 (socket says 172.21.15.86:57658) 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 -- 172.21.15.86:0/2286125024 learned_addr learned my addr 172.21.15.86:0/2286125024 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59660/0 (socket says 172.21.15.86:59660) 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): get_auth_request con 0x7f19200689d0 auth_method 0 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): get_auth_request con 0x7f19200694c0 auth_method 0 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:52.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1925e20640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f192013aff0 0x7f1920135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1925e20640 10 monclient(hunting): get_auth_request con 0x7f192013aff0 auth_method 0 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1925e20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1925e20640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1925e20640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient(hunting): handle_auth_done global_id 234345 payload 824 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 -- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 msgr2=0x7f192011ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 -- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f192013aff0 msgr2=0x7f1920135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f192013aff0 0x7f1920135b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient: _finish_hunting 0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 monclient: found mon.noname-a 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f192561f640 1 -- 172.21.15.86:0/2286125024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192013b3d0 con 0x7f19200689d0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.017+0000 7f1924e1e640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f192561f640 1 -- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 msgr2=0x7f192011cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f192561f640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c470 tx=0x7f191c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c017070 con 0x7f19200689d0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: dump: 2024-09-17T15:49:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f191c0070d0 con 0x7f19200689d0 2024-09-17T15:49:52.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191c0074c0 con 0x7f19200689d0 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 10 monclient: dump: 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:52.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 5 monclient: authenticate success, global_id 234345 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _renew_subs 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f19177fe640 4 set_mon_vals no callback set 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192009d460 con 0x7f19200689d0 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _renew_subs 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1920136820 con 0x7f19200689d0 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f192006f840 con 0x7f19200689d0 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f191c01a070 con 0x7f19200689d0 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 0x7f190c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.021+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(798..798 src has 696..798) ==== 4767+0+0 (secure 0 0 0) 0x7f191c04f580 con 0x7f19200689d0 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.025+0000 7f1924e1e640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 0x7f190c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.025+0000 7f1924e1e640 10 monclient: get_auth_request con 0x7f190c03bb60 auth_method 0 2024-09-17T15:49:52.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.025+0000 7f1924e1e640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 0x7f190c03e020 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1920069a70 tx=0x7f19180a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:52.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.029+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f191c01bc50 con 0x7f19200689d0 2024-09-17T15:49:52.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.029+0000 7f1915ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:52.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.029+0000 7f1915ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:52.137 INFO:tasks.rados.rados.0.smithi203.stdout:3651: expect (ObjNum 1254 snap 380 seq_num 1254) 2024-09-17T15:49:52.164 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2580311 2024-09-17T15:49:52.188 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1850033 2024-09-17T15:49:52.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.261+0000 7f1927883640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:49:52.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.261+0000 7f1927883640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:49:52.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.261+0000 7f1927883640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:52.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.261+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f192006d4f0 con 0x7f19200689d0 2024-09-17T15:49:52.264 INFO:tasks.rados.rados.0.smithi203.stdout:3652: finishing rollback tid 0 to smithi20328086-6 2024-09-17T15:49:52.264 INFO:tasks.rados.rados.0.smithi203.stdout:3652: finishing rollback tid 1 to smithi20328086-6 2024-09-17T15:49:52.264 INFO:tasks.rados.rados.0.smithi203.stdout:3652: finishing rollback tid 2 to smithi20328086-6 2024-09-17T15:49:52.264 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1672 (ObjNum 1253 snap 380 seq_num 1253) dirty exists 2024-09-17T15:49:52.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f1915ffb640 1 -- 172.21.15.86:0/2286125024 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f191c010300 con 0x7f19200689d0 2024-09-17T15:49:52.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f1915ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:49:52.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f1915ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=2976298756,config=3541143443,health=3319432238,kv=2002333117,logm=732795263}) 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=2447134731,config=3541143443,health=3319432238,kv=2732825231,logm=1342541973}) 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=2976298756,config=3541143443,health=3319432238,kv=2002333117,logm=732795263}) 2024-09-17T15:49:52.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=2762875454,config=3541143443,health=4185752094,kv=398854677,logm=1871843385}) 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 msgr2=0x7f190c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 0x7f190c03e020 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1920069a70 tx=0x7f19180a7070 comp rx=0 tx=0).stop 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 10 monclient: shutdown 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 msgr2=0x7f192011cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f191c00c470 tx=0x7f191c00c930 comp rx=0 tx=0).stop 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 shutdown_connections 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f190c03bb60 0x7f190c03e020 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f192013aff0 0x7f1920135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19200694c0 0x7f192011ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 --2- 172.21.15.86:0/2286125024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19200689d0 0x7f192011cda0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:52.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 >> 172.21.15.86:0/2286125024 conn(0x7f19200a6440 msgr2=0x7f19200a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:52.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616336579}) 2024-09-17T15:49:52.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274616566}) 2024-09-17T15:49:52.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616336579}) 2024-09-17T15:49:52.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3328703183}) 2024-09-17T15:49:52.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 shutdown_connections 2024-09-17T15:49:52.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:52.269+0000 7f1927883640 1 -- 172.21.15.86:0/2286125024 wait complete. 2024-09-17T15:49:52.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002753608}) 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829534016}) 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002753608}) 2024-09-17T15:49:52.712 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.713+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958448239}) 2024-09-17T15:49:52.715 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.715 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316518967}) 2024-09-17T15:49:52.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084191227}) 2024-09-17T15:49:52.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316518967}) 2024-09-17T15:49:52.716 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.717+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1794098715}) 2024-09-17T15:49:52.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831667610}) 2024-09-17T15:49:52.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906758800}) 2024-09-17T15:49:52.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.772 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831667610}) 2024-09-17T15:49:52.773 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.773+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446199447}) 2024-09-17T15:49:52.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715014640}) 2024-09-17T15:49:52.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170942720}) 2024-09-17T15:49:52.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715014640}) 2024-09-17T15:49:52.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2202061306}) 2024-09-17T15:49:52.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953380217}) 2024-09-17T15:49:52.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769626415}) 2024-09-17T15:49:52.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953380217}) 2024-09-17T15:49:52.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679881806}) 2024-09-17T15:49:52.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458704065}) 2024-09-17T15:49:52.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171190300}) 2024-09-17T15:49:52.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:52.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458704065}) 2024-09-17T15:49:52.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:52.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828275494}) 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3650: done (3 left) 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3651: done (2 left) 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3652: done (1 left) 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3653: done (0 left) 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3654: rollback oid 33 current snap is 393 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 33 to 389 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3655: copy_from oid 36 from oid 13 current snap is 393 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3656: write (excl) oid 32 current snap is 393 2024-09-17T15:49:52.874 INFO:tasks.rados.rados.0.smithi203.stdout:3656: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2024-09-17T15:49:52.896 INFO:tasks.rados.rados.0.smithi203.stdout:3656: writing smithi20328086-32 from 696153 to 1286928 tid 1 2024-09-17T15:49:52.918 INFO:tasks.rados.rados.0.smithi203.stdout:3656: writing smithi20328086-32 from 1927293 to 2648733 tid 2 2024-09-17T15:49:52.930 INFO:tasks.rados.rados.0.smithi203.stdout:3656: writing smithi20328086-32 from 3206596 to 3496153 tid 3 2024-09-17T15:49:52.930 INFO:tasks.rados.rados.0.smithi203.stdout:3657: rollback oid 34 current snap is 393 2024-09-17T15:49:52.930 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 34 to 389 2024-09-17T15:49:52.930 INFO:tasks.rados.rados.0.smithi203.stdout:3655: finishing copy_from to smithi20328086-36 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3655: got expected ENOENT (src dne) 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3655: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3654: finishing rollback tid 0 to smithi20328086-33 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3654: finishing rollback tid 1 to smithi20328086-33 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3654: finishing rollback tid 2 to smithi20328086-33 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1421 (ObjNum 1239 snap 373 seq_num 1239) dirty exists 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3656: finishing write tid 1 to smithi20328086-32 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3656: finishing write tid 2 to smithi20328086-32 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3654: done (3 left) 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3655: done (2 left) 2024-09-17T15:49:52.931 INFO:tasks.rados.rados.0.smithi203.stdout:3658: snap_create 2024-09-17T15:49:52.937 INFO:tasks.rados.rados.0.smithi203.stdout:3656: finishing write tid 3 to smithi20328086-32 2024-09-17T15:49:52.938 INFO:tasks.rados.rados.0.smithi203.stdout:3656: finishing write tid 4 to smithi20328086-32 2024-09-17T15:49:52.938 INFO:tasks.rados.rados.0.smithi203.stdout:3656: finishing write tid 5 to smithi20328086-32 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:3656: oid 32 updating version 0 to 1735 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:3656: oid 32 updating version 1735 to 1736 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:3656: oid 32 version 1736 is already newer than 1733 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:3656: oid 32 version 1736 is already newer than 1734 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1736 (ObjNum 1279 snap 393 seq_num 1279) dirty exists 2024-09-17T15:49:52.939 INFO:tasks.rados.rados.0.smithi203.stdout:3656: left oid 32 (ObjNum 1279 snap 393 seq_num 1279) 2024-09-17T15:49:52.946 INFO:tasks.rados.rados.0.smithi203.stdout:3657: finishing rollback tid 0 to smithi20328086-34 2024-09-17T15:49:52.946 INFO:tasks.rados.rados.0.smithi203.stdout:3657: finishing rollback tid 1 to smithi20328086-34 2024-09-17T15:49:52.947 INFO:tasks.rados.rados.0.smithi203.stdout:3657: finishing rollback tid 2 to smithi20328086-34 2024-09-17T15:49:52.947 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1675 (ObjNum 1246 snap 378 seq_num 1246) dirty exists 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249999260}) 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554089682}) 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249999260}) 2024-09-17T15:49:53.040 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.041+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1778719347}) 2024-09-17T15:49:53.051 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243613450}) 2024-09-17T15:49:53.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648264873}) 2024-09-17T15:49:53.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=243613450}) 2024-09-17T15:49:53.052 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.053+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774141990}) 2024-09-17T15:49:53.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065859522}) 2024-09-17T15:49:53.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053494846}) 2024-09-17T15:49:53.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065859522}) 2024-09-17T15:49:53.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.265+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3222334690}) 2024-09-17T15:49:53.268 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.268 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255015606}) 2024-09-17T15:49:53.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=676235377}) 2024-09-17T15:49:53.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255015606}) 2024-09-17T15:49:53.269 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015234502}) 2024-09-17T15:49:53.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072887987}) 2024-09-17T15:49:53.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049498633}) 2024-09-17T15:49:53.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4072887987}) 2024-09-17T15:49:53.271 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.269+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033215282}) 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498446366}) 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3116192744}) 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498446366}) 2024-09-17T15:49:53.273 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3195322696}) 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288119336}) 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36479948}) 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288119336}) 2024-09-17T15:49:53.275 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.273+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367960703}) 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168801510}) 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418688823}) 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168801510}) 2024-09-17T15:49:53.277 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1508430027}) 2024-09-17T15:49:53.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769111704}) 2024-09-17T15:49:53.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=529706098}) 2024-09-17T15:49:53.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769111704}) 2024-09-17T15:49:53.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.277+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656858782}) 2024-09-17T15:49:53.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081770861}) 2024-09-17T15:49:53.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019196250}) 2024-09-17T15:49:53.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081770861}) 2024-09-17T15:49:53.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.281+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1438303471}) 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235009079}) 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182496247}) 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2235009079}) 2024-09-17T15:49:53.284 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578149046}) 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730210235}) 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2722232979}) 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1730210235}) 2024-09-17T15:49:53.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.285+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2731002044}) 2024-09-17T15:49:53.286 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:49:53.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856382238}) 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2877268728}) 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856382238}) 2024-09-17T15:49:53.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1101008528}) 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609886462}) 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645549666}) 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3609886462}) 2024-09-17T15:49:53.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.289+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933581010}) 2024-09-17T15:49:53.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033613574}) 2024-09-17T15:49:53.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932835612}) 2024-09-17T15:49:53.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033613574}) 2024-09-17T15:49:53.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279202027}) 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109559501}) 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632181231}) 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109559501}) 2024-09-17T15:49:53.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.293+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=665266387}) 2024-09-17T15:49:53.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620429443}) 2024-09-17T15:49:53.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=93343754}) 2024-09-17T15:49:53.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.298 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=620429443}) 2024-09-17T15:49:53.298 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.297+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4074472979}) 2024-09-17T15:49:53.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145486068}) 2024-09-17T15:49:53.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1855782494}) 2024-09-17T15:49:53.300 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145486068}) 2024-09-17T15:49:53.301 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3087954106}) 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819535646}) 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3351081366}) 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819535646}) 2024-09-17T15:49:53.303 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.301+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492198140}) 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657045493}) 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=668788850}) 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657045493}) 2024-09-17T15:49:53.306 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.305+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2150833427}) 2024-09-17T15:49:53.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11590908}) 2024-09-17T15:49:53.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3942985087}) 2024-09-17T15:49:53.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11590908}) 2024-09-17T15:49:53.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335762408}) 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994840293}) 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920353341}) 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994840293}) 2024-09-17T15:49:53.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.309+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643004366}) 2024-09-17T15:49:53.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955971176}) 2024-09-17T15:49:53.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133792236}) 2024-09-17T15:49:53.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2955971176}) 2024-09-17T15:49:53.314 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080821436}) 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167939023}) 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1621525101}) 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167939023}) 2024-09-17T15:49:53.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.313+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935486890}) 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589439303}) 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4080541987}) 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589439303}) 2024-09-17T15:49:53.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=765611136}) 2024-09-17T15:49:53.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789209255}) 2024-09-17T15:49:53.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287584713}) 2024-09-17T15:49:53.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.317+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789209255}) 2024-09-17T15:49:53.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098806432}) 2024-09-17T15:49:53.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190160484}) 2024-09-17T15:49:53.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=727769370}) 2024-09-17T15:49:53.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190160484}) 2024-09-17T15:49:53.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.321+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783548697}) 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291150132}) 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638427046}) 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291150132}) 2024-09-17T15:49:53.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1831595}) 2024-09-17T15:49:53.326 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.326 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826664172}) 2024-09-17T15:49:53.327 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1968342446}) 2024-09-17T15:49:53.327 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.327 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826664172}) 2024-09-17T15:49:53.327 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.325+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366848712}) 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109061935}) 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3040218742}) 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=109061935}) 2024-09-17T15:49:53.329 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2065026832}) 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805044068}) 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269404354}) 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805044068}) 2024-09-17T15:49:53.331 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.329+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598131208}) 2024-09-17T15:49:53.333 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.333 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650548991}) 2024-09-17T15:49:53.333 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116465940}) 2024-09-17T15:49:53.334 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.334 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650548991}) 2024-09-17T15:49:53.334 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648670810}) 2024-09-17T15:49:53.335 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.335 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582405299}) 2024-09-17T15:49:53.336 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2482364292}) 2024-09-17T15:49:53.336 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.336 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582405299}) 2024-09-17T15:49:53.336 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.333+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825389409}) 2024-09-17T15:49:53.337 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931972020}) 2024-09-17T15:49:53.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467792746}) 2024-09-17T15:49:53.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931972020}) 2024-09-17T15:49:53.338 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.337+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608537126}) 2024-09-17T15:49:53.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084296427}) 2024-09-17T15:49:53.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334796117}) 2024-09-17T15:49:53.343 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084296427}) 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=173847632}) 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371561578}) 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2153798943}) 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371561578}) 2024-09-17T15:49:53.344 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.341+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=281436863}) 2024-09-17T15:49:53.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605020640}) 2024-09-17T15:49:53.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1911705320}) 2024-09-17T15:49:53.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.345 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605020640}) 2024-09-17T15:49:53.346 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813867161}) 2024-09-17T15:49:53.346 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.346 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817412369}) 2024-09-17T15:49:53.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=527118569}) 2024-09-17T15:49:53.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817412369}) 2024-09-17T15:49:53.347 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.345+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283401537}) 2024-09-17T15:49:53.348 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.349 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130650211}) 2024-09-17T15:49:53.349 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253422917}) 2024-09-17T15:49:53.349 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.349 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3130650211}) 2024-09-17T15:49:53.349 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151378346}) 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358435594}) 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461395209}) 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358435594}) 2024-09-17T15:49:53.351 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.349+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318426430}) 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415133415}) 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2875977397}) 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415133415}) 2024-09-17T15:49:53.353 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749938455}) 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920304462}) 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4147029616}) 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920304462}) 2024-09-17T15:49:53.355 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.353+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3775078713}) 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679591231}) 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661433220}) 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679591231}) 2024-09-17T15:49:53.357 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1312705763}) 2024-09-17T15:49:53.359 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.359 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651009345}) 2024-09-17T15:49:53.359 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1441718891}) 2024-09-17T15:49:53.359 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.359 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651009345}) 2024-09-17T15:49:53.360 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.357+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590128639}) 2024-09-17T15:49:53.361 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.361 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606971370}) 2024-09-17T15:49:53.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035524234}) 2024-09-17T15:49:53.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606971370}) 2024-09-17T15:49:53.362 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579529114}) 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82984183}) 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955207192}) 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.361+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82984183}) 2024-09-17T15:49:53.364 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3151572525}) 2024-09-17T15:49:53.366 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.366 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075813655}) 2024-09-17T15:49:53.366 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057910061}) 2024-09-17T15:49:53.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075813655}) 2024-09-17T15:49:53.367 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707928259}) 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.365+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627840762}) 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=13636762}) 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627840762}) 2024-09-17T15:49:53.368 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1322279927}) 2024-09-17T15:49:53.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.369 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144589784}) 2024-09-17T15:49:53.370 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2914601207}) 2024-09-17T15:49:53.370 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.370 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144589784}) 2024-09-17T15:49:53.370 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=799256649}) 2024-09-17T15:49:53.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801856095}) 2024-09-17T15:49:53.371 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2597692562}) 2024-09-17T15:49:53.372 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.372 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.369+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801856095}) 2024-09-17T15:49:53.372 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=906660859}) 2024-09-17T15:49:53.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.373 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172442188}) 2024-09-17T15:49:53.374 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2255461072}) 2024-09-17T15:49:53.374 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.374 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172442188}) 2024-09-17T15:49:53.374 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777310218}) 2024-09-17T15:49:53.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705760652}) 2024-09-17T15:49:53.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746017722}) 2024-09-17T15:49:53.375 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.376 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705760652}) 2024-09-17T15:49:53.376 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.373+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251665973}) 2024-09-17T15:49:53.377 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.377 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538167038}) 2024-09-17T15:49:53.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1962629393}) 2024-09-17T15:49:53.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538167038}) 2024-09-17T15:49:53.378 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3682328337}) 2024-09-17T15:49:53.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140587158}) 2024-09-17T15:49:53.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781291955}) 2024-09-17T15:49:53.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.379 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140587158}) 2024-09-17T15:49:53.380 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.377+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915971674}) 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556745775}) 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2990010214}) 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556745775}) 2024-09-17T15:49:53.381 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237468186}) 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764768939}) 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644476885}) 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764768939}) 2024-09-17T15:49:53.383 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.381+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599226031}) 2024-09-17T15:49:53.385 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.385 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031880045}) 2024-09-17T15:49:53.385 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757857255}) 2024-09-17T15:49:53.385 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.385 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2031880045}) 2024-09-17T15:49:53.386 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686220770}) 2024-09-17T15:49:53.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.387 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078195726}) 2024-09-17T15:49:53.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=855802409}) 2024-09-17T15:49:53.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078195726}) 2024-09-17T15:49:53.388 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.385+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2034220013}) 2024-09-17T15:49:53.389 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.389 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293322116}) 2024-09-17T15:49:53.389 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205917462}) 2024-09-17T15:49:53.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293322116}) 2024-09-17T15:49:53.390 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397116620}) 2024-09-17T15:49:53.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215032255}) 2024-09-17T15:49:53.391 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753259712}) 2024-09-17T15:49:53.392 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.392 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215032255}) 2024-09-17T15:49:53.392 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.389+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2305525277}) 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143800336}) 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1322250106}) 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143800336}) 2024-09-17T15:49:53.393 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=674076234}) 2024-09-17T15:49:53.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401358697}) 2024-09-17T15:49:53.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194088817}) 2024-09-17T15:49:53.395 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.396 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401358697}) 2024-09-17T15:49:53.396 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.393+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557815085}) 2024-09-17T15:49:53.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910220330}) 2024-09-17T15:49:53.397 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879009783}) 2024-09-17T15:49:53.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910220330}) 2024-09-17T15:49:53.398 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=837932452}) 2024-09-17T15:49:53.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546610526}) 2024-09-17T15:49:53.399 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237761818}) 2024-09-17T15:49:53.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546610526}) 2024-09-17T15:49:53.400 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.397+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2103238361}) 2024-09-17T15:49:53.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992229316}) 2024-09-17T15:49:53.401 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2191403296}) 2024-09-17T15:49:53.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992229316}) 2024-09-17T15:49:53.402 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886649083}) 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684505391}) 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795353733}) 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684505391}) 2024-09-17T15:49:53.404 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.401+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452287815}) 2024-09-17T15:49:53.405 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.405 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948625534}) 2024-09-17T15:49:53.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2989400839}) 2024-09-17T15:49:53.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948625534}) 2024-09-17T15:49:53.406 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=143573567}) 2024-09-17T15:49:53.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407293828}) 2024-09-17T15:49:53.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468883212}) 2024-09-17T15:49:53.408 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.405+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.409 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407293828}) 2024-09-17T15:49:53.409 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497955836}) 2024-09-17T15:49:53.409 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771464442}) 2024-09-17T15:49:53.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2380532237}) 2024-09-17T15:49:53.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771464442}) 2024-09-17T15:49:53.410 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428185117}) 2024-09-17T15:49:53.411 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855861156}) 2024-09-17T15:49:53.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354679151}) 2024-09-17T15:49:53.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.409+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855861156}) 2024-09-17T15:49:53.412 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651558586}) 2024-09-17T15:49:53.413 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620245295}) 2024-09-17T15:49:53.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771874180}) 2024-09-17T15:49:53.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620245295}) 2024-09-17T15:49:53.414 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353591146}) 2024-09-17T15:49:53.415 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.415 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201548230}) 2024-09-17T15:49:53.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2767205812}) 2024-09-17T15:49:53.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.413+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201548230}) 2024-09-17T15:49:53.416 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1983345264}) 2024-09-17T15:49:53.417 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4016087286}) 2024-09-17T15:49:53.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510029765}) 2024-09-17T15:49:53.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4016087286}) 2024-09-17T15:49:53.418 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2302620483}) 2024-09-17T15:49:53.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809852324}) 2024-09-17T15:49:53.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663831722}) 2024-09-17T15:49:53.419 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.417+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.420 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809852324}) 2024-09-17T15:49:53.420 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153641943}) 2024-09-17T15:49:53.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965212822}) 2024-09-17T15:49:53.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3117166013}) 2024-09-17T15:49:53.421 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.422 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965212822}) 2024-09-17T15:49:53.422 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731605487}) 2024-09-17T15:49:53.423 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.424 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733365794}) 2024-09-17T15:49:53.424 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2095099900}) 2024-09-17T15:49:53.424 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.424 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733365794}) 2024-09-17T15:49:53.424 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.421+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=175912662}) 2024-09-17T15:49:53.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.425 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764371134}) 2024-09-17T15:49:53.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3699580979}) 2024-09-17T15:49:53.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764371134}) 2024-09-17T15:49:53.426 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3179997894}) 2024-09-17T15:49:53.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537946817}) 2024-09-17T15:49:53.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410661600}) 2024-09-17T15:49:53.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.427 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1537946817}) 2024-09-17T15:49:53.428 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.425+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115750788}) 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676483955}) 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3391495759}) 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676483955}) 2024-09-17T15:49:53.429 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158097007}) 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081016982}) 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1917174466}) 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081016982}) 2024-09-17T15:49:53.431 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.429+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004206876}) 2024-09-17T15:49:53.432 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.432 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808944406}) 2024-09-17T15:49:53.432 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071299463}) 2024-09-17T15:49:53.432 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808944406}) 2024-09-17T15:49:53.433 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117893649}) 2024-09-17T15:49:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: get_monmap_and_config 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: build_initial_monmap 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: monmap: 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:53.438047+0000 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:53.438047+0000 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:53.435 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:53.435 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:53.435 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451502213}) 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546950080}) 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2451502213}) 2024-09-17T15:49:53.436 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499007760}) 2024-09-17T15:49:53.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 Processor -- start 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 -- start start 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: init 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5e80a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5e813d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e80a6530 con 0x7ff5e80a50e0 2024-09-17T15:49:53.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e809c150 con 0x7ff5e813d2f0 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e809c430 con 0x7ff5e80a54c0 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5eefbe640 10 monclient(hunting): authenticate will time out at 2326.403720s 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5ecd5a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff5e80a50e0 0x7ff5e809ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48480/0 (socket says 172.21.15.86:48480) 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5ecd5a640 1 -- 172.21.15.86:0/442475211 learned_addr learned my addr 172.21.15.86:0/442475211 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5e7fff640 1 --2- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5e7fff640 10 monclient(hunting): get_auth_request con 0x7ff5e80a54c0 auth_method 0 2024-09-17T15:49:53.440 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.433+0000 7ff5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.433+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246466954}) 2024-09-17T15:49:53.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2355922049}) 2024-09-17T15:49:53.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=246466954}) 2024-09-17T15:49:53.441 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=17822459}) 2024-09-17T15:49:53.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_done global_id 165008 payload 274 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 1 -- 172.21.15.86:0/442475211 >> v1:172.21.15.203:6789/0 conn(0x7ff5e813d2f0 legacy=0x7ff5e813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 1 -- 172.21.15.86:0/442475211 >> v1:172.21.15.86:6789/0 conn(0x7ff5e80a50e0 legacy=0x7ff5e809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 1 monclient: found mon.noname-b 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:53.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.437+0000 7ff5e7fff640 1 -- 172.21.15.86:0/442475211 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e80a6940 con 0x7ff5e80a54c0 2024-09-17T15:49:53.444 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.444 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510086286}) 2024-09-17T15:49:53.444 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070262957}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510086286}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.437+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896828614}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=956719789,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1188138759,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4228708718,osd_snap=3772246274}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=1,nvmeofgw=3,osd_metadata=5,osd_snap=26} crc {logm=4211998874,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2641783023,mon_config_key=639531838,monmap=1228700967,nvmeofgw=598337686,osd_metadata=3734937928,osd_snap=2857303171}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=25,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24} crc {logm=956719789,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1188138759,mon_config_key=639531838,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4228708718,osd_snap=3772246274}) 2024-09-17T15:49:53.445 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=28,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=2666749724,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=2641783023,mon_config_key=1859314278,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=4228708718,osd_snap=2872008563}) 2024-09-17T15:49:53.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e7fff640 1 -- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 msgr2=0x7ff5e80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e7fff640 1 --2- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e80a58d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40097f0 tx=0x7ff5d401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdfa671d8d29ff54 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 1 -- 172.21.15.86:0/442475211 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40b0070 con 0x7ff5e80a54c0 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: dump: 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 1 -- 172.21.15.86:0/442475211 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d40ab600 con 0x7ff5e80a54c0 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 5 monclient: authenticate success, global_id 165008 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 1 -- 172.21.15.86:0/442475211 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40ab9d0 con 0x7ff5e80a54c0 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5e77fe640 10 monclient: dump: 2024-09-17T15:49:53.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:53.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 4 set_mon_vals no callback set 2024-09-17T15:49:53.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 10 monclient: shutdown 2024-09-17T15:49:53.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:53.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 -- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 msgr2=0x7ff5e80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e80a58d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff5d40097f0 tx=0x7ff5d401c3a0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4252661087}) 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=15961378}) 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4252661087}) 2024-09-17T15:49:53.451 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.441+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=2991672452}) 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 -- 172.21.15.86:0/442475211 shutdown_connections 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/442475211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e80a58d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 -- 172.21.15.86:0/442475211 >> 172.21.15.86:0/442475211 conn(0x7ff5e80aaae0 msgr2=0x7ff5e80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 -- 172.21.15.86:0/442475211 shutdown_connections 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.441+0000 7ff5eefbe640 1 -- 172.21.15.86:0/442475211 wait complete. 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient: build_initial_monmap 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient: monmap: 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:53.453 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 Processor -- start 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 -- start start 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient: init 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:53.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5e80a50e0 0x7ff5e8137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e80a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-c con 0x7ff5e80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 0x7ff5e80a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): picked mon.noname-a con 0x7ff5e813d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e8143a10 con 0x7ff5e813d2f0 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e8143890 con 0x7ff5e80a50e0 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e8143b90 con 0x7ff5e80a54c0 2024-09-17T15:49:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:53.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5eefbe640 10 monclient(hunting): authenticate will time out at 2326.413008s 2024-09-17T15:49:53.456 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=3898490100,osdmap=1755466761}) 2024-09-17T15:49:53.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=3898490100,osdmap=3920348413}) 2024-09-17T15:49:53.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=3898490100,osdmap=1755466761}) 2024-09-17T15:49:53.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.445+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=3898490100,osdmap=2480289287}) 2024-09-17T15:49:53.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59684/0 (socket says 172.21.15.86:59684) 2024-09-17T15:49:53.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 1 -- 172.21.15.86:0/1321486251 learned_addr learned my addr 172.21.15.86:0/1321486251 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): get_auth_request con 0x7ff5e80a54c0 auth_method 0 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 0x7ff5e80a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): get_auth_request con 0x7ff5e813d2f0 auth_method 0 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5e80a50e0 0x7ff5e8137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): get_auth_request con 0x7ff5e80a50e0 auth_method 0 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ecd5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient(hunting): handle_auth_done global_id 234348 payload 824 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 -- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 msgr2=0x7ff5e8138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 -- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5e80a50e0 msgr2=0x7ff5e8137ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5e80a50e0 0x7ff5e8137ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient: _finish_hunting 0 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 monclient: found mon.noname-a 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5ed55b640 1 -- 172.21.15.86:0/1321486251 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e8121740 con 0x7ff5e813d2f0 2024-09-17T15:49:53.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.445+0000 7ff5e7fff640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5ed55b640 1 -- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 msgr2=0x7ff5e80a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5ed55b640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 0x7ff5e80a18d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c780 tx=0x7ff5e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0016020 con 0x7ff5e813d2f0 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: dump: 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.464 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e000ac00 con 0x7ff5e813d2f0 2024-09-17T15:49:53.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000ce30 con 0x7ff5e813d2f0 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 5 monclient: authenticate success, global_id 234348 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _renew_subs 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e80a7c30 con 0x7ff5e813d2f0 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e6ffd640 4 set_mon_vals no callback set 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 10 monclient: dump: 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.467 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _renew_subs 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e80a3550 con 0x7ff5e813d2f0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8120d90 con 0x7ff5e813d2f0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff5e0019020 con 0x7ff5e813d2f0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 0x7ff5cc03e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.449+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(799..799 src has 696..799) ==== 4739+0+0 (secure 0 0 0) 0x7ff5e004fe00 con 0x7ff5e813d2f0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.453+0000 7ff5ecd5a640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 0x7ff5cc03e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.453+0000 7ff5ecd5a640 10 monclient: get_auth_request con 0x7ff5cc03bba0 auth_method 0 2024-09-17T15:49:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.453+0000 7ff5ecd5a640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 0x7ff5cc03e060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc007ca0 tx=0x7ff5dc007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.457+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff5e0054070 con 0x7ff5e813d2f0 2024-09-17T15:49:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.457+0000 7ff5e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:53.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.457+0000 7ff5e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:53.469 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=1822424019}) 2024-09-17T15:49:53.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=75} crc {osdmap=4077789954}) 2024-09-17T15:49:53.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:49:53.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=1822424019}) 2024-09-17T15:49:53.470 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:49:53.457+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=77} crc {osdmap=3383039}) 2024-09-17T15:49:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5e80a58d0 con 0x7ff5e813d2f0 2024-09-17T15:49:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5e57fa640 1 -- 172.21.15.86:0/1321486251 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff5e0020030 con 0x7ff5e813d2f0 2024-09-17T15:49:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:53.693 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 msgr2=0x7ff5cc03e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 0x7ff5cc03e060 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc007ca0 tx=0x7ff5dc007bf0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 10 monclient: shutdown 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 msgr2=0x7ff5e80a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.693+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 0x7ff5e80a18d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c780 tx=0x7ff5e000cc40 comp rx=0 tx=0).stop 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 shutdown_connections 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff5cc03bba0 0x7ff5cc03e060 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff5e813d2f0 0x7ff5e80a18d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5e80a54c0 0x7ff5e8138400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 --2- 172.21.15.86:0/1321486251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5e80a50e0 0x7ff5e8137ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 >> 172.21.15.86:0/1321486251 conn(0x7ff5e80aaae0 msgr2=0x7ff5e80a0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:53.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 shutdown_connections 2024-09-17T15:49:53.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.697+0000 7ff5eefbe640 1 -- 172.21.15.86:0/1321486251 wait complete. 2024-09-17T15:49:53.713 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:49:53.713 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:49:53.714 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:49:53.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:49:53.873 INFO:tasks.rados.rados.0.smithi203.stdout:3656: done (2 left) 2024-09-17T15:49:53.874 INFO:tasks.rados.rados.0.smithi203.stdout:3657: done (1 left) 2024-09-17T15:49:53.874 INFO:tasks.rados.rados.0.smithi203.stdout:3658: done (0 left) 2024-09-17T15:49:53.874 INFO:tasks.rados.rados.0.smithi203.stdout:3659: rollback oid 29 current snap is 394 2024-09-17T15:49:53.874 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 29 to 382 2024-09-17T15:49:53.874 INFO:tasks.rados.rados.0.smithi203.stdout:3660: snap_create 2024-09-17T15:49:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: get_monmap_and_config 2024-09-17T15:49:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: build_initial_monmap 2024-09-17T15:49:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: monmap: 2024-09-17T15:49:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:53.885496+0000 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:53.885496+0000 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:53.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.883 INFO:tasks.rados.rados.0.smithi203.stdout:3659: finishing rollback tid 1 to smithi20328086-29 2024-09-17T15:49:53.883 INFO:tasks.rados.rados.0.smithi203.stdout:3659: finishing rollback tid 2 to smithi20328086-29 2024-09-17T15:49:53.883 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1555 (ObjNum 1232 snap 371 seq_num 1232) dirty exists 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 Processor -- start 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 -- start start 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: init 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f899c0699e0 0x7f899c132a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-a con 0x7f899c0699e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f899c132f80 0x7f899c135a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-c con 0x7f899c132f80 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:53.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f899c135fc0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f899c0a1910 con 0x7f899c135fc0 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f899c069dc0 con 0x7f899c0699e0 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f899c0a8290 con 0x7f899c132f80 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f89a15e9640 10 monclient(hunting): authenticate will time out at 2326.851012s 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f899c0699e0 0x7f899c132a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f899c135fc0 0x7f899c13cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48482/0 (socket says 172.21.15.86:48482) 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899b7fe640 1 -- 172.21.15.86:0/2120819989 learned_addr learned my addr 172.21.15.86:0/2120819989 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): get_auth_request con 0x7f899c0699e0 auth_method 0 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f8993fff640 1 --2- 172.21.15.86:0/2120819989 >> v2:172.21.15.203:3300/0 conn(0x7f899c132f80 0x7f899c135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f8993fff640 10 monclient(hunting): get_auth_request con 0x7f899c132f80 auth_method 0 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f8993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f8993fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f8993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient(hunting): handle_auth_done global_id 234351 payload 274 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 -- 172.21.15.86:0/2120819989 >> v2:172.21.15.203:3300/0 conn(0x7f899c132f80 msgr2=0x7f899c135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 --2- 172.21.15.86:0/2120819989 >> v2:172.21.15.203:3300/0 conn(0x7f899c132f80 0x7f899c135a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 -- 172.21.15.86:0/2120819989 >> v1:172.21.15.86:6789/0 conn(0x7f899c135fc0 legacy=0x7f899c13cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient: _finish_hunting 0 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 monclient: found mon.noname-a 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:53.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.881+0000 7f899affd640 1 -- 172.21.15.86:0/2120819989 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899c0a1070 con 0x7f899c0699e0 2024-09-17T15:49:53.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 -- 172.21.15.86:0/2120819989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c0699e0 msgr2=0x7f899c132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 --2- 172.21.15.86:0/2120819989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c0699e0 0x7f899c132a40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f899c070ce0 tx=0x7f898c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8549fef1650525a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 1 -- 172.21.15.86:0/2120819989 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c0b0070 con 0x7f899c0699e0 2024-09-17T15:49:53.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: dump: 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 1 -- 172.21.15.86:0/2120819989 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f898c0ab600 con 0x7f899c0699e0 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 5 monclient: authenticate success, global_id 234351 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 1 -- 172.21.15.86:0/2120819989 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c0ab9d0 con 0x7f899c0699e0 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 4 set_mon_vals no callback set 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899a7fc640 10 monclient: dump: 2024-09-17T15:49:53.891 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: shutdown 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- 172.21.15.86:0/2120819989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c0699e0 msgr2=0x7f899c132a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- 172.21.15.86:0/2120819989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c0699e0 0x7f899c132a40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f899c070ce0 tx=0x7f898c01c320 comp rx=0 tx=0).stop 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- 172.21.15.86:0/2120819989 shutdown_connections 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- 172.21.15.86:0/2120819989 >> v2:172.21.15.203:3300/0 conn(0x7f899c132f80 0x7f899c135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- 172.21.15.86:0/2120819989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c0699e0 0x7f899c132a40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- 172.21.15.86:0/2120819989 >> 172.21.15.86:0/2120819989 conn(0x7f899c0a3ef0 msgr2=0x7f899c0a42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- 172.21.15.86:0/2120819989 shutdown_connections 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- 172.21.15.86:0/2120819989 wait complete. 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: build_initial_monmap 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: monmap: 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:53.894 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.894 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:53.894 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:53.894 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:53.894 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 Processor -- start 2024-09-17T15:49:53.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- start start 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: init 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 0x7f899c0723b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-c con 0x7f899c0699e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f899c132f80 0x7f899c0728f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-b con 0x7f899c132f80 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 0x7f899c1d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): picked mon.noname-a con 0x7f899c135fc0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:53.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f899c140cf0 con 0x7f899c135fc0 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f899c133390 con 0x7f899c132f80 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f899c140e70 con 0x7f899c0699e0 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f89a15e9640 10 monclient(hunting): authenticate will time out at 2326.859061s 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 0x7f899c0723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 0x7f899c1d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 0x7f899c0723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59688/0 (socket says 172.21.15.86:59688) 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 -- 172.21.15.86:0/3901493694 learned_addr learned my addr 172.21.15.86:0/3901493694 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:53.901 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): get_auth_request con 0x7f899c0699e0 auth_method 0 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): get_auth_request con 0x7f899c135fc0 auth_method 0 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f8993fff640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f899c132f80 0x7f899c0728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient(hunting): handle_auth_done global_id 234354 payload 824 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 msgr2=0x7f899c0723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 0x7f899c0723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f899c132f80 msgr2=0x7f899c0728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f899c132f80 0x7f899c0728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 monclient: found mon.noname-a 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f899c1358a0 con 0x7f899c135fc0 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f899affd640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f899c0699e0 0x7f899c0723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:53.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f8993fff640 10 monclient: get_auth_request con 0x7f899c132f80 auth_method 0 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.889+0000 7f8993fff640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f899c132f80 0x7f899c0728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 msgr2=0x7f899c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899b7fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 0x7f899c1d0020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f899400caf0 tx=0x7f8994005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899400ad80 con 0x7f899c135fc0 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: dump: 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.904 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8994012400 con 0x7f899c135fc0 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 5 monclient: authenticate success, global_id 234354 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 monclient: _renew_subs 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f899c09e5e0 con 0x7f899c135fc0 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89940127f0 con 0x7f899c135fc0 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 10 monclient: dump: 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.906 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89997fa640 4 set_mon_vals no callback set 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 monclient: _renew_subs 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f899c1375f0 con 0x7f899c135fc0 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 0x7f899c1d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f899c074670 con 0x7f899c1d0560 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 0x7f899c1d0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient: get_auth_request con 0x7f899c1d0560 auth_method 0 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899b7fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 0x7f899c1d0020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f899400caf0 tx=0x7f8994005ad0 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f8994050440 osd_map(800..800 src has 696..800) 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f8994012a50 con 0x7f899c135fc0 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 10 monclient(hunting): handle_auth_done global_id 234357 payload 824 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f899affd640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f899c09d0f0 con 0x7f899c1d0560 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f89937fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f897c03bb90 0x7f897c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f8993fff640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f897c03bb90 0x7f897c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.893+0000 7f8993fff640 10 monclient: get_auth_request con 0x7f897c03bb90 auth_method 0 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f899affd640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 msgr2=0x7f899c1d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f899affd640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 0x7f899c1d0940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f898c01a9d0 tx=0x7f898c0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c0ba030 con 0x7f899c1d0560 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 10 monclient: dump: 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.909 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f898c0b6660 con 0x7f899c1d0560 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 msgr2=0x7f899c1d0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 0x7f899c1d0940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f898c01a9d0 tx=0x7f898c0190c0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 msgr2=0x7f899c1d0940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f89937fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c1d0560 0x7f899c1d0940 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f898c01a9d0 tx=0x7f898c0190c0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.897+0000 7f8993fff640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f897c03bb90 0x7f897c03e050 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 0x7f899c09ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f899c132a40 con 0x7f899c09e8f0 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 0x7f899c09ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient: get_auth_request con 0x7f899c09e8f0 auth_method 0 2024-09-17T15:49:53.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 10 monclient(hunting): handle_auth_done global_id 234360 payload 824 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 1 -- 172.21.15.86:0/3901493694 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f899c1d0e80 con 0x7f899c09e8f0 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 msgr2=0x7f899c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899affd640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 0x7f899c09ed00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f898c000c10 tx=0x7f898c0a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c0ba070 con 0x7f899c09e8f0 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:53.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 10 monclient: dump: 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f898c003830 con 0x7f899c09e8f0 2024-09-17T15:49:53.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 msgr2=0x7f899c09ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 0x7f899c09ed00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f898c000c10 tx=0x7f898c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 msgr2=0x7f899c09ed00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c09e8f0 0x7f899c09ed00 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f898c000c10 tx=0x7f898c0a8000 comp rx=0 tx=0).stop 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:53.915 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 123100, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:49:53.916 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:53.917 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:53.918 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.919 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:53.920 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:53.921 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:53.922 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:53.923 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:53.924 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:53.925 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.927 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:49:53.928 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:53.929 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f897c03bb90 msgr2=0x7f897c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f897c03bb90 0x7f897c03e050 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8984009840 tx=0x7f8984008040 comp rx=0 tx=0).stop 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 10 monclient: shutdown 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 msgr2=0x7f899c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 --2- 172.21.15.86:0/3901493694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f899c135fc0 0x7f899c1d0020 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f899400caf0 tx=0x7f8994005ad0 comp rx=0 tx=0).stop 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 reap_dead start 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f899b7fe640 1 -- 172.21.15.86:0/3901493694 reap_dead start 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89937fe640 1 -- 172.21.15.86:0/3901493694 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (unknown 0 0 0) 0x7f8994050440 con 0x7f899c135fc0 2024-09-17T15:49:53.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 shutdown_connections 2024-09-17T15:49:53.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 >> 172.21.15.86:0/3901493694 conn(0x7f899c0a3ef0 msgr2=0x7f899c1308c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:53.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 shutdown_connections 2024-09-17T15:49:53.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:53.905+0000 7f89a15e9640 1 -- 172.21.15.86:0/3901493694 wait complete. 2024-09-17T15:49:53.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:53.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:49:53.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:49:54.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: get_monmap_and_config 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: monmap: 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:54.049102+0000 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:54.049102+0000 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:54.045 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:54.046 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:54.046 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 Processor -- start 2024-09-17T15:49:54.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 -- start start 2024-09-17T15:49:54.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: init 2024-09-17T15:49:54.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92480a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f92480a5350 0x7f92480a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): picked mon.noname-c con 0x7f92480a5350 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f924813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): picked mon.noname-b con 0x7f924813d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92480a68d0 con 0x7f92480a4f70 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f92480a64a0 con 0x7f92480a5350 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f924809c230 con 0x7f924813d180 2024-09-17T15:49:54.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924e281640 10 monclient(hunting): authenticate will time out at 2327.014704s 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f9247fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f92480a4f70 0x7f924809cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48484/0 (socket says 172.21.15.86:48484) 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f9247fff640 1 -- 172.21.15.86:0/265765758 learned_addr learned my addr 172.21.15.86:0/265765758 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 --2- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f924813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): get_auth_request con 0x7f924813d180 auth_method 0 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f92477fe640 1 --2- 172.21.15.86:0/265765758 >> v2:172.21.15.203:3300/0 conn(0x7f92480a5350 0x7f92480a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f92477fe640 10 monclient(hunting): get_auth_request con 0x7f92480a5350 auth_method 0 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f92477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f92477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f92477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient(hunting): handle_auth_done global_id 165017 payload 274 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 -- 172.21.15.86:0/265765758 >> v2:172.21.15.203:3300/0 conn(0x7f92480a5350 msgr2=0x7f92480a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 --2- 172.21.15.86:0/265765758 >> v2:172.21.15.203:3300/0 conn(0x7f92480a5350 0x7f92480a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 -- 172.21.15.86:0/265765758 >> v1:172.21.15.86:6789/0 conn(0x7f92480a4f70 legacy=0x7f924809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 monclient: found mon.noname-b 2024-09-17T15:49:54.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.045+0000 7f924c81e640 1 -- 172.21.15.86:0/265765758 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924809c510 con 0x7f924813d180 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 1 -- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 msgr2=0x7f924813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 1 --2- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f924813f570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9240009d90 tx=0x7f924001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b74ec7244cb4e6a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 1 -- 172.21.15.86:0/265765758 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92400b2070 con 0x7f924813d180 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: dump: 2024-09-17T15:49:54.054 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 1 -- 172.21.15.86:0/265765758 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92400a5890 con 0x7f924813d180 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 1 -- 172.21.15.86:0/265765758 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92400a5c60 con 0x7f924813d180 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 5 monclient: authenticate success, global_id 165017 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 4 set_mon_vals no callback set 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f9246ffd640 10 monclient: dump: 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: shutdown 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 msgr2=0x7f924813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f924813f570 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9240009d90 tx=0x7f924001c830 comp rx=0 tx=0).stop 2024-09-17T15:49:54.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- 172.21.15.86:0/265765758 shutdown_connections 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- 172.21.15.86:0/265765758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f924813f570 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- 172.21.15.86:0/265765758 >> v2:172.21.15.203:3300/0 conn(0x7f92480a5350 0x7f92480a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- 172.21.15.86:0/265765758 >> 172.21.15.86:0/265765758 conn(0x7f92480a7240 msgr2=0x7f92480a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- 172.21.15.86:0/265765758 shutdown_connections 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- 172.21.15.86:0/265765758 wait complete. 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: monmap: 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 Processor -- start 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- start start 2024-09-17T15:49:54.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: init 2024-09-17T15:49:54.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:49:54.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92480a4f70 0x7f92481350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): picked mon.noname-b con 0x7f92480a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92480a5350 0x7f9248135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): picked mon.noname-a con 0x7f92480a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f92481cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): picked mon.noname-c con 0x7f924813d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9248142740 con 0x7f92480a5350 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f92481425c0 con 0x7f92480a4f70 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f92481428c0 con 0x7f924813d180 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924e281640 10 monclient(hunting): authenticate will time out at 2327.023877s 2024-09-17T15:49:54.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f92481cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f92477fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92480a5350 0x7f9248135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f92481cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59698/0 (socket says 172.21.15.86:59698) 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 learned_addr learned my addr 172.21.15.86:0/862320629 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f92477fe640 10 monclient(hunting): get_auth_request con 0x7f92480a5350 auth_method 0 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f92477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f92477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f92477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 10 monclient(hunting): get_auth_request con 0x7f924813d180 auth_method 0 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.053+0000 7f924c81e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f92477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92480a4f70 0x7f92481350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 10 monclient(hunting): get_auth_request con 0x7f92480a4f70 auth_method 0 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient(hunting): handle_auth_done global_id 165020 payload 824 2024-09-17T15:49:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92480a4f70 msgr2=0x7f92481350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92480a4f70 0x7f92481350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92480a5350 msgr2=0x7f9248135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92480a5350 0x7f9248135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 monclient: found mon.noname-c 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9248135b40 con 0x7f924813d180 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92480a4f70 0x7f92481350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 msgr2=0x7f92481cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.057+0000 7f924c81e640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f92481cfd30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92400b0040 tx=0x7f92400195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92400b2050 con 0x7f924813d180 2024-09-17T15:49:54.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: dump: 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92400be070 con 0x7f924813d180 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92400adc60 con 0x7f924813d180 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 5 monclient: authenticate success, global_id 165020 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 10 monclient: dump: 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.070 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f92467fc640 4 set_mon_vals no callback set 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 monclient: _renew_subs 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92480a0510 con 0x7f924813d180 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 monclient: _renew_subs 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f924809f1d0 con 0x7f924813d180 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 0x7f92481d0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f92480a5750 con 0x7f924809feb0 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f92400b9450 con 0x7f924813d180 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 0x7f92481d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient: get_auth_request con 0x7f924809feb0 auth_method 0 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f922803bbb0 0x7f922803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f92400ea990 con 0x7f924813d180 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 10 monclient(hunting): handle_auth_done global_id 165023 payload 824 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92480a5ff0 con 0x7f924809feb0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f92477fe640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f922803bbb0 0x7f922803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f92477fe640 10 monclient: get_auth_request con 0x7f922803bbb0 auth_method 0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 msgr2=0x7f92481d0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.065+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 0x7f92481d0270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f923000d6f0 tx=0x7f923000dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230005cf0 con 0x7f924809feb0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 10 monclient: dump: 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.073 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9230017070 con 0x7f924809feb0 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 msgr2=0x7f92481d0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 0x7f92481d0270 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f923000d6f0 tx=0x7f923000dbb0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 msgr2=0x7f92481d0270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:54.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f9244ff9640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924809feb0 0x7f92481d0270 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.069+0000 7f92477fe640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f922803bbb0 0x7f922803e070 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f924809bf50 tx=0x7f9238008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f924e281640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f924e281640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 0x7f92480a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f924e281640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f92481d0bc0 con 0x7f92481d07b0 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 0x7f92480a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient: get_auth_request con 0x7f92481d07b0 auth_method 0 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 10 monclient(hunting): handle_auth_done global_id 165026 payload 824 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.073+0000 7f9247fff640 1 -- 172.21.15.86:0/862320629 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92480a0290 con 0x7f92481d07b0 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9247fff640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 msgr2=0x7f92480a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9247fff640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 0x7f92480a1970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f923000d6f0 tx=0x7f9230004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230007630 con 0x7f92481d07b0 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 10 monclient: dump: 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.078 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f9230007630 con 0x7f92481d07b0 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 msgr2=0x7f92480a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 0x7f92480a1970 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f923000d6f0 tx=0x7f9230004950 comp rx=0 tx=0).stop 2024-09-17T15:49:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 msgr2=0x7f92480a1970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f9244ff9640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92481d07b0 0x7f92480a1970 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f923000d6f0 tx=0x7f9230004950 comp rx=0 tx=0).stop 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 497264, 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:54.080 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.081 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.082 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:54.083 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:54.084 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:54.085 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.086 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:54.087 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:54.089 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.090 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.091 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:54.092 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:54.093 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.094 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:54.095 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:54.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:54.097 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 6 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.099 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:54.099 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f922803bbb0 msgr2=0x7f922803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f922803bbb0 0x7f922803e070 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f924809bf50 tx=0x7f9238008040 comp rx=0 tx=0).stop 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 10 monclient: shutdown 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 msgr2=0x7f92481cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 --2- 172.21.15.86:0/862320629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f924813d180 0x7f92481cfd30 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f92400b0040 tx=0x7f92400195d0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 reap_dead start 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924c81e640 1 -- 172.21.15.86:0/862320629 reap_dead start 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 shutdown_connections 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 >> 172.21.15.86:0/862320629 conn(0x7f92480a7240 msgr2=0x7f924813ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 shutdown_connections 2024-09-17T15:49:54.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.077+0000 7f924e281640 1 -- 172.21.15.86:0/862320629 wait complete. 2024-09-17T15:49:54.101 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:54.101 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:49:54.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:49:54.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: get_monmap_and_config 2024-09-17T15:49:54.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: monmap: 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:54.220286+0000 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:54.220286+0000 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:54.217 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:54.218 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:54.218 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:54.218 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 Processor -- start 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 -- start start 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: init 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa33406dee0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 0x7fa3340a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-b con 0x7fa33406f8c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa33413d420 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa33406fca0 con 0x7fa33413d420 2024-09-17T15:49:54.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3340a4950 con 0x7fa33406dee0 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa33406e2c0 con 0x7fa33406f8c0 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa33aee4640 10 monclient(hunting): authenticate will time out at 2327.185915s 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa339481640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa33413d420 0x7fa33413f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48488/0 (socket says 172.21.15.86:48488) 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa339481640 1 -- 172.21.15.86:0/3812001283 learned_addr learned my addr 172.21.15.86:0/3812001283 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 1 --2- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 0x7fa3340a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): get_auth_request con 0x7fa33406f8c0 auth_method 0 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient(hunting): handle_auth_done global_id 165029 payload 274 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 1 -- 172.21.15.86:0/3812001283 >> v1:172.21.15.203:6789/0 conn(0x7fa33406dee0 legacy=0x7fa3340a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 1 -- 172.21.15.86:0/3812001283 >> v1:172.21.15.86:6789/0 conn(0x7fa33413d420 legacy=0x7fa33413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 1 monclient: found mon.noname-b 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.217+0000 7fa333fff640 1 -- 172.21.15.86:0/3812001283 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33406ec00 con 0x7fa33406f8c0 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa333fff640 1 -- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 msgr2=0x7fa3340a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa333fff640 1 --2- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 0x7fa3340a3580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa324009dc0 tx=0x7fa32401c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fc43d7bba30ac7e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 1 -- 172.21.15.86:0/3812001283 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240b2070 con 0x7fa33406f8c0 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:49:54.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: dump: 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.225 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 1 -- 172.21.15.86:0/3812001283 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3240a58e0 con 0x7fa33406f8c0 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa33aee4640 5 monclient: authenticate success, global_id 165029 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa33aee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 1 -- 172.21.15.86:0/3812001283 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240a5cb0 con 0x7fa33406f8c0 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa33aee4640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa33aee4640 4 set_mon_vals no callback set 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.221+0000 7fa3337fe640 10 monclient: dump: 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: shutdown 2024-09-17T15:49:54.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 msgr2=0x7fa3340a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 0x7fa3340a3580 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa324009dc0 tx=0x7fa32401c820 comp rx=0 tx=0).stop 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- 172.21.15.86:0/3812001283 shutdown_connections 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3812001283 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33406f8c0 0x7fa3340a3580 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- 172.21.15.86:0/3812001283 >> 172.21.15.86:0/3812001283 conn(0x7fa33409bc70 msgr2=0x7fa3340a8470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- 172.21.15.86:0/3812001283 shutdown_connections 2024-09-17T15:49:54.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- 172.21.15.86:0/3812001283 wait complete. 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: monmap: 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:54.233 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 Processor -- start 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- start start 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: init 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 0x7fa33409d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-a con 0x7fa33406dee0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:54.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa33406f8c0 0x7fa33409d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-b con 0x7fa33406f8c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 0x7fa33409ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): picked mon.noname-c con 0x7fa33413d420 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa3341438d0 con 0x7fa33406dee0 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa334143750 con 0x7fa33406f8c0 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa334143a50 con 0x7fa33413d420 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa33aee4640 10 monclient(hunting): authenticate will time out at 2327.194295s 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 0x7fa33409d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 0x7fa33409ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 0x7fa33409d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57712/0 (socket says 172.21.15.86:57712) 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 1 -- 172.21.15.86:0/3445105935 learned_addr learned my addr 172.21.15.86:0/3445105935 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): get_auth_request con 0x7fa33406dee0 auth_method 0 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): get_auth_request con 0x7fa33413d420 auth_method 0 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa333fff640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa33406f8c0 0x7fa33409d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient(hunting): handle_auth_done global_id 165032 payload 824 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa33406f8c0 msgr2=0x7fa33409d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa33406f8c0 0x7fa33409d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 msgr2=0x7fa33409d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 0x7fa33409d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 monclient: found mon.noname-c 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3341409c0 con 0x7fa33413d420 2024-09-17T15:49:54.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa338c80640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa33406dee0 0x7fa33409d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa333fff640 10 monclient: get_auth_request con 0x7fa33406f8c0 auth_method 0 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.225+0000 7fa333fff640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa33406f8c0 0x7fa33409d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 msgr2=0x7fa33409ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa339481640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 0x7fa33409ddb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa3340a4800 tx=0x7fa32c00c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c017070 con 0x7fa33413d420 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:54.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: dump: 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa32c0070d0 con 0x7fa33413d420 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32c0074c0 con 0x7fa33413d420 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 10 monclient: dump: 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa332ffd640 4 set_mon_vals no callback set 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 5 monclient: authenticate success, global_id 165032 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 monclient: _renew_subs 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33413aa80 con 0x7fa33413d420 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 monclient: _renew_subs 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3340738a0 con 0x7fa33413d420 2024-09-17T15:49:54.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 0x7fa3341d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa33406a0e0 con 0x7fa3341cfd10 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa32c01a070 con 0x7fa33413d420 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 0x7fa3341d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient: get_auth_request con 0x7fa3341cfd10 auth_method 0 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa31403bb60 0x7fa31403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7fa32c04f270 con 0x7fa33413d420 2024-09-17T15:49:54.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa333fff640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa31403bb60 0x7fa31403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa333fff640 10 monclient: get_auth_request con 0x7fa31403bb60 auth_method 0 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 10 monclient(hunting): handle_auth_done global_id 254125 payload 824 2024-09-17T15:49:54.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.229+0000 7fa338c80640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3340a2e70 con 0x7fa3341cfd10 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa338c80640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 msgr2=0x7fa3341d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa338c80640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 0x7fa3341d00f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00fb10 tx=0x7fa31c0045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa333fff640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa31403bb60 0x7fa31403e020 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa334071840 tx=0x7fa3240a7070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c018030 con 0x7fa3341cfd10 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:54.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 10 monclient: dump: 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa31c00de50 con 0x7fa3341cfd10 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 msgr2=0x7fa3341d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 0x7fa3341d00f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00fb10 tx=0x7fa31c0045a0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 msgr2=0x7fa3341d00f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.233+0000 7fa3317fa640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341cfd10 0x7fa3341d00f0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00fb10 tx=0x7fa31c0045a0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa33aee4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 0x7fa33413d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa33aee4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa33413acc0 con 0x7fa3341d0630 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 0x7fa33413d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient: get_auth_request con 0x7fa3341d0630 auth_method 0 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 10 monclient(hunting): handle_auth_done global_id 254128 payload 824 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.241+0000 7fa338c80640 1 -- 172.21.15.86:0/3445105935 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa334139840 con 0x7fa3341d0630 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa338c80640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 msgr2=0x7fa33413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa338c80640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 0x7fa33413d800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa31c018000 tx=0x7fa31c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c01ee40 con 0x7fa3341d0630 2024-09-17T15:49:54.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:54.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 10 monclient: dump: 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fa31c0049c0 con 0x7fa3341d0630 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:49:54.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 msgr2=0x7fa33413d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 0x7fa33413d800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa31c018000 tx=0x7fa31c007f90 comp rx=0 tx=0).stop 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 msgr2=0x7fa33413d800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa3317fa640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa3341d0630 0x7fa33413d800 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa31c018000 tx=0x7fa31c007f90 comp rx=0 tx=0).stop 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:49:54.251 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 206, 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10191, 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.252 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 6, 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.253 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.254 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:49:54.255 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:49:54.256 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.257 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:49:54.260 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.262 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:49:54.263 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:49:54.264 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:49:54.264 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:49:54.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:49:54.264 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:49:54.264 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:49:54.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa31403bb60 msgr2=0x7fa31403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa31403bb60 0x7fa31403e020 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fa334071840 tx=0x7fa3240a7070 comp rx=0 tx=0).stop 2024-09-17T15:49:54.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 10 monclient: shutdown 2024-09-17T15:49:54.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:54.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 msgr2=0x7fa33409ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 --2- 172.21.15.86:0/3445105935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa33413d420 0x7fa33409ddb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa3340a4800 tx=0x7fa32c00c870 comp rx=0 tx=0).stop 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 reap_dead start 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa339481640 1 -- 172.21.15.86:0/3445105935 reap_dead start 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 shutdown_connections 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 >> 172.21.15.86:0/3445105935 conn(0x7fa33409bc70 msgr2=0x7fa3340a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 shutdown_connections 2024-09-17T15:49:54.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.245+0000 7fa33aee4640 1 -- 172.21.15.86:0/3445105935 wait complete. 2024-09-17T15:49:54.266 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:49:54.267 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:49:54.267 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:49:54.267 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:49:54.267 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:49:54.267 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:49:54.267 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:49:54.284 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:49:54.284 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:49:54.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:54.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: get_monmap_and_config 2024-09-17T15:49:54.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: monmap: 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:54.445811+0000 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:54.445811+0000 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:54.443 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 Processor -- start 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 -- start start 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: init 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36a80689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f36a80694c0 0x7f36a806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-c con 0x7f36a80694c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 0x7f36a813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-b con 0x7f36a813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a8074470 con 0x7f36a80689d0 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f36a809c900 con 0x7f36a80694c0 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f36a809cb90 con 0x7f36a813aff0 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36af8e2640 10 monclient(hunting): authenticate will time out at 2327.411287s 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ade7f640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 msgr2=0x7f36a813d3e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ade7f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 0x7f36a813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ad67e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f36a80689d0 0x7f36a8068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:48504/0 (socket says 172.21.15.86:48504) 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ad67e640 1 -- 172.21.15.86:0/3373143163 learned_addr learned my addr 172.21.15.86:0/3373143163 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 1 --2- 172.21.15.86:0/3373143163 >> v2:172.21.15.203:3300/0 conn(0x7f36a80694c0 0x7f36a806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): get_auth_request con 0x7f36a80694c0 auth_method 0 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.441+0000 7f36ace7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 10 monclient(hunting): handle_auth_done global_id 254131 payload 274 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 1 -- 172.21.15.86:0/3373143163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 msgr2=0x7f36a813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 1 --2- 172.21.15.86:0/3373143163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 0x7f36a813d3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 1 -- 172.21.15.86:0/3373143163 >> v1:172.21.15.86:6789/0 conn(0x7f36a80689d0 legacy=0x7f36a8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 1 monclient: found mon.noname-c 2024-09-17T15:49:54.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:49:54.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.445+0000 7f36ace7d640 1 -- 172.21.15.86:0/3373143163 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a809ce70 con 0x7f36a80694c0 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ace7d640 1 -- 172.21.15.86:0/3373143163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80694c0 msgr2=0x7f36a806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ace7d640 1 --2- 172.21.15.86:0/3373143163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80694c0 0x7f36a806f840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36a40097c0 tx=0x7f36a401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f82e80e189c02b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 1 -- 172.21.15.86:0/3373143163 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a40b0070 con 0x7f36a80694c0 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:49:54.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: dump: 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 1 -- 172.21.15.86:0/3373143163 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a40ab600 con 0x7f36a80694c0 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 5 monclient: authenticate success, global_id 254131 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 4 set_mon_vals no callback set 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 1 -- 172.21.15.86:0/3373143163 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a809c2d0 con 0x7f36a80694c0 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f369ffff640 10 monclient: dump: 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.455 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: shutdown 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- 172.21.15.86:0/3373143163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80694c0 msgr2=0x7f36a806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- 172.21.15.86:0/3373143163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80694c0 0x7f36a806f840 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36a40097c0 tx=0x7f36a401c380 comp rx=0 tx=0).stop 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- 172.21.15.86:0/3373143163 shutdown_connections 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- 172.21.15.86:0/3373143163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a813aff0 0x7f36a813d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- 172.21.15.86:0/3373143163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80694c0 0x7f36a806f840 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.456 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- 172.21.15.86:0/3373143163 >> 172.21.15.86:0/3373143163 conn(0x7f36a80a6440 msgr2=0x7f36a80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- 172.21.15.86:0/3373143163 shutdown_connections 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- 172.21.15.86:0/3373143163 wait complete. 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: build_initial_monmap 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: monmap: 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:54.457 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 Processor -- start 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- start start 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: init 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80689d0 0x7f36a81cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-b con 0x7f36a80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 0x7f36a8135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-c con 0x7f36a80694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:54.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): picked mon.noname-a con 0x7f36a813aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f36a8140670 con 0x7f36a813aff0 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36a81404f0 con 0x7f36a80689d0 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f36a81407f0 con 0x7f36a80694c0 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36af8e2640 10 monclient(hunting): authenticate will time out at 2327.419810s 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ace7d640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 msgr2=0x7f36a8135e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ace7d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 0x7f36a8135e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:54.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:57734/0 (socket says 172.21.15.86:57734) 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 1 -- 172.21.15.86:0/2098902932 learned_addr learned my addr 172.21.15.86:0/2098902932 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 10 monclient(hunting): get_auth_request con 0x7f36a813aff0 auth_method 0 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.449+0000 7f36ade7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ad67e640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80689d0 0x7f36a81cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ad67e640 10 monclient(hunting): get_auth_request con 0x7f36a80689d0 auth_method 0 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ad67e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:54.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ad67e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient(hunting): handle_auth_done global_id 234369 payload 824 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 -- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 msgr2=0x7f36a8135e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 0x7f36a8135e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 -- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80689d0 msgr2=0x7f36a81cfd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80689d0 0x7f36a81cfd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient: _finish_hunting 0 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 monclient: found mon.noname-a 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 -- 172.21.15.86:0/2098902932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a80736b0 con 0x7f36a813aff0 2024-09-17T15:49:54.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 -- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 msgr2=0x7f36a8136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:54.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36a000be60 tx=0x7f36a000bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36ade7f640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36a000be60 tx=0x7f36a000bf50 comp rx=0 tx=0).queue_received will delay after 1.72659e+09 on 0x7f36a0016630 mon_map magic: 0 2024-09-17T15:49:54.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0016630 con 0x7f36a813aff0 2024-09-17T15:49:54.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: dump: 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a0010030 con 0x7f36a813aff0 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 5 monclient: authenticate success, global_id 234369 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _renew_subs 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a8122810 con 0x7f36a813aff0 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369effd640 4 set_mon_vals no callback set 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _renew_subs 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a8122530 con 0x7f36a813aff0 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a806f840 con 0x7f36a813aff0 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0016a10 con 0x7f36a813aff0 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 10 monclient: dump: 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:54.466 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.453+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f36a00153e0 con 0x7f36a813aff0 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.457+0000 7f369dffb640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 0x7f368c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.457+0000 7f36ad67e640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 0x7f368c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.457+0000 7f36ad67e640 10 monclient: get_auth_request con 0x7f368c03bb10 auth_method 0 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.457+0000 7f36ad67e640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 0x7f368c03dfd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3698007fd0 tx=0x7f3698009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:54.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:54.461+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f36a0050900 con 0x7f36a813aff0 2024-09-17T15:49:55.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.077+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f36a004f320 con 0x7f36a813aff0 2024-09-17T15:49:55.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.077+0000 7f369dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:55.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.077+0000 7f369dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:55.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f36af8e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:55.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f36af8e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:55.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f36af8e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:55.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f36a806d4f0 con 0x7f36a813aff0 2024-09-17T15:49:55.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f369dffb640 1 -- 172.21.15.86:0/2098902932 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f36a0013cf0 con 0x7f36a813aff0 2024-09-17T15:49:55.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f369dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:55.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f369dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:55.306 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.305+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 msgr2=0x7f368c03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 0x7f368c03dfd0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3698007fd0 tx=0x7f3698009040 comp rx=0 tx=0).stop 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 10 monclient: shutdown 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 msgr2=0x7f36a8136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36a000be60 tx=0x7f36a000bf50 comp rx=0 tx=0).stop 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 shutdown_connections 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f368c03bb10 0x7f368c03dfd0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36a813aff0 0x7f36a8136380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:55.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36a80694c0 0x7f36a8135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:55.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 --2- 172.21.15.86:0/2098902932 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36a80689d0 0x7f36a81cfd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:55.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 >> 172.21.15.86:0/2098902932 conn(0x7f36a80a6440 msgr2=0x7f36a809df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:55.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 shutdown_connections 2024-09-17T15:49:55.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:55.309+0000 7f36af8e2640 1 -- 172.21.15.86:0/2098902932 wait complete. 2024-09-17T15:49:58.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:49:58.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: get_monmap_and_config 2024-09-17T15:49:58.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: monmap: 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:49:58.457601+0000 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:49:58.457601+0000 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:49:58.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 Processor -- start 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 -- start start 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: init 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fddf80689d0 0x7fddf806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): picked mon.noname-c con 0x7fddf80689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:49:58.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddf8068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fddf813aff0 0x7fddf813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): picked mon.noname-a con 0x7fddf813aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddf809c8b0 con 0x7fddf8068db0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fddf8074500 con 0x7fddf813aff0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fddf809cb90 con 0x7fddf80689d0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddff621640 10 monclient(hunting): authenticate will time out at 2331.423402s 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fddf813aff0 0x7fddf813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fddf813aff0 0x7fddf813d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39360/0 (socket says 172.21.15.86:39360) 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 1 -- 172.21.15.86:0/823325099 learned_addr learned my addr 172.21.15.86:0/823325099 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): get_auth_request con 0x7fddf813aff0 auth_method 0 2024-09-17T15:49:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfd3bd640 1 --2- 172.21.15.86:0/823325099 >> v2:172.21.15.203:3300/0 conn(0x7fddf80689d0 0x7fddf806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfd3bd640 10 monclient(hunting): get_auth_request con 0x7fddf80689d0 auth_method 0 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfd3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfd3bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfd3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.453+0000 7fddfdbbe640 10 monclient(hunting): handle_auth_done global_id 234372 payload 274 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 1 -- 172.21.15.86:0/823325099 >> v2:172.21.15.203:3300/0 conn(0x7fddf80689d0 msgr2=0x7fddf806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:58.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 1 --2- 172.21.15.86:0/823325099 >> v2:172.21.15.203:3300/0 conn(0x7fddf80689d0 0x7fddf806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 1 -- 172.21.15.86:0/823325099 >> v1:172.21.15.203:6789/0 conn(0x7fddf8068db0 legacy=0x7fddf8069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 1 monclient: found mon.noname-a 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.457+0000 7fddfdbbe640 1 -- 172.21.15.86:0/823325099 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf809ce70 con 0x7fddf813aff0 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfdbbe640 1 -- 172.21.15.86:0/823325099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf813aff0 msgr2=0x7fddf813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfdbbe640 1 --2- 172.21.15.86:0/823325099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf813aff0 0x7fddf813d3e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fddf8069940 tx=0x7fddf001c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15b45987902f14a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 1 -- 172.21.15.86:0/823325099 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf809c2d0 con 0x7fddf813aff0 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: dump: 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:58.464 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 1 -- 172.21.15.86:0/823325099 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddf00a5730 con 0x7fddf813aff0 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 5 monclient: authenticate success, global_id 234372 2024-09-17T15:49:58.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: get_monmap_and_config success 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 1 -- 172.21.15.86:0/823325099 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddf00a5b00 con 0x7fddf813aff0 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 4 set_mon_vals no callback set 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddeffff640 10 monclient: dump: 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.466 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: shutdown 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- 172.21.15.86:0/823325099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf813aff0 msgr2=0x7fddf813d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- 172.21.15.86:0/823325099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf813aff0 0x7fddf813d3e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fddf8069940 tx=0x7fddf001c890 comp rx=0 tx=0).stop 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- 172.21.15.86:0/823325099 shutdown_connections 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- 172.21.15.86:0/823325099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf813aff0 0x7fddf813d3e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- 172.21.15.86:0/823325099 >> v2:172.21.15.203:3300/0 conn(0x7fddf80689d0 0x7fddf806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- 172.21.15.86:0/823325099 >> 172.21.15.86:0/823325099 conn(0x7fddf80a6440 msgr2=0x7fddf80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.467 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- 172.21.15.86:0/823325099 shutdown_connections 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- 172.21.15.86:0/823325099 wait complete. 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: build_initial_monmap 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: monmap: 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.468 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 Processor -- start 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- start start 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: init 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: _reopen_session rank -1 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddf80689d0 0x7fddf81cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): picked mon.noname-b con 0x7fddf80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): picked mon.noname-a con 0x7fddf8068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 0x7fddf8070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): picked mon.noname-c con 0x7fddf813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fddf81405f0 con 0x7fddf8068db0 2024-09-17T15:49:58.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fddf8140470 con 0x7fddf80689d0 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): start opening mon connection 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fddf8140770 con 0x7fddf813aff0 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): _renew_subs 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddff621640 10 monclient(hunting): authenticate will time out at 2331.430511s 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfdbbe640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 msgr2=0x7fddf8070e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfdbbe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 0x7fddf8070e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39364/0 (socket says 172.21.15.86:39364) 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 -- 172.21.15.86:0/359747960 learned_addr learned my addr 172.21.15.86:0/359747960 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): get_auth_request con 0x7fddf8068db0 auth_method 0 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:58.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfd3bd640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddf80689d0 0x7fddf81cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfd3bd640 10 monclient(hunting): get_auth_request con 0x7fddf80689d0 auth_method 0 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfd3bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfd3bd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfd3bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:49:58.471 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient(hunting): handle_auth_done global_id 234375 payload 824 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 -- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 msgr2=0x7fddf8070e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 0x7fddf8070e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 -- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddf80689d0 msgr2=0x7fddf81cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddf80689d0 0x7fddf81cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient: _finish_hunting 0 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 monclient: found mon.noname-a 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.472 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.461+0000 7fddfcbbc640 1 -- 172.21.15.86:0/359747960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf811d4d0 con 0x7fddf8068db0 2024-09-17T15:49:58.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddfcbbc640 1 -- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 msgr2=0x7fddf8132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:49:58.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddfcbbc640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdde800c450 tx=0x7fdde800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:58.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde80119a0 con 0x7fddf8068db0 2024-09-17T15:49:58.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: dump: 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.475 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: _finish_auth 0 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: _check_auth_tickets 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 5 monclient: authenticate success, global_id 234375 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 10 monclient: _renew_subs 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde8017070 con 0x7fddf8068db0 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddf809d460 con 0x7fddf8068db0 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde8011da0 con 0x7fddf8068db0 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddedffb640 10 monclient: dump: 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:49:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 10 monclient: _renew_subs 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddeeffd640 4 set_mon_vals no callback set 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.465+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddf81ce870 con 0x7fddf8068db0 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddff621640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddff621640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddff621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddf806f840 con 0x7fddf8068db0 2024-09-17T15:49:58.478 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdde801c460 con 0x7fddf8068db0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddedffb640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 0x7fdddc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7fdde804fdb0 con 0x7fddf8068db0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddfd3bd640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 0x7fdddc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.469+0000 7fddfd3bd640 10 monclient: get_auth_request con 0x7fdddc03bb60 auth_method 0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.473+0000 7fddfd3bd640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 0x7fdddc03e020 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fddf400b9d0 tx=0x7fddf4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.473+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdde801ac60 con 0x7fddf8068db0 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.473+0000 7fddedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:49:58.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.473+0000 7fddedffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:49:58.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddff621640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddff621640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddff621640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fddf806d4f0 con 0x7fddf8068db0 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddedffb640 1 -- 172.21.15.86:0/359747960 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fdde8010cf0 con 0x7fddf8068db0 2024-09-17T15:49:58.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.697+0000 7fddedffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:49:58.699 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 msgr2=0x7fdddc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 0x7fdddc03e020 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fddf400b9d0 tx=0x7fddf4008040 comp rx=0 tx=0).stop 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 10 monclient: shutdown 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 msgr2=0x7fddf8132b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdde800c450 tx=0x7fdde800c910 comp rx=0 tx=0).stop 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 shutdown_connections 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fdddc03bb60 0x7fdddc03e020 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fddf813aff0 0x7fddf8070e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fddf8068db0 0x7fddf8132b40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 --2- 172.21.15.86:0/359747960 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fddf80689d0 0x7fddf81cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 >> 172.21.15.86:0/359747960 conn(0x7fddf80a6440 msgr2=0x7fddf813bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 shutdown_connections 2024-09-17T15:49:58.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:49:58.701+0000 7fddff621640 1 -- 172.21.15.86:0/359747960 wait complete. 2024-09-17T15:50:01.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: get_monmap_and_config 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: monmap: 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:01.849490+0000 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:01.849490+0000 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:01.846 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 Processor -- start 2024-09-17T15:50:01.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 -- start start 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: init 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f60640689d0 0x7f606406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-c con 0x7f60640689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6064068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f606413aff0 0x7f606413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-a con 0x7f606413aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606409c8b0 con 0x7f6064068db0 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6064074500 con 0x7f606413aff0 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:01.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f606409cb90 con 0x7f60640689d0 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60697d2640 10 monclient(hunting): authenticate will time out at 2334.815132s 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f606413aff0 0x7f606413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f606413aff0 0x7f606413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39380/0 (socket says 172.21.15.86:39380) 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 -- 172.21.15.86:0/4272915326 learned_addr learned my addr 172.21.15.86:0/4272915326 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): get_auth_request con 0x7f606413aff0 auth_method 0 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f6062ffd640 1 --2- 172.21.15.86:0/4272915326 >> v2:172.21.15.203:3300/0 conn(0x7f60640689d0 0x7f606406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f6062ffd640 10 monclient(hunting): get_auth_request con 0x7f60640689d0 auth_method 0 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f6062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f6062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f6062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient(hunting): handle_auth_done global_id 234378 payload 274 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 -- 172.21.15.86:0/4272915326 >> v2:172.21.15.203:3300/0 conn(0x7f60640689d0 msgr2=0x7f606406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 --2- 172.21.15.86:0/4272915326 >> v2:172.21.15.203:3300/0 conn(0x7f60640689d0 0x7f606406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 -- 172.21.15.86:0/4272915326 >> v1:172.21.15.203:6789/0 conn(0x7f6064068db0 legacy=0x7f6064069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 monclient: found mon.noname-a 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:01.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.844+0000 7f60637fe640 1 -- 172.21.15.86:0/4272915326 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606409ce70 con 0x7f606413aff0 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 -- 172.21.15.86:0/4272915326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 msgr2=0x7f606413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 --2- 172.21.15.86:0/4272915326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f606413d3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6064069940 tx=0x7f605401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c1ff9e1de3171b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 1 -- 172.21.15.86:0/4272915326 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60540b1070 con 0x7f606413aff0 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: dump: 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:01.853 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 1 -- 172.21.15.86:0/4272915326 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60540ac600 con 0x7f606413aff0 2024-09-17T15:50:01.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 5 monclient: authenticate success, global_id 234378 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 1 -- 172.21.15.86:0/4272915326 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f606409c2d0 con 0x7f606413aff0 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 4 set_mon_vals no callback set 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6061ffb640 10 monclient: dump: 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: shutdown 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- 172.21.15.86:0/4272915326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 msgr2=0x7f606413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- 172.21.15.86:0/4272915326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f606413d3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6064069940 tx=0x7f605401c3c0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- 172.21.15.86:0/4272915326 shutdown_connections 2024-09-17T15:50:01.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- 172.21.15.86:0/4272915326 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f606413d3e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- 172.21.15.86:0/4272915326 >> v2:172.21.15.203:3300/0 conn(0x7f60640689d0 0x7f606406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- 172.21.15.86:0/4272915326 >> 172.21.15.86:0/4272915326 conn(0x7f60640a6440 msgr2=0x7f60640a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:01.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- 172.21.15.86:0/4272915326 shutdown_connections 2024-09-17T15:50:01.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- 172.21.15.86:0/4272915326 wait complete. 2024-09-17T15:50:01.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: build_initial_monmap 2024-09-17T15:50:01.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: monmap: 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:01.861 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 Processor -- start 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- start start 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: init 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 0x7f60641cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-c con 0x7f60640689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6064068db0 0x7f6064132b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-b con 0x7f6064068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): picked mon.noname-a con 0x7f606413aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f60641405f0 con 0x7f606413aff0 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6064140470 con 0x7f6064068db0 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6064140770 con 0x7f60640689d0 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60697d2640 10 monclient(hunting): authenticate will time out at 2334.823184s 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6062ffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 msgr2=0x7f60641cda40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f6062ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 0x7f60641cda40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:39396/0 (socket says 172.21.15.86:39396) 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 -- 172.21.15.86:0/2749306502 learned_addr learned my addr 172.21.15.86:0/2749306502 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): get_auth_request con 0x7f606413aff0 auth_method 0 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60627fc640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6064068db0 0x7f6064132b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60627fc640 10 monclient(hunting): get_auth_request con 0x7f6064068db0 auth_method 0 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:01.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60627fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient(hunting): handle_auth_done global_id 234381 payload 824 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 -- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 msgr2=0x7f60641cda40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 0x7f60641cda40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 -- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6064068db0 msgr2=0x7f6064132b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6064068db0 0x7f6064132b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 monclient: found mon.noname-a 2024-09-17T15:50:01.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.852+0000 7f60637fe640 1 -- 172.21.15.86:0/2749306502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606411d4d0 con 0x7f606413aff0 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.856+0000 7f60637fe640 1 -- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 msgr2=0x7f6064070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.856+0000 7f60637fe640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f605401cc60 tx=0x7f605401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.856+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60540b1040 con 0x7f606413aff0 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: dump: 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.866 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60540ac400 con 0x7f606413aff0 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60540b67e0 con 0x7f606413aff0 2024-09-17T15:50:01.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 10 monclient: dump: 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:01.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60617fa640 4 set_mon_vals no callback set 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 5 monclient: authenticate success, global_id 234381 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _renew_subs 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606409d460 con 0x7f606413aff0 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _renew_subs 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:01.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60641ce8c0 con 0x7f606413aff0 2024-09-17T15:50:01.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606406f840 con 0x7f606413aff0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f60540be030 con 0x7f606413aff0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 0x7f604803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f60540e98f0 con 0x7f606413aff0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f6062ffd640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 0x7f604803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.860+0000 7f6062ffd640 10 monclient: get_auth_request con 0x7f604803bb60 auth_method 0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.864+0000 7f6062ffd640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 0x7f604803e020 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f605c009660 tx=0x7f605c009930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.864+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f60540c5060 con 0x7f606413aff0 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.864+0000 7f605b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:01.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:01.864+0000 7f605b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:02.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f60697d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f60697d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f60697d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f606406d4f0 con 0x7f606413aff0 2024-09-17T15:50:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f605b7fe640 1 -- 172.21.15.86:0/2749306502 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f60540160d0 con 0x7f606413aff0 2024-09-17T15:50:02.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f605b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:02.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.088+0000 7f605b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:02.093 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":206,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:02.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 msgr2=0x7f604803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 0x7f604803e020 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f605c009660 tx=0x7f605c009930 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 10 monclient: shutdown 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 msgr2=0x7f6064070e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f605401cc60 tx=0x7f605401ac00 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 shutdown_connections 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f604803bb60 0x7f604803e020 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f606413aff0 0x7f6064070e60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6064068db0 0x7f6064132b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 --2- 172.21.15.86:0/2749306502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60640689d0 0x7f60641cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 >> 172.21.15.86:0/2749306502 conn(0x7f60640a6440 msgr2=0x7f606413bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:02.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 shutdown_connections 2024-09-17T15:50:02.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:02.092+0000 7f60697d2640 1 -- 172.21.15.86:0/2749306502 wait complete. 2024-09-17T15:50:05.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: get_monmap_and_config 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: build_initial_monmap 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: monmap: 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:05.245380+0000 2024-09-17T15:50:05.241 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:05.245380+0000 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:05.242 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 Processor -- start 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 -- start start 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: init 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e58069260 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:05.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-c con 0x7f3e58074c30 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-a con 0x7f3e5809c6c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5809e410 con 0x7f3e58069260 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5809d150 con 0x7f3e5809c6c0 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5809ce80 con 0x7f3e58074c30 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5e4f0640 10 monclient(hunting): authenticate will time out at 2338.210900s 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e57fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3e58069260 0x7f3e5809ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37736/0 (socket says 172.21.15.86:37736) 2024-09-17T15:50:05.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 learned_addr learned my addr 172.21.15.86:0/1204677329 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e5809c6c0 auth_method 0 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e58074c30 auth_method 0 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:05.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:50:05.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.240+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:05.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.244+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 >> v1:172.21.15.86:6789/0 conn(0x7f3e58069260 legacy=0x7f3e5809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e5809c6c0 auth_method 0 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:05.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e58074c30 auth_method 0 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:05.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:50:05.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.444+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e5809c6c0 auth_method 0 2024-09-17T15:50:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:05.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:05.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e58074c30 auth_method 0 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:05.848+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:06.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:06.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e58074c30 auth_method 0 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e5809c6c0 auth_method 0 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:06.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:50:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:50:06.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:06.652+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:50:08.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): tick 2024-09-17T15:50:08.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:50:08.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 monclient(hunting): continuing hunt 2024-09-17T15:50:08.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:50:08.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 msgr2=0x7f3e5809c180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.203:3300/0 conn(0x7f3e58074c30 0x7f3e5809c180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 -- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 msgr2=0x7f3e5813ba80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 --2- 172.21.15.86:0/1204677329 >> v2:172.21.15.86:3300/0 conn(0x7f3e5809c6c0 0x7f3e5813ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 -- 172.21.15.86:0/1204677329 >> v1:172.21.15.86:6789/0 conn(0x7f3e58069260 legacy=0x7f3e5809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient: _add_conns ranks=[2,1] 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 0x7f3e44005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e44003770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e440062b0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 -- 172.21.15.86:0/1204677329 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5809e6f0 con 0x7f3e440062b0 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 1 -- 172.21.15.86:0/1204677329 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5809d150 con 0x7f3e44003770 2024-09-17T15:50:08.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e55ffb640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:08.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 msgr2=0x7f3e44005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:08.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.240+0000 7f3e57fff640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 0x7f3e44005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:08.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.244+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> v1:172.21.15.203:6789/0 conn(0x7f3e440062b0 legacy=0x7f3e44008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:08.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.444+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 msgr2=0x7f3e44005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:08.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.444+0000 7f3e57fff640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 0x7f3e44005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:08.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.848+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 msgr2=0x7f3e44005b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:08.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:08.848+0000 7f3e57fff640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 0x7f3e44005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974570082 0 0) 0x7f3e5809e6f0 con 0x7f3e440062b0 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): _negotiate 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): my global_id is 254149 2024-09-17T15:50:09.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e40003530 con 0x7f3e440062b0 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2807957062 0 0) 0x7f3e40003530 con 0x7f3e440062b0 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 msgr2=0x7f3e44005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e44003770 0x7f3e44005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4400b180 con 0x7f3e440062b0 2024-09-17T15:50:09.127 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f3e4800f040 con 0x7f3e440062b0 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: dump: 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.128 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 >> v1:172.21.15.203:6789/0 conn(0x7f3e440062b0 legacy=0x7f3e44008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 1 -- 172.21.15.86:0/1204677329 reap_dead start 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e40003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): picked mon.c con 0x7f3e40003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): picked mon.a con 0x7f3e40004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e4000ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): picked mon.b con 0x7f3e40005420 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5809d150 con 0x7f3e40004810 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4000c330 con 0x7f3e40005420 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e40010940 con 0x7f3e40003a30 2024-09-17T15:50:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e56ffd640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e57fff640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 msgr2=0x7f3e40003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e57fff640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e40003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e40004810 auth_method 0 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e4000ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:09.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e40005420 auth_method 0 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5ca8d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient(hunting): handle_auth_done global_id 254149 payload 293 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 msgr2=0x7f3e40003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e40003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 msgr2=0x7f3e4000ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e4000ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 monclient: found mon.a 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e40003530 con 0x7f3e40004810 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5e4f0640 5 monclient: authenticate success, global_id 254149 2024-09-17T15:50:09.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.124+0000 7f3e5e4f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e577fe640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 msgr2=0x7f3e40004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e577fe640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e40004c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000ee0 tx=0x7f3e4801d8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554cea42841d1cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4800ee60 con 0x7f3e40004810 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: dump: 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.135 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e480ab630 con 0x7f3e40004810 2024-09-17T15:50:09.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 1 -- 172.21.15.86:0/1204677329 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480aba00 con 0x7f3e40004810 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e56ffd640 10 monclient: dump: 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.137 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e5e4f0640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:09.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e5e4f0640 4 set_mon_vals no callback set 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.132+0000 7f3e5e4f0640 10 monclient: shutdown 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 msgr2=0x7f3e40004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e40004c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3e48000ee0 tx=0x7f3e4801d8b0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/1204677329 shutdown_connections 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e4000ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e40004c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/1204677329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e40003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/1204677329 >> 172.21.15.86:0/1204677329 conn(0x7f3e580a6440 msgr2=0x7f3e580a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/1204677329 shutdown_connections 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/1204677329 wait complete. 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient: build_initial_monmap 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient: monmap: 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:09.143 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 Processor -- start 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- start start 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient: init 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e581c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-c con 0x7f3e40003a30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-a con 0x7f3e40004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e581cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): picked mon.noname-b con 0x7f3e40005420 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5809be00 con 0x7f3e40004810 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5809bc80 con 0x7f3e40005420 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5809bf80 con 0x7f3e40003a30 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5e4f0640 10 monclient(hunting): authenticate will time out at 2342.106728s 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e57fff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 msgr2=0x7f3e581c7e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e57fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e581c7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58576/0 (socket says 172.21.15.86:58576) 2024-09-17T15:50:09.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 -- 172.21.15.86:0/3359720394 learned_addr learned my addr 172.21.15.86:0/3359720394 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): get_auth_request con 0x7f3e40004810 auth_method 0 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5ca8d640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e581cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request con 0x7f3e40005420 auth_method 0 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5ca8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e5ca8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient(hunting): handle_auth_done global_id 234384 payload 824 2024-09-17T15:50:09.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 -- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 msgr2=0x7f3e581c7e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e581c7e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 -- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 msgr2=0x7f3e581cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e581cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 monclient: found mon.noname-a 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.136+0000 7f3e577fe640 1 -- 172.21.15.86:0/3359720394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e58121950 con 0x7f3e40004810 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e577fe640 1 -- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 msgr2=0x7f3e58133fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e577fe640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e4801c010 tx=0x7f3e4801a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4800ee60 con 0x7f3e40004810 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: dump: 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4800e570 con 0x7f3e40004810 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480bc640 con 0x7f3e40004810 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 5 monclient: authenticate success, global_id 234384 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _renew_subs 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e58120610 con 0x7f3e40004810 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 10 monclient: dump: 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.151 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e567fc640 4 set_mon_vals no callback set 2024-09-17T15:50:09.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _renew_subs 2024-09-17T15:50:09.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e58121170 con 0x7f3e40004810 2024-09-17T15:50:09.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3e480c5440 con 0x7f3e40004810 2024-09-17T15:50:09.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 0x7f3e4403e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e581200f0 con 0x7f3e40004810 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f3e480eb3e0 con 0x7f3e40004810 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.140+0000 7f3e57fff640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 0x7f3e4403e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.144+0000 7f3e57fff640 10 monclient: get_auth_request con 0x7f3e4403c4d0 auth_method 0 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.144+0000 7f3e57fff640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 0x7f3e4403e990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0014b0 tx=0x7f3e4c007a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.148+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f3e581200f0 con 0x7f3e40004810 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.148+0000 7f3e54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.148+0000 7f3e54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:09.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e5e4f0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:09.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e5e4f0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:09.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e5e4f0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e5809c310 con 0x7f3e40004810 2024-09-17T15:50:09.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e54ff9640 1 -- 172.21.15.86:0/3359720394 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f3e480af020 con 0x7f3e40004810 2024-09-17T15:50:09.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:09.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.388+0000 7f3e54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:09.390 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":210,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:09.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 msgr2=0x7f3e4403e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 0x7f3e4403e990 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c0014b0 tx=0x7f3e4c007a90 comp rx=0 tx=0).stop 2024-09-17T15:50:09.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 10 monclient: shutdown 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 msgr2=0x7f3e58133fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3e4801c010 tx=0x7f3e4801a750 comp rx=0 tx=0).stop 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 shutdown_connections 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3e4403c4d0 0x7f3e4403e990 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3e40005420 0x7f3e581cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e40004810 0x7f3e58133fa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 --2- 172.21.15.86:0/3359720394 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e40003a30 0x7f3e581c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 >> 172.21.15.86:0/3359720394 conn(0x7f3e580a6440 msgr2=0x7f3e5806ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 shutdown_connections 2024-09-17T15:50:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.392+0000 7f3e5e4f0640 1 -- 172.21.15.86:0/3359720394 wait complete. 2024-09-17T15:50:09.409 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:50:09.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:50:09.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.524+0000 7f62bde42640 10 monclient: get_monmap_and_config 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.524+0000 7f62bde42640 10 monclient: build_initial_monmap 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient: monmap: 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:09.532040+0000 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:09.532040+0000 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:09.528 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:09.529 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:09.529 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 Processor -- start 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 -- start start 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient: init 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62b809d770 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62b809e260 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f62b813b620 0x7f62b813da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-c con 0x7f62b813b620 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b809ff00 con 0x7f62b809d770 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b809ec30 con 0x7f62b809e260 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f62b809e7f0 con 0x7f62b813b620 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62bde42640 10 monclient(hunting): authenticate will time out at 2342.497546s 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b77fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f62b809d770 0x7f62b809db50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46184/0 (socket says 172.21.15.86:46184) 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b77fe640 1 -- 172.21.15.86:0/2595550995 learned_addr learned my addr 172.21.15.86:0/2595550995 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 1 --2- 172.21.15.86:0/2595550995 >> v2:172.21.15.203:3300/0 conn(0x7f62b813b620 0x7f62b813da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): get_auth_request con 0x7f62b813b620 auth_method 0 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.528+0000 7f62b77fe640 1 -- 172.21.15.86:0/2595550995 >> v1:172.21.15.86:6789/0 conn(0x7f62b809d770 legacy=0x7f62b809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 1 -- 172.21.15.86:0/2595550995 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 151365919 0 0) 0x7f62b809ff00 con 0x7f62b809d770 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 10 monclient(hunting): _negotiate 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 10 monclient(hunting): my global_id is 234387 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b67fc640 1 -- 172.21.15.86:0/2595550995 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a0003530 con 0x7f62b809d770 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b6ffd640 1 -- 172.21.15.86:0/2595550995 >> v1:172.21.15.203:6789/0 conn(0x7f62b809e260 legacy=0x7f62b8139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 10 monclient(hunting): handle_auth_done global_id 254155 payload 274 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 1 -- 172.21.15.86:0/2595550995 >> v1:172.21.15.203:6789/0 conn(0x7f62b809e260 legacy=0x7f62b8139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 1 -- 172.21.15.86:0/2595550995 >> v1:172.21.15.86:6789/0 conn(0x7f62b809d770 legacy=0x7f62b809db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 1 monclient: found mon.noname-c 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:50:09.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.532+0000 7f62b7fff640 1 -- 172.21.15.86:0/2595550995 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b809ef10 con 0x7f62b813b620 2024-09-17T15:50:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b7fff640 1 -- 172.21.15.86:0/2595550995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 msgr2=0x7f62b813da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.536 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b7fff640 1 --2- 172.21.15.86:0/2595550995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 0x7f62b813da10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e90 tx=0x7f62b001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24161824bcbe505d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 1 -- 172.21.15.86:0/2595550995 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b00a66f0 con 0x7f62b813b620 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: dump: 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.540 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 1 -- 172.21.15.86:0/2595550995 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b00a6890 con 0x7f62b813b620 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 1 -- 172.21.15.86:0/2595550995 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b00a6c60 con 0x7f62b813b620 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b67fc640 10 monclient: dump: 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.541 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 5 monclient: authenticate success, global_id 254155 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 4 set_mon_vals no callback set 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: shutdown 2024-09-17T15:50:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- 172.21.15.86:0/2595550995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 msgr2=0x7f62b813da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 --2- 172.21.15.86:0/2595550995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 0x7f62b813da10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009e90 tx=0x7f62b001c930 comp rx=0 tx=0).stop 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- 172.21.15.86:0/2595550995 shutdown_connections 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 --2- 172.21.15.86:0/2595550995 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 0x7f62b813da10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- 172.21.15.86:0/2595550995 >> 172.21.15.86:0/2595550995 conn(0x7f62b80a2f40 msgr2=0x7f62b80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- 172.21.15.86:0/2595550995 shutdown_connections 2024-09-17T15:50:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- 172.21.15.86:0/2595550995 wait complete. 2024-09-17T15:50:09.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: build_initial_monmap 2024-09-17T15:50:09.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: monmap: 2024-09-17T15:50:09.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:09.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:09.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 Processor -- start 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- start start 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: init 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:50:09.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 0x7f62b8133570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-a con 0x7f62b809d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62b809e260 0x7f62b8133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-c con 0x7f62b809e260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 0x7f62b811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): picked mon.noname-b con 0x7f62b813b620 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81408f0 con 0x7f62b809d770 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f62b8140770 con 0x7f62b813b620 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f62b8140a70 con 0x7f62b809e260 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62bde42640 10 monclient(hunting): authenticate will time out at 2342.507855s 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b6ffd640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62b809e260 msgr2=0x7f62b8133ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b6ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62b809e260 0x7f62b8133ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 0x7f62b8133570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 0x7f62b8133570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58582/0 (socket says 172.21.15.86:58582) 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 1 -- 172.21.15.86:0/3401180553 learned_addr learned my addr 172.21.15.86:0/3401180553 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 10 monclient(hunting): get_auth_request con 0x7f62b809d770 auth_method 0 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.536+0000 7f62b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient(hunting): handle_auth_done global_id 234390 payload 824 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62b809e260 msgr2=0x7f62b8133ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62b809e260 0x7f62b8133ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 msgr2=0x7f62b811c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f62b813b620 0x7f62b811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 monclient: found mon.noname-a 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b8134690 con 0x7f62b809d770 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 msgr2=0x7f62b8133570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b77fe640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 0x7f62b8133570 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c700 tx=0x7f62a800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8017070 con 0x7f62b809d770 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: dump: 2024-09-17T15:50:09.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a8005d10 con 0x7f62b809d770 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a80072b0 con 0x7f62b809d770 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 10 monclient: dump: 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.554 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 5 monclient: authenticate success, global_id 234390 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 monclient: _renew_subs 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b5ffb640 4 set_mon_vals no callback set 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b809d460 con 0x7f62b809d770 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 monclient: _renew_subs 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b811ff50 con 0x7f62b809d770 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 0x7f62b802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f62b8120120 con 0x7f62b811d5c0 2024-09-17T15:50:09.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 0x7f62b802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient: get_auth_request con 0x7f62b811d5c0 auth_method 0 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f62a801a070 con 0x7f62b809d770 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_done global_id 234393 payload 824 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.540+0000 7f62b6ffd640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62b809db50 con 0x7f62b811d5c0 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62a403bb10 0x7f62a403dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f62a804f390 con 0x7f62b809d770 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f62b7fff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62a403bb10 0x7f62a403dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f62b7fff640 10 monclient: get_auth_request con 0x7f62a403bb10 auth_method 0 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f62b6ffd640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 msgr2=0x7f62b802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f62b6ffd640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 0x7f62b802d450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f62b813b0d0 tx=0x7f62ac00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac017070 con 0x7f62b811d5c0 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 10 monclient: dump: 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.557 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f62ac0057d0 con 0x7f62b811d5c0 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 msgr2=0x7f62b802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 0x7f62b802d450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f62b813b0d0 tx=0x7f62ac00c810 comp rx=0 tx=0).stop 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 msgr2=0x7f62b802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f629ffff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d5c0 0x7f62b802d450 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f62b813b0d0 tx=0x7f62ac00c810 comp rx=0 tx=0).stop 2024-09-17T15:50:09.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.544+0000 7f62b7fff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62a403bb10 0x7f62a403dfd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f62b00ac000 tx=0x7f62b0004ac0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62bde42640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62bde42640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 0x7f62b802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62bde42640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f62b81375f0 con 0x7f62b811d9a0 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 0x7f62b802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient: get_auth_request con 0x7f62b811d9a0 auth_method 0 2024-09-17T15:50:09.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 10 monclient(hunting): handle_auth_done global_id 234396 payload 824 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 1 -- 172.21.15.86:0/3401180553 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f62b806d4f0 con 0x7f62b811d9a0 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 msgr2=0x7f62b802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f62b6ffd640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 0x7f62b802d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62ac014000 tx=0x7f62ac000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac017070 con 0x7f62b811d9a0 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 10 monclient: dump: 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f62ac018040 con 0x7f62b811d9a0 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:09.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 msgr2=0x7f62b802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 0x7f62b802d990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62ac014000 tx=0x7f62ac000bd0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 msgr2=0x7f62b802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.552+0000 7f629ffff640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b811d9a0 0x7f62b802d990 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62ac014000 tx=0x7f62ac000bd0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 210, 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 138747, 2024-09-17T15:50:09.563 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.565 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:09.566 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.567 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:09.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.570 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.571 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.572 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.573 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.574 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.575 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:09.576 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62a403bb10 msgr2=0x7f62a403dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f62a403bb10 0x7f62a403dfd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f62b00ac000 tx=0x7f62b0004ac0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 10 monclient: shutdown 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 msgr2=0x7f62b8133570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 --2- 172.21.15.86:0/3401180553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62b809d770 0x7f62b8133570 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c700 tx=0x7f62a800cbc0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62b7fff640 1 -- 172.21.15.86:0/3401180553 reap_dead start 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62b7fff640 1 -- 172.21.15.86:0/3401180553 reap_dead start 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 shutdown_connections 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 >> 172.21.15.86:0/3401180553 conn(0x7f62b80a2f40 msgr2=0x7f62b80a1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 shutdown_connections 2024-09-17T15:50:09.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.556+0000 7f62bde42640 1 -- 172.21.15.86:0/3401180553 wait complete. 2024-09-17T15:50:09.579 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:50:09.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:50:09.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.688+0000 7f2a73b52640 10 monclient: get_monmap_and_config 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.688+0000 7f2a73b52640 10 monclient: build_initial_monmap 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.688+0000 7f2a73b52640 10 monclient: monmap: 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:09.695420+0000 2024-09-17T15:50:09.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:09.695420+0000 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:09.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 Processor -- start 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 -- start start 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient: init 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a6c09d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a6c09e260 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a6c13da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-b con 0x7f2a6c13b620 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6c09ff00 con 0x7f2a6c09e260 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6c09ec30 con 0x7f2a6c09d770 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c09e7f0 con 0x7f2a6c13b620 2024-09-17T15:50:09.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 10 monclient(hunting): authenticate will time out at 2342.660907s 2024-09-17T15:50:09.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a720ef640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 msgr2=0x7f2a6c13da10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a720ef640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a6c13da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a710ed640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2a6c09e260 0x7f2a6c139f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46200/0 (socket says 172.21.15.86:46200) 2024-09-17T15:50:09.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a710ed640 1 -- 172.21.15.86:0/1394966720 learned_addr learned my addr 172.21.15.86:0/1394966720 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a710ed640 1 -- 172.21.15.86:0/1394966720 >> v1:172.21.15.86:6789/0 conn(0x7f2a6c09e260 legacy=0x7f2a6c139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913280850 0 0) 0x7f2a6c09ff00 con 0x7f2a6c09e260 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): _negotiate 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): my global_id is 234399 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a58003530 con 0x7f2a6c09e260 2024-09-17T15:50:09.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a718ee640 1 -- 172.21.15.86:0/1394966720 >> v1:172.21.15.203:6789/0 conn(0x7f2a6c09d770 legacy=0x7f2a6c09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1795157006 0 0) 0x7f2a58003530 con 0x7f2a6c09e260 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 msgr2=0x7f2a6c13da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a6c13da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 >> v1:172.21.15.203:6789/0 conn(0x7f2a6c09d770 legacy=0x7f2a6c09db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c09ef10 con 0x7f2a6c09e260 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f2a5c0019b0 con 0x7f2a6c09e260 2024-09-17T15:50:09.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 5 monclient: authenticate success, global_id 234399 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a73b52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: dump: 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 >> v1:172.21.15.86:6789/0 conn(0x7f2a6c09e260 legacy=0x7f2a6c139f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58003ab0 0x7f2a58003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): picked mon.b con 0x7f2a58003ab0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a58004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): picked mon.c con 0x7f2a6c13b620 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 0x7f2a5800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): picked mon.a con 0x7f2a58004fb0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.692+0000 7f2a708ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a710ed640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 msgr2=0x7f2a58004850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5800bf10 con 0x7f2a58004fb0 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5800c090 con 0x7f2a58003ab0 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2a5800c210 con 0x7f2a6c13b620 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a710ed640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a58004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a708ec640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a73b52640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 0x7f2a5800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): get_auth_request con 0x7f2a58004fb0 auth_method 0 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:09.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a718ee640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58003ab0 0x7f2a58003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient(hunting): handle_auth_done global_id 234399 payload 293 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 msgr2=0x7f2a58004850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c13b620 0x7f2a58004850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58003ab0 msgr2=0x7f2a58003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58003ab0 0x7f2a58003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 monclient: found mon.a 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 1 -- 172.21.15.86:0/1394966720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a58003530 con 0x7f2a58004fb0 2024-09-17T15:50:09.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a720ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a718ee640 10 monclient: get_auth_request con 0x7f2a58003ab0 auth_method 0 2024-09-17T15:50:09.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.696+0000 7f2a718ee640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58003ab0 0x7f2a58003ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:50:09.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 msgr2=0x7f2a5800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 0x7f2a5800b9d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a64003990 tx=0x7f2a6401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57921ece9601f39d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64009520 con 0x7f2a58004fb0 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: dump: 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.711 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a64003bc0 con 0x7f2a58004fb0 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 1 -- 172.21.15.86:0/1394966720 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64005070 con 0x7f2a58004fb0 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 4 set_mon_vals no callback set 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a708ec640 10 monclient: dump: 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.712 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: shutdown 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 msgr2=0x7f2a5800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 --2- 172.21.15.86:0/1394966720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a58004fb0 0x7f2a5800b9d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a64003990 tx=0x7f2a6401c420 comp rx=0 tx=0).stop 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/1394966720 reap_dead start 2024-09-17T15:50:09.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- 172.21.15.86:0/1394966720 shutdown_connections 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- 172.21.15.86:0/1394966720 >> 172.21.15.86:0/1394966720 conn(0x7f2a6c0a2f40 msgr2=0x7f2a6c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- 172.21.15.86:0/1394966720 shutdown_connections 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- 172.21.15.86:0/1394966720 wait complete. 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: build_initial_monmap 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: monmap: 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:09.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 Processor -- start 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- start start 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: init 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58004fb0 0x7f2a6c1cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-b con 0x7f2a58004fb0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c1cdf20 0x7f2a6c1ce360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-c con 0x7f2a6c1cdf20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 0x7f2a6c134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): picked mon.noname-a con 0x7f2a6c134850 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c13e3a0 con 0x7f2a6c134850 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c13e220 con 0x7f2a58004fb0 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c13e520 con 0x7f2a6c1cdf20 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a73b52640 10 monclient(hunting): authenticate will time out at 2342.671042s 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a710ed640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c1cdf20 msgr2=0x7f2a6c1ce360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a710ed640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c1cdf20 0x7f2a6c1ce360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 0x7f2a6c134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 0x7f2a6c134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58626/0 (socket says 172.21.15.86:58626) 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 learned_addr learned my addr 172.21.15.86:0/210481907 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:09.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): get_auth_request con 0x7f2a6c134850 auth_method 0 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58004fb0 0x7f2a6c1cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient(hunting): handle_auth_done global_id 234402 payload 824 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c1cdf20 msgr2=0x7f2a6c1ce360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a6c1cdf20 0x7f2a6c1ce360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58004fb0 msgr2=0x7f2a6c1cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58004fb0 0x7f2a6c1cd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient: _finish_hunting 0 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 monclient: found mon.noname-a 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c09d460 con 0x7f2a6c134850 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a718ee640 10 monclient: get_auth_request con 0x7f2a58004fb0 auth_method 0 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.700+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a58004fb0 0x7f2a6c1cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 msgr2=0x7f2a6c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a720ef640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 0x7f2a6c134c90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2a6401c900 tx=0x7f2a58005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64009500 con 0x7f2a6c134850 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: dump: 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.718 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a64003970 con 0x7f2a6c134850 2024-09-17T15:50:09.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a64005580 con 0x7f2a6c134850 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 10 monclient: dump: 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.720 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a63fff640 4 set_mon_vals no callback set 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 5 monclient: authenticate success, global_id 234402 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 monclient: _renew_subs 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c13abd0 con 0x7f2a6c134850 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 monclient: _renew_subs 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c11e120 con 0x7f2a6c134850 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 0x7f2a6c135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c0a5180 con 0x7f2a6c1351d0 2024-09-17T15:50:09.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2a640b6030 con 0x7f2a6c134850 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a5003bb90 0x7f2a5003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 0x7f2a6c135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(800..800 src has 696..800) ==== 4739+0+0 (secure 0 0 0) 0x7f2a640e8e40 con 0x7f2a6c134850 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient: get_auth_request con 0x7f2a6c1351d0 auth_method 0 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.704+0000 7f2a710ed640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a5003bb90 0x7f2a5003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a710ed640 10 monclient: get_auth_request con 0x7f2a5003bb90 auth_method 0 2024-09-17T15:50:09.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a718ee640 10 monclient(hunting): handle_auth_done global_id 254164 payload 824 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a718ee640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a6c09e7f0 con 0x7f2a6c1351d0 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a718ee640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 msgr2=0x7f2a6c135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 0x7f2a6c135ee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ea10 tx=0x7f2a6800eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a710ed640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a5003bb90 0x7f2a5003e050 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c001c90 tx=0x7f2a5c000ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6800c9c0 con 0x7f2a6c1351d0 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 10 monclient: dump: 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.723 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.724 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.724 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.724 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.724 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2a68018070 con 0x7f2a6c1351d0 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 msgr2=0x7f2a6c135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 0x7f2a6c135ee0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ea10 tx=0x7f2a6800eed0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 msgr2=0x7f2a6c135ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.708+0000 7f2a627fc640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c1351d0 0x7f2a6c135ee0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a73b52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a73b52640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 0x7f2a6c136830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a73b52640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:09.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c06d4f0 con 0x7f2a6c136420 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 0x7f2a6c136830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient: get_auth_request con 0x7f2a6c136420 auth_method 0 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 10 monclient(hunting): handle_auth_done global_id 254167 payload 824 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.716+0000 7f2a718ee640 1 -- 172.21.15.86:0/210481907 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a6c1355b0 con 0x7f2a6c136420 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a718ee640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 msgr2=0x7f2a6c136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a718ee640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 0x7f2a6c136830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ea10 tx=0x7f2a68001840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6801de00 con 0x7f2a6c136420 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 10 monclient: dump: 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:09.728 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f2a68007490 con 0x7f2a6c136420 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 msgr2=0x7f2a6c136830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 0x7f2a6c136830 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ea10 tx=0x7f2a68001840 comp rx=0 tx=0).stop 2024-09-17T15:50:09.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 msgr2=0x7f2a6c136830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a627fc640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a6c136420 0x7f2a6c136830 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800ea10 tx=0x7f2a68001840 comp rx=0 tx=0).stop 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 210, 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 25666, 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:09.730 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.731 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.732 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:09.733 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:09.734 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.735 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.736 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:09.737 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.738 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.739 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.740 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:50:09.741 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:09.742 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a5003bb90 msgr2=0x7f2a5003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a5003bb90 0x7f2a5003e050 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c001c90 tx=0x7f2a5c000ed0 comp rx=0 tx=0).stop 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 10 monclient: shutdown 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 msgr2=0x7f2a6c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 --2- 172.21.15.86:0/210481907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a6c134850 0x7f2a6c134c90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2a6401c900 tx=0x7f2a58005840 comp rx=0 tx=0).stop 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 reap_dead start 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a720ef640 1 -- 172.21.15.86:0/210481907 reap_dead start 2024-09-17T15:50:09.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 shutdown_connections 2024-09-17T15:50:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 >> 172.21.15.86:0/210481907 conn(0x7f2a6c0a2f40 msgr2=0x7f2a6c071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 shutdown_connections 2024-09-17T15:50:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:09.720+0000 7f2a73b52640 1 -- 172.21.15.86:0/210481907 wait complete. 2024-09-17T15:50:09.746 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:50:09.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:50:10.119 INFO:tasks.rados.rados.0.smithi203.stdout:3659: done (1 left) 2024-09-17T15:50:10.119 INFO:tasks.rados.rados.0.smithi203.stdout:3660: done (0 left) 2024-09-17T15:50:10.119 INFO:tasks.rados.rados.0.smithi203.stdout:3661: snap_remove snap 376 2024-09-17T15:50:10.859 INFO:tasks.rados.rados.0.smithi203.stdout:3661: done (0 left) 2024-09-17T15:50:10.859 INFO:tasks.rados.rados.0.smithi203.stdout:3662: write oid 36 current snap is 395 2024-09-17T15:50:10.859 INFO:tasks.rados.rados.0.smithi203.stdout:3662: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2024-09-17T15:50:10.884 INFO:tasks.rados.rados.0.smithi203.stdout:3662: writing smithi20328086-36 from 712960 to 1378984 tid 1 2024-09-17T15:50:10.903 INFO:tasks.rados.rados.0.smithi203.stdout:3662: writing smithi20328086-36 from 1985775 to 2567226 tid 2 2024-09-17T15:50:10.915 INFO:tasks.rados.rados.0.smithi203.stdout:3662: writing smithi20328086-36 from 3234019 to 3512960 tid 3 2024-09-17T15:50:10.915 INFO:tasks.rados.rados.0.smithi203.stdout:3663: write oid 28 current snap is 395 2024-09-17T15:50:10.916 INFO:tasks.rados.rados.0.smithi203.stdout:3662: finishing write tid 1 to smithi20328086-36 2024-09-17T15:50:10.916 INFO:tasks.rados.rados.0.smithi203.stdout:3662: finishing write tid 2 to smithi20328086-36 2024-09-17T15:50:10.916 INFO:tasks.rados.rados.0.smithi203.stdout:3663: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-17T15:50:10.932 INFO:tasks.rados.rados.0.smithi203.stdout:3663: writing smithi20328086-28 from 729767 to 1471040 tid 1 2024-09-17T15:50:10.946 INFO:tasks.rados.rados.0.smithi203.stdout:3663: writing smithi20328086-28 from 1927904 to 2653013 tid 2 2024-09-17T15:50:10.955 INFO:tasks.rados.rados.0.smithi203.stdout:3663: writing smithi20328086-28 from 3145089 to 3529767 tid 3 2024-09-17T15:50:10.955 INFO:tasks.rados.rados.0.smithi203.stdout:3664: copy_from oid 42 from oid 22 current snap is 395 2024-09-17T15:50:10.955 INFO:tasks.rados.rados.0.smithi203.stdout:3662: finishing write tid 3 to smithi20328086-36 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: finishing write tid 4 to smithi20328086-36 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: finishing write tid 5 to smithi20328086-36 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: oid 36 updating version 0 to 1917 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: oid 36 version 1917 is already newer than 1915 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: oid 36 version 1917 is already newer than 1916 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: oid 36 updating version 1917 to 1918 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 1918 (ObjNum 1280 snap 395 seq_num 1280) dirty exists 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: left oid 36 (ObjNum 1280 snap 395 seq_num 1280) 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3663: finishing write tid 1 to smithi20328086-28 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3663: finishing write tid 2 to smithi20328086-28 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3662: done (2 left) 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3665: read oid 16 snap -1 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3665: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:10.956 INFO:tasks.rados.rados.0.smithi203.stdout:3666: delete oid 32 current snap is 395 2024-09-17T15:50:10.964 INFO:tasks.rados.rados.0.smithi203.stdout:3665: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:10.984 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 747610 2024-09-17T15:50:11.045 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:11.091 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:11.131 INFO:tasks.rados.rados.0.smithi203.stdout:3663: finishing write tid 3 to smithi20328086-28 2024-09-17T15:50:11.131 INFO:tasks.rados.rados.0.smithi203.stdout:3663: finishing write tid 4 to smithi20328086-28 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: finishing write tid 5 to smithi20328086-28 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: oid 28 updating version 0 to 1923 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: oid 28 version 1923 is already newer than 1920 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: oid 28 version 1923 is already newer than 1922 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: oid 28 version 1923 is already newer than 1921 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1923 (ObjNum 1281 snap 395 seq_num 1281) dirty exists 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3663: left oid 28 (ObjNum 1281 snap 395 seq_num 1281) 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3664: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:50:11.132 INFO:tasks.rados.rados.0.smithi203.stdout:3664: finishing copy_from to smithi20328086-42 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1345 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3663: done (3 left) 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3664: done (2 left) 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3665: done (1 left) 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3666: done (0 left) 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3667: read oid 1 snap -1 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3667: expect (ObjNum 1257 snap 380 seq_num 1257) 2024-09-17T15:50:11.133 INFO:tasks.rados.rados.0.smithi203.stdout:3668: snap_create 2024-09-17T15:50:11.136 INFO:tasks.rados.rados.0.smithi203.stdout:3667: expect (ObjNum 1257 snap 380 seq_num 1257) 2024-09-17T15:50:11.304 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 440469 2024-09-17T15:50:13.079 INFO:tasks.rados.rados.0.smithi203.stdout:3667: done (1 left) 2024-09-17T15:50:13.079 INFO:tasks.rados.rados.0.smithi203.stdout:3668: done (0 left) 2024-09-17T15:50:13.079 INFO:tasks.rados.rados.0.smithi203.stdout:3669: write oid 33 current snap is 396 2024-09-17T15:50:13.079 INFO:tasks.rados.rados.0.smithi203.stdout:3669: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-17T15:50:13.097 INFO:tasks.rados.rados.0.smithi203.stdout:3669: writing smithi20328086-33 from 763381 to 1255152 tid 1 2024-09-17T15:50:13.121 INFO:tasks.rados.rados.0.smithi203.stdout:3669: writing smithi20328086-33 from 2044868 to 2773646 tid 2 2024-09-17T15:50:13.132 INFO:tasks.rados.rados.0.smithi203.stdout:3669: writing smithi20328086-33 from 3199935 to 3563381 tid 3 2024-09-17T15:50:13.132 INFO:tasks.rados.rados.0.smithi203.stdout:3670: write (excl) oid 38 current snap is 396 2024-09-17T15:50:13.132 INFO:tasks.rados.rados.0.smithi203.stdout:3670: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-17T15:50:13.146 INFO:tasks.rados.rados.0.smithi203.stdout:3670: writing smithi20328086-38 from 780188 to 1347208 tid 1 2024-09-17T15:50:13.157 INFO:tasks.rados.rados.0.smithi203.stdout:3670: writing smithi20328086-38 from 2103350 to 2575786 tid 2 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3670: writing smithi20328086-38 from 3227358 to 3580188 tid 3 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3671: read oid 30 snap -1 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3671: expect (ObjNum 1275 snap 391 seq_num 1275) 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3672: read oid 32 snap -1 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3672: expect deleted 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3673: read oid 4 snap -1 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3673: expect deleted 2024-09-17T15:50:13.166 INFO:tasks.rados.rados.0.smithi203.stdout:3674: write oid 43 current snap is 396 2024-09-17T15:50:13.167 INFO:tasks.rados.rados.0.smithi203.stdout:3669: finishing write tid 1 to smithi20328086-33 2024-09-17T15:50:13.167 INFO:tasks.rados.rados.0.smithi203.stdout:3674: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-17T15:50:13.177 INFO:tasks.rados.rados.0.smithi203.stdout:3674: writing smithi20328086-43 from 796995 to 1555617 tid 1 2024-09-17T15:50:13.187 INFO:tasks.rados.rados.0.smithi203.stdout:3674: writing smithi20328086-43 from 2161832 to 2894279 tid 2 2024-09-17T15:50:13.187 INFO:tasks.rados.rados.0.smithi203.stdout:3675: snap_remove snap 390 2024-09-17T15:50:14.086 INFO:tasks.rados.rados.0.smithi203.stdout:3675: done (6 left) 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3676: delete oid 13 current snap is 396 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: finishing write tid 2 to smithi20328086-33 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: finishing write tid 3 to smithi20328086-33 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: finishing write tid 4 to smithi20328086-33 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: finishing write tid 5 to smithi20328086-33 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: oid 33 updating version 0 to 1423 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: oid 33 updating version 1423 to 1424 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: oid 33 version 1424 is already newer than 1422 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: oid 33 updating version 1424 to 1425 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1425 (ObjNum 1283 snap 396 seq_num 1283) dirty exists 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3669: left oid 33 (ObjNum 1283 snap 396 seq_num 1283) 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3670: finishing write tid 1 to smithi20328086-38 2024-09-17T15:50:14.087 INFO:tasks.rados.rados.0.smithi203.stdout:3670: finishing write tid 2 to smithi20328086-38 2024-09-17T15:50:14.088 INFO:tasks.rados.rados.0.smithi203.stdout:3670: finishing write tid 3 to smithi20328086-38 2024-09-17T15:50:14.088 INFO:tasks.rados.rados.0.smithi203.stdout:3671: expect (ObjNum 1275 snap 391 seq_num 1275) 2024-09-17T15:50:14.221 INFO:tasks.rados.rados.0.smithi203.stdout:3670: finishing write tid 4 to smithi20328086-38 2024-09-17T15:50:14.221 INFO:tasks.rados.rados.0.smithi203.stdout:3670: finishing write tid 5 to smithi20328086-38 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3670: oid 38 updating version 0 to 1398 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3670: oid 38 version 1398 is already newer than 1396 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3670: oid 38 version 1398 is already newer than 1395 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3670: oid 38 version 1398 is already newer than 1397 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1398 (ObjNum 1284 snap 396 seq_num 1284) dirty exists 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3670: left oid 38 (ObjNum 1284 snap 396 seq_num 1284) 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: finishing write tid 1 to smithi20328086-43 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: finishing write tid 2 to smithi20328086-43 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: finishing write tid 3 to smithi20328086-43 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: finishing write tid 4 to smithi20328086-43 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: oid 43 updating version 0 to 704 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: oid 43 version 704 is already newer than 703 2024-09-17T15:50:14.222 INFO:tasks.rados.rados.0.smithi203.stdout:3674: oid 43 updating version 704 to 705 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 705 (ObjNum 1285 snap 396 seq_num 1285) dirty exists 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3674: left oid 43 (ObjNum 1285 snap 396 seq_num 1285) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3669: done (6 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3670: done (5 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3671: done (4 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3672: done (3 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3673: done (2 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3674: done (1 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3676: done (0 left) 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3677: write oid 35 current snap is 396 2024-09-17T15:50:14.223 INFO:tasks.rados.rados.0.smithi203.stdout:3677: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-17T15:50:14.234 INFO:tasks.rados.rados.0.smithi203.stdout:3677: writing smithi20328086-35 from 430609 to 939729 tid 1 2024-09-17T15:50:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3677: writing smithi20328086-35 from 1478796 to 2214912 tid 2 2024-09-17T15:50:14.270 INFO:tasks.rados.rados.0.smithi203.stdout:3677: writing smithi20328086-35 from 2909627 to 3511884 tid 3 2024-09-17T15:50:14.270 INFO:tasks.rados.rados.0.smithi203.stdout:3678: rollback oid 44 current snap is 396 2024-09-17T15:50:14.270 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 395 2024-09-17T15:50:14.271 INFO:tasks.rados.rados.0.smithi203.stdout:3677: finishing write tid 1 to smithi20328086-35 2024-09-17T15:50:14.271 INFO:tasks.rados.rados.0.smithi203.stdout:3677: finishing write tid 2 to smithi20328086-35 2024-09-17T15:50:14.271 INFO:tasks.rados.rados.0.smithi203.stdout:3679: read oid 50 snap 393 2024-09-17T15:50:14.271 INFO:tasks.rados.rados.0.smithi203.stdout:3679: expect deleted 2024-09-17T15:50:14.271 INFO:tasks.rados.rados.0.smithi203.stdout:3680: delete oid 40 current snap is 396 2024-09-17T15:50:14.279 INFO:tasks.rados.rados.0.smithi203.stdout:3677: finishing write tid 3 to smithi20328086-35 2024-09-17T15:50:14.279 INFO:tasks.rados.rados.0.smithi203.stdout:3677: finishing write tid 4 to smithi20328086-35 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: finishing write tid 5 to smithi20328086-35 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: oid 35 updating version 0 to 1678 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: oid 35 updating version 1678 to 1679 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: oid 35 version 1679 is already newer than 1677 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: oid 35 version 1679 is already newer than 1676 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1679 (ObjNum 1287 snap 396 seq_num 1287) dirty exists 2024-09-17T15:50:14.280 INFO:tasks.rados.rados.0.smithi203.stdout:3677: left oid 35 (ObjNum 1287 snap 396 seq_num 1287) 2024-09-17T15:50:14.286 INFO:tasks.rados.rados.0.smithi203.stdout:3678: finishing rollback tid 0 to smithi20328086-44 2024-09-17T15:50:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3678: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:50:14.291 INFO:tasks.rados.rados.0.smithi203.stdout:3678: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:50:14.291 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1666 (ObjNum 1169 snap 359 seq_num 1169) dirty exists 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3677: done (3 left) 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3678: done (2 left) 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3679: done (1 left) 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3680: done (0 left) 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3681: write oid 21 current snap is 396 2024-09-17T15:50:14.293 INFO:tasks.rados.rados.0.smithi203.stdout:3681: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-17T15:50:14.307 INFO:tasks.rados.rados.0.smithi203.stdout:3681: writing smithi20328086-21 from 464223 to 1123841 tid 1 2024-09-17T15:50:14.326 INFO:tasks.rados.rados.0.smithi203.stdout:3681: writing smithi20328086-21 from 1879407 to 2619192 tid 2 2024-09-17T15:50:14.338 INFO:tasks.rados.rados.0.smithi203.stdout:3681: writing smithi20328086-21 from 3248120 to 3664223 tid 3 2024-09-17T15:50:14.338 INFO:tasks.rados.rados.0.smithi203.stdout:3682: snap_remove snap 382 2024-09-17T15:50:15.099 INFO:tasks.rados.rados.0.smithi203.stdout:3682: done (1 left) 2024-09-17T15:50:15.099 INFO:tasks.rados.rados.0.smithi203.stdout:3683: write (excl) oid 33 current snap is 396 2024-09-17T15:50:15.100 INFO:tasks.rados.rados.0.smithi203.stdout:3681: finishing write tid 1 to smithi20328086-21 2024-09-17T15:50:15.100 INFO:tasks.rados.rados.0.smithi203.stdout:3683: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-17T15:50:15.118 INFO:tasks.rados.rados.0.smithi203.stdout:3683: writing smithi20328086-33 from 481030 to 1215897 tid 1 2024-09-17T15:50:15.134 INFO:tasks.rados.rados.0.smithi203.stdout:3683: writing smithi20328086-33 from 1937889 to 2421332 tid 2 2024-09-17T15:50:15.147 INFO:tasks.rados.rados.0.smithi203.stdout:3683: writing smithi20328086-33 from 2875543 to 3427969 tid 3 2024-09-17T15:50:15.147 INFO:tasks.rados.rados.0.smithi203.stdout:3684: snap_remove snap 381 2024-09-17T15:50:15.859 INFO:tasks.rados.rados.0.smithi203.stdout:3684: done (2 left) 2024-09-17T15:50:15.859 INFO:tasks.rados.rados.0.smithi203.stdout:3685: write oid 28 current snap is 396 2024-09-17T15:50:15.859 INFO:tasks.rados.rados.0.smithi203.stdout:3685: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-17T15:50:15.875 INFO:tasks.rados.rados.0.smithi203.stdout:3685: writing smithi20328086-28 from 497837 to 907953 tid 1 2024-09-17T15:50:15.894 INFO:tasks.rados.rados.0.smithi203.stdout:3685: writing smithi20328086-28 from 1596371 to 2339825 tid 2 2024-09-17T15:50:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:3685: writing smithi20328086-28 from 2902966 to 3666664 tid 3 2024-09-17T15:50:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:3686: write (excl) oid 44 current snap is 396 2024-09-17T15:50:15.909 INFO:tasks.rados.rados.0.smithi203.stdout:3686: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-17T15:50:15.921 INFO:tasks.rados.rados.0.smithi203.stdout:3686: writing smithi20328086-44 from 514644 to 1000009 tid 1 2024-09-17T15:50:15.931 INFO:tasks.rados.rados.0.smithi203.stdout:3686: writing smithi20328086-44 from 1654853 to 2141965 tid 2 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3686: writing smithi20328086-44 from 2930389 to 3505359 tid 3 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3687: read oid 32 snap -1 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3687: expect deleted 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3688: snap_create 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3681: finishing write tid 2 to smithi20328086-21 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3681: finishing write tid 3 to smithi20328086-21 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3681: finishing write tid 4 to smithi20328086-21 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3681: finishing write tid 5 to smithi20328086-21 2024-09-17T15:50:15.943 INFO:tasks.rados.rados.0.smithi203.stdout:3681: oid 21 updating version 0 to 1931 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3681: oid 21 version 1931 is already newer than 1929 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3681: oid 21 updating version 1931 to 1932 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3681: oid 21 version 1932 is already newer than 1930 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1932 (ObjNum 1289 snap 396 seq_num 1289) dirty exists 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3681: left oid 21 (ObjNum 1289 snap 396 seq_num 1289) 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: finishing write tid 1 to smithi20328086-33 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: finishing write tid 2 to smithi20328086-33 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: finishing write tid 3 to smithi20328086-33 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: finishing write tid 4 to smithi20328086-33 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: finishing write tid 5 to smithi20328086-33 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: oid 33 updating version 0 to 1426 2024-09-17T15:50:15.944 INFO:tasks.rados.rados.0.smithi203.stdout:3683: oid 33 updating version 1426 to 1429 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3683: oid 33 version 1429 is already newer than 1428 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3683: oid 33 version 1429 is already newer than 1427 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1429 (ObjNum 1290 snap 396 seq_num 1290) dirty exists 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3683: left oid 33 (ObjNum 1290 snap 396 seq_num 1290) 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: finishing write tid 1 to smithi20328086-28 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: finishing write tid 2 to smithi20328086-28 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: finishing write tid 3 to smithi20328086-28 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: finishing write tid 4 to smithi20328086-28 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: finishing write tid 5 to smithi20328086-28 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: oid 28 updating version 0 to 1949 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: oid 28 version 1949 is already newer than 1945 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: oid 28 version 1949 is already newer than 1940 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:3685: oid 28 version 1949 is already newer than 1948 2024-09-17T15:50:15.945 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1949 (ObjNum 1291 snap 396 seq_num 1291) dirty exists 2024-09-17T15:50:15.946 INFO:tasks.rados.rados.0.smithi203.stdout:3685: left oid 28 (ObjNum 1291 snap 396 seq_num 1291) 2024-09-17T15:50:15.946 INFO:tasks.rados.rados.0.smithi203.stdout:3686: finishing write tid 1 to smithi20328086-44 2024-09-17T15:50:15.946 INFO:tasks.rados.rados.0.smithi203.stdout:3686: finishing write tid 2 to smithi20328086-44 2024-09-17T15:50:15.948 INFO:tasks.rados.rados.0.smithi203.stdout:3686: finishing write tid 3 to smithi20328086-44 2024-09-17T15:50:15.948 INFO:tasks.rados.rados.0.smithi203.stdout:3686: finishing write tid 4 to smithi20328086-44 2024-09-17T15:50:15.948 INFO:tasks.rados.rados.0.smithi203.stdout:3686: finishing write tid 5 to smithi20328086-44 2024-09-17T15:50:15.948 INFO:tasks.rados.rados.0.smithi203.stdout:3686: oid 44 updating version 0 to 1679 2024-09-17T15:50:15.949 INFO:tasks.rados.rados.0.smithi203.stdout:3686: oid 44 version 1679 is already newer than 1678 2024-09-17T15:50:15.949 INFO:tasks.rados.rados.0.smithi203.stdout:3686: oid 44 version 1679 is already newer than 1677 2024-09-17T15:50:15.949 INFO:tasks.rados.rados.0.smithi203.stdout:3686: oid 44 version 1679 is already newer than 1676 2024-09-17T15:50:15.949 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1679 (ObjNum 1292 snap 396 seq_num 1292) dirty exists 2024-09-17T15:50:15.949 INFO:tasks.rados.rados.0.smithi203.stdout:3686: left oid 44 (ObjNum 1292 snap 396 seq_num 1292) 2024-09-17T15:50:16.875 INFO:tasks.rados.rados.0.smithi203.stdout:3681: done (5 left) 2024-09-17T15:50:16.875 INFO:tasks.rados.rados.0.smithi203.stdout:3683: done (4 left) 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3685: done (3 left) 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3686: done (2 left) 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3687: done (1 left) 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3688: done (0 left) 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3689: rollback oid 39 current snap is 397 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 39 to 380 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3690: read oid 17 snap 396 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3690: expect deleted 2024-09-17T15:50:16.876 INFO:tasks.rados.rados.0.smithi203.stdout:3691: delete oid 7 current snap is 397 2024-09-17T15:50:16.885 INFO:tasks.rados.rados.0.smithi203.stdout:3690: done (2 left) 2024-09-17T15:50:16.885 INFO:tasks.rados.rados.0.smithi203.stdout:3691: done (1 left) 2024-09-17T15:50:16.885 INFO:tasks.rados.rados.0.smithi203.stdout:3692: rollback oid 2 current snap is 397 2024-09-17T15:50:16.885 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 388 2024-09-17T15:50:16.885 INFO:tasks.rados.rados.0.smithi203.stdout:3693: copy_from oid 34 from oid 50 current snap is 397 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3694: copy_from oid 15 from oid 33 current snap is 397 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3695: read oid 16 snap -1 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3695: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3696: snap_create 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3693: finishing copy_from to smithi20328086-34 2024-09-17T15:50:16.886 INFO:tasks.rados.rados.0.smithi203.stdout:3693: got expected ENOENT (src dne) 2024-09-17T15:50:16.887 INFO:tasks.rados.rados.0.smithi203.stdout:3693: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:50:16.888 INFO:tasks.rados.rados.0.smithi203.stdout:3689: finishing rollback tid 0 to smithi20328086-39 2024-09-17T15:50:16.893 INFO:tasks.rados.rados.0.smithi203.stdout:3695: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:17.011 INFO:tasks.rados.rados.0.smithi203.stdout:3689: finishing rollback tid 1 to smithi20328086-39 2024-09-17T15:50:17.011 INFO:tasks.rados.rados.0.smithi203.stdout:3689: finishing rollback tid 2 to smithi20328086-39 2024-09-17T15:50:17.011 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1877 (ObjNum 1253 snap 380 seq_num 1253) dirty exists 2024-09-17T15:50:17.011 INFO:tasks.rados.rados.0.smithi203.stdout:3692: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:3692: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:3692: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1831 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:3694: finishing copy_from racing read to smithi20328086-15 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:3694: finishing copy_from to smithi20328086-15 2024-09-17T15:50:17.012 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1429 (ObjNum 1290 snap 396 seq_num 1290) dirty exists 2024-09-17T15:50:17.879 INFO:tasks.rados.rados.0.smithi203.stdout:3689: done (5 left) 2024-09-17T15:50:17.879 INFO:tasks.rados.rados.0.smithi203.stdout:3692: done (4 left) 2024-09-17T15:50:17.879 INFO:tasks.rados.rados.0.smithi203.stdout:3693: done (3 left) 2024-09-17T15:50:17.879 INFO:tasks.rados.rados.0.smithi203.stdout:3694: done (2 left) 2024-09-17T15:50:17.880 INFO:tasks.rados.rados.0.smithi203.stdout:3695: done (1 left) 2024-09-17T15:50:17.880 INFO:tasks.rados.rados.0.smithi203.stdout:3696: done (0 left) 2024-09-17T15:50:17.880 INFO:tasks.rados.rados.0.smithi203.stdout:3697: delete oid 1 current snap is 398 2024-09-17T15:50:17.891 INFO:tasks.rados.rados.0.smithi203.stdout:3697: done (0 left) 2024-09-17T15:50:17.892 INFO:tasks.rados.rados.0.smithi203.stdout:3698: snap_remove snap 387 2024-09-17T15:50:18.905 INFO:tasks.rados.rados.0.smithi203.stdout:3698: done (0 left) 2024-09-17T15:50:18.905 INFO:tasks.rados.rados.0.smithi203.stdout:3699: write oid 33 current snap is 398 2024-09-17T15:50:18.905 INFO:tasks.rados.rados.0.smithi203.stdout:3699: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-09-17T15:50:18.919 INFO:tasks.rados.rados.0.smithi203.stdout:3699: writing smithi20328086-33 from 565065 to 992530 tid 1 2024-09-17T15:50:18.933 INFO:tasks.rados.rados.0.smithi203.stdout:3699: writing smithi20328086-33 from 1430299 to 2064738 tid 2 2024-09-17T15:50:18.945 INFO:tasks.rados.rados.0.smithi203.stdout:3699: writing smithi20328086-33 from 2496305 to 3021444 tid 3 2024-09-17T15:50:18.955 INFO:tasks.rados.rados.0.smithi203.stdout:3699: writing smithi20328086-33 from 3472104 to 3765065 tid 4 2024-09-17T15:50:18.955 INFO:tasks.rados.rados.0.smithi203.stdout:3700: snap_create 2024-09-17T15:50:18.955 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 1 to smithi20328086-33 2024-09-17T15:50:18.955 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 2 to smithi20328086-33 2024-09-17T15:50:18.955 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 3 to smithi20328086-33 2024-09-17T15:50:18.958 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 4 to smithi20328086-33 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 5 to smithi20328086-33 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: finishing write tid 6 to smithi20328086-33 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: oid 33 updating version 0 to 1431 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: oid 33 updating version 1431 to 1433 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: oid 33 version 1433 is already newer than 1432 2024-09-17T15:50:18.959 INFO:tasks.rados.rados.0.smithi203.stdout:3699: oid 33 version 1433 is already newer than 1430 2024-09-17T15:50:18.960 INFO:tasks.rados.rados.0.smithi203.stdout:3699: oid 33 updating version 1433 to 1434 2024-09-17T15:50:18.960 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1434 (ObjNum 1295 snap 398 seq_num 1295) dirty exists 2024-09-17T15:50:18.960 INFO:tasks.rados.rados.0.smithi203.stdout:3699: left oid 33 (ObjNum 1295 snap 398 seq_num 1295) 2024-09-17T15:50:19.898 INFO:tasks.rados.rados.0.smithi203.stdout:3699: done (1 left) 2024-09-17T15:50:19.898 INFO:tasks.rados.rados.0.smithi203.stdout:3700: done (0 left) 2024-09-17T15:50:19.898 INFO:tasks.rados.rados.0.smithi203.stdout:3701: copy_from oid 22 from oid 6 current snap is 399 2024-09-17T15:50:19.898 INFO:tasks.rados.rados.0.smithi203.stdout:3702: write (excl) oid 46 current snap is 399 2024-09-17T15:50:19.898 INFO:tasks.rados.rados.0.smithi203.stdout:3702: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-17T15:50:19.910 INFO:tasks.rados.rados.0.smithi203.stdout:3702: writing smithi20328086-46 from 581872 to 1084586 tid 1 2024-09-17T15:50:19.920 INFO:tasks.rados.rados.0.smithi203.stdout:3702: writing smithi20328086-46 from 1488781 to 1983231 tid 2 2024-09-17T15:50:19.935 INFO:tasks.rados.rados.0.smithi203.stdout:3702: writing smithi20328086-46 from 2640081 to 3376492 tid 3 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:3703: rollback oid 43 current snap is 399 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 397 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:3702: finishing write tid 1 to smithi20328086-46 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:3702: finishing write tid 2 to smithi20328086-46 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:3704: write (excl) oid 10 current snap is 399 2024-09-17T15:50:19.936 INFO:tasks.rados.rados.0.smithi203.stdout:3704: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-17T15:50:19.949 INFO:tasks.rados.rados.0.smithi203.stdout:3704: writing smithi20328086-10 from 598679 to 1176642 tid 1 2024-09-17T15:50:19.961 INFO:tasks.rados.rados.0.smithi203.stdout:3704: writing smithi20328086-10 from 1947263 to 2585371 tid 2 2024-09-17T15:50:19.971 INFO:tasks.rados.rados.0.smithi203.stdout:3704: writing smithi20328086-10 from 3351151 to 3798679 tid 3 2024-09-17T15:50:19.971 INFO:tasks.rados.rados.0.smithi203.stdout:3705: snap_create 2024-09-17T15:50:19.971 INFO:tasks.rados.rados.0.smithi203.stdout:3701: finishing copy_from racing read to smithi20328086-22 2024-09-17T15:50:19.971 INFO:tasks.rados.rados.0.smithi203.stdout:3701: finishing copy_from to smithi20328086-22 2024-09-17T15:50:19.971 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1672 (ObjNum 1253 snap 380 seq_num 1253) dirty exists 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: finishing write tid 3 to smithi20328086-46 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: finishing write tid 4 to smithi20328086-46 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: finishing write tid 5 to smithi20328086-46 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: oid 46 updating version 0 to 1443 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: oid 46 updating version 1443 to 1446 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: oid 46 version 1446 is already newer than 1445 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: oid 46 version 1446 is already newer than 1444 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1446 (ObjNum 1296 snap 399 seq_num 1296) dirty exists 2024-09-17T15:50:19.972 INFO:tasks.rados.rados.0.smithi203.stdout:3702: left oid 46 (ObjNum 1296 snap 399 seq_num 1296) 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:3703: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:3703: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:3703: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 708 (ObjNum 1285 snap 396 seq_num 1285) dirty exists 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:3704: finishing write tid 1 to smithi20328086-10 2024-09-17T15:50:19.973 INFO:tasks.rados.rados.0.smithi203.stdout:3704: finishing write tid 2 to smithi20328086-10 2024-09-17T15:50:19.978 INFO:tasks.rados.rados.0.smithi203.stdout:3704: finishing write tid 3 to smithi20328086-10 2024-09-17T15:50:19.978 INFO:tasks.rados.rados.0.smithi203.stdout:3704: finishing write tid 4 to smithi20328086-10 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:3704: finishing write tid 5 to smithi20328086-10 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:3704: oid 10 updating version 0 to 1402 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:3704: oid 10 version 1402 is already newer than 1399 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:3704: oid 10 version 1402 is already newer than 1400 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:3704: oid 10 version 1402 is already newer than 1401 2024-09-17T15:50:19.979 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1402 (ObjNum 1297 snap 399 seq_num 1297) dirty exists 2024-09-17T15:50:19.980 INFO:tasks.rados.rados.0.smithi203.stdout:3704: left oid 10 (ObjNum 1297 snap 399 seq_num 1297) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3701: done (4 left) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3702: done (3 left) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3703: done (2 left) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3704: done (1 left) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3705: done (0 left) 2024-09-17T15:50:20.863 INFO:tasks.rados.rados.0.smithi203.stdout:3706: read oid 8 snap -1 2024-09-17T15:50:20.864 INFO:tasks.rados.rados.0.smithi203.stdout:3706: expect (ObjNum 1277 snap 392 seq_num 1277) 2024-09-17T15:50:20.864 INFO:tasks.rados.rados.0.smithi203.stdout:3707: write oid 10 current snap is 400 2024-09-17T15:50:20.864 INFO:tasks.rados.rados.0.smithi203.stdout:3707: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-09-17T15:50:20.872 INFO:tasks.rados.rados.0.smithi203.stdout:3707: writing smithi20328086-10 from 615486 to 1268698 tid 1 2024-09-17T15:50:20.880 INFO:tasks.rados.rados.0.smithi203.stdout:3707: writing smithi20328086-10 from 1889392 to 2387511 tid 2 2024-09-17T15:50:20.889 INFO:tasks.rados.rados.0.smithi203.stdout:3707: writing smithi20328086-10 from 2978574 to 3453882 tid 3 2024-09-17T15:50:20.889 INFO:tasks.rados.rados.0.smithi203.stdout:3708: write (excl) oid 22 current snap is 400 2024-09-17T15:50:20.889 INFO:tasks.rados.rados.0.smithi203.stdout:3708: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-09-17T15:50:20.898 INFO:tasks.rados.rados.0.smithi203.stdout:3708: writing smithi20328086-22 from 632293 to 1360754 tid 1 2024-09-17T15:50:20.908 INFO:tasks.rados.rados.0.smithi203.stdout:3708: writing smithi20328086-22 from 1947874 to 2589651 tid 2 2024-09-17T15:50:20.916 INFO:tasks.rados.rados.0.smithi203.stdout:3708: writing smithi20328086-22 from 3005997 to 3692577 tid 3 2024-09-17T15:50:20.916 INFO:tasks.rados.rados.0.smithi203.stdout:3709: read oid 17 snap 393 2024-09-17T15:50:20.916 INFO:tasks.rados.rados.0.smithi203.stdout:3709: expect deleted 2024-09-17T15:50:20.916 INFO:tasks.rados.rados.0.smithi203.stdout:3710: write (excl) oid 39 current snap is 400 2024-09-17T15:50:20.916 INFO:tasks.rados.rados.0.smithi203.stdout:3710: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-17T15:50:20.923 INFO:tasks.rados.rados.0.smithi203.stdout:3710: writing smithi20328086-39 from 649100 to 1052810 tid 1 2024-09-17T15:50:20.931 INFO:tasks.rados.rados.0.smithi203.stdout:3710: writing smithi20328086-39 from 1606356 to 2108144 tid 2 2024-09-17T15:50:20.937 INFO:tasks.rados.rados.0.smithi203.stdout:3710: writing smithi20328086-39 from 2633420 to 3131272 tid 3 2024-09-17T15:50:20.942 INFO:tasks.rados.rados.0.smithi203.stdout:3710: writing smithi20328086-39 from 3636005 to 3849100 tid 4 2024-09-17T15:50:20.942 INFO:tasks.rados.rados.0.smithi203.stdout:3711: write oid 24 current snap is 400 2024-09-17T15:50:20.942 INFO:tasks.rados.rados.0.smithi203.stdout:3711: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-17T15:50:20.949 INFO:tasks.rados.rados.0.smithi203.stdout:3711: writing smithi20328086-24 from 665907 to 1261219 tid 1 2024-09-17T15:50:20.956 INFO:tasks.rados.rados.0.smithi203.stdout:3711: writing smithi20328086-24 from 1781191 to 2426637 tid 2 2024-09-17T15:50:20.964 INFO:tasks.rados.rados.0.smithi203.stdout:3711: writing smithi20328086-24 from 3177196 to 3865907 tid 3 2024-09-17T15:50:20.965 INFO:tasks.rados.rados.0.smithi203.stdout:3712: write oid 48 current snap is 400 2024-09-17T15:50:20.965 INFO:tasks.rados.rados.0.smithi203.stdout:3712: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-17T15:50:20.972 INFO:tasks.rados.rados.0.smithi203.stdout:3712: writing smithi20328086-48 from 682714 to 1353275 tid 1 2024-09-17T15:50:20.979 INFO:tasks.rados.rados.0.smithi203.stdout:3712: writing smithi20328086-48 from 2123320 to 2628777 tid 2 2024-09-17T15:50:20.986 INFO:tasks.rados.rados.0.smithi203.stdout:3712: writing smithi20328086-48 from 3088266 to 3725015 tid 3 2024-09-17T15:50:20.986 INFO:tasks.rados.rados.0.smithi203.stdout:3713: copy_from oid 26 from oid 9 current snap is 400 2024-09-17T15:50:20.986 INFO:tasks.rados.rados.0.smithi203.stdout:3706: expect (ObjNum 1277 snap 392 seq_num 1277) 2024-09-17T15:50:20.986 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:21.142 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1402477 2024-09-17T15:50:21.278 INFO:tasks.rados.rados.0.smithi203.stdout:3707: finishing write tid 1 to smithi20328086-10 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: finishing write tid 2 to smithi20328086-10 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: finishing write tid 3 to smithi20328086-10 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: finishing write tid 4 to smithi20328086-10 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: finishing write tid 5 to smithi20328086-10 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: oid 10 updating version 0 to 1405 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: oid 10 version 1405 is already newer than 1403 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: oid 10 updating version 1405 to 1406 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: oid 10 version 1406 is already newer than 1404 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1406 (ObjNum 1298 snap 400 seq_num 1298) dirty exists 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3707: left oid 10 (ObjNum 1298 snap 400 seq_num 1298) 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3708: finishing write tid 1 to smithi20328086-22 2024-09-17T15:50:21.279 INFO:tasks.rados.rados.0.smithi203.stdout:3708: finishing write tid 2 to smithi20328086-22 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: finishing write tid 3 to smithi20328086-22 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: finishing write tid 4 to smithi20328086-22 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: finishing write tid 5 to smithi20328086-22 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: oid 22 updating version 0 to 1676 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: oid 22 version 1676 is already newer than 1673 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: oid 22 version 1676 is already newer than 1675 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: oid 22 version 1676 is already newer than 1674 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1676 (ObjNum 1299 snap 400 seq_num 1299) dirty exists 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3708: left oid 22 (ObjNum 1299 snap 400 seq_num 1299) 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 1 to smithi20328086-39 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 2 to smithi20328086-39 2024-09-17T15:50:21.280 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 3 to smithi20328086-39 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 4 to smithi20328086-39 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 5 to smithi20328086-39 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: finishing write tid 6 to smithi20328086-39 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: oid 39 updating version 0 to 1883 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: oid 39 updating version 1883 to 1887 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: oid 39 version 1887 is already newer than 1886 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: oid 39 version 1887 is already newer than 1884 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: oid 39 version 1887 is already newer than 1885 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1887 (ObjNum 1300 snap 400 seq_num 1300) dirty exists 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3710: left oid 39 (ObjNum 1300 snap 400 seq_num 1300) 2024-09-17T15:50:21.281 INFO:tasks.rados.rados.0.smithi203.stdout:3711: finishing write tid 1 to smithi20328086-24 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: finishing write tid 2 to smithi20328086-24 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: finishing write tid 3 to smithi20328086-24 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: finishing write tid 4 to smithi20328086-24 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: finishing write tid 5 to smithi20328086-24 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: oid 24 updating version 0 to 1559 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: oid 24 version 1559 is already newer than 1556 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: oid 24 version 1559 is already newer than 1558 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: oid 24 version 1559 is already newer than 1557 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1559 (ObjNum 1301 snap 400 seq_num 1301) dirty exists 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3711: left oid 24 (ObjNum 1301 snap 400 seq_num 1301) 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3712: finishing write tid 1 to smithi20328086-48 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3712: finishing write tid 2 to smithi20328086-48 2024-09-17T15:50:21.282 INFO:tasks.rados.rados.0.smithi203.stdout:3712: finishing write tid 3 to smithi20328086-48 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: finishing write tid 4 to smithi20328086-48 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3706: done (7 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3707: done (6 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3708: done (5 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3709: done (4 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3710: done (3 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3711: done (2 left) 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3714: copy_from oid 37 from oid 30 current snap is 400 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: finishing write tid 5 to smithi20328086-48 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: oid 48 updating version 0 to 1890 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: oid 48 updating version 1890 to 1891 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: oid 48 version 1891 is already newer than 1889 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:3712: oid 48 updating version 1891 to 1892 2024-09-17T15:50:21.283 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 1892 (ObjNum 1302 snap 400 seq_num 1302) dirty exists 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3712: left oid 48 (ObjNum 1302 snap 400 seq_num 1302) 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3713: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3713: finishing copy_from to smithi20328086-26 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1553 (ObjNum 1269 snap 388 seq_num 1269) dirty exists 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3712: done (2 left) 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3713: done (1 left) 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3715: write (excl) oid 38 current snap is 400 2024-09-17T15:50:21.284 INFO:tasks.rados.rados.0.smithi203.stdout:3715: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-17T15:50:21.287 INFO:tasks.rados.rados.0.smithi203.stdout:3715: writing smithi20328086-38 from 699521 to 1445331 tid 1 2024-09-17T15:50:21.298 INFO:tasks.rados.rados.0.smithi203.stdout:3715: writing smithi20328086-38 from 2181802 to 2830917 tid 2 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3715: writing smithi20328086-38 from 3515689 to 3899521 tid 3 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3716: read oid 16 snap -1 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3716: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3717: read oid 26 snap -1 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3717: expect (ObjNum 1269 snap 388 seq_num 1269) 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3718: copy_from oid 34 from oid 9 current snap is 400 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3714: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:50:21.306 INFO:tasks.rados.rados.0.smithi203.stdout:3714: finishing copy_from to smithi20328086-37 2024-09-17T15:50:21.307 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1807 (ObjNum 1275 snap 391 seq_num 1275) dirty exists 2024-09-17T15:50:21.307 INFO:tasks.rados.rados.0.smithi203.stdout:3715: finishing write tid 1 to smithi20328086-38 2024-09-17T15:50:21.307 INFO:tasks.rados.rados.0.smithi203.stdout:3714: done (4 left) 2024-09-17T15:50:21.307 INFO:tasks.rados.rados.0.smithi203.stdout:3719: write oid 13 current snap is 400 2024-09-17T15:50:21.307 INFO:tasks.rados.rados.0.smithi203.stdout:3719: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-17T15:50:21.313 INFO:tasks.rados.rados.0.smithi203.stdout:3719: writing smithi20328086-13 from 716328 to 1137387 tid 1 2024-09-17T15:50:21.322 INFO:tasks.rados.rados.0.smithi203.stdout:3719: writing smithi20328086-13 from 1840284 to 2349410 tid 2 2024-09-17T15:50:21.332 INFO:tasks.rados.rados.0.smithi203.stdout:3719: writing smithi20328086-13 from 3143112 to 3802405 tid 3 2024-09-17T15:50:21.332 INFO:tasks.rados.rados.0.smithi203.stdout:3720: write (excl) oid 11 current snap is 400 2024-09-17T15:50:21.332 INFO:tasks.rados.rados.0.smithi203.stdout:3720: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2024-09-17T15:50:21.340 INFO:tasks.rados.rados.0.smithi203.stdout:3720: writing smithi20328086-11 from 733135 to 1229443 tid 1 2024-09-17T15:50:21.348 INFO:tasks.rados.rados.0.smithi203.stdout:3720: writing smithi20328086-11 from 1898766 to 2551550 tid 2 2024-09-17T15:50:21.355 INFO:tasks.rados.rados.0.smithi203.stdout:3720: writing smithi20328086-11 from 3170535 to 3641100 tid 3 2024-09-17T15:50:21.356 INFO:tasks.rados.rados.0.smithi203.stdout:3721: snap_create 2024-09-17T15:50:21.356 INFO:tasks.rados.rados.0.smithi203.stdout:3716: expect (ObjNum 1219 snap 368 seq_num 1219) 2024-09-17T15:50:21.442 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1611035 2024-09-17T15:50:21.469 INFO:tasks.rados.rados.0.smithi203.stdout:3715: finishing write tid 2 to smithi20328086-38 2024-09-17T15:50:21.469 INFO:tasks.rados.rados.0.smithi203.stdout:3717: expect (ObjNum 1269 snap 388 seq_num 1269) 2024-09-17T15:50:21.591 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:21.657 INFO:tasks.rados.rados.0.smithi203.stdout:3715: finishing write tid 3 to smithi20328086-38 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: finishing write tid 4 to smithi20328086-38 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: finishing write tid 5 to smithi20328086-38 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: oid 38 updating version 0 to 1408 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: oid 38 updating version 1408 to 1410 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: oid 38 version 1410 is already newer than 1407 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: oid 38 version 1410 is already newer than 1409 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1410 (ObjNum 1303 snap 400 seq_num 1303) dirty exists 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3715: left oid 38 (ObjNum 1303 snap 400 seq_num 1303) 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3719: finishing write tid 1 to smithi20328086-13 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3719: finishing write tid 2 to smithi20328086-13 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3718: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:3718: finishing copy_from to smithi20328086-34 2024-09-17T15:50:21.658 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1553 (ObjNum 1269 snap 388 seq_num 1269) dirty exists 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: finishing write tid 3 to smithi20328086-13 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: finishing write tid 4 to smithi20328086-13 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: finishing write tid 5 to smithi20328086-13 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: oid 13 updating version 0 to 1414 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: oid 13 version 1414 is already newer than 1411 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: oid 13 version 1414 is already newer than 1412 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: oid 13 version 1414 is already newer than 1413 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1414 (ObjNum 1304 snap 400 seq_num 1304) dirty exists 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3719: left oid 13 (ObjNum 1304 snap 400 seq_num 1304) 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3720: finishing write tid 1 to smithi20328086-11 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3720: finishing write tid 2 to smithi20328086-11 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3720: finishing write tid 3 to smithi20328086-11 2024-09-17T15:50:21.659 INFO:tasks.rados.rados.0.smithi203.stdout:3720: finishing write tid 4 to smithi20328086-11 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: finishing write tid 5 to smithi20328086-11 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: oid 11 updating version 0 to 1683 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: oid 11 updating version 1683 to 1685 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: oid 11 version 1685 is already newer than 1682 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: oid 11 version 1685 is already newer than 1684 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 1685 (ObjNum 1305 snap 400 seq_num 1305) dirty exists 2024-09-17T15:50:21.660 INFO:tasks.rados.rados.0.smithi203.stdout:3720: left oid 11 (ObjNum 1305 snap 400 seq_num 1305) 2024-09-17T15:50:21.916 INFO:tasks.rados.rados.0.smithi203.stdout:3715: done (6 left) 2024-09-17T15:50:21.916 INFO:tasks.rados.rados.0.smithi203.stdout:3716: done (5 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3717: done (4 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3718: done (3 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3719: done (2 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3720: done (1 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3721: done (0 left) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3722: copy_from oid 21 from oid 28 current snap is 401 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3723: read oid 24 snap -1 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3723: expect (ObjNum 1301 snap 400 seq_num 1301) 2024-09-17T15:50:21.917 INFO:tasks.rados.rados.0.smithi203.stdout:3724: delete oid 6 current snap is 401 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3724: done (2 left) 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3725: copy_from oid 2 from oid 37 current snap is 401 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3726: rollback oid 41 current snap is 401 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 41 to 385 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3727: rollback oid 27 current snap is 401 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 27 to 398 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3728: read oid 12 snap -1 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3728: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:50:21.930 INFO:tasks.rados.rados.0.smithi203.stdout:3729: snap_remove snap 391 2024-09-17T15:50:22.927 INFO:tasks.rados.rados.0.smithi203.stdout:3729: done (6 left) 2024-09-17T15:50:22.927 INFO:tasks.rados.rados.0.smithi203.stdout:3730: read oid 49 snap 380 2024-09-17T15:50:22.927 INFO:tasks.rados.rados.0.smithi203.stdout:3730: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:50:22.927 INFO:tasks.rados.rados.0.smithi203.stdout:3731: read oid 18 snap -1 2024-09-17T15:50:22.927 INFO:tasks.rados.rados.0.smithi203.stdout:3731: expect (ObjNum 1241 snap 373 seq_num 1241) 2024-09-17T15:50:22.928 INFO:tasks.rados.rados.0.smithi203.stdout:3732: copy_from oid 26 from oid 40 current snap is 401 2024-09-17T15:50:22.928 INFO:tasks.rados.rados.0.smithi203.stdout:3727: finishing rollback tid 1 to smithi20328086-27 2024-09-17T15:50:22.928 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 0 (ObjNum 33 snap 0 seq_num 3586005048) dirty dne 2024-09-17T15:50:22.928 INFO:tasks.rados.rados.0.smithi203.stdout:3723: expect (ObjNum 1301 snap 400 seq_num 1301) 2024-09-17T15:50:22.928 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.080 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.143 INFO:tasks.rados.rados.0.smithi203.stdout:3728: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:50:23.206 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3726: finishing rollback tid 0 to smithi20328086-41 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3726: finishing rollback tid 1 to smithi20328086-41 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3726: finishing rollback tid 2 to smithi20328086-41 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 41 v 711 (ObjNum 1260 snap 382 seq_num 1260) dirty exists 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3722: finishing copy_from racing read to smithi20328086-21 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3725: finishing copy_from racing read to smithi20328086-2 2024-09-17T15:50:23.244 INFO:tasks.rados.rados.0.smithi203.stdout:3725: finishing copy_from to smithi20328086-2 2024-09-17T15:50:23.245 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1807 (ObjNum 1275 snap 391 seq_num 1275) dirty exists 2024-09-17T15:50:23.245 INFO:tasks.rados.rados.0.smithi203.stdout:3722: finishing copy_from to smithi20328086-21 2024-09-17T15:50:23.245 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 1949 (ObjNum 1291 snap 396 seq_num 1291) dirty exists 2024-09-17T15:50:23.245 INFO:tasks.rados.rados.0.smithi203.stdout:3731: expect (ObjNum 1241 snap 373 seq_num 1241) 2024-09-17T15:50:23.245 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.325 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.369 INFO:tasks.rados.rados.0.smithi203.stdout:3730: expect (ObjNum 1020 snap 316 seq_num 1020) 2024-09-17T15:50:23.369 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3732: finishing copy_from to smithi20328086-26 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3732: got expected ENOENT (src dne) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3732: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3722: done (8 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3723: done (7 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3725: done (6 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3726: done (5 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3727: done (4 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3728: done (3 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3730: done (2 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3731: done (1 left) 2024-09-17T15:50:23.530 INFO:tasks.rados.rados.0.smithi203.stdout:3732: done (0 left) 2024-09-17T15:50:23.531 INFO:tasks.rados.rados.0.smithi203.stdout:3733: delete oid 21 current snap is 401 2024-09-17T15:50:23.534 INFO:tasks.rados.rados.0.smithi203.stdout:3733: done (0 left) 2024-09-17T15:50:23.535 INFO:tasks.rados.rados.0.smithi203.stdout:3734: write oid 35 current snap is 401 2024-09-17T15:50:23.535 INFO:tasks.rados.rados.0.smithi203.stdout:3734: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-17T15:50:23.554 INFO:tasks.rados.rados.0.smithi203.stdout:3734: writing smithi20328086-35 from 783556 to 1221964 tid 1 2024-09-17T15:50:23.568 INFO:tasks.rados.rados.0.smithi203.stdout:3734: writing smithi20328086-35 from 1674212 to 2074323 tid 2 2024-09-17T15:50:23.585 INFO:tasks.rados.rados.0.smithi203.stdout:3734: writing smithi20328086-35 from 2736451 to 3157185 tid 3 2024-09-17T15:50:23.596 INFO:tasks.rados.rados.0.smithi203.stdout:3734: writing smithi20328086-35 from 3882270 to 3983556 tid 4 2024-09-17T15:50:23.596 INFO:tasks.rados.rados.0.smithi203.stdout:3735: write (excl) oid 3 current snap is 401 2024-09-17T15:50:23.596 INFO:tasks.rados.rados.0.smithi203.stdout:3735: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-17T15:50:23.606 INFO:tasks.rados.rados.0.smithi203.stdout:3735: writing smithi20328086-3 from 400363 to 914020 tid 1 2024-09-17T15:50:23.616 INFO:tasks.rados.rados.0.smithi203.stdout:3735: writing smithi20328086-3 from 1332694 to 1992816 tid 2 2024-09-17T15:50:23.616 INFO:tasks.rados.rados.0.smithi203.stdout:3736: snap_create 2024-09-17T15:50:23.617 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 1 to smithi20328086-35 2024-09-17T15:50:23.617 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 2 to smithi20328086-35 2024-09-17T15:50:23.617 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 3 to smithi20328086-35 2024-09-17T15:50:23.617 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 4 to smithi20328086-35 2024-09-17T15:50:23.618 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 5 to smithi20328086-35 2024-09-17T15:50:23.618 INFO:tasks.rados.rados.0.smithi203.stdout:3734: finishing write tid 6 to smithi20328086-35 2024-09-17T15:50:23.618 INFO:tasks.rados.rados.0.smithi203.stdout:3734: oid 35 updating version 0 to 1683 2024-09-17T15:50:23.618 INFO:tasks.rados.rados.0.smithi203.stdout:3734: oid 35 updating version 1683 to 1684 2024-09-17T15:50:23.618 INFO:tasks.rados.rados.0.smithi203.stdout:3734: oid 35 version 1684 is already newer than 1680 2024-09-17T15:50:23.619 INFO:tasks.rados.rados.0.smithi203.stdout:3734: oid 35 version 1684 is already newer than 1682 2024-09-17T15:50:23.619 INFO:tasks.rados.rados.0.smithi203.stdout:3734: oid 35 version 1684 is already newer than 1681 2024-09-17T15:50:23.619 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1684 (ObjNum 1308 snap 401 seq_num 1308) dirty exists 2024-09-17T15:50:23.619 INFO:tasks.rados.rados.0.smithi203.stdout:3734: left oid 35 (ObjNum 1308 snap 401 seq_num 1308) 2024-09-17T15:50:23.619 INFO:tasks.rados.rados.0.smithi203.stdout:3735: finishing write tid 1 to smithi20328086-3 2024-09-17T15:50:23.624 INFO:tasks.rados.rados.0.smithi203.stdout:3735: finishing write tid 2 to smithi20328086-3 2024-09-17T15:50:23.624 INFO:tasks.rados.rados.0.smithi203.stdout:3735: finishing write tid 3 to smithi20328086-3 2024-09-17T15:50:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:3735: finishing write tid 4 to smithi20328086-3 2024-09-17T15:50:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:3735: oid 3 updating version 0 to 1448 2024-09-17T15:50:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:3735: oid 3 updating version 1448 to 1449 2024-09-17T15:50:23.625 INFO:tasks.rados.rados.0.smithi203.stdout:3735: oid 3 version 1449 is already newer than 1447 2024-09-17T15:50:23.626 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1449 (ObjNum 1309 snap 401 seq_num 1309) dirty exists 2024-09-17T15:50:23.626 INFO:tasks.rados.rados.0.smithi203.stdout:3735: left oid 3 (ObjNum 1309 snap 401 seq_num 1309) 2024-09-17T15:50:23.932 INFO:tasks.rados.rados.0.smithi203.stdout:3734: done (2 left) 2024-09-17T15:50:23.932 INFO:tasks.rados.rados.0.smithi203.stdout:3735: done (1 left) 2024-09-17T15:50:23.932 INFO:tasks.rados.rados.0.smithi203.stdout:3736: done (0 left) 2024-09-17T15:50:23.932 INFO:tasks.rados.rados.0.smithi203.stdout:3737: write (excl) oid 35 current snap is 402 2024-09-17T15:50:23.932 INFO:tasks.rados.rados.0.smithi203.stdout:3737: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-09-17T15:50:23.940 INFO:tasks.rados.rados.0.smithi203.stdout:3737: writing smithi20328086-35 from 417170 to 1006076 tid 1 2024-09-17T15:50:23.948 INFO:tasks.rados.rados.0.smithi203.stdout:3737: writing smithi20328086-35 from 1674823 to 2017170 tid 2 2024-09-17T15:50:23.948 INFO:tasks.rados.rados.0.smithi203.stdout:3738: write (excl) oid 2 current snap is 402 2024-09-17T15:50:23.949 INFO:tasks.rados.rados.0.smithi203.stdout:3738: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-17T15:50:23.957 INFO:tasks.rados.rados.0.smithi203.stdout:3738: writing smithi20328086-2 from 433977 to 1098132 tid 1 2024-09-17T15:50:23.965 INFO:tasks.rados.rados.0.smithi203.stdout:3738: writing smithi20328086-2 from 1733305 to 2033977 tid 2 2024-09-17T15:50:23.965 INFO:tasks.rados.rados.0.smithi203.stdout:3739: snap_remove snap 386 2024-09-17T15:50:24.942 INFO:tasks.rados.rados.0.smithi203.stdout:3739: done (2 left) 2024-09-17T15:50:24.942 INFO:tasks.rados.rados.0.smithi203.stdout:3740: delete oid 9 current snap is 402 2024-09-17T15:50:24.942 INFO:tasks.rados.rados.0.smithi203.stdout:3737: finishing write tid 1 to smithi20328086-35 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: finishing write tid 2 to smithi20328086-35 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: finishing write tid 3 to smithi20328086-35 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: finishing write tid 4 to smithi20328086-35 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: oid 35 updating version 0 to 1686 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: oid 35 updating version 1686 to 1687 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: oid 35 version 1687 is already newer than 1685 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1687 (ObjNum 1310 snap 402 seq_num 1310) dirty exists 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3737: left oid 35 (ObjNum 1310 snap 402 seq_num 1310) 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3738: finishing write tid 1 to smithi20328086-2 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3738: finishing write tid 2 to smithi20328086-2 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3738: finishing write tid 3 to smithi20328086-2 2024-09-17T15:50:24.943 INFO:tasks.rados.rados.0.smithi203.stdout:3738: finishing write tid 4 to smithi20328086-2 2024-09-17T15:50:24.944 INFO:tasks.rados.rados.0.smithi203.stdout:3738: oid 2 updating version 0 to 1832 2024-09-17T15:50:24.944 INFO:tasks.rados.rados.0.smithi203.stdout:3738: oid 2 updating version 1832 to 1834 2024-09-17T15:50:24.944 INFO:tasks.rados.rados.0.smithi203.stdout:3738: oid 2 version 1834 is already newer than 1833 2024-09-17T15:50:24.944 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1834 (ObjNum 1311 snap 402 seq_num 1311) dirty exists 2024-09-17T15:50:24.944 INFO:tasks.rados.rados.0.smithi203.stdout:3738: left oid 2 (ObjNum 1311 snap 402 seq_num 1311) 2024-09-17T15:50:24.961 INFO:tasks.rados.rados.0.smithi203.stdout:3737: done (2 left) 2024-09-17T15:50:24.961 INFO:tasks.rados.rados.0.smithi203.stdout:3738: done (1 left) 2024-09-17T15:50:24.961 INFO:tasks.rados.rados.0.smithi203.stdout:3740: done (0 left) 2024-09-17T15:50:24.962 INFO:tasks.rados.rados.0.smithi203.stdout:3741: snap_remove snap 398 2024-09-17T15:50:25.866 INFO:tasks.rados.rados.0.smithi203.stdout:3741: done (0 left) 2024-09-17T15:50:25.866 INFO:tasks.rados.rados.0.smithi203.stdout:3742: read oid 7 snap -1 2024-09-17T15:50:25.867 INFO:tasks.rados.rados.0.smithi203.stdout:3742: expect deleted 2024-09-17T15:50:25.867 INFO:tasks.rados.rados.0.smithi203.stdout:3743: copy_from oid 35 from oid 46 current snap is 402 2024-09-17T15:50:25.868 INFO:tasks.rados.rados.0.smithi203.stdout:3744: read oid 47 snap -1 2024-09-17T15:50:25.868 INFO:tasks.rados.rados.0.smithi203.stdout:3744: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:50:25.868 INFO:tasks.rados.rados.0.smithi203.stdout:3745: snap_remove snap 399 2024-09-17T15:50:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:3745: done (3 left) 2024-09-17T15:50:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:3746: read oid 37 snap -1 2024-09-17T15:50:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:3746: expect (ObjNum 1275 snap 391 seq_num 1275) 2024-09-17T15:50:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:3747: write (excl) oid 28 current snap is 402 2024-09-17T15:50:26.879 INFO:tasks.rados.rados.0.smithi203.stdout:3747: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-17T15:50:26.890 INFO:tasks.rados.rados.0.smithi203.stdout:3747: writing smithi20328086-28 from 467591 to 882244 tid 1 2024-09-17T15:50:26.906 INFO:tasks.rados.rados.0.smithi203.stdout:3747: writing smithi20328086-28 from 1450269 to 2067591 tid 2 2024-09-17T15:50:26.906 INFO:tasks.rados.rados.0.smithi203.stdout:3748: snap_remove snap 393 2024-09-17T15:50:27.880 INFO:tasks.rados.rados.0.smithi203.stdout:3744: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:50:27.973 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1954620 2024-09-17T15:50:28.035 INFO:tasks.rados.rados.0.smithi203.stdout:3743: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:50:28.035 INFO:tasks.rados.rados.0.smithi203.stdout:3743: finishing copy_from to smithi20328086-35 2024-09-17T15:50:28.035 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1446 (ObjNum 1296 snap 399 seq_num 1296) dirty exists 2024-09-17T15:50:28.035 INFO:tasks.rados.rados.0.smithi203.stdout:3746: expect (ObjNum 1275 snap 391 seq_num 1275) 2024-09-17T15:50:28.086 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3387841 2024-09-17T15:50:28.097 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1883436 2024-09-17T15:50:28.113 INFO:tasks.rados.rados.0.smithi203.stdout:3747: finishing write tid 1 to smithi20328086-28 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: finishing write tid 2 to smithi20328086-28 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: finishing write tid 3 to smithi20328086-28 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: finishing write tid 4 to smithi20328086-28 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: oid 28 updating version 0 to 1987 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: oid 28 updating version 1987 to 1988 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: oid 28 version 1988 is already newer than 1986 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 1988 (ObjNum 1313 snap 402 seq_num 1313) dirty exists 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3747: left oid 28 (ObjNum 1313 snap 402 seq_num 1313) 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3742: done (5 left) 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3743: done (4 left) 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3744: done (3 left) 2024-09-17T15:50:28.114 INFO:tasks.rados.rados.0.smithi203.stdout:3746: done (2 left) 2024-09-17T15:50:28.115 INFO:tasks.rados.rados.0.smithi203.stdout:3747: done (1 left) 2024-09-17T15:50:28.115 INFO:tasks.rados.rados.0.smithi203.stdout:3748: done (0 left) 2024-09-17T15:50:28.115 INFO:tasks.rados.rados.0.smithi203.stdout:3749: write oid 26 current snap is 402 2024-09-17T15:50:28.115 INFO:tasks.rados.rados.0.smithi203.stdout:3749: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-17T15:50:28.129 INFO:tasks.rados.rados.0.smithi203.stdout:3749: writing smithi20328086-26 from 484398 to 974300 tid 1 2024-09-17T15:50:28.143 INFO:tasks.rados.rados.0.smithi203.stdout:3749: writing smithi20328086-26 from 1392398 to 1803516 tid 2 2024-09-17T15:50:28.143 INFO:tasks.rados.rados.0.smithi203.stdout:3750: write (excl) oid 19 current snap is 402 2024-09-17T15:50:28.144 INFO:tasks.rados.rados.0.smithi203.stdout:3749: finishing write tid 1 to smithi20328086-26 2024-09-17T15:50:28.144 INFO:tasks.rados.rados.0.smithi203.stdout:3750: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-17T15:50:28.161 INFO:tasks.rados.rados.0.smithi203.stdout:3750: writing smithi20328086-19 from 501205 to 1066356 tid 1 2024-09-17T15:50:28.178 INFO:tasks.rados.rados.0.smithi203.stdout:3750: writing smithi20328086-19 from 1850880 to 2101205 tid 2 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3751: rollback oid 42 current snap is 402 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 42 to 388 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: finishing write tid 2 to smithi20328086-26 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: finishing write tid 3 to smithi20328086-26 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: finishing write tid 4 to smithi20328086-26 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: oid 26 updating version 0 to 2000 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: oid 26 version 2000 is already newer than 1999 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: oid 26 version 2000 is already newer than 1998 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 2000 (ObjNum 1314 snap 402 seq_num 1314) dirty exists 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3749: left oid 26 (ObjNum 1314 snap 402 seq_num 1314) 2024-09-17T15:50:28.179 INFO:tasks.rados.rados.0.smithi203.stdout:3750: finishing write tid 1 to smithi20328086-19 2024-09-17T15:50:28.180 INFO:tasks.rados.rados.0.smithi203.stdout:3749: done (2 left) 2024-09-17T15:50:28.180 INFO:tasks.rados.rados.0.smithi203.stdout:3752: write oid 32 current snap is 402 2024-09-17T15:50:28.180 INFO:tasks.rados.rados.0.smithi203.stdout:3752: seq_num 1316 ranges {518012=756753,2025715=92297} 2024-09-17T15:50:28.193 INFO:tasks.rados.rados.0.smithi203.stdout:3752: writing smithi20328086-32 from 518012 to 1274765 tid 1 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3752: writing smithi20328086-32 from 2025715 to 2118012 tid 2 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3753: copy_from oid 46 from oid 39 current snap is 402 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: finishing write tid 2 to smithi20328086-19 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: finishing write tid 3 to smithi20328086-19 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3751: finishing rollback tid 0 to smithi20328086-42 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: finishing write tid 4 to smithi20328086-19 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: oid 19 updating version 0 to 1918 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: oid 19 updating version 1918 to 1919 2024-09-17T15:50:28.202 INFO:tasks.rados.rados.0.smithi203.stdout:3750: oid 19 version 1919 is already newer than 1917 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1919 (ObjNum 1315 snap 402 seq_num 1315) dirty exists 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3750: left oid 19 (ObjNum 1315 snap 402 seq_num 1315) 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3751: finishing rollback tid 1 to smithi20328086-42 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3751: finishing rollback tid 2 to smithi20328086-42 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1452 (ObjNum 1230 snap 371 seq_num 1230) dirty exists 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3752: finishing write tid 1 to smithi20328086-32 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3750: done (3 left) 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3751: done (2 left) 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3754: write (excl) oid 27 current snap is 402 2024-09-17T15:50:28.203 INFO:tasks.rados.rados.0.smithi203.stdout:3754: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-17T15:50:28.210 INFO:tasks.rados.rados.0.smithi203.stdout:3754: writing smithi20328086-27 from 534819 to 966821 tid 1 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3754: writing smithi20328086-27 from 1684197 to 2134819 tid 2 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3755: snap_create 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3752: finishing write tid 2 to smithi20328086-32 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3752: finishing write tid 3 to smithi20328086-32 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3752: finishing write tid 4 to smithi20328086-32 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3752: oid 32 updating version 0 to 1739 2024-09-17T15:50:28.219 INFO:tasks.rados.rados.0.smithi203.stdout:3752: oid 32 updating version 1739 to 1740 2024-09-17T15:50:28.220 INFO:tasks.rados.rados.0.smithi203.stdout:3752: oid 32 version 1740 is already newer than 1738 2024-09-17T15:50:28.220 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1740 (ObjNum 1316 snap 402 seq_num 1316) dirty exists 2024-09-17T15:50:28.220 INFO:tasks.rados.rados.0.smithi203.stdout:3752: left oid 32 (ObjNum 1316 snap 402 seq_num 1316) 2024-09-17T15:50:28.237 INFO:tasks.rados.rados.0.smithi203.stdout:3753: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:50:28.237 INFO:tasks.rados.rados.0.smithi203.stdout:3754: finishing write tid 1 to smithi20328086-27 2024-09-17T15:50:28.246 INFO:tasks.rados.rados.0.smithi203.stdout:3753: finishing copy_from to smithi20328086-46 2024-09-17T15:50:28.247 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1887 (ObjNum 1300 snap 400 seq_num 1300) dirty exists 2024-09-17T15:50:28.247 INFO:tasks.rados.rados.0.smithi203.stdout:3754: finishing write tid 2 to smithi20328086-27 2024-09-17T15:50:28.247 INFO:tasks.rados.rados.0.smithi203.stdout:3754: finishing write tid 3 to smithi20328086-27 2024-09-17T15:50:28.248 INFO:tasks.rados.rados.0.smithi203.stdout:3754: finishing write tid 4 to smithi20328086-27 2024-09-17T15:50:28.248 INFO:tasks.rados.rados.0.smithi203.stdout:3754: oid 27 updating version 0 to 1889 2024-09-17T15:50:28.248 INFO:tasks.rados.rados.0.smithi203.stdout:3754: oid 27 version 1889 is already newer than 1453 2024-09-17T15:50:28.248 INFO:tasks.rados.rados.0.smithi203.stdout:3754: oid 27 version 1889 is already newer than 1888 2024-09-17T15:50:28.249 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1889 (ObjNum 1317 snap 402 seq_num 1317) dirty exists 2024-09-17T15:50:28.249 INFO:tasks.rados.rados.0.smithi203.stdout:3754: left oid 27 (ObjNum 1317 snap 402 seq_num 1317) 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3752: done (3 left) 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3753: done (2 left) 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3754: done (1 left) 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3755: done (0 left) 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3756: write oid 15 current snap is 403 2024-09-17T15:50:28.887 INFO:tasks.rados.rados.0.smithi203.stdout:3756: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-17T15:50:28.895 INFO:tasks.rados.rados.0.smithi203.stdout:3756: writing smithi20328086-15 from 551626 to 1058877 tid 1 2024-09-17T15:50:28.905 INFO:tasks.rados.rados.0.smithi203.stdout:3756: writing smithi20328086-15 from 1626326 to 2044782 tid 2 2024-09-17T15:50:28.905 INFO:tasks.rados.rados.0.smithi203.stdout:3757: write oid 10 current snap is 403 2024-09-17T15:50:28.905 INFO:tasks.rados.rados.0.smithi203.stdout:3757: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-09-17T15:50:28.915 INFO:tasks.rados.rados.0.smithi203.stdout:3757: writing smithi20328086-10 from 568433 to 1150933 tid 1 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3757: writing smithi20328086-10 from 1684808 to 2168433 tid 2 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3758: read oid 24 snap 401 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3758: expect (ObjNum 1301 snap 400 seq_num 1301) 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3759: copy_from oid 46 from oid 48 current snap is 403 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3756: finishing write tid 1 to smithi20328086-15 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3756: finishing write tid 2 to smithi20328086-15 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3756: finishing write tid 3 to smithi20328086-15 2024-09-17T15:50:28.924 INFO:tasks.rados.rados.0.smithi203.stdout:3756: finishing write tid 4 to smithi20328086-15 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3756: oid 15 updating version 0 to 1810 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3756: oid 15 version 1810 is already newer than 1808 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3756: oid 15 version 1810 is already newer than 1809 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1810 (ObjNum 1318 snap 403 seq_num 1318) dirty exists 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3756: left oid 15 (ObjNum 1318 snap 403 seq_num 1318) 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3756: done (3 left) 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3760: rollback oid 8 current snap is 403 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 8 to 380 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3761: copy_from oid 26 from oid 47 current snap is 403 2024-09-17T15:50:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:3762: snap_create 2024-09-17T15:50:28.935 INFO:tasks.rados.rados.0.smithi203.stdout:3758: expect (ObjNum 1301 snap 400 seq_num 1301) 2024-09-17T15:50:29.004 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2244679 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: finishing write tid 1 to smithi20328086-10 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: finishing write tid 2 to smithi20328086-10 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: finishing write tid 3 to smithi20328086-10 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: finishing write tid 4 to smithi20328086-10 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: oid 10 updating version 0 to 1416 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: oid 10 updating version 1416 to 1417 2024-09-17T15:50:29.156 INFO:tasks.rados.rados.0.smithi203.stdout:3757: oid 10 version 1417 is already newer than 1415 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1417 (ObjNum 1319 snap 403 seq_num 1319) dirty exists 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3757: left oid 10 (ObjNum 1319 snap 403 seq_num 1319) 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3760: finishing rollback tid 0 to smithi20328086-8 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3760: finishing rollback tid 1 to smithi20328086-8 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3760: finishing rollback tid 2 to smithi20328086-8 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1649 (ObjNum 1157 snap 357 seq_num 1157) dirty exists 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3759: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3759: finishing copy_from to smithi20328086-46 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1892 (ObjNum 1302 snap 400 seq_num 1302) dirty exists 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3761: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:3761: finishing copy_from to smithi20328086-26 2024-09-17T15:50:29.157 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1540 (ObjNum 1117 snap 344 seq_num 1117) dirty exists 2024-09-17T15:50:29.747 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:50:29.747 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:50:29.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:50:29.750 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:50:29.750 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:50:29.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.868+0000 7fc3d8600640 10 monclient: get_monmap_and_config 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.868+0000 7fc3d8600640 10 monclient: build_initial_monmap 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.868+0000 7fc3d8600640 10 monclient: monmap: 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:29.875155+0000 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:29.875155+0000 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:29.872 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 Processor -- start 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 -- start start 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient: init 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3d00689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d009c180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3d009c560 0x7fc3d013c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d009c560 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d009cba0 con 0x7fc3d00689d0 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fc3d0068db0 con 0x7fc3d009c560 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d009df20 con 0x7fc3d009c180 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d8600640 10 monclient(hunting): authenticate will time out at 2362.840790s 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3d009c560 0x7fc3d013c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3d009c560 0x7fc3d013c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58172/0 (socket says 172.21.15.86:58172) 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50532/0 (socket says 172.21.15.86:50532) 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 -- 172.21.15.86:0/2578395203 learned_addr learned my addr 172.21.15.86:0/2578395203 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:29.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): get_auth_request con 0x7fc3d009c560 auth_method 0 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 10 monclient(hunting): get_auth_request con 0x7fc3d009c180 auth_method 0 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 msgr2=0x7fc3d013a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 msgr2=0x7fc3d013a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:29.876 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d5b9b640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient(hunting): handle_auth_done global_id 234405 payload 274 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 -- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 msgr2=0x7fc3d013a4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 -- 172.21.15.86:0/2578395203 >> v1:172.21.15.203:6789/0 conn(0x7fc3d00689d0 legacy=0x7fc3d009ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient: _finish_hunting 0 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 monclient: found mon.noname-a 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:29.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.872+0000 7fc3d6b9d640 1 -- 172.21.15.86:0/2578395203 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d009e340 con 0x7fc3d009c560 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d6b9d640 1 -- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c560 msgr2=0x7fc3d013c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d6b9d640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c560 0x7fc3d013c9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009e80 tx=0x7fc3c801c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7c671e62f2c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 1 -- 172.21.15.86:0/2578395203 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80a64f0 con 0x7fc3d009c560 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: dump: 2024-09-17T15:50:29.881 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: _finish_auth 0 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: _check_auth_tickets 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 5 monclient: authenticate success, global_id 234405 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 1 -- 172.21.15.86:0/2578395203 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c80a6690 con 0x7fc3d009c560 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 1 -- 172.21.15.86:0/2578395203 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c80a6a60 con 0x7fc3d009c560 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d539a640 10 monclient: dump: 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 4 set_mon_vals no callback set 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: shutdown 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c560 msgr2=0x7fc3d013c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c560 0x7fc3d013c9f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009e80 tx=0x7fc3c801c660 comp rx=0 tx=0).stop 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- 172.21.15.86:0/2578395203 shutdown_connections 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c560 0x7fc3d013c9f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- 172.21.15.86:0/2578395203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d009c180 0x7fc3d013a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- 172.21.15.86:0/2578395203 >> 172.21.15.86:0/2578395203 conn(0x7fc3d006cf70 msgr2=0x7fc3d006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- 172.21.15.86:0/2578395203 shutdown_connections 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- 172.21.15.86:0/2578395203 wait complete. 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: build_initial_monmap 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: monmap: 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.885 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 Processor -- start 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- start start 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: init 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d009c180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3d009c560 0x7fc3d013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d009c560 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d009ea00 con 0x7fc3d009c180 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0074750 con 0x7fc3d009c560 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d009eb80 con 0x7fc3d00689d0 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d8600640 10 monclient(hunting): authenticate will time out at 2362.847745s 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58182/0 (socket says 172.21.15.86:58182) 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/3907788561 learned_addr learned my addr 172.21.15.86:0/3907788561 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:29.887 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 10 monclient(hunting): get_auth_request con 0x7fc3d00689d0 auth_method 0 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): get_auth_request con 0x7fc3d009c180 auth_method 0 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d5b9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.876+0000 7fc3d639c640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 msgr2=0x7fc3d01d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d639c640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 msgr2=0x7fc3d01d02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d639c640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:29.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d639c640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 10 monclient(hunting): handle_auth_done global_id 234408 payload 824 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 msgr2=0x7fc3d01d02b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d6b9d640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3d009c560 0x7fc3d013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3d009c560 msgr2=0x7fc3d013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3d009c560 0x7fc3d013df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 10 monclient: _finish_hunting 0 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 monclient: found mon.noname-a 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/3907788561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d01269a0 con 0x7fc3d009c180 2024-09-17T15:50:29.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 msgr2=0x7fc3d01d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d5b9b640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000e7e0 tx=0x7fc3c000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0017020 con 0x7fc3d009c180 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: dump: 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.892 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 5 monclient: authenticate success, global_id 234408 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c000c890 con 0x7fc3d009c180 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _renew_subs 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d00a6910 con 0x7fc3d009c180 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:29.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c000f040 con 0x7fc3d009c180 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 10 monclient: dump: 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:29.894 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d4b99640 4 set_mon_vals no callback set 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _renew_subs 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d01266c0 con 0x7fc3d009c180 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc3c001a020 con 0x7fc3d009c180 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d0125920 con 0x7fc3d009c180 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.880+0000 7fc3c6ffd640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 0x7fc3b403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.884+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(819..819 src has 696..819) ==== 4787+0+0 (secure 0 0 0) 0x7fc3c004ff30 con 0x7fc3d009c180 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.884+0000 7fc3d639c640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 0x7fc3b403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.884+0000 7fc3d639c640 10 monclient: get_auth_request con 0x7fc3b403bbb0 auth_method 0 2024-09-17T15:50:29.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.884+0000 7fc3d639c640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 0x7fc3b403e070 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.888+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc3d0125920 con 0x7fc3d009c180 2024-09-17T15:50:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.888+0000 7fc3c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:29.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:29.888+0000 7fc3c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:29.896 INFO:tasks.rados.rados.0.smithi203.stdout:3757: done (5 left) 2024-09-17T15:50:29.896 INFO:tasks.rados.rados.0.smithi203.stdout:3758: done (4 left) 2024-09-17T15:50:29.896 INFO:tasks.rados.rados.0.smithi203.stdout:3759: done (3 left) 2024-09-17T15:50:29.896 INFO:tasks.rados.rados.0.smithi203.stdout:3760: done (2 left) 2024-09-17T15:50:29.897 INFO:tasks.rados.rados.0.smithi203.stdout:3761: done (1 left) 2024-09-17T15:50:29.897 INFO:tasks.rados.rados.0.smithi203.stdout:3762: done (0 left) 2024-09-17T15:50:29.897 INFO:tasks.rados.rados.0.smithi203.stdout:3763: delete oid 16 current snap is 404 2024-09-17T15:50:29.905 INFO:tasks.rados.rados.0.smithi203.stdout:3763: done (0 left) 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:3764: rollback oid 40 current snap is 404 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 40 to 394 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:3765: read oid 28 snap 380 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:3765: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:3766: write (excl) oid 33 current snap is 404 2024-09-17T15:50:29.906 INFO:tasks.rados.rados.0.smithi203.stdout:3766: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-17T15:50:29.917 INFO:tasks.rados.rados.0.smithi203.stdout:3766: writing smithi20328086-33 from 602047 to 1335045 tid 1 2024-09-17T15:50:29.926 INFO:tasks.rados.rados.0.smithi203.stdout:3766: writing smithi20328086-33 from 1801772 to 2202047 tid 2 2024-09-17T15:50:29.926 INFO:tasks.rados.rados.0.smithi203.stdout:3767: write (excl) oid 3 current snap is 404 2024-09-17T15:50:29.926 INFO:tasks.rados.rados.0.smithi203.stdout:3764: finishing rollback tid 1 to smithi20328086-40 2024-09-17T15:50:29.926 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 40 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-17T15:50:29.926 INFO:tasks.rados.rados.0.smithi203.stdout:3765: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:50:29.927 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:29.965 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 311601 2024-09-17T15:50:30.000 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:30.038 INFO:tasks.rados.rados.0.smithi203.stdout:3766: finishing write tid 1 to smithi20328086-33 2024-09-17T15:50:30.038 INFO:tasks.rados.rados.0.smithi203.stdout:3766: finishing write tid 2 to smithi20328086-33 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: finishing write tid 3 to smithi20328086-33 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: finishing write tid 4 to smithi20328086-33 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: oid 33 updating version 0 to 1437 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: oid 33 version 1437 is already newer than 1435 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: oid 33 version 1437 is already newer than 1436 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1437 (ObjNum 1321 snap 404 seq_num 1321) dirty exists 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3766: left oid 33 (ObjNum 1321 snap 404 seq_num 1321) 2024-09-17T15:50:30.039 INFO:tasks.rados.rados.0.smithi203.stdout:3767: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-17T15:50:30.051 INFO:tasks.rados.rados.0.smithi203.stdout:3767: writing smithi20328086-3 from 618854 to 1027101 tid 1 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3767: writing smithi20328086-3 from 1743901 to 2169695 tid 2 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3764: done (3 left) 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3765: done (2 left) 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3766: done (1 left) 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3768: write (excl) oid 34 current snap is 404 2024-09-17T15:50:30.065 INFO:tasks.rados.rados.0.smithi203.stdout:3768: seq_num 1323 ranges {635661=599849,1918736=316925} 2024-09-17T15:50:30.080 INFO:tasks.rados.rados.0.smithi203.stdout:3768: writing smithi20328086-34 from 635661 to 1235510 tid 1 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3768: writing smithi20328086-34 from 1918736 to 2235661 tid 2 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3769: copy_from oid 19 from oid 18 current snap is 404 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: finishing write tid 1 to smithi20328086-3 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: finishing write tid 2 to smithi20328086-3 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: finishing write tid 3 to smithi20328086-3 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3770: rollback oid 1 current snap is 404 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 388 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: finishing write tid 4 to smithi20328086-3 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: oid 3 updating version 0 to 1894 2024-09-17T15:50:30.092 INFO:tasks.rados.rados.0.smithi203.stdout:3767: oid 3 version 1894 is already newer than 1893 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:3767: oid 3 updating version 1894 to 1895 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1895 (ObjNum 1322 snap 404 seq_num 1322) dirty exists 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:3767: left oid 3 (ObjNum 1322 snap 404 seq_num 1322) 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:3768: finishing write tid 1 to smithi20328086-34 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:3767: done (3 left) 2024-09-17T15:50:30.093 INFO:tasks.rados.rados.0.smithi203.stdout:3771: snap_remove snap 380 2024-09-17T15:50:30.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3d8600640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:30.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3d8600640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:30.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3d8600640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:30.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3d013d900 con 0x7fc3d009c180 2024-09-17T15:50:30.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3c6ffd640 1 -- 172.21.15.86:0/3907788561 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc3c0021020 con 0x7fc3d009c180 2024-09-17T15:50:30.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:30.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.128+0000 7fc3c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:30.132 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":210,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 msgr2=0x7fc3b403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 0x7fc3b403e070 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc007fd0 tx=0x7fc3cc009040 comp rx=0 tx=0).stop 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 10 monclient: shutdown 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 msgr2=0x7fc3d01d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000e7e0 tx=0x7fc3c000eca0 comp rx=0 tx=0).stop 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 shutdown_connections 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc3b403bbb0 0x7fc3b403e070 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3d009c560 0x7fc3d013df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3d009c180 0x7fc3d01d07f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 --2- 172.21.15.86:0/3907788561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3d00689d0 0x7fc3d01d02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 >> 172.21.15.86:0/3907788561 conn(0x7fc3d006cf70 msgr2=0x7fc3d00a1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 shutdown_connections 2024-09-17T15:50:30.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:30.132+0000 7fc3d8600640 1 -- 172.21.15.86:0/3907788561 wait complete. 2024-09-17T15:50:30.907 INFO:tasks.rados.rados.0.smithi203.stdout:3771: done (3 left) 2024-09-17T15:50:30.907 INFO:tasks.rados.rados.0.smithi203.stdout:3772: write (excl) oid 50 current snap is 404 2024-09-17T15:50:30.907 INFO:tasks.rados.rados.0.smithi203.stdout:3772: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-17T15:50:30.912 INFO:tasks.rados.rados.0.smithi203.stdout:3772: writing smithi20328086-50 from 652468 to 1327566 tid 1 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3772: writing smithi20328086-50 from 1977218 to 2252468 tid 2 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3773: snap_create 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: finishing write tid 2 to smithi20328086-34 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: finishing write tid 3 to smithi20328086-34 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: finishing write tid 4 to smithi20328086-34 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: oid 34 updating version 0 to 1689 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: oid 34 version 1689 is already newer than 1688 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: oid 34 updating version 1689 to 1690 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1690 (ObjNum 1323 snap 404 seq_num 1323) dirty exists 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3768: left oid 34 (ObjNum 1323 snap 404 seq_num 1323) 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3770: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:50:30.918 INFO:tasks.rados.rados.0.smithi203.stdout:3770: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:50:30.919 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1676 (ObjNum 1257 snap 380 seq_num 1257) dirty exists 2024-09-17T15:50:30.919 INFO:tasks.rados.rados.0.smithi203.stdout:3769: finishing copy_from racing read to smithi20328086-19 2024-09-17T15:50:30.919 INFO:tasks.rados.rados.0.smithi203.stdout:3769: finishing copy_from to smithi20328086-19 2024-09-17T15:50:30.919 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1639 (ObjNum 1241 snap 373 seq_num 1241) dirty exists 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: finishing write tid 1 to smithi20328086-50 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: finishing write tid 2 to smithi20328086-50 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: finishing write tid 3 to smithi20328086-50 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: finishing write tid 4 to smithi20328086-50 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: oid 50 updating version 0 to 1743 2024-09-17T15:50:30.931 INFO:tasks.rados.rados.0.smithi203.stdout:3772: oid 50 version 1743 is already newer than 1741 2024-09-17T15:50:30.932 INFO:tasks.rados.rados.0.smithi203.stdout:3772: oid 50 version 1743 is already newer than 1742 2024-09-17T15:50:30.932 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1743 (ObjNum 1324 snap 404 seq_num 1324) dirty exists 2024-09-17T15:50:30.932 INFO:tasks.rados.rados.0.smithi203.stdout:3772: left oid 50 (ObjNum 1324 snap 404 seq_num 1324) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3768: done (4 left) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3769: done (3 left) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3770: done (2 left) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3772: done (1 left) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3773: done (0 left) 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3774: write (excl) oid 6 current snap is 405 2024-09-17T15:50:31.907 INFO:tasks.rados.rados.0.smithi203.stdout:3774: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-17T15:50:31.922 INFO:tasks.rados.rados.0.smithi203.stdout:3774: writing smithi20328086-6 from 669275 to 1419622 tid 1 2024-09-17T15:50:31.931 INFO:tasks.rados.rados.0.smithi203.stdout:3774: writing smithi20328086-6 from 2035700 to 2269275 tid 2 2024-09-17T15:50:31.931 INFO:tasks.rados.rados.0.smithi203.stdout:3775: write (excl) oid 9 current snap is 405 2024-09-17T15:50:31.931 INFO:tasks.rados.rados.0.smithi203.stdout:3775: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-17T15:50:31.943 INFO:tasks.rados.rados.0.smithi203.stdout:3775: writing smithi20328086-9 from 686082 to 1111678 tid 1 2024-09-17T15:50:31.953 INFO:tasks.rados.rados.0.smithi203.stdout:3775: writing smithi20328086-9 from 1577829 to 2010961 tid 2 2024-09-17T15:50:31.953 INFO:tasks.rados.rados.0.smithi203.stdout:3776: delete oid 18 current snap is 405 2024-09-17T15:50:31.953 INFO:tasks.rados.rados.0.smithi203.stdout:3774: finishing write tid 1 to smithi20328086-6 2024-09-17T15:50:31.953 INFO:tasks.rados.rados.0.smithi203.stdout:3774: finishing write tid 2 to smithi20328086-6 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: finishing write tid 3 to smithi20328086-6 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: finishing write tid 4 to smithi20328086-6 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: oid 6 updating version 0 to 1679 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: oid 6 version 1679 is already newer than 1678 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: oid 6 version 1679 is already newer than 1677 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 6 v 1679 (ObjNum 1325 snap 405 seq_num 1325) dirty exists 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3774: left oid 6 (ObjNum 1325 snap 405 seq_num 1325) 2024-09-17T15:50:31.954 INFO:tasks.rados.rados.0.smithi203.stdout:3775: finishing write tid 1 to smithi20328086-9 2024-09-17T15:50:31.961 INFO:tasks.rados.rados.0.smithi203.stdout:3775: finishing write tid 2 to smithi20328086-9 2024-09-17T15:50:31.961 INFO:tasks.rados.rados.0.smithi203.stdout:3775: finishing write tid 3 to smithi20328086-9 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:3775: finishing write tid 4 to smithi20328086-9 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:3775: oid 9 updating version 0 to 1561 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:3775: oid 9 updating version 1561 to 1562 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:3775: oid 9 updating version 1562 to 1563 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1563 (ObjNum 1326 snap 405 seq_num 1326) dirty exists 2024-09-17T15:50:31.962 INFO:tasks.rados.rados.0.smithi203.stdout:3775: left oid 9 (ObjNum 1326 snap 405 seq_num 1326) 2024-09-17T15:50:31.968 INFO:tasks.rados.rados.0.smithi203.stdout:3774: done (2 left) 2024-09-17T15:50:31.968 INFO:tasks.rados.rados.0.smithi203.stdout:3775: done (1 left) 2024-09-17T15:50:31.968 INFO:tasks.rados.rados.0.smithi203.stdout:3776: done (0 left) 2024-09-17T15:50:31.968 INFO:tasks.rados.rados.0.smithi203.stdout:3777: write (excl) oid 28 current snap is 405 2024-09-17T15:50:31.968 INFO:tasks.rados.rados.0.smithi203.stdout:3777: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-17T15:50:31.981 INFO:tasks.rados.rados.0.smithi203.stdout:3777: writing smithi20328086-28 from 719696 to 1295790 tid 1 2024-09-17T15:50:31.992 INFO:tasks.rados.rados.0.smithi203.stdout:3777: writing smithi20328086-28 from 2094793 to 2319696 tid 2 2024-09-17T15:50:31.992 INFO:tasks.rados.rados.0.smithi203.stdout:3778: read oid 40 snap -1 2024-09-17T15:50:31.992 INFO:tasks.rados.rados.0.smithi203.stdout:3778: expect deleted 2024-09-17T15:50:31.992 INFO:tasks.rados.rados.0.smithi203.stdout:3779: snap_remove snap 397 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3779: done (2 left) 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3780: delete oid 36 current snap is 405 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: finishing write tid 1 to smithi20328086-28 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: finishing write tid 2 to smithi20328086-28 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: finishing write tid 3 to smithi20328086-28 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: finishing write tid 4 to smithi20328086-28 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: oid 28 updating version 0 to 2013 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: oid 28 updating version 2013 to 2014 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: oid 28 version 2014 is already newer than 2012 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 2014 (ObjNum 1328 snap 405 seq_num 1328) dirty exists 2024-09-17T15:50:32.923 INFO:tasks.rados.rados.0.smithi203.stdout:3777: left oid 28 (ObjNum 1328 snap 405 seq_num 1328) 2024-09-17T15:50:32.949 INFO:tasks.rados.rados.0.smithi203.stdout:3777: done (2 left) 2024-09-17T15:50:32.949 INFO:tasks.rados.rados.0.smithi203.stdout:3778: done (1 left) 2024-09-17T15:50:32.949 INFO:tasks.rados.rados.0.smithi203.stdout:3780: done (0 left) 2024-09-17T15:50:32.949 INFO:tasks.rados.rados.0.smithi203.stdout:3781: snap_remove snap 385 2024-09-17T15:50:33.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:33.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: get_monmap_and_config 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: build_initial_monmap 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: monmap: 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:33.281811+0000 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:33.281811+0000 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:33.278 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:33.279 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:33.279 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 Processor -- start 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 -- start start 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: init 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f89140689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:33.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8914068db0 0x7f89140691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-c con 0x7f8914068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8914139f00 0x7f891413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-a con 0x7f8914139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891409d770 con 0x7f89140689d0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f891409ed00 con 0x7f8914139f00 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f891409efe0 con 0x7f8914068db0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f891aa41640 10 monclient(hunting): authenticate will time out at 2366.247294s 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8913fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f89140689d0 0x7f891409ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49356/0 (socket says 172.21.15.86:49356) 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8913fff640 1 -- 172.21.15.86:0/3888748380 learned_addr learned my addr 172.21.15.86:0/3888748380 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 --2- 172.21.15.86:0/3888748380 >> v2:172.21.15.86:3300/0 conn(0x7f8914139f00 0x7f891413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): get_auth_request con 0x7f8914139f00 auth_method 0 2024-09-17T15:50:33.282 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f89137fe640 1 --2- 172.21.15.86:0/3888748380 >> v2:172.21.15.203:3300/0 conn(0x7f8914068db0 0x7f89140691c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f89137fe640 10 monclient(hunting): get_auth_request con 0x7f8914068db0 auth_method 0 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f89137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f89137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f89137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient(hunting): handle_auth_done global_id 234411 payload 274 2024-09-17T15:50:33.283 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 -- 172.21.15.86:0/3888748380 >> v2:172.21.15.203:3300/0 conn(0x7f8914068db0 msgr2=0x7f89140691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 --2- 172.21.15.86:0/3888748380 >> v2:172.21.15.203:3300/0 conn(0x7f8914068db0 0x7f89140691c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 -- 172.21.15.86:0/3888748380 >> v1:172.21.15.86:6789/0 conn(0x7f89140689d0 legacy=0x7f891409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient: _finish_hunting 0 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 monclient: found mon.noname-a 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.284 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.276+0000 7f8918fde640 1 -- 172.21.15.86:0/3888748380 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914069700 con 0x7f8914139f00 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 -- 172.21.15.86:0/3888748380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 msgr2=0x7f891413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 --2- 172.21.15.86:0/3888748380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f891413c2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89140a06b0 tx=0x7f890c01c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5e548702885375 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 1 -- 172.21.15.86:0/3888748380 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c0b2070 con 0x7f8914139f00 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: dump: 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.288 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 1 -- 172.21.15.86:0/3888748380 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890c0a5760 con 0x7f8914139f00 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 5 monclient: authenticate success, global_id 234411 2024-09-17T15:50:33.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 4 set_mon_vals no callback set 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 1 -- 172.21.15.86:0/3888748380 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c0a5b30 con 0x7f8914139f00 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8912ffd640 10 monclient: dump: 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:33.290 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: shutdown 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- 172.21.15.86:0/3888748380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 msgr2=0x7f891413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- 172.21.15.86:0/3888748380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f891413c2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f89140a06b0 tx=0x7f890c01c7d0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- 172.21.15.86:0/3888748380 shutdown_connections 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- 172.21.15.86:0/3888748380 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f891413c2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- 172.21.15.86:0/3888748380 >> v2:172.21.15.203:3300/0 conn(0x7f8914068db0 0x7f89140691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- 172.21.15.86:0/3888748380 >> 172.21.15.86:0/3888748380 conn(0x7f89140a6440 msgr2=0x7f89140a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:33.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- 172.21.15.86:0/3888748380 shutdown_connections 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- 172.21.15.86:0/3888748380 wait complete. 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: build_initial_monmap 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: monmap: 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 Processor -- start 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- start start 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: init 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 0x7f8914131650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-b con 0x7f89140689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-c con 0x7f8914068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): picked mon.noname-a con 0x7f8914139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f891413f3e0 con 0x7f8914139f00 2024-09-17T15:50:33.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f891413f260 con 0x7f89140689d0 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f891413f560 con 0x7f8914068db0 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f891aa41640 10 monclient(hunting): authenticate will time out at 2366.254464s 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58254/0 (socket says 172.21.15.86:58254) 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 -- 172.21.15.86:0/1888438388 learned_addr learned my addr 172.21.15.86:0/1888438388 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): get_auth_request con 0x7f8914139f00 auth_method 0 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:33.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 10 monclient(hunting): get_auth_request con 0x7f8914068db0 auth_method 0 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 msgr2=0x7f8914131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 msgr2=0x7f8914131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f89137fe640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8913fff640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 0x7f8914131650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient(hunting): handle_auth_done global_id 234414 payload 824 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 msgr2=0x7f8914131b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 msgr2=0x7f8914131650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 0x7f8914131650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient: _finish_hunting 0 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 monclient: found mon.noname-a 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8918fde640 1 -- 172.21.15.86:0/1888438388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914138b40 con 0x7f8914139f00 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8913fff640 10 monclient: get_auth_request con 0x7f89140689d0 auth_method 0 2024-09-17T15:50:33.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.284+0000 7f8913fff640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 0x7f8914131650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:50:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8918fde640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 msgr2=0x7f89141320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8918fde640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f890c0b0040 tx=0x7f890c019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:33.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c0b2050 con 0x7f8914139f00 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: dump: 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:33.299 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: _finish_auth 0 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890c0be070 con 0x7f8914139f00 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c0addd0 con 0x7f8914139f00 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 5 monclient: authenticate success, global_id 234414 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _renew_subs 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89140a1250 con 0x7f8914139f00 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:33.300 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 10 monclient: dump: 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f89127fc640 4 set_mon_vals no callback set 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _renew_subs 2024-09-17T15:50:33.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891411f560 con 0x7f8914139f00 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f890c0c3460 con 0x7f8914139f00 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89140691c0 con 0x7f8914139f00 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 0x7f890003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(823..823 src has 794..823) ==== 4831+0+0 (secure 0 0 0) 0x7f890c0eb220 con 0x7f8914139f00 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8913fff640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 0x7f890003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.288+0000 7f8913fff640 10 monclient: get_auth_request con 0x7f890003bb60 auth_method 0 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.292+0000 7f8913fff640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 0x7f890003e020 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f890800a850 tx=0x7f8908008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.292+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f890c0164e0 con 0x7f8914139f00 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.292+0000 7f8910ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:33.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.292+0000 7f8910ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:33.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f891aa41640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:33.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f891aa41640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:33.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f891aa41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:33.529 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f891406d4f0 con 0x7f8914139f00 2024-09-17T15:50:33.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f8910ff9640 1 -- 172.21.15.86:0/1888438388 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f890c0e9bf0 con 0x7f8914139f00 2024-09-17T15:50:33.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f8910ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:33.530 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.528+0000 7f8910ff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:33.530 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":210,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 msgr2=0x7f890003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 0x7f890003e020 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f890800a850 tx=0x7f8908008040 comp rx=0 tx=0).stop 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 10 monclient: shutdown 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 msgr2=0x7f89141320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f890c0b0040 tx=0x7f890c019570 comp rx=0 tx=0).stop 2024-09-17T15:50:33.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 shutdown_connections 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f890003bb60 0x7f890003e020 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8914139f00 0x7f89141320d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8914068db0 0x7f8914131b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 --2- 172.21.15.86:0/1888438388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89140689d0 0x7f8914131650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 >> 172.21.15.86:0/1888438388 conn(0x7f89140a6440 msgr2=0x7f89140a23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 shutdown_connections 2024-09-17T15:50:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:33.532+0000 7f891aa41640 1 -- 172.21.15.86:0/1888438388 wait complete. 2024-09-17T15:50:33.927 INFO:tasks.rados.rados.0.smithi203.stdout:3781: done (0 left) 2024-09-17T15:50:33.927 INFO:tasks.rados.rados.0.smithi203.stdout:3782: snap_remove snap 403 2024-09-17T15:50:34.944 INFO:tasks.rados.rados.0.smithi203.stdout:3782: done (0 left) 2024-09-17T15:50:34.944 INFO:tasks.rados.rados.0.smithi203.stdout:3783: delete oid 32 current snap is 405 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3783: done (0 left) 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3784: read oid 20 snap 375 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3784: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3785: read oid 43 snap -1 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3785: expect (ObjNum 1285 snap 396 seq_num 1285) 2024-09-17T15:50:34.959 INFO:tasks.rados.rados.0.smithi203.stdout:3786: snap_remove snap 396 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:3786: done (2 left) 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:3787: rollback oid 2 current snap is 405 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 402 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:3788: snap_create 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:3784: expect (ObjNum 1221 snap 368 seq_num 1221) 2024-09-17T15:50:35.876 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:36.043 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:36.102 INFO:tasks.rados.rados.0.smithi203.stdout:3785: expect (ObjNum 1285 snap 396 seq_num 1285) 2024-09-17T15:50:36.165 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:36.298 INFO:tasks.rados.rados.0.smithi203.stdout:3787: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:50:36.298 INFO:tasks.rados.rados.0.smithi203.stdout:3787: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:50:36.298 INFO:tasks.rados.rados.0.smithi203.stdout:3787: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:50:36.298 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1837 (ObjNum 1311 snap 402 seq_num 1311) dirty exists 2024-09-17T15:50:36.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:36.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: get_monmap_and_config 2024-09-17T15:50:36.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: build_initial_monmap 2024-09-17T15:50:36.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: monmap: 2024-09-17T15:50:36.677 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:36.677 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:36.681197+0000 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:36.681197+0000 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:36.678 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 Processor -- start 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 -- start start 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: init 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf80689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bf8068db0 0x7f7bf8069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf8068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bf8134a40 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:36.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf809d770 con 0x7f7bf8134a40 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf809ed00 con 0x7f7bf8068db0 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf809efe0 con 0x7f7bf80689d0 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bfd081640 10 monclient(hunting): authenticate will time out at 2369.647199s 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bf8068db0 0x7f7bf8069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bf8068db0 0x7f7bf8069190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58260/0 (socket says 172.21.15.86:58260) 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50558/0 (socket says 172.21.15.86:50558) 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 -- 172.21.15.86:0/4232004225 learned_addr learned my addr 172.21.15.86:0/4232004225 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): get_auth_request con 0x7f7bf80689d0 auth_method 0 2024-09-17T15:50:36.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): get_auth_request con 0x7f7bf8068db0 auth_method 0 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient(hunting): handle_auth_done global_id 234417 payload 274 2024-09-17T15:50:36.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 -- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 msgr2=0x7f7bf809ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 -- 172.21.15.86:0/4232004225 >> v1:172.21.15.203:6789/0 conn(0x7f7bf8134a40 legacy=0x7f7bf8134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient: _finish_hunting 0 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 monclient: found mon.noname-a 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf659c640 1 -- 172.21.15.86:0/4232004225 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf80a04e0 con 0x7f7bf8068db0 2024-09-17T15:50:36.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.676+0000 7f7bf6d9d640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:36.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.680+0000 7f7bf659c640 1 -- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 msgr2=0x7f7bf8069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:36.687 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.680+0000 7f7bf659c640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf8069190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009e50 tx=0x7f7be001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b9af63226d0496 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 1 -- 172.21.15.86:0/4232004225 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00a57e0 con 0x7f7bf8068db0 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: dump: 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:36.688 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 1 -- 172.21.15.86:0/4232004225 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be00a5980 con 0x7f7bf8068db0 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 5 monclient: authenticate success, global_id 234417 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 1 -- 172.21.15.86:0/4232004225 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00a5d50 con 0x7f7bf8068db0 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 4 set_mon_vals no callback set 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:36.689 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf5d9b640 10 monclient: dump: 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: shutdown 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 msgr2=0x7f7bf8069190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:36.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf8069190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009e50 tx=0x7f7be001c980 comp rx=0 tx=0).stop 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- 172.21.15.86:0/4232004225 shutdown_connections 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf8069190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- 172.21.15.86:0/4232004225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf80689d0 0x7f7bf809ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- 172.21.15.86:0/4232004225 >> 172.21.15.86:0/4232004225 conn(0x7f7bf80a6440 msgr2=0x7f7bf80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- 172.21.15.86:0/4232004225 shutdown_connections 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- 172.21.15.86:0/4232004225 wait complete. 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: build_initial_monmap 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: monmap: 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:36.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 Processor -- start 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- start start 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: init 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf80689d0 0x7f7bf813bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-b con 0x7f7bf80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-a con 0x7f7bf8068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:36.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): picked mon.noname-c con 0x7f7bf8134a40 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf813fad0 con 0x7f7bf8068db0 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf813f950 con 0x7f7bf80689d0 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf813fc50 con 0x7f7bf8134a40 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bfd081640 10 monclient(hunting): authenticate will time out at 2369.654474s 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58262/0 (socket says 172.21.15.86:58262) 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50570/0 (socket says 172.21.15.86:50570) 2024-09-17T15:50:36.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 -- 172.21.15.86:0/2017444075 learned_addr learned my addr 172.21.15.86:0/2017444075 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): get_auth_request con 0x7f7bf8068db0 auth_method 0 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): get_auth_request con 0x7f7bf8134a40 auth_method 0 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:36.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient(hunting): handle_auth_done global_id 234420 payload 824 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf6d9d640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf80689d0 0x7f7bf813bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 -- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 msgr2=0x7f7bf8140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 -- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf80689d0 msgr2=0x7f7bf813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf80689d0 0x7f7bf813bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient: _finish_hunting 0 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 monclient: found mon.noname-a 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf659c640 1 -- 172.21.15.86:0/2017444075 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf813b720 con 0x7f7bf8068db0 2024-09-17T15:50:36.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.684+0000 7f7bf759e640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:36.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bf659c640 1 -- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 msgr2=0x7f7bf813fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:36.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bf659c640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7be001d220 tx=0x7f7be0019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:36.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00bc020 con 0x7f7bf8068db0 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: dump: 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be00b8880 con 0x7f7bf8068db0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 5 monclient: authenticate success, global_id 234420 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _renew_subs 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf81419a0 con 0x7f7bf8068db0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00b8c50 con 0x7f7bf8068db0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 10 monclient: dump: 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bf559a640 4 set_mon_vals no callback set 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _renew_subs 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf813b160 con 0x7f7bf8068db0 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:36.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf809ffa0 con 0x7f7bf8068db0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7be00bf020 con 0x7f7bf8068db0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 0x7f7bd803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(826..826 src has 794..826) ==== 4891+0+0 (secure 0 0 0) 0x7f7be00eae90 con 0x7f7bf8068db0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bf6d9d640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 0x7f7bd803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.688+0000 7f7bf6d9d640 10 monclient: get_auth_request con 0x7f7bd803bbb0 auth_method 0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.692+0000 7f7bf6d9d640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 0x7f7bd803e070 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009580 tx=0x7f7be8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.692+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7be00c6030 con 0x7f7bf8068db0 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.692+0000 7f7bef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:36.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.692+0000 7f7bef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:36.876 INFO:tasks.rados.rados.0.smithi203.stdout:3784: done (3 left) 2024-09-17T15:50:36.876 INFO:tasks.rados.rados.0.smithi203.stdout:3785: done (2 left) 2024-09-17T15:50:36.876 INFO:tasks.rados.rados.0.smithi203.stdout:3787: done (1 left) 2024-09-17T15:50:36.876 INFO:tasks.rados.rados.0.smithi203.stdout:3788: done (0 left) 2024-09-17T15:50:36.876 INFO:tasks.rados.rados.0.smithi203.stdout:3789: delete oid 38 current snap is 406 2024-09-17T15:50:36.892 INFO:tasks.rados.rados.0.smithi203.stdout:3789: done (0 left) 2024-09-17T15:50:36.892 INFO:tasks.rados.rados.0.smithi203.stdout:3790: write oid 9 current snap is 406 2024-09-17T15:50:36.892 INFO:tasks.rados.rados.0.smithi203.stdout:3790: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-09-17T15:50:36.914 INFO:tasks.rados.rados.0.smithi203.stdout:3790: writing smithi20328086-9 from 786924 to 1380367 tid 1 2024-09-17T15:50:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.928+0000 7f7bfd081640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.928+0000 7f7bfd081640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.928+0000 7f7bfd081640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.928+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7bf813ab30 con 0x7f7bf8068db0 2024-09-17T15:50:36.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bef7fe640 1 -- 172.21.15.86:0/2017444075 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f7be0016600 con 0x7f7bf8068db0 2024-09-17T15:50:36.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:36.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:36.933 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":212,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:36.933 INFO:tasks.rados.rados.0.smithi203.stdout:3790: writing smithi20328086-9 from 1928721 to 2386924 tid 2 2024-09-17T15:50:36.933 INFO:tasks.rados.rados.0.smithi203.stdout:3791: read oid 47 snap -1 2024-09-17T15:50:36.933 INFO:tasks.rados.rados.0.smithi203.stdout:3791: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:50:36.933 INFO:tasks.rados.rados.0.smithi203.stdout:3792: snap_create 2024-09-17T15:50:36.933 INFO:tasks.rados.rados.0.smithi203.stdout:3790: finishing write tid 1 to smithi20328086-9 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 msgr2=0x7f7bd803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 0x7f7bd803e070 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009580 tx=0x7f7be8008040 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 10 monclient: shutdown 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 msgr2=0x7f7bf813fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7be001d220 tx=0x7f7be0019720 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 shutdown_connections 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bd803bbb0 0x7f7bd803e070 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bf8134a40 0x7f7bf8140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bf8068db0 0x7f7bf813fef0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 --2- 172.21.15.86:0/2017444075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bf80689d0 0x7f7bf813bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:36.935 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 >> 172.21.15.86:0/2017444075 conn(0x7f7bf80a6440 msgr2=0x7f7bf80740c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:36.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 shutdown_connections 2024-09-17T15:50:36.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:36.932+0000 7f7bfd081640 1 -- 172.21.15.86:0/2017444075 wait complete. 2024-09-17T15:50:36.939 INFO:tasks.rados.rados.0.smithi203.stdout:3791: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:50:36.953 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:50:36.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:50:36.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:50:36.967 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1256502 2024-09-17T15:50:37.029 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 718172 2024-09-17T15:50:37.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.072+0000 7faa2b916640 10 monclient: get_monmap_and_config 2024-09-17T15:50:37.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.072+0000 7faa2b916640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.072+0000 7faa2b916640 10 monclient: monmap: 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:37.079466+0000 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:37.079466+0000 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:37.076 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 Processor -- start 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 -- start start 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient: init 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa240a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faa240a1760 0x7faa2413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-c con 0x7faa240a1760 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa2413b450 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa240a2130 con 0x7faa240a0c70 2024-09-17T15:50:37.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa240a26f0 con 0x7faa2413b450 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7faa240a2410 con 0x7faa240a1760 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa2b916640 10 monclient(hunting): authenticate will time out at 2370.045240s 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa296b2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faa240a0c70 0x7faa240a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49366/0 (socket says 172.21.15.86:49366) 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 learned_addr learned my addr 172.21.15.86:0/961995225 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 1 --2- 172.21.15.86:0/961995225 >> v2:172.21.15.203:3300/0 conn(0x7faa240a1760 0x7faa2413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): get_auth_request con 0x7faa240a1760 auth_method 0 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 >> v1:172.21.15.86:6789/0 conn(0x7faa240a0c70 legacy=0x7faa240a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287144222 0 0) 0x7faa240a2130 con 0x7faa240a0c70 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 10 monclient(hunting): my global_id is 234423 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa10003530 con 0x7faa240a0c70 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.076+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 564481814 0 0) 0x7faa10003530 con 0x7faa240a0c70 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 >> v2:172.21.15.203:3300/0 conn(0x7faa240a1760 msgr2=0x7faa2413af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 --2- 172.21.15.86:0/961995225 >> v2:172.21.15.203:3300/0 conn(0x7faa240a1760 0x7faa2413af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 >> v1:172.21.15.203:6789/0 conn(0x7faa2413b450 legacy=0x7faa2413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 1 -- 172.21.15.86:0/961995225 >> v1:172.21.15.203:6789/0 conn(0x7faa2413b450 legacy=0x7faa2413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2409f9f0 con 0x7faa240a0c70 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7faa200019b0 con 0x7faa240a0c70 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: dump: 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.085 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 >> v1:172.21.15.86:6789/0 conn(0x7faa240a0c70 legacy=0x7faa240a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa10003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): picked mon.a con 0x7faa10003ba0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa10004810 0x7faa10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): picked mon.c con 0x7faa10004810 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa100099c0 0x7faa1000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): picked mon.b con 0x7faa100099c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:37.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faa240a2410 con 0x7faa10003ba0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa1000c3a0 con 0x7faa100099c0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faa1000c5a0 con 0x7faa10004810 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa1bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa10003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): get_auth_request con 0x7faa10003ba0 auth_method 0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa10004810 0x7faa10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): get_auth_request con 0x7faa10004810 auth_method 0 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa100099c0 0x7faa1000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 10 monclient(hunting): get_auth_request con 0x7faa100099c0 auth_method 0 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient(hunting): handle_auth_done global_id 234423 payload 293 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa10004810 msgr2=0x7faa10004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa10004810 0x7faa10004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa100099c0 msgr2=0x7faa1000bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa100099c0 0x7faa1000bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 monclient: found mon.a 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa29eb3640 1 -- 172.21.15.86:0/961995225 reap_dead start 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa10003530 con 0x7faa10003ba0 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa296b2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa2b916640 5 monclient: authenticate success, global_id 234423 2024-09-17T15:50:37.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.080+0000 7faa2b916640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 -- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 msgr2=0x7faa10003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa10003fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa20001bd0 tx=0x7faa2001cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f474e2dc7a48bd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2000c0e0 con 0x7faa10003ba0 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: dump: 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.096 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa200a9070 con 0x7faa10003ba0 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:37.097 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 4 set_mon_vals no callback set 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 1 -- 172.21.15.86:0/961995225 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa200a5e50 con 0x7faa10003ba0 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa1bfff640 10 monclient: dump: 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.098 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: shutdown 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 msgr2=0x7faa10003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa10003fc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faa20001bd0 tx=0x7faa2001cd60 comp rx=0 tx=0).stop 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- 172.21.15.86:0/961995225 shutdown_connections 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 --2- 172.21.15.86:0/961995225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa10003fc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- 172.21.15.86:0/961995225 >> 172.21.15.86:0/961995225 conn(0x7faa240a2f40 msgr2=0x7faa240a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- 172.21.15.86:0/961995225 shutdown_connections 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- 172.21.15.86:0/961995225 wait complete. 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: monmap: 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.099 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 Processor -- start 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- start start 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: init 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:50:37.100 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa24120300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-a con 0x7faa10003ba0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa241ce350 0x7faa24133130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-b con 0x7faa241ce350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa24133670 0x7faa241ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): picked mon.noname-c con 0x7faa24133670 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faa240a00d0 con 0x7faa10003ba0 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faa2409ff50 con 0x7faa241ce350 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faa24136260 con 0x7faa24133670 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.101 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa2b916640 10 monclient(hunting): authenticate will time out at 2370.055213s 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa24120300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa24133670 0x7faa241ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa24120300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58278/0 (socket says 172.21.15.86:58278) 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 -- 172.21.15.86:0/2915711513 learned_addr learned my addr 172.21.15.86:0/2915711513 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): get_auth_request con 0x7faa10003ba0 auth_method 0 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): get_auth_request con 0x7faa24133670 auth_method 0 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient(hunting): handle_auth_done global_id 234426 payload 824 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa241ce350 0x7faa24133130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa24133670 msgr2=0x7faa241ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa24133670 0x7faa241ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa241ce350 msgr2=0x7faa24133130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa241ce350 0x7faa24133130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 monclient: found mon.noname-a 2024-09-17T15:50:37.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa296b2640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24121b00 con 0x7faa10003ba0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa28eb1640 10 monclient: get_auth_request con 0x7faa241ce350 auth_method 0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa29eb3640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa24133670 0x7faa241ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.084+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faa241ce350 0x7faa24133130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa296b2640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 msgr2=0x7faa24120300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa296b2640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa24120300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faa200010f0 tx=0x7faa10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa200017b0 con 0x7faa10003ba0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: dump: 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.104 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa20001950 con 0x7faa10003ba0 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 5 monclient: authenticate success, global_id 234426 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 monclient: _renew_subs 2024-09-17T15:50:37.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa24120c70 con 0x7faa10003ba0 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20007370 con 0x7faa10003ba0 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 10 monclient: dump: 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.106 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa1b7fe640 4 set_mon_vals no callback set 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 monclient: _renew_subs 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa24121540 con 0x7faa10003ba0 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 0x7faa241cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faa241cb790 con 0x7faa241cae70 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faa200b6030 con 0x7faa10003ba0 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 0x7faa241cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient: get_auth_request con 0x7faa241cae70 auth_method 0 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faa0803bbb0 0x7faa0803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(827..827 src has 794..827) ==== 4787+0+0 (secure 0 0 0) 0x7faa200e9250 con 0x7faa10003ba0 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 10 monclient(hunting): handle_auth_done global_id 234429 payload 824 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa28eb1640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa24136560 con 0x7faa241cae70 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa29eb3640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faa0803bbb0 0x7faa0803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.088+0000 7faa29eb3640 10 monclient: get_auth_request con 0x7faa0803bbb0 auth_method 0 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa28eb1640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 msgr2=0x7faa241cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.108 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 0x7faa241cb250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa24052890 tx=0x7faa14013d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa14011bd0 con 0x7faa241cae70 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 10 monclient: dump: 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.109 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa14011bd0 con 0x7faa241cae70 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 msgr2=0x7faa241cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 0x7faa241cb250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa24052890 tx=0x7faa14013d10 comp rx=0 tx=0).stop 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 msgr2=0x7faa241cb250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa19ffb640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa241cae70 0x7faa241cb250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.110 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.092+0000 7faa29eb3640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faa0803bbb0 0x7faa0803e070 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faa240a40b0 tx=0x7faa1c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: finishing write tid 2 to smithi20328086-9 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: finishing write tid 3 to smithi20328086-9 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: finishing write tid 4 to smithi20328086-9 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: oid 9 updating version 0 to 1564 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: oid 9 updating version 1564 to 1565 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: oid 9 updating version 1565 to 1566 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 9 v 1566 (ObjNum 1332 snap 406 seq_num 1332) dirty exists 2024-09-17T15:50:37.111 INFO:tasks.rados.rados.0.smithi203.stdout:3790: left oid 9 (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 0x7faa24071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faa2411f7a0 con 0x7faa24071910 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 0x7faa24071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient: get_auth_request con 0x7faa24071910 auth_method 0 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 10 monclient(hunting): handle_auth_done global_id 234432 payload 824 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 1 -- 172.21.15.86:0/2915711513 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa2406d4f0 con 0x7faa24071910 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 msgr2=0x7faa24071d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa28eb1640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 0x7faa24071d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faa24052890 tx=0x7faa14004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1402d020 con 0x7faa24071910 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 10 monclient: dump: 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.114 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7faa140153c0 con 0x7faa24071910 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 msgr2=0x7faa24071d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 0x7faa24071d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faa24052890 tx=0x7faa14004740 comp rx=0 tx=0).stop 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 msgr2=0x7faa24071d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa19ffb640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa24071910 0x7faa24071d40 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faa24052890 tx=0x7faa14004740 comp rx=0 tx=0).stop 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 166295, 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:37.116 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.117 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.118 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.119 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.120 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:37.121 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.122 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.123 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.124 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:37.127 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:37.128 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.129 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:37.129 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faa0803bbb0 msgr2=0x7faa0803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faa0803bbb0 0x7faa0803e070 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7faa240a40b0 tx=0x7faa1c009040 comp rx=0 tx=0).stop 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 10 monclient: shutdown 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 msgr2=0x7faa24120300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 --2- 172.21.15.86:0/2915711513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa10003ba0 0x7faa24120300 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faa200010f0 tx=0x7faa10005840 comp rx=0 tx=0).stop 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa29eb3640 1 -- 172.21.15.86:0/2915711513 reap_dead start 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa29eb3640 1 -- 172.21.15.86:0/2915711513 reap_dead start 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 shutdown_connections 2024-09-17T15:50:37.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 >> 172.21.15.86:0/2915711513 conn(0x7faa240a2f40 msgr2=0x7faa241c08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 shutdown_connections 2024-09-17T15:50:37.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.100+0000 7faa2b916640 1 -- 172.21.15.86:0/2915711513 wait complete. 2024-09-17T15:50:37.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:37.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:50:37.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:50:37.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: get_monmap_and_config 2024-09-17T15:50:37.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: monmap: 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:37.252507+0000 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:37.252507+0000 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:37.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 Processor -- start 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 -- start start 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: init 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b3009d770 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3009e260 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3b3013a440 0x7f3b3013c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3013a440 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b3009ff00 con 0x7f3b3009d770 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3b3009e7f0 con 0x7f3b3013a440 2024-09-17T15:50:37.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b300a2110 con 0x7f3b3009e260 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b372a8640 10 monclient(hunting): authenticate will time out at 2370.218121s 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50608/0 (socket says 172.21.15.86:50608) 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 -- 172.21.15.86:0/2023277918 learned_addr learned my addr 172.21.15.86:0/2023277918 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): get_auth_request con 0x7f3b3009e260 auth_method 0 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b35845640 1 --2- 172.21.15.86:0/2023277918 >> v2:172.21.15.203:3300/0 conn(0x7f3b3013a440 0x7f3b3013c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b35845640 10 monclient(hunting): get_auth_request con 0x7f3b3013a440 auth_method 0 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b35845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b35845640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b35845640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient(hunting): handle_auth_done global_id 264110 payload 274 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 -- 172.21.15.86:0/2023277918 >> v2:172.21.15.203:3300/0 conn(0x7f3b3013a440 msgr2=0x7f3b3013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 --2- 172.21.15.86:0/2023277918 >> v2:172.21.15.203:3300/0 conn(0x7f3b3013a440 0x7f3b3013c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 -- 172.21.15.86:0/2023277918 >> v1:172.21.15.203:6789/0 conn(0x7f3b3009d770 legacy=0x7f3b3009db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 monclient: found mon.noname-b 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:50:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.248+0000 7f3b34843640 1 -- 172.21.15.86:0/2023277918 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3009ec30 con 0x7f3b3009e260 2024-09-17T15:50:37.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b34843640 1 -- 172.21.15.86:0/2023277918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 msgr2=0x7f3b30139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b34843640 1 --2- 172.21.15.86:0/2023277918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b2401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b950eaabf9a2990 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 1 -- 172.21.15.86:0/2023277918 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240b0070 con 0x7f3b3009e260 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: dump: 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.257 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 1 -- 172.21.15.86:0/2023277918 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b240ab600 con 0x7f3b3009e260 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 1 -- 172.21.15.86:0/2023277918 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240ab9d0 con 0x7f3b3009e260 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 5 monclient: authenticate success, global_id 264110 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 4 set_mon_vals no callback set 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:37.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b2ffff640 10 monclient: dump: 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 10 monclient: shutdown 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.259 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 -- 172.21.15.86:0/2023277918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 msgr2=0x7f3b30139f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 --2- 172.21.15.86:0/2023277918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009870 tx=0x7f3b2401c420 comp rx=0 tx=0).stop 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 -- 172.21.15.86:0/2023277918 shutdown_connections 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 --2- 172.21.15.86:0/2023277918 >> v2:172.21.15.203:3300/0 conn(0x7f3b3013a440 0x7f3b3013c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 --2- 172.21.15.86:0/2023277918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3009e260 0x7f3b30139f00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 -- 172.21.15.86:0/2023277918 >> 172.21.15.86:0/2023277918 conn(0x7f3b300a2f40 msgr2=0x7f3b300a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 -- 172.21.15.86:0/2023277918 shutdown_connections 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.252+0000 7f3b372a8640 1 -- 172.21.15.86:0/2023277918 wait complete. 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient: monmap: 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:37.260 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 Processor -- start 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 -- start start 2024-09-17T15:50:37.261 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient: init 2024-09-17T15:50:37.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 0x7f3b30132910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3009d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b3009e260 0x7f3b30132e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-b con 0x7f3b3009e260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 0x7f3b30072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3013a440 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3013f590 con 0x7f3b3009d770 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3b300a0740 con 0x7f3b3009e260 2024-09-17T15:50:37.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3013f710 con 0x7f3b3013a440 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b372a8640 10 monclient(hunting): authenticate will time out at 2370.225485s 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 0x7f3b30132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 0x7f3b30072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 0x7f3b30072900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50610/0 (socket says 172.21.15.86:50610) 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 1 -- 172.21.15.86:0/1308408914 learned_addr learned my addr 172.21.15.86:0/1308408914 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 0x7f3b30132910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58296/0 (socket says 172.21.15.86:58296) 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): get_auth_request con 0x7f3b3013a440 auth_method 0 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): get_auth_request con 0x7f3b3009d770 auth_method 0 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b3009e260 0x7f3b30132e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 10 monclient(hunting): get_auth_request con 0x7f3b3009e260 auth_method 0 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient(hunting): handle_auth_done global_id 234435 payload 824 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 msgr2=0x7f3b30072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 0x7f3b30072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b3009e260 msgr2=0x7f3b30132e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b3009e260 0x7f3b30132e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 monclient: found mon.noname-a 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35044640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3013c000 con 0x7f3b3009d770 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b35845640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b3013a440 0x7f3b30072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.256+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3b3009e260 0x7f3b30132e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T15:50:37.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b35044640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 msgr2=0x7f3b30132910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b35044640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 0x7f3b30132910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c700 tx=0x7f3b2800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28016020 con 0x7f3b3009d770 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: dump: 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.270 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b2800ac60 con 0x7f3b3009d770 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 5 monclient: authenticate success, global_id 234435 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 monclient: _renew_subs 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3013c2e0 con 0x7f3b3009d770 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b28012660 con 0x7f3b3009d770 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 10 monclient: dump: 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 monclient: _renew_subs 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.272 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3013bd20 con 0x7f3b3009d770 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2f7fe640 4 set_mon_vals no callback set 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 0x7f3b3002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3009db50 con 0x7f3b30073600 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 0x7f3b3002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient: get_auth_request con 0x7f3b30073600 auth_method 0 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 10 monclient(hunting): handle_auth_done global_id 264116 payload 824 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b34843640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b30135470 con 0x7f3b30073600 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f3b28019020 con 0x7f3b3009d770 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b1c03bbe0 0x7f3b1c03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(827..827 src has 794..827) ==== 4787+0+0 (secure 0 0 0) 0x7f3b280501b0 con 0x7f3b3009d770 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b35845640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b1c03bbe0 0x7f3b1c03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.260+0000 7f3b35845640 10 monclient: get_auth_request con 0x7f3b1c03bbe0 auth_method 0 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b34843640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 msgr2=0x7f3b3002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 0x7f3b3002d450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b240ab000 tx=0x7f3b240191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240ba020 con 0x7f3b30073600 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 10 monclient: dump: 2024-09-17T15:50:37.274 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3b240b6470 con 0x7f3b30073600 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 msgr2=0x7f3b3002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.275 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 0x7f3b3002d450 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b240ab000 tx=0x7f3b240191c0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 msgr2=0x7f3b3002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b2dffb640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b30073600 0x7f3b3002d450 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b240ab000 tx=0x7f3b240191c0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.264+0000 7f3b35845640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b1c03bbe0 0x7f3b1c03e0a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3b20007e50 tx=0x7f3b20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b372a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:37.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b372a8640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 0x7f3b3002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b372a8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3006d4f0 con 0x7f3b300739e0 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 0x7f3b3002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient: get_auth_request con 0x7f3b300739e0 auth_method 0 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 10 monclient(hunting): handle_auth_done global_id 264119 payload 824 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.268+0000 7f3b34843640 1 -- 172.21.15.86:0/1308408914 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b3013aa90 con 0x7f3b300739e0 2024-09-17T15:50:37.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b34843640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 msgr2=0x7f3b3002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b34843640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 0x7f3b3002d990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3b2401ccc0 tx=0x7f3b240a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b240ba030 con 0x7f3b300739e0 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 10 monclient: dump: 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.279 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f3b240c1220 con 0x7f3b300739e0 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 msgr2=0x7f3b3002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 0x7f3b3002d990 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3b2401ccc0 tx=0x7f3b240a8000 comp rx=0 tx=0).stop 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 msgr2=0x7f3b3002d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b2dffb640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b300739e0 0x7f3b3002d990 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3b2401ccc0 tx=0x7f3b240a8000 comp rx=0 tx=0).stop 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:37.280 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 7439, 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:37.281 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.282 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.283 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.284 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.286 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.287 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.289 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.291 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.292 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:37.293 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b1c03bbe0 msgr2=0x7f3b1c03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f3b1c03bbe0 0x7f3b1c03e0a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f3b20007e50 tx=0x7f3b20008040 comp rx=0 tx=0).stop 2024-09-17T15:50:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 10 monclient: shutdown 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 msgr2=0x7f3b30132910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 --2- 172.21.15.86:0/1308408914 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b3009d770 0x7f3b30132910 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800c700 tx=0x7f3b2800cbc0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b35845640 1 -- 172.21.15.86:0/1308408914 reap_dead start 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b35845640 1 -- 172.21.15.86:0/1308408914 reap_dead start 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 shutdown_connections 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 >> 172.21.15.86:0/1308408914 conn(0x7f3b300a2f40 msgr2=0x7f3b30139680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 shutdown_connections 2024-09-17T15:50:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.272+0000 7f3b372a8640 1 -- 172.21.15.86:0/1308408914 wait complete. 2024-09-17T15:50:37.295 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:37.296 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:50:37.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: get_monmap_and_config 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: monmap: 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:37.415019+0000 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:37.415019+0000 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.411 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 Processor -- start 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 -- start start 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: init 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6640a50e0 0x7fa66409cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa6640a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6640a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa66413d2f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6640a6280 con 0x7fa6640a54c0 2024-09-17T15:50:37.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66409c0a0 con 0x7fa66413d2f0 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa66409d2a0 con 0x7fa6640a50e0 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66c1f1640 10 monclient(hunting): authenticate will time out at 2370.380525s 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6640a50e0 0x7fa66409cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66978c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa6640a54c0 0x7fa6640a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49372/0 (socket says 172.21.15.86:49372) 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6640a50e0 0x7fa66409cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58308/0 (socket says 172.21.15.86:58308) 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa66978c640 1 -- 172.21.15.86:0/3741964301 learned_addr learned my addr 172.21.15.86:0/3741964301 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): get_auth_request con 0x7fa6640a50e0 auth_method 0 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.415 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient(hunting): handle_auth_done global_id 234438 payload 274 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 -- 172.21.15.86:0/3741964301 >> v1:172.21.15.203:6789/0 conn(0x7fa66413d2f0 legacy=0x7fa66413f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 -- 172.21.15.86:0/3741964301 >> v1:172.21.15.86:6789/0 conn(0x7fa6640a54c0 legacy=0x7fa6640a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 monclient: found mon.noname-a 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.416 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.412+0000 7fa669f8d640 1 -- 172.21.15.86:0/3741964301 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66409c3d0 con 0x7fa6640a50e0 2024-09-17T15:50:37.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 1 -- 172.21.15.86:0/3741964301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6640a50e0 msgr2=0x7fa66409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 1 --2- 172.21.15.86:0/3741964301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6640a50e0 0x7fa66409cd60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097c0 tx=0x7fa66001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6b7dc1d86e5789 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 1 -- 172.21.15.86:0/3741964301 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600b0070 con 0x7fa6640a50e0 2024-09-17T15:50:37.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.417 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: dump: 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 5 monclient: authenticate success, global_id 234438 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 1 -- 172.21.15.86:0/3741964301 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6600ab600 con 0x7fa6640a50e0 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 1 -- 172.21.15.86:0/3741964301 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600ab9d0 con 0x7fa6640a50e0 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 4 set_mon_vals no callback set 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa668f8b640 10 monclient: dump: 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.419 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.420 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: shutdown 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/3741964301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6640a50e0 msgr2=0x7fa66409cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/3741964301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6640a50e0 0x7fa66409cd60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6600097c0 tx=0x7fa66001c360 comp rx=0 tx=0).stop 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/3741964301 shutdown_connections 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/3741964301 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6640a50e0 0x7fa66409cd60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/3741964301 >> 172.21.15.86:0/3741964301 conn(0x7fa6640aaae0 msgr2=0x7fa6640aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/3741964301 shutdown_connections 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/3741964301 wait complete. 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: monmap: 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:37.426 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 Processor -- start 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- start start 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: init 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640a50e0 0x7fa6641cff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa6640a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa6640a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa66413d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641439b0 con 0x7fa66413d2f0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa664143830 con 0x7fa6640a50e0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa664143b30 con 0x7fa6640a54c0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): authenticate will time out at 2370.388239s 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58310/0 (socket says 172.21.15.86:58310) 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50630/0 (socket says 172.21.15.86:50630) 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 learned_addr learned my addr 172.21.15.86:0/450396347 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): get_auth_request con 0x7fa66413d2f0 auth_method 0 2024-09-17T15:50:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): get_auth_request con 0x7fa6640a54c0 auth_method 0 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 msgr2=0x7fa664138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640a50e0 0x7fa6641cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 msgr2=0x7fa664138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 10 monclient(hunting): get_auth_request con 0x7fa6640a50e0 auth_method 0 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66978c640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa669f8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient(hunting): handle_auth_done global_id 234441 payload 824 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 msgr2=0x7fa664138240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6640a54c0 0x7fa664138240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640a50e0 msgr2=0x7fa6641cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6640a50e0 0x7fa6641cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 monclient: found mon.noname-a 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6640a18d0 con 0x7fa66413d2f0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 msgr2=0x7fa66409ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66a78e640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c780 tx=0x7fa65c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c016020 con 0x7fa66413d2f0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: dump: 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.431 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 5 monclient: authenticate success, global_id 234441 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _renew_subs 2024-09-17T15:50:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c00ac00 con 0x7fa66413d2f0 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6640a0330 con 0x7fa66413d2f0 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65c012420 con 0x7fa66413d2f0 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65a7fc640 10 monclient: dump: 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.433 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa65bfff640 4 set_mon_vals no callback set 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _renew_subs 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641219e0 con 0x7fa66413d2f0 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 0x7fa66409ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.420+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6640a8060 con 0x7fa664138780 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa65c019020 con 0x7fa66413d2f0 2024-09-17T15:50:37.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 0x7fa66409ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa64803bb90 0x7fa64803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient: get_auth_request con 0x7fa664138780 auth_method 0 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(827..827 src has 794..827) ==== 4787+0+0 (secure 0 0 0) 0x7fa65c04fba0 con 0x7fa66413d2f0 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa66978c640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa64803bb90 0x7fa64803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa66978c640 10 monclient: get_auth_request con 0x7fa64803bb90 auth_method 0 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 10 monclient(hunting): handle_auth_done global_id 254185 payload 824 2024-09-17T15:50:37.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa66409cd60 con 0x7fa664138780 2024-09-17T15:50:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 msgr2=0x7fa66409ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa669f8d640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 0x7fa66409ef40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa6600ad000 tx=0x7fa660019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa66978c640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa64803bb90 0x7fa64803e050 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa654007ca0 tx=0x7fa654007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600bb070 con 0x7fa664138780 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.424+0000 7fa65a7fc640 10 monclient: dump: 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6600c0450 con 0x7fa664138780 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 msgr2=0x7fa66409ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 0x7fa66409ef40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa6600ad000 tx=0x7fa660019100 comp rx=0 tx=0).stop 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 msgr2=0x7fa66409ef40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.428+0000 7fa65a7fc640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664138780 0x7fa66409ef40 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa6600ad000 tx=0x7fa660019100 comp rx=0 tx=0).stop 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 0x7fa6641392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa66409f860 con 0x7fa66409f480 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 0x7fa6641392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient: get_auth_request con 0x7fa66409f480 auth_method 0 2024-09-17T15:50:37.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 10 monclient(hunting): handle_auth_done global_id 254188 payload 824 2024-09-17T15:50:37.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 1 -- 172.21.15.86:0/450396347 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6640a0e70 con 0x7fa66409f480 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 msgr2=0x7fa6641392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa669f8d640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 0x7fa6641392d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c10 tx=0x7fa660009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6600b0040 con 0x7fa66409f480 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 10 monclient: dump: 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.441 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fa660003e10 con 0x7fa66409f480 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:37.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 msgr2=0x7fa6641392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 53385, 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:37.443 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.444 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.445 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.446 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:37.447 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:37.448 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.449 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:37.451 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.454 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:37.455 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 0x7fa6641392d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c10 tx=0x7fa660009190 comp rx=0 tx=0).stop 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 msgr2=0x7fa6641392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa65a7fc640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa66409f480 0x7fa6641392d0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c10 tx=0x7fa660009190 comp rx=0 tx=0).stop 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa64803bb90 msgr2=0x7fa64803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa64803bb90 0x7fa64803e050 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa654007ca0 tx=0x7fa654007bf0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 10 monclient: shutdown 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 msgr2=0x7fa66409ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.457 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c780 tx=0x7fa65c00cc40 comp rx=0 tx=0).stop 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66a78e640 1 -- 172.21.15.86:0/450396347 reap_dead start 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 shutdown_connections 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 --2- 172.21.15.86:0/450396347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa66413d2f0 0x7fa66409ea00 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 >> 172.21.15.86:0/450396347 conn(0x7fa6640aaae0 msgr2=0x7fa6640a3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 shutdown_connections 2024-09-17T15:50:37.458 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.436+0000 7fa66c1f1640 1 -- 172.21.15.86:0/450396347 wait complete. 2024-09-17T15:50:37.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:37.458 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:50:37.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:50:37.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: get_monmap_and_config 2024-09-17T15:50:37.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: monmap: 2024-09-17T15:50:37.570 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.570 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:37.574388+0000 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:37.574388+0000 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:37.571 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 Processor -- start 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 -- start start 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: init 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7180689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 0x7fd71806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-b con 0x7fd7180694c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd71813aff0 0x7fd71813d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-c con 0x7fd71813aff0 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd718074470 con 0x7fd7180689d0 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd71809c8b0 con 0x7fd71813aff0 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd71809cb90 con 0x7fd7180694c0 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd71d2da640 10 monclient(hunting): authenticate will time out at 2370.539916s 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd716ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd7180689d0 0x7fd718068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49380/0 (socket says 172.21.15.86:49380) 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd716ffd640 1 -- 172.21.15.86:0/3109288680 learned_addr learned my addr 172.21.15.86:0/3109288680 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 --2- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 0x7fd71806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): get_auth_request con 0x7fd7180694c0 auth_method 0 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7177fe640 1 --2- 172.21.15.86:0/3109288680 >> v2:172.21.15.203:3300/0 conn(0x7fd71813aff0 0x7fd71813d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7177fe640 10 monclient(hunting): get_auth_request con 0x7fd71813aff0 auth_method 0 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient(hunting): handle_auth_done global_id 264125 payload 274 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 -- 172.21.15.86:0/3109288680 >> v2:172.21.15.203:3300/0 conn(0x7fd71813aff0 msgr2=0x7fd71813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 --2- 172.21.15.86:0/3109288680 >> v2:172.21.15.203:3300/0 conn(0x7fd71813aff0 0x7fd71813d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 -- 172.21.15.86:0/3109288680 >> v1:172.21.15.86:6789/0 conn(0x7fd7180689d0 legacy=0x7fd718068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 monclient: found mon.noname-b 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:50:37.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.572+0000 7fd7167fc640 1 -- 172.21.15.86:0/3109288680 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71809ce70 con 0x7fd7180694c0 2024-09-17T15:50:37.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 1 -- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 msgr2=0x7fd71806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 1 --2- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 0x7fd71806f840 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd7080097c0 tx=0x7fd70801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de8f2ac69849a74e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 1 -- 172.21.15.86:0/3109288680 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b0070 con 0x7fd7180694c0 2024-09-17T15:50:37.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.578 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: dump: 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 1 -- 172.21.15.86:0/3109288680 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7080ab600 con 0x7fd7180694c0 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 1 -- 172.21.15.86:0/3109288680 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71809c2d0 con 0x7fd7180694c0 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 5 monclient: authenticate success, global_id 264125 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 4 set_mon_vals no callback set 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd715ffb640 10 monclient: dump: 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.580 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: shutdown 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 msgr2=0x7fd71806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 0x7fd71806f840 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd7080097c0 tx=0x7fd70801c360 comp rx=0 tx=0).stop 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- 172.21.15.86:0/3109288680 shutdown_connections 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- 172.21.15.86:0/3109288680 >> v2:172.21.15.203:3300/0 conn(0x7fd71813aff0 0x7fd71813d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- 172.21.15.86:0/3109288680 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7180694c0 0x7fd71806f840 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- 172.21.15.86:0/3109288680 >> 172.21.15.86:0/3109288680 conn(0x7fd7180a6440 msgr2=0x7fd7180a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- 172.21.15.86:0/3109288680 shutdown_connections 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- 172.21.15.86:0/3109288680 wait complete. 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: build_initial_monmap 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: monmap: 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:37.582 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 Processor -- start 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- start start 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: init 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7180689d0 0x7fd71811eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-b con 0x7fd7180689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-a con 0x7fd7180694c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71813aff0 0x7fd71811c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): picked mon.noname-c con 0x7fd71813aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd718140610 con 0x7fd7180694c0 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd718140490 con 0x7fd7180689d0 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd718140790 con 0x7fd71813aff0 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd71d2da640 10 monclient(hunting): authenticate will time out at 2370.547672s 2024-09-17T15:50:37.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:58316/0 (socket says 172.21.15.86:58316) 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 1 -- 172.21.15.86:0/2872385916 learned_addr learned my addr 172.21.15.86:0/2872385916 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): get_auth_request con 0x7fd7180694c0 auth_method 0 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71813aff0 0x7fd71811c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): get_auth_request con 0x7fd71813aff0 auth_method 0 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7167fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.576+0000 7fd716ffd640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7180689d0 0x7fd71811eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd716ffd640 10 monclient(hunting): get_auth_request con 0x7fd7180689d0 auth_method 0 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd716ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd716ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd716ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 10 monclient(hunting): handle_auth_done global_id 234444 payload 824 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 -- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71813aff0 msgr2=0x7fd71811c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71813aff0 0x7fd71811c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 -- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7180689d0 msgr2=0x7fd71811eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7180689d0 0x7fd71811eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 10 monclient: _finish_hunting 0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 monclient: found mon.noname-a 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 -- 172.21.15.86:0/2872385916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd718120380 con 0x7fd7180694c0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 -- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 msgr2=0x7fd7181359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7167fc640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd70801cc00 tx=0x7fd70801aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b0050 con 0x7fd7180694c0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: dump: 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7080bc070 con 0x7fd7180694c0 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:37.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080b5c80 con 0x7fd7180694c0 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 5 monclient: authenticate success, global_id 234444 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _renew_subs 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71809d460 con 0x7fd7180694c0 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd7157fa640 4 set_mon_vals no callback set 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 10 monclient: dump: 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:37.593 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _renew_subs 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7181200a0 con 0x7fd7180694c0 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd71806f840 con 0x7fd7180694c0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd7080ab4c0 con 0x7fd7180694c0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 0x7fd70403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.580+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(827..827 src has 794..827) ==== 4787+0+0 (secure 0 0 0) 0x7fd7080ea7b0 con 0x7fd7180694c0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.584+0000 7fd716ffd640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 0x7fd70403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.584+0000 7fd716ffd640 10 monclient: get_auth_request con 0x7fd70403bb60 auth_method 0 2024-09-17T15:50:37.595 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.584+0000 7fd716ffd640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 0x7fd70403e020 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd70c007fd0 tx=0x7fd70c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.588+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd708016070 con 0x7fd7180694c0 2024-09-17T15:50:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.588+0000 7fd6ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:37.596 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.588+0000 7fd6ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd71d2da640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:50:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd71d2da640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:50:37.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd71d2da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:37.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd7181ce7b0 con 0x7fd7180694c0 2024-09-17T15:50:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd6ff7fe640 1 -- 172.21.15.86:0/2872385916 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd7080c1c60 con 0x7fd7180694c0 2024-09-17T15:50:37.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd6ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:50:37.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.827+0000 7fd6ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:37.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3593604572,config=3541143443,health=577633902,kv=2814306489,logm=3168015809}) 2024-09-17T15:50:37.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=11,kv=11,logm=48} crc {auth=869843146,config=3541143443,health=4110111587,kv=3591778543,logm=4200465962}) 2024-09-17T15:50:37.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3593604572,config=3541143443,health=577633902,kv=2814306489,logm=3168015809}) 2024-09-17T15:50:37.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.827+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=11,logm=49} crc {auth=3637451415,config=3541143443,health=1019750300,kv=3591778543,logm=924175843}) 2024-09-17T15:50:37.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 msgr2=0x7fd70403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 0x7fd70403e020 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd70c007fd0 tx=0x7fd70c009040 comp rx=0 tx=0).stop 2024-09-17T15:50:37.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 10 monclient: shutdown 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 msgr2=0x7fd7181359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd70801cc00 tx=0x7fd70801aba0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 shutdown_connections 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd70403bb60 0x7fd70403e020 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd71813aff0 0x7fd71811c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7180694c0 0x7fd7181359f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 --2- 172.21.15.86:0/2872385916 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7180689d0 0x7fd71811eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:37.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 >> 172.21.15.86:0/2872385916 conn(0x7fd7180a6440 msgr2=0x7fd7180a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:37.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135985274}) 2024-09-17T15:50:37.835 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1147518267}) 2024-09-17T15:50:37.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 shutdown_connections 2024-09-17T15:50:37.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:37.831+0000 7fd71d2da640 1 -- 172.21.15.86:0/2872385916 wait complete. 2024-09-17T15:50:37.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1135985274}) 2024-09-17T15:50:37.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.831+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2654856964}) 2024-09-17T15:50:37.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308235364}) 2024-09-17T15:50:37.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385721193}) 2024-09-17T15:50:37.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308235364}) 2024-09-17T15:50:37.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284356454}) 2024-09-17T15:50:37.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545725406}) 2024-09-17T15:50:37.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2840066382}) 2024-09-17T15:50:37.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545725406}) 2024-09-17T15:50:37.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.835+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=394171304}) 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716800913}) 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509818555}) 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716800913}) 2024-09-17T15:50:37.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318012318}) 2024-09-17T15:50:37.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323094179}) 2024-09-17T15:50:37.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285769514}) 2024-09-17T15:50:37.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323094179}) 2024-09-17T15:50:37.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.839+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2560978863}) 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910717149}) 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3152321889}) 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910717149}) 2024-09-17T15:50:37.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1956257458}) 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010527668}) 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1796156352}) 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2010527668}) 2024-09-17T15:50:37.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.843+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582542484}) 2024-09-17T15:50:37.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226272755}) 2024-09-17T15:50:37.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902398720}) 2024-09-17T15:50:37.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226272755}) 2024-09-17T15:50:37.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439433218}) 2024-09-17T15:50:37.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916519051}) 2024-09-17T15:50:37.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1637950386}) 2024-09-17T15:50:37.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916519051}) 2024-09-17T15:50:37.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.847+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052101564}) 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029081113}) 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4228685310}) 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029081113}) 2024-09-17T15:50:37.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106486462}) 2024-09-17T15:50:37.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685100343}) 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631683531}) 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685100343}) 2024-09-17T15:50:37.854 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770347343}) 2024-09-17T15:50:37.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947376794}) 2024-09-17T15:50:37.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523860355}) 2024-09-17T15:50:37.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947376794}) 2024-09-17T15:50:37.856 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.851+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985842937}) 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597486834}) 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481305859}) 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2597486834}) 2024-09-17T15:50:37.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463780251}) 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403684361}) 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491552632}) 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403684361}) 2024-09-17T15:50:37.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.855+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007585256}) 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696342141}) 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3614870759}) 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696342141}) 2024-09-17T15:50:37.872 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.871+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=543281722}) 2024-09-17T15:50:37.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260058863}) 2024-09-17T15:50:37.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927750077}) 2024-09-17T15:50:37.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=260058863}) 2024-09-17T15:50:37.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.875+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=78419524}) 2024-09-17T15:50:37.878 INFO:tasks.rados.rados.0.smithi203.stdout:3790: done (2 left) 2024-09-17T15:50:37.878 INFO:tasks.rados.rados.0.smithi203.stdout:3791: done (1 left) 2024-09-17T15:50:37.878 INFO:tasks.rados.rados.0.smithi203.stdout:3792: done (0 left) 2024-09-17T15:50:37.878 INFO:tasks.rados.rados.0.smithi203.stdout:3793: snap_remove snap 394 2024-09-17T15:50:37.956 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746966112}) 2024-09-17T15:50:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548018745}) 2024-09-17T15:50:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746966112}) 2024-09-17T15:50:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.955+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2676363988}) 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.963+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.963+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484890730}) 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.963+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333545008}) 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.963+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.967+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1484890730}) 2024-09-17T15:50:37.968 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:37.967+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876139409}) 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103953674}) 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198198317}) 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103953674}) 2024-09-17T15:50:38.142 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397591976}) 2024-09-17T15:50:38.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565912815}) 2024-09-17T15:50:38.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=716089911}) 2024-09-17T15:50:38.144 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565912815}) 2024-09-17T15:50:38.145 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3104669325}) 2024-09-17T15:50:38.146 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160284791}) 2024-09-17T15:50:38.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2898485797}) 2024-09-17T15:50:38.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3160284791}) 2024-09-17T15:50:38.147 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.143+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846961473}) 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186700008}) 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=533776540}) 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186700008}) 2024-09-17T15:50:38.149 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507206646}) 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245201556}) 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512551988}) 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4245201556}) 2024-09-17T15:50:38.151 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.147+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1139799696}) 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048207217}) 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450221404}) 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3048207217}) 2024-09-17T15:50:38.153 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730232776}) 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666026423}) 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250167014}) 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3666026423}) 2024-09-17T15:50:38.155 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.151+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403491692}) 2024-09-17T15:50:38.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839258454}) 2024-09-17T15:50:38.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546634393}) 2024-09-17T15:50:38.157 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839258454}) 2024-09-17T15:50:38.158 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318131708}) 2024-09-17T15:50:38.159 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20508350}) 2024-09-17T15:50:38.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283903502}) 2024-09-17T15:50:38.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20508350}) 2024-09-17T15:50:38.160 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.155+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466700975}) 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226263226}) 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688877843}) 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226263226}) 2024-09-17T15:50:38.162 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.163+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946859853}) 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993007918}) 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961745964}) 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993007918}) 2024-09-17T15:50:38.164 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273614118}) 2024-09-17T15:50:38.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149961096}) 2024-09-17T15:50:38.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260548521}) 2024-09-17T15:50:38.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.166 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149961096}) 2024-09-17T15:50:38.167 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808050046}) 2024-09-17T15:50:38.168 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094483467}) 2024-09-17T15:50:38.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3636754575}) 2024-09-17T15:50:38.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094483467}) 2024-09-17T15:50:38.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365737224}) 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725418134}) 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188228228}) 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725418134}) 2024-09-17T15:50:38.171 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.167+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1646251149}) 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592020196}) 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476588761}) 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3592020196}) 2024-09-17T15:50:38.173 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.175+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359864625}) 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159925695}) 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002289407}) 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159925695}) 2024-09-17T15:50:38.585 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663075571}) 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242405408}) 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1932784477}) 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242405408}) 2024-09-17T15:50:38.587 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.587+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513800305}) 2024-09-17T15:50:38.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500338684}) 2024-09-17T15:50:38.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995704137}) 2024-09-17T15:50:38.589 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500338684}) 2024-09-17T15:50:38.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315653793}) 2024-09-17T15:50:38.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933009444}) 2024-09-17T15:50:38.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747937183}) 2024-09-17T15:50:38.591 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=933009444}) 2024-09-17T15:50:38.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.591+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3890262291}) 2024-09-17T15:50:38.593 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663645199}) 2024-09-17T15:50:38.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992989067}) 2024-09-17T15:50:38.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663645199}) 2024-09-17T15:50:38.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475729774}) 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.595+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218087460}) 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3376084457}) 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2218087460}) 2024-09-17T15:50:38.596 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2949810666}) 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611048371}) 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369349402}) 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611048371}) 2024-09-17T15:50:38.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.599+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=544566039}) 2024-09-17T15:50:38.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630915189}) 2024-09-17T15:50:38.600 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662186431}) 2024-09-17T15:50:38.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630915189}) 2024-09-17T15:50:38.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2377502792}) 2024-09-17T15:50:38.602 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857325821}) 2024-09-17T15:50:38.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610952638}) 2024-09-17T15:50:38.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857325821}) 2024-09-17T15:50:38.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.603+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606790152}) 2024-09-17T15:50:38.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934628290}) 2024-09-17T15:50:38.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=85929463}) 2024-09-17T15:50:38.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1934628290}) 2024-09-17T15:50:38.605 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.607+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533729390}) 2024-09-17T15:50:38.855 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:50:38.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:38.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728463427}) 2024-09-17T15:50:38.876 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211619959}) 2024-09-17T15:50:38.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728463427}) 2024-09-17T15:50:38.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.879+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976229859}) 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217885790}) 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3102019753}) 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217885790}) 2024-09-17T15:50:38.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:38.883+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721601274}) 2024-09-17T15:50:38.885 INFO:tasks.rados.rados.0.smithi203.stdout:3793: done (0 left) 2024-09-17T15:50:38.886 INFO:tasks.rados.rados.0.smithi203.stdout:3794: read oid 35 snap -1 2024-09-17T15:50:38.886 INFO:tasks.rados.rados.0.smithi203.stdout:3794: expect (ObjNum 1296 snap 399 seq_num 1296) 2024-09-17T15:50:38.886 INFO:tasks.rados.rados.0.smithi203.stdout:3795: delete oid 16 current snap is 407 2024-09-17T15:50:38.903 INFO:tasks.rados.rados.0.smithi203.stdout:3794: expect (ObjNum 1296 snap 399 seq_num 1296) 2024-09-17T15:50:38.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: get_monmap_and_config 2024-09-17T15:50:38.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: build_initial_monmap 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: monmap: 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:38.993587+0000 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:38.993587+0000 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:38.990 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:38.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 Processor -- start 2024-09-17T15:50:38.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 -- start start 2024-09-17T15:50:38.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: init 2024-09-17T15:50:38.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e300689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e300694c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6e3006f840 0x7f6e3013e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-a con 0x7f6e3006f840 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e30074470 con 0x7f6e300689d0 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e3009c8b0 con 0x7f6e300694c0 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6e3009cb90 con 0x7f6e3006f840 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e35f40640 10 monclient(hunting): authenticate will time out at 2371.959193s 2024-09-17T15:50:38.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6e300689d0 0x7f6e30068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45776/0 (socket says 172.21.15.86:45776) 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2f7fe640 1 -- 172.21.15.86:0/77874341 learned_addr learned my addr 172.21.15.86:0/77874341 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 1 --2- 172.21.15.86:0/77874341 >> v2:172.21.15.86:3300/0 conn(0x7f6e3006f840 0x7f6e3013e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): get_auth_request con 0x7f6e3006f840 auth_method 0 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_done global_id 234447 payload 274 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 1 -- 172.21.15.86:0/77874341 >> v1:172.21.15.203:6789/0 conn(0x7f6e300694c0 legacy=0x7f6e3013aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 1 -- 172.21.15.86:0/77874341 >> v1:172.21.15.86:6789/0 conn(0x7f6e300689d0 legacy=0x7f6e30068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:38.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 1 monclient: found mon.noname-a 2024-09-17T15:50:38.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:38.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:38.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.991+0000 7f6e2ffff640 1 -- 172.21.15.86:0/77874341 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3009ce70 con 0x7f6e3006f840 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.995+0000 7f6e2ffff640 1 -- 172.21.15.86:0/77874341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e3006f840 msgr2=0x7f6e3013e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.995+0000 7f6e2ffff640 1 --2- 172.21.15.86:0/77874341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e3006f840 0x7f6e3013e720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e280097c0 tx=0x7f6e2801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c88f09b52ce9ff server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 1 -- 172.21.15.86:0/77874341 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e280b0070 con 0x7f6e3006f840 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: dump: 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 1 -- 172.21.15.86:0/77874341 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e280ab600 con 0x7f6e3006f840 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 5 monclient: authenticate success, global_id 234447 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:38.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 1 -- 172.21.15.86:0/77874341 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3009c2d0 con 0x7f6e3006f840 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2e7fc640 10 monclient: dump: 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:38.998 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:38.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:38.999 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 4 set_mon_vals no callback set 2024-09-17T15:50:39.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: shutdown 2024-09-17T15:50:39.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- 172.21.15.86:0/77874341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e3006f840 msgr2=0x7f6e3013e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 --2- 172.21.15.86:0/77874341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e3006f840 0x7f6e3013e720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e280097c0 tx=0x7f6e2801c360 comp rx=0 tx=0).stop 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- 172.21.15.86:0/77874341 shutdown_connections 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 --2- 172.21.15.86:0/77874341 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e3006f840 0x7f6e3013e720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- 172.21.15.86:0/77874341 >> 172.21.15.86:0/77874341 conn(0x7f6e300a6440 msgr2=0x7f6e300a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- 172.21.15.86:0/77874341 shutdown_connections 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- 172.21.15.86:0/77874341 wait complete. 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: monmap: 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:39.004 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 Processor -- start 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- start start 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: init 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:50:39.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300689d0 0x7f6e30138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-b con 0x7f6e300689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-a con 0x7f6e300694c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): picked mon.noname-c con 0x7f6e3006f840 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6e301418a0 con 0x7f6e300694c0 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e30141720 con 0x7f6e300689d0 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6e30141a20 con 0x7f6e3006f840 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e35f40640 10 monclient(hunting): authenticate will time out at 2371.966216s 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47902/0 (socket says 172.21.15.86:47902) 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53174/0 (socket says 172.21.15.86:53174) 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 1 -- 172.21.15.86:0/3793370020 learned_addr learned my addr 172.21.15.86:0/3793370020 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): get_auth_request con 0x7f6e3006f840 auth_method 0 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): get_auth_request con 0x7f6e300694c0 auth_method 0 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2f7fe640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300689d0 0x7f6e30138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2f7fe640 10 monclient(hunting): get_auth_request con 0x7f6e300689d0 auth_method 0 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient(hunting): handle_auth_done global_id 234450 payload 824 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 -- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 msgr2=0x7f6e30139530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 -- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300689d0 msgr2=0x7f6e30138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300689d0 0x7f6e30138ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2effd640 1 -- 172.21.15.86:0/3793370020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3013c5f0 con 0x7f6e300694c0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:38.999+0000 7f6e2ffff640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e2effd640 1 -- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 msgr2=0x7f6e30138ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e2effd640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c700 tx=0x7f6e2000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2001e070 con 0x7f6e300694c0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: dump: 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 5 monclient: authenticate success, global_id 234450 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _renew_subs 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e2000ac60 con 0x7f6e300694c0 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3013c8d0 con 0x7f6e300694c0 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e200117a0 con 0x7f6e300694c0 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 10 monclient: dump: 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.011 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e2dffb640 4 set_mon_vals no callback set 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _renew_subs 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e3013b0d0 con 0x7f6e300694c0 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e30068db0 con 0x7f6e300694c0 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6e20021070 con 0x7f6e300694c0 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 0x7f6e1003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(829..829 src has 794..829) ==== 4831+0+0 (secure 0 0 0) 0x7f6e20053c00 con 0x7f6e300694c0 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e2f7fe640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 0x7f6e1003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.003+0000 7f6e2f7fe640 10 monclient: get_auth_request con 0x7f6e1003bb60 auth_method 0 2024-09-17T15:50:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.007+0000 7f6e2f7fe640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 0x7f6e1003e020 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6e18007fd0 tx=0x7f6e18009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.007+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6e20017c60 con 0x7f6e300694c0 2024-09-17T15:50:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.007+0000 7f6e27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.007+0000 7f6e27fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:39.050 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 199366 2024-09-17T15:50:39.113 INFO:tasks.rados.rados.0.smithi203.stdout:3794: done (1 left) 2024-09-17T15:50:39.113 INFO:tasks.rados.rados.0.smithi203.stdout:3795: done (0 left) 2024-09-17T15:50:39.113 INFO:tasks.rados.rados.0.smithi203.stdout:3796: delete oid 19 current snap is 407 2024-09-17T15:50:39.119 INFO:tasks.rados.rados.0.smithi203.stdout:3796: done (0 left) 2024-09-17T15:50:39.119 INFO:tasks.rados.rados.0.smithi203.stdout:3797: read oid 34 snap -1 2024-09-17T15:50:39.119 INFO:tasks.rados.rados.0.smithi203.stdout:3797: expect (ObjNum 1323 snap 404 seq_num 1323) 2024-09-17T15:50:39.120 INFO:tasks.rados.rados.0.smithi203.stdout:3798: snap_remove snap 383 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127417636}) 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=265212880}) 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3127417636}) 2024-09-17T15:50:39.169 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.171+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973865878}) 2024-09-17T15:50:39.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999142660}) 2024-09-17T15:50:39.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=83476913}) 2024-09-17T15:50:39.179 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.179 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999142660}) 2024-09-17T15:50:39.179 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.179+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111806682}) 2024-09-17T15:50:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e35f40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:50:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e35f40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e35f40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6e3013a570 con 0x7f6e300694c0 2024-09-17T15:50:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e27fff640 1 -- 172.21.15.86:0/3793370020 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6e2000ce10 con 0x7f6e300694c0 2024-09-17T15:50:39.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:50:39.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.247+0000 7f6e27fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":212,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 msgr2=0x7f6e1003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 0x7f6e1003e020 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6e18007fd0 tx=0x7f6e18009040 comp rx=0 tx=0).stop 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 10 monclient: shutdown 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 msgr2=0x7f6e30138ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e2000c700 tx=0x7f6e2000cbc0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 shutdown_connections 2024-09-17T15:50:39.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6e1003bb60 0x7f6e1003e020 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e3006f840 0x7f6e30139530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e300694c0 0x7f6e30138ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 --2- 172.21.15.86:0/3793370020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300689d0 0x7f6e30138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 >> 172.21.15.86:0/3793370020 conn(0x7f6e300a6440 msgr2=0x7f6e30070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 shutdown_connections 2024-09-17T15:50:39.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.251+0000 7f6e35f40640 1 -- 172.21.15.86:0/3793370020 wait complete. 2024-09-17T15:50:39.267 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:50:39.267 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:50:39.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:50:39.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.315+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.319+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081074348}) 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.319+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=458691700}) 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.319+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.319+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081074348}) 2024-09-17T15:50:39.316 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.319+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530051736}) 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: get_monmap_and_config 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: monmap: 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:39.430172+0000 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:39.430172+0000 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.426 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:39.427 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:39.427 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:39.427 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:39.427 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:39.427 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 Processor -- start 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 -- start start 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: init 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72200a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f72200a5350 0x7f72200a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f72200a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f722013d180 0x7f722013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f722013d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72200a68d0 con 0x7f72200a4f70 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f72200a64a0 con 0x7f72200a5350 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f722009c230 con 0x7f722013d180 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7224ef4640 10 monclient(hunting): authenticate will time out at 2372.395657s 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f72200a5350 0x7f72200a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f72200a5350 0x7f72200a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53186/0 (socket says 172.21.15.86:53186) 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 1 -- 172.21.15.86:0/4160101854 learned_addr learned my addr 172.21.15.86:0/4160101854 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): get_auth_request con 0x7f72200a5350 auth_method 0 2024-09-17T15:50:39.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.427+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f721ed9d640 1 --2- 172.21.15.86:0/4160101854 >> v2:172.21.15.203:3300/0 conn(0x7f722013d180 0x7f722013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f721ed9d640 10 monclient(hunting): get_auth_request con 0x7f722013d180 auth_method 0 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f721ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f721ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f721ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 10 monclient(hunting): handle_auth_done global_id 234453 payload 274 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 1 -- 172.21.15.86:0/4160101854 >> v2:172.21.15.203:3300/0 conn(0x7f722013d180 msgr2=0x7f722013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 1 --2- 172.21.15.86:0/4160101854 >> v2:172.21.15.203:3300/0 conn(0x7f722013d180 0x7f722013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 1 -- 172.21.15.86:0/4160101854 >> v1:172.21.15.203:6789/0 conn(0x7f72200a4f70 legacy=0x7f722009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.431+0000 7f7217fff640 1 -- 172.21.15.86:0/4160101854 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722009c510 con 0x7f72200a5350 2024-09-17T15:50:39.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 -- 172.21.15.86:0/4160101854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 msgr2=0x7f72200a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 --2- 172.21.15.86:0/4160101854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a5750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72200a6380 tx=0x7f720801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1842abe3a13bb672 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 1 -- 172.21.15.86:0/4160101854 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080b1070 con 0x7f72200a5350 2024-09-17T15:50:39.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: dump: 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 1 -- 172.21.15.86:0/4160101854 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72080ac620 con 0x7f72200a5350 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 5 monclient: authenticate success, global_id 234453 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 4 set_mon_vals no callback set 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 1 -- 172.21.15.86:0/4160101854 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080ac9f0 con 0x7f72200a5350 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721dd9b640 10 monclient: dump: 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: shutdown 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- 172.21.15.86:0/4160101854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 msgr2=0x7f72200a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- 172.21.15.86:0/4160101854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a5750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f72200a6380 tx=0x7f720801c3e0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- 172.21.15.86:0/4160101854 shutdown_connections 2024-09-17T15:50:39.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- 172.21.15.86:0/4160101854 >> v2:172.21.15.203:3300/0 conn(0x7f722013d180 0x7f722013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- 172.21.15.86:0/4160101854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a5750 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- 172.21.15.86:0/4160101854 >> 172.21.15.86:0/4160101854 conn(0x7f72200a7240 msgr2=0x7f72200a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- 172.21.15.86:0/4160101854 shutdown_connections 2024-09-17T15:50:39.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- 172.21.15.86:0/4160101854 wait complete. 2024-09-17T15:50:39.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: monmap: 2024-09-17T15:50:39.441 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 Processor -- start 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- start start 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: init 2024-09-17T15:50:39.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72200a4f70 0x7f72201215e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-b con 0x7f72200a4f70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-a con 0x7f72200a5350 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f722013d180 0x7f7220138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): picked mon.noname-c con 0x7f722013d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7220142660 con 0x7f72200a5350 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72201424e0 con 0x7f72200a4f70 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f72201427e0 con 0x7f722013d180 2024-09-17T15:50:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7224ef4640 10 monclient(hunting): authenticate will time out at 2372.403284s 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53196/0 (socket says 172.21.15.86:53196) 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 -- 172.21.15.86:0/3626157596 learned_addr learned my addr 172.21.15.86:0/3626157596 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): get_auth_request con 0x7f72200a5350 auth_method 0 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f722013d180 0x7f7220138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): get_auth_request con 0x7f722013d180 auth_method 0 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient(hunting): handle_auth_done global_id 234456 payload 824 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f722013d180 msgr2=0x7f7220138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72200a4f70 0x7f72201215e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f722013d180 0x7f7220138450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72200a4f70 msgr2=0x7f72201215e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72200a4f70 0x7f72201215e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f7217fff640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722013d830 con 0x7f72200a5350 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721e59c640 10 monclient: get_auth_request con 0x7f72200a4f70 auth_method 0 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.435+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72200a4f70 0x7f72201215e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7217fff640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 msgr2=0x7f72200a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7217fff640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a3430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f720801a010 tx=0x7f720801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080bb020 con 0x7f72200a5350 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: dump: 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.447 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72080b7420 con 0x7f72200a5350 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72080b1040 con 0x7f72200a5350 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 10 monclient: dump: 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.448 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 5 monclient: authenticate success, global_id 234456 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721cd99640 4 set_mon_vals no callback set 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 monclient: _renew_subs 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72200a1760 con 0x7f72200a5350 2024-09-17T15:50:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 monclient: _renew_subs 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7220122540 con 0x7f72200a5350 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 0x7f72201cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f72200a5750 con 0x7f7220138990 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 0x7f72201cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient: get_auth_request con 0x7f7220138990 auth_method 0 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f72080be020 con 0x7f72200a5350 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 10 monclient(hunting): handle_auth_done global_id 234459 payload 824 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f721e59c640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72200a5ff0 con 0x7f7220138990 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f720003bbb0 0x7f720003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.439+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(829..829 src has 794..829) ==== 4831+0+0 (secure 0 0 0) 0x7f72080ea080 con 0x7f72200a5350 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f721ed9d640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f720003bbb0 0x7f720003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.451 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f721ed9d640 10 monclient: get_auth_request con 0x7f720003bbb0 auth_method 0 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097844325}) 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899530917}) 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097844325}) 2024-09-17T15:50:39.452 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406666765}) 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f721e59c640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 msgr2=0x7f72201cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 0x7f72201cfe30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f721000d480 tx=0x7f721000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210012a10 con 0x7f7220138990 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 10 monclient: dump: 2024-09-17T15:50:39.453 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7210012a10 con 0x7f7220138990 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:39.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 msgr2=0x7f72201cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 0x7f72201cfe30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f721000d480 tx=0x7f721000d940 comp rx=0 tx=0).stop 2024-09-17T15:50:39.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 msgr2=0x7f72201cfe30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f7216ffd640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7220138990 0x7f72201cfe30 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f721000d480 tx=0x7f721000d940 comp rx=0 tx=0).stop 2024-09-17T15:50:39.455 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f721ed9d640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f720003bbb0 0x7f720003e070 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f72200a4b50 tx=0x7f7218009820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.456 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901732087}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991553423}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901732087}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.443+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=444283574}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206958158}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4277555767}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4206958158}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473820641}) 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.457 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819623661}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3408012731}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819623661}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.447+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859185330}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2470495978}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765750459}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2470495978}) 2024-09-17T15:50:39.458 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883539352}) 2024-09-17T15:50:39.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7224ef4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:39.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7224ef4640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 0x7f72201d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.459 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7224ef4640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f72201d0cc0 con 0x7f72201d0370 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 0x7f72201d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient: get_auth_request con 0x7f72201d0370 auth_method 0 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 10 monclient(hunting): handle_auth_done global_id 234462 payload 824 2024-09-17T15:50:39.460 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 1 -- 172.21.15.86:0/3626157596 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7220069920 con 0x7f72201d0370 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854658539}) 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584507586}) 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=854658539}) 2024-09-17T15:50:39.461 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43660561}) 2024-09-17T15:50:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 msgr2=0x7f72201d0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f721e59c640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 0x7f72201d0780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7210017000 tx=0x7f7210003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210028020 con 0x7f72201d0370 2024-09-17T15:50:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7216ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7216ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.451+0000 7f7216ffd640 10 monclient: dump: 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f7210007650 con 0x7f72201d0370 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 msgr2=0x7f72201d0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 0x7f72201d0780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7210017000 tx=0x7f7210003770 comp rx=0 tx=0).stop 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 msgr2=0x7f72201d0780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7216ffd640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72201d0370 0x7f72201d0780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:39.464 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 168647, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:39.465 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:39.466 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.467 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:39.468 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.469 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.471 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:39.474 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.476 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.478 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:39.478 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:39.478 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.478 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967175027}) 2024-09-17T15:50:39.479 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3422703454}) 2024-09-17T15:50:39.479 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.479 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967175027}) 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f720003bbb0 msgr2=0x7f720003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f720003bbb0 0x7f720003e070 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f72200a4b50 tx=0x7f7218009820 comp rx=0 tx=0).stop 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 10 monclient: shutdown 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 msgr2=0x7f72200a3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 --2- 172.21.15.86:0/3626157596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72200a5350 0x7f72200a3430 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f720801a010 tx=0x7f720801ac20 comp rx=0 tx=0).stop 2024-09-17T15:50:39.480 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129419600}) 2024-09-17T15:50:39.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f721ed9d640 1 -- 172.21.15.86:0/3626157596 reap_dead start 2024-09-17T15:50:39.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f721ed9d640 1 -- 172.21.15.86:0/3626157596 reap_dead start 2024-09-17T15:50:39.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 shutdown_connections 2024-09-17T15:50:39.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 >> 172.21.15.86:0/3626157596 conn(0x7f72200a7240 msgr2=0x7f722013c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 shutdown_connections 2024-09-17T15:50:39.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f7224ef4640 1 -- 172.21.15.86:0/3626157596 wait complete. 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082400515}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725721967}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082400515}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.455+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=876376726}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622149598}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923761997}) 2024-09-17T15:50:39.486 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622149598}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595887470}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450716890}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4280563606}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450716890}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4037687107}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273608483}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=412695905}) 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.487 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=273608483}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.459+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2787316172}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418080169}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4123475490}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1418080169}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1973213596}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836302384}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1346080438}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836302384}) 2024-09-17T15:50:39.488 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.463+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016928722}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139924693}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325364637}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139924693}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623223725}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270079300}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065172854}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270079300}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.467+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597233056}) 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.489 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3885965024}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1825492665}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3885965024}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929907516}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175184461}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037497933}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175184461}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534377252}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.471+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034407440}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3885667981}) 2024-09-17T15:50:39.490 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.491 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034407440}) 2024-09-17T15:50:39.491 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2131395544}) 2024-09-17T15:50:39.492 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:39.492 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:50:39.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355582225}) 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727637316}) 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3355582225}) 2024-09-17T15:50:39.498 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3844302064}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296660467}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4033202812}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296660467}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.475+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974291198}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571098009}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670363345}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571098009}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940266077}) 2024-09-17T15:50:39.499 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086677372}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727489316}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086677372}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.479+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=491769780}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227871734}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612553988}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227871734}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3750130400}) 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.500 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550997970}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406535573}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550997970}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.483+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3961084757}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726912383}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204008804}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726912383}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3168449903}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736573358}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277721104}) 2024-09-17T15:50:39.501 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736573358}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.487+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3512142073}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737255801}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3403708448}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1737255801}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466971028}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692526401}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1486843175}) 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.502 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692526401}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3576890733}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753038604}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795179266}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753038604}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.491+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3021681750}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027439016}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=373963438}) 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.503 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027439016}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278282180}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469672931}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394503378}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469672931}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.495+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024777371}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589603106}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853803746}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589603106}) 2024-09-17T15:50:39.504 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790985681}) 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914765716}) 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087214929}) 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914765716}) 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.499+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1688049118}) 2024-09-17T15:50:39.507 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460893292}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4053735716}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3460893292}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=503164329}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112037963}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641791127}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112037963}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.503+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266055306}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666834478}) 2024-09-17T15:50:39.508 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1477228082}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666834478}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959546073}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716643547}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329986722}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=716643547}) 2024-09-17T15:50:39.509 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.507+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1707289342}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486295629}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174289492}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486295629}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2403794824}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191959185}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387744372}) 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.511 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191959185}) 2024-09-17T15:50:39.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708641440}) 2024-09-17T15:50:39.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.512 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714665597}) 2024-09-17T15:50:39.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885089429}) 2024-09-17T15:50:39.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714665597}) 2024-09-17T15:50:39.513 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.511+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1619422406}) 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478426106}) 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325693881}) 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478426106}) 2024-09-17T15:50:39.514 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=882212485}) 2024-09-17T15:50:39.515 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.515 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799618805}) 2024-09-17T15:50:39.515 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53053200}) 2024-09-17T15:50:39.515 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.515 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799618805}) 2024-09-17T15:50:39.516 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.515+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051946912}) 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533353960}) 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377534005}) 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533353960}) 2024-09-17T15:50:39.517 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066845942}) 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856639732}) 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65259092}) 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856639732}) 2024-09-17T15:50:39.519 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.519+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2081427089}) 2024-09-17T15:50:39.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872029621}) 2024-09-17T15:50:39.520 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137272393}) 2024-09-17T15:50:39.521 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.521 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872029621}) 2024-09-17T15:50:39.521 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730098925}) 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326389237}) 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=772445187}) 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326389237}) 2024-09-17T15:50:39.522 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.523+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=845890797}) 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284884426}) 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727196912}) 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284884426}) 2024-09-17T15:50:39.524 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2800505595}) 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616773302}) 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1444960667}) 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616773302}) 2024-09-17T15:50:39.526 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170384381}) 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087290289}) 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=874234066}) 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087290289}) 2024-09-17T15:50:39.528 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.527+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4203345381}) 2024-09-17T15:50:39.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510012827}) 2024-09-17T15:50:39.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046816247}) 2024-09-17T15:50:39.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510012827}) 2024-09-17T15:50:39.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416932382}) 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626700265}) 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=545032983}) 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3626700265}) 2024-09-17T15:50:39.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845742412}) 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441606001}) 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850860547}) 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441606001}) 2024-09-17T15:50:39.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.539+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=698166576}) 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=48} crc {logm=1207647095,mdsmap=2815901553,mgr=333540547,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4224228207,mon_config_key=639531838,monmap=1314516931,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2247246626,osd_snap=3272853051}) 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=47} crc {logm=1207647095,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1765527147,mon_config_key=3643836368,monmap=4191199434,nvmeofgw=598337686,osd_metadata=1259201597,osd_pg_creating=2247246626,osd_snap=2616173325}) 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=48} crc {logm=1207647095,mdsmap=2815901553,mgr=333540547,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=4224228207,mon_config_key=639531838,monmap=1314516931,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2247246626,osd_snap=3272853051}) 2024-09-17T15:50:39.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=2,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=48} crc {logm=1207647095,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3294157872,mon_config_key=2837151273,monmap=4191199434,nvmeofgw=598337686,osd_metadata=4043941930,osd_pg_creating=2247246626,osd_snap=3355866239}) 2024-09-17T15:50:39.543 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=573657667}) 2024-09-17T15:50:39.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=1272420808}) 2024-09-17T15:50:39.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=573657667}) 2024-09-17T15:50:39.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.543+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=64901921}) 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=69} crc {osd_snap=451503433,osdmap=3393235686}) 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=71} crc {osd_snap=30234031,osdmap=1495424811}) 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=3,osdmap=69} crc {osd_snap=451503433,osdmap=3393235686}) 2024-09-17T15:50:39.557 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:50:39.559+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=70} crc {osd_snap=987078162,osdmap=3781379689}) 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: get_monmap_and_config 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: monmap: 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:39.628099+0000 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:39.628099+0000 2024-09-17T15:50:39.624 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:39.625 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 Processor -- start 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 -- start start 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: init 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05900a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05900a5bd0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f059009cd60 0x7f0590140a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): picked mon.noname-c con 0x7f059009cd60 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05900a6280 con 0x7f05900a5bd0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059009c0a0 con 0x7f05900a50e0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f059009c380 con 0x7f059009cd60 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f059737a640 10 monclient(hunting): authenticate will time out at 2372.593516s 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0594915640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f05900a5bd0 0x7f059013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45788/0 (socket says 172.21.15.86:45788) 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0594915640 1 -- 172.21.15.86:0/2152930210 learned_addr learned my addr 172.21.15.86:0/2152930210 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 1 --2- 172.21.15.86:0/2152930210 >> v2:172.21.15.203:3300/0 conn(0x7f059009cd60 0x7f0590140a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): get_auth_request con 0x7f059009cd60 auth_method 0 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient(hunting): handle_auth_done global_id 254203 payload 274 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 1 -- 172.21.15.86:0/2152930210 >> v1:172.21.15.203:6789/0 conn(0x7f05900a50e0 legacy=0x7f05900a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 1 -- 172.21.15.86:0/2152930210 >> v1:172.21.15.86:6789/0 conn(0x7f05900a5bd0 legacy=0x7f059013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 1 monclient: found mon.noname-c 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:50:39.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.627+0000 7f0595917640 1 -- 172.21.15.86:0/2152930210 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059009c660 con 0x7f059009cd60 2024-09-17T15:50:39.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f0595917640 1 -- 172.21.15.86:0/2152930210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f059009cd60 msgr2=0x7f0590140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f0595917640 1 --2- 172.21.15.86:0/2152930210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f059009cd60 0x7f0590140a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05800097c0 tx=0x7f058001c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3372818a7766588 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 1 -- 172.21.15.86:0/2152930210 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800b0070 con 0x7f059009cd60 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.633 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f059737a640 5 monclient: authenticate success, global_id 254203 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f059737a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 1 -- 172.21.15.86:0/2152930210 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05800ab600 con 0x7f059009cd60 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 1 -- 172.21.15.86:0/2152930210 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800a4e60 con 0x7f059009cd60 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.631+0000 7f058ffff640 10 monclient: dump: 2024-09-17T15:50:39.634 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 4 set_mon_vals no callback set 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: shutdown 2024-09-17T15:50:39.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- 172.21.15.86:0/2152930210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f059009cd60 msgr2=0x7f0590140a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 --2- 172.21.15.86:0/2152930210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f059009cd60 0x7f0590140a20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f05800097c0 tx=0x7f058001c330 comp rx=0 tx=0).stop 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- 172.21.15.86:0/2152930210 shutdown_connections 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 --2- 172.21.15.86:0/2152930210 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f059009cd60 0x7f0590140a20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- 172.21.15.86:0/2152930210 >> 172.21.15.86:0/2152930210 conn(0x7f05900aaae0 msgr2=0x7f05900aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- 172.21.15.86:0/2152930210 shutdown_connections 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- 172.21.15.86:0/2152930210 wait complete. 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: monmap: 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:39.636 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:39.637 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 Processor -- start 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- start start 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: init 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 0x7f05901cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): picked mon.noname-a con 0x7f059009cd60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05900a50e0 0x7f059013f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): picked mon.noname-c con 0x7f05900a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05900a5bd0 0x7f059013f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): picked mon.noname-b con 0x7f05900a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0590143900 con 0x7f059009cd60 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0590143780 con 0x7f05900a5bd0 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0590143a80 con 0x7f05900a50e0 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f059737a640 10 monclient(hunting): authenticate will time out at 2372.602211s 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05900a50e0 0x7f059013f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 0x7f05901cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 0x7f05901cfca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53210/0 (socket says 172.21.15.86:53210) 2024-09-17T15:50:39.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 -- 172.21.15.86:0/1274597428 learned_addr learned my addr 172.21.15.86:0/1274597428 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): get_auth_request con 0x7f059009cd60 auth_method 0 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05900a50e0 0x7f059013f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:47916/0 (socket says 172.21.15.86:47916) 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): get_auth_request con 0x7f05900a50e0 auth_method 0 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595917640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05900a5bd0 0x7f059013f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595917640 10 monclient(hunting): get_auth_request con 0x7f05900a5bd0 auth_method 0 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595917640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595917640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient(hunting): handle_auth_done global_id 234465 payload 824 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05900a50e0 msgr2=0x7f059013f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05900a50e0 0x7f059013f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05900a5bd0 msgr2=0x7f059013f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05900a5bd0 0x7f059013f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.635+0000 7f0595116640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059013ab20 con 0x7f059009cd60 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0595116640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 msgr2=0x7f05901cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0595116640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 0x7f05901cfca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f058800c7c0 tx=0x7f058800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588016020 con 0x7f059009cd60 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: dump: 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.646 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058800ac60 con 0x7f059009cd60 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 5 monclient: authenticate success, global_id 234465 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 monclient: _renew_subs 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05900a23c0 con 0x7f059009cd60 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0588012660 con 0x7f059009cd60 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 10 monclient: dump: 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.648 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 monclient: _renew_subs 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059013c5e0 con 0x7f059009cd60 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058f7fe640 4 set_mon_vals no callback set 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 0x7f05901396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0590140320 con 0x7f05901392d0 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 0x7f05901396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient: get_auth_request con 0x7f05901392d0 auth_method 0 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0588019020 con 0x7f059009cd60 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 10 monclient(hunting): handle_auth_done global_id 264140 payload 824 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0594915640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05900a54c0 con 0x7f05901392d0 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f057c03bb90 0x7f057c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(829..829 src has 794..829) ==== 4831+0+0 (secure 0 0 0) 0x7f0588050300 con 0x7f059009cd60 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0595917640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f057c03bb90 0x7f057c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.639+0000 7f0595917640 10 monclient: get_auth_request con 0x7f057c03bb90 auth_method 0 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f0594915640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 msgr2=0x7f05901396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f0594915640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 0x7f05901396b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f058400eaa0 tx=0x7f058400eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058400ee60 con 0x7f05901392d0 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 10 monclient: dump: 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0584019070 con 0x7f05901392d0 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 msgr2=0x7f05901396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 0x7f05901396b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f058400eaa0 tx=0x7f058400eb90 comp rx=0 tx=0).stop 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 msgr2=0x7f05901396b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f058dffb640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05901392d0 0x7f05901396b0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f058400eaa0 tx=0x7f058400eb90 comp rx=0 tx=0).stop 2024-09-17T15:50:39.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.643+0000 7f0595917640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f057c03bb90 0x7f057c03e050 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f059006cc20 tx=0x7f05800a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f059737a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:39.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f059737a640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 0x7f0590135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f059737a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0590074be0 con 0x7f0590139bf0 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 0x7f0590135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient: get_auth_request con 0x7f0590139bf0 auth_method 0 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 10 monclient(hunting): handle_auth_done global_id 264143 payload 824 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.647+0000 7f0594915640 1 -- 172.21.15.86:0/1274597428 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f059013a000 con 0x7f0590139bf0 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f0594915640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 msgr2=0x7f0590135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f0594915640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 0x7f0590135110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05840070c0 tx=0x7f0584004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058401ea40 con 0x7f0590139bf0 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 10 monclient: dump: 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f0584014480 con 0x7f0590139bf0 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 msgr2=0x7f0590135110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 0x7f0590135110 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05840070c0 tx=0x7f0584004950 comp rx=0 tx=0).stop 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 msgr2=0x7f0590135110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f058dffb640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0590139bf0 0x7f0590135110 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05840070c0 tx=0x7f0584004950 comp rx=0 tx=0).stop 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:39.657 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 9819, 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.658 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:39.659 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.660 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:39.661 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.662 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:39.663 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.665 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.666 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.669 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.670 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:39.671 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.671 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.671 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:39.671 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f057c03bb90 msgr2=0x7f057c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f057c03bb90 0x7f057c03e050 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f059006cc20 tx=0x7f05800a6000 comp rx=0 tx=0).stop 2024-09-17T15:50:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 10 monclient: shutdown 2024-09-17T15:50:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 msgr2=0x7f05901cfca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 --2- 172.21.15.86:0/1274597428 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f059009cd60 0x7f05901cfca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f058800c7c0 tx=0x7f058800cc80 comp rx=0 tx=0).stop 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f0595917640 1 -- 172.21.15.86:0/1274597428 reap_dead start 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f0595917640 1 -- 172.21.15.86:0/1274597428 reap_dead start 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 shutdown_connections 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 >> 172.21.15.86:0/1274597428 conn(0x7f05900aaae0 msgr2=0x7f05900a3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 shutdown_connections 2024-09-17T15:50:39.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.651+0000 7f059737a640 1 -- 172.21.15.86:0/1274597428 wait complete. 2024-09-17T15:50:39.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:39.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:50:39.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:50:39.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.839+0000 7f0b871f6640 10 monclient: get_monmap_and_config 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.839+0000 7f0b871f6640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.839+0000 7f0b871f6640 10 monclient: monmap: 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:39.843623+0000 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:39.843623+0000 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:39.840 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:39.841 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 Processor -- start 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 -- start start 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient: init 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b800a0c70 0x7f0b800a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b800a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0b800a1590 0x7f0b80135a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b800a1590 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:39.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 0x7f0b80136420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b80135f90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0b800a2410 con 0x7f0b800a0c70 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0b800a2130 con 0x7f0b800a1590 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0b800a26f0 con 0x7f0b80135f90 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b871f6640 10 monclient(hunting): authenticate will time out at 2372.809076s 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b800a0c70 0x7f0b800a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 0x7f0b80136420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b800a0c70 0x7f0b800a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53216/0 (socket says 172.21.15.86:53216) 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 -- 172.21.15.86:0/3262367648 learned_addr learned my addr 172.21.15.86:0/3262367648 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): get_auth_request con 0x7f0b80135f90 auth_method 0 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): get_auth_request con 0x7f0b800a0c70 auth_method 0 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b7ffff640 1 --2- 172.21.15.86:0/3262367648 >> v2:172.21.15.203:3300/0 conn(0x7f0b800a1590 0x7f0b80135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request con 0x7f0b800a1590 auth_method 0 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient(hunting): handle_auth_done global_id 234468 payload 274 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 -- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 msgr2=0x7f0b80136420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 0x7f0b80136420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 -- 172.21.15.86:0/3262367648 >> v2:172.21.15.203:3300/0 conn(0x7f0b800a1590 msgr2=0x7f0b80135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3262367648 >> v2:172.21.15.203:3300/0 conn(0x7f0b800a1590 0x7f0b80135a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b84f92640 1 -- 172.21.15.86:0/3262367648 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8009f9f0 con 0x7f0b800a0c70 2024-09-17T15:50:39.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.843+0000 7f0b85793640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 0x7f0b80136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b84f92640 1 -- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 msgr2=0x7f0b800a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b800a1050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b80074ae0 tx=0x7f0b7401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c41b13c28ee117 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 1 -- 172.21.15.86:0/3262367648 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740b1070 con 0x7f0b800a0c70 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: dump: 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.854 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 1 -- 172.21.15.86:0/3262367648 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b740ac620 con 0x7f0b800a0c70 2024-09-17T15:50:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 5 monclient: authenticate success, global_id 234468 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 1 -- 172.21.15.86:0/3262367648 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740ac9f0 con 0x7f0b800a0c70 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 4 set_mon_vals no callback set 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b7f7fe640 10 monclient: dump: 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.856 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 10 monclient: shutdown 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 -- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 msgr2=0x7f0b800a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b800a1050 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0b80074ae0 tx=0x7f0b7401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 -- 172.21.15.86:0/3262367648 shutdown_connections 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b80135f90 0x7f0b80136420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3262367648 >> v2:172.21.15.203:3300/0 conn(0x7f0b800a1590 0x7f0b80135a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3262367648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b800a1050 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 -- 172.21.15.86:0/3262367648 >> 172.21.15.86:0/3262367648 conn(0x7f0b800a2f40 msgr2=0x7f0b800a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 -- 172.21.15.86:0/3262367648 shutdown_connections 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 1 -- 172.21.15.86:0/3262367648 wait complete. 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 10 monclient: build_initial_monmap 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.847+0000 7f0b871f6640 10 monclient: monmap: 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:50:39.858 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 Processor -- start 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- start start 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: init 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b801cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b800a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b800a1590 0x7f0b80135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b800a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135f90 0x7f0b801cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b80135f90 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800a00d0 con 0x7f0b800a0c70 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b8009ff50 con 0x7f0b80135f90 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0b8013f740 con 0x7f0b800a1590 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient(hunting): authenticate will time out at 2372.816915s 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b800a1590 0x7f0b80135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.860 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b801cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b801cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:53222/0 (socket says 172.21.15.86:53222) 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 -- 172.21.15.86:0/3213613707 learned_addr learned my addr 172.21.15.86:0/3213613707 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request con 0x7f0b800a1590 auth_method 0 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): get_auth_request con 0x7f0b800a0c70 auth_method 0 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.861 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b85793640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135f90 0x7f0b801cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b85793640 10 monclient(hunting): get_auth_request con 0x7f0b80135f90 auth_method 0 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b85793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b85793640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b85793640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient(hunting): handle_auth_done global_id 234471 payload 824 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b800a1590 msgr2=0x7f0b80135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b800a1590 0x7f0b80135140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135f90 msgr2=0x7f0b801cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135f90 0x7f0b801cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient: _finish_hunting 0 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 monclient: found mon.noname-a 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b80122b00 con 0x7f0b800a0c70 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 msgr2=0x7f0b801cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b84f92640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b801cc850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b7401aa90 tx=0x7f0b74019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740b1040 con 0x7f0b800a0c70 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: dump: 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.863 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b740ba070 con 0x7f0b800a0c70 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 5 monclient: authenticate success, global_id 234471 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _renew_subs 2024-09-17T15:50:39.864 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8009ec40 con 0x7f0b800a0c70 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740b5440 con 0x7f0b800a0c70 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7d7fa640 10 monclient: dump: 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.865 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _renew_subs 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b7effd640 4 set_mon_vals no callback set 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.851+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b80074b50 con 0x7f0b800a0c70 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b871f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 0x7f0b801d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80135a60 con 0x7f0b80135680 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0b740b5600 con 0x7f0b800a0c70 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7d7fa640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6403bb60 0x7f0b6403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(829..829 src has 794..829) ==== 4831+0+0 (secure 0 0 0) 0x7f0b740e99e0 con 0x7f0b800a0c70 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 0x7f0b801d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient: get_auth_request con 0x7f0b80135680 auth_method 0 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b85793640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6403bb60 0x7f0b6403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b85793640 10 monclient: get_auth_request con 0x7f0b6403bb60 auth_method 0 2024-09-17T15:50:39.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_done global_id 254215 payload 824 2024-09-17T15:50:39.868 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.855+0000 7f0b7ffff640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b8009c860 con 0x7f0b80135680 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7ffff640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 msgr2=0x7f0b801d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7ffff640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 0x7f0b801d02f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c018020 con 0x7f0b80135680 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b85793640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6403bb60 0x7f0b6403e020 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0b78007580 tx=0x7f0b7800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 10 monclient: dump: 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0b6c004450 con 0x7f0b80135680 2024-09-17T15:50:39.871 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 msgr2=0x7f0b801d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 0x7f0b801d02f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c010c80 comp rx=0 tx=0).stop 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 msgr2=0x7f0b801d02f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.859+0000 7f0b7d7fa640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b80135680 0x7f0b801d02f0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c010c80 comp rx=0 tx=0).stop 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b871f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 0x7f0b80071490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b871f6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0b801d0c40 con 0x7f0b801d0830 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 0x7f0b80071490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient: get_auth_request con 0x7f0b801d0830 auth_method 0 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:39.872 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.863+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7ffff640 10 monclient(hunting): handle_auth_done global_id 254218 payload 824 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7ffff640 1 -- 172.21.15.86:0/3213613707 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0b801216b0 con 0x7f0b801d0830 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7ffff640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 msgr2=0x7f0b80071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7ffff640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 0x7f0b80071490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c018070 con 0x7f0b801d0830 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 10 monclient: dump: 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:39.873 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f0b6c00ecb0 con 0x7f0b801d0830 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 msgr2=0x7f0b80071490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 0x7f0b80071490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.874 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 msgr2=0x7f0b80071490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b7d7fa640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b801d0830 0x7f0b80071490 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b80121290 tx=0x7f0b6c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 212, 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 55817, 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:50:39.875 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 5, 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.876 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.877 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:50:39.878 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:50:39.879 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:50:39.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:50:39.881 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.882 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.883 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:50:39.884 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:50:39.885 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:50:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:50:39.887 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:50:39.888 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:50:39.889 INFO:tasks.rados.rados.0.smithi203.stdout:3798: done (1 left) 2024-09-17T15:50:39.889 INFO:tasks.rados.rados.0.smithi203.stdout:3799: rollback oid 31 current snap is 407 2024-09-17T15:50:39.889 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 31 to 401 2024-09-17T15:50:39.890 INFO:tasks.rados.rados.0.smithi203.stdout:3797: expect (ObjNum 1323 snap 404 seq_num 1323) 2024-09-17T15:50:39.890 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6403bb60 msgr2=0x7f0b6403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0b6403bb60 0x7f0b6403e020 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0b78007580 tx=0x7f0b7800b040 comp rx=0 tx=0).stop 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 10 monclient: shutdown 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 msgr2=0x7f0b801cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:39.890 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 --2- 172.21.15.86:0/3213613707 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b800a0c70 0x7f0b801cc850 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0b7401aa90 tx=0x7f0b74019180 comp rx=0 tx=0).stop 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b85793640 1 -- 172.21.15.86:0/3213613707 reap_dead start 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b85793640 1 -- 172.21.15.86:0/3213613707 reap_dead start 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 shutdown_connections 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.867+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 >> 172.21.15.86:0/3213613707 conn(0x7f0b800a2f40 msgr2=0x7f0b80070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.871+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 shutdown_connections 2024-09-17T15:50:39.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:39.871+0000 7f0b871f6640 1 -- 172.21.15.86:0/3213613707 wait complete. 2024-09-17T15:50:39.891 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:50:39.891 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-17T15:50:39.892 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-17T15:50:39.892 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-17T15:50:39.892 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-17T15:50:39.892 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:50:39.892 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:50:39.906 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:50:39.906 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:50:39.907 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:50:39.907 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:50:40.004 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:3799: finishing rollback tid 0 to smithi20328086-31 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:3799: finishing rollback tid 1 to smithi20328086-31 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:3799: finishing rollback tid 2 to smithi20328086-31 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1679 (ObjNum 1266 snap 385 seq_num 1266) dirty exists 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:3797: done (1 left) 2024-09-17T15:50:40.059 INFO:tasks.rados.rados.0.smithi203.stdout:3799: done (0 left) 2024-09-17T15:50:40.060 INFO:tasks.rados.rados.0.smithi203.stdout:3800: read oid 6 snap -1 2024-09-17T15:50:40.060 INFO:tasks.rados.rados.0.smithi203.stdout:3800: expect (ObjNum 1325 snap 405 seq_num 1325) 2024-09-17T15:50:40.060 INFO:tasks.rados.rados.0.smithi203.stdout:3801: delete oid 19 current snap is 407 2024-09-17T15:50:40.060 INFO:tasks.rados.rados.0.smithi203.stdout:3800: expect (ObjNum 1325 snap 405 seq_num 1325) 2024-09-17T15:50:40.077 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 886846 2024-09-17T15:50:40.104 INFO:tasks.rados.rados.0.smithi203.stdout:3800: done (1 left) 2024-09-17T15:50:40.105 INFO:tasks.rados.rados.0.smithi203.stdout:3801: done (0 left) 2024-09-17T15:50:40.105 INFO:tasks.rados.rados.0.smithi203.stdout:3802: write (excl) oid 24 current snap is 407 2024-09-17T15:50:40.105 INFO:tasks.rados.rados.0.smithi203.stdout:3802: seq_num 1336 ranges {454152=494439,1646296=735124} 2024-09-17T15:50:40.110 INFO:tasks.rados.rados.0.smithi203.stdout:3802: writing smithi20328086-24 from 454152 to 948591 tid 1 2024-09-17T15:50:40.120 INFO:tasks.rados.rados.0.smithi203.stdout:3802: writing smithi20328086-24 from 1646296 to 2381420 tid 2 2024-09-17T15:50:40.120 INFO:tasks.rados.rados.0.smithi203.stdout:3803: rollback oid 2 current snap is 407 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 2 to 400 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:3804: rollback oid 15 current snap is 407 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 15 to 402 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:3805: rollback oid 45 current snap is 407 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 400 2024-09-17T15:50:40.121 INFO:tasks.rados.rados.0.smithi203.stdout:3806: delete oid 11 current snap is 407 2024-09-17T15:50:40.131 INFO:tasks.rados.rados.0.smithi203.stdout:3805: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:50:40.132 INFO:tasks.rados.rados.0.smithi203.stdout:3805: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:50:40.132 INFO:tasks.rados.rados.0.smithi203.stdout:3804: finishing rollback tid 0 to smithi20328086-15 2024-09-17T15:50:40.132 INFO:tasks.rados.rados.0.smithi203.stdout:3805: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:50:40.132 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1348 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:50:40.132 INFO:tasks.rados.rados.0.smithi203.stdout:3803: finishing rollback tid 0 to smithi20328086-2 2024-09-17T15:50:40.133 INFO:tasks.rados.rados.0.smithi203.stdout:3805: done (4 left) 2024-09-17T15:50:40.133 INFO:tasks.rados.rados.0.smithi203.stdout:3806: done (3 left) 2024-09-17T15:50:40.133 INFO:tasks.rados.rados.0.smithi203.stdout:3807: rollback oid 14 current snap is 407 2024-09-17T15:50:40.133 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 375 2024-09-17T15:50:40.133 INFO:tasks.rados.rados.0.smithi203.stdout:3808: write oid 29 current snap is 407 2024-09-17T15:50:40.134 INFO:tasks.rados.rados.0.smithi203.stdout:3808: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-17T15:50:40.141 INFO:tasks.rados.rados.0.smithi203.stdout:3808: writing smithi20328086-29 from 487766 to 1132703 tid 1 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3808: writing smithi20328086-29 from 1646907 to 2385700 tid 2 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3809: read oid 26 snap 395 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3809: expect (ObjNum 1245 snap 374 seq_num 1245) 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3810: read oid 48 snap -1 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3810: expect (ObjNum 1302 snap 400 seq_num 1302) 2024-09-17T15:50:40.151 INFO:tasks.rados.rados.0.smithi203.stdout:3811: read oid 50 snap -1 2024-09-17T15:50:40.152 INFO:tasks.rados.rados.0.smithi203.stdout:3811: expect (ObjNum 1324 snap 404 seq_num 1324) 2024-09-17T15:50:40.152 INFO:tasks.rados.rados.0.smithi203.stdout:3812: write (excl) oid 4 current snap is 407 2024-09-17T15:50:40.152 INFO:tasks.rados.rados.0.smithi203.stdout:3812: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-17T15:50:40.159 INFO:tasks.rados.rados.0.smithi203.stdout:3812: writing smithi20328086-4 from 504573 to 941112 tid 1 2024-09-17T15:50:40.166 INFO:tasks.rados.rados.0.smithi203.stdout:3812: writing smithi20328086-4 from 1421742 to 2020546 tid 2 2024-09-17T15:50:40.170 INFO:tasks.rados.rados.0.smithi203.stdout:3812: writing smithi20328086-4 from 2437505 to 2504573 tid 3 2024-09-17T15:50:40.170 INFO:tasks.rados.rados.0.smithi203.stdout:3813: snap_remove snap 404 2024-09-17T15:50:54.907 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:50:54.908 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:50:54.908 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:50:54.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:50:55.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: get_monmap_and_config 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: build_initial_monmap 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: monmap: 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:50:55.041824+0000 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:50:55.041824+0000 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:50:55.038 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:50:55.039 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 Processor -- start 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 -- start start 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: init 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-17T15:50:55.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-a con 0x7f50a40689d0 addr v2:172.21.15.86:3300/0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-b con 0x7f50a4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-c con 0x7f50a4135ae0 addr v2:172.21.15.203:3300/0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f50a4074500 con 0x7f50a40689d0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f50a409c8b0 con 0x7f50a4135ae0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f50a409cb90 con 0x7f50a4068db0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50ab7dc640 10 monclient(hunting): authenticate will time out at 2388.007473s 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a8d77640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 msgr2=0x7f50a40691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:55.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a8d77640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51506/0 (socket says 172.21.15.86:51506) 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 learned_addr learned my addr 172.21.15.86:0/2530579140 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f50a40689d0 auth_method 0 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9d79640 10 monclient(hunting): get_auth_request con 0x7f50a4135ae0 auth_method 0 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9d79640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.039+0000 7f50a9d79640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.043+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.043+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.043+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.043+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:55.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 msgr2=0x7f50a40691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:55.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f50a40689d0 auth_method 0 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.243+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 10 monclient(hunting): get_auth_request con 0x7f50a4135ae0 auth_method 0 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.247+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f50a40689d0 auth_method 0 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 msgr2=0x7f50a40691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:55.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:55.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:55.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.647+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:55.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:55.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 10 monclient(hunting): get_auth_request con 0x7f50a4135ae0 auth_method 0 2024-09-17T15:50:55.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:55.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:55.651+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:50:56.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 msgr2=0x7f50a40691d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f50a40689d0 auth_method 0 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.451+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:50:56.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:56.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 10 monclient(hunting): get_auth_request con 0x7f50a4135ae0 auth_method 0 2024-09-17T15:50:56.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:56.453 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:56.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:56.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:50:56.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:50:56.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:50:56.454 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:56.455+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): tick 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 monclient(hunting): continuing hunt 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 msgr2=0x7f50a40691d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50a4068db0 0x7f50a40691d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 msgr2=0x7f50a4135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.203:3300/0 conn(0x7f50a4135ae0 0x7f50a4135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 -- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 msgr2=0x7f50a406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 --2- 172.21.15.86:0/2530579140 >> v2:172.21.15.86:3300/0 conn(0x7f50a40689d0 0x7f50a406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:58.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient: _add_conns ranks=[0,1] 2024-09-17T15:50:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50880036a0 addr v1:172.21.15.86:6789/0 2024-09-17T15:50:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50880073d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:50:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 -- 172.21.15.86:0/2530579140 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50a409ce70 con 0x7f50880036a0 2024-09-17T15:50:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 1 -- 172.21.15.86:0/2530579140 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508800c130 con 0x7f50880073d0 2024-09-17T15:50:58.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.039+0000 7f509affd640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.043+0000 7f50a9578640 1 -- 172.21.15.86:0/2530579140 >> v1:172.21.15.86:6789/0 conn(0x7f50880036a0 legacy=0x7f5088005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:58.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:58.043+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> v1:172.21.15.203:6789/0 conn(0x7f50880073d0 legacy=0x7f5088009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1679447137 0 0) 0x7f508800c130 con 0x7f50880073d0 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): my global_id is 254221 2024-09-17T15:50:59.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5090003530 con 0x7f50880073d0 2024-09-17T15:50:59.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839657759 0 0) 0x7f50a409ce70 con 0x7f50880036a0 2024-09-17T15:50:59.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _negotiate 2024-09-17T15:50:59.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:59.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:50:59.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 10 monclient(hunting): my global_id is 234474 2024-09-17T15:50:59.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.979+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f508800c130 con 0x7f50880036a0 2024-09-17T15:50:59.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4034016699 0 0) 0x7f5090003530 con 0x7f50880073d0 2024-09-17T15:50:59.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 >> v1:172.21.15.86:6789/0 conn(0x7f50880036a0 legacy=0x7f5088005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:59.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:50:59.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 monclient: found mon.noname-c-legacy 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.203:6789/0 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508800d390 con 0x7f50880073d0 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _finish_auth 0 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f50a0004610 con 0x7f50880073d0 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: dump: 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:59.987 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] but i'm connected to v1:172.21.15.203:6789/0, reconnecting 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _reopen_session rank -1 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 >> v1:172.21.15.203:6789/0 conn(0x7f50880073d0 legacy=0x7f5088009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 reap_dead start 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:50:59.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f5090003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): picked mon.b con 0x7f5090003c10 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f5090004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): picked mon.a con 0x7f5090004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f509000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): picked mon.c con 0x7f50900099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f509000c3a0 con 0x7f5090004810 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f509000c550 con 0x7f5090003c10 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): start opening mon connection 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f509000c750 con 0x7f50900099c0 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): _renew_subs 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:50:59.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9d79640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 msgr2=0x7f509000bdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9d79640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f509000bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f5090004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f5090003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f5090003c10 auth_method 0 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): get_auth_request con 0x7f5090004810 auth_method 0 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:50:59.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient(hunting): handle_auth_done global_id 254221 payload 293 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 msgr2=0x7f509000bdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f509000bdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 msgr2=0x7f5090003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f5090003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient: _finish_hunting 0 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 monclient: found mon.a 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:50:59.991 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5090003530 con 0x7f5090004810 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient: _finish_auth 0 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 10 monclient: _check_auth_tickets 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a8d77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50ab7dc640 5 monclient: authenticate success, global_id 254221 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50ab7dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:50:59.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.987+0000 7f50a9578640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f5090003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:50:59.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50a8d77640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 msgr2=0x7f5090004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:50:59.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f5090004c90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003990 tx=0x7f50a001cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365712a6c3b79450 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a000c200 con 0x7f5090004810 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: dump: 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:59.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a00a9070 con 0x7f5090004810 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 1 -- 172.21.15.86:0/2530579140 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a0004b20 con 0x7f5090004810 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 10 monclient: get_monmap_and_config success 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 4 set_mon_vals no callback set 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f509bfff640 10 monclient: dump: 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:50:59.996 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:50:59.997 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 10 monclient: shutdown 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 msgr2=0x7f5090004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.995+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f5090004c90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003990 tx=0x7f50a001cee0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2530579140 shutdown_connections 2024-09-17T15:51:00.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f509000bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f5090004c90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2530579140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f5090003ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2530579140 >> 172.21.15.86:0/2530579140 conn(0x7f50a40a6440 msgr2=0x7f50a40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2530579140 shutdown_connections 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2530579140 wait complete. 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient: build_initial_monmap 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient: monmap: 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:00.001 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 Processor -- start 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- start start 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient: init 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f50a41cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-b con 0x7f5090003c10 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-a con 0x7f5090004810 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:00.002 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f50a4134850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): picked mon.noname-c con 0x7f50900099c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f50a4136280 con 0x7f5090004810 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f50a4136100 con 0x7f5090003c10 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f50a4136400 con 0x7f50900099c0 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50ab7dc640 10 monclient(hunting): authenticate will time out at 2392.966858s 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9d79640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 msgr2=0x7f50a4134850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9d79640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f50a4134850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:00.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35526/0 (socket says 172.21.15.86:35526) 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 -- 172.21.15.86:0/2736022135 learned_addr learned my addr 172.21.15.86:0/2736022135 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): get_auth_request con 0x7f5090004810 auth_method 0 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9578640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f50a41cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9578640 10 monclient(hunting): get_auth_request con 0x7f5090003c10 auth_method 0 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9578640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9578640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a9578640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient(hunting): handle_auth_done global_id 234477 payload 824 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 -- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 msgr2=0x7f50a4134850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f50a4134850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 -- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 msgr2=0x7f50a41cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f50a41cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 monclient: found mon.noname-a 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.005 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:50:59.999+0000 7f50a8d77640 1 -- 172.21.15.86:0/2736022135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a4121430 con 0x7f5090004810 2024-09-17T15:51:00.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50a8d77640 1 -- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 msgr2=0x7f50a4134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50a8d77640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003e80 tx=0x7f5090005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a00bd070 con 0x7f5090004810 2024-09-17T15:51:00.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: dump: 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.008 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50a0004410 con 0x7f5090004810 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 5 monclient: authenticate success, global_id 234477 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _renew_subs 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a409f720 con 0x7f5090004810 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:00.009 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a000c200 con 0x7f5090004810 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 10 monclient: dump: 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f509affd640 4 set_mon_vals no callback set 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _renew_subs 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a4121cd0 con 0x7f5090004810 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f50a00c0070 con 0x7f5090004810 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a4136790 con 0x7f5090004810 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 0x7f508803eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.003+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(830..830 src has 794..830) ==== 4847+0+0 (secure 0 0 0) 0x7f50a00e9880 con 0x7f5090004810 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.007+0000 7f50a9578640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 0x7f508803eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.007+0000 7f50a9578640 10 monclient: get_auth_request con 0x7f508803c630 auth_method 0 2024-09-17T15:51:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.007+0000 7f50a9578640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 0x7f508803eaf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f50a4121360 tx=0x7f5094008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.011+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f50a4121430 con 0x7f5090004810 2024-09-17T15:51:00.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.011+0000 7f5099ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.011+0000 7f5099ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:00.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f50a41cfd30 con 0x7f5090004810 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f5099ffb640 1 -- 172.21.15.86:0/2736022135 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f50a0016c90 con 0x7f5090004810 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f5099ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:00.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f5099ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:00.238 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":214,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:00.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 msgr2=0x7f508803eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 0x7f508803eaf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f50a4121360 tx=0x7f5094008040 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 10 monclient: shutdown 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 msgr2=0x7f50a4134310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f50a0003e80 tx=0x7f5090005840 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 shutdown_connections 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f508803c630 0x7f508803eaf0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50900099c0 0x7f50a4134850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5090004810 0x7f50a4134310 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 --2- 172.21.15.86:0/2736022135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5090003c10 0x7f50a41cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 >> 172.21.15.86:0/2736022135 conn(0x7f50a40a6440 msgr2=0x7f50a409ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 shutdown_connections 2024-09-17T15:51:00.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.239+0000 7f50ab7dc640 1 -- 172.21.15.86:0/2736022135 wait complete. 2024-09-17T15:51:00.256 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:51:00.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:51:00.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:00.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: get_monmap_and_config 2024-09-17T15:51:00.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: build_initial_monmap 2024-09-17T15:51:00.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: monmap: 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:00.377516+0000 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:00.377516+0000 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:00.374 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 Processor -- start 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 -- start start 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: init 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f127c0a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f127c0a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): picked mon.noname-b con 0x7f127c0a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f127c13d180 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c0a68d0 con 0x7f127c13d180 2024-09-17T15:51:00.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f127c0a64a0 con 0x7f127c0a4f70 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f127c09c230 con 0x7f127c0a5350 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f128270c640 10 monclient(hunting): authenticate will time out at 2393.342815s 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f1273fff640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 msgr2=0x7f127c0a5750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f1273fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f127c0a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f1280ca9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f127c13d180 0x7f127c13f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39178/0 (socket says 172.21.15.86:39178) 2024-09-17T15:51:00.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.375+0000 7f1280ca9640 1 -- 172.21.15.86:0/845723981 learned_addr learned my addr 172.21.15.86:0/845723981 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 1 -- 172.21.15.86:0/845723981 >> v1:172.21.15.86:6789/0 conn(0x7f127c13d180 legacy=0x7f127c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 -- 172.21.15.86:0/845723981 >> v1:172.21.15.203:6789/0 conn(0x7f127c0a4f70 legacy=0x7f127c09cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3221134474 0 0) 0x7f127c0a68d0 con 0x7f127c13d180 2024-09-17T15:51:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): my global_id is 234480 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1268003530 con 0x7f127c13d180 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721350693 0 0) 0x7f127c0a64a0 con 0x7f127c0a4f70 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): my global_id is 254227 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f127c0a68d0 con 0x7f127c0a4f70 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 592457842 0 0) 0x7f1268003530 con 0x7f127c13d180 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 msgr2=0x7f127c0a5750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f127c0a5750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 >> v1:172.21.15.203:6789/0 conn(0x7f127c0a4f70 legacy=0x7f127c09cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c09c510 con 0x7f127c13d180 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f1274001d30 con 0x7f127c13d180 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f128270c640 5 monclient: authenticate success, global_id 234480 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f128270c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:00.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: dump: 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 >> v1:172.21.15.86:6789/0 conn(0x7f127c13d180 legacy=0x7f127c13f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 0x7f1268004090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): picked mon.a con 0x7f1268003c90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f1268004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): picked mon.c con 0x7f127c0a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1268004fd0 0x7f126800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): picked mon.b con 0x7f1268004fd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f127c09c230 con 0x7f1268003c90 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f126800bf10 con 0x7f1268004fd0 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f126800c090 con 0x7f127c0a5350 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1273fff640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 msgr2=0x7f1268004870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f128270c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1273fff640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f1268004870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 0x7f1268004090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): get_auth_request con 0x7f1268003c90 auth_method 0 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.386 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1268004fd0 0x7f126800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 10 monclient(hunting): get_auth_request con 0x7f1268004fd0 auth_method 0 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f1280ca9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient(hunting): handle_auth_done global_id 234480 payload 293 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 msgr2=0x7f1268004870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c0a5350 0x7f1268004870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1268004fd0 msgr2=0x7f126800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1268004fd0 0x7f126800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 monclient: found mon.a 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 1 -- 172.21.15.86:0/845723981 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1268003530 con 0x7f1268003c90 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.379+0000 7f127bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127bfff640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 msgr2=0x7f1268004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127bfff640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 0x7f1268004090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f126c007c50 tx=0x7f126c01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47143f07e81c9fd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ad070 con 0x7f1268003c90 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: dump: 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f128270c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126c0072d0 con 0x7f1268003c90 2024-09-17T15:51:00.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f128270c640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f128270c640 4 set_mon_vals no callback set 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 1 -- 172.21.15.86:0/845723981 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c007950 con 0x7f1268003c90 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.383+0000 7f127b7fe640 10 monclient: dump: 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: shutdown 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 msgr2=0x7f1268004090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 --2- 172.21.15.86:0/845723981 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1268003c90 0x7f1268004090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f126c007c50 tx=0x7f126c01c6a0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1280ca9640 1 -- 172.21.15.86:0/845723981 reap_dead start 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- 172.21.15.86:0/845723981 shutdown_connections 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- 172.21.15.86:0/845723981 >> 172.21.15.86:0/845723981 conn(0x7f127c0a7240 msgr2=0x7f127c0a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- 172.21.15.86:0/845723981 shutdown_connections 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- 172.21.15.86:0/845723981 wait complete. 2024-09-17T15:51:00.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: build_initial_monmap 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: monmap: 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 Processor -- start 2024-09-17T15:51:00.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- start start 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: init 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f127c120bc0 0x7f127c120fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): picked mon.noname-b con 0x7f127c120bc0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 0x7f127c1214e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): picked mon.noname-a con 0x7f127c124980 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c1d0c60 0x7f127c1d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): picked mon.noname-c con 0x7f127c1d0c60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f127c141920 con 0x7f127c124980 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f127c1417a0 con 0x7f127c120bc0 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f127c141aa0 con 0x7f127c1d0c60 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f128270c640 10 monclient(hunting): authenticate will time out at 2393.354369s 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1280ca9640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c1d0c60 msgr2=0x7f127c1d1060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1280ca9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c1d0c60 0x7f127c1d1060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 0x7f127c1214e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 0x7f127c1214e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35556/0 (socket says 172.21.15.86:35556) 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 -- 172.21.15.86:0/1230715692 learned_addr learned my addr 172.21.15.86:0/1230715692 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): get_auth_request con 0x7f127c124980 auth_method 0 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f127c120bc0 0x7f127c120fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient(hunting): handle_auth_done global_id 234483 payload 824 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c1d0c60 msgr2=0x7f127c1d1060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f127c1d0c60 0x7f127c1d1060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f127c120bc0 msgr2=0x7f127c120fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f127c120bc0 0x7f127c120fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 monclient: found mon.noname-a 2024-09-17T15:51:00.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f1273fff640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127c1228e0 con 0x7f127c124980 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f127bfff640 10 monclient: get_auth_request con 0x7f127c120bc0 auth_method 0 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.387+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f127c120bc0 0x7f127c120fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f1273fff640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 msgr2=0x7f127c1214e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f1273fff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 0x7f127c1214e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f126400b580 tx=0x7f126400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264013020 con 0x7f127c124980 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: dump: 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.403 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1264005070 con 0x7f127c124980 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 5 monclient: authenticate success, global_id 234483 2024-09-17T15:51:00.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 monclient: _renew_subs 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127c123680 con 0x7f127c124980 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1264005460 con 0x7f127c124980 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 10 monclient: dump: 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.405 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127affd640 4 set_mon_vals no callback set 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 monclient: _renew_subs 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f127c0a1760 con 0x7f127c124980 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 0x7f127c09e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f127c09cef0 con 0x7f127c1d15a0 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 0x7f127c09e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.406 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient: get_auth_request con 0x7f127c1d15a0 auth_method 0 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1264016020 con 0x7f127c124980 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f125c03bb60 0x7f125c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 10 monclient(hunting): handle_auth_done global_id 234486 payload 824 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f127bfff640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f127c0a5ff0 con 0x7f127c1d15a0 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(830..830 src has 794..830) ==== 4847+0+0 (secure 0 0 0) 0x7f126404fe30 con 0x7f127c124980 2024-09-17T15:51:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f1280ca9640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f125c03bb60 0x7f125c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.391+0000 7f1280ca9640 10 monclient: get_auth_request con 0x7f125c03bb60 auth_method 0 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f127bfff640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 msgr2=0x7f127c09e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 0x7f127c09e820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f126c0aa000 tx=0x7f1268005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ad040 con 0x7f127c1d15a0 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 10 monclient: dump: 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.408 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f126c007610 con 0x7f127c1d15a0 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:00.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 msgr2=0x7f127c09e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 0x7f127c09e820 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f126c0aa000 tx=0x7f1268005840 comp rx=0 tx=0).stop 2024-09-17T15:51:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 msgr2=0x7f127c09e820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f12797fa640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c1d15a0 0x7f127c09e820 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f126c0aa000 tx=0x7f1268005840 comp rx=0 tx=0).stop 2024-09-17T15:51:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.395+0000 7f1280ca9640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f125c03bb60 0x7f125c03e020 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f127c09cce0 tx=0x7f1274004b70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f128270c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f128270c640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 0x7f127c09f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f128270c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f127c069920 con 0x7f127c09ed60 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 0x7f127c09f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient: get_auth_request con 0x7f127c09ed60 auth_method 0 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.412 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 10 monclient(hunting): handle_auth_done global_id 234489 payload 824 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.399+0000 7f127bfff640 1 -- 172.21.15.86:0/1230715692 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f127c1d1980 con 0x7f127c09ed60 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f127bfff640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 msgr2=0x7f127c09f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f127bfff640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 0x7f127c09f170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f126c01cf40 tx=0x7f126c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ad040 con 0x7f127c09ed60 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 10 monclient: dump: 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.413 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f126c0a9e20 con 0x7f127c09ed60 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 msgr2=0x7f127c09f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 0x7f127c09f170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f126c01cf40 tx=0x7f126c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 msgr2=0x7f127c09f170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:00.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f12797fa640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c09ed60 0x7f127c09f170 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f126c01cf40 tx=0x7f126c0a7000 comp rx=0 tx=0).stop 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 214, 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 189598, 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:51:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:00.429 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f125c03bb60 msgr2=0x7f125c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f125c03bb60 0x7f125c03e020 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f127c09cce0 tx=0x7f1274004b70 comp rx=0 tx=0).stop 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 10 monclient: shutdown 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 msgr2=0x7f127c1214e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 --2- 172.21.15.86:0/1230715692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f127c124980 0x7f127c1214e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f126400b580 tx=0x7f126400ba40 comp rx=0 tx=0).stop 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f1280ca9640 1 -- 172.21.15.86:0/1230715692 reap_dead start 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f1280ca9640 1 -- 172.21.15.86:0/1230715692 reap_dead start 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 shutdown_connections 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 >> 172.21.15.86:0/1230715692 conn(0x7f127c0a7240 msgr2=0x7f127c139ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 shutdown_connections 2024-09-17T15:51:00.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.403+0000 7f128270c640 1 -- 172.21.15.86:0/1230715692 wait complete. 2024-09-17T15:51:00.431 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:51:00.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-17T15:51:00.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: get_monmap_and_config 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: build_initial_monmap 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: monmap: 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:00.548910+0000 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:00.548910+0000 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.545 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:00.546 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:00.546 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:00.546 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:00.546 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:00.546 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 Processor -- start 2024-09-17T15:51:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 -- start start 2024-09-17T15:51:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: init 2024-09-17T15:51:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2b80a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb2b80a1760 0x7fb2b813af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb2b80a1760 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2b813b450 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b80a2130 con 0x7fb2b813b450 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2b80a26f0 con 0x7fb2b80a0c70 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fb2b80a2410 con 0x7fb2b80a1760 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2c07c0640 10 monclient(hunting): authenticate will time out at 2393.514705s 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bed5d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb2b813b450 0x7fb2b813d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39186/0 (socket says 172.21.15.86:39186) 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bed5d640 1 -- 172.21.15.86:0/931925355 learned_addr learned my addr 172.21.15.86:0/931925355 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:00.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/931925355 >> v2:172.21.15.203:3300/0 conn(0x7fb2b80a1760 0x7fb2b813af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): get_auth_request con 0x7fb2b80a1760 auth_method 0 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_done global_id 254230 payload 274 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/931925355 >> v1:172.21.15.203:6789/0 conn(0x7fb2b80a0c70 legacy=0x7fb2b80a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/931925355 >> v1:172.21.15.86:6789/0 conn(0x7fb2b813b450 legacy=0x7fb2b813d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 1 monclient: found mon.noname-c 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:51:00.550 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.547+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/931925355 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b809f9f0 con 0x7fb2b80a1760 2024-09-17T15:51:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/931925355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b80a1760 msgr2=0x7fb2b813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/931925355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b80a1760 0x7fb2b813af10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80097c0 tx=0x7fb2a801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=462c97021a2afb28 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 1 -- 172.21.15.86:0/931925355 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a80b2070 con 0x7fb2b80a1760 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: dump: 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 1 -- 172.21.15.86:0/931925355 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2a80ad640 con 0x7fb2b80a1760 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 1 -- 172.21.15.86:0/931925355 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a80ada10 con 0x7fb2b80a1760 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 5 monclient: authenticate success, global_id 254230 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 4 set_mon_vals no callback set 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bd55a640 10 monclient: dump: 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.555 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: shutdown 2024-09-17T15:51:00.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- 172.21.15.86:0/931925355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b80a1760 msgr2=0x7fb2b813af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/931925355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b80a1760 0x7fb2b813af10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb2a80097c0 tx=0x7fb2a801c3f0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- 172.21.15.86:0/931925355 shutdown_connections 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/931925355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b80a1760 0x7fb2b813af10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- 172.21.15.86:0/931925355 >> 172.21.15.86:0/931925355 conn(0x7fb2b80a2f40 msgr2=0x7fb2b80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- 172.21.15.86:0/931925355 shutdown_connections 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- 172.21.15.86:0/931925355 wait complete. 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: build_initial_monmap 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: monmap: 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.561 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 Processor -- start 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- start start 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: init 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-a con 0x7fb2b80a0c70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2b80a1760 0x7fb2b806fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-c con 0x7fb2b80a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b813b450 0x7fb2b80703a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): picked mon.noname-b con 0x7fb2b813b450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8141900 con 0x7fb2b80a0c70 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8141780 con 0x7fb2b813b450 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fb2b8141a80 con 0x7fb2b80a1760 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2c07c0640 10 monclient(hunting): authenticate will time out at 2393.523352s 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bdd5b640 1 -- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2b80a1760 msgr2=0x7fb2b806fe60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.86:3301/0 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bdd5b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2b80a1760 0x7fb2b806fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:35580/0 (socket says 172.21.15.86:35580) 2024-09-17T15:51:00.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 -- 172.21.15.86:0/3567188657 learned_addr learned my addr 172.21.15.86:0/3567188657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): get_auth_request con 0x7fb2b80a0c70 auth_method 0 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bed5d640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b813b450 0x7fb2b80703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient(hunting): handle_auth_done global_id 234492 payload 824 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2b80a1760 msgr2=0x7fb2b806fe60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2b80a1760 0x7fb2b806fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b813b450 msgr2=0x7fb2b80703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b813b450 0x7fb2b80703a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 monclient: found mon.noname-a 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2be55c640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b809d460 con 0x7fb2b80a0c70 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bed5d640 10 monclient: get_auth_request con 0x7fb2b813b450 auth_method 0 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.555+0000 7fb2bed5d640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b813b450 0x7fb2b80703a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2be55c640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 msgr2=0x7fb2b806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2be55c640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400c7c0 tx=0x7fb2b400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4016020 con 0x7fb2b80a0c70 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: dump: 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b400ac60 con 0x7fb2b80a0c70 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 5 monclient: authenticate success, global_id 234492 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 monclient: _renew_subs 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b80725b0 con 0x7fb2b80a0c70 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4012660 con 0x7fb2b80a0c70 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 10 monclient: dump: 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.567 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 monclient: _renew_subs 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bcd59640 4 set_mon_vals no callback set 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b8072f90 con 0x7fb2b80a0c70 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fb2b4019020 con 0x7fb2b80a0c70 2024-09-17T15:51:00.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 0x7fb2b81321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b80a1050 con 0x7fb2b8070db0 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb29c03bb40 0x7fb29c03e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(830..830 src has 794..830) ==== 4847+0+0 (secure 0 0 0) 0x7fb2b40500d0 con 0x7fb2b80a0c70 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 0x7fb2b81321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient: get_auth_request con 0x7fb2b8070db0 auth_method 0 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bed5d640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb29c03bb40 0x7fb29c03e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.559+0000 7fb2bed5d640 10 monclient: get_auth_request con 0x7fb29c03bb40 auth_method 0 2024-09-17T15:51:00.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_done global_id 254233 payload 824 2024-09-17T15:51:00.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2b8072110 con 0x7fb2b8070db0 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 msgr2=0x7fb2b81321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 0x7fb2b81321d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb2b806f840 tx=0x7fb2a801ac30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2bed5d640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb29c03bb40 0x7fb29c03e000 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2b813dd70 tx=0x7fb2b0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a80b2070 con 0x7fb2b8070db0 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 10 monclient: dump: 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.571 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb2a80b3040 con 0x7fb2b8070db0 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 msgr2=0x7fb2b81321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 0x7fb2b81321d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb2b806f840 tx=0x7fb2a801ac30 comp rx=0 tx=0).stop 2024-09-17T15:51:00.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 msgr2=0x7fb2b81321d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.563+0000 7fb2aeffd640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b8070db0 0x7fb2b81321d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2c07c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 0x7fb2b802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2c07c0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b8136350 con 0x7fb2b802d450 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 0x7fb2b802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient: get_auth_request con 0x7fb2b802d450 auth_method 0 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:00.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:00.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 10 monclient(hunting): handle_auth_done global_id 254236 payload 824 2024-09-17T15:51:00.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.571+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/3567188657 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2b806d4f0 con 0x7fb2b802d450 2024-09-17T15:51:00.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2bdd5b640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 msgr2=0x7fb2b802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2bdd5b640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 0x7fb2b802d860 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2a801cc90 tx=0x7fb2a8019190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8003a30 con 0x7fb2b802d450 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 10 monclient: dump: 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:00.575 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7fb2a80c4e30 con 0x7fb2b802d450 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 msgr2=0x7fb2b802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 0x7fb2b802d860 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2a801cc90 tx=0x7fb2a8019190 comp rx=0 tx=0).stop 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 msgr2=0x7fb2b802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2aeffd640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb2b802d450 0x7fb2b802d860 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb2a801cc90 tx=0x7fb2a8019190 comp rx=0 tx=0).stop 2024-09-17T15:51:00.576 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 214, 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 76520, 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: 1 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:00.577 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:00.578 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:00.579 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:00.580 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:00.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.582 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:00.583 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.584 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:00.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:00.586 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:00.587 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:00.589 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:00.589 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb29c03bb40 msgr2=0x7fb29c03e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fb29c03bb40 0x7fb29c03e000 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb2b813dd70 tx=0x7fb2b0009040 comp rx=0 tx=0).stop 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 10 monclient: shutdown 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 msgr2=0x7fb2b806f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400c7c0 tx=0x7fb2b400cc80 comp rx=0 tx=0).stop 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2bed5d640 1 -- 172.21.15.86:0/3567188657 reap_dead start 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 shutdown_connections 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 --2- 172.21.15.86:0/3567188657 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2b80a0c70 0x7fb2b806f920 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 >> 172.21.15.86:0/3567188657 conn(0x7fb2b80a2f40 msgr2=0x7fb2b813de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 shutdown_connections 2024-09-17T15:51:00.590 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:00.575+0000 7fb2c07c0640 1 -- 172.21.15.86:0/3567188657 wait complete. 2024-09-17T15:51:00.591 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:51:00.592 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:51:00.879 INFO:tasks.rados.rados.0.smithi203.stdout:3813: done (9 left) 2024-09-17T15:51:00.879 INFO:tasks.rados.rados.0.smithi203.stdout:3814: rollback oid 32 current snap is 407 2024-09-17T15:51:00.879 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 32 to 389 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3815: delete oid 35 current snap is 407 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3804: finishing rollback tid 1 to smithi20328086-15 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3803: finishing rollback tid 1 to smithi20328086-2 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3804: finishing rollback tid 2 to smithi20328086-15 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 15 v 1813 (ObjNum 1290 snap 396 seq_num 1290) dirty exists 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3803: finishing rollback tid 2 to smithi20328086-2 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1840 (ObjNum 1020 snap 316 seq_num 1020) dirty exists 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3807: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1669 (ObjNum 129 snap 0 seq_num 3606607520) dirty dne 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: finishing write tid 1 to smithi20328086-24 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: finishing write tid 2 to smithi20328086-24 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: finishing write tid 3 to smithi20328086-24 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: finishing write tid 4 to smithi20328086-24 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: oid 24 updating version 0 to 1569 2024-09-17T15:51:00.880 INFO:tasks.rados.rados.0.smithi203.stdout:3802: oid 24 version 1569 is already newer than 1568 2024-09-17T15:51:00.881 INFO:tasks.rados.rados.0.smithi203.stdout:3802: oid 24 version 1569 is already newer than 1567 2024-09-17T15:51:00.881 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1569 (ObjNum 1336 snap 407 seq_num 1336) dirty exists 2024-09-17T15:51:00.881 INFO:tasks.rados.rados.0.smithi203.stdout:3802: left oid 24 (ObjNum 1336 snap 407 seq_num 1336) 2024-09-17T15:51:00.881 INFO:tasks.rados.rados.0.smithi203.stdout:3808: finishing write tid 1 to smithi20328086-29 2024-09-17T15:51:00.881 INFO:tasks.rados.rados.0.smithi203.stdout:3809: expect (ObjNum 1245 snap 374 seq_num 1245) 2024-09-17T15:51:00.945 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2894794 2024-09-17T15:51:01.039 INFO:tasks.rados.rados.0.smithi203.stdout:3810: expect (ObjNum 1302 snap 400 seq_num 1302) 2024-09-17T15:51:01.039 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:01.246 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3067262 2024-09-17T15:51:01.272 INFO:tasks.rados.rados.0.smithi203.stdout:3808: finishing write tid 2 to smithi20328086-29 2024-09-17T15:51:01.272 INFO:tasks.rados.rados.0.smithi203.stdout:3811: expect (ObjNum 1324 snap 404 seq_num 1324) 2024-09-17T15:51:01.301 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 739855 2024-09-17T15:51:01.316 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 27299 2024-09-17T15:51:01.340 INFO:tasks.rados.rados.0.smithi203.stdout:3808: finishing write tid 3 to smithi20328086-29 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3808: finishing write tid 4 to smithi20328086-29 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3808: oid 29 updating version 0 to 1570 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3808: oid 29 updating version 1570 to 1571 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3808: oid 29 updating version 1571 to 1572 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1572 (ObjNum 1338 snap 407 seq_num 1338) dirty exists 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3808: left oid 29 (ObjNum 1338 snap 407 seq_num 1338) 2024-09-17T15:51:01.341 INFO:tasks.rados.rados.0.smithi203.stdout:3812: finishing write tid 1 to smithi20328086-4 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: finishing write tid 2 to smithi20328086-4 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: finishing write tid 3 to smithi20328086-4 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: finishing write tid 4 to smithi20328086-4 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: finishing write tid 5 to smithi20328086-4 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: oid 4 updating version 0 to 1350 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: oid 4 updating version 1350 to 1351 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: oid 4 version 1351 is already newer than 1349 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: oid 4 updating version 1351 to 1352 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1352 (ObjNum 1339 snap 407 seq_num 1339) dirty exists 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3812: left oid 4 (ObjNum 1339 snap 407 seq_num 1339) 2024-09-17T15:51:01.342 INFO:tasks.rados.rados.0.smithi203.stdout:3814: finishing rollback tid 1 to smithi20328086-32 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3814: finishing rollback tid 2 to smithi20328086-32 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1746 (ObjNum 1240 snap 373 seq_num 1240) dirty exists 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3802: done (10 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3803: done (9 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3804: done (8 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3807: done (7 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3808: done (6 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3809: done (5 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3810: done (4 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3811: done (3 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3812: done (2 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3814: done (1 left) 2024-09-17T15:51:01.343 INFO:tasks.rados.rados.0.smithi203.stdout:3815: done (0 left) 2024-09-17T15:51:01.344 INFO:tasks.rados.rados.0.smithi203.stdout:3816: write (excl) oid 8 current snap is 407 2024-09-17T15:51:01.344 INFO:tasks.rados.rados.0.smithi203.stdout:3816: seq_num 1341 ranges {538187=587037,1538706=602473} 2024-09-17T15:51:01.359 INFO:tasks.rados.rados.0.smithi203.stdout:3816: writing smithi20328086-8 from 538187 to 1125224 tid 1 2024-09-17T15:51:01.376 INFO:tasks.rados.rados.0.smithi203.stdout:3816: writing smithi20328086-8 from 1538706 to 2141179 tid 2 2024-09-17T15:51:01.376 INFO:tasks.rados.rados.0.smithi203.stdout:3817: snap_remove snap 402 2024-09-17T15:51:01.376 INFO:tasks.rados.rados.0.smithi203.stdout:3816: finishing write tid 1 to smithi20328086-8 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3817: done (1 left) 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3818: delete oid 37 current snap is 407 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3816: finishing write tid 2 to smithi20328086-8 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3816: finishing write tid 3 to smithi20328086-8 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3816: finishing write tid 4 to smithi20328086-8 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3816: oid 8 updating version 0 to 1652 2024-09-17T15:51:02.812 INFO:tasks.rados.rados.0.smithi203.stdout:3816: oid 8 version 1652 is already newer than 1651 2024-09-17T15:51:02.813 INFO:tasks.rados.rados.0.smithi203.stdout:3816: oid 8 updating version 1652 to 1653 2024-09-17T15:51:02.813 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1653 (ObjNum 1341 snap 407 seq_num 1341) dirty exists 2024-09-17T15:51:02.813 INFO:tasks.rados.rados.0.smithi203.stdout:3816: left oid 8 (ObjNum 1341 snap 407 seq_num 1341) 2024-09-17T15:51:02.819 INFO:tasks.rados.rados.0.smithi203.stdout:3816: done (1 left) 2024-09-17T15:51:02.820 INFO:tasks.rados.rados.0.smithi203.stdout:3818: done (0 left) 2024-09-17T15:51:02.820 INFO:tasks.rados.rados.0.smithi203.stdout:3819: delete oid 20 current snap is 407 2024-09-17T15:51:02.827 INFO:tasks.rados.rados.0.smithi203.stdout:3819: done (0 left) 2024-09-17T15:51:02.827 INFO:tasks.rados.rados.0.smithi203.stdout:3820: snap_remove snap 392 2024-09-17T15:51:02.898 INFO:tasks.rados.rados.0.smithi203.stdout:3820: done (0 left) 2024-09-17T15:51:02.898 INFO:tasks.rados.rados.0.smithi203.stdout:3821: delete oid 23 current snap is 407 2024-09-17T15:51:02.912 INFO:tasks.rados.rados.0.smithi203.stdout:3821: done (0 left) 2024-09-17T15:51:02.913 INFO:tasks.rados.rados.0.smithi203.stdout:3822: snap_create 2024-09-17T15:51:03.951 INFO:tasks.rados.rados.0.smithi203.stdout:3822: done (0 left) 2024-09-17T15:51:03.952 INFO:tasks.rados.rados.0.smithi203.stdout:3823: snap_remove snap 375 2024-09-17T15:51:04.915 INFO:tasks.rados.rados.0.smithi203.stdout:3823: done (0 left) 2024-09-17T15:51:04.915 INFO:tasks.rados.rados.0.smithi203.stdout:3824: snap_remove snap 406 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:3824: done (0 left) 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:3825: read oid 16 snap -1 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:3825: expect deleted 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:3826: rollback oid 47 current snap is 408 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 47 to 407 2024-09-17T15:51:05.881 INFO:tasks.rados.rados.0.smithi203.stdout:3827: snap_create 2024-09-17T15:51:05.903 INFO:tasks.rados.rados.0.smithi203.stdout:3826: finishing rollback tid 0 to smithi20328086-47 2024-09-17T15:51:05.903 INFO:tasks.rados.rados.0.smithi203.stdout:3826: finishing rollback tid 1 to smithi20328086-47 2024-09-17T15:51:05.905 INFO:tasks.rados.rados.0.smithi203.stdout:3826: finishing rollback tid 2 to smithi20328086-47 2024-09-17T15:51:05.905 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 47 v 1656 (ObjNum 1117 snap 344 seq_num 1117) dirty exists 2024-09-17T15:51:06.884 INFO:tasks.rados.rados.0.smithi203.stdout:3825: done (2 left) 2024-09-17T15:51:06.884 INFO:tasks.rados.rados.0.smithi203.stdout:3826: done (1 left) 2024-09-17T15:51:06.884 INFO:tasks.rados.rados.0.smithi203.stdout:3827: done (0 left) 2024-09-17T15:51:06.884 INFO:tasks.rados.rados.0.smithi203.stdout:3828: read oid 46 snap 400 2024-09-17T15:51:06.884 INFO:tasks.rados.rados.0.smithi203.stdout:3828: expect (ObjNum 1296 snap 399 seq_num 1296) 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3829: rollback oid 24 current snap is 409 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 24 to 401 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3830: read oid 8 snap -1 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3830: expect (ObjNum 1341 snap 407 seq_num 1341) 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3831: read oid 12 snap 407 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3831: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3832: write oid 34 current snap is 409 2024-09-17T15:51:06.885 INFO:tasks.rados.rados.0.smithi203.stdout:3832: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-17T15:51:06.913 INFO:tasks.rados.rados.0.smithi203.stdout:3832: writing smithi20328086-34 from 605415 to 1093448 tid 1 2024-09-17T15:51:06.931 INFO:tasks.rados.rados.0.smithi203.stdout:3832: writing smithi20328086-34 from 1656281 to 2149739 tid 2 2024-09-17T15:51:06.931 INFO:tasks.rados.rados.0.smithi203.stdout:3833: copy_from oid 20 from oid 9 current snap is 409 2024-09-17T15:51:06.931 INFO:tasks.rados.rados.0.smithi203.stdout:3831: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:51:07.017 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1369042 2024-09-17T15:51:07.044 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:3829: finishing rollback tid 0 to smithi20328086-24 2024-09-17T15:51:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:3829: finishing rollback tid 1 to smithi20328086-24 2024-09-17T15:51:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:3829: finishing rollback tid 2 to smithi20328086-24 2024-09-17T15:51:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 24 v 1575 (ObjNum 1301 snap 400 seq_num 1301) dirty exists 2024-09-17T15:51:07.082 INFO:tasks.rados.rados.0.smithi203.stdout:3830: expect (ObjNum 1341 snap 407 seq_num 1341) 2024-09-17T15:51:07.147 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2349932 2024-09-17T15:51:07.154 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.195 INFO:tasks.rados.rados.0.smithi203.stdout:3828: expect (ObjNum 1296 snap 399 seq_num 1296) 2024-09-17T15:51:07.195 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.324 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.388 INFO:tasks.rados.rados.0.smithi203.stdout:3832: finishing write tid 1 to smithi20328086-34 2024-09-17T15:51:07.388 INFO:tasks.rados.rados.0.smithi203.stdout:3832: finishing write tid 2 to smithi20328086-34 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: finishing write tid 3 to smithi20328086-34 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: finishing write tid 4 to smithi20328086-34 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: oid 34 updating version 0 to 1694 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: oid 34 version 1694 is already newer than 1692 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: oid 34 version 1694 is already newer than 1693 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1694 (ObjNum 1345 snap 409 seq_num 1345) dirty exists 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3832: left oid 34 (ObjNum 1345 snap 409 seq_num 1345) 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3833: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3833: finishing copy_from to smithi20328086-20 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1566 (ObjNum 1332 snap 406 seq_num 1332) dirty exists 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3828: done (5 left) 2024-09-17T15:51:07.389 INFO:tasks.rados.rados.0.smithi203.stdout:3829: done (4 left) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3830: done (3 left) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3831: done (2 left) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3832: done (1 left) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3833: done (0 left) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3834: copy_from oid 7 from oid 39 current snap is 409 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3835: read oid 22 snap -1 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3835: expect (ObjNum 1299 snap 400 seq_num 1299) 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3836: write (excl) oid 43 current snap is 409 2024-09-17T15:51:07.390 INFO:tasks.rados.rados.0.smithi203.stdout:3836: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-17T15:51:07.410 INFO:tasks.rados.rados.0.smithi203.stdout:3836: writing smithi20328086-43 from 622222 to 1301857 tid 1 2024-09-17T15:51:07.431 INFO:tasks.rados.rados.0.smithi203.stdout:3836: writing smithi20328086-43 from 1831116 to 2584585 tid 2 2024-09-17T15:51:07.432 INFO:tasks.rados.rados.0.smithi203.stdout:3834: finishing copy_from racing read to smithi20328086-7 2024-09-17T15:51:07.432 INFO:tasks.rados.rados.0.smithi203.stdout:3835: expect (ObjNum 1299 snap 400 seq_num 1299) 2024-09-17T15:51:07.491 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1882890 2024-09-17T15:51:07.627 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 3317418 2024-09-17T15:51:07.638 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.726 INFO:tasks.rados.rados.0.smithi203.stdout:3834: finishing copy_from to smithi20328086-7 2024-09-17T15:51:07.726 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 7 v 1887 (ObjNum 1300 snap 400 seq_num 1300) dirty exists 2024-09-17T15:51:07.726 INFO:tasks.rados.rados.0.smithi203.stdout:3836: finishing write tid 1 to smithi20328086-43 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: finishing write tid 2 to smithi20328086-43 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: finishing write tid 3 to smithi20328086-43 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: finishing write tid 4 to smithi20328086-43 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: oid 43 updating version 0 to 741 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: oid 43 version 741 is already newer than 740 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: oid 43 version 741 is already newer than 739 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 741 (ObjNum 1346 snap 409 seq_num 1346) dirty exists 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: left oid 43 (ObjNum 1346 snap 409 seq_num 1346) 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3837: read oid 20 snap -1 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3837: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3834: done (3 left) 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3835: done (2 left) 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3836: done (1 left) 2024-09-17T15:51:07.727 INFO:tasks.rados.rados.0.smithi203.stdout:3838: read oid 32 snap -1 2024-09-17T15:51:07.728 INFO:tasks.rados.rados.0.smithi203.stdout:3838: expect (ObjNum 1240 snap 373 seq_num 1240) 2024-09-17T15:51:07.728 INFO:tasks.rados.rados.0.smithi203.stdout:3839: write oid 42 current snap is 409 2024-09-17T15:51:07.728 INFO:tasks.rados.rados.0.smithi203.stdout:3839: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-17T15:51:07.751 INFO:tasks.rados.rados.0.smithi203.stdout:3839: writing smithi20328086-42 from 639029 to 1393913 tid 1 2024-09-17T15:51:07.772 INFO:tasks.rados.rados.0.smithi203.stdout:3839: writing smithi20328086-42 from 2173245 to 2639029 tid 2 2024-09-17T15:51:07.772 INFO:tasks.rados.rados.0.smithi203.stdout:3840: copy_from oid 28 from oid 21 current snap is 409 2024-09-17T15:51:07.772 INFO:tasks.rados.rados.0.smithi203.stdout:3837: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:07.786 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1759019 2024-09-17T15:51:07.840 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1952750 2024-09-17T15:51:07.844 INFO:tasks.rados.rados.0.smithi203.stdout:3838: expect (ObjNum 1240 snap 373 seq_num 1240) 2024-09-17T15:51:07.863 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: finishing write tid 1 to smithi20328086-42 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3840: finishing copy_from to smithi20328086-28 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3840: got expected ENOENT (src dne) 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3840: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: finishing write tid 2 to smithi20328086-42 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: finishing write tid 3 to smithi20328086-42 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: finishing write tid 4 to smithi20328086-42 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: oid 42 updating version 0 to 1897 2024-09-17T15:51:07.924 INFO:tasks.rados.rados.0.smithi203.stdout:3839: oid 42 updating version 1897 to 1898 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3839: oid 42 version 1898 is already newer than 1896 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 42 v 1898 (ObjNum 1347 snap 409 seq_num 1347) dirty exists 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3839: left oid 42 (ObjNum 1347 snap 409 seq_num 1347) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3837: done (3 left) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3838: done (2 left) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3839: done (1 left) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3840: done (0 left) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3841: read oid 43 snap -1 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3841: expect (ObjNum 1346 snap 409 seq_num 1346) 2024-09-17T15:51:07.925 INFO:tasks.rados.rados.0.smithi203.stdout:3842: delete oid 6 current snap is 409 2024-09-17T15:51:07.928 INFO:tasks.rados.rados.0.smithi203.stdout:3842: done (1 left) 2024-09-17T15:51:07.928 INFO:tasks.rados.rados.0.smithi203.stdout:3843: write (excl) oid 26 current snap is 409 2024-09-17T15:51:07.928 INFO:tasks.rados.rados.0.smithi203.stdout:3843: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-17T15:51:07.947 INFO:tasks.rados.rados.0.smithi203.stdout:3843: writing smithi20328086-26 from 672643 to 1178025 tid 1 2024-09-17T15:51:07.967 INFO:tasks.rados.rados.0.smithi203.stdout:3843: writing smithi20328086-26 from 1890209 to 2391005 tid 2 2024-09-17T15:51:07.967 INFO:tasks.rados.rados.0.smithi203.stdout:3844: delete oid 36 current snap is 409 2024-09-17T15:51:07.967 INFO:tasks.rados.rados.0.smithi203.stdout:3841: expect (ObjNum 1346 snap 409 seq_num 1346) 2024-09-17T15:51:08.093 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1264484 2024-09-17T15:51:08.130 INFO:tasks.rados.rados.0.smithi203.stdout:3843: finishing write tid 1 to smithi20328086-26 2024-09-17T15:51:08.130 INFO:tasks.rados.rados.0.smithi203.stdout:3843: finishing write tid 2 to smithi20328086-26 2024-09-17T15:51:08.130 INFO:tasks.rados.rados.0.smithi203.stdout:3843: finishing write tid 3 to smithi20328086-26 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: finishing write tid 4 to smithi20328086-26 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: oid 26 updating version 0 to 2089 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: oid 26 version 2089 is already newer than 2088 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: oid 26 updating version 2089 to 2090 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 2090 (ObjNum 1349 snap 409 seq_num 1349) dirty exists 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: left oid 26 (ObjNum 1349 snap 409 seq_num 1349) 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3841: done (2 left) 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3843: done (1 left) 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3844: done (0 left) 2024-09-17T15:51:08.131 INFO:tasks.rados.rados.0.smithi203.stdout:3845: snap_remove snap 400 2024-09-17T15:51:09.096 INFO:tasks.rados.rados.0.smithi203.stdout:3845: done (0 left) 2024-09-17T15:51:09.096 INFO:tasks.rados.rados.0.smithi203.stdout:3846: copy_from oid 34 from oid 10 current snap is 409 2024-09-17T15:51:09.096 INFO:tasks.rados.rados.0.smithi203.stdout:3847: copy_from oid 42 from oid 19 current snap is 409 2024-09-17T15:51:09.096 INFO:tasks.rados.rados.0.smithi203.stdout:3848: snap_create 2024-09-17T15:51:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:3847: finishing copy_from to smithi20328086-42 2024-09-17T15:51:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:3847: got expected ENOENT (src dne) 2024-09-17T15:51:09.107 INFO:tasks.rados.rados.0.smithi203.stdout:3847: finishing copy_from racing read to smithi20328086-42 2024-09-17T15:51:09.133 INFO:tasks.rados.rados.0.smithi203.stdout:3846: finishing copy_from racing read to smithi20328086-34 2024-09-17T15:51:09.136 INFO:tasks.rados.rados.0.smithi203.stdout:3846: finishing copy_from to smithi20328086-34 2024-09-17T15:51:09.136 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1417 (ObjNum 1319 snap 403 seq_num 1319) dirty exists 2024-09-17T15:51:10.105 INFO:tasks.rados.rados.0.smithi203.stdout:3846: done (2 left) 2024-09-17T15:51:10.105 INFO:tasks.rados.rados.0.smithi203.stdout:3847: done (1 left) 2024-09-17T15:51:10.105 INFO:tasks.rados.rados.0.smithi203.stdout:3848: done (0 left) 2024-09-17T15:51:10.105 INFO:tasks.rados.rados.0.smithi203.stdout:3849: snap_remove snap 401 2024-09-17T15:51:10.883 INFO:tasks.rados.rados.0.smithi203.stdout:3849: done (0 left) 2024-09-17T15:51:10.883 INFO:tasks.rados.rados.0.smithi203.stdout:3850: write oid 21 current snap is 410 2024-09-17T15:51:10.883 INFO:tasks.rados.rados.0.smithi203.stdout:3850: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-17T15:51:10.910 INFO:tasks.rados.rados.0.smithi203.stdout:3850: writing smithi20328086-21 from 706257 to 1362137 tid 1 2024-09-17T15:51:10.930 INFO:tasks.rados.rados.0.smithi203.stdout:3850: writing smithi20328086-21 from 1890820 to 2395285 tid 2 2024-09-17T15:51:10.930 INFO:tasks.rados.rados.0.smithi203.stdout:3851: write (excl) oid 3 current snap is 410 2024-09-17T15:51:10.930 INFO:tasks.rados.rados.0.smithi203.stdout:3851: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-17T15:51:10.950 INFO:tasks.rados.rados.0.smithi203.stdout:3851: writing smithi20328086-3 from 723064 to 1454193 tid 1 2024-09-17T15:51:10.964 INFO:tasks.rados.rados.0.smithi203.stdout:3851: writing smithi20328086-3 from 1949302 to 2713778 tid 2 2024-09-17T15:51:10.964 INFO:tasks.rados.rados.0.smithi203.stdout:3852: write (excl) oid 35 current snap is 410 2024-09-17T15:51:10.964 INFO:tasks.rados.rados.0.smithi203.stdout:3852: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-17T15:51:10.973 INFO:tasks.rados.rados.0.smithi203.stdout:3852: writing smithi20328086-35 from 739871 to 1146249 tid 1 2024-09-17T15:51:10.980 INFO:tasks.rados.rados.0.smithi203.stdout:3852: writing smithi20328086-35 from 1607784 to 2115918 tid 2 2024-09-17T15:51:10.980 INFO:tasks.rados.rados.0.smithi203.stdout:3853: write oid 5 current snap is 410 2024-09-17T15:51:10.980 INFO:tasks.rados.rados.0.smithi203.stdout:3853: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-17T15:51:10.989 INFO:tasks.rados.rados.0.smithi203.stdout:3853: writing smithi20328086-5 from 756678 to 1354658 tid 1 2024-09-17T15:51:10.996 INFO:tasks.rados.rados.0.smithi203.stdout:3853: writing smithi20328086-5 from 1782619 to 2550764 tid 2 2024-09-17T15:51:10.996 INFO:tasks.rados.rados.0.smithi203.stdout:3854: snap_remove snap 395 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3854: done (4 left) 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3855: snap_create 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: finishing write tid 1 to smithi20328086-21 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: finishing write tid 2 to smithi20328086-21 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: finishing write tid 3 to smithi20328086-21 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: finishing write tid 4 to smithi20328086-21 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: oid 21 updating version 0 to 2105 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: oid 21 version 2105 is already newer than 2100 2024-09-17T15:51:11.895 INFO:tasks.rados.rados.0.smithi203.stdout:3850: oid 21 updating version 2105 to 2106 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 21 v 2106 (ObjNum 1351 snap 410 seq_num 1351) dirty exists 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3850: left oid 21 (ObjNum 1351 snap 410 seq_num 1351) 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: finishing write tid 1 to smithi20328086-3 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: finishing write tid 2 to smithi20328086-3 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: finishing write tid 3 to smithi20328086-3 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: finishing write tid 4 to smithi20328086-3 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: oid 3 updating version 0 to 1899 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: oid 3 updating version 1899 to 1900 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: oid 3 updating version 1900 to 1901 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1901 (ObjNum 1352 snap 410 seq_num 1352) dirty exists 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3851: left oid 3 (ObjNum 1352 snap 410 seq_num 1352) 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3852: finishing write tid 1 to smithi20328086-35 2024-09-17T15:51:11.896 INFO:tasks.rados.rados.0.smithi203.stdout:3852: finishing write tid 2 to smithi20328086-35 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: finishing write tid 3 to smithi20328086-35 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: finishing write tid 4 to smithi20328086-35 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: oid 35 updating version 0 to 1697 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: oid 35 version 1697 is already newer than 1695 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: oid 35 version 1697 is already newer than 1696 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1697 (ObjNum 1353 snap 410 seq_num 1353) dirty exists 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3852: left oid 35 (ObjNum 1353 snap 410 seq_num 1353) 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: finishing write tid 1 to smithi20328086-5 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: finishing write tid 2 to smithi20328086-5 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: finishing write tid 3 to smithi20328086-5 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: finishing write tid 4 to smithi20328086-5 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: oid 5 updating version 0 to 1438 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: oid 5 updating version 1438 to 1440 2024-09-17T15:51:11.897 INFO:tasks.rados.rados.0.smithi203.stdout:3853: oid 5 version 1440 is already newer than 1439 2024-09-17T15:51:11.898 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1440 (ObjNum 1354 snap 410 seq_num 1354) dirty exists 2024-09-17T15:51:11.898 INFO:tasks.rados.rados.0.smithi203.stdout:3853: left oid 5 (ObjNum 1354 snap 410 seq_num 1354) 2024-09-17T15:51:12.905 INFO:tasks.rados.rados.0.smithi203.stdout:3850: done (4 left) 2024-09-17T15:51:12.905 INFO:tasks.rados.rados.0.smithi203.stdout:3851: done (3 left) 2024-09-17T15:51:12.906 INFO:tasks.rados.rados.0.smithi203.stdout:3852: done (2 left) 2024-09-17T15:51:12.906 INFO:tasks.rados.rados.0.smithi203.stdout:3853: done (1 left) 2024-09-17T15:51:12.906 INFO:tasks.rados.rados.0.smithi203.stdout:3855: done (0 left) 2024-09-17T15:51:12.906 INFO:tasks.rados.rados.0.smithi203.stdout:3856: snap_create 2024-09-17T15:51:13.907 INFO:tasks.rados.rados.0.smithi203.stdout:3856: done (0 left) 2024-09-17T15:51:13.907 INFO:tasks.rados.rados.0.smithi203.stdout:3857: read oid 6 snap -1 2024-09-17T15:51:13.907 INFO:tasks.rados.rados.0.smithi203.stdout:3857: expect deleted 2024-09-17T15:51:13.907 INFO:tasks.rados.rados.0.smithi203.stdout:3858: write oid 49 current snap is 412 2024-09-17T15:51:13.907 INFO:tasks.rados.rados.0.smithi203.stdout:3858: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-17T15:51:13.921 INFO:tasks.rados.rados.0.smithi203.stdout:3858: writing smithi20328086-49 from 773485 to 1446714 tid 1 2024-09-17T15:51:13.934 INFO:tasks.rados.rados.0.smithi203.stdout:3858: writing smithi20328086-49 from 2124748 to 2636551 tid 2 2024-09-17T15:51:13.935 INFO:tasks.rados.rados.0.smithi203.stdout:3859: write (excl) oid 31 current snap is 412 2024-09-17T15:51:13.935 INFO:tasks.rados.rados.0.smithi203.stdout:3859: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-09-17T15:51:13.950 INFO:tasks.rados.rados.0.smithi203.stdout:3859: writing smithi20328086-31 from 790292 to 1538770 tid 1 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3859: writing smithi20328086-31 from 2183230 to 2790292 tid 2 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3860: read oid 12 snap -1 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3860: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3861: copy_from oid 20 from oid 24 current snap is 412 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: finishing write tid 1 to smithi20328086-49 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: finishing write tid 2 to smithi20328086-49 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: finishing write tid 3 to smithi20328086-49 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: finishing write tid 4 to smithi20328086-49 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: oid 49 updating version 0 to 2119 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: oid 49 version 2119 is already newer than 2118 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:3858: oid 49 updating version 2119 to 2120 2024-09-17T15:51:13.964 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 2120 (ObjNum 1355 snap 412 seq_num 1355) dirty exists 2024-09-17T15:51:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:3858: left oid 49 (ObjNum 1355 snap 412 seq_num 1355) 2024-09-17T15:51:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:3859: finishing write tid 1 to smithi20328086-31 2024-09-17T15:51:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:3857: done (4 left) 2024-09-17T15:51:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:3858: done (3 left) 2024-09-17T15:51:13.965 INFO:tasks.rados.rados.0.smithi203.stdout:3862: delete oid 2 current snap is 412 2024-09-17T15:51:13.970 INFO:tasks.rados.rados.0.smithi203.stdout:3860: expect (ObjNum 1227 snap 368 seq_num 1227) 2024-09-17T15:51:14.066 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2090883 2024-09-17T15:51:14.072 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:14.111 INFO:tasks.rados.rados.0.smithi203.stdout:3859: finishing write tid 2 to smithi20328086-31 2024-09-17T15:51:14.111 INFO:tasks.rados.rados.0.smithi203.stdout:3859: finishing write tid 3 to smithi20328086-31 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: finishing write tid 4 to smithi20328086-31 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: oid 31 updating version 0 to 1682 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: oid 31 version 1682 is already newer than 1680 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: oid 31 version 1682 is already newer than 1681 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 31 v 1682 (ObjNum 1356 snap 412 seq_num 1356) dirty exists 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: left oid 31 (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3861: finishing copy_from racing read to smithi20328086-20 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3861: finishing copy_from to smithi20328086-20 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 20 v 1575 (ObjNum 1301 snap 400 seq_num 1301) dirty exists 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3859: done (3 left) 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3860: done (2 left) 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3861: done (1 left) 2024-09-17T15:51:14.112 INFO:tasks.rados.rados.0.smithi203.stdout:3862: done (0 left) 2024-09-17T15:51:14.113 INFO:tasks.rados.rados.0.smithi203.stdout:3863: read oid 5 snap -1 2024-09-17T15:51:14.113 INFO:tasks.rados.rados.0.smithi203.stdout:3863: expect (ObjNum 1354 snap 410 seq_num 1354) 2024-09-17T15:51:14.113 INFO:tasks.rados.rados.0.smithi203.stdout:3864: copy_from oid 46 from oid 8 current snap is 412 2024-09-17T15:51:14.113 INFO:tasks.rados.rados.0.smithi203.stdout:3865: write oid 35 current snap is 412 2024-09-17T15:51:14.113 INFO:tasks.rados.rados.0.smithi203.stdout:3865: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-17T15:51:14.122 INFO:tasks.rados.rados.0.smithi203.stdout:3865: writing smithi20328086-35 from 423906 to 922882 tid 1 2024-09-17T15:51:14.137 INFO:tasks.rados.rados.0.smithi203.stdout:3865: writing smithi20328086-35 from 1500194 to 2159324 tid 2 2024-09-17T15:51:14.145 INFO:tasks.rados.rados.0.smithi203.stdout:3865: writing smithi20328086-35 from 2609483 to 2823906 tid 3 2024-09-17T15:51:14.145 INFO:tasks.rados.rados.0.smithi203.stdout:3866: copy_from oid 24 from oid 2 current snap is 412 2024-09-17T15:51:14.145 INFO:tasks.rados.rados.0.smithi203.stdout:3863: expect (ObjNum 1354 snap 410 seq_num 1354) 2024-09-17T15:51:14.145 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:14.220 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3864: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3864: finishing copy_from to smithi20328086-46 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1653 (ObjNum 1341 snap 407 seq_num 1341) dirty exists 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3865: finishing write tid 1 to smithi20328086-35 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3865: finishing write tid 2 to smithi20328086-35 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3866: finishing copy_from to smithi20328086-24 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3866: got expected ENOENT (src dne) 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3866: finishing copy_from racing read to smithi20328086-24 2024-09-17T15:51:14.252 INFO:tasks.rados.rados.0.smithi203.stdout:3865: finishing write tid 3 to smithi20328086-35 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: finishing write tid 4 to smithi20328086-35 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: finishing write tid 5 to smithi20328086-35 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: oid 35 updating version 0 to 1698 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: oid 35 updating version 1698 to 1700 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: oid 35 version 1700 is already newer than 1699 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: oid 35 updating version 1700 to 1701 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1701 (ObjNum 1358 snap 412 seq_num 1358) dirty exists 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: left oid 35 (ObjNum 1358 snap 412 seq_num 1358) 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3863: done (3 left) 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3864: done (2 left) 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3865: done (1 left) 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3866: done (0 left) 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3867: copy_from oid 35 from oid 18 current snap is 412 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3868: copy_from oid 37 from oid 50 current snap is 412 2024-09-17T15:51:14.253 INFO:tasks.rados.rados.0.smithi203.stdout:3869: write oid 13 current snap is 412 2024-09-17T15:51:14.254 INFO:tasks.rados.rados.0.smithi203.stdout:3869: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-09-17T15:51:14.265 INFO:tasks.rados.rados.0.smithi203.stdout:3869: writing smithi20328086-13 from 440713 to 1014938 tid 1 2024-09-17T15:51:14.276 INFO:tasks.rados.rados.0.smithi203.stdout:3869: writing smithi20328086-13 from 1442323 to 1961464 tid 2 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3869: writing smithi20328086-13 from 2636906 to 2840713 tid 3 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3870: delete oid 33 current snap is 412 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3867: finishing copy_from to smithi20328086-35 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3867: got expected ENOENT (src dne) 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3867: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3868: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:3868: finishing copy_from to smithi20328086-37 2024-09-17T15:51:14.289 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 1743 (ObjNum 1324 snap 404 seq_num 1324) dirty exists 2024-09-17T15:51:14.290 INFO:tasks.rados.rados.0.smithi203.stdout:3869: finishing write tid 1 to smithi20328086-13 2024-09-17T15:51:14.290 INFO:tasks.rados.rados.0.smithi203.stdout:3869: finishing write tid 2 to smithi20328086-13 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3867: done (3 left) 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3868: done (2 left) 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3870: done (1 left) 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3871: read oid 9 snap 405 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3871: expect (ObjNum 1326 snap 405 seq_num 1326) 2024-09-17T15:51:14.294 INFO:tasks.rados.rados.0.smithi203.stdout:3872: snap_create 2024-09-17T15:51:14.295 INFO:tasks.rados.rados.0.smithi203.stdout:3869: finishing write tid 3 to smithi20328086-13 2024-09-17T15:51:14.295 INFO:tasks.rados.rados.0.smithi203.stdout:3869: finishing write tid 4 to smithi20328086-13 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: finishing write tid 5 to smithi20328086-13 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: oid 13 updating version 0 to 1422 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: oid 13 version 1422 is already newer than 1419 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: oid 13 version 1422 is already newer than 1420 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: oid 13 version 1422 is already newer than 1421 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 13 v 1422 (ObjNum 1359 snap 412 seq_num 1359) dirty exists 2024-09-17T15:51:14.296 INFO:tasks.rados.rados.0.smithi203.stdout:3869: left oid 13 (ObjNum 1359 snap 412 seq_num 1359) 2024-09-17T15:51:14.301 INFO:tasks.rados.rados.0.smithi203.stdout:3871: expect (ObjNum 1326 snap 405 seq_num 1326) 2024-09-17T15:51:14.353 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:14.921 INFO:tasks.rados.rados.0.smithi203.stdout:3869: done (2 left) 2024-09-17T15:51:14.921 INFO:tasks.rados.rados.0.smithi203.stdout:3871: done (1 left) 2024-09-17T15:51:14.921 INFO:tasks.rados.rados.0.smithi203.stdout:3872: done (0 left) 2024-09-17T15:51:14.921 INFO:tasks.rados.rados.0.smithi203.stdout:3873: write oid 50 current snap is 413 2024-09-17T15:51:14.922 INFO:tasks.rados.rados.0.smithi203.stdout:3873: seq_num 1361 ranges {474327=441076,1675640=522810,2808105=66222} 2024-09-17T15:51:14.934 INFO:tasks.rados.rados.0.smithi203.stdout:3873: writing smithi20328086-50 from 474327 to 915403 tid 1 2024-09-17T15:51:14.952 INFO:tasks.rados.rados.0.smithi203.stdout:3873: writing smithi20328086-50 from 1675640 to 2198450 tid 2 2024-09-17T15:51:14.962 INFO:tasks.rados.rados.0.smithi203.stdout:3873: writing smithi20328086-50 from 2808105 to 2874327 tid 3 2024-09-17T15:51:14.962 INFO:tasks.rados.rados.0.smithi203.stdout:3874: snap_create 2024-09-17T15:51:14.962 INFO:tasks.rados.rados.0.smithi203.stdout:3873: finishing write tid 1 to smithi20328086-50 2024-09-17T15:51:14.962 INFO:tasks.rados.rados.0.smithi203.stdout:3873: finishing write tid 2 to smithi20328086-50 2024-09-17T15:51:14.967 INFO:tasks.rados.rados.0.smithi203.stdout:3873: finishing write tid 3 to smithi20328086-50 2024-09-17T15:51:14.967 INFO:tasks.rados.rados.0.smithi203.stdout:3873: finishing write tid 4 to smithi20328086-50 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: finishing write tid 5 to smithi20328086-50 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: oid 50 updating version 0 to 1748 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: oid 50 updating version 1748 to 1750 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: oid 50 version 1750 is already newer than 1749 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: oid 50 version 1750 is already newer than 1747 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1750 (ObjNum 1361 snap 413 seq_num 1361) dirty exists 2024-09-17T15:51:14.968 INFO:tasks.rados.rados.0.smithi203.stdout:3873: left oid 50 (ObjNum 1361 snap 413 seq_num 1361) 2024-09-17T15:51:15.885 INFO:tasks.rados.rados.0.smithi203.stdout:3873: done (1 left) 2024-09-17T15:51:15.885 INFO:tasks.rados.rados.0.smithi203.stdout:3874: done (0 left) 2024-09-17T15:51:15.886 INFO:tasks.rados.rados.0.smithi203.stdout:3875: write oid 33 current snap is 414 2024-09-17T15:51:15.886 INFO:tasks.rados.rados.0.smithi203.stdout:3875: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-09-17T15:51:15.900 INFO:tasks.rados.rados.0.smithi203.stdout:3875: writing smithi20328086-33 from 491134 to 1007459 tid 1 2024-09-17T15:51:15.917 INFO:tasks.rados.rados.0.smithi203.stdout:3875: writing smithi20328086-33 from 1734122 to 2400590 tid 2 2024-09-17T15:51:15.917 INFO:tasks.rados.rados.0.smithi203.stdout:3876: delete oid 10 current snap is 414 2024-09-17T15:51:15.917 INFO:tasks.rados.rados.0.smithi203.stdout:3875: finishing write tid 1 to smithi20328086-33 2024-09-17T15:51:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:3875: finishing write tid 2 to smithi20328086-33 2024-09-17T15:51:15.925 INFO:tasks.rados.rados.0.smithi203.stdout:3875: finishing write tid 3 to smithi20328086-33 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:3875: finishing write tid 4 to smithi20328086-33 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:3875: oid 33 updating version 0 to 1443 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:3875: oid 33 updating version 1443 to 1444 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:3875: oid 33 version 1444 is already newer than 1442 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 33 v 1444 (ObjNum 1362 snap 414 seq_num 1362) dirty exists 2024-09-17T15:51:15.926 INFO:tasks.rados.rados.0.smithi203.stdout:3875: left oid 33 (ObjNum 1362 snap 414 seq_num 1362) 2024-09-17T15:51:15.935 INFO:tasks.rados.rados.0.smithi203.stdout:3875: done (1 left) 2024-09-17T15:51:15.935 INFO:tasks.rados.rados.0.smithi203.stdout:3876: done (0 left) 2024-09-17T15:51:15.935 INFO:tasks.rados.rados.0.smithi203.stdout:3877: snap_create 2024-09-17T15:51:16.939 INFO:tasks.rados.rados.0.smithi203.stdout:3877: done (0 left) 2024-09-17T15:51:16.939 INFO:tasks.rados.rados.0.smithi203.stdout:3878: delete oid 23 current snap is 415 2024-09-17T15:51:16.949 INFO:tasks.rados.rados.0.smithi203.stdout:3878: done (0 left) 2024-09-17T15:51:16.949 INFO:tasks.rados.rados.0.smithi203.stdout:3879: read oid 40 snap -1 2024-09-17T15:51:16.949 INFO:tasks.rados.rados.0.smithi203.stdout:3879: expect deleted 2024-09-17T15:51:16.949 INFO:tasks.rados.rados.0.smithi203.stdout:3880: write (excl) oid 22 current snap is 415 2024-09-17T15:51:16.949 INFO:tasks.rados.rados.0.smithi203.stdout:3880: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-17T15:51:16.969 INFO:tasks.rados.rados.0.smithi203.stdout:3880: writing smithi20328086-22 from 541555 to 1283627 tid 1 2024-09-17T15:51:16.986 INFO:tasks.rados.rados.0.smithi203.stdout:3880: writing smithi20328086-22 from 1793215 to 2323363 tid 2 2024-09-17T15:51:16.996 INFO:tasks.rados.rados.0.smithi203.stdout:3880: writing smithi20328086-22 from 2801444 to 2941555 tid 3 2024-09-17T15:51:16.996 INFO:tasks.rados.rados.0.smithi203.stdout:3881: write (excl) oid 23 current snap is 415 2024-09-17T15:51:16.996 INFO:tasks.rados.rados.0.smithi203.stdout:3881: seq_num 1366 ranges {558362=417321,1451697=673806,2712514=245848} 2024-09-17T15:51:17.008 INFO:tasks.rados.rados.0.smithi203.stdout:3881: writing smithi20328086-23 from 558362 to 975683 tid 1 2024-09-17T15:51:17.020 INFO:tasks.rados.rados.0.smithi203.stdout:3881: writing smithi20328086-23 from 1451697 to 2125503 tid 2 2024-09-17T15:51:17.028 INFO:tasks.rados.rados.0.smithi203.stdout:3881: writing smithi20328086-23 from 2712514 to 2958362 tid 3 2024-09-17T15:51:17.028 INFO:tasks.rados.rados.0.smithi203.stdout:3882: snap_remove snap 412 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3882: done (3 left) 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3883: read oid 31 snap -1 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3883: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3884: snap_create 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3880: finishing write tid 1 to smithi20328086-22 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3880: finishing write tid 2 to smithi20328086-22 2024-09-17T15:51:17.946 INFO:tasks.rados.rados.0.smithi203.stdout:3880: finishing write tid 3 to smithi20328086-22 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: finishing write tid 4 to smithi20328086-22 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: finishing write tid 5 to smithi20328086-22 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: oid 22 updating version 0 to 1686 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: oid 22 version 1686 is already newer than 1683 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: oid 22 version 1686 is already newer than 1684 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: oid 22 version 1686 is already newer than 1685 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1686 (ObjNum 1365 snap 415 seq_num 1365) dirty exists 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3880: left oid 22 (ObjNum 1365 snap 415 seq_num 1365) 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3881: finishing write tid 1 to smithi20328086-23 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3881: finishing write tid 2 to smithi20328086-23 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3881: finishing write tid 3 to smithi20328086-23 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3881: finishing write tid 4 to smithi20328086-23 2024-09-17T15:51:17.947 INFO:tasks.rados.rados.0.smithi203.stdout:3881: finishing write tid 5 to smithi20328086-23 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:3881: oid 23 updating version 0 to 1728 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:3881: oid 23 version 1728 is already newer than 1726 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:3881: oid 23 version 1728 is already newer than 1727 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:3881: oid 23 updating version 1728 to 1729 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1729 (ObjNum 1366 snap 415 seq_num 1366) dirty exists 2024-09-17T15:51:17.948 INFO:tasks.rados.rados.0.smithi203.stdout:3881: left oid 23 (ObjNum 1366 snap 415 seq_num 1366) 2024-09-17T15:51:17.968 INFO:tasks.rados.rados.0.smithi203.stdout:3883: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:18.063 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 571514 2024-09-17T15:51:18.103 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:18.957 INFO:tasks.rados.rados.0.smithi203.stdout:3879: done (4 left) 2024-09-17T15:51:18.957 INFO:tasks.rados.rados.0.smithi203.stdout:3880: done (3 left) 2024-09-17T15:51:18.957 INFO:tasks.rados.rados.0.smithi203.stdout:3881: done (2 left) 2024-09-17T15:51:18.958 INFO:tasks.rados.rados.0.smithi203.stdout:3883: done (1 left) 2024-09-17T15:51:18.958 INFO:tasks.rados.rados.0.smithi203.stdout:3884: done (0 left) 2024-09-17T15:51:18.958 INFO:tasks.rados.rados.0.smithi203.stdout:3885: copy_from oid 39 from oid 9 current snap is 416 2024-09-17T15:51:18.958 INFO:tasks.rados.rados.0.smithi203.stdout:3886: delete oid 30 current snap is 416 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3886: done (1 left) 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3887: read oid 15 snap 389 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3887: expect (ObjNum 1268 snap 385 seq_num 1268) 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3888: rollback oid 14 current snap is 416 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 14 to 410 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3889: write (excl) oid 36 current snap is 416 2024-09-17T15:51:18.970 INFO:tasks.rados.rados.0.smithi203.stdout:3889: seq_num 1368 ranges {591976=567819,1852308=677475} 2024-09-17T15:51:18.982 INFO:tasks.rados.rados.0.smithi203.stdout:3889: writing smithi20328086-36 from 591976 to 1159795 tid 1 2024-09-17T15:51:18.997 INFO:tasks.rados.rados.0.smithi203.stdout:3889: writing smithi20328086-36 from 1852308 to 2529783 tid 2 2024-09-17T15:51:18.997 INFO:tasks.rados.rados.0.smithi203.stdout:3890: write oid 3 current snap is 416 2024-09-17T15:51:18.997 INFO:tasks.rados.rados.0.smithi203.stdout:3890: seq_num 1369 ranges {608783=759421,2027143=421133} 2024-09-17T15:51:19.006 INFO:tasks.rados.rados.0.smithi203.stdout:3890: writing smithi20328086-3 from 608783 to 1368204 tid 1 2024-09-17T15:51:19.017 INFO:tasks.rados.rados.0.smithi203.stdout:3890: writing smithi20328086-3 from 2027143 to 2448276 tid 2 2024-09-17T15:51:19.017 INFO:tasks.rados.rados.0.smithi203.stdout:3891: rollback oid 19 current snap is 416 2024-09-17T15:51:19.017 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 19 to 411 2024-09-17T15:51:19.017 INFO:tasks.rados.rados.0.smithi203.stdout:3887: expect (ObjNum 1268 snap 385 seq_num 1268) 2024-09-17T15:51:19.064 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2038851 2024-09-17T15:51:19.165 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2609074 2024-09-17T15:51:19.177 INFO:tasks.rados.rados.0.smithi203.stdout:3888: finishing rollback tid 1 to smithi20328086-14 2024-09-17T15:51:19.177 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1669 (ObjNum 65 snap 0 seq_num 1) dirty dne 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3885: finishing copy_from racing read to smithi20328086-39 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3885: finishing copy_from to smithi20328086-39 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 1566 (ObjNum 1332 snap 406 seq_num 1332) dirty exists 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3889: finishing write tid 1 to smithi20328086-36 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3885: done (5 left) 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3887: done (4 left) 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3888: done (3 left) 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3892: copy_from oid 25 from oid 2 current snap is 416 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3889: finishing write tid 2 to smithi20328086-36 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3889: finishing write tid 3 to smithi20328086-36 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3889: finishing write tid 4 to smithi20328086-36 2024-09-17T15:51:19.178 INFO:tasks.rados.rados.0.smithi203.stdout:3889: oid 36 updating version 0 to 2122 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3889: oid 36 updating version 2122 to 2123 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3889: oid 36 version 2123 is already newer than 2121 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 2123 (ObjNum 1368 snap 416 seq_num 1368) dirty exists 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3889: left oid 36 (ObjNum 1368 snap 416 seq_num 1368) 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: finishing write tid 1 to smithi20328086-3 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3891: finishing rollback tid 1 to smithi20328086-19 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 19 v 1974 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: finishing write tid 2 to smithi20328086-3 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: finishing write tid 3 to smithi20328086-3 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: finishing write tid 4 to smithi20328086-3 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: oid 3 updating version 0 to 1902 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: oid 3 updating version 1902 to 1904 2024-09-17T15:51:19.179 INFO:tasks.rados.rados.0.smithi203.stdout:3890: oid 3 version 1904 is already newer than 1903 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1904 (ObjNum 1369 snap 416 seq_num 1369) dirty exists 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:3890: left oid 3 (ObjNum 1369 snap 416 seq_num 1369) 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:3889: done (3 left) 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:3890: done (2 left) 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:3891: done (1 left) 2024-09-17T15:51:19.180 INFO:tasks.rados.rados.0.smithi203.stdout:3893: snap_remove snap 407 2024-09-17T15:51:19.963 INFO:tasks.rados.rados.0.smithi203.stdout:3893: done (1 left) 2024-09-17T15:51:19.963 INFO:tasks.rados.rados.0.smithi203.stdout:3894: snap_remove snap 405 2024-09-17T15:51:20.592 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-17T15:51:20.592 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T15:51:20.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T15:51:20.596 INFO:tasks.ceph.mon.c:Started 2024-09-17T15:51:20.596 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-17T15:51:20.596 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-17T15:51:20.596 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-17T15:51:26.634 INFO:tasks.rados.rados.0.smithi203.stdout:3894: done (1 left) 2024-09-17T15:51:26.635 INFO:tasks.rados.rados.0.smithi203.stdout:3895: delete oid 27 current snap is 416 2024-09-17T15:51:26.635 INFO:tasks.rados.rados.0.smithi203.stdout:3892: finishing copy_from to smithi20328086-25 2024-09-17T15:51:26.635 INFO:tasks.rados.rados.0.smithi203.stdout:3892: got expected ENOENT (src dne) 2024-09-17T15:51:26.635 INFO:tasks.rados.rados.0.smithi203.stdout:3892: finishing copy_from racing read to smithi20328086-25 2024-09-17T15:51:26.646 INFO:tasks.rados.rados.0.smithi203.stdout:3892: done (1 left) 2024-09-17T15:51:26.646 INFO:tasks.rados.rados.0.smithi203.stdout:3895: done (0 left) 2024-09-17T15:51:26.646 INFO:tasks.rados.rados.0.smithi203.stdout:3896: rollback oid 48 current snap is 416 2024-09-17T15:51:26.646 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 413 2024-09-17T15:51:26.646 INFO:tasks.rados.rados.0.smithi203.stdout:3897: delete oid 49 current snap is 416 2024-09-17T15:51:26.667 INFO:tasks.rados.rados.0.smithi203.stdout:3896: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:51:26.667 INFO:tasks.rados.rados.0.smithi203.stdout:3896: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:51:26.668 INFO:tasks.rados.rados.0.smithi203.stdout:3897: done (1 left) 2024-09-17T15:51:26.668 INFO:tasks.rados.rados.0.smithi203.stdout:3898: read oid 18 snap -1 2024-09-17T15:51:26.668 INFO:tasks.rados.rados.0.smithi203.stdout:3898: expect deleted 2024-09-17T15:51:26.668 INFO:tasks.rados.rados.0.smithi203.stdout:3899: write oid 27 current snap is 416 2024-09-17T15:51:26.668 INFO:tasks.rados.rados.0.smithi203.stdout:3899: seq_num 1372 ranges {659204=585168,1686236=684813} 2024-09-17T15:51:26.676 INFO:tasks.rados.rados.0.smithi203.stdout:3899: writing smithi20328086-27 from 659204 to 1244372 tid 1 2024-09-17T15:51:26.684 INFO:tasks.rados.rados.0.smithi203.stdout:3899: writing smithi20328086-27 from 1686236 to 2371049 tid 2 2024-09-17T15:51:26.684 INFO:tasks.rados.rados.0.smithi203.stdout:3900: write oid 22 current snap is 416 2024-09-17T15:51:26.684 INFO:tasks.rados.rados.0.smithi203.stdout:3900: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-09-17T15:51:26.692 INFO:tasks.rados.rados.0.smithi203.stdout:3900: writing smithi20328086-22 from 676011 to 1336428 tid 1 2024-09-17T15:51:26.698 INFO:tasks.rados.rados.0.smithi203.stdout:3900: writing smithi20328086-22 from 1744718 to 2173189 tid 2 2024-09-17T15:51:26.704 INFO:tasks.rados.rados.0.smithi203.stdout:3900: writing smithi20328086-22 from 2788122 to 3076011 tid 3 2024-09-17T15:51:26.704 INFO:tasks.rados.rados.0.smithi203.stdout:3901: read oid 44 snap 408 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3901: expect (ObjNum 1292 snap 396 seq_num 1292) 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3902: copy_from oid 28 from oid 40 current snap is 416 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3898: done (5 left) 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3903: read oid 31 snap -1 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3903: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3904: read oid 1 snap -1 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3904: expect (ObjNum 1257 snap 380 seq_num 1257) 2024-09-17T15:51:26.705 INFO:tasks.rados.rados.0.smithi203.stdout:3905: snap_remove snap 415 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3905: done (7 left) 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3906: read oid 49 snap -1 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3906: expect deleted 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3907: rollback oid 10 current snap is 416 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 10 to 388 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3896: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 2043 (ObjNum 1302 snap 400 seq_num 1302) dirty exists 2024-09-17T15:51:27.642 INFO:tasks.rados.rados.0.smithi203.stdout:3899: finishing write tid 1 to smithi20328086-27 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: finishing write tid 2 to smithi20328086-27 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: finishing write tid 3 to smithi20328086-27 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: finishing write tid 4 to smithi20328086-27 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: oid 27 updating version 0 to 1908 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: oid 27 version 1908 is already newer than 1906 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: oid 27 version 1908 is already newer than 1907 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 27 v 1908 (ObjNum 1372 snap 416 seq_num 1372) dirty exists 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3899: left oid 27 (ObjNum 1372 snap 416 seq_num 1372) 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3900: finishing write tid 1 to smithi20328086-22 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3900: finishing write tid 2 to smithi20328086-22 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3900: finishing write tid 3 to smithi20328086-22 2024-09-17T15:51:27.643 INFO:tasks.rados.rados.0.smithi203.stdout:3900: finishing write tid 4 to smithi20328086-22 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3896: done (8 left) 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3899: done (7 left) 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3908: read oid 47 snap -1 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3908: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3909: write (excl) oid 36 current snap is 416 2024-09-17T15:51:27.644 INFO:tasks.rados.rados.0.smithi203.stdout:3909: seq_num 1374 ranges {692818=735666,2203200=688482} 2024-09-17T15:51:27.654 INFO:tasks.rados.rados.0.smithi203.stdout:3909: writing smithi20328086-36 from 692818 to 1428484 tid 1 2024-09-17T15:51:27.669 INFO:tasks.rados.rados.0.smithi203.stdout:3909: writing smithi20328086-36 from 2203200 to 2891682 tid 2 2024-09-17T15:51:27.669 INFO:tasks.rados.rados.0.smithi203.stdout:3910: snap_remove snap 411 2024-09-17T15:51:28.646 INFO:tasks.rados.rados.0.smithi203.stdout:3910: done (9 left) 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3911: rollback oid 11 current snap is 416 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 388 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: finishing write tid 5 to smithi20328086-22 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: oid 22 updating version 0 to 1687 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: oid 22 updating version 1687 to 1690 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: oid 22 version 1690 is already newer than 1689 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: oid 22 version 1690 is already newer than 1688 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1690 (ObjNum 1373 snap 416 seq_num 1373) dirty exists 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3900: left oid 22 (ObjNum 1373 snap 416 seq_num 1373) 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:3904: expect (ObjNum 1257 snap 380 seq_num 1257) 2024-09-17T15:51:28.647 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:28.765 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 23143 2024-09-17T15:51:28.867 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2657240 2024-09-17T15:51:28.879 INFO:tasks.rados.rados.0.smithi203.stdout:3902: finishing copy_from to smithi20328086-28 2024-09-17T15:51:28.879 INFO:tasks.rados.rados.0.smithi203.stdout:3902: got expected ENOENT (src dne) 2024-09-17T15:51:28.879 INFO:tasks.rados.rados.0.smithi203.stdout:3902: finishing copy_from racing read to smithi20328086-28 2024-09-17T15:51:28.879 INFO:tasks.rados.rados.0.smithi203.stdout:3903: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:28.925 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:29.019 INFO:tasks.rados.rados.0.smithi203.stdout:3901: expect (ObjNum 1292 snap 396 seq_num 1292) 2024-09-17T15:51:29.019 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:3907: finishing rollback tid 1 to smithi20328086-10 2024-09-17T15:51:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 10 v 1423 (ObjNum 113 snap 0 seq_num 3892337440) dirty dne 2024-09-17T15:51:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:3908: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:51:29.336 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:29.374 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:29.423 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1532012 2024-09-17T15:51:29.444 INFO:tasks.rados.rados.0.smithi203.stdout:3909: finishing write tid 1 to smithi20328086-36 2024-09-17T15:51:29.444 INFO:tasks.rados.rados.0.smithi203.stdout:3909: finishing write tid 2 to smithi20328086-36 2024-09-17T15:51:29.444 INFO:tasks.rados.rados.0.smithi203.stdout:3909: finishing write tid 3 to smithi20328086-36 2024-09-17T15:51:29.444 INFO:tasks.rados.rados.0.smithi203.stdout:3909: finishing write tid 4 to smithi20328086-36 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3909: oid 36 updating version 0 to 2138 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3909: oid 36 version 2138 is already newer than 2136 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3909: oid 36 version 2138 is already newer than 2137 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 36 v 2138 (ObjNum 1374 snap 416 seq_num 1374) dirty exists 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3909: left oid 36 (ObjNum 1374 snap 416 seq_num 1374) 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3911: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3900: done (9 left) 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3901: done (8 left) 2024-09-17T15:51:29.445 INFO:tasks.rados.rados.0.smithi203.stdout:3902: done (7 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3903: done (6 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3904: done (5 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3906: done (4 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3907: done (3 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3908: done (2 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3909: done (1 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3911: done (0 left) 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3912: write oid 28 current snap is 416 2024-09-17T15:51:29.446 INFO:tasks.rados.rados.0.smithi203.stdout:3912: seq_num 1375 ranges {709625=410915,1745329=432140,2726615=383010} 2024-09-17T15:51:29.462 INFO:tasks.rados.rados.0.smithi203.stdout:3912: writing smithi20328086-28 from 709625 to 1120540 tid 1 2024-09-17T15:51:29.480 INFO:tasks.rados.rados.0.smithi203.stdout:3912: writing smithi20328086-28 from 1745329 to 2177469 tid 2 2024-09-17T15:51:29.495 INFO:tasks.rados.rados.0.smithi203.stdout:3912: writing smithi20328086-28 from 2726615 to 3109625 tid 3 2024-09-17T15:51:29.495 INFO:tasks.rados.rados.0.smithi203.stdout:3913: read oid 6 snap -1 2024-09-17T15:51:29.495 INFO:tasks.rados.rados.0.smithi203.stdout:3913: expect deleted 2024-09-17T15:51:29.495 INFO:tasks.rados.rados.0.smithi203.stdout:3912: finishing write tid 1 to smithi20328086-28 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3912: finishing write tid 2 to smithi20328086-28 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3914: read oid 29 snap -1 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3914: expect (ObjNum 1338 snap 407 seq_num 1338) 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3915: rollback oid 23 current snap is 416 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 23 to 389 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3916: write (excl) oid 46 current snap is 416 2024-09-17T15:51:29.496 INFO:tasks.rados.rados.0.smithi203.stdout:3916: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-09-17T15:51:29.511 INFO:tasks.rados.rados.0.smithi203.stdout:3916: writing smithi20328086-46 from 726432 to 1212596 tid 1 2024-09-17T15:51:29.528 INFO:tasks.rados.rados.0.smithi203.stdout:3916: writing smithi20328086-46 from 1803811 to 2495962 tid 2 2024-09-17T15:51:29.528 INFO:tasks.rados.rados.0.smithi203.stdout:3917: write oid 25 current snap is 416 2024-09-17T15:51:29.528 INFO:tasks.rados.rados.0.smithi203.stdout:3917: seq_num 1377 ranges {743239=677766,1978646=435809,2897814=245425} 2024-09-17T15:51:29.542 INFO:tasks.rados.rados.0.smithi203.stdout:3917: writing smithi20328086-25 from 743239 to 1421005 tid 1 2024-09-17T15:51:29.551 INFO:tasks.rados.rados.0.smithi203.stdout:3917: writing smithi20328086-25 from 1978646 to 2414455 tid 2 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3917: writing smithi20328086-25 from 2897814 to 3143239 tid 3 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3918: rollback oid 35 current snap is 416 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 35 to 410 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3915: finishing rollback tid 0 to smithi20328086-23 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3915: finishing rollback tid 1 to smithi20328086-23 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 23 v 1740 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3912: finishing write tid 3 to smithi20328086-28 2024-09-17T15:51:29.558 INFO:tasks.rados.rados.0.smithi203.stdout:3914: expect (ObjNum 1338 snap 407 seq_num 1338) 2024-09-17T15:51:29.698 INFO:tasks.rados.rados.0.smithi203.stdout:3912: finishing write tid 4 to smithi20328086-28 2024-09-17T15:51:29.698 INFO:tasks.rados.rados.0.smithi203.stdout:3912: finishing write tid 5 to smithi20328086-28 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3912: oid 28 updating version 0 to 2143 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3912: oid 28 updating version 2143 to 2144 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3912: oid 28 version 2144 is already newer than 2142 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3912: oid 28 updating version 2144 to 2145 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 28 v 2145 (ObjNum 1375 snap 416 seq_num 1375) dirty exists 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3912: left oid 28 (ObjNum 1375 snap 416 seq_num 1375) 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: finishing write tid 1 to smithi20328086-46 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: finishing write tid 2 to smithi20328086-46 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: finishing write tid 3 to smithi20328086-46 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: finishing write tid 4 to smithi20328086-46 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: oid 46 updating version 0 to 1909 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: oid 46 updating version 1909 to 1910 2024-09-17T15:51:29.699 INFO:tasks.rados.rados.0.smithi203.stdout:3916: oid 46 updating version 1910 to 1911 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1911 (ObjNum 1376 snap 416 seq_num 1376) dirty exists 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3916: left oid 46 (ObjNum 1376 snap 416 seq_num 1376) 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: finishing write tid 1 to smithi20328086-25 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: finishing write tid 2 to smithi20328086-25 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: finishing write tid 3 to smithi20328086-25 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: finishing write tid 4 to smithi20328086-25 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: finishing write tid 5 to smithi20328086-25 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: oid 25 updating version 0 to 1576 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: oid 25 updating version 1576 to 1579 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: oid 25 version 1579 is already newer than 1577 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: oid 25 version 1579 is already newer than 1578 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1579 (ObjNum 1377 snap 416 seq_num 1377) dirty exists 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3917: left oid 25 (ObjNum 1377 snap 416 seq_num 1377) 2024-09-17T15:51:29.700 INFO:tasks.rados.rados.0.smithi203.stdout:3918: finishing rollback tid 0 to smithi20328086-35 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3918: finishing rollback tid 1 to smithi20328086-35 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3918: finishing rollback tid 2 to smithi20328086-35 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1704 (ObjNum 1353 snap 410 seq_num 1353) dirty exists 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3912: done (6 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3913: done (5 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3914: done (4 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3915: done (3 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3916: done (2 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3917: done (1 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3918: done (0 left) 2024-09-17T15:51:29.701 INFO:tasks.rados.rados.0.smithi203.stdout:3919: snap_create 2024-09-17T15:51:30.754 INFO:tasks.rados.rados.0.smithi203.stdout:3919: done (0 left) 2024-09-17T15:51:30.754 INFO:tasks.rados.rados.0.smithi203.stdout:3920: rollback oid 25 current snap is 417 2024-09-17T15:51:30.754 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 25 to 388 2024-09-17T15:51:30.755 INFO:tasks.rados.rados.0.smithi203.stdout:3921: snap_remove snap 389 2024-09-17T15:51:30.889 INFO:tasks.rados.rados.0.smithi203.stdout:3921: done (1 left) 2024-09-17T15:51:30.889 INFO:tasks.rados.rados.0.smithi203.stdout:3922: write (excl) oid 29 current snap is 417 2024-09-17T15:51:30.889 INFO:tasks.rados.rados.0.smithi203.stdout:3922: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-09-17T15:51:30.911 INFO:tasks.rados.rados.0.smithi203.stdout:3922: writing smithi20328086-29 from 760046 to 1513061 tid 1 2024-09-17T15:51:30.928 INFO:tasks.rados.rados.0.smithi203.stdout:3922: writing smithi20328086-29 from 2037128 to 2732948 tid 2 2024-09-17T15:51:30.928 INFO:tasks.rados.rados.0.smithi203.stdout:3923: write oid 8 current snap is 417 2024-09-17T15:51:30.929 INFO:tasks.rados.rados.0.smithi203.stdout:3923: seq_num 1379 ranges {776853=428264,1979257=439478,2836307=340546} 2024-09-17T15:51:30.939 INFO:tasks.rados.rados.0.smithi203.stdout:3923: writing smithi20328086-8 from 776853 to 1205117 tid 1 2024-09-17T15:51:30.949 INFO:tasks.rados.rados.0.smithi203.stdout:3923: writing smithi20328086-8 from 1979257 to 2418735 tid 2 2024-09-17T15:51:30.955 INFO:tasks.rados.rados.0.smithi203.stdout:3923: writing smithi20328086-8 from 2836307 to 3176853 tid 3 2024-09-17T15:51:30.956 INFO:tasks.rados.rados.0.smithi203.stdout:3924: write oid 32 current snap is 417 2024-09-17T15:51:30.956 INFO:tasks.rados.rados.0.smithi203.stdout:3924: seq_num 1380 ranges {793660=503513,2037739=583136} 2024-09-17T15:51:30.965 INFO:tasks.rados.rados.0.smithi203.stdout:3924: writing smithi20328086-32 from 793660 to 1297173 tid 1 2024-09-17T15:51:30.975 INFO:tasks.rados.rados.0.smithi203.stdout:3924: writing smithi20328086-32 from 2037739 to 2620875 tid 2 2024-09-17T15:51:30.975 INFO:tasks.rados.rados.0.smithi203.stdout:3925: copy_from oid 16 from oid 6 current snap is 417 2024-09-17T15:51:30.975 INFO:tasks.rados.rados.0.smithi203.stdout:3920: finishing rollback tid 0 to smithi20328086-25 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3920: finishing rollback tid 1 to smithi20328086-25 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3920: finishing rollback tid 2 to smithi20328086-25 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 25 v 1583 (ObjNum 1246 snap 378 seq_num 1246) dirty exists 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: finishing write tid 1 to smithi20328086-29 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: finishing write tid 2 to smithi20328086-29 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: finishing write tid 3 to smithi20328086-29 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: finishing write tid 4 to smithi20328086-29 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: oid 29 updating version 0 to 1591 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: oid 29 version 1591 is already newer than 1590 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:3922: oid 29 version 1591 is already newer than 1585 2024-09-17T15:51:30.976 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 29 v 1591 (ObjNum 1378 snap 417 seq_num 1378) dirty exists 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3922: left oid 29 (ObjNum 1378 snap 417 seq_num 1378) 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: finishing write tid 1 to smithi20328086-8 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: finishing write tid 2 to smithi20328086-8 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: finishing write tid 3 to smithi20328086-8 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3920: done (4 left) 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3922: done (3 left) 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3926: rollback oid 48 current snap is 417 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 48 to 413 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: finishing write tid 4 to smithi20328086-8 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: finishing write tid 5 to smithi20328086-8 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: oid 8 updating version 0 to 1657 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: oid 8 updating version 1657 to 1659 2024-09-17T15:51:30.977 INFO:tasks.rados.rados.0.smithi203.stdout:3923: oid 8 version 1659 is already newer than 1658 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3923: oid 8 updating version 1659 to 1660 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 8 v 1660 (ObjNum 1379 snap 417 seq_num 1379) dirty exists 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3923: left oid 8 (ObjNum 1379 snap 417 seq_num 1379) 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3924: finishing write tid 1 to smithi20328086-32 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3923: done (3 left) 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3927: write (excl) oid 38 current snap is 417 2024-09-17T15:51:30.978 INFO:tasks.rados.rados.0.smithi203.stdout:3927: seq_num 1381 ranges {410467=578762,1696221=443147,2891153=319314} 2024-09-17T15:51:30.982 INFO:tasks.rados.rados.0.smithi203.stdout:3927: writing smithi20328086-38 from 410467 to 989229 tid 1 2024-09-17T15:51:30.992 INFO:tasks.rados.rados.0.smithi203.stdout:3927: writing smithi20328086-38 from 1696221 to 2139368 tid 2 2024-09-17T15:51:31.001 INFO:tasks.rados.rados.0.smithi203.stdout:3927: writing smithi20328086-38 from 2891153 to 3210467 tid 3 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3928: copy_from oid 26 from oid 24 current snap is 417 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: finishing write tid 2 to smithi20328086-32 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3925: finishing copy_from to smithi20328086-16 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3925: got expected ENOENT (src dne) 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3925: finishing copy_from racing read to smithi20328086-16 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: finishing write tid 3 to smithi20328086-32 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: finishing write tid 4 to smithi20328086-32 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: oid 32 updating version 0 to 1751 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: oid 32 updating version 1751 to 1752 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: oid 32 updating version 1752 to 1753 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 32 v 1753 (ObjNum 1380 snap 417 seq_num 1380) dirty exists 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3924: left oid 32 (ObjNum 1380 snap 417 seq_num 1380) 2024-09-17T15:51:31.002 INFO:tasks.rados.rados.0.smithi203.stdout:3926: finishing rollback tid 0 to smithi20328086-48 2024-09-17T15:51:31.003 INFO:tasks.rados.rados.0.smithi203.stdout:3924: done (4 left) 2024-09-17T15:51:31.003 INFO:tasks.rados.rados.0.smithi203.stdout:3925: done (3 left) 2024-09-17T15:51:31.003 INFO:tasks.rados.rados.0.smithi203.stdout:3929: snap_remove snap 410 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3929: done (3 left) 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3930: write (excl) oid 3 current snap is 417 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3926: finishing rollback tid 1 to smithi20328086-48 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3926: finishing rollback tid 2 to smithi20328086-48 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 2069 (ObjNum 1302 snap 400 seq_num 1302) dirty exists 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3927: finishing write tid 1 to smithi20328086-38 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3927: finishing write tid 2 to smithi20328086-38 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3927: finishing write tid 3 to smithi20328086-38 2024-09-17T15:51:31.900 INFO:tasks.rados.rados.0.smithi203.stdout:3930: seq_num 1382 ranges {427274=654011,1754703=586805,2918576=308698} 2024-09-17T15:51:31.914 INFO:tasks.rados.rados.0.smithi203.stdout:3930: writing smithi20328086-3 from 427274 to 1081285 tid 1 2024-09-17T15:51:31.928 INFO:tasks.rados.rados.0.smithi203.stdout:3930: writing smithi20328086-3 from 1754703 to 2341508 tid 2 2024-09-17T15:51:31.938 INFO:tasks.rados.rados.0.smithi203.stdout:3930: writing smithi20328086-3 from 2918576 to 3227274 tid 3 2024-09-17T15:51:31.938 INFO:tasks.rados.rados.0.smithi203.stdout:3926: done (3 left) 2024-09-17T15:51:31.938 INFO:tasks.rados.rados.0.smithi203.stdout:3931: read oid 37 snap -1 2024-09-17T15:51:31.938 INFO:tasks.rados.rados.0.smithi203.stdout:3931: expect (ObjNum 1324 snap 404 seq_num 1324) 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3932: delete oid 6 current snap is 417 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: finishing write tid 4 to smithi20328086-38 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: finishing write tid 5 to smithi20328086-38 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: oid 38 updating version 0 to 1425 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: oid 38 version 1425 is already newer than 1424 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: oid 38 updating version 1425 to 1426 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: oid 38 updating version 1426 to 1427 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 38 v 1427 (ObjNum 1381 snap 417 seq_num 1381) dirty exists 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3927: left oid 38 (ObjNum 1381 snap 417 seq_num 1381) 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3928: finishing copy_from racing read to smithi20328086-26 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3928: finishing copy_from to smithi20328086-26 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 26 v 1575 (ObjNum 1301 snap 400 seq_num 1301) dirty exists 2024-09-17T15:51:31.939 INFO:tasks.rados.rados.0.smithi203.stdout:3930: finishing write tid 1 to smithi20328086-3 2024-09-17T15:51:31.942 INFO:tasks.rados.rados.0.smithi203.stdout:3931: expect (ObjNum 1324 snap 404 seq_num 1324) 2024-09-17T15:51:32.005 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1753012 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: finishing write tid 2 to smithi20328086-3 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: finishing write tid 3 to smithi20328086-3 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: finishing write tid 4 to smithi20328086-3 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: finishing write tid 5 to smithi20328086-3 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: oid 3 updating version 0 to 1913 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: oid 3 version 1913 is already newer than 1912 2024-09-17T15:51:32.040 INFO:tasks.rados.rados.0.smithi203.stdout:3930: oid 3 updating version 1913 to 1914 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3930: oid 3 updating version 1914 to 1915 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1915 (ObjNum 1382 snap 417 seq_num 1382) dirty exists 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3930: left oid 3 (ObjNum 1382 snap 417 seq_num 1382) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3927: done (4 left) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3928: done (3 left) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3930: done (2 left) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3931: done (1 left) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3932: done (0 left) 2024-09-17T15:51:32.041 INFO:tasks.rados.rados.0.smithi203.stdout:3933: snap_create 2024-09-17T15:51:32.906 INFO:tasks.rados.rados.0.smithi203.stdout:3933: done (0 left) 2024-09-17T15:51:32.906 INFO:tasks.rados.rados.0.smithi203.stdout:3934: delete oid 37 current snap is 418 2024-09-17T15:51:33.723 INFO:tasks.rados.rados.0.smithi203.stdout:3934: done (0 left) 2024-09-17T15:51:33.723 INFO:tasks.rados.rados.0.smithi203.stdout:3935: read oid 39 snap 409 2024-09-17T15:51:33.723 INFO:tasks.rados.rados.0.smithi203.stdout:3935: expect (ObjNum 1300 snap 400 seq_num 1300) 2024-09-17T15:51:33.723 INFO:tasks.rados.rados.0.smithi203.stdout:3936: snap_create 2024-09-17T15:51:33.736 INFO:tasks.rados.rados.0.smithi203.stdout:3935: expect (ObjNum 1300 snap 400 seq_num 1300) 2024-09-17T15:51:33.805 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2723547 2024-09-17T15:51:33.975 INFO:tasks.rados.rados.0.smithi203.stdout:3935: done (1 left) 2024-09-17T15:51:33.975 INFO:tasks.rados.rados.0.smithi203.stdout:3936: done (0 left) 2024-09-17T15:51:33.975 INFO:tasks.rados.rados.0.smithi203.stdout:3937: write oid 34 current snap is 419 2024-09-17T15:51:33.975 INFO:tasks.rados.rados.0.smithi203.stdout:3937: seq_num 1385 ranges {477695=596111,1530149=450485,2600845=666679} 2024-09-17T15:51:33.992 INFO:tasks.rados.rados.0.smithi203.stdout:3937: writing smithi20328086-34 from 477695 to 1073806 tid 1 2024-09-17T15:51:34.007 INFO:tasks.rados.rados.0.smithi203.stdout:3937: writing smithi20328086-34 from 1530149 to 1980634 tid 2 2024-09-17T15:51:34.028 INFO:tasks.rados.rados.0.smithi203.stdout:3937: writing smithi20328086-34 from 2600845 to 3267524 tid 3 2024-09-17T15:51:34.028 INFO:tasks.rados.rados.0.smithi203.stdout:3938: write (excl) oid 48 current snap is 419 2024-09-17T15:51:34.028 INFO:tasks.rados.rados.0.smithi203.stdout:3937: finishing write tid 1 to smithi20328086-34 2024-09-17T15:51:34.028 INFO:tasks.rados.rados.0.smithi203.stdout:3937: finishing write tid 2 to smithi20328086-34 2024-09-17T15:51:34.028 INFO:tasks.rados.rados.0.smithi203.stdout:3938: seq_num 1386 ranges {494502=671360,1588631=594143,2628268=477951} 2024-09-17T15:51:34.043 INFO:tasks.rados.rados.0.smithi203.stdout:3938: writing smithi20328086-48 from 494502 to 1165862 tid 1 2024-09-17T15:51:34.056 INFO:tasks.rados.rados.0.smithi203.stdout:3938: writing smithi20328086-48 from 1588631 to 2182774 tid 2 2024-09-17T15:51:34.067 INFO:tasks.rados.rados.0.smithi203.stdout:3938: writing smithi20328086-48 from 2628268 to 3106219 tid 3 2024-09-17T15:51:34.067 INFO:tasks.rados.rados.0.smithi203.stdout:3939: write (excl) oid 1 current snap is 419 2024-09-17T15:51:34.067 INFO:tasks.rados.rados.0.smithi203.stdout:3939: seq_num 1387 ranges {511309=746609,2047113=454154,3055691=255618} 2024-09-17T15:51:34.079 INFO:tasks.rados.rados.0.smithi203.stdout:3939: writing smithi20328086-1 from 511309 to 1257918 tid 1 2024-09-17T15:51:34.089 INFO:tasks.rados.rados.0.smithi203.stdout:3939: writing smithi20328086-1 from 2047113 to 2501267 tid 2 2024-09-17T15:51:34.095 INFO:tasks.rados.rados.0.smithi203.stdout:3939: writing smithi20328086-1 from 3055691 to 3311309 tid 3 2024-09-17T15:51:34.095 INFO:tasks.rados.rados.0.smithi203.stdout:3940: snap_create 2024-09-17T15:51:34.095 INFO:tasks.rados.rados.0.smithi203.stdout:3937: finishing write tid 3 to smithi20328086-34 2024-09-17T15:51:34.095 INFO:tasks.rados.rados.0.smithi203.stdout:3937: finishing write tid 4 to smithi20328086-34 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: finishing write tid 5 to smithi20328086-34 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: oid 34 updating version 0 to 1708 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: oid 34 version 1708 is already newer than 1707 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: oid 34 version 1708 is already newer than 1705 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: oid 34 version 1708 is already newer than 1706 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 34 v 1708 (ObjNum 1385 snap 419 seq_num 1385) dirty exists 2024-09-17T15:51:34.096 INFO:tasks.rados.rados.0.smithi203.stdout:3937: left oid 34 (ObjNum 1385 snap 419 seq_num 1385) 2024-09-17T15:51:34.840 INFO:tasks.rados.rados.0.smithi203.stdout:3938: finishing write tid 1 to smithi20328086-48 2024-09-17T15:51:34.840 INFO:tasks.rados.rados.0.smithi203.stdout:3939: finishing write tid 1 to smithi20328086-1 2024-09-17T15:51:34.840 INFO:tasks.rados.rados.0.smithi203.stdout:3939: finishing write tid 2 to smithi20328086-1 2024-09-17T15:51:34.840 INFO:tasks.rados.rados.0.smithi203.stdout:3939: finishing write tid 3 to smithi20328086-1 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: finishing write tid 4 to smithi20328086-1 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: finishing write tid 5 to smithi20328086-1 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: oid 1 updating version 0 to 1751 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: oid 1 updating version 1751 to 1753 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: oid 1 version 1753 is already newer than 1752 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: oid 1 updating version 1753 to 1754 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1754 (ObjNum 1387 snap 419 seq_num 1387) dirty exists 2024-09-17T15:51:34.841 INFO:tasks.rados.rados.0.smithi203.stdout:3939: left oid 1 (ObjNum 1387 snap 419 seq_num 1387) 2024-09-17T15:51:34.843 INFO:tasks.rados.rados.0.smithi203.stdout:3938: finishing write tid 2 to smithi20328086-48 2024-09-17T15:51:34.845 INFO:tasks.rados.rados.0.smithi203.stdout:3938: finishing write tid 3 to smithi20328086-48 2024-09-17T15:51:34.846 INFO:tasks.rados.rados.0.smithi203.stdout:3938: finishing write tid 4 to smithi20328086-48 2024-09-17T15:51:34.846 INFO:tasks.rados.rados.0.smithi203.stdout:3938: finishing write tid 5 to smithi20328086-48 2024-09-17T15:51:34.846 INFO:tasks.rados.rados.0.smithi203.stdout:3938: oid 48 updating version 0 to 2077 2024-09-17T15:51:34.847 INFO:tasks.rados.rados.0.smithi203.stdout:3938: oid 48 updating version 2077 to 2080 2024-09-17T15:51:34.847 INFO:tasks.rados.rados.0.smithi203.stdout:3938: oid 48 version 2080 is already newer than 2078 2024-09-17T15:51:34.847 INFO:tasks.rados.rados.0.smithi203.stdout:3938: oid 48 version 2080 is already newer than 2079 2024-09-17T15:51:34.847 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 48 v 2080 (ObjNum 1386 snap 419 seq_num 1386) dirty exists 2024-09-17T15:51:34.847 INFO:tasks.rados.rados.0.smithi203.stdout:3938: left oid 48 (ObjNum 1386 snap 419 seq_num 1386) 2024-09-17T15:51:34.923 INFO:tasks.rados.rados.0.smithi203.stdout:3937: done (3 left) 2024-09-17T15:51:34.923 INFO:tasks.rados.rados.0.smithi203.stdout:3938: done (2 left) 2024-09-17T15:51:34.923 INFO:tasks.rados.rados.0.smithi203.stdout:3939: done (1 left) 2024-09-17T15:51:34.923 INFO:tasks.rados.rados.0.smithi203.stdout:3940: done (0 left) 2024-09-17T15:51:34.923 INFO:tasks.rados.rados.0.smithi203.stdout:3941: snap_create 2024-09-17T15:51:35.598 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-17T15:51:35.598 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-17T15:51:35.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:51:35.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: get_monmap_and_config 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: build_initial_monmap 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: monmap: 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:35.729772+0000 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:35.729772+0000 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:35.727 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:35.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 Processor -- start 2024-09-17T15:51:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 -- start start 2024-09-17T15:51:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: init 2024-09-17T15:51:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:35.729 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa820073fc0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa82009bc00 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa820074ab0 0x7fa820140740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-a con 0x7fa820074ab0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82009d210 con 0x7fa820073fc0 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82009c3c0 con 0x7fa82009bc00 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fa82009c6a0 con 0x7fa820074ab0 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa8261de640 10 monclient(hunting): authenticate will time out at 2428.695301s 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa820073fc0 0x7fa8200743a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46356/0 (socket says 172.21.15.86:46356) 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81f7fe640 1 -- 172.21.15.86:0/1309124339 learned_addr learned my addr 172.21.15.86:0/1309124339 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:35.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 1 --2- 172.21.15.86:0/1309124339 >> v2:172.21.15.86:3300/0 conn(0x7fa820074ab0 0x7fa820140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): get_auth_request con 0x7fa820074ab0 auth_method 0 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient(hunting): handle_auth_done global_id 234495 payload 274 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 1 -- 172.21.15.86:0/1309124339 >> v1:172.21.15.203:6789/0 conn(0x7fa82009bc00 legacy=0x7fa82013d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 1 -- 172.21.15.86:0/1309124339 >> v1:172.21.15.86:6789/0 conn(0x7fa820073fc0 legacy=0x7fa8200743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 1 monclient: found mon.noname-a 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:35.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:35.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.726+0000 7fa81ffff640 1 -- 172.21.15.86:0/1309124339 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82009c980 con 0x7fa820074ab0 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 -- 172.21.15.86:0/1309124339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa820074ab0 msgr2=0x7fa820140740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 --2- 172.21.15.86:0/1309124339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa820074ab0 0x7fa820140740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa818009d90 tx=0x7fa81801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aeb28541d9f5061 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 1 -- 172.21.15.86:0/1309124339 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b2070 con 0x7fa820074ab0 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: dump: 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.736 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 1 -- 172.21.15.86:0/1309124339 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8180a5970 con 0x7fa820074ab0 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 5 monclient: authenticate success, global_id 234495 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:35.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 1 -- 172.21.15.86:0/1309124339 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180a5d40 con 0x7fa820074ab0 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 4 set_mon_vals no callback set 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81e7fc640 10 monclient: dump: 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:35.738 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: shutdown 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- 172.21.15.86:0/1309124339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa820074ab0 msgr2=0x7fa820140740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 --2- 172.21.15.86:0/1309124339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa820074ab0 0x7fa820140740 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa818009d90 tx=0x7fa81801c830 comp rx=0 tx=0).stop 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- 172.21.15.86:0/1309124339 shutdown_connections 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 --2- 172.21.15.86:0/1309124339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa820074ab0 0x7fa820140740 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- 172.21.15.86:0/1309124339 >> 172.21.15.86:0/1309124339 conn(0x7fa8200a6440 msgr2=0x7fa8200a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- 172.21.15.86:0/1309124339 shutdown_connections 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- 172.21.15.86:0/1309124339 wait complete. 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: build_initial_monmap 2024-09-17T15:51:35.739 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: monmap: 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 Processor -- start 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- start start 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: init 2024-09-17T15:51:35.740 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-c con 0x7fa820073fc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa820074ab0 0x7fa82011dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-b con 0x7fa820074ab0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): picked mon.noname-a con 0x7fa82009bc00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa820143830 con 0x7fa82009bc00 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa8201436b0 con 0x7fa820074ab0 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa8201439b0 con 0x7fa820073fc0 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa8261de640 10 monclient(hunting): authenticate will time out at 2428.702295s 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50158/0 (socket says 172.21.15.86:50158) 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59060/0 (socket says 172.21.15.86:59060) 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 -- 172.21.15.86:0/826739097 learned_addr learned my addr 172.21.15.86:0/826739097 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): get_auth_request con 0x7fa82009bc00 auth_method 0 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): get_auth_request con 0x7fa820073fc0 auth_method 0 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:35.742 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81effd640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa820074ab0 0x7fa82011dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81effd640 10 monclient(hunting): get_auth_request con 0x7fa820074ab0 auth_method 0 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:35.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient(hunting): handle_auth_done global_id 274100 payload 824 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 -- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa820074ab0 msgr2=0x7fa82011dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa820074ab0 0x7fa82011dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 -- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 msgr2=0x7fa820137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:35.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 monclient: found mon.noname-c 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81f7fe640 1 -- 172.21.15.86:0/826739097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82009d460 con 0x7fa820073fc0 2024-09-17T15:51:35.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.734+0000 7fa81ffff640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa81f7fe640 1 -- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 msgr2=0x7fa82011fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa81f7fe640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c410 tx=0x7fa81400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa814017070 con 0x7fa820073fc0 2024-09-17T15:51:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:35.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: dump: 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:35.748 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8140070d0 con 0x7fa820073fc0 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 5 monclient: authenticate success, global_id 274100 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _renew_subs 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa820121660 con 0x7fa820073fc0 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140074c0 con 0x7fa820073fc0 2024-09-17T15:51:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 10 monclient: dump: 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _renew_subs 2024-09-17T15:51:35.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa820120dc0 con 0x7fa820073fc0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa81dffb640 4 set_mon_vals no callback set 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa820074420 con 0x7fa820073fc0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa81401a070 con 0x7fa820073fc0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 0x7fa80c03e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(858..858 src has 794..858) ==== 4755+0+0 (secure 0 0 0) 0x7fa81404f3f0 con 0x7fa820073fc0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa81effd640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 0x7fa80c03e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.738+0000 7fa81effd640 10 monclient: get_auth_request con 0x7fa80c03bb90 auth_method 0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.742+0000 7fa81effd640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 0x7fa80c03e050 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa82011fdf0 tx=0x7fa810009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.742+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa82009d460 con 0x7fa820073fc0 2024-09-17T15:51:35.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.742+0000 7fa807fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:35.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.742+0000 7fa807fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:35.891 INFO:tasks.rados.rados.0.smithi203.stdout:3941: done (0 left) 2024-09-17T15:51:35.891 INFO:tasks.rados.rados.0.smithi203.stdout:3942: write oid 4 current snap is 421 2024-09-17T15:51:35.891 INFO:tasks.rados.rados.0.smithi203.stdout:3942: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-09-17T15:51:35.900 INFO:tasks.rados.rados.0.smithi203.stdout:3942: writing smithi20328086-4 from 528116 to 949974 tid 1 2024-09-17T15:51:35.911 INFO:tasks.rados.rados.0.smithi203.stdout:3942: writing smithi20328086-4 from 1589242 to 2187054 tid 2 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3942: writing smithi20328086-4 from 2966761 to 3328116 tid 3 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3943: read oid 33 snap -1 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3943: expect (ObjNum 1362 snap 414 seq_num 1362) 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3944: rollback oid 1 current snap is 421 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 1 to 420 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3942: finishing write tid 1 to smithi20328086-4 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3942: finishing write tid 2 to smithi20328086-4 2024-09-17T15:51:35.923 INFO:tasks.rados.rados.0.smithi203.stdout:3945: snap_create 2024-09-17T15:51:35.930 INFO:tasks.rados.rados.0.smithi203.stdout:3942: finishing write tid 3 to smithi20328086-4 2024-09-17T15:51:35.930 INFO:tasks.rados.rados.0.smithi203.stdout:3942: finishing write tid 4 to smithi20328086-4 2024-09-17T15:51:35.930 INFO:tasks.rados.rados.0.smithi203.stdout:3942: finishing write tid 5 to smithi20328086-4 2024-09-17T15:51:35.930 INFO:tasks.rados.rados.0.smithi203.stdout:3942: oid 4 updating version 0 to 1354 2024-09-17T15:51:35.930 INFO:tasks.rados.rados.0.smithi203.stdout:3942: oid 4 updating version 1354 to 1356 2024-09-17T15:51:35.931 INFO:tasks.rados.rados.0.smithi203.stdout:3942: oid 4 version 1356 is already newer than 1355 2024-09-17T15:51:35.931 INFO:tasks.rados.rados.0.smithi203.stdout:3942: oid 4 version 1356 is already newer than 1353 2024-09-17T15:51:35.931 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 4 v 1356 (ObjNum 1388 snap 421 seq_num 1388) dirty exists 2024-09-17T15:51:35.931 INFO:tasks.rados.rados.0.smithi203.stdout:3942: left oid 4 (ObjNum 1388 snap 421 seq_num 1388) 2024-09-17T15:51:35.945 INFO:tasks.rados.rados.0.smithi203.stdout:3943: expect (ObjNum 1362 snap 414 seq_num 1362) 2024-09-17T15:51:35.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.982+0000 7fa8261de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:35.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.982+0000 7fa8261de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:35.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:35.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa82006d4f0 con 0x7fa820073fc0 2024-09-17T15:51:35.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa807fff640 1 -- 172.21.15.86:0/826739097 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fa81401bc00 con 0x7fa820073fc0 2024-09-17T15:51:35.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa807fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:35.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa807fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:35.986 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 msgr2=0x7fa80c03e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 0x7fa80c03e050 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa82011fdf0 tx=0x7fa810009040 comp rx=0 tx=0).stop 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 10 monclient: shutdown 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 msgr2=0x7fa82011fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa81400c410 tx=0x7fa81400c8d0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 shutdown_connections 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa80c03bb90 0x7fa80c03e050 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa82009bc00 0x7fa820137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa820074ab0 0x7fa82011dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 --2- 172.21.15.86:0/826739097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa820073fc0 0x7fa82011fe60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 >> 172.21.15.86:0/826739097 conn(0x7fa8200a6440 msgr2=0x7fa8200a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 shutdown_connections 2024-09-17T15:51:35.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:35.986+0000 7fa8261de640 1 -- 172.21.15.86:0/826739097 wait complete. 2024-09-17T15:51:36.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:51:36.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:51:36.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:36.010 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:36.086 INFO:tasks.rados.rados.0.smithi203.stdout:3944: finishing rollback tid 0 to smithi20328086-1 2024-09-17T15:51:36.086 INFO:tasks.rados.rados.0.smithi203.stdout:3944: finishing rollback tid 1 to smithi20328086-1 2024-09-17T15:51:36.086 INFO:tasks.rados.rados.0.smithi203.stdout:3944: finishing rollback tid 2 to smithi20328086-1 2024-09-17T15:51:36.086 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 1 v 1758 (ObjNum 1387 snap 419 seq_num 1387) dirty exists 2024-09-17T15:51:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: get_monmap_and_config 2024-09-17T15:51:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: monmap: 2024-09-17T15:51:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:36.132200+0000 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:36.132200+0000 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:36.129 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 Processor -- start 2024-09-17T15:51:36.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 -- start start 2024-09-17T15:51:36.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: init 2024-09-17T15:51:36.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-b con 0x7ff1000a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1000a54c0 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff10013d2f0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1000a6280 con 0x7ff10013d2f0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10009c0a0 con 0x7ff1000a54c0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff10009d2a0 con 0x7ff1000a50e0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff104f90640 10 monclient(hunting): authenticate will time out at 2429.097662s 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fed9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff10013d2f0 0x7ff10013f6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46362/0 (socket says 172.21.15.86:46362) 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59074/0 (socket says 172.21.15.86:59074) 2024-09-17T15:51:36.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fed9d640 1 -- 172.21.15.86:0/3450969541 learned_addr learned my addr 172.21.15.86:0/3450969541 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): get_auth_request con 0x7ff1000a50e0 auth_method 0 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_done global_id 274103 payload 274 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 -- 172.21.15.86:0/3450969541 >> v1:172.21.15.203:6789/0 conn(0x7ff1000a54c0 legacy=0x7ff1000a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 -- 172.21.15.86:0/3450969541 >> v1:172.21.15.86:6789/0 conn(0x7ff10013d2f0 legacy=0x7ff10013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 monclient: found mon.noname-b 2024-09-17T15:51:36.133 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.130+0000 7ff0fe59c640 1 -- 172.21.15.86:0/3450969541 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10009c380 con 0x7ff1000a50e0 2024-09-17T15:51:36.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fe59c640 1 -- 172.21.15.86:0/3450969541 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 msgr2=0x7ff10009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fe59c640 1 --2- 172.21.15.86:0/3450969541 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097f0 tx=0x7ff0f801c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85544f012ecead98 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 1 -- 172.21.15.86:0/3450969541 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80b0070 con 0x7ff1000a50e0 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: dump: 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 1 -- 172.21.15.86:0/3450969541 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f80ab600 con 0x7ff1000a50e0 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 5 monclient: authenticate success, global_id 274103 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 4 set_mon_vals no callback set 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 1 -- 172.21.15.86:0/3450969541 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80ab9d0 con 0x7ff1000a50e0 2024-09-17T15:51:36.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff0fd59a640 10 monclient: dump: 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 10 monclient: shutdown 2024-09-17T15:51:36.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 -- 172.21.15.86:0/3450969541 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 msgr2=0x7ff10009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 --2- 172.21.15.86:0/3450969541 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097f0 tx=0x7ff0f801c370 comp rx=0 tx=0).stop 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 -- 172.21.15.86:0/3450969541 shutdown_connections 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 --2- 172.21.15.86:0/3450969541 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1000a50e0 0x7ff10009cd60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 -- 172.21.15.86:0/3450969541 >> 172.21.15.86:0/3450969541 conn(0x7ff1000aaae0 msgr2=0x7ff1000aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 -- 172.21.15.86:0/3450969541 shutdown_connections 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 1 -- 172.21.15.86:0/3450969541 wait complete. 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.134+0000 7ff104f90640 10 monclient: monmap: 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:36.139 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.140 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.140 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:36.140 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:36.140 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:36.140 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 Processor -- start 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- start start 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: init 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 0x7ff100137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-a con 0x7ff1000a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1000a54c0 0x7ff100138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-b con 0x7ff1000a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 0x7ff10009e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): picked mon.noname-c con 0x7ff10013d2f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff100143830 con 0x7ff1000a50e0 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff1001439b0 con 0x7ff1000a54c0 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff100143b30 con 0x7ff10013d2f0 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient(hunting): authenticate will time out at 2429.104846s 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 0x7ff100137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 0x7ff10009e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 0x7ff100137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50168/0 (socket says 172.21.15.86:50168) 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 0x7ff10009e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59082/0 (socket says 172.21.15.86:59082) 2024-09-17T15:51:36.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 -- 172.21.15.86:0/1290406570 learned_addr learned my addr 172.21.15.86:0/1290406570 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): get_auth_request con 0x7ff1000a50e0 auth_method 0 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): get_auth_request con 0x7ff10013d2f0 auth_method 0 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fdd9b640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1000a54c0 0x7ff100138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fdd9b640 10 monclient(hunting): get_auth_request con 0x7ff1000a54c0 auth_method 0 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient(hunting): handle_auth_done global_id 234501 payload 824 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 msgr2=0x7ff10009e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 0x7ff10009e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1000a54c0 msgr2=0x7ff100138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1000a54c0 0x7ff100138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 monclient: found mon.noname-a 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1000a0180 con 0x7ff1000a50e0 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fed9d640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff10013d2f0 0x7ff10009e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 msgr2=0x7ff100137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fe59c640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 0x7ff100137b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80ae040 tx=0x7ff0f801abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80b0050 con 0x7ff1000a50e0 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: dump: 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.149 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f80bc070 con 0x7ff1000a50e0 2024-09-17T15:51:36.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 5 monclient: authenticate success, global_id 234501 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _renew_subs 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff100121a50 con 0x7ff1000a50e0 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f80b5c80 con 0x7ff1000a50e0 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0f6ffd640 10 monclient: dump: 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.151 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff0fcd99640 4 set_mon_vals no callback set 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _renew_subs 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.138+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001385c0 con 0x7ff1000a50e0 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff104f90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff104f90640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 0x7ff10009f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1001216d0 con 0x7ff10009ed90 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff0f80ab4c0 con 0x7ff1000a50e0 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 0x7ff10009f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient: get_auth_request con 0x7ff10009ed90 auth_method 0 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0e003bbb0 0x7ff0e003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(859..859 src has 794..859) ==== 4783+0+0 (secure 0 0 0) 0x7ff0f80ea900 con 0x7ff1000a50e0 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_done global_id 234504 payload 824 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff100140970 con 0x7ff10009ed90 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fed9d640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0e003bbb0 0x7ff0e003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fed9d640 10 monclient: get_auth_request con 0x7ff0e003bbb0 auth_method 0 2024-09-17T15:51:36.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 msgr2=0x7ff10009f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0fdd9b640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 0x7ff10009f170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800eea0 con 0x7ff10009ed90 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 10 monclient: dump: 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0e8006b60 con 0x7ff10009ed90 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 msgr2=0x7ff10009f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 0x7ff10009f170 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e800ec50 comp rx=0 tx=0).stop 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 msgr2=0x7ff10009f170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.142+0000 7ff0f6ffd640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009ed90 0x7ff10009f170 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e800ec50 comp rx=0 tx=0).stop 2024-09-17T15:51:36.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.146+0000 7ff0fed9d640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0e003bbb0 0x7ff0e003e070 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0007ca0 tx=0x7ff0f0007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff104f90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff104f90640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 0x7ff1001392d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff104f90640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff1001211b0 con 0x7ff10009f6b0 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 0x7ff1001392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient: get_auth_request con 0x7ff10009f6b0 auth_method 0 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 10 monclient(hunting): handle_auth_done global_id 234507 payload 824 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.150+0000 7ff0fdd9b640 1 -- 172.21.15.86:0/1290406570 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1000a18d0 con 0x7ff10009f6b0 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0fdd9b640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 msgr2=0x7ff1001392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0fdd9b640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 0x7ff1001392d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e8003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8015430 con 0x7ff10009f6b0 2024-09-17T15:51:36.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 10 monclient: dump: 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7ff0e80155d0 con 0x7ff10009f6b0 2024-09-17T15:51:36.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 msgr2=0x7ff1001392d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 0x7ff1001392d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e8003af0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 msgr2=0x7ff1001392d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0f6ffd640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff10009f6b0 0x7ff1001392d0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff0e800e790 tx=0x7ff0e8003af0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:51:36.160 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 225349, 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.161 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:36.162 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.163 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.164 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.165 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:36.166 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:36.168 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.169 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.170 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.171 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.172 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:51:36.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:36.174 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:36.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0e003bbb0 msgr2=0x7ff0e003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.175 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff0e003bbb0 0x7ff0e003e070 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0007ca0 tx=0x7ff0f0007bf0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 10 monclient: shutdown 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 msgr2=0x7ff100137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 --2- 172.21.15.86:0/1290406570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff1000a50e0 0x7ff100137b40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80ae040 tx=0x7ff0f801abb0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0fed9d640 1 -- 172.21.15.86:0/1290406570 reap_dead start 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff0fed9d640 1 -- 172.21.15.86:0/1290406570 reap_dead start 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 shutdown_connections 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 >> 172.21.15.86:0/1290406570 conn(0x7ff1000aaae0 msgr2=0x7ff1000a34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 shutdown_connections 2024-09-17T15:51:36.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.154+0000 7ff104f90640 1 -- 172.21.15.86:0/1290406570 wait complete. 2024-09-17T15:51:36.177 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:36.177 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:51:36.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:51:36.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: get_monmap_and_config 2024-09-17T15:51:36.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: monmap: 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:36.296246+0000 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:36.296246+0000 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:36.311 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:36.312 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 Processor -- start 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 -- start start 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: init 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-b con 0x7f4f280a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 0x7f4f280a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-c con 0x7f4f280a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f28140690 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f28073ff0 con 0x7f4f28140690 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f4f2809bb20 con 0x7f4f280a54c0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f280a6d70 con 0x7f4f280a50e0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2d21c640 10 monclient(hunting): authenticate will time out at 2429.261713s 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59098/0 (socket says 172.21.15.86:59098) 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 -- 172.21.15.86:0/4271396233 learned_addr learned my addr 172.21.15.86:0/4271396233 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request con 0x7f4f280a50e0 auth_method 0 2024-09-17T15:51:36.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2659c640 1 --2- 172.21.15.86:0/4271396233 >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 0x7f4f280a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_done global_id 274109 payload 274 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 -- 172.21.15.86:0/4271396233 >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 msgr2=0x7f4f280a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/4271396233 >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 0x7f4f280a58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 -- 172.21.15.86:0/4271396233 >> v1:172.21.15.203:6789/0 conn(0x7f4f28140690 legacy=0x7f4f28142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.327 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 monclient: found mon.noname-b 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f26d9d640 1 -- 172.21.15.86:0/4271396233 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2809cbc0 con 0x7f4f280a50e0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2659c640 10 monclient: get_auth_request con 0x7f4f280a54c0 auth_method 0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.294+0000 7f4f2659c640 1 --2- 172.21.15.86:0/4271396233 >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 0x7f4f280a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f26d9d640 1 -- 172.21.15.86:0/4271396233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 msgr2=0x7f4f2809d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/4271396233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f280a6690 tx=0x7f4f2001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee3aa5e0e012059 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 1 -- 172.21.15.86:0/4271396233 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200a5520 con 0x7f4f280a50e0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: dump: 2024-09-17T15:51:36.328 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 1 -- 172.21.15.86:0/4271396233 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f200a56c0 con 0x7f4f280a50e0 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 5 monclient: authenticate success, global_id 274109 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 1 -- 172.21.15.86:0/4271396233 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200a5a90 con 0x7f4f280a50e0 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f25d9b640 10 monclient: dump: 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.330 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 4 set_mon_vals no callback set 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 10 monclient: shutdown 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 -- 172.21.15.86:0/4271396233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 msgr2=0x7f4f2809d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/4271396233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4f280a6690 tx=0x7f4f2001c8d0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 -- 172.21.15.86:0/4271396233 shutdown_connections 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/4271396233 >> v2:172.21.15.203:3300/0 conn(0x7f4f280a54c0 0x7f4f280a58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/4271396233 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a50e0 0x7f4f2809d880 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 -- 172.21.15.86:0/4271396233 >> 172.21.15.86:0/4271396233 conn(0x7f4f280aaae0 msgr2=0x7f4f280aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 -- 172.21.15.86:0/4271396233 shutdown_connections 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.298+0000 7f4f2d21c640 1 -- 172.21.15.86:0/4271396233 wait complete. 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient: monmap: 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:36.332 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 Processor -- start 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 -- start start 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient: init 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f280a50e0 0x7f4f281d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-b con 0x7f4f280a50e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-c con 0x7f4f280a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28140690 0x7f4f28070110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): picked mon.noname-a con 0x7f4f28140690 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28146bd0 con 0x7f4f28140690 2024-09-17T15:51:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4f28146d50 con 0x7f4f280a50e0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f28146ed0 con 0x7f4f280a54c0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2d21c640 10 monclient(hunting): authenticate will time out at 2429.269413s 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59110/0 (socket says 172.21.15.86:59110) 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 -- 172.21.15.86:0/1174582667 learned_addr learned my addr 172.21.15.86:0/1174582667 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): get_auth_request con 0x7f4f280a54c0 auth_method 0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28140690 0x7f4f28070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): get_auth_request con 0x7f4f28140690 auth_method 0 2024-09-17T15:51:36.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f280a50e0 0x7f4f281d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request con 0x7f4f280a50e0 auth_method 0 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient(hunting): handle_auth_done global_id 274112 payload 824 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f280a50e0 msgr2=0x7f4f281d41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4f280a50e0 0x7f4f281d41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28140690 msgr2=0x7f4f28070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f28140690 0x7f4f28070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 monclient: found mon.noname-c 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.302+0000 7f4f2659c640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f28129b20 con 0x7f4f280a54c0 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2659c640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 msgr2=0x7f4f2813f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2659c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000c7c0 tx=0x7f4f1000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10007c40 con 0x7f4f280a54c0 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: dump: 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f10007de0 con 0x7f4f280a54c0 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 5 monclient: authenticate success, global_id 274112 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 monclient: _renew_subs 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f28129540 con 0x7f4f280a54c0 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10011630 con 0x7f4f280a54c0 2024-09-17T15:51:36.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 10 monclient: dump: 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2559a640 4 set_mon_vals no callback set 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 monclient: _renew_subs 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f28072910 con 0x7f4f280a54c0 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 0x7f4f28070650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f2809c170 con 0x7f4f2813fc90 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f4f100117d0 con 0x7f4f280a54c0 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 0x7f4f28070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient: get_auth_request con 0x7f4f2813fc90 auth_method 0 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_done global_id 274115 payload 824 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f0803bbe0 0x7f4f0803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f2809d880 con 0x7f4f2813fc90 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(859..859 src has 794..859) ==== 4783+0+0 (secure 0 0 0) 0x7f4f100502b0 con 0x7f4f280a54c0 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2759e640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f0803bbe0 0x7f4f0803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f2759e640 10 monclient: get_auth_request con 0x7f4f0803bbe0 auth_method 0 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 msgr2=0x7f4f28070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.306+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 0x7f4f28070650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f2809de80 tx=0x7f4f20019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200b6620 con 0x7f4f2813fc90 2024-09-17T15:51:36.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 10 monclient: dump: 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.342 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4f200ad3f0 con 0x7f4f2813fc90 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 msgr2=0x7f4f28070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 0x7f4f28070650 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f2809de80 tx=0x7f4f20019670 comp rx=0 tx=0).stop 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 msgr2=0x7f4f28070650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f1f7fe640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f2813fc90 0x7f4f28070650 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4f2809de80 tx=0x7f4f20019670 comp rx=0 tx=0).stop 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.310+0000 7f4f2759e640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f0803bbe0 0x7f4f0803e0a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4f18007fd0 tx=0x7f4f18007dc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f2d21c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 0x7f4f2809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f2d21c640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f4f28070fa0 con 0x7f4f28070b90 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 0x7f4f2809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient: get_auth_request con 0x7f4f28070b90 auth_method 0 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 10 monclient(hunting): handle_auth_done global_id 274118 payload 824 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.314+0000 7f4f26d9d640 1 -- 172.21.15.86:0/1174582667 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f28071b90 con 0x7f4f28070b90 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f26d9d640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 msgr2=0x7f4f2809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f26d9d640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 0x7f4f2809e5e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f2001d170 tx=0x7f4f200a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f200c29f0 con 0x7f4f28070b90 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 10 monclient: dump: 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f4f200c8070 con 0x7f4f28070b90 2024-09-17T15:51:36.345 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 msgr2=0x7f4f2809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 0x7f4f2809e5e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4f2001d170 tx=0x7f4f200a7000 comp rx=0 tx=0).stop 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 msgr2=0x7f4f2809e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f1f7fe640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f28070b90 0x7f4f2809e5e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 15639, 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:36.359 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:36.360 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f0803bbe0 msgr2=0x7f4f0803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f4f0803bbe0 0x7f4f0803e0a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f4f18007fd0 tx=0x7f4f18007dc0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 10 monclient: shutdown 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 msgr2=0x7f4f2813f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4f1000c7c0 tx=0x7f4f1000cc80 comp rx=0 tx=0).stop 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2759e640 1 -- 172.21.15.86:0/1174582667 reap_dead start 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 shutdown_connections 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 --2- 172.21.15.86:0/1174582667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f280a54c0 0x7f4f2813f750 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 >> 172.21.15.86:0/1174582667 conn(0x7f4f280aaae0 msgr2=0x7f4f280a45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 shutdown_connections 2024-09-17T15:51:36.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.318+0000 7f4f2d21c640 1 -- 172.21.15.86:0/1174582667 wait complete. 2024-09-17T15:51:36.362 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:36.362 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:51:36.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 10 monclient: get_monmap_and_config 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 10 monclient: monmap: 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:36.482668+0000 2024-09-17T15:51:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:36.482668+0000 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:36.480 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 1 Processor -- start 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 1 -- start start 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.478+0000 7f348aaa6640 10 monclient: init 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f348412f5f0 0x7f348412f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-c con 0x7f348412f5f0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3484135940 0x7f3484135d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-a con 0x7f3484135940 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34841362d0 0x7f348413ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-b con 0x7f34841362d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f34840703e0 con 0x7f3484135940 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f3484070160 con 0x7f348412f5f0 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f3484136740 con 0x7f34841362d0 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f348aaa6640 10 monclient(hunting): authenticate will time out at 2429.448675s 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3484135940 0x7f3484135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34841362d0 0x7f348413ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3484135940 0x7f3484135d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50210/0 (socket says 172.21.15.86:50210) 2024-09-17T15:51:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 -- 172.21.15.86:0/3865246348 learned_addr learned my addr 172.21.15.86:0/3865246348 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): get_auth_request con 0x7f3484135940 auth_method 0 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): get_auth_request con 0x7f34841362d0 auth_method 0 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3488842640 1 --2- 172.21.15.86:0/3865246348 >> v2:172.21.15.203:3300/0 conn(0x7f348412f5f0 0x7f348412f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3488842640 10 monclient(hunting): get_auth_request con 0x7f348412f5f0 auth_method 0 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3488842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3488842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3488842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient(hunting): handle_auth_done global_id 234513 payload 274 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 -- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34841362d0 msgr2=0x7f348413ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 --2- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34841362d0 0x7f348413ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 -- 172.21.15.86:0/3865246348 >> v2:172.21.15.203:3300/0 conn(0x7f348412f5f0 msgr2=0x7f348412f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 --2- 172.21.15.86:0/3865246348 >> v2:172.21.15.203:3300/0 conn(0x7f348412f5f0 0x7f348412f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 monclient: found mon.noname-a 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:36.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.482+0000 7f3483fff640 1 -- 172.21.15.86:0/3865246348 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3484073f60 con 0x7f3484135940 2024-09-17T15:51:36.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f3483fff640 1 -- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3484135940 msgr2=0x7f3484135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f3483fff640 1 --2- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3484135940 0x7f3484135d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34840a7bd0 tx=0x7f347401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee3ed83306e6adb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 1 -- 172.21.15.86:0/3865246348 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34740a5620 con 0x7f3484135940 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: dump: 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.494 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 1 -- 172.21.15.86:0/3865246348 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34740a57c0 con 0x7f3484135940 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 5 monclient: authenticate success, global_id 234513 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:36.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 1 -- 172.21.15.86:0/3865246348 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34740a5b90 con 0x7f3484135940 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 4 set_mon_vals no callback set 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f34837fe640 10 monclient: dump: 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.496 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: shutdown 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3484135940 msgr2=0x7f3484135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3484135940 0x7f3484135d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34840a7bd0 tx=0x7f347401c8b0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- 172.21.15.86:0/3865246348 shutdown_connections 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34841362d0 0x7f348413ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- 172.21.15.86:0/3865246348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3484135940 0x7f3484135d90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- 172.21.15.86:0/3865246348 >> v2:172.21.15.203:3300/0 conn(0x7f348412f5f0 0x7f348412f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- 172.21.15.86:0/3865246348 >> 172.21.15.86:0/3865246348 conn(0x7f34840a3630 msgr2=0x7f34840a3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- 172.21.15.86:0/3865246348 shutdown_connections 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- 172.21.15.86:0/3865246348 wait complete. 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: monmap: 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.498 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 Processor -- start 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 -- start start 2024-09-17T15:51:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: init 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348412f5f0 0x7f348412e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-b con 0x7f348412f5f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 0x7f348412ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-c con 0x7f3484135940 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.486+0000 7f348aaa6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 0x7f3484136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): picked mon.noname-a con 0x7f34841362d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f3484074920 con 0x7f34841362d0 2024-09-17T15:51:36.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3484068c30 con 0x7f348412f5f0 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f34841399b0 con 0x7f3484135940 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): authenticate will time out at 2429.456046s 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 0x7f3484136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 0x7f348412ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 0x7f3484136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50218/0 (socket says 172.21.15.86:50218) 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 1 -- 172.21.15.86:0/1610457035 learned_addr learned my addr 172.21.15.86:0/1610457035 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): get_auth_request con 0x7f34841362d0 auth_method 0 2024-09-17T15:51:36.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): get_auth_request con 0x7f3484135940 auth_method 0 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3488842640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348412f5f0 0x7f348412e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3488842640 10 monclient(hunting): get_auth_request con 0x7f348412f5f0 auth_method 0 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3488842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3488842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3488842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient(hunting): handle_auth_done global_id 274124 payload 824 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348412f5f0 msgr2=0x7f348412e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348412f5f0 0x7f348412e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 msgr2=0x7f3484136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 0x7f3484136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 monclient: found mon.noname-c 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f348413cc40 con 0x7f3484135940 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3489043640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34841362d0 0x7f3484136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 msgr2=0x7f348412ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3483fff640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 0x7f348412ec20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34740ab000 tx=0x7f3474019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34740bc070 con 0x7f3484135940 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: dump: 2024-09-17T15:51:36.504 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34740b8880 con 0x7f3484135940 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34740a5620 con 0x7f3484135940 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f34817fa640 10 monclient: dump: 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.506 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f3482ffd640 4 set_mon_vals no callback set 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 5 monclient: authenticate success, global_id 274124 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient: _renew_subs 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3484075000 con 0x7f3484135940 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient: _renew_subs 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34841396d0 con 0x7f3484135940 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 0x7f3484137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.490+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3484139350 con 0x7f3484136e90 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f34740b8c70 con 0x7f3484135940 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 0x7f3484137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient: get_auth_request con 0x7f3484136e90 auth_method 0 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f346403bbb0 0x7f346403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(859..859 src has 794..859) ==== 4783+0+0 (secure 0 0 0) 0x7f34740ea370 con 0x7f3484135940 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3489043640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f346403bbb0 0x7f346403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3489043640 10 monclient: get_auth_request con 0x7f346403bbb0 auth_method 0 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 10 monclient(hunting): handle_auth_done global_id 254254 payload 824 2024-09-17T15:51:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34841377b0 con 0x7f3484136e90 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 msgr2=0x7f3484137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3488842640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 0x7f3484137270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f346c00f4b0 tx=0x7f346c00f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c019020 con 0x7f3484136e90 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f34817fa640 10 monclient: dump: 2024-09-17T15:51:36.512 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.513 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.494+0000 7f3489043640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f346403bbb0 0x7f346403e070 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f34841347e0 tx=0x7f347c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f346c014a80 con 0x7f3484136e90 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 msgr2=0x7f3484137270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 0x7f3484137270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f346c00f4b0 tx=0x7f346c00f970 comp rx=0 tx=0).stop 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 msgr2=0x7f3484137270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.498+0000 7f34817fa640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3484136e90 0x7f3484137270 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f346c00f4b0 tx=0x7f346c00f970 comp rx=0 tx=0).stop 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f348aaa6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f348aaa6640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 0x7f348409be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f348aaa6640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3484074aa0 con 0x7f348409ba70 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 0x7f348409be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient: get_auth_request con 0x7f348409ba70 auth_method 0 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 10 monclient(hunting): handle_auth_done global_id 254257 payload 824 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.502+0000 7f3488842640 1 -- 172.21.15.86:0/1610457035 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3484130030 con 0x7f348409ba70 2024-09-17T15:51:36.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f3488842640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 msgr2=0x7f348409be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f3488842640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 0x7f348409be50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f346c01a000 tx=0x7f346c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c019070 con 0x7f348409ba70 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 10 monclient: dump: 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.516 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f346c004230 con 0x7f348409ba70 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 msgr2=0x7f348409be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 0x7f348409be50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f346c01a000 tx=0x7f346c0046d0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 msgr2=0x7f348409be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:36.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f34817fa640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f348409ba70 0x7f348409be50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 112451, 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:36.518 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.519 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:36.520 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.521 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:36.522 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:36.523 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:36.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.525 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.527 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.529 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.530 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:36.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:36.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:36.533 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:36.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:36.533 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:36.533 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:36.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f346403bbb0 msgr2=0x7f346403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f346403bbb0 0x7f346403e070 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f34841347e0 tx=0x7f347c009190 comp rx=0 tx=0).stop 2024-09-17T15:51:36.534 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 10 monclient: shutdown 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 msgr2=0x7f348412ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 --2- 172.21.15.86:0/1610457035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3484135940 0x7f348412ec20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f34740ab000 tx=0x7f3474019650 comp rx=0 tx=0).stop 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f3489043640 1 -- 172.21.15.86:0/1610457035 reap_dead start 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f3489043640 1 -- 172.21.15.86:0/1610457035 reap_dead start 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 shutdown_connections 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 >> 172.21.15.86:0/1610457035 conn(0x7f34840a3630 msgr2=0x7f3484134950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 shutdown_connections 2024-09-17T15:51:36.535 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.506+0000 7f348aaa6640 1 -- 172.21.15.86:0/1610457035 wait complete. 2024-09-17T15:51:36.535 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:36.536 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:51:36.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:51:36.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.646+0000 7fc84b22d640 10 monclient: get_monmap_and_config 2024-09-17T15:51:36.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.646+0000 7fc84b22d640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.646+0000 7fc84b22d640 10 monclient: monmap: 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:36.651488+0000 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:36.651488+0000 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:36.648 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:36.649 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:36.649 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:36.649 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 Processor -- start 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 -- start start 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient: init 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc8440689d0 0x7fc84406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-c con 0x7fc8440689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:36.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc844068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc84413aff0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84409c8b0 con 0x7fc84413aff0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc84409cb90 con 0x7fc844068db0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fc844074500 con 0x7fc8440689d0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc84b22d640 10 monclient(hunting): authenticate will time out at 2429.616905s 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc8497ca640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc84413aff0 0x7fc84413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46372/0 (socket says 172.21.15.86:46372) 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc8497ca640 1 -- 172.21.15.86:0/3670896039 learned_addr learned my addr 172.21.15.86:0/3670896039 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 1 --2- 172.21.15.86:0/3670896039 >> v2:172.21.15.203:3300/0 conn(0x7fc8440689d0 0x7fc84406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): get_auth_request con 0x7fc8440689d0 auth_method 0 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient(hunting): handle_auth_done global_id 254260 payload 274 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 1 -- 172.21.15.86:0/3670896039 >> v1:172.21.15.203:6789/0 conn(0x7fc844068db0 legacy=0x7fc844069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 1 -- 172.21.15.86:0/3670896039 >> v1:172.21.15.86:6789/0 conn(0x7fc84413aff0 legacy=0x7fc84413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 1 monclient: found mon.noname-c 2024-09-17T15:51:36.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:51:36.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.650+0000 7fc848fc9640 1 -- 172.21.15.86:0/3670896039 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84409ce70 con 0x7fc8440689d0 2024-09-17T15:51:36.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc848fc9640 1 -- 172.21.15.86:0/3670896039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 msgr2=0x7fc84406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc848fc9640 1 --2- 172.21.15.86:0/3670896039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc84406f840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc84406fdb0 tx=0x7fc83801c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9320101b6f851ce8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 1 -- 172.21.15.86:0/3670896039 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8380b0070 con 0x7fc8440689d0 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: dump: 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.656 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 1 -- 172.21.15.86:0/3670896039 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8380ab600 con 0x7fc8440689d0 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 5 monclient: authenticate success, global_id 254260 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 4 set_mon_vals no callback set 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 1 -- 172.21.15.86:0/3670896039 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84409c2d0 con 0x7fc8440689d0 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc8437fe640 10 monclient: dump: 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.658 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 10 monclient: shutdown 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 -- 172.21.15.86:0/3670896039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 msgr2=0x7fc84406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 --2- 172.21.15.86:0/3670896039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc84406f840 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc84406fdb0 tx=0x7fc83801c340 comp rx=0 tx=0).stop 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 -- 172.21.15.86:0/3670896039 shutdown_connections 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 --2- 172.21.15.86:0/3670896039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc84406f840 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 -- 172.21.15.86:0/3670896039 >> 172.21.15.86:0/3670896039 conn(0x7fc8440a6440 msgr2=0x7fc8440a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 -- 172.21.15.86:0/3670896039 shutdown_connections 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.654+0000 7fc84b22d640 1 -- 172.21.15.86:0/3670896039 wait complete. 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient: build_initial_monmap 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient: monmap: 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:36.660 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:36.661 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 Processor -- start 2024-09-17T15:51:36.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 -- start start 2024-09-17T15:51:36.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient: init 2024-09-17T15:51:36.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:36.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc844072b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-b con 0x7fc8440689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-c con 0x7fc844068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): picked mon.noname-a con 0x7fc84413aff0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc844141c30 con 0x7fc84413aff0 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc844141ab0 con 0x7fc8440689d0 2024-09-17T15:51:36.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc844141db0 con 0x7fc844068db0 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc84b22d640 10 monclient(hunting): authenticate will time out at 2429.625372s 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50226/0 (socket says 172.21.15.86:50226) 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:59164/0 (socket says 172.21.15.86:59164) 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 1 -- 172.21.15.86:0/2155706381 learned_addr learned my addr 172.21.15.86:0/2155706381 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): get_auth_request con 0x7fc84413aff0 auth_method 0 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): get_auth_request con 0x7fc844068db0 auth_method 0 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc848fc9640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc844072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc848fc9640 10 monclient(hunting): get_auth_request con 0x7fc8440689d0 auth_method 0 2024-09-17T15:51:36.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc848fc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc848fc9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc848fc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient(hunting): handle_auth_done global_id 274127 payload 824 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 -- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 msgr2=0x7fc844072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc844072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 -- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 msgr2=0x7fc84409d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 monclient: found mon.noname-c 2024-09-17T15:51:36.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc843fff640 1 -- 172.21.15.86:0/2155706381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc844121290 con 0x7fc844068db0 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.658+0000 7fc8497ca640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc843fff640 1 -- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 msgr2=0x7fc844135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc843fff640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c3e0 tx=0x7fc83000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8300119a0 con 0x7fc844068db0 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: dump: 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.670 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:36.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc830017070 con 0x7fc844068db0 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc830011d80 con 0x7fc844068db0 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 5 monclient: authenticate success, global_id 274127 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _renew_subs 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc842ffd640 4 set_mon_vals no callback set 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc844120fb0 con 0x7fc844068db0 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 10 monclient: dump: 2024-09-17T15:51:36.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _renew_subs 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8441360f0 con 0x7fc844068db0 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc84406f840 con 0x7fc844068db0 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc83001c460 con 0x7fc844068db0 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 0x7fc82803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(859..859 src has 794..859) ==== 4783+0+0 (secure 0 0 0) 0x7fc83004fd90 con 0x7fc844068db0 2024-09-17T15:51:36.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc848fc9640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 0x7fc82803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:36.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.662+0000 7fc848fc9640 10 monclient: get_auth_request con 0x7fc82803bb60 auth_method 0 2024-09-17T15:51:36.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.666+0000 7fc848fc9640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 0x7fc82803e020 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc8380ab000 tx=0x7fc838008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:36.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.666+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fc830010cd0 con 0x7fc844068db0 2024-09-17T15:51:36.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.666+0000 7fc8417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:36.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.666+0000 7fc8417fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:36.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.894+0000 7fc84b22d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:51:36.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.894+0000 7fc84b22d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:51:36.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.894+0000 7fc84b22d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:36.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.894+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc8441363b0 con 0x7fc844068db0 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=12,kv=11,logm=46} crc {auth=4202762319,config=555582061,health=2612668575,kv=3591778543,logm=3819074969}) 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=4202762319,config=3541143443,health=2612668575,kv=3591778543,logm=2554012436}) 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=12,kv=11,logm=46} crc {auth=4202762319,config=555582061,health=2612668575,kv=3591778543,logm=3819074969}) 2024-09-17T15:51:36.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=4202762319,config=3541143443,health=3259022870,kv=215610202,logm=1600624055}) 2024-09-17T15:51:36.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.898+0000 7fc8417fa640 1 -- 172.21.15.86:0/2155706381 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc830012420 con 0x7fc844068db0 2024-09-17T15:51:36.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.898+0000 7fc8417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:51:36.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.898+0000 7fc8417fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425528188}) 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537578235}) 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3425528188}) 2024-09-17T15:51:36.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532515317}) 2024-09-17T15:51:36.902 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 msgr2=0x7fc82803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 0x7fc82803e020 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc8380ab000 tx=0x7fc838008040 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 10 monclient: shutdown 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 msgr2=0x7fc844135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc83000c3e0 tx=0x7fc83000c8a0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 shutdown_connections 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc82803bb60 0x7fc82803e020 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc84413aff0 0x7fc84409d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc844068db0 0x7fc844135bb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 --2- 172.21.15.86:0/2155706381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8440689d0 0x7fc844072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:36.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 >> 172.21.15.86:0/2155706381 conn(0x7fc8440a6440 msgr2=0x7fc84413c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:36.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 shutdown_connections 2024-09-17T15:51:36.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:36.902+0000 7fc84b22d640 1 -- 172.21.15.86:0/2155706381 wait complete. 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779752460}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2937785593}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2779752460}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403401536}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138357497}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786870374}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138357497}) 2024-09-17T15:51:36.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383491509}) 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809289983}) 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173135516}) 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809289983}) 2024-09-17T15:51:36.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3673106790}) 2024-09-17T15:51:36.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108543249}) 2024-09-17T15:51:36.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4292302698}) 2024-09-17T15:51:36.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108543249}) 2024-09-17T15:51:36.912 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215307732}) 2024-09-17T15:51:36.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561659202}) 2024-09-17T15:51:36.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457299661}) 2024-09-17T15:51:36.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561659202}) 2024-09-17T15:51:36.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129007568}) 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821158668}) 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683280324}) 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821158668}) 2024-09-17T15:51:36.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167785349}) 2024-09-17T15:51:36.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619406125}) 2024-09-17T15:51:36.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644999265}) 2024-09-17T15:51:36.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619406125}) 2024-09-17T15:51:36.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397240775}) 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253312459}) 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391866363}) 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=253312459}) 2024-09-17T15:51:36.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2938616146}) 2024-09-17T15:51:36.922 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916671028}) 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2964710955}) 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916671028}) 2024-09-17T15:51:36.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861458640}) 2024-09-17T15:51:36.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923447630}) 2024-09-17T15:51:36.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149952473}) 2024-09-17T15:51:36.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:36.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923447630}) 2024-09-17T15:51:36.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:36.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2038885730}) 2024-09-17T15:51:36.937 INFO:tasks.rados.rados.0.smithi203.stdout:3942: done (3 left) 2024-09-17T15:51:36.937 INFO:tasks.rados.rados.0.smithi203.stdout:3943: done (2 left) 2024-09-17T15:51:36.937 INFO:tasks.rados.rados.0.smithi203.stdout:3944: done (1 left) 2024-09-17T15:51:36.937 INFO:tasks.rados.rados.0.smithi203.stdout:3945: done (0 left) 2024-09-17T15:51:36.937 INFO:tasks.rados.rados.0.smithi203.stdout:3946: snap_remove snap 413 2024-09-17T15:51:37.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226999586}) 2024-09-17T15:51:37.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289804218}) 2024-09-17T15:51:37.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226999586}) 2024-09-17T15:51:37.604 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.602+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918376004}) 2024-09-17T15:51:37.878 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329903766}) 2024-09-17T15:51:37.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1330749218}) 2024-09-17T15:51:37.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329903766}) 2024-09-17T15:51:37.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.878+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027718928}) 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991124594}) 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179424362}) 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991124594}) 2024-09-17T15:51:37.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690721840}) 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485947797}) 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627899270}) 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485947797}) 2024-09-17T15:51:37.884 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.882+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375011542}) 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879807016}) 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005610462}) 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3879807016}) 2024-09-17T15:51:37.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.886+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=755715435}) 2024-09-17T15:51:37.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712570581}) 2024-09-17T15:51:37.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812358681}) 2024-09-17T15:51:37.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712570581}) 2024-09-17T15:51:37.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=732041307}) 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705677426}) 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4013962100}) 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=705677426}) 2024-09-17T15:51:37.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.890+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169991071}) 2024-09-17T15:51:37.893 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.893 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800230172}) 2024-09-17T15:51:37.893 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833353125}) 2024-09-17T15:51:37.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800230172}) 2024-09-17T15:51:37.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2539096361}) 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62791646}) 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2814222608}) 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.894+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62791646}) 2024-09-17T15:51:37.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173813762}) 2024-09-17T15:51:37.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239945414}) 2024-09-17T15:51:37.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467419114}) 2024-09-17T15:51:37.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239945414}) 2024-09-17T15:51:37.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.898+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709137262}) 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773786655}) 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286482933}) 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773786655}) 2024-09-17T15:51:37.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2791931817}) 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985081722}) 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964679278}) 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985081722}) 2024-09-17T15:51:37.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.902+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=248796219}) 2024-09-17T15:51:37.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006753704}) 2024-09-17T15:51:37.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049445915}) 2024-09-17T15:51:37.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006753704}) 2024-09-17T15:51:37.906 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406871703}) 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185999519}) 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.906+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2406125547}) 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185999519}) 2024-09-17T15:51:37.908 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276790631}) 2024-09-17T15:51:37.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.910 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152947868}) 2024-09-17T15:51:37.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603980685}) 2024-09-17T15:51:37.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152947868}) 2024-09-17T15:51:37.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.910+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196916634}) 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974254408}) 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699731500}) 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3974254408}) 2024-09-17T15:51:37.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808146787}) 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617519364}) 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3309678583}) 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617519364}) 2024-09-17T15:51:37.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.914+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439022164}) 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632085283}) 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1000226692}) 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632085283}) 2024-09-17T15:51:37.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.918+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143039011}) 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142868253}) 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909776274}) 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142868253}) 2024-09-17T15:51:37.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1920330018}) 2024-09-17T15:51:37.922 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.922 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40761628}) 2024-09-17T15:51:37.922 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661973578}) 2024-09-17T15:51:37.922 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40761628}) 2024-09-17T15:51:37.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.922+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3930801292}) 2024-09-17T15:51:37.923 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:51:37.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389162735}) 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1069028329}) 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389162735}) 2024-09-17T15:51:37.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281287126}) 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559474936}) 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1359088033}) 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559474936}) 2024-09-17T15:51:37.927 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.926+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520734883}) 2024-09-17T15:51:37.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824659127}) 2024-09-17T15:51:37.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357593772}) 2024-09-17T15:51:37.929 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1824659127}) 2024-09-17T15:51:37.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.930+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1361804136}) 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346918246}) 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=731547585}) 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.934+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346918246}) 2024-09-17T15:51:37.936 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.938+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513071966}) 2024-09-17T15:51:37.957 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546456157}) 2024-09-17T15:51:37.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3207769996}) 2024-09-17T15:51:37.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546456157}) 2024-09-17T15:51:37.958 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.958+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375445249}) 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.970+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.974+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521422105}) 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.974+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1881314599}) 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.974+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.974+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521422105}) 2024-09-17T15:51:37.972 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:37.974+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=250469206}) 2024-09-17T15:51:37.976 INFO:tasks.rados.rados.0.smithi203.stdout:3946: done (0 left) 2024-09-17T15:51:37.976 INFO:tasks.rados.rados.0.smithi203.stdout:3947: copy_from oid 37 from oid 40 current snap is 422 2024-09-17T15:51:37.976 INFO:tasks.rados.rados.0.smithi203.stdout:3948: write (excl) oid 22 current snap is 422 2024-09-17T15:51:37.976 INFO:tasks.rados.rados.0.smithi203.stdout:3948: seq_num 1389 ranges {544923=497107,1647724=457823,2710537=428120} 2024-09-17T15:51:37.992 INFO:tasks.rados.rados.0.smithi203.stdout:3948: writing smithi20328086-22 from 544923 to 1042030 tid 1 2024-09-17T15:51:38.005 INFO:tasks.rados.rados.0.smithi203.stdout:3948: writing smithi20328086-22 from 1647724 to 2105547 tid 2 2024-09-17T15:51:38.020 INFO:tasks.rados.rados.0.smithi203.stdout:3948: writing smithi20328086-22 from 2710537 to 3138657 tid 3 2024-09-17T15:51:38.020 INFO:tasks.rados.rados.0.smithi203.stdout:3949: copy_from oid 46 from oid 33 current snap is 422 2024-09-17T15:51:38.020 INFO:tasks.rados.rados.0.smithi203.stdout:3947: finishing copy_from to smithi20328086-37 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3947: got expected ENOENT (src dne) 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3947: finishing copy_from racing read to smithi20328086-37 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3948: finishing write tid 1 to smithi20328086-22 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3948: finishing write tid 2 to smithi20328086-22 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3947: done (2 left) 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3950: read oid 1 snap -1 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3950: expect (ObjNum 1387 snap 419 seq_num 1387) 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3951: read oid 4 snap 388 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3951: expect deleted 2024-09-17T15:51:38.021 INFO:tasks.rados.rados.0.smithi203.stdout:3952: delete oid 21 current snap is 422 2024-09-17T15:51:38.025 INFO:tasks.rados.rados.0.smithi203.stdout:3948: finishing write tid 3 to smithi20328086-22 2024-09-17T15:51:38.025 INFO:tasks.rados.rados.0.smithi203.stdout:3951: done (4 left) 2024-09-17T15:51:38.025 INFO:tasks.rados.rados.0.smithi203.stdout:3952: done (3 left) 2024-09-17T15:51:38.025 INFO:tasks.rados.rados.0.smithi203.stdout:3953: delete oid 11 current snap is 422 2024-09-17T15:51:38.026 INFO:tasks.rados.rados.0.smithi203.stdout:3948: finishing write tid 4 to smithi20328086-22 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: finishing write tid 5 to smithi20328086-22 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: oid 22 updating version 0 to 1694 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: oid 22 version 1694 is already newer than 1692 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: oid 22 version 1694 is already newer than 1693 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: oid 22 version 1694 is already newer than 1691 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1694 (ObjNum 1389 snap 422 seq_num 1389) dirty exists 2024-09-17T15:51:38.027 INFO:tasks.rados.rados.0.smithi203.stdout:3948: left oid 22 (ObjNum 1389 snap 422 seq_num 1389) 2024-09-17T15:51:38.031 INFO:tasks.rados.rados.0.smithi203.stdout:3948: done (3 left) 2024-09-17T15:51:38.031 INFO:tasks.rados.rados.0.smithi203.stdout:3953: done (2 left) 2024-09-17T15:51:38.031 INFO:tasks.rados.rados.0.smithi203.stdout:3954: rollback oid 43 current snap is 422 2024-09-17T15:51:38.031 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 43 to 408 2024-09-17T15:51:38.031 INFO:tasks.rados.rados.0.smithi203.stdout:3955: delete oid 50 current snap is 422 2024-09-17T15:51:38.034 INFO:tasks.rados.rados.0.smithi203.stdout:3950: expect (ObjNum 1387 snap 419 seq_num 1387) 2024-09-17T15:51:38.040 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 598101 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: get_monmap_and_config 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: monmap: 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:38.069273+0000 2024-09-17T15:51:38.065 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:38.069273+0000 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:38.066 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 Processor -- start 2024-09-17T15:51:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 -- start start 2024-09-17T15:51:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: init 2024-09-17T15:51:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f53500689d0 0x7f535009ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-c con 0x7f53500689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5350068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-b con 0x7f5350139f00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f535009d770 con 0x7f5350068db0 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f535009ed00 con 0x7f53500689d0 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f535009efe0 con 0x7f5350139f00 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5356c94640 10 monclient(hunting): authenticate will time out at 2431.034836s 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56610/0 (socket says 172.21.15.86:56610) 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 -- 172.21.15.86:0/2296076905 learned_addr learned my addr 172.21.15.86:0/2296076905 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): get_auth_request con 0x7f5350139f00 auth_method 0 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5354a30640 1 --2- 172.21.15.86:0/2296076905 >> v2:172.21.15.203:3300/0 conn(0x7f53500689d0 0x7f535009ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5354a30640 10 monclient(hunting): get_auth_request con 0x7f53500689d0 auth_method 0 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5354a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5354a30640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5354a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient(hunting): handle_auth_done global_id 274130 payload 274 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 -- 172.21.15.86:0/2296076905 >> v2:172.21.15.203:3300/0 conn(0x7f53500689d0 msgr2=0x7f535009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 --2- 172.21.15.86:0/2296076905 >> v2:172.21.15.203:3300/0 conn(0x7f53500689d0 0x7f535009ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 -- 172.21.15.86:0/2296076905 >> v1:172.21.15.203:6789/0 conn(0x7f5350068db0 legacy=0x7f53500691c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 monclient: found mon.noname-b 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.066+0000 7f5355231640 1 -- 172.21.15.86:0/2296076905 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53500a04e0 con 0x7f5350139f00 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.070+0000 7f5355231640 1 -- 172.21.15.86:0/2296076905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 msgr2=0x7f535013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.070+0000 7f5355231640 1 --2- 172.21.15.86:0/2296076905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53500a0770 tx=0x7f534001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d139ab5bfd5f03b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 1 -- 172.21.15.86:0/2296076905 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400b0070 con 0x7f5350139f00 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: dump: 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 1 -- 172.21.15.86:0/2296076905 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53400ab600 con 0x7f5350139f00 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 1 -- 172.21.15.86:0/2296076905 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53400ab9d0 con 0x7f5350139f00 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 5 monclient: authenticate success, global_id 274130 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 4 set_mon_vals no callback set 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534f7fe640 10 monclient: dump: 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: shutdown 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- 172.21.15.86:0/2296076905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 msgr2=0x7f535013c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- 172.21.15.86:0/2296076905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f53500a0770 tx=0x7f534001c490 comp rx=0 tx=0).stop 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- 172.21.15.86:0/2296076905 shutdown_connections 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- 172.21.15.86:0/2296076905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350139f00 0x7f535013c2f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- 172.21.15.86:0/2296076905 >> v2:172.21.15.203:3300/0 conn(0x7f53500689d0 0x7f535009ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- 172.21.15.86:0/2296076905 >> 172.21.15.86:0/2296076905 conn(0x7f53500a6440 msgr2=0x7f53500a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- 172.21.15.86:0/2296076905 shutdown_connections 2024-09-17T15:51:38.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- 172.21.15.86:0/2296076905 wait complete. 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: monmap: 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:38.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 Processor -- start 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- start start 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: init 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53500689d0 0x7f53501d0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-b con 0x7f53500689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-c con 0x7f5350068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5350139f00 0x7f53501d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): picked mon.noname-a con 0x7f5350139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f535009beb0 con 0x7f5350139f00 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f535009bd30 con 0x7f53500689d0 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f535009c030 con 0x7f5350068db0 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5356c94640 10 monclient(hunting): authenticate will time out at 2431.042649s 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56626/0 (socket says 172.21.15.86:56626) 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 -- 172.21.15.86:0/2166006108 learned_addr learned my addr 172.21.15.86:0/2166006108 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): get_auth_request con 0x7f5350068db0 auth_method 0 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5350139f00 0x7f53501d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): get_auth_request con 0x7f5350139f00 auth_method 0 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5354a30640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53500689d0 0x7f53501d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5354a30640 10 monclient(hunting): get_auth_request con 0x7f53500689d0 auth_method 0 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5354a30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5354a30640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5354a30640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f5355231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient(hunting): handle_auth_done global_id 274133 payload 824 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 -- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53500689d0 msgr2=0x7f53501d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53500689d0 0x7f53501d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 -- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5350139f00 msgr2=0x7f53501d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5350139f00 0x7f53501d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 monclient: found mon.noname-c 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.074+0000 7f534ffff640 1 -- 172.21.15.86:0/2166006108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f535013bee0 con 0x7f5350068db0 2024-09-17T15:51:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534ffff640 1 -- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 msgr2=0x7f53501d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534ffff640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f533800c7c0 tx=0x7f533800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338016020 con 0x7f5350068db0 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: dump: 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.086 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 5 monclient: authenticate success, global_id 274133 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f533800abe0 con 0x7f5350068db0 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _renew_subs 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5338012420 con 0x7f5350068db0 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f535009d460 con 0x7f5350068db0 2024-09-17T15:51:38.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534e7fc640 4 set_mon_vals no callback set 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 10 monclient: dump: 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _renew_subs 2024-09-17T15:51:38.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5350074060 con 0x7f5350068db0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f535009ffa0 con 0x7f5350068db0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5338019020 con 0x7f5350068db0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 0x7f533003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f533804ffd0 con 0x7f5350068db0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5354a30640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 0x7f533003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.078+0000 7f5354a30640 10 monclient: get_auth_request con 0x7f533003bb90 auth_method 0 2024-09-17T15:51:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.082+0000 7f5354a30640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 0x7f533003e050 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f534800b9d0 tx=0x7f5348008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.082+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5338020030 con 0x7f5350068db0 2024-09-17T15:51:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.082+0000 7f534d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.082+0000 7f534d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:38.111 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 67387 2024-09-17T15:51:38.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815076929}) 2024-09-17T15:51:38.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575527037}) 2024-09-17T15:51:38.176 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1815076929}) 2024-09-17T15:51:38.177 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=438751728}) 2024-09-17T15:51:38.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156430424}) 2024-09-17T15:51:38.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1249008884}) 2024-09-17T15:51:38.178 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.179 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156430424}) 2024-09-17T15:51:38.179 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.178+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370065849}) 2024-09-17T15:51:38.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812308107}) 2024-09-17T15:51:38.180 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=373981733}) 2024-09-17T15:51:38.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3812308107}) 2024-09-17T15:51:38.181 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3599058001}) 2024-09-17T15:51:38.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90633598}) 2024-09-17T15:51:38.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691218948}) 2024-09-17T15:51:38.182 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90633598}) 2024-09-17T15:51:38.183 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.182+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243472478}) 2024-09-17T15:51:38.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341894498}) 2024-09-17T15:51:38.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2358639372}) 2024-09-17T15:51:38.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.184 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341894498}) 2024-09-17T15:51:38.185 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236850353}) 2024-09-17T15:51:38.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270772307}) 2024-09-17T15:51:38.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1856143972}) 2024-09-17T15:51:38.186 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270772307}) 2024-09-17T15:51:38.187 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.186+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774169531}) 2024-09-17T15:51:38.188 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.188 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394835427}) 2024-09-17T15:51:38.188 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058902837}) 2024-09-17T15:51:38.188 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394835427}) 2024-09-17T15:51:38.189 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711116959}) 2024-09-17T15:51:38.190 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.190 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104790007}) 2024-09-17T15:51:38.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524095387}) 2024-09-17T15:51:38.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=104790007}) 2024-09-17T15:51:38.191 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.190+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083706692}) 2024-09-17T15:51:38.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031783100}) 2024-09-17T15:51:38.192 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754616741}) 2024-09-17T15:51:38.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031783100}) 2024-09-17T15:51:38.193 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858832390}) 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547322926}) 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866834976}) 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547322926}) 2024-09-17T15:51:38.194 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.194+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281636834}) 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083741095}) 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=743256284}) 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3083741095}) 2024-09-17T15:51:38.196 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082101500}) 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689620252}) 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156479932}) 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689620252}) 2024-09-17T15:51:38.198 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.198+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299094114}) 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784470635}) 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065147326}) 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784470635}) 2024-09-17T15:51:38.200 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798770720}) 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573103557}) 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395246740}) 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573103557}) 2024-09-17T15:51:38.202 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091243437}) 2024-09-17T15:51:38.203 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099683228}) 2024-09-17T15:51:38.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833719583}) 2024-09-17T15:51:38.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099683228}) 2024-09-17T15:51:38.204 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.202+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926890099}) 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785379763}) 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204362998}) 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785379763}) 2024-09-17T15:51:38.206 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.206+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289055461}) 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785948335}) 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717278739}) 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785948335}) 2024-09-17T15:51:38.208 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706462448}) 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855733807}) 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157601865}) 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=855733807}) 2024-09-17T15:51:38.210 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821304131}) 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220596125}) 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625357329}) 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220596125}) 2024-09-17T15:51:38.212 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.210+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201450243}) 2024-09-17T15:51:38.213 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679237893}) 2024-09-17T15:51:38.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709282878}) 2024-09-17T15:51:38.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679237893}) 2024-09-17T15:51:38.214 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3671417455}) 2024-09-17T15:51:38.215 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211022367}) 2024-09-17T15:51:38.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147881022}) 2024-09-17T15:51:38.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211022367}) 2024-09-17T15:51:38.216 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.214+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365564419}) 2024-09-17T15:51:38.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.217 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473324272}) 2024-09-17T15:51:38.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849703939}) 2024-09-17T15:51:38.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=473324272}) 2024-09-17T15:51:38.218 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93063694}) 2024-09-17T15:51:38.219 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.219 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512943140}) 2024-09-17T15:51:38.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267689272}) 2024-09-17T15:51:38.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512943140}) 2024-09-17T15:51:38.220 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.218+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1749808769}) 2024-09-17T15:51:38.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079753459}) 2024-09-17T15:51:38.221 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2026444094}) 2024-09-17T15:51:38.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079753459}) 2024-09-17T15:51:38.222 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096238889}) 2024-09-17T15:51:38.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717737847}) 2024-09-17T15:51:38.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965525805}) 2024-09-17T15:51:38.223 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.224 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717737847}) 2024-09-17T15:51:38.224 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.222+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=482622987}) 2024-09-17T15:51:38.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.225 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782152265}) 2024-09-17T15:51:38.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960951872}) 2024-09-17T15:51:38.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782152265}) 2024-09-17T15:51:38.226 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190258919}) 2024-09-17T15:51:38.228 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.228 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756888471}) 2024-09-17T15:51:38.228 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567879512}) 2024-09-17T15:51:38.228 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.228 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756888471}) 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.226+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4113230454}) 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499044245}) 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454813163}) 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499044245}) 2024-09-17T15:51:38.229 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3014077788}) 2024-09-17T15:51:38.231 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541122797}) 2024-09-17T15:51:38.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1959684453}) 2024-09-17T15:51:38.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541122797}) 2024-09-17T15:51:38.232 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.230+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242775790}) 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478549537}) 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705762608}) 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478549537}) 2024-09-17T15:51:38.233 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046352800}) 2024-09-17T15:51:38.235 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2297918 2024-09-17T15:51:38.235 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.235 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050490721}) 2024-09-17T15:51:38.235 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=890309779}) 2024-09-17T15:51:38.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050490721}) 2024-09-17T15:51:38.236 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.234+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865173131}) 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326137829}) 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781725449}) 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=326137829}) 2024-09-17T15:51:38.237 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1166029056}) 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133840094}) 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3774039650}) 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133840094}) 2024-09-17T15:51:38.239 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.238+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247388270}) 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048767798}) 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236527541}) 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048767798}) 2024-09-17T15:51:38.241 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2292959611}) 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288528646}) 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790010762}) 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288528646}) 2024-09-17T15:51:38.243 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.242+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849161521}) 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356797989}) 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2209690259}) 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356797989}) 2024-09-17T15:51:38.245 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4006236510}) 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470179251}) 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2745544714}) 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470179251}) 2024-09-17T15:51:38.247 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.246+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240739857}) 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598335226}) 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612743154}) 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598335226}) 2024-09-17T15:51:38.249 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774244994}) 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836609550}) 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372949267}) 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=836609550}) 2024-09-17T15:51:38.251 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.250+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332710594}) 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813792942}) 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148579297}) 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1813792942}) 2024-09-17T15:51:38.253 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1460238156}) 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660549684}) 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695618384}) 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1660549684}) 2024-09-17T15:51:38.255 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.254+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504974294}) 2024-09-17T15:51:38.257 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.257 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649084217}) 2024-09-17T15:51:38.257 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2955345665}) 2024-09-17T15:51:38.257 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.257 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649084217}) 2024-09-17T15:51:38.258 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1464788176}) 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565129559}) 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3295952401}) 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565129559}) 2024-09-17T15:51:38.259 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.258+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819375837}) 2024-09-17T15:51:38.260 INFO:tasks.rados.rados.0.smithi203.stdout:3954: finishing rollback tid 0 to smithi20328086-43 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3954: finishing rollback tid 1 to smithi20328086-43 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3949: finishing copy_from racing read to smithi20328086-46 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3954: finishing rollback tid 2 to smithi20328086-43 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 43 v 763 (ObjNum 1285 snap 396 seq_num 1285) dirty exists 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3949: finishing copy_from to smithi20328086-46 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 46 v 1444 (ObjNum 1362 snap 414 seq_num 1362) dirty exists 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3949: done (3 left) 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3950: done (2 left) 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3954: done (1 left) 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3955: done (0 left) 2024-09-17T15:51:38.261 INFO:tasks.rados.rados.0.smithi203.stdout:3956: rollback oid 3 current snap is 422 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 3 to 417 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:3957: rollback oid 44 current snap is 422 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 44 to 420 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:3958: read oid 39 snap -1 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:3958: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:38.262 INFO:tasks.rados.rados.0.smithi203.stdout:3959: delete oid 36 current snap is 422 2024-09-17T15:51:38.262 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.263 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198713535}) 2024-09-17T15:51:38.263 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1045980657}) 2024-09-17T15:51:38.263 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.263 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198713535}) 2024-09-17T15:51:38.263 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.262+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429326447}) 2024-09-17T15:51:38.266 INFO:tasks.rados.rados.0.smithi203.stdout:3959: done (3 left) 2024-09-17T15:51:38.266 INFO:tasks.rados.rados.0.smithi203.stdout:3960: rollback oid 11 current snap is 422 2024-09-17T15:51:38.266 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 11 to 420 2024-09-17T15:51:38.266 INFO:tasks.rados.rados.0.smithi203.stdout:3961: snap_remove snap 418 2024-09-17T15:51:38.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.266 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621505498}) 2024-09-17T15:51:38.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=156052255}) 2024-09-17T15:51:38.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621505498}) 2024-09-17T15:51:38.267 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.266+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3416848375}) 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930190724}) 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160889638}) 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930190724}) 2024-09-17T15:51:38.270 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.270+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235523359}) 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861826063}) 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077275809}) 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861826063}) 2024-09-17T15:51:38.272 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398296162}) 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472050087}) 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976949518}) 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472050087}) 2024-09-17T15:51:38.274 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1695885292}) 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463179869}) 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025832334}) 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.274+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2463179869}) 2024-09-17T15:51:38.276 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4267805382}) 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692495594}) 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2687403805}) 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692495594}) 2024-09-17T15:51:38.278 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102107543}) 2024-09-17T15:51:38.279 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467237075}) 2024-09-17T15:51:38.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914462775}) 2024-09-17T15:51:38.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467237075}) 2024-09-17T15:51:38.280 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.278+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965138428}) 2024-09-17T15:51:38.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843215096}) 2024-09-17T15:51:38.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988468679}) 2024-09-17T15:51:38.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.281 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843215096}) 2024-09-17T15:51:38.282 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3376701550}) 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389889575}) 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1686094232}) 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3389889575}) 2024-09-17T15:51:38.283 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.282+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454757605}) 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830052710}) 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3225038095}) 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830052710}) 2024-09-17T15:51:38.285 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529299267}) 2024-09-17T15:51:38.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408545731}) 2024-09-17T15:51:38.286 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3834875403}) 2024-09-17T15:51:38.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408545731}) 2024-09-17T15:51:38.287 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.286+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1667803176}) 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987931852}) 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=713724338}) 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1987931852}) 2024-09-17T15:51:38.288 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032560449}) 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892841203}) 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949559150}) 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2892841203}) 2024-09-17T15:51:38.290 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781831665}) 2024-09-17T15:51:38.291 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478382546}) 2024-09-17T15:51:38.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3423081086}) 2024-09-17T15:51:38.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478382546}) 2024-09-17T15:51:38.292 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.290+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3481105922}) 2024-09-17T15:51:38.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537849522}) 2024-09-17T15:51:38.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=492523908}) 2024-09-17T15:51:38.293 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537849522}) 2024-09-17T15:51:38.294 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=749596857}) 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884740184}) 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693843313}) 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884740184}) 2024-09-17T15:51:38.295 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.294+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4149373709}) 2024-09-17T15:51:38.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984230323}) 2024-09-17T15:51:38.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741157389}) 2024-09-17T15:51:38.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.297 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2984230323}) 2024-09-17T15:51:38.298 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.298+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752167488}) 2024-09-17T15:51:38.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201517774}) 2024-09-17T15:51:38.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070742640}) 2024-09-17T15:51:38.307 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201517774}) 2024-09-17T15:51:38.308 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.306+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206192517}) 2024-09-17T15:51:38.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239174784}) 2024-09-17T15:51:38.309 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3271888609}) 2024-09-17T15:51:38.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4239174784}) 2024-09-17T15:51:38.310 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546408059}) 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505908491}) 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028591467}) 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505908491}) 2024-09-17T15:51:38.311 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.310+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681674540}) 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062350830}) 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2253421436}) 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062350830}) 2024-09-17T15:51:38.313 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365060759}) 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265276033}) 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460694934}) 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265276033}) 2024-09-17T15:51:38.315 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.314+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355347276}) 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399316718}) 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=82521935}) 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=399316718}) 2024-09-17T15:51:38.317 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43671766}) 2024-09-17T15:51:38.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106293253}) 2024-09-17T15:51:38.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074284923}) 2024-09-17T15:51:38.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.318 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106293253}) 2024-09-17T15:51:38.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3975699093}) 2024-09-17T15:51:38.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766202535}) 2024-09-17T15:51:38.319 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468149055}) 2024-09-17T15:51:38.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766202535}) 2024-09-17T15:51:38.320 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.318+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590088076}) 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=2} crc {logm=1742433070,mdsmap=3140514009,mgr=1452637520,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=34761450,mon_config_key=268771840,monmap=4191199434,nvmeofgw=598337686,osd_metadata=52536041}) 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=14,monmap=2} crc {logm=4060841564,mdsmap=2815901553,mgr=333540546,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3421286596,mon_config_key=639531838,monmap=2119989493}) 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=2} crc {logm=1742433070,mdsmap=3140514009,mgr=1452637520,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=34761450,mon_config_key=268771840,monmap=4191199434,nvmeofgw=598337686,osd_metadata=52536041}) 2024-09-17T15:51:38.322 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=61,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=2} crc {logm=2236697505,mdsmap=2815901553,mgr=2224701651,mgr_command_descs=1770323759,mgrstat=2756308903,mon_config_key=1003290438,monmap=4191199434,nvmeofgw=1596038039}) 2024-09-17T15:51:38.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=3366526702,osd_snap=2661579049}) 2024-09-17T15:51:38.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=3366526702,osd_snap=3078614533}) 2024-09-17T15:51:38.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.323 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=3366526702,osd_snap=2661579049}) 2024-09-17T15:51:38.324 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.322+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_snap=96} crc {osd_metadata=831666901,osd_snap=4161680986}) 2024-09-17T15:51:38.325 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.325 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=63,osdmap=37} crc {osd_snap=2650325115,osdmap=1050937279}) 2024-09-17T15:51:38.325 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=61,osdmap=39} crc {osd_snap=3695762048,osdmap=9656442}) 2024-09-17T15:51:38.325 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.325 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=63,osdmap=37} crc {osd_snap=2650325115,osdmap=1050937279}) 2024-09-17T15:51:38.326 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.326+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=61,osdmap=39} crc {osd_snap=3110622038,osdmap=2034629618}) 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=79} crc {osdmap=2172885103}) 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=672402335}) 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=79} crc {osdmap=2172885103}) 2024-09-17T15:51:38.339 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:51:38.338+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=1595900166}) 2024-09-17T15:51:38.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f5356c94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:38.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f5356c94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:38.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f5356c94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:38.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5350120cf0 con 0x7f5350068db0 2024-09-17T15:51:38.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f534d7fa640 1 -- 172.21.15.86:0/2166006108 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f5338010cf0 con 0x7f5350068db0 2024-09-17T15:51:38.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f534d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:38.363 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.362+0000 7f534d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:38.363 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 msgr2=0x7f533003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 0x7f533003e050 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f534800b9d0 tx=0x7f5348008040 comp rx=0 tx=0).stop 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 10 monclient: shutdown 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 msgr2=0x7f53501d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f533800c7c0 tx=0x7f533800cc80 comp rx=0 tx=0).stop 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 shutdown_connections 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f533003bb90 0x7f533003e050 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5350139f00 0x7f53501d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5350068db0 0x7f53501d0840 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 --2- 172.21.15.86:0/2166006108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53500689d0 0x7f53501d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 >> 172.21.15.86:0/2166006108 conn(0x7f53500a6440 msgr2=0x7f53500a3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 shutdown_connections 2024-09-17T15:51:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.366+0000 7f5356c94640 1 -- 172.21.15.86:0/2166006108 wait complete. 2024-09-17T15:51:38.382 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:51:38.382 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-17T15:51:38.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:51:38.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: get_monmap_and_config 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: monmap: 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:38.552094+0000 2024-09-17T15:51:38.548 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:38.552094+0000 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:38.549 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 Processor -- start 2024-09-17T15:51:38.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 -- start start 2024-09-17T15:51:38.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: init 2024-09-17T15:51:38.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f46b00a50e0 0x7f46b009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-c con 0x7f46b00a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46b00a54c0 0x7f46b00a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-a con 0x7f46b00a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46b0137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46b00a6280 con 0x7f46b0137ea0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f46b009c0a0 con 0x7f46b00a54c0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f46b009c380 con 0x7f46b00a50e0 2024-09-17T15:51:38.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46b60cf640 10 monclient(hunting): authenticate will time out at 2431.517630s 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46b00a54c0 0x7f46b00a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f46b00a54c0 0x7f46b00a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55094/0 (socket says 172.21.15.86:55094) 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 -- 172.21.15.86:0/1510650655 learned_addr learned my addr 172.21.15.86:0/1510650655 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): get_auth_request con 0x7f46b00a54c0 auth_method 0 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46af7fe640 1 --2- 172.21.15.86:0/1510650655 >> v2:172.21.15.203:3300/0 conn(0x7f46b00a50e0 0x7f46b009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46af7fe640 10 monclient(hunting): get_auth_request con 0x7f46b00a50e0 auth_method 0 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient(hunting): handle_auth_done global_id 234525 payload 274 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 -- 172.21.15.86:0/1510650655 >> v2:172.21.15.203:3300/0 conn(0x7f46b00a50e0 msgr2=0x7f46b009cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1510650655 >> v2:172.21.15.203:3300/0 conn(0x7f46b00a50e0 0x7f46b009cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 -- 172.21.15.86:0/1510650655 >> v1:172.21.15.203:6789/0 conn(0x7f46b0137ea0 legacy=0x7f46b0138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.550+0000 7f46aeffd640 1 -- 172.21.15.86:0/1510650655 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b009c660 con 0x7f46b00a54c0 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46aeffd640 1 -- 172.21.15.86:0/1510650655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a54c0 msgr2=0x7f46b00a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1510650655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a54c0 0x7f46b00a58d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46b00aa620 tx=0x7f469801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227a7d66a5949b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 1 -- 172.21.15.86:0/1510650655 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980b0070 con 0x7f46b00a54c0 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: dump: 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.557 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 1 -- 172.21.15.86:0/1510650655 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46980a4e50 con 0x7f46b00a54c0 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46b60cf640 5 monclient: authenticate success, global_id 234525 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46b60cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46b60cf640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46b60cf640 4 set_mon_vals no callback set 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 1 -- 172.21.15.86:0/1510650655 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980ab820 con 0x7f46b00a54c0 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.554+0000 7f46ae7fc640 10 monclient: dump: 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: shutdown 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- 172.21.15.86:0/1510650655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a54c0 msgr2=0x7f46b00a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1510650655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a54c0 0x7f46b00a58d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46b00aa620 tx=0x7f469801c470 comp rx=0 tx=0).stop 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- 172.21.15.86:0/1510650655 shutdown_connections 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1510650655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a54c0 0x7f46b00a58d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1510650655 >> v2:172.21.15.203:3300/0 conn(0x7f46b00a50e0 0x7f46b009cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- 172.21.15.86:0/1510650655 >> 172.21.15.86:0/1510650655 conn(0x7f46b00aaae0 msgr2=0x7f46b00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- 172.21.15.86:0/1510650655 shutdown_connections 2024-09-17T15:51:38.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- 172.21.15.86:0/1510650655 wait complete. 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: monmap: 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.566 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 Processor -- start 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- start start 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: init 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 0x7f46b01d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-a con 0x7f46b00a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0137ea0 0x7f46b01d0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-b con 0x7f46b0137ea0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:38.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46b01d0cc0 0x7f46b013c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): picked mon.noname-c con 0x7f46b01d0cc0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0142cb0 con 0x7f46b00a50e0 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f46b0142b30 con 0x7f46b0137ea0 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f46b0142e30 con 0x7f46b01d0cc0 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46b60cf640 10 monclient(hunting): authenticate will time out at 2431.526156s 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 0x7f46b01d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46b01d0cc0 0x7f46b013c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 0x7f46b01d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55108/0 (socket says 172.21.15.86:55108) 2024-09-17T15:51:38.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 -- 172.21.15.86:0/1504296447 learned_addr learned my addr 172.21.15.86:0/1504296447 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): get_auth_request con 0x7f46b00a50e0 auth_method 0 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): get_auth_request con 0x7f46b01d0cc0 auth_method 0 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0137ea0 0x7f46b01d0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46aeffd640 10 monclient(hunting): get_auth_request con 0x7f46b0137ea0 auth_method 0 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient(hunting): handle_auth_done global_id 234528 payload 824 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46b01d0cc0 msgr2=0x7f46b013c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46b01d0cc0 0x7f46b013c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0137ea0 msgr2=0x7f46b01d0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46b0137ea0 0x7f46b01d0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.558+0000 7f46af7fe640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46b013dd40 con 0x7f46b00a50e0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46af7fe640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 msgr2=0x7f46b01d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46af7fe640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 0x7f46b01d0050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f46a800d830 tx=0x7f46a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a800b8d0 con 0x7f46b00a50e0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: dump: 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.571 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.572 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 5 monclient: authenticate success, global_id 234528 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a8019070 con 0x7f46b00a50e0 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 monclient: _renew_subs 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46b013e8a0 con 0x7f46b00a50e0 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a800bc70 con 0x7f46b00a50e0 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 10 monclient: dump: 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.573 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46adffb640 4 set_mon_vals no callback set 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 monclient: _renew_subs 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46b0121260 con 0x7f46b00a50e0 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 0x7f46b013cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f46b006a0e0 con 0x7f46b013c7c0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 0x7f46b013cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient: get_auth_request con 0x7f46b013c7c0 auth_method 0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f46a801e460 con 0x7f46b00a50e0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 10 monclient(hunting): handle_auth_done global_id 234531 payload 824 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46aeffd640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46b013d0e0 con 0x7f46b013c7c0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f469403bbe0 0x7f469403e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f46a8050390 con 0x7f46b00a50e0 2024-09-17T15:51:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46affff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f469403bbe0 0x7f469403e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.562+0000 7f46affff640 10 monclient: get_auth_request con 0x7f469403bbe0 auth_method 0 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46aeffd640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 msgr2=0x7f46b013cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 0x7f46b013cba0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f46b006f8c0 tx=0x7f4698019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980ba030 con 0x7f46b013c7c0 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 10 monclient: dump: 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.576 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4698009490 con 0x7f46b013c7c0 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 msgr2=0x7f46b013cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 0x7f46b013cba0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f46b006f8c0 tx=0x7f4698019210 comp rx=0 tx=0).stop 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 msgr2=0x7f46b013cba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46a7fff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b013c7c0 0x7f46b013cba0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.566+0000 7f46affff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f469403bbe0 0x7f469403e0a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f46b009d9f0 tx=0x7f46a0003d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46b60cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:38.579 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 0x7f46b00a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46b60cf640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f46b00a58d0 con 0x7f46b00a2a00 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 0x7f46b00a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient: get_auth_request con 0x7f46b00a2a00 auth_method 0 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 10 monclient(hunting): handle_auth_done global_id 234534 payload 824 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.570+0000 7f46aeffd640 1 -- 172.21.15.86:0/1504296447 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46b0074be0 con 0x7f46b00a2a00 2024-09-17T15:51:38.580 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46aeffd640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 msgr2=0x7f46b00a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46aeffd640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 0x7f46b00a2de0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f469801cd10 tx=0x7f46980a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46980ba070 con 0x7f46b00a2a00 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 10 monclient: dump: 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.581 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f46980c0e50 con 0x7f46b00a2a00 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 msgr2=0x7f46b00a2de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 0x7f46b00a2de0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f469801cd10 tx=0x7f46980a8000 comp rx=0 tx=0).stop 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 msgr2=0x7f46b00a2de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46a7fff640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a2a00 0x7f46b00a2de0 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f469801cd10 tx=0x7f46980a8000 comp rx=0 tx=0).stop 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:38.582 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 227769, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 2, 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:38.583 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:38.584 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.585 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:38.586 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.587 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:38.588 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.589 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.590 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.591 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:38.592 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.593 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:38.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.595 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:38.596 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f469403bbe0 msgr2=0x7f469403e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f469403bbe0 0x7f469403e0a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f46b009d9f0 tx=0x7f46a0003d50 comp rx=0 tx=0).stop 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 10 monclient: shutdown 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 msgr2=0x7f46b01d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 --2- 172.21.15.86:0/1504296447 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46b00a50e0 0x7f46b01d0050 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f46a800d830 tx=0x7f46a800dcf0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46affff640 1 -- 172.21.15.86:0/1504296447 reap_dead start 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46affff640 1 -- 172.21.15.86:0/1504296447 reap_dead start 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 shutdown_connections 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 >> 172.21.15.86:0/1504296447 conn(0x7f46b00aaae0 msgr2=0x7f46b00a8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 shutdown_connections 2024-09-17T15:51:38.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.574+0000 7f46b60cf640 1 -- 172.21.15.86:0/1504296447 wait complete. 2024-09-17T15:51:38.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:38.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:51:38.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: get_monmap_and_config 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: monmap: 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:38.724084+0000 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:38.724084+0000 2024-09-17T15:51:38.720 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:38.721 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 Processor -- start 2024-09-17T15:51:38.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 -- start start 2024-09-17T15:51:38.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: init 2024-09-17T15:51:38.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c0a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c0a54c0 0x7f6c9c0a58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c0a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c9c137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c9c0a6280 con 0x7f6c9c137ea0 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6c9c09c0a0 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c09c380 con 0x7f6c9c0a54c0 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6ca0fec640 10 monclient(hunting): authenticate will time out at 2431.689555s 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55138/0 (socket says 172.21.15.86:55138) 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1469658484 learned_addr learned my addr 172.21.15.86:0/1469658484 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): get_auth_request con 0x7f6c9c0a50e0 auth_method 0 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c0a54c0 0x7f6c9c0a58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): get_auth_request con 0x7f6c9c0a54c0 auth_method 0 2024-09-17T15:51:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_done global_id 234537 payload 274 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c0a54c0 msgr2=0x7f6c9c0a58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c0a54c0 0x7f6c9c0a58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1469658484 >> v1:172.21.15.86:6789/0 conn(0x7f6c9c137ea0 legacy=0x7f6c9c1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.722+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1469658484 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c09c660 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 msgr2=0x7f6c9c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9a59c640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009870 tx=0x7f6c8401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ae9a0d694e7ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 1 -- 172.21.15.86:0/1469658484 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840b0070 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: dump: 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.730 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 5 monclient: authenticate success, global_id 234537 2024-09-17T15:51:38.731 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 1 -- 172.21.15.86:0/1469658484 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c840a4e50 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 4 set_mon_vals no callback set 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 1 -- 172.21.15.86:0/1469658484 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840ab820 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6c9959a640 10 monclient: dump: 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 10 monclient: shutdown 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 msgr2=0x7f6c9c09cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6c84009870 tx=0x7f6c8401c4b0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1469658484 shutdown_connections 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c0a54c0 0x7f6c9c0a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1469658484 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c09cd60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.733 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1469658484 >> 172.21.15.86:0/1469658484 conn(0x7f6c9c0aaae0 msgr2=0x7f6c9c0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1469658484 shutdown_connections 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1469658484 wait complete. 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.726+0000 7f6ca0fec640 10 monclient: monmap: 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.734 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 Processor -- start 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 -- start start 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient: init 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c1d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-a con 0x7f6c9c0a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c9c0a54c0 0x7f6c9c136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9c0a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137ea0 0x7f6c9c13d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.735 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): picked mon.noname-c con 0x7f6c9c137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c142cb0 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c142b30 con 0x7f6c9c0a54c0 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c142e30 con 0x7f6c9c137ea0 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6ca0fec640 10 monclient(hunting): authenticate will time out at 2431.696622s 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c1d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c1d0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55142/0 (socket says 172.21.15.86:55142) 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1591305948 learned_addr learned my addr 172.21.15.86:0/1591305948 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): get_auth_request con 0x7f6c9c0a50e0 auth_method 0 2024-09-17T15:51:38.736 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137ea0 0x7f6c9c13d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): get_auth_request con 0x7f6c9c137ea0 auth_method 0 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.737 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c9c0a54c0 0x7f6c9c136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient(hunting): handle_auth_done global_id 234540 payload 824 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137ea0 msgr2=0x7f6c9c13d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137ea0 0x7f6c9c13d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c9c0a54c0 msgr2=0x7f6c9c136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c9c0a54c0 0x7f6c9c136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c0a4520 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c99d9b640 10 monclient: get_auth_request con 0x7f6c9c0a54c0 auth_method 0 2024-09-17T15:51:38.738 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c9c0a54c0 0x7f6c9c136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 msgr2=0x7f6c9c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c9a59c640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c1d0020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6c840ae040 tx=0x7f6c84019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840b9070 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: dump: 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.743 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c840b4420 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c840b0040 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c92ffd640 10 monclient: dump: 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.730+0000 7f6c98d99640 4 set_mon_vals no callback set 2024-09-17T15:51:38.745 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 5 monclient: authenticate success, global_id 234540 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 monclient: _renew_subs 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c0a18d0 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 monclient: _renew_subs 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c121310 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 0x7f6c9c13d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c120d30 con 0x7f6c9c137480 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 0x7f6c9c13d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient: get_auth_request con 0x7f6c9c137480 auth_method 0 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.746 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6c840bc070 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_done global_id 274145 payload 824 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c9c09cd60 con 0x7f6c9c137480 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c7c03bbe0 0x7f6c7c03e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f6c840e9090 con 0x7f6c9c0a50e0 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c9ad9d640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c7c03bbe0 0x7f6c7c03e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c9ad9d640 10 monclient: get_auth_request con 0x7f6c7c03bbe0 auth_method 0 2024-09-17T15:51:38.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 msgr2=0x7f6c9c13d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 0x7f6c9c13d5a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00f3b0 tx=0x7f6c8c00f870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c009e50 con 0x7f6c9c137480 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 10 monclient: dump: 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.748 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.749 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.749 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6c8c0039a0 con 0x7f6c9c137480 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 msgr2=0x7f6c9c13d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 0x7f6c9c13d5a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00f3b0 tx=0x7f6c8c00f870 comp rx=0 tx=0).stop 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 msgr2=0x7f6c9c13d5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.734+0000 7f6c92ffd640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c137480 0x7f6c9c13d5a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.738+0000 7f6c9ad9d640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c7c03bbe0 0x7f6c7c03e0a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c1d10e0 tx=0x7f6c94007dc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6ca0fec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 0x7f6c9c0a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6ca0fec640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6c9c13def0 con 0x7f6c9c13dae0 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 0x7f6c9c0a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient: get_auth_request con 0x7f6c9c13dae0 auth_method 0 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 10 monclient(hunting): handle_auth_done global_id 274148 payload 824 2024-09-17T15:51:38.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.742+0000 7f6c99d9b640 1 -- 172.21.15.86:0/1591305948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6c9c074be0 con 0x7f6c9c13dae0 2024-09-17T15:51:38.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c99d9b640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 msgr2=0x7f6c9c0a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c99d9b640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 0x7f6c9c0a05f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00f3b0 tx=0x7f6c8c004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c01fb80 con 0x7f6c9c13dae0 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 10 monclient: dump: 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.754 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f6c8c01fb80 con 0x7f6c9c13dae0 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 msgr2=0x7f6c9c0a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 0x7f6c9c0a05f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00f3b0 tx=0x7f6c8c004a50 comp rx=0 tx=0).stop 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 msgr2=0x7f6c9c0a05f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c92ffd640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9c13dae0 0x7f6c9c0a05f0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00f3b0 tx=0x7f6c8c004a50 comp rx=0 tx=0).stop 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:38.755 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 18067, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:38.756 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:38.757 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.758 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:38.759 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.760 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:38.761 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.762 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:38.763 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:38.764 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.765 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.766 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:38.767 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c7c03bbe0 msgr2=0x7f6c7c03e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6c7c03bbe0 0x7f6c7c03e0a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c1d10e0 tx=0x7f6c94007dc0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 10 monclient: shutdown 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 msgr2=0x7f6c9c1d0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 --2- 172.21.15.86:0/1591305948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9c0a50e0 0x7f6c9c1d0020 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6c840ae040 tx=0x7f6c84019250 comp rx=0 tx=0).stop 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c9ad9d640 1 -- 172.21.15.86:0/1591305948 reap_dead start 2024-09-17T15:51:38.769 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6c9ad9d640 1 -- 172.21.15.86:0/1591305948 reap_dead start 2024-09-17T15:51:38.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 shutdown_connections 2024-09-17T15:51:38.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 >> 172.21.15.86:0/1591305948 conn(0x7f6c9c0aaae0 msgr2=0x7f6c9c0a2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 shutdown_connections 2024-09-17T15:51:38.770 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.746+0000 7f6ca0fec640 1 -- 172.21.15.86:0/1591305948 wait complete. 2024-09-17T15:51:38.770 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:38.770 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:51:38.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: get_monmap_and_config 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: monmap: 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:38.892123+0000 2024-09-17T15:51:38.888 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:38.892123+0000 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:38.889 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 Processor -- start 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 -- start start 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: init 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51fc0a50e0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 0x7f51fc09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-c con 0x7f51fc0a5bd0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f51fc13d2f0 0x7f51fc13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-a con 0x7f51fc13d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:38.891 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51fc0a6280 con 0x7f51fc0a50e0 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f51fc09d2a0 con 0x7f51fc13d2f0 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f51fc09c0a0 con 0x7f51fc0a5bd0 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f5200c32640 10 monclient(hunting): authenticate will time out at 2431.857366s 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f51fc13d2f0 0x7f51fc13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fa59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f51fc0a50e0 0x7f51fc0a54c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38008/0 (socket says 172.21.15.86:38008) 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f51fc13d2f0 0x7f51fc13f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55154/0 (socket says 172.21.15.86:55154) 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fa59c640 1 -- 172.21.15.86:0/1543011155 learned_addr learned my addr 172.21.15.86:0/1543011155 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): get_auth_request con 0x7f51fc13d2f0 auth_method 0 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51f9d9b640 1 --2- 172.21.15.86:0/1543011155 >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 0x7f51fc09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient(hunting): handle_auth_done global_id 234543 payload 274 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 -- 172.21.15.86:0/1543011155 >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 msgr2=0x7f51fc09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 --2- 172.21.15.86:0/1543011155 >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 0x7f51fc09cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 -- 172.21.15.86:0/1543011155 >> v1:172.21.15.86:6789/0 conn(0x7f51fc0a50e0 legacy=0x7f51fc0a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51fad9d640 1 -- 172.21.15.86:0/1543011155 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc09c380 con 0x7f51fc13d2f0 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51f9d9b640 10 monclient: get_auth_request con 0x7f51fc0a5bd0 auth_method 0 2024-09-17T15:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.890+0000 7f51f9d9b640 1 --2- 172.21.15.86:0/1543011155 >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 0x7f51fc09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51fad9d640 1 -- 172.21.15.86:0/1543011155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 msgr2=0x7f51fc13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51fad9d640 1 --2- 172.21.15.86:0/1543011155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc13f6e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f51fc06cc20 tx=0x7f51ec01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85543463212821ec server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 1 -- 172.21.15.86:0/1543011155 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b0070 con 0x7f51fc13d2f0 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: dump: 2024-09-17T15:51:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 1 -- 172.21.15.86:0/1543011155 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec0ab600 con 0x7f51fc13d2f0 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 5 monclient: authenticate success, global_id 234543 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 1 -- 172.21.15.86:0/1543011155 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0ab9d0 con 0x7f51fc13d2f0 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f51f959a640 10 monclient: dump: 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 4 set_mon_vals no callback set 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 10 monclient: shutdown 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 -- 172.21.15.86:0/1543011155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 msgr2=0x7f51fc13f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 --2- 172.21.15.86:0/1543011155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc13f6e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f51fc06cc20 tx=0x7f51ec01c310 comp rx=0 tx=0).stop 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 -- 172.21.15.86:0/1543011155 shutdown_connections 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 --2- 172.21.15.86:0/1543011155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc13f6e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 --2- 172.21.15.86:0/1543011155 >> v2:172.21.15.203:3300/0 conn(0x7f51fc0a5bd0 0x7f51fc09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 -- 172.21.15.86:0/1543011155 >> 172.21.15.86:0/1543011155 conn(0x7f51fc0aaae0 msgr2=0x7f51fc0aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 -- 172.21.15.86:0/1543011155 shutdown_connections 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 -- 172.21.15.86:0/1543011155 wait complete. 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 10 monclient: build_initial_monmap 2024-09-17T15:51:38.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 10 monclient: monmap: 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:38.899 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.894+0000 7f5200c32640 1 Processor -- start 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- start start 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: init 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0a50e0 0x7f51fc121300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-c con 0x7f51fc0a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc0a5bd0 0x7f51fc138170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-b con 0x7f51fc0a5bd0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc1cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): picked mon.noname-a con 0x7f51fc13d2f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.904 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc142650 con 0x7f51fc13d2f0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc0a6ad0 con 0x7f51fc0a5bd0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f51fc1427d0 con 0x7f51fc0a50e0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient(hunting): authenticate will time out at 2431.864514s 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc1cfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc1cfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55160/0 (socket says 172.21.15.86:55160) 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 learned_addr learned my addr 172.21.15.86:0/2748727468 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): get_auth_request con 0x7f51fc13d2f0 auth_method 0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0a50e0 0x7f51fc121300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.905 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): get_auth_request con 0x7f51fc0a50e0 auth_method 0 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f9d9b640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc0a5bd0 0x7f51fc138170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f9d9b640 10 monclient(hunting): get_auth_request con 0x7f51fc0a5bd0 auth_method 0 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.906 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient(hunting): handle_auth_done global_id 234546 payload 824 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0a50e0 msgr2=0x7f51fc121300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0a50e0 0x7f51fc121300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc0a5bd0 msgr2=0x7f51fc138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc0a5bd0 0x7f51fc138170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 monclient: found mon.noname-a 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc0a1180 con 0x7f51fc13d2f0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0a50e0 0x7f51fc121300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 msgr2=0x7f51fc1cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51fad9d640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc1cfa70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01cbb0 tx=0x7f51ec0190b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0ba020 con 0x7f51fc13d2f0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.907 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: dump: 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:38.908 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ec0b5870 con 0x7f51fc13d2f0 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b0040 con 0x7f51fc13d2f0 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 5 monclient: authenticate success, global_id 234546 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _renew_subs 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51fc122260 con 0x7f51fc13d2f0 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f8d99640 4 set_mon_vals no callback set 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f51f2ffd640 10 monclient: dump: 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.909 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _renew_subs 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.898+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51fc0a0af0 con 0x7f51fc13d2f0 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f5200c32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f5200c32640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 0x7f51fc1cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc06a0e0 con 0x7f51fc1386b0 2024-09-17T15:51:38.910 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f51ec0bd020 con 0x7f51fc13d2f0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 0x7f51fc1cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51f2ffd640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51e003bbb0 0x7f51e003e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient: get_auth_request con 0x7f51fc1386b0 auth_method 0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f51ec0ea560 con 0x7f51fc13d2f0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51f9d9b640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51e003bbb0 0x7f51e003e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51f9d9b640 10 monclient: get_auth_request con 0x7f51e003bbb0 auth_method 0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 10 monclient(hunting): handle_auth_done global_id 254281 payload 824 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51fc09cd60 con 0x7f51fc1386b0 2024-09-17T15:51:38.911 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 msgr2=0x7f51fc1cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.902+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 0x7f51fc1cffb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d380 tx=0x7f51f400d840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f9d9b640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51e003bbb0 0x7f51e003e070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51fc13d960 tx=0x7f51e4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4019070 con 0x7f51fc1386b0 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 10 monclient: dump: 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.913 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.913 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51f4007530 con 0x7f51fc1386b0 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 msgr2=0x7f51fc1cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 0x7f51fc1cffb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d380 tx=0x7f51f400d840 comp rx=0 tx=0).stop 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 msgr2=0x7f51fc1cffb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.906+0000 7f51f2ffd640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc1386b0 0x7f51fc1cffb0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f51f400d380 tx=0x7f51f400d840 comp rx=0 tx=0).stop 2024-09-17T15:51:38.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f5200c32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f5200c32640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 0x7f51fc1d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f5200c32640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f51fc138e70 con 0x7f51fc09fc20 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 0x7f51fc1d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient: get_auth_request con 0x7f51fc09fc20 auth_method 0 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 10 monclient(hunting): handle_auth_done global_id 254284 payload 824 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.910+0000 7f51fa59c640 1 -- 172.21.15.86:0/2748727468 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51fc074be0 con 0x7f51fc09fc20 2024-09-17T15:51:38.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51fa59c640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 msgr2=0x7f51fc1d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:38.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51fa59c640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 0x7f51fc1d04f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51f400f240 tx=0x7f51f4004cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:38.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4019070 con 0x7f51fc09fc20 2024-09-17T15:51:38.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:38.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 10 monclient: dump: 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f51f401d8c0 con 0x7f51fc09fc20 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:38.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 msgr2=0x7f51fc1d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 0x7f51fc1d04f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51f400f240 tx=0x7f51f4004cd0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 msgr2=0x7f51fc1d04f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.914+0000 7f51f2ffd640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51fc09fc20 0x7f51fc1d04f0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f51f400f240 tx=0x7f51f4004cd0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 218, 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 114859, 2024-09-17T15:51:38.918 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 3, 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.920 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:38.921 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:38.922 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.924 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:38.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:38.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.929 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:38.930 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51e003bbb0 msgr2=0x7f51e003e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f51e003bbb0 0x7f51e003e070 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51fc13d960 tx=0x7f51e4008040 comp rx=0 tx=0).stop 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 10 monclient: shutdown 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 msgr2=0x7f51fc1cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 --2- 172.21.15.86:0/2748727468 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc13d2f0 0x7f51fc1cfa70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01cbb0 tx=0x7f51ec0190b0 comp rx=0 tx=0).stop 2024-09-17T15:51:38.931 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 reap_dead start 2024-09-17T15:51:38.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f51fad9d640 1 -- 172.21.15.86:0/2748727468 reap_dead start 2024-09-17T15:51:38.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 shutdown_connections 2024-09-17T15:51:38.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 >> 172.21.15.86:0/2748727468 conn(0x7f51fc0aaae0 msgr2=0x7f51fc0a3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:38.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 shutdown_connections 2024-09-17T15:51:38.932 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:38.918+0000 7f5200c32640 1 -- 172.21.15.86:0/2748727468 wait complete. 2024-09-17T15:51:38.933 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:51:38.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-17T15:51:38.933 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-17T15:51:38.933 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-17T15:51:38.934 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-17T15:51:38.934 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:51:38.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:51:38.962 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:51:38.962 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-17T15:51:38.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 10 monclient: get_monmap_and_config 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 10 monclient: build_initial_monmap 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 10 monclient: monmap: 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:39.070 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:39.074195+0000 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:39.074195+0000 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:39.071 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 1 Processor -- start 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 1 -- start start 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.070+0000 7fd3f6922640 10 monclient: init 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f009c6e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3f009bf30 0x7fd3f0134930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f009bf30 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 0x7fd3f0135300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f0134e70 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd3f009de70 con 0x7fd3f009bf30 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd3f00a3160 con 0x7fd3f0134e70 2024-09-17T15:51:39.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f009cac0 con 0x7fd3f009c6e0 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f6922640 10 monclient(hunting): authenticate will time out at 2432.040381s 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3f009bf30 0x7fd3f0134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f4ebf640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 msgr2=0x7fd3f0135300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56688/0 (socket says 172.21.15.86:56688) 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd3f009bf30 0x7fd3f0134930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55172/0 (socket says 172.21.15.86:55172) 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 1 -- 172.21.15.86:0/4235011820 learned_addr learned my addr 172.21.15.86:0/4235011820 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): get_auth_request con 0x7fd3f009c6e0 auth_method 0 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:39.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3f4ebf640 1 --2- 172.21.15.86:0/4235011820 >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 0x7fd3f0135300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): get_auth_request con 0x7fd3f009bf30 auth_method 0 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient(hunting): handle_auth_done global_id 234549 payload 274 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 -- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 msgr2=0x7fd3f00a0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:39.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 -- 172.21.15.86:0/4235011820 >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 msgr2=0x7fd3f0135300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 --2- 172.21.15.86:0/4235011820 >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 0x7fd3f0135300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 monclient: found mon.noname-a 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3ef7fe640 1 -- 172.21.15.86:0/4235011820 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f00a3390 con 0x7fd3f009bf30 2024-09-17T15:51:39.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.074+0000 7fd3effff640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3ef7fe640 1 -- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f009bf30 msgr2=0x7fd3f0134930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3ef7fe640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f009bf30 0x7fd3f0134930 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd3f009c380 tx=0x7fd3e4020670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0b25c73b906bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 1 -- 172.21.15.86:0/4235011820 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e40b0020 con 0x7fd3f009bf30 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: dump: 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 1 -- 172.21.15.86:0/4235011820 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e400b450 con 0x7fd3f009bf30 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 5 monclient: authenticate success, global_id 234549 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 1 -- 172.21.15.86:0/4235011820 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e400b820 con 0x7fd3f009bf30 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 4 set_mon_vals no callback set 2024-09-17T15:51:39.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3eeffd640 10 monclient: dump: 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: shutdown 2024-09-17T15:51:39.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f009bf30 msgr2=0x7fd3f0134930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f009bf30 0x7fd3f0134930 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd3f009c380 tx=0x7fd3e4020670 comp rx=0 tx=0).stop 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- 172.21.15.86:0/4235011820 shutdown_connections 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- 172.21.15.86:0/4235011820 >> v2:172.21.15.203:3300/0 conn(0x7fd3f0134e70 0x7fd3f0135300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f009bf30 0x7fd3f0134930 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- 172.21.15.86:0/4235011820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009c6e0 0x7fd3f00a0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- 172.21.15.86:0/4235011820 >> 172.21.15.86:0/4235011820 conn(0x7fd3f00a7110 msgr2=0x7fd3f00a7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- 172.21.15.86:0/4235011820 shutdown_connections 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- 172.21.15.86:0/4235011820 wait complete. 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: build_initial_monmap 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: monmap: 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:39.084 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 Processor -- start 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- start start 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: init 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f009bf30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:39.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 0x7fd3f0133580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f009c6e0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f0134e70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f00a1190 con 0x7fd3f0134e70 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd3f00a1010 con 0x7fd3f009c6e0 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd3f00a2360 con 0x7fd3f009bf30 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f6922640 10 monclient(hunting): authenticate will time out at 2432.047644s 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:39.086 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3ef7fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 msgr2=0x7fd3f0133580 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3ef7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 0x7fd3f0133580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55186/0 (socket says 172.21.15.86:55186) 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56698/0 (socket says 172.21.15.86:56698) 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 1 -- 172.21.15.86:0/3567376016 learned_addr learned my addr 172.21.15.86:0/3567376016 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): get_auth_request con 0x7fd3f0134e70 auth_method 0 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): get_auth_request con 0x7fd3f009bf30 auth_method 0 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:39.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.078+0000 7fd3f4ebf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 10 monclient(hunting): handle_auth_done global_id 274157 payload 824 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 -- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 msgr2=0x7fd3f0133580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 0x7fd3f0133580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 -- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 msgr2=0x7fd3f0139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 monclient: found mon.noname-c 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:39.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 -- 172.21.15.86:0/3567376016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f011ec30 con 0x7fd3f009bf30 2024-09-17T15:51:39.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f4ebf640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:39.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 -- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 msgr2=0x7fd3f0133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:39.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3effff640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000fb80 tx=0x7fd3e000d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0004230 con 0x7fd3f009bf30 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: dump: 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:39.091 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: _finish_auth 0 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: _check_auth_tickets 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 5 monclient: authenticate success, global_id 274157 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3e00043d0 con 0x7fd3f009bf30 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _renew_subs 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0010870 con 0x7fd3f009bf30 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3f011ef10 con 0x7fd3f009bf30 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:39.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 10 monclient: dump: 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ee7fc640 4 set_mon_vals no callback set 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _renew_subs 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:39.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f006f100 con 0x7fd3f009bf30 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3f00a2770 con 0x7fd3f009bf30 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd3e001f020 con 0x7fd3f009bf30 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 0x7fd3dc03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.082+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7fd3e00503a0 con 0x7fd3f009bf30 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.086+0000 7fd3ef7fe640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 0x7fd3dc03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.086+0000 7fd3ef7fe640 10 monclient: get_auth_request con 0x7fd3dc03bb10 auth_method 0 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.086+0000 7fd3ef7fe640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 0x7fd3dc03dfd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00691f0 tx=0x7fd3e40ab000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.090+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd3e001fd80 con 0x7fd3f009bf30 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.090+0000 7fd3ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:39.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.090+0000 7fd3ecff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:39.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3f6922640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:39.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3f6922640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3f6922640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd3f006deb0 con 0x7fd3f009bf30 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3ecff9640 1 -- 172.21.15.86:0/3567376016 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd3e00500d0 con 0x7fd3f009bf30 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.318+0000 7fd3ecff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:39.319 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:39.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 msgr2=0x7fd3dc03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 0x7fd3dc03dfd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00691f0 tx=0x7fd3e40ab000 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 10 monclient: shutdown 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 msgr2=0x7fd3f0133040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000fb80 tx=0x7fd3e000d720 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 shutdown_connections 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd3dc03bb10 0x7fd3dc03dfd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd3f0134e70 0x7fd3f0139780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3f009c6e0 0x7fd3f0133580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 --2- 172.21.15.86:0/3567376016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3f009bf30 0x7fd3f0133040 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 >> 172.21.15.86:0/3567376016 conn(0x7fd3f00a7110 msgr2=0x7fd3f00a59a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 shutdown_connections 2024-09-17T15:51:39.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:39.322+0000 7fd3f6922640 1 -- 172.21.15.86:0/3567376016 wait complete. 2024-09-17T15:51:42.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:42.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: get_monmap_and_config 2024-09-17T15:51:42.469 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: build_initial_monmap 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: monmap: 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:42.473427+0000 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:42.473427+0000 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:42.470 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:42.471 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:42.471 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 Processor -- start 2024-09-17T15:51:42.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 -- start start 2024-09-17T15:51:42.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: init 2024-09-17T15:51:42.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:42.473 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-b con 0x7f6be80689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 0x7f6be80691d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-a con 0x7f6be8068db0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 0x7f6be8135f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-c con 0x7f6be8135ae0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f6be8074500 con 0x7f6be8068db0 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f6be809c8b0 con 0x7f6be8135ae0 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.474 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6be809cb90 con 0x7f6be80689d0 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6becc7d640 10 monclient(hunting): authenticate will time out at 2435.438957s 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be6d9d640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 msgr2=0x7f6be8135f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be6d9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 0x7f6be8135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 0x7f6be80691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56714/0 (socket says 172.21.15.86:56714) 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 0x7f6be80691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55188/0 (socket says 172.21.15.86:55188) 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 -- 172.21.15.86:0/4231300773 learned_addr learned my addr 172.21.15.86:0/4231300773 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): get_auth_request con 0x7f6be80689d0 auth_method 0 2024-09-17T15:51:42.475 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): get_auth_request con 0x7f6be8068db0 auth_method 0 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:42.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient(hunting): handle_auth_done global_id 274160 payload 274 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 -- 172.21.15.86:0/4231300773 >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 msgr2=0x7f6be8135f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 --2- 172.21.15.86:0/4231300773 >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 0x7f6be8135f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 -- 172.21.15.86:0/4231300773 >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 msgr2=0x7f6be80691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 --2- 172.21.15.86:0/4231300773 >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 0x7f6be80691d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient: _finish_hunting 0 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 monclient: found mon.noname-b 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:42.477 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.470+0000 7f6be659c640 1 -- 172.21.15.86:0/4231300773 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be809ce70 con 0x7f6be80689d0 2024-09-17T15:51:42.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.474+0000 7f6be659c640 1 -- 172.21.15.86:0/4231300773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 msgr2=0x7f6be806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.474+0000 7f6be659c640 1 --2- 172.21.15.86:0/4231300773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6be809cac0 tx=0x7f6bd801cda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c346fa55c67807 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 1 -- 172.21.15.86:0/4231300773 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80b0070 con 0x7f6be80689d0 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: dump: 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.483 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: _finish_auth 0 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: _check_auth_tickets 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 1 -- 172.21.15.86:0/4231300773 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd80ab600 con 0x7f6be80689d0 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 5 monclient: authenticate success, global_id 274160 2024-09-17T15:51:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 4 set_mon_vals no callback set 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 1 -- 172.21.15.86:0/4231300773 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd80ab9d0 con 0x7f6be80689d0 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be559a640 10 monclient: dump: 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: shutdown 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- 172.21.15.86:0/4231300773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 msgr2=0x7f6be806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:42.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- 172.21.15.86:0/4231300773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6be809cac0 tx=0x7f6bd801cda0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- 172.21.15.86:0/4231300773 shutdown_connections 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- 172.21.15.86:0/4231300773 >> v2:172.21.15.203:3300/0 conn(0x7f6be8135ae0 0x7f6be8135f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- 172.21.15.86:0/4231300773 >> v2:172.21.15.86:3300/0 conn(0x7f6be8068db0 0x7f6be80691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- 172.21.15.86:0/4231300773 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be80689d0 0x7f6be806f840 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- 172.21.15.86:0/4231300773 >> 172.21.15.86:0/4231300773 conn(0x7f6be80a6440 msgr2=0x7f6be80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- 172.21.15.86:0/4231300773 shutdown_connections 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- 172.21.15.86:0/4231300773 wait complete. 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: build_initial_monmap 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: monmap: 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:42.487 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 Processor -- start 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- start start 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: init 2024-09-17T15:51:42.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 0x7f6be81cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-b con 0x7f6be80689d0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-c con 0x7f6be8068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 0x7f6be813a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): picked mon.noname-a con 0x7f6be8135ae0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f6be809c2d0 con 0x7f6be8135ae0 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6be809be70 con 0x7f6be80689d0 2024-09-17T15:51:42.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f6be813f820 con 0x7f6be8068db0 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6becc7d640 10 monclient(hunting): authenticate will time out at 2435.446321s 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 0x7f6be813a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:56718/0 (socket says 172.21.15.86:56718) 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 1 -- 172.21.15.86:0/2076424163 learned_addr learned my addr 172.21.15.86:0/2076424163 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): get_auth_request con 0x7f6be8068db0 auth_method 0 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:42.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): get_auth_request con 0x7f6be8135ae0 auth_method 0 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be659c640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 msgr2=0x7f6be81cde30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be659c640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 0x7f6be81cde30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:42.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient(hunting): handle_auth_done global_id 234558 payload 824 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 msgr2=0x7f6be8132f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 msgr2=0x7f6be81cde30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 0x7f6be81cde30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 monclient: found mon.noname-a 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be6d9d640 1 -- 172.21.15.86:0/2076424163 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6be81ced90 con 0x7f6be8135ae0 2024-09-17T15:51:42.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.478+0000 7f6be5d9b640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6be6d9d640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 msgr2=0x7f6be813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6be6d9d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 0x7f6be813a440 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6be000bd90 tx=0x7f6be0009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be00066f0 con 0x7f6be8135ae0 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: dump: 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.493 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be0006890 con 0x7f6be8135ae0 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:42.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0015440 con 0x7f6be8135ae0 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 10 monclient: dump: 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:42.495 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 5 monclient: authenticate success, global_id 234558 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _renew_subs 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6be809d460 con 0x7f6be8135ae0 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6be4d99640 4 set_mon_vals no callback set 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _renew_subs 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6be811e610 con 0x7f6be8135ae0 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:42.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6be80691d0 con 0x7f6be8135ae0 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f6be001f3f0 con 0x7f6be8135ae0 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 0x7f6bc803dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f6be0050b10 con 0x7f6be8135ae0 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6be659c640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 0x7f6bc803dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:42.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.482+0000 7f6be659c640 10 monclient: get_auth_request con 0x7f6bc803bb10 auth_method 0 2024-09-17T15:51:42.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.486+0000 7f6be659c640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 0x7f6bc803dfd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6be8069ad0 tx=0x7f6bd8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:42.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.486+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f6be809d460 con 0x7f6be8135ae0 2024-09-17T15:51:42.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.486+0000 7f6bd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:42.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.486+0000 7f6bd6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:42.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6becc7d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:42.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6becc7d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:42.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6becc7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:42.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6be806d4f0 con 0x7f6be8135ae0 2024-09-17T15:51:42.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6bd6ffd640 1 -- 172.21.15.86:0/2076424163 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6be001e3a0 con 0x7f6be8135ae0 2024-09-17T15:51:42.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6bd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:42.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.702+0000 7f6bd6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:42.705 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":218,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 msgr2=0x7f6bc803dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 0x7f6bc803dfd0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f6be8069ad0 tx=0x7f6bd8008040 comp rx=0 tx=0).stop 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 10 monclient: shutdown 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 msgr2=0x7f6be813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 0x7f6be813a440 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6be000bd90 tx=0x7f6be0009eb0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 shutdown_connections 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f6bc803bb10 0x7f6bc803dfd0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6be8135ae0 0x7f6be813a440 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6be8068db0 0x7f6be8132f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 --2- 172.21.15.86:0/2076424163 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6be80689d0 0x7f6be81cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:42.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 >> 172.21.15.86:0/2076424163 conn(0x7f6be80a6440 msgr2=0x7f6be80a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:42.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 shutdown_connections 2024-09-17T15:51:42.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:42.706+0000 7f6becc7d640 1 -- 172.21.15.86:0/2076424163 wait complete. 2024-09-17T15:51:45.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:51:45.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: get_monmap_and_config 2024-09-17T15:51:45.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: build_initial_monmap 2024-09-17T15:51:45.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: monmap: 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:45.857244+0000 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:45.857244+0000 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:45.854 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:45.855 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 Processor -- start 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 -- start start 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: init 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b500689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:45.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-c con 0x7f9b50068db0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b50139f00 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b5009d770 con 0x7f9b500689d0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b5009ed00 con 0x7f9b50139f00 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9b5009efe0 con 0x7f9b50068db0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b57246640 10 monclient(hunting): authenticate will time out at 2438.822913s 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b54fe2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9b500689d0 0x7f9b5009ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38012/0 (socket says 172.21.15.86:38012) 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b4ffff640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 msgr2=0x7f9b500691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 learned_addr learned my addr 172.21.15.86:0/2427910698 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.203:6789/0 conn(0x7f9b50139f00 legacy=0x7f9b5013c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:51:45.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.854+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:45.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:45.858+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.86:6789/0 conn(0x7f9b500689d0 legacy=0x7f9b5009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:46.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.058+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 msgr2=0x7f9b500691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:46.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.058+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.203:6789/0 conn(0x7f9b50139f00 legacy=0x7f9b5013c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:51:46.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.058+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:51:46.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.462+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 msgr2=0x7f9b500691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:46.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.462+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.203:6789/0 conn(0x7f9b50139f00 legacy=0x7f9b5013c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:51:46.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:46.462+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:51:47.265 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:47.266+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 msgr2=0x7f9b500691c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:47.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:47.266+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.203:6789/0 conn(0x7f9b50139f00 legacy=0x7f9b5013c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:51:47.266 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:47.266+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): tick 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 monclient(hunting): continuing hunt 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 msgr2=0x7f9b500691c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b50068db0 0x7f9b500691c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.203:6789/0 conn(0x7f9b50139f00 legacy=0x7f9b5013c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.86:6789/0 conn(0x7f9b500689d0 legacy=0x7f9b5009ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient: _add_conns ranks=[2,3] 2024-09-17T15:51:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9b30003750 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f9b30006280 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9b5009efe0 con 0x7f9b30006280 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9b30008e00 con 0x7f9b30003750 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:48.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request con 0x7f9b30006280 auth_method 0 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.854+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:48.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:48.858+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:49.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:49.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.058+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request con 0x7f9b30006280 auth_method 0 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:49.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:49.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:49.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:49.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.062+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:51:49.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:49.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:49.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:49.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:49.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:49.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:49.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:49.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:49.462 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.462+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:51:49.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:49.464 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request con 0x7f9b30006280 auth_method 0 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:49.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:49.466+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:50.263 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:50.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:50.264 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.266+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:51:50.268 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request con 0x7f9b30006280 auth_method 0 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:50.269 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:50.270+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:51:51.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): tick 2024-09-17T15:51:51.856 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 monclient(hunting): continuing hunt 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 msgr2=0x7f9b30008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.86:3300/0 conn(0x7f9b30006280 0x7f9b30008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 reap_dead start 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f9b30003750 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 0x7f9b3000b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f9b300066b0 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:51.857 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b3000b990 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b50069700 con 0x7f9b3000b990 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f9b5009efe0 con 0x7f9b300066b0 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9b30008e00 con 0x7f9b30003750 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4e7fc640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 msgr2=0x7f9b3000b450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.854+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 0x7f9b3000b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:51.858 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:51.859 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:51.858+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.86:6789/0 conn(0x7f9b3000b990 legacy=0x7f9b3000dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.058+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 msgr2=0x7f9b3000b450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.058+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 0x7f9b3000b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:51:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.062+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:51:52.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.462+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 msgr2=0x7f9b3000b450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:52.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.462+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 0x7f9b3000b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:51:52.461 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.462+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 DispatchQueue::entry inject delay of 0.002000 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:51:52.465 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:52.466+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:51:53.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.058+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035577154 0 0) 0x7f9b50069700 con 0x7f9b3000b990 2024-09-17T15:51:53.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.058+0000 7f9b4f7fe640 10 monclient(hunting): _negotiate 2024-09-17T15:51:53.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.058+0000 7f9b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.058+0000 7f9b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.058+0000 7f9b4f7fe640 10 monclient(hunting): my global_id is 234561 2024-09-17T15:51:53.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.062+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b3c003530 con 0x7f9b3000b990 2024-09-17T15:51:53.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 123325011 0 0) 0x7f9b3c003530 con 0x7f9b3000b990 2024-09-17T15:51:53.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b30005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:53.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b30005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 msgr2=0x7f9b3000b450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 --2- 172.21.15.86:0/2427910698 >> v2:172.21.15.203:3300/0 conn(0x7f9b300066b0 0x7f9b3000b450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.86:6789/0 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5009ed00 con 0x7f9b3000b990 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3159590149 0 0) 0x7f9b48004440 con 0x7f9b3000b990 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-17T15:51:53.070 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: dump: 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] but i'm connected to v1:172.21.15.86:6789/0, reconnecting 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 >> v1:172.21.15.86:6789/0 conn(0x7f9b3000b990 legacy=0x7f9b3000dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 0x7f9b3c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): picked mon.a con 0x7f9b3c003a30 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b3c004810 0x7f9b3c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): picked mon.b con 0x7f9b3c004810 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b3c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): picked mon.c con 0x7f9b30003750 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5009efe0 con 0x7f9b3c003a30 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b30008e00 con 0x7f9b3c004810 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c0055b0 con 0x7f9b30003750 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.072 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.066+0000 7f9b4f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 0x7f9b3c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request con 0x7f9b3c003a30 auth_method 0 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b4ffff640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b3c004810 msgr2=0x7f9b3c004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b3c004810 0x7f9b3c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b3c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:51:53.073 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): get_auth_request con 0x7f9b30003750 auth_method 0 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient(hunting): handle_auth_done global_id 234561 payload 293 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 msgr2=0x7f9b3c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b30003750 0x7f9b3c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b3c004810 msgr2=0x7f9b3c004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b3c004810 0x7f9b3c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 monclient: found mon.a 2024-09-17T15:51:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c003530 con 0x7f9b3c003a30 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b54fe2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b57246640 5 monclient: authenticate success, global_id 234561 2024-09-17T15:51:53.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.070+0000 7f9b57246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:53.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b54fe2640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 msgr2=0x7f9b3c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 0x7f9b3c003e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9b40005f40 tx=0x7f9b4001dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ece365e4d85b9b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b400ac070 con 0x7f9b3c003a30 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: dump: 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.078 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b400a6c30 con 0x7f9b3c003a30 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 1 -- 172.21.15.86:0/2427910698 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b40005b90 con 0x7f9b3c003a30 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b4f7fe640 10 monclient: dump: 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.079 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.080 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.080 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:53.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 4 set_mon_vals no callback set 2024-09-17T15:51:53.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 10 monclient: shutdown 2024-09-17T15:51:53.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 -- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 msgr2=0x7f9b3c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 --2- 172.21.15.86:0/2427910698 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b3c003a30 0x7f9b3c003e50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9b40005f40 tx=0x7f9b4001dfa0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b557e3640 1 -- 172.21.15.86:0/2427910698 reap_dead start 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 -- 172.21.15.86:0/2427910698 shutdown_connections 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 -- 172.21.15.86:0/2427910698 >> 172.21.15.86:0/2427910698 conn(0x7f9b500a6440 msgr2=0x7f9b500a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 -- 172.21.15.86:0/2427910698 shutdown_connections 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.078+0000 7f9b57246640 1 -- 172.21.15.86:0/2427910698 wait complete. 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient: monmap: 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:53.082 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 Processor -- start 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 -- start start 2024-09-17T15:51:53.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient: init 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 0x7f9b50133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-b con 0x7f9b501339b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-a con 0x7f9b5011fb70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:53.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 0x7f9b5011dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): picked mon.noname-c con 0x7f9b5011da90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9b501355e0 con 0x7f9b5011fb70 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b50135460 con 0x7f9b501339b0 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9b50135760 con 0x7f9b5011da90 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b57246640 10 monclient(hunting): authenticate will time out at 2446.050416s 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b54fe2640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 msgr2=0x7f9b50133d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b54fe2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 0x7f9b50133d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:53.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34032/0 (socket says 172.21.15.86:34032) 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 -- 172.21.15.86:0/3531716184 learned_addr learned my addr 172.21.15.86:0/3531716184 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request con 0x7f9b5011fb70 auth_method 0 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 0x7f9b5011dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): get_auth_request con 0x7f9b5011da90 auth_method 0 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient(hunting): handle_auth_done global_id 234564 payload 824 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 -- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 msgr2=0x7f9b5011dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 0x7f9b5011dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 -- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 msgr2=0x7f9b50133d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 0x7f9b50133d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 monclient: found mon.noname-a 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b4ffff640 1 -- 172.21.15.86:0/3531716184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b50120ad0 con 0x7f9b5011fb70 2024-09-17T15:51:53.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.082+0000 7f9b557e3640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 0x7f9b5011dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4ffff640 1 -- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 msgr2=0x7f9b5011d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4ffff640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bb70 tx=0x7f9b38005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3801b020 con 0x7f9b5011fb70 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: dump: 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 5 monclient: authenticate success, global_id 234564 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _renew_subs 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b38017600 con 0x7f9b5011fb70 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5009d460 con 0x7f9b5011fb70 2024-09-17T15:51:53.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38006e60 con 0x7f9b5011fb70 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 10 monclient: dump: 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4effd640 4 set_mon_vals no callback set 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _renew_subs 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b50121370 con 0x7f9b5011fb70 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9b3801e020 con 0x7f9b5011fb70 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b50134330 con 0x7f9b5011fb70 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 0x7f9b30041310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f9b38050d80 con 0x7f9b5011fb70 2024-09-17T15:51:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 0x7f9b30041310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.086+0000 7f9b54fe2640 10 monclient: get_auth_request con 0x7f9b3003ee50 auth_method 0 2024-09-17T15:51:53.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.090+0000 7f9b54fe2640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 0x7f9b30041310 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005500 tx=0x7f9b400014b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.094+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f9b38025020 con 0x7f9b5011fb70 2024-09-17T15:51:53.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.094+0000 7f9b4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.094+0000 7f9b4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:53.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9b50120f20 con 0x7f9b5011fb70 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b4d7fa640 1 -- 172.21.15.86:0/3531716184 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9b50120f20 con 0x7f9b5011fb70 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:51:53.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:53.334 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":222,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:51:53.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 msgr2=0x7f9b30041310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 0x7f9b30041310 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c005500 tx=0x7f9b400014b0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 10 monclient: shutdown 2024-09-17T15:51:53.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 msgr2=0x7f9b5011d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.334+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800bb70 tx=0x7f9b38005480 comp rx=0 tx=0).stop 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 shutdown_connections 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9b3003ee50 0x7f9b30041310 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9b5011da90 0x7f9b5011dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9b5011fb70 0x7f9b5011d550 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 --2- 172.21.15.86:0/3531716184 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b501339b0 0x7f9b50133d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 >> 172.21.15.86:0/3531716184 conn(0x7f9b500a6440 msgr2=0x7f9b500a0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 shutdown_connections 2024-09-17T15:51:53.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.338+0000 7f9b57246640 1 -- 172.21.15.86:0/3531716184 wait complete. 2024-09-17T15:51:53.352 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-17T15:51:53.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-17T15:51:53.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: get_monmap_and_config 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: monmap: 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.476 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:53.480086+0000 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:53.480086+0000 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:53.477 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 Processor -- start 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 -- start start 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: init 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-17T15:51:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a00a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a00a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 0x7fa7a013d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a013b450 addr v2:172.21.15.203:3300/0 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a00a2130 con 0x7fa7a00a0c70 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fa7a00a2410 con 0x7fa7a013b450 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a00a26f0 con 0x7fa7a00a1760 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa7a5497640 10 monclient(hunting): authenticate will time out at 2446.445518s 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79f7fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 msgr2=0x7fa7a013d840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:53.480 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79f7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 0x7fa7a013d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79effd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7fa7a00a0c70 legacy=0x7fa7a00a1050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:34730/0 (socket says 172.21.15.86:34730) 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 -- 172.21.15.86:0/1763521467 learned_addr learned my addr 172.21.15.86:0/1763521467 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): get_auth_request con 0x7fa7a00a1760 auth_method 0 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.481 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_done global_id 274169 payload 274 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 -- 172.21.15.86:0/1763521467 >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 msgr2=0x7fa7a013d840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 --2- 172.21.15.86:0/1763521467 >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 0x7fa7a013d840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 -- 172.21.15.86:0/1763521467 >> v1:172.21.15.203:6789/0 conn(0x7fa7a00a0c70 legacy=0x7fa7a00a1050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 monclient: found mon.noname-b 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.482 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.478+0000 7fa79e7fc640 1 -- 172.21.15.86:0/1763521467 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a009f9f0 con 0x7fa7a00a1760 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79e7fc640 1 -- 172.21.15.86:0/1763521467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 msgr2=0x7fa7a013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79e7fc640 1 --2- 172.21.15.86:0/1763521467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa790009870 tx=0x7fa79001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d4a9bf0a55fd38d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 1 -- 172.21.15.86:0/1763521467 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900b1070 con 0x7fa7a00a1760 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: dump: 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.483 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 1 -- 172.21.15.86:0/1763521467 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7900a8050 con 0x7fa7a00a1760 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 1 -- 172.21.15.86:0/1763521467 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900ac860 con 0x7fa7a00a1760 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 5 monclient: authenticate success, global_id 274169 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 4 set_mon_vals no callback set 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa79dffb640 10 monclient: dump: 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.485 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.486 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.486 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.486 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.486 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 10 monclient: shutdown 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 -- 172.21.15.86:0/1763521467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 msgr2=0x7fa7a013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 --2- 172.21.15.86:0/1763521467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa790009870 tx=0x7fa79001c440 comp rx=0 tx=0).stop 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 -- 172.21.15.86:0/1763521467 shutdown_connections 2024-09-17T15:51:53.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 --2- 172.21.15.86:0/1763521467 >> v2:172.21.15.203:3300/0 conn(0x7fa7a013b450 0x7fa7a013d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 --2- 172.21.15.86:0/1763521467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a013af10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 -- 172.21.15.86:0/1763521467 >> 172.21.15.86:0/1763521467 conn(0x7fa7a00a2f40 msgr2=0x7fa7a00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 -- 172.21.15.86:0/1763521467 shutdown_connections 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.482+0000 7fa7a5497640 1 -- 172.21.15.86:0/1763521467 wait complete. 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient: monmap: 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 Processor -- start 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 -- start start 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient: init 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a00a0c70 0x7fa7a01cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a00a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a0135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a00a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a013b450 0x7fa7a0136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a013b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0140720 con 0x7fa7a013b450 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01405a0 con 0x7fa7a00a0c70 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a01408a0 con 0x7fa7a00a1760 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa7a5497640 10 monclient(hunting): authenticate will time out at 2446.453514s 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a0135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a0135fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:34746/0 (socket says 172.21.15.86:34746) 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a013b450 0x7fa7a0136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 -- 172.21.15.86:0/2058865835 learned_addr learned my addr 172.21.15.86:0/2058865835 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): get_auth_request con 0x7fa7a00a1760 auth_method 0 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79effd640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a00a0c70 msgr2=0x7fa7a01cde40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): get_auth_request con 0x7fa7a013b450 auth_method 0 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79effd640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a00a0c70 0x7fa7a01cde40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient(hunting): handle_auth_done global_id 274172 payload 824 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a00a0c70 msgr2=0x7fa7a01cde40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa7a00a0c70 0x7fa7a01cde40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a013b450 msgr2=0x7fa7a0136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a013b450 0x7fa7a0136520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 monclient: found mon.noname-c 2024-09-17T15:51:53.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79e7fc640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a013b9c0 con 0x7fa7a00a1760 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.486+0000 7fa79f7fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a013b450 0x7fa7a0136520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79e7fc640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 msgr2=0x7fa7a0135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79e7fc640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a0135fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa79001cce0 tx=0x7fa79001ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900bb020 con 0x7fa7a00a1760 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: dump: 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.497 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7900b7420 con 0x7fa7a00a1760 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900b1040 con 0x7fa7a00a1760 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 5 monclient: authenticate success, global_id 274172 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 monclient: _renew_subs 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a009d460 con 0x7fa7a00a1760 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79d7fa640 4 set_mon_vals no callback set 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 10 monclient: dump: 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.499 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 monclient: _renew_subs 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a0131740 con 0x7fa7a00a1760 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 0x7fa7a0130350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a00a1050 con 0x7fa7a012ff70 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 0x7fa7a0130350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient: get_auth_request con 0x7fa7a012ff70 auth_method 0 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa7900be020 con 0x7fa7a00a1760 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 10 monclient(hunting): handle_auth_done global_id 234570 payload 824 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7a01322a0 con 0x7fa7a012ff70 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa78c03bb60 0x7fa78c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7fa7900e9f00 con 0x7fa7a00a1760 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79f7fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa78c03bb60 0x7fa78c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79f7fe640 10 monclient: get_auth_request con 0x7fa78c03bb60 auth_method 0 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 msgr2=0x7fa7a0130350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.490+0000 7fa79effd640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 0x7fa7a0130350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa79400b450 tx=0x7fa79400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794005270 con 0x7fa7a012ff70 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 10 monclient: dump: 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.504 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa794015070 con 0x7fa7a012ff70 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 msgr2=0x7fa7a0130350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 0x7fa7a0130350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa79400b450 tx=0x7fa79400b910 comp rx=0 tx=0).stop 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 msgr2=0x7fa7a0130350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:53.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa7877fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a012ff70 0x7fa7a0130350 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa79400b450 tx=0x7fa79400b910 comp rx=0 tx=0).stop 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.494+0000 7fa79f7fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa78c03bb60 0x7fa78c03e020 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa798009fd0 tx=0x7fa79800d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa7a5497640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa7a5497640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 0x7fa7a0136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa7a5497640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0133560 con 0x7fa7a0130890 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 0x7fa7a0136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient: get_auth_request con 0x7fa7a0130890 auth_method 0 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 10 monclient(hunting): handle_auth_done global_id 234573 payload 824 2024-09-17T15:51:53.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.498+0000 7fa79effd640 1 -- 172.21.15.86:0/2058865835 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7a006d4f0 con 0x7fa7a0130890 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa79effd640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 msgr2=0x7fa7a0136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa79effd640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 0x7fa7a0136ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa79400b450 tx=0x7fa7940059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79401ae60 con 0x7fa7a0130890 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 10 monclient: dump: 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.507 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fa794010480 con 0x7fa7a0130890 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 msgr2=0x7fa7a0136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 0x7fa7a0136ef0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa79400b450 tx=0x7fa7940059e0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 msgr2=0x7fa7a0136ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:53.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7877fe640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7a0130890 0x7fa7a0136ef0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 222, 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 242697, 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:53.509 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.510 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.511 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:53.512 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.513 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:53.515 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.516 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.517 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.518 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:51:53.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 5 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:53.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.522 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.522 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:53.522 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:53.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa78c03bb60 msgr2=0x7fa78c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa78c03bb60 0x7fa78c03e020 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa798009fd0 tx=0x7fa79800d040 comp rx=0 tx=0).stop 2024-09-17T15:51:53.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 10 monclient: shutdown 2024-09-17T15:51:53.523 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 msgr2=0x7fa7a0135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 --2- 172.21.15.86:0/2058865835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7a00a1760 0x7fa7a0135fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa79001cce0 tx=0x7fa79001ac80 comp rx=0 tx=0).stop 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa79f7fe640 1 -- 172.21.15.86:0/2058865835 reap_dead start 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa79f7fe640 1 -- 172.21.15.86:0/2058865835 reap_dead start 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 shutdown_connections 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 >> 172.21.15.86:0/2058865835 conn(0x7fa7a00a2f40 msgr2=0x7fa7a00a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 shutdown_connections 2024-09-17T15:51:53.524 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.502+0000 7fa7a5497640 1 -- 172.21.15.86:0/2058865835 wait complete. 2024-09-17T15:51:53.524 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:51:53.524 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-17T15:51:53.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: get_monmap_and_config 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: monmap: 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.644 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:51:53.648191+0000 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:51:53.648191+0000 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:51:53.645 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 Processor -- start 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 -- start start 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: init 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e2009cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-b con 0x7f1e200a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e200a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e2013d2f0 0x7f1e2013f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-a con 0x7f1e2013d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:51:53.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e200a6280 con 0x7f1e200a54c0 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f1e2009d2a0 con 0x7f1e2013d2f0 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2009c0a0 con 0x7f1e200a50e0 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e267ce640 10 monclient(hunting): authenticate will time out at 2446.613908s 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e2009cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1e200a54c0 0x7f1e200a58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42662/0 (socket says 172.21.15.86:42662) 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1f7fe640 1 -- 172.21.15.86:0/2541780758 learned_addr learned my addr 172.21.15.86:0/2541780758 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): get_auth_request con 0x7f1e200a50e0 auth_method 0 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 1 --2- 172.21.15.86:0/2541780758 >> v2:172.21.15.86:3300/0 conn(0x7f1e2013d2f0 0x7f1e2013f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): get_auth_request con 0x7f1e2013d2f0 auth_method 0 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e24d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_done global_id 274175 payload 274 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2541780758 >> v2:172.21.15.86:3300/0 conn(0x7f1e2013d2f0 msgr2=0x7f1e2013f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:53.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 --2- 172.21.15.86:0/2541780758 >> v2:172.21.15.86:3300/0 conn(0x7f1e2013d2f0 0x7f1e2013f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2541780758 >> v1:172.21.15.86:6789/0 conn(0x7f1e200a54c0 legacy=0x7f1e200a58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 monclient: found mon.noname-b 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.646+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2541780758 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e2009c380 con 0x7f1e200a50e0 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2541780758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 msgr2=0x7f1e2009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1ffff640 1 --2- 172.21.15.86:0/2541780758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e2009cd60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009d90 tx=0x7f1e1801c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ddf86848474dbb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 1 -- 172.21.15.86:0/2541780758 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180b2070 con 0x7f1e200a50e0 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: dump: 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.655 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 1 -- 172.21.15.86:0/2541780758 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e180a5760 con 0x7f1e200a50e0 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 5 monclient: authenticate success, global_id 274175 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 10 monclient: get_monmap_and_config success 2024-09-17T15:51:53.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 4 set_mon_vals no callback set 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 1 -- 172.21.15.86:0/2541780758 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180a5b30 con 0x7f1e200a50e0 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e1effd640 10 monclient: dump: 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.657 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 10 monclient: shutdown 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 -- 172.21.15.86:0/2541780758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 msgr2=0x7f1e2009cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2541780758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e2009cd60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1e18009d90 tx=0x7f1e1801c800 comp rx=0 tx=0).stop 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 -- 172.21.15.86:0/2541780758 shutdown_connections 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2541780758 >> v2:172.21.15.86:3300/0 conn(0x7f1e2013d2f0 0x7f1e2013f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2541780758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e2009cd60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 -- 172.21.15.86:0/2541780758 >> 172.21.15.86:0/2541780758 conn(0x7f1e200aaae0 msgr2=0x7f1e200aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 -- 172.21.15.86:0/2541780758 shutdown_connections 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.650+0000 7f1e267ce640 1 -- 172.21.15.86:0/2541780758 wait complete. 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient: build_initial_monmap 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient: monmap: 2024-09-17T15:51:53.658 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 Processor -- start 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 -- start start 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient: init 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient: _reopen_session rank -1 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:51:53.659 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e201340c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-c con 0x7f1e200a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e200a54c0 0x7f1e20134600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-a con 0x7f1e200a54c0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e2013d2f0 0x7f1e200a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): picked mon.noname-b con 0x7f1e2013d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f1e20142650 con 0x7f1e200a54c0 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1e200a6b00 con 0x7f1e2013d2f0 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1e201427d0 con 0x7f1e200a50e0 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): _renew_subs 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e267ce640 10 monclient(hunting): authenticate will time out at 2446.621108s 2024-09-17T15:51:53.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e201340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:34758/0 (socket says 172.21.15.86:34758) 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2188825440 learned_addr learned my addr 172.21.15.86:0/2188825440 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): get_auth_request con 0x7f1e200a50e0 auth_method 0 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e200a54c0 0x7f1e20134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): get_auth_request con 0x7f1e200a54c0 auth_method 0 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e24d6b640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e2013d2f0 msgr2=0x7f1e200a1f20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:51:53.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e24d6b640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e2013d2f0 0x7f1e200a1f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient(hunting): handle_auth_done global_id 274178 payload 824 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e2013d2f0 msgr2=0x7f1e200a1f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1e2013d2f0 0x7f1e200a1f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e200a54c0 msgr2=0x7f1e20134600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e200a54c0 0x7f1e20134600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 monclient: found mon.noname-c 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:51:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e20139b50 con 0x7f1e200a50e0 2024-09-17T15:51:53.663 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e200a54c0 0x7f1e20134600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:51:53.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 msgr2=0x7f1e201340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.654+0000 7f1e1ffff640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e201340c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e1801d0a0 tx=0x7f1e180195a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180bd020 con 0x7f1e200a50e0 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: dump: 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: _finish_auth 0 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e180b8680 con 0x7f1e200a50e0 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 5 monclient: authenticate success, global_id 274178 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 monclient: _renew_subs 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e20139e30 con 0x7f1e200a50e0 2024-09-17T15:51:53.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e180b2040 con 0x7f1e200a50e0 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: dump: 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.669 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 monclient: _renew_subs 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1e7fc640 4 set_mon_vals no callback set 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e200a18d0 con 0x7f1e200a50e0 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 0x7f1e200a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2006a0e0 con 0x7f1e200a2460 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 0x7f1e200a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient: get_auth_request con 0x7f1e200a2460 auth_method 0 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f1e180c0020 con 0x7f1e200a50e0 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_done global_id 274181 payload 824 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e2013da90 con 0x7f1e200a2460 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e0003bb60 0x7f1e0003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(861..861 src has 794..861) ==== 4799+0+0 (secure 0 0 0) 0x7f1e180eacd0 con 0x7f1e200a50e0 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e24d6b640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e0003bb60 0x7f1e0003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e24d6b640 10 monclient: get_auth_request con 0x7f1e0003bb60 auth_method 0 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 msgr2=0x7f1e200a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 0x7f1e200a2840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e08007270 tx=0x7f1e08007730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800fce0 con 0x7f1e200a2460 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: dump: 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e0800fce0 con 0x7f1e200a2460 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 msgr2=0x7f1e200a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 0x7f1e200a2840 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e08007270 tx=0x7f1e08007730 comp rx=0 tx=0).stop 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 msgr2=0x7f1e200a2840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.658+0000 7f1e1cff9640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2460 0x7f1e200a2840 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.662+0000 7f1e24d6b640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e0003bb60 0x7f1e0003e020 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1e2013a850 tx=0x7f1e10009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e267ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 0x7f1e2009e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e267ce640 10 monclient(hunting): start opening mon connection 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f1e2013a200 con 0x7f1e200a2d80 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 0x7f1e2009e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:51:53.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient: get_auth_request con 0x7f1e200a2d80 auth_method 0 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 10 monclient(hunting): handle_auth_done global_id 274184 payload 824 2024-09-17T15:51:53.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.666+0000 7f1e1f7fe640 1 -- 172.21.15.86:0/2188825440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e2013d6d0 con 0x7f1e200a2d80 2024-09-17T15:51:53.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1f7fe640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 msgr2=0x7f1e2009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:51:53.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1f7fe640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 0x7f1e2009e5e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1e08017000 tx=0x7f1e08004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:51:53.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e08006450 con 0x7f1e200a2d80 2024-09-17T15:51:53.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 10 monclient: dump: 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:51:53.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f1e08006450 con 0x7f1e200a2d80 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 msgr2=0x7f1e2009e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 0x7f1e2009e5e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1e08017000 tx=0x7f1e08004950 comp rx=0 tx=0).stop 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 msgr2=0x7f1e2009e5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e1cff9640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a2d80 0x7f1e2009e5e0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1e08017000 tx=0x7f1e08004950 comp rx=0 tx=0).stop 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:53.677 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 222, 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 32991, 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 0, 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.678 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:51:53.679 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.680 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:51:53.681 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:51:53.682 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.683 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:51:53.684 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:51:53.685 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.686 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.688 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:51:53.689 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 3 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:51:53.690 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e0003bb60 msgr2=0x7f1e0003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f1e0003bb60 0x7f1e0003e020 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1e2013a850 tx=0x7f1e10009190 comp rx=0 tx=0).stop 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 10 monclient: shutdown 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 msgr2=0x7f1e201340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 --2- 172.21.15.86:0/2188825440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e200a50e0 0x7f1e201340c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e1801d0a0 tx=0x7f1e180195a0 comp rx=0 tx=0).stop 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e24d6b640 1 -- 172.21.15.86:0/2188825440 reap_dead start 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e24d6b640 1 -- 172.21.15.86:0/2188825440 reap_dead start 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 shutdown_connections 2024-09-17T15:51:53.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 >> 172.21.15.86:0/2188825440 conn(0x7f1e200aaae0 msgr2=0x7f1e2013c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:51:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 shutdown_connections 2024-09-17T15:51:53.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:51:53.670+0000 7f1e267ce640 1 -- 172.21.15.86:0/2188825440 wait complete. 2024-09-17T15:51:53.693 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-17T15:51:53.693 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3961: done (4 left) 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3962: read oid 50 snap 419 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3962: expect (ObjNum 1361 snap 413 seq_num 1361) 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3963: read oid 31 snap -1 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3963: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3964: delete oid 43 current snap is 422 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:3958: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:54.039 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 226881 2024-09-17T15:51:54.104 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:54.138 INFO:tasks.rados.rados.0.smithi203.stdout:3960: finishing rollback tid 1 to smithi20328086-11 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 11 v 0 (ObjNum 241 snap 0 seq_num 3606570480) dirty dne 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3956: finishing rollback tid 0 to smithi20328086-3 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3956: finishing rollback tid 1 to smithi20328086-3 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3956: finishing rollback tid 2 to smithi20328086-3 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 3 v 1918 (ObjNum 1382 snap 417 seq_num 1382) dirty exists 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3957: finishing rollback tid 0 to smithi20328086-44 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3957: finishing rollback tid 1 to smithi20328086-44 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3957: finishing rollback tid 2 to smithi20328086-44 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 44 v 1733 (ObjNum 1292 snap 396 seq_num 1292) dirty exists 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:3963: expect (ObjNum 1356 snap 412 seq_num 1356) 2024-09-17T15:51:54.139 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:54.185 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:54.273 INFO:tasks.rados.rados.0.smithi203.stdout:3962: expect (ObjNum 1361 snap 413 seq_num 1361) 2024-09-17T15:51:54.273 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 246057 2024-09-17T15:51:54.360 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2226235 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3956: done (6 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3957: done (5 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3958: done (4 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3960: done (3 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3962: done (2 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3963: done (1 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3964: done (0 left) 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3965: write oid 37 current snap is 422 2024-09-17T15:51:54.371 INFO:tasks.rados.rados.0.smithi203.stdout:3965: seq_num 1395 ranges {645765=664954,1998616=752477,3158722=287043} 2024-09-17T15:51:54.392 INFO:tasks.rados.rados.0.smithi203.stdout:3965: writing smithi20328086-37 from 645765 to 1310719 tid 1 2024-09-17T15:51:54.416 INFO:tasks.rados.rados.0.smithi203.stdout:3965: writing smithi20328086-37 from 1998616 to 2751093 tid 2 2024-09-17T15:51:54.426 INFO:tasks.rados.rados.0.smithi203.stdout:3965: writing smithi20328086-37 from 3158722 to 3445765 tid 3 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:3966: rollback oid 45 current snap is 422 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 45 to 420 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:3965: finishing write tid 1 to smithi20328086-37 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:3965: finishing write tid 2 to smithi20328086-37 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:3967: rollback oid 5 current snap is 422 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 5 to 421 2024-09-17T15:51:54.427 INFO:tasks.rados.rados.0.smithi203.stdout:3968: snap_create 2024-09-17T15:51:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3966: finishing rollback tid 0 to smithi20328086-45 2024-09-17T15:51:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3966: finishing rollback tid 1 to smithi20328086-45 2024-09-17T15:51:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3966: finishing rollback tid 2 to smithi20328086-45 2024-09-17T15:51:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 45 v 1359 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:51:54.435 INFO:tasks.rados.rados.0.smithi203.stdout:3965: finishing write tid 3 to smithi20328086-37 2024-09-17T15:51:54.436 INFO:tasks.rados.rados.0.smithi203.stdout:3965: finishing write tid 4 to smithi20328086-37 2024-09-17T15:51:54.436 INFO:tasks.rados.rados.0.smithi203.stdout:3967: finishing rollback tid 0 to smithi20328086-5 2024-09-17T15:51:54.436 INFO:tasks.rados.rados.0.smithi203.stdout:3967: finishing rollback tid 1 to smithi20328086-5 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: finishing write tid 5 to smithi20328086-37 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: oid 37 updating version 0 to 2090 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: oid 37 updating version 2090 to 2091 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: oid 37 updating version 2091 to 2092 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: oid 37 version 2092 is already newer than 2089 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 37 v 2092 (ObjNum 1395 snap 422 seq_num 1395) dirty exists 2024-09-17T15:51:54.437 INFO:tasks.rados.rados.0.smithi203.stdout:3965: left oid 37 (ObjNum 1395 snap 422 seq_num 1395) 2024-09-17T15:51:54.438 INFO:tasks.rados.rados.0.smithi203.stdout:3967: finishing rollback tid 2 to smithi20328086-5 2024-09-17T15:51:54.438 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 5 v 1447 (ObjNum 1354 snap 410 seq_num 1354) dirty exists 2024-09-17T15:51:55.042 INFO:tasks.rados.rados.0.smithi203.stdout:3965: done (3 left) 2024-09-17T15:51:55.042 INFO:tasks.rados.rados.0.smithi203.stdout:3966: done (2 left) 2024-09-17T15:51:55.042 INFO:tasks.rados.rados.0.smithi203.stdout:3967: done (1 left) 2024-09-17T15:51:55.042 INFO:tasks.rados.rados.0.smithi203.stdout:3968: done (0 left) 2024-09-17T15:51:55.042 INFO:tasks.rados.rados.0.smithi203.stdout:3969: delete oid 34 current snap is 423 2024-09-17T15:51:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:3969: done (0 left) 2024-09-17T15:51:55.049 INFO:tasks.rados.rados.0.smithi203.stdout:3970: snap_create 2024-09-17T15:51:55.894 INFO:tasks.rados.rados.0.smithi203.stdout:3970: done (0 left) 2024-09-17T15:51:55.894 INFO:tasks.rados.rados.0.smithi203.stdout:3971: read oid 7 snap -1 2024-09-17T15:51:55.894 INFO:tasks.rados.rados.0.smithi203.stdout:3971: expect (ObjNum 1300 snap 400 seq_num 1300) 2024-09-17T15:51:55.894 INFO:tasks.rados.rados.0.smithi203.stdout:3972: write oid 22 current snap is 424 2024-09-17T15:51:55.894 INFO:tasks.rados.rados.0.smithi203.stdout:3972: seq_num 1397 ranges {679379=415452,1599227=756146,3097215=382164} 2024-09-17T15:51:55.912 INFO:tasks.rados.rados.0.smithi203.stdout:3972: writing smithi20328086-22 from 679379 to 1094831 tid 1 2024-09-17T15:51:55.932 INFO:tasks.rados.rados.0.smithi203.stdout:3972: writing smithi20328086-22 from 1599227 to 2355373 tid 2 2024-09-17T15:51:55.949 INFO:tasks.rados.rados.0.smithi203.stdout:3972: writing smithi20328086-22 from 3097215 to 3479379 tid 3 2024-09-17T15:51:55.949 INFO:tasks.rados.rados.0.smithi203.stdout:3973: read oid 10 snap -1 2024-09-17T15:51:55.949 INFO:tasks.rados.rados.0.smithi203.stdout:3973: expect deleted 2024-09-17T15:51:55.949 INFO:tasks.rados.rados.0.smithi203.stdout:3974: snap_remove snap 417 2024-09-17T15:51:56.905 INFO:tasks.rados.rados.0.smithi203.stdout:3974: done (3 left) 2024-09-17T15:51:56.905 INFO:tasks.rados.rados.0.smithi203.stdout:3975: snap_create 2024-09-17T15:51:56.905 INFO:tasks.rados.rados.0.smithi203.stdout:3971: expect (ObjNum 1300 snap 400 seq_num 1300) 2024-09-17T15:51:56.905 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:57.098 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:57.174 INFO:tasks.rados.rados.0.smithi203.stdout:3972: finishing write tid 1 to smithi20328086-22 2024-09-17T15:51:57.174 INFO:tasks.rados.rados.0.smithi203.stdout:3972: finishing write tid 2 to smithi20328086-22 2024-09-17T15:51:57.174 INFO:tasks.rados.rados.0.smithi203.stdout:3972: finishing write tid 3 to smithi20328086-22 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: finishing write tid 4 to smithi20328086-22 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: finishing write tid 5 to smithi20328086-22 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: oid 22 updating version 0 to 1697 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: oid 22 updating version 1697 to 1698 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: oid 22 version 1698 is already newer than 1695 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: oid 22 version 1698 is already newer than 1696 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 22 v 1698 (ObjNum 1397 snap 424 seq_num 1397) dirty exists 2024-09-17T15:51:57.175 INFO:tasks.rados.rados.0.smithi203.stdout:3972: left oid 22 (ObjNum 1397 snap 424 seq_num 1397) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3971: done (3 left) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3972: done (2 left) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3973: done (1 left) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3975: done (0 left) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3976: copy_from oid 6 from oid 34 current snap is 425 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3977: read oid 9 snap -1 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3977: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:57.907 INFO:tasks.rados.rados.0.smithi203.stdout:3978: snap_remove snap 421 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3978: done (2 left) 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3979: delete oid 1 current snap is 425 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3976: finishing copy_from to smithi20328086-6 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3976: got expected ENOENT (src dne) 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3976: finishing copy_from racing read to smithi20328086-6 2024-09-17T15:51:58.926 INFO:tasks.rados.rados.0.smithi203.stdout:3977: expect (ObjNum 1332 snap 406 seq_num 1332) 2024-09-17T15:51:58.941 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 1897279 2024-09-17T15:51:58.951 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:51:59.012 INFO:tasks.rados.rados.0.smithi203.stdout:3976: done (2 left) 2024-09-17T15:51:59.012 INFO:tasks.rados.rados.0.smithi203.stdout:3977: done (1 left) 2024-09-17T15:51:59.012 INFO:tasks.rados.rados.0.smithi203.stdout:3979: done (0 left) 2024-09-17T15:51:59.013 INFO:tasks.rados.rados.0.smithi203.stdout:3980: snap_remove snap 422 2024-09-17T15:51:59.937 INFO:tasks.rados.rados.0.smithi203.stdout:3980: done (0 left) 2024-09-17T15:51:59.937 INFO:tasks.rados.rados.0.smithi203.stdout:3981: snap_create 2024-09-17T15:52:00.948 INFO:tasks.rados.rados.0.smithi203.stdout:3981: done (0 left) 2024-09-17T15:52:00.948 INFO:tasks.rados.rados.0.smithi203.stdout:3982: snap_create 2024-09-17T15:52:01.948 INFO:tasks.rados.rados.0.smithi203.stdout:3982: done (0 left) 2024-09-17T15:52:02.046 INFO:tasks.rados.rados.0.smithi203.stdout:3983: write (excl) oid 18 current snap is 427 2024-09-17T15:52:02.047 INFO:tasks.rados.rados.0.smithi203.stdout:3983: seq_num 1399 ranges {712993=682303,1832544=759815,3268414=244579} 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3983: writing smithi20328086-18 from 712993 to 1395296 tid 1 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3983: writing smithi20328086-18 from 1832544 to 2592359 tid 2 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3983: writing smithi20328086-18 from 3268414 to 3512993 tid 3 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3984: rollback oid 50 current snap is 427 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:rollback oid 50 to 420 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3983: finishing write tid 1 to smithi20328086-18 2024-09-17T15:52:02.049 INFO:tasks.rados.rados.0.smithi203.stdout:3983: finishing write tid 2 to smithi20328086-18 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3985: delete oid 16 current snap is 427 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: finishing write tid 3 to smithi20328086-18 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3984: finishing rollback tid 1 to smithi20328086-50 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: finishing write tid 4 to smithi20328086-18 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3984: finishing rollback tid 2 to smithi20328086-50 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 50 v 1756 (ObjNum 1361 snap 413 seq_num 1361) dirty exists 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: finishing write tid 5 to smithi20328086-18 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: oid 18 updating version 0 to 1662 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: oid 18 updating version 1662 to 1664 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: oid 18 version 1664 is already newer than 1661 2024-09-17T15:52:02.050 INFO:tasks.rados.rados.0.smithi203.stdout:3983: oid 18 version 1664 is already newer than 1663 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 18 v 1664 (ObjNum 1399 snap 427 seq_num 1399) dirty exists 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3983: left oid 18 (ObjNum 1399 snap 427 seq_num 1399) 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3983: done (2 left) 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3984: done (1 left) 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3985: done (0 left) 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3986: read oid 47 snap -1 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3986: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3987: write oid 39 current snap is 427 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3987: seq_num 1401 ranges {746607=432801,1833155=763484,3206907=339700} 2024-09-17T15:52:02.051 INFO:tasks.rados.rados.0.smithi203.stdout:3987: writing smithi20328086-39 from 746607 to 1179408 tid 1 2024-09-17T15:52:02.052 INFO:tasks.rados.rados.0.smithi203.stdout:3987: writing smithi20328086-39 from 1833155 to 2596639 tid 2 2024-09-17T15:52:02.061 INFO:tasks.rados.rados.0.smithi203.stdout:3987: writing smithi20328086-39 from 3206907 to 3546607 tid 3 2024-09-17T15:52:02.061 INFO:tasks.rados.rados.0.smithi203.stdout:3988: copy_from oid 36 from oid 43 current snap is 427 2024-09-17T15:52:02.061 INFO:tasks.rados.rados.0.smithi203.stdout:3986: expect (ObjNum 1117 snap 344 seq_num 1117) 2024-09-17T15:52:02.128 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2630848 2024-09-17T15:52:02.155 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 2034705 2024-09-17T15:52:02.175 INFO:tasks.rados.rados.0.smithi203.stdout:seeking to 0 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3987: finishing write tid 1 to smithi20328086-39 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3987: finishing write tid 2 to smithi20328086-39 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3988: finishing copy_from to smithi20328086-36 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3988: got expected ENOENT (src dne) 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3988: finishing copy_from racing read to smithi20328086-36 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3986: done (2 left) 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3988: done (1 left) 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3989: delete oid 49 current snap is 427 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3987: finishing write tid 3 to smithi20328086-39 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3987: finishing write tid 4 to smithi20328086-39 2024-09-17T15:52:02.217 INFO:tasks.rados.rados.0.smithi203.stdout:3987: finishing write tid 5 to smithi20328086-39 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:3987: oid 39 updating version 0 to 2100 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:3987: oid 39 version 2100 is already newer than 2099 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:3987: oid 39 updating version 2100 to 2101 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:3987: oid 39 updating version 2101 to 2102 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 39 v 2102 (ObjNum 1401 snap 427 seq_num 1401) dirty exists 2024-09-17T15:52:02.218 INFO:tasks.rados.rados.0.smithi203.stdout:3987: left oid 39 (ObjNum 1401 snap 427 seq_num 1401) 2024-09-17T15:52:02.222 INFO:tasks.rados.rados.0.smithi203.stdout:3987: done (1 left) 2024-09-17T15:52:02.222 INFO:tasks.rados.rados.0.smithi203.stdout:3989: done (0 left) 2024-09-17T15:52:02.222 INFO:tasks.rados.rados.0.smithi203.stdout:3990: snap_create 2024-09-17T15:52:02.963 INFO:tasks.rados.rados.0.smithi203.stdout:3990: done (0 left) 2024-09-17T15:52:02.964 INFO:tasks.rados.rados.0.smithi203.stdout:3991: write (excl) oid 14 current snap is 428 2024-09-17T15:52:02.964 INFO:tasks.rados.rados.0.smithi203.stdout:3991: seq_num 1403 ranges {780221=583299,1950119=767153,3261753=318468} 2024-09-17T15:52:02.985 INFO:tasks.rados.rados.0.smithi203.stdout:3991: writing smithi20328086-14 from 780221 to 1363520 tid 1 2024-09-17T15:52:03.008 INFO:tasks.rados.rados.0.smithi203.stdout:3991: writing smithi20328086-14 from 1950119 to 2717272 tid 2 2024-09-17T15:52:03.020 INFO:tasks.rados.rados.0.smithi203.stdout:3991: writing smithi20328086-14 from 3261753 to 3580221 tid 3 2024-09-17T15:52:03.021 INFO:tasks.rados.rados.0.smithi203.stdout:3992: delete oid 38 current snap is 428 2024-09-17T15:52:03.021 INFO:tasks.rados.rados.0.smithi203.stdout:3991: finishing write tid 1 to smithi20328086-14 2024-09-17T15:52:03.021 INFO:tasks.rados.rados.0.smithi203.stdout:3991: finishing write tid 2 to smithi20328086-14 2024-09-17T15:52:03.027 INFO:tasks.rados.rados.0.smithi203.stdout:3992: done (1 left) 2024-09-17T15:52:03.027 INFO:tasks.rados.rados.0.smithi203.stdout:3993: write oid 2 current snap is 428 2024-09-17T15:52:03.028 INFO:tasks.rados.rados.0.smithi203.stdout:3993: seq_num 1405 ranges {413835=733797,1550730=770822,2800246=557531} 2024-09-17T15:52:03.041 INFO:tasks.rados.rados.0.smithi203.stdout:3993: writing smithi20328086-2 from 413835 to 1147632 tid 1 2024-09-17T15:52:03.056 INFO:tasks.rados.rados.0.smithi203.stdout:3993: writing smithi20328086-2 from 1550730 to 2321552 tid 2 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3993: writing smithi20328086-2 from 2800246 to 3357777 tid 3 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3994: copy_from oid 35 from oid 45 current snap is 428 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: finishing write tid 3 to smithi20328086-14 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: finishing write tid 4 to smithi20328086-14 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: finishing write tid 5 to smithi20328086-14 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: oid 14 updating version 0 to 1771 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: oid 14 updating version 1771 to 1772 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: oid 14 version 1772 is already newer than 1770 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: oid 14 version 1772 is already newer than 1769 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 14 v 1772 (ObjNum 1403 snap 428 seq_num 1403) dirty exists 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3991: left oid 14 (ObjNum 1403 snap 428 seq_num 1403) 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3993: finishing write tid 1 to smithi20328086-2 2024-09-17T15:52:03.068 INFO:tasks.rados.rados.0.smithi203.stdout:3993: finishing write tid 2 to smithi20328086-2 2024-09-17T15:52:03.069 INFO:tasks.rados.rados.0.smithi203.stdout:3991: done (2 left) 2024-09-17T15:52:03.069 INFO:tasks.rados.rados.0.smithi203.stdout:3995: snap_create 2024-09-17T15:52:03.075 INFO:tasks.rados.rados.0.smithi203.stdout:3993: finishing write tid 3 to smithi20328086-2 2024-09-17T15:52:03.075 INFO:tasks.rados.rados.0.smithi203.stdout:3993: finishing write tid 4 to smithi20328086-2 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: finishing write tid 5 to smithi20328086-2 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: oid 2 updating version 0 to 1892 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: oid 2 version 1892 is already newer than 1891 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: oid 2 version 1892 is already newer than 1889 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: oid 2 version 1892 is already newer than 1890 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 2 v 1892 (ObjNum 1405 snap 428 seq_num 1405) dirty exists 2024-09-17T15:52:03.076 INFO:tasks.rados.rados.0.smithi203.stdout:3993: left oid 2 (ObjNum 1405 snap 428 seq_num 1405) 2024-09-17T15:52:03.090 INFO:tasks.rados.rados.0.smithi203.stdout:3994: finishing copy_from racing read to smithi20328086-35 2024-09-17T15:52:03.092 INFO:tasks.rados.rados.0.smithi203.stdout:3994: finishing copy_from to smithi20328086-35 2024-09-17T15:52:03.092 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 35 v 1359 (ObjNum 1226 snap 368 seq_num 1226) dirty exists 2024-09-17T15:52:03.972 INFO:tasks.rados.rados.0.smithi203.stdout:3993: done (2 left) 2024-09-17T15:52:03.972 INFO:tasks.rados.rados.0.smithi203.stdout:3994: done (1 left) 2024-09-17T15:52:03.972 INFO:tasks.rados.rados.0.smithi203.stdout:3995: done (0 left) 2024-09-17T15:52:03.972 INFO:tasks.rados.rados.0.smithi203.stdout:3996: snap_create 2024-09-17T15:52:04.981 INFO:tasks.rados.rados.0.smithi203.stdout:3996: done (0 left) 2024-09-17T15:52:04.981 INFO:tasks.rados.rados.0.smithi203.stdout:3997: delete oid 9 current snap is 430 2024-09-17T15:52:04.992 INFO:tasks.rados.rados.0.smithi203.stdout:3997: done (0 left) 2024-09-17T15:52:04.992 INFO:tasks.rados.rados.0.smithi203.stdout:3998: copy_from oid 49 from oid 8 current snap is 430 2024-09-17T15:52:04.992 INFO:tasks.rados.rados.0.smithi203.stdout:3999: read oid 18 snap -1 2024-09-17T15:52:04.992 INFO:tasks.rados.rados.0.smithi203.stdout:3999: expect (ObjNum 1399 snap 427 seq_num 1399) 2024-09-17T15:52:04.992 INFO:tasks.rados.rados.0.smithi203.stdout: waiting on 2 2024-09-17T15:52:05.002 INFO:tasks.rados.rados.0.smithi203.stdout:3999: expect (ObjNum 1399 snap 427 seq_num 1399) 2024-09-17T15:52:05.157 INFO:tasks.rados.rados.0.smithi203.stdout:3998: finishing copy_from racing read to smithi20328086-49 2024-09-17T15:52:05.157 INFO:tasks.rados.rados.0.smithi203.stdout:3998: finishing copy_from to smithi20328086-49 2024-09-17T15:52:05.157 INFO:tasks.rados.rados.0.smithi203.stdout:update_object_version oid 49 v 1660 (ObjNum 1379 snap 417 seq_num 1379) dirty exists 2024-09-17T15:52:05.157 INFO:tasks.rados.rados.0.smithi203.stdout:3998: done (1 left) 2024-09-17T15:52:05.157 INFO:tasks.rados.rados.0.smithi203.stdout:3999: done (0 left) 2024-09-17T15:52:05.159 INFO:tasks.rados.rados.0.smithi203.stderr:0 errors. 2024-09-17T15:52:05.160 INFO:tasks.rados.rados.0.smithi203.stderr: 2024-09-17T15:52:05.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:52:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: get_monmap_and_config 2024-09-17T15:52:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: monmap: 2024-09-17T15:52:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:05.310230+0000 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:05.310230+0000 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:05.307 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 Processor -- start 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 -- start start 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: init 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 0x7f5f2406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5f240689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:05.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f24068db0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f2413aff0 0x7f5f2413d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f5f2413aff0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f2409c8b0 con 0x7f5f24068db0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f5f24074500 con 0x7f5f2413aff0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2409cb90 con 0x7f5f240689d0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f29ec9640 10 monclient(hunting): authenticate will time out at 2458.275798s 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.305+0000 7f5f23fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f2413aff0 0x7f5f2413d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f2413aff0 0x7f5f2413d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55516/0 (socket says 172.21.15.86:55516) 2024-09-17T15:52:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f237fe640 1 -- >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 msgr2=0x7f5f2406f840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f22ffd640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f5f24068db0 legacy=0x7f5f24069200 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 -- 172.21.15.86:0/2320263726 learned_addr learned my addr 172.21.15.86:0/2320263726 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): get_auth_request con 0x7f5f2413aff0 auth_method 0 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f237fe640 1 --2- 172.21.15.86:0/2320263726 >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 0x7f5f2406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient(hunting): handle_auth_done global_id 234582 payload 274 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 -- 172.21.15.86:0/2320263726 >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 msgr2=0x7f5f2406f840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 --2- 172.21.15.86:0/2320263726 >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 0x7f5f2406f840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 -- 172.21.15.86:0/2320263726 >> v1:172.21.15.203:6789/0 conn(0x7f5f24068db0 legacy=0x7f5f24069200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 monclient: found mon.noname-a 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.309+0000 7f5f23fff640 1 -- 172.21.15.86:0/2320263726 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2409ce70 con 0x7f5f2413aff0 2024-09-17T15:52:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 -- 172.21.15.86:0/2320263726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2413aff0 msgr2=0x7f5f2413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 --2- 172.21.15.86:0/2320263726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2413aff0 0x7f5f2413d3e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5f24069940 tx=0x7f5f1401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6eb8e66ca014f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 1 -- 172.21.15.86:0/2320263726 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f140b1070 con 0x7f5f2413aff0 2024-09-17T15:52:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: dump: 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.318 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 5 monclient: authenticate success, global_id 234582 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 1 -- 172.21.15.86:0/2320263726 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f140a8050 con 0x7f5f2413aff0 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 4 set_mon_vals no callback set 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 1 -- 172.21.15.86:0/2320263726 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2409c2d0 con 0x7f5f2413aff0 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f227fc640 10 monclient: dump: 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: shutdown 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- 172.21.15.86:0/2320263726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2413aff0 msgr2=0x7f5f2413d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/2320263726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2413aff0 0x7f5f2413d3e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5f24069940 tx=0x7f5f1401c400 comp rx=0 tx=0).stop 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- 172.21.15.86:0/2320263726 shutdown_connections 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/2320263726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f2413aff0 0x7f5f2413d3e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/2320263726 >> v2:172.21.15.203:3300/0 conn(0x7f5f240689d0 0x7f5f2406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- 172.21.15.86:0/2320263726 >> 172.21.15.86:0/2320263726 conn(0x7f5f240a6440 msgr2=0x7f5f240a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- 172.21.15.86:0/2320263726 shutdown_connections 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- 172.21.15.86:0/2320263726 wait complete. 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: monmap: 2024-09-17T15:52:05.321 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 Processor -- start 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- start start 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: init 2024-09-17T15:52:05.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f5f240689d0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 0x7f5f2409d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f5f24068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 0x7f5f2409dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f5f2413aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241405f0 con 0x7f5f240689d0 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5f24140470 con 0x7f5f24068db0 2024-09-17T15:52:05.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f5f24140770 con 0x7f5f2413aff0 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f29ec9640 10 monclient(hunting): authenticate will time out at 2458.283041s 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f22ffd640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 msgr2=0x7f5f2409d7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55522/0 (socket says 172.21.15.86:55522) 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 1 -- 172.21.15.86:0/3238935221 learned_addr learned my addr 172.21.15.86:0/3238935221 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): get_auth_request con 0x7f5f240689d0 auth_method 0 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 0x7f5f2409dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f22ffd640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 0x7f5f2409d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): get_auth_request con 0x7f5f2413aff0 auth_method 0 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.325 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient(hunting): handle_auth_done global_id 274187 payload 824 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 -- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 msgr2=0x7f5f2409d7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 0x7f5f2409d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 -- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 msgr2=0x7f5f24133230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 monclient: found mon.noname-c 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f23fff640 1 -- 172.21.15.86:0/3238935221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f24072210 con 0x7f5f2413aff0 2024-09-17T15:52:05.326 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.313+0000 7f5f237fe640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f23fff640 1 -- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 msgr2=0x7f5f2409dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f23fff640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 0x7f5f2409dce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f140ac000 tx=0x7f5f1401ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f140bb030 con 0x7f5f2413aff0 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: dump: 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.329 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f140acd90 con 0x7f5f2413aff0 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 5 monclient: authenticate success, global_id 274187 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _renew_subs 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f240724f0 con 0x7f5f2413aff0 2024-09-17T15:52:05.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f140b1040 con 0x7f5f2413aff0 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 10 monclient: dump: 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.331 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f21ffb640 4 set_mon_vals no callback set 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _renew_subs 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f2413d440 con 0x7f5f2413aff0 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f2409d460 con 0x7f5f2413aff0 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f5f140be070 con 0x7f5f2413aff0 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 0x7f5f0403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(873..873 src has 794..873) ==== 4755+0+0 (secure 0 0 0) 0x7f5f140e98b0 con 0x7f5f2413aff0 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f237fe640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 0x7f5f0403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.317+0000 7f5f237fe640 10 monclient: get_auth_request con 0x7f5f0403bb90 auth_method 0 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.321+0000 7f5f237fe640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 0x7f5f0403e050 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0094b0 tx=0x7f5f1c009400 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.321+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f5f140c5060 con 0x7f5f2413aff0 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.321+0000 7f5f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.321+0000 7f5f1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:05.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.493+0000 7f5f29ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:05.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.493+0000 7f5f29ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:05.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.493+0000 7f5f29ec9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.495 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.493+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f2406f840 con 0x7f5f2413aff0 2024-09-17T15:52:05.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f1bfff640 1 -- 172.21.15.86:0/3238935221 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v873) ==== 74+0+14232 (secure 0 0 0) 0x7f5f14016110 con 0x7f5f2413aff0 2024-09-17T15:52:05.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:05.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:05.497 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:05.497 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":873,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:52:04.971114+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:27:23.674476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:27:24.613917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-17T15:27:50.724298+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"873","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":435,"snap_epoch":873,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.592045+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.760848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.159435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.733344+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.364717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.514816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":423,"length":2},{"begin":427,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 msgr2=0x7f5f0403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 0x7f5f0403e050 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0094b0 tx=0x7f5f1c009400 comp rx=0 tx=0).stop 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 10 monclient: shutdown 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 msgr2=0x7f5f2409dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 0x7f5f2409dce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f140ac000 tx=0x7f5f1401ac40 comp rx=0 tx=0).stop 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 shutdown_connections 2024-09-17T15:52:05.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f5f0403bb90 0x7f5f0403e050 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f2413aff0 0x7f5f2409dce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5f24068db0 0x7f5f2409d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 --2- 172.21.15.86:0/3238935221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f240689d0 0x7f5f24133230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 >> 172.21.15.86:0/3238935221 conn(0x7f5f240a6440 msgr2=0x7f5f2413c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 shutdown_connections 2024-09-17T15:52:05.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.497+0000 7f5f29ec9640 1 -- 172.21.15.86:0/3238935221 wait complete. 2024-09-17T15:52:05.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:05.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.629+0000 7f2aa627b640 10 monclient: get_monmap_and_config 2024-09-17T15:52:05.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.629+0000 7f2aa627b640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.629+0000 7f2aa627b640 10 monclient: monmap: 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:05.635111+0000 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:05.635111+0000 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:05.635 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.629+0000 7f2aa627b640 1 Processor -- start 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.629+0000 7f2aa627b640 1 -- start start 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient: init 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2aa00a50e0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2aa00a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 0x7f2aa01406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-b con 0x7f2aa013e300 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa00a6530 con 0x7f2aa00a54c0 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa009c150 con 0x7f2aa00a50e0 2024-09-17T15:52:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa009bdc0 con 0x7f2aa013e300 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa627b640 10 monclient(hunting): authenticate will time out at 2458.600670s 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2a9ffff640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7f2aa00a50e0 legacy=0x7f2aa009ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 0x7f2aa01406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2a9f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2aa00a54c0 0x7f2aa00a58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51324/0 (socket says 172.21.15.86:51324) 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2a9f7fe640 1 -- 172.21.15.86:0/2037259707 learned_addr learned my addr 172.21.15.86:0/2037259707 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): get_auth_request con 0x7f2aa013e300 auth_method 0 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient(hunting): handle_auth_done global_id 274190 payload 274 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 1 -- 172.21.15.86:0/2037259707 >> v1:172.21.15.203:6789/0 conn(0x7f2aa00a50e0 legacy=0x7f2aa009ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 1 -- 172.21.15.86:0/2037259707 >> v1:172.21.15.86:6789/0 conn(0x7f2aa00a54c0 legacy=0x7f2aa00a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 1 monclient: found mon.noname-b 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.633+0000 7f2aa4818640 1 -- 172.21.15.86:0/2037259707 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa009c430 con 0x7f2aa013e300 2024-09-17T15:52:05.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 1 -- 172.21.15.86:0/2037259707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 msgr2=0x7f2aa01406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 1 --2- 172.21.15.86:0/2037259707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 0x7f2aa01406f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a980097f0 tx=0x7f2a9801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef5ba1211415d63 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 1 -- 172.21.15.86:0/2037259707 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a980b0070 con 0x7f2aa013e300 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: dump: 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.643 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 5 monclient: authenticate success, global_id 274190 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 1 -- 172.21.15.86:0/2037259707 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a980ab600 con 0x7f2aa013e300 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 4 set_mon_vals no callback set 2024-09-17T15:52:05.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 1 -- 172.21.15.86:0/2037259707 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a980ab9d0 con 0x7f2aa013e300 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9effd640 10 monclient: dump: 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.645 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: shutdown 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- 172.21.15.86:0/2037259707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 msgr2=0x7f2aa01406f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 --2- 172.21.15.86:0/2037259707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 0x7f2aa01406f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2a980097f0 tx=0x7f2a9801c3a0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- 172.21.15.86:0/2037259707 shutdown_connections 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 --2- 172.21.15.86:0/2037259707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa013e300 0x7f2aa01406f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- 172.21.15.86:0/2037259707 >> 172.21.15.86:0/2037259707 conn(0x7f2aa00aaae0 msgr2=0x7f2aa00aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- 172.21.15.86:0/2037259707 shutdown_connections 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- 172.21.15.86:0/2037259707 wait complete. 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: monmap: 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.646 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 Processor -- start 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- start start 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: init 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-c con 0x7f2aa00a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 0x7f2aa0137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-b con 0x7f2aa00a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aa013e300 0x7f2aa013b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): picked mon.noname-a con 0x7f2aa013e300 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa0143e80 con 0x7f2aa013e300 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2aa0143d00 con 0x7f2aa00a54c0 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f2aa0144000 con 0x7f2aa00a50e0 2024-09-17T15:52:05.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa627b640 10 monclient(hunting): authenticate will time out at 2458.607679s 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9f7fe640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 msgr2=0x7f2aa0137e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 0x7f2aa0137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aa013e300 0x7f2aa013b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:38848/0 (socket says 172.21.15.86:38848) 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 1 -- 172.21.15.86:0/277187949 learned_addr learned my addr 172.21.15.86:0/277187949 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 10 monclient(hunting): get_auth_request con 0x7f2aa00a50e0 auth_method 0 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 10 monclient(hunting): get_auth_request con 0x7f2aa013e300 auth_method 0 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.637+0000 7f2aa4818640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa4818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient(hunting): handle_auth_done global_id 274193 payload 824 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 -- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 msgr2=0x7f2aa0137e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 0x7f2aa0137e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 -- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aa013e300 msgr2=0x7f2aa013b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aa013e300 0x7f2aa013b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 monclient: found mon.noname-c 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 -- 172.21.15.86:0/277187949 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa01212b0 con 0x7f2aa00a50e0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 -- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 msgr2=0x7f2aa0137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9ffff640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400b4e0 tx=0x7f2a9400b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94005070 con 0x7f2aa00a50e0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: dump: 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.651 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a94015070 con 0x7f2aa00a50e0 2024-09-17T15:52:05.652 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 5 monclient: authenticate success, global_id 274193 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _renew_subs 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa0121590 con 0x7f2aa00a50e0 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a94005470 con 0x7f2aa00a50e0 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 10 monclient: dump: 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.653 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _renew_subs 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9e7fc640 4 set_mon_vals no callback set 2024-09-17T15:52:05.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2aa0120fd0 con 0x7f2aa00a50e0 2024-09-17T15:52:05.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:05.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2aa009ce10 con 0x7f2aa00a50e0 2024-09-17T15:52:05.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.641+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f2a94005690 con 0x7f2aa00a50e0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.645+0000 7f2a9cff9640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 0x7f2a8c03dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.645+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(873..873 src has 794..873) ==== 4755+0+0 (secure 0 0 0) 0x7f2a9404fdb0 con 0x7f2aa00a50e0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.645+0000 7f2a9f7fe640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 0x7f2a8c03dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.645+0000 7f2a9f7fe640 10 monclient: get_auth_request con 0x7f2a8c03bb10 auth_method 0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.645+0000 7f2a9f7fe640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 0x7f2a8c03dfd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2aa01214c0 tx=0x7f2a90005110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.649+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f2a94010e20 con 0x7f2aa00a50e0 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.649+0000 7f2a9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.649+0000 7f2a9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:05.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.801+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2aa0074be0 con 0x7f2a8c03bb10 2024-09-17T15:52:05.806 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.805+0000 7f2a9cff9640 1 -- 172.21.15.86:0/277187949 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+77418 (secure 0 0 0) 0x7f2aa0074be0 con 0x7f2a8c03bb10 2024-09-17T15:52:05.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:05.808 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 msgr2=0x7f2a8c03dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 0x7f2a8c03dfd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2aa01214c0 tx=0x7f2a90005110 comp rx=0 tx=0).stop 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 10 monclient: shutdown 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 msgr2=0x7f2aa0137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a9400b4e0 tx=0x7f2a9400b9a0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 shutdown_connections 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f2a8c03bb10 0x7f2a8c03dfd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2aa013e300 0x7f2aa013b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2aa00a54c0 0x7f2aa0137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 --2- 172.21.15.86:0/277187949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2aa00a50e0 0x7f2aa0137950 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 >> 172.21.15.86:0/277187949 conn(0x7f2aa00aaae0 msgr2=0x7f2aa00a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:05.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 shutdown_connections 2024-09-17T15:52:05.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.809+0000 7f2aa627b640 1 -- 172.21.15.86:0/277187949 wait complete. 2024-09-17T15:52:05.823 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2024-09-17T15:52:04.980014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":288578164,"num_objects":139,"num_object_clones":89,"num_object_copies":278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":12,"num_read":7556,"num_read_kb":5656265,"num_write":5759,"num_write_kb":2171904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20478,"ondisk_log_size":20478,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":659376,"kb_used_data":351456,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565571664,"statfs":{"total":579820584960,"available":579145383936,"internally_reserved":0,"allocated":359890944,"data_stored":478860576,"data_compressed":7183760,"data_compressed_allocated":5517312,"data_compressed_original":19632346,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18333011,"num_objects":4,"num_object_clones":4,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-2,"num_read":18,"num_read_kb":7150,"num_write":24,"num_write_kb":9342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031449"},"pg_stats":[{"pgid":"3.f","version":"870'2102","reported_seq":6251,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:02.995133+0000","last_change":"2024-09-17T15:52:00.331121+0000","last_active":"2024-09-17T15:52:02.995133+0000","last_peered":"2024-09-17T15:52:02.995133+0000","last_clean":"2024-09-17T15:52:02.995133+0000","last_became_active":"2024-09-17T15:27:52.718109+0000","last_became_peered":"2024-09-17T15:27:52.718109+0000","last_unstale":"2024-09-17T15:52:02.995133+0000","last_undegraded":"2024-09-17T15:52:02.995133+0000","last_fullsized":"2024-09-17T15:52:02.995133+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:52:04.021119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00186652,"stat_sum":{"num_bytes":31160580,"num_objects":15,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":789,"num_read_kb":561343,"num_write":546,"num_write_kb":220862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.e","version":"871'1772","reported_seq":5927,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.033564+0000","last_change":"2024-09-17T15:52:00.334879+0000","last_active":"2024-09-17T15:52:03.033564+0000","last_peered":"2024-09-17T15:52:03.033564+0000","last_clean":"2024-09-17T15:52:03.033564+0000","last_became_active":"2024-09-17T15:27:52.711536+0000","last_became_peered":"2024-09-17T15:27:52.711536+0000","last_unstale":"2024-09-17T15:52:03.033564+0000","last_undegraded":"2024-09-17T15:52:03.033564+0000","last_fullsized":"2024-09-17T15:52:03.033564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1772,"log_dups_size":0,"ondisk_log_size":1772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:33:42.515544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061178259999999998,"stat_sum":{"num_bytes":28585651,"num_objects":13,"num_object_clones":9,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":637,"num_read_kb":455961,"num_write":494,"num_write_kb":181753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.d","version":"861'1601","reported_seq":5757,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111576+0000","last_change":"2024-09-17T15:52:00.330242+0000","last_active":"2024-09-17T15:52:04.111576+0000","last_peered":"2024-09-17T15:52:04.111576+0000","last_clean":"2024-09-17T15:52:04.111576+0000","last_became_active":"2024-09-17T15:27:52.720463+0000","last_became_peered":"2024-09-17T15:27:52.720463+0000","last_unstale":"2024-09-17T15:52:04.111576+0000","last_undegraded":"2024-09-17T15:52:04.111576+0000","last_fullsized":"2024-09-17T15:52:04.111576+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:53:37.056186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002080677,"stat_sum":{"num_bytes":22471461,"num_objects":10,"num_object_clones":6,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":599,"num_read_kb":509709,"num_write":383,"num_write_kb":170847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.c","version":"867'770","reported_seq":4997,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:02.994593+0000","last_change":"2024-09-17T15:52:00.330083+0000","last_active":"2024-09-17T15:52:02.994593+0000","last_peered":"2024-09-17T15:52:02.994593+0000","last_clean":"2024-09-17T15:52:02.994593+0000","last_became_active":"2024-09-17T15:27:52.716001+0000","last_became_peered":"2024-09-17T15:27:52.716001+0000","last_unstale":"2024-09-17T15:52:02.994593+0000","last_undegraded":"2024-09-17T15:52:02.994593+0000","last_fullsized":"2024-09-17T15:52:02.994593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":770,"log_dups_size":0,"ondisk_log_size":770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:48:28.202724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001561259,"stat_sum":{"num_bytes":9697988,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":328,"num_read_kb":259065,"num_write":233,"num_write_kb":87771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.b","version":"867'1582","reported_seq":5709,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.011724+0000","last_change":"2024-09-17T15:51:59.940472+0000","last_active":"2024-09-17T15:52:03.011724+0000","last_peered":"2024-09-17T15:52:03.011724+0000","last_clean":"2024-09-17T15:52:03.011724+0000","last_became_active":"2024-09-17T15:27:52.718753+0000","last_became_peered":"2024-09-17T15:27:52.718753+0000","last_unstale":"2024-09-17T15:52:03.011724+0000","last_undegraded":"2024-09-17T15:52:03.011724+0000","last_fullsized":"2024-09-17T15:52:03.011724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1582,"log_dups_size":0,"ondisk_log_size":1582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:25:02.395126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023677759999999998,"stat_sum":{"num_bytes":31611982,"num_objects":15,"num_object_clones":11,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":533,"num_read_kb":362257,"num_write":427,"num_write_kb":169547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.a","version":"867'806","reported_seq":5030,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111424+0000","last_change":"2024-09-17T15:52:00.330056+0000","last_active":"2024-09-17T15:52:04.111424+0000","last_peered":"2024-09-17T15:52:04.111424+0000","last_clean":"2024-09-17T15:52:04.111424+0000","last_became_active":"2024-09-17T15:27:52.722270+0000","last_became_peered":"2024-09-17T15:27:52.722270+0000","last_unstale":"2024-09-17T15:52:04.111424+0000","last_undegraded":"2024-09-17T15:52:04.111424+0000","last_fullsized":"2024-09-17T15:52:04.111424+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:23:48.238150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019453739999999999,"stat_sum":{"num_bytes":10097448,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":340,"num_read_kb":255309,"num_write":240,"num_write_kb":98392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.9","version":"871'952","reported_seq":5184,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.426107+0000","last_change":"2024-09-17T15:51:59.940464+0000","last_active":"2024-09-17T15:52:04.426107+0000","last_peered":"2024-09-17T15:52:04.426107+0000","last_clean":"2024-09-17T15:52:04.426107+0000","last_became_active":"2024-09-17T15:27:52.717142+0000","last_became_peered":"2024-09-17T15:27:52.717142+0000","last_unstale":"2024-09-17T15:52:04.426107+0000","last_undegraded":"2024-09-17T15:52:04.426107+0000","last_fullsized":"2024-09-17T15:52:04.426107+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":952,"log_dups_size":0,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:58:53.495179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013514,"stat_sum":{"num_bytes":17446986,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":396,"num_read_kb":349777,"num_write":261,"num_write_kb":98826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.8","version":"868'947","reported_seq":5150,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.426225+0000","last_change":"2024-09-17T15:52:00.334290+0000","last_active":"2024-09-17T15:52:04.426225+0000","last_peered":"2024-09-17T15:52:04.426225+0000","last_clean":"2024-09-17T15:52:04.426225+0000","last_became_active":"2024-09-17T15:27:52.719593+0000","last_became_peered":"2024-09-17T15:27:52.719593+0000","last_unstale":"2024-09-17T15:52:04.426225+0000","last_undegraded":"2024-09-17T15:52:04.426225+0000","last_fullsized":"2024-09-17T15:52:04.426225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T18:39:03.065574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0072634830000000003,"stat_sum":{"num_bytes":15875362,"num_objects":8,"num_object_clones":6,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":350,"num_read_kb":251179,"num_write":267,"num_write_kb":102652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"1.0","version":"14'72","reported_seq":1803,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111238+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:52:04.111238+0000","last_peered":"2024-09-17T15:52:04.111238+0000","last_clean":"2024-09-17T15:52:04.111238+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:04.111238+0000","last_undegraded":"2024-09-17T15:52:04.111238+0000","last_fullsized":"2024-09-17T15:52:04.111238+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1715,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:02.994366+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:52:02.994366+0000","last_peered":"2024-09-17T15:52:02.994366+0000","last_clean":"2024-09-17T15:52:02.994366+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:02.994366+0000","last_undegraded":"2024-09-17T15:52:02.994366+0000","last_fullsized":"2024-09-17T15:52:02.994366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"870'2186","reported_seq":6375,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.011472+0000","last_change":"2024-09-17T15:51:59.940446+0000","last_active":"2024-09-17T15:52:03.011472+0000","last_peered":"2024-09-17T15:52:03.011472+0000","last_clean":"2024-09-17T15:52:03.011472+0000","last_became_active":"2024-09-17T15:27:52.720661+0000","last_became_peered":"2024-09-17T15:27:52.720661+0000","last_unstale":"2024-09-17T15:52:03.011472+0000","last_undegraded":"2024-09-17T15:52:03.011472+0000","last_fullsized":"2024-09-17T15:52:03.011472+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":2186,"log_dups_size":0,"ondisk_log_size":2186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:32:37.421959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001415845,"stat_sum":{"num_bytes":32855858,"num_objects":16,"num_object_clones":11,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":870,"num_read_kb":656060,"num_write":630,"num_write_kb":243858,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":1716,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111200+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:52:04.111200+0000","last_peered":"2024-09-17T15:52:04.111200+0000","last_clean":"2024-09-17T15:52:04.111200+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:04.111200+0000","last_undegraded":"2024-09-17T15:52:04.111200+0000","last_fullsized":"2024-09-17T15:52:04.111200+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"867'1739","reported_seq":5835,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.973978+0000","last_change":"2024-09-17T15:52:00.329178+0000","last_active":"2024-09-17T15:52:03.973978+0000","last_peered":"2024-09-17T15:52:03.973978+0000","last_clean":"2024-09-17T15:52:03.973978+0000","last_became_active":"2024-09-17T15:27:52.716024+0000","last_became_peered":"2024-09-17T15:27:52.716024+0000","last_unstale":"2024-09-17T15:52:03.973978+0000","last_undegraded":"2024-09-17T15:52:03.973978+0000","last_fullsized":"2024-09-17T15:52:03.973978+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1739,"log_dups_size":0,"ondisk_log_size":1739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:27:15.354283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00151739,"stat_sum":{"num_bytes":5262091,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":510,"num_read_kb":412895,"num_write":465,"num_write_kb":158637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":1716,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.425856+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:52:04.425856+0000","last_peered":"2024-09-17T15:52:04.425856+0000","last_clean":"2024-09-17T15:52:04.425856+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:04.425856+0000","last_undegraded":"2024-09-17T15:52:04.425856+0000","last_fullsized":"2024-09-17T15:52:04.425856+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"871'950","reported_seq":5114,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.012212+0000","last_change":"2024-09-17T15:51:59.939283+0000","last_active":"2024-09-17T15:52:03.012212+0000","last_peered":"2024-09-17T15:52:03.012212+0000","last_clean":"2024-09-17T15:52:03.012212+0000","last_became_active":"2024-09-17T15:27:52.719585+0000","last_became_peered":"2024-09-17T15:27:52.719585+0000","last_unstale":"2024-09-17T15:52:03.012212+0000","last_undegraded":"2024-09-17T15:52:03.012212+0000","last_fullsized":"2024-09-17T15:52:03.012212+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:25:49.282778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001330143,"stat_sum":{"num_bytes":13802835,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":262,"num_read_kb":145182,"num_write":282,"num_write_kb":107027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.2","version":"17'2","reported_seq":1716,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:02.994337+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:52:02.994337+0000","last_peered":"2024-09-17T15:52:02.994337+0000","last_clean":"2024-09-17T15:52:02.994337+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:02.994337+0000","last_undegraded":"2024-09-17T15:52:02.994337+0000","last_fullsized":"2024-09-17T15:52:02.994337+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"867'751","reported_seq":4969,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.425986+0000","last_change":"2024-09-17T15:51:59.940016+0000","last_active":"2024-09-17T15:52:04.425986+0000","last_peered":"2024-09-17T15:52:04.425986+0000","last_clean":"2024-09-17T15:52:04.425986+0000","last_became_active":"2024-09-17T15:27:52.714241+0000","last_became_peered":"2024-09-17T15:27:52.714241+0000","last_unstale":"2024-09-17T15:52:04.425986+0000","last_undegraded":"2024-09-17T15:52:04.425986+0000","last_fullsized":"2024-09-17T15:52:04.425986+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:47:52.566523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099206500000000009,"stat_sum":{"num_bytes":6953168,"num_objects":5,"num_object_clones":3,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":249,"num_read_kb":188256,"num_write":229,"num_write_kb":83716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"3.5","version":"861'1097","reported_seq":5371,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:02.994653+0000","last_change":"2024-09-17T15:52:00.329945+0000","last_active":"2024-09-17T15:52:02.994653+0000","last_peered":"2024-09-17T15:52:02.994653+0000","last_clean":"2024-09-17T15:52:02.994653+0000","last_became_active":"2024-09-17T15:27:52.719434+0000","last_became_peered":"2024-09-17T15:27:52.719434+0000","last_unstale":"2024-09-17T15:52:02.994653+0000","last_undegraded":"2024-09-17T15:52:02.994653+0000","last_fullsized":"2024-09-17T15:52:02.994653+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:30:58.608326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00155627,"stat_sum":{"num_bytes":13043420,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":534,"num_read_kb":448228,"num_write":349,"num_write_kb":132018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":1716,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.012176+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:52:03.012176+0000","last_peered":"2024-09-17T15:52:03.012176+0000","last_clean":"2024-09-17T15:52:03.012176+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:03.012176+0000","last_undegraded":"2024-09-17T15:52:03.012176+0000","last_fullsized":"2024-09-17T15:52:03.012176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"870'1258","reported_seq":5456,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.012338+0000","last_change":"2024-09-17T15:51:59.939356+0000","last_active":"2024-09-17T15:52:03.012338+0000","last_peered":"2024-09-17T15:52:03.012338+0000","last_clean":"2024-09-17T15:52:03.012338+0000","last_became_active":"2024-09-17T15:27:52.719112+0000","last_became_peered":"2024-09-17T15:27:52.719112+0000","last_unstale":"2024-09-17T15:52:03.012338+0000","last_undegraded":"2024-09-17T15:52:03.012338+0000","last_fullsized":"2024-09-17T15:52:03.012338+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":1258,"log_dups_size":0,"ondisk_log_size":1258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:36:54.685061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001244041,"stat_sum":{"num_bytes":15943763,"num_objects":7,"num_object_clones":4,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":494,"num_read_kb":345178,"num_write":324,"num_write_kb":116280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":1715,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111364+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:52:04.111364+0000","last_peered":"2024-09-17T15:52:04.111364+0000","last_clean":"2024-09-17T15:52:04.111364+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:04.111364+0000","last_undegraded":"2024-09-17T15:52:04.111364+0000","last_fullsized":"2024-09-17T15:52:04.111364+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"870'945","reported_seq":5125,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.426803+0000","last_change":"2024-09-17T15:52:00.328174+0000","last_active":"2024-09-17T15:52:04.426803+0000","last_peered":"2024-09-17T15:52:04.426803+0000","last_clean":"2024-09-17T15:52:04.426803+0000","last_became_active":"2024-09-17T15:27:52.716928+0000","last_became_peered":"2024-09-17T15:27:52.716928+0000","last_unstale":"2024-09-17T15:52:04.426803+0000","last_undegraded":"2024-09-17T15:52:04.426803+0000","last_fullsized":"2024-09-17T15:52:04.426803+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:03:12.488582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012873680000000001,"stat_sum":{"num_bytes":13833529,"num_objects":6,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":289,"num_read_kb":211970,"num_write":250,"num_write_kb":85134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":1716,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.011788+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:52:03.011788+0000","last_peered":"2024-09-17T15:52:03.011788+0000","last_clean":"2024-09-17T15:52:03.011788+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:03.011788+0000","last_undegraded":"2024-09-17T15:52:03.011788+0000","last_fullsized":"2024-09-17T15:52:03.011788+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"871'945","reported_seq":5121,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:03.974153+0000","last_change":"2024-09-17T15:52:00.335138+0000","last_active":"2024-09-17T15:52:03.974153+0000","last_peered":"2024-09-17T15:52:03.974153+0000","last_clean":"2024-09-17T15:52:03.974153+0000","last_became_active":"2024-09-17T15:27:52.716361+0000","last_became_peered":"2024-09-17T15:27:52.716361+0000","last_unstale":"2024-09-17T15:52:03.974153+0000","last_undegraded":"2024-09-17T15:52:03.974153+0000","last_fullsized":"2024-09-17T15:52:03.974153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:51.690434+0000","last_clean_scrub_stamp":"2024-09-17T15:27:51.690434+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:03:26.906586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073756469999999999,"stat_sum":{"num_bytes":19345655,"num_objects":9,"num_object_clones":7,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":310,"num_read_kb":243841,"num_write":276,"num_write_kb":112954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a7","length":"2"},{"start":"1ab","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":1715,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-17T15:52:04.111593+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:52:04.111593+0000","last_peered":"2024-09-17T15:52:04.111593+0000","last_clean":"2024-09-17T15:52:04.111593+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:04.111593+0000","last_undegraded":"2024-09-17T15:52:04.111593+0000","last_fullsized":"2024-09-17T15:52:04.111593+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":287987777,"num_objects":135,"num_object_clones":89,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":12,"num_read":7490,"num_read_kb":5656210,"num_write":5656,"num_write_kb":2170274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355246080,"data_stored":476650356,"data_compressed":7231698,"data_compressed_allocated":5554176,"data_compressed_original":18804954,"omap_allocated":0,"internal_metadata":0},"log_size":20403,"ondisk_log_size":20403,"up":32,"acting":32,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640555,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130568,"kb_used_data":73008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94241272,"statfs":{"total":96636764160,"available":96503062528,"internally_reserved":0,"allocated":74760192,"data_stored":95323041,"data_compressed":1847008,"data_compressed_allocated":1327104,"data_compressed_original":4946029,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640555,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97004,"kb_used_data":39444,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94274836,"statfs":{"total":96636764160,"available":96537432064,"internally_reserved":0,"allocated":40390656,"data_stored":56507576,"data_compressed":982531,"data_compressed_allocated":737280,"data_compressed_original":2424832,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640555,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91676,"kb_used_data":42436,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94280164,"statfs":{"total":96636764160,"available":96542887936,"internally_reserved":0,"allocated":43454464,"data_stored":56841358,"data_compressed":856951,"data_compressed_allocated":471040,"data_compressed_original":2856622,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640555,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112004,"kb_used_data":66924,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94259836,"statfs":{"total":96636764160,"available":96522072064,"internally_reserved":0,"allocated":68530176,"data_stored":90717851,"data_compressed":1319075,"data_compressed_allocated":1060864,"data_compressed_original":3473408,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640555,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128808,"kb_used_data":83732,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94243032,"statfs":{"total":96636764160,"available":96504864768,"internally_reserved":0,"allocated":85741568,"data_stored":113195803,"data_compressed":1437967,"data_compressed_allocated":1212416,"data_compressed_original":3846591,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640554,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99316,"kb_used_data":45912,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94272524,"statfs":{"total":96636764160,"available":96535064576,"internally_reserved":0,"allocated":47013888,"data_stored":66274947,"data_compressed":740228,"data_compressed_allocated":708608,"data_compressed_original":2084864,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45973504,"data_stored":65321970,"data_compressed":709760,"data_compressed_allocated":667648,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86069248,"data_stored":114142523,"data_compressed":1546841,"data_compressed_allocated":1331200,"data_compressed_original":4043199,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67563520,"data_stored":90299097,"data_compressed":1319075,"data_compressed_allocated":1060864,"data_compressed_original":3473408,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":42426368,"data_stored":55893676,"data_compressed":826483,"data_compressed_allocated":430080,"data_compressed_original":2344622,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":39424000,"data_stored":56088822,"data_compressed":982531,"data_compressed_allocated":737280,"data_compressed_original":2424832,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73789440,"data_stored":94904268,"data_compressed":1847008,"data_compressed_allocated":1327104,"data_compressed_original":4946029,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:05.825 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-17T15:52:05.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 10 monclient: get_monmap_and_config 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 10 monclient: monmap: 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:05.955140+0000 2024-09-17T15:52:05.951 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:05.955140+0000 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:05.952 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.954 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 1 Processor -- start 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 1 -- start start 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.949+0000 7fecd444b640 10 monclient: init 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feccc137e10 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feccc1381f0 0x7feccc138640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-a con 0x7feccc1381f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feccc13d340 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feccc09c4a0 con 0x7feccc13d340 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feccc09c780 con 0x7feccc137e10 2024-09-17T15:52:05.955 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7feccc0a62e0 con 0x7feccc1381f0 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd444b640 10 monclient(hunting): authenticate will time out at 2458.920548s 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd21e7640 1 -- >> v1:172.21.15.203:6789/0 conn(0x7feccc137e10 legacy=0x7feccc09d160 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.203:6789/0 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feccc1381f0 0x7feccc138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd29e8640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feccc13d340 0x7feccc13f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51330/0 (socket says 172.21.15.86:51330) 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd29e8640 1 -- 172.21.15.86:0/2009488923 learned_addr learned my addr 172.21.15.86:0/2009488923 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): get_auth_request con 0x7feccc1381f0 auth_method 0 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.956 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient(hunting): handle_auth_done global_id 234591 payload 274 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 1 -- 172.21.15.86:0/2009488923 >> v1:172.21.15.203:6789/0 conn(0x7feccc137e10 legacy=0x7feccc09d160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 1 -- 172.21.15.86:0/2009488923 >> v1:172.21.15.86:6789/0 conn(0x7feccc13d340 legacy=0x7feccc13f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 1 monclient: found mon.noname-a 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.957 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.953+0000 7fecd19e6640 1 -- 172.21.15.86:0/2009488923 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc138b80 con 0x7feccc1381f0 2024-09-17T15:52:05.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd19e6640 1 -- 172.21.15.86:0/2009488923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc1381f0 msgr2=0x7feccc138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd19e6640 1 --2- 172.21.15.86:0/2009488923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc1381f0 0x7feccc138640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0097c0 tx=0x7fecbc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201ab84f740a5810 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 1 -- 172.21.15.86:0/2009488923 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc0b0070 con 0x7feccc1381f0 2024-09-17T15:52:05.962 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: dump: 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.963 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 5 monclient: authenticate success, global_id 234591 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 1 -- 172.21.15.86:0/2009488923 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecbc0ab600 con 0x7feccc1381f0 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 1 -- 172.21.15.86:0/2009488923 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc0ab9d0 con 0x7feccc1381f0 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 4 set_mon_vals no callback set 2024-09-17T15:52:05.964 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd11e5640 10 monclient: dump: 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.965 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: shutdown 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- 172.21.15.86:0/2009488923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc1381f0 msgr2=0x7feccc138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 --2- 172.21.15.86:0/2009488923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc1381f0 0x7feccc138640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0097c0 tx=0x7fecbc01c360 comp rx=0 tx=0).stop 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- 172.21.15.86:0/2009488923 shutdown_connections 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 --2- 172.21.15.86:0/2009488923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc1381f0 0x7feccc138640 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- 172.21.15.86:0/2009488923 >> 172.21.15.86:0/2009488923 conn(0x7feccc0a7070 msgr2=0x7feccc0a7480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- 172.21.15.86:0/2009488923 shutdown_connections 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- 172.21.15.86:0/2009488923 wait complete. 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: build_initial_monmap 2024-09-17T15:52:05.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: monmap: 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.967 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 Processor -- start 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- start start 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: init 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-a con 0x7feccc137e10 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 0x7feccc139320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-b con 0x7feccc1381f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 0x7feccc139860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): picked mon.noname-c con 0x7feccc13d340 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:05.968 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7feccc143a00 con 0x7feccc137e10 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feccc143880 con 0x7feccc1381f0 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7feccc143b80 con 0x7feccc13d340 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd444b640 10 monclient(hunting): authenticate will time out at 2458.927172s 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 0x7feccc139860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd19e6640 1 -- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 msgr2=0x7feccc139320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.203:3300/0 2024-09-17T15:52:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:55550/0 (socket says 172.21.15.86:55550) 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 -- 172.21.15.86:0/1764218085 learned_addr learned my addr 172.21.15.86:0/1764218085 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd19e6640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 0x7feccc139320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): get_auth_request con 0x7feccc137e10 auth_method 0 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): get_auth_request con 0x7feccc13d340 auth_method 0 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient(hunting): handle_auth_done global_id 234594 payload 824 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 -- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 msgr2=0x7feccc139860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 0x7feccc139860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 -- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 msgr2=0x7feccc139320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 0x7feccc139320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient: _finish_hunting 0 2024-09-17T15:52:05.971 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 monclient: found mon.noname-a 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd21e7640 1 -- 172.21.15.86:0/1764218085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc13cdc0 con 0x7feccc137e10 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.957+0000 7fecd29e8640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 0x7feccc139860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd21e7640 1 -- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 msgr2=0x7feccc136bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd21e7640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fecc800b530 tx=0x7fecc800b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8013020 con 0x7feccc137e10 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: dump: 2024-09-17T15:52:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc8005070 con 0x7feccc137e10 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 5 monclient: authenticate success, global_id 234594 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _renew_subs 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc0a9580 con 0x7feccc137e10 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc8005460 con 0x7feccc137e10 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 10 monclient: dump: 2024-09-17T15:52:05.974 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:05.975 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd09e4640 4 set_mon_vals no callback set 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _renew_subs 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc13c800 con 0x7feccc137e10 2024-09-17T15:52:05.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:05.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feccc138640 con 0x7feccc137e10 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fecc8016020 con 0x7feccc137e10 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 0x7fecb003dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(874..874 src has 794..874) ==== 4783+0+0 (secure 0 0 0) 0x7fecc804fb70 con 0x7feccc137e10 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd19e6640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 0x7fecb003dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.961+0000 7fecd19e6640 10 monclient: get_auth_request con 0x7fecb003bb10 auth_method 0 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.965+0000 7fecd19e6640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 0x7fecb003dfd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feccc09cd40 tx=0x7fecbc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.965+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fecc801d020 con 0x7feccc137e10 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.965+0000 7fecc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:05.965+0000 7fecc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:06.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.137+0000 7fecd444b640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-17T15:52:06.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.137+0000 7fecd444b640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-17T15:52:06.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.137+0000 7fecd444b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:06.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.137+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} v 0) -- 0x7feccc13b890 con 0x7feccc137e10 2024-09-17T15:52:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.981+0000 7fecc2ffd640 1 -- 172.21.15.86:0/1764218085 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]=0 pool 'unique_pool_0' removed v875) ==== 175+0+0 (secure 0 0 0) 0x7fecc8018c50 con 0x7feccc137e10 2024-09-17T15:52:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.981+0000 7fecc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-17T15:52:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.981+0000 7fecc2ffd640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' removed 2024-09-17T15:52:06.983 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'unique_pool_0' removed 2024-09-17T15:52:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 msgr2=0x7fecb003dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 0x7fecb003dfd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7feccc09cd40 tx=0x7fecbc008040 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 10 monclient: shutdown 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 msgr2=0x7feccc136bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fecc800b530 tx=0x7fecc800b9f0 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 shutdown_connections 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fecb003bb10 0x7fecb003dfd0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc13d340 0x7feccc139860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feccc1381f0 0x7feccc139320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 --2- 172.21.15.86:0/1764218085 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc137e10 0x7feccc136bd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:06.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 >> 172.21.15.86:0/1764218085 conn(0x7feccc0a7070 msgr2=0x7feccc0a2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 shutdown_connections 2024-09-17T15:52:06.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:06.985+0000 7fecd444b640 1 -- 172.21.15.86:0/1764218085 wait complete. 2024-09-17T15:52:07.006 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-17T15:52:07.020 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-17T15:52:13.697 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-17T15:52:13.698 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T15:52:13.698 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T15:52:13.701 INFO:tasks.ceph.mon.b:Started 2024-09-17T15:52:13.701 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:52:13.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: get_monmap_and_config 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: build_initial_monmap 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: monmap: 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:13.833694+0000 2024-09-17T15:52:13.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:13.833694+0000 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 Processor -- start 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 -- start start 2024-09-17T15:52:13.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: init 2024-09-17T15:52:13.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:13.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-17T15:52:13.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31e40689d0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e4068db0 0x7f31e4069190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-b con 0x7f31e4068db0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31e4139f00 0x7f31e413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-a con 0x7f31e4139f00 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e409d770 con 0x7f31e40689d0 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f31e409ed00 con 0x7f31e4139f00 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f31e409efe0 con 0x7f31e4068db0 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e9519640 10 monclient(hunting): authenticate will time out at 2466.799172s 2024-09-17T15:52:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31e4139f00 0x7f31e413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f31e40689d0 0x7f31e409ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43688/0 (socket says 172.21.15.86:43688) 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e2ffd640 1 -- 172.21.15.86:0/2755730737 learned_addr learned my addr 172.21.15.86:0/2755730737 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): get_auth_request con 0x7f31e4139f00 auth_method 0 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e4068db0 0x7f31e4069190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): get_auth_request con 0x7f31e4068db0 auth_method 0 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:13.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient(hunting): handle_auth_done global_id 234597 payload 274 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 -- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e4068db0 msgr2=0x7f31e4069190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e4068db0 0x7f31e4069190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 -- 172.21.15.86:0/2755730737 >> v1:172.21.15.86:6789/0 conn(0x7f31e40689d0 legacy=0x7f31e409ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:13.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.829+0000 7f31e37fe640 1 -- 172.21.15.86:0/2755730737 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e40696d0 con 0x7f31e4139f00 2024-09-17T15:52:13.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.833+0000 7f31e37fe640 1 -- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 msgr2=0x7f31e413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:13.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.833+0000 7f31e37fe640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e413c2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009d90 tx=0x7f31dc01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a97b265d0262d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:13.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 1 -- 172.21.15.86:0/2755730737 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc0b2070 con 0x7f31e4139f00 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: dump: 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:13.841 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 1 -- 172.21.15.86:0/2755730737 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31dc0a5810 con 0x7f31e4139f00 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 1 -- 172.21.15.86:0/2755730737 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc0a5be0 con 0x7f31e4139f00 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 5 monclient: authenticate success, global_id 234597 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 4 set_mon_vals no callback set 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e1ffb640 10 monclient: dump: 2024-09-17T15:52:13.842 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: shutdown 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 msgr2=0x7f31e413c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e413c2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f31dc009d90 tx=0x7f31dc01c8c0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- 172.21.15.86:0/2755730737 shutdown_connections 2024-09-17T15:52:13.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e413c2f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- 172.21.15.86:0/2755730737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e4068db0 0x7f31e4069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- 172.21.15.86:0/2755730737 >> 172.21.15.86:0/2755730737 conn(0x7f31e40a6440 msgr2=0x7f31e40a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- 172.21.15.86:0/2755730737 shutdown_connections 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- 172.21.15.86:0/2755730737 wait complete. 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: build_initial_monmap 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: monmap: 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:13.844 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 Processor -- start 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- start start 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: init 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 0x7f31e41cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-c con 0x7f31e40689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.845 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-b con 0x7f31e4068db0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): picked mon.noname-a con 0x7f31e4139f00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f31e413f3e0 con 0x7f31e4139f00 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31e413f260 con 0x7f31e4068db0 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f31e413f560 con 0x7f31e40689d0 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e9519640 10 monclient(hunting): authenticate will time out at 2466.806370s 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 0x7f31e41cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54774/0 (socket says 172.21.15.86:54774) 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 0x7f31e41cf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42250/0 (socket says 172.21.15.86:42250) 2024-09-17T15:52:13.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 -- 172.21.15.86:0/1793507180 learned_addr learned my addr 172.21.15.86:0/1793507180 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): get_auth_request con 0x7f31e4139f00 auth_method 0 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): get_auth_request con 0x7f31e40689d0 auth_method 0 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 10 monclient(hunting): get_auth_request con 0x7f31e4068db0 auth_method 0 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient(hunting): handle_auth_done global_id 234600 payload 824 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 msgr2=0x7f31e41cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 0x7f31e41cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 msgr2=0x7f31e4134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 msgr2=0x7f31e4134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 msgr2=0x7f31e4134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e27fc640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:13.848 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:13.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:13.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:13.849 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.837+0000 7f31e37fe640 1 -- 172.21.15.86:0/1793507180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e4122690 con 0x7f31e4139f00 2024-09-17T15:52:13.850 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e37fe640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 msgr2=0x7f31e4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e37fe640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0b0040 tx=0x7f31dc019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc0b2050 con 0x7f31e4139f00 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: dump: 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:13.851 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31dc0be070 con 0x7f31e4139f00 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31dc0add30 con 0x7f31e4139f00 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 10 monclient: dump: 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:13.852 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e0ff9640 4 set_mon_vals no callback set 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 5 monclient: authenticate success, global_id 234600 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _renew_subs 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e41d0860 con 0x7f31e4139f00 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _renew_subs 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e40a15a0 con 0x7f31e4139f00 2024-09-17T15:52:13.853 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e409d460 con 0x7f31e4139f00 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f31dc0b9450 con 0x7f31e4139f00 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 0x7f31d003e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f31dc0ea8d0 con 0x7f31e4139f00 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e2ffd640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 0x7f31d003e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.841+0000 7f31e2ffd640 10 monclient: get_auth_request con 0x7f31d003bbe0 auth_method 0 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.845+0000 7f31e2ffd640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 0x7f31d003e0a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f31d8007ca0 tx=0x7f31d8007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.845+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f31dc0c7020 con 0x7f31e4139f00 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.845+0000 7f31cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:13.854 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:13.845+0000 7f31cb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31e9519640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:52:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31e9519640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31e9519640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f31e4069190 con 0x7f31e4139f00 2024-09-17T15:52:14.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31cb7fe640 1 -- 172.21.15.86:0/1793507180 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f31dc016540 con 0x7f31e4139f00 2024-09-17T15:52:14.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:14.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.081+0000 7f31cb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:14.082 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":222,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 msgr2=0x7f31d003e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 0x7f31d003e0a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f31d8007ca0 tx=0x7f31d8007bf0 comp rx=0 tx=0).stop 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 10 monclient: shutdown 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 msgr2=0x7f31e4135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:14.084 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0b0040 tx=0x7f31dc019660 comp rx=0 tx=0).stop 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 shutdown_connections 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31d003bbe0 0x7f31d003e0a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31e4139f00 0x7f31e4135ee0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31e4068db0 0x7f31e4134fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 --2- 172.21.15.86:0/1793507180 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31e40689d0 0x7f31e41cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 >> 172.21.15.86:0/1793507180 conn(0x7f31e40a6440 msgr2=0x7f31e40a1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 shutdown_connections 2024-09-17T15:52:14.085 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:14.085+0000 7f31e9519640 1 -- 172.21.15.86:0/1793507180 wait complete. 2024-09-17T15:52:17.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: get_monmap_and_config 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: build_initial_monmap 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: monmap: 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:17.233954+0000 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:17.233954+0000 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:17.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 Processor -- start 2024-09-17T15:52:17.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 -- start start 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: init 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-b con 0x7f73080a2d90 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-a con 0x7f730806d0b0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f730813c520 0x7f730813e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-c con 0x7f730813c520 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f730806d9d0 con 0x7f730806d0b0 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f73080a3170 con 0x7f730813c520 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f73080a3430 con 0x7f73080a2d90 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730cebd640 10 monclient(hunting): authenticate will time out at 2470.199329s 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42258/0 (socket says 172.21.15.86:42258) 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:54776/0 (socket says 172.21.15.86:54776) 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 -- 172.21.15.86:0/4033614657 learned_addr learned my addr 172.21.15.86:0/4033614657 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): get_auth_request con 0x7f73080a2d90 auth_method 0 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): get_auth_request con 0x7f730806d0b0 auth_method 0 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7306d9d640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.203:3300/0 conn(0x7f730813c520 0x7f730813e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7306d9d640 10 monclient(hunting): get_auth_request con 0x7f730813c520 auth_method 0 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7306d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7306d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient(hunting): handle_auth_done global_id 274205 payload 274 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 -- 172.21.15.86:0/4033614657 >> v2:172.21.15.203:3300/0 conn(0x7f730813c520 msgr2=0x7f730813e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.203:3300/0 conn(0x7f730813c520 0x7f730813e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 -- 172.21.15.86:0/4033614657 >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 msgr2=0x7f730806d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient: _finish_hunting 0 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 monclient: found mon.noname-b 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f730659c640 1 -- 172.21.15.86:0/4033614657 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730809bf40 con 0x7f73080a2d90 2024-09-17T15:52:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.229+0000 7f7305d9b640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:17.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 1 -- 172.21.15.86:0/4033614657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 msgr2=0x7f730806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 1 --2- 172.21.15.86:0/4033614657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73080a3a90 tx=0x7f72f801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a12e51314b149aa server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 1 -- 172.21.15.86:0/4033614657 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80b0070 con 0x7f73080a2d90 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: dump: 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.244 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: _finish_auth 0 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: _check_auth_tickets 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 1 -- 172.21.15.86:0/4033614657 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72f80ab600 con 0x7f73080a2d90 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 5 monclient: authenticate success, global_id 274205 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:17.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 1 -- 172.21.15.86:0/4033614657 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f80ab9d0 con 0x7f73080a2d90 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 4 set_mon_vals no callback set 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730559a640 10 monclient: dump: 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.246 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: shutdown 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- 172.21.15.86:0/4033614657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 msgr2=0x7f730806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- 172.21.15.86:0/4033614657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73080a3a90 tx=0x7f72f801c470 comp rx=0 tx=0).stop 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- 172.21.15.86:0/4033614657 shutdown_connections 2024-09-17T15:52:17.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.203:3300/0 conn(0x7f730813c520 0x7f730813e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- 172.21.15.86:0/4033614657 >> v2:172.21.15.86:3300/0 conn(0x7f730806d0b0 0x7f730806d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- 172.21.15.86:0/4033614657 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73080a2d90 0x7f730806cb70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- 172.21.15.86:0/4033614657 >> 172.21.15.86:0/4033614657 conn(0x7f73080a3e80 msgr2=0x7f73080a4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- 172.21.15.86:0/4033614657 shutdown_connections 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- 172.21.15.86:0/4033614657 wait complete. 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: build_initial_monmap 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: monmap: 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:17.248 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 Processor -- start 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- start start 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: init 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:17.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f730806d0b0 0x7f73081cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-b con 0x7f730806d0b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73080a2d90 0x7f73080997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-a con 0x7f73080a2d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): picked mon.noname-c con 0x7f730813c520 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f73081417f0 con 0x7f73080a2d90 2024-09-17T15:52:17.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7308141670 con 0x7f730806d0b0 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7308141970 con 0x7f730813c520 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730cebd640 10 monclient(hunting): authenticate will time out at 2470.206826s 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73080a2d90 0x7f73080997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:42266/0 (socket says 172.21.15.86:42266) 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 -- 172.21.15.86:0/2581098554 learned_addr learned my addr 172.21.15.86:0/2581098554 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:17.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): get_auth_request con 0x7f730813c520 auth_method 0 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): get_auth_request con 0x7f73080a2d90 auth_method 0 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7305d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:17.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f730806d0b0 0x7f73081cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 10 monclient(hunting): get_auth_request con 0x7f730806d0b0 auth_method 0 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f730659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient(hunting): handle_auth_done global_id 274208 payload 824 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 -- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f730806d0b0 msgr2=0x7f73081cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f730806d0b0 0x7f73081cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 -- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73080a2d90 msgr2=0x7f73080997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73080a2d90 0x7f73080997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:17.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 monclient: found mon.noname-c 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.237+0000 7f7306d9d640 1 -- 172.21.15.86:0/2581098554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f730809a220 con 0x7f730813c520 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f7306d9d640 1 -- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 msgr2=0x7f7308099ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f7306d9d640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7300007e30 tx=0x7f730000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300013790 con 0x7f730813c520 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: dump: 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7300019070 con 0x7f730813c520 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300013b70 con 0x7f730813c520 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 5 monclient: authenticate success, global_id 274208 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _renew_subs 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7308074b00 con 0x7f730813c520 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:17.255 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f7304d99640 4 set_mon_vals no callback set 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 10 monclient: dump: 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _renew_subs 2024-09-17T15:52:17.256 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7308092c60 con 0x7f730813c520 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f730806d490 con 0x7f730813c520 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7300013d10 con 0x7f730813c520 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 0x7f72e803e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:17.257 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f73000505e0 con 0x7f730813c520 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730659c640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 0x7f72e803e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.241+0000 7f730659c640 10 monclient: get_auth_request con 0x7f72e803bb90 auth_method 0 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.245+0000 7f730659c640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 0x7f72e803e050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7308070500 tx=0x7f72f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.245+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7300012cf0 con 0x7f730813c520 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.245+0000 7f72f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:17.258 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.245+0000 7f72f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:17.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.481+0000 7f730cebd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:52:17.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:17.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:17.484 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f730813ea20 con 0x7f730813c520 2024-09-17T15:52:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f72f6ffd640 1 -- 172.21.15.86:0/2581098554 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f7300014420 con 0x7f730813c520 2024-09-17T15:52:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f72f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f72f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:17.485 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":222,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 msgr2=0x7f72e803e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 0x7f72e803e050 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7308070500 tx=0x7f72f8008040 comp rx=0 tx=0).stop 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 10 monclient: shutdown 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 msgr2=0x7f7308099ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7300007e30 tx=0x7f730000e4b0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 shutdown_connections 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f72e803bb90 0x7f72e803e050 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f730813c520 0x7f7308099ce0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73080a2d90 0x7f73080997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 --2- 172.21.15.86:0/2581098554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f730806d0b0 0x7f73081cfca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 >> 172.21.15.86:0/2581098554 conn(0x7f73080a3e80 msgr2=0x7f73080a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 shutdown_connections 2024-09-17T15:52:17.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:17.485+0000 7f730cebd640 1 -- 172.21.15.86:0/2581098554 wait complete. 2024-09-17T15:52:20.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: get_monmap_and_config 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: build_initial_monmap 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: monmap: 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:20.636994+0000 2024-09-17T15:52:20.633 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:20.636994+0000 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:20.634 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 Processor -- start 2024-09-17T15:52:20.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 -- start start 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: init 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe64009ba70 0x7fe640068c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-a con 0x7fe64009ba70 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 0x7fe64009be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-b con 0x7fe64009eea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe64009c390 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64009f2b0 con 0x7fe64009c390 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fe64009d7e0 con 0x7fe64009ba70 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:20.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe64009e4c0 con 0x7fe64009eea0 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe6477b1640 10 monclient(hunting): authenticate will time out at 2473.602511s 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe64009ba70 0x7fe640068c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 0x7fe64009be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe64009ba70 0x7fe640068c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34522/0 (socket says 172.21.15.86:34522) 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 0x7fe64009be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50562/0 (socket says 172.21.15.86:50562) 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 -- 172.21.15.86:0/886300185 learned_addr learned my addr 172.21.15.86:0/886300185 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): get_auth_request con 0x7fe64009ba70 auth_method 0 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:20.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): get_auth_request con 0x7fe64009eea0 auth_method 0 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient(hunting): handle_auth_done global_id 234609 payload 274 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 -- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 msgr2=0x7fe64009be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 --2- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 0x7fe64009be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 -- 172.21.15.86:0/886300185 >> v1:172.21.15.203:6789/0 conn(0x7fe64009c390 legacy=0x7fe64013c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:20.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:20.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 monclient: found mon.noname-a 2024-09-17T15:52:20.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:20.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.633+0000 7fe64554d640 1 -- 172.21.15.86:0/886300185 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64009dc20 con 0x7fe64009ba70 2024-09-17T15:52:20.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe64554d640 1 -- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009ba70 msgr2=0x7fe640068c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:20.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe64554d640 1 --2- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009ba70 0x7fe640068c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe63c0098a0 tx=0x7fe63c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5795d8be7f3b8301 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:20.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 1 -- 172.21.15.86:0/886300185 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0b0070 con 0x7fe64009ba70 2024-09-17T15:52:20.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: dump: 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:20.644 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: _finish_auth 0 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe6477b1640 5 monclient: authenticate success, global_id 234609 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 1 -- 172.21.15.86:0/886300185 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c0ab600 con 0x7fe64009ba70 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe6477b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe6477b1640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 1 -- 172.21.15.86:0/886300185 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0ab9d0 con 0x7fe64009ba70 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe6477b1640 4 set_mon_vals no callback set 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:20.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.637+0000 7fe637fff640 10 monclient: dump: 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: shutdown 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009ba70 msgr2=0x7fe640068c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:20.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009ba70 0x7fe640068c30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe63c0098a0 tx=0x7fe63c01c400 comp rx=0 tx=0).stop 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- 172.21.15.86:0/886300185 shutdown_connections 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009eea0 0x7fe64009be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- 172.21.15.86:0/886300185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009ba70 0x7fe640068c30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- 172.21.15.86:0/886300185 >> 172.21.15.86:0/886300185 conn(0x7fe6400a6440 msgr2=0x7fe6400a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- 172.21.15.86:0/886300185 shutdown_connections 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- 172.21.15.86:0/886300185 wait complete. 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: build_initial_monmap 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: monmap: 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:20.647 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 Processor -- start 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- start start 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: init 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe64009ba70 0x7fe640071240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-b con 0x7fe64009ba70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009c390 0x7fe640071780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-c con 0x7fe64009c390 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): picked mon.noname-a con 0x7fe64009eea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe6401473a0 con 0x7fe64009eea0 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe640147220 con 0x7fe64009ba70 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe640147520 con 0x7fe64009c390 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe6477b1640 10 monclient(hunting): authenticate will time out at 2473.609831s 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009c390 0x7fe640071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.649 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34530/0 (socket says 172.21.15.86:34530) 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 -- 172.21.15.86:0/676050672 learned_addr learned my addr 172.21.15.86:0/676050672 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): get_auth_request con 0x7fe64009eea0 auth_method 0 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): get_auth_request con 0x7fe64009c390 auth_method 0 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:20.650 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe644d4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient(hunting): handle_auth_done global_id 234612 payload 824 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe64554d640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe64009ba70 0x7fe640071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 -- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009c390 msgr2=0x7fe640071780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009c390 0x7fe640071780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 -- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe64009ba70 msgr2=0x7fe640071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe64009ba70 0x7fe640071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient: _finish_hunting 0 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 monclient: found mon.noname-a 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.651 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.641+0000 7fe645d4e640 1 -- 172.21.15.86:0/676050672 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64013e3d0 con 0x7fe64009eea0 2024-09-17T15:52:20.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe645d4e640 1 -- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 msgr2=0x7fe6401d48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe645d4e640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe63800c780 tx=0x7fe63800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63800ac60 con 0x7fe64009eea0 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: dump: 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:20.654 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63800ae00 con 0x7fe64009eea0 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638004bb0 con 0x7fe64009eea0 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 5 monclient: authenticate success, global_id 234612 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _renew_subs 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64009d460 con 0x7fe64009eea0 2024-09-17T15:52:20.655 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6377fe640 4 set_mon_vals no callback set 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 10 monclient: dump: 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:20.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _renew_subs 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64013db30 con 0x7fe64009eea0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe638005030 con 0x7fe64009eea0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe640068c30 con 0x7fe64009eea0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 0x7fe62403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fe63804ffd0 con 0x7fe64009eea0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe64554d640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 0x7fe62403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.645+0000 7fe64554d640 10 monclient: get_auth_request con 0x7fe62403bbb0 auth_method 0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.649+0000 7fe64554d640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 0x7fe62403e070 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe640074d60 tx=0x7fe63c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.649+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe638020070 con 0x7fe64009eea0 2024-09-17T15:52:20.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.649+0000 7fe635ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:20.658 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.649+0000 7fe635ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:20.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe6477b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:52:20.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe6477b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:20.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe6477b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:20.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe640071d80 con 0x7fe64009eea0 2024-09-17T15:52:20.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe635ffb640 1 -- 172.21.15.86:0/676050672 <== mon.0 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe638004e10 con 0x7fe64009eea0 2024-09-17T15:52:20.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe635ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:20.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.885+0000 7fe635ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:20.886 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":224,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:52:20.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 msgr2=0x7fe62403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:20.888 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 0x7fe62403e070 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe640074d60 tx=0x7fe63c008040 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 10 monclient: shutdown 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 msgr2=0x7fe6401d48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe63800c780 tx=0x7fe63800cc40 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 shutdown_connections 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe62403bbb0 0x7fe62403e070 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe64009eea0 0x7fe6401d48f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe64009c390 0x7fe640071780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 --2- 172.21.15.86:0/676050672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe64009ba70 0x7fe640071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 >> 172.21.15.86:0/676050672 conn(0x7fe6400a6440 msgr2=0x7fe6400a1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 shutdown_connections 2024-09-17T15:52:20.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:20.889+0000 7fe6477b1640 1 -- 172.21.15.86:0/676050672 wait complete. 2024-09-17T15:52:20.913 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:52:20.913 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-17T15:52:20.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-17T15:52:21.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 10 monclient: get_monmap_and_config 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 10 monclient: monmap: 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:21.034611+0000 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:21.034611+0000 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:21.031 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:21.032 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:21.032 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:21.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 1 Processor -- start 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 1 -- start start 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 10 monclient: init 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.029+0000 7fc311d2b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc30c0a0c70 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 0x7fc30c13af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c0a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc30c13b450 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30c0a2130 con 0x7fc30c0a0c70 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc30c0a26f0 con 0x7fc30c13b450 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c0a2410 con 0x7fc30c0a1760 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc311d2b640 10 monclient(hunting): authenticate will time out at 2474.000123s 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30b7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc30c0a0c70 0x7fc30c0a1050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56540/0 (socket says 172.21.15.86:56540) 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30b7fe640 1 -- 172.21.15.86:0/750500416 learned_addr learned my addr 172.21.15.86:0/750500416 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 1 --2- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 0x7fc30c13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): get_auth_request con 0x7fc30c0a1760 auth_method 0 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient(hunting): handle_auth_done global_id 274217 payload 274 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 1 -- 172.21.15.86:0/750500416 >> v1:172.21.15.203:6789/0 conn(0x7fc30c13b450 legacy=0x7fc30c13d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 1 -- 172.21.15.86:0/750500416 >> v1:172.21.15.86:6789/0 conn(0x7fc30c0a0c70 legacy=0x7fc30c0a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 1 monclient: found mon.noname-b 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.033+0000 7fc30affd640 1 -- 172.21.15.86:0/750500416 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c09f9f0 con 0x7fc30c0a1760 2024-09-17T15:52:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30affd640 1 -- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 msgr2=0x7fc30c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30affd640 1 --2- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 0x7fc30c13af10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009dc0 tx=0x7fc2fc01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9893262ae713717a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 1 -- 172.21.15.86:0/750500416 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b2070 con 0x7fc30c0a1760 2024-09-17T15:52:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: dump: 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 1 -- 172.21.15.86:0/750500416 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0a5930 con 0x7fc30c0a1760 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 1 -- 172.21.15.86:0/750500416 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0a5d00 con 0x7fc30c0a1760 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 5 monclient: authenticate success, global_id 274217 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 4 set_mon_vals no callback set 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30a7fc640 10 monclient: dump: 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: shutdown 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 msgr2=0x7fc30c13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 --2- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 0x7fc30c13af10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009dc0 tx=0x7fc2fc01c850 comp rx=0 tx=0).stop 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- 172.21.15.86:0/750500416 shutdown_connections 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 --2- 172.21.15.86:0/750500416 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a1760 0x7fc30c13af10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- 172.21.15.86:0/750500416 >> 172.21.15.86:0/750500416 conn(0x7fc30c0a2f40 msgr2=0x7fc30c0a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- 172.21.15.86:0/750500416 shutdown_connections 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- 172.21.15.86:0/750500416 wait complete. 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: monmap: 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:21.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 Processor -- start 2024-09-17T15:52:21.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- start start 2024-09-17T15:52:21.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: init 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 0x7fc30c11dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-c con 0x7fc30c0a0c70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc30c0a1760 0x7fc30c1ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-b con 0x7fc30c0a1760 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c13b450 0x7fc30c11c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): picked mon.noname-a con 0x7fc30c13b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:21.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c141900 con 0x7fc30c13b450 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c141780 con 0x7fc30c0a1760 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fc30c141a80 con 0x7fc30c0a0c70 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc311d2b640 10 monclient(hunting): authenticate will time out at 2474.007278s 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 0x7fc30c11dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 0x7fc30c11dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50592/0 (socket says 172.21.15.86:50592) 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 -- 172.21.15.86:0/3973408340 learned_addr learned my addr 172.21.15.86:0/3973408340 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): get_auth_request con 0x7fc30c0a0c70 auth_method 0 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c13b450 0x7fc30c11c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): get_auth_request con 0x7fc30c13b450 auth_method 0 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient(hunting): handle_auth_done global_id 274220 payload 824 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30affd640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc30c0a1760 0x7fc30c1ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc30c0a1760 msgr2=0x7fc30c1ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc30c0a1760 0x7fc30c1ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c13b450 msgr2=0x7fc30c11c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c13b450 0x7fc30c11c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 monclient: found mon.noname-c 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.037+0000 7fc30b7fe640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30c11f550 con 0x7fc30c0a0c70 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30b7fe640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 msgr2=0x7fc30c11dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30b7fe640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 0x7fc30c11dd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400c470 tx=0x7fc2f400c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f4017070 con 0x7fc30c0a0c70 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: dump: 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f40070d0 con 0x7fc30c0a0c70 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f40074c0 con 0x7fc30c0a0c70 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 10 monclient: dump: 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.053 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc309ffb640 4 set_mon_vals no callback set 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 5 monclient: authenticate success, global_id 274220 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 monclient: _renew_subs 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30c09d460 con 0x7fc30c0a0c70 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 monclient: _renew_subs 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30c11f270 con 0x7fc30c0a0c70 2024-09-17T15:52:21.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 0x7fc30c02d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c0a1050 con 0x7fc30c1ce790 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 0x7fc30c02d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient: get_auth_request con 0x7fc30c1ce790 auth_method 0 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fc2f401a070 con 0x7fc30c0a0c70 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.041+0000 7fc30affd640 10 monclient(hunting): handle_auth_done global_id 234618 payload 824 2024-09-17T15:52:21.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30affd640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc30c13a610 con 0x7fc30c1ce790 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2ec03bb60 0x7fc2ec03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fc2f404f400 con 0x7fc30c0a0c70 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30bfff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2ec03bb60 0x7fc2ec03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30bfff640 10 monclient: get_auth_request con 0x7fc2ec03bb60 auth_method 0 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30affd640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 msgr2=0x7fc30c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30affd640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 0x7fc30c02d450 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc30c06d160 tx=0x7fc2fc0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0bc070 con 0x7fc30c1ce790 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 10 monclient: dump: 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.056 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc2fc0ad640 con 0x7fc30c1ce790 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 msgr2=0x7fc30c02d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 0x7fc30c02d450 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc30c06d160 tx=0x7fc2fc0195f0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 msgr2=0x7fc30c02d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc303fff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c1ce790 0x7fc30c02d450 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.045+0000 7fc30bfff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2ec03bb60 0x7fc2ec03e020 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc304004ef0 tx=0x7fc304007d10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 0x7fc30c02dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fc30c11edb0 con 0x7fc30c02d990 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 0x7fc30c02dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient: get_auth_request con 0x7fc30c02d990 auth_method 0 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 10 monclient(hunting): handle_auth_done global_id 234621 payload 824 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 1 -- 172.21.15.86:0/3973408340 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc30c11d5c0 con 0x7fc30c02d990 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 msgr2=0x7fc30c02dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc30affd640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 0x7fc30c02dd70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009dc0 tx=0x7fc2fc0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b2040 con 0x7fc30c02d990 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 10 monclient: dump: 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.062 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 <== mon.0 v2:172.21.15.86:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7fc2fc004540 con 0x7fc30c02d990 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 msgr2=0x7fc30c02dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 0x7fc30c02dd70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009dc0 tx=0x7fc2fc0a7000 comp rx=0 tx=0).stop 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 msgr2=0x7fc30c02dd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc303fff640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc30c02d990 0x7fc30c02dd70 secure :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc009dc0 tx=0x7fc2fc0a7000 comp rx=0 tx=0).stop 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:21.063 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "leader", 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 224, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 270248, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:21.064 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.065 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.066 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:21.067 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.068 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.069 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.070 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.071 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.072 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.073 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:52:21.074 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.075 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 4 2024-09-17T15:52:21.076 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr": [ 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:52:21.077 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2ec03bb60 msgr2=0x7fc2ec03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fc2ec03bb60 0x7fc2ec03e020 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fc304004ef0 tx=0x7fc304007d10 comp rx=0 tx=0).stop 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 10 monclient: shutdown 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 msgr2=0x7fc30c11dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.053+0000 7fc311d2b640 1 --2- 172.21.15.86:0/3973408340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc30c0a0c70 0x7fc30c11dd50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc2f400c470 tx=0x7fc2f400c930 comp rx=0 tx=0).stop 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc30bfff640 1 -- 172.21.15.86:0/3973408340 reap_dead start 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc30bfff640 1 -- 172.21.15.86:0/3973408340 reap_dead start 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 shutdown_connections 2024-09-17T15:52:21.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 >> 172.21.15.86:0/3973408340 conn(0x7fc30c0a2f40 msgr2=0x7fc30c13ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:21.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 shutdown_connections 2024-09-17T15:52:21.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.057+0000 7fc311d2b640 1 -- 172.21.15.86:0/3973408340 wait complete. 2024-09-17T15:52:21.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:52:21.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-17T15:52:21.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: get_monmap_and_config 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: monmap: 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:21.196800+0000 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:21.196800+0000 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.193 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:21.194 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 Processor -- start 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 -- start start 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: init 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a200a4f70 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a200a5350 0x7f9a200a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-a con 0x7f9a200a5350 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 0x7f9a2013f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.196 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2013d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a200a68d0 con 0x7f9a200a4f70 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f9a200a64a0 con 0x7f9a200a5350 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a2009c230 con 0x7f9a2013d180 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a27623640 10 monclient(hunting): authenticate will time out at 2474.163080s 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a200a5350 0x7f9a200a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 0x7f9a2013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a200a5350 0x7f9a200a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34554/0 (socket says 172.21.15.86:34554) 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 -- 172.21.15.86:0/2728161631 learned_addr learned my addr 172.21.15.86:0/2728161631 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 0x7f9a2013f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50602/0 (socket says 172.21.15.86:50602) 2024-09-17T15:52:21.197 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request con 0x7f9a200a5350 auth_method 0 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): get_auth_request con 0x7f9a2013d180 auth_method 0 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a25bc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_done global_id 234624 payload 274 2024-09-17T15:52:21.198 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 -- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 msgr2=0x7f9a2013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 0x7f9a2013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 -- 172.21.15.86:0/2728161631 >> v1:172.21.15.203:6789/0 conn(0x7f9a200a4f70 legacy=0x7f9a2009cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 monclient: found mon.noname-a 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:21.199 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.193+0000 7f9a24bbe640 1 -- 172.21.15.86:0/2728161631 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2009c510 con 0x7f9a200a5350 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a24bbe640 1 -- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a5350 msgr2=0x7f9a200a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a5350 0x7f9a200a5750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009870 tx=0x7f9a1001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82cbf5ec2e00aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 1 -- 172.21.15.86:0/2728161631 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a100b1070 con 0x7f9a200a5350 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: dump: 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.200 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 1 -- 172.21.15.86:0/2728161631 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a100ac620 con 0x7f9a200a5350 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 1 -- 172.21.15.86:0/2728161631 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a100ac9f0 con 0x7f9a200a5350 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a27623640 5 monclient: authenticate success, global_id 234624 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a27623640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a27623640 4 set_mon_vals no callback set 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.197+0000 7f9a17fff640 10 monclient: dump: 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.202 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: shutdown 2024-09-17T15:52:21.207 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a5350 msgr2=0x7f9a200a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a5350 0x7f9a200a5750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009870 tx=0x7f9a1001c420 comp rx=0 tx=0).stop 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- 172.21.15.86:0/2728161631 shutdown_connections 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a2013d180 0x7f9a2013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- 172.21.15.86:0/2728161631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a5350 0x7f9a200a5750 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- 172.21.15.86:0/2728161631 >> 172.21.15.86:0/2728161631 conn(0x7f9a200a7240 msgr2=0x7f9a200a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- 172.21.15.86:0/2728161631 shutdown_connections 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- 172.21.15.86:0/2728161631 wait complete. 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: monmap: 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.208 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 Processor -- start 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- start start 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: init 2024-09-17T15:52:21.209 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 0x7f9a201350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-a con 0x7f9a200a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a5350 0x7f9a20135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-c con 0x7f9a200a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a2013d180 0x7f9a201cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2013d180 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f9a20142740 con 0x7f9a200a4f70 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a201425c0 con 0x7f9a2013d180 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.210 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a201428c0 con 0x7f9a200a5350 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a27623640 10 monclient(hunting): authenticate will time out at 2474.170508s 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 0x7f9a201350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a5350 0x7f9a20135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 0x7f9a201350c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34568/0 (socket says 172.21.15.86:34568) 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 -- 172.21.15.86:0/4037749877 learned_addr learned my addr 172.21.15.86:0/4037749877 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): get_auth_request con 0x7f9a200a4f70 auth_method 0 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request con 0x7f9a200a5350 auth_method 0 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.211 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient(hunting): handle_auth_done global_id 234627 payload 824 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a5350 msgr2=0x7f9a20135600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a5350 0x7f9a20135600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a25bc0640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a2013d180 0x7f9a201cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a2013d180 msgr2=0x7f9a201cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a2013d180 0x7f9a201cfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 monclient: found mon.noname-a 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a253bf640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a20135b40 con 0x7f9a200a4f70 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.201+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a5350 0x7f9a20135600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a253bf640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 msgr2=0x7f9a201350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a253bf640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 0x7f9a201350c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00c450 tx=0x7f9a1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c012a40 con 0x7f9a200a4f70 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: dump: 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.213 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a1c017070 con 0x7f9a200a4f70 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.214 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c012e40 con 0x7f9a200a4f70 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 5 monclient: authenticate success, global_id 234627 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 monclient: _renew_subs 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2013b040 con 0x7f9a200a4f70 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a177fe640 4 set_mon_vals no callback set 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 10 monclient: dump: 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.215 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 monclient: _renew_subs 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a200a1220 con 0x7f9a200a4f70 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 0x7f9a201d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a201d0af0 con 0x7f9a201d01d0 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f9a1c01c460 con 0x7f9a200a4f70 2024-09-17T15:52:21.216 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 0x7f9a201d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient: get_auth_request con 0x7f9a201d01d0 auth_method 0 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a0403bbb0 0x7f9a0403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f9a1c04fb80 con 0x7f9a200a4f70 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_done global_id 274229 payload 824 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a24bbe640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a200a5750 con 0x7f9a201d01d0 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a25bc0640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a0403bbb0 0x7f9a0403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.217 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.205+0000 7f9a25bc0640 10 monclient: get_auth_request con 0x7f9a0403bbb0 auth_method 0 2024-09-17T15:52:21.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a24bbe640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 msgr2=0x7f9a201d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 0x7f9a201d05b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a100ae000 tx=0x7f9a100191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a100b1040 con 0x7f9a201d01d0 2024-09-17T15:52:21.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.220 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 10 monclient: dump: 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a100b8660 con 0x7f9a201d01d0 2024-09-17T15:52:21.221 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 msgr2=0x7f9a201d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 0x7f9a201d05b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a100ae000 tx=0x7f9a100191c0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 msgr2=0x7f9a201d05b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a15ffb640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a201d01d0 0x7f9a201d05b0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9a100ae000 tx=0x7f9a100191c0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.209+0000 7f9a25bc0640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a0403bbb0 0x7f9a0403e070 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9a2013a9b0 tx=0x7f9a1800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a27623640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a27623640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 0x7f9a200a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a27623640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f9a20139840 con 0x7f9a200a1760 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 0x7f9a200a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient: get_auth_request con 0x7f9a200a1760 auth_method 0 2024-09-17T15:52:21.222 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.213+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a24bbe640 10 monclient(hunting): handle_auth_done global_id 274232 payload 824 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a24bbe640 1 -- 172.21.15.86:0/4037749877 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a20069920 con 0x7f9a200a1760 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a24bbe640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 msgr2=0x7f9a200a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a24bbe640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 0x7f9a200a1b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009870 tx=0x7f9a10008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a100c6070 con 0x7f9a200a1760 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 10 monclient: dump: 2024-09-17T15:52:21.223 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 <== mon.2 v2:172.21.15.86:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f9a100b6d60 con 0x7f9a200a1760 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 msgr2=0x7f9a200a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 0x7f9a200a1b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009870 tx=0x7f9a10008040 comp rx=0 tx=0).stop 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 msgr2=0x7f9a200a1b40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a15ffb640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a200a1760 0x7f9a200a1b40 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9a10009870 tx=0x7f9a10008040 comp rx=0 tx=0).stop 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:52:21.225 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 224, 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 60538, 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.226 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 1, 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.227 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.228 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:52:21.229 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:21.230 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:21.231 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.232 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.233 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.235 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:21.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.237 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "osd": [ 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:21.238 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 2 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:52:21.239 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a0403bbb0 msgr2=0x7f9a0403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f9a0403bbb0 0x7f9a0403e070 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f9a2013a9b0 tx=0x7f9a1800b040 comp rx=0 tx=0).stop 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 10 monclient: shutdown 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 msgr2=0x7f9a201350c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 --2- 172.21.15.86:0/4037749877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a200a4f70 0x7f9a201350c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00c450 tx=0x7f9a1c00c910 comp rx=0 tx=0).stop 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a25bc0640 1 -- 172.21.15.86:0/4037749877 reap_dead start 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a25bc0640 1 -- 172.21.15.86:0/4037749877 reap_dead start 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 shutdown_connections 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 >> 172.21.15.86:0/4037749877 conn(0x7f9a200a7240 msgr2=0x7f9a2013ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 shutdown_connections 2024-09-17T15:52:21.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.217+0000 7f9a27623640 1 -- 172.21.15.86:0/4037749877 wait complete. 2024-09-17T15:52:21.242 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:52:21.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-17T15:52:21.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-17T15:52:21.352 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: get_monmap_and_config 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: monmap: 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:21.356640+0000 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:21.356640+0000 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:21.354 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 Processor -- start 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 -- start start 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: init 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:52:21.356 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-b con 0x7f31c00a01a0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 0x7f31c00a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-a con 0x7f31c00a0580 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31c00a11b0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31c0074e00 con 0x7f31c00a11b0 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f31c009e350 con 0x7f31c00a0580 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f31c009e630 con 0x7f31c00a01a0 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c7e72640 10 monclient(hunting): authenticate will time out at 2474.322348s 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50638/0 (socket says 172.21.15.86:50638) 2024-09-17T15:52:21.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 -- 172.21.15.86:0/897395096 learned_addr learned my addr 172.21.15.86:0/897395096 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request con 0x7f31c00a01a0 auth_method 0 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c540d640 1 --2- 172.21.15.86:0/897395096 >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 0x7f31c00a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_done global_id 274235 payload 274 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 -- 172.21.15.86:0/897395096 >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 msgr2=0x7f31c00a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/897395096 >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 0x7f31c00a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 -- 172.21.15.86:0/897395096 >> v1:172.21.15.86:6789/0 conn(0x7f31c00a11b0 legacy=0x7f31c013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:21.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 monclient: found mon.noname-b 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c5c0e640 1 -- 172.21.15.86:0/897395096 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c009e910 con 0x7f31c00a01a0 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c540d640 10 monclient: get_auth_request con 0x7f31c00a0580 auth_method 0 2024-09-17T15:52:21.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.353+0000 7f31c540d640 1 --2- 172.21.15.86:0/897395096 >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 0x7f31c00a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c5c0e640 1 -- 172.21.15.86:0/897395096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 msgr2=0x7f31c009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/897395096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006fd0 tx=0x7f31b001e8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=227ef9b1f27aa87d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 1 -- 172.21.15.86:0/897395096 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00ad070 con 0x7f31c00a01a0 2024-09-17T15:52:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.361 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: dump: 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.364 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 5 monclient: authenticate success, global_id 274235 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 1 -- 172.21.15.86:0/897395096 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31b0004230 con 0x7f31c00a01a0 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 1 -- 172.21.15.86:0/897395096 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00a7740 con 0x7f31c00a01a0 2024-09-17T15:52:21.365 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 4 set_mon_vals no callback set 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c4c0c640 10 monclient: dump: 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.366 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 10 monclient: shutdown 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 -- 172.21.15.86:0/897395096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 msgr2=0x7f31c009f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 --2- 172.21.15.86:0/897395096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31b0006fd0 tx=0x7f31b001e8b0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 -- 172.21.15.86:0/897395096 shutdown_connections 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 --2- 172.21.15.86:0/897395096 >> v2:172.21.15.86:3300/0 conn(0x7f31c00a0580 0x7f31c00a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 --2- 172.21.15.86:0/897395096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a01a0 0x7f31c009f010 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 -- 172.21.15.86:0/897395096 >> 172.21.15.86:0/897395096 conn(0x7f31c00a2f40 msgr2=0x7f31c00a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 -- 172.21.15.86:0/897395096 shutdown_connections 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.357+0000 7f31c7e72640 1 -- 172.21.15.86:0/897395096 wait complete. 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient: build_initial_monmap 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient: monmap: 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:21.367 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 Processor -- start 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 -- start start 2024-09-17T15:52:21.368 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient: init 2024-09-17T15:52:21.372 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31c00a01a0 0x7f31c0137180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-a con 0x7f31c00a01a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 0x7f31c01376c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-c con 0x7f31c00a0580 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c00a11b0 0x7f31c0122210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): picked mon.noname-b con 0x7f31c00a11b0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f31c013e5e0 con 0x7f31c00a01a0 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31c013e460 con 0x7f31c00a11b0 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.373 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f31c013e760 con 0x7f31c00a0580 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c7e72640 10 monclient(hunting): authenticate will time out at 2474.329913s 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31c00a01a0 0x7f31c0137180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 0x7f31c01376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31c00a01a0 0x7f31c0137180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34578/0 (socket says 172.21.15.86:34578) 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 learned_addr learned my addr 172.21.15.86:0/2968910774 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:21.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request con 0x7f31c00a01a0 auth_method 0 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): get_auth_request con 0x7f31c00a0580 auth_method 0 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient(hunting): handle_auth_done global_id 274238 payload 824 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c640f640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c00a11b0 0x7f31c0122210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c00a11b0 msgr2=0x7f31c0122210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c00a11b0 0x7f31c0122210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31c00a01a0 msgr2=0x7f31c0137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31c00a01a0 0x7f31c0137180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 monclient: found mon.noname-c 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.361+0000 7f31c540d640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31c0138c50 con 0x7f31c00a0580 2024-09-17T15:52:21.376 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c540d640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 msgr2=0x7f31c01376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c540d640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 0x7f31c01376c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c7c0 tx=0x7f31bc00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc015070 con 0x7f31c00a0580 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: dump: 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.377 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 5 monclient: authenticate success, global_id 274238 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31bc004220 con 0x7f31c00a0580 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 monclient: _renew_subs 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31c0125d10 con 0x7f31c00a0580 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31bc0077d0 con 0x7f31c00a0580 2024-09-17T15:52:21.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b7fff640 4 set_mon_vals no callback set 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 10 monclient: dump: 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:21.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 monclient: _renew_subs 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31c0073df0 con 0x7f31c00a0580 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31c009f010 con 0x7f31c0122f10 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f31bc007970 con 0x7f31c00a0580 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a403bb60 0x7f31a403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f31bc04fcd0 con 0x7f31c00a0580 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c0122f10 auth_method 0 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c640f640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a403bb60 0x7f31a403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.365+0000 7f31c640f640 10 monclient: get_auth_request con 0x7f31a403bb60 auth_method 0 2024-09-17T15:52:21.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.369+0000 7f31c640f640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a403bb60 0x7f31a403e020 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f31c0138230 tx=0x7f31b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:21.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c0122f10 auth_method 0 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:52:21.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.569+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c0122f10 auth_method 0 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:52:21.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:52:21.974 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:21.973+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c0122f10 auth_method 0 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-17T15:52:22.777 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-17T15:52:22.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-17T15:52:22.778 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:22.777+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-17T15:52:24.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.365+0000 7f31b57fa640 10 monclient: tick 2024-09-17T15:52:24.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.365+0000 7f31b57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:52:24.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.365+0000 7f31b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:24.362 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.365+0000 7f31b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:24.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c0122f10 auth_method 0 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_done global_id 284098 payload 824 2024-09-17T15:52:24.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.381+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31c00008e0 con 0x7f31c0122f10 2024-09-17T15:52:24.384 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31b001f150 tx=0x7f31b001b650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00c0030 con 0x7f31c0122f10 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 10 monclient: dump: 2024-09-17T15:52:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.386 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31b00a7a30 con 0x7f31c0122f10 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31b001f150 tx=0x7f31b001b650 comp rx=0 tx=0).stop 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 msgr2=0x7f31c01232f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:24.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.385+0000 7f31b67fc640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c0122f10 0x7f31c01232f0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f31b001f150 tx=0x7f31b001b650 comp rx=0 tx=0).stop 2024-09-17T15:52:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c7e72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-17T15:52:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c7e72640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 0x7f31c002d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c7e72640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31c0137d50 con 0x7f31c002d450 2024-09-17T15:52:24.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 0x7f31c002d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient: get_auth_request con 0x7f31c002d450 auth_method 0 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 10 monclient(hunting): handle_auth_done global_id 284101 payload 824 2024-09-17T15:52:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.393+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31c013bcd0 con 0x7f31c002d450 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31c5c0e640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 msgr2=0x7f31c002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31c5c0e640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 0x7f31c002d860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f31c009f190 tx=0x7f31b00a9000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b00c0040 con 0x7f31c002d450 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 10 monclient: dump: 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.398 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 <== mon.1 v2:172.21.15.203:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f31b00bc860 con 0x7f31c002d450 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 msgr2=0x7f31c002d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 0x7f31c002d860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f31c009f190 tx=0x7f31b00a9000 comp rx=0 tx=0).stop 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 msgr2=0x7f31c002d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.397+0000 7f31b67fc640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31c002d450 0x7f31c002d860 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f31c009f190 tx=0x7f31b00a9000 comp rx=0 tx=0).stop 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:52:24.400 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "state": "peon", 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "election_epoch": 224, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "uptime": 10615, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum": [ 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: 0, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: 1, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: 2 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_age": 4, 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "required_con": "2450521708007866372", 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "required_mon": [ 2024-09-17T15:52:24.401 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_con": "4540719139924082687", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "quorum_mon": [ 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:24.402 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "outside_quorum": [], 2024-09-17T15:52:24.403 INFO:teuthology.orchestra.run.smithi086.stdout: "extra_probe_peers": [], 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "sync_provider": [], 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": { 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "epoch": 1, 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "b4e90329-4d4c-4253-907f-d2ba7ae38072", 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "modified": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "created": "2024-09-17T15:26:41.267126Z", 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release": 19, 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "min_mon_release_name": "squid", 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "election_strategy": 1, 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "disallowed_leaders: ": "", 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false, 2024-09-17T15:52:24.404 INFO:teuthology.orchestra.run.smithi086.stdout: "tiebreaker_mon": "", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "removed_ranks: ": "", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "features": { 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "persistent": [ 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "kraken", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "luminous", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "mimic", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "osdmap-prune", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "nautilus", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "octopus", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "pacific", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "elector-pinging", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "quincy", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "reef", 2024-09-17T15:52:24.405 INFO:teuthology.orchestra.run.smithi086.stdout: "squid" 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "optional": [] 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "mons": [ 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 0, 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "a", 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3300", 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789", 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6789/0", 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6789/0", 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 1, 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "b", 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:3300", 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789", 2024-09-17T15:52:24.408 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.203:6789/0", 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.203:6789/0", 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "rank": 2, 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "c", 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrs": { 2024-09-17T15:52:24.409 INFO:teuthology.orchestra.run.smithi086.stdout: "addrvec": [ 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v2", 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:3301", 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "type": "v1", 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790", 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "nonce": 0 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "addr": "172.21.15.86:6790/0", 2024-09-17T15:52:24.410 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "172.21.15.86:6790/0", 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "priority": 0, 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "weight": 0, 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "{}" 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "feature_map": { 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "mon": [ 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.411 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: "client": [ 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: "features": "0x3f03dffffffdffff", 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: "release": "squid", 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: "num": 1 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout: "stretch_mode": false 2024-09-17T15:52:24.412 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:24.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a403bb60 msgr2=0x7f31a403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f31a403bb60 0x7f31a403e020 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f31c0138230 tx=0x7f31b8009040 comp rx=0 tx=0).stop 2024-09-17T15:52:24.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 10 monclient: shutdown 2024-09-17T15:52:24.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:24.413 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 msgr2=0x7f31c01376c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 --2- 172.21.15.86:0/2968910774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31c00a0580 0x7f31c01376c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f31bc00c7c0 tx=0x7f31bc00cc80 comp rx=0 tx=0).stop 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c640f640 1 -- 172.21.15.86:0/2968910774 reap_dead start 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c640f640 1 -- 172.21.15.86:0/2968910774 reap_dead start 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 shutdown_connections 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 >> 172.21.15.86:0/2968910774 conn(0x7f31c00a2f40 msgr2=0x7f31c00a5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 shutdown_connections 2024-09-17T15:52:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.401+0000 7f31c7e72640 1 -- 172.21.15.86:0/2968910774 wait complete. 2024-09-17T15:52:24.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-17T15:52:24.426 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-17T15:52:24.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-17T15:52:24.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.549+0000 7f37509aa640 10 monclient: get_monmap_and_config 2024-09-17T15:52:24.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.549+0000 7f37509aa640 10 monclient: build_initial_monmap 2024-09-17T15:52:24.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.549+0000 7f37509aa640 10 monclient: monmap: 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:24.555505+0000 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:24.555505+0000 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:24.552 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 Processor -- start 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 -- start start 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient: init 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f374c0a5070 0x7f374c09cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-c con 0x7f374c0a5070 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f374c0a5450 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 0x7f374c13f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.555 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-b con 0x7f374c13d280 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c0a6f40 con 0x7f374c0a5450 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f374c0a65e0 con 0x7f374c0a5070 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f374c09c2a0 con 0x7f374c13d280 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f37509aa640 10 monclient(hunting): authenticate will time out at 2477.521344s 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f3749d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f374c0a5450 0x7f374c0a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56548/0 (socket says 172.21.15.86:56548) 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f3749d9b640 1 -- 172.21.15.86:0/3907530419 learned_addr learned my addr 172.21.15.86:0/3907530419 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 --2- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 0x7f374c13f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): get_auth_request con 0x7f374c13d280 auth_method 0 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 1 --2- 172.21.15.86:0/3907530419 >> v2:172.21.15.203:3300/0 conn(0x7f374c0a5070 0x7f374c09cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): get_auth_request con 0x7f374c0a5070 auth_method 0 2024-09-17T15:52:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.557 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:24.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient(hunting): handle_auth_done global_id 274241 payload 274 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 -- 172.21.15.86:0/3907530419 >> v2:172.21.15.203:3300/0 conn(0x7f374c0a5070 msgr2=0x7f374c09cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 --2- 172.21.15.86:0/3907530419 >> v2:172.21.15.203:3300/0 conn(0x7f374c0a5070 0x7f374c09cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 -- 172.21.15.86:0/3907530419 >> v1:172.21.15.86:6789/0 conn(0x7f374c0a5450 legacy=0x7f374c0a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 monclient: found mon.noname-b 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.553+0000 7f374ad9d640 1 -- 172.21.15.86:0/3907530419 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c09c580 con 0x7f374c13d280 2024-09-17T15:52:24.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374ad9d640 1 -- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 msgr2=0x7f374c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374ad9d640 1 --2- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 0x7f374c13f670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3744009d90 tx=0x7f374401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec5f2876853a1a59 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 1 -- 172.21.15.86:0/3907530419 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37440ac070 con 0x7f374c13d280 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: dump: 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.563 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: _finish_auth 0 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: _check_auth_tickets 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 5 monclient: authenticate success, global_id 274241 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 1 -- 172.21.15.86:0/3907530419 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37440a6680 con 0x7f374c13d280 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 4 set_mon_vals no callback set 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 1 -- 172.21.15.86:0/3907530419 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37440a6a50 con 0x7f374c13d280 2024-09-17T15:52:24.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f374959a640 10 monclient: dump: 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 10 monclient: shutdown 2024-09-17T15:52:24.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 -- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 msgr2=0x7f374c13f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 --2- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 0x7f374c13f670 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3744009d90 tx=0x7f374401c830 comp rx=0 tx=0).stop 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 -- 172.21.15.86:0/3907530419 shutdown_connections 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 --2- 172.21.15.86:0/3907530419 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c13d280 0x7f374c13f670 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 --2- 172.21.15.86:0/3907530419 >> v2:172.21.15.203:3300/0 conn(0x7f374c0a5070 0x7f374c09cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 -- 172.21.15.86:0/3907530419 >> 172.21.15.86:0/3907530419 conn(0x7f374c0742a0 msgr2=0x7f374c0746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 -- 172.21.15.86:0/3907530419 shutdown_connections 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 1 -- 172.21.15.86:0/3907530419 wait complete. 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 10 monclient: build_initial_monmap 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.557+0000 7f37509aa640 10 monclient: monmap: 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:24.566 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 Processor -- start 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- start start 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: init 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-c con 0x7f374c0a5070 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 0x7f374c137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-b con 0x7f374c0a5450 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): picked mon.noname-a con 0x7f374c13d280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f374c142760 con 0x7f374c13d280 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f374c1425e0 con 0x7f374c0a5450 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f374c1428e0 con 0x7f374c0a5070 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient(hunting): authenticate will time out at 2477.528747s 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34592/0 (socket says 172.21.15.86:34592) 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50674/0 (socket says 172.21.15.86:50674) 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 1 -- 172.21.15.86:0/2556677240 learned_addr learned my addr 172.21.15.86:0/2556677240 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): get_auth_request con 0x7f374c13d280 auth_method 0 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): get_auth_request con 0x7f374c0a5070 auth_method 0 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:24.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f3749d9b640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 0x7f374c137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient(hunting): handle_auth_done global_id 274244 payload 824 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 -- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 msgr2=0x7f374c137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 0x7f374c137ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 -- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 msgr2=0x7f374c13b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.570 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient: _finish_hunting 0 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 monclient: found mon.noname-c 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 -- 172.21.15.86:0/2556677240 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c0a1860 con 0x7f374c0a5070 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374ad9d640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f3749d9b640 10 monclient: get_auth_request con 0x7f374c0a5450 auth_method 0 2024-09-17T15:52:24.571 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f3749d9b640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 0x7f374c137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 -- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 msgr2=0x7f374c121180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f374a59c640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e7f0 tx=0x7f373c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c017070 con 0x7f374c0a5070 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: dump: 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.573 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: _finish_auth 0 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c00cad0 con 0x7f374c0a5070 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 5 monclient: authenticate success, global_id 274244 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _renew_subs 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c1223c0 con 0x7f374c0a5070 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c013470 con 0x7f374c0a5070 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f373affd640 10 monclient: dump: 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:24.575 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _renew_subs 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f3748d99640 4 set_mon_vals no callback set 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.561+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c138530 con 0x7f374c0a5070 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f37509aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f37509aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f37509aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374c09cf60 con 0x7f374c0a5070 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f373c01a070 con 0x7f374c0a5070 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f373affd640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 0x7f372c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f373c04fd20 con 0x7f374c0a5070 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f3749d9b640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 0x7f372c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:24.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.565+0000 7f3749d9b640 10 monclient: get_auth_request con 0x7f372c03bb60 auth_method 0 2024-09-17T15:52:24.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.569+0000 7f3749d9b640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 0x7f372c03e020 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3740007fd0 tx=0x7f3740009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:24.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.569+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f374c0a1860 con 0x7f374c0a5070 2024-09-17T15:52:24.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.569+0000 7f373affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:24.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.569+0000 7f373affd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:24.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.793+0000 7f37509aa640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-17T15:52:24.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.793+0000 7f37509aa640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-17T15:52:24.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.793+0000 7f37509aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:24.796 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.793+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f374c13cb90 con 0x7f374c0a5070 2024-09-17T15:52:24.808 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=4050011463,config=3541143443,health=1084904687,kv=3591778543,logm=2031128059}) 2024-09-17T15:52:24.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=10,logm=46} crc {auth=3778254672,config=3541143443,health=1084904687,kv=3867796606,logm=407481249}) 2024-09-17T15:52:24.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=11,logm=45} crc {auth=4050011463,config=3541143443,health=1084904687,kv=3591778543,logm=2031128059}) 2024-09-17T15:52:24.809 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=11,kv=11,logm=45} crc {auth=2357180003,config=3541143443,health=3650330791,kv=3591778543,logm=3594116106}) 2024-09-17T15:52:24.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f373affd640 1 -- 172.21.15.86:0/2556677240 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f373c023d00 con 0x7f374c0a5070 2024-09-17T15:52:24.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f373affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-17T15:52:24.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f373affd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141130628}) 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513719152}) 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141130628}) 2024-09-17T15:52:24.811 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=912422116}) 2024-09-17T15:52:24.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 msgr2=0x7f372c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 0x7f372c03e020 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3740007fd0 tx=0x7f3740009040 comp rx=0 tx=0).stop 2024-09-17T15:52:24.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 10 monclient: shutdown 2024-09-17T15:52:24.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 msgr2=0x7f374c121180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e7f0 tx=0x7f373c00ecb0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 shutdown_connections 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f372c03bb60 0x7f372c03e020 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374c13d280 0x7f374c13b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f374c0a5450 0x7f374c137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 --2- 172.21.15.86:0/2556677240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374c0a5070 0x7f374c121180 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e7f0 tx=0x7f373c00ecb0 comp rx=0 tx=0).stop 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.809+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 >> 172.21.15.86:0/2556677240 conn(0x7f374c0742a0 msgr2=0x7f374c0a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 shutdown_connections 2024-09-17T15:52:24.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f37509aa640 1 -- 172.21.15.86:0/2556677240 wait complete. 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275164461}) 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768449228}) 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275164461}) 2024-09-17T15:52:24.814 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.813+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4251418884}) 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675533889}) 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624212975}) 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675533889}) 2024-09-17T15:52:24.816 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=553257883}) 2024-09-17T15:52:24.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983616661}) 2024-09-17T15:52:24.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987903495}) 2024-09-17T15:52:24.819 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983616661}) 2024-09-17T15:52:24.820 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.817+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382932207}) 2024-09-17T15:52:24.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552364727}) 2024-09-17T15:52:24.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981772231}) 2024-09-17T15:52:24.821 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552364727}) 2024-09-17T15:52:24.822 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2489513966}) 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215077973}) 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534707496}) 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215077973}) 2024-09-17T15:52:24.823 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.821+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1650918113}) 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293629330}) 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893836422}) 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293629330}) 2024-09-17T15:52:24.825 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273369510}) 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254421236}) 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717473120}) 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254421236}) 2024-09-17T15:52:24.827 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.825+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3081490101}) 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508451427}) 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052835027}) 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508451427}) 2024-09-17T15:52:24.829 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940825865}) 2024-09-17T15:52:24.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43827011}) 2024-09-17T15:52:24.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2973728349}) 2024-09-17T15:52:24.830 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43827011}) 2024-09-17T15:52:24.831 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.829+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594796321}) 2024-09-17T15:52:24.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-17T15:52:24.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.832 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764737026}) 2024-09-17T15:52:24.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135575202}) 2024-09-17T15:52:24.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2764737026}) 2024-09-17T15:52:24.833 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862975365}) 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341958887}) 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960283795}) 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341958887}) 2024-09-17T15:52:24.834 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.833+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=992245488}) 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551184503}) 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254706345}) 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551184503}) 2024-09-17T15:52:24.836 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2740515627}) 2024-09-17T15:52:24.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448899649}) 2024-09-17T15:52:24.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593317100}) 2024-09-17T15:52:24.837 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1448899649}) 2024-09-17T15:52:24.838 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=806841292}) 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180317178}) 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1249778160}) 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3180317178}) 2024-09-17T15:52:24.839 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.837+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126832284}) 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444176061}) 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2809836782}) 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444176061}) 2024-09-17T15:52:24.841 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685196476}) 2024-09-17T15:52:24.842 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477084124}) 2024-09-17T15:52:24.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=186473595}) 2024-09-17T15:52:24.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3477084124}) 2024-09-17T15:52:24.843 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.841+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603129177}) 2024-09-17T15:52:24.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.844 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486077872}) 2024-09-17T15:52:24.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395214237}) 2024-09-17T15:52:24.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486077872}) 2024-09-17T15:52:24.845 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1561851539}) 2024-09-17T15:52:24.846 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706933748}) 2024-09-17T15:52:24.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314853793}) 2024-09-17T15:52:24.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706933748}) 2024-09-17T15:52:24.847 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.845+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958303564}) 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161756049}) 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=17943491}) 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161756049}) 2024-09-17T15:52:24.848 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2361529005}) 2024-09-17T15:52:24.849 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789595304}) 2024-09-17T15:52:24.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1660519642}) 2024-09-17T15:52:24.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1789595304}) 2024-09-17T15:52:24.850 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=397129079}) 2024-09-17T15:52:24.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642316277}) 2024-09-17T15:52:24.851 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3840716027}) 2024-09-17T15:52:24.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1642316277}) 2024-09-17T15:52:24.852 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.849+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1522133748}) 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242459328}) 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=566991548}) 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3242459328}) 2024-09-17T15:52:24.853 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2629086023}) 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323448356}) 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695826394}) 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323448356}) 2024-09-17T15:52:24.855 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.853+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259890899}) 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240854072}) 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=420476461}) 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240854072}) 2024-09-17T15:52:24.857 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791628961}) 2024-09-17T15:52:24.858 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038375860}) 2024-09-17T15:52:24.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2393716254}) 2024-09-17T15:52:24.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.859 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1038375860}) 2024-09-17T15:52:24.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.857+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508019385}) 2024-09-17T15:52:24.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.860 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990449348}) 2024-09-17T15:52:24.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1633365683}) 2024-09-17T15:52:24.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=990449348}) 2024-09-17T15:52:24.861 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2971561430}) 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090280195}) 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381042072}) 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3090280195}) 2024-09-17T15:52:24.862 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024888613}) 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.861+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421840072}) 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081411757}) 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421840072}) 2024-09-17T15:52:24.864 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4154555234}) 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005947553}) 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773806277}) 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3005947553}) 2024-09-17T15:52:24.866 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590654655}) 2024-09-17T15:52:24.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056582333}) 2024-09-17T15:52:24.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029814301}) 2024-09-17T15:52:24.867 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056582333}) 2024-09-17T15:52:24.868 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.865+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571251040}) 2024-09-17T15:52:24.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74228493}) 2024-09-17T15:52:24.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077756880}) 2024-09-17T15:52:24.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.869 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74228493}) 2024-09-17T15:52:24.870 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757027843}) 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156994713}) 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014932374}) 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2156994713}) 2024-09-17T15:52:24.871 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.869+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1691361375}) 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949769641}) 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413041113}) 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949769641}) 2024-09-17T15:52:24.873 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=717518347}) 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577691555}) 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1671172652}) 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3577691555}) 2024-09-17T15:52:24.875 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.873+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653114735}) 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221866492}) 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2829862305}) 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221866492}) 2024-09-17T15:52:24.877 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270716070}) 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905597297}) 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855246936}) 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905597297}) 2024-09-17T15:52:24.879 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.877+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1045996176}) 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50144053}) 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419779633}) 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50144053}) 2024-09-17T15:52:24.881 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2099872234}) 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696785831}) 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3347156374}) 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3696785831}) 2024-09-17T15:52:24.883 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.881+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326880459}) 2024-09-17T15:52:24.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416295479}) 2024-09-17T15:52:24.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1067307824}) 2024-09-17T15:52:24.885 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416295479}) 2024-09-17T15:52:24.886 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342575637}) 2024-09-17T15:52:24.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725353981}) 2024-09-17T15:52:24.887 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4212828894}) 2024-09-17T15:52:24.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1725353981}) 2024-09-17T15:52:24.888 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.885+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398564144}) 2024-09-17T15:52:24.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.889 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778153650}) 2024-09-17T15:52:24.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579316287}) 2024-09-17T15:52:24.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778153650}) 2024-09-17T15:52:24.890 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3842237062}) 2024-09-17T15:52:24.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.891 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850472121}) 2024-09-17T15:52:24.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=19264573}) 2024-09-17T15:52:24.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850472121}) 2024-09-17T15:52:24.892 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.889+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1453592966}) 2024-09-17T15:52:24.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459749877}) 2024-09-17T15:52:24.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2209321148}) 2024-09-17T15:52:24.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.894 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459749877}) 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780904390}) 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114221544}) 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299544908}) 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114221544}) 2024-09-17T15:52:24.895 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=296795392}) 2024-09-17T15:52:24.896 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647455188}) 2024-09-17T15:52:24.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943675250}) 2024-09-17T15:52:24.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647455188}) 2024-09-17T15:52:24.897 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.893+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462216591}) 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791715736}) 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990122213}) 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791715736}) 2024-09-17T15:52:24.898 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858124069}) 2024-09-17T15:52:24.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004061465}) 2024-09-17T15:52:24.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074743603}) 2024-09-17T15:52:24.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.899 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004061465}) 2024-09-17T15:52:24.900 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.897+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252155724}) 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749124933}) 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523501928}) 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=749124933}) 2024-09-17T15:52:24.901 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866348672}) 2024-09-17T15:52:24.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165625218}) 2024-09-17T15:52:24.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578736308}) 2024-09-17T15:52:24.903 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165625218}) 2024-09-17T15:52:24.904 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.901+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342279926}) 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656963815}) 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382347138}) 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656963815}) 2024-09-17T15:52:24.905 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805587741}) 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756116295}) 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2827346895}) 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756116295}) 2024-09-17T15:52:24.907 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.905+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3317486418}) 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478415478}) 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1544233216}) 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478415478}) 2024-09-17T15:52:24.909 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2185223915}) 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732462355}) 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3770731248}) 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732462355}) 2024-09-17T15:52:24.911 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.909+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2974807351}) 2024-09-17T15:52:24.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507788151}) 2024-09-17T15:52:24.913 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118212802}) 2024-09-17T15:52:24.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507788151}) 2024-09-17T15:52:24.914 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887943418}) 2024-09-17T15:52:24.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023947263}) 2024-09-17T15:52:24.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2825476203}) 2024-09-17T15:52:24.915 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023947263}) 2024-09-17T15:52:24.916 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.913+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218084350}) 2024-09-17T15:52:24.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232899545}) 2024-09-17T15:52:24.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563622316}) 2024-09-17T15:52:24.917 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232899545}) 2024-09-17T15:52:24.918 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963518029}) 2024-09-17T15:52:24.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927365975}) 2024-09-17T15:52:24.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2742535029}) 2024-09-17T15:52:24.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.919 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3927365975}) 2024-09-17T15:52:24.920 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.917+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616107411}) 2024-09-17T15:52:24.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439991451}) 2024-09-17T15:52:24.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3369606018}) 2024-09-17T15:52:24.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.921 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439991451}) 2024-09-17T15:52:24.922 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690236950}) 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241544060}) 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043896324}) 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3241544060}) 2024-09-17T15:52:24.923 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.921+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2813468086}) 2024-09-17T15:52:24.924 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919449062}) 2024-09-17T15:52:24.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3587159642}) 2024-09-17T15:52:24.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919449062}) 2024-09-17T15:52:24.925 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676914944}) 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047324089}) 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315302675}) 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047324089}) 2024-09-17T15:52:24.926 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.925+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3167582265}) 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092117817}) 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539257349}) 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4092117817}) 2024-09-17T15:52:24.928 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750472021}) 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414438507}) 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=421863439}) 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414438507}) 2024-09-17T15:52:24.930 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280466175}) 2024-09-17T15:52:24.931 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865944396}) 2024-09-17T15:52:24.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675836419}) 2024-09-17T15:52:24.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865944396}) 2024-09-17T15:52:24.932 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.929+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=106298636}) 2024-09-17T15:52:24.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43456738}) 2024-09-17T15:52:24.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1862565322}) 2024-09-17T15:52:24.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.933 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43456738}) 2024-09-17T15:52:24.934 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274401566}) 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857601105}) 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2272922825}) 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=857601105}) 2024-09-17T15:52:24.935 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.933+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954870863}) 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922918262}) 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=630617307}) 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922918262}) 2024-09-17T15:52:24.937 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3843784618}) 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317098178}) 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437107290}) 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317098178}) 2024-09-17T15:52:24.939 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.937+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715388832}) 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998887806}) 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359866449}) 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998887806}) 2024-09-17T15:52:24.941 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046335435}) 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296101481}) 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167217089}) 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296101481}) 2024-09-17T15:52:24.943 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:24.941+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185230604}) 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360132945}) 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038802904}) 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3360132945}) 2024-09-17T15:52:25.529 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2922210223}) 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714592859}) 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884834514}) 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714592859}) 2024-09-17T15:52:25.531 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.529+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653948872}) 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447884765}) 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999029905}) 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447884765}) 2024-09-17T15:52:25.533 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064359421}) 2024-09-17T15:52:25.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.535 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331750754}) 2024-09-17T15:52:25.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3887940241}) 2024-09-17T15:52:25.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.536 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331750754}) 2024-09-17T15:52:25.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.533+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233131310}) 2024-09-17T15:52:25.537 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553945955}) 2024-09-17T15:52:25.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387241321}) 2024-09-17T15:52:25.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553945955}) 2024-09-17T15:52:25.538 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954824913}) 2024-09-17T15:52:25.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.539 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315330998}) 2024-09-17T15:52:25.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250216834}) 2024-09-17T15:52:25.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315330998}) 2024-09-17T15:52:25.540 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.537+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708853144}) 2024-09-17T15:52:25.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233082229}) 2024-09-17T15:52:25.541 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115959196}) 2024-09-17T15:52:25.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233082229}) 2024-09-17T15:52:25.542 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926866535}) 2024-09-17T15:52:25.543 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.543 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468908210}) 2024-09-17T15:52:25.543 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094432331}) 2024-09-17T15:52:25.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468908210}) 2024-09-17T15:52:25.544 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.541+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567009790}) 2024-09-17T15:52:25.545 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718341368}) 2024-09-17T15:52:25.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811022914}) 2024-09-17T15:52:25.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718341368}) 2024-09-17T15:52:25.546 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101296865}) 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643850572}) 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521752591}) 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.545+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1643850572}) 2024-09-17T15:52:25.548 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030302032}) 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145423187}) 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466421284}) 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145423187}) 2024-09-17T15:52:25.550 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.549+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289094689}) 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123586777}) 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220878311}) 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123586777}) 2024-09-17T15:52:25.552 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357316826}) 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304060071}) 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3143353002}) 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304060071}) 2024-09-17T15:52:25.554 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.553+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2620642174}) 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1614138055}) 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442558849}) 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1614138055}) 2024-09-17T15:52:25.556 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663971001}) 2024-09-17T15:52:25.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864998975}) 2024-09-17T15:52:25.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952097230}) 2024-09-17T15:52:25.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.558 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864998975}) 2024-09-17T15:52:25.559 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.557+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3180246111}) 2024-09-17T15:52:25.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699759587}) 2024-09-17T15:52:25.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816115563}) 2024-09-17T15:52:25.560 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.561 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2699759587}) 2024-09-17T15:52:25.561 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447250842}) 2024-09-17T15:52:25.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063480620}) 2024-09-17T15:52:25.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290613821}) 2024-09-17T15:52:25.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.562 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063480620}) 2024-09-17T15:52:25.563 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.561+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3920281732}) 2024-09-17T15:52:25.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.564 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631921392}) 2024-09-17T15:52:25.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3233340941}) 2024-09-17T15:52:25.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631921392}) 2024-09-17T15:52:25.565 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428345368}) 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339138171}) 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=644051030}) 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3339138171}) 2024-09-17T15:52:25.567 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.565+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2100953802}) 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367279495}) 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249223885}) 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367279495}) 2024-09-17T15:52:25.569 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685543655}) 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506122619}) 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466163447}) 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3506122619}) 2024-09-17T15:52:25.571 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.569+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977202334}) 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425775174}) 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226026218}) 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425775174}) 2024-09-17T15:52:25.573 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1806512555}) 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536485678}) 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=968560839}) 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=536485678}) 2024-09-17T15:52:25.575 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.573+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423127582}) 2024-09-17T15:52:25.577 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.577 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965372817}) 2024-09-17T15:52:25.577 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436082062}) 2024-09-17T15:52:25.577 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965372817}) 2024-09-17T15:52:25.578 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733854616}) 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254305804}) 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2120753140}) 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2254305804}) 2024-09-17T15:52:25.580 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.577+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769958334}) 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892827858}) 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=478560290}) 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1892827858}) 2024-09-17T15:52:25.582 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396437667}) 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.581+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887019542}) 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615552911}) 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887019542}) 2024-09-17T15:52:25.584 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2775452443}) 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344542531}) 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3681466518}) 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344542531}) 2024-09-17T15:52:25.586 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.585+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928035546}) 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028359802}) 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301812311}) 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3028359802}) 2024-09-17T15:52:25.588 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548865821}) 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12202461}) 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3420389277}) 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12202461}) 2024-09-17T15:52:25.590 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.589+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000978773}) 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191638836}) 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3079361949}) 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191638836}) 2024-09-17T15:52:25.592 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757667834}) 2024-09-17T15:52:25.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.594 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893517486}) 2024-09-17T15:52:25.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510887306}) 2024-09-17T15:52:25.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893517486}) 2024-09-17T15:52:25.595 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.593+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323278476}) 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145014661}) 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1848335817}) 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145014661}) 2024-09-17T15:52:25.597 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681971146}) 2024-09-17T15:52:25.598 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393216246}) 2024-09-17T15:52:25.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375403473}) 2024-09-17T15:52:25.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1393216246}) 2024-09-17T15:52:25.599 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.597+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166219567}) 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869000485}) 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580779503}) 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=869000485}) 2024-09-17T15:52:25.601 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3371836230}) 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090990815}) 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509591380}) 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090990815}) 2024-09-17T15:52:25.603 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.601+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1928668621}) 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=2} crc {logm=1986535455,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3363155253,mon_config_key=1453524256,monmap=3140514009,nvmeofgw=598337686,osd_metadata=52536041}) 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=1} crc {logm=3151152598,mdsmap=2815901553,mgr=333540546,mgr_metadata=2984965231,mgrstat=36821827,mon_config_key=3068761350,monmap=4191199434,nvmeofgw=598337686,osd_metadata=84285716}) 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=2} crc {logm=1986535455,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=3363155253,mon_config_key=1453524256,monmap=3140514009,nvmeofgw=598337686,osd_metadata=52536041}) 2024-09-17T15:52:25.606 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.605+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=2} crc {logm=3138034693,mdsmap=2815901553,mgr=1288229768,mgr_command_descs=1770323759,mgr_metadata=2984965231,mgrstat=1769337699,mon_config_key=639531838,monmap=2119989493}) 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=2936913951,osd_snap=1415394853}) 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=2936913951,osd_snap=277085212}) 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=2936913951,osd_snap=1415394853}) 2024-09-17T15:52:25.608 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osd_snap=95} crc {osd_metadata=831666901,osd_pg_creating=2936913951,osd_snap=349662687}) 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=64,osdmap=36} crc {osd_snap=841601073,osdmap=3000300149}) 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=64,osdmap=36} crc {osd_snap=804621019,osdmap=3200905361}) 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=64,osdmap=36} crc {osd_snap=841601073,osdmap=3000300149}) 2024-09-17T15:52:25.610 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.609+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=64,osdmap=36} crc {osd_snap=1571281626,osdmap=3739736630}) 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1428159974}) 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=99} crc {osdmap=1061159738}) 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1428159974}) 2024-09-17T15:52:25.623 INFO:tasks.ceph.mon.a.smithi086.stderr:2024-09-17T15:52:25.621+0000 7f649fe70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=4060245775}) 2024-09-17T15:52:25.833 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-17T15:52:25.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-17T15:52:25.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: get_monmap_and_config 2024-09-17T15:52:25.965 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: build_initial_monmap 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: monmap: 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:25.969612+0000 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:25.969612+0000 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:25.966 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:25.967 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 Processor -- start 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 -- start start 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: init 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b80a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2b8137ea0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:25.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2b809d350 con 0x7ff2b8137ea0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff2b80a6530 con 0x7ff2b80a54c0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b809c150 con 0x7ff2b80a50e0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2bd186640 10 monclient(hunting): authenticate will time out at 2478.934775s 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50682/0 (socket says 172.21.15.86:50682) 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/2404082280 learned_addr learned my addr 172.21.15.86:0/2404082280 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): get_auth_request con 0x7ff2b80a50e0 auth_method 0 2024-09-17T15:52:25.977 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 1 --2- 172.21.15.86:0/2404082280 >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34604/0 (socket says 172.21.15.86:34604) 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b759e640 1 --1- 172.21.15.86:0/2404082280 >> v1:172.21.15.86:6789/0 conn(0x7ff2b8137ea0 0x7ff2b8138330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56562/0 (socket says 172.21.15.86:56562) 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): get_auth_request con 0x7ff2b80a54c0 auth_method 0 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:25.978 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_done global_id 274247 payload 274 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/2404082280 >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 msgr2=0x7ff2b80a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 --2- 172.21.15.86:0/2404082280 >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/2404082280 >> v1:172.21.15.86:6789/0 conn(0x7ff2b8137ea0 legacy=0x7ff2b8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 monclient: found mon.noname-b 2024-09-17T15:52:25.979 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/2404082280 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b809c430 con 0x7ff2b80a50e0 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.965+0000 7ff2b659c640 1 --2- 172.21.15.86:0/2404082280 >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/2404082280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 msgr2=0x7ff2b809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b6d9d640 1 --2- 172.21.15.86:0/2404082280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2b806f9b0 tx=0x7ff2ac01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58b8205bde023ae server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 1 -- 172.21.15.86:0/2404082280 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0b0070 con 0x7ff2b80a50e0 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 10 monclient: dump: 2024-09-17T15:52:25.980 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:25.981 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2bd186640 5 monclient: authenticate success, global_id 274247 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.969+0000 7ff2bd186640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 1 -- 172.21.15.86:0/2404082280 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2ac0ab600 con 0x7ff2b80a50e0 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:25.982 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 1 -- 172.21.15.86:0/2404082280 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0ab9d0 con 0x7ff2b80a50e0 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 4 set_mon_vals no callback set 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b5d9b640 10 monclient: dump: 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:25.983 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: shutdown 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- 172.21.15.86:0/2404082280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 msgr2=0x7ff2b809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- 172.21.15.86:0/2404082280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2b806f9b0 tx=0x7ff2ac01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- 172.21.15.86:0/2404082280 shutdown_connections 2024-09-17T15:52:25.984 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- 172.21.15.86:0/2404082280 >> v2:172.21.15.86:3300/0 conn(0x7ff2b80a54c0 0x7ff2b80a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- 172.21.15.86:0/2404082280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b809ce10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- 172.21.15.86:0/2404082280 >> 172.21.15.86:0/2404082280 conn(0x7ff2b80aaae0 msgr2=0x7ff2b80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- 172.21.15.86:0/2404082280 shutdown_connections 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- 172.21.15.86:0/2404082280 wait complete. 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: build_initial_monmap 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: monmap: 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:25.985 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 Processor -- start 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- start start 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: init 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-c con 0x7ff2b80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2b80a54c0 0x7ff2b8136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b80a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2b8137ea0 0x7ff2b813ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b8137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b8142cb0 con 0x7ff2b8137ea0 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b8142b30 con 0x7ff2b80a54c0 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b8142e30 con 0x7ff2b80a50e0 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2bd186640 10 monclient(hunting): authenticate will time out at 2478.942316s 2024-09-17T15:52:25.987 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2b8137ea0 0x7ff2b813ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50696/0 (socket says 172.21.15.86:50696) 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/1065053375 learned_addr learned my addr 172.21.15.86:0/1065053375 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): get_auth_request con 0x7ff2b80a50e0 auth_method 0 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): get_auth_request con 0x7ff2b8137ea0 auth_method 0 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient(hunting): handle_auth_done global_id 274250 payload 824 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2b80a54c0 msgr2=0x7ff2b8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2b80a54c0 0x7ff2b8136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2b8137ea0 msgr2=0x7ff2b813ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:25.989 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2b8137ea0 0x7ff2b813ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:25.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:25.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 monclient: found mon.noname-c 2024-09-17T15:52:25.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:25.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:25.990 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.973+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/1065053375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b813fdd0 con 0x7ff2b80a50e0 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2b6d9d640 1 -- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 msgr2=0x7ff2b813f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2b6d9d640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0ae040 tx=0x7ff2ac019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0b0040 con 0x7ff2b80a50e0 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: dump: 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:25.992 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 5 monclient: authenticate success, global_id 274250 2024-09-17T15:52:25.993 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _renew_subs 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b8121670 con 0x7ff2b80a50e0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _renew_subs 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b80a18d0 con 0x7ff2b80a50e0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2ac0b9070 con 0x7ff2b80a50e0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2ac0b37d0 con 0x7ff2b80a50e0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 10 monclient: dump: 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2b559a640 4 set_mon_vals no callback set 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff298005090 con 0x7ff2b80a50e0 2024-09-17T15:52:25.995 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.977+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff2ac0b3970 con 0x7ff2b80a50e0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.981+0000 7ff29f7fe640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 0x7ff2a403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.981+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7ff2ac0e9b70 con 0x7ff2b80a50e0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.981+0000 7ff2b659c640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 0x7ff2a403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.981+0000 7ff2b659c640 10 monclient: get_auth_request con 0x7ff2a403bb60 auth_method 0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.981+0000 7ff2b659c640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 0x7ff2a403e020 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006e90 tx=0x7ff2a8006de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.985+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7ff2ac0160f0 con 0x7ff2b80a50e0 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.985+0000 7ff29f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:25.996 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:25.985+0000 7ff29f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.221+0000 7ff2bd186640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-17T15:52:26.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.221+0000 7ff2bd186640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:26.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.221+0000 7ff2bd186640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.221+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff298005380 con 0x7ff2b80a50e0 2024-09-17T15:52:26.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff29f7fe640 1 -- 172.21.15.86:0/1065053375 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff2ac0abe20 con 0x7ff2b80a50e0 2024-09-17T15:52:26.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff29f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-17T15:52:26.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff29f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:26.224 INFO:teuthology.orchestra.run.smithi086.stdout:{"election_epoch":224,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","modified":"2024-09-17T15:26:41.267126Z","created":"2024-09-17T15:26:41.267126Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3301","nonce":0},{"type":"v1","addr":"172.21.15.86:6790","nonce":0}]},"addr":"172.21.15.86:6790/0","public_addr":"172.21.15.86:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-17T15:52:26.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 msgr2=0x7ff2a403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 0x7ff2a403e020 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8006e90 tx=0x7ff2a8006de0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 10 monclient: shutdown 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 msgr2=0x7ff2b813f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac0ae040 tx=0x7ff2ac019180 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 shutdown_connections 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff2a403bb60 0x7ff2a403e020 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff2b8137ea0 0x7ff2b813ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2b80a54c0 0x7ff2b8136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 --2- 172.21.15.86:0/1065053375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff2b80a50e0 0x7ff2b813f440 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 >> 172.21.15.86:0/1065053375 conn(0x7ff2b80aaae0 msgr2=0x7ff2b80a0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 shutdown_connections 2024-09-17T15:52:26.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.225+0000 7ff2bd186640 1 -- 172.21.15.86:0/1065053375 wait complete. 2024-09-17T15:52:26.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-17T15:52:26.243 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T15:52:26.252 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T15:52:26.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:26.374 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 10 monclient: monmap: 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:26.378753+0000 2024-09-17T15:52:26.375 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:26.378753+0000 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:26.376 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.378 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 1 Processor -- start 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 1 -- start start 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.373+0000 7f7bdfbf0640 10 monclient: init 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 0x7f7bd809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-c con 0x7f7bd80a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bd80a54c0 0x7f7bd80a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-a con 0x7f7bd80a54c0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd8138330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-b con 0x7f7bd8137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd809bdc0 con 0x7f7bd80a54c0 2024-09-17T15:52:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd809d350 con 0x7f7bd80a50e0 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd80a6530 con 0x7f7bd8137ea0 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdfbf0640 10 monclient(hunting): authenticate will time out at 2479.344492s 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bd80a54c0 0x7f7bd80a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd8138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7bd80a54c0 0x7f7bd80a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34624/0 (socket says 172.21.15.86:34624) 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 -- 172.21.15.86:0/2534289974 learned_addr learned my addr 172.21.15.86:0/2534289974 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): get_auth_request con 0x7f7bd80a54c0 auth_method 0 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.380 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): get_auth_request con 0x7f7bd8137ea0 auth_method 0 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient(hunting): handle_auth_done global_id 234642 payload 274 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd98c640 1 --2- 172.21.15.86:0/2534289974 >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 0x7f7bd809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 -- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 msgr2=0x7f7bd8138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd8138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 -- 172.21.15.86:0/2534289974 >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 msgr2=0x7f7bd809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 --2- 172.21.15.86:0/2534289974 >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 0x7f7bd809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 monclient: found mon.noname-a 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd18b640 1 -- 172.21.15.86:0/2534289974 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd809c150 con 0x7f7bd80a54c0 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd98c640 10 monclient: get_auth_request con 0x7f7bd80a50e0 auth_method 0 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bdd98c640 1 --2- 172.21.15.86:0/2534289974 >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 0x7f7bd809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:26.382 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.377+0000 7f7bde18d640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd8138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:26.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd18b640 1 -- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a54c0 msgr2=0x7f7bd80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd18b640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a54c0 0x7f7bd80a58d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80a8280 tx=0x7f7bd001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7e7ccb6a6f99a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 1 -- 172.21.15.86:0/2534289974 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00b0070 con 0x7f7bd80a54c0 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: dump: 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 1 -- 172.21.15.86:0/2534289974 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bd00ab600 con 0x7f7bd80a54c0 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 5 monclient: authenticate success, global_id 234642 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 1 -- 172.21.15.86:0/2534289974 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bd00ab9d0 con 0x7f7bd80a54c0 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 4 set_mon_vals no callback set 2024-09-17T15:52:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdc98a640 10 monclient: dump: 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: shutdown 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.390 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a54c0 msgr2=0x7f7bd80a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a54c0 0x7f7bd80a58d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7bd80a8280 tx=0x7f7bd001c3b0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/2534289974 shutdown_connections 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd8138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/2534289974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a54c0 0x7f7bd80a58d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/2534289974 >> v2:172.21.15.203:3300/0 conn(0x7f7bd80a50e0 0x7f7bd809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/2534289974 >> 172.21.15.86:0/2534289974 conn(0x7f7bd80aaae0 msgr2=0x7f7bd80aaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/2534289974 shutdown_connections 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/2534289974 wait complete. 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.391 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: monmap: 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 Processor -- start 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- start start 2024-09-17T15:52:26.392 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: init 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-a con 0x7f7bd80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bd80a54c0 0x7f7bd81370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-b con 0x7f7bd80a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd813c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): picked mon.noname-c con 0x7f7bd8137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd806f660 con 0x7f7bd80a50e0 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd806f4e0 con 0x7f7bd80a54c0 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd8141b20 con 0x7f7bd8137ea0 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdfbf0640 10 monclient(hunting): authenticate will time out at 2479.351590s 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34638/0 (socket says 172.21.15.86:34638) 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 1 -- 172.21.15.86:0/1683324726 learned_addr learned my addr 172.21.15.86:0/1683324726 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): get_auth_request con 0x7f7bd80a50e0 auth_method 0 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd813c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): get_auth_request con 0x7f7bd8137ea0 auth_method 0 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.394 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd98c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bdd18b640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bd80a54c0 0x7f7bd81370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.381+0000 7f7bde18d640 10 monclient(hunting): handle_auth_done global_id 274256 payload 824 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 -- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bd80a54c0 msgr2=0x7f7bd81370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bd80a54c0 0x7f7bd81370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 -- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 msgr2=0x7f7bd81cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.395 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 monclient: found mon.noname-c 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 -- 172.21.15.86:0/1683324726 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd8122a60 con 0x7f7bd8137ea0 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdd98c640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 -- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 msgr2=0x7f7bd813c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bde18d640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd813c290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800e960 tx=0x7f7bc800ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8017070 con 0x7f7bd8137ea0 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: dump: 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.397 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc800cc10 con 0x7f7bd8137ea0 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 5 monclient: authenticate success, global_id 274256 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8004020 con 0x7f7bd8137ea0 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 10 monclient: dump: 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.398 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bcffff640 4 set_mon_vals no callback set 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _renew_subs 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd80a0fd0 con 0x7f7bd8137ea0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _renew_subs 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd813dd50 con 0x7f7bd8137ea0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7bc80041c0 con 0x7f7bd8137ea0 2024-09-17T15:52:26.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.385+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd80a58d0 con 0x7f7bd8137ea0 2024-09-17T15:52:26.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.389+0000 7f7bce7fc640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 0x7f7bc403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.389+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f7bc804fba0 con 0x7f7bd8137ea0 2024-09-17T15:52:26.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.389+0000 7f7bdd98c640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 0x7f7bc403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.389+0000 7f7bdd98c640 10 monclient: get_auth_request con 0x7f7bc403bb60 auth_method 0 2024-09-17T15:52:26.400 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.389+0000 7f7bdd98c640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 0x7f7bc403e020 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d9e0 tx=0x7f7bd40078d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.393+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f7bc804e730 con 0x7f7bd8137ea0 2024-09-17T15:52:26.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.393+0000 7f7bce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.393+0000 7f7bce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.557+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7bd8122330 con 0x7f7bc403bb60 2024-09-17T15:52:26.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bce7fc640 1 -- 172.21.15.86:0/1683324726 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33968 (secure 0 0 0) 0x7f7bd8122330 con 0x7f7bc403bb60 2024-09-17T15:52:26.561 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:26.561 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:26.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 msgr2=0x7f7bc403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 0x7f7bc403e020 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7bd400d9e0 tx=0x7f7bd40078d0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 10 monclient: shutdown 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 msgr2=0x7f7bd813c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd813c290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800e960 tx=0x7f7bc800ee20 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 shutdown_connections 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7bc403bb60 0x7f7bc403e020 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7bd8137ea0 0x7f7bd813c290 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bd80a54c0 0x7f7bd81370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 --2- 172.21.15.86:0/1683324726 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7bd80a50e0 0x7f7bd81cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 >> 172.21.15.86:0/1683324726 conn(0x7f7bd80aaae0 msgr2=0x7f7bd813e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 shutdown_connections 2024-09-17T15:52:26.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.561+0000 7f7bdfbf0640 1 -- 172.21.15.86:0/1683324726 wait complete. 2024-09-17T15:52:26.579 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1623,"stamp":"2024-09-17T15:52:26.049743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":314076,"kb_used_data":6176,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565916964,"statfs":{"total":579820584960,"available":579498971136,"internally_reserved":0,"allocated":6324224,"data_stored":3721246,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.0","version":"14'72","reported_seq":1811,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375474+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:52:11.375474+0000","last_peered":"2024-09-17T15:52:11.375474+0000","last_clean":"2024-09-17T15:52:11.375474+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:11.375474+0000","last_undegraded":"2024-09-17T15:52:11.375474+0000","last_fullsized":"2024-09-17T15:52:11.375474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1725,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375309+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:52:11.375309+0000","last_peered":"2024-09-17T15:52:11.375309+0000","last_clean":"2024-09-17T15:52:11.375309+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:11.375309+0000","last_undegraded":"2024-09-17T15:52:11.375309+0000","last_fullsized":"2024-09-17T15:52:11.375309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:10.899559+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:52:10.899559+0000","last_peered":"2024-09-17T15:52:10.899559+0000","last_clean":"2024-09-17T15:52:10.899559+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:10.899559+0000","last_undegraded":"2024-09-17T15:52:10.899559+0000","last_fullsized":"2024-09-17T15:52:10.899559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374668+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:52:11.374668+0000","last_peered":"2024-09-17T15:52:11.374668+0000","last_clean":"2024-09-17T15:52:11.374668+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:11.374668+0000","last_undegraded":"2024-09-17T15:52:11.374668+0000","last_fullsized":"2024-09-17T15:52:11.374668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375366+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:52:11.375366+0000","last_peered":"2024-09-17T15:52:11.375366+0000","last_clean":"2024-09-17T15:52:11.375366+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:11.375366+0000","last_undegraded":"2024-09-17T15:52:11.375366+0000","last_fullsized":"2024-09-17T15:52:11.375366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374688+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:52:11.374688+0000","last_peered":"2024-09-17T15:52:11.374688+0000","last_clean":"2024-09-17T15:52:11.374688+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:11.374688+0000","last_undegraded":"2024-09-17T15:52:11.374688+0000","last_fullsized":"2024-09-17T15:52:11.374688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375404+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:52:11.375404+0000","last_peered":"2024-09-17T15:52:11.375404+0000","last_clean":"2024-09-17T15:52:11.375404+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:11.375404+0000","last_undegraded":"2024-09-17T15:52:11.375404+0000","last_fullsized":"2024-09-17T15:52:11.375404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50304,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321536,"statfs":{"total":96636764160,"available":96585252864,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46088,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325752,"statfs":{"total":96636764160,"available":96589570048,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:26.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:26.700 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.697+0000 7fdefd1ba640 10 monclient: get_monmap_and_config 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.697+0000 7fdefd1ba640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.697+0000 7fdefd1ba640 10 monclient: monmap: 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:26.703750+0000 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:26.703750+0000 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.701 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 Processor -- start 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 -- start start 2024-09-17T15:52:26.702 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient: init 2024-09-17T15:52:26.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 0x7fdef809cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-a con 0x7fdef80a5070 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-b con 0x7fdef80a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdef8137e30 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef80a6f40 con 0x7fdef8137e30 2024-09-17T15:52:26.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fdef80a65e0 con 0x7fdef80a5070 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdef809c2a0 con 0x7fdef80a5450 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdefd1ba640 10 monclient(hunting): authenticate will time out at 2479.669143s 2024-09-17T15:52:26.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 0x7fdef809cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50728/0 (socket says 172.21.15.86:50728) 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 -- 172.21.15.86:0/4117854297 learned_addr learned my addr 172.21.15.86:0/4117854297 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): get_auth_request con 0x7fdef80a5450 auth_method 0 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): get_auth_request con 0x7fdef80a5070 auth_method 0 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient(hunting): handle_auth_done global_id 274259 payload 274 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 -- 172.21.15.86:0/4117854297 >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 msgr2=0x7fdef809cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 --2- 172.21.15.86:0/4117854297 >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 0x7fdef809cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 -- 172.21.15.86:0/4117854297 >> v1:172.21.15.203:6789/0 conn(0x7fdef8137e30 legacy=0x7fdef8138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 monclient: found mon.noname-b 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef659c640 1 -- 172.21.15.86:0/4117854297 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef809c580 con 0x7fdef80a5450 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.701+0000 7fdef6d9d640 1 --2- 172.21.15.86:0/4117854297 >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 0x7fdef809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef659c640 1 -- 172.21.15.86:0/4117854297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 msgr2=0x7fdef80a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef659c640 1 --2- 172.21.15.86:0/4117854297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009870 tx=0x7fdee801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3430d5fb939e4608 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 1 -- 172.21.15.86:0/4117854297 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80b0070 con 0x7fdef80a5450 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: dump: 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 1 -- 172.21.15.86:0/4117854297 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee80ab600 con 0x7fdef80a5450 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 5 monclient: authenticate success, global_id 274259 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 1 -- 172.21.15.86:0/4117854297 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80ab9d0 con 0x7fdef80a5450 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 4 set_mon_vals no callback set 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdef5d9b640 10 monclient: dump: 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 10 monclient: shutdown 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- 172.21.15.86:0/4117854297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 msgr2=0x7fdef80a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/4117854297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee8009870 tx=0x7fdee801c420 comp rx=0 tx=0).stop 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- 172.21.15.86:0/4117854297 shutdown_connections 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/4117854297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef80a5860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/4117854297 >> v2:172.21.15.86:3300/0 conn(0x7fdef80a5070 0x7fdef809cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- 172.21.15.86:0/4117854297 >> 172.21.15.86:0/4117854297 conn(0x7fdef80742a0 msgr2=0x7fdef80746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- 172.21.15.86:0/4117854297 shutdown_connections 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- 172.21.15.86:0/4117854297 wait complete. 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 10 monclient: build_initial_monmap 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 10 monclient: monmap: 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 Processor -- start 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 1 -- start start 2024-09-17T15:52:26.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.705+0000 7fdefd1ba640 10 monclient: init 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 0x7fdef81369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-a con 0x7fdef80a5070 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-c con 0x7fdef80a5450 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 0x7fdef813c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): picked mon.noname-b con 0x7fdef8137e30 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:26.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fdef8142c40 con 0x7fdef80a5070 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdef8142ac0 con 0x7fdef8137e30 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fdef8142dc0 con 0x7fdef80a5450 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient(hunting): authenticate will time out at 2479.676312s 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50740/0 (socket says 172.21.15.86:50740) 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 0x7fdef81369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 -- 172.21.15.86:0/1749051907 learned_addr learned my addr 172.21.15.86:0/1749051907 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): get_auth_request con 0x7fdef80a5450 auth_method 0 2024-09-17T15:52:26.722 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): get_auth_request con 0x7fdef80a5070 auth_method 0 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef759e640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 0x7fdef813c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient(hunting): handle_auth_done global_id 274262 payload 824 2024-09-17T15:52:26.723 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 -- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 msgr2=0x7fdef813c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 0x7fdef813c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 -- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 msgr2=0x7fdef81369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 0x7fdef81369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient: _finish_hunting 0 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 monclient: found mon.noname-c 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 -- 172.21.15.86:0/1749051907 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef8120cc0 con 0x7fdef80a5450 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef6d9d640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 0x7fdef81369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef759e640 10 monclient: get_auth_request con 0x7fdef8137e30 auth_method 0 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef759e640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 0x7fdef813c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 -- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 msgr2=0x7fdef8136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef659c640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdee801aad0 tx=0x7fdee80191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80b0070 con 0x7fdef80a5450 2024-09-17T15:52:26.724 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: dump: 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:26.725 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee80b1040 con 0x7fdef80a5450 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee80abe60 con 0x7fdef80a5450 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 5 monclient: authenticate success, global_id 274262 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _renew_subs 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef8121970 con 0x7fdef80a5450 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdef559a640 4 set_mon_vals no callback set 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdeef7fe640 10 monclient: dump: 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:26.726 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _renew_subs 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.709+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef8121690 con 0x7fdef80a5450 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fdee80aaa20 con 0x7fdef80a5450 2024-09-17T15:52:26.727 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdefd1ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdefd1ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdefd1ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef809cf60 con 0x7fdef80a5450 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdeef7fe640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 0x7fded803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fdee80e9b60 con 0x7fdef80a5450 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdef6d9d640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 0x7fded803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdef6d9d640 10 monclient: get_auth_request con 0x7fded803bbb0 auth_method 0 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.713+0000 7fdef6d9d640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 0x7fded803e070 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009580 tx=0x7fdee0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.717+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fdee8016130 con 0x7fdef80a5450 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.717+0000 7fdeef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:26.728 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.717+0000 7fdeef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:26.875 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.873+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdef8069920 con 0x7fded803bbb0 2024-09-17T15:52:26.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdeef7fe640 1 -- 172.21.15.86:0/1749051907 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33968 (secure 0 0 0) 0x7fdef8069920 con 0x7fded803bbb0 2024-09-17T15:52:26.878 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:26.878 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 msgr2=0x7fded803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 0x7fded803e070 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009580 tx=0x7fdee0009190 comp rx=0 tx=0).stop 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 10 monclient: shutdown 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 msgr2=0x7fdef8136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdee801aad0 tx=0x7fdee80191c0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 shutdown_connections 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fded803bbb0 0x7fded803e070 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdef8137e30 0x7fdef813c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.880 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef80a5450 0x7fdef8136f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 --2- 172.21.15.86:0/1749051907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef80a5070 0x7fdef81369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:26.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 >> 172.21.15.86:0/1749051907 conn(0x7fdef80742a0 msgr2=0x7fdef80a2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:26.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 shutdown_connections 2024-09-17T15:52:26.881 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:26.877+0000 7fdefd1ba640 1 -- 172.21.15.86:0/1749051907 wait complete. 2024-09-17T15:52:26.897 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1623,"stamp":"2024-09-17T15:52:26.049743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":314076,"kb_used_data":6176,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565916964,"statfs":{"total":579820584960,"available":579498971136,"internally_reserved":0,"allocated":6324224,"data_stored":3721246,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.0","version":"14'72","reported_seq":1811,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375474+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:52:11.375474+0000","last_peered":"2024-09-17T15:52:11.375474+0000","last_clean":"2024-09-17T15:52:11.375474+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:11.375474+0000","last_undegraded":"2024-09-17T15:52:11.375474+0000","last_fullsized":"2024-09-17T15:52:11.375474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1725,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375309+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:52:11.375309+0000","last_peered":"2024-09-17T15:52:11.375309+0000","last_clean":"2024-09-17T15:52:11.375309+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:11.375309+0000","last_undegraded":"2024-09-17T15:52:11.375309+0000","last_fullsized":"2024-09-17T15:52:11.375309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:10.899559+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:52:10.899559+0000","last_peered":"2024-09-17T15:52:10.899559+0000","last_clean":"2024-09-17T15:52:10.899559+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:10.899559+0000","last_undegraded":"2024-09-17T15:52:10.899559+0000","last_fullsized":"2024-09-17T15:52:10.899559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374668+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:52:11.374668+0000","last_peered":"2024-09-17T15:52:11.374668+0000","last_clean":"2024-09-17T15:52:11.374668+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:11.374668+0000","last_undegraded":"2024-09-17T15:52:11.374668+0000","last_fullsized":"2024-09-17T15:52:11.374668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375366+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:52:11.375366+0000","last_peered":"2024-09-17T15:52:11.375366+0000","last_clean":"2024-09-17T15:52:11.375366+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:11.375366+0000","last_undegraded":"2024-09-17T15:52:11.375366+0000","last_fullsized":"2024-09-17T15:52:11.375366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374688+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:52:11.374688+0000","last_peered":"2024-09-17T15:52:11.374688+0000","last_clean":"2024-09-17T15:52:11.374688+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:11.374688+0000","last_undegraded":"2024-09-17T15:52:11.374688+0000","last_fullsized":"2024-09-17T15:52:11.374688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375404+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:52:11.375404+0000","last_peered":"2024-09-17T15:52:11.375404+0000","last_clean":"2024-09-17T15:52:11.375404+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:11.375404+0000","last_undegraded":"2024-09-17T15:52:11.375404+0000","last_fullsized":"2024-09-17T15:52:11.375404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50304,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321536,"statfs":{"total":96636764160,"available":96585252864,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46088,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325752,"statfs":{"total":96636764160,"available":96589570048,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:26.898 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T15:52:26.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:27.006 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 10 monclient: get_monmap_and_config 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 10 monclient: build_initial_monmap 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 10 monclient: monmap: 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:27.010663+0000 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:27.010663+0000 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:27.008 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 1 Processor -- start 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 1 -- start start 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.005+0000 7faeee395640 10 monclient: init 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 0x7faee809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): picked mon.noname-c con 0x7faee8069260 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): picked mon.noname-b con 0x7faee8074c30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faee809c4f0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee809e410 con 0x7faee809c4f0 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7faee809d150 con 0x7faee8069260 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faee809ce80 con 0x7faee8074c30 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faeee395640 10 monclient(hunting): authenticate will time out at 2479.976216s 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50754/0 (socket says 172.21.15.86:50754) 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 -- 172.21.15.86:0/246791557 learned_addr learned my addr 172.21.15.86:0/246791557 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): get_auth_request con 0x7faee8074c30 auth_method 0 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.011 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee7fff640 1 --2- 172.21.15.86:0/246791557 >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 0x7faee809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient(hunting): handle_auth_done global_id 274265 payload 274 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 -- 172.21.15.86:0/246791557 >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 msgr2=0x7faee809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 --2- 172.21.15.86:0/246791557 >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 0x7faee809ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 -- 172.21.15.86:0/246791557 >> v1:172.21.15.203:6789/0 conn(0x7faee809c4f0 legacy=0x7faee813aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 monclient: found mon.noname-b 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:27.012 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee77fe640 1 -- 172.21.15.86:0/246791557 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee809e6f0 con 0x7faee8074c30 2024-09-17T15:52:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee7fff640 10 monclient: get_auth_request con 0x7faee8069260 auth_method 0 2024-09-17T15:52:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.009+0000 7faee7fff640 1 --2- 172.21.15.86:0/246791557 >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 0x7faee809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:27.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 1 -- 172.21.15.86:0/246791557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 msgr2=0x7faee809bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:27.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 1 --2- 172.21.15.86:0/246791557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faed000aa10 tx=0x7faed001d7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87dfeec05de99a6a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 1 -- 172.21.15.86:0/246791557 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed000b040 con 0x7faee8074c30 2024-09-17T15:52:27.016 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: dump: 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.017 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 1 -- 172.21.15.86:0/246791557 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed00a9710 con 0x7faee8074c30 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 1 -- 172.21.15.86:0/246791557 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed00a9ae0 con 0x7faee8074c30 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 5 monclient: authenticate success, global_id 274265 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 4 set_mon_vals no callback set 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee6ffd640 10 monclient: dump: 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.018 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: shutdown 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- 172.21.15.86:0/246791557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 msgr2=0x7faee809bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- 172.21.15.86:0/246791557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faed000aa10 tx=0x7faed001d7f0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- 172.21.15.86:0/246791557 shutdown_connections 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- 172.21.15.86:0/246791557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee809bfb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- 172.21.15.86:0/246791557 >> v2:172.21.15.203:3300/0 conn(0x7faee8069260 0x7faee809ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- 172.21.15.86:0/246791557 >> 172.21.15.86:0/246791557 conn(0x7faee80a6440 msgr2=0x7faee80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- 172.21.15.86:0/246791557 shutdown_connections 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- 172.21.15.86:0/246791557 wait complete. 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: build_initial_monmap 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: monmap: 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:27.020 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 Processor -- start 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- start start 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: init 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faee8069260 0x7faee8136c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): picked mon.noname-b con 0x7faee8069260 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee813ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): picked mon.noname-c con 0x7faee8074c30 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): picked mon.noname-a con 0x7faee809c4f0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faee813e600 con 0x7faee809c4f0 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faee813e480 con 0x7faee8069260 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faee813e780 con 0x7faee8074c30 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeee395640 10 monclient(hunting): authenticate will time out at 2479.983022s 2024-09-17T15:52:27.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:34670/0 (socket says 172.21.15.86:34670) 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 -- 172.21.15.86:0/2403312777 learned_addr learned my addr 172.21.15.86:0/2403312777 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): get_auth_request con 0x7faee809c4f0 auth_method 0 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee813ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): get_auth_request con 0x7faee8074c30 auth_method 0 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faee8069260 0x7faee8136c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): get_auth_request con 0x7faee8069260 auth_method 0 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faee7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient(hunting): handle_auth_done global_id 234654 payload 824 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 -- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 msgr2=0x7faee813ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee813ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 -- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faee8069260 msgr2=0x7faee8136c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faee8069260 0x7faee8136c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient: _finish_hunting 0 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 monclient: found mon.noname-a 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.013+0000 7faeec932640 1 -- 172.21.15.86:0/2403312777 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee8122e30 con 0x7faee809c4f0 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeec932640 1 -- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 msgr2=0x7faee813f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeec932640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faed800c780 tx=0x7faed800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed800ac60 con 0x7faee809c4f0 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: dump: 2024-09-17T15:52:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: _finish_auth 0 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:27.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed800ae00 con 0x7faee809c4f0 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed8004bb0 con 0x7faee809c4f0 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 5 monclient: authenticate success, global_id 234654 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 10 monclient: dump: 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.029 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _renew_subs 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee8123110 con 0x7faee809c4f0 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee67fc640 4 set_mon_vals no callback set 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _renew_subs 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee809d460 con 0x7faee809c4f0 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faee809ba70 con 0x7faee809c4f0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faed8005030 con 0x7faee809c4f0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 0x7faec803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7faed804ffd0 con 0x7faee809c4f0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee7fff640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 0x7faec803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.017+0000 7faee7fff640 10 monclient: get_auth_request con 0x7faec803bbe0 auth_method 0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.021+0000 7faee7fff640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 0x7faec803e0a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7faee000ab60 tx=0x7faee0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.021+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faed8020070 con 0x7faee809c4f0 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.021+0000 7faee4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.021+0000 7faee4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:27.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.177+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faee806d4f0 con 0x7faec803bbe0 2024-09-17T15:52:27.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.177+0000 7faee4ff9640 1 -- 172.21.15.86:0/2403312777 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33968 (secure 0 0 0) 0x7faee806d4f0 con 0x7faec803bbe0 2024-09-17T15:52:27.179 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:27.179 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 msgr2=0x7faec803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 0x7faec803e0a0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7faee000ab60 tx=0x7faee0008040 comp rx=0 tx=0).stop 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 10 monclient: shutdown 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 msgr2=0x7faee813f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7faed800c780 tx=0x7faed800cc40 comp rx=0 tx=0).stop 2024-09-17T15:52:27.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 shutdown_connections 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faec803bbe0 0x7faec803e0a0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faee809c4f0 0x7faee813f150 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faee8074c30 0x7faee813ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 --2- 172.21.15.86:0/2403312777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faee8069260 0x7faee8136c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 >> 172.21.15.86:0/2403312777 conn(0x7faee80a6440 msgr2=0x7faee80a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 shutdown_connections 2024-09-17T15:52:27.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.181+0000 7faeee395640 1 -- 172.21.15.86:0/2403312777 wait complete. 2024-09-17T15:52:27.196 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1623,"stamp":"2024-09-17T15:52:26.049743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":314076,"kb_used_data":6176,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565916964,"statfs":{"total":579820584960,"available":579498971136,"internally_reserved":0,"allocated":6324224,"data_stored":3721246,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.0","version":"14'72","reported_seq":1811,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375474+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:52:11.375474+0000","last_peered":"2024-09-17T15:52:11.375474+0000","last_clean":"2024-09-17T15:52:11.375474+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:11.375474+0000","last_undegraded":"2024-09-17T15:52:11.375474+0000","last_fullsized":"2024-09-17T15:52:11.375474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1725,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375309+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:52:11.375309+0000","last_peered":"2024-09-17T15:52:11.375309+0000","last_clean":"2024-09-17T15:52:11.375309+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:11.375309+0000","last_undegraded":"2024-09-17T15:52:11.375309+0000","last_fullsized":"2024-09-17T15:52:11.375309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:10.899559+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:52:10.899559+0000","last_peered":"2024-09-17T15:52:10.899559+0000","last_clean":"2024-09-17T15:52:10.899559+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:10.899559+0000","last_undegraded":"2024-09-17T15:52:10.899559+0000","last_fullsized":"2024-09-17T15:52:10.899559+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375300+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:52:11.375300+0000","last_peered":"2024-09-17T15:52:11.375300+0000","last_clean":"2024-09-17T15:52:11.375300+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:11.375300+0000","last_undegraded":"2024-09-17T15:52:11.375300+0000","last_fullsized":"2024-09-17T15:52:11.375300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374668+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:52:11.374668+0000","last_peered":"2024-09-17T15:52:11.374668+0000","last_clean":"2024-09-17T15:52:11.374668+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:11.374668+0000","last_undegraded":"2024-09-17T15:52:11.374668+0000","last_fullsized":"2024-09-17T15:52:11.374668+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375366+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:52:11.375366+0000","last_peered":"2024-09-17T15:52:11.375366+0000","last_clean":"2024-09-17T15:52:11.375366+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:11.375366+0000","last_undegraded":"2024-09-17T15:52:11.375366+0000","last_fullsized":"2024-09-17T15:52:11.375366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374688+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:52:11.374688+0000","last_peered":"2024-09-17T15:52:11.374688+0000","last_clean":"2024-09-17T15:52:11.374688+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:11.374688+0000","last_undegraded":"2024-09-17T15:52:11.374688+0000","last_fullsized":"2024-09-17T15:52:11.374688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375404+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:52:11.375404+0000","last_peered":"2024-09-17T15:52:11.375404+0000","last_clean":"2024-09-17T15:52:11.375404+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:11.375404+0000","last_undegraded":"2024-09-17T15:52:11.375404+0000","last_fullsized":"2024-09-17T15:52:11.375404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50304,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321536,"statfs":{"total":96636764160,"available":96585252864,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640559,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46088,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325752,"statfs":{"total":96636764160,"available":96589570048,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:27.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T15:52:27.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 10 monclient: get_monmap_and_config 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 10 monclient: build_initial_monmap 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 10 monclient: monmap: 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:27.318836+0000 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:27.318836+0000 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:27.315 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:27.316 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 1 Processor -- start 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 1 -- start start 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.313+0000 7fd19d5e8640 10 monclient: init 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1980a5070 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:27.318 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1980a5b60 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd19809cf60 0x7fd1981409b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd19809cf60 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1980a6f40 con 0x7fd1980a5070 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1980a65e0 con 0x7fd1980a5b60 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd19809d390 con 0x7fd19809cf60 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd19d5e8640 10 monclient(hunting): authenticate will time out at 2480.284310s 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd196ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd1980a5070 0x7fd1980a5450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56576/0 (socket says 172.21.15.86:56576) 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd196ffd640 1 -- 172.21.15.86:0/99988505 learned_addr learned my addr 172.21.15.86:0/99988505 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 1 --2- 172.21.15.86:0/99988505 >> v2:172.21.15.203:3300/0 conn(0x7fd19809cf60 0x7fd1981409b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): get_auth_request con 0x7fd19809cf60 auth_method 0 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.319 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient(hunting): handle_auth_done global_id 284110 payload 274 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 1 -- 172.21.15.86:0/99988505 >> v1:172.21.15.203:6789/0 conn(0x7fd1980a5b60 legacy=0x7fd19813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 1 -- 172.21.15.86:0/99988505 >> v1:172.21.15.86:6789/0 conn(0x7fd1980a5070 legacy=0x7fd1980a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 1 monclient: found mon.noname-c 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:52:27.320 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.317+0000 7fd1977fe640 1 -- 172.21.15.86:0/99988505 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19809c2a0 con 0x7fd19809cf60 2024-09-17T15:52:27.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd1977fe640 1 -- 172.21.15.86:0/99988505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd19809cf60 msgr2=0x7fd1981409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:27.321 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd1977fe640 1 --2- 172.21.15.86:0/99988505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd19809cf60 0x7fd1981409b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd190009d90 tx=0x7fd19001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7083fc99106b0ae2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 1 -- 172.21.15.86:0/99988505 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1900b2070 con 0x7fd19809cf60 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: dump: 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.322 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 1 -- 172.21.15.86:0/99988505 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1900a5940 con 0x7fd19809cf60 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:27.323 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 1 -- 172.21.15.86:0/99988505 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1900a5d10 con 0x7fd19809cf60 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 5 monclient: authenticate success, global_id 284110 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 4 set_mon_vals no callback set 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd195ffb640 10 monclient: dump: 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.324 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.325 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.325 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 10 monclient: shutdown 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 -- 172.21.15.86:0/99988505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd19809cf60 msgr2=0x7fd1981409b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/99988505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd19809cf60 0x7fd1981409b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd190009d90 tx=0x7fd19001c800 comp rx=0 tx=0).stop 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 -- 172.21.15.86:0/99988505 shutdown_connections 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/99988505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd19809cf60 0x7fd1981409b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 -- 172.21.15.86:0/99988505 >> 172.21.15.86:0/99988505 conn(0x7fd1980742a0 msgr2=0x7fd1980746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 -- 172.21.15.86:0/99988505 shutdown_connections 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 1 -- 172.21.15.86:0/99988505 wait complete. 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 10 monclient: build_initial_monmap 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.321+0000 7fd19d5e8640 10 monclient: monmap: 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:27.329 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 Processor -- start 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 -- start start 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient: init 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-c con 0x7fd19809cf60 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 0x7fd198138200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-b con 0x7fd1980a5070 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 0x7fd19813dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): picked mon.noname-a con 0x7fd1980a5b60 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd198143890 con 0x7fd1980a5b60 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd198143710 con 0x7fd1980a5070 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd198143a10 con 0x7fd19809cf60 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd19d5e8640 10 monclient(hunting): authenticate will time out at 2480.293046s 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 0x7fd19813dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:50776/0 (socket says 172.21.15.86:50776) 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 -- 172.21.15.86:0/53932014 learned_addr learned my addr 172.21.15.86:0/53932014 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): get_auth_request con 0x7fd19809cf60 auth_method 0 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): get_auth_request con 0x7fd1980a5b60 auth_method 0 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1967fc640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 0x7fd198138200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient(hunting): handle_auth_done global_id 274271 payload 824 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 -- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 msgr2=0x7fd198138200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 0x7fd198138200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 -- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 msgr2=0x7fd19813dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 0x7fd19813dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 monclient: found mon.noname-c 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.333 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 -- 172.21.15.86:0/53932014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1980a44d0 con 0x7fd19809cf60 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1967fc640 10 monclient: get_auth_request con 0x7fd1980a5070 auth_method 0 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1977fe640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 0x7fd19813dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd1967fc640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 0x7fd198138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 -- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 msgr2=0x7fd198137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd196ffd640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c470 tx=0x7fd18800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.325+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188007070 con 0x7fd19809cf60 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: dump: 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.334 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd188007210 con 0x7fd19809cf60 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 5 monclient: authenticate success, global_id 274271 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _renew_subs 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.335 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd198121260 con 0x7fd19809cf60 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188007600 con 0x7fd19809cf60 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 10 monclient: dump: 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:27.336 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd1957fa640 4 set_mon_vals no callback set 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _renew_subs 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd198121820 con 0x7fd19809cf60 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.337 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1980a54d0 con 0x7fd19809cf60 2024-09-17T15:52:27.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd18801b400 con 0x7fd19809cf60 2024-09-17T15:52:27.338 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 0x7fd18403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fd18804fb00 con 0x7fd19809cf60 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd1967fc640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 0x7fd18403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.329+0000 7fd1967fc640 10 monclient: get_auth_request con 0x7fd18403bb90 auth_method 0 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.333+0000 7fd1967fc640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 0x7fd18403e050 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd198140c40 tx=0x7fd18c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.333+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fd1980a44d0 con 0x7fd19809cf60 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.333+0000 7fd17f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:27.339 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.333+0000 7fd17f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd19d5e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd19d5e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd19d5e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd198120f80 con 0x7fd19809cf60 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd17f7fe640 1 -- 172.21.15.86:0/53932014 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v876) ==== 74+0+12375 (secure 0 0 0) 0x7fd188010d10 con 0x7fd19809cf60 2024-09-17T15:52:27.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd17f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-17T15:52:27.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.501+0000 7fd17f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:27.504 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:27.504 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":876,"fsid":"b4e90329-4d4c-4253-907f-d2ba7ae38072","created":"2024-09-17T15:27:12.713178+0000","modified":"2024-09-17T15:52:10.893775+0000","last_up_change":"2024-09-17T15:27:23.470675+0000","last_in_change":"2024-09-17T15:27:15.454129+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T15:27:23.674476+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T15:27:24.613917+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09486956-c4be-472a-a7b7-ae1b3e855a71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6801","nonce":3461875681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6803","nonce":3461875681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6807","nonce":3461875681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":3461875681},{"type":"v1","addr":"172.21.15.86:6805","nonce":3461875681}]},"public_addr":"172.21.15.86:6801/3461875681","cluster_addr":"172.21.15.86:6803/3461875681","heartbeat_back_addr":"172.21.15.86:6807/3461875681","heartbeat_front_addr":"172.21.15.86:6805/3461875681","state":["exists","up"]},{"osd":1,"uuid":"c2fa0454-8b28-4acd-8c63-83331707cc17","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6809","nonce":2659992775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6811","nonce":2659992775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6815","nonce":2659992775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2659992775},{"type":"v1","addr":"172.21.15.86:6813","nonce":2659992775}]},"public_addr":"172.21.15.86:6809/2659992775","cluster_addr":"172.21.15.86:6811/2659992775","heartbeat_back_addr":"172.21.15.86:6815/2659992775","heartbeat_front_addr":"172.21.15.86:6813/2659992775","state":["exists","up"]},{"osd":2,"uuid":"7eafb358-7785-40f5-a8a4-dc97082623ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6817","nonce":3359134885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6819","nonce":3359134885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6823","nonce":3359134885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":3359134885},{"type":"v1","addr":"172.21.15.86:6821","nonce":3359134885}]},"public_addr":"172.21.15.86:6817/3359134885","cluster_addr":"172.21.15.86:6819/3359134885","heartbeat_back_addr":"172.21.15.86:6823/3359134885","heartbeat_front_addr":"172.21.15.86:6821/3359134885","state":["exists","up"]},{"osd":3,"uuid":"32ba0b4f-a5b8-4eee-8404-2823bf95ebc3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6809","nonce":2535065757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6811","nonce":2535065757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6815","nonce":2535065757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":2535065757},{"type":"v1","addr":"172.21.15.203:6813","nonce":2535065757}]},"public_addr":"172.21.15.203:6809/2535065757","cluster_addr":"172.21.15.203:6811/2535065757","heartbeat_back_addr":"172.21.15.203:6815/2535065757","heartbeat_front_addr":"172.21.15.203:6813/2535065757","state":["exists","up"]},{"osd":4,"uuid":"b14d69b1-ee94-4f5a-b210-7893cb8790ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6817","nonce":4139704649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6819","nonce":4139704649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6823","nonce":4139704649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":4139704649},{"type":"v1","addr":"172.21.15.203:6821","nonce":4139704649}]},"public_addr":"172.21.15.203:6817/4139704649","cluster_addr":"172.21.15.203:6819/4139704649","heartbeat_back_addr":"172.21.15.203:6823/4139704649","heartbeat_front_addr":"172.21.15.203:6821/4139704649","state":["exists","up"]},{"osd":5,"uuid":"37cddbcf-8bae-470a-97cf-2ef98e2e7fa1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6801","nonce":3502598683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6803","nonce":3502598683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6807","nonce":3502598683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3502598683},{"type":"v1","addr":"172.21.15.203:6805","nonce":3502598683}]},"public_addr":"172.21.15.203:6801/3502598683","cluster_addr":"172.21.15.203:6803/3502598683","heartbeat_back_addr":"172.21.15.203:6807/3502598683","heartbeat_front_addr":"172.21.15.203:6805/3502598683","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.592045+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.760848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.159435+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.733344+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:22.364717+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T15:27:21.514816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":428,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 msgr2=0x7fd18403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 0x7fd18403e050 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd198140c40 tx=0x7fd18c009040 comp rx=0 tx=0).stop 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 10 monclient: shutdown 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 msgr2=0x7fd198137cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c470 tx=0x7fd18800c930 comp rx=0 tx=0).stop 2024-09-17T15:52:27.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 shutdown_connections 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd18403bb90 0x7fd18403e050 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1980a5b60 0x7fd19813dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1980a5070 0x7fd198138200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 --2- 172.21.15.86:0/53932014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd19809cf60 0x7fd198137cc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 >> 172.21.15.86:0/53932014 conn(0x7fd1980742a0 msgr2=0x7fd1980a2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 shutdown_connections 2024-09-17T15:52:27.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:27.505+0000 7fd19d5e8640 1 -- 172.21.15.86:0/53932014 wait complete. 2024-09-17T15:52:28.522 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T15:52:28.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:28.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: get_monmap_and_config 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: build_initial_monmap 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: monmap: 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:28.664626+0000 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:28.664626+0000 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:28.661 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:28.662 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:28.662 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 Processor -- start 2024-09-17T15:52:28.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 -- start start 2024-09-17T15:52:28.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: init 2024-09-17T15:52:28.664 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 0x7f7ed809d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ed80a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ed80a54c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ed813d2f0 0x7f7ed813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ed813d2f0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed809d590 con 0x7f7ed80a54c0 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed80a6ba0 con 0x7f7ed813d2f0 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed8069d50 con 0x7f7ed80a50e0 2024-09-17T15:52:28.665 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edf25a640 10 monclient(hunting): authenticate will time out at 2481.630866s 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ed813d2f0 0x7f7ed813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 0x7f7ed809d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7ecffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7ed80a54c0 0x7f7ed80a58a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36386/0 (socket says 172.21.15.86:36386) 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7ecffff640 1 -- 172.21.15.86:0/425832654 learned_addr learned my addr 172.21.15.86:0/425832654 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): get_auth_request con 0x7f7ed80a50e0 auth_method 0 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): get_auth_request con 0x7f7ed813d2f0 auth_method 0 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient(hunting): handle_auth_done global_id 234660 payload 274 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 -- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 msgr2=0x7f7ed809d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 0x7f7ed809d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 -- 172.21.15.86:0/425832654 >> v1:172.21.15.86:6789/0 conn(0x7f7ed80a54c0 legacy=0x7f7ed80a58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient: _finish_hunting 0 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 monclient: found mon.noname-a 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:28.667 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edd7f7640 1 -- 172.21.15.86:0/425832654 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed809c030 con 0x7f7ed813d2f0 2024-09-17T15:52:28.668 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.661+0000 7f7edcff6640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 0x7f7ed809d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7edd7f7640 1 -- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed813d2f0 msgr2=0x7f7ed813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7edd7f7640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed813d2f0 0x7f7ed813f6e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009d90 tx=0x7f7ed001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4be6c9310a91d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 1 -- 172.21.15.86:0/425832654 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed00ac070 con 0x7f7ed813d2f0 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: dump: 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.671 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 1 -- 172.21.15.86:0/425832654 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed00a6680 con 0x7f7ed813d2f0 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 1 -- 172.21.15.86:0/425832654 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed00a6a50 con 0x7f7ed813d2f0 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7edf25a640 5 monclient: authenticate success, global_id 234660 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7edf25a640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7edf25a640 4 set_mon_vals no callback set 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.665+0000 7f7ecf7fe640 10 monclient: dump: 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.672 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: shutdown 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed813d2f0 msgr2=0x7f7ed813f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed813d2f0 0x7f7ed813f6e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7ed0009d90 tx=0x7f7ed001c830 comp rx=0 tx=0).stop 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- 172.21.15.86:0/425832654 shutdown_connections 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed813d2f0 0x7f7ed813f6e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- 172.21.15.86:0/425832654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a50e0 0x7f7ed809d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.673 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- 172.21.15.86:0/425832654 >> 172.21.15.86:0/425832654 conn(0x7f7ed806ed20 msgr2=0x7f7ed806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- 172.21.15.86:0/425832654 shutdown_connections 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- 172.21.15.86:0/425832654 wait complete. 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: build_initial_monmap 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: monmap: 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 Processor -- start 2024-09-17T15:52:28.674 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- start start 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: init 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed80a50e0 0x7f7ed81351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-a con 0x7f7ed80a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 0x7f7ed8135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-c con 0x7f7ed80a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ed813d2f0 0x7f7ed81cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): picked mon.noname-b con 0x7f7ed813d2f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed81427d0 con 0x7f7ed80a50e0 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed8142650 con 0x7f7ed813d2f0 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed8142950 con 0x7f7ed80a54c0 2024-09-17T15:52:28.675 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edf25a640 10 monclient(hunting): authenticate will time out at 2481.637886s 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed80a50e0 0x7f7ed81351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 0x7f7ed8135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed80a50e0 0x7f7ed81351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51642/0 (socket says 172.21.15.86:51642) 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 1 -- 172.21.15.86:0/1941254181 learned_addr learned my addr 172.21.15.86:0/1941254181 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): get_auth_request con 0x7f7ed80a50e0 auth_method 0 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): get_auth_request con 0x7f7ed80a54c0 auth_method 0 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.676 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7edcff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient(hunting): handle_auth_done global_id 274277 payload 824 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ed813d2f0 msgr2=0x7f7ed81cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ed813d2f0 0x7f7ed81cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed80a50e0 msgr2=0x7f7ed81351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ed80a50e0 0x7f7ed81351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 monclient: found mon.noname-c 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:28.677 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.669+0000 7f7ecffff640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed813b190 con 0x7f7ed80a54c0 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecffff640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 msgr2=0x7f7ed8135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecffff640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 0x7f7ed8135700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800c410 tx=0x7f7ec800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec8017070 con 0x7f7ed80a54c0 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: dump: 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.680 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: _finish_auth 0 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: _check_auth_tickets 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec80070d0 con 0x7f7ed80a54c0 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 5 monclient: authenticate success, global_id 274277 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 10 monclient: _renew_subs 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed813d6d0 con 0x7f7ed80a54c0 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec80074c0 con 0x7f7ed80a54c0 2024-09-17T15:52:28.681 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: dump: 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.682 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 10 monclient: _renew_subs 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed813db90 con 0x7f7ed80a54c0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7eceffd640 4 set_mon_vals no callback set 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ed809d050 con 0x7f7ed80a54c0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f7ec801a070 con 0x7f7ed80a54c0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ec403bb90 0x7f7ec403e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f7ec804f1b0 con 0x7f7ed80a54c0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7f7ec4041a00 0x7f7ec4043e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ec801fe50 con 0x7f7ec4041a00 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7f7ec8056050 con 0x7f7ed80a54c0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7ecd7fa640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edd7f7640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7f7ec4041a00 0x7f7ec4043e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edd7f7640 10 monclient: get_auth_request con 0x7f7ec4041a00 auth_method 0 2024-09-17T15:52:28.683 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edcff6640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ec403bb90 0x7f7ec403e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.673+0000 7f7edcff6640 10 monclient: get_auth_request con 0x7f7ec403bb90 auth_method 0 2024-09-17T15:52:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.677+0000 7f7edcff6640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ec403bb90 0x7f7ec403e050 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7ed813b3a0 tx=0x7f7ed4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.677+0000 7f7edd7f7640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7f7ec4041a00 0x7f7ec4043e20 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.677+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== osd.0 v2:172.21.15.86:6800/3461875681 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ec801fe50 con 0x7f7ec4041a00 2024-09-17T15:52:28.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 --> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7ed813aa80 con 0x7f7ec4041a00 2024-09-17T15:52:28.692 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7ecd7fa640 1 -- 172.21.15.86:0/1941254181 <== osd.0 v2:172.21.15.86:6800/3461875681 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7ed813aa80 con 0x7f7ec4041a00 2024-09-17T15:52:28.693 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:28.693 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:28.693 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7f7ec4041a00 msgr2=0x7f7ec4043e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:6800/3461875681,v1:172.21.15.86:6801/3461875681] conn(0x7f7ec4041a00 0x7f7ec4043e20 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ec403bb90 msgr2=0x7f7ec403e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f7ec403bb90 0x7f7ec403e050 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f7ed813b3a0 tx=0x7f7ed4009190 comp rx=0 tx=0).stop 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 10 monclient: shutdown 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 msgr2=0x7f7ed8135700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 --2- 172.21.15.86:0/1941254181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ed80a54c0 0x7f7ed8135700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7ec800c410 tx=0x7f7ec800c8d0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edd7f7640 1 -- 172.21.15.86:0/1941254181 reap_dead start 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 shutdown_connections 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 >> 172.21.15.86:0/1941254181 conn(0x7f7ed806ed20 msgr2=0x7f7ed813c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 shutdown_connections 2024-09-17T15:52:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.693+0000 7f7edf25a640 1 -- 172.21.15.86:0/1941254181 wait complete. 2024-09-17T15:52:28.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.817+0000 7faead4f9640 10 monclient: get_monmap_and_config 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.817+0000 7faead4f9640 10 monclient: build_initial_monmap 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.817+0000 7faead4f9640 10 monclient: monmap: 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:28.823934+0000 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:28.823934+0000 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:28.820 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:28.821 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:28.821 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 Processor -- start 2024-09-17T15:52:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 -- start start 2024-09-17T15:52:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient: init 2024-09-17T15:52:28.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faea80a4f70 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 0x7faea80a5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-b con 0x7faea80a5350 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faea813d180 0x7faea813f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-a con 0x7faea813d180 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea80a68d0 con 0x7faea80a4f70 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7faea80a64a0 con 0x7faea813d180 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faea809c230 con 0x7faea80a5350 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faead4f9640 10 monclient(hunting): authenticate will time out at 2481.789120s 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 0x7faea80a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faea813d180 0x7faea813f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea6ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faea80a4f70 0x7faea809cef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36402/0 (socket says 172.21.15.86:36402) 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea6ffd640 1 -- 172.21.15.86:0/2453919881 learned_addr learned my addr 172.21.15.86:0/2453919881 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): get_auth_request con 0x7faea80a5350 auth_method 0 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): get_auth_request con 0x7faea813d180 auth_method 0 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient(hunting): handle_auth_done global_id 234666 payload 274 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 -- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 msgr2=0x7faea80a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 0x7faea80a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 -- 172.21.15.86:0/2453919881 >> v1:172.21.15.86:6789/0 conn(0x7faea80a4f70 legacy=0x7faea809cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:28.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea77fe640 1 -- 172.21.15.86:0/2453919881 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea809c510 con 0x7faea813d180 2024-09-17T15:52:28.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.821+0000 7faea67fc640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 0x7faea80a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea77fe640 1 -- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 msgr2=0x7faea813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea77fe640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea813f570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae98009d90 tx=0x7fae9801c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77befb05d7aa64cf server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 1 -- 172.21.15.86:0/2453919881 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980b2070 con 0x7faea813d180 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:28.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: dump: 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:28.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 1 -- 172.21.15.86:0/2453919881 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae980a5870 con 0x7faea813d180 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 5 monclient: authenticate success, global_id 234666 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 1 -- 172.21.15.86:0/2453919881 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980a5c40 con 0x7faea813d180 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 4 set_mon_vals no callback set 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faea5ffb640 10 monclient: dump: 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.829 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 10 monclient: shutdown 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 msgr2=0x7faea813f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea813f570 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae98009d90 tx=0x7fae9801c7d0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- 172.21.15.86:0/2453919881 shutdown_connections 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea813f570 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 --2- 172.21.15.86:0/2453919881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a5350 0x7faea80a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- 172.21.15.86:0/2453919881 >> 172.21.15.86:0/2453919881 conn(0x7faea80a7240 msgr2=0x7faea80a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- 172.21.15.86:0/2453919881 shutdown_connections 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- 172.21.15.86:0/2453919881 wait complete. 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 10 monclient: build_initial_monmap 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 10 monclient: monmap: 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 Processor -- start 2024-09-17T15:52:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 1 -- start start 2024-09-17T15:52:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.825+0000 7faead4f9640 10 monclient: init 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-c con 0x7faea80a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea80a5350 0x7faea8138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-b con 0x7faea80a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): picked mon.noname-a con 0x7faea813d180 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faea8142710 con 0x7faea813d180 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faea8142590 con 0x7faea80a5350 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faea8142890 con 0x7faea80a4f70 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient(hunting): authenticate will time out at 2481.796068s 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51668/0 (socket says 172.21.15.86:51668) 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 -- 172.21.15.86:0/1644187727 learned_addr learned my addr 172.21.15.86:0/1644187727 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): get_auth_request con 0x7faea813d180 auth_method 0 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44186/0 (socket says 172.21.15.86:44186) 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): get_auth_request con 0x7faea80a4f70 auth_method 0 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea67fc640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea80a5350 0x7faea8138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient(hunting): handle_auth_done global_id 234669 payload 824 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 -- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 msgr2=0x7faea81cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 -- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea80a5350 msgr2=0x7faea8138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea80a5350 0x7faea8138030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:28.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 -- 172.21.15.86:0/1644187727 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea80a1210 con 0x7faea813d180 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea6ffd640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 -- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 msgr2=0x7faea809f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea77fe640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fae9801d070 tx=0x7fae980195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980bb020 con 0x7faea813d180 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: dump: 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae980ad850 con 0x7faea813d180 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae980b2040 con 0x7faea813d180 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 5 monclient: authenticate success, global_id 234669 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _renew_subs 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea813d990 con 0x7faea813d180 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faea57fa640 4 set_mon_vals no callback set 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 10 monclient: dump: 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:28.841 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _renew_subs 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea81227e0 con 0x7faea813d180 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea80a5750 con 0x7faea813d180 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.829+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fae980be020 con 0x7faea813d180 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.833+0000 7fae9f7fe640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 0x7fae8803e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.833+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fae980ea450 con 0x7faea813d180 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.833+0000 7faea6ffd640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 0x7fae8803e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.833+0000 7faea6ffd640 10 monclient: get_auth_request con 0x7fae8803bbe0 auth_method 0 2024-09-17T15:52:28.842 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.833+0000 7faea6ffd640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 0x7fae8803e0a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7faea0007ca0 tx=0x7faea0007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:28.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.837+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fae980c5070 con 0x7faea813d180 2024-09-17T15:52:28.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.837+0000 7fae9f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:28.843 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.837+0000 7fae9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:28.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.997+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7faea80a5ff0 con 0x7fae8803bbe0 2024-09-17T15:52:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:28.997+0000 7fae9f7fe640 1 -- 172.21.15.86:0/1644187727 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7faea80a5ff0 con 0x7fae8803bbe0 2024-09-17T15:52:28.998 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 msgr2=0x7fae8803e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 0x7fae8803e0a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7faea0007ca0 tx=0x7faea0007bf0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 10 monclient: shutdown 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 msgr2=0x7faea809f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.000 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fae9801d070 tx=0x7fae980195d0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 shutdown_connections 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fae8803bbe0 0x7fae8803e0a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faea813d180 0x7faea809f750 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea80a5350 0x7faea8138030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 --2- 172.21.15.86:0/1644187727 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faea80a4f70 0x7faea81cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 >> 172.21.15.86:0/1644187727 conn(0x7faea80a7240 msgr2=0x7faea813c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 shutdown_connections 2024-09-17T15:52:29.001 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.001+0000 7faead4f9640 1 -- 172.21.15.86:0/1644187727 wait complete. 2024-09-17T15:52:29.015 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T15:52:29.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: get_monmap_and_config 2024-09-17T15:52:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: monmap: 2024-09-17T15:52:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:29.129291+0000 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:29.129291+0000 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:29.126 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.128 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 Processor -- start 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 -- start start 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: init 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0380a0c00 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff03806f920 0x7ff03813aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-a con 0x7ff03806f920 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0380a16f0 0x7ff03813d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-b con 0x7ff0380a16f0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0380a2130 con 0x7ff0380a0c00 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7ff0380a2410 con 0x7ff03806f920 2024-09-17T15:52:29.129 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff0380a26f0 con 0x7ff0380a16f0 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff03db76640 10 monclient(hunting): authenticate will time out at 2482.094751s 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff03806f920 0x7ff03813aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0380a16f0 0x7ff03813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff0377fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff0380a0c00 0x7ff0380a0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36414/0 (socket says 172.21.15.86:36414) 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff0377fe640 1 -- 172.21.15.86:0/2182335196 learned_addr learned my addr 172.21.15.86:0/2182335196 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): get_auth_request con 0x7ff03806f920 auth_method 0 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): get_auth_request con 0x7ff0380a16f0 auth_method 0 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient(hunting): handle_auth_done global_id 234672 payload 274 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 -- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0380a16f0 msgr2=0x7ff03813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 --2- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0380a16f0 0x7ff03813d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 -- 172.21.15.86:0/2182335196 >> v1:172.21.15.86:6789/0 conn(0x7ff0380a0c00 legacy=0x7ff0380a0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 monclient: found mon.noname-a 2024-09-17T15:52:29.131 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.132 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.125+0000 7ff036ffd640 1 -- 172.21.15.86:0/2182335196 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0380a29d0 con 0x7ff03806f920 2024-09-17T15:52:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff036ffd640 1 -- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff03806f920 msgr2=0x7ff03813aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff036ffd640 1 --2- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff03806f920 0x7ff03813aec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0097c0 tx=0x7ff02c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbab17ae0ca012c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 1 -- 172.21.15.86:0/2182335196 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0b0070 con 0x7ff03806f920 2024-09-17T15:52:29.135 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: dump: 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.136 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 1 -- 172.21.15.86:0/2182335196 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0ab600 con 0x7ff03806f920 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 1 -- 172.21.15.86:0/2182335196 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0ab9d0 con 0x7ff03806f920 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.137 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff0367fc640 10 monclient: dump: 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.138 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff03db76640 5 monclient: authenticate success, global_id 234672 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff03db76640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.129+0000 7ff03db76640 4 set_mon_vals no callback set 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: shutdown 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff03806f920 msgr2=0x7ff03813aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff03806f920 0x7ff03813aec0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0097c0 tx=0x7ff02c01c360 comp rx=0 tx=0).stop 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- 172.21.15.86:0/2182335196 shutdown_connections 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0380a16f0 0x7ff03813d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- 172.21.15.86:0/2182335196 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff03806f920 0x7ff03813aec0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- 172.21.15.86:0/2182335196 >> 172.21.15.86:0/2182335196 conn(0x7ff0380a2d00 msgr2=0x7ff0380a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- 172.21.15.86:0/2182335196 shutdown_connections 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- 172.21.15.86:0/2182335196 wait complete. 2024-09-17T15:52:29.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: monmap: 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 Processor -- start 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- start start 2024-09-17T15:52:29.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: init 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff03806f920 0x7ff038132410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-c con 0x7ff03806f920 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 0x7ff038132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-a con 0x7ff0380a0c00 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0380a16f0 0x7ff038139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): picked mon.noname-b con 0x7ff0380a16f0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7ff038140580 con 0x7ff0380a0c00 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff038140400 con 0x7ff0380a16f0 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7ff038140700 con 0x7ff03806f920 2024-09-17T15:52:29.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff03db76640 10 monclient(hunting): authenticate will time out at 2482.101929s 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff03806f920 0x7ff038132410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 0x7ff038132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 0x7ff038132950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51682/0 (socket says 172.21.15.86:51682) 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff03806f920 0x7ff038132410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44200/0 (socket says 172.21.15.86:44200) 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 -- 172.21.15.86:0/1239229400 learned_addr learned my addr 172.21.15.86:0/1239229400 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): get_auth_request con 0x7ff0380a0c00 auth_method 0 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): get_auth_request con 0x7ff03806f920 auth_method 0 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient(hunting): handle_auth_done global_id 234675 payload 824 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff03806f920 msgr2=0x7ff038132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff03806f920 0x7ff038132410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0380a16f0 msgr2=0x7ff038139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0380a16f0 0x7ff038139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 monclient: found mon.noname-a 2024-09-17T15:52:29.143 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.133+0000 7ff036ffd640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03811cff0 con 0x7ff0380a0c00 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff036ffd640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 msgr2=0x7ff038132950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff036ffd640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 0x7ff038132950 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa10 tx=0x7ff02c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0b9070 con 0x7ff0380a0c00 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:29.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: dump: 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 5 monclient: authenticate success, global_id 234675 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02c0b5690 con 0x7ff0380a0c00 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 10 monclient: _renew_subs 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0b0040 con 0x7ff0380a0c00 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03811d2d0 con 0x7ff0380a0c00 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: dump: 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.148 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff0357fa640 4 set_mon_vals no callback set 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 10 monclient: _renew_subs 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03813b740 con 0x7ff0380a0c00 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0380a0fe0 con 0x7ff0380a0c00 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7ff02c0bc070 con 0x7ff0380a0c00 2024-09-17T15:52:29.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff02403bb60 0x7ff02403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7ff02c0e9f40 con 0x7ff0380a0c00 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7ff0240419d0 0x7ff024043df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff02c001d40 con 0x7ff0240419d0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7ff02c0ec510 con 0x7ff0380a0c00 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff01ffff640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff037fff640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7ff0240419d0 0x7ff024043df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff037fff640 10 monclient: get_auth_request con 0x7ff0240419d0 auth_method 0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff0377fe640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff02403bb60 0x7ff02403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.137+0000 7ff0377fe640 10 monclient: get_auth_request con 0x7ff02403bb60 auth_method 0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.141+0000 7ff0377fe640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff02403bb60 0x7ff02403e020 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff028007ca0 tx=0x7ff028007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.141+0000 7ff037fff640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7ff0240419d0 0x7ff024043df0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.141+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== osd.1 v2:172.21.15.86:6808/2659992775 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff02c001d40 con 0x7ff0240419d0 2024-09-17T15:52:29.155 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.153+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 --> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff038132e90 con 0x7ff0240419d0 2024-09-17T15:52:29.156 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff01ffff640 1 -- 172.21.15.86:0/1239229400 <== osd.1 v2:172.21.15.86:6808/2659992775 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff038132e90 con 0x7ff0240419d0 2024-09-17T15:52:29.156 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:29.156 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:29.156 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7ff0240419d0 msgr2=0x7ff024043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:6808/2659992775,v1:172.21.15.86:6809/2659992775] conn(0x7ff0240419d0 0x7ff024043df0 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff02403bb60 msgr2=0x7ff02403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7ff02403bb60 0x7ff02403e020 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff028007ca0 tx=0x7ff028007bf0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 10 monclient: shutdown 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 msgr2=0x7ff038132950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 --2- 172.21.15.86:0/1239229400 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0380a0c00 0x7ff038132950 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff02c01aa10 tx=0x7ff02c019100 comp rx=0 tx=0).stop 2024-09-17T15:52:29.157 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff037fff640 1 -- 172.21.15.86:0/1239229400 reap_dead start 2024-09-17T15:52:29.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 shutdown_connections 2024-09-17T15:52:29.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 >> 172.21.15.86:0/1239229400 conn(0x7ff0380a2d00 msgr2=0x7ff038135c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 shutdown_connections 2024-09-17T15:52:29.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.157+0000 7ff03db76640 1 -- 172.21.15.86:0/1239229400 wait complete. 2024-09-17T15:52:29.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: get_monmap_and_config 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: monmap: 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:29.276951+0000 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:29.276951+0000 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:29.273 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:29.274 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:29.274 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 Processor -- start 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 -- start start 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: init 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 0x7fe9d409cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d40a4f70 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9d40a5350 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe9d413d180 0x7fe9d413f570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d413d180 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9d409d430 con 0x7fe9d40a5350 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.276 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d409c7f0 con 0x7fe9d413d180 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d40a68d0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9db7a3640 10 monclient(hunting): authenticate will time out at 2482.242380s 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d8d3e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe9d40a5350 0x7fe9d40a5750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36420/0 (socket says 172.21.15.86:36420) 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d8d3e640 1 -- 172.21.15.86:0/1746828049 learned_addr learned my addr 172.21.15.86:0/1746828049 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 --2- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 0x7fe9d409cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): get_auth_request con 0x7fe9d40a4f70 auth_method 0 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d9d40640 1 --2- 172.21.15.86:0/1746828049 >> v2:172.21.15.203:3300/0 conn(0x7fe9d413d180 0x7fe9d413f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.277 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient(hunting): handle_auth_done global_id 274292 payload 274 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 -- 172.21.15.86:0/1746828049 >> v2:172.21.15.203:3300/0 conn(0x7fe9d413d180 msgr2=0x7fe9d413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 --2- 172.21.15.86:0/1746828049 >> v2:172.21.15.203:3300/0 conn(0x7fe9d413d180 0x7fe9d413f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 -- 172.21.15.86:0/1746828049 >> v1:172.21.15.86:6789/0 conn(0x7fe9d40a5350 legacy=0x7fe9d40a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 monclient: found mon.noname-b 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.278 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.273+0000 7fe9d953f640 1 -- 172.21.15.86:0/1746828049 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d40a64a0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9d953f640 1 -- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 msgr2=0x7fe9d409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9d953f640 1 --2- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 0x7fe9d409cef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9d409e420 tx=0x7fe9c401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5534990e8b8807b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 1 -- 172.21.15.86:0/1746828049 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40b0070 con 0x7fe9d40a4f70 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: dump: 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.279 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 1 -- 172.21.15.86:0/1746828049 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c40ab600 con 0x7fe9d40a4f70 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 5 monclient: authenticate success, global_id 274292 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:29.280 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 1 -- 172.21.15.86:0/1746828049 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40ab9d0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 4 set_mon_vals no callback set 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9cbfff640 10 monclient: dump: 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.281 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 10 monclient: shutdown 2024-09-17T15:52:29.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 -- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 msgr2=0x7fe9d409cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 0x7fe9d409cef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9d409e420 tx=0x7fe9c401c330 comp rx=0 tx=0).stop 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 -- 172.21.15.86:0/1746828049 shutdown_connections 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/1746828049 >> v2:172.21.15.203:3300/0 conn(0x7fe9d413d180 0x7fe9d413f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/1746828049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d40a4f70 0x7fe9d409cef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 -- 172.21.15.86:0/1746828049 >> 172.21.15.86:0/1746828049 conn(0x7fe9d40a7240 msgr2=0x7fe9d40a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 -- 172.21.15.86:0/1746828049 shutdown_connections 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.277+0000 7fe9db7a3640 1 -- 172.21.15.86:0/1746828049 wait complete. 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient: monmap: 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:29.286 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 Processor -- start 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 -- start start 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient: init 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-a con 0x7fe9d40a4f70 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 0x7fe9d41d5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-b con 0x7fe9d40a5350 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 0x7fe9d40a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): picked mon.noname-c con 0x7fe9d413d180 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d41424e0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d406f9a0 con 0x7fe9d40a5350 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d4142660 con 0x7fe9d413d180 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9db7a3640 10 monclient(hunting): authenticate will time out at 2482.249254s 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.288 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 0x7fe9d40a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51698/0 (socket says 172.21.15.86:51698) 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 -- 172.21.15.86:0/2208388364 learned_addr learned my addr 172.21.15.86:0/2208388364 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): get_auth_request con 0x7fe9d413d180 auth_method 0 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): get_auth_request con 0x7fe9d40a4f70 auth_method 0 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.289 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient(hunting): handle_auth_done global_id 234678 payload 824 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 -- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 msgr2=0x7fe9d40a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 0x7fe9d40a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d8d3e640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 0x7fe9d41d5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 -- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 msgr2=0x7fe9d41d5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 0x7fe9d41d5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 monclient: found mon.noname-a 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 -- 172.21.15.86:0/2208388364 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d413d810 con 0x7fe9d40a4f70 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d9d40640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 0x7fe9d40a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:29.290 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d8d3e640 10 monclient: get_auth_request con 0x7fe9d40a5350 auth_method 0 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d8d3e640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 0x7fe9d41d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 -- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 msgr2=0x7fe9d41d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.281+0000 7fe9d953f640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40ae040 tx=0x7fe9c40190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40b8020 con 0x7fe9d40a4f70 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: dump: 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.291 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c40b4620 con 0x7fe9d40a4f70 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 5 monclient: authenticate success, global_id 234678 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _renew_subs 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d413dd20 con 0x7fe9d40a4f70 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9c40b0040 con 0x7fe9d40a4f70 2024-09-17T15:52:29.292 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 10 monclient: dump: 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9cb7fe640 4 set_mon_vals no callback set 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _renew_subs 2024-09-17T15:52:29.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d4071300 con 0x7fe9d40a4f70 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d409cef0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fe9c40bb020 con 0x7fe9d40a4f70 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 0x7fe9b803e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fe9c40e9ac0 con 0x7fe9d40a4f70 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9d8d3e640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 0x7fe9b803e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.294 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.285+0000 7fe9d8d3e640 10 monclient: get_auth_request con 0x7fe9b803bbb0 auth_method 0 2024-09-17T15:52:29.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.289+0000 7fe9d8d3e640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 0x7fe9b803e070 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.289+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fe9c40c2030 con 0x7fe9d40a4f70 2024-09-17T15:52:29.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.289+0000 7fe9c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.295 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.289+0000 7fe9c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:29.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.445+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fe9d4073610 con 0x7fe9b803bbb0 2024-09-17T15:52:29.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.445+0000 7fe9c9ffb640 1 -- 172.21.15.86:0/2208388364 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe9d4073610 con 0x7fe9b803bbb0 2024-09-17T15:52:29.446 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 msgr2=0x7fe9b803e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 0x7fe9b803e070 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0007fd0 tx=0x7fe9d0009040 comp rx=0 tx=0).stop 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 10 monclient: shutdown 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 msgr2=0x7fe9d41d4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40ae040 tx=0x7fe9c40190d0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 shutdown_connections 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fe9b803bbb0 0x7fe9b803e070 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d413d180 0x7fe9d40a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9d40a5350 0x7fe9d41d5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 --2- 172.21.15.86:0/2208388364 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d40a4f70 0x7fe9d41d4b00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 >> 172.21.15.86:0/2208388364 conn(0x7fe9d40a7240 msgr2=0x7fe9d41386d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 shutdown_connections 2024-09-17T15:52:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.449+0000 7fe9db7a3640 1 -- 172.21.15.86:0/2208388364 wait complete. 2024-09-17T15:52:29.464 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T15:52:29.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 10 monclient: get_monmap_and_config 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 10 monclient: monmap: 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:29.586956+0000 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:29.586956+0000 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.583 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:29.584 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:29.584 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:29.584 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:29.584 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:29.584 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 1 Processor -- start 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 1 -- start start 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.581+0000 7f47b0994640 10 monclient: init 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f47ac134d20 0x7f47ac135100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-c con 0x7f47ac134d20 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47ac09d700 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47ac09e280 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac09ec30 con 0x7f47ac09d700 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47ac09ef10 con 0x7f47ac09e280 2024-09-17T15:52:29.586 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f47ac09f1f0 con 0x7f47ac134d20 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47b0994640 10 monclient(hunting): authenticate will time out at 2482.552472s 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47a9d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f47ac09d700 0x7f47ac09db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36430/0 (socket says 172.21.15.86:36430) 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3522626257 learned_addr learned my addr 172.21.15.86:0/3522626257 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 1 --2- 172.21.15.86:0/3522626257 >> v2:172.21.15.203:3300/0 conn(0x7f47ac134d20 0x7f47ac135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): get_auth_request con 0x7f47ac134d20 auth_method 0 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.587 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient(hunting): handle_auth_done global_id 284113 payload 274 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 1 -- 172.21.15.86:0/3522626257 >> v1:172.21.15.203:6789/0 conn(0x7f47ac09e280 legacy=0x7f47ac13cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 1 -- 172.21.15.86:0/3522626257 >> v1:172.21.15.86:6789/0 conn(0x7f47ac09d700 legacy=0x7f47ac09db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 1 monclient: found mon.noname-c 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.203:3300/0 2024-09-17T15:52:29.588 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.585+0000 7f47aa59c640 1 -- 172.21.15.86:0/3522626257 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac0a2110 con 0x7f47ac134d20 2024-09-17T15:52:29.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47aa59c640 1 -- 172.21.15.86:0/3522626257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac134d20 msgr2=0x7f47ac135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.589 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47aa59c640 1 --2- 172.21.15.86:0/3522626257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac134d20 0x7f47ac135100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f479c0097c0 tx=0x7f479c01e570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f039efaa16670db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 1 -- 172.21.15.86:0/3522626257 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c0ad600 con 0x7f47ac134d20 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: dump: 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.591 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 1 -- 172.21.15.86:0/3522626257 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f479c0b2070 con 0x7f47ac134d20 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 1 -- 172.21.15.86:0/3522626257 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c0ad9c0 con 0x7f47ac134d20 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47a959a640 10 monclient: dump: 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.592 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 5 monclient: authenticate success, global_id 284113 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 4 set_mon_vals no callback set 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 10 monclient: shutdown 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 -- 172.21.15.86:0/3522626257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac134d20 msgr2=0x7f47ac135100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.593 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 --2- 172.21.15.86:0/3522626257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac134d20 0x7f47ac135100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f479c0097c0 tx=0x7f479c01e570 comp rx=0 tx=0).stop 2024-09-17T15:52:29.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 -- 172.21.15.86:0/3522626257 shutdown_connections 2024-09-17T15:52:29.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 --2- 172.21.15.86:0/3522626257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac134d20 0x7f47ac135100 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 -- 172.21.15.86:0/3522626257 >> 172.21.15.86:0/3522626257 conn(0x7f47ac0a2d00 msgr2=0x7f47ac0a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 -- 172.21.15.86:0/3522626257 shutdown_connections 2024-09-17T15:52:29.594 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 1 -- 172.21.15.86:0/3522626257 wait complete. 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.589+0000 7f47b0994640 10 monclient: monmap: 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:29.598 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 Processor -- start 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- start start 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: init 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac09d700 0x7f47ac13ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-b con 0x7f47ac09d700 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:29.599 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 0x7f47ac133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-a con 0x7f47ac09e280 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47ac134d20 0x7f47ac134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): picked mon.noname-c con 0x7f47ac134d20 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac0a1410 con 0x7f47ac09e280 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47ac0a1590 con 0x7f47ac09d700 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f47ac0a1710 con 0x7f47ac134d20 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient(hunting): authenticate will time out at 2482.560778s 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 0x7f47ac133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 0x7f47ac133c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51710/0 (socket says 172.21.15.86:51710) 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47ac134d20 0x7f47ac134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3969250608 learned_addr learned my addr 172.21.15.86:0/3969250608 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): get_auth_request con 0x7f47ac09e280 auth_method 0 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): get_auth_request con 0x7f47ac134d20 auth_method 0 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aa59c640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac09d700 0x7f47ac13ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aa59c640 10 monclient(hunting): get_auth_request con 0x7f47ac09d700 auth_method 0 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient(hunting): handle_auth_done global_id 234681 payload 824 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47ac134d20 msgr2=0x7f47ac134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47ac134d20 0x7f47ac134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac09d700 msgr2=0x7f47ac13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47ac09d700 0x7f47ac13ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 monclient: found mon.noname-a 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac09d3f0 con 0x7f47ac09e280 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 msgr2=0x7f47ac133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a9d9b640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 0x7f47ac133c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c470 tx=0x7f47a000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0007070 con 0x7f47ac09e280 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: dump: 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.603 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a0007210 con 0x7f47ac09e280 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a0007600 con 0x7f47ac09e280 2024-09-17T15:52:29.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 5 monclient: authenticate success, global_id 234681 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _renew_subs 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47ac13c160 con 0x7f47ac09e280 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f479affd640 10 monclient: dump: 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47a8d99640 4 set_mon_vals no callback set 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _renew_subs 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.593+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47ac13b0c0 con 0x7f47ac09e280 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47b0994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f47ac135100 con 0x7f47ac09e280 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f47a001b400 con 0x7f47ac09e280 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f478c03bb60 0x7f478c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f47a004fb60 con 0x7f47ac09e280 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7f478c0419d0 0x7f478c043df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47a0011e50 con 0x7f478c0419d0 2024-09-17T15:52:29.606 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7f47a00542a0 con 0x7f47ac09e280 2024-09-17T15:52:29.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f479affd640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:29.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47aad9d640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7f478c0419d0 0x7f478c043df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47aa59c640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f478c03bb60 0x7f478c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47aad9d640 10 monclient: get_auth_request con 0x7f478c0419d0 auth_method 0 2024-09-17T15:52:29.607 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.597+0000 7f47aa59c640 10 monclient: get_auth_request con 0x7f478c03bb60 auth_method 0 2024-09-17T15:52:29.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.601+0000 7f47aa59c640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f478c03bb60 0x7f478c03e020 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0a0520 tx=0x7f479c0a9000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.601+0000 7f47aad9d640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7f478c0419d0 0x7f478c043df0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.608 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.601+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== osd.2 v2:172.21.15.86:6816/3359134885 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f47a0011e50 con 0x7f478c0419d0 2024-09-17T15:52:29.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 --> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f47ac1d0290 con 0x7f478c0419d0 2024-09-17T15:52:29.616 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f479affd640 1 -- 172.21.15.86:0/3969250608 <== osd.2 v2:172.21.15.86:6816/3359134885 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f47ac1d0290 con 0x7f478c0419d0 2024-09-17T15:52:29.616 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:29.616 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:29.616 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7f478c0419d0 msgr2=0x7f478c043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:6816/3359134885,v1:172.21.15.86:6817/3359134885] conn(0x7f478c0419d0 0x7f478c043df0 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f478c03bb60 msgr2=0x7f478c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f478c03bb60 0x7f478c03e020 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f47ac0a0520 tx=0x7f479c0a9000 comp rx=0 tx=0).stop 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 10 monclient: shutdown 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 msgr2=0x7f47ac133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 --2- 172.21.15.86:0/3969250608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47ac09e280 0x7f47ac133c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f47a000c470 tx=0x7f47a000c930 comp rx=0 tx=0).stop 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47aad9d640 1 -- 172.21.15.86:0/3969250608 reap_dead start 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 shutdown_connections 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 >> 172.21.15.86:0/3969250608 conn(0x7f47ac0a2d00 msgr2=0x7f47ac0a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 shutdown_connections 2024-09-17T15:52:29.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.617+0000 7f47b0994640 1 -- 172.21.15.86:0/3969250608 wait complete. 2024-09-17T15:52:29.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T15:52:29.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: get_monmap_and_config 2024-09-17T15:52:29.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.744 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: monmap: 2024-09-17T15:52:29.744 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:29.748269+0000 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:29.748269+0000 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:29.745 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 Processor -- start 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 -- start start 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: init 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83f40a0c70 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:29.747 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f413af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-b con 0x7f83f40a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83f413b450 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f40a2130 con 0x7f83f413b450 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f40a26f0 con 0x7f83f40a0c70 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83f40a2410 con 0x7f83f40a1760 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83fb643640 10 monclient(hunting): authenticate will time out at 2482.713719s 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f9be0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f83f413b450 0x7f83f413d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36438/0 (socket says 172.21.15.86:36438) 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f9be0640 1 -- 172.21.15.86:0/1575525790 learned_addr learned my addr 172.21.15.86:0/1575525790 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 1 --2- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f413af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): get_auth_request con 0x7f83f40a1760 auth_method 0 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.748 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient(hunting): handle_auth_done global_id 274301 payload 274 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 1 -- 172.21.15.86:0/1575525790 >> v1:172.21.15.203:6789/0 conn(0x7f83f40a0c70 legacy=0x7f83f40a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 1 -- 172.21.15.86:0/1575525790 >> v1:172.21.15.86:6789/0 conn(0x7f83f413b450 legacy=0x7f83f413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 1 monclient: found mon.noname-b 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.749 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.745+0000 7f83f8bde640 1 -- 172.21.15.86:0/1575525790 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f409f9f0 con 0x7f83f40a1760 2024-09-17T15:52:29.750 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83f8bde640 1 -- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 msgr2=0x7f83f413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83f8bde640 1 --2- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f413af10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83e40097f0 tx=0x7f83e401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=754a246b58134a21 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 1 -- 172.21.15.86:0/1575525790 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e40b0070 con 0x7f83f40a1760 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: dump: 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.751 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 1 -- 172.21.15.86:0/1575525790 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e40ab600 con 0x7f83f40a1760 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 1 -- 172.21.15.86:0/1575525790 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e40ab9d0 con 0x7f83f40a1760 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83ebfff640 10 monclient: dump: 2024-09-17T15:52:29.752 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 5 monclient: authenticate success, global_id 274301 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:29.753 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 4 set_mon_vals no callback set 2024-09-17T15:52:29.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 10 monclient: shutdown 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 -- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 msgr2=0x7f83f413af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 --2- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f413af10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f83e40097f0 tx=0x7f83e401c3a0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 -- 172.21.15.86:0/1575525790 shutdown_connections 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 --2- 172.21.15.86:0/1575525790 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f413af10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 -- 172.21.15.86:0/1575525790 >> 172.21.15.86:0/1575525790 conn(0x7f83f40a2f40 msgr2=0x7f83f40a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 -- 172.21.15.86:0/1575525790 shutdown_connections 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.749+0000 7f83fb643640 1 -- 172.21.15.86:0/1575525790 wait complete. 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient: build_initial_monmap 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient: monmap: 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:29.758 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 Processor -- start 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 -- start start 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient: init 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:52:29.759 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 0x7f83f4132220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-b con 0x7f83f40a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-c con 0x7f83f40a1760 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 0x7f83f409dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): picked mon.noname-a con 0x7f83f413b450 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4141900 con 0x7f83f413b450 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f83f4141780 con 0x7f83f40a0c70 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f83f4141a80 con 0x7f83f40a1760 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:29.760 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83fb643640 10 monclient(hunting): authenticate will time out at 2482.721043s 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 0x7f83f409dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44246/0 (socket says 172.21.15.86:44246) 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 -- 172.21.15.86:0/3582697684 learned_addr learned my addr 172.21.15.86:0/3582697684 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): get_auth_request con 0x7f83f40a1760 auth_method 0 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): get_auth_request con 0x7f83f413b450 auth_method 0 2024-09-17T15:52:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient(hunting): handle_auth_done global_id 274304 payload 824 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f93df640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 0x7f83f4132220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 -- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 msgr2=0x7f83f4132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 0x7f83f4132220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.762 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 -- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 msgr2=0x7f83f409dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 0x7f83f409dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient: _finish_hunting 0 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 monclient: found mon.noname-c 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 -- 172.21.15.86:0/3582697684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f41335c0 con 0x7f83f40a1760 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f93df640 10 monclient: get_auth_request con 0x7f83f40a0c70 auth_method 0 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f9be0640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 0x7f83f409dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f93df640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 0x7f83f4132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 -- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 msgr2=0x7f83f409d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83f8bde640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83e40ae040 tx=0x7f83e401abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e40b0050 con 0x7f83f40a1760 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.753+0000 7f83e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.763 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: dump: 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.764 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e40bc070 con 0x7f83f40a1760 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 5 monclient: authenticate success, global_id 274304 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _renew_subs 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f4132d60 con 0x7f83f40a1760 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e40b5a80 con 0x7f83f40a1760 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 10 monclient: dump: 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:29.765 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _renew_subs 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f413ba30 con 0x7f83f40a1760 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83eb7fe640 4 set_mon_vals no callback set 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f40a1050 con 0x7f83f40a1760 2024-09-17T15:52:29.766 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f83e40b5c60 con 0x7f83f40a1760 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 0x7f83d803e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f83e40ea690 con 0x7f83f40a1760 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83f93df640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 0x7f83d803e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.757+0000 7f83f93df640 10 monclient: get_auth_request con 0x7f83d803bb60 auth_method 0 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.761+0000 7f83f93df640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 0x7f83d803e020 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f83f0003e40 tx=0x7f83f0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.761+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f83e40160b0 con 0x7f83f40a1760 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.761+0000 7f83e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:29.767 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.761+0000 7f83e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:29.933 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.933+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f83f409eb10 con 0x7f83d803bb60 2024-09-17T15:52:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.933+0000 7f83e9ffb640 1 -- 172.21.15.86:0/3582697684 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f83f409eb10 con 0x7f83d803bb60 2024-09-17T15:52:29.934 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T15:52:29.936 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 msgr2=0x7f83d803e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 0x7f83d803e020 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f83f0003e40 tx=0x7f83f0009040 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 10 monclient: shutdown 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 msgr2=0x7f83f409d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f83e40ae040 tx=0x7f83e401abe0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 shutdown_connections 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f83d803bb60 0x7f83d803e020 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83f413b450 0x7f83f409dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83f40a1760 0x7f83f409d770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 --2- 172.21.15.86:0/3582697684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f83f40a0c70 0x7f83f4132220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 >> 172.21.15.86:0/3582697684 conn(0x7f83f40a2f40 msgr2=0x7f83f413de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 shutdown_connections 2024-09-17T15:52:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:29.937+0000 7f83fb643640 1 -- 172.21.15.86:0/3582697684 wait complete. 2024-09-17T15:52:29.952 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T15:52:29.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 10 monclient: get_monmap_and_config 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 10 monclient: monmap: 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:30.075026+0000 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:30.075026+0000 2024-09-17T15:52:30.071 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:30.072 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 1 Processor -- start 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 1 -- start start 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.069+0000 7faadef75640 10 monclient: init 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faad80a0c00 0x7faad80a0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): picked mon.noname-a con 0x7faad80a0c00 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:30.074 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faad806f920 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faad80a1520 0x7faad81360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): picked mon.noname-c con 0x7faad80a1520 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad80a2410 con 0x7faad806f920 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7faad80a21c0 con 0x7faad80a0c00 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7faad80a26f0 con 0x7faad80a1520 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadef75640 10 monclient(hunting): authenticate will time out at 2483.040481s 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faad80a0c00 0x7faad80a0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faad7fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faad806f920 0x7faad81359e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36442/0 (socket says 172.21.15.86:36442) 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faad7fff640 1 -- 172.21.15.86:0/787753490 learned_addr learned my addr 172.21.15.86:0/787753490 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:30.075 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): get_auth_request con 0x7faad80a0c00 auth_method 0 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient(hunting): handle_auth_done global_id 234687 payload 274 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 -- 172.21.15.86:0/787753490 >> v2:172.21.15.203:3300/0 conn(0x7faad80a1520 msgr2=0x7faad81360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 --2- 172.21.15.86:0/787753490 >> v2:172.21.15.203:3300/0 conn(0x7faad80a1520 0x7faad81360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 -- 172.21.15.86:0/787753490 >> v1:172.21.15.86:6789/0 conn(0x7faad806f920 legacy=0x7faad81359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 monclient: found mon.noname-a 2024-09-17T15:52:30.076 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.077 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.073+0000 7faadcd11640 1 -- 172.21.15.86:0/787753490 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad80a29d0 con 0x7faad80a0c00 2024-09-17T15:52:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadcd11640 1 -- 172.21.15.86:0/787753490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a0c00 msgr2=0x7faad80a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.078 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadcd11640 1 --2- 172.21.15.86:0/787753490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a0c00 0x7faad80a0fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7faad809fc00 tx=0x7faac801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f778cb48d8f4ee15 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 1 -- 172.21.15.86:0/787753490 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac80b0070 con 0x7faad80a0c00 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: dump: 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.079 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 1 -- 172.21.15.86:0/787753490 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac80ab600 con 0x7faad80a0c00 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 5 monclient: authenticate success, global_id 234687 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 1 -- 172.21.15.86:0/787753490 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac80ab9d0 con 0x7faad80a0c00 2024-09-17T15:52:30.080 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 4 set_mon_vals no callback set 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad77fe640 10 monclient: dump: 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.081 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: shutdown 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- 172.21.15.86:0/787753490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a0c00 msgr2=0x7faad80a0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- 172.21.15.86:0/787753490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a0c00 0x7faad80a0fe0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7faad809fc00 tx=0x7faac801c340 comp rx=0 tx=0).stop 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- 172.21.15.86:0/787753490 shutdown_connections 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- 172.21.15.86:0/787753490 >> v2:172.21.15.203:3300/0 conn(0x7faad80a1520 0x7faad81360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- 172.21.15.86:0/787753490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a0c00 0x7faad80a0fe0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- 172.21.15.86:0/787753490 >> 172.21.15.86:0/787753490 conn(0x7faad80a2d00 msgr2=0x7faad80a30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- 172.21.15.86:0/787753490 shutdown_connections 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- 172.21.15.86:0/787753490 wait complete. 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: monmap: 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.082 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 Processor -- start 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- start start 2024-09-17T15:52:30.083 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: init 2024-09-17T15:52:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:52:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad806f920 0x7faad81cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.087 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): picked mon.noname-b con 0x7faad806f920 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad80a0c00 0x7faad81358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): picked mon.noname-c con 0x7faad80a0c00 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 0x7faad811c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): picked mon.noname-a con 0x7faad80a1520 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faad81367b0 con 0x7faad80a1520 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faad8136630 con 0x7faad806f920 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faad8140700 con 0x7faad80a0c00 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadef75640 10 monclient(hunting): authenticate will time out at 2483.047640s 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 0x7faad811c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.088 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad80a0c00 0x7faad81358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 0x7faad811c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51728/0 (socket says 172.21.15.86:51728) 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 learned_addr learned my addr 172.21.15.86:0/2861467490 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): get_auth_request con 0x7faad80a1520 auth_method 0 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): get_auth_request con 0x7faad80a0c00 auth_method 0 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.089 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faadd512640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.077+0000 7faad7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadcd11640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad806f920 0x7faad81cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 10 monclient(hunting): handle_auth_done global_id 234690 payload 824 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad80a0c00 msgr2=0x7faad81358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad80a0c00 0x7faad81358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad806f920 msgr2=0x7faad81cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad806f920 0x7faad81cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 monclient: found mon.noname-a 2024-09-17T15:52:30.090 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad809d3f0 con 0x7faad80a1520 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadcd11640 10 monclient: get_auth_request con 0x7faad806f920 auth_method 0 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad7fff640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faad80a0c00 0x7faad81358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadcd11640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faad806f920 0x7faad81cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 msgr2=0x7faad811c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadd512640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 0x7faad811c850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faad000b970 tx=0x7faad000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0019070 con 0x7faad80a1520 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: dump: 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 5 monclient: authenticate success, global_id 234690 2024-09-17T15:52:30.092 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 10 monclient: _renew_subs 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad000cd70 con 0x7faad80a1520 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad811df30 con 0x7faad80a1520 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0014680 con 0x7faad80a1520 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: dump: 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.093 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 10 monclient: _renew_subs 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad6ffd640 4 set_mon_vals no callback set 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad811ea90 con 0x7faad80a1520 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faad80a0fe0 con 0x7faad80a1520 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faad001c070 con 0x7faad80a1520 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faac403bb60 0x7faac403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.094 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7faad004fb00 con 0x7faad80a1520 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7faac40419d0 0x7faac4043df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faac40444c0 con 0x7faac40419d0 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7faad00542a0 con 0x7faad80a1520 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.081+0000 7faad57fa640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.085+0000 7faadcd11640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faac403bb60 0x7faac403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.085+0000 7faadcd11640 10 monclient: get_auth_request con 0x7faac403bb60 auth_method 0 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.085+0000 7faad7fff640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7faac40419d0 0x7faac4043df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.085+0000 7faad7fff640 10 monclient: get_auth_request con 0x7faac40419d0 auth_method 0 2024-09-17T15:52:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.085+0000 7faadcd11640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faac403bb60 0x7faac403e020 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faac80ab000 tx=0x7faac8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.089+0000 7faad7fff640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7faac40419d0 0x7faac4043df0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.096 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.089+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== osd.3 v2:172.21.15.203:6808/2535065757 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faac40444c0 con 0x7faac40419d0 2024-09-17T15:52:30.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.101+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 --> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faad81ce860 con 0x7faac40419d0 2024-09-17T15:52:30.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faad57fa640 1 -- 172.21.15.86:0/2861467490 <== osd.3 v2:172.21.15.203:6808/2535065757 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faad81ce860 con 0x7faac40419d0 2024-09-17T15:52:30.104 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:30.104 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:30.104 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7faac40419d0 msgr2=0x7faac4043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6808/2535065757,v1:172.21.15.203:6809/2535065757] conn(0x7faac40419d0 0x7faac4043df0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faac403bb60 msgr2=0x7faac403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faac403bb60 0x7faac403e020 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faac80ab000 tx=0x7faac8008040 comp rx=0 tx=0).stop 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 10 monclient: shutdown 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 msgr2=0x7faad811c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 --2- 172.21.15.86:0/2861467490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faad80a1520 0x7faad811c850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faad000b970 tx=0x7faad000be30 comp rx=0 tx=0).stop 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadd512640 1 -- 172.21.15.86:0/2861467490 reap_dead start 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 shutdown_connections 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 >> 172.21.15.86:0/2861467490 conn(0x7faad80a2d00 msgr2=0x7faad8071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 shutdown_connections 2024-09-17T15:52:30.106 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.105+0000 7faadef75640 1 -- 172.21.15.86:0/2861467490 wait complete. 2024-09-17T15:52:30.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T15:52:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: get_monmap_and_config 2024-09-17T15:52:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: monmap: 2024-09-17T15:52:30.229 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:30.233414+0000 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:30.233414+0000 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:30.230 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 Processor -- start 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 -- start start 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: init 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 0x7f0a7409f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a740a01a0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a740a0580 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a740a1380 0x7f0a7413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a740a1380 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a74074e00 con 0x7f0a740a0580 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0a7409e350 con 0x7f0a740a1380 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0a7409e630 con 0x7f0a740a01a0 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a796a1640 10 monclient(hunting): authenticate will time out at 2483.199378s 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a740a1380 0x7f0a7413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a740a1380 0x7f0a7413ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51740/0 (socket says 172.21.15.86:51740) 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 -- 172.21.15.86:0/3985926798 learned_addr learned my addr 172.21.15.86:0/3985926798 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): get_auth_request con 0x7f0a740a1380 auth_method 0 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a72ffd640 1 --2- 172.21.15.86:0/3985926798 >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 0x7f0a7409f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient(hunting): handle_auth_done global_id 234693 payload 274 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 -- 172.21.15.86:0/3985926798 >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 msgr2=0x7f0a7409f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 --2- 172.21.15.86:0/3985926798 >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 0x7f0a7409f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 -- 172.21.15.86:0/3985926798 >> v1:172.21.15.203:6789/0 conn(0x7f0a740a0580 legacy=0x7f0a740a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a737fe640 1 -- 172.21.15.86:0/3985926798 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7409e910 con 0x7f0a740a1380 2024-09-17T15:52:30.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a72ffd640 10 monclient: get_auth_request con 0x7f0a740a01a0 auth_method 0 2024-09-17T15:52:30.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.229+0000 7f0a72ffd640 1 --2- 172.21.15.86:0/3985926798 >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 0x7f0a7409f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 1 -- 172.21.15.86:0/3985926798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a1380 msgr2=0x7f0a7413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 1 --2- 172.21.15.86:0/3985926798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a1380 0x7f0a7413ba80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009870 tx=0x7f0a6c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0eef63ecd17dbd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 1 -- 172.21.15.86:0/3985926798 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0b1070 con 0x7f0a740a1380 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: dump: 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.237 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 1 -- 172.21.15.86:0/3985926798 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a6c0ac620 con 0x7f0a740a1380 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 5 monclient: authenticate success, global_id 234693 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 1 -- 172.21.15.86:0/3985926798 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0ac9f0 con 0x7f0a740a1380 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 4 set_mon_vals no callback set 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a71ffb640 10 monclient: dump: 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: shutdown 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- 172.21.15.86:0/3985926798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a1380 msgr2=0x7f0a7413ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- 172.21.15.86:0/3985926798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a1380 0x7f0a7413ba80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009870 tx=0x7f0a6c01c3e0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- 172.21.15.86:0/3985926798 shutdown_connections 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- 172.21.15.86:0/3985926798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a1380 0x7f0a7413ba80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- 172.21.15.86:0/3985926798 >> v2:172.21.15.203:3300/0 conn(0x7f0a740a01a0 0x7f0a7409f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- 172.21.15.86:0/3985926798 >> 172.21.15.86:0/3985926798 conn(0x7f0a740a2f40 msgr2=0x7f0a740a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- 172.21.15.86:0/3985926798 shutdown_connections 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- 172.21.15.86:0/3985926798 wait complete. 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: monmap: 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:30.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 Processor -- start 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- start start 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: init 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-a con 0x7f0a740a01a0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-c con 0x7f0a740a0580 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a740a1380 0x7f0a7413b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): picked mon.noname-b con 0x7f0a740a1380 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7409ce40 con 0x7f0a740a01a0 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7409ccc0 con 0x7f0a740a1380 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7409cfc0 con 0x7f0a740a0580 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a796a1640 10 monclient(hunting): authenticate will time out at 2483.207391s 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44258/0 (socket says 172.21.15.86:44258) 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 -- 172.21.15.86:0/2441578495 learned_addr learned my addr 172.21.15.86:0/2441578495 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): get_auth_request con 0x7f0a740a0580 auth_method 0 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51750/0 (socket says 172.21.15.86:51750) 2024-09-17T15:52:30.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): get_auth_request con 0x7f0a740a01a0 auth_method 0 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a740a1380 0x7f0a7413b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 10 monclient(hunting): get_auth_request con 0x7f0a740a1380 auth_method 0 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient(hunting): handle_auth_done global_id 274310 payload 824 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 -- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a740a1380 msgr2=0x7f0a7413b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a740a1380 0x7f0a7413b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 -- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 msgr2=0x7f0a741cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 monclient: found mon.noname-c 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a727fc640 1 -- 172.21.15.86:0/2441578495 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a741d0180 con 0x7f0a740a0580 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.237+0000 7f0a72ffd640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a727fc640 1 -- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 msgr2=0x7f0a7413af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a727fc640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c410 tx=0x7f0a6800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a680119a0 con 0x7f0a740a0580 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: dump: 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.250 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a68017070 con 0x7f0a740a0580 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 5 monclient: authenticate success, global_id 274310 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68011d80 con 0x7f0a740a0580 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _renew_subs 2024-09-17T15:52:30.251 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a741265f0 con 0x7f0a740a0580 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a717fa640 4 set_mon_vals no callback set 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 10 monclient: dump: 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.252 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _renew_subs 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a74128390 con 0x7f0a740a0580 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7409f010 con 0x7f0a740a0580 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0a6801c460 con 0x7f0a740a0580 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 0x7f0a6003e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.241+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f0a6804fa50 con 0x7f0a740a0580 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.245+0000 7f0a72ffd640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 0x7f0a6003e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.245+0000 7f0a72ffd640 10 monclient: get_auth_request con 0x7f0a6003bb60 auth_method 0 2024-09-17T15:52:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.245+0000 7f0a72ffd640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 0x7f0a6003e020 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0a7413bec0 tx=0x7f0a64007810 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.249+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0a68021020 con 0x7f0a740a0580 2024-09-17T15:52:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.249+0000 7f0a5b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.254 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.249+0000 7f0a5b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:30.404 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f0a7406d4f0 con 0x7f0a6003bb60 2024-09-17T15:52:30.405 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a5b7fe640 1 -- 172.21.15.86:0/2441578495 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0a7406d4f0 con 0x7f0a6003bb60 2024-09-17T15:52:30.405 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T15:52:30.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 msgr2=0x7f0a6003e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 0x7f0a6003e020 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f0a7413bec0 tx=0x7f0a64007810 comp rx=0 tx=0).stop 2024-09-17T15:52:30.407 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 10 monclient: shutdown 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 msgr2=0x7f0a7413af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0a6800c410 tx=0x7f0a6800c8d0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 shutdown_connections 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0a6003bb60 0x7f0a6003e020 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a740a1380 0x7f0a7413b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a740a0580 0x7f0a7413af90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 --2- 172.21.15.86:0/2441578495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a740a01a0 0x7f0a741cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 >> 172.21.15.86:0/2441578495 conn(0x7f0a740a2f40 msgr2=0x7f0a740a5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 shutdown_connections 2024-09-17T15:52:30.408 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.405+0000 7f0a796a1640 1 -- 172.21.15.86:0/2441578495 wait complete. 2024-09-17T15:52:30.422 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T15:52:30.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:30.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: get_monmap_and_config 2024-09-17T15:52:30.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: monmap: 2024-09-17T15:52:30.541 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.541 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:30.545279+0000 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:30.545279+0000 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:30.542 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 Processor -- start 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 -- start start 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: init 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f23c40a50e0 0x7f23c409d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-c con 0x7f23c40a50e0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-b con 0x7f23c40a54c0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23c4137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:30.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23c409d590 con 0x7f23c4137ea0 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f23c40a6ba0 con 0x7f23c40a50e0 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f23c4069d50 con 0x7f23c40a54c0 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23cad53640 10 monclient(hunting): authenticate will time out at 2483.510757s 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44260/0 (socket says 172.21.15.86:44260) 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 -- 172.21.15.86:0/2119928735 learned_addr learned my addr 172.21.15.86:0/2119928735 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): get_auth_request con 0x7f23c40a54c0 auth_method 0 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient(hunting): handle_auth_done global_id 274313 payload 274 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 -- 172.21.15.86:0/2119928735 >> v2:172.21.15.203:3300/0 conn(0x7f23c40a50e0 msgr2=0x7f23c409d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c8aef640 1 --2- 172.21.15.86:0/2119928735 >> v2:172.21.15.203:3300/0 conn(0x7f23c40a50e0 0x7f23c409d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 --2- 172.21.15.86:0/2119928735 >> v2:172.21.15.203:3300/0 conn(0x7f23c40a50e0 0x7f23c409d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 -- 172.21.15.86:0/2119928735 >> v1:172.21.15.203:6789/0 conn(0x7f23c4137ea0 legacy=0x7f23c4138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 monclient: found mon.noname-b 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.541+0000 7f23c3fff640 1 -- 172.21.15.86:0/2119928735 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c409c030 con 0x7f23c40a54c0 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.545+0000 7f23c3fff640 1 -- 172.21.15.86:0/2119928735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 msgr2=0x7f23c40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.545+0000 7f23c3fff640 1 --2- 172.21.15.86:0/2119928735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d7b3e8a96a29bab server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 1 -- 172.21.15.86:0/2119928735 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0b0070 con 0x7f23c40a54c0 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:30.549 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: dump: 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.550 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 1 -- 172.21.15.86:0/2119928735 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ac0ab600 con 0x7f23c40a54c0 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 1 -- 172.21.15.86:0/2119928735 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0ab9d0 con 0x7f23c40a54c0 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 5 monclient: authenticate success, global_id 274313 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 4 set_mon_vals no callback set 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c37fe640 10 monclient: dump: 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.551 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: shutdown 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- 172.21.15.86:0/2119928735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 msgr2=0x7f23c40a58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- 172.21.15.86:0/2119928735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac01c420 comp rx=0 tx=0).stop 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- 172.21.15.86:0/2119928735 shutdown_connections 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- 172.21.15.86:0/2119928735 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a54c0 0x7f23c40a58d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- 172.21.15.86:0/2119928735 >> v2:172.21.15.203:3300/0 conn(0x7f23c40a50e0 0x7f23c409d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- 172.21.15.86:0/2119928735 >> 172.21.15.86:0/2119928735 conn(0x7f23c406ed20 msgr2=0x7f23c406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- 172.21.15.86:0/2119928735 shutdown_connections 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- 172.21.15.86:0/2119928735 wait complete. 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: monmap: 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:30.553 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:30.554 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 Processor -- start 2024-09-17T15:52:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- start start 2024-09-17T15:52:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: init 2024-09-17T15:52:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 0x7f23c41d0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-c con 0x7f23c40a50e0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23c40a54c0 0x7f23c4137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-b con 0x7f23c40a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23c4137ea0 0x7f23c413c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): picked mon.noname-a con 0x7f23c4137ea0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f23c4142cb0 con 0x7f23c4137ea0 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23c4142b30 con 0x7f23c40a54c0 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f23c4142e30 con 0x7f23c40a50e0 2024-09-17T15:52:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23cad53640 10 monclient(hunting): authenticate will time out at 2483.518741s 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 0x7f23c41d0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 0x7f23c41d0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44270/0 (socket says 172.21.15.86:44270) 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 -- 172.21.15.86:0/1086850880 learned_addr learned my addr 172.21.15.86:0/1086850880 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): get_auth_request con 0x7f23c40a50e0 auth_method 0 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23c4137ea0 0x7f23c413c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): get_auth_request con 0x7f23c4137ea0 auth_method 0 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.561 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c3fff640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23c40a54c0 0x7f23c4137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c3fff640 10 monclient(hunting): get_auth_request con 0x7f23c40a54c0 auth_method 0 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient(hunting): handle_auth_done global_id 274316 payload 824 2024-09-17T15:52:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23c40a54c0 msgr2=0x7f23c4137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23c40a54c0 0x7f23c4137020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23c4137ea0 msgr2=0x7f23c413c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23c4137ea0 0x7f23c413c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 monclient: found mon.noname-c 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c8aef640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23c40a9e90 con 0x7f23c40a50e0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.549+0000 7f23c92f0640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23c4137ea0 0x7f23c413c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c8aef640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 msgr2=0x7f23c41d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c8aef640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 0x7f23c41d0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c7f0 tx=0x7f23bc00ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc015070 con 0x7f23c40a50e0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: dump: 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23bc0079d0 con 0x7f23c40a50e0 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 5 monclient: authenticate success, global_id 274316 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 10 monclient: _renew_subs 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23c41216d0 con 0x7f23c40a50e0 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc007dc0 con 0x7f23c40a50e0 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: dump: 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c27fc640 4 set_mon_vals no callback set 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 10 monclient: _renew_subs 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23c41d0ff0 con 0x7f23c40a50e0 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f23c40a1090 con 0x7f23c40a50e0 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f23bc018070 con 0x7f23c40a50e0 2024-09-17T15:52:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f23a403bbb0 0x7f23a403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f23bc04fd70 con 0x7f23c40a50e0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f23a4041a30 0x7f23a4043e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23bc014e50 con 0x7f23a4041a30 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7f23bc055070 con 0x7f23c40a50e0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c17fa640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c3fff640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f23a403bbb0 0x7f23a403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c3fff640 10 monclient: get_auth_request con 0x7f23a403bbb0 auth_method 0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c92f0640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f23a4041a30 0x7f23a4043e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.553+0000 7f23c92f0640 10 monclient: get_auth_request con 0x7f23a4041a30 auth_method 0 2024-09-17T15:52:30.567 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.557+0000 7f23c3fff640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f23a403bbb0 0x7f23a403e070 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.561+0000 7f23c92f0640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f23a4041a30 0x7f23a4043e50 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.568 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.561+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== osd.4 v2:172.21.15.203:6816/4139704649 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f23bc014e50 con 0x7f23a4041a30 2024-09-17T15:52:30.574 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 --> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f23c409d050 con 0x7f23a4041a30 2024-09-17T15:52:30.575 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23c17fa640 1 -- 172.21.15.86:0/1086850880 <== osd.4 v2:172.21.15.203:6816/4139704649 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f23c409d050 con 0x7f23a4041a30 2024-09-17T15:52:30.575 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:30.575 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:30.576 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:30.576 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f23a4041a30 msgr2=0x7f23a4043e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6816/4139704649,v1:172.21.15.203:6817/4139704649] conn(0x7f23a4041a30 0x7f23a4043e50 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f23a403bbb0 msgr2=0x7f23a403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f23a403bbb0 0x7f23a403e070 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f23ac009870 tx=0x7f23ac008040 comp rx=0 tx=0).stop 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 10 monclient: shutdown 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 msgr2=0x7f23c41d0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 --2- 172.21.15.86:0/1086850880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23c40a50e0 0x7f23c41d0170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00c7f0 tx=0x7f23bc00ccb0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23c92f0640 1 -- 172.21.15.86:0/1086850880 reap_dead start 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 shutdown_connections 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 >> 172.21.15.86:0/1086850880 conn(0x7f23c406ed20 msgr2=0x7f23c40a8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 shutdown_connections 2024-09-17T15:52:30.577 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.573+0000 7f23cad53640 1 -- 172.21.15.86:0/1086850880 wait complete. 2024-09-17T15:52:30.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T15:52:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: get_monmap_and_config 2024-09-17T15:52:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: monmap: 2024-09-17T15:52:30.690 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:30.694409+0000 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:30.694409+0000 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:30.691 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:30.692 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 Processor -- start 2024-09-17T15:52:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 -- start start 2024-09-17T15:52:30.693 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: init 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 0x7fa83409db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-b con 0x7fa83409d770 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa83409e090 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa834135150 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83409ff00 con 0x7fa83409e090 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83409ec30 con 0x7fa834135150 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa83409e7f0 con 0x7fa83409d770 2024-09-17T15:52:30.694 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa83bb29640 10 monclient(hunting): authenticate will time out at 2483.659748s 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8398c5640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 0x7fa83409db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8390c4640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa83409e090 0x7fa834134a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36448/0 (socket says 172.21.15.86:36448) 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8390c4640 1 -- 172.21.15.86:0/4221062397 learned_addr learned my addr 172.21.15.86:0/4221062397 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8398c5640 10 monclient(hunting): get_auth_request con 0x7fa83409d770 auth_method 0 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8398c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8398c5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.689+0000 7fa8398c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.695 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient(hunting): handle_auth_done global_id 274319 payload 274 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 1 -- 172.21.15.86:0/4221062397 >> v1:172.21.15.203:6789/0 conn(0x7fa834135150 legacy=0x7fa83413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 1 -- 172.21.15.86:0/4221062397 >> v1:172.21.15.86:6789/0 conn(0x7fa83409e090 legacy=0x7fa834134a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 1 monclient: found mon.noname-b 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.693+0000 7fa8398c5640 1 -- 172.21.15.86:0/4221062397 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83409ef10 con 0x7fa83409d770 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 -- 172.21.15.86:0/4221062397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 msgr2=0x7fa83409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 --2- 172.21.15.86:0/4221062397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 0x7fa83409db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0097f0 tx=0x7fa82c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee6b5dd0dc0ed81c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 1 -- 172.21.15.86:0/4221062397 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0b0070 con 0x7fa83409d770 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: dump: 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.703 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 5 monclient: authenticate success, global_id 274319 2024-09-17T15:52:30.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 1 -- 172.21.15.86:0/4221062397 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c0ab600 con 0x7fa83409d770 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 4 set_mon_vals no callback set 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 1 -- 172.21.15.86:0/4221062397 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0ab9d0 con 0x7fa83409d770 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8388c3640 10 monclient: dump: 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.705 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: shutdown 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.706 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- 172.21.15.86:0/4221062397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 msgr2=0x7fa83409db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 --2- 172.21.15.86:0/4221062397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 0x7fa83409db50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0097f0 tx=0x7fa82c01c3a0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- 172.21.15.86:0/4221062397 shutdown_connections 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 --2- 172.21.15.86:0/4221062397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409d770 0x7fa83409db50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- 172.21.15.86:0/4221062397 >> 172.21.15.86:0/4221062397 conn(0x7fa8340a2f40 msgr2=0x7fa8340a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- 172.21.15.86:0/4221062397 shutdown_connections 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- 172.21.15.86:0/4221062397 wait complete. 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: build_initial_monmap 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: monmap: 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:30.707 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 Processor -- start 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- start start 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: init 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:30.708 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-a con 0x7fa83409d770 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-c con 0x7fa83409e090 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa834135150 0x7fa83411c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): picked mon.noname-b con 0x7fa834135150 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fa83409ce60 con 0x7fa83409d770 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa83409cce0 con 0x7fa834135150 2024-09-17T15:52:30.709 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fa834140e60 con 0x7fa83409e090 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83bb29640 10 monclient(hunting): authenticate will time out at 2483.666511s 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51766/0 (socket says 172.21.15.86:51766) 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44290/0 (socket says 172.21.15.86:44290) 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 -- 172.21.15.86:0/335961853 learned_addr learned my addr 172.21.15.86:0/335961853 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): get_auth_request con 0x7fa83409d770 auth_method 0 2024-09-17T15:52:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): get_auth_request con 0x7fa83409e090 auth_method 0 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:30.711 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient(hunting): handle_auth_done global_id 234702 payload 824 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 -- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 msgr2=0x7fa834133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa83a0c6640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa834135150 0x7fa83411c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 -- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa834135150 msgr2=0x7fa83411c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa834135150 0x7fa83411c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient: _finish_hunting 0 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 monclient: found mon.noname-a 2024-09-17T15:52:30.712 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8398c5640 1 -- 172.21.15.86:0/335961853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83409d460 con 0x7fa83409d770 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.697+0000 7fa8390c4640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8398c5640 1 -- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 msgr2=0x7fa8341d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8398c5640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa82c01c1d0 tx=0x7fa82c019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0b9070 con 0x7fa83409d770 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: dump: 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.713 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.714 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: _finish_auth 0 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: _check_auth_tickets 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 5 monclient: authenticate success, global_id 234702 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c0b5690 con 0x7fa83409d770 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _renew_subs 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0b0040 con 0x7fa83409d770 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83411fd60 con 0x7fa83409d770 2024-09-17T15:52:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 10 monclient: dump: 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:30.716 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa833fff640 4 set_mon_vals no callback set 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _renew_subs 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa834120320 con 0x7fa83409d770 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83411f290 con 0x7fa83409d770 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fa82c0bc070 con 0x7fa83409d770 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 0x7fa82003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:30.717 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fa82c0ea230 con 0x7fa83409d770 2024-09-17T15:52:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8390c4640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 0x7fa82003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.701+0000 7fa8390c4640 10 monclient: get_auth_request con 0x7fa82003bb90 auth_method 0 2024-09-17T15:52:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.705+0000 7fa8390c4640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 0x7fa82003e050 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa828007ca0 tx=0x7fa828007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.705+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7fa82c0160b0 con 0x7fa83409d770 2024-09-17T15:52:30.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.705+0000 7fa8327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:30.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.705+0000 7fa8327fc640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:30.866 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.865+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fa83409db50 con 0x7fa82003bb90 2024-09-17T15:52:30.867 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.865+0000 7fa8327fc640 1 -- 172.21.15.86:0/335961853 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa83409db50 con 0x7fa82003bb90 2024-09-17T15:52:30.867 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T15:52:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 msgr2=0x7fa82003e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 0x7fa82003e050 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa828007ca0 tx=0x7fa828007bf0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 10 monclient: shutdown 2024-09-17T15:52:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:30.869 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 msgr2=0x7fa8341d0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa82c01c1d0 tx=0x7fa82c019140 comp rx=0 tx=0).stop 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 shutdown_connections 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fa82003bb90 0x7fa82003e050 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa834135150 0x7fa83411c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa83409e090 0x7fa834133e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 --2- 172.21.15.86:0/335961853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa83409d770 0x7fa8341d0310 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 >> 172.21.15.86:0/335961853 conn(0x7fa8340a2f40 msgr2=0x7fa83409c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 shutdown_connections 2024-09-17T15:52:30.870 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:30.869+0000 7fa83bb29640 1 -- 172.21.15.86:0/335961853 wait complete. 2024-09-17T15:52:30.885 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T15:52:30.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: get_monmap_and_config 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: monmap: 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:31.020972+0000 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:31.020972+0000 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.017 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:31.018 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 Processor -- start 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 -- start start 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: init 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5480a50e0 0x7fd54809d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-a con 0x7fd5480a50e0 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd5480a54c0 0x7fd5480a58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-c con 0x7fd5480a54c0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:31.021 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd548137ea0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd54809d590 con 0x7fd548137ea0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7fd5480a6ba0 con 0x7fd5480a50e0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7fd548069d50 con 0x7fd5480a54c0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd54e25d640 10 monclient(hunting): authenticate will time out at 2483.987255s 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5480a50e0 0x7fd54809d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5480a50e0 0x7fd54809d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51776/0 (socket says 172.21.15.86:51776) 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 -- 172.21.15.86:0/2979118049 learned_addr learned my addr 172.21.15.86:0/2979118049 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): get_auth_request con 0x7fd5480a50e0 auth_method 0 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.022 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd53effd640 1 --2- 172.21.15.86:0/2979118049 >> v2:172.21.15.203:3300/0 conn(0x7fd5480a54c0 0x7fd5480a58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd53effd640 10 monclient(hunting): get_auth_request con 0x7fd5480a54c0 auth_method 0 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd53effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd53effd640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd53effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient(hunting): handle_auth_done global_id 234705 payload 274 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 -- 172.21.15.86:0/2979118049 >> v2:172.21.15.203:3300/0 conn(0x7fd5480a54c0 msgr2=0x7fd5480a58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 --2- 172.21.15.86:0/2979118049 >> v2:172.21.15.203:3300/0 conn(0x7fd5480a54c0 0x7fd5480a58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.023 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 -- 172.21.15.86:0/2979118049 >> v1:172.21.15.203:6789/0 conn(0x7fd548137ea0 legacy=0x7fd548138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.024 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.017+0000 7fd5477fe640 1 -- 172.21.15.86:0/2979118049 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54809c030 con 0x7fd5480a50e0 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd5477fe640 1 -- 172.21.15.86:0/2979118049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 msgr2=0x7fd54809d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd5477fe640 1 --2- 172.21.15.86:0/2979118049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd54809d050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd54809bf60 tx=0x7fd53001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a1cf3dd7d4017 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 1 -- 172.21.15.86:0/2979118049 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300b0070 con 0x7fd5480a50e0 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: dump: 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.026 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 1 -- 172.21.15.86:0/2979118049 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5300ab600 con 0x7fd5480a50e0 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 5 monclient: authenticate success, global_id 234705 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:31.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 1 -- 172.21.15.86:0/2979118049 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300ab9d0 con 0x7fd5480a50e0 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd546ffd640 10 monclient: dump: 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.028 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.029 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.029 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.029 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:31.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 4 set_mon_vals no callback set 2024-09-17T15:52:31.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 10 monclient: shutdown 2024-09-17T15:52:31.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 -- 172.21.15.86:0/2979118049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 msgr2=0x7fd54809d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2979118049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd54809d050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd54809bf60 tx=0x7fd53001c3e0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 -- 172.21.15.86:0/2979118049 shutdown_connections 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2979118049 >> v2:172.21.15.203:3300/0 conn(0x7fd5480a54c0 0x7fd5480a58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2979118049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd54809d050 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 -- 172.21.15.86:0/2979118049 >> 172.21.15.86:0/2979118049 conn(0x7fd54806ed20 msgr2=0x7fd54806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 -- 172.21.15.86:0/2979118049 shutdown_connections 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 1 -- 172.21.15.86:0/2979118049 wait complete. 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.025+0000 7fd54e25d640 10 monclient: monmap: 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:31.034 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 Processor -- start 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- start start 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: init 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd5481d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-a con 0x7fd5480a50e0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:31.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5480a54c0 0x7fd5481d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-b con 0x7fd5480a54c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd548137ea0 0x7fd54813c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): picked mon.noname-c con 0x7fd548137ea0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7fd548142cb0 con 0x7fd5480a50e0 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd548142b30 con 0x7fd5480a54c0 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7fd548142e30 con 0x7fd548137ea0 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient(hunting): authenticate will time out at 2483.996738s 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd5481d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd5481d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51790/0 (socket says 172.21.15.86:51790) 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 -- 172.21.15.86:0/2061638516 learned_addr learned my addr 172.21.15.86:0/2061638516 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): get_auth_request con 0x7fd5480a50e0 auth_method 0 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd548137ea0 0x7fd54813c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): get_auth_request con 0x7fd548137ea0 auth_method 0 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd547fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient(hunting): handle_auth_done global_id 234708 payload 824 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd548137ea0 msgr2=0x7fd54813c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd548137ea0 0x7fd54813c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5480a54c0 msgr2=0x7fd5481d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5480a54c0 0x7fd5481d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd548121880 con 0x7fd5480a50e0 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 msgr2=0x7fd5481d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5477fe640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd5481d0380 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd53001aa90 tx=0x7fd530019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300b0040 con 0x7fd5480a50e0 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: dump: 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5300aa5b0 con 0x7fd5480a50e0 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5300aa9a0 con 0x7fd5480a50e0 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 5 monclient: authenticate success, global_id 234708 2024-09-17T15:52:31.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _renew_subs 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5480aa340 con 0x7fd5480a50e0 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd5467fc640 4 set_mon_vals no callback set 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd544ff9640 10 monclient: dump: 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.041 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _renew_subs 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.029+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5481215a0 con 0x7fd5480a50e0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd54e25d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd54809d050 con 0x7fd5480a50e0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7fd5300aab40 con 0x7fd5480a50e0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd52c03bc10 0x7fd52c03e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7fd5300e98a0 con 0x7fd5480a50e0 2024-09-17T15:52:31.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7fd52c041b00 0x7fd52c043f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5300b7ec0 con 0x7fd52c041b00 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=876) ==== 24+0+0 (secure 0 0 0) 0x7fd5300ee2a0 con 0x7fd5480a50e0 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd544ff9640 10 monclient: handle_get_version_reply finishing 1 version 876 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd53effd640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd52c03bc10 0x7fd52c03e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd53effd640 10 monclient: get_auth_request con 0x7fd52c03bc10 auth_method 0 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd547fff640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7fd52c041b00 0x7fd52c043f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.033+0000 7fd547fff640 10 monclient: get_auth_request con 0x7fd52c041b00 auth_method 0 2024-09-17T15:52:31.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.037+0000 7fd53effd640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd52c03bc10 0x7fd52c03e0d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd54809e720 tx=0x7fd538008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.037+0000 7fd547fff640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7fd52c041b00 0x7fd52c043f20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.037+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== osd.5 v2:172.21.15.203:6800/3502598683 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd5300b7ec0 con 0x7fd52c041b00 2024-09-17T15:52:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.049+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 --> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd54813e960 con 0x7fd52c041b00 2024-09-17T15:52:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd544ff9640 1 -- 172.21.15.86:0/2061638516 <== osd.5 v2:172.21.15.203:6800/3502598683 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd54813e960 con 0x7fd52c041b00 2024-09-17T15:52:31.052 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-09-17T15:52:31.053 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T15:52:31.053 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7fd52c041b00 msgr2=0x7fd52c043f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6800/3502598683,v1:172.21.15.203:6801/3502598683] conn(0x7fd52c041b00 0x7fd52c043f20 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd52c03bc10 msgr2=0x7fd52c03e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7fd52c03bc10 0x7fd52c03e0d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fd54809e720 tx=0x7fd538008040 comp rx=0 tx=0).stop 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 10 monclient: shutdown 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 msgr2=0x7fd5481d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 --2- 172.21.15.86:0/2061638516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd5480a50e0 0x7fd5481d0380 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd53001aa90 tx=0x7fd530019180 comp rx=0 tx=0).stop 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd547fff640 1 -- 172.21.15.86:0/2061638516 reap_dead start 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 shutdown_connections 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 >> 172.21.15.86:0/2061638516 conn(0x7fd54806ed20 msgr2=0x7fd54813f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 shutdown_connections 2024-09-17T15:52:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.053+0000 7fd54e25d640 1 -- 172.21.15.86:0/2061638516 wait complete. 2024-09-17T15:52:31.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T15:52:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: get_monmap_and_config 2024-09-17T15:52:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: monmap: 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:31.170348+0000 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:31.170348+0000 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:31.167 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 Processor -- start 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 -- start start 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: init 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f280a0c70 0x7f0f280a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-a con 0x7f0f280a0c70 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f280a1590 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 0x7f0f2813df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f28136160 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:31.170 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f280a2130 con 0x7f0f280a1590 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f0f280a2410 con 0x7f0f280a0c70 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f0f280a26f0 con 0x7f0f28136160 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2f63f640 10 monclient(hunting): authenticate will time out at 2484.135802s 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f280a0c70 0x7f0f280a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f280a0c70 0x7f0f280a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51800/0 (socket says 172.21.15.86:51800) 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 1 -- 172.21.15.86:0/3302093452 learned_addr learned my addr 172.21.15.86:0/3302093452 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 10 monclient(hunting): get_auth_request con 0x7f0f280a0c70 auth_method 0 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.165+0000 7f0f2d3db640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.171 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2dbdc640 1 --2- 172.21.15.86:0/3302093452 >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 0x7f0f2813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient(hunting): handle_auth_done global_id 234711 payload 274 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 1 -- 172.21.15.86:0/3302093452 >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 msgr2=0x7f0f2813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/3302093452 >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 0x7f0f2813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 1 -- 172.21.15.86:0/3302093452 >> v1:172.21.15.203:6789/0 conn(0x7f0f280a1590 legacy=0x7f0f28135a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2d3db640 1 -- 172.21.15.86:0/3302093452 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2809f9f0 con 0x7f0f280a0c70 2024-09-17T15:52:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2dbdc640 10 monclient: get_auth_request con 0x7f0f28136160 auth_method 0 2024-09-17T15:52:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.169+0000 7f0f2dbdc640 1 --2- 172.21.15.86:0/3302093452 >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 0x7f0f2813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2d3db640 1 -- 172.21.15.86:0/3302093452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f280a0c70 msgr2=0x7f0f280a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/3302093452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f280a0c70 0x7f0f280a1050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0f2806f840 tx=0x7f0f2401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc15fd485efac2b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 1 -- 172.21.15.86:0/3302093452 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240b1070 con 0x7f0f280a0c70 2024-09-17T15:52:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.176 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: dump: 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.177 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 1 -- 172.21.15.86:0/3302093452 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f240ac620 con 0x7f0f280a0c70 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 1 -- 172.21.15.86:0/3302093452 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f240ac9f0 con 0x7f0f280a0c70 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f1ffff640 10 monclient: dump: 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.178 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 5 monclient: authenticate success, global_id 234711 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 4 set_mon_vals no callback set 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: shutdown 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- 172.21.15.86:0/3302093452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f280a0c70 msgr2=0x7f0f280a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/3302093452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f280a0c70 0x7f0f280a1050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0f2806f840 tx=0x7f0f2401c3e0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- 172.21.15.86:0/3302093452 shutdown_connections 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/3302093452 >> v2:172.21.15.203:3300/0 conn(0x7f0f28136160 0x7f0f2813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/3302093452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f280a0c70 0x7f0f280a1050 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.179 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- 172.21.15.86:0/3302093452 >> 172.21.15.86:0/3302093452 conn(0x7f0f280a2f40 msgr2=0x7f0f280a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- 172.21.15.86:0/3302093452 shutdown_connections 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- 172.21.15.86:0/3302093452 wait complete. 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: monmap: 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:31.180 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 Processor -- start 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- start start 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: init 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 0x7f0f2812ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-b con 0x7f0f280a0c70 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 0x7f0f2812f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f280a1590 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): picked mon.noname-a con 0x7f0f28136160 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.181 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f0f280a00d0 con 0x7f0f28136160 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0f2809ff50 con 0x7f0f280a0c70 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f0f28140c90 con 0x7f0f280a1590 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2f63f640 10 monclient(hunting): authenticate will time out at 2484.142663s 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:51812/0 (socket says 172.21.15.86:51812) 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 0x7f0f2812f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 -- 172.21.15.86:0/27002068 learned_addr learned my addr 172.21.15.86:0/27002068 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): get_auth_request con 0x7f0f28136160 auth_method 0 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.182 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): get_auth_request con 0x7f0f280a1590 auth_method 0 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 0x7f0f2812ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.183 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient(hunting): handle_auth_done global_id 234714 payload 824 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 -- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 msgr2=0x7f0f2812f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 0x7f0f2812f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 -- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 msgr2=0x7f0f2812ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 0x7f0f2812ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2dbdc640 1 -- 172.21.15.86:0/27002068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2809ef00 con 0x7f0f28136160 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2d3db640 10 monclient: get_auth_request con 0x7f0f280a0c70 auth_method 0 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 0x7f0f2812ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:31.184 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.173+0000 7f0f2cbda640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 0x7f0f2812f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2dbdc640 1 -- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 msgr2=0x7f0f281cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.186 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2dbdc640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000ce40 tx=0x7f0f2000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2001a070 con 0x7f0f28136160 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: dump: 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.187 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2000da50 con 0x7f0f28136160 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 5 monclient: authenticate success, global_id 234714 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _renew_subs 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f2809d770 con 0x7f0f28136160 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2000de40 con 0x7f0f28136160 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.188 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 10 monclient: dump: 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1f7fe640 4 set_mon_vals no callback set 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _renew_subs 2024-09-17T15:52:31.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f2813c340 con 0x7f0f28136160 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f280a1050 con 0x7f0f28136160 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f0f2001d070 con 0x7f0f28136160 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 0x7f0f0c03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.177+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f0f20050a20 con 0x7f0f28136160 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.181+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 0x7f0f0c03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.181+0000 7f0f2d3db640 10 monclient: get_auth_request con 0x7f0f0c03bb60 auth_method 0 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.181+0000 7f0f2d3db640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 0x7f0f0c03e020 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0f240ac000 tx=0x7f0f24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.190 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.185+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f0f20024020 con 0x7f0f28136160 2024-09-17T15:52:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.185+0000 7f0f1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.191 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.185+0000 7f0f1dffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.353+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f0f2813c700 con 0x7f0f0c03bb60 2024-09-17T15:52:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.353+0000 7f0f1dffb640 1 -- 172.21.15.86:0/27002068 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0f2813c700 con 0x7f0f0c03bb60 2024-09-17T15:52:31.355 INFO:teuthology.orchestra.run.smithi086.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T15:52:31.357 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 msgr2=0x7f0f0c03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 0x7f0f0c03e020 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f0f240ac000 tx=0x7f0f24008040 comp rx=0 tx=0).stop 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 10 monclient: shutdown 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 msgr2=0x7f0f281cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0f2000ce40 tx=0x7f0f2000a710 comp rx=0 tx=0).stop 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 shutdown_connections 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f0f0c03bb60 0x7f0f0c03e020 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.358 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f28136160 0x7f0f281cfdc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f280a1590 0x7f0f2812f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 --2- 172.21.15.86:0/27002068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0f280a0c70 0x7f0f2812ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 >> 172.21.15.86:0/27002068 conn(0x7f0f280a2f40 msgr2=0x7f0f280a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 shutdown_connections 2024-09-17T15:52:31.359 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.357+0000 7f0f2f63f640 1 -- 172.21.15.86:0/27002068 wait complete. 2024-09-17T15:52:31.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 10 monclient: get_monmap_and_config 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 10 monclient: monmap: 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:31.487017+0000 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:31.487017+0000 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.483 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:31.484 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.485 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 1 Processor -- start 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 1 -- start start 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.481+0000 7faed0591640 10 monclient: init 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faec80689d0 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faec80694c0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faec806f840 0x7faec813e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): picked mon.noname-a con 0x7faec806f840 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faec8074470 con 0x7faec80694c0 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faec809c8b0 con 0x7faec80689d0 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7faec809cb90 con 0x7faec806f840 2024-09-17T15:52:31.486 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faed0591640 10 monclient(hunting): authenticate will time out at 2484.452444s 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faec806f840 0x7faec813e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faecdb2c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faec80694c0 0x7faec813aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36464/0 (socket says 172.21.15.86:36464) 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faecdb2c640 1 -- 172.21.15.86:0/1404468007 learned_addr learned my addr 172.21.15.86:0/1404468007 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): get_auth_request con 0x7faec806f840 auth_method 0 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient(hunting): handle_auth_done global_id 234717 payload 274 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 1 -- 172.21.15.86:0/1404468007 >> v1:172.21.15.203:6789/0 conn(0x7faec80689d0 legacy=0x7faec8068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 1 -- 172.21.15.86:0/1404468007 >> v1:172.21.15.86:6789/0 conn(0x7faec80694c0 legacy=0x7faec813aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.488 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.485+0000 7faeceb2e640 1 -- 172.21.15.86:0/1404468007 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec809ce70 con 0x7faec806f840 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 -- 172.21.15.86:0/1404468007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 msgr2=0x7faec813e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 --2- 172.21.15.86:0/1404468007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec813e720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faec0009d90 tx=0x7faec001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfd5e42e87ceb20 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 1 -- 172.21.15.86:0/1404468007 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00b2070 con 0x7faec806f840 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: dump: 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.489 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 5 monclient: authenticate success, global_id 234717 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 1 -- 172.21.15.86:0/1404468007 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec00a5970 con 0x7faec806f840 2024-09-17T15:52:31.490 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 1 -- 172.21.15.86:0/1404468007 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec809c2d0 con 0x7faec806f840 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 4 set_mon_vals no callback set 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecd32b640 10 monclient: dump: 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.491 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.492 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.492 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: shutdown 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- 172.21.15.86:0/1404468007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 msgr2=0x7faec813e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 --2- 172.21.15.86:0/1404468007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec813e720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faec0009d90 tx=0x7faec001c830 comp rx=0 tx=0).stop 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- 172.21.15.86:0/1404468007 shutdown_connections 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 --2- 172.21.15.86:0/1404468007 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec813e720 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- 172.21.15.86:0/1404468007 >> 172.21.15.86:0/1404468007 conn(0x7faec80a6440 msgr2=0x7faec80a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.496 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- 172.21.15.86:0/1404468007 shutdown_connections 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- 172.21.15.86:0/1404468007 wait complete. 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: build_initial_monmap 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: monmap: 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 Processor -- start 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- start start 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: init 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 0x7faec8138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): picked mon.noname-c con 0x7faec80689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faec80694c0 0x7faec81391f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): picked mon.noname-b con 0x7faec80694c0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec8139730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): picked mon.noname-a con 0x7faec806f840 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7faec81418a0 con 0x7faec806f840 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7faec8141720 con 0x7faec80694c0 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7faec8141a20 con 0x7faec80689d0 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faed0591640 10 monclient(hunting): authenticate will time out at 2484.459391s 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 0x7faec8138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec8139730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 0x7faec8138cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:44302/0 (socket says 172.21.15.86:44302) 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 1 -- 172.21.15.86:0/924793518 learned_addr learned my addr 172.21.15.86:0/924793518 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): get_auth_request con 0x7faec80689d0 auth_method 0 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): get_auth_request con 0x7faec806f840 auth_method 0 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faece32d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient(hunting): handle_auth_done global_id 234720 payload 824 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faecdb2c640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faec80694c0 0x7faec81391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 -- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 msgr2=0x7faec8138cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 0x7faec8138cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 -- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faec80694c0 msgr2=0x7faec81391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faec80694c0 0x7faec81391f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient: _finish_hunting 0 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 monclient: found mon.noname-a 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.489+0000 7faeceb2e640 1 -- 172.21.15.86:0/924793518 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec813c7a0 con 0x7faec806f840 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faeceb2e640 1 -- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 msgr2=0x7faec8139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faeceb2e640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec8139730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faec00ac000 tx=0x7faec00195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00bd070 con 0x7faec806f840 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: dump: 2024-09-17T15:52:31.501 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec00b8880 con 0x7faec806f840 2024-09-17T15:52:31.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 5 monclient: authenticate success, global_id 234720 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _renew_subs 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faec80705f0 con 0x7faec806f840 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec00b2040 con 0x7faec806f840 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 10 monclient: dump: 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:31.503 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _renew_subs 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faec813df60 con 0x7faec806f840 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec7fff640 4 set_mon_vals no callback set 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.86:3300/0 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faec8068db0 con 0x7faec806f840 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7faec00b8cb0 con 0x7faec806f840 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 0x7faeb403e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7faec00eac70 con 0x7faec806f840 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faece32d640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 0x7faeb403e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.493+0000 7faece32d640 10 monclient: get_auth_request con 0x7faeb403bbb0 auth_method 0 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.497+0000 7faece32d640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 0x7faeb403e070 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faeb80050b0 tx=0x7faeb8003700 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.497+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mon.0 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7faec0016540 con 0x7faec806f840 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.497+0000 7faec6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:31.505 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.497+0000 7faec6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:31.653 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.653+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faec813c390 con 0x7faeb403bbb0 2024-09-17T15:52:31.656 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.653+0000 7faec6ffd640 1 -- 172.21.15.86:0/924793518 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33968 (secure 0 0 0) 0x7faec813c390 con 0x7faeb403bbb0 2024-09-17T15:52:31.656 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:31.656 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 msgr2=0x7faeb403e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 0x7faeb403e070 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faeb80050b0 tx=0x7faeb8003700 comp rx=0 tx=0).stop 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 10 monclient: shutdown 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 msgr2=0x7faec8139730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec8139730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faec00ac000 tx=0x7faec00195d0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 shutdown_connections 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7faeb403bbb0 0x7faeb403e070 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faec806f840 0x7faec8139730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faec80694c0 0x7faec81391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.660 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 --2- 172.21.15.86:0/924793518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faec80689d0 0x7faec8138cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:31.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 >> 172.21.15.86:0/924793518 conn(0x7faec80a6440 msgr2=0x7faec80a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:31.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 shutdown_connections 2024-09-17T15:52:31.661 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:31.657+0000 7faed0591640 1 -- 172.21.15.86:0/924793518 wait complete. 2024-09-17T15:52:31.675 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1625,"stamp":"2024-09-17T15:52:30.050793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":314076,"kb_used_data":6176,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565916964,"statfs":{"total":579820584960,"available":579498971136,"internally_reserved":0,"allocated":6324224,"data_stored":3721246,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"1.0","version":"14'72","reported_seq":1811,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375474+0000","last_change":"2024-09-17T15:27:25.516013+0000","last_active":"2024-09-17T15:52:11.375474+0000","last_peered":"2024-09-17T15:52:11.375474+0000","last_clean":"2024-09-17T15:52:11.375474+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:11.375474+0000","last_undegraded":"2024-09-17T15:52:11.375474+0000","last_fullsized":"2024-09-17T15:52:11.375474+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:24.487928+0000","last_clean_scrub_stamp":"2024-09-17T15:27:24.487928+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:52:20.361772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375451+0000","last_change":"2024-09-17T15:27:29.638683+0000","last_active":"2024-09-17T15:52:11.375451+0000","last_peered":"2024-09-17T15:52:11.375451+0000","last_clean":"2024-09-17T15:52:11.375451+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:11.375451+0000","last_undegraded":"2024-09-17T15:52:11.375451+0000","last_fullsized":"2024-09-17T15:52:11.375451+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:07:05.722715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1724,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375309+0000","last_change":"2024-09-17T15:27:29.739037+0000","last_active":"2024-09-17T15:52:11.375309+0000","last_peered":"2024-09-17T15:52:11.375309+0000","last_clean":"2024-09-17T15:52:11.375309+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:11.375309+0000","last_undegraded":"2024-09-17T15:52:11.375309+0000","last_fullsized":"2024-09-17T15:52:11.375309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:20:50.320484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1725,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:10.899664+0000","last_change":"2024-09-17T15:27:29.632182+0000","last_active":"2024-09-17T15:52:10.899664+0000","last_peered":"2024-09-17T15:52:10.899664+0000","last_clean":"2024-09-17T15:52:10.899664+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:10.899664+0000","last_undegraded":"2024-09-17T15:52:10.899664+0000","last_fullsized":"2024-09-17T15:52:10.899664+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:41:56.203307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":1727,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375635+0000","last_change":"2024-09-17T15:27:29.639322+0000","last_active":"2024-09-17T15:52:11.375635+0000","last_peered":"2024-09-17T15:52:11.375635+0000","last_clean":"2024-09-17T15:52:11.375635+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:11.375635+0000","last_undegraded":"2024-09-17T15:52:11.375635+0000","last_fullsized":"2024-09-17T15:52:11.375635+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:11:07.429808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1727,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374803+0000","last_change":"2024-09-17T15:27:29.642636+0000","last_active":"2024-09-17T15:52:11.374803+0000","last_peered":"2024-09-17T15:52:11.374803+0000","last_clean":"2024-09-17T15:52:11.374803+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:11.374803+0000","last_undegraded":"2024-09-17T15:52:11.374803+0000","last_fullsized":"2024-09-17T15:52:11.374803+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:01:56.831183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375366+0000","last_change":"2024-09-17T15:27:29.739533+0000","last_active":"2024-09-17T15:52:11.375366+0000","last_peered":"2024-09-17T15:52:11.375366+0000","last_clean":"2024-09-17T15:52:11.375366+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:11.375366+0000","last_undegraded":"2024-09-17T15:52:11.375366+0000","last_fullsized":"2024-09-17T15:52:11.375366+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:42:02.597159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1726,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.374688+0000","last_change":"2024-09-17T15:27:29.642483+0000","last_active":"2024-09-17T15:52:11.374688+0000","last_peered":"2024-09-17T15:52:11.374688+0000","last_clean":"2024-09-17T15:52:11.374688+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:11.374688+0000","last_undegraded":"2024-09-17T15:52:11.374688+0000","last_fullsized":"2024-09-17T15:52:11.374688+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T22:33:25.392180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1723,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:11.375404+0000","last_change":"2024-09-17T15:27:29.738956+0000","last_active":"2024-09-17T15:52:11.375404+0000","last_peered":"2024-09-17T15:52:11.375404+0000","last_clean":"2024-09-17T15:52:11.375404+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:11.375404+0000","last_undegraded":"2024-09-17T15:52:11.375404+0000","last_fullsized":"2024-09-17T15:52:11.375404+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:27:25.502442+0000","last_clean_scrub_stamp":"2024-09-17T15:27:25.502442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T20:33:49.319674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640560,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58568,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313272,"statfs":{"total":96636764160,"available":96576790528,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640560,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640560,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50304,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321536,"statfs":{"total":96636764160,"available":96585252864,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640560,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640560,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46088,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325752,"statfs":{"total":96636764160,"available":96589570048,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640559,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T15:27:24.487928+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=24, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.679 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T15:27:25.502442+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=27, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=15, tm_min=52, tm_sec=27, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T15:52:31.680 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T15:52:51.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T15:52:51.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.805+0000 7f09c7354640 10 monclient: get_monmap_and_config 2024-09-17T15:52:51.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.805+0000 7f09c7354640 10 monclient: build_initial_monmap 2024-09-17T15:52:51.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.805+0000 7f09c7354640 10 monclient: monmap: 2024-09-17T15:52:51.808 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:51.811737+0000 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:51.811737+0000 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:51.810 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:51.810 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 Processor -- start 2024-09-17T15:52:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 -- start start 2024-09-17T15:52:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient: init 2024-09-17T15:52:51.813 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-17T15:52:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 0x7f09c006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-c con 0x7f09c00689d0 addr v2:172.21.15.203:3300/0 2024-09-17T15:52:51.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09c0068db0 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 0x7f09c013d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-b con 0x7f09c013aff0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c0074470 con 0x7f09c0068db0 2024-09-17T15:52:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:51.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 -- 0x7f09c009c900 con 0x7f09c00689d0 2024-09-17T15:52:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f09c009cb90 con 0x7f09c013aff0 2024-09-17T15:52:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c7354640 10 monclient(hunting): authenticate will time out at 2504.777248s 2024-09-17T15:52:51.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c48ef640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f09c0068db0 0x7f09c0069200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50600/0 (socket says 172.21.15.86:50600) 2024-09-17T15:52:51.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c48ef640 1 -- 172.21.15.86:0/3782327871 learned_addr learned my addr 172.21.15.86:0/3782327871 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:51.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 --2- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 0x7f09c013d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:51.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): get_auth_request con 0x7f09c013aff0 auth_method 0 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c50f0640 1 --2- 172.21.15.86:0/3782327871 >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 0x7f09c006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient(hunting): handle_auth_done global_id 274334 payload 274 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 -- 172.21.15.86:0/3782327871 >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 msgr2=0x7f09c006f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 --2- 172.21.15.86:0/3782327871 >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 0x7f09c006f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.818 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 -- 172.21.15.86:0/3782327871 >> v1:172.21.15.86:6789/0 conn(0x7f09c0068db0 legacy=0x7f09c0069200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient: _finish_hunting 0 2024-09-17T15:52:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 monclient: found mon.noname-b 2024-09-17T15:52:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.86:3301/0 2024-09-17T15:52:51.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c58f1640 1 -- 172.21.15.86:0/3782327871 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c009ce70 con 0x7f09c013aff0 2024-09-17T15:52:51.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c50f0640 10 monclient: get_auth_request con 0x7f09c00689d0 auth_method 0 2024-09-17T15:52:51.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.809+0000 7f09c50f0640 1 --2- 172.21.15.86:0/3782327871 >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 0x7f09c006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c58f1640 1 -- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 msgr2=0x7f09c013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c58f1640 1 --2- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 0x7f09c013d3e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009d90 tx=0x7f09b801c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372a46111413afcc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 1 -- 172.21.15.86:0/3782327871 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b80b2070 con 0x7f09c013aff0 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-17T15:52:51.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: dump: 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: _finish_auth 0 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: _check_auth_tickets 2024-09-17T15:52:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 1 -- 172.21.15.86:0/3782327871 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b80a5970 con 0x7f09c013aff0 2024-09-17T15:52:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 5 monclient: authenticate success, global_id 274334 2024-09-17T15:52:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:51.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 1 -- 172.21.15.86:0/3782327871 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c009c2d0 con 0x7f09c013aff0 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 4 set_mon_vals no callback set 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09b7fff640 10 monclient: dump: 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:51.830 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 10 monclient: shutdown 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 -- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 msgr2=0x7f09c013d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 --2- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 0x7f09c013d3e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009d90 tx=0x7f09b801c830 comp rx=0 tx=0).stop 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 -- 172.21.15.86:0/3782327871 shutdown_connections 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 --2- 172.21.15.86:0/3782327871 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c013aff0 0x7f09c013d3e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 --2- 172.21.15.86:0/3782327871 >> v2:172.21.15.203:3300/0 conn(0x7f09c00689d0 0x7f09c006f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 -- 172.21.15.86:0/3782327871 >> 172.21.15.86:0/3782327871 conn(0x7f09c00a6440 msgr2=0x7f09c00a6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 -- 172.21.15.86:0/3782327871 shutdown_connections 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.813+0000 7f09c7354640 1 -- 172.21.15.86:0/3782327871 wait complete. 2024-09-17T15:52:51.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient: build_initial_monmap 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient: monmap: 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:51.832 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 Processor -- start 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 -- start start 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient: init 2024-09-17T15:52:51.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-c con 0x7f09c00689d0 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09c0068db0 0x7f09c009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-a con 0x7f09c0068db0 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09c013aff0 0x7f09c009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): picked mon.noname-b con 0x7f09c013aff0 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f09c0140670 con 0x7f09c0068db0 2024-09-17T15:52:51.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09c01404f0 con 0x7f09c013aff0 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f09c01407f0 con 0x7f09c00689d0 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c7354640 10 monclient(hunting): authenticate will time out at 2504.785125s 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09c0068db0 0x7f09c009d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:36114/0 (socket says 172.21.15.86:36114) 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 -- 172.21.15.86:0/649722315 learned_addr learned my addr 172.21.15.86:0/649722315 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): get_auth_request con 0x7f09c00689d0 auth_method 0 2024-09-17T15:52:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): get_auth_request con 0x7f09c0068db0 auth_method 0 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:51.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c48ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient(hunting): handle_auth_done global_id 274337 payload 824 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 -- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09c013aff0 msgr2=0x7f09c009dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09c013aff0 0x7f09c009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 -- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09c0068db0 msgr2=0x7f09c009d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09c0068db0 0x7f09c009d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient: _finish_hunting 0 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 monclient: found mon.noname-c 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 -- 172.21.15.86:0/649722315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c009eb40 con 0x7f09c00689d0 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 -- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 msgr2=0x7f09c0072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09c50f0640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c7c0 tx=0x7f09bc00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc007bc0 con 0x7f09c00689d0 2024-09-17T15:52:51.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.817+0000 7f09b5ffb640 10 monclient: dump: 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: _finish_auth 0 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: _check_auth_tickets 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:51.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09bc007d60 con 0x7f09c00689d0 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 5 monclient: authenticate success, global_id 274337 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _renew_subs 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c00a3850 con 0x7f09c00689d0 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc011630 con 0x7f09c00689d0 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 10 monclient: dump: 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b77fe640 4 set_mon_vals no callback set 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _renew_subs 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c0071670 con 0x7f09c00689d0 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f09bc0117d0 con 0x7f09c00689d0 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:51.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c006f840 con 0x7f09c00689d0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 0x7f09a403e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f09bc0500b0 con 0x7f09c00689d0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c48ef640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 0x7f09a403e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.821+0000 7f09c48ef640 10 monclient: get_auth_request con 0x7f09a403bb60 auth_method 0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.825+0000 7f09c48ef640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 0x7f09a403e020 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007ca0 tx=0x7f09b0007bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.825+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f09c00a3850 con 0x7f09c00689d0 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.825+0000 7f09b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:51.841 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.825+0000 7f09b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:51.992 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.993+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 --> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f09c006fe90 con 0x7f09a403bb60 2024-09-17T15:52:51.994 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.993+0000 7f09b5ffb640 1 -- 172.21.15.86:0/649722315 <== mgr.4099 v2:172.21.15.203:6824/2899236933 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33986 (secure 0 0 0) 0x7f09c006fe90 con 0x7f09a403bb60 2024-09-17T15:52:51.995 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-09-17T15:52:51.995 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 msgr2=0x7f09a403e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 0x7f09a403e020 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f09b0007ca0 tx=0x7f09b0007bf0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 10 monclient: shutdown 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 msgr2=0x7f09c0072dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:51.997 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f09bc00c7c0 tx=0x7f09bc00cc80 comp rx=0 tx=0).stop 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 shutdown_connections 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f09a403bb60 0x7f09a403e020 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09c013aff0 0x7f09c009dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09c0068db0 0x7f09c009d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 --2- 172.21.15.86:0/649722315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09c00689d0 0x7f09c0072dd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 >> 172.21.15.86:0/649722315 conn(0x7f09c00a6440 msgr2=0x7f09c00a2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 shutdown_connections 2024-09-17T15:52:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:51.997+0000 7f09c7354640 1 -- 172.21.15.86:0/649722315 wait complete. 2024-09-17T15:52:52.013 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2024-09-17T15:52:50.056195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":314052,"kb_used_data":6176,"kb_used_omap":9,"kb_used_meta":307766,"kb_avail":565916988,"statfs":{"total":579820584960,"available":579498995712,"internally_reserved":0,"allocated":6324224,"data_stored":3721246,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":315153057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003180"},"pg_stats":[{"pgid":"1.0","version":"876'74","reported_seq":1821,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:30.884433+0000","last_change":"2024-09-17T15:52:30.884433+0000","last_active":"2024-09-17T15:52:30.884433+0000","last_peered":"2024-09-17T15:52:30.884433+0000","last_clean":"2024-09-17T15:52:30.884433+0000","last_became_active":"2024-09-17T15:27:25.515144+0000","last_became_peered":"2024-09-17T15:27:25.515144+0000","last_unstale":"2024-09-17T15:52:30.884433+0000","last_undegraded":"2024-09-17T15:52:30.884433+0000","last_fullsized":"2024-09-17T15:52:30.884433+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"876'74","last_scrub_stamp":"2024-09-17T15:52:30.884223+0000","last_deep_scrub":"876'74","last_deep_scrub_stamp":"2024-09-17T15:52:30.884223+0000","last_clean_scrub_stamp":"2024-09-17T15:52:30.884223+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T17:27:54.181647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"876'2","reported_seq":1735,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:32.558631+0000","last_change":"2024-09-17T15:52:32.558631+0000","last_active":"2024-09-17T15:52:32.558631+0000","last_peered":"2024-09-17T15:52:32.558631+0000","last_clean":"2024-09-17T15:52:32.558631+0000","last_became_active":"2024-09-17T15:27:27.630834+0000","last_became_peered":"2024-09-17T15:27:27.630834+0000","last_unstale":"2024-09-17T15:52:32.558631+0000","last_undegraded":"2024-09-17T15:52:32.558631+0000","last_fullsized":"2024-09-17T15:52:32.558631+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"876'2","last_scrub_stamp":"2024-09-17T15:52:32.558461+0000","last_deep_scrub":"876'2","last_deep_scrub_stamp":"2024-09-17T15:52:32.558461+0000","last_clean_scrub_stamp":"2024-09-17T15:52:32.558461+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:16:56.017881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001096238,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1732,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:32.817530+0000","last_change":"2024-09-17T15:52:32.817530+0000","last_active":"2024-09-17T15:52:32.817530+0000","last_peered":"2024-09-17T15:52:32.817530+0000","last_clean":"2024-09-17T15:52:32.817530+0000","last_became_active":"2024-09-17T15:27:26.663824+0000","last_became_peered":"2024-09-17T15:27:26.663824+0000","last_unstale":"2024-09-17T15:52:32.817530+0000","last_undegraded":"2024-09-17T15:52:32.817530+0000","last_fullsized":"2024-09-17T15:52:32.817530+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:32.817357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:32.817357+0000","last_clean_scrub_stamp":"2024-09-17T15:52:32.817357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:34:31.819824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001205437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1733,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:30.279363+0000","last_change":"2024-09-17T15:52:30.279363+0000","last_active":"2024-09-17T15:52:30.279363+0000","last_peered":"2024-09-17T15:52:30.279363+0000","last_clean":"2024-09-17T15:52:30.279363+0000","last_became_active":"2024-09-17T15:27:26.666614+0000","last_became_peered":"2024-09-17T15:27:26.666614+0000","last_unstale":"2024-09-17T15:52:30.279363+0000","last_undegraded":"2024-09-17T15:52:30.279363+0000","last_fullsized":"2024-09-17T15:52:30.279363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:30.279227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:30.279227+0000","last_clean_scrub_stamp":"2024-09-17T15:52:30.279227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:08:40.627386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001178604,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"876'3","reported_seq":1736,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:31.556606+0000","last_change":"2024-09-17T15:52:31.556606+0000","last_active":"2024-09-17T15:52:31.556606+0000","last_peered":"2024-09-17T15:52:31.556606+0000","last_clean":"2024-09-17T15:52:31.556606+0000","last_became_active":"2024-09-17T15:27:27.628461+0000","last_became_peered":"2024-09-17T15:27:27.628461+0000","last_unstale":"2024-09-17T15:52:31.556606+0000","last_undegraded":"2024-09-17T15:52:31.556606+0000","last_fullsized":"2024-09-17T15:52:31.556606+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"876'3","last_scrub_stamp":"2024-09-17T15:52:31.556432+0000","last_deep_scrub":"876'3","last_deep_scrub_stamp":"2024-09-17T15:52:31.556432+0000","last_clean_scrub_stamp":"2024-09-17T15:52:31.556432+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T19:22:49.997410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019104250000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1735,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:30.755244+0000","last_change":"2024-09-17T15:52:30.755244+0000","last_active":"2024-09-17T15:52:30.755244+0000","last_peered":"2024-09-17T15:52:30.755244+0000","last_clean":"2024-09-17T15:52:30.755244+0000","last_became_active":"2024-09-17T15:27:26.663205+0000","last_became_peered":"2024-09-17T15:27:26.663205+0000","last_unstale":"2024-09-17T15:52:30.755244+0000","last_undegraded":"2024-09-17T15:52:30.755244+0000","last_fullsized":"2024-09-17T15:52:30.755244+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:30.755104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:30.755104+0000","last_clean_scrub_stamp":"2024-09-17T15:52:30.755104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T23:26:52.504612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001355303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1732,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:31.838139+0000","last_change":"2024-09-17T15:52:31.837980+0000","last_active":"2024-09-17T15:52:31.838139+0000","last_peered":"2024-09-17T15:52:31.838139+0000","last_clean":"2024-09-17T15:52:31.838139+0000","last_became_active":"2024-09-17T15:27:26.624782+0000","last_became_peered":"2024-09-17T15:27:26.624782+0000","last_unstale":"2024-09-17T15:52:31.838139+0000","last_undegraded":"2024-09-17T15:52:31.838139+0000","last_fullsized":"2024-09-17T15:52:31.838139+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:31.837808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:31.837808+0000","last_clean_scrub_stamp":"2024-09-17T15:52:31.837808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:50:10.271286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016580340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1734,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:29.774698+0000","last_change":"2024-09-17T15:52:29.774698+0000","last_active":"2024-09-17T15:52:29.774698+0000","last_peered":"2024-09-17T15:52:29.774698+0000","last_clean":"2024-09-17T15:52:29.774698+0000","last_became_active":"2024-09-17T15:27:26.663732+0000","last_became_peered":"2024-09-17T15:27:26.663732+0000","last_unstale":"2024-09-17T15:52:29.774698+0000","last_undegraded":"2024-09-17T15:52:29.774698+0000","last_fullsized":"2024-09-17T15:52:29.774698+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:29.774626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:29.774626+0000","last_clean_scrub_stamp":"2024-09-17T15:52:29.774626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T21:01:46.916989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00125023,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1731,"reported_epoch":876,"state":"active+clean","last_fresh":"2024-09-17T15:52:33.772467+0000","last_change":"2024-09-17T15:52:33.772467+0000","last_active":"2024-09-17T15:52:33.772467+0000","last_peered":"2024-09-17T15:52:33.772467+0000","last_clean":"2024-09-17T15:52:33.772467+0000","last_became_active":"2024-09-17T15:27:27.025282+0000","last_became_peered":"2024-09-17T15:27:27.025282+0000","last_unstale":"2024-09-17T15:52:33.772467+0000","last_undegraded":"2024-09-17T15:52:33.772467+0000","last_fullsized":"2024-09-17T15:52:33.772467+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T15:52:33.772281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T15:52:33.772281+0000","last_clean_scrub_stamp":"2024-09-17T15:52:33.772281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:44:58.141017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011468750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60936,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58564,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313276,"statfs":{"total":96636764160,"available":96576794624,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50300,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321540,"statfs":{"total":96636764160,"available":96585256960,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46080,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325760,"statfs":{"total":96636764160,"available":96589578240,"internally_reserved":0,"allocated":1032192,"data_stored":443892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640564,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46084,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325756,"statfs":{"total":96636764160,"available":96589574144,"internally_reserved":0,"allocated":1036288,"data_stored":443911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640563,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54460,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317380,"statfs":{"total":96636764160,"available":96580997120,"internally_reserved":0,"allocated":1093632,"data_stored":972820,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30468,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T15:52:52.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T15:52:52.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: get_monmap_and_config 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: build_initial_monmap 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: monmap: 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:52:52.145664+0000 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:52:52.145664+0000 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a-legacy 2024-09-17T15:52:52.142 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c-legacy 2024-09-17T15:52:52.143 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-b 2024-09-17T15:52:52.143 INFO:teuthology.orchestra.run.smithi086.stderr:3: v2:172.21.15.86:3300/0 mon.noname-a 2024-09-17T15:52:52.143 INFO:teuthology.orchestra.run.smithi086.stderr:4: v2:172.21.15.203:3300/0 mon.noname-c 2024-09-17T15:52:52.143 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 Processor -- start 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 -- start start 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: init 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33e8134d90 0x7f33e80a1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:52.145 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-a con 0x7f33e8134d90 addr v2:172.21.15.86:3300/0 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33e8135170 addr v1:172.21.15.86:6789/0 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33e8135580 addr v1:172.21.15.203:6789/0 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e8138a00 con 0x7f33e8135170 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e8073b10 con 0x7f33e8135580 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 -- 0x7f33e80a21b0 con 0x7f33e8134d90 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33f053d640 10 monclient(hunting): authenticate will time out at 2505.111118s 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33e8134d90 0x7f33e80a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33edad8640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f33e8135170 0x7f33e8137be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50616/0 (socket says 172.21.15.86:50616) 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33edad8640 1 -- 172.21.15.86:0/3621553354 learned_addr learned my addr 172.21.15.86:0/3621553354 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): get_auth_request con 0x7f33e8134d90 auth_method 0 2024-09-17T15:52:52.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_done global_id 234726 payload 274 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 1 -- 172.21.15.86:0/3621553354 >> v1:172.21.15.203:6789/0 conn(0x7f33e8135580 legacy=0x7f33e81382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 1 -- 172.21.15.86:0/3621553354 >> v1:172.21.15.86:6789/0 conn(0x7f33e8135170 legacy=0x7f33e8137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient: _finish_hunting 0 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 1 monclient: found mon.noname-a 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.86:3300/0 2024-09-17T15:52:52.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.141+0000 7f33ee2d9640 1 -- 172.21.15.86:0/3621553354 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e80a1ed0 con 0x7f33e8134d90 2024-09-17T15:52:52.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.145+0000 7f33ee2d9640 1 -- 172.21.15.86:0/3621553354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 msgr2=0x7f33e80a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:52.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.145+0000 7f33ee2d9640 1 --2- 172.21.15.86:0/3621553354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e80a1130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009820 tx=0x7f33e401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4878c3d93f21f8a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:52.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 1 -- 172.21.15.86:0/3621553354 <== mon.0 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e40b0070 con 0x7f33e8134d90 2024-09-17T15:52:52.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:52.149 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: dump: 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: _finish_auth 0 2024-09-17T15:52:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: _check_auth_tickets 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 1 -- 172.21.15.86:0/3621553354 <== mon.0 v2:172.21.15.86:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e40ab600 con 0x7f33e8134d90 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 5 monclient: authenticate success, global_id 234726 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: get_monmap_and_config success 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 1 -- 172.21.15.86:0/3621553354 <== mon.0 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e40ab9d0 con 0x7f33e8134d90 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 4 set_mon_vals no callback set 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ed2d7640 10 monclient: dump: 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:52.151 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: shutdown 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- 172.21.15.86:0/3621553354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 msgr2=0x7f33e80a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:52.152 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 --2- 172.21.15.86:0/3621553354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e80a1130 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009820 tx=0x7f33e401c340 comp rx=0 tx=0).stop 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- 172.21.15.86:0/3621553354 shutdown_connections 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 --2- 172.21.15.86:0/3621553354 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e80a1130 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- 172.21.15.86:0/3621553354 >> 172.21.15.86:0/3621553354 conn(0x7f33e80a2f40 msgr2=0x7f33e80a3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- 172.21.15.86:0/3621553354 shutdown_connections 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- 172.21.15.86:0/3621553354 wait complete. 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: build_initial_monmap 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: monmap: 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.noname-a 2024-09-17T15:52:52.153 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.noname-b 2024-09-17T15:52:52.154 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.noname-c 2024-09-17T15:52:52.154 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 Processor -- start 2024-09-17T15:52:52.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- start start 2024-09-17T15:52:52.154 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: init 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: _reopen_session rank -1 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-a con 0x7f33e8134d90 addr [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 0x7f33e81404e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-c con 0x7f33e8135170 addr [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e8135580 0x7f33e8139c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): picked mon.noname-b con 0x7f33e8135580 addr [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 -- 0x7f33e8143b40 con 0x7f33e8134d90 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:52.158 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f33e81439c0 con 0x7f33e8135580 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): start opening mon connection 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 -- 0x7f33e8143cc0 con 0x7f33e8135170 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): _renew_subs 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33f053d640 10 monclient(hunting): authenticate will time out at 2505.118271s 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 0x7f33e81404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:50148/0 (socket says 172.21.15.86:50148) 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): get_auth_request con 0x7f33e8135170 auth_method 0 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 1 -- 172.21.15.86:0/1827144915 learned_addr learned my addr 172.21.15.86:0/1827144915 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): get_auth_request con 0x7f33e8134d90 auth_method 0 2024-09-17T15:52:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33eeada640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e8135580 0x7f33e8139c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33eeada640 10 monclient(hunting): get_auth_request con 0x7f33e8135580 auth_method 0 2024-09-17T15:52:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33eeada640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33eeada640 10 monclient(hunting): _init_auth method 2 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33eeada640 10 monclient(hunting): _init_auth creating new auth 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient(hunting): handle_auth_done global_id 274340 payload 824 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 -- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e8135580 msgr2=0x7f33e8139c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e8135580 0x7f33e8139c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 -- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 msgr2=0x7f33e813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient: _finish_hunting 0 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 monclient: found mon.noname-c 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.86:3301/0 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33edad8640 1 -- 172.21.15.86:0/1827144915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e809d460 con 0x7f33e8135170 2024-09-17T15:52:52.161 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.149+0000 7f33ee2d9640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33edad8640 1 -- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 msgr2=0x7f33e81404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33edad8640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 0x7f33e81404e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c410 tx=0x7f33dc00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33dc017070 con 0x7f33e8135170 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: dump: 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.162 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: _finish_auth 0 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: _check_auth_tickets 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33dc0070d0 con 0x7f33e8135170 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: handle_config config(0 keys) 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0074c0 con 0x7f33e8135170 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 10 monclient: dump: 2024-09-17T15:52:52.163 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:fsid b4e90329-4d4c-4253-907f-d2ba7ae38072 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-09-17T15:26:41.267126+0000 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 5 monclient: authenticate success, global_id 274340 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _renew_subs 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33ecad6640 4 set_mon_vals no callback set 2024-09-17T15:52:52.164 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e813d780 con 0x7f33e8135170 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _renew_subs 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e80a5aa0 con 0x7f33e8135170 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33e80a1130 con 0x7f33e8135170 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) ==== 48944+0+0 (secure 0 0 0) 0x7f33dc01a070 con 0x7f33e8135170 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 0x7f33cc03e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(876..876 src has 794..876) ==== 4374+0+0 (secure 0 0 0) 0x7f33dc04f630 con 0x7f33e8135170 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33ee2d9640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 0x7f33cc03e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.153+0000 7f33ee2d9640 10 monclient: get_auth_request con 0x7f33cc03bb60 auth_method 0 2024-09-17T15:52:52.165 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.157+0000 7f33ee2d9640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 0x7f33cc03e020 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f33e401c170 tx=0x7f33e40a6000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T15:52:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.157+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187803 (secure 0 0 0) 0x7f33dc021020 con 0x7f33e8135170 2024-09-17T15:52:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.157+0000 7f33daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-17T15:52:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.157+0000 7f33daffd640 10 monclient: _finish_command 1 = system:0 2024-09-17T15:52:52.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.329+0000 7f33f053d640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:52:52.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.329+0000 7f33f053d640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:52:52.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.329+0000 7f33f053d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.86:3301/0 2024-09-17T15:52:52.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.329+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f33e80a5fd0 con 0x7f33e8135170 2024-09-17T15:52:52.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f33dc00e620 con 0x7f33e8135170 2024-09-17T15:52:52.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33daffd640 10 monclient: handle_config config(1 keys) 2024-09-17T15:52:52.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33ecad6640 4 set_mon_vals no callback set 2024-09-17T15:52:52.340 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33ecad6640 10 set_mon_vals mon_health_to_clog = false 2024-09-17T15:52:52.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33daffd640 1 -- 172.21.15.86:0/1827144915 <== mon.2 v2:172.21.15.86:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7f33e809d460 con 0x7f33e8135170 2024-09-17T15:52:52.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-17T15:52:52.341 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33daffd640 10 monclient: _finish_command 2 = system:0 2024-09-17T15:52:52.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 msgr2=0x7f33cc03e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:52.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 0x7f33cc03e020 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f33e401c170 tx=0x7f33e40a6000 comp rx=0 tx=0).stop 2024-09-17T15:52:52.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 10 monclient: shutdown 2024-09-17T15:52:52.343 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 msgr2=0x7f33e81404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 0x7f33e81404e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33dc00c410 tx=0x7f33dc00c8d0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 shutdown_connections 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:6824/2899236933,v1:172.21.15.203:6825/2899236933] conn(0x7f33cc03bb60 0x7f33cc03e020 secure :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f33e401c170 tx=0x7f33e40a6000 comp rx=0 tx=0).stop 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33e8135580 0x7f33e8139c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33e8135170 0x7f33e81404e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 --2- 172.21.15.86:0/1827144915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33e8134d90 0x7f33e813ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 >> 172.21.15.86:0/1827144915 conn(0x7f33e80a2f40 msgr2=0x7f33e8137da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 shutdown_connections 2024-09-17T15:52:52.344 INFO:teuthology.orchestra.run.smithi086.stderr:2024-09-17T15:52:52.341+0000 7f33f053d640 1 -- 172.21.15.86:0/1827144915 wait complete. 2024-09-17T15:52:52.360 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T15:52:52.360 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T15:52:52.361 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T15:52:52.361 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.456 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T15:52:52.457 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T15:52:52.457 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.541 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T15:52:52.541 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T15:52:52.541 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.630 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T15:52:52.630 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T15:52:52.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.716 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T15:52:52.717 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T15:52:52.717 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.804 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T15:52:52.804 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T15:52:52.805 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.903 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T15:52:52.903 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T15:52:52.903 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T15:52:52.903 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.934 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T15:52:52.934 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T15:52:52.934 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T15:52:52.934 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:52.966 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T15:52:52.966 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T15:52:52.966 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:53.031 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T15:52:53.031 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T15:52:53.031 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T15:52:53.051 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T15:52:53.051 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T15:52:53.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-17T15:52:53.096 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:52:53.096 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T15:52:53.368 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:52:53.369 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T15:52:53.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:52:53.643 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T15:52:53.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:52:53.851 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T15:52:54.115 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:52:54.115 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T15:52:54.280 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:52:54.280 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T15:52:54.405 INFO:tasks.ceph:Archiving mon data... 2024-09-17T15:52:54.406 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/data/mon.a.tgz 2024-09-17T15:52:54.407 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-09-17T15:52:54.413 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.17PJtcXRYi 2024-09-17T15:52:54.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.17PJtcXRYi 2024-09-17T15:52:54.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.17PJtcXRYi 2024-09-17T15:52:54.669 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.17PJtcXRYi is 832KB 2024-09-17T15:52:54.738 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.17PJtcXRYi 2024-09-17T15:52:54.744 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/data/mon.c.tgz 2024-09-17T15:52:54.744 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-09-17T15:52:54.792 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.D8KbYK5m2g 2024-09-17T15:52:54.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.D8KbYK5m2g 2024-09-17T15:52:54.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.D8KbYK5m2g 2024-09-17T15:52:55.041 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.D8KbYK5m2g is 831KB 2024-09-17T15:52:55.106 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.D8KbYK5m2g 2024-09-17T15:52:55.112 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/data/mon.b.tgz 2024-09-17T15:52:55.112 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-09-17T15:52:55.118 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.94sfxRSySd 2024-09-17T15:52:55.118 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.94sfxRSySd 2024-09-17T15:52:55.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.94sfxRSySd 2024-09-17T15:52:55.368 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.94sfxRSySd is 830KB 2024-09-17T15:52:55.434 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.94sfxRSySd 2024-09-17T15:52:55.440 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T15:52:55.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T15:52:55.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T15:52:55.503 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T15:52:55.545 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T15:52:55.545 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T15:52:55.546 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi086/crash 2024-09-17T15:52:55.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T15:52:55.562 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi203/crash 2024-09-17T15:52:55.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T15:52:55.599 INFO:tasks.ceph:Compressing logs... 2024-09-17T15:52:55.600 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T15:52:55.609 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T15:52:55.634 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35894.log 2024-09-17T15:52:55.636 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31282.log 2024-09-17T15:52:55.636 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32848.log 2024-09-17T15:52:55.636 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40103.log 2024-09-17T15:52:55.636 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35894.log.gz 2024-09-17T15:52:55.638 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32848.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.31282.log.gz 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr: /var/log/ceph/ceph-client.admin.36576.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31511.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32848.log.gz 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30640.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31511.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40103.log.gz 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29641.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30640.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36576.log.gz 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31511.log.gz 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29537.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-09-17T15:52:55.639 INFO:teuthology.orchestra.run.smithi086.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.30640.log.gz 2024-09-17T15:52:55.640 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38882.log 2024-09-17T15:52:55.656 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29302.log 2024-09-17T15:52:55.657 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29641.log.gz 2024-09-17T15:52:55.657 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29537.log.gz 2024-09-17T15:52:55.657 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33529.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.38882.log.gz 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29302.log.gz 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38541.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.33529.log.gz 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 2024-09-17T15:52:55.658 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31973.log 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38541.log.gz 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29693.log 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31973.log.gz 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32794.log 2024-09-17T15:52:55.659 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29693.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33477.log 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38385.log 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29693.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29918.log 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.32794.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33477.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41442.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-09-17T15:52:55.660 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36697.log 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34496.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29918.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31921.log 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32051.log 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36723.log 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36697.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36077.log 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31921.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36723.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32051.log.gz 2024-09-17T15:52:55.661 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32025.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36723.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31746.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36077.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31746.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32025.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29762.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28368.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31746.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34027.log 2024-09-17T15:52:55.662 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33923.log 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29762.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28394.log 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34879.log 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28394.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34027.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28342.log 88.5% 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33923.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28394.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31895.log 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41650.log 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34879.log.gz 2024-09-17T15:52:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38567.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28342.log.gz 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/ceph-client.admin.41650.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40233.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31895.log.gz 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41650.log.gz 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log/var/log/ceph/ceph-client.admin.40233.log: 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38176.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40233.log.gz 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- 89.9% /var/log/ceph/ceph-client.admin.37820.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39343.log: -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 90.3% 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38567.log.gz 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33347.log 2024-09-17T15:52:55.664 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37820.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-09-17T15:52:55.665 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30956.log 2024-09-17T15:52:55.665 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38176.log.gz 2024-09-17T15:52:55.665 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-09-17T15:52:55.665 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34853.log 2024-09-17T15:52:55.671 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25155.log 2024-09-17T15:52:55.687 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25227.log 2024-09-17T15:52:55.687 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25251.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25179.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.25155.log.gz 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25251.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25227.log.gz 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25179.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.25251.log.gz 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25179.log.gz 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T15:52:55.688 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.28086.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.0.28086.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25275.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25203.log 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25275.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25275.log.gz 2024-09-17T15:52:55.689 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.25203.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25203.log.gz 2024-09-17T15:52:55.703 INFO:teuthology.orchestra.run.smithi203.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T15:52:55.780 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33347.log: /var/log/ceph/ceph-client.admin.30956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37609.log 2024-09-17T15:52:55.780 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33347.log.gz 2024-09-17T15:52:55.780 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30022.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30956.log.gz 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34853.log.gz 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35485.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36602.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37609.log.gz 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28446.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36602.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35485.log.gz 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30022.log.gz 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28921.log 2024-09-17T15:52:55.781 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28446.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36602.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28446.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28921.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37426.log 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30796.log 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28921.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8%/var/log/ceph/ceph-client.admin.30796.log: -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37426.log.gz 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32158.log 2024-09-17T15:52:55.782 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30588.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30796.log.gz 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35842.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32158.log.gz 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37192.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30588.log.gz 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41468.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31537.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29866.log 2024-09-17T15:52:55.783 INFO:teuthology.orchestra.run.smithi086.stderr: 90.6% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37192.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35842.log.gz 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41468.log.gz 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32335.log 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29866.log: /var/log/ceph/ceph-client.admin.40616.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31537.log.gz 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33134.log 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29866.log.gz 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30562.log 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34905.log 2024-09-17T15:52:55.784 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32335.log.gz 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34522.log 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30562.log: /var/log/ceph/ceph-client.admin.34905.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33134.log.gz 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38515.log 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30562.log.gz 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34905.log.gz 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41338.log 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz/var/log/ceph/ceph-client.admin.28603.log: 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27978.log 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41338.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-09-17T15:52:55.785 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38515.log.gz 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35119.log 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28178.log 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27978.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41338.log.gz 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31230.log 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27978.log.gz 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29103.log 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35119.log.gz 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30248.log 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29103.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28178.log.gz 2024-09-17T15:52:55.786 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29355.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29103.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31230.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28761.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29355.log: /var/log/ceph/ceph-client.admin.40842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25307.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30248.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28761.log: /var/log/ceph/ceph-client.admin.38776.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29355.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.25307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32217.log 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.25307.log.gz 2024-09-17T15:52:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38776.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28761.log.gz 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34392.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38776.log.gz/var/log/ceph/ceph-client.admin.32217.log: 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29129.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz/var/log/ceph/ceph-client.admin.34392.log: 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32217.log.gz 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30352.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34392.log.gz 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29129.log: /var/log/ceph/ceph-client.admin.30352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37452.log 2024-09-17T15:52:55.788 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29129.log.gz 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30352.log.gz 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29944.log 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39630.log 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34931.log 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29944.log: /var/log/ceph/ceph-client.admin.39630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33949.log 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37452.log.gz 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39630.log.gz 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37166.log 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29944.log.gz 2024-09-17T15:52:55.789 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34931.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.33949.log: -- /var/log/ceph/ceph-client.admin.39265.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37166.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34931.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37010.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33949.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37166.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29970.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39265.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.37010.log: /var/log/ceph/ceph-client.admin.34314.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29511.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37010.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34314.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29970.log.gz 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32491.log 2024-09-17T15:52:55.790 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39395.log 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34314.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28018.log 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29511.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- 90.2% /var/log/ceph/ceph-client.admin.30484.log -- replaced with /var/log/ceph/ceph-client.admin.32491.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39395.log: /var/log/ceph/ceph-client.admin.28018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35589.log 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39395.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37348.log 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28018.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29051.log 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30484.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35589.log.gz 2024-09-17T15:52:55.791 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37716.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37768.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29051.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37348.log.gz 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37716.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39213.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37716.log.gz 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29051.log.gz 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37768.log.gz 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28168.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37322.log 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39213.log.gz 2024-09-17T15:52:55.792 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36446.log 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41598.log 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37322.log: /var/log/ceph/ceph-client.admin.36446.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31589.log 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28168.log.gz 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37244.log 88.3% 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37322.log.gz 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36446.log.gz 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33108.log 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31589.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41598.log.gz 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5/var/log/ceph/ceph-client.admin.37244.log: --verbose -- /var/log/ceph/ceph-client.admin.36776.log 2024-09-17T15:52:55.793 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31589.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37690.log 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36776.log: 90.1% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33108.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37244.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36776.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37114.log 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37690.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36024.log 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-09-17T15:52:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32900.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34001.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37114.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39500.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36024.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37036.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32900.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37950.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34001.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31720.log 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.39500.log.gz 2024-09-17T15:52:55.795 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37036.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28655.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37950.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40920.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31720.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38724.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37556.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28655.log: /var/log/ceph/ceph-client.admin.38724.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34983.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38724.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28655.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37556.log.gz 2024-09-17T15:52:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35277.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34983.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz/var/log/ceph/ceph-client.admin.39239.log: 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32978.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34983.log.gz 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36290.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35277.log.gz 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30822.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32978.log.gz 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36290.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41364.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2024-09-17T15:52:55.797 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30822.log: /var/log/ceph/ceph-client.admin.41364.log: 89.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36290.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30822.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41364.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35249.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30614.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38059.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30196.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30378.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38059.log: /var/log/ceph/ceph-client.admin.30196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32283.log 2024-09-17T15:52:55.798 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30614.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30378.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.38059.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35381.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30196.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/ceph-client.admin.32283.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30378.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.32283.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30458.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35381.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37976.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40972.log: /var/log/ceph/ceph-client.admin.30458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33373.log 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2024-09-17T15:52:55.799 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41180.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30458.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37872.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.37976.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41180.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33373.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37872.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41180.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37872.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41806.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32543.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30874.log 2024-09-17T15:52:55.800 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37140.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41806.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32543.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39030.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35698.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.30874.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37846.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37140.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31947.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31947.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.39030.log.gz --verbose 2024-09-17T15:52:55.801 INFO:teuthology.orchestra.run.smithi086.stderr: -- /var/log/ceph/ceph-client.admin.37400.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37846.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31947.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32741.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28180.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39921.log: /var/log/ceph/ceph-client.admin.32741.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37400.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35763.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32741.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32387.log 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28180.log.gz 2024-09-17T15:52:55.802 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39973.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35763.log.gz 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32647.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32387.log.gz 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30718.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41208.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32647.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz/var/log/ceph/ceph-client.admin.30718.log: 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27900.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32647.log.gz 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31048.log 2024-09-17T15:52:55.803 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30718.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29996.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41208.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38828.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27900.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33704.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31048.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/ceph-client.admin.38828.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32768.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29996.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33678.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32768.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38828.log.gz 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33704.log.gz/var/log/ceph/ceph-client.admin.33678.log: 2024-09-17T15:52:55.804 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39656.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32768.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38934.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33678.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38934.log: /var/log/ceph/ceph-client.admin.33613.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39656.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30926.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38934.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33082.log 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33613.log.gz 2024-09-17T15:52:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32439.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30926.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28629.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32439.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29181.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28629.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.33082.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39760.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32439.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40434.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28629.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29589.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29181.log.gz 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38124.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29589.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36854.log 2024-09-17T15:52:55.806 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39760.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37218.log 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29589.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36186.log 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36854.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38124.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28179.log 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36186.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36854.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39369.log 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37218.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38620.log 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36186.log.gz 2024-09-17T15:52:55.807 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34957.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28179.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39369.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/ceph-client.admin.34957.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39999.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38620.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29381.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34957.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30430.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36632.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39999.log.gz 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32621.log 2024-09-17T15:52:55.808 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0%/var/log/ceph/ceph-client.admin.36632.log: -- replaced with /var/log/ceph/ceph-client.admin.29381.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32569.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30430.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31485.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36632.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31485.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32621.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28316.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32569.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40181.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31485.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:gzip/var/log/ceph/ceph-client.admin.28316.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33451.log 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-09-17T15:52:55.809 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29237.log 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28316.log.gz 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40181.log.gz 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33451.log.gz 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29667.log 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37898.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29237.log.gz 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27825.log 2024-09-17T15:52:55.810 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27825.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29667.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39682.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27825.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41050.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36984.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38307.log 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4%/var/log/ceph/ceph-client.admin.41050.log: -- replaced with /var/log/ceph/ceph-client.admin.39682.log.gz 2024-09-17T15:52:55.811 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35197.log 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36984.log: /var/log/ceph/ceph-client.admin.38307.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41050.log.gz 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36984.log.gz 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38307.log.gz 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33793.log 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35790.log 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr: 90.6%/var/log/ceph/ceph-client.admin.32952.log: -- replaced with /var/log/ceph/ceph-client.admin.35197.log.gz 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36932.log 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.33793.log.gz 2024-09-17T15:52:55.812 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35790.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35790.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30404.log 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35998.log 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38333.log 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36932.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30404.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38333.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35998.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38333.log.gz 2024-09-17T15:52:55.813 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31308.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27800.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31204.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36420.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31308.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35041.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31204.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27800.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36420.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31204.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36420.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36342.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29407.log 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35041.log.gz 2024-09-17T15:52:55.814 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29407.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36342.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29407.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32820.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40590.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32465.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33004.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32465.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.32820.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34053.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40590.log.gz 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-09-17T15:52:55.815 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33004.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32465.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34053.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33004.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34053.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31021.log 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30144.log 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36238.log 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3%/var/log/ceph/ceph-client.admin.30144.log: -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31021.log.gz 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2024-09-17T15:52:55.816 INFO:teuthology.orchestra.run.smithi086.stderr:gzip 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30144.log.gz -5 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36238.log: /var/log/ceph/ceph-client.admin.36212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31256.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36238.log.gz 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35223.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40025.log: /var/log/ceph/ceph-client.admin.31256.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31256.log.gz 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34262.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39734.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34262.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35223.log.gz 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-09-17T15:52:55.817 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39734.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34262.log.gz 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39734.log.gz 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35511.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36316.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34236.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz/var/log/ceph/ceph-client.admin.36316.log: 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29077.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35511.log.gz 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29207.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34236.log: 90.9%/var/log/ceph/ceph-client.admin.29077.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.36316.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.28420.log 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29207.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34236.log.gz 2024-09-17T15:52:55.818 INFO:teuthology.orchestra.run.smithi086.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29077.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29207.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31386.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31360.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25232.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28420.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36524.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31386.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.25232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29025.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31360.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36524.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.25232.log.gz 2024-09-17T15:52:55.819 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29025.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36524.log.gz 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29025.log.gz 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41754.log 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32874.log 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz/var/log/ceph/ceph-client.admin.40868.log: 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28024.log 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41754.log.gz 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32874.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32874.log.gz 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30510.log 2024-09-17T15:52:55.820 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28707.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36368.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30510.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28024.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30510.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34079.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28707.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29329.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8%/var/log/ceph/ceph-client.admin.39082.log: -- replaced with /var/log/ceph/ceph-client.admin.36368.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34079.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38594.log 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34079.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-09-17T15:52:55.821 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29485.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29329.log.gz 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35407.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38463.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38594.log.gz 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29485.log: /var/log/ceph/ceph-client.admin.35407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41286.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38463.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29485.log.gz 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35407.log.gz 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39578.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38463.log.gz 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33581.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34131.log 2024-09-17T15:52:55.822 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41286.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33581.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39578.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34131.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33581.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28498.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28167.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28498.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 90.6% -- replaced with /var/log/ceph/ceph-client.admin.34131.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28498.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34340.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28023.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28167.log: /var/log/ceph/ceph-client.admin.34340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28176.log 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28023.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28167.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34340.log.gz 2024-09-17T15:52:55.823 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28023.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32517.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39108.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4%/var/log/ceph/ceph-client.admin.29892.log: -- replaced with /var/log/ceph/ceph-client.admin.28176.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32517.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32517.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33056.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32184.log 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39108.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41494.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33056.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2024-09-17T15:52:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36958.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36472.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32129.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32184.log.gz 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36472.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36958.log.gz 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36472.log.gz 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39134.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27952.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8%/var/log/ceph/ceph-client.admin.27952.log: -- replaced with /var/log/ceph/ceph-client.admin.32129.log.gz 2024-09-17T15:52:55.825 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39134.log.gz 2024-09-17T15:52:55.826 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27952.log.gz 2024-09-17T15:52:55.826 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38281.log 2024-09-17T15:52:55.826 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27926.log 2024-09-17T15:52:55.826 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37924.log: /var/log/ceph/ceph-client.admin.38281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35563.log 2024-09-17T15:52:55.827 INFO:teuthology.orchestra.run.smithi203.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.0.28086.log.gz 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27926.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38281.log.gz 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27926.log.gz 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31074.log 2024-09-17T15:52:55.910 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35563.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35093.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40077.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33767.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31074.log: /var/log/ceph/ceph-client.admin.40077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37530.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33767.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40077.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31074.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- 91.1% /var/log/ceph/ceph-client.admin.40642.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33767.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29615.log 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40642.log: /var/log/ceph/ceph-client.admin.31999.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-09-17T15:52:55.911 INFO:teuthology.orchestra.run.smithi086.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.37530.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28787.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31615.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28895.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29615.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31615.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28787.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31615.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28895.log: /var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41102.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz 90.1% 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28895.log.gz 2024-09-17T15:52:55.912 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37062.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37794.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39708.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37794.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37062.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37794.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32926.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38802.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32926.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39708.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38437.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35145.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38802.log.gz 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34720.log 2024-09-17T15:52:55.913 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30666.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1%/var/log/ceph/ceph-client.admin.34720.log: -- replaced with /var/log/ceph/ceph-client.admin.38437.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40946.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34720.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28869.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30666.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29788.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28869.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40946.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr: 91.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28009.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28869.log.gz 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2024-09-17T15:52:55.914 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30118.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29788.log.gz 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40512.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28009.log.gz 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31152.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30118.log.gz 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38750.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-09-17T15:52:55.915 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38750.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31152.log.gz 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41234.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38750.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30770.log 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34418.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31563.log 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33164.log 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41024.log 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31563.log: 89.7%/var/log/ceph/ceph-client.admin.33164.log: -- replaced with /var/log/ceph/ceph-client.admin.30770.log.gz 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31563.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33164.log.gz 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.916 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31334.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41728.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36498.log: /var/log/ceph/ceph-client.admin.31334.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.41024.log.gz 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35920.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41728.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.31334.log.gz 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41728.log.gz 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36131.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37504.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33030.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35920.log.gz 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34210.log 2024-09-17T15:52:55.917 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36131.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33030.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37504.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33030.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38255.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34210.log: /var/log/ceph/ceph-client.admin.38255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28472.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34210.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38255.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36906.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28472.log: /var/log/ceph/ceph-client.admin.41624.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28472.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38698.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36906.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36906.log.gz 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30170.log 2024-09-17T15:52:55.918 INFO:teuthology.orchestra.run.smithi086.stderr: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39786.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38698.log: /var/log/ceph/ceph-client.admin.30170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39786.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38698.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30170.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29840.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39786.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36802.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35642.log: /var/log/ceph/ceph-client.admin.29840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30744.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36802.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29840.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36802.log.gz 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28553.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37374.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32309.log 2024-09-17T15:52:55.919 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30744.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37374.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28553.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.37374.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35616.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35668.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32309.log: /var/log/ceph/ceph-client.admin.35616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37478.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35668.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32309.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.35616.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35668.log.gz 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33321.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37478.log: /var/log/ceph/ceph-client.admin.33321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30692.log 2024-09-17T15:52:55.920 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33425.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37478.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33321.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31641.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31178.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30692.log: /var/log/ceph/ceph-client.admin.31641.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30692.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31641.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33975.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31178.log: gzip/var/log/ceph/ceph-client.admin.33975.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38002.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31178.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33975.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-09-17T15:52:55.921 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34366.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39003.log: /var/log/ceph/ceph-client.admin.39947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28843.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34366.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 91.5% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34366.log.gz 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29814.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36104.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39869.log: 92.0% 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29814.log.gz 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28843.log.gz 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32077.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27850.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36104.log: /var/log/ceph/ceph-client.admin.32077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39291.log 2024-09-17T15:52:55.922 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.27850.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32077.log.gz 90.5% -- replaced with /var/log/ceph/ceph-client.admin.36104.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27850.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34157.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39291.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35303.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34157.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39422.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36050.log: /var/log/ceph/ceph-client.admin.36550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28524.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39422.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33399.log 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-09-17T15:52:55.923 INFO:teuthology.orchestra.run.smithi086.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39422.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28524.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28524.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30326.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30300.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33399.log: /var/log/ceph/ceph-client.admin.30326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38229.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30300.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30326.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41832.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30300.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34801.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36749.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34801.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38229.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41832.log.gz 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28681.log 2024-09-17T15:52:55.924 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25115.log 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.34801.log.gz 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36749.log: /var/log/ceph/ceph-client.admin.28681.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36749.log.gz 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28681.log.gz 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40894.log 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.25115.log: /var/log/ceph/ceph-client.admin.40894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36880.log 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35816.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40894.log.gz 2024-09-17T15:52:55.925 INFO:teuthology.orchestra.run.smithi086.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.25115.log.gz 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37296.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28554.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36880.log: /var/log/ceph/ceph-client.admin.37296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34288.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28554.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36880.log.gz 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr: 90.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.37296.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.28815.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28554.log.gz 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34288.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34288.log.gz 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-09-17T15:52:55.926 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28815.log: /var/log/ceph/ceph-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41676.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38854.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28815.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31817.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32361.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41676.log: /var/log/ceph/ceph-client.admin.31817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41676.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32361.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31817.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34827.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37635.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32361.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31667.log 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-09-17T15:52:55.927 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38028.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33871.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31667.log: /var/log/ceph/ceph-client.admin.38028.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34827.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31667.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38028.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30536.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38203.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33871.log: gzip -5/var/log/ceph/ceph-client.admin.30536.log: --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38203.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30536.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33871.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33190.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31843.log 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.38203.log.gz/var/log/ceph/ceph-client.admin.33503.log: 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-09-17T15:52:55.928 INFO:teuthology.orchestra.run.smithi086.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33190.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37088.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29459.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31843.log: /var/log/ceph/ceph-client.admin.37088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37664.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29459.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31843.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.37088.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38672.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29459.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35868.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37664.log: /var/log/ceph/ceph-client.admin.38672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36394.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35868.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37664.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38672.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35868.log.gz 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39812.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33819.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37270.log 2024-09-17T15:52:55.929 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36394.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33819.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33819.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40051.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41260.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37270.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40051.log: /var/log/ceph/ceph-client.admin.39161.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41260.log: 89.4% 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37270.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40051.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41260.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34105.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33845.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39161.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33845.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34105.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33845.log.gz 2024-09-17T15:52:55.930 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30048.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28947.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35946.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41076.log: /var/log/ceph/ceph-client.admin.28947.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30048.log.gz 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28947.log.gz 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35946.log: /var/log/ceph/ceph-client.admin.28973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39187.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34470.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35946.log.gz 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36157.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37582.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39187.log: /var/log/ceph/ceph-client.admin.36157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32676.log 2024-09-17T15:52:55.931 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37582.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36157.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.37582.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30222.log 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31694.log 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32676.log: /var/log/ceph/ceph-client.admin.30222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31694.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32676.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30222.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35171.log 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:gzip 91.1% -- replaced with /var/log/ceph/ceph-client.admin.31694.log.gz -5 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38646.log 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40382.log: /var/log/ceph/ceph-client.admin.35171.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35171.log.gz 2024-09-17T15:52:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38646.log: /var/log/ceph/ceph-client.admin.39843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40207.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38646.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38359.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41572.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35067.log: /var/log/ceph/ceph-client.admin.38359.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38359.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40155.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31126.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41572.log: /var/log/ceph/ceph-client.admin.40155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41546.log 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31126.log: 88.4% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40155.log.gz 2024-09-17T15:52:55.933 INFO:teuthology.orchestra.run.smithi086.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31126.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31100.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28022.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41546.log: /var/log/ceph/ceph-client.admin.31100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32413.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28022.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41546.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31100.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28022.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30848.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32103.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32413.log: /var/log/ceph/ceph-client.admin.30848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33295.log 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32103.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30848.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32413.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32103.log.gz 2024-09-17T15:52:55.934 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29433.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34183.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33295.log: gzip -5/var/log/ceph/ceph-client.admin.29433.log: --verbose -- /var/log/ceph/ceph-client.admin.38489.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34183.log: 90.1% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33295.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29433.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34183.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28999.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38411.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33897.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28999.log: /var/log/ceph/ceph-client.admin.38411.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38489.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28999.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38411.log.gz 2024-09-17T15:52:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36828.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.33897.log: /var/log/ceph/ceph-client.admin.41856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.36828.log: 88.1% 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33897.log.gz 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.36828.log.gz 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28016.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40486.log: /var/log/ceph/ceph-client.admin.34655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29563.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.28016.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28016.log.gz 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29155.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-09-17T15:52:55.936 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.29563.log: /var/log/ceph/ceph-client.admin.29155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35329.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40564.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29563.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: 90.9%gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21517.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29155.log.gz 89.4% 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35329.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35329.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38150.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31869.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.tmp-client.admin.21517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35972.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38150.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21517.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.31869.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34548.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38150.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31869.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35972.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35972.log.gz 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40816.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32595.log 2024-09-17T15:52:55.937 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34548.log: /var/log/ceph/ceph-client.admin.40816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35459.log 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.32595.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34548.log.gz 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40816.log.gz 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.32595.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30900.log 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.35459.log: /var/log/ceph/ceph-client.admin.33269.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35459.log.gz 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-09-17T15:52:55.938 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.30900.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30900.log.gz 2024-09-17T15:52:55.941 INFO:teuthology.orchestra.run.smithi203.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T15:52:56.162 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T15:52:56.460 INFO:teuthology.orchestra.run.smithi086.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T15:52:56.795 INFO:teuthology.orchestra.run.smithi086.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T15:52:59.260 INFO:teuthology.orchestra.run.smithi203.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T15:52:59.319 INFO:teuthology.orchestra.run.smithi086.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T15:52:59.621 INFO:teuthology.orchestra.run.smithi203.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T15:53:00.297 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T15:53:00.696 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T15:53:00.697 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-09-17T15:53:00.697 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m5.047s 2024-09-17T15:53:00.697 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m13.036s 2024-09-17T15:53:00.697 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.455s 2024-09-17T15:53:01.319 INFO:teuthology.orchestra.run.smithi086.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T15:53:01.320 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-09-17T15:53:01.320 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m5.709s 2024-09-17T15:53:01.320 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m15.734s 2024-09-17T15:53:01.320 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.731s 2024-09-17T15:53:01.321 INFO:tasks.ceph:Archiving logs... 2024-09-17T15:53:01.321 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi086/log 2024-09-17T15:53:01.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T15:53:03.999 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi203/log 2024-09-17T15:53:04.000 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T15:53:05.883 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T15:53:05.892 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T15:53:05.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T15:53:05.895 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T15:53:05.968 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T15:53:05.981 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T15:53:05.981 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:53:05.984 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================== 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout:*hv01.front.sepi 67.205.162.81 3 u 7 64 377 0.097 -6.158 1.192 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout:+hv02.front.sepi 63.231.80.2 3 u - 64 377 0.088 -6.884 1.212 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout:+hv03.front.sepi 74.6.168.72 3 u 66 64 377 0.077 -6.798 1.515 2024-09-17T15:53:05.994 INFO:teuthology.orchestra.run.smithi086.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================== 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout:*hv01.front.sepi 67.205.162.81 3 u 4 64 377 0.092 -7.094 1.313 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout:+hv02.front.sepi 63.231.80.2 3 u 11 64 377 0.108 -8.786 1.815 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout:+hv03.front.sepi 74.6.168.72 3 u 4 64 377 0.086 -7.993 1.886 2024-09-17T15:53:05.998 INFO:teuthology.orchestra.run.smithi203.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-17T15:53:05.999 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T15:53:06.018 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T15:53:06.019 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T15:53:06.068 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T15:53:06.123 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T15:53:06.166 INFO:teuthology.task.internal:Duration was 2194.813038 seconds 2024-09-17T15:53:06.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T15:53:06.219 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T15:53:06.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T15:53:06.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T15:53:06.260 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T15:53:06.260 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-09-17T15:53:06.260 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T15:53:06.309 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-09-17T15:53:06.310 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T15:53:06.320 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T15:53:06.320 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T15:53:06.353 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T15:53:06.395 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T15:53:06.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T15:53:06.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T15:53:06.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T15:53:06.502 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T15:53:06.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T15:53:06.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T15:53:06.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T15:53:06.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:53:06.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-09-17T15:53:06.572 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-09-17T15:53:06.812 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:53:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:06.818 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T15:53:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T15:53:06.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T15:53:06.869 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T15:53:06.870 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi086 2024-09-17T15:53:06.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T15:53:06.893 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7908954/remote/smithi203 2024-09-17T15:53:06.894 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T15:53:06.924 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T15:53:06.924 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:06.933 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T15:53:06.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T15:53:06.979 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T15:53:06.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T15:53:07.011 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T15:53:07.011 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T15:53:07.014 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T15:53:07.018 INFO:teuthology.orchestra.run.smithi086.stdout: 658465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 15:53 /home/ubuntu/cephtest 2024-09-17T15:53:07.020 INFO:teuthology.orchestra.run.smithi203.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 15:53 /home/ubuntu/cephtest 2024-09-17T15:53:07.021 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T15:53:07.061 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T15:53:07.163 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} duration: 2194.813037633896 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-17T15:53:07.164 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T15:53:07.238 INFO:teuthology.run:pass